* [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a document describing XRT Alveo drivers
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-19 22:26 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions Lizhi Hou
` (18 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Describe XRT driver architecture and provide basic overview of
Xilinx Alveo platform.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
Documentation/fpga/index.rst | 1 +
Documentation/fpga/xrt.rst | 842 +++++++++++++++++++++++++++++++++++
2 files changed, 843 insertions(+)
create mode 100644 Documentation/fpga/xrt.rst
diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst
index f80f95667ca2..30134357b70d 100644
--- a/Documentation/fpga/index.rst
+++ b/Documentation/fpga/index.rst
@@ -8,6 +8,7 @@ fpga
:maxdepth: 1
dfl
+ xrt
.. only:: subproject and html
diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
new file mode 100644
index 000000000000..9bc2d2785cb9
--- /dev/null
+++ b/Documentation/fpga/xrt.rst
@@ -0,0 +1,842 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================================
+XRTV2 Linux Kernel Driver Overview
+==================================
+
+Authors:
+
+* Sonal Santan <sonal.santan@xilinx.com>
+* Max Zhen <max.zhen@xilinx.com>
+* Lizhi Hou <lizhi.hou@xilinx.com>
+
+XRTV2 drivers are second generation `XRT <https://github.com/Xilinx/XRT>`_
+drivers which support `Alveo <https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
+PCIe platforms from Xilinx.
+
+XRTV2 drivers support *subsystem* style data driven platforms where driver's
+configuration and behavior is determined by meta data provided by the platform
+(in *device tree* format). Primary management physical function (MPF) driver
+is called **xmgmt**. Primary user physical function (UPF) driver is called
+**xuser** and is under development. xrt driver framework and HW subsystem
+drivers are packaged into a library module called **xrt-lib**, which is
+shared by **xmgmt** and **xuser** (under development). The xrt driver framework
+implements a pseudo-bus which is used to discover HW subsystems and facilitate
+inter HW subsystem interaction.
+
+Driver Modules
+==============
+
+xrt-lib.ko
+----------
+
+Repository of all subsystem drivers and pure software modules that can potentially
+be shared between xmgmt and xuser. All these drivers are structured as Linux
+*platform driver* and are instantiated by xmgmt (or xuser under development) based
+on meta data associated with hardware. The metadata is in the form of device tree
+as mentioned before. Each platform driver statically defines a subsystem node
+array by using node name or a string in its ``compatible`` property. And this
+array is eventually translated to IOMEM resources of the platform device.
+
+The xrt-lib core infrastructure provides hooks to platform drivers for device node
+management, user file operations and ioctl callbacks. The core also provides pseudo-bus
+functionality for platform driver registration, discovery and inter platform driver
+ioctl calls.
+
+.. note::
+ See code in ``include/xleaf.h``
+
+
+xmgmt.ko
+--------
+
+The xmgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
+PCIE device. It consists of one *root* driver, one or more *group* drivers
+and one or more *xleaf* drivers. The root and MPF specific xleaf drivers are
+in xmgmt.ko. The group driver and other xleaf drivers are in xrt-lib.ko.
+
+The instantiation of specific group driver or xleaf driver is completely data
+driven based on meta data (mostly in device tree format) found through VSEC
+capability and inside firmware files, such as platform xsabin or user xclbin file.
+The root driver manages life cycle of multiple group drivers, which, in turn,
+manages multiple xleaf drivers. This allows a single set of driver code to support
+all kinds of subsystems exposed by different shells. The difference among all
+these subsystems will be handled in xleaf drivers with root and group drivers
+being part of the infrastructure and provide common services for all leaves
+found on all platforms.
+
+The driver object model looks like the following::
+
+ +-----------+
+ | xroot |
+ +-----+-----+
+ |
+ +-----------+-----------+
+ | |
+ v v
+ +-----------+ +-----------+
+ | group | ... | group |
+ +-----+-----+ +------+----+
+ | |
+ | |
+ +-----+----+ +-----+----+
+ | | | |
+ v v v v
+ +-------+ +-------+ +-------+ +-------+
+ | xleaf |..| xleaf | | xleaf |..| xleaf |
+ +-------+ +-------+ +-------+ +-------+
+
+As an example for Xilinx Alveo U50 before user xclbin download, the tree
+looks like the following::
+
+ +-----------+
+ | xmgmt |
+ +-----+-----+
+ |
+ +-------------------------+--------------------+
+ | | |
+ v v v
+ +--------+ +--------+ +--------+
+ | group0 | | group1 | | group2 |
+ +----+---+ +----+---+ +---+----+
+ | | |
+ | | |
+ +-----+-----+ +----+-----+---+ +-----+-----+----+--------+
+ | | | | | | | | |
+ v v | v v | v v |
+ +------------+ +------+ | +------+ +------+ | +------+ +-----------+ |
+ | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | |
+ +------------+ +------+ | +------+ +------+ | +------+ +-----------+ |
+ | +---------+ | +------+ +-----------+ |
+ +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+
+ +---------+ | +------+ +-----------+
+ | +-------+
+ +->| CALIB |
+ +-------+
+
+After an xclbin is download, group3 will be added and the tree looks like the
+following::
+
+ +-----------+
+ | xmgmt |
+ +-----+-----+
+ |
+ +-------------------------+--------------------+-----------------+
+ | | | |
+ v v v |
+ +--------+ +--------+ +--------+ |
+ | group0 | | group1 | | group2 | |
+ +----+---+ +----+---+ +---+----+ |
+ | | | |
+ | | | |
+ +-----+-----+ +-----+-----+---+ +-----+-----+----+--------+ |
+ | | | | | | | | | |
+ v v | v v | v v | |
+ +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
+ | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | | |
+ +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
+ | +---------+ | +------+ +-----------+ | |
+ +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+ |
+ +---------+ | +------+ +-----------+ |
+ | +-------+ |
+ +->| CALIB | |
+ +-------+ |
+ +---+----+ |
+ | group3 |<--------------------------------------------+
+ +--------+
+ |
+ |
+ +-------+--------+---+--+--------+------+-------+
+ | | | | | | |
+ v | v | v | v
+ +--------+ | +--------+ | +--------+ | +-----+
+ | CLOCK0 | | | CLOCK1 | | | CLOCK2 | | | UCS |
+ +--------+ v +--------+ v +--------+ v +-----+
+ +-------------+ +-------------+ +-------------+
+ | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
+ +-------------+ +-------------+ +-------------+
+
+
+xmgmt-root
+^^^^^^^^^^
+
+The xmgmt-root driver is a PCIe device driver attached to MPF. It's part of the
+infrastructure of the MPF driver and resides in xmgmt.ko. This driver
+
+* manages one or more group drivers
+* provides access to functionalities that requires pci_dev, such as PCIE config
+ space access, to other xleaf drivers through root calls
+* together with group driver, facilities event callbacks for other xleaf drivers
+* together with group driver, facilities inter-leaf driver calls for other xleaf
+ drivers
+
+When root driver starts, it will explicitly create an initial group instance,
+which contains xleaf drivers that will trigger the creation of other group
+instances. The root driver will wait for all group and leaves to be created
+before it returns from it's probe routine and claim success of the
+initialization of the entire xmgmt driver.
+
+.. note::
+ See code in ``lib/xroot.c`` and ``mgmt/root.c``
+
+
+group
+^^^^^
+
+The group driver is a platform device driver whose life cycle is managed by
+root and does not have real IO mem or IRQ resources. It's part of the
+infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
+
+* manages one or more xleaf drivers so that multiple leaves can be managed as a
+ group
+* provides access to root from leaves, so that root calls, event notifications
+ and inter-leaf calls can happen
+
+In xmgmt, an initial group driver instance will be created by root, which
+contains leaves that will trigger group instances to be created to manage
+groups of leaves found on different partitions on hardware, such as VSEC, Shell,
+and User.
+
+Every *fpga_region* has a group object associated with it. The group is
+created when xclbin image is loaded on the fpga_region. The existing group
+is destroyed when a new xclbin image is loaded. The fpga_region persists
+across xclbin downloads.
+
+.. note::
+ See code in ``lib/group.c``
+
+
+xleaf
+^^^^^
+
+The xleaf driver is a platform device driver whose life cycle is managed by
+a group driver and may or may not have real IO mem or IRQ resources. They
+are the real meat of xmgmt and contains platform specific code to Shell and
+User found on a MPF.
+
+A xleaf driver may not have real hardware resources when it merely acts as a
+driver that manages certain in-memory states for xmgmt. These in-memory states
+could be shared by multiple other leaves.
+
+Leaf drivers assigned to specific hardware resources drive specific subsystem in
+the device. To manipulate the subsystem or carry out a task, a xleaf driver may
+ask help from root via root calls and/or from other leaves via inter-leaf calls.
+
+A xleaf can also broadcast events through infrastructure code for other leaves
+to process. It can also receive event notification from infrastructure about
+certain events, such as post-creation or pre-exit of a particular xleaf.
+
+.. note::
+ See code in ``lib/xleaf/*.c``
+
+
+FPGA Manager Interaction
+========================
+
+fpga_manager
+------------
+
+An instance of fpga_manager is created by xmgmt_main and is used for xclbin
+image download. fpga_manager requires the full xclbin image before it can
+start programming the FPGA configuration engine via ICAP platform driver.
+
+fpga_region
+-----------
+
+For every interface exposed by currently loaded xclbin/xsabin in the *parent*
+fpga_region a new instance of fpga_region is created like a *child* region.
+The device tree of the *parent* fpga_region defines the
+resources for a new instance of fpga_bridge which isolates the parent from
+child fpga_region. This new instance of fpga_bridge will be used when a
+xclbin image is loaded on the child fpga_region. After the xclbin image is
+downloaded to the fpga_region, an instance of group is created for the
+fpga_region using the device tree obtained as part of xclbin. If this device
+tree defines any child interfaces then it can trigger the creation of
+fpga_bridge and fpga_region for the next region in the chain.
+
+fpga_bridge
+-----------
+
+Like fpga_region, matching fpga_bridge is also created by walking the device
+tree of the parent group.
+
+Driver Interfaces
+=================
+
+xmgmt Driver Ioctls
+-------------------
+
+Ioctls exposed by xmgmt driver to user space are enumerated in the following
+table:
+
+== ===================== ============================ ==========================
+# Functionality ioctl request code data format
+== ===================== ============================ ==========================
+1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
+== ===================== ============================ ==========================
+
+User xclbin can be downloaded by using xbmgmt tool from XRT open source suite. See
+example usage below::
+
+ xbmgmt partition --program --path /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xclbin --force
+
+xmgmt Driver Sysfs
+------------------
+
+xmgmt driver exposes a rich set of sysfs interfaces. Subsystem platform
+drivers export sysfs node for every platform instance.
+
+Every partition also exports its UUIDs. See below for examples::
+
+ /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
+ /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
+
+
+hwmon
+-----
+
+xmgmt driver exposes standard hwmon interface to report voltage, current,
+temperature, power, etc. These can easily be viewed using *sensors* command
+line utility.
+
+Alveo Platform Overview
+=======================
+
+Alveo platforms are architected as two physical FPGA partitions: *Shell* and
+*User*. The Shell provides basic infrastructure for the Alveo platform like
+PCIe connectivity, board management, Dynamic Function Exchange (DFX), sensors,
+clocking, reset, and security. User partition contains user compiled FPGA
+binary which is loaded by a process called DFX also known as partial
+reconfiguration.
+
+Physical partitions require strict HW compatibility with each other for DFX to
+work properly. Every physical partition has two interface UUIDs: *parent* UUID
+and *child* UUID. For simple single stage platforms, Shell → User forms parent
+child relationship. For complex two stage platforms, Base → Shell → User forms
+the parent child relationship chain.
+
+.. note::
+ Partition compatibility matching is key design component of Alveo platforms
+ and XRT. Partitions have child and parent relationship. A loaded partition
+ exposes child partition UUID to advertise its compatibility requirement for
+ child partition. When loading a child partition the xmgmt management driver
+ matches parent UUID of the child partition against child UUID exported by
+ the parent. Parent and child partition UUIDs are stored in the *xclbin*
+ (for user) or *xsabin* (for base and shell). Except for root UUID, VSEC,
+ hardware itself does not know about UUIDs. UUIDs are stored in xsabin and
+ xclbin.
+
+
+The physical partitions and their loading is illustrated below::
+
+ SHELL USER
+ +-----------+ +-------------------+
+ | | | |
+ | VSEC UUID | CHILD PARENT | LOGIC UUID |
+ | o------->|<--------o |
+ | | UUID UUID | |
+ +-----+-----+ +--------+----------+
+ | |
+ . .
+ | |
+ +---+---+ +------+--------+
+ | POR | | USER COMPILED |
+ | FLASH | | XCLBIN |
+ +-------+ +---------------+
+
+
+Loading Sequence
+----------------
+
+The Shell partition is loaded from flash at system boot time. It establishes the
+PCIe link and exposes two physical functions to the BIOS. After OS boot, xmgmt
+driver attaches to PCIe physical function 0 exposed by the Shell and then looks
+for VSEC in PCIe extended configuration space. Using VSEC it determines the logic
+UUID of Shell and uses the UUID to load matching *xsabin* file from Linux firmware
+directory. The xsabin file contains metadata to discover peripherals that are part
+of Shell and firmware(s) for any embedded soft processors in Shell.
+
+The Shell exports child interface UUID which is used for compatibility check when
+loading user compiled xclbin over the User partition as part of DFX. When a user
+requests loading of a specific xclbin the xmgmt management driver reads the parent
+interface UUID specified in the xclbin and matches it with child interface UUID
+exported by Shell to determine if xclbin is compatible with the Shell. If match
+fails loading of xclbin is denied.
+
+xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command. When loading
+xclbin, xmgmt driver performs the following *logical* operations:
+
+1. Copy xclbin from user to kernel memory
+2. Sanity check the xclbin contents
+3. Isolate the User partition
+4. Download the bitstream using the FPGA config engine (ICAP)
+5. De-isolate the User partition
+6. Program the clocks (ClockWiz) driving the User partition
+7. Wait for memory controller (MIG) calibration
+8. Return the loading status back to the caller
+
+`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
+provides more detailed information on platform loading.
+
+
+xsabin
+------
+
+Each Alveo platform comes packaged with its own xsabin. The xsabin is trusted
+component of the platform. For format details refer to :ref:`xsabin_xclbin_container_format`
+below. xsabin contains basic information like UUIDs, platform name and metadata in the
+form of device tree. See :ref:`device_tree_usage` below for details and example.
+
+xclbin
+------
+
+xclbin is compiled by end user using
+`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_
+tool set from Xilinx. The xclbin contains sections describing user compiled
+acceleration engines/kernels, memory subsystems, clocking information etc. It also
+contains bitstream for the user partition, UUIDs, platform name, etc. xclbin uses
+the same container format as xsabin which is described below.
+
+
+.. _xsabin_xclbin_container_format:
+
+xsabin/xclbin Container Format
+------------------------------
+
+xclbin/xsabin is ELF-like binary container format. It is structured as series of
+sections. There is a file header followed by several section headers which is
+followed by sections. A section header points to an actual section. There is an
+optional signature at the end. The format is defined by header file ``xclbin.h``.
+The following figure illustrates a typical xclbin::
+
+
+ +---------------------+
+ | |
+ | HEADER |
+ +---------------------+
+ | SECTION HEADER |
+ | |
+ +---------------------+
+ | ... |
+ | |
+ +---------------------+
+ | SECTION HEADER |
+ | |
+ +---------------------+
+ | SECTION |
+ | |
+ +---------------------+
+ | ... |
+ | |
+ +---------------------+
+ | SECTION |
+ | |
+ +---------------------+
+ | SIGNATURE |
+ | (OPTIONAL) |
+ +---------------------+
+
+
+xclbin/xsabin files can be packaged, un-packaged and inspected using XRT utility
+called **xclbinutil**. xclbinutil is part of XRT open source software stack. The
+source code for xclbinutil can be found at
+https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
+
+For example to enumerate the contents of a xclbin/xsabin use the *--info* switch
+as shown below::
+
+
+ xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
+ xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
+
+
+.. _device_tree_usage:
+
+Device Tree Usage
+-----------------
+
+As mentioned previously xsabin stores metadata which advertise HW subsystems present
+in a partition. The metadata is stored in device tree format with well defined schema.
+XRT management driver uses this information to bind *platform drivers* to the subsystem
+instantiations. The platform drivers are found in **xrt-lib.ko** kernel module defined
+later.
+
+Logic UUID
+^^^^^^^^^^
+A partition is identified uniquely through ``logic_uuid`` property::
+
+ /dts-v1/;
+ / {
+ logic_uuid = "0123456789abcdef0123456789abcdef";
+ ...
+ }
+
+Schema Version
+^^^^^^^^^^^^^^
+Schema version is defined through ``schema_version`` node. And it contains ``major``
+and ``minor`` properties as below::
+
+ /dts-v1/;
+ / {
+ schema_version {
+ major = <0x01>;
+ minor = <0x00>;
+ };
+ ...
+ }
+
+Partition UUIDs
+^^^^^^^^^^^^^^^
+As said earlier, each partition may have parent and child UUIDs. These UUIDs are
+defined by ``interfaces`` node and ``interface_uuid`` property::
+
+ /dts-v1/;
+ / {
+ interfaces {
+ @0 {
+ interface_uuid = "0123456789abcdef0123456789abcdef";
+ };
+ @1 {
+ interface_uuid = "fedcba9876543210fedcba9876543210";
+ };
+ ...
+ };
+ ...
+ }
+
+
+Subsystem Instantiations
+^^^^^^^^^^^^^^^^^^^^^^^^
+Subsystem instantiations are captured as children of ``addressable_endpoints``
+node::
+
+ /dts-v1/;
+ / {
+ addressable_endpoints {
+ abc {
+ ...
+ };
+ def {
+ ...
+ };
+ ...
+ }
+ }
+
+Subnode 'abc' and 'def' are the name of subsystem nodes
+
+Subsystem Node
+^^^^^^^^^^^^^^
+Each subsystem node and its properties define a hardware instance::
+
+
+ addressable_endpoints {
+ abc {
+ reg = <0xa 0xb>
+ pcie_physical_function = <0x0>;
+ pcie_bar_mapping = <0x2>;
+ compatible = "abc def";
+ firmware {
+ firmware_product_name = "abc"
+ firmware_branch_name = "def"
+ firmware_version_major = <1>
+ firmware_version_minor = <2>
+ };
+ }
+ ...
+ }
+
+:reg:
+ Property defines address range. '<0xa 0xb>' is BAR offset and length pair, both
+ are 64-bit integer.
+:pcie_physical_function:
+ Property specifies which PCIe physical function the subsystem node resides.
+:pcie_bar_mapping:
+ Property specifies which PCIe BAR the subsystem node resides. '<0x2>' is BAR
+ index and it is 0 if this property is not defined.
+:compatible:
+ Property is a list of strings. The first string in the list specifies the exact
+ subsystem node. The following strings represent other devices that the device
+ is compatible with.
+:firmware:
+ Subnode defines the firmware required by this subsystem node.
+
+Alveo U50 Platform Example
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+::
+
+ /dts-v1/;
+
+ /{
+ logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
+
+ schema_version {
+ major = <0x01>;
+ minor = <0x00>;
+ };
+
+ interfaces {
+
+ @0 {
+ interface_uuid = "862c7020a250293e32036f19956669e5";
+ };
+ };
+
+ addressable_endpoints {
+
+ ep_blp_rom_00 {
+ reg = <0x00 0x1f04000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+ };
+
+ ep_card_flash_program_00 {
+ reg = <0x00 0x1f06000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
+ interrupts = <0x03 0x03>;
+ };
+
+ ep_cmc_firmware_mem_00 {
+ reg = <0x00 0x1e20000 0x00 0x20000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+ firmware {
+ firmware_product_name = "cmc";
+ firmware_branch_name = "u50";
+ firmware_version_major = <0x01>;
+ firmware_version_minor = <0x00>;
+ };
+ };
+
+ ep_cmc_intc_00 {
+ reg = <0x00 0x1e03000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+ interrupts = <0x04 0x04>;
+ };
+
+ ep_cmc_mutex_00 {
+ reg = <0x00 0x1e02000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_cmc_regmap_00 {
+ reg = <0x00 0x1e08000 0x00 0x2000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+ firmware {
+ firmware_product_name = "sc-fw";
+ firmware_branch_name = "u50";
+ firmware_version_major = <0x05>;
+ };
+ };
+
+ ep_cmc_reset_00 {
+ reg = <0x00 0x1e01000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_ddr_mem_calib_00 {
+ reg = <0x00 0x63000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_debug_bscan_mgmt_00 {
+ reg = <0x00 0x1e90000 0x00 0x10000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
+ };
+
+ ep_ert_base_address_00 {
+ reg = <0x00 0x21000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_ert_command_queue_mgmt_00 {
+ reg = <0x00 0x40000 0x00 0x10000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
+ };
+
+ ep_ert_command_queue_user_00 {
+ reg = <0x00 0x40000 0x00 0x10000>;
+ pcie_physical_function = <0x01>;
+ compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
+ };
+
+ ep_ert_firmware_mem_00 {
+ reg = <0x00 0x30000 0x00 0x8000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+ firmware {
+ firmware_product_name = "ert";
+ firmware_branch_name = "v20";
+ firmware_version_major = <0x01>;
+ };
+ };
+
+ ep_ert_intc_00 {
+ reg = <0x00 0x23000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+ interrupts = <0x05 0x05>;
+ };
+
+ ep_ert_reset_00 {
+ reg = <0x00 0x22000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_ert_sched_00 {
+ reg = <0x00 0x50000 0x00 0x1000>;
+ pcie_physical_function = <0x01>;
+ compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
+ interrupts = <0x09 0x0c>;
+ };
+
+ ep_fpga_configuration_00 {
+ reg = <0x00 0x1e88000 0x00 0x8000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
+ interrupts = <0x02 0x02>;
+ };
+
+ ep_icap_reset_00 {
+ reg = <0x00 0x1f07000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_msix_00 {
+ reg = <0x00 0x00 0x00 0x20000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
+ pcie_bar_mapping = <0x02>;
+ };
+
+ ep_pcie_link_mon_00 {
+ reg = <0x00 0x1f05000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_pr_isolate_plp_00 {
+ reg = <0x00 0x1f01000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_pr_isolate_ulp_00 {
+ reg = <0x00 0x1000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+ };
+
+ ep_uuid_rom_00 {
+ reg = <0x00 0x64000 0x00 0x1000>;
+ pcie_physical_function = <0x00>;
+ compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+ };
+
+ ep_xdma_00 {
+ reg = <0x00 0x00 0x00 0x10000>;
+ pcie_physical_function = <0x01>;
+ compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
+ pcie_bar_mapping = <0x02>;
+ };
+ };
+
+ }
+
+
+
+Deployment Models
+=================
+
+Baremetal
+---------
+
+In bare-metal deployments both MPF and UPF are visible and accessible. xmgmt
+driver binds to MPF. xmgmt driver operations are privileged and available to
+system administrator. The full stack is illustrated below::
+
+ HOST
+
+ [XMGMT] [XUSER]
+ | |
+ | |
+ +-----+ +-----+
+ | MPF | | UPF |
+ | | | |
+ | PF0 | | PF1 |
+ +--+--+ +--+--+
+ ......... ^................. ^..........
+ | |
+ | PCIe DEVICE |
+ | |
+ +--+------------------+--+
+ | SHELL |
+ | |
+ +------------------------+
+ | USER |
+ | |
+ | |
+ | |
+ | |
+ +------------------------+
+
+
+
+Virtualized
+-----------
+
+In virtualized deployments privileged MPF is assigned to host but unprivileged
+UPF is assigned to guest VM via PCIe pass-through. xmgmt driver in host binds
+to MPF. xmgmt driver operations are privileged and only accessible by hosting
+service provider. The full stack is illustrated below::
+
+
+ .............
+ HOST . VM .
+ . .
+ [XMGMT] . [XUSER] .
+ | . | .
+ | . | .
+ +-----+ . +-----+ .
+ | MPF | . | UPF | .
+ | | . | | .
+ | PF0 | . | PF1 | .
+ +--+--+ . +--+--+ .
+ ......... ^................. ^..........
+ | |
+ | PCIe DEVICE |
+ | |
+ +--+------------------+--+
+ | SHELL |
+ | |
+ +------------------------+
+ | USER |
+ | |
+ | |
+ | |
+ | |
+ +------------------------+
+
+
+
+
+
+Platform Security Considerations
+================================
+
+`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
+discusses the deployment options and security implications in great detail.
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a document describing XRT Alveo drivers
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a document describing XRT Alveo drivers Lizhi Hou
@ 2021-02-19 22:26 ` Tom Rix
2021-03-01 6:48 ` Sonal Santan
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-19 22:26 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
From the documentation, there are a couple of big questions and a bunch of word smithing.
pseudo-bus : do we need a bus ?
xrt-lib real platform devices that aren't fpga, do they need to move to another subsystem ?
Overall looks good, love the ascii art!
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Describe XRT driver architecture and provide basic overview of
> Xilinx Alveo platform.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> Documentation/fpga/index.rst | 1 +
> Documentation/fpga/xrt.rst | 842 +++++++++++++++++++++++++++++++++++
> 2 files changed, 843 insertions(+)
> create mode 100644 Documentation/fpga/xrt.rst
>
> diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst
> index f80f95667ca2..30134357b70d 100644
> --- a/Documentation/fpga/index.rst
> +++ b/Documentation/fpga/index.rst
> @@ -8,6 +8,7 @@ fpga
> :maxdepth: 1
>
> dfl
> + xrt
>
> .. only:: subproject and html
>
> diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
> new file mode 100644
> index 000000000000..9bc2d2785cb9
> --- /dev/null
> +++ b/Documentation/fpga/xrt.rst
> @@ -0,0 +1,842 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================================
> +XRTV2 Linux Kernel Driver Overview
> +==================================
> +
> +Authors:
> +
> +* Sonal Santan <sonal.santan@xilinx.com>
> +* Max Zhen <max.zhen@xilinx.com>
> +* Lizhi Hou <lizhi.hou@xilinx.com>
> +
> +XRTV2 drivers are second generation `XRT <https://github.com/Xilinx/XRT>`_
> +drivers which support `Alveo <https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
> +PCIe platforms from Xilinx.
> +
> +XRTV2 drivers support *subsystem* style data driven platforms where driver's
where the driver's
> +configuration and behavior is determined by meta data provided by the platform
> +(in *device tree* format). Primary management physical function (MPF) driver
> +is called **xmgmt**. Primary user physical function (UPF) driver is called
> +**xuser** and is under development. xrt driver framework and HW subsystem
> +drivers are packaged into a library module called **xrt-lib**, which is
> +shared by **xmgmt** and **xuser** (under development). The xrt driver framework
xuser still under development ?
> +implements a pseudo-bus which is used to discover HW subsystems and facilitate
A pseudo-bus.
It would be good if this was close to what was done for dfl here
https://lore.kernel.org/linux-fpga/1605159759-3439-1-git-send-email-yilun.xu@intel.com/
> +inter HW subsystem interaction.
> +
> +Driver Modules
> +==============
> +
> +xrt-lib.ko
> +----------
> +
> +Repository of all subsystem drivers and pure software modules that can potentially
subsystem drivers
drivers in fpga/ should be for managing just the fpganess of the fpga.
soft devices ex/ a soft tty should go to their respective subsystem location
Are there any in this patchset you think might move ?
Maybe we can defer reviewing those now.
> +be shared between xmgmt and xuser. All these drivers are structured as Linux
> +*platform driver* and are instantiated by xmgmt (or xuser under development) based
> +on meta data associated with hardware. The metadata is in the form of device tree
with the hardware
form of a device tree
> +as mentioned before. Each platform driver statically defines a subsystem node
> +array by using node name or a string in its ``compatible`` property. And this
> +array is eventually translated to IOMEM resources of the platform device.
> +
> +The xrt-lib core infrastructure provides hooks to platform drivers for device node
> +management, user file operations and ioctl callbacks. The core also provides pseudo-bus
> +functionality for platform driver registration, discovery and inter platform driver
> +ioctl calls.
core infrastructure.
The interfaces to the infrastructure are not in include/linux/fpga/
Maybe this needs to change.
> +
> +.. note::
> + See code in ``include/xleaf.h``
> +
> +
> +xmgmt.ko
> +--------
> +
> +The xmgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
> +PCIE device. It consists of one *root* driver, one or more *group* drivers
> +and one or more *xleaf* drivers. The root and MPF specific xleaf drivers are
> +in xmgmt.ko. The group driver and other xleaf drivers are in xrt-lib.ko.
I am not sure if *.ko is correct, these will also be intree.
> +
> +The instantiation of specific group driver or xleaf driver is completely data
of a specific
> +driven based on meta data (mostly in device tree format) found through VSEC
mostly ? what is the deviation from device tree ?
> +capability and inside firmware files, such as platform xsabin or user xclbin file.
> +The root driver manages life cycle of multiple group drivers, which, in turn,
the life cycle
> +manages multiple xleaf drivers. This allows a single set of driver code to support
set of drivers
drop 'code'
> +all kinds of subsystems exposed by different shells. The difference among all
> +these subsystems will be handled in xleaf drivers with root and group drivers
> +being part of the infrastructure and provide common services for all leaves
> +found on all platforms.
> +
> +The driver object model looks like the following::
> +
> + +-----------+
> + | xroot |
> + +-----+-----+
> + |
> + +-----------+-----------+
> + | |
> + v v
> + +-----------+ +-----------+
> + | group | ... | group |
> + +-----+-----+ +------+----+
> + | |
> + | |
> + +-----+----+ +-----+----+
> + | | | |
> + v v v v
> + +-------+ +-------+ +-------+ +-------+
> + | xleaf |..| xleaf | | xleaf |..| xleaf |
> + +-------+ +-------+ +-------+ +-------+
> +
> +As an example for Xilinx Alveo U50 before user xclbin download, the tree
> +looks like the following::
> +
> + +-----------+
> + | xmgmt |
> + +-----+-----+
> + |
> + +-------------------------+--------------------+
> + | | |
> + v v v
> + +--------+ +--------+ +--------+
> + | group0 | | group1 | | group2 |
> + +----+---+ +----+---+ +---+----+
> + | | |
> + | | |
> + +-----+-----+ +----+-----+---+ +-----+-----+----+--------+
> + | | | | | | | | |
> + v v | v v | v v |
> + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ |
> + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | |
> + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ |
> + | +---------+ | +------+ +-----------+ |
> + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+
> + +---------+ | +------+ +-----------+
> + | +-------+
> + +->| CALIB |
> + +-------+
> +
Nice ascii art!
> +After an xclbin is download, group3 will be added and the tree looks like the
> +following::
> +
> + +-----------+
> + | xmgmt |
> + +-----+-----+
> + |
> + +-------------------------+--------------------+-----------------+
> + | | | |
> + v v v |
> + +--------+ +--------+ +--------+ |
> + | group0 | | group1 | | group2 | |
> + +----+---+ +----+---+ +---+----+ |
> + | | | |
> + | | | |
> + +-----+-----+ +-----+-----+---+ +-----+-----+----+--------+ |
> + | | | | | | | | | |
> + v v | v v | v v | |
> + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
> + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | | |
> + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
> + | +---------+ | +------+ +-----------+ | |
> + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+ |
> + +---------+ | +------+ +-----------+ |
> + | +-------+ |
> + +->| CALIB | |
> + +-------+ |
> + +---+----+ |
> + | group3 |<--------------------------------------------+
> + +--------+
> + |
> + |
> + +-------+--------+---+--+--------+------+-------+
> + | | | | | | |
> + v | v | v | v
> + +--------+ | +--------+ | +--------+ | +-----+
> + | CLOCK0 | | | CLOCK1 | | | CLOCK2 | | | UCS |
> + +--------+ v +--------+ v +--------+ v +-----+
> + +-------------+ +-------------+ +-------------+
> + | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
> + +-------------+ +-------------+ +-------------+
> +
> +
> +xmgmt-root
> +^^^^^^^^^^
> +
> +The xmgmt-root driver is a PCIe device driver attached to MPF. It's part of the
> +infrastructure of the MPF driver and resides in xmgmt.ko. This driver
> +
> +* manages one or more group drivers
> +* provides access to functionalities that requires pci_dev, such as PCIE config
> + space access, to other xleaf drivers through root calls
> +* together with group driver, facilities event callbacks for other xleaf drivers
> +* together with group driver, facilities inter-leaf driver calls for other xleaf
Maybe drop 'together with group driver'
> + drivers
> +
> +When root driver starts, it will explicitly create an initial group instance,
> +which contains xleaf drivers that will trigger the creation of other group
> +instances. The root driver will wait for all group and leaves to be created
> +before it returns from it's probe routine and claim success of the
> +initialization of the entire xmgmt driver.
What happens if there a failure in one leaf ? Does the whole board go down ?
> +
> +.. note::
> + See code in ``lib/xroot.c`` and ``mgmt/root.c``
> +
> +
> +group
> +^^^^^
> +
> +The group driver is a platform device driver whose life cycle is managed by
Maybe call this a 'pseudo device'
> +root and does not have real IO mem or IRQ resources. It's part of the
> +infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
> +
> +* manages one or more xleaf drivers so that multiple leaves can be managed as a
> + group
can drop 'so that multiple leaves can be managed as a group' to me, this is the same as 'one or more'
> +* provides access to root from leaves, so that root calls, event notifications
> + and inter-leaf calls can happen
> +
> +In xmgmt, an initial group driver instance will be created by root, which
by the root
> +contains leaves that will trigger group instances to be created to manage
> +groups of leaves found on different partitions on hardware, such as VSEC, Shell,
> +and User.
> +
> +Every *fpga_region* has a group object associated with it. The group is
> +created when xclbin image is loaded on the fpga_region. The existing group
> +is destroyed when a new xclbin image is loaded. The fpga_region persists
> +across xclbin downloads.
The connection of a 'group' node to a fpga region region is fairly important, maybe move this section earlier. 'group' as an fpganess thing would be kept in fpga/ subsystem.
> +
> +.. note::
> + See code in ``lib/group.c``
> +
> +
> +xleaf
> +^^^^^
> +
> +The xleaf driver is a platform device driver whose life cycle is managed by
> +a group driver and may or may not have real IO mem or IRQ resources. They
> +are the real meat of xmgmt and contains platform specific code to Shell and
> +User found on a MPF.
> +
Maybe a split is pseudo device leaves, those without real IO mem, stay in fpga/ others go ?
> +A xleaf driver may not have real hardware resources when it merely acts as a
> +driver that manages certain in-memory states for xmgmt. These in-memory states
> +could be shared by multiple other leaves.
> +
This implies locking and some message passing.
> +Leaf drivers assigned to specific hardware resources drive specific subsystem in
drive a specific
> +the device. To manipulate the subsystem or carry out a task, a xleaf driver may
> +ask help from root via root calls and/or from other leaves via inter-leaf calls.
> +
> +A xleaf can also broadcast events through infrastructure code for other leaves
> +to process. It can also receive event notification from infrastructure about
> +certain events, such as post-creation or pre-exit of a particular xleaf.
I would like to see some examples of how the inter node communications work.
> +
> +.. note::
> + See code in ``lib/xleaf/*.c``
> +
> +
> +FPGA Manager Interaction
> +========================
> +
> +fpga_manager
> +------------
> +
> +An instance of fpga_manager is created by xmgmt_main and is used for xclbin
for the xclbin
> +image download. fpga_manager requires the full xclbin image before it can
> +start programming the FPGA configuration engine via ICAP platform driver.
via the ICAP
what is ICAP ?
> +
> +fpga_region
> +-----------
> +
> +For every interface exposed by currently loaded xclbin/xsabin in the *parent*
by the currently
> +fpga_region a new instance of fpga_region is created like a *child* region.
fpga_region,
> +The device tree of the *parent* fpga_region defines the
> +resources for a new instance of fpga_bridge which isolates the parent from
and isolates
> +child fpga_region. This new instance of fpga_bridge will be used when a
> +xclbin image is loaded on the child fpga_region. After the xclbin image is
> +downloaded to the fpga_region, an instance of group is created for the
> +fpga_region using the device tree obtained as part of xclbin. If this device
of the xclbin
> +tree defines any child interfaces then it can trigger the creation of
interfaces, then
> +fpga_bridge and fpga_region for the next region in the chain.
a fpga_bridge and a fpga_region
> +
> +fpga_bridge
> +-----------
> +
> +Like fpga_region, matching fpga_bridge is also created by walking the device
Like the fpga_region, a matchin
> +tree of the parent group.
> +
> +Driver Interfaces
> +=================
> +
> +xmgmt Driver Ioctls
> +-------------------
> +
> +Ioctls exposed by xmgmt driver to user space are enumerated in the following
> +table:
> +
> +== ===================== ============================ ==========================
> +# Functionality ioctl request code data format
> +== ===================== ============================ ==========================
> +1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
> +== ===================== ============================ ==========================
This data format is described below, maybe swap this section with that so
folks will know what xmgmnt_ioc_bitstream_axlf is before this section.
> +
> +User xclbin can be downloaded by using xbmgmt tool from XRT open source suite. See
A user xclbin
using the xbmgmt
from the XRT
> +example usage below::
> +
> + xbmgmt partition --program --path /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xclbin --force
> +
> +xmgmt Driver Sysfs
> +------------------
> +
> +xmgmt driver exposes a rich set of sysfs interfaces. Subsystem platform
> +drivers export sysfs node for every platform instance.
> +
> +Every partition also exports its UUIDs. See below for examples::
> +
> + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
> + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
> +
> +
> +hwmon
> +-----
> +
> +xmgmt driver exposes standard hwmon interface to report voltage, current,
> +temperature, power, etc. These can easily be viewed using *sensors* command
> +line utility.
> +
> +Alveo Platform Overview
> +=======================
> +
> +Alveo platforms are architected as two physical FPGA partitions: *Shell* and
> +*User*. The Shell provides basic infrastructure for the Alveo platform like
> +PCIe connectivity, board management, Dynamic Function Exchange (DFX), sensors,
> +clocking, reset, and security. User partition contains user compiled FPGA
the user compiled
> +binary which is loaded by a process called DFX also known as partial
> +reconfiguration.
> +
> +Physical partitions require strict HW compatibility with each other for DFX to
> +work properly.
swap order
For DFX to work properly physical partitions ..
> Every physical partition has two interface UUIDs: *parent* UUID
> +and *child* UUID. For simple single stage platforms, Shell → User forms parent
> +child relationship. For complex two stage platforms, Base → Shell → User forms
> +the parent child relationship chain.
this bit is confusing. is this related to uuid?
> +
> +.. note::
> + Partition compatibility matching is key design component of Alveo platforms
> + and XRT. Partitions have child and parent relationship. A loaded partition
have a child
> + exposes child partition UUID to advertise its compatibility requirement for
the child's
can drop 'for child partition'
> + child partition. When loading a child partition the xmgmt management driver
When loading a child partition,
> + matches parent UUID of the child partition against child UUID exported by
> + the parent. Parent and child partition UUIDs are stored in the *xclbin*
> + (for user) or *xsabin* (for base and shell).
this is confusing, is this part of the file image format ?
Maybe save/move till the image layout.
> Except for root UUID, VSEC,
> + hardware itself does not know about UUIDs. UUIDs are stored in xsabin and
> + xclbin.
This is confusing too, not sure how to untangle.
> +
> +
> +The physical partitions and their loading is illustrated below::
> +
> + SHELL USER
> + +-----------+ +-------------------+
> + | | | |
> + | VSEC UUID | CHILD PARENT | LOGIC UUID |
> + | o------->|<--------o |
> + | | UUID UUID | |
> + +-----+-----+ +--------+----------+
> + | |
> + . .
> + | |
> + +---+---+ +------+--------+
> + | POR | | USER COMPILED |
> + | FLASH | | XCLBIN |
> + +-------+ +---------------+
> +
> +
> +Loading Sequence
> +----------------
> +
> +The Shell partition is loaded from flash at system boot time. It establishes the
> +PCIe link and exposes two physical functions to the BIOS. After OS boot, xmgmt
the OS boots, the xmgmt
> +driver attaches to PCIe physical function 0 exposed by the Shell and then looks
> +for VSEC in PCIe extended configuration space. Using VSEC it determines the logic
the PCIe
The driver uses VSEC to determine the UUID of Shell. The UUID is also used to load a matching ...
> +UUID of Shell and uses the UUID to load matching *xsabin* file from Linux firmware
> +directory. The xsabin file contains metadata to discover peripherals that are part
> +of Shell and firmware(s) for any embedded soft processors in Shell.
the firmware needed for any ...
> +
> +The Shell exports child interface UUID which is used for compatibility check when
export a child
for a compatibility check
> +loading user compiled xclbin over the User partition as part of DFX. When a user
> +requests loading of a specific xclbin the xmgmt management driver reads the parent
xclbin, the
> +interface UUID specified in the xclbin and matches it with child interface UUID
> +exported by Shell to determine if xclbin is compatible with the Shell. If match
> +fails loading of xclbin is denied.
> +
> +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command. When loading
> +xclbin, xmgmt driver performs the following *logical* operations:
> +
> +1. Copy xclbin from user to kernel memory
> +2. Sanity check the xclbin contents
> +3. Isolate the User partition
> +4. Download the bitstream using the FPGA config engine (ICAP)
> +5. De-isolate the User partition
> +6. Program the clocks (ClockWiz) driving the User partition
maybe drop '(ClockWiz)'
> +7. Wait for memory controller (MIG) calibration
for the
> +8. Return the loading status back to the caller
> +
> +`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
> +provides more detailed information on platform loading.
> +
the link works.
> +
> +xsabin
> +------
> +
> +Each Alveo platform comes packaged with its own xsabin. The xsabin is trusted
is a trusted
> +component of the platform. For format details refer to :ref:`xsabin_xclbin_container_format`
> +below. xsabin contains basic information like UUIDs, platform name and metadata in the
> +form of device tree. See :ref:`device_tree_usage` below for details and example.
of a device
> +
> +xclbin
> +------
> +
> +xclbin is compiled by end user using
> +`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_
this link works, seems reasonable landing
> +tool set from Xilinx. The xclbin contains sections describing user compiled
> +acceleration engines/kernels, memory subsystems, clocking information etc. It also
> +contains bitstream for the user partition, UUIDs, platform name, etc. xclbin uses
bitstreams
> +the same container format as xsabin which is described below.
> +
> +
> +.. _xsabin_xclbin_container_format:
> +
> +xsabin/xclbin Container Format
> +------------------------------
> +
> +xclbin/xsabin is ELF-like binary container format. It is structured as series of
> +sections. There is a file header followed by several section headers which is
> +followed by sections. A section header points to an actual section. There is an
> +optional signature at the end. The format is defined by header file ``xclbin.h``.
> +The following figure illustrates a typical xclbin::
> +
> +
> + +---------------------+
> + | |
> + | HEADER |
> + +---------------------+
> + | SECTION HEADER |
> + | |
> + +---------------------+
> + | ... |
> + | |
> + +---------------------+
> + | SECTION HEADER |
> + | |
> + +---------------------+
> + | SECTION |
> + | |
> + +---------------------+
> + | ... |
> + | |
> + +---------------------+
> + | SECTION |
> + | |
> + +---------------------+
> + | SIGNATURE |
> + | (OPTIONAL) |
> + +---------------------+
> +
> +
> +xclbin/xsabin files can be packaged, un-packaged and inspected using XRT utility
> +called **xclbinutil**. xclbinutil is part of XRT open source software stack. The
> +source code for xclbinutil can be found at
> +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
> +
Works, but maybe the location of a manpage or doc would be better.
> +For example to enumerate the contents of a xclbin/xsabin use the *--info* switch
> +as shown below::
> +
> +
> + xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
> + xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
> +
> +
> +.. _device_tree_usage:
> +
> +Device Tree Usage
> +-----------------
> +
> +As mentioned previously xsabin stores metadata which advertise HW subsystems present
> +in a partition. The metadata is stored in device tree format with well defined schema.
> +XRT management driver uses this information to bind *platform drivers* to the subsystem
> +instantiations. The platform drivers are found in **xrt-lib.ko** kernel module defined
> +later.
> +
> +Logic UUID
> +^^^^^^^^^^
> +A partition is identified uniquely through ``logic_uuid`` property::
> +
> + /dts-v1/;
> + / {
> + logic_uuid = "0123456789abcdef0123456789abcdef";
> + ...
> + }
> +
> +Schema Version
> +^^^^^^^^^^^^^^
> +Schema version is defined through ``schema_version`` node. And it contains ``major``
> +and ``minor`` properties as below::
> +
> + /dts-v1/;
> + / {
> + schema_version {
> + major = <0x01>;
> + minor = <0x00>;
> + };
> + ...
> + }
> +
> +Partition UUIDs
> +^^^^^^^^^^^^^^^
> +As said earlier, each partition may have parent and child UUIDs. These UUIDs are
> +defined by ``interfaces`` node and ``interface_uuid`` property::
> +
> + /dts-v1/;
> + / {
> + interfaces {
> + @0 {
> + interface_uuid = "0123456789abcdef0123456789abcdef";
> + };
> + @1 {
> + interface_uuid = "fedcba9876543210fedcba9876543210";
> + };
> + ...
> + };
> + ...
> + }
> +
> +
> +Subsystem Instantiations
> +^^^^^^^^^^^^^^^^^^^^^^^^
> +Subsystem instantiations are captured as children of ``addressable_endpoints``
> +node::
> +
> + /dts-v1/;
> + / {
> + addressable_endpoints {
> + abc {
> + ...
> + };
> + def {
> + ...
> + };
> + ...
> + }
> + }
> +
> +Subnode 'abc' and 'def' are the name of subsystem nodes
> +
> +Subsystem Node
> +^^^^^^^^^^^^^^
> +Each subsystem node and its properties define a hardware instance::
> +
> +
> + addressable_endpoints {
> + abc {
> + reg = <0xa 0xb>
> + pcie_physical_function = <0x0>;
> + pcie_bar_mapping = <0x2>;
> + compatible = "abc def";
> + firmware {
> + firmware_product_name = "abc"
> + firmware_branch_name = "def"
> + firmware_version_major = <1>
> + firmware_version_minor = <2>
> + };
> + }
> + ...
> + }
> +
> +:reg:
> + Property defines address range. '<0xa 0xb>' is BAR offset and length pair, both
> + are 64-bit integer.
> +:pcie_physical_function:
> + Property specifies which PCIe physical function the subsystem node resides.
> +:pcie_bar_mapping:
> + Property specifies which PCIe BAR the subsystem node resides. '<0x2>' is BAR
> + index and it is 0 if this property is not defined.
> +:compatible:
> + Property is a list of strings. The first string in the list specifies the exact
> + subsystem node. The following strings represent other devices that the device
> + is compatible with.
> +:firmware:
> + Subnode defines the firmware required by this subsystem node.
> +
> +Alveo U50 Platform Example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^
> +::
> +
> + /dts-v1/;
> +
> + /{
> + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> +
> + schema_version {
> + major = <0x01>;
> + minor = <0x00>;
> + };
> +
> + interfaces {
> +
> + @0 {
> + interface_uuid = "862c7020a250293e32036f19956669e5";
> + };
> + };
> +
> + addressable_endpoints {
> +
> + ep_blp_rom_00 {
> + reg = <0x00 0x1f04000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> + };
> +
> + ep_card_flash_program_00 {
> + reg = <0x00 0x1f06000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
> + interrupts = <0x03 0x03>;
> + };
> +
> + ep_cmc_firmware_mem_00 {
> + reg = <0x00 0x1e20000 0x00 0x20000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> + firmware {
> + firmware_product_name = "cmc";
> + firmware_branch_name = "u50";
> + firmware_version_major = <0x01>;
> + firmware_version_minor = <0x00>;
> + };
> + };
> +
> + ep_cmc_intc_00 {
> + reg = <0x00 0x1e03000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> + interrupts = <0x04 0x04>;
> + };
> +
> + ep_cmc_mutex_00 {
> + reg = <0x00 0x1e02000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_cmc_regmap_00 {
> + reg = <0x00 0x1e08000 0x00 0x2000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> + firmware {
> + firmware_product_name = "sc-fw";
> + firmware_branch_name = "u50";
> + firmware_version_major = <0x05>;
> + };
> + };
> +
> + ep_cmc_reset_00 {
> + reg = <0x00 0x1e01000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_ddr_mem_calib_00 {
> + reg = <0x00 0x63000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_debug_bscan_mgmt_00 {
> + reg = <0x00 0x1e90000 0x00 0x10000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
> + };
> +
> + ep_ert_base_address_00 {
> + reg = <0x00 0x21000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_ert_command_queue_mgmt_00 {
> + reg = <0x00 0x40000 0x00 0x10000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> + };
> +
> + ep_ert_command_queue_user_00 {
> + reg = <0x00 0x40000 0x00 0x10000>;
> + pcie_physical_function = <0x01>;
> + compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
> + };
> +
> + ep_ert_firmware_mem_00 {
> + reg = <0x00 0x30000 0x00 0x8000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> +
> + firmware {
> + firmware_product_name = "ert";
> + firmware_branch_name = "v20";
> + firmware_version_major = <0x01>;
> + };
> + };
> +
> + ep_ert_intc_00 {
> + reg = <0x00 0x23000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> + interrupts = <0x05 0x05>;
> + };
> +
> + ep_ert_reset_00 {
> + reg = <0x00 0x22000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_ert_sched_00 {
> + reg = <0x00 0x50000 0x00 0x1000>;
> + pcie_physical_function = <0x01>;
> + compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> + interrupts = <0x09 0x0c>;
> + };
> +
> + ep_fpga_configuration_00 {
> + reg = <0x00 0x1e88000 0x00 0x8000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
> + interrupts = <0x02 0x02>;
> + };
> +
> + ep_icap_reset_00 {
> + reg = <0x00 0x1f07000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_msix_00 {
> + reg = <0x00 0x00 0x00 0x20000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> + pcie_bar_mapping = <0x02>;
> + };
> +
> + ep_pcie_link_mon_00 {
> + reg = <0x00 0x1f05000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_pr_isolate_plp_00 {
> + reg = <0x00 0x1f01000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_pr_isolate_ulp_00 {
> + reg = <0x00 0x1000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> + };
> +
> + ep_uuid_rom_00 {
> + reg = <0x00 0x64000 0x00 0x1000>;
> + pcie_physical_function = <0x00>;
> + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> + };
> +
> + ep_xdma_00 {
> + reg = <0x00 0x00 0x00 0x10000>;
> + pcie_physical_function = <0x01>;
> + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> + pcie_bar_mapping = <0x02>;
> + };
> + };
> +
> + }
> +
> +
> +
> +Deployment Models
> +=================
> +
> +Baremetal
> +---------
> +
> +In bare-metal deployments both MPF and UPF are visible and accessible. xmgmt
In bare-meta deployments,
> +driver binds to MPF. xmgmt driver operations are privileged and available to
> +system administrator. The full stack is illustrated below::
> +
> + HOST
> +
> + [XMGMT] [XUSER]
> + | |
> + | |
> + +-----+ +-----+
> + | MPF | | UPF |
> + | | | |
> + | PF0 | | PF1 |
> + +--+--+ +--+--+
> + ......... ^................. ^..........
> + | |
> + | PCIe DEVICE |
> + | |
> + +--+------------------+--+
> + | SHELL |
> + | |
> + +------------------------+
> + | USER |
> + | |
> + | |
> + | |
> + | |
> + +------------------------+
> +
> +
> +
> +Virtualized
> +-----------
> +
> +In virtualized deployments privileged MPF is assigned to host but unprivileged
In virtualized deployments, the
> +UPF is assigned to guest VM via PCIe pass-through. xmgmt driver in host binds
in the host
> +to MPF. xmgmt driver operations are privileged and only accessible by hosting
to the MPF
> +service provider. The full stack is illustrated below::
> +
> +
> + .............
> + HOST . VM .
> + . .
> + [XMGMT] . [XUSER] .
> + | . | .
> + | . | .
> + +-----+ . +-----+ .
> + | MPF | . | UPF | .
> + | | . | | .
> + | PF0 | . | PF1 | .
> + +--+--+ . +--+--+ .
> + ......... ^................. ^..........
> + | |
> + | PCIe DEVICE |
> + | |
> + +--+------------------+--+
> + | SHELL |
> + | |
> + +------------------------+
> + | USER |
> + | |
> + | |
> + | |
> + | |
> + +------------------------+
> +
> +
> +
> +
> +
> +Platform Security Considerations
> +================================
> +
> +`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
> +discusses the deployment options and security implications in great detail.
This link works and looks great.
Tom
^ permalink raw reply [flat|nested] 87+ messages in thread* RE: [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a document describing XRT Alveo drivers
2021-02-19 22:26 ` Tom Rix
@ 2021-03-01 6:48 ` Sonal Santan
2021-03-06 17:19 ` Moritz Fischer
0 siblings, 1 reply; 87+ messages in thread
From: Sonal Santan @ 2021-03-01 6:48 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, Max Zhen, Michal Simek,
Stefano Stabellini, devicetree, mdf, robh, Max Zhen
Hello Tom,
> -----Original Message-----
> From: Tom Rix <trix@redhat.com>
> Sent: Friday, February 19, 2021 2:26 PM
> To: Lizhi Hou <lizhih@xilinx.com>; linux-kernel@vger.kernel.org
> Cc: Lizhi Hou <lizhih@xilinx.com>; linux-fpga@vger.kernel.org; Max Zhen
> <maxz@xilinx.com>; Sonal Santan <sonals@xilinx.com>; Michal Simek
> <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
> devicetree@vger.kernel.org; mdf@kernel.org; robh@kernel.org; Max Zhen
> <maxz@xilinx.com>
> Subject: Re: [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a
> document describing XRT Alveo drivers
>
> From the documentation, there are a couple of big questions and a bunch of
> word smithing.
>
> pseudo-bus : do we need a bus ?
We are looking for guidance here.
>
> xrt-lib real platform devices that aren't fpga, do they need to move to another
> subsystem ?
>
Drivers for the IPs that show up in the Alveo shell are not generic enough. They
fit into the framework that XRT uses. Is the idea that that they can be used in a
different context?
> Overall looks good, love the ascii art!
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
> > Describe XRT driver architecture and provide basic overview of Xilinx
> > Alveo platform.
> >
> > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> > Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> > ---
> > Documentation/fpga/index.rst | 1 +
> > Documentation/fpga/xrt.rst | 842
> +++++++++++++++++++++++++++++++++++
> > 2 files changed, 843 insertions(+)
> > create mode 100644 Documentation/fpga/xrt.rst
> >
> > diff --git a/Documentation/fpga/index.rst
> > b/Documentation/fpga/index.rst index f80f95667ca2..30134357b70d 100644
> > --- a/Documentation/fpga/index.rst
> > +++ b/Documentation/fpga/index.rst
> > @@ -8,6 +8,7 @@ fpga
> > :maxdepth: 1
> >
> > dfl
> > + xrt
> >
> > .. only:: subproject and html
> >
> > diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
> > new file mode 100644 index 000000000000..9bc2d2785cb9
> > --- /dev/null
> > +++ b/Documentation/fpga/xrt.rst
> > @@ -0,0 +1,842 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +==================================
> > +XRTV2 Linux Kernel Driver Overview
> > +==================================
> > +
> > +Authors:
> > +
> > +* Sonal Santan <sonal.santan@xilinx.com>
> > +* Max Zhen <max.zhen@xilinx.com>
> > +* Lizhi Hou <lizhi.hou@xilinx.com>
> > +
> > +XRTV2 drivers are second generation `XRT
> > +<https://github.com/Xilinx/XRT>`_ drivers which support `Alveo
> > +<https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
> > +PCIe platforms from Xilinx.
> > +
> > +XRTV2 drivers support *subsystem* style data driven platforms where
> > +driver's
> where the driver's
> > +configuration and behavior is determined by meta data provided by the
> > +platform (in *device tree* format). Primary management physical
> > +function (MPF) driver is called **xmgmt**. Primary user physical
> > +function (UPF) driver is called
> > +**xuser** and is under development. xrt driver framework and HW
> > +subsystem drivers are packaged into a library module called
> > +**xrt-lib**, which is shared by **xmgmt** and **xuser** (under
> > +development). The xrt driver framework
> xuser still under development ?
> > +implements a pseudo-bus which is used to discover HW subsystems and
> > +facilitate
>
> A pseudo-bus.
>
> It would be good if this was close to what was done for dfl here
>
> https://lore.kernel.org/linux-fpga/1605159759-3439-1-git-send-email-
> yilun.xu@intel.com/
>
I am wondering if we can phase in the migration to formal bus architecture
based on struct bus_type as a follow on set of patches?
> > +inter HW subsystem interaction.
> > +
> > +Driver Modules
> > +==============
> > +
> > +xrt-lib.ko
> > +----------
> > +
> > +Repository of all subsystem drivers and pure software modules that
> > +can potentially
>
> subsystem drivers
>
> drivers in fpga/ should be for managing just the fpganess of the fpga.
>
> soft devices ex/ a soft tty should go to their respective subsystem location
>
> Are there any in this patchset you think might move ?
We have already shrunk the patch to only include FPGA centric pieces
necessary to get the bitstream download implemented. Should we explore
the question of subsystem drivers when we add support for more features of
the Alveo shell?
>
> Maybe we can defer reviewing those now.
>
> > +be shared between xmgmt and xuser. All these drivers are structured
> > +as Linux *platform driver* and are instantiated by xmgmt (or xuser
> > +under development) based on meta data associated with hardware. The
> > +metadata is in the form of device tree
>
> with the hardware
>
> form of a device tree
>
Will change
> > +as mentioned before. Each platform driver statically defines a
> > +subsystem node array by using node name or a string in its
> > +``compatible`` property. And this array is eventually translated to IOMEM
> resources of the platform device.
> > +
> > +The xrt-lib core infrastructure provides hooks to platform drivers
> > +for device node management, user file operations and ioctl callbacks.
> > +The core also provides pseudo-bus functionality for platform driver
> > +registration, discovery and inter platform driver ioctl calls.
>
> core infrastructure.
>
Will update.
> The interfaces to the infrastructure are not in include/linux/fpga/
>
> Maybe this needs to change.
>
Were you thinking of moving XRT infrastructure header files from
drivers/fpga/xrt/include to include/linux/fpga?
> > +
> > +.. note::
> > + See code in ``include/xleaf.h``
> > +
> > +
> > +xmgmt.ko
> > +--------
> > +
> > +The xmgmt driver is a PCIe device driver driving MPF found on
> > +Xilinx's Alveo PCIE device. It consists of one *root* driver, one or
> > +more *group* drivers and one or more *xleaf* drivers. The root and
> > +MPF specific xleaf drivers are in xmgmt.ko. The group driver and other xleaf
> drivers are in xrt-lib.ko.
> I am not sure if *.ko is correct, these will also be intree.
> > +
> > +The instantiation of specific group driver or xleaf driver is
> > +completely data
> of a specific
> > +driven based on meta data (mostly in device tree format) found
> > +through VSEC
> mostly ? what is the deviation from device tree ?
> > +capability and inside firmware files, such as platform xsabin or user xclbin
> file.
> > +The root driver manages life cycle of multiple group drivers, which,
> > +in turn,
> the life cycle
> > +manages multiple xleaf drivers. This allows a single set of driver
> > +code to support
>
> set of drivers
>
> drop 'code'
>
Will update
> > +all kinds of subsystems exposed by different shells. The difference
> > +among all these subsystems will be handled in xleaf drivers with root
> > +and group drivers being part of the infrastructure and provide common
> > +services for all leaves found on all platforms.
> > +
> > +The driver object model looks like the following::
> > +
> > + +-----------+
> > + | xroot |
> > + +-----+-----+
> > + |
> > + +-----------+-----------+
> > + | |
> > + v v
> > + +-----------+ +-----------+
> > + | group | ... | group |
> > + +-----+-----+ +------+----+
> > + | |
> > + | |
> > + +-----+----+ +-----+----+
> > + | | | |
> > + v v v v
> > + +-------+ +-------+ +-------+ +-------+
> > + | xleaf |..| xleaf | | xleaf |..| xleaf |
> > + +-------+ +-------+ +-------+ +-------+
> > +
> > +As an example for Xilinx Alveo U50 before user xclbin download, the
> > +tree looks like the following::
> > +
> > + +-----------+
> > + | xmgmt |
> > + +-----+-----+
> > + |
> > + +-------------------------+--------------------+
> > + | | |
> > + v v v
> > + +--------+ +--------+ +--------+
> > + | group0 | | group1 | | group2 |
> > + +----+---+ +----+---+ +---+----+
> > + | | |
> > + | | |
> > + +-----+-----+ +----+-----+---+ +-----+-----+----+--------+
> > + | | | | | | | | |
> > + v v | v v | v v |
> > + +------------+ +------+ | +------+ +------+ | +------+
> > + +------------+ +-----------+ |
> > + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | |
> > + | AXI-GATE0 | |
> > + +------------+ +------+ | +------+ +------+ | +------+
> > + +------------+ +-----------+ |
> > + | +---------+ | +------+ +-----------+ |
> > + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+
> > + +---------+ | +------+ +-----------+
> > + | +-------+
> > + +->| CALIB |
> > + +-------+
> > +
> Nice ascii art!
> > +After an xclbin is download, group3 will be added and the tree looks
> > +like the
> > +following::
> > +
> > + +-----------+
> > + | xmgmt |
> > + +-----+-----+
> > + |
> > + +-------------------------+--------------------+-----------------+
> > + | | | |
> > + v v v |
> > + +--------+ +--------+ +--------+ |
> > + | group0 | | group1 | | group2 | |
> > + +----+---+ +----+---+ +---+----+ |
> > + | | | |
> > + | | | |
> > + +-----+-----+ +-----+-----+---+ +-----+-----+----+--------+ |
> > + | | | | | | | | | |
> > + v v | v v | v v | |
> > + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
> > + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | |
> |
> > + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
> > + | +---------+ | +------+ +-----------+ | |
> > + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+ |
> > + +---------+ | +------+ +-----------+ |
> > + | +-------+ |
> > + +->| CALIB | |
> > + +-------+ |
> > + +---+----+ |
> > + | group3 |<--------------------------------------------+
> > + +--------+
> > + |
> > + |
> > + +-------+--------+---+--+--------+------+-------+
> > + | | | | | | |
> > + v | v | v | v
> > + +--------+ | +--------+ | +--------+ | +-----+
> > + | CLOCK0 | | | CLOCK1 | | | CLOCK2 | | | UCS |
> > + +--------+ v +--------+ v +--------+ v +-----+
> > + +-------------+ +-------------+ +-------------+
> > + | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
> > + +-------------+ +-------------+ +-------------+
> > +
> > +
> > +xmgmt-root
> > +^^^^^^^^^^
> > +
> > +The xmgmt-root driver is a PCIe device driver attached to MPF. It's
> > +part of the infrastructure of the MPF driver and resides in xmgmt.ko.
> > +This driver
> > +
> > +* manages one or more group drivers
> > +* provides access to functionalities that requires pci_dev, such as
> > +PCIE config
> > + space access, to other xleaf drivers through root calls
> > +* together with group driver, facilities event callbacks for other
> > +xleaf drivers
> > +* together with group driver, facilities inter-leaf driver calls for
> > +other xleaf
> Maybe drop 'together with group driver'
Will update
> > + drivers
> > +
> > +When root driver starts, it will explicitly create an initial group
> > +instance, which contains xleaf drivers that will trigger the creation
> > +of other group instances. The root driver will wait for all group and
> > +leaves to be created before it returns from it's probe routine and
> > +claim success of the initialization of the entire xmgmt driver.
> What happens if there a failure in one leaf ? Does the whole board go down ?
> > +
> > +.. note::
> > + See code in ``lib/xroot.c`` and ``mgmt/root.c``
> > +
> > +
> > +group
> > +^^^^^
> > +
> > +The group driver is a platform device driver whose life cycle is
> > +managed by
> Maybe call this a 'pseudo device'
Will update
> > +root and does not have real IO mem or IRQ resources. It's part of the
> > +infrastructure of the MPF driver and resides in xrt-lib.ko. This
> > +driver
> > +
> > +* manages one or more xleaf drivers so that multiple leaves can be
> > +managed as a
> > + group
> can drop 'so that multiple leaves can be managed as a group' to me, this is the
> same as 'one or more'
Will do
> > +* provides access to root from leaves, so that root calls, event
> > +notifications
> > + and inter-leaf calls can happen
> > +
> > +In xmgmt, an initial group driver instance will be created by root,
> > +which
> by the root
> > +contains leaves that will trigger group instances to be created to
> > +manage groups of leaves found on different partitions on hardware,
> > +such as VSEC, Shell, and User.
> > +
> > +Every *fpga_region* has a group object associated with it. The group
> > +is created when xclbin image is loaded on the fpga_region. The
> > +existing group is destroyed when a new xclbin image is loaded. The
> > +fpga_region persists across xclbin downloads.
> The connection of a 'group' node to a fpga region region is fairly important,
> maybe move this section earlier. 'group' as an fpganess thing would be kept in
> fpga/ subsystem.
Will update
> > +
> > +.. note::
> > + See code in ``lib/group.c``
> > +
> > +
> > +xleaf
> > +^^^^^
> > +
> > +The xleaf driver is a platform device driver whose life cycle is
> > +managed by a group driver and may or may not have real IO mem or IRQ
> > +resources. They are the real meat of xmgmt and contains platform
> > +specific code to Shell and User found on a MPF.
> > +
>
> Maybe a split is pseudo device leaves, those without real IO mem, stay in
> fpga/ others go ?
>
This goes back to the earlier question of what minimal set of platform drivers
should stay in fpga subsystem. There are some like bridge or configuration
engine (also called icap) which have their own IO mem but do not have a life
outside of fpga subsystem.
> > +A xleaf driver may not have real hardware resources when it merely
> > +acts as a driver that manages certain in-memory states for xmgmt.
> > +These in-memory states could be shared by multiple other leaves.
> > +
> This implies locking and some message passing.
> > +Leaf drivers assigned to specific hardware resources drive specific
> > +subsystem in
> drive a specific
> > +the device. To manipulate the subsystem or carry out a task, a xleaf
> > +driver may ask help from root via root calls and/or from other leaves via
> inter-leaf calls.
> > +
> > +A xleaf can also broadcast events through infrastructure code for
> > +other leaves to process. It can also receive event notification from
> > +infrastructure about certain events, such as post-creation or pre-exit of a
> particular xleaf.
> I would like to see some examples of how the inter node communications work.
Would update to show an example.
> > +
> > +.. note::
> > + See code in ``lib/xleaf/*.c``
> > +
> > +
> > +FPGA Manager Interaction
> > +========================
> > +
> > +fpga_manager
> > +------------
> > +
> > +An instance of fpga_manager is created by xmgmt_main and is used for
> > +xclbin
> for the xclbin
> > +image download. fpga_manager requires the full xclbin image before it
> > +can start programming the FPGA configuration engine via ICAP platform
> driver.
>
> via the ICAP
>
> what is ICAP ?
Will update. ICAP stands for Internal Configuration Access Port used for configuring
the fpga.
>
> > +
> > +fpga_region
> > +-----------
> > +
> > +For every interface exposed by currently loaded xclbin/xsabin in the
> > +*parent*
> by the currently
> > +fpga_region a new instance of fpga_region is created like a *child* region.
> fpga_region,
> > +The device tree of the *parent* fpga_region defines the resources for
> > +a new instance of fpga_bridge which isolates the parent from
> and isolates
> > +child fpga_region. This new instance of fpga_bridge will be used when
> > +a xclbin image is loaded on the child fpga_region. After the xclbin
> > +image is downloaded to the fpga_region, an instance of group is
> > +created for the fpga_region using the device tree obtained as part of
> > +xclbin. If this device
> of the xclbin
> > +tree defines any child interfaces then it can trigger the creation of
> interfaces, then
> > +fpga_bridge and fpga_region for the next region in the chain.
> a fpga_bridge and a fpga_region
> > +
> > +fpga_bridge
> > +-----------
> > +
> > +Like fpga_region, matching fpga_bridge is also created by walking the
> > +device
> Like the fpga_region, a matchin
> > +tree of the parent group.
> > +
> > +Driver Interfaces
> > +=================
> > +
> > +xmgmt Driver Ioctls
> > +-------------------
> > +
> > +Ioctls exposed by xmgmt driver to user space are enumerated in the
> > +following
> > +table:
> > +
> > +== ===================== ============================
> ==========================
> > +# Functionality ioctl request code data format
> > +== ===================== ============================
> ==========================
> > +1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF
> xmgmt_ioc_bitstream_axlf
> > +== ===================== ============================
> > +==========================
>
> This data format is described below, maybe swap this section with that so
>
> folks will know what xmgmnt_ioc_bitstream_axlf is before this section.
>
Will update.
> > +
> > +User xclbin can be downloaded by using xbmgmt tool from XRT open
> > +source suite. See
>
> A user xclbin
>
> using the xbmgmt
>
> from the XRT
>
Will update
> > +example usage below::
> > +
> > + xbmgmt partition --program --path
> > + /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xc
> > + lbin --force
> > +
> > +xmgmt Driver Sysfs
> > +------------------
> > +
> > +xmgmt driver exposes a rich set of sysfs interfaces. Subsystem
> > +platform drivers export sysfs node for every platform instance.
> > +
> > +Every partition also exports its UUIDs. See below for examples::
> > +
> > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
> > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
> > +
> > +
> > +hwmon
> > +-----
> > +
> > +xmgmt driver exposes standard hwmon interface to report voltage,
> > +current, temperature, power, etc. These can easily be viewed using
> > +*sensors* command line utility.
> > +
> > +Alveo Platform Overview
> > +=======================
> > +
> > +Alveo platforms are architected as two physical FPGA partitions:
> > +*Shell* and *User*. The Shell provides basic infrastructure for the
> > +Alveo platform like PCIe connectivity, board management, Dynamic
> > +Function Exchange (DFX), sensors, clocking, reset, and security. User
> > +partition contains user compiled FPGA
> the user compiled
> > +binary which is loaded by a process called DFX also known as partial
> > +reconfiguration.
> > +
> > +Physical partitions require strict HW compatibility with each other
> > +for DFX to work properly.
>
> swap order
>
> For DFX to work properly physical partitions ..
>
>
Will update
> > Every physical partition has two interface UUIDs: *parent* UUID
> > +and *child* UUID. For simple single stage platforms, Shell → User
> > +forms parent child relationship. For complex two stage platforms,
> > +Base → Shell → User forms the parent child relationship chain.
> this bit is confusing. is this related to uuid?
> > +
> > +.. note::
> > + Partition compatibility matching is key design component of Alveo
> platforms
> > + and XRT. Partitions have child and parent relationship. A loaded
> > +partition
> have a child
> > + exposes child partition UUID to advertise its compatibility
> > + requirement for
>
> the child's
>
> can drop 'for child partition'
Will update
>
> > + child partition. When loading a child partition the xmgmt
> > + management driver
> When loading a child partition,
> > + matches parent UUID of the child partition against child UUID exported by
> > + the parent. Parent and child partition UUIDs are stored in the *xclbin*
> > + (for user) or *xsabin* (for base and shell).
>
> this is confusing, is this part of the file image format ?
>
> Maybe save/move till the image layout.
Yes these IDs are stored in xclbin image format. Will move the sections around
as suggested.
>
> > Except for root UUID, VSEC,
> > + hardware itself does not know about UUIDs. UUIDs are stored in xsabin
> and
> > + xclbin.
> This is confusing too, not sure how to untangle.
Will reword.
> > +
> > +
> > +The physical partitions and their loading is illustrated below::
> > +
> > + SHELL USER
> > + +-----------+ +-------------------+
> > + | | | |
> > + | VSEC UUID | CHILD PARENT | LOGIC UUID |
> > + | o------->|<--------o |
> > + | | UUID UUID | |
> > + +-----+-----+ +--------+----------+
> > + | |
> > + . .
> > + | |
> > + +---+---+ +------+--------+
> > + | POR | | USER COMPILED |
> > + | FLASH | | XCLBIN |
> > + +-------+ +---------------+
> > +
> > +
> > +Loading Sequence
> > +----------------
> > +
> > +The Shell partition is loaded from flash at system boot time. It
> > +establishes the PCIe link and exposes two physical functions to the
> > +BIOS. After OS boot, xmgmt
> the OS boots, the xmgmt
> > +driver attaches to PCIe physical function 0 exposed by the Shell and
> > +then looks for VSEC in PCIe extended configuration space. Using VSEC
> > +it determines the logic
>
> the PCIe
>
> The driver uses VSEC to determine the UUID of Shell. The UUID is also used to
> load a matching ...
>
Will update
> > +UUID of Shell and uses the UUID to load matching *xsabin* file from
> > +Linux firmware directory. The xsabin file contains metadata to
> > +discover peripherals that are part of Shell and firmware(s) for any
> embedded soft processors in Shell.
> the firmware needed for any ...
Will update
> > +
> > +The Shell exports child interface UUID which is used for
> > +compatibility check when
>
> export a child
>
> for a compatibility check
>
Will update
> > +loading user compiled xclbin over the User partition as part of DFX.
> > +When a user requests loading of a specific xclbin the xmgmt
> > +management driver reads the parent
> xclbin, the
> > +interface UUID specified in the xclbin and matches it with child
> > +interface UUID exported by Shell to determine if xclbin is compatible
> > +with the Shell. If match fails loading of xclbin is denied.
> > +
> > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
> > +When loading xclbin, xmgmt driver performs the following *logical*
> operations:
> > +
> > +1. Copy xclbin from user to kernel memory 2. Sanity check the xclbin
> > +contents 3. Isolate the User partition 4. Download the bitstream
> > +using the FPGA config engine (ICAP) 5. De-isolate the User partition
> > +6. Program the clocks (ClockWiz) driving the User partition
> maybe drop '(ClockWiz)'
> > +7. Wait for memory controller (MIG) calibration
> for the
> > +8. Return the loading status back to the caller
> > +
> > +`Platform Loading Overview
> > +<https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`
> > +_ provides more detailed information on platform loading.
> > +
> the link works.
> > +
> > +xsabin
> > +------
> > +
> > +Each Alveo platform comes packaged with its own xsabin. The xsabin is
> > +trusted
> is a trusted
> > +component of the platform. For format details refer to
> > +:ref:`xsabin_xclbin_container_format`
> > +below. xsabin contains basic information like UUIDs, platform name
> > +and metadata in the form of device tree. See :ref:`device_tree_usage`
> below for details and example.
> of a device
> > +
> > +xclbin
> > +------
> > +
> > +xclbin is compiled by end user using
> > +`Vitis
> > +<https://www.xilinx.com/products/design-tools/vitis/vitis-platform.ht
> > +ml>`_
> this link works, seems reasonable landing
> > +tool set from Xilinx. The xclbin contains sections describing user
> > +compiled acceleration engines/kernels, memory subsystems, clocking
> > +information etc. It also contains bitstream for the user partition,
> > +UUIDs, platform name, etc. xclbin uses
> bitstreams
> > +the same container format as xsabin which is described below.
> > +
> > +
> > +.. _xsabin_xclbin_container_format:
> > +
> > +xsabin/xclbin Container Format
> > +------------------------------
> > +
> > +xclbin/xsabin is ELF-like binary container format. It is structured
> > +as series of sections. There is a file header followed by several
> > +section headers which is followed by sections. A section header
> > +points to an actual section. There is an optional signature at the end. The
> format is defined by header file ``xclbin.h``.
> > +The following figure illustrates a typical xclbin::
> > +
> > +
> > + +---------------------+
> > + | |
> > + | HEADER |
> > + +---------------------+
> > + | SECTION HEADER |
> > + | |
> > + +---------------------+
> > + | ... |
> > + | |
> > + +---------------------+
> > + | SECTION HEADER |
> > + | |
> > + +---------------------+
> > + | SECTION |
> > + | |
> > + +---------------------+
> > + | ... |
> > + | |
> > + +---------------------+
> > + | SECTION |
> > + | |
> > + +---------------------+
> > + | SIGNATURE |
> > + | (OPTIONAL) |
> > + +---------------------+
> > +
> > +
> > +xclbin/xsabin files can be packaged, un-packaged and inspected using
> > +XRT utility called **xclbinutil**. xclbinutil is part of XRT open
> > +source software stack. The source code for xclbinutil can be found at
> > +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbi
> > +nutil
> > +
> Works, but maybe the location of a manpage or doc would be better.
> > +For example to enumerate the contents of a xclbin/xsabin use the
> > +*--info* switch as shown below::
> > +
> > +
> > + xclbinutil --info --input
> > + /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
> > + xclbinutil --info --input
> > + /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsab
> > + in
> > +
> > +
> > +.. _device_tree_usage:
> > +
> > +Device Tree Usage
> > +-----------------
> > +
> > +As mentioned previously xsabin stores metadata which advertise HW
> > +subsystems present in a partition. The metadata is stored in device tree
> format with well defined schema.
> > +XRT management driver uses this information to bind *platform
> > +drivers* to the subsystem instantiations. The platform drivers are
> > +found in **xrt-lib.ko** kernel module defined later.
> > +
> > +Logic UUID
> > +^^^^^^^^^^
> > +A partition is identified uniquely through ``logic_uuid`` property::
> > +
> > + /dts-v1/;
> > + / {
> > + logic_uuid = "0123456789abcdef0123456789abcdef";
> > + ...
> > + }
> > +
> > +Schema Version
> > +^^^^^^^^^^^^^^
> > +Schema version is defined through ``schema_version`` node. And it
> > +contains ``major`` and ``minor`` properties as below::
> > +
> > + /dts-v1/;
> > + / {
> > + schema_version {
> > + major = <0x01>;
> > + minor = <0x00>;
> > + };
> > + ...
> > + }
> > +
> > +Partition UUIDs
> > +^^^^^^^^^^^^^^^
> > +As said earlier, each partition may have parent and child UUIDs.
> > +These UUIDs are defined by ``interfaces`` node and ``interface_uuid``
> property::
> > +
> > + /dts-v1/;
> > + / {
> > + interfaces {
> > + @0 {
> > + interface_uuid = "0123456789abcdef0123456789abcdef";
> > + };
> > + @1 {
> > + interface_uuid = "fedcba9876543210fedcba9876543210";
> > + };
> > + ...
> > + };
> > + ...
> > + }
> > +
> > +
> > +Subsystem Instantiations
> > +^^^^^^^^^^^^^^^^^^^^^^^^
> > +Subsystem instantiations are captured as children of
> > +``addressable_endpoints``
> > +node::
> > +
> > + /dts-v1/;
> > + / {
> > + addressable_endpoints {
> > + abc {
> > + ...
> > + };
> > + def {
> > + ...
> > + };
> > + ...
> > + }
> > + }
> > +
> > +Subnode 'abc' and 'def' are the name of subsystem nodes
> > +
> > +Subsystem Node
> > +^^^^^^^^^^^^^^
> > +Each subsystem node and its properties define a hardware instance::
> > +
> > +
> > + addressable_endpoints {
> > + abc {
> > + reg = <0xa 0xb>
> > + pcie_physical_function = <0x0>;
> > + pcie_bar_mapping = <0x2>;
> > + compatible = "abc def";
> > + firmware {
> > + firmware_product_name = "abc"
> > + firmware_branch_name = "def"
> > + firmware_version_major = <1>
> > + firmware_version_minor = <2>
> > + };
> > + }
> > + ...
> > + }
> > +
> > +:reg:
> > + Property defines address range. '<0xa 0xb>' is BAR offset and length
> > +pair, both are 64-bit integer.
> > +:pcie_physical_function:
> > + Property specifies which PCIe physical function the subsystem node resides.
> > +:pcie_bar_mapping:
> > + Property specifies which PCIe BAR the subsystem node resides.
> > +'<0x2>' is BAR index and it is 0 if this property is not defined.
> > +:compatible:
> > + Property is a list of strings. The first string in the list
> > +specifies the exact subsystem node. The following strings represent
> > +other devices that the device is compatible with.
> > +:firmware:
> > + Subnode defines the firmware required by this subsystem node.
> > +
> > +Alveo U50 Platform Example
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +::
> > +
> > + /dts-v1/;
> > +
> > + /{
> > + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> > +
> > + schema_version {
> > + major = <0x01>;
> > + minor = <0x00>;
> > + };
> > +
> > + interfaces {
> > +
> > + @0 {
> > + interface_uuid = "862c7020a250293e32036f19956669e5";
> > + };
> > + };
> > +
> > + addressable_endpoints {
> > +
> > + ep_blp_rom_00 {
> > + reg = <0x00 0x1f04000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> 1.0\0axi_bram_ctrl";
> > + };
> > +
> > + ep_card_flash_program_00 {
> > + reg = <0x00 0x1f06000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_quad_spi-
> 1.0\0axi_quad_spi";
> > + interrupts = <0x03 0x03>;
> > + };
> > +
> > + ep_cmc_firmware_mem_00 {
> > + reg = <0x00 0x1e20000 0x00 0x20000>;
> > + pcie_physical_function = <0x00>;
> > + compatible =
> > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > +
> > + firmware {
> > + firmware_product_name = "cmc";
> > + firmware_branch_name = "u50";
> > + firmware_version_major = <0x01>;
> > + firmware_version_minor = <0x00>;
> > + };
> > + };
> > +
> > + ep_cmc_intc_00 {
> > + reg = <0x00 0x1e03000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > + interrupts = <0x04 0x04>;
> > + };
> > +
> > + ep_cmc_mutex_00 {
> > + reg = <0x00 0x1e02000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_cmc_regmap_00 {
> > + reg = <0x00 0x1e08000 0x00 0x2000>;
> > + pcie_physical_function = <0x00>;
> > + compatible =
> > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > +
> > + firmware {
> > + firmware_product_name = "sc-fw";
> > + firmware_branch_name = "u50";
> > + firmware_version_major = <0x05>;
> > + };
> > + };
> > +
> > + ep_cmc_reset_00 {
> > + reg = <0x00 0x1e01000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_ddr_mem_calib_00 {
> > + reg = <0x00 0x63000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_debug_bscan_mgmt_00 {
> > + reg = <0x00 0x1e90000 0x00 0x10000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-debug_bridge-
> 1.0\0debug_bridge";
> > + };
> > +
> > + ep_ert_base_address_00 {
> > + reg = <0x00 0x21000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_ert_command_queue_mgmt_00 {
> > + reg = <0x00 0x40000 0x00 0x10000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-ert_command_queue-
> 1.0\0ert_command_queue";
> > + };
> > +
> > + ep_ert_command_queue_user_00 {
> > + reg = <0x00 0x40000 0x00 0x10000>;
> > + pcie_physical_function = <0x01>;
> > + compatible = "xilinx.com,reg_abs-ert_command_queue-
> 1.0\0ert_command_queue";
> > + };
> > +
> > + ep_ert_firmware_mem_00 {
> > + reg = <0x00 0x30000 0x00 0x8000>;
> > + pcie_physical_function = <0x00>;
> > + compatible =
> > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > +
> > + firmware {
> > + firmware_product_name = "ert";
> > + firmware_branch_name = "v20";
> > + firmware_version_major = <0x01>;
> > + };
> > + };
> > +
> > + ep_ert_intc_00 {
> > + reg = <0x00 0x23000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > + interrupts = <0x05 0x05>;
> > + };
> > +
> > + ep_ert_reset_00 {
> > + reg = <0x00 0x22000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_ert_sched_00 {
> > + reg = <0x00 0x50000 0x00 0x1000>;
> > + pcie_physical_function = <0x01>;
> > + compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> > + interrupts = <0x09 0x0c>;
> > + };
> > +
> > + ep_fpga_configuration_00 {
> > + reg = <0x00 0x1e88000 0x00 0x8000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_hwicap-
> 1.0\0axi_hwicap";
> > + interrupts = <0x02 0x02>;
> > + };
> > +
> > + ep_icap_reset_00 {
> > + reg = <0x00 0x1f07000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_msix_00 {
> > + reg = <0x00 0x00 0x00 0x20000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> > + pcie_bar_mapping = <0x02>;
> > + };
> > +
> > + ep_pcie_link_mon_00 {
> > + reg = <0x00 0x1f05000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_pr_isolate_plp_00 {
> > + reg = <0x00 0x1f01000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_pr_isolate_ulp_00 {
> > + reg = <0x00 0x1000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > + };
> > +
> > + ep_uuid_rom_00 {
> > + reg = <0x00 0x64000 0x00 0x1000>;
> > + pcie_physical_function = <0x00>;
> > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> 1.0\0axi_bram_ctrl";
> > + };
> > +
> > + ep_xdma_00 {
> > + reg = <0x00 0x00 0x00 0x10000>;
> > + pcie_physical_function = <0x01>;
> > + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> > + pcie_bar_mapping = <0x02>;
> > + };
> > + };
> > +
> > + }
> > +
> > +
> > +
> > +Deployment Models
> > +=================
> > +
> > +Baremetal
> > +---------
> > +
> > +In bare-metal deployments both MPF and UPF are visible and
> > +accessible. xmgmt
> In bare-meta deployments,
> > +driver binds to MPF. xmgmt driver operations are privileged and
> > +available to system administrator. The full stack is illustrated below::
> > +
> > + HOST
> > +
> > + [XMGMT] [XUSER]
> > + | |
> > + | |
> > + +-----+ +-----+
> > + | MPF | | UPF |
> > + | | | |
> > + | PF0 | | PF1 |
> > + +--+--+ +--+--+
> > + ......... ^................. ^..........
> > + | |
> > + | PCIe DEVICE |
> > + | |
> > + +--+------------------+--+
> > + | SHELL |
> > + | |
> > + +------------------------+
> > + | USER |
> > + | |
> > + | |
> > + | |
> > + | |
> > + +------------------------+
> > +
> > +
> > +
> > +Virtualized
> > +-----------
> > +
> > +In virtualized deployments privileged MPF is assigned to host but
> > +unprivileged
> In virtualized deployments, the
> > +UPF is assigned to guest VM via PCIe pass-through. xmgmt driver in
> > +host binds
> in the host
> > +to MPF. xmgmt driver operations are privileged and only accessible by
> > +hosting
> to the MPF
> > +service provider. The full stack is illustrated below::
> > +
> > +
> > + .............
> > + HOST . VM .
> > + . .
> > + [XMGMT] . [XUSER] .
> > + | . | .
> > + | . | .
> > + +-----+ . +-----+ .
> > + | MPF | . | UPF | .
> > + | | . | | .
> > + | PF0 | . | PF1 | .
> > + +--+--+ . +--+--+ .
> > + ......... ^................. ^..........
> > + | |
> > + | PCIe DEVICE |
> > + | |
> > + +--+------------------+--+
> > + | SHELL |
> > + | |
> > + +------------------------+
> > + | USER |
> > + | |
> > + | |
> > + | |
> > + | |
> > + +------------------------+
> > +
> > +
> > +
> > +
> > +
> > +Platform Security Considerations
> > +================================
> > +
> > +`Security of Alveo Platform
> > +<https://xilinx.github.io/XRT/master/html/security.html>`_
> > +discusses the deployment options and security implications in great detail.
>
> This link works and looks great.
>
> Tom
Thanks for the detailed review of the document. I am working on incorporating the
feedback. One outstanding question is about usage of formal bus in XRT and if we
should phase that in as a follow-on. It would also determine if IP drivers should
move to other subsystems.
-Sonal
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a document describing XRT Alveo drivers
2021-03-01 6:48 ` Sonal Santan
@ 2021-03-06 17:19 ` Moritz Fischer
2021-03-08 20:12 ` Sonal Santan
0 siblings, 1 reply; 87+ messages in thread
From: Moritz Fischer @ 2021-03-06 17:19 UTC (permalink / raw)
To: Sonal Santan
Cc: Tom Rix, Lizhi Hou, linux-kernel, linux-fpga, Max Zhen,
Michal Simek, Stefano Stabellini, devicetree, mdf, robh
On Mon, Mar 01, 2021 at 06:48:46AM +0000, Sonal Santan wrote:
> Hello Tom,
>
> > -----Original Message-----
> > From: Tom Rix <trix@redhat.com>
> > Sent: Friday, February 19, 2021 2:26 PM
> > To: Lizhi Hou <lizhih@xilinx.com>; linux-kernel@vger.kernel.org
> > Cc: Lizhi Hou <lizhih@xilinx.com>; linux-fpga@vger.kernel.org; Max Zhen
> > <maxz@xilinx.com>; Sonal Santan <sonals@xilinx.com>; Michal Simek
> > <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
> > devicetree@vger.kernel.org; mdf@kernel.org; robh@kernel.org; Max Zhen
> > <maxz@xilinx.com>
> > Subject: Re: [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a
> > document describing XRT Alveo drivers
> >
> > From the documentation, there are a couple of big questions and a bunch of
> > word smithing.
> >
> > pseudo-bus : do we need a bus ?
> We are looking for guidance here.
> >
> > xrt-lib real platform devices that aren't fpga, do they need to move to another
> > subsystem ?
> >
>
> Drivers for the IPs that show up in the Alveo shell are not generic enough. They
> fit into the framework that XRT uses. Is the idea that that they can be used in a
> different context?
>
> > Overall looks good, love the ascii art!
> >
> > On 2/17/21 10:40 PM, Lizhi Hou wrote:
> > > Describe XRT driver architecture and provide basic overview of Xilinx
> > > Alveo platform.
> > >
> > > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > > Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> > > Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> > > ---
> > > Documentation/fpga/index.rst | 1 +
> > > Documentation/fpga/xrt.rst | 842
> > +++++++++++++++++++++++++++++++++++
> > > 2 files changed, 843 insertions(+)
> > > create mode 100644 Documentation/fpga/xrt.rst
> > >
> > > diff --git a/Documentation/fpga/index.rst
> > > b/Documentation/fpga/index.rst index f80f95667ca2..30134357b70d 100644
> > > --- a/Documentation/fpga/index.rst
> > > +++ b/Documentation/fpga/index.rst
> > > @@ -8,6 +8,7 @@ fpga
> > > :maxdepth: 1
> > >
> > > dfl
> > > + xrt
> > >
> > > .. only:: subproject and html
> > >
> > > diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
> > > new file mode 100644 index 000000000000..9bc2d2785cb9
> > > --- /dev/null
> > > +++ b/Documentation/fpga/xrt.rst
> > > @@ -0,0 +1,842 @@
> > > +.. SPDX-License-Identifier: GPL-2.0
> > > +
> > > +==================================
> > > +XRTV2 Linux Kernel Driver Overview
> > > +==================================
> > > +
> > > +Authors:
> > > +
> > > +* Sonal Santan <sonal.santan@xilinx.com>
> > > +* Max Zhen <max.zhen@xilinx.com>
> > > +* Lizhi Hou <lizhi.hou@xilinx.com>
> > > +
> > > +XRTV2 drivers are second generation `XRT
> > > +<https://github.com/Xilinx/XRT>`_ drivers which support `Alveo
> > > +<https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
> > > +PCIe platforms from Xilinx.
> > > +
> > > +XRTV2 drivers support *subsystem* style data driven platforms where
> > > +driver's
> > where the driver's
> > > +configuration and behavior is determined by meta data provided by the
> > > +platform (in *device tree* format). Primary management physical
> > > +function (MPF) driver is called **xmgmt**. Primary user physical
> > > +function (UPF) driver is called
> > > +**xuser** and is under development. xrt driver framework and HW
> > > +subsystem drivers are packaged into a library module called
> > > +**xrt-lib**, which is shared by **xmgmt** and **xuser** (under
> > > +development). The xrt driver framework
> > xuser still under development ?
> > > +implements a pseudo-bus which is used to discover HW subsystems and
> > > +facilitate
> >
> > A pseudo-bus.
> >
> > It would be good if this was close to what was done for dfl here
> >
> > https://lore.kernel.org/linux-fpga/1605159759-3439-1-git-send-email-
> > yilun.xu@intel.com/
> >
>
> I am wondering if we can phase in the migration to formal bus architecture
> based on struct bus_type as a follow on set of patches?
I'd rather have it done early on. If we know there's a better way of
doing something and we don't the code should go into staging.
This gives us an out if it doesn't happen, otherwise the kernel
community would depend on corporate goodwill to appropriately staff it.
Note, I'm not trying to say there's any ill will anywhere, Xilinx has
been traditionally good about this :)
>
> > > +inter HW subsystem interaction.
> > > +
> > > +Driver Modules
> > > +==============
> > > +
> > > +xrt-lib.ko
> > > +----------
> > > +
> > > +Repository of all subsystem drivers and pure software modules that
> > > +can potentially
> >
> > subsystem drivers
> >
> > drivers in fpga/ should be for managing just the fpganess of the fpga.
> >
> > soft devices ex/ a soft tty should go to their respective subsystem location
> >
> > Are there any in this patchset you think might move ?
>
> We have already shrunk the patch to only include FPGA centric pieces
> necessary to get the bitstream download implemented. Should we explore
> the question of subsystem drivers when we add support for more features of
> the Alveo shell?
>
> >
> > Maybe we can defer reviewing those now.
> >
> > > +be shared between xmgmt and xuser. All these drivers are structured
> > > +as Linux *platform driver* and are instantiated by xmgmt (or xuser
> > > +under development) based on meta data associated with hardware. The
> > > +metadata is in the form of device tree
> >
> > with the hardware
> >
> > form of a device tree
> >
>
> Will change
>
> > > +as mentioned before. Each platform driver statically defines a
> > > +subsystem node array by using node name or a string in its
> > > +``compatible`` property. And this array is eventually translated to IOMEM
> > resources of the platform device.
> > > +
> > > +The xrt-lib core infrastructure provides hooks to platform drivers
> > > +for device node management, user file operations and ioctl callbacks.
> > > +The core also provides pseudo-bus functionality for platform driver
> > > +registration, discovery and inter platform driver ioctl calls.
> >
> > core infrastructure.
> >
>
> Will update.
>
> > The interfaces to the infrastructure are not in include/linux/fpga/
> >
> > Maybe this needs to change.
> >
>
> Were you thinking of moving XRT infrastructure header files from
> drivers/fpga/xrt/include to include/linux/fpga?
>
> > > +
> > > +.. note::
> > > + See code in ``include/xleaf.h``
> > > +
> > > +
> > > +xmgmt.ko
> > > +--------
> > > +
> > > +The xmgmt driver is a PCIe device driver driving MPF found on
> > > +Xilinx's Alveo PCIE device. It consists of one *root* driver, one or
> > > +more *group* drivers and one or more *xleaf* drivers. The root and
> > > +MPF specific xleaf drivers are in xmgmt.ko. The group driver and other xleaf
> > drivers are in xrt-lib.ko.
> > I am not sure if *.ko is correct, these will also be intree.
> > > +
> > > +The instantiation of specific group driver or xleaf driver is
> > > +completely data
> > of a specific
> > > +driven based on meta data (mostly in device tree format) found
> > > +through VSEC
> > mostly ? what is the deviation from device tree ?
> > > +capability and inside firmware files, such as platform xsabin or user xclbin
> > file.
> > > +The root driver manages life cycle of multiple group drivers, which,
> > > +in turn,
> > the life cycle
> > > +manages multiple xleaf drivers. This allows a single set of driver
> > > +code to support
> >
> > set of drivers
> >
> > drop 'code'
> >
>
> Will update
>
> > > +all kinds of subsystems exposed by different shells. The difference
> > > +among all these subsystems will be handled in xleaf drivers with root
> > > +and group drivers being part of the infrastructure and provide common
> > > +services for all leaves found on all platforms.
> > > +
> > > +The driver object model looks like the following::
> > > +
> > > + +-----------+
> > > + | xroot |
> > > + +-----+-----+
> > > + |
> > > + +-----------+-----------+
> > > + | |
> > > + v v
> > > + +-----------+ +-----------+
> > > + | group | ... | group |
> > > + +-----+-----+ +------+----+
> > > + | |
> > > + | |
> > > + +-----+----+ +-----+----+
> > > + | | | |
> > > + v v v v
> > > + +-------+ +-------+ +-------+ +-------+
> > > + | xleaf |..| xleaf | | xleaf |..| xleaf |
> > > + +-------+ +-------+ +-------+ +-------+
> > > +
> > > +As an example for Xilinx Alveo U50 before user xclbin download, the
> > > +tree looks like the following::
> > > +
> > > + +-----------+
> > > + | xmgmt |
> > > + +-----+-----+
> > > + |
> > > + +-------------------------+--------------------+
> > > + | | |
> > > + v v v
> > > + +--------+ +--------+ +--------+
> > > + | group0 | | group1 | | group2 |
> > > + +----+---+ +----+---+ +---+----+
> > > + | | |
> > > + | | |
> > > + +-----+-----+ +----+-----+---+ +-----+-----+----+--------+
> > > + | | | | | | | | |
> > > + v v | v v | v v |
> > > + +------------+ +------+ | +------+ +------+ | +------+
> > > + +------------+ +-----------+ |
> > > + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | |
> > > + | AXI-GATE0 | |
> > > + +------------+ +------+ | +------+ +------+ | +------+
> > > + +------------+ +-----------+ |
> > > + | +---------+ | +------+ +-----------+ |
> > > + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+
> > > + +---------+ | +------+ +-----------+
> > > + | +-------+
> > > + +->| CALIB |
> > > + +-------+
> > > +
> > Nice ascii art!
> > > +After an xclbin is download, group3 will be added and the tree looks
> > > +like the
> > > +following::
> > > +
> > > + +-----------+
> > > + | xmgmt |
> > > + +-----+-----+
> > > + |
> > > + +-------------------------+--------------------+-----------------+
> > > + | | | |
> > > + v v v |
> > > + +--------+ +--------+ +--------+ |
> > > + | group0 | | group1 | | group2 | |
> > > + +----+---+ +----+---+ +---+----+ |
> > > + | | | |
> > > + | | | |
> > > + +-----+-----+ +-----+-----+---+ +-----+-----+----+--------+ |
> > > + | | | | | | | | | |
> > > + v v | v v | v v | |
> > > + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
> > > + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | |
> > |
> > > + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
> > > + | +---------+ | +------+ +-----------+ | |
> > > + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+ |
> > > + +---------+ | +------+ +-----------+ |
> > > + | +-------+ |
> > > + +->| CALIB | |
> > > + +-------+ |
> > > + +---+----+ |
> > > + | group3 |<--------------------------------------------+
> > > + +--------+
> > > + |
> > > + |
> > > + +-------+--------+---+--+--------+------+-------+
> > > + | | | | | | |
> > > + v | v | v | v
> > > + +--------+ | +--------+ | +--------+ | +-----+
> > > + | CLOCK0 | | | CLOCK1 | | | CLOCK2 | | | UCS |
> > > + +--------+ v +--------+ v +--------+ v +-----+
> > > + +-------------+ +-------------+ +-------------+
> > > + | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
> > > + +-------------+ +-------------+ +-------------+
> > > +
> > > +
> > > +xmgmt-root
> > > +^^^^^^^^^^
> > > +
> > > +The xmgmt-root driver is a PCIe device driver attached to MPF. It's
> > > +part of the infrastructure of the MPF driver and resides in xmgmt.ko.
> > > +This driver
> > > +
> > > +* manages one or more group drivers
> > > +* provides access to functionalities that requires pci_dev, such as
> > > +PCIE config
> > > + space access, to other xleaf drivers through root calls
> > > +* together with group driver, facilities event callbacks for other
> > > +xleaf drivers
> > > +* together with group driver, facilities inter-leaf driver calls for
> > > +other xleaf
> > Maybe drop 'together with group driver'
>
> Will update
>
> > > + drivers
> > > +
> > > +When root driver starts, it will explicitly create an initial group
> > > +instance, which contains xleaf drivers that will trigger the creation
> > > +of other group instances. The root driver will wait for all group and
> > > +leaves to be created before it returns from it's probe routine and
> > > +claim success of the initialization of the entire xmgmt driver.
> > What happens if there a failure in one leaf ? Does the whole board go down ?
> > > +
> > > +.. note::
> > > + See code in ``lib/xroot.c`` and ``mgmt/root.c``
> > > +
> > > +
> > > +group
> > > +^^^^^
> > > +
> > > +The group driver is a platform device driver whose life cycle is
> > > +managed by
> > Maybe call this a 'pseudo device'
>
> Will update
>
> > > +root and does not have real IO mem or IRQ resources. It's part of the
> > > +infrastructure of the MPF driver and resides in xrt-lib.ko. This
> > > +driver
> > > +
> > > +* manages one or more xleaf drivers so that multiple leaves can be
> > > +managed as a
> > > + group
> > can drop 'so that multiple leaves can be managed as a group' to me, this is the
> > same as 'one or more'
>
> Will do
>
> > > +* provides access to root from leaves, so that root calls, event
> > > +notifications
> > > + and inter-leaf calls can happen
> > > +
> > > +In xmgmt, an initial group driver instance will be created by root,
> > > +which
> > by the root
> > > +contains leaves that will trigger group instances to be created to
> > > +manage groups of leaves found on different partitions on hardware,
> > > +such as VSEC, Shell, and User.
> > > +
> > > +Every *fpga_region* has a group object associated with it. The group
> > > +is created when xclbin image is loaded on the fpga_region. The
> > > +existing group is destroyed when a new xclbin image is loaded. The
> > > +fpga_region persists across xclbin downloads.
> > The connection of a 'group' node to a fpga region region is fairly important,
> > maybe move this section earlier. 'group' as an fpganess thing would be kept in
> > fpga/ subsystem.
>
> Will update
>
> > > +
> > > +.. note::
> > > + See code in ``lib/group.c``
> > > +
> > > +
> > > +xleaf
> > > +^^^^^
> > > +
> > > +The xleaf driver is a platform device driver whose life cycle is
> > > +managed by a group driver and may or may not have real IO mem or IRQ
> > > +resources. They are the real meat of xmgmt and contains platform
> > > +specific code to Shell and User found on a MPF.
> > > +
> >
> > Maybe a split is pseudo device leaves, those without real IO mem, stay in
> > fpga/ others go ?
> >
>
> This goes back to the earlier question of what minimal set of platform drivers
> should stay in fpga subsystem. There are some like bridge or configuration
> engine (also called icap) which have their own IO mem but do not have a life
> outside of fpga subsystem.
>
> > > +A xleaf driver may not have real hardware resources when it merely
> > > +acts as a driver that manages certain in-memory states for xmgmt.
> > > +These in-memory states could be shared by multiple other leaves.
> > > +
> > This implies locking and some message passing.
> > > +Leaf drivers assigned to specific hardware resources drive specific
> > > +subsystem in
> > drive a specific
> > > +the device. To manipulate the subsystem or carry out a task, a xleaf
> > > +driver may ask help from root via root calls and/or from other leaves via
> > inter-leaf calls.
> > > +
> > > +A xleaf can also broadcast events through infrastructure code for
> > > +other leaves to process. It can also receive event notification from
> > > +infrastructure about certain events, such as post-creation or pre-exit of a
> > particular xleaf.
> > I would like to see some examples of how the inter node communications work.
>
> Would update to show an example.
>
> > > +
> > > +.. note::
> > > + See code in ``lib/xleaf/*.c``
> > > +
> > > +
> > > +FPGA Manager Interaction
> > > +========================
> > > +
> > > +fpga_manager
> > > +------------
> > > +
> > > +An instance of fpga_manager is created by xmgmt_main and is used for
> > > +xclbin
> > for the xclbin
> > > +image download. fpga_manager requires the full xclbin image before it
> > > +can start programming the FPGA configuration engine via ICAP platform
> > driver.
> >
> > via the ICAP
> >
> > what is ICAP ?
>
> Will update. ICAP stands for Internal Configuration Access Port used for configuring
> the fpga.
>
> >
> > > +
> > > +fpga_region
> > > +-----------
> > > +
> > > +For every interface exposed by currently loaded xclbin/xsabin in the
> > > +*parent*
> > by the currently
> > > +fpga_region a new instance of fpga_region is created like a *child* region.
> > fpga_region,
> > > +The device tree of the *parent* fpga_region defines the resources for
> > > +a new instance of fpga_bridge which isolates the parent from
> > and isolates
> > > +child fpga_region. This new instance of fpga_bridge will be used when
> > > +a xclbin image is loaded on the child fpga_region. After the xclbin
> > > +image is downloaded to the fpga_region, an instance of group is
> > > +created for the fpga_region using the device tree obtained as part of
> > > +xclbin. If this device
> > of the xclbin
> > > +tree defines any child interfaces then it can trigger the creation of
> > interfaces, then
> > > +fpga_bridge and fpga_region for the next region in the chain.
> > a fpga_bridge and a fpga_region
> > > +
> > > +fpga_bridge
> > > +-----------
> > > +
> > > +Like fpga_region, matching fpga_bridge is also created by walking the
> > > +device
> > Like the fpga_region, a matchin
> > > +tree of the parent group.
> > > +
> > > +Driver Interfaces
> > > +=================
> > > +
> > > +xmgmt Driver Ioctls
> > > +-------------------
> > > +
> > > +Ioctls exposed by xmgmt driver to user space are enumerated in the
> > > +following
> > > +table:
> > > +
> > > +== ===================== ============================
> > ==========================
> > > +# Functionality ioctl request code data format
> > > +== ===================== ============================
> > ==========================
> > > +1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF
> > xmgmt_ioc_bitstream_axlf
> > > +== ===================== ============================
> > > +==========================
> >
> > This data format is described below, maybe swap this section with that so
> >
> > folks will know what xmgmnt_ioc_bitstream_axlf is before this section.
> >
>
> Will update.
>
> > > +
> > > +User xclbin can be downloaded by using xbmgmt tool from XRT open
> > > +source suite. See
> >
> > A user xclbin
> >
> > using the xbmgmt
> >
> > from the XRT
> >
>
> Will update
>
> > > +example usage below::
> > > +
> > > + xbmgmt partition --program --path
> > > + /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xc
> > > + lbin --force
> > > +
> > > +xmgmt Driver Sysfs
> > > +------------------
> > > +
> > > +xmgmt driver exposes a rich set of sysfs interfaces. Subsystem
> > > +platform drivers export sysfs node for every platform instance.
> > > +
> > > +Every partition also exports its UUIDs. See below for examples::
> > > +
> > > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
> > > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
> > > +
> > > +
> > > +hwmon
> > > +-----
> > > +
> > > +xmgmt driver exposes standard hwmon interface to report voltage,
> > > +current, temperature, power, etc. These can easily be viewed using
> > > +*sensors* command line utility.
> > > +
> > > +Alveo Platform Overview
> > > +=======================
> > > +
> > > +Alveo platforms are architected as two physical FPGA partitions:
> > > +*Shell* and *User*. The Shell provides basic infrastructure for the
> > > +Alveo platform like PCIe connectivity, board management, Dynamic
> > > +Function Exchange (DFX), sensors, clocking, reset, and security. User
> > > +partition contains user compiled FPGA
> > the user compiled
> > > +binary which is loaded by a process called DFX also known as partial
> > > +reconfiguration.
> > > +
> > > +Physical partitions require strict HW compatibility with each other
> > > +for DFX to work properly.
> >
> > swap order
> >
> > For DFX to work properly physical partitions ..
> >
> >
>
> Will update
>
> > > Every physical partition has two interface UUIDs: *parent* UUID
> > > +and *child* UUID. For simple single stage platforms, Shell → User
> > > +forms parent child relationship. For complex two stage platforms,
> > > +Base → Shell → User forms the parent child relationship chain.
> > this bit is confusing. is this related to uuid?
> > > +
> > > +.. note::
> > > + Partition compatibility matching is key design component of Alveo
> > platforms
> > > + and XRT. Partitions have child and parent relationship. A loaded
> > > +partition
> > have a child
> > > + exposes child partition UUID to advertise its compatibility
> > > + requirement for
> >
> > the child's
> >
> > can drop 'for child partition'
>
> Will update
>
> >
> > > + child partition. When loading a child partition the xmgmt
> > > + management driver
> > When loading a child partition,
> > > + matches parent UUID of the child partition against child UUID exported by
> > > + the parent. Parent and child partition UUIDs are stored in the *xclbin*
> > > + (for user) or *xsabin* (for base and shell).
> >
> > this is confusing, is this part of the file image format ?
> >
> > Maybe save/move till the image layout.
>
> Yes these IDs are stored in xclbin image format. Will move the sections around
> as suggested.
>
> >
> > > Except for root UUID, VSEC,
> > > + hardware itself does not know about UUIDs. UUIDs are stored in xsabin
> > and
> > > + xclbin.
> > This is confusing too, not sure how to untangle.
>
> Will reword.
>
> > > +
> > > +
> > > +The physical partitions and their loading is illustrated below::
> > > +
> > > + SHELL USER
> > > + +-----------+ +-------------------+
> > > + | | | |
> > > + | VSEC UUID | CHILD PARENT | LOGIC UUID |
> > > + | o------->|<--------o |
> > > + | | UUID UUID | |
> > > + +-----+-----+ +--------+----------+
> > > + | |
> > > + . .
> > > + | |
> > > + +---+---+ +------+--------+
> > > + | POR | | USER COMPILED |
> > > + | FLASH | | XCLBIN |
> > > + +-------+ +---------------+
> > > +
> > > +
> > > +Loading Sequence
> > > +----------------
> > > +
> > > +The Shell partition is loaded from flash at system boot time. It
> > > +establishes the PCIe link and exposes two physical functions to the
> > > +BIOS. After OS boot, xmgmt
> > the OS boots, the xmgmt
> > > +driver attaches to PCIe physical function 0 exposed by the Shell and
> > > +then looks for VSEC in PCIe extended configuration space. Using VSEC
> > > +it determines the logic
> >
> > the PCIe
> >
> > The driver uses VSEC to determine the UUID of Shell. The UUID is also used to
> > load a matching ...
> >
>
> Will update
>
> > > +UUID of Shell and uses the UUID to load matching *xsabin* file from
> > > +Linux firmware directory. The xsabin file contains metadata to
> > > +discover peripherals that are part of Shell and firmware(s) for any
> > embedded soft processors in Shell.
> > the firmware needed for any ...
>
> Will update
>
> > > +
> > > +The Shell exports child interface UUID which is used for
> > > +compatibility check when
> >
> > export a child
> >
> > for a compatibility check
> >
>
> Will update
>
> > > +loading user compiled xclbin over the User partition as part of DFX.
> > > +When a user requests loading of a specific xclbin the xmgmt
> > > +management driver reads the parent
> > xclbin, the
> > > +interface UUID specified in the xclbin and matches it with child
> > > +interface UUID exported by Shell to determine if xclbin is compatible
> > > +with the Shell. If match fails loading of xclbin is denied.
> > > +
> > > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
> > > +When loading xclbin, xmgmt driver performs the following *logical*
> > operations:
> > > +
> > > +1. Copy xclbin from user to kernel memory 2. Sanity check the xclbin
> > > +contents 3. Isolate the User partition 4. Download the bitstream
> > > +using the FPGA config engine (ICAP) 5. De-isolate the User partition
> > > +6. Program the clocks (ClockWiz) driving the User partition
> > maybe drop '(ClockWiz)'
> > > +7. Wait for memory controller (MIG) calibration
> > for the
> > > +8. Return the loading status back to the caller
> > > +
> > > +`Platform Loading Overview
> > > +<https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`
> > > +_ provides more detailed information on platform loading.
> > > +
> > the link works.
> > > +
> > > +xsabin
> > > +------
> > > +
> > > +Each Alveo platform comes packaged with its own xsabin. The xsabin is
> > > +trusted
> > is a trusted
> > > +component of the platform. For format details refer to
> > > +:ref:`xsabin_xclbin_container_format`
> > > +below. xsabin contains basic information like UUIDs, platform name
> > > +and metadata in the form of device tree. See :ref:`device_tree_usage`
> > below for details and example.
> > of a device
> > > +
> > > +xclbin
> > > +------
> > > +
> > > +xclbin is compiled by end user using
> > > +`Vitis
> > > +<https://www.xilinx.com/products/design-tools/vitis/vitis-platform.ht
> > > +ml>`_
> > this link works, seems reasonable landing
> > > +tool set from Xilinx. The xclbin contains sections describing user
> > > +compiled acceleration engines/kernels, memory subsystems, clocking
> > > +information etc. It also contains bitstream for the user partition,
> > > +UUIDs, platform name, etc. xclbin uses
> > bitstreams
> > > +the same container format as xsabin which is described below.
> > > +
> > > +
> > > +.. _xsabin_xclbin_container_format:
> > > +
> > > +xsabin/xclbin Container Format
> > > +------------------------------
> > > +
> > > +xclbin/xsabin is ELF-like binary container format. It is structured
> > > +as series of sections. There is a file header followed by several
> > > +section headers which is followed by sections. A section header
> > > +points to an actual section. There is an optional signature at the end. The
> > format is defined by header file ``xclbin.h``.
> > > +The following figure illustrates a typical xclbin::
> > > +
> > > +
> > > + +---------------------+
> > > + | |
> > > + | HEADER |
> > > + +---------------------+
> > > + | SECTION HEADER |
> > > + | |
> > > + +---------------------+
> > > + | ... |
> > > + | |
> > > + +---------------------+
> > > + | SECTION HEADER |
> > > + | |
> > > + +---------------------+
> > > + | SECTION |
> > > + | |
> > > + +---------------------+
> > > + | ... |
> > > + | |
> > > + +---------------------+
> > > + | SECTION |
> > > + | |
> > > + +---------------------+
> > > + | SIGNATURE |
> > > + | (OPTIONAL) |
> > > + +---------------------+
> > > +
> > > +
> > > +xclbin/xsabin files can be packaged, un-packaged and inspected using
> > > +XRT utility called **xclbinutil**. xclbinutil is part of XRT open
> > > +source software stack. The source code for xclbinutil can be found at
> > > +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbi
> > > +nutil
> > > +
> > Works, but maybe the location of a manpage or doc would be better.
> > > +For example to enumerate the contents of a xclbin/xsabin use the
> > > +*--info* switch as shown below::
> > > +
> > > +
> > > + xclbinutil --info --input
> > > + /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
> > > + xclbinutil --info --input
> > > + /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsab
> > > + in
> > > +
> > > +
> > > +.. _device_tree_usage:
> > > +
> > > +Device Tree Usage
> > > +-----------------
> > > +
> > > +As mentioned previously xsabin stores metadata which advertise HW
> > > +subsystems present in a partition. The metadata is stored in device tree
> > format with well defined schema.
> > > +XRT management driver uses this information to bind *platform
> > > +drivers* to the subsystem instantiations. The platform drivers are
> > > +found in **xrt-lib.ko** kernel module defined later.
> > > +
> > > +Logic UUID
> > > +^^^^^^^^^^
> > > +A partition is identified uniquely through ``logic_uuid`` property::
> > > +
> > > + /dts-v1/;
> > > + / {
> > > + logic_uuid = "0123456789abcdef0123456789abcdef";
> > > + ...
> > > + }
> > > +
> > > +Schema Version
> > > +^^^^^^^^^^^^^^
> > > +Schema version is defined through ``schema_version`` node. And it
> > > +contains ``major`` and ``minor`` properties as below::
> > > +
> > > + /dts-v1/;
> > > + / {
> > > + schema_version {
> > > + major = <0x01>;
> > > + minor = <0x00>;
> > > + };
> > > + ...
> > > + }
> > > +
> > > +Partition UUIDs
> > > +^^^^^^^^^^^^^^^
> > > +As said earlier, each partition may have parent and child UUIDs.
> > > +These UUIDs are defined by ``interfaces`` node and ``interface_uuid``
> > property::
> > > +
> > > + /dts-v1/;
> > > + / {
> > > + interfaces {
> > > + @0 {
> > > + interface_uuid = "0123456789abcdef0123456789abcdef";
> > > + };
> > > + @1 {
> > > + interface_uuid = "fedcba9876543210fedcba9876543210";
> > > + };
> > > + ...
> > > + };
> > > + ...
> > > + }
> > > +
> > > +
> > > +Subsystem Instantiations
> > > +^^^^^^^^^^^^^^^^^^^^^^^^
> > > +Subsystem instantiations are captured as children of
> > > +``addressable_endpoints``
> > > +node::
> > > +
> > > + /dts-v1/;
> > > + / {
> > > + addressable_endpoints {
> > > + abc {
> > > + ...
> > > + };
> > > + def {
> > > + ...
> > > + };
> > > + ...
> > > + }
> > > + }
> > > +
> > > +Subnode 'abc' and 'def' are the name of subsystem nodes
> > > +
> > > +Subsystem Node
> > > +^^^^^^^^^^^^^^
> > > +Each subsystem node and its properties define a hardware instance::
> > > +
> > > +
> > > + addressable_endpoints {
> > > + abc {
> > > + reg = <0xa 0xb>
> > > + pcie_physical_function = <0x0>;
> > > + pcie_bar_mapping = <0x2>;
> > > + compatible = "abc def";
> > > + firmware {
> > > + firmware_product_name = "abc"
> > > + firmware_branch_name = "def"
> > > + firmware_version_major = <1>
> > > + firmware_version_minor = <2>
> > > + };
> > > + }
> > > + ...
> > > + }
> > > +
> > > +:reg:
> > > + Property defines address range. '<0xa 0xb>' is BAR offset and length
> > > +pair, both are 64-bit integer.
> > > +:pcie_physical_function:
> > > + Property specifies which PCIe physical function the subsystem node resides.
> > > +:pcie_bar_mapping:
> > > + Property specifies which PCIe BAR the subsystem node resides.
> > > +'<0x2>' is BAR index and it is 0 if this property is not defined.
> > > +:compatible:
> > > + Property is a list of strings. The first string in the list
> > > +specifies the exact subsystem node. The following strings represent
> > > +other devices that the device is compatible with.
> > > +:firmware:
> > > + Subnode defines the firmware required by this subsystem node.
> > > +
> > > +Alveo U50 Platform Example
> > > +^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > +::
> > > +
> > > + /dts-v1/;
> > > +
> > > + /{
> > > + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> > > +
> > > + schema_version {
> > > + major = <0x01>;
> > > + minor = <0x00>;
> > > + };
> > > +
> > > + interfaces {
> > > +
> > > + @0 {
> > > + interface_uuid = "862c7020a250293e32036f19956669e5";
> > > + };
> > > + };
> > > +
> > > + addressable_endpoints {
> > > +
> > > + ep_blp_rom_00 {
> > > + reg = <0x00 0x1f04000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> > 1.0\0axi_bram_ctrl";
> > > + };
> > > +
> > > + ep_card_flash_program_00 {
> > > + reg = <0x00 0x1f06000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_quad_spi-
> > 1.0\0axi_quad_spi";
> > > + interrupts = <0x03 0x03>;
> > > + };
> > > +
> > > + ep_cmc_firmware_mem_00 {
> > > + reg = <0x00 0x1e20000 0x00 0x20000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible =
> > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > + firmware {
> > > + firmware_product_name = "cmc";
> > > + firmware_branch_name = "u50";
> > > + firmware_version_major = <0x01>;
> > > + firmware_version_minor = <0x00>;
> > > + };
> > > + };
> > > +
> > > + ep_cmc_intc_00 {
> > > + reg = <0x00 0x1e03000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > > + interrupts = <0x04 0x04>;
> > > + };
> > > +
> > > + ep_cmc_mutex_00 {
> > > + reg = <0x00 0x1e02000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_cmc_regmap_00 {
> > > + reg = <0x00 0x1e08000 0x00 0x2000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible =
> > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > + firmware {
> > > + firmware_product_name = "sc-fw";
> > > + firmware_branch_name = "u50";
> > > + firmware_version_major = <0x05>;
> > > + };
> > > + };
> > > +
> > > + ep_cmc_reset_00 {
> > > + reg = <0x00 0x1e01000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_ddr_mem_calib_00 {
> > > + reg = <0x00 0x63000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_debug_bscan_mgmt_00 {
> > > + reg = <0x00 0x1e90000 0x00 0x10000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-debug_bridge-
> > 1.0\0debug_bridge";
> > > + };
> > > +
> > > + ep_ert_base_address_00 {
> > > + reg = <0x00 0x21000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_ert_command_queue_mgmt_00 {
> > > + reg = <0x00 0x40000 0x00 0x10000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-ert_command_queue-
> > 1.0\0ert_command_queue";
> > > + };
> > > +
> > > + ep_ert_command_queue_user_00 {
> > > + reg = <0x00 0x40000 0x00 0x10000>;
> > > + pcie_physical_function = <0x01>;
> > > + compatible = "xilinx.com,reg_abs-ert_command_queue-
> > 1.0\0ert_command_queue";
> > > + };
> > > +
> > > + ep_ert_firmware_mem_00 {
> > > + reg = <0x00 0x30000 0x00 0x8000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible =
> > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > +
> > > + firmware {
> > > + firmware_product_name = "ert";
> > > + firmware_branch_name = "v20";
> > > + firmware_version_major = <0x01>;
> > > + };
> > > + };
> > > +
> > > + ep_ert_intc_00 {
> > > + reg = <0x00 0x23000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > > + interrupts = <0x05 0x05>;
> > > + };
> > > +
> > > + ep_ert_reset_00 {
> > > + reg = <0x00 0x22000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_ert_sched_00 {
> > > + reg = <0x00 0x50000 0x00 0x1000>;
> > > + pcie_physical_function = <0x01>;
> > > + compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
> > > + interrupts = <0x09 0x0c>;
> > > + };
> > > +
> > > + ep_fpga_configuration_00 {
> > > + reg = <0x00 0x1e88000 0x00 0x8000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_hwicap-
> > 1.0\0axi_hwicap";
> > > + interrupts = <0x02 0x02>;
> > > + };
> > > +
> > > + ep_icap_reset_00 {
> > > + reg = <0x00 0x1f07000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_msix_00 {
> > > + reg = <0x00 0x00 0x00 0x20000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> > > + pcie_bar_mapping = <0x02>;
> > > + };
> > > +
> > > + ep_pcie_link_mon_00 {
> > > + reg = <0x00 0x1f05000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_pr_isolate_plp_00 {
> > > + reg = <0x00 0x1f01000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_pr_isolate_ulp_00 {
> > > + reg = <0x00 0x1000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
> > > + };
> > > +
> > > + ep_uuid_rom_00 {
> > > + reg = <0x00 0x64000 0x00 0x1000>;
> > > + pcie_physical_function = <0x00>;
> > > + compatible = "xilinx.com,reg_abs-axi_bram_ctrl-
> > 1.0\0axi_bram_ctrl";
> > > + };
> > > +
> > > + ep_xdma_00 {
> > > + reg = <0x00 0x00 0x00 0x10000>;
> > > + pcie_physical_function = <0x01>;
> > > + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> > > + pcie_bar_mapping = <0x02>;
> > > + };
> > > + };
> > > +
> > > + }
> > > +
> > > +
> > > +
> > > +Deployment Models
> > > +=================
> > > +
> > > +Baremetal
> > > +---------
> > > +
> > > +In bare-metal deployments both MPF and UPF are visible and
> > > +accessible. xmgmt
> > In bare-meta deployments,
> > > +driver binds to MPF. xmgmt driver operations are privileged and
> > > +available to system administrator. The full stack is illustrated below::
> > > +
> > > + HOST
> > > +
> > > + [XMGMT] [XUSER]
> > > + | |
> > > + | |
> > > + +-----+ +-----+
> > > + | MPF | | UPF |
> > > + | | | |
> > > + | PF0 | | PF1 |
> > > + +--+--+ +--+--+
> > > + ......... ^................. ^..........
> > > + | |
> > > + | PCIe DEVICE |
> > > + | |
> > > + +--+------------------+--+
> > > + | SHELL |
> > > + | |
> > > + +------------------------+
> > > + | USER |
> > > + | |
> > > + | |
> > > + | |
> > > + | |
> > > + +------------------------+
> > > +
> > > +
> > > +
> > > +Virtualized
> > > +-----------
> > > +
> > > +In virtualized deployments privileged MPF is assigned to host but
> > > +unprivileged
> > In virtualized deployments, the
> > > +UPF is assigned to guest VM via PCIe pass-through. xmgmt driver in
> > > +host binds
> > in the host
> > > +to MPF. xmgmt driver operations are privileged and only accessible by
> > > +hosting
> > to the MPF
> > > +service provider. The full stack is illustrated below::
> > > +
> > > +
> > > + .............
> > > + HOST . VM .
> > > + . .
> > > + [XMGMT] . [XUSER] .
> > > + | . | .
> > > + | . | .
> > > + +-----+ . +-----+ .
> > > + | MPF | . | UPF | .
> > > + | | . | | .
> > > + | PF0 | . | PF1 | .
> > > + +--+--+ . +--+--+ .
> > > + ......... ^................. ^..........
> > > + | |
> > > + | PCIe DEVICE |
> > > + | |
> > > + +--+------------------+--+
> > > + | SHELL |
> > > + | |
> > > + +------------------------+
> > > + | USER |
> > > + | |
> > > + | |
> > > + | |
> > > + | |
> > > + +------------------------+
> > > +
> > > +
> > > +
> > > +
> > > +
> > > +Platform Security Considerations
> > > +================================
> > > +
> > > +`Security of Alveo Platform
> > > +<https://xilinx.github.io/XRT/master/html/security.html>`_
> > > +discusses the deployment options and security implications in great detail.
> >
> > This link works and looks great.
> >
> > Tom
>
> Thanks for the detailed review of the document. I am working on incorporating the
> feedback. One outstanding question is about usage of formal bus in XRT and if we
> should phase that in as a follow-on. It would also determine if IP drivers should
> move to other subsystems.
>
> -Sonal
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* RE: [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a document describing XRT Alveo drivers
2021-03-06 17:19 ` Moritz Fischer
@ 2021-03-08 20:12 ` Sonal Santan
0 siblings, 0 replies; 87+ messages in thread
From: Sonal Santan @ 2021-03-08 20:12 UTC (permalink / raw)
To: Moritz Fischer
Cc: Tom Rix, Lizhi Hou, linux-kernel, linux-fpga, Max Zhen,
Michal Simek, Stefano Stabellini, devicetree, robh
Hello Moritz,
> -----Original Message-----
> From: Moritz Fischer <mdf@kernel.org>
> Sent: Saturday, March 6, 2021 9:19 AM
> To: Sonal Santan <sonals@xilinx.com>
> Cc: Tom Rix <trix@redhat.com>; Lizhi Hou <lizhih@xilinx.com>; linux-
> kernel@vger.kernel.org; linux-fpga@vger.kernel.org; Max Zhen
> <maxz@xilinx.com>; Michal Simek <michals@xilinx.com>; Stefano Stabellini
> <stefanos@xilinx.com>; devicetree@vger.kernel.org; mdf@kernel.org;
> robh@kernel.org
> Subject: Re: [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a
> document describing XRT Alveo drivers
>
> On Mon, Mar 01, 2021 at 06:48:46AM +0000, Sonal Santan wrote:
> > Hello Tom,
> >
> > > -----Original Message-----
> > > From: Tom Rix <trix@redhat.com>
> > > Sent: Friday, February 19, 2021 2:26 PM
> > > To: Lizhi Hou <lizhih@xilinx.com>; linux-kernel@vger.kernel.org
> > > Cc: Lizhi Hou <lizhih@xilinx.com>; linux-fpga@vger.kernel.org; Max
> > > Zhen <maxz@xilinx.com>; Sonal Santan <sonals@xilinx.com>; Michal
> > > Simek <michals@xilinx.com>; Stefano Stabellini
> > > <stefanos@xilinx.com>; devicetree@vger.kernel.org; mdf@kernel.org;
> > > robh@kernel.org; Max Zhen <maxz@xilinx.com>
> > > Subject: Re: [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a
> > > document describing XRT Alveo drivers
> > >
> > > From the documentation, there are a couple of big questions and a
> > > bunch of word smithing.
> > >
> > > pseudo-bus : do we need a bus ?
> > We are looking for guidance here.
> > >
> > > xrt-lib real platform devices that aren't fpga, do they need to move
> > > to another subsystem ?
> > >
> >
> > Drivers for the IPs that show up in the Alveo shell are not generic
> > enough. They fit into the framework that XRT uses. Is the idea that
> > that they can be used in a different context?
> >
> > > Overall looks good, love the ascii art!
> > >
> > > On 2/17/21 10:40 PM, Lizhi Hou wrote:
> > > > Describe XRT driver architecture and provide basic overview of
> > > > Xilinx Alveo platform.
> > > >
> > > > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > > > Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> > > > Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> > > > ---
> > > > Documentation/fpga/index.rst | 1 +
> > > > Documentation/fpga/xrt.rst | 842
> > > +++++++++++++++++++++++++++++++++++
> > > > 2 files changed, 843 insertions(+) create mode 100644
> > > > Documentation/fpga/xrt.rst
> > > >
> > > > diff --git a/Documentation/fpga/index.rst
> > > > b/Documentation/fpga/index.rst index f80f95667ca2..30134357b70d
> > > > 100644
> > > > --- a/Documentation/fpga/index.rst
> > > > +++ b/Documentation/fpga/index.rst
> > > > @@ -8,6 +8,7 @@ fpga
> > > > :maxdepth: 1
> > > >
> > > > dfl
> > > > + xrt
> > > >
> > > > .. only:: subproject and html
> > > >
> > > > diff --git a/Documentation/fpga/xrt.rst
> > > > b/Documentation/fpga/xrt.rst new file mode 100644 index
> > > > 000000000000..9bc2d2785cb9
> > > > --- /dev/null
> > > > +++ b/Documentation/fpga/xrt.rst
> > > > @@ -0,0 +1,842 @@
> > > > +.. SPDX-License-Identifier: GPL-2.0
> > > > +
> > > > +==================================
> > > > +XRTV2 Linux Kernel Driver Overview
> > > > +==================================
> > > > +
> > > > +Authors:
> > > > +
> > > > +* Sonal Santan <sonal.santan@xilinx.com>
> > > > +* Max Zhen <max.zhen@xilinx.com>
> > > > +* Lizhi Hou <lizhi.hou@xilinx.com>
> > > > +
> > > > +XRTV2 drivers are second generation `XRT
> > > > +<https://github.com/Xilinx/XRT>`_ drivers which support `Alveo
> > > > +<https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
> > > > +PCIe platforms from Xilinx.
> > > > +
> > > > +XRTV2 drivers support *subsystem* style data driven platforms
> > > > +where driver's
> > > where the driver's
> > > > +configuration and behavior is determined by meta data provided by
> > > > +the platform (in *device tree* format). Primary management
> > > > +physical function (MPF) driver is called **xmgmt**. Primary user
> > > > +physical function (UPF) driver is called
> > > > +**xuser** and is under development. xrt driver framework and HW
> > > > +subsystem drivers are packaged into a library module called
> > > > +**xrt-lib**, which is shared by **xmgmt** and **xuser** (under
> > > > +development). The xrt driver framework
> > > xuser still under development ?
> > > > +implements a pseudo-bus which is used to discover HW subsystems
> > > > +and facilitate
> > >
> > > A pseudo-bus.
> > >
> > > It would be good if this was close to what was done for dfl here
> > >
> > > https://lore.kernel.org/linux-fpga/1605159759-3439-1-git-send-email-
> > > yilun.xu@intel.com/
> > >
> >
> > I am wondering if we can phase in the migration to formal bus
> > architecture based on struct bus_type as a follow on set of patches?
>
> I'd rather have it done early on. If we know there's a better way of doing
> something and we don't the code should go into staging.
>
> This gives us an out if it doesn't happen, otherwise the kernel community would
> depend on corporate goodwill to appropriately staff it.
>
> Note, I'm not trying to say there's any ill will anywhere, Xilinx has been
> traditionally good about this :)
Thanks for the suggestion. Since the bus change is a bigger change we will
incorporate it into V5 version of the patch series. Meanwhile we will post V4
version of patch series which will addresses other feedback we have received
so far.
> >
> > > > +inter HW subsystem interaction.
> > > > +
> > > > +Driver Modules
> > > > +==============
> > > > +
> > > > +xrt-lib.ko
> > > > +----------
> > > > +
> > > > +Repository of all subsystem drivers and pure software modules
> > > > +that can potentially
> > >
> > > subsystem drivers
> > >
> > > drivers in fpga/ should be for managing just the fpganess of the fpga.
> > >
> > > soft devices ex/ a soft tty should go to their respective subsystem
> > > location
> > >
> > > Are there any in this patchset you think might move ?
> >
> > We have already shrunk the patch to only include FPGA centric pieces
> > necessary to get the bitstream download implemented. Should we explore
> > the question of subsystem drivers when we add support for more
> > features of the Alveo shell?
> >
> > >
> > > Maybe we can defer reviewing those now.
> > >
> > > > +be shared between xmgmt and xuser. All these drivers are
> > > > +structured as Linux *platform driver* and are instantiated by
> > > > +xmgmt (or xuser under development) based on meta data associated
> > > > +with hardware. The metadata is in the form of device tree
> > >
> > > with the hardware
> > >
> > > form of a device tree
> > >
> >
> > Will change
> >
> > > > +as mentioned before. Each platform driver statically defines a
> > > > +subsystem node array by using node name or a string in its
> > > > +``compatible`` property. And this array is eventually translated
> > > > +to IOMEM
> > > resources of the platform device.
> > > > +
> > > > +The xrt-lib core infrastructure provides hooks to platform
> > > > +drivers for device node management, user file operations and ioctl
> callbacks.
> > > > +The core also provides pseudo-bus functionality for platform
> > > > +driver registration, discovery and inter platform driver ioctl calls.
> > >
> > > core infrastructure.
> > >
> >
> > Will update.
> >
> > > The interfaces to the infrastructure are not in include/linux/fpga/
> > >
> > > Maybe this needs to change.
> > >
> >
> > Were you thinking of moving XRT infrastructure header files from
> > drivers/fpga/xrt/include to include/linux/fpga?
> >
> > > > +
> > > > +.. note::
> > > > + See code in ``include/xleaf.h``
> > > > +
> > > > +
> > > > +xmgmt.ko
> > > > +--------
> > > > +
> > > > +The xmgmt driver is a PCIe device driver driving MPF found on
> > > > +Xilinx's Alveo PCIE device. It consists of one *root* driver, one
> > > > +or more *group* drivers and one or more *xleaf* drivers. The root
> > > > +and MPF specific xleaf drivers are in xmgmt.ko. The group driver
> > > > +and other xleaf
> > > drivers are in xrt-lib.ko.
> > > I am not sure if *.ko is correct, these will also be intree.
> > > > +
> > > > +The instantiation of specific group driver or xleaf driver is
> > > > +completely data
> > > of a specific
> > > > +driven based on meta data (mostly in device tree format) found
> > > > +through VSEC
> > > mostly ? what is the deviation from device tree ?
> > > > +capability and inside firmware files, such as platform xsabin or
> > > > +user xclbin
> > > file.
> > > > +The root driver manages life cycle of multiple group drivers,
> > > > +which, in turn,
> > > the life cycle
> > > > +manages multiple xleaf drivers. This allows a single set of
> > > > +driver code to support
> > >
> > > set of drivers
> > >
> > > drop 'code'
> > >
> >
> > Will update
> >
> > > > +all kinds of subsystems exposed by different shells. The
> > > > +difference among all these subsystems will be handled in xleaf
> > > > +drivers with root and group drivers being part of the
> > > > +infrastructure and provide common services for all leaves found on all
> platforms.
> > > > +
> > > > +The driver object model looks like the following::
> > > > +
> > > > + +-----------+
> > > > + | xroot |
> > > > + +-----+-----+
> > > > + |
> > > > + +-----------+-----------+
> > > > + | |
> > > > + v v
> > > > + +-----------+ +-----------+
> > > > + | group | ... | group |
> > > > + +-----+-----+ +------+----+
> > > > + | |
> > > > + | |
> > > > + +-----+----+ +-----+----+
> > > > + | | | |
> > > > + v v v v
> > > > + +-------+ +-------+ +-------+ +-------+
> > > > + | xleaf |..| xleaf | | xleaf |..| xleaf |
> > > > + +-------+ +-------+ +-------+ +-------+
> > > > +
> > > > +As an example for Xilinx Alveo U50 before user xclbin download,
> > > > +the tree looks like the following::
> > > > +
> > > > + +-----------+
> > > > + | xmgmt |
> > > > + +-----+-----+
> > > > + |
> > > > + +-------------------------+--------------------+
> > > > + | | |
> > > > + v v v
> > > > + +--------+ +--------+ +--------+
> > > > + | group0 | | group1 | | group2 |
> > > > + +----+---+ +----+---+ +---+----+
> > > > + | | |
> > > > + | | |
> > > > + +-----+-----+ +----+-----+---+ +-----+-----+----+--------+
> > > > + | | | | | | | | |
> > > > + v v | v v | v v |
> > > > + +------------+ +------+ | +------+ +------+ | +------+
> > > > + +------------+ +-----------+ |
> > > > + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | |
> > > > + | AXI-GATE0 | |
> > > > + +------------+ +------+ | +------+ +------+ | +------+
> > > > + +------------+ +-----------+ |
> > > > + | +---------+ | +------+ +-----------+ |
> > > > + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+
> > > > + +---------+ | +------+ +-----------+
> > > > + | +-------+
> > > > + +->| CALIB |
> > > > + +-------+
> > > > +
> > > Nice ascii art!
> > > > +After an xclbin is download, group3 will be added and the tree
> > > > +looks like the
> > > > +following::
> > > > +
> > > > + +-----------+
> > > > + | xmgmt |
> > > > + +-----+-----+
> > > > + |
> > > > + +-------------------------+--------------------+-----------------+
> > > > + | | | |
> > > > + v v v |
> > > > + +--------+ +--------+ +--------+ |
> > > > + | group0 | | group1 | | group2 | |
> > > > + +----+---+ +----+---+ +---+----+ |
> > > > + | | | |
> > > > + | | | |
> > > > + +-----+-----+ +-----+-----+---+ +-----+-----+----+--------+ |
> > > > + | | | | | | | | | |
> > > > + v v | v v | v v | |
> > > > + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
> > > > + | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | |
> > > > + | AXI-GATE0 | |
> > > |
> > > > + +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
> > > > + | +---------+ | +------+ +-----------+ | |
> > > > + +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+ |
> > > > + +---------+ | +------+ +-----------+ |
> > > > + | +-------+ |
> > > > + +->| CALIB | |
> > > > + +-------+ |
> > > > + +---+----+ |
> > > > + | group3 |<--------------------------------------------+
> > > > + +--------+
> > > > + |
> > > > + |
> > > > + +-------+--------+---+--+--------+------+-------+
> > > > + | | | | | | |
> > > > + v | v | v | v
> > > > + +--------+ | +--------+ | +--------+ | +-----+
> > > > + | CLOCK0 | | | CLOCK1 | | | CLOCK2 | | | UCS |
> > > > + +--------+ v +--------+ v +--------+ v +-----+
> > > > + +-------------+ +-------------+ +-------------+
> > > > + | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
> > > > + +-------------+ +-------------+ +-------------+
> > > > +
> > > > +
> > > > +xmgmt-root
> > > > +^^^^^^^^^^
> > > > +
> > > > +The xmgmt-root driver is a PCIe device driver attached to MPF.
> > > > +It's part of the infrastructure of the MPF driver and resides in xmgmt.ko.
> > > > +This driver
> > > > +
> > > > +* manages one or more group drivers
> > > > +* provides access to functionalities that requires pci_dev, such
> > > > +as PCIE config
> > > > + space access, to other xleaf drivers through root calls
> > > > +* together with group driver, facilities event callbacks for
> > > > +other xleaf drivers
> > > > +* together with group driver, facilities inter-leaf driver calls
> > > > +for other xleaf
> > > Maybe drop 'together with group driver'
> >
> > Will update
> >
> > > > + drivers
> > > > +
> > > > +When root driver starts, it will explicitly create an initial
> > > > +group instance, which contains xleaf drivers that will trigger
> > > > +the creation of other group instances. The root driver will wait
> > > > +for all group and leaves to be created before it returns from
> > > > +it's probe routine and claim success of the initialization of the entire
> xmgmt driver.
> > > What happens if there a failure in one leaf ? Does the whole board go down
> ?
> > > > +
> > > > +.. note::
> > > > + See code in ``lib/xroot.c`` and ``mgmt/root.c``
> > > > +
> > > > +
> > > > +group
> > > > +^^^^^
> > > > +
> > > > +The group driver is a platform device driver whose life cycle is
> > > > +managed by
> > > Maybe call this a 'pseudo device'
> >
> > Will update
> >
> > > > +root and does not have real IO mem or IRQ resources. It's part of
> > > > +the infrastructure of the MPF driver and resides in xrt-lib.ko.
> > > > +This driver
> > > > +
> > > > +* manages one or more xleaf drivers so that multiple leaves can
> > > > +be managed as a
> > > > + group
> > > can drop 'so that multiple leaves can be managed as a group' to me,
> > > this is the same as 'one or more'
> >
> > Will do
> >
> > > > +* provides access to root from leaves, so that root calls, event
> > > > +notifications
> > > > + and inter-leaf calls can happen
> > > > +
> > > > +In xmgmt, an initial group driver instance will be created by
> > > > +root, which
> > > by the root
> > > > +contains leaves that will trigger group instances to be created
> > > > +to manage groups of leaves found on different partitions on
> > > > +hardware, such as VSEC, Shell, and User.
> > > > +
> > > > +Every *fpga_region* has a group object associated with it. The
> > > > +group is created when xclbin image is loaded on the fpga_region.
> > > > +The existing group is destroyed when a new xclbin image is
> > > > +loaded. The fpga_region persists across xclbin downloads.
> > > The connection of a 'group' node to a fpga region region is fairly
> > > important, maybe move this section earlier. 'group' as an fpganess
> > > thing would be kept in fpga/ subsystem.
> >
> > Will update
> >
> > > > +
> > > > +.. note::
> > > > + See code in ``lib/group.c``
> > > > +
> > > > +
> > > > +xleaf
> > > > +^^^^^
> > > > +
> > > > +The xleaf driver is a platform device driver whose life cycle is
> > > > +managed by a group driver and may or may not have real IO mem or
> > > > +IRQ resources. They are the real meat of xmgmt and contains
> > > > +platform specific code to Shell and User found on a MPF.
> > > > +
> > >
> > > Maybe a split is pseudo device leaves, those without real IO mem,
> > > stay in fpga/ others go ?
> > >
> >
> > This goes back to the earlier question of what minimal set of platform
> > drivers should stay in fpga subsystem. There are some like bridge or
> > configuration engine (also called icap) which have their own IO mem
> > but do not have a life outside of fpga subsystem.
> >
> > > > +A xleaf driver may not have real hardware resources when it
> > > > +merely acts as a driver that manages certain in-memory states for
> xmgmt.
> > > > +These in-memory states could be shared by multiple other leaves.
> > > > +
> > > This implies locking and some message passing.
> > > > +Leaf drivers assigned to specific hardware resources drive
> > > > +specific subsystem in
> > > drive a specific
> > > > +the device. To manipulate the subsystem or carry out a task, a
> > > > +xleaf driver may ask help from root via root calls and/or from
> > > > +other leaves via
> > > inter-leaf calls.
> > > > +
> > > > +A xleaf can also broadcast events through infrastructure code for
> > > > +other leaves to process. It can also receive event notification
> > > > +from infrastructure about certain events, such as post-creation
> > > > +or pre-exit of a
> > > particular xleaf.
> > > I would like to see some examples of how the inter node communications
> work.
> >
> > Would update to show an example.
> >
> > > > +
> > > > +.. note::
> > > > + See code in ``lib/xleaf/*.c``
> > > > +
> > > > +
> > > > +FPGA Manager Interaction
> > > > +========================
> > > > +
> > > > +fpga_manager
> > > > +------------
> > > > +
> > > > +An instance of fpga_manager is created by xmgmt_main and is used
> > > > +for xclbin
> > > for the xclbin
> > > > +image download. fpga_manager requires the full xclbin image
> > > > +before it can start programming the FPGA configuration engine via
> > > > +ICAP platform
> > > driver.
> > >
> > > via the ICAP
> > >
> > > what is ICAP ?
> >
> > Will update. ICAP stands for Internal Configuration Access Port used
> > for configuring the fpga.
> >
> > >
> > > > +
> > > > +fpga_region
> > > > +-----------
> > > > +
> > > > +For every interface exposed by currently loaded xclbin/xsabin in
> > > > +the
> > > > +*parent*
> > > by the currently
> > > > +fpga_region a new instance of fpga_region is created like a *child*
> region.
> > > fpga_region,
> > > > +The device tree of the *parent* fpga_region defines the resources
> > > > +for a new instance of fpga_bridge which isolates the parent from
> > > and isolates
> > > > +child fpga_region. This new instance of fpga_bridge will be used
> > > > +when a xclbin image is loaded on the child fpga_region. After the
> > > > +xclbin image is downloaded to the fpga_region, an instance of
> > > > +group is created for the fpga_region using the device tree
> > > > +obtained as part of xclbin. If this device
> > > of the xclbin
> > > > +tree defines any child interfaces then it can trigger the
> > > > +creation of
> > > interfaces, then
> > > > +fpga_bridge and fpga_region for the next region in the chain.
> > > a fpga_bridge and a fpga_region
> > > > +
> > > > +fpga_bridge
> > > > +-----------
> > > > +
> > > > +Like fpga_region, matching fpga_bridge is also created by walking
> > > > +the device
> > > Like the fpga_region, a matchin
> > > > +tree of the parent group.
> > > > +
> > > > +Driver Interfaces
> > > > +=================
> > > > +
> > > > +xmgmt Driver Ioctls
> > > > +-------------------
> > > > +
> > > > +Ioctls exposed by xmgmt driver to user space are enumerated in
> > > > +the following
> > > > +table:
> > > > +
> > > > +== ===================== ============================
> > > ==========================
> > > > +# Functionality ioctl request code data format
> > > > +== ===================== ============================
> > > ==========================
> > > > +1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF
> > > xmgmt_ioc_bitstream_axlf
> > > > +== ===================== ============================
> > > > +==========================
> > >
> > > This data format is described below, maybe swap this section with
> > > that so
> > >
> > > folks will know what xmgmnt_ioc_bitstream_axlf is before this section.
> > >
> >
> > Will update.
> >
> > > > +
> > > > +User xclbin can be downloaded by using xbmgmt tool from XRT open
> > > > +source suite. See
> > >
> > > A user xclbin
> > >
> > > using the xbmgmt
> > >
> > > from the XRT
> > >
> >
> > Will update
> >
> > > > +example usage below::
> > > > +
> > > > + xbmgmt partition --program --path
> > > > + /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verif
> > > > + y.xc
> > > > + lbin --force
> > > > +
> > > > +xmgmt Driver Sysfs
> > > > +------------------
> > > > +
> > > > +xmgmt driver exposes a rich set of sysfs interfaces. Subsystem
> > > > +platform drivers export sysfs node for every platform instance.
> > > > +
> > > > +Every partition also exports its UUIDs. See below for examples::
> > > > +
> > > > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
> > > > + /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
> > > > +
> > > > +
> > > > +hwmon
> > > > +-----
> > > > +
> > > > +xmgmt driver exposes standard hwmon interface to report voltage,
> > > > +current, temperature, power, etc. These can easily be viewed
> > > > +using
> > > > +*sensors* command line utility.
> > > > +
> > > > +Alveo Platform Overview
> > > > +=======================
> > > > +
> > > > +Alveo platforms are architected as two physical FPGA partitions:
> > > > +*Shell* and *User*. The Shell provides basic infrastructure for
> > > > +the Alveo platform like PCIe connectivity, board management,
> > > > +Dynamic Function Exchange (DFX), sensors, clocking, reset, and
> > > > +security. User partition contains user compiled FPGA
> > > the user compiled
> > > > +binary which is loaded by a process called DFX also known as
> > > > +partial reconfiguration.
> > > > +
> > > > +Physical partitions require strict HW compatibility with each
> > > > +other for DFX to work properly.
> > >
> > > swap order
> > >
> > > For DFX to work properly physical partitions ..
> > >
> > >
> >
> > Will update
> >
> > > > Every physical partition has two interface UUIDs: *parent* UUID
> > > > +and *child* UUID. For simple single stage platforms, Shell → User
> > > > +forms parent child relationship. For complex two stage platforms,
> > > > +Base → Shell → User forms the parent child relationship chain.
> > > this bit is confusing. is this related to uuid?
> > > > +
> > > > +.. note::
> > > > + Partition compatibility matching is key design component of
> > > > +Alveo
> > > platforms
> > > > + and XRT. Partitions have child and parent relationship. A
> > > > +loaded partition
> > > have a child
> > > > + exposes child partition UUID to advertise its compatibility
> > > > + requirement for
> > >
> > > the child's
> > >
> > > can drop 'for child partition'
> >
> > Will update
> >
> > >
> > > > + child partition. When loading a child partition the xmgmt
> > > > + management driver
> > > When loading a child partition,
> > > > + matches parent UUID of the child partition against child UUID exported
> by
> > > > + the parent. Parent and child partition UUIDs are stored in the *xclbin*
> > > > + (for user) or *xsabin* (for base and shell).
> > >
> > > this is confusing, is this part of the file image format ?
> > >
> > > Maybe save/move till the image layout.
> >
> > Yes these IDs are stored in xclbin image format. Will move the
> > sections around as suggested.
> >
> > >
> > > > Except for root UUID, VSEC,
> > > > + hardware itself does not know about UUIDs. UUIDs are stored in
> > > > + xsabin
> > > and
> > > > + xclbin.
> > > This is confusing too, not sure how to untangle.
> >
> > Will reword.
> >
> > > > +
> > > > +
> > > > +The physical partitions and their loading is illustrated below::
> > > > +
> > > > + SHELL USER
> > > > + +-----------+ +-------------------+
> > > > + | | | |
> > > > + | VSEC UUID | CHILD PARENT | LOGIC UUID |
> > > > + | o------->|<--------o |
> > > > + | | UUID UUID | |
> > > > + +-----+-----+ +--------+----------+
> > > > + | |
> > > > + . .
> > > > + | |
> > > > + +---+---+ +------+--------+
> > > > + | POR | | USER COMPILED |
> > > > + | FLASH | | XCLBIN |
> > > > + +-------+ +---------------+
> > > > +
> > > > +
> > > > +Loading Sequence
> > > > +----------------
> > > > +
> > > > +The Shell partition is loaded from flash at system boot time. It
> > > > +establishes the PCIe link and exposes two physical functions to
> > > > +the BIOS. After OS boot, xmgmt
> > > the OS boots, the xmgmt
> > > > +driver attaches to PCIe physical function 0 exposed by the Shell
> > > > +and then looks for VSEC in PCIe extended configuration space.
> > > > +Using VSEC it determines the logic
> > >
> > > the PCIe
> > >
> > > The driver uses VSEC to determine the UUID of Shell. The UUID is
> > > also used to load a matching ...
> > >
> >
> > Will update
> >
> > > > +UUID of Shell and uses the UUID to load matching *xsabin* file
> > > > +from Linux firmware directory. The xsabin file contains metadata
> > > > +to discover peripherals that are part of Shell and firmware(s)
> > > > +for any
> > > embedded soft processors in Shell.
> > > the firmware needed for any ...
> >
> > Will update
> >
> > > > +
> > > > +The Shell exports child interface UUID which is used for
> > > > +compatibility check when
> > >
> > > export a child
> > >
> > > for a compatibility check
> > >
> >
> > Will update
> >
> > > > +loading user compiled xclbin over the User partition as part of DFX.
> > > > +When a user requests loading of a specific xclbin the xmgmt
> > > > +management driver reads the parent
> > > xclbin, the
> > > > +interface UUID specified in the xclbin and matches it with child
> > > > +interface UUID exported by Shell to determine if xclbin is
> > > > +compatible with the Shell. If match fails loading of xclbin is denied.
> > > > +
> > > > +xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl
> command.
> > > > +When loading xclbin, xmgmt driver performs the following
> > > > +*logical*
> > > operations:
> > > > +
> > > > +1. Copy xclbin from user to kernel memory 2. Sanity check the
> > > > +xclbin contents 3. Isolate the User partition 4. Download the
> > > > +bitstream using the FPGA config engine (ICAP) 5. De-isolate the
> > > > +User partition 6. Program the clocks (ClockWiz) driving the User
> > > > +partition
> > > maybe drop '(ClockWiz)'
> > > > +7. Wait for memory controller (MIG) calibration
> > > for the
> > > > +8. Return the loading status back to the caller
> > > > +
> > > > +`Platform Loading Overview
> > > > +<https://xilinx.github.io/XRT/master/html/platforms_partitions.ht
> > > > +ml>` _ provides more detailed information on platform loading.
> > > > +
> > > the link works.
> > > > +
> > > > +xsabin
> > > > +------
> > > > +
> > > > +Each Alveo platform comes packaged with its own xsabin. The
> > > > +xsabin is trusted
> > > is a trusted
> > > > +component of the platform. For format details refer to
> > > > +:ref:`xsabin_xclbin_container_format`
> > > > +below. xsabin contains basic information like UUIDs, platform
> > > > +name and metadata in the form of device tree. See
> > > > +:ref:`device_tree_usage`
> > > below for details and example.
> > > of a device
> > > > +
> > > > +xclbin
> > > > +------
> > > > +
> > > > +xclbin is compiled by end user using `Vitis
> > > > +<https://www.xilinx.com/products/design-tools/vitis/vitis-platfor
> > > > +m.ht
> > > > +ml>`_
> > > this link works, seems reasonable landing
> > > > +tool set from Xilinx. The xclbin contains sections describing
> > > > +user compiled acceleration engines/kernels, memory subsystems,
> > > > +clocking information etc. It also contains bitstream for the user
> > > > +partition, UUIDs, platform name, etc. xclbin uses
> > > bitstreams
> > > > +the same container format as xsabin which is described below.
> > > > +
> > > > +
> > > > +.. _xsabin_xclbin_container_format:
> > > > +
> > > > +xsabin/xclbin Container Format
> > > > +------------------------------
> > > > +
> > > > +xclbin/xsabin is ELF-like binary container format. It is
> > > > +structured as series of sections. There is a file header followed
> > > > +by several section headers which is followed by sections. A
> > > > +section header points to an actual section. There is an optional
> > > > +signature at the end. The
> > > format is defined by header file ``xclbin.h``.
> > > > +The following figure illustrates a typical xclbin::
> > > > +
> > > > +
> > > > + +---------------------+
> > > > + | |
> > > > + | HEADER |
> > > > + +---------------------+
> > > > + | SECTION HEADER |
> > > > + | |
> > > > + +---------------------+
> > > > + | ... |
> > > > + | |
> > > > + +---------------------+
> > > > + | SECTION HEADER |
> > > > + | |
> > > > + +---------------------+
> > > > + | SECTION |
> > > > + | |
> > > > + +---------------------+
> > > > + | ... |
> > > > + | |
> > > > + +---------------------+
> > > > + | SECTION |
> > > > + | |
> > > > + +---------------------+
> > > > + | SIGNATURE |
> > > > + | (OPTIONAL) |
> > > > + +---------------------+
> > > > +
> > > > +
> > > > +xclbin/xsabin files can be packaged, un-packaged and inspected
> > > > +using XRT utility called **xclbinutil**. xclbinutil is part of
> > > > +XRT open source software stack. The source code for xclbinutil
> > > > +can be found at
> > > > +https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/x
> > > > +clbi
> > > > +nutil
> > > > +
> > > Works, but maybe the location of a manpage or doc would be better.
> > > > +For example to enumerate the contents of a xclbin/xsabin use the
> > > > +*--info* switch as shown below::
> > > > +
> > > > +
> > > > + xclbinutil --info --input
> > > > + /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
> > > > + xclbinutil --info --input
> > > > + /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.
> > > > + xsab
> > > > + in
> > > > +
> > > > +
> > > > +.. _device_tree_usage:
> > > > +
> > > > +Device Tree Usage
> > > > +-----------------
> > > > +
> > > > +As mentioned previously xsabin stores metadata which advertise HW
> > > > +subsystems present in a partition. The metadata is stored in
> > > > +device tree
> > > format with well defined schema.
> > > > +XRT management driver uses this information to bind *platform
> > > > +drivers* to the subsystem instantiations. The platform drivers
> > > > +are found in **xrt-lib.ko** kernel module defined later.
> > > > +
> > > > +Logic UUID
> > > > +^^^^^^^^^^
> > > > +A partition is identified uniquely through ``logic_uuid`` property::
> > > > +
> > > > + /dts-v1/;
> > > > + / {
> > > > + logic_uuid = "0123456789abcdef0123456789abcdef";
> > > > + ...
> > > > + }
> > > > +
> > > > +Schema Version
> > > > +^^^^^^^^^^^^^^
> > > > +Schema version is defined through ``schema_version`` node. And it
> > > > +contains ``major`` and ``minor`` properties as below::
> > > > +
> > > > + /dts-v1/;
> > > > + / {
> > > > + schema_version {
> > > > + major = <0x01>;
> > > > + minor = <0x00>;
> > > > + };
> > > > + ...
> > > > + }
> > > > +
> > > > +Partition UUIDs
> > > > +^^^^^^^^^^^^^^^
> > > > +As said earlier, each partition may have parent and child UUIDs.
> > > > +These UUIDs are defined by ``interfaces`` node and
> > > > +``interface_uuid``
> > > property::
> > > > +
> > > > + /dts-v1/;
> > > > + / {
> > > > + interfaces {
> > > > + @0 {
> > > > + interface_uuid = "0123456789abcdef0123456789abcdef";
> > > > + };
> > > > + @1 {
> > > > + interface_uuid = "fedcba9876543210fedcba9876543210";
> > > > + };
> > > > + ...
> > > > + };
> > > > + ...
> > > > + }
> > > > +
> > > > +
> > > > +Subsystem Instantiations
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +Subsystem instantiations are captured as children of
> > > > +``addressable_endpoints``
> > > > +node::
> > > > +
> > > > + /dts-v1/;
> > > > + / {
> > > > + addressable_endpoints {
> > > > + abc {
> > > > + ...
> > > > + };
> > > > + def {
> > > > + ...
> > > > + };
> > > > + ...
> > > > + }
> > > > + }
> > > > +
> > > > +Subnode 'abc' and 'def' are the name of subsystem nodes
> > > > +
> > > > +Subsystem Node
> > > > +^^^^^^^^^^^^^^
> > > > +Each subsystem node and its properties define a hardware instance::
> > > > +
> > > > +
> > > > + addressable_endpoints {
> > > > + abc {
> > > > + reg = <0xa 0xb>
> > > > + pcie_physical_function = <0x0>;
> > > > + pcie_bar_mapping = <0x2>;
> > > > + compatible = "abc def";
> > > > + firmware {
> > > > + firmware_product_name = "abc"
> > > > + firmware_branch_name = "def"
> > > > + firmware_version_major = <1>
> > > > + firmware_version_minor = <2>
> > > > + };
> > > > + }
> > > > + ...
> > > > + }
> > > > +
> > > > +:reg:
> > > > + Property defines address range. '<0xa 0xb>' is BAR offset and
> > > > +length pair, both are 64-bit integer.
> > > > +:pcie_physical_function:
> > > > + Property specifies which PCIe physical function the subsystem node
> resides.
> > > > +:pcie_bar_mapping:
> > > > + Property specifies which PCIe BAR the subsystem node resides.
> > > > +'<0x2>' is BAR index and it is 0 if this property is not defined.
> > > > +:compatible:
> > > > + Property is a list of strings. The first string in the list
> > > > +specifies the exact subsystem node. The following strings
> > > > +represent other devices that the device is compatible with.
> > > > +:firmware:
> > > > + Subnode defines the firmware required by this subsystem node.
> > > > +
> > > > +Alveo U50 Platform Example
> > > > +^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > > +::
> > > > +
> > > > + /dts-v1/;
> > > > +
> > > > + /{
> > > > + logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
> > > > +
> > > > + schema_version {
> > > > + major = <0x01>;
> > > > + minor = <0x00>;
> > > > + };
> > > > +
> > > > + interfaces {
> > > > +
> > > > + @0 {
> > > > + interface_uuid = "862c7020a250293e32036f19956669e5";
> > > > + };
> > > > + };
> > > > +
> > > > + addressable_endpoints {
> > > > +
> > > > + ep_blp_rom_00 {
> > > > + reg = <0x00 0x1f04000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-axi_bram_ctrl-
> > > 1.0\0axi_bram_ctrl";
> > > > + };
> > > > +
> > > > + ep_card_flash_program_00 {
> > > > + reg = <0x00 0x1f06000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-axi_quad_spi-
> > > 1.0\0axi_quad_spi";
> > > > + interrupts = <0x03 0x03>;
> > > > + };
> > > > +
> > > > + ep_cmc_firmware_mem_00 {
> > > > + reg = <0x00 0x1e20000 0x00 0x20000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > > +
> > > > + firmware {
> > > > + firmware_product_name = "cmc";
> > > > + firmware_branch_name = "u50";
> > > > + firmware_version_major = <0x01>;
> > > > + firmware_version_minor = <0x00>;
> > > > + };
> > > > + };
> > > > +
> > > > + ep_cmc_intc_00 {
> > > > + reg = <0x00 0x1e03000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > > > + interrupts = <0x04 0x04>;
> > > > + };
> > > > +
> > > > + ep_cmc_mutex_00 {
> > > > + reg = <0x00 0x1e02000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_cmc_regmap_00 {
> > > > + reg = <0x00 0x1e08000 0x00 0x2000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > > +
> > > > + firmware {
> > > > + firmware_product_name = "sc-fw";
> > > > + firmware_branch_name = "u50";
> > > > + firmware_version_major = <0x05>;
> > > > + };
> > > > + };
> > > > +
> > > > + ep_cmc_reset_00 {
> > > > + reg = <0x00 0x1e01000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_ddr_mem_calib_00 {
> > > > + reg = <0x00 0x63000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_debug_bscan_mgmt_00 {
> > > > + reg = <0x00 0x1e90000 0x00 0x10000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-debug_bridge-
> > > 1.0\0debug_bridge";
> > > > + };
> > > > +
> > > > + ep_ert_base_address_00 {
> > > > + reg = <0x00 0x21000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_ert_command_queue_mgmt_00 {
> > > > + reg = <0x00 0x40000 0x00 0x10000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-ert_command_queue-
> > > 1.0\0ert_command_queue";
> > > > + };
> > > > +
> > > > + ep_ert_command_queue_user_00 {
> > > > + reg = <0x00 0x40000 0x00 0x10000>;
> > > > + pcie_physical_function = <0x01>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-ert_command_queue-
> > > 1.0\0ert_command_queue";
> > > > + };
> > > > +
> > > > + ep_ert_firmware_mem_00 {
> > > > + reg = <0x00 0x30000 0x00 0x8000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
> > > > +
> > > > + firmware {
> > > > + firmware_product_name = "ert";
> > > > + firmware_branch_name = "v20";
> > > > + firmware_version_major = <0x01>;
> > > > + };
> > > > + };
> > > > +
> > > > + ep_ert_intc_00 {
> > > > + reg = <0x00 0x23000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
> > > > + interrupts = <0x05 0x05>;
> > > > + };
> > > > +
> > > > + ep_ert_reset_00 {
> > > > + reg = <0x00 0x22000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_ert_sched_00 {
> > > > + reg = <0x00 0x50000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x01>;
> > > > + compatible = "xilinx.com,reg_abs-ert_sched-
> 1.0\0ert_sched";
> > > > + interrupts = <0x09 0x0c>;
> > > > + };
> > > > +
> > > > + ep_fpga_configuration_00 {
> > > > + reg = <0x00 0x1e88000 0x00 0x8000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-axi_hwicap-
> > > 1.0\0axi_hwicap";
> > > > + interrupts = <0x02 0x02>;
> > > > + };
> > > > +
> > > > + ep_icap_reset_00 {
> > > > + reg = <0x00 0x1f07000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_msix_00 {
> > > > + reg = <0x00 0x00 0x00 0x20000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
> > > > + pcie_bar_mapping = <0x02>;
> > > > + };
> > > > +
> > > > + ep_pcie_link_mon_00 {
> > > > + reg = <0x00 0x1f05000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_pr_isolate_plp_00 {
> > > > + reg = <0x00 0x1f01000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_pr_isolate_ulp_00 {
> > > > + reg = <0x00 0x1000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible = "xilinx.com,reg_abs-axi_gpio-
> 1.0\0axi_gpio";
> > > > + };
> > > > +
> > > > + ep_uuid_rom_00 {
> > > > + reg = <0x00 0x64000 0x00 0x1000>;
> > > > + pcie_physical_function = <0x00>;
> > > > + compatible =
> > > > + "xilinx.com,reg_abs-axi_bram_ctrl-
> > > 1.0\0axi_bram_ctrl";
> > > > + };
> > > > +
> > > > + ep_xdma_00 {
> > > > + reg = <0x00 0x00 0x00 0x10000>;
> > > > + pcie_physical_function = <0x01>;
> > > > + compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
> > > > + pcie_bar_mapping = <0x02>;
> > > > + };
> > > > + };
> > > > +
> > > > + }
> > > > +
> > > > +
> > > > +
> > > > +Deployment Models
> > > > +=================
> > > > +
> > > > +Baremetal
> > > > +---------
> > > > +
> > > > +In bare-metal deployments both MPF and UPF are visible and
> > > > +accessible. xmgmt
> > > In bare-meta deployments,
> > > > +driver binds to MPF. xmgmt driver operations are privileged and
> > > > +available to system administrator. The full stack is illustrated below::
> > > > +
> > > > + HOST
> > > > +
> > > > + [XMGMT] [XUSER]
> > > > + | |
> > > > + | |
> > > > + +-----+ +-----+
> > > > + | MPF | | UPF |
> > > > + | | | |
> > > > + | PF0 | | PF1 |
> > > > + +--+--+ +--+--+
> > > > + ......... ^................. ^..........
> > > > + | |
> > > > + | PCIe DEVICE |
> > > > + | |
> > > > + +--+------------------+--+
> > > > + | SHELL |
> > > > + | |
> > > > + +------------------------+
> > > > + | USER |
> > > > + | |
> > > > + | |
> > > > + | |
> > > > + | |
> > > > + +------------------------+
> > > > +
> > > > +
> > > > +
> > > > +Virtualized
> > > > +-----------
> > > > +
> > > > +In virtualized deployments privileged MPF is assigned to host but
> > > > +unprivileged
> > > In virtualized deployments, the
> > > > +UPF is assigned to guest VM via PCIe pass-through. xmgmt driver
> > > > +in host binds
> > > in the host
> > > > +to MPF. xmgmt driver operations are privileged and only
> > > > +accessible by hosting
> > > to the MPF
> > > > +service provider. The full stack is illustrated below::
> > > > +
> > > > +
> > > > + .............
> > > > + HOST . VM .
> > > > + . .
> > > > + [XMGMT] . [XUSER] .
> > > > + | . | .
> > > > + | . | .
> > > > + +-----+ . +-----+ .
> > > > + | MPF | . | UPF | .
> > > > + | | . | | .
> > > > + | PF0 | . | PF1 | .
> > > > + +--+--+ . +--+--+ .
> > > > + ......... ^................. ^..........
> > > > + | |
> > > > + | PCIe DEVICE |
> > > > + | |
> > > > + +--+------------------+--+
> > > > + | SHELL |
> > > > + | |
> > > > + +------------------------+
> > > > + | USER |
> > > > + | |
> > > > + | |
> > > > + | |
> > > > + | |
> > > > + +------------------------+
> > > > +
> > > > +
> > > > +
> > > > +
> > > > +
> > > > +Platform Security Considerations
> > > > +================================
> > > > +
> > > > +`Security of Alveo Platform
> > > > +<https://xilinx.github.io/XRT/master/html/security.html>`_
> > > > +discusses the deployment options and security implications in great
> detail.
> > >
> > > This link works and looks great.
> > >
> > > Tom
> >
> > Thanks for the detailed review of the document. I am working on
> > incorporating the feedback. One outstanding question is about usage of
> > formal bus in XRT and if we should phase that in as a follow-on. It
> > would also determine if IP drivers should move to other subsystems.
> >
> > -Sonal
>
> - Moritz
-Sonal
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a document describing XRT Alveo drivers Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-20 17:07 ` Tom Rix
2021-02-23 1:23 ` Fernando Pacheco
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file " Lizhi Hou
` (17 subsequent siblings)
19 siblings, 2 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
XRT drivers use device tree as metadata format to discover HW subsystems
behind PCIe BAR. Thus libfdt functions are called for driver to parse
device tree blob.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/metadata.h | 229 ++++++++++++
drivers/fpga/xrt/metadata/metadata.c | 524 +++++++++++++++++++++++++++
2 files changed, 753 insertions(+)
create mode 100644 drivers/fpga/xrt/include/metadata.h
create mode 100644 drivers/fpga/xrt/metadata/metadata.c
diff --git a/drivers/fpga/xrt/include/metadata.h b/drivers/fpga/xrt/include/metadata.h
new file mode 100644
index 000000000000..b929bc469b73
--- /dev/null
+++ b/drivers/fpga/xrt/include/metadata.h
@@ -0,0 +1,229 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_METADATA_H
+#define _XRT_METADATA_H
+
+#include <linux/device.h>
+#include <linux/vmalloc.h>
+#include <linux/uuid.h>
+
+#define XRT_MD_INVALID_LENGTH (~0UL)
+
+#define XRT_MD_PROP_COMPATIBLE "compatible"
+#define XRT_MD_PROP_PF_NUM "pcie_physical_function"
+#define XRT_MD_PROP_BAR_IDX "pcie_bar_mapping"
+#define XRT_MD_PROP_IO_OFFSET "reg"
+#define XRT_MD_PROP_INTERRUPTS "interrupts"
+#define XRT_MD_PROP_INTERFACE_UUID "interface_uuid"
+#define XRT_MD_PROP_LOGIC_UUID "logic_uuid"
+#define XRT_MD_PROP_VERSION_MAJOR "firmware_version_major"
+
+#define XRT_MD_PROP_HWICAP "axi_hwicap"
+#define XRT_MD_PROP_PDI_CONFIG "pdi_config_mem"
+
+#define XRT_MD_NODE_ENDPOINTS "addressable_endpoints"
+#define XRT_MD_INTERFACES_PATH "/interfaces"
+
+#define XRT_MD_NODE_FIRMWARE "firmware"
+#define XRT_MD_NODE_INTERFACES "interfaces"
+#define XRT_MD_NODE_PARTITION_INFO "partition_info"
+
+#define XRT_MD_NODE_FLASH "ep_card_flash_program_00"
+#define XRT_MD_NODE_XVC_PUB "ep_debug_bscan_user_00"
+#define XRT_MD_NODE_XVC_PRI "ep_debug_bscan_mgmt_00"
+#define XRT_MD_NODE_SYSMON "ep_cmp_sysmon_00"
+#define XRT_MD_NODE_AF_BLP_CTRL_MGMT "ep_firewall_blp_ctrl_mgmt_00"
+#define XRT_MD_NODE_AF_BLP_CTRL_USER "ep_firewall_blp_ctrl_user_00"
+#define XRT_MD_NODE_AF_CTRL_MGMT "ep_firewall_ctrl_mgmt_00"
+#define XRT_MD_NODE_AF_CTRL_USER "ep_firewall_ctrl_user_00"
+#define XRT_MD_NODE_AF_CTRL_DEBUG "ep_firewall_ctrl_debug_00"
+#define XRT_MD_NODE_AF_DATA_H2C "ep_firewall_data_h2c_00"
+#define XRT_MD_NODE_AF_DATA_C2H "ep_firewall_data_c2h_00"
+#define XRT_MD_NODE_AF_DATA_P2P "ep_firewall_data_p2p_00"
+#define XRT_MD_NODE_AF_DATA_M2M "ep_firewall_data_m2m_00"
+#define XRT_MD_NODE_CMC_REG "ep_cmc_regmap_00"
+#define XRT_MD_NODE_CMC_RESET "ep_cmc_reset_00"
+#define XRT_MD_NODE_CMC_MUTEX "ep_cmc_mutex_00"
+#define XRT_MD_NODE_CMC_FW_MEM "ep_cmc_firmware_mem_00"
+#define XRT_MD_NODE_ERT_FW_MEM "ep_ert_firmware_mem_00"
+#define XRT_MD_NODE_ERT_CQ_MGMT "ep_ert_command_queue_mgmt_00"
+#define XRT_MD_NODE_ERT_CQ_USER "ep_ert_command_queue_user_00"
+#define XRT_MD_NODE_MAILBOX_MGMT "ep_mailbox_mgmt_00"
+#define XRT_MD_NODE_MAILBOX_USER "ep_mailbox_user_00"
+#define XRT_MD_NODE_GATE_PLP "ep_pr_isolate_plp_00"
+#define XRT_MD_NODE_GATE_ULP "ep_pr_isolate_ulp_00"
+#define XRT_MD_NODE_PCIE_MON "ep_pcie_link_mon_00"
+#define XRT_MD_NODE_DDR_CALIB "ep_ddr_mem_calib_00"
+#define XRT_MD_NODE_CLK_KERNEL1 "ep_aclk_kernel_00"
+#define XRT_MD_NODE_CLK_KERNEL2 "ep_aclk_kernel_01"
+#define XRT_MD_NODE_CLK_KERNEL3 "ep_aclk_hbm_00"
+#define XRT_MD_NODE_KDMA_CTRL "ep_kdma_ctrl_00"
+#define XRT_MD_NODE_FPGA_CONFIG "ep_fpga_configuration_00"
+#define XRT_MD_NODE_ERT_SCHED "ep_ert_sched_00"
+#define XRT_MD_NODE_XDMA "ep_xdma_00"
+#define XRT_MD_NODE_MSIX "ep_msix_00"
+#define XRT_MD_NODE_QDMA "ep_qdma_00"
+#define XRT_MD_XRT_MD_NODE_QDMA4 "ep_qdma4_00"
+#define XRT_MD_NODE_STM "ep_stream_traffic_manager_00"
+#define XRT_MD_NODE_STM4 "ep_stream_traffic_manager4_00"
+#define XRT_MD_NODE_CLK_SHUTDOWN "ep_aclk_shutdown_00"
+#define XRT_MD_NODE_ERT_BASE "ep_ert_base_address_00"
+#define XRT_MD_NODE_ERT_RESET "ep_ert_reset_00"
+#define XRT_MD_NODE_CLKFREQ_K1 "ep_freq_cnt_aclk_kernel_00"
+#define XRT_MD_NODE_CLKFREQ_K2 "ep_freq_cnt_aclk_kernel_01"
+#define XRT_MD_NODE_CLKFREQ_HBM "ep_freq_cnt_aclk_hbm_00"
+#define XRT_MD_NODE_GAPPING "ep_gapping_demand_00"
+#define XRT_MD_NODE_UCS_CONTROL_STATUS "ep_ucs_control_status_00"
+#define XRT_MD_NODE_P2P "ep_p2p_00"
+#define XRT_MD_NODE_REMAP_P2P "ep_remap_p2p_00"
+#define XRT_MD_NODE_DDR4_RESET_GATE "ep_ddr_mem_srsr_gate_00"
+#define XRT_MD_NODE_ADDR_TRANSLATOR "ep_remap_data_c2h_00"
+#define XRT_MD_NODE_MAILBOX_XRT "ep_mailbox_user_to_ert_00"
+#define XRT_MD_NODE_PMC_INTR "ep_pmc_intr_00"
+#define XRT_MD_NODE_PMC_MUX "ep_pmc_mux_00"
+
+/* driver defined endpoints */
+#define XRT_MD_NODE_VSEC "drv_ep_vsec_00"
+#define XRT_MD_NODE_VSEC_GOLDEN "drv_ep_vsec_golden_00"
+#define XRT_MD_NODE_BLP_ROM "drv_ep_blp_rom_00"
+#define XRT_MD_NODE_MAILBOX_VSEC "ep_mailbox_vsec_00"
+#define XRT_MD_NODE_PLAT_INFO "drv_ep_platform_info_mgmt_00"
+#define XRT_MD_NODE_TEST "drv_ep_test_00"
+#define XRT_MD_NODE_MGMT_MAIN "drv_ep_mgmt_main_00"
+#define XRT_MD_NODE_FLASH_VSEC "drv_ep_card_flash_program_00"
+#define XRT_MD_NODE_GOLDEN_VER "drv_ep_golden_ver_00"
+#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_BLP "partition_info_0"
+#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_PLP "partition_info_1"
+
+#define XRT_MD_NODE_DDR_SRSR "drv_ep_ddr_srsr"
+#define XRT_MD_REGMAP_DDR_SRSR "drv_ddr_srsr"
+
+#define XRT_MD_PROP_OFFSET "drv_offset"
+#define XRT_MD_PROP_CLK_FREQ "drv_clock_frequency"
+#define XRT_MD_PROP_CLK_CNT "drv_clock_frequency_counter"
+#define XRT_MD_PROP_VBNV "vbnv"
+#define XRT_MD_PROP_VROM "vrom"
+#define XRT_MD_PROP_PARTITION_LEVEL "partition_level"
+
+struct xrt_md_endpoint {
+ const char *ep_name;
+ u32 bar;
+ long bar_off;
+ ulong size;
+ char *regmap;
+ char *regmap_ver;
+};
+
+/* Note: res_id is defined by leaf driver and must start with 0. */
+struct xrt_iores_map {
+ char *res_name;
+ int res_id;
+};
+
+static inline int xrt_md_res_name2id(const struct xrt_iores_map *res_map,
+ int entry_num, const char *res_name)
+{
+ int i;
+
+ for (i = 0; i < entry_num; i++) {
+ if (!strcmp(res_name, res_map->res_name))
+ return res_map->res_id;
+ res_map++;
+ }
+ return -1;
+}
+
+static inline const char *
+xrt_md_res_id2name(const struct xrt_iores_map *res_map, int entry_num, int id)
+{
+ int i;
+
+ for (i = 0; i < entry_num; i++) {
+ if (res_map->res_id == id)
+ return res_map->res_name;
+ res_map++;
+ }
+ return NULL;
+}
+
+unsigned long xrt_md_size(struct device *dev, const char *blob);
+int xrt_md_create(struct device *dev, char **blob);
+int xrt_md_add_endpoint(struct device *dev, char *blob,
+ struct xrt_md_endpoint *ep);
+int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
+ char *regmap_name);
+int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
+ const char *regmap_name, const char *prop,
+ const void **val, int *size);
+int xrt_md_set_prop(struct device *dev, char *blob, const char *ep_name,
+ const char *regmap_name, const char *prop,
+ const void *val, int size);
+int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
+ const char *ep_name, const char *regmap_name,
+ const char *new_ep_name);
+int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name,
+ char **next_ep, char **next_regmap);
+int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
+ const char *regmap_name, const char **ep_name);
+int xrt_md_find_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name,
+ const char **epname);
+void xrt_md_pack(struct device *dev, char *blob);
+char *xrt_md_dup(struct device *dev, const char *blob);
+int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
+ u32 *num_uuids, uuid_t *intf_uuids);
+static inline int xrt_md_copy_all_endpoints(struct device *dev, char *blob, const char *src_blob)
+{
+ return xrt_md_copy_endpoint(dev, blob, src_blob, XRT_MD_NODE_ENDPOINTS,
+ NULL, NULL);
+}
+
+/*
+ * Firmware provides 128 bit hash string as unque id of partition/interface.
+ * This string will be canonical textual representation in the future.
+ * Before that, introducing these two functions below to translate
+ * hash string to uuid_t for released hardware.
+ */
+static inline void xrt_md_trans_uuid2str(const uuid_t *uuid, char *uuidstr)
+{
+ int i, p;
+ u8 *u = (u8 *)uuid;
+
+ for (p = 0, i = sizeof(uuid_t) - 1; i >= 0; p++, i--)
+ (void)snprintf(&uuidstr[p * 2], 3, "%02x", u[i]);
+}
+
+static inline int xrt_md_trans_str2uuid(struct device *dev, const char *uuidstr, uuid_t *p_uuid)
+{
+ char *p;
+ const char *str;
+ char tmp[3] = { 0 };
+ int i, ret;
+
+ memset(p_uuid, 0, sizeof(*p_uuid));
+ p = (char *)p_uuid;
+ str = uuidstr + strlen(uuidstr) - 2;
+
+ for (i = 0; i < sizeof(*p_uuid) && str >= uuidstr; i++) {
+ tmp[0] = *str;
+ tmp[1] = *(str + 1);
+ ret = kstrtou8(tmp, 16, p);
+ if (ret)
+ return -EINVAL;
+ p++;
+ str -= 2;
+ }
+
+ return 0;
+}
+
+#endif
diff --git a/drivers/fpga/xrt/metadata/metadata.c b/drivers/fpga/xrt/metadata/metadata.c
new file mode 100644
index 000000000000..5d106396f438
--- /dev/null
+++ b/drivers/fpga/xrt/metadata/metadata.c
@@ -0,0 +1,524 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Metadata parse APIs
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#include <linux/libfdt_env.h>
+#include "libfdt.h"
+#include "metadata.h"
+
+#define MAX_BLOB_SIZE (4096 * 25)
+
+static int xrt_md_setprop(struct device *dev, char *blob, int offset,
+ const char *prop, const void *val, int size);
+static int xrt_md_overlay(struct device *dev, char *blob, int target,
+ const char *overlay_blob, int overlay_offset);
+static int xrt_md_get_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name,
+ int *ep_offset);
+
+unsigned long xrt_md_size(struct device *dev, const char *blob)
+{
+ unsigned long len = (long)fdt_totalsize(blob);
+
+ len = (len > MAX_BLOB_SIZE) ? XRT_MD_INVALID_LENGTH : len;
+ return len;
+}
+EXPORT_SYMBOL_GPL(xrt_md_size);
+
+int xrt_md_create(struct device *dev, char **blob)
+{
+ int ret = 0;
+
+ WARN_ON(!blob);
+
+ *blob = vmalloc(MAX_BLOB_SIZE);
+ if (!*blob)
+ return -ENOMEM;
+
+ ret = fdt_create_empty_tree(*blob, MAX_BLOB_SIZE);
+ if (ret) {
+ dev_err(dev, "format blob failed, ret = %d", ret);
+ goto failed;
+ }
+
+ ret = fdt_next_node(*blob, -1, NULL);
+ if (ret < 0) {
+ dev_err(dev, "No Node, ret = %d", ret);
+ goto failed;
+ }
+
+ ret = fdt_add_subnode(*blob, ret, XRT_MD_NODE_ENDPOINTS);
+ if (ret < 0) {
+ dev_err(dev, "add node failed, ret = %d", ret);
+ goto failed;
+ }
+
+ return 0;
+
+failed:
+ vfree(*blob);
+ *blob = NULL;
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_md_create);
+
+static int xrt_md_add_node(struct device *dev, char *blob, int parent_offset,
+ const char *ep_name)
+{
+ int ret;
+
+ ret = fdt_add_subnode(blob, parent_offset, ep_name);
+ if (ret < 0 && ret != -FDT_ERR_EXISTS)
+ dev_err(dev, "failed to add node %s. %d", ep_name, ret);
+
+ return ret;
+}
+
+int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
+ char *regmap_name)
+{
+ int ret;
+ int ep_offset;
+
+ ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name, &ep_offset);
+ if (ret) {
+ dev_err(dev, "can not find ep %s", ep_name);
+ return -EINVAL;
+ }
+
+ ret = fdt_del_node(blob, ep_offset);
+ if (ret)
+ dev_err(dev, "delete node %s failed, ret %d", ep_name, ret);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_md_del_endpoint);
+
+static int __xrt_md_add_endpoint(struct device *dev, char *blob,
+ struct xrt_md_endpoint *ep, int *offset, bool root)
+{
+ int ret = 0;
+ int ep_offset = 0;
+ u32 val, count = 0;
+ u64 io_range[2];
+ char comp[128];
+
+ if (!ep->ep_name) {
+ dev_err(dev, "empty name");
+ return -EINVAL;
+ }
+
+ if (!root) {
+ ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
+ &ep_offset);
+ if (ret) {
+ dev_err(dev, "invalid blob, ret = %d", ret);
+ return -EINVAL;
+ }
+ }
+
+ ep_offset = xrt_md_add_node(dev, blob, ep_offset, ep->ep_name);
+ if (ep_offset < 0) {
+ dev_err(dev, "add endpoint failed, ret = %d", ret);
+ return -EINVAL;
+ }
+ if (offset)
+ *offset = ep_offset;
+
+ if (ep->size != 0) {
+ val = cpu_to_be32(ep->bar);
+ ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_BAR_IDX,
+ &val, sizeof(u32));
+ if (ret) {
+ dev_err(dev, "set %s failed, ret %d",
+ XRT_MD_PROP_BAR_IDX, ret);
+ goto failed;
+ }
+ io_range[0] = cpu_to_be64((u64)ep->bar_off);
+ io_range[1] = cpu_to_be64((u64)ep->size);
+ ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_IO_OFFSET,
+ io_range, sizeof(io_range));
+ if (ret) {
+ dev_err(dev, "set %s failed, ret %d",
+ XRT_MD_PROP_IO_OFFSET, ret);
+ goto failed;
+ }
+ }
+
+ if (ep->regmap) {
+ if (ep->regmap_ver) {
+ count = snprintf(comp, sizeof(comp),
+ "%s-%s", ep->regmap, ep->regmap_ver);
+ count++;
+ }
+
+ count += snprintf(comp + count, sizeof(comp) - count,
+ "%s", ep->regmap);
+ count++;
+
+ ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_COMPATIBLE,
+ comp, count);
+ if (ret) {
+ dev_err(dev, "set %s failed, ret %d",
+ XRT_MD_PROP_COMPATIBLE, ret);
+ goto failed;
+ }
+ }
+
+failed:
+ if (ret)
+ xrt_md_del_endpoint(dev, blob, ep->ep_name, NULL);
+
+ return ret;
+}
+
+int xrt_md_add_endpoint(struct device *dev, char *blob,
+ struct xrt_md_endpoint *ep)
+{
+ return __xrt_md_add_endpoint(dev, blob, ep, NULL, false);
+}
+EXPORT_SYMBOL_GPL(xrt_md_add_endpoint);
+
+static int xrt_md_get_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name,
+ int *ep_offset)
+{
+ int offset;
+ const char *name;
+
+ for (offset = fdt_next_node(blob, -1, NULL);
+ offset >= 0;
+ offset = fdt_next_node(blob, offset, NULL)) {
+ name = fdt_get_name(blob, offset, NULL);
+ if (!name || strncmp(name, ep_name, strlen(ep_name) + 1))
+ continue;
+ if (!regmap_name ||
+ !fdt_node_check_compatible(blob, offset, regmap_name))
+ break;
+ }
+ if (offset < 0)
+ return -ENODEV;
+
+ *ep_offset = offset;
+
+ return 0;
+}
+
+int xrt_md_find_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name,
+ const char **epname)
+{
+ int offset;
+ int ret;
+
+ ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
+ &offset);
+ if (!ret && epname && offset >= 0)
+ *epname = fdt_get_name(blob, offset, NULL);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_md_find_endpoint);
+
+int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
+ const char *regmap_name, const char *prop,
+ const void **val, int *size)
+{
+ int offset;
+ int ret;
+
+ if (val)
+ *val = NULL;
+ if (ep_name) {
+ ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
+ &offset);
+ if (ret) {
+ dev_err(dev, "cannot get ep %s, regmap %s, ret = %d",
+ ep_name, regmap_name, ret);
+ return -EINVAL;
+ }
+ } else {
+ offset = fdt_next_node(blob, -1, NULL);
+ if (offset < 0) {
+ dev_err(dev, "internal error, ret = %d", offset);
+ return -EINVAL;
+ }
+ }
+
+ if (val) {
+ *val = fdt_getprop(blob, offset, prop, size);
+ if (!*val) {
+ dev_dbg(dev, "get ep %s, prop %s failed", ep_name, prop);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_md_get_prop);
+
+static int xrt_md_setprop(struct device *dev, char *blob, int offset,
+ const char *prop, const void *val, int size)
+{
+ int ret;
+
+ ret = fdt_setprop(blob, offset, prop, val, size);
+ if (ret)
+ dev_err(dev, "failed to set prop %d", ret);
+
+ return ret;
+}
+
+int xrt_md_set_prop(struct device *dev, char *blob,
+ const char *ep_name, const char *regmap_name,
+ const char *prop, const void *val, int size)
+{
+ int offset;
+ int ret;
+
+ if (ep_name) {
+ ret = xrt_md_get_endpoint(dev, blob, ep_name,
+ regmap_name, &offset);
+ if (ret) {
+ dev_err(dev, "cannot get node %s, ret = %d",
+ ep_name, ret);
+ return -EINVAL;
+ }
+ } else {
+ offset = fdt_next_node(blob, -1, NULL);
+ if (offset < 0) {
+ dev_err(dev, "internal error, ret = %d", offset);
+ return -EINVAL;
+ }
+ }
+
+ ret = xrt_md_setprop(dev, blob, offset, prop, val, size);
+ if (ret)
+ dev_err(dev, "set prop %s failed, ret = %d", prop, ret);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_md_set_prop);
+
+int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
+ const char *ep_name, const char *regmap_name,
+ const char *new_ep_name)
+{
+ int offset, target;
+ int ret;
+ struct xrt_md_endpoint ep = {0};
+ const char *newepnm = new_ep_name ? new_ep_name : ep_name;
+
+ ret = xrt_md_get_endpoint(dev, src_blob, ep_name, regmap_name,
+ &offset);
+ if (ret)
+ return -EINVAL;
+
+ ret = xrt_md_get_endpoint(dev, blob, newepnm, regmap_name, &target);
+ if (ret) {
+ ep.ep_name = newepnm;
+ ret = __xrt_md_add_endpoint(dev, blob, &ep, &target,
+ fdt_parent_offset(src_blob, offset) == 0);
+ if (ret)
+ return -EINVAL;
+ }
+
+ ret = xrt_md_overlay(dev, blob, target, src_blob, offset);
+ if (ret)
+ dev_err(dev, "overlay failed, ret = %d", ret);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_md_copy_endpoint);
+
+char *xrt_md_dup(struct device *dev, const char *blob)
+{
+ int ret;
+ char *dup_blob;
+
+ ret = xrt_md_create(dev, &dup_blob);
+ if (ret)
+ return NULL;
+ ret = xrt_md_overlay(dev, dup_blob, -1, blob, -1);
+ if (ret) {
+ vfree(dup_blob);
+ return NULL;
+ }
+
+ return dup_blob;
+}
+EXPORT_SYMBOL_GPL(xrt_md_dup);
+
+static int xrt_md_overlay(struct device *dev, char *blob, int target,
+ const char *overlay_blob, int overlay_offset)
+{
+ int property, subnode;
+ int ret;
+
+ WARN_ON(!blob || !overlay_blob);
+
+ if (!blob) {
+ dev_err(dev, "blob is NULL");
+ return -EINVAL;
+ }
+
+ if (target < 0) {
+ target = fdt_next_node(blob, -1, NULL);
+ if (target < 0) {
+ dev_err(dev, "invalid target");
+ return -EINVAL;
+ }
+ }
+ if (overlay_offset < 0) {
+ overlay_offset = fdt_next_node(overlay_blob, -1, NULL);
+ if (overlay_offset < 0) {
+ dev_err(dev, "invalid overlay");
+ return -EINVAL;
+ }
+ }
+
+ fdt_for_each_property_offset(property, overlay_blob, overlay_offset) {
+ const char *name;
+ const void *prop;
+ int prop_len;
+
+ prop = fdt_getprop_by_offset(overlay_blob, property, &name,
+ &prop_len);
+ if (!prop || prop_len >= MAX_BLOB_SIZE) {
+ dev_err(dev, "internal error");
+ return -EINVAL;
+ }
+
+ ret = xrt_md_setprop(dev, blob, target, name, prop,
+ prop_len);
+ if (ret) {
+ dev_err(dev, "setprop failed, ret = %d", ret);
+ return ret;
+ }
+ }
+
+ fdt_for_each_subnode(subnode, overlay_blob, overlay_offset) {
+ const char *name = fdt_get_name(overlay_blob, subnode, NULL);
+ int nnode;
+
+ nnode = xrt_md_add_node(dev, blob, target, name);
+ if (nnode == -FDT_ERR_EXISTS)
+ nnode = fdt_subnode_offset(blob, target, name);
+ if (nnode < 0) {
+ dev_err(dev, "add node failed, ret = %d", nnode);
+ return nnode;
+ }
+
+ ret = xrt_md_overlay(dev, blob, nnode, overlay_blob, subnode);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
+ const char *ep_name, const char *regmap_name,
+ char **next_ep, char **next_regmap)
+{
+ int offset, ret;
+
+ if (!ep_name) {
+ ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
+ &offset);
+ } else {
+ ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
+ &offset);
+ }
+
+ if (ret) {
+ *next_ep = NULL;
+ *next_regmap = NULL;
+ return -EINVAL;
+ }
+
+ offset = ep_name ? fdt_next_subnode(blob, offset) :
+ fdt_first_subnode(blob, offset);
+ if (offset < 0) {
+ *next_ep = NULL;
+ *next_regmap = NULL;
+ return -EINVAL;
+ }
+
+ *next_ep = (char *)fdt_get_name(blob, offset, NULL);
+ *next_regmap = (char *)fdt_stringlist_get(blob, offset, XRT_MD_PROP_COMPATIBLE,
+ 0, NULL);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_md_get_next_endpoint);
+
+int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
+ const char *regmap_name, const char **ep_name)
+{
+ int ep_offset;
+
+ ep_offset = fdt_node_offset_by_compatible(blob, -1, regmap_name);
+ if (ep_offset < 0) {
+ *ep_name = NULL;
+ return -ENOENT;
+ }
+
+ *ep_name = (char *)fdt_get_name(blob, ep_offset, NULL);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_md_get_compatible_endpoint);
+
+void xrt_md_pack(struct device *dev, char *blob)
+{
+ int ret;
+
+ ret = fdt_pack(blob);
+ if (ret)
+ dev_err(dev, "pack failed %d", ret);
+}
+EXPORT_SYMBOL_GPL(xrt_md_pack);
+
+int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
+ u32 *num_uuids, uuid_t *intf_uuids)
+{
+ int offset, count = 0;
+ int ret;
+ const char *uuid_str;
+
+ ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_INTERFACES, NULL, &offset);
+ if (ret)
+ return -ENOENT;
+
+ for (offset = fdt_first_subnode(blob, offset);
+ offset >= 0;
+ offset = fdt_next_subnode(blob, offset)) {
+ uuid_str = fdt_getprop(blob, offset, XRT_MD_PROP_INTERFACE_UUID,
+ NULL);
+ if (!uuid_str) {
+ dev_err(dev, "empty intf uuid node");
+ return -EINVAL;
+ }
+
+ if (intf_uuids && count < *num_uuids) {
+ ret = xrt_md_trans_str2uuid(dev, uuid_str,
+ &intf_uuids[count]);
+ if (ret)
+ return -EINVAL;
+ }
+ count++;
+ }
+
+ *num_uuids = count;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_md_get_intf_uuids);
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions Lizhi Hou
@ 2021-02-20 17:07 ` Tom Rix
2021-02-23 6:05 ` Lizhi Hou
2021-02-23 1:23 ` Fernando Pacheco
1 sibling, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-20 17:07 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> XRT drivers use device tree as metadata format to discover HW subsystems
> behind PCIe BAR. Thus libfdt functions are called for driver to parse
for the driver to parse the
> device tree blob.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/metadata.h | 229 ++++++++++++
> drivers/fpga/xrt/metadata/metadata.c | 524 +++++++++++++++++++++++++++
> 2 files changed, 753 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/metadata.h
> create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>
> diff --git a/drivers/fpga/xrt/include/metadata.h b/drivers/fpga/xrt/include/metadata.h
> new file mode 100644
> index 000000000000..b929bc469b73
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/metadata.h
> @@ -0,0 +1,229 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_METADATA_H
> +#define _XRT_METADATA_H
> +
> +#include <linux/device.h>
> +#include <linux/vmalloc.h>
> +#include <linux/uuid.h>
> +
> +#define XRT_MD_INVALID_LENGTH (~0UL)
> +
These #defines could be in alphabetical order
Some #define with embedded acronyms could be expanded
ex/ XRT_MD_NODE_CMC_REG , what is CMC ?
> +#define XRT_MD_PROP_COMPATIBLE "compatible"
> +#define XRT_MD_PROP_PF_NUM "pcie_physical_function"
> +#define XRT_MD_PROP_BAR_IDX "pcie_bar_mapping"
> +#define XRT_MD_PROP_IO_OFFSET "reg"
> +#define XRT_MD_PROP_INTERRUPTS "interrupts"
> +#define XRT_MD_PROP_INTERFACE_UUID "interface_uuid"
> +#define XRT_MD_PROP_LOGIC_UUID "logic_uuid"
> +#define XRT_MD_PROP_VERSION_MAJOR "firmware_version_major"
> +
> +#define XRT_MD_PROP_HWICAP "axi_hwicap"
> +#define XRT_MD_PROP_PDI_CONFIG "pdi_config_mem"
> +
> +#define XRT_MD_NODE_ENDPOINTS "addressable_endpoints"
> +#define XRT_MD_INTERFACES_PATH "/interfaces"
> +
> +#define XRT_MD_NODE_FIRMWARE "firmware"
> +#define XRT_MD_NODE_INTERFACES "interfaces"
> +#define XRT_MD_NODE_PARTITION_INFO "partition_info"
> +
> +#define XRT_MD_NODE_FLASH "ep_card_flash_program_00"
> +#define XRT_MD_NODE_XVC_PUB "ep_debug_bscan_user_00"
> +#define XRT_MD_NODE_XVC_PRI "ep_debug_bscan_mgmt_00"
> +#define XRT_MD_NODE_SYSMON "ep_cmp_sysmon_00"
> +#define XRT_MD_NODE_AF_BLP_CTRL_MGMT "ep_firewall_blp_ctrl_mgmt_00"
> +#define XRT_MD_NODE_AF_BLP_CTRL_USER "ep_firewall_blp_ctrl_user_00"
> +#define XRT_MD_NODE_AF_CTRL_MGMT "ep_firewall_ctrl_mgmt_00"
> +#define XRT_MD_NODE_AF_CTRL_USER "ep_firewall_ctrl_user_00"
> +#define XRT_MD_NODE_AF_CTRL_DEBUG "ep_firewall_ctrl_debug_00"
> +#define XRT_MD_NODE_AF_DATA_H2C "ep_firewall_data_h2c_00"
> +#define XRT_MD_NODE_AF_DATA_C2H "ep_firewall_data_c2h_00"
> +#define XRT_MD_NODE_AF_DATA_P2P "ep_firewall_data_p2p_00"
> +#define XRT_MD_NODE_AF_DATA_M2M "ep_firewall_data_m2m_00"
> +#define XRT_MD_NODE_CMC_REG "ep_cmc_regmap_00"
> +#define XRT_MD_NODE_CMC_RESET "ep_cmc_reset_00"
> +#define XRT_MD_NODE_CMC_MUTEX "ep_cmc_mutex_00"
> +#define XRT_MD_NODE_CMC_FW_MEM "ep_cmc_firmware_mem_00"
> +#define XRT_MD_NODE_ERT_FW_MEM "ep_ert_firmware_mem_00"
> +#define XRT_MD_NODE_ERT_CQ_MGMT "ep_ert_command_queue_mgmt_00"
> +#define XRT_MD_NODE_ERT_CQ_USER "ep_ert_command_queue_user_00"
> +#define XRT_MD_NODE_MAILBOX_MGMT "ep_mailbox_mgmt_00"
> +#define XRT_MD_NODE_MAILBOX_USER "ep_mailbox_user_00"
> +#define XRT_MD_NODE_GATE_PLP "ep_pr_isolate_plp_00"
> +#define XRT_MD_NODE_GATE_ULP "ep_pr_isolate_ulp_00"
> +#define XRT_MD_NODE_PCIE_MON "ep_pcie_link_mon_00"
> +#define XRT_MD_NODE_DDR_CALIB "ep_ddr_mem_calib_00"
> +#define XRT_MD_NODE_CLK_KERNEL1 "ep_aclk_kernel_00"
> +#define XRT_MD_NODE_CLK_KERNEL2 "ep_aclk_kernel_01"
> +#define XRT_MD_NODE_CLK_KERNEL3 "ep_aclk_hbm_00"
> +#define XRT_MD_NODE_KDMA_CTRL "ep_kdma_ctrl_00"
> +#define XRT_MD_NODE_FPGA_CONFIG "ep_fpga_configuration_00"
> +#define XRT_MD_NODE_ERT_SCHED "ep_ert_sched_00"
> +#define XRT_MD_NODE_XDMA "ep_xdma_00"
> +#define XRT_MD_NODE_MSIX "ep_msix_00"
> +#define XRT_MD_NODE_QDMA "ep_qdma_00"
> +#define XRT_MD_XRT_MD_NODE_QDMA4 "ep_qdma4_00"
> +#define XRT_MD_NODE_STM "ep_stream_traffic_manager_00"
> +#define XRT_MD_NODE_STM4 "ep_stream_traffic_manager4_00"
> +#define XRT_MD_NODE_CLK_SHUTDOWN "ep_aclk_shutdown_00"
> +#define XRT_MD_NODE_ERT_BASE "ep_ert_base_address_00"
> +#define XRT_MD_NODE_ERT_RESET "ep_ert_reset_00"
> +#define XRT_MD_NODE_CLKFREQ_K1 "ep_freq_cnt_aclk_kernel_00"
> +#define XRT_MD_NODE_CLKFREQ_K2 "ep_freq_cnt_aclk_kernel_01"
> +#define XRT_MD_NODE_CLKFREQ_HBM "ep_freq_cnt_aclk_hbm_00"
> +#define XRT_MD_NODE_GAPPING "ep_gapping_demand_00"
> +#define XRT_MD_NODE_UCS_CONTROL_STATUS "ep_ucs_control_status_00"
> +#define XRT_MD_NODE_P2P "ep_p2p_00"
> +#define XRT_MD_NODE_REMAP_P2P "ep_remap_p2p_00"
> +#define XRT_MD_NODE_DDR4_RESET_GATE "ep_ddr_mem_srsr_gate_00"
> +#define XRT_MD_NODE_ADDR_TRANSLATOR "ep_remap_data_c2h_00"
> +#define XRT_MD_NODE_MAILBOX_XRT "ep_mailbox_user_to_ert_00"
> +#define XRT_MD_NODE_PMC_INTR "ep_pmc_intr_00"
> +#define XRT_MD_NODE_PMC_MUX "ep_pmc_mux_00"
> +
> +/* driver defined endpoints */
> +#define XRT_MD_NODE_VSEC "drv_ep_vsec_00"
> +#define XRT_MD_NODE_VSEC_GOLDEN "drv_ep_vsec_golden_00"
> +#define XRT_MD_NODE_BLP_ROM "drv_ep_blp_rom_00"
> +#define XRT_MD_NODE_MAILBOX_VSEC "ep_mailbox_vsec_00"
> +#define XRT_MD_NODE_PLAT_INFO "drv_ep_platform_info_mgmt_00"
> +#define XRT_MD_NODE_TEST "drv_ep_test_00"
> +#define XRT_MD_NODE_MGMT_MAIN "drv_ep_mgmt_main_00"
> +#define XRT_MD_NODE_FLASH_VSEC "drv_ep_card_flash_program_00"
> +#define XRT_MD_NODE_GOLDEN_VER "drv_ep_golden_ver_00"
> +#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_BLP "partition_info_0"
> +#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_PLP "partition_info_1"
> +
> +#define XRT_MD_NODE_DDR_SRSR "drv_ep_ddr_srsr"
> +#define XRT_MD_REGMAP_DDR_SRSR "drv_ddr_srsr"
> +
> +#define XRT_MD_PROP_OFFSET "drv_offset"
> +#define XRT_MD_PROP_CLK_FREQ "drv_clock_frequency"
> +#define XRT_MD_PROP_CLK_CNT "drv_clock_frequency_counter"
> +#define XRT_MD_PROP_VBNV "vbnv"
> +#define XRT_MD_PROP_VROM "vrom"
> +#define XRT_MD_PROP_PARTITION_LEVEL "partition_level"
> +
> +struct xrt_md_endpoint {
> + const char *ep_name;
> + u32 bar;
> + long bar_off;
> + ulong size;
> + char *regmap;
> + char *regmap_ver;
> +};
could the existing 'struct regmap' be used here ?
> +
> +/* Note: res_id is defined by leaf driver and must start with 0. */
> +struct xrt_iores_map {
> + char *res_name;
> + int res_id;
> +};
> +
> +static inline int xrt_md_res_name2id(const struct xrt_iores_map *res_map,
> + int entry_num, const char *res_name)
> +{
> + int i;
> +
> + for (i = 0; i < entry_num; i++) {
> + if (!strcmp(res_name, res_map->res_name))
Use the 'n' variant strncmp for better safety.
Fix generally.
> + return res_map->res_id;
> + res_map++;
> + }
> + return -1;
> +}
> +
> +static inline const char *
> +xrt_md_res_id2name(const struct xrt_iores_map *res_map, int entry_num, int id)
> +{
> + int i;
> +
> + for (i = 0; i < entry_num; i++) {
> + if (res_map->res_id == id)
> + return res_map->res_name;
> + res_map++;
> + }
> + return NULL;
> +}
> +
> +unsigned long xrt_md_size(struct device *dev, const char *blob);
> +int xrt_md_create(struct device *dev, char **blob);
> +int xrt_md_add_endpoint(struct device *dev, char *blob,
> + struct xrt_md_endpoint *ep);
> +int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
> + char *regmap_name);
> +int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
> + const char *regmap_name, const char *prop,
> + const void **val, int *size);
> +int xrt_md_set_prop(struct device *dev, char *blob, const char *ep_name,
> + const char *regmap_name, const char *prop,
> + const void *val, int size);
> +int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
> + const char *ep_name, const char *regmap_name,
> + const char *new_ep_name);
> +int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + char **next_ep, char **next_regmap);
> +int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
> + const char *regmap_name, const char **ep_name);
> +int xrt_md_find_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + const char **epname);
> +void xrt_md_pack(struct device *dev, char *blob);
> +char *xrt_md_dup(struct device *dev, const char *blob);
> +int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
> + u32 *num_uuids, uuid_t *intf_uuids);
> +static inline int xrt_md_copy_all_endpoints(struct device *dev, char *blob, const char *src_blob)
> +{
> + return xrt_md_copy_endpoint(dev, blob, src_blob, XRT_MD_NODE_ENDPOINTS,
> + NULL, NULL);
A wrapping a single call seems like an unnecessary abstraction layer.
can this be reduced/removed ?
> +}
> +
> +/*
> + * Firmware provides 128 bit hash string as unque id of partition/interface.
The firmware provides a 128 bit hash string as a unique id to the partition/interface.
Existing hw does not yet use the cononical form, so it is necessary to use a translation function.
> + * This string will be canonical textual representation in the future.
> + * Before that, introducing these two functions below to translate
> + * hash string to uuid_t for released hardware.
Is there an existing version string the new hw will use to check which way to go ?
> + */
> +static inline void xrt_md_trans_uuid2str(const uuid_t *uuid, char *uuidstr)
> +{
> + int i, p;
> + u8 *u = (u8 *)uuid;
> +
> + for (p = 0, i = sizeof(uuid_t) - 1; i >= 0; p++, i--)
This loop needs to be improved.
Consider if sizeof(uuid_t) changed, accessing u[] would overflow.
> + (void)snprintf(&uuidstr[p * 2], 3, "%02x", u[i]);
(void) cast isn't needed.
> +}
> +
> +static inline int xrt_md_trans_str2uuid(struct device *dev, const char *uuidstr, uuid_t *p_uuid)
> +{
> + char *p;
> + const char *str;
> + char tmp[3] = { 0 };
> + int i, ret;
> +
> + memset(p_uuid, 0, sizeof(*p_uuid));
> + p = (char *)p_uuid;
access with p_uuid->b[] would remove use of 'p'
> + str = uuidstr + strlen(uuidstr) - 2;
> +
> + for (i = 0; i < sizeof(*p_uuid) && str >= uuidstr; i++) {
is not filling p_uuid completely really ok ?
> + tmp[0] = *str;
> + tmp[1] = *(str + 1);
> + ret = kstrtou8(tmp, 16, p);
> + if (ret)
> + return -EINVAL;
> + p++;
> + str -= 2;
> + }
> +
> + return 0;
> +}
> +
> +#endif
> diff --git a/drivers/fpga/xrt/metadata/metadata.c b/drivers/fpga/xrt/metadata/metadata.c
> new file mode 100644
> index 000000000000..5d106396f438
> --- /dev/null
> +++ b/drivers/fpga/xrt/metadata/metadata.c
> @@ -0,0 +1,524 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA Metadata parse APIs
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#include <linux/libfdt_env.h>
> +#include "libfdt.h"
> +#include "metadata.h"
> +
> +#define MAX_BLOB_SIZE (4096 * 25)
> +
> +static int xrt_md_setprop(struct device *dev, char *blob, int offset,
> + const char *prop, const void *val, int size);
> +static int xrt_md_overlay(struct device *dev, char *blob, int target,
> + const char *overlay_blob, int overlay_offset);
> +static int xrt_md_get_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + int *ep_offset);
consider reordering functions so these fwd decl's are not needed
> +
> +unsigned long xrt_md_size(struct device *dev, const char *blob)
> +{
> + unsigned long len = (long)fdt_totalsize(blob);
> +
> + len = (len > MAX_BLOB_SIZE) ? XRT_MD_INVALID_LENGTH : len;
> + return len;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_size);
> +
> +int xrt_md_create(struct device *dev, char **blob)
> +{
> + int ret = 0;
> +
> + WARN_ON(!blob);
> +
> + *blob = vmalloc(MAX_BLOB_SIZE);
why vmalloc instead of vzalloc ?
> + if (!*blob)
> + return -ENOMEM;
> +
> + ret = fdt_create_empty_tree(*blob, MAX_BLOB_SIZE);
> + if (ret) {
> + dev_err(dev, "format blob failed, ret = %d", ret);
> + goto failed;
> + }
> +
> + ret = fdt_next_node(*blob, -1, NULL);
A variable called 'offset' would make more sense here because it is used later in fdt_add_subnode as the parentoffset parameter.
> + if (ret < 0) {
> + dev_err(dev, "No Node, ret = %d", ret);
> + goto failed;
> + }
> +
> + ret = fdt_add_subnode(*blob, ret, XRT_MD_NODE_ENDPOINTS);
why isn't the parentoffset '0' ?
> + if (ret < 0) {
> + dev_err(dev, "add node failed, ret = %d", ret);
> + goto failed;
> + }
> +
> + return 0;
> +
> +failed:
> + vfree(*blob);
> + *blob = NULL;
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_create);
> +
> +static int xrt_md_add_node(struct device *dev, char *blob, int parent_offset,
> + const char *ep_name)
> +{
> + int ret;
> +
> + ret = fdt_add_subnode(blob, parent_offset, ep_name);
> + if (ret < 0 && ret != -FDT_ERR_EXISTS)
> + dev_err(dev, "failed to add node %s. %d", ep_name, ret);
> +
> + return ret;
> +}
> +
> +int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
> + char *regmap_name)
> +{
> + int ret;
> + int ep_offset;
> +
> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name, &ep_offset);
> + if (ret) {
> + dev_err(dev, "can not find ep %s", ep_name);
> + return -EINVAL;
> + }
> +
> + ret = fdt_del_node(blob, ep_offset);
> + if (ret)
> + dev_err(dev, "delete node %s failed, ret %d", ep_name, ret);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_del_endpoint);
> +
> +static int __xrt_md_add_endpoint(struct device *dev, char *blob,
> + struct xrt_md_endpoint *ep, int *offset, bool root)
> +{
> + int ret = 0;
> + int ep_offset = 0;
> + u32 val, count = 0;
> + u64 io_range[2];
> + char comp[128];
> +
> + if (!ep->ep_name) {
> + dev_err(dev, "empty name");
> + return -EINVAL;
> + }
> +
> + if (!root) {
> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
> + &ep_offset);
> + if (ret) {
> + dev_err(dev, "invalid blob, ret = %d", ret);
> + return -EINVAL;
> + }
> + }
> +
> + ep_offset = xrt_md_add_node(dev, blob, ep_offset, ep->ep_name);
> + if (ep_offset < 0) {
> + dev_err(dev, "add endpoint failed, ret = %d", ret);
> + return -EINVAL;
> + }
> + if (offset)
> + *offset = ep_offset;
> +
> + if (ep->size != 0) {
> + val = cpu_to_be32(ep->bar);
> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_BAR_IDX,
> + &val, sizeof(u32));
> + if (ret) {
> + dev_err(dev, "set %s failed, ret %d",
> + XRT_MD_PROP_BAR_IDX, ret);
> + goto failed;
> + }
> + io_range[0] = cpu_to_be64((u64)ep->bar_off);
> + io_range[1] = cpu_to_be64((u64)ep->size);
is there a type mismatch between bar (u32) and size (u64) ?
> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_IO_OFFSET,
> + io_range, sizeof(io_range));
> + if (ret) {
> + dev_err(dev, "set %s failed, ret %d",
> + XRT_MD_PROP_IO_OFFSET, ret);
> + goto failed;
> + }
> + }
> +
> + if (ep->regmap) {
> + if (ep->regmap_ver) {
> + count = snprintf(comp, sizeof(comp),
> + "%s-%s", ep->regmap, ep->regmap_ver);
> + count++;
> + }
> +
> + count += snprintf(comp + count, sizeof(comp) - count,
> + "%s", ep->regmap);
> + count++;
unlikely, but overflow is not checked.
are multiple null's in this string ok ?
> +
> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_COMPATIBLE,
> + comp, count);
> + if (ret) {
> + dev_err(dev, "set %s failed, ret %d",
> + XRT_MD_PROP_COMPATIBLE, ret);
> + goto failed;
> + }
> + }
> +
> +failed:
> + if (ret)
> + xrt_md_del_endpoint(dev, blob, ep->ep_name, NULL);
> +
> + return ret;
> +}
> +
> +int xrt_md_add_endpoint(struct device *dev, char *blob,
> + struct xrt_md_endpoint *ep)
> +{
> + return __xrt_md_add_endpoint(dev, blob, ep, NULL, false);
ok, user doesn't add root's endpoint.
could an assert be added ?
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_add_endpoint);
> +
> +static int xrt_md_get_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + int *ep_offset)
> +{
> + int offset;
> + const char *name;
> +
> + for (offset = fdt_next_node(blob, -1, NULL);
> + offset >= 0;
> + offset = fdt_next_node(blob, offset, NULL)) {
> + name = fdt_get_name(blob, offset, NULL);
> + if (!name || strncmp(name, ep_name, strlen(ep_name) + 1))
strlen() + 1 ?
> + continue;
> + if (!regmap_name ||
> + !fdt_node_check_compatible(blob, offset, regmap_name))
> + break;
> + }
> + if (offset < 0)
> + return -ENODEV;
> +
> + *ep_offset = offset;
> +
> + return 0;
> +}
> +
> +int xrt_md_find_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + const char **epname)
> +{
> + int offset;
> + int ret;
> +
> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
> + &offset);
> + if (!ret && epname && offset >= 0)
The offset >= 0 check isn't needed, !ret is enough
> + *epname = fdt_get_name(blob, offset, NULL);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_find_endpoint);
> +
> +int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
> + const char *regmap_name, const char *prop,
> + const void **val, int *size)
> +{
> + int offset;
> + int ret;
> +
> + if (val)
> + *val = NULL;
Seems like no point in making this call if !val
Return -EINVAL if !val and remove the if (val) check below.
> + if (ep_name) {
> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
> + &offset);
> + if (ret) {
> + dev_err(dev, "cannot get ep %s, regmap %s, ret = %d",
> + ep_name, regmap_name, ret);
> + return -EINVAL;
> + }
> + } else {
> + offset = fdt_next_node(blob, -1, NULL);
> + if (offset < 0) {
> + dev_err(dev, "internal error, ret = %d", offset);
> + return -EINVAL;
> + }
> + }
> +
> + if (val) {
> + *val = fdt_getprop(blob, offset, prop, size);
> + if (!*val) {
> + dev_dbg(dev, "get ep %s, prop %s failed", ep_name, prop);
> + return -EINVAL;
> + }
> + }
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_get_prop);
> +
> +static int xrt_md_setprop(struct device *dev, char *blob, int offset,
> + const char *prop, const void *val, int size)
> +{
> + int ret;
> +
> + ret = fdt_setprop(blob, offset, prop, val, size);
> + if (ret)
> + dev_err(dev, "failed to set prop %d", ret);
> +
> + return ret;
> +}
> +
> +int xrt_md_set_prop(struct device *dev, char *blob,
> + const char *ep_name, const char *regmap_name,
> + const char *prop, const void *val, int size)
> +{
> + int offset;
> + int ret;
> +
> + if (ep_name) {
> + ret = xrt_md_get_endpoint(dev, blob, ep_name,
> + regmap_name, &offset);
> + if (ret) {
> + dev_err(dev, "cannot get node %s, ret = %d",
> + ep_name, ret);
> + return -EINVAL;
> + }
> + } else {
> + offset = fdt_next_node(blob, -1, NULL);
> + if (offset < 0) {
> + dev_err(dev, "internal error, ret = %d", offset);
> + return -EINVAL;
> + }
> + }
This if-else block is a cut-n-paste from above.
It is good to convert common logic blocks to macros or inline functions.
> +
> + ret = xrt_md_setprop(dev, blob, offset, prop, val, size);
> + if (ret)
> + dev_err(dev, "set prop %s failed, ret = %d", prop, ret);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_set_prop);
> +
> +int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
> + const char *ep_name, const char *regmap_name,
> + const char *new_ep_name)
> +{
> + int offset, target;
> + int ret;
> + struct xrt_md_endpoint ep = {0};
> + const char *newepnm = new_ep_name ? new_ep_name : ep_name;
How is this valid ?
The xrt_md_get_endpoint searches by ep_name and if there names are not unique the second one will never be found.
> +
> + ret = xrt_md_get_endpoint(dev, src_blob, ep_name, regmap_name,
> + &offset);
> + if (ret)
> + return -EINVAL;
> +
> + ret = xrt_md_get_endpoint(dev, blob, newepnm, regmap_name, &target);
> + if (ret) {
> + ep.ep_name = newepnm;
> + ret = __xrt_md_add_endpoint(dev, blob, &ep, &target,
> + fdt_parent_offset(src_blob, offset) == 0);
> + if (ret)
> + return -EINVAL;
> + }
> +
> + ret = xrt_md_overlay(dev, blob, target, src_blob, offset);
> + if (ret)
> + dev_err(dev, "overlay failed, ret = %d", ret);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_copy_endpoint);
> +
> +char *xrt_md_dup(struct device *dev, const char *blob)
> +{
> + int ret;
> + char *dup_blob;
> +
> + ret = xrt_md_create(dev, &dup_blob);
> + if (ret)
> + return NULL;
> + ret = xrt_md_overlay(dev, dup_blob, -1, blob, -1);
would memcpy-ing the blob work ?
> + if (ret) {
> + vfree(dup_blob);
> + return NULL;
> + }
> +
> + return dup_blob;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_dup);
> +
> +static int xrt_md_overlay(struct device *dev, char *blob, int target,
> + const char *overlay_blob, int overlay_offset)
> +{
> + int property, subnode;
> + int ret;
> +
> + WARN_ON(!blob || !overlay_blob);
> +
> + if (!blob) {
> + dev_err(dev, "blob is NULL");
> + return -EINVAL;
> + }
> +
> + if (target < 0) {
> + target = fdt_next_node(blob, -1, NULL);
> + if (target < 0) {
> + dev_err(dev, "invalid target");
> + return -EINVAL;
> + }
> + }
> + if (overlay_offset < 0) {
> + overlay_offset = fdt_next_node(overlay_blob, -1, NULL);
> + if (overlay_offset < 0) {
> + dev_err(dev, "invalid overlay");
> + return -EINVAL;
> + }
> + }
> +
> + fdt_for_each_property_offset(property, overlay_blob, overlay_offset) {
> + const char *name;
> + const void *prop;
> + int prop_len;
> +
> + prop = fdt_getprop_by_offset(overlay_blob, property, &name,
> + &prop_len);
> + if (!prop || prop_len >= MAX_BLOB_SIZE) {
could add prop_len < 0 as another sanity check
> + dev_err(dev, "internal error");
> + return -EINVAL;
> + }
> +
> + ret = xrt_md_setprop(dev, blob, target, name, prop,
> + prop_len);
> + if (ret) {
> + dev_err(dev, "setprop failed, ret = %d", ret);
> + return ret;
overlay_blob is half done, as an error handling shouldn't it be undone ?
> + }
> + }
> +
> + fdt_for_each_subnode(subnode, overlay_blob, overlay_offset) {
> + const char *name = fdt_get_name(overlay_blob, subnode, NULL);
> + int nnode;
> +
> + nnode = xrt_md_add_node(dev, blob, target, name);
> + if (nnode == -FDT_ERR_EXISTS)
> + nnode = fdt_subnode_offset(blob, target, name);
> + if (nnode < 0) {
> + dev_err(dev, "add node failed, ret = %d", nnode);
> + return nnode;
> + }
> +
> + ret = xrt_md_overlay(dev, blob, nnode, overlay_blob, subnode);
eek, recursion.
Any chance this will blow the stack ?
> + if (ret)
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + char **next_ep, char **next_regmap)
> +{
> + int offset, ret;
> +
> + if (!ep_name) {
> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
> + &offset);
> + } else {
> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
> + &offset);
> + }
> +
> + if (ret) {
> + *next_ep = NULL;
> + *next_regmap = NULL;
could initialize next_ep and next_regmap to NULL outside the check.
> + return -EINVAL;
> + }
> +
> + offset = ep_name ? fdt_next_subnode(blob, offset) :
> + fdt_first_subnode(blob, offset);
> + if (offset < 0) {
> + *next_ep = NULL;
> + *next_regmap = NULL;
> + return -EINVAL;
> + }
> +
> + *next_ep = (char *)fdt_get_name(blob, offset, NULL);
> + *next_regmap = (char *)fdt_stringlist_get(blob, offset, XRT_MD_PROP_COMPATIBLE,
> + 0, NULL);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_get_next_endpoint);
> +
> +int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
> + const char *regmap_name, const char **ep_name)
> +{
> + int ep_offset;
> +
> + ep_offset = fdt_node_offset_by_compatible(blob, -1, regmap_name);
> + if (ep_offset < 0) {
> + *ep_name = NULL;
> + return -ENOENT;
> + }
> +
> + *ep_name = (char *)fdt_get_name(blob, ep_offset, NULL);
why the cast ?
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_get_compatible_endpoint);
> +
> +void xrt_md_pack(struct device *dev, char *blob)
> +{
> + int ret;
> +
> + ret = fdt_pack(blob);
> + if (ret)
> + dev_err(dev, "pack failed %d", ret);
maybe return int
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_pack);
> +
The input/output of num_uuids parameter is tricky add a comment
> +int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
> + u32 *num_uuids, uuid_t *intf_uuids)
what is intf ? change to 'interface' ?
> +{
> + int offset, count = 0;
> + int ret;
> + const char *uuid_str;
> +
> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_INTERFACES, NULL, &offset);
> + if (ret)
> + return -ENOENT;
> +
> + for (offset = fdt_first_subnode(blob, offset);
> + offset >= 0;
> + offset = fdt_next_subnode(blob, offset)) {
> + uuid_str = fdt_getprop(blob, offset, XRT_MD_PROP_INTERFACE_UUID,
> + NULL);
> + if (!uuid_str) {
> + dev_err(dev, "empty intf uuid node");
> + return -EINVAL;
> + }
> +
> + if (intf_uuids && count < *num_uuids) {
> + ret = xrt_md_trans_str2uuid(dev, uuid_str,
> + &intf_uuids[count]);
> + if (ret)
> + return -EINVAL;
> + }
> + count++;
keep going even when count > num_uuids ?
that seems like an error.
Tom
> + }
> +
> + *num_uuids = count;
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_get_intf_uuids);
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions
2021-02-20 17:07 ` Tom Rix
@ 2021-02-23 6:05 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-23 6:05 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 02/20/2021 09:07 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> XRT drivers use device tree as metadata format to discover HW subsystems
>> behind PCIe BAR. Thus libfdt functions are called for driver to parse
> for the driver to parse the
will fix.
>> device tree blob.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/metadata.h | 229 ++++++++++++
>> drivers/fpga/xrt/metadata/metadata.c | 524 +++++++++++++++++++++++++++
>> 2 files changed, 753 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/metadata.h
>> create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>>
>> diff --git a/drivers/fpga/xrt/include/metadata.h b/drivers/fpga/xrt/include/metadata.h
>> new file mode 100644
>> index 000000000000..b929bc469b73
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/metadata.h
>> @@ -0,0 +1,229 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for Xilinx Runtime (XRT) driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_METADATA_H
>> +#define _XRT_METADATA_H
>> +
>> +#include <linux/device.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/uuid.h>
>> +
>> +#define XRT_MD_INVALID_LENGTH (~0UL)
>> +
> These #defines could be in alphabetical order
>
> Some #define with embedded acronyms could be expanded
>
> ex/ XRT_MD_NODE_CMC_REG , what is CMC ?
Will reorder. Expanding might make macro name too long. I will add
comment as below:
/*
* IP nodes
* AF: AXI Firewall
* CMC: Card Management Controller
* ERT: Embedded Runtime
* PLP: Provider Reconfigurable Partition
* ULP: User Reconfigurable Partition
*/
>
>> +#define XRT_MD_PROP_COMPATIBLE "compatible"
>> +#define XRT_MD_PROP_PF_NUM "pcie_physical_function"
>> +#define XRT_MD_PROP_BAR_IDX "pcie_bar_mapping"
>> +#define XRT_MD_PROP_IO_OFFSET "reg"
>> +#define XRT_MD_PROP_INTERRUPTS "interrupts"
>> +#define XRT_MD_PROP_INTERFACE_UUID "interface_uuid"
>> +#define XRT_MD_PROP_LOGIC_UUID "logic_uuid"
>> +#define XRT_MD_PROP_VERSION_MAJOR "firmware_version_major"
>> +
>> +#define XRT_MD_PROP_HWICAP "axi_hwicap"
>> +#define XRT_MD_PROP_PDI_CONFIG "pdi_config_mem"
>> +
>> +#define XRT_MD_NODE_ENDPOINTS "addressable_endpoints"
>> +#define XRT_MD_INTERFACES_PATH "/interfaces"
>> +
>> +#define XRT_MD_NODE_FIRMWARE "firmware"
>> +#define XRT_MD_NODE_INTERFACES "interfaces"
>> +#define XRT_MD_NODE_PARTITION_INFO "partition_info"
>> +
>> +#define XRT_MD_NODE_FLASH "ep_card_flash_program_00"
>> +#define XRT_MD_NODE_XVC_PUB "ep_debug_bscan_user_00"
>> +#define XRT_MD_NODE_XVC_PRI "ep_debug_bscan_mgmt_00"
>> +#define XRT_MD_NODE_SYSMON "ep_cmp_sysmon_00"
>> +#define XRT_MD_NODE_AF_BLP_CTRL_MGMT "ep_firewall_blp_ctrl_mgmt_00"
>> +#define XRT_MD_NODE_AF_BLP_CTRL_USER "ep_firewall_blp_ctrl_user_00"
>> +#define XRT_MD_NODE_AF_CTRL_MGMT "ep_firewall_ctrl_mgmt_00"
>> +#define XRT_MD_NODE_AF_CTRL_USER "ep_firewall_ctrl_user_00"
>> +#define XRT_MD_NODE_AF_CTRL_DEBUG "ep_firewall_ctrl_debug_00"
>> +#define XRT_MD_NODE_AF_DATA_H2C "ep_firewall_data_h2c_00"
>> +#define XRT_MD_NODE_AF_DATA_C2H "ep_firewall_data_c2h_00"
>> +#define XRT_MD_NODE_AF_DATA_P2P "ep_firewall_data_p2p_00"
>> +#define XRT_MD_NODE_AF_DATA_M2M "ep_firewall_data_m2m_00"
>> +#define XRT_MD_NODE_CMC_REG "ep_cmc_regmap_00"
>> +#define XRT_MD_NODE_CMC_RESET "ep_cmc_reset_00"
>> +#define XRT_MD_NODE_CMC_MUTEX "ep_cmc_mutex_00"
>> +#define XRT_MD_NODE_CMC_FW_MEM "ep_cmc_firmware_mem_00"
>> +#define XRT_MD_NODE_ERT_FW_MEM "ep_ert_firmware_mem_00"
>> +#define XRT_MD_NODE_ERT_CQ_MGMT "ep_ert_command_queue_mgmt_00"
>> +#define XRT_MD_NODE_ERT_CQ_USER "ep_ert_command_queue_user_00"
>> +#define XRT_MD_NODE_MAILBOX_MGMT "ep_mailbox_mgmt_00"
>> +#define XRT_MD_NODE_MAILBOX_USER "ep_mailbox_user_00"
>> +#define XRT_MD_NODE_GATE_PLP "ep_pr_isolate_plp_00"
>> +#define XRT_MD_NODE_GATE_ULP "ep_pr_isolate_ulp_00"
>> +#define XRT_MD_NODE_PCIE_MON "ep_pcie_link_mon_00"
>> +#define XRT_MD_NODE_DDR_CALIB "ep_ddr_mem_calib_00"
>> +#define XRT_MD_NODE_CLK_KERNEL1 "ep_aclk_kernel_00"
>> +#define XRT_MD_NODE_CLK_KERNEL2 "ep_aclk_kernel_01"
>> +#define XRT_MD_NODE_CLK_KERNEL3 "ep_aclk_hbm_00"
>> +#define XRT_MD_NODE_KDMA_CTRL "ep_kdma_ctrl_00"
>> +#define XRT_MD_NODE_FPGA_CONFIG "ep_fpga_configuration_00"
>> +#define XRT_MD_NODE_ERT_SCHED "ep_ert_sched_00"
>> +#define XRT_MD_NODE_XDMA "ep_xdma_00"
>> +#define XRT_MD_NODE_MSIX "ep_msix_00"
>> +#define XRT_MD_NODE_QDMA "ep_qdma_00"
>> +#define XRT_MD_XRT_MD_NODE_QDMA4 "ep_qdma4_00"
>> +#define XRT_MD_NODE_STM "ep_stream_traffic_manager_00"
>> +#define XRT_MD_NODE_STM4 "ep_stream_traffic_manager4_00"
>> +#define XRT_MD_NODE_CLK_SHUTDOWN "ep_aclk_shutdown_00"
>> +#define XRT_MD_NODE_ERT_BASE "ep_ert_base_address_00"
>> +#define XRT_MD_NODE_ERT_RESET "ep_ert_reset_00"
>> +#define XRT_MD_NODE_CLKFREQ_K1 "ep_freq_cnt_aclk_kernel_00"
>> +#define XRT_MD_NODE_CLKFREQ_K2 "ep_freq_cnt_aclk_kernel_01"
>> +#define XRT_MD_NODE_CLKFREQ_HBM "ep_freq_cnt_aclk_hbm_00"
>> +#define XRT_MD_NODE_GAPPING "ep_gapping_demand_00"
>> +#define XRT_MD_NODE_UCS_CONTROL_STATUS "ep_ucs_control_status_00"
>> +#define XRT_MD_NODE_P2P "ep_p2p_00"
>> +#define XRT_MD_NODE_REMAP_P2P "ep_remap_p2p_00"
>> +#define XRT_MD_NODE_DDR4_RESET_GATE "ep_ddr_mem_srsr_gate_00"
>> +#define XRT_MD_NODE_ADDR_TRANSLATOR "ep_remap_data_c2h_00"
>> +#define XRT_MD_NODE_MAILBOX_XRT "ep_mailbox_user_to_ert_00"
>> +#define XRT_MD_NODE_PMC_INTR "ep_pmc_intr_00"
>> +#define XRT_MD_NODE_PMC_MUX "ep_pmc_mux_00"
>> +
>> +/* driver defined endpoints */
>> +#define XRT_MD_NODE_VSEC "drv_ep_vsec_00"
>> +#define XRT_MD_NODE_VSEC_GOLDEN "drv_ep_vsec_golden_00"
>> +#define XRT_MD_NODE_BLP_ROM "drv_ep_blp_rom_00"
>> +#define XRT_MD_NODE_MAILBOX_VSEC "ep_mailbox_vsec_00"
>> +#define XRT_MD_NODE_PLAT_INFO "drv_ep_platform_info_mgmt_00"
>> +#define XRT_MD_NODE_TEST "drv_ep_test_00"
>> +#define XRT_MD_NODE_MGMT_MAIN "drv_ep_mgmt_main_00"
>> +#define XRT_MD_NODE_FLASH_VSEC "drv_ep_card_flash_program_00"
>> +#define XRT_MD_NODE_GOLDEN_VER "drv_ep_golden_ver_00"
>> +#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_BLP "partition_info_0"
>> +#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_PLP "partition_info_1"
>> +
>> +#define XRT_MD_NODE_DDR_SRSR "drv_ep_ddr_srsr"
>> +#define XRT_MD_REGMAP_DDR_SRSR "drv_ddr_srsr"
>> +
>> +#define XRT_MD_PROP_OFFSET "drv_offset"
>> +#define XRT_MD_PROP_CLK_FREQ "drv_clock_frequency"
>> +#define XRT_MD_PROP_CLK_CNT "drv_clock_frequency_counter"
>> +#define XRT_MD_PROP_VBNV "vbnv"
>> +#define XRT_MD_PROP_VROM "vrom"
>> +#define XRT_MD_PROP_PARTITION_LEVEL "partition_level"
>> +
>> +struct xrt_md_endpoint {
>> + const char *ep_name;
>> + u32 bar;
>> + long bar_off;
>> + ulong size;
>> + char *regmap;
>> + char *regmap_ver;
>> +};
> could the existing 'struct regmap' be used here ?
This structure is just a direct translation of firmware node. It is
simple description of hardware endpoint.
>> +
>> +/* Note: res_id is defined by leaf driver and must start with 0. */
>> +struct xrt_iores_map {
>> + char *res_name;
>> + int res_id;
>> +};
>> +
>> +static inline int xrt_md_res_name2id(const struct xrt_iores_map *res_map,
>> + int entry_num, const char *res_name)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < entry_num; i++) {
>> + if (!strcmp(res_name, res_map->res_name))
> Use the 'n' variant strncmp for better safety.
>
> Fix generally.
Will replace all strcmp()
>
>> + return res_map->res_id;
>> + res_map++;
>> + }
>> + return -1;
>> +}
>> +
>> +static inline const char *
>> +xrt_md_res_id2name(const struct xrt_iores_map *res_map, int entry_num, int id)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < entry_num; i++) {
>> + if (res_map->res_id == id)
>> + return res_map->res_name;
>> + res_map++;
>> + }
>> + return NULL;
>> +}
>> +
>> +unsigned long xrt_md_size(struct device *dev, const char *blob);
>> +int xrt_md_create(struct device *dev, char **blob);
>> +int xrt_md_add_endpoint(struct device *dev, char *blob,
>> + struct xrt_md_endpoint *ep);
>> +int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
>> + char *regmap_name);
>> +int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
>> + const char *regmap_name, const char *prop,
>> + const void **val, int *size);
>> +int xrt_md_set_prop(struct device *dev, char *blob, const char *ep_name,
>> + const char *regmap_name, const char *prop,
>> + const void *val, int size);
>> +int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char *new_ep_name);
>> +int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + char **next_ep, char **next_regmap);
>> +int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
>> + const char *regmap_name, const char **ep_name);
>> +int xrt_md_find_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char **epname);
>> +void xrt_md_pack(struct device *dev, char *blob);
>> +char *xrt_md_dup(struct device *dev, const char *blob);
>> +int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
>> + u32 *num_uuids, uuid_t *intf_uuids);
>> +static inline int xrt_md_copy_all_endpoints(struct device *dev, char *blob, const char *src_blob)
>> +{
>> + return xrt_md_copy_endpoint(dev, blob, src_blob, XRT_MD_NODE_ENDPOINTS,
>> + NULL, NULL);
> A wrapping a single call seems like an unnecessary abstraction layer.
>
> can this be reduced/removed ?
Will remove.
>
>> +}
>> +
>> +/*
>> + * Firmware provides 128 bit hash string as unque id of partition/interface.
> The firmware provides a 128 bit hash string as a unique id to the partition/interface.
>
> Existing hw does not yet use the cononical form, so it is necessary to use a translation function.
Will fix
>
>> + * This string will be canonical textual representation in the future.
>> + * Before that, introducing these two functions below to translate
>> + * hash string to uuid_t for released hardware.
> Is there an existing version string the new hw will use to check which way to go ?
We do not have the new hw yet.
>> + */
>> +static inline void xrt_md_trans_uuid2str(const uuid_t *uuid, char *uuidstr)
>> +{
>> + int i, p;
>> + u8 *u = (u8 *)uuid;
>> +
>> + for (p = 0, i = sizeof(uuid_t) - 1; i >= 0; p++, i--)
> This loop needs to be improved.
>
> Consider if sizeof(uuid_t) changed, accessing u[] would overflow.
Will fix all the issues of uuid2str()/str2uuid() you pointed out. The
rough idea is to use export_uuid() and import_uuid().
>
>> + (void)snprintf(&uuidstr[p * 2], 3, "%02x", u[i]);
> (void) cast isn't needed.
>> +}
>> +
>> +static inline int xrt_md_trans_str2uuid(struct device *dev, const char *uuidstr, uuid_t *p_uuid)
>> +{
>> + char *p;
>> + const char *str;
>> + char tmp[3] = { 0 };
>> + int i, ret;
>> +
>> + memset(p_uuid, 0, sizeof(*p_uuid));
>> + p = (char *)p_uuid;
> access with p_uuid->b[] would remove use of 'p'
>> + str = uuidstr + strlen(uuidstr) - 2;
>> +
>> + for (i = 0; i < sizeof(*p_uuid) && str >= uuidstr; i++) {
> is not filling p_uuid completely really ok ?
>> + tmp[0] = *str;
>> + tmp[1] = *(str + 1);
>> + ret = kstrtou8(tmp, 16, p);
>> + if (ret)
>> + return -EINVAL;
>> + p++;
>> + str -= 2;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +#endif
>> diff --git a/drivers/fpga/xrt/metadata/metadata.c b/drivers/fpga/xrt/metadata/metadata.c
>> new file mode 100644
>> index 000000000000..5d106396f438
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/metadata/metadata.c
>> @@ -0,0 +1,524 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA Metadata parse APIs
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#include <linux/libfdt_env.h>
>> +#include "libfdt.h"
>> +#include "metadata.h"
>> +
>> +#define MAX_BLOB_SIZE (4096 * 25)
>> +
>> +static int xrt_md_setprop(struct device *dev, char *blob, int offset,
>> + const char *prop, const void *val, int size);
>> +static int xrt_md_overlay(struct device *dev, char *blob, int target,
>> + const char *overlay_blob, int overlay_offset);
>> +static int xrt_md_get_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + int *ep_offset);
> consider reordering functions so these fwd decl's are not needed
Will remove decl's.
>> +
>> +unsigned long xrt_md_size(struct device *dev, const char *blob)
>> +{
>> + unsigned long len = (long)fdt_totalsize(blob);
>> +
>> + len = (len > MAX_BLOB_SIZE) ? XRT_MD_INVALID_LENGTH : len;
>> + return len;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_size);
>> +
>> +int xrt_md_create(struct device *dev, char **blob)
>> +{
>> + int ret = 0;
>> +
>> + WARN_ON(!blob);
>> +
>> + *blob = vmalloc(MAX_BLOB_SIZE);
> why vmalloc instead of vzalloc ?
Will use vzalloc.
>> + if (!*blob)
>> + return -ENOMEM;
>> +
>> + ret = fdt_create_empty_tree(*blob, MAX_BLOB_SIZE);
>> + if (ret) {
>> + dev_err(dev, "format blob failed, ret = %d", ret);
>> + goto failed;
>> + }
>> +
>> + ret = fdt_next_node(*blob, -1, NULL);
> A variable called 'offset' would make more sense here because it is used later in fdt_add_subnode as the parentoffset parameter.
the following fdt_add_subnode should use 0 as the parentoffset
parameter. So I will keep 'ret' here.
>
>> + if (ret < 0) {
>> + dev_err(dev, "No Node, ret = %d", ret);
>> + goto failed;
>> + }
>> +
>> + ret = fdt_add_subnode(*blob, ret, XRT_MD_NODE_ENDPOINTS);
> why isn't the parentoffset '0' ?
Yes, 0 should be used as parentoffset.
>
>
>> + if (ret < 0) {
>> + dev_err(dev, "add node failed, ret = %d", ret);
>> + goto failed;
>> + }
>> +
>> + return 0;
>> +
>> +failed:
>> + vfree(*blob);
>> + *blob = NULL;
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_create);
>> +
>> +static int xrt_md_add_node(struct device *dev, char *blob, int parent_offset,
>> + const char *ep_name)
>> +{
>> + int ret;
>> +
>> + ret = fdt_add_subnode(blob, parent_offset, ep_name);
>> + if (ret < 0 && ret != -FDT_ERR_EXISTS)
>> + dev_err(dev, "failed to add node %s. %d", ep_name, ret);
>> +
>> + return ret;
>> +}
>> +
>> +int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
>> + char *regmap_name)
>> +{
>> + int ret;
>> + int ep_offset;
>> +
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name, &ep_offset);
>> + if (ret) {
>> + dev_err(dev, "can not find ep %s", ep_name);
>> + return -EINVAL;
>> + }
>> +
>> + ret = fdt_del_node(blob, ep_offset);
>> + if (ret)
>> + dev_err(dev, "delete node %s failed, ret %d", ep_name, ret);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_del_endpoint);
>> +
>> +static int __xrt_md_add_endpoint(struct device *dev, char *blob,
>> + struct xrt_md_endpoint *ep, int *offset, bool root)
>> +{
>> + int ret = 0;
>> + int ep_offset = 0;
>> + u32 val, count = 0;
>> + u64 io_range[2];
>> + char comp[128];
>> +
>> + if (!ep->ep_name) {
>> + dev_err(dev, "empty name");
>> + return -EINVAL;
>> + }
>> +
>> + if (!root) {
>> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
>> + &ep_offset);
>> + if (ret) {
>> + dev_err(dev, "invalid blob, ret = %d", ret);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + ep_offset = xrt_md_add_node(dev, blob, ep_offset, ep->ep_name);
>> + if (ep_offset < 0) {
>> + dev_err(dev, "add endpoint failed, ret = %d", ret);
>> + return -EINVAL;
>> + }
>> + if (offset)
>> + *offset = ep_offset;
>> +
>> + if (ep->size != 0) {
>> + val = cpu_to_be32(ep->bar);
>> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_BAR_IDX,
>> + &val, sizeof(u32));
>> + if (ret) {
>> + dev_err(dev, "set %s failed, ret %d",
>> + XRT_MD_PROP_BAR_IDX, ret);
>> + goto failed;
>> + }
>> + io_range[0] = cpu_to_be64((u64)ep->bar_off);
>> + io_range[1] = cpu_to_be64((u64)ep->size);
> is there a type mismatch between bar (u32) and size (u64) ?
Do you mean ep->bar? 'ep->bar' is bar index. So it does not have type
mismatch issue.
>> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_IO_OFFSET,
>> + io_range, sizeof(io_range));
>> + if (ret) {
>> + dev_err(dev, "set %s failed, ret %d",
>> + XRT_MD_PROP_IO_OFFSET, ret);
>> + goto failed;
>> + }
>> + }
>> +
>> + if (ep->regmap) {
>> + if (ep->regmap_ver) {
>> + count = snprintf(comp, sizeof(comp),
>> + "%s-%s", ep->regmap, ep->regmap_ver);
>> + count++;
>> + }
>> +
>> + count += snprintf(comp + count, sizeof(comp) - count,
>> + "%s", ep->regmap);
>> + count++;
> unlikely, but overflow is not checked.
Will add overflow check.
>
> are multiple null's in this string ok ?
yes.
>
>> +
>> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_COMPATIBLE,
>> + comp, count);
>> + if (ret) {
>> + dev_err(dev, "set %s failed, ret %d",
>> + XRT_MD_PROP_COMPATIBLE, ret);
>> + goto failed;
>> + }
>> + }
>> +
>> +failed:
>> + if (ret)
>> + xrt_md_del_endpoint(dev, blob, ep->ep_name, NULL);
>> +
>> + return ret;
>> +}
>> +
>> +int xrt_md_add_endpoint(struct device *dev, char *blob,
>> + struct xrt_md_endpoint *ep)
>> +{
>> + return __xrt_md_add_endpoint(dev, blob, ep, NULL, false);
> ok, user doesn't add root's endpoint.
>
> could an assert be added ?
The word 'root' is misleading here. Will change this argument to 'char
*parent'
>
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_add_endpoint);
>> +
>> +static int xrt_md_get_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + int *ep_offset)
>> +{
>> + int offset;
>> + const char *name;
>> +
>> + for (offset = fdt_next_node(blob, -1, NULL);
>> + offset >= 0;
>> + offset = fdt_next_node(blob, offset, NULL)) {
>> + name = fdt_get_name(blob, offset, NULL);
>> + if (!name || strncmp(name, ep_name, strlen(ep_name) + 1))
> strlen() + 1 ?
This is to make sure "12345" and "123" are different.
>> + continue;
>> + if (!regmap_name ||
>> + !fdt_node_check_compatible(blob, offset, regmap_name))
>> + break;
>> + }
>> + if (offset < 0)
>> + return -ENODEV;
>> +
>> + *ep_offset = offset;
>> +
>> + return 0;
>> +}
>> +
>> +int xrt_md_find_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char **epname)
>> +{
>> + int offset;
>> + int ret;
>> +
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
>> + &offset);
>> + if (!ret && epname && offset >= 0)
> The offset >= 0 check isn't needed, !ret is enough
Will fix.
>> + *epname = fdt_get_name(blob, offset, NULL);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_find_endpoint);
>> +
>> +int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
>> + const char *regmap_name, const char *prop,
>> + const void **val, int *size)
>> +{
>> + int offset;
>> + int ret;
>> +
>> + if (val)
>> + *val = NULL;
> Seems like no point in making this call if !val
>
> Return -EINVAL if !val and remove the if (val) check below.
Will fix
>
>> + if (ep_name) {
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
>> + &offset);
>> + if (ret) {
>> + dev_err(dev, "cannot get ep %s, regmap %s, ret = %d",
>> + ep_name, regmap_name, ret);
>> + return -EINVAL;
>> + }
>> + } else {
>> + offset = fdt_next_node(blob, -1, NULL);
>> + if (offset < 0) {
>> + dev_err(dev, "internal error, ret = %d", offset);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + if (val) {
>> + *val = fdt_getprop(blob, offset, prop, size);
>> + if (!*val) {
>> + dev_dbg(dev, "get ep %s, prop %s failed", ep_name, prop);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_get_prop);
>> +
>> +static int xrt_md_setprop(struct device *dev, char *blob, int offset,
>> + const char *prop, const void *val, int size)
>> +{
>> + int ret;
>> +
>> + ret = fdt_setprop(blob, offset, prop, val, size);
>> + if (ret)
>> + dev_err(dev, "failed to set prop %d", ret);
>> +
>> + return ret;
>> +}
>> +
>> +int xrt_md_set_prop(struct device *dev, char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char *prop, const void *val, int size)
>> +{
>> + int offset;
>> + int ret;
>> +
>> + if (ep_name) {
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name,
>> + regmap_name, &offset);
>> + if (ret) {
>> + dev_err(dev, "cannot get node %s, ret = %d",
>> + ep_name, ret);
>> + return -EINVAL;
>> + }
>> + } else {
>> + offset = fdt_next_node(blob, -1, NULL);
>> + if (offset < 0) {
>> + dev_err(dev, "internal error, ret = %d", offset);
>> + return -EINVAL;
>> + }
>> + }
> This if-else block is a cut-n-paste from above.
>
> It is good to convert common logic blocks to macros or inline functions.
Will define a in-line function.
>
>> +
>> + ret = xrt_md_setprop(dev, blob, offset, prop, val, size);
>> + if (ret)
>> + dev_err(dev, "set prop %s failed, ret = %d", prop, ret);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_set_prop);
>> +
>> +int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char *new_ep_name)
>> +{
>> + int offset, target;
>> + int ret;
>> + struct xrt_md_endpoint ep = {0};
>> + const char *newepnm = new_ep_name ? new_ep_name : ep_name;
> How is this valid ?
>
> The xrt_md_get_endpoint searches by ep_name and if there names are not unique the second one will never be found.
ep_name in the firmware will always be unique. It is documented in xrt.rst
>
>> +
>> + ret = xrt_md_get_endpoint(dev, src_blob, ep_name, regmap_name,
>> + &offset);
>> + if (ret)
>> + return -EINVAL;
>> +
>> + ret = xrt_md_get_endpoint(dev, blob, newepnm, regmap_name, &target);
>> + if (ret) {
>> + ep.ep_name = newepnm;
>> + ret = __xrt_md_add_endpoint(dev, blob, &ep, &target,
>> + fdt_parent_offset(src_blob, offset) == 0);
>> + if (ret)
>> + return -EINVAL;
>> + }
>> +
>> + ret = xrt_md_overlay(dev, blob, target, src_blob, offset);
>> + if (ret)
>> + dev_err(dev, "overlay failed, ret = %d", ret);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_copy_endpoint);
>> +
>> +char *xrt_md_dup(struct device *dev, const char *blob)
>> +{
>> + int ret;
>> + char *dup_blob;
>> +
>> + ret = xrt_md_create(dev, &dup_blob);
>> + if (ret)
>> + return NULL;
>> + ret = xrt_md_overlay(dev, dup_blob, -1, blob, -1);
> would memcpy-ing the blob work ?
Sure. Memcpy works. I will remove xrt_md_dup() and use memcpy.
>> + if (ret) {
>> + vfree(dup_blob);
>> + return NULL;
>> + }
>> +
>> + return dup_blob;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_dup);
>> +
>> +static int xrt_md_overlay(struct device *dev, char *blob, int target,
>> + const char *overlay_blob, int overlay_offset)
>> +{
>> + int property, subnode;
>> + int ret;
>> +
>> + WARN_ON(!blob || !overlay_blob);
>> +
>> + if (!blob) {
>> + dev_err(dev, "blob is NULL");
>> + return -EINVAL;
>> + }
>> +
>> + if (target < 0) {
>> + target = fdt_next_node(blob, -1, NULL);
>> + if (target < 0) {
>> + dev_err(dev, "invalid target");
>> + return -EINVAL;
>> + }
>> + }
>> + if (overlay_offset < 0) {
>> + overlay_offset = fdt_next_node(overlay_blob, -1, NULL);
>> + if (overlay_offset < 0) {
>> + dev_err(dev, "invalid overlay");
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + fdt_for_each_property_offset(property, overlay_blob, overlay_offset) {
>> + const char *name;
>> + const void *prop;
>> + int prop_len;
>> +
>> + prop = fdt_getprop_by_offset(overlay_blob, property, &name,
>> + &prop_len);
>> + if (!prop || prop_len >= MAX_BLOB_SIZE) {
> could add prop_len < 0 as another sanity check
Will add.
>> + dev_err(dev, "internal error");
>> + return -EINVAL;
>> + }
>> +
>> + ret = xrt_md_setprop(dev, blob, target, name, prop,
>> + prop_len);
>> + if (ret) {
>> + dev_err(dev, "setprop failed, ret = %d", ret);
>> + return ret;
> overlay_blob is half done, as an error handling shouldn't it be undone ?
The error will only cause a half done blob. Nothing need to be undone.
>> + }
>> + }
>> +
>> + fdt_for_each_subnode(subnode, overlay_blob, overlay_offset) {
>> + const char *name = fdt_get_name(overlay_blob, subnode, NULL);
>> + int nnode;
>> +
>> + nnode = xrt_md_add_node(dev, blob, target, name);
>> + if (nnode == -FDT_ERR_EXISTS)
>> + nnode = fdt_subnode_offset(blob, target, name);
>> + if (nnode < 0) {
>> + dev_err(dev, "add node failed, ret = %d", nnode);
>> + return nnode;
>> + }
>> +
>> + ret = xrt_md_overlay(dev, blob, nnode, overlay_blob, subnode);
> eek, recursion.
>
> Any chance this will blow the stack ?
Will add max recursive depth (5).
>
>> + if (ret)
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + char **next_ep, char **next_regmap)
>> +{
>> + int offset, ret;
>> +
>> + if (!ep_name) {
>> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
>> + &offset);
>> + } else {
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
>> + &offset);
>> + }
>> +
>> + if (ret) {
>> + *next_ep = NULL;
>> + *next_regmap = NULL;
> could initialize next_ep and next_regmap to NULL outside the check.
Will move outside.
>> + return -EINVAL;
>> + }
>> +
>> + offset = ep_name ? fdt_next_subnode(blob, offset) :
>> + fdt_first_subnode(blob, offset);
>> + if (offset < 0) {
>> + *next_ep = NULL;
>> + *next_regmap = NULL;
>> + return -EINVAL;
>> + }
>> +
>> + *next_ep = (char *)fdt_get_name(blob, offset, NULL);
>> + *next_regmap = (char *)fdt_stringlist_get(blob, offset, XRT_MD_PROP_COMPATIBLE,
>> + 0, NULL);
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_get_next_endpoint);
>> +
>> +int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
>> + const char *regmap_name, const char **ep_name)
>> +{
>> + int ep_offset;
>> +
>> + ep_offset = fdt_node_offset_by_compatible(blob, -1, regmap_name);
>> + if (ep_offset < 0) {
>> + *ep_name = NULL;
>> + return -ENOENT;
>> + }
>> +
>> + *ep_name = (char *)fdt_get_name(blob, ep_offset, NULL);
> why the cast ?
Will remove.
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_get_compatible_endpoint);
>> +
>> +void xrt_md_pack(struct device *dev, char *blob)
>> +{
>> + int ret;
>> +
>> + ret = fdt_pack(blob);
>> + if (ret)
>> + dev_err(dev, "pack failed %d", ret);
> maybe return int
Will return int.
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_pack);
>> +
> The input/output of num_uuids parameter is tricky add a comment
Will modify to 'u32 num_uuids' and return the actual number
>> +int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
>> + u32 *num_uuids, uuid_t *intf_uuids)
> what is intf ? change to 'interface' ?
yes. And will change to 'interface'.
>> +{
>> + int offset, count = 0;
>> + int ret;
>> + const char *uuid_str;
>> +
>> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_INTERFACES, NULL, &offset);
>> + if (ret)
>> + return -ENOENT;
>> +
>> + for (offset = fdt_first_subnode(blob, offset);
>> + offset >= 0;
>> + offset = fdt_next_subnode(blob, offset)) {
>> + uuid_str = fdt_getprop(blob, offset, XRT_MD_PROP_INTERFACE_UUID,
>> + NULL);
>> + if (!uuid_str) {
>> + dev_err(dev, "empty intf uuid node");
>> + return -EINVAL;
>> + }
>> +
>> + if (intf_uuids && count < *num_uuids) {
>> + ret = xrt_md_trans_str2uuid(dev, uuid_str,
>> + &intf_uuids[count]);
>> + if (ret)
>> + return -EINVAL;
>> + }
>> + count++;
> keep going even when count > num_uuids ?
>
> that seems like an error.
Will add check.
Thanks,
Lizhi
>
> Tom
>
>> + }
>> +
>> + *num_uuids = count;
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_get_intf_uuids);
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions Lizhi Hou
2021-02-20 17:07 ` Tom Rix
@ 2021-02-23 1:23 ` Fernando Pacheco
2021-02-25 20:27 ` Lizhi Hou
1 sibling, 1 reply; 87+ messages in thread
From: Fernando Pacheco @ 2021-02-23 1:23 UTC (permalink / raw)
To: Lizhi Hou
Cc: linux-kernel, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, mdf, robh, Max Zhen
On Wed, Feb 17, 2021 at 10:40:03PM -0800, Lizhi Hou wrote:
> XRT drivers use device tree as metadata format to discover HW subsystems
> behind PCIe BAR. Thus libfdt functions are called for driver to parse
> device tree blob.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/metadata.h | 229 ++++++++++++
> drivers/fpga/xrt/metadata/metadata.c | 524 +++++++++++++++++++++++++++
> 2 files changed, 753 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/metadata.h
> create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>
> diff --git a/drivers/fpga/xrt/include/metadata.h b/drivers/fpga/xrt/include/metadata.h
> new file mode 100644
> index 000000000000..b929bc469b73
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/metadata.h
> @@ -0,0 +1,229 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_METADATA_H
> +#define _XRT_METADATA_H
> +
> +#include <linux/device.h>
> +#include <linux/vmalloc.h>
> +#include <linux/uuid.h>
> +
> +#define XRT_MD_INVALID_LENGTH (~0UL)
> +
> +#define XRT_MD_PROP_COMPATIBLE "compatible"
> +#define XRT_MD_PROP_PF_NUM "pcie_physical_function"
> +#define XRT_MD_PROP_BAR_IDX "pcie_bar_mapping"
> +#define XRT_MD_PROP_IO_OFFSET "reg"
> +#define XRT_MD_PROP_INTERRUPTS "interrupts"
> +#define XRT_MD_PROP_INTERFACE_UUID "interface_uuid"
> +#define XRT_MD_PROP_LOGIC_UUID "logic_uuid"
> +#define XRT_MD_PROP_VERSION_MAJOR "firmware_version_major"
> +
> +#define XRT_MD_PROP_HWICAP "axi_hwicap"
> +#define XRT_MD_PROP_PDI_CONFIG "pdi_config_mem"
> +
> +#define XRT_MD_NODE_ENDPOINTS "addressable_endpoints"
> +#define XRT_MD_INTERFACES_PATH "/interfaces"
> +
> +#define XRT_MD_NODE_FIRMWARE "firmware"
> +#define XRT_MD_NODE_INTERFACES "interfaces"
> +#define XRT_MD_NODE_PARTITION_INFO "partition_info"
> +
> +#define XRT_MD_NODE_FLASH "ep_card_flash_program_00"
> +#define XRT_MD_NODE_XVC_PUB "ep_debug_bscan_user_00"
> +#define XRT_MD_NODE_XVC_PRI "ep_debug_bscan_mgmt_00"
> +#define XRT_MD_NODE_SYSMON "ep_cmp_sysmon_00"
> +#define XRT_MD_NODE_AF_BLP_CTRL_MGMT "ep_firewall_blp_ctrl_mgmt_00"
> +#define XRT_MD_NODE_AF_BLP_CTRL_USER "ep_firewall_blp_ctrl_user_00"
> +#define XRT_MD_NODE_AF_CTRL_MGMT "ep_firewall_ctrl_mgmt_00"
> +#define XRT_MD_NODE_AF_CTRL_USER "ep_firewall_ctrl_user_00"
> +#define XRT_MD_NODE_AF_CTRL_DEBUG "ep_firewall_ctrl_debug_00"
> +#define XRT_MD_NODE_AF_DATA_H2C "ep_firewall_data_h2c_00"
> +#define XRT_MD_NODE_AF_DATA_C2H "ep_firewall_data_c2h_00"
> +#define XRT_MD_NODE_AF_DATA_P2P "ep_firewall_data_p2p_00"
> +#define XRT_MD_NODE_AF_DATA_M2M "ep_firewall_data_m2m_00"
> +#define XRT_MD_NODE_CMC_REG "ep_cmc_regmap_00"
> +#define XRT_MD_NODE_CMC_RESET "ep_cmc_reset_00"
> +#define XRT_MD_NODE_CMC_MUTEX "ep_cmc_mutex_00"
> +#define XRT_MD_NODE_CMC_FW_MEM "ep_cmc_firmware_mem_00"
> +#define XRT_MD_NODE_ERT_FW_MEM "ep_ert_firmware_mem_00"
> +#define XRT_MD_NODE_ERT_CQ_MGMT "ep_ert_command_queue_mgmt_00"
> +#define XRT_MD_NODE_ERT_CQ_USER "ep_ert_command_queue_user_00"
> +#define XRT_MD_NODE_MAILBOX_MGMT "ep_mailbox_mgmt_00"
> +#define XRT_MD_NODE_MAILBOX_USER "ep_mailbox_user_00"
> +#define XRT_MD_NODE_GATE_PLP "ep_pr_isolate_plp_00"
> +#define XRT_MD_NODE_GATE_ULP "ep_pr_isolate_ulp_00"
> +#define XRT_MD_NODE_PCIE_MON "ep_pcie_link_mon_00"
> +#define XRT_MD_NODE_DDR_CALIB "ep_ddr_mem_calib_00"
> +#define XRT_MD_NODE_CLK_KERNEL1 "ep_aclk_kernel_00"
> +#define XRT_MD_NODE_CLK_KERNEL2 "ep_aclk_kernel_01"
> +#define XRT_MD_NODE_CLK_KERNEL3 "ep_aclk_hbm_00"
> +#define XRT_MD_NODE_KDMA_CTRL "ep_kdma_ctrl_00"
> +#define XRT_MD_NODE_FPGA_CONFIG "ep_fpga_configuration_00"
> +#define XRT_MD_NODE_ERT_SCHED "ep_ert_sched_00"
> +#define XRT_MD_NODE_XDMA "ep_xdma_00"
> +#define XRT_MD_NODE_MSIX "ep_msix_00"
> +#define XRT_MD_NODE_QDMA "ep_qdma_00"
> +#define XRT_MD_XRT_MD_NODE_QDMA4 "ep_qdma4_00"
> +#define XRT_MD_NODE_STM "ep_stream_traffic_manager_00"
> +#define XRT_MD_NODE_STM4 "ep_stream_traffic_manager4_00"
> +#define XRT_MD_NODE_CLK_SHUTDOWN "ep_aclk_shutdown_00"
> +#define XRT_MD_NODE_ERT_BASE "ep_ert_base_address_00"
> +#define XRT_MD_NODE_ERT_RESET "ep_ert_reset_00"
> +#define XRT_MD_NODE_CLKFREQ_K1 "ep_freq_cnt_aclk_kernel_00"
> +#define XRT_MD_NODE_CLKFREQ_K2 "ep_freq_cnt_aclk_kernel_01"
> +#define XRT_MD_NODE_CLKFREQ_HBM "ep_freq_cnt_aclk_hbm_00"
> +#define XRT_MD_NODE_GAPPING "ep_gapping_demand_00"
> +#define XRT_MD_NODE_UCS_CONTROL_STATUS "ep_ucs_control_status_00"
> +#define XRT_MD_NODE_P2P "ep_p2p_00"
> +#define XRT_MD_NODE_REMAP_P2P "ep_remap_p2p_00"
> +#define XRT_MD_NODE_DDR4_RESET_GATE "ep_ddr_mem_srsr_gate_00"
> +#define XRT_MD_NODE_ADDR_TRANSLATOR "ep_remap_data_c2h_00"
> +#define XRT_MD_NODE_MAILBOX_XRT "ep_mailbox_user_to_ert_00"
> +#define XRT_MD_NODE_PMC_INTR "ep_pmc_intr_00"
> +#define XRT_MD_NODE_PMC_MUX "ep_pmc_mux_00"
> +
> +/* driver defined endpoints */
> +#define XRT_MD_NODE_VSEC "drv_ep_vsec_00"
> +#define XRT_MD_NODE_VSEC_GOLDEN "drv_ep_vsec_golden_00"
> +#define XRT_MD_NODE_BLP_ROM "drv_ep_blp_rom_00"
> +#define XRT_MD_NODE_MAILBOX_VSEC "ep_mailbox_vsec_00"
Should this be "drv_ep_mailbox_vsec_00"?
> +#define XRT_MD_NODE_PLAT_INFO "drv_ep_platform_info_mgmt_00"
> +#define XRT_MD_NODE_TEST "drv_ep_test_00"
> +#define XRT_MD_NODE_MGMT_MAIN "drv_ep_mgmt_main_00"
> +#define XRT_MD_NODE_FLASH_VSEC "drv_ep_card_flash_program_00"
> +#define XRT_MD_NODE_GOLDEN_VER "drv_ep_golden_ver_00"
> +#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_BLP "partition_info_0"
> +#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_PLP "partition_info_1"
> +
> +#define XRT_MD_NODE_DDR_SRSR "drv_ep_ddr_srsr"
> +#define XRT_MD_REGMAP_DDR_SRSR "drv_ddr_srsr"
> +
> +#define XRT_MD_PROP_OFFSET "drv_offset"
> +#define XRT_MD_PROP_CLK_FREQ "drv_clock_frequency"
> +#define XRT_MD_PROP_CLK_CNT "drv_clock_frequency_counter"
> +#define XRT_MD_PROP_VBNV "vbnv"
> +#define XRT_MD_PROP_VROM "vrom"
> +#define XRT_MD_PROP_PARTITION_LEVEL "partition_level"
> +
> +struct xrt_md_endpoint {
> + const char *ep_name;
> + u32 bar;
> + long bar_off;
> + ulong size;
> + char *regmap;
> + char *regmap_ver;
> +};
> +
> +/* Note: res_id is defined by leaf driver and must start with 0. */
> +struct xrt_iores_map {
> + char *res_name;
> + int res_id;
> +};
> +
> +static inline int xrt_md_res_name2id(const struct xrt_iores_map *res_map,
> + int entry_num, const char *res_name)
> +{
> + int i;
> +
> + for (i = 0; i < entry_num; i++) {
> + if (!strcmp(res_name, res_map->res_name))
> + return res_map->res_id;
> + res_map++;
> + }
> + return -1;
> +}
> +
> +static inline const char *
> +xrt_md_res_id2name(const struct xrt_iores_map *res_map, int entry_num, int id)
> +{
> + int i;
> +
> + for (i = 0; i < entry_num; i++) {
> + if (res_map->res_id == id)
> + return res_map->res_name;
> + res_map++;
> + }
> + return NULL;
> +}
> +
> +unsigned long xrt_md_size(struct device *dev, const char *blob);
> +int xrt_md_create(struct device *dev, char **blob);
> +int xrt_md_add_endpoint(struct device *dev, char *blob,
> + struct xrt_md_endpoint *ep);
> +int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
> + char *regmap_name);
> +int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
> + const char *regmap_name, const char *prop,
> + const void **val, int *size);
> +int xrt_md_set_prop(struct device *dev, char *blob, const char *ep_name,
> + const char *regmap_name, const char *prop,
> + const void *val, int size);
> +int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
> + const char *ep_name, const char *regmap_name,
> + const char *new_ep_name);
> +int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + char **next_ep, char **next_regmap);
> +int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
> + const char *regmap_name, const char **ep_name);
> +int xrt_md_find_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + const char **epname);
> +void xrt_md_pack(struct device *dev, char *blob);
> +char *xrt_md_dup(struct device *dev, const char *blob);
> +int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
> + u32 *num_uuids, uuid_t *intf_uuids);
> +static inline int xrt_md_copy_all_endpoints(struct device *dev, char *blob, const char *src_blob)
> +{
> + return xrt_md_copy_endpoint(dev, blob, src_blob, XRT_MD_NODE_ENDPOINTS,
> + NULL, NULL);
> +}
> +
> +/*
> + * Firmware provides 128 bit hash string as unque id of partition/interface.
> + * This string will be canonical textual representation in the future.
> + * Before that, introducing these two functions below to translate
> + * hash string to uuid_t for released hardware.
> + */
> +static inline void xrt_md_trans_uuid2str(const uuid_t *uuid, char *uuidstr)
> +{
> + int i, p;
> + u8 *u = (u8 *)uuid;
> +
> + for (p = 0, i = sizeof(uuid_t) - 1; i >= 0; p++, i--)
> + (void)snprintf(&uuidstr[p * 2], 3, "%02x", u[i]);
> +}
> +
> +static inline int xrt_md_trans_str2uuid(struct device *dev, const char *uuidstr, uuid_t *p_uuid)
> +{
> + char *p;
> + const char *str;
> + char tmp[3] = { 0 };
> + int i, ret;
> +
> + memset(p_uuid, 0, sizeof(*p_uuid));
> + p = (char *)p_uuid;
> + str = uuidstr + strlen(uuidstr) - 2;
> +
> + for (i = 0; i < sizeof(*p_uuid) && str >= uuidstr; i++) {
> + tmp[0] = *str;
> + tmp[1] = *(str + 1);
> + ret = kstrtou8(tmp, 16, p);
> + if (ret)
> + return -EINVAL;
> + p++;
> + str -= 2;
> + }
> +
> + return 0;
> +}
> +
> +#endif
> diff --git a/drivers/fpga/xrt/metadata/metadata.c b/drivers/fpga/xrt/metadata/metadata.c
> new file mode 100644
> index 000000000000..5d106396f438
> --- /dev/null
> +++ b/drivers/fpga/xrt/metadata/metadata.c
> @@ -0,0 +1,524 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA Metadata parse APIs
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#include <linux/libfdt_env.h>
> +#include "libfdt.h"
> +#include "metadata.h"
> +
> +#define MAX_BLOB_SIZE (4096 * 25)
> +
> +static int xrt_md_setprop(struct device *dev, char *blob, int offset,
> + const char *prop, const void *val, int size);
> +static int xrt_md_overlay(struct device *dev, char *blob, int target,
> + const char *overlay_blob, int overlay_offset);
> +static int xrt_md_get_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + int *ep_offset);
> +
> +unsigned long xrt_md_size(struct device *dev, const char *blob)
> +{
> + unsigned long len = (long)fdt_totalsize(blob);
> +
> + len = (len > MAX_BLOB_SIZE) ? XRT_MD_INVALID_LENGTH : len;
how about (avoiding len = len):
if (len > MAX_BLOB_SIZE)
return XRT_MD_INVALID_LENGTH;
return len;
> + return len;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_size);
> +
> +int xrt_md_create(struct device *dev, char **blob)
> +{
> + int ret = 0;
> +
> + WARN_ON(!blob);
Shouldn't this be signaled to caller as a -EINVAL?
> +
> + *blob = vmalloc(MAX_BLOB_SIZE);
> + if (!*blob)
> + return -ENOMEM;
> +
> + ret = fdt_create_empty_tree(*blob, MAX_BLOB_SIZE);
> + if (ret) {
> + dev_err(dev, "format blob failed, ret = %d", ret);
> + goto failed;
> + }
> +
> + ret = fdt_next_node(*blob, -1, NULL);
> + if (ret < 0) {
> + dev_err(dev, "No Node, ret = %d", ret);
> + goto failed;
> + }
> +
> + ret = fdt_add_subnode(*blob, ret, XRT_MD_NODE_ENDPOINTS);
> + if (ret < 0) {
> + dev_err(dev, "add node failed, ret = %d", ret);
> + goto failed;
> + }
> +
> + return 0;
> +
> +failed:
> + vfree(*blob);
> + *blob = NULL;
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_create);
> +
> +static int xrt_md_add_node(struct device *dev, char *blob, int parent_offset,
> + const char *ep_name)
> +{
> + int ret;
> +
> + ret = fdt_add_subnode(blob, parent_offset, ep_name);
> + if (ret < 0 && ret != -FDT_ERR_EXISTS)
> + dev_err(dev, "failed to add node %s. %d", ep_name, ret);
> +
> + return ret;
> +}
> +
> +int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
> + char *regmap_name)
> +{
> + int ret;
> + int ep_offset;
> +
> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name, &ep_offset);
> + if (ret) {
> + dev_err(dev, "can not find ep %s", ep_name);
> + return -EINVAL;
> + }
> +
> + ret = fdt_del_node(blob, ep_offset);
> + if (ret)
> + dev_err(dev, "delete node %s failed, ret %d", ep_name, ret);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_del_endpoint);
> +
> +static int __xrt_md_add_endpoint(struct device *dev, char *blob,
> + struct xrt_md_endpoint *ep, int *offset, bool root)
> +{
> + int ret = 0;
> + int ep_offset = 0;
> + u32 val, count = 0;
> + u64 io_range[2];
> + char comp[128];
> +
> + if (!ep->ep_name) {
> + dev_err(dev, "empty name");
> + return -EINVAL;
> + }
> +
> + if (!root) {
> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
> + &ep_offset);
> + if (ret) {
> + dev_err(dev, "invalid blob, ret = %d", ret);
> + return -EINVAL;
> + }
> + }
> +
> + ep_offset = xrt_md_add_node(dev, blob, ep_offset, ep->ep_name);
A little hard to follow when you re-use ep_offset like this. Maybe
parent_offset to match xrt_md_add_node parameter?
> + if (ep_offset < 0) {
> + dev_err(dev, "add endpoint failed, ret = %d", ret);
> + return -EINVAL;
> + }
> + if (offset)
> + *offset = ep_offset;
> +
> + if (ep->size != 0) {
> + val = cpu_to_be32(ep->bar);
> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_BAR_IDX,
> + &val, sizeof(u32));
> + if (ret) {
> + dev_err(dev, "set %s failed, ret %d",
> + XRT_MD_PROP_BAR_IDX, ret);
> + goto failed;
> + }
> + io_range[0] = cpu_to_be64((u64)ep->bar_off);
> + io_range[1] = cpu_to_be64((u64)ep->size);
> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_IO_OFFSET,
> + io_range, sizeof(io_range));
> + if (ret) {
> + dev_err(dev, "set %s failed, ret %d",
> + XRT_MD_PROP_IO_OFFSET, ret);
> + goto failed;
> + }
> + }
> +
> + if (ep->regmap) {
> + if (ep->regmap_ver) {
> + count = snprintf(comp, sizeof(comp),
> + "%s-%s", ep->regmap, ep->regmap_ver);
> + count++;
> + }
> +
> + count += snprintf(comp + count, sizeof(comp) - count,
> + "%s", ep->regmap);
> + count++;
> +
> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_COMPATIBLE,
> + comp, count);
> + if (ret) {
> + dev_err(dev, "set %s failed, ret %d",
> + XRT_MD_PROP_COMPATIBLE, ret);
> + goto failed;
> + }
> + }
> +
> +failed:
> + if (ret)
> + xrt_md_del_endpoint(dev, blob, ep->ep_name, NULL);
> +
> + return ret;
> +}
> +
> +int xrt_md_add_endpoint(struct device *dev, char *blob,
> + struct xrt_md_endpoint *ep)
> +{
> + return __xrt_md_add_endpoint(dev, blob, ep, NULL, false);
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_add_endpoint);
> +
> +static int xrt_md_get_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + int *ep_offset)
> +{
> + int offset;
> + const char *name;
> +
> + for (offset = fdt_next_node(blob, -1, NULL);
> + offset >= 0;
> + offset = fdt_next_node(blob, offset, NULL)) {
> + name = fdt_get_name(blob, offset, NULL);
> + if (!name || strncmp(name, ep_name, strlen(ep_name) + 1))
> + continue;
> + if (!regmap_name ||
> + !fdt_node_check_compatible(blob, offset, regmap_name))
> + break;
> + }
> + if (offset < 0)
> + return -ENODEV;
> +
> + *ep_offset = offset;
> +
> + return 0;
> +}
> +
> +int xrt_md_find_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + const char **epname)
> +{
> + int offset;
> + int ret;
> +
> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
> + &offset);
> + if (!ret && epname && offset >= 0)
> + *epname = fdt_get_name(blob, offset, NULL);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_find_endpoint);
> +
> +int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
> + const char *regmap_name, const char *prop,
> + const void **val, int *size)
> +{
> + int offset;
> + int ret;
> +
> + if (val)
> + *val = NULL;
> + if (ep_name) {
> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
> + &offset);
> + if (ret) {
> + dev_err(dev, "cannot get ep %s, regmap %s, ret = %d",
> + ep_name, regmap_name, ret);
> + return -EINVAL;
> + }
> + } else {
> + offset = fdt_next_node(blob, -1, NULL);
> + if (offset < 0) {
> + dev_err(dev, "internal error, ret = %d", offset);
> + return -EINVAL;
> + }
> + }
> +
> + if (val) {
> + *val = fdt_getprop(blob, offset, prop, size);
> + if (!*val) {
> + dev_dbg(dev, "get ep %s, prop %s failed", ep_name, prop);
> + return -EINVAL;
> + }
> + }
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_get_prop);
> +
> +static int xrt_md_setprop(struct device *dev, char *blob, int offset,
> + const char *prop, const void *val, int size)
> +{
> + int ret;
> +
> + ret = fdt_setprop(blob, offset, prop, val, size);
> + if (ret)
> + dev_err(dev, "failed to set prop %d", ret);
> +
> + return ret;
> +}
> +
> +int xrt_md_set_prop(struct device *dev, char *blob,
> + const char *ep_name, const char *regmap_name,
> + const char *prop, const void *val, int size)
> +{
> + int offset;
> + int ret;
> +
> + if (ep_name) {
> + ret = xrt_md_get_endpoint(dev, blob, ep_name,
> + regmap_name, &offset);
> + if (ret) {
> + dev_err(dev, "cannot get node %s, ret = %d",
> + ep_name, ret);
> + return -EINVAL;
> + }
> + } else {
> + offset = fdt_next_node(blob, -1, NULL);
> + if (offset < 0) {
> + dev_err(dev, "internal error, ret = %d", offset);
> + return -EINVAL;
> + }
> + }
> +
> + ret = xrt_md_setprop(dev, blob, offset, prop, val, size);
> + if (ret)
> + dev_err(dev, "set prop %s failed, ret = %d", prop, ret);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_set_prop);
> +
> +int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
> + const char *ep_name, const char *regmap_name,
> + const char *new_ep_name)
> +{
> + int offset, target;
> + int ret;
> + struct xrt_md_endpoint ep = {0};
> + const char *newepnm = new_ep_name ? new_ep_name : ep_name;
> +
> + ret = xrt_md_get_endpoint(dev, src_blob, ep_name, regmap_name,
> + &offset);
> + if (ret)
> + return -EINVAL;
> +
> + ret = xrt_md_get_endpoint(dev, blob, newepnm, regmap_name, &target);
> + if (ret) {
> + ep.ep_name = newepnm;
> + ret = __xrt_md_add_endpoint(dev, blob, &ep, &target,
> + fdt_parent_offset(src_blob, offset) == 0);
> + if (ret)
> + return -EINVAL;
> + }
> +
> + ret = xrt_md_overlay(dev, blob, target, src_blob, offset);
> + if (ret)
> + dev_err(dev, "overlay failed, ret = %d", ret);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_copy_endpoint);
> +
> +char *xrt_md_dup(struct device *dev, const char *blob)
> +{
> + int ret;
> + char *dup_blob;
> +
> + ret = xrt_md_create(dev, &dup_blob);
> + if (ret)
> + return NULL;
> + ret = xrt_md_overlay(dev, dup_blob, -1, blob, -1);
> + if (ret) {
> + vfree(dup_blob);
> + return NULL;
> + }
> +
> + return dup_blob;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_dup);
> +
> +static int xrt_md_overlay(struct device *dev, char *blob, int target,
> + const char *overlay_blob, int overlay_offset)
> +{
> + int property, subnode;
> + int ret;
> +
> + WARN_ON(!blob || !overlay_blob);
Why check !blob twice? Also, is a WARN_ON justified here
(and in xrt_md_create)?
Fernando
> +
> + if (!blob) {
> + dev_err(dev, "blob is NULL");
> + return -EINVAL;
> + }
> +
> + if (target < 0) {
> + target = fdt_next_node(blob, -1, NULL);
> + if (target < 0) {
> + dev_err(dev, "invalid target");
> + return -EINVAL;
> + }
> + }
> + if (overlay_offset < 0) {
> + overlay_offset = fdt_next_node(overlay_blob, -1, NULL);
> + if (overlay_offset < 0) {
> + dev_err(dev, "invalid overlay");
> + return -EINVAL;
> + }
> + }
> +
> + fdt_for_each_property_offset(property, overlay_blob, overlay_offset) {
> + const char *name;
> + const void *prop;
> + int prop_len;
> +
> + prop = fdt_getprop_by_offset(overlay_blob, property, &name,
> + &prop_len);
> + if (!prop || prop_len >= MAX_BLOB_SIZE) {
> + dev_err(dev, "internal error");
> + return -EINVAL;
> + }
> +
> + ret = xrt_md_setprop(dev, blob, target, name, prop,
> + prop_len);
> + if (ret) {
> + dev_err(dev, "setprop failed, ret = %d", ret);
> + return ret;
> + }
> + }
> +
> + fdt_for_each_subnode(subnode, overlay_blob, overlay_offset) {
> + const char *name = fdt_get_name(overlay_blob, subnode, NULL);
> + int nnode;
> +
> + nnode = xrt_md_add_node(dev, blob, target, name);
> + if (nnode == -FDT_ERR_EXISTS)
> + nnode = fdt_subnode_offset(blob, target, name);
> + if (nnode < 0) {
> + dev_err(dev, "add node failed, ret = %d", nnode);
> + return nnode;
> + }
> +
> + ret = xrt_md_overlay(dev, blob, nnode, overlay_blob, subnode);
> + if (ret)
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
> + const char *ep_name, const char *regmap_name,
> + char **next_ep, char **next_regmap)
> +{
> + int offset, ret;
> +
> + if (!ep_name) {
> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
> + &offset);
> + } else {
> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
> + &offset);
> + }
> +
> + if (ret) {
> + *next_ep = NULL;
> + *next_regmap = NULL;
> + return -EINVAL;
> + }
> +
> + offset = ep_name ? fdt_next_subnode(blob, offset) :
> + fdt_first_subnode(blob, offset);
> + if (offset < 0) {
> + *next_ep = NULL;
> + *next_regmap = NULL;
> + return -EINVAL;
> + }
> +
> + *next_ep = (char *)fdt_get_name(blob, offset, NULL);
> + *next_regmap = (char *)fdt_stringlist_get(blob, offset, XRT_MD_PROP_COMPATIBLE,
> + 0, NULL);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_get_next_endpoint);
> +
> +int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
> + const char *regmap_name, const char **ep_name)
> +{
> + int ep_offset;
> +
> + ep_offset = fdt_node_offset_by_compatible(blob, -1, regmap_name);
> + if (ep_offset < 0) {
> + *ep_name = NULL;
> + return -ENOENT;
> + }
> +
> + *ep_name = (char *)fdt_get_name(blob, ep_offset, NULL);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_get_compatible_endpoint);
> +
> +void xrt_md_pack(struct device *dev, char *blob)
> +{
> + int ret;
> +
> + ret = fdt_pack(blob);
> + if (ret)
> + dev_err(dev, "pack failed %d", ret);
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_pack);
> +
> +int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
> + u32 *num_uuids, uuid_t *intf_uuids)
> +{
> + int offset, count = 0;
> + int ret;
> + const char *uuid_str;
> +
> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_INTERFACES, NULL, &offset);
> + if (ret)
> + return -ENOENT;
> +
> + for (offset = fdt_first_subnode(blob, offset);
> + offset >= 0;
> + offset = fdt_next_subnode(blob, offset)) {
> + uuid_str = fdt_getprop(blob, offset, XRT_MD_PROP_INTERFACE_UUID,
> + NULL);
> + if (!uuid_str) {
> + dev_err(dev, "empty intf uuid node");
> + return -EINVAL;
> + }
> +
> + if (intf_uuids && count < *num_uuids) {
> + ret = xrt_md_trans_str2uuid(dev, uuid_str,
> + &intf_uuids[count]);
> + if (ret)
> + return -EINVAL;
> + }
> + count++;
> + }
> +
> + *num_uuids = count;
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_md_get_intf_uuids);
> --
> 2.18.4
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions
2021-02-23 1:23 ` Fernando Pacheco
@ 2021-02-25 20:27 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-25 20:27 UTC (permalink / raw)
To: Fernando Pacheco, Lizhi Hou
Cc: linux-kernel, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Hi Fernando,
On 02/22/2021 05:23 PM, Fernando Pacheco wrote:
> On Wed, Feb 17, 2021 at 10:40:03PM -0800, Lizhi Hou wrote:
>> XRT drivers use device tree as metadata format to discover HW subsystems
>> behind PCIe BAR. Thus libfdt functions are called for driver to parse
>> device tree blob.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/metadata.h | 229 ++++++++++++
>> drivers/fpga/xrt/metadata/metadata.c | 524 +++++++++++++++++++++++++++
>> 2 files changed, 753 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/metadata.h
>> create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>>
>> diff --git a/drivers/fpga/xrt/include/metadata.h b/drivers/fpga/xrt/include/metadata.h
>> new file mode 100644
>> index 000000000000..b929bc469b73
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/metadata.h
>> @@ -0,0 +1,229 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for Xilinx Runtime (XRT) driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_METADATA_H
>> +#define _XRT_METADATA_H
>> +
>> +#include <linux/device.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/uuid.h>
>> +
>> +#define XRT_MD_INVALID_LENGTH (~0UL)
>> +
>> +#define XRT_MD_PROP_COMPATIBLE "compatible"
>> +#define XRT_MD_PROP_PF_NUM "pcie_physical_function"
>> +#define XRT_MD_PROP_BAR_IDX "pcie_bar_mapping"
>> +#define XRT_MD_PROP_IO_OFFSET "reg"
>> +#define XRT_MD_PROP_INTERRUPTS "interrupts"
>> +#define XRT_MD_PROP_INTERFACE_UUID "interface_uuid"
>> +#define XRT_MD_PROP_LOGIC_UUID "logic_uuid"
>> +#define XRT_MD_PROP_VERSION_MAJOR "firmware_version_major"
>> +
>> +#define XRT_MD_PROP_HWICAP "axi_hwicap"
>> +#define XRT_MD_PROP_PDI_CONFIG "pdi_config_mem"
>> +
>> +#define XRT_MD_NODE_ENDPOINTS "addressable_endpoints"
>> +#define XRT_MD_INTERFACES_PATH "/interfaces"
>> +
>> +#define XRT_MD_NODE_FIRMWARE "firmware"
>> +#define XRT_MD_NODE_INTERFACES "interfaces"
>> +#define XRT_MD_NODE_PARTITION_INFO "partition_info"
>> +
>> +#define XRT_MD_NODE_FLASH "ep_card_flash_program_00"
>> +#define XRT_MD_NODE_XVC_PUB "ep_debug_bscan_user_00"
>> +#define XRT_MD_NODE_XVC_PRI "ep_debug_bscan_mgmt_00"
>> +#define XRT_MD_NODE_SYSMON "ep_cmp_sysmon_00"
>> +#define XRT_MD_NODE_AF_BLP_CTRL_MGMT "ep_firewall_blp_ctrl_mgmt_00"
>> +#define XRT_MD_NODE_AF_BLP_CTRL_USER "ep_firewall_blp_ctrl_user_00"
>> +#define XRT_MD_NODE_AF_CTRL_MGMT "ep_firewall_ctrl_mgmt_00"
>> +#define XRT_MD_NODE_AF_CTRL_USER "ep_firewall_ctrl_user_00"
>> +#define XRT_MD_NODE_AF_CTRL_DEBUG "ep_firewall_ctrl_debug_00"
>> +#define XRT_MD_NODE_AF_DATA_H2C "ep_firewall_data_h2c_00"
>> +#define XRT_MD_NODE_AF_DATA_C2H "ep_firewall_data_c2h_00"
>> +#define XRT_MD_NODE_AF_DATA_P2P "ep_firewall_data_p2p_00"
>> +#define XRT_MD_NODE_AF_DATA_M2M "ep_firewall_data_m2m_00"
>> +#define XRT_MD_NODE_CMC_REG "ep_cmc_regmap_00"
>> +#define XRT_MD_NODE_CMC_RESET "ep_cmc_reset_00"
>> +#define XRT_MD_NODE_CMC_MUTEX "ep_cmc_mutex_00"
>> +#define XRT_MD_NODE_CMC_FW_MEM "ep_cmc_firmware_mem_00"
>> +#define XRT_MD_NODE_ERT_FW_MEM "ep_ert_firmware_mem_00"
>> +#define XRT_MD_NODE_ERT_CQ_MGMT "ep_ert_command_queue_mgmt_00"
>> +#define XRT_MD_NODE_ERT_CQ_USER "ep_ert_command_queue_user_00"
>> +#define XRT_MD_NODE_MAILBOX_MGMT "ep_mailbox_mgmt_00"
>> +#define XRT_MD_NODE_MAILBOX_USER "ep_mailbox_user_00"
>> +#define XRT_MD_NODE_GATE_PLP "ep_pr_isolate_plp_00"
>> +#define XRT_MD_NODE_GATE_ULP "ep_pr_isolate_ulp_00"
>> +#define XRT_MD_NODE_PCIE_MON "ep_pcie_link_mon_00"
>> +#define XRT_MD_NODE_DDR_CALIB "ep_ddr_mem_calib_00"
>> +#define XRT_MD_NODE_CLK_KERNEL1 "ep_aclk_kernel_00"
>> +#define XRT_MD_NODE_CLK_KERNEL2 "ep_aclk_kernel_01"
>> +#define XRT_MD_NODE_CLK_KERNEL3 "ep_aclk_hbm_00"
>> +#define XRT_MD_NODE_KDMA_CTRL "ep_kdma_ctrl_00"
>> +#define XRT_MD_NODE_FPGA_CONFIG "ep_fpga_configuration_00"
>> +#define XRT_MD_NODE_ERT_SCHED "ep_ert_sched_00"
>> +#define XRT_MD_NODE_XDMA "ep_xdma_00"
>> +#define XRT_MD_NODE_MSIX "ep_msix_00"
>> +#define XRT_MD_NODE_QDMA "ep_qdma_00"
>> +#define XRT_MD_XRT_MD_NODE_QDMA4 "ep_qdma4_00"
>> +#define XRT_MD_NODE_STM "ep_stream_traffic_manager_00"
>> +#define XRT_MD_NODE_STM4 "ep_stream_traffic_manager4_00"
>> +#define XRT_MD_NODE_CLK_SHUTDOWN "ep_aclk_shutdown_00"
>> +#define XRT_MD_NODE_ERT_BASE "ep_ert_base_address_00"
>> +#define XRT_MD_NODE_ERT_RESET "ep_ert_reset_00"
>> +#define XRT_MD_NODE_CLKFREQ_K1 "ep_freq_cnt_aclk_kernel_00"
>> +#define XRT_MD_NODE_CLKFREQ_K2 "ep_freq_cnt_aclk_kernel_01"
>> +#define XRT_MD_NODE_CLKFREQ_HBM "ep_freq_cnt_aclk_hbm_00"
>> +#define XRT_MD_NODE_GAPPING "ep_gapping_demand_00"
>> +#define XRT_MD_NODE_UCS_CONTROL_STATUS "ep_ucs_control_status_00"
>> +#define XRT_MD_NODE_P2P "ep_p2p_00"
>> +#define XRT_MD_NODE_REMAP_P2P "ep_remap_p2p_00"
>> +#define XRT_MD_NODE_DDR4_RESET_GATE "ep_ddr_mem_srsr_gate_00"
>> +#define XRT_MD_NODE_ADDR_TRANSLATOR "ep_remap_data_c2h_00"
>> +#define XRT_MD_NODE_MAILBOX_XRT "ep_mailbox_user_to_ert_00"
>> +#define XRT_MD_NODE_PMC_INTR "ep_pmc_intr_00"
>> +#define XRT_MD_NODE_PMC_MUX "ep_pmc_mux_00"
>> +
>> +/* driver defined endpoints */
>> +#define XRT_MD_NODE_VSEC "drv_ep_vsec_00"
>> +#define XRT_MD_NODE_VSEC_GOLDEN "drv_ep_vsec_golden_00"
>> +#define XRT_MD_NODE_BLP_ROM "drv_ep_blp_rom_00"
>> +#define XRT_MD_NODE_MAILBOX_VSEC "ep_mailbox_vsec_00"
> Should this be "drv_ep_mailbox_vsec_00"?
Will fix this.
>
>> +#define XRT_MD_NODE_PLAT_INFO "drv_ep_platform_info_mgmt_00"
>> +#define XRT_MD_NODE_TEST "drv_ep_test_00"
>> +#define XRT_MD_NODE_MGMT_MAIN "drv_ep_mgmt_main_00"
>> +#define XRT_MD_NODE_FLASH_VSEC "drv_ep_card_flash_program_00"
>> +#define XRT_MD_NODE_GOLDEN_VER "drv_ep_golden_ver_00"
>> +#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_BLP "partition_info_0"
>> +#define XRT_MD_XRT_MD_NODE_PARTITION_INFO_PLP "partition_info_1"
>> +
>> +#define XRT_MD_NODE_DDR_SRSR "drv_ep_ddr_srsr"
>> +#define XRT_MD_REGMAP_DDR_SRSR "drv_ddr_srsr"
>> +
>> +#define XRT_MD_PROP_OFFSET "drv_offset"
>> +#define XRT_MD_PROP_CLK_FREQ "drv_clock_frequency"
>> +#define XRT_MD_PROP_CLK_CNT "drv_clock_frequency_counter"
>> +#define XRT_MD_PROP_VBNV "vbnv"
>> +#define XRT_MD_PROP_VROM "vrom"
>> +#define XRT_MD_PROP_PARTITION_LEVEL "partition_level"
>> +
>> +struct xrt_md_endpoint {
>> + const char *ep_name;
>> + u32 bar;
>> + long bar_off;
>> + ulong size;
>> + char *regmap;
>> + char *regmap_ver;
>> +};
>> +
>> +/* Note: res_id is defined by leaf driver and must start with 0. */
>> +struct xrt_iores_map {
>> + char *res_name;
>> + int res_id;
>> +};
>> +
>> +static inline int xrt_md_res_name2id(const struct xrt_iores_map *res_map,
>> + int entry_num, const char *res_name)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < entry_num; i++) {
>> + if (!strcmp(res_name, res_map->res_name))
>> + return res_map->res_id;
>> + res_map++;
>> + }
>> + return -1;
>> +}
>> +
>> +static inline const char *
>> +xrt_md_res_id2name(const struct xrt_iores_map *res_map, int entry_num, int id)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < entry_num; i++) {
>> + if (res_map->res_id == id)
>> + return res_map->res_name;
>> + res_map++;
>> + }
>> + return NULL;
>> +}
>> +
>> +unsigned long xrt_md_size(struct device *dev, const char *blob);
>> +int xrt_md_create(struct device *dev, char **blob);
>> +int xrt_md_add_endpoint(struct device *dev, char *blob,
>> + struct xrt_md_endpoint *ep);
>> +int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
>> + char *regmap_name);
>> +int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
>> + const char *regmap_name, const char *prop,
>> + const void **val, int *size);
>> +int xrt_md_set_prop(struct device *dev, char *blob, const char *ep_name,
>> + const char *regmap_name, const char *prop,
>> + const void *val, int size);
>> +int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char *new_ep_name);
>> +int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + char **next_ep, char **next_regmap);
>> +int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
>> + const char *regmap_name, const char **ep_name);
>> +int xrt_md_find_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char **epname);
>> +void xrt_md_pack(struct device *dev, char *blob);
>> +char *xrt_md_dup(struct device *dev, const char *blob);
>> +int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
>> + u32 *num_uuids, uuid_t *intf_uuids);
>> +static inline int xrt_md_copy_all_endpoints(struct device *dev, char *blob, const char *src_blob)
>> +{
>> + return xrt_md_copy_endpoint(dev, blob, src_blob, XRT_MD_NODE_ENDPOINTS,
>> + NULL, NULL);
>> +}
>> +
>> +/*
>> + * Firmware provides 128 bit hash string as unque id of partition/interface.
>> + * This string will be canonical textual representation in the future.
>> + * Before that, introducing these two functions below to translate
>> + * hash string to uuid_t for released hardware.
>> + */
>> +static inline void xrt_md_trans_uuid2str(const uuid_t *uuid, char *uuidstr)
>> +{
>> + int i, p;
>> + u8 *u = (u8 *)uuid;
>> +
>> + for (p = 0, i = sizeof(uuid_t) - 1; i >= 0; p++, i--)
>> + (void)snprintf(&uuidstr[p * 2], 3, "%02x", u[i]);
>> +}
>> +
>> +static inline int xrt_md_trans_str2uuid(struct device *dev, const char *uuidstr, uuid_t *p_uuid)
>> +{
>> + char *p;
>> + const char *str;
>> + char tmp[3] = { 0 };
>> + int i, ret;
>> +
>> + memset(p_uuid, 0, sizeof(*p_uuid));
>> + p = (char *)p_uuid;
>> + str = uuidstr + strlen(uuidstr) - 2;
>> +
>> + for (i = 0; i < sizeof(*p_uuid) && str >= uuidstr; i++) {
>> + tmp[0] = *str;
>> + tmp[1] = *(str + 1);
>> + ret = kstrtou8(tmp, 16, p);
>> + if (ret)
>> + return -EINVAL;
>> + p++;
>> + str -= 2;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +#endif
>> diff --git a/drivers/fpga/xrt/metadata/metadata.c b/drivers/fpga/xrt/metadata/metadata.c
>> new file mode 100644
>> index 000000000000..5d106396f438
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/metadata/metadata.c
>> @@ -0,0 +1,524 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA Metadata parse APIs
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#include <linux/libfdt_env.h>
>> +#include "libfdt.h"
>> +#include "metadata.h"
>> +
>> +#define MAX_BLOB_SIZE (4096 * 25)
>> +
>> +static int xrt_md_setprop(struct device *dev, char *blob, int offset,
>> + const char *prop, const void *val, int size);
>> +static int xrt_md_overlay(struct device *dev, char *blob, int target,
>> + const char *overlay_blob, int overlay_offset);
>> +static int xrt_md_get_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + int *ep_offset);
>> +
>> +unsigned long xrt_md_size(struct device *dev, const char *blob)
>> +{
>> + unsigned long len = (long)fdt_totalsize(blob);
>> +
>> + len = (len > MAX_BLOB_SIZE) ? XRT_MD_INVALID_LENGTH : len;
> how about (avoiding len = len):
>
> if (len > MAX_BLOB_SIZE)
> return XRT_MD_INVALID_LENGTH;
>
> return len;
Agree. Will fix.
>> + return len;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_size);
>> +
>> +int xrt_md_create(struct device *dev, char **blob)
>> +{
>> + int ret = 0;
>> +
>> + WARN_ON(!blob);
> Shouldn't this be signaled to caller as a -EINVAL?
Agree. Will fix it.
>
>> +
>> + *blob = vmalloc(MAX_BLOB_SIZE);
>> + if (!*blob)
>> + return -ENOMEM;
>> +
>> + ret = fdt_create_empty_tree(*blob, MAX_BLOB_SIZE);
>> + if (ret) {
>> + dev_err(dev, "format blob failed, ret = %d", ret);
>> + goto failed;
>> + }
>> +
>> + ret = fdt_next_node(*blob, -1, NULL);
>> + if (ret < 0) {
>> + dev_err(dev, "No Node, ret = %d", ret);
>> + goto failed;
>> + }
>> +
>> + ret = fdt_add_subnode(*blob, ret, XRT_MD_NODE_ENDPOINTS);
>> + if (ret < 0) {
>> + dev_err(dev, "add node failed, ret = %d", ret);
>> + goto failed;
>> + }
>> +
>> + return 0;
>> +
>> +failed:
>> + vfree(*blob);
>> + *blob = NULL;
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_create);
>> +
>> +static int xrt_md_add_node(struct device *dev, char *blob, int parent_offset,
>> + const char *ep_name)
>> +{
>> + int ret;
>> +
>> + ret = fdt_add_subnode(blob, parent_offset, ep_name);
>> + if (ret < 0 && ret != -FDT_ERR_EXISTS)
>> + dev_err(dev, "failed to add node %s. %d", ep_name, ret);
>> +
>> + return ret;
>> +}
>> +
>> +int xrt_md_del_endpoint(struct device *dev, char *blob, const char *ep_name,
>> + char *regmap_name)
>> +{
>> + int ret;
>> + int ep_offset;
>> +
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name, &ep_offset);
>> + if (ret) {
>> + dev_err(dev, "can not find ep %s", ep_name);
>> + return -EINVAL;
>> + }
>> +
>> + ret = fdt_del_node(blob, ep_offset);
>> + if (ret)
>> + dev_err(dev, "delete node %s failed, ret %d", ep_name, ret);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_del_endpoint);
>> +
>> +static int __xrt_md_add_endpoint(struct device *dev, char *blob,
>> + struct xrt_md_endpoint *ep, int *offset, bool root)
>> +{
>> + int ret = 0;
>> + int ep_offset = 0;
>> + u32 val, count = 0;
>> + u64 io_range[2];
>> + char comp[128];
>> +
>> + if (!ep->ep_name) {
>> + dev_err(dev, "empty name");
>> + return -EINVAL;
>> + }
>> +
>> + if (!root) {
>> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
>> + &ep_offset);
>> + if (ret) {
>> + dev_err(dev, "invalid blob, ret = %d", ret);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + ep_offset = xrt_md_add_node(dev, blob, ep_offset, ep->ep_name);
> A little hard to follow when you re-use ep_offset like this. Maybe
> parent_offset to match xrt_md_add_node parameter?
Will add parent_offset.
>
>> + if (ep_offset < 0) {
>> + dev_err(dev, "add endpoint failed, ret = %d", ret);
>> + return -EINVAL;
>> + }
>> + if (offset)
>> + *offset = ep_offset;
>> +
>> + if (ep->size != 0) {
>> + val = cpu_to_be32(ep->bar);
>> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_BAR_IDX,
>> + &val, sizeof(u32));
>> + if (ret) {
>> + dev_err(dev, "set %s failed, ret %d",
>> + XRT_MD_PROP_BAR_IDX, ret);
>> + goto failed;
>> + }
>> + io_range[0] = cpu_to_be64((u64)ep->bar_off);
>> + io_range[1] = cpu_to_be64((u64)ep->size);
>> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_IO_OFFSET,
>> + io_range, sizeof(io_range));
>> + if (ret) {
>> + dev_err(dev, "set %s failed, ret %d",
>> + XRT_MD_PROP_IO_OFFSET, ret);
>> + goto failed;
>> + }
>> + }
>> +
>> + if (ep->regmap) {
>> + if (ep->regmap_ver) {
>> + count = snprintf(comp, sizeof(comp),
>> + "%s-%s", ep->regmap, ep->regmap_ver);
>> + count++;
>> + }
>> +
>> + count += snprintf(comp + count, sizeof(comp) - count,
>> + "%s", ep->regmap);
>> + count++;
>> +
>> + ret = xrt_md_setprop(dev, blob, ep_offset, XRT_MD_PROP_COMPATIBLE,
>> + comp, count);
>> + if (ret) {
>> + dev_err(dev, "set %s failed, ret %d",
>> + XRT_MD_PROP_COMPATIBLE, ret);
>> + goto failed;
>> + }
>> + }
>> +
>> +failed:
>> + if (ret)
>> + xrt_md_del_endpoint(dev, blob, ep->ep_name, NULL);
>> +
>> + return ret;
>> +}
>> +
>> +int xrt_md_add_endpoint(struct device *dev, char *blob,
>> + struct xrt_md_endpoint *ep)
>> +{
>> + return __xrt_md_add_endpoint(dev, blob, ep, NULL, false);
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_add_endpoint);
>> +
>> +static int xrt_md_get_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + int *ep_offset)
>> +{
>> + int offset;
>> + const char *name;
>> +
>> + for (offset = fdt_next_node(blob, -1, NULL);
>> + offset >= 0;
>> + offset = fdt_next_node(blob, offset, NULL)) {
>> + name = fdt_get_name(blob, offset, NULL);
>> + if (!name || strncmp(name, ep_name, strlen(ep_name) + 1))
>> + continue;
>> + if (!regmap_name ||
>> + !fdt_node_check_compatible(blob, offset, regmap_name))
>> + break;
>> + }
>> + if (offset < 0)
>> + return -ENODEV;
>> +
>> + *ep_offset = offset;
>> +
>> + return 0;
>> +}
>> +
>> +int xrt_md_find_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char **epname)
>> +{
>> + int offset;
>> + int ret;
>> +
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
>> + &offset);
>> + if (!ret && epname && offset >= 0)
>> + *epname = fdt_get_name(blob, offset, NULL);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_find_endpoint);
>> +
>> +int xrt_md_get_prop(struct device *dev, const char *blob, const char *ep_name,
>> + const char *regmap_name, const char *prop,
>> + const void **val, int *size)
>> +{
>> + int offset;
>> + int ret;
>> +
>> + if (val)
>> + *val = NULL;
>> + if (ep_name) {
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
>> + &offset);
>> + if (ret) {
>> + dev_err(dev, "cannot get ep %s, regmap %s, ret = %d",
>> + ep_name, regmap_name, ret);
>> + return -EINVAL;
>> + }
>> + } else {
>> + offset = fdt_next_node(blob, -1, NULL);
>> + if (offset < 0) {
>> + dev_err(dev, "internal error, ret = %d", offset);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + if (val) {
>> + *val = fdt_getprop(blob, offset, prop, size);
>> + if (!*val) {
>> + dev_dbg(dev, "get ep %s, prop %s failed", ep_name, prop);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_get_prop);
>> +
>> +static int xrt_md_setprop(struct device *dev, char *blob, int offset,
>> + const char *prop, const void *val, int size)
>> +{
>> + int ret;
>> +
>> + ret = fdt_setprop(blob, offset, prop, val, size);
>> + if (ret)
>> + dev_err(dev, "failed to set prop %d", ret);
>> +
>> + return ret;
>> +}
>> +
>> +int xrt_md_set_prop(struct device *dev, char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char *prop, const void *val, int size)
>> +{
>> + int offset;
>> + int ret;
>> +
>> + if (ep_name) {
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name,
>> + regmap_name, &offset);
>> + if (ret) {
>> + dev_err(dev, "cannot get node %s, ret = %d",
>> + ep_name, ret);
>> + return -EINVAL;
>> + }
>> + } else {
>> + offset = fdt_next_node(blob, -1, NULL);
>> + if (offset < 0) {
>> + dev_err(dev, "internal error, ret = %d", offset);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + ret = xrt_md_setprop(dev, blob, offset, prop, val, size);
>> + if (ret)
>> + dev_err(dev, "set prop %s failed, ret = %d", prop, ret);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_set_prop);
>> +
>> +int xrt_md_copy_endpoint(struct device *dev, char *blob, const char *src_blob,
>> + const char *ep_name, const char *regmap_name,
>> + const char *new_ep_name)
>> +{
>> + int offset, target;
>> + int ret;
>> + struct xrt_md_endpoint ep = {0};
>> + const char *newepnm = new_ep_name ? new_ep_name : ep_name;
>> +
>> + ret = xrt_md_get_endpoint(dev, src_blob, ep_name, regmap_name,
>> + &offset);
>> + if (ret)
>> + return -EINVAL;
>> +
>> + ret = xrt_md_get_endpoint(dev, blob, newepnm, regmap_name, &target);
>> + if (ret) {
>> + ep.ep_name = newepnm;
>> + ret = __xrt_md_add_endpoint(dev, blob, &ep, &target,
>> + fdt_parent_offset(src_blob, offset) == 0);
>> + if (ret)
>> + return -EINVAL;
>> + }
>> +
>> + ret = xrt_md_overlay(dev, blob, target, src_blob, offset);
>> + if (ret)
>> + dev_err(dev, "overlay failed, ret = %d", ret);
>> +
>> + return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_copy_endpoint);
>> +
>> +char *xrt_md_dup(struct device *dev, const char *blob)
>> +{
>> + int ret;
>> + char *dup_blob;
>> +
>> + ret = xrt_md_create(dev, &dup_blob);
>> + if (ret)
>> + return NULL;
>> + ret = xrt_md_overlay(dev, dup_blob, -1, blob, -1);
>> + if (ret) {
>> + vfree(dup_blob);
>> + return NULL;
>> + }
>> +
>> + return dup_blob;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_dup);
>> +
>> +static int xrt_md_overlay(struct device *dev, char *blob, int target,
>> + const char *overlay_blob, int overlay_offset)
>> +{
>> + int property, subnode;
>> + int ret;
>> +
>> + WARN_ON(!blob || !overlay_blob);
> Why check !blob twice? Also, is a WARN_ON justified here
> (and in xrt_md_create)?
Using 'if' statement to check is good enough and I will remove WARN_ON
from both places.
Thanks,
Lizhi
>
> Fernando
>
>> +
>> + if (!blob) {
>> + dev_err(dev, "blob is NULL");
>> + return -EINVAL;
>> + }
>> +
>> + if (target < 0) {
>> + target = fdt_next_node(blob, -1, NULL);
>> + if (target < 0) {
>> + dev_err(dev, "invalid target");
>> + return -EINVAL;
>> + }
>> + }
>> + if (overlay_offset < 0) {
>> + overlay_offset = fdt_next_node(overlay_blob, -1, NULL);
>> + if (overlay_offset < 0) {
>> + dev_err(dev, "invalid overlay");
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + fdt_for_each_property_offset(property, overlay_blob, overlay_offset) {
>> + const char *name;
>> + const void *prop;
>> + int prop_len;
>> +
>> + prop = fdt_getprop_by_offset(overlay_blob, property, &name,
>> + &prop_len);
>> + if (!prop || prop_len >= MAX_BLOB_SIZE) {
>> + dev_err(dev, "internal error");
>> + return -EINVAL;
>> + }
>> +
>> + ret = xrt_md_setprop(dev, blob, target, name, prop,
>> + prop_len);
>> + if (ret) {
>> + dev_err(dev, "setprop failed, ret = %d", ret);
>> + return ret;
>> + }
>> + }
>> +
>> + fdt_for_each_subnode(subnode, overlay_blob, overlay_offset) {
>> + const char *name = fdt_get_name(overlay_blob, subnode, NULL);
>> + int nnode;
>> +
>> + nnode = xrt_md_add_node(dev, blob, target, name);
>> + if (nnode == -FDT_ERR_EXISTS)
>> + nnode = fdt_subnode_offset(blob, target, name);
>> + if (nnode < 0) {
>> + dev_err(dev, "add node failed, ret = %d", nnode);
>> + return nnode;
>> + }
>> +
>> + ret = xrt_md_overlay(dev, blob, nnode, overlay_blob, subnode);
>> + if (ret)
>> + return ret;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +int xrt_md_get_next_endpoint(struct device *dev, const char *blob,
>> + const char *ep_name, const char *regmap_name,
>> + char **next_ep, char **next_regmap)
>> +{
>> + int offset, ret;
>> +
>> + if (!ep_name) {
>> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_ENDPOINTS, NULL,
>> + &offset);
>> + } else {
>> + ret = xrt_md_get_endpoint(dev, blob, ep_name, regmap_name,
>> + &offset);
>> + }
>> +
>> + if (ret) {
>> + *next_ep = NULL;
>> + *next_regmap = NULL;
>> + return -EINVAL;
>> + }
>> +
>> + offset = ep_name ? fdt_next_subnode(blob, offset) :
>> + fdt_first_subnode(blob, offset);
>> + if (offset < 0) {
>> + *next_ep = NULL;
>> + *next_regmap = NULL;
>> + return -EINVAL;
>> + }
>> +
>> + *next_ep = (char *)fdt_get_name(blob, offset, NULL);
>> + *next_regmap = (char *)fdt_stringlist_get(blob, offset, XRT_MD_PROP_COMPATIBLE,
>> + 0, NULL);
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_get_next_endpoint);
>> +
>> +int xrt_md_get_compatible_endpoint(struct device *dev, const char *blob,
>> + const char *regmap_name, const char **ep_name)
>> +{
>> + int ep_offset;
>> +
>> + ep_offset = fdt_node_offset_by_compatible(blob, -1, regmap_name);
>> + if (ep_offset < 0) {
>> + *ep_name = NULL;
>> + return -ENOENT;
>> + }
>> +
>> + *ep_name = (char *)fdt_get_name(blob, ep_offset, NULL);
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_get_compatible_endpoint);
>> +
>> +void xrt_md_pack(struct device *dev, char *blob)
>> +{
>> + int ret;
>> +
>> + ret = fdt_pack(blob);
>> + if (ret)
>> + dev_err(dev, "pack failed %d", ret);
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_pack);
>> +
>> +int xrt_md_get_intf_uuids(struct device *dev, const char *blob,
>> + u32 *num_uuids, uuid_t *intf_uuids)
>> +{
>> + int offset, count = 0;
>> + int ret;
>> + const char *uuid_str;
>> +
>> + ret = xrt_md_get_endpoint(dev, blob, XRT_MD_NODE_INTERFACES, NULL, &offset);
>> + if (ret)
>> + return -ENOENT;
>> +
>> + for (offset = fdt_first_subnode(blob, offset);
>> + offset >= 0;
>> + offset = fdt_next_subnode(blob, offset)) {
>> + uuid_str = fdt_getprop(blob, offset, XRT_MD_PROP_INTERFACE_UUID,
>> + NULL);
>> + if (!uuid_str) {
>> + dev_err(dev, "empty intf uuid node");
>> + return -EINVAL;
>> + }
>> +
>> + if (intf_uuids && count < *num_uuids) {
>> + ret = xrt_md_trans_str2uuid(dev, uuid_str,
>> + &intf_uuids[count]);
>> + if (ret)
>> + return -EINVAL;
>> + }
>> + count++;
>> + }
>> +
>> + *num_uuids = count;
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_md_get_intf_uuids);
>> --
>> 2.18.4
>>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 01/18] Documentation: fpga: Add a document describing XRT Alveo drivers Lizhi Hou
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 02/18] fpga: xrt: driver metadata helper functions Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-21 17:12 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager Lizhi Hou
` (16 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Alveo FPGA firmware and partial reconfigure file are in xclbin format. Add
code to enumerate and extract sections from xclbin files. xclbin.h is cross
platform and used across all platforms and OS
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xclbin-helper.h | 52 +++
drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++++++++++++++
include/uapi/linux/xrt/xclbin.h | 408 +++++++++++++++++++++++
3 files changed, 854 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
create mode 100644 drivers/fpga/xrt/lib/xclbin.c
create mode 100644 include/uapi/linux/xrt/xclbin.h
diff --git a/drivers/fpga/xrt/include/xclbin-helper.h b/drivers/fpga/xrt/include/xclbin-helper.h
new file mode 100644
index 000000000000..68218efc9d0b
--- /dev/null
+++ b/drivers/fpga/xrt/include/xclbin-helper.h
@@ -0,0 +1,52 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * David Zhang <davidzha@xilinx.com>
+ * Sonal Santan <sonal.santan@xilinx.com>
+ */
+
+#ifndef _XRT_XCLBIN_H
+#define _XRT_XCLBIN_H
+
+#include <linux/types.h>
+#include <linux/device.h>
+#include <linux/xrt/xclbin.h>
+
+#define ICAP_XCLBIN_V2 "xclbin2"
+#define DMA_HWICAP_BITFILE_BUFFER_SIZE 1024
+#define MAX_XCLBIN_SIZE (1024 * 1024 * 1024) /* Assuming xclbin <= 1G, always */
+
+enum axlf_section_kind;
+struct axlf;
+
+/**
+ * Bitstream header information as defined by Xilinx tools.
+ * Please note that this struct definition is not owned by the driver.
+ */
+struct hw_icap_bit_header {
+ unsigned int header_length; /* Length of header in 32 bit words */
+ unsigned int bitstream_length; /* Length of bitstream to read in bytes*/
+ unsigned char *design_name; /* Design name get from bitstream */
+ unsigned char *part_name; /* Part name read from bitstream */
+ unsigned char *date; /* Date read from bitstream header */
+ unsigned char *time; /* Bitstream creation time */
+ unsigned int magic_length; /* Length of the magic numbers */
+ unsigned char *version; /* Version string */
+};
+
+const char *xrt_xclbin_kind_to_string(enum axlf_section_kind kind);
+int xrt_xclbin_get_section(const struct axlf *xclbin,
+ enum axlf_section_kind kind, void **data,
+ uint64_t *len);
+int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb);
+int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
+ unsigned int size,
+ struct hw_icap_bit_header *header);
+void xrt_xclbin_free_header(struct hw_icap_bit_header *header);
+const char *xrt_clock_type2epname(enum CLOCK_TYPE type);
+
+#endif /* _XRT_XCLBIN_H */
diff --git a/drivers/fpga/xrt/lib/xclbin.c b/drivers/fpga/xrt/lib/xclbin.c
new file mode 100644
index 000000000000..47dc6ca25c1b
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xclbin.c
@@ -0,0 +1,394 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Driver XCLBIN parser
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors: David Zhang <davidzha@xilinx.com>
+ */
+
+#include <asm/errno.h>
+#include <linux/vmalloc.h>
+#include <linux/device.h>
+#include "xclbin-helper.h"
+#include "metadata.h"
+
+/* Used for parsing bitstream header */
+#define XHI_EVEN_MAGIC_BYTE 0x0f
+#define XHI_ODD_MAGIC_BYTE 0xf0
+
+/* Extra mode for IDLE */
+#define XHI_OP_IDLE -1
+#define XHI_BIT_HEADER_FAILURE -1
+
+/* The imaginary module length register */
+#define XHI_MLR 15
+
+static inline unsigned char xhi_data_and_inc(const unsigned char *d, int *i, int sz)
+{
+ unsigned char data;
+
+ if (*i >= sz)
+ return -1;
+
+ data = d[*i];
+ (*i)++;
+
+ return data;
+}
+
+static const struct axlf_section_header *
+xrt_xclbin_get_section_hdr(const struct axlf *xclbin,
+ enum axlf_section_kind kind)
+{
+ int i = 0;
+
+ for (i = 0; i < xclbin->m_header.m_numSections; i++) {
+ if (xclbin->m_sections[i].m_sectionKind == kind)
+ return &xclbin->m_sections[i];
+ }
+
+ return NULL;
+}
+
+static int
+xrt_xclbin_check_section_hdr(const struct axlf_section_header *header,
+ u64 xclbin_len)
+{
+ int ret;
+
+ ret = (header->m_sectionOffset + header->m_sectionSize) > xclbin_len ? -EINVAL : 0;
+
+ return ret;
+}
+
+static int xrt_xclbin_section_info(const struct axlf *xclbin,
+ enum axlf_section_kind kind,
+ u64 *offset, u64 *size)
+{
+ const struct axlf_section_header *mem_header = NULL;
+ u64 xclbin_len;
+ int err = 0;
+
+ mem_header = xrt_xclbin_get_section_hdr(xclbin, kind);
+ if (!mem_header)
+ return -EINVAL;
+
+ xclbin_len = xclbin->m_header.m_length;
+ if (xclbin_len > MAX_XCLBIN_SIZE)
+ return -EINVAL;
+
+ err = xrt_xclbin_check_section_hdr(mem_header, xclbin_len);
+ if (err)
+ return err;
+
+ *offset = mem_header->m_sectionOffset;
+ *size = mem_header->m_sectionSize;
+
+ return 0;
+}
+
+/* caller should free the allocated memory for **data */
+int xrt_xclbin_get_section(const struct axlf *buf,
+ enum axlf_section_kind kind,
+ void **data, u64 *len)
+{
+ const struct axlf *xclbin = (const struct axlf *)buf;
+ void *section = NULL;
+ int err = 0;
+ u64 offset = 0;
+ u64 size = 0;
+
+ err = xrt_xclbin_section_info(xclbin, kind, &offset, &size);
+ if (err)
+ return err;
+
+ section = vmalloc(size);
+ if (!section)
+ return -ENOMEM;
+
+ memcpy(section, ((const char *)xclbin) + offset, size);
+
+ *data = section;
+ if (len)
+ *len = size;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_xclbin_get_section);
+
+/* parse bitstream header */
+int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
+ unsigned int size,
+ struct hw_icap_bit_header *header)
+{
+ unsigned int index;
+ unsigned int len;
+ unsigned int tmp;
+ unsigned int i;
+
+ memset(header, 0, sizeof(*header));
+ /* Start Index at start of bitstream */
+ index = 0;
+
+ /* Initialize HeaderLength. If header returned early inidicates
+ * failure.
+ */
+ header->header_length = XHI_BIT_HEADER_FAILURE;
+
+ /* Get "Magic" length */
+ header->magic_length = xhi_data_and_inc(data, &index, size);
+ header->magic_length = (header->magic_length << 8) | xhi_data_and_inc(data, &index, size);
+
+ /* Read in "magic" */
+ for (i = 0; i < header->magic_length - 1; i++) {
+ tmp = xhi_data_and_inc(data, &index, size);
+ if (i % 2 == 0 && tmp != XHI_EVEN_MAGIC_BYTE)
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ if (i % 2 == 1 && tmp != XHI_ODD_MAGIC_BYTE)
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+ }
+
+ /* Read null end of magic data. */
+ tmp = xhi_data_and_inc(data, &index, size);
+
+ /* Read 0x01 (short) */
+ tmp = xhi_data_and_inc(data, &index, size);
+ tmp = (tmp << 8) | xhi_data_and_inc(data, &index, size);
+
+ /* Check the "0x01" half word */
+ if (tmp != 0x01)
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Read 'a' */
+ tmp = xhi_data_and_inc(data, &index, size);
+ if (tmp != 'a')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get Design Name length */
+ len = xhi_data_and_inc(data, &index, size);
+ len = (len << 8) | xhi_data_and_inc(data, &index, size);
+
+ /* allocate space for design name and final null character. */
+ header->design_name = vmalloc(len);
+ if (!header->design_name)
+ return -ENOMEM;
+
+ /* Read in Design Name */
+ for (i = 0; i < len; i++)
+ header->design_name[i] = xhi_data_and_inc(data, &index, size);
+
+ if (header->design_name[len - 1] != '\0')
+ return -1;
+
+ header->version = strstr(header->design_name, "Version=") + strlen("Version=");
+
+ /* Read 'b' */
+ tmp = xhi_data_and_inc(data, &index, size);
+ if (tmp != 'b')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get Part Name length */
+ len = xhi_data_and_inc(data, &index, size);
+ len = (len << 8) | xhi_data_and_inc(data, &index, size);
+
+ /* allocate space for part name and final null character. */
+ header->part_name = vmalloc(len);
+ if (!header->part_name)
+ return -ENOMEM;
+
+ /* Read in part name */
+ for (i = 0; i < len; i++)
+ header->part_name[i] = xhi_data_and_inc(data, &index, size);
+
+ if (header->part_name[len - 1] != '\0')
+ return -1;
+
+ /* Read 'c' */
+ tmp = xhi_data_and_inc(data, &index, size);
+ if (tmp != 'c')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get date length */
+ len = xhi_data_and_inc(data, &index, size);
+ len = (len << 8) | xhi_data_and_inc(data, &index, size);
+
+ /* allocate space for date and final null character. */
+ header->date = vmalloc(len);
+ if (!header->date)
+ return -ENOMEM;
+
+ /* Read in date name */
+ for (i = 0; i < len; i++)
+ header->date[i] = xhi_data_and_inc(data, &index, size);
+
+ if (header->date[len - 1] != '\0')
+ return -1;
+
+ /* Read 'd' */
+ tmp = xhi_data_and_inc(data, &index, size);
+ if (tmp != 'd')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get time length */
+ len = xhi_data_and_inc(data, &index, size);
+ len = (len << 8) | xhi_data_and_inc(data, &index, size);
+
+ /* allocate space for time and final null character. */
+ header->time = vmalloc(len);
+ if (!header->time)
+ return -ENOMEM;
+
+ /* Read in time name */
+ for (i = 0; i < len; i++)
+ header->time[i] = xhi_data_and_inc(data, &index, size);
+
+ if (header->time[len - 1] != '\0')
+ return -1;
+
+ /* Read 'e' */
+ tmp = xhi_data_and_inc(data, &index, size);
+ if (tmp != 'e')
+ return -1; /* INVALID_FILE_HEADER_ERROR */
+
+ /* Get byte length of bitstream */
+ header->bitstream_length = xhi_data_and_inc(data, &index, size);
+ header->bitstream_length = (header->bitstream_length << 8) |
+ xhi_data_and_inc(data, &index, size);
+ header->bitstream_length = (header->bitstream_length << 8) |
+ xhi_data_and_inc(data, &index, size);
+ header->bitstream_length = (header->bitstream_length << 8) |
+ xhi_data_and_inc(data, &index, size);
+
+ header->header_length = index;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_xclbin_parse_bitstream_header);
+
+void xrt_xclbin_free_header(struct hw_icap_bit_header *header)
+{
+ vfree(header->design_name);
+ vfree(header->part_name);
+ vfree(header->date);
+ vfree(header->time);
+}
+EXPORT_SYMBOL_GPL(xrt_xclbin_free_header);
+
+struct xrt_clock_desc {
+ char *clock_ep_name;
+ u32 clock_xclbin_type;
+ char *clkfreq_ep_name;
+} clock_desc[] = {
+ {
+ .clock_ep_name = XRT_MD_NODE_CLK_KERNEL1,
+ .clock_xclbin_type = CT_DATA,
+ .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K1,
+ },
+ {
+ .clock_ep_name = XRT_MD_NODE_CLK_KERNEL2,
+ .clock_xclbin_type = CT_KERNEL,
+ .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K2,
+ },
+ {
+ .clock_ep_name = XRT_MD_NODE_CLK_KERNEL3,
+ .clock_xclbin_type = CT_SYSTEM,
+ .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_HBM,
+ },
+};
+
+const char *xrt_clock_type2epname(enum CLOCK_TYPE type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
+ if (clock_desc[i].clock_xclbin_type == type)
+ return clock_desc[i].clock_ep_name;
+ }
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(xrt_clock_type2epname);
+
+static const char *clock_type2clkfreq_name(u32 type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
+ if (clock_desc[i].clock_xclbin_type == type)
+ return clock_desc[i].clkfreq_ep_name;
+ }
+ return NULL;
+}
+
+static int xrt_xclbin_add_clock_metadata(struct device *dev,
+ const struct axlf *xclbin,
+ char *dtb)
+{
+ int i;
+ u16 freq;
+ struct clock_freq_topology *clock_topo;
+ int rc = xrt_xclbin_get_section(xclbin, CLOCK_FREQ_TOPOLOGY,
+ (void **)&clock_topo, NULL);
+
+ if (rc)
+ return 0;
+
+ for (i = 0; i < clock_topo->m_count; i++) {
+ u8 type = clock_topo->m_clock_freq[i].m_type;
+ const char *ep_name = xrt_clock_type2epname(type);
+ const char *counter_name = clock_type2clkfreq_name(type);
+
+ if (!ep_name || !counter_name)
+ continue;
+
+ freq = cpu_to_be16(clock_topo->m_clock_freq[i].m_freq_Mhz);
+ rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_FREQ,
+ &freq, sizeof(freq));
+ if (rc)
+ break;
+
+ rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_CNT,
+ counter_name, strlen(counter_name) + 1);
+ if (rc)
+ break;
+ }
+
+ vfree(clock_topo);
+
+ return rc;
+}
+
+int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb)
+{
+ char *md = NULL, *newmd = NULL;
+ u64 len;
+ int rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA,
+ (void **)&md, &len);
+
+ if (rc)
+ goto done;
+
+ /* Sanity check the dtb section. */
+ if (xrt_md_size(dev, md) > len) {
+ rc = -EINVAL;
+ goto done;
+ }
+
+ newmd = xrt_md_dup(dev, md);
+ if (!newmd) {
+ rc = -EFAULT;
+ goto done;
+ }
+ /* Convert various needed xclbin sections into dtb. */
+ rc = xrt_xclbin_add_clock_metadata(dev, xclbin, newmd);
+
+done:
+ if (rc == 0)
+ *dtb = newmd;
+ else
+ vfree(newmd);
+ vfree(md);
+ return rc;
+}
+EXPORT_SYMBOL_GPL(xrt_xclbin_get_metadata);
diff --git a/include/uapi/linux/xrt/xclbin.h b/include/uapi/linux/xrt/xclbin.h
new file mode 100644
index 000000000000..53f140123ef1
--- /dev/null
+++ b/include/uapi/linux/xrt/xclbin.h
@@ -0,0 +1,408 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Xilinx FPGA compiled binary container format
+ *
+ * Copyright (C) 2015-2021, Xilinx Inc
+ */
+
+#ifndef _XCLBIN_H_
+#define _XCLBIN_H_
+
+#ifdef _WIN32
+ #include <cstdint>
+ #include <algorithm>
+ #include "windows/uuid.h"
+#else
+ #if defined(__KERNEL__)
+ #include <linux/types.h>
+ #include <linux/uuid.h>
+ #include <linux/version.h>
+ #elif defined(__cplusplus)
+ #include <cstdlib>
+ #include <cstdint>
+ #include <algorithm>
+ #include <uuid/uuid.h>
+ #else
+ #include <stdlib.h>
+ #include <stdint.h>
+ #include <uuid/uuid.h>
+ #endif
+#endif
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * DOC: Container format for Xilinx FPGA images
+ * The container stores bitstreams, metadata and firmware images.
+ * xclbin/xsabin is ELF-like binary container format. It is structured
+ * series of sections. There is a file header followed by several section
+ * headers which is followed by sections. A section header points to an
+ * actual section. There is an optional signature at the end. The
+ * following figure illustrates a typical xclbin:
+ *
+ * +---------------------+
+ * | |
+ * | HEADER |
+ * +---------------------+
+ * | SECTION HEADER |
+ * | |
+ * +---------------------+
+ * | ... |
+ * | |
+ * +---------------------+
+ * | SECTION HEADER |
+ * | |
+ * +---------------------+
+ * | SECTION |
+ * | |
+ * +---------------------+
+ * | ... |
+ * | |
+ * +---------------------+
+ * | SECTION |
+ * | |
+ * +---------------------+
+ * | SIGNATURE |
+ * | (OPTIONAL) |
+ * +---------------------+
+ */
+
+enum XCLBIN_MODE {
+ XCLBIN_FLAT,
+ XCLBIN_PR,
+ XCLBIN_TANDEM_STAGE2,
+ XCLBIN_TANDEM_STAGE2_WITH_PR,
+ XCLBIN_HW_EMU,
+ XCLBIN_SW_EMU,
+ XCLBIN_MODE_MAX
+};
+
+enum axlf_section_kind {
+ BITSTREAM = 0,
+ CLEARING_BITSTREAM,
+ EMBEDDED_METADATA,
+ FIRMWARE,
+ DEBUG_DATA,
+ SCHED_FIRMWARE,
+ MEM_TOPOLOGY,
+ CONNECTIVITY,
+ IP_LAYOUT,
+ DEBUG_IP_LAYOUT,
+ DESIGN_CHECK_POINT,
+ CLOCK_FREQ_TOPOLOGY,
+ MCS,
+ BMC,
+ BUILD_METADATA,
+ KEYVALUE_METADATA,
+ USER_METADATA,
+ DNA_CERTIFICATE,
+ PDI,
+ BITSTREAM_PARTIAL_PDI,
+ PARTITION_METADATA,
+ EMULATION_DATA,
+ SYSTEM_METADATA,
+ SOFT_KERNEL,
+ ASK_FLASH,
+ AIE_METADATA,
+ ASK_GROUP_TOPOLOGY,
+ ASK_GROUP_CONNECTIVITY
+};
+
+enum MEM_TYPE {
+ MEM_DDR3,
+ MEM_DDR4,
+ MEM_DRAM,
+ MEM_STREAMING,
+ MEM_PREALLOCATED_GLOB,
+ MEM_ARE,
+ MEM_HBM,
+ MEM_BRAM,
+ MEM_URAM,
+ MEM_STREAMING_CONNECTION
+};
+
+enum IP_TYPE {
+ IP_MB = 0,
+ IP_KERNEL,
+ IP_DNASC,
+ IP_DDR4_CONTROLLER,
+ IP_MEM_DDR4,
+ IP_MEM_HBM
+};
+
+struct axlf_section_header {
+ uint32_t m_sectionKind; /* Section type */
+ char m_sectionName[16]; /* Examples: "stage2", "clear1", */
+ /* "clear2", "ocl1", "ocl2, */
+ /* "ublaze", "sched" */
+ uint64_t m_sectionOffset; /* File offset of section data */
+ uint64_t m_sectionSize; /* Size of section data */
+};
+
+struct axlf_header {
+ uint64_t m_length; /* Total size of the xclbin file */
+ uint64_t m_timeStamp; /* Number of seconds since epoch */
+ /* when xclbin was created */
+ uint64_t m_featureRomTimeStamp; /* TimeSinceEpoch of the featureRom */
+ uint16_t m_versionPatch; /* Patch Version */
+ uint8_t m_versionMajor; /* Major Version - Version: 2.1.0*/
+ uint8_t m_versionMinor; /* Minor Version */
+ uint32_t m_mode; /* XCLBIN_MODE */
+ union {
+ struct {
+ uint64_t m_platformId; /* 64 bit platform ID: */
+ /* vendor-device-subvendor-subdev */
+ uint64_t m_featureId; /* 64 bit feature id */
+ } rom;
+ unsigned char rom_uuid[16]; /* feature ROM UUID for which */
+ /* this xclbin was generated */
+ };
+ unsigned char m_platformVBNV[64]; /* e.g. */
+ /* xilinx:xil-accel-rd-ku115:4ddr-xpr:3.4: null terminated */
+ union {
+ char m_next_axlf[16]; /* Name of next xclbin file */
+ /* in the daisy chain */
+ uuid_t uuid; /* uuid of this xclbin*/
+ };
+ char m_debug_bin[16]; /* Name of binary with debug */
+ /* information */
+ uint32_t m_numSections; /* Number of section headers */
+};
+
+struct axlf {
+ char m_magic[8]; /* Should be "xclbin2\0" */
+ int32_t m_signature_length; /* Length of the signature. */
+ /* -1 indicates no signature */
+ unsigned char reserved[28]; /* Note: Initialized to 0xFFs */
+
+ unsigned char m_keyBlock[256]; /* Signature for validation */
+ /* of binary */
+ uint64_t m_uniqueId; /* axlf's uniqueId, use it to */
+ /* skip redownload etc */
+ struct axlf_header m_header; /* Inline header */
+ struct axlf_section_header m_sections[1]; /* One or more section */
+ /* headers follow */
+};
+
+/* bitstream information */
+struct xlnx_bitstream {
+ uint8_t m_freq[8];
+ char bits[1];
+};
+
+/**** MEMORY TOPOLOGY SECTION ****/
+struct mem_data {
+ uint8_t m_type; /* enum corresponding to mem_type. */
+ uint8_t m_used; /* if 0 this bank is not present */
+ union {
+ uint64_t m_size; /* if mem_type DDR, then size in KB; */
+ uint64_t route_id; /* if streaming then "route_id" */
+ };
+ union {
+ uint64_t m_base_address;/* if DDR then the base address; */
+ uint64_t flow_id; /* if streaming then "flow id" */
+ };
+ unsigned char m_tag[16]; /* DDR: BANK0,1,2,3, has to be null */
+ /* terminated; if streaming then stream0, 1 etc */
+};
+
+struct mem_topology {
+ int32_t m_count; /* Number of mem_data */
+ struct mem_data m_mem_data[1]; /* Should be sorted on mem_type */
+};
+
+/**** CONNECTIVITY SECTION ****/
+/* Connectivity of each argument of Kernel. It will be in terms of argument
+ * index associated. For associating kernel instances with arguments and
+ * banks, start at the connectivity section. Using the m_ip_layout_index
+ * access the ip_data.m_name. Now we can associate this kernel instance
+ * with its original kernel name and get the connectivity as well. This
+ * enables us to form related groups of kernel instances.
+ */
+
+struct connection {
+ int32_t arg_index; /* From 0 to n, may not be contiguous as scalars */
+ /* skipped */
+ int32_t m_ip_layout_index; /* index into the ip_layout section. */
+ /* ip_layout.m_ip_data[index].m_type == IP_KERNEL */
+ int32_t mem_data_index; /* index of the m_mem_data . Flag error is */
+ /* m_used false. */
+};
+
+struct connectivity {
+ int32_t m_count;
+ struct connection m_connection[1];
+};
+
+/**** IP_LAYOUT SECTION ****/
+
+/* IP Kernel */
+#define IP_INT_ENABLE_MASK 0x0001
+#define IP_INTERRUPT_ID_MASK 0x00FE
+#define IP_INTERRUPT_ID_SHIFT 0x1
+
+enum IP_CONTROL {
+ AP_CTRL_HS = 0,
+ AP_CTRL_CHAIN = 1,
+ AP_CTRL_NONE = 2,
+ AP_CTRL_ME = 3,
+ ACCEL_ADAPTER = 4
+};
+
+#define IP_CONTROL_MASK 0xFF00
+#define IP_CONTROL_SHIFT 0x8
+
+/* IPs on AXI lite - their types, names, and base addresses.*/
+struct ip_data {
+ uint32_t m_type; /* map to IP_TYPE enum */
+ union {
+ uint32_t properties; /* Default: 32-bits to indicate ip */
+ /* specific property. */
+ /* m_type: IP_KERNEL
+ * m_int_enable : Bit - 0x0000_0001;
+ * m_interrupt_id : Bits - 0x0000_00FE;
+ * m_ip_control : Bits = 0x0000_FF00;
+ */
+ struct { /* m_type: IP_MEM_* */
+ uint16_t m_index;
+ uint8_t m_pc_index;
+ uint8_t unused;
+ } indices;
+ };
+ uint64_t m_base_address;
+ uint8_t m_name[64]; /* eg Kernel name corresponding to KERNEL */
+ /* instance, can embed CU name in future. */
+};
+
+struct ip_layout {
+ int32_t m_count;
+ struct ip_data m_ip_data[1]; /* All the ip_data needs to be sorted */
+ /* by m_base_address. */
+};
+
+/*** Debug IP section layout ****/
+enum DEBUG_IP_TYPE {
+ UNDEFINED = 0,
+ LAPC,
+ ILA,
+ AXI_MM_MONITOR,
+ AXI_TRACE_FUNNEL,
+ AXI_MONITOR_FIFO_LITE,
+ AXI_MONITOR_FIFO_FULL,
+ ACCEL_MONITOR,
+ AXI_STREAM_MONITOR,
+ AXI_STREAM_PROTOCOL_CHECKER,
+ TRACE_S2MM,
+ AXI_DMA,
+ TRACE_S2MM_FULL
+};
+
+struct debug_ip_data {
+ uint8_t m_type; /* type of enum DEBUG_IP_TYPE */
+ uint8_t m_index_lowbyte;
+ uint8_t m_properties;
+ uint8_t m_major;
+ uint8_t m_minor;
+ uint8_t m_index_highbyte;
+ uint8_t m_reserved[2];
+ uint64_t m_base_address;
+ char m_name[128];
+};
+
+struct debug_ip_layout {
+ uint16_t m_count;
+ struct debug_ip_data m_debug_ip_data[1];
+};
+
+/* Supported clock frequency types */
+enum CLOCK_TYPE {
+ CT_UNUSED = 0, /* Initialized value */
+ CT_DATA = 1, /* Data clock */
+ CT_KERNEL = 2, /* Kernel clock */
+ CT_SYSTEM = 3 /* System Clock */
+};
+
+/* Clock Frequency Entry */
+struct clock_freq {
+ uint16_t m_freq_Mhz; /* Frequency in MHz */
+ uint8_t m_type; /* Clock type (enum CLOCK_TYPE) */
+ uint8_t m_unused[5]; /* Not used - padding */
+ char m_name[128]; /* Clock Name */
+};
+
+/* Clock frequency section */
+struct clock_freq_topology {
+ int16_t m_count; /* Number of entries */
+ struct clock_freq m_clock_freq[1]; /* Clock array */
+};
+
+/* Supported MCS file types */
+enum MCS_TYPE {
+ MCS_UNKNOWN = 0, /* Initialized value */
+ MCS_PRIMARY = 1, /* The primary mcs file data */
+ MCS_SECONDARY = 2, /* The secondary mcs file data */
+};
+
+/* One chunk of MCS data */
+struct mcs_chunk {
+ uint8_t m_type; /* MCS data type */
+ uint8_t m_unused[7]; /* padding */
+ uint64_t m_offset; /* data offset from the start of */
+ /* the section */
+ uint64_t m_size; /* data size */
+};
+
+/* MCS data section */
+struct mcs {
+ int8_t m_count; /* Number of chunks */
+ int8_t m_unused[7]; /* padding */
+ struct mcs_chunk m_chunk[1]; /* MCS chunks followed by data */
+};
+
+/* bmc data section */
+struct bmc {
+ uint64_t m_offset; /* data offset from the start of */
+ /* the section */
+ uint64_t m_size; /* data size (bytes) */
+ char m_image_name[64]; /* Name of the image */
+ /* (e.g., MSP432P401R) */
+ char m_device_name[64]; /* Device ID (e.g., VCU1525) */
+ char m_version[64];
+ char m_md5value[33]; /* MD5 Expected Value */
+ /* (e.g., 56027182079c0bd621761b7dab5a27ca)*/
+ char m_padding[7]; /* Padding */
+};
+
+/* soft kernel data section, used by classic driver */
+struct soft_kernel {
+ /** Prefix Syntax:
+ * mpo - member, pointer, offset
+ * This variable represents a zero terminated string
+ * that is offseted from the beginning of the section.
+ * The pointer to access the string is initialized as follows:
+ * char * pCharString = (address_of_section) + (mpo value)
+ */
+ uint32_t mpo_name; /* Name of the soft kernel */
+ uint32_t m_image_offset; /* Image offset */
+ uint32_t m_image_size; /* Image size */
+ uint32_t mpo_version; /* Version */
+ uint32_t mpo_md5_value; /* MD5 checksum */
+ uint32_t mpo_symbol_name; /* Symbol name */
+ uint32_t m_num_instances; /* Number of instances */
+ uint8_t padding[36]; /* Reserved for future use */
+ uint8_t reservedExt[16]; /* Reserved for future extended data */
+};
+
+enum CHECKSUM_TYPE {
+ CST_UNKNOWN = 0,
+ CST_SDBM = 1,
+ CST_LAST
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file " Lizhi Hou
@ 2021-02-21 17:12 ` Tom Rix
2021-02-21 18:33 ` Moritz Fischer
2021-02-26 21:23 ` Lizhi Hou
0 siblings, 2 replies; 87+ messages in thread
From: Tom Rix @ 2021-02-21 17:12 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Alveo FPGA firmware and partial reconfigure file are in xclbin format.
This code enumerates and extracts
> Add
> code to enumerate and extract sections from xclbin files. xclbin.h is cross
> platform and used across all platforms and OS
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xclbin-helper.h | 52 +++
> drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++++++++++++++
> include/uapi/linux/xrt/xclbin.h | 408 +++++++++++++++++++++++
> 3 files changed, 854 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
> create mode 100644 drivers/fpga/xrt/lib/xclbin.c
> create mode 100644 include/uapi/linux/xrt/xclbin.h
>
> diff --git a/drivers/fpga/xrt/include/xclbin-helper.h b/drivers/fpga/xrt/include/xclbin-helper.h
> new file mode 100644
> index 000000000000..68218efc9d0b
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xclbin-helper.h
> @@ -0,0 +1,52 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * David Zhang <davidzha@xilinx.com>
> + * Sonal Santan <sonal.santan@xilinx.com>
> + */
> +
> +#ifndef _XRT_XCLBIN_H
> +#define _XRT_XCLBIN_H
The header guard should match the filename.
> +
> +#include <linux/types.h>
> +#include <linux/device.h>
> +#include <linux/xrt/xclbin.h>
> +
> +#define ICAP_XCLBIN_V2 "xclbin2"
> +#define DMA_HWICAP_BITFILE_BUFFER_SIZE 1024
> +#define MAX_XCLBIN_SIZE (1024 * 1024 * 1024) /* Assuming xclbin <= 1G, always */
#defines should have a prefix, maybe XRT_ or XCLBIN_
> +
> +enum axlf_section_kind;
> +struct axlf;
> +
> +/**
> + * Bitstream header information as defined by Xilinx tools.
> + * Please note that this struct definition is not owned by the driver.
> + */
> +struct hw_icap_bit_header {
File headers usually have fixed length fields like uint32_t
Is this a structure the real header is converted into ?
> + unsigned int header_length; /* Length of header in 32 bit words */
> + unsigned int bitstream_length; /* Length of bitstream to read in bytes*/
> + unsigned char *design_name; /* Design name get from bitstream */
> + unsigned char *part_name; /* Part name read from bitstream */
> + unsigned char *date; /* Date read from bitstream header */
> + unsigned char *time; /* Bitstream creation time */
> + unsigned int magic_length; /* Length of the magic numbers */
> + unsigned char *version; /* Version string */
> +};
> +
> +const char *xrt_xclbin_kind_to_string(enum axlf_section_kind kind);
Only add decl's that are using in multiple files.
This is only defined in xclbin.c, why does it need to be in the header ?
> +int xrt_xclbin_get_section(const struct axlf *xclbin,
> + enum axlf_section_kind kind, void **data,
> + uint64_t *len);
> +int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb);
> +int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
> + unsigned int size,
> + struct hw_icap_bit_header *header);
> +void xrt_xclbin_free_header(struct hw_icap_bit_header *header);
> +const char *xrt_clock_type2epname(enum CLOCK_TYPE type);
CLOCK_TYPE needs a prefix, something like XCLBIN_CLOCK_TYPE
> +
> +#endif /* _XRT_XCLBIN_H */
> diff --git a/drivers/fpga/xrt/lib/xclbin.c b/drivers/fpga/xrt/lib/xclbin.c
> new file mode 100644
> index 000000000000..47dc6ca25c1b
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xclbin.c
> @@ -0,0 +1,394 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA Driver XCLBIN parser
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors: David Zhang <davidzha@xilinx.com>
> + */
> +
> +#include <asm/errno.h>
> +#include <linux/vmalloc.h>
> +#include <linux/device.h>
> +#include "xclbin-helper.h"
> +#include "metadata.h"
> +
What is XHI ? Maybe expand this, at the lease should comment
> +/* Used for parsing bitstream header */
> +#define XHI_EVEN_MAGIC_BYTE 0x0f
> +#define XHI_ODD_MAGIC_BYTE 0xf0
> +
> +/* Extra mode for IDLE */
> +#define XHI_OP_IDLE -1
> +#define XHI_BIT_HEADER_FAILURE -1
> +
> +/* The imaginary module length register */
> +#define XHI_MLR 15
> +
> +static inline unsigned char xhi_data_and_inc(const unsigned char *d, int *i, int sz)
could move to the *.h
> +{_
> + unsigned char data;
> +
> + if (*i >= sz)
> + return -1;
The return value of this funtion is not always checked, at the least add a dev_err here
> +
> + data = d[*i];
> + (*i)++;
> +
> + return data;
> +}
> +
> +static const struct axlf_section_header *
> +xrt_xclbin_get_section_hdr(const struct axlf *xclbin,
> + enum axlf_section_kind kind)
> +{
> + int i = 0;
> +
> + for (i = 0; i < xclbin->m_header.m_numSections; i++) {
> + if (xclbin->m_sections[i].m_sectionKind == kind)
> + return &xclbin->m_sections[i];
> + }
> +
> + return NULL;
> +}
> +
> +static int
> +xrt_xclbin_check_section_hdr(const struct axlf_section_header *header,
> + u64 xclbin_len)
> +{
> + int ret;
> +
> + ret = (header->m_sectionOffset + header->m_sectionSize) > xclbin_len ? -EINVAL : 0;
Tristate is harder to read, consider replacing with if()
int ret = 0
if ()
ret =
> +
> + return ret;
> +}
> +
> +static int xrt_xclbin_section_info(const struct axlf *xclbin,
> + enum axlf_section_kind kind,
> + u64 *offset, u64 *size)
> +{
> + const struct axlf_section_header *mem_header = NULL;
> + u64 xclbin_len;
> + int err = 0;
> +
> + mem_header = xrt_xclbin_get_section_hdr(xclbin, kind);
> + if (!mem_header)
> + return -EINVAL;
> +
> + xclbin_len = xclbin->m_header.m_length;
> + if (xclbin_len > MAX_XCLBIN_SIZE)
> + return -EINVAL;
This check can be added to the function call..
or the sanity checking added to the earier call to *get_section_hdr
There a number of small functions that can be combined.
> +
> + err = xrt_xclbin_check_section_hdr(mem_header, xclbin_len);
> + if (err)
> + return err;
> +
> + *offset = mem_header->m_sectionOffset;
> + *size = mem_header->m_sectionSize;
> +
> + return 0;
> +}
> +
> +/* caller should free the allocated memory for **data */
must free
This comment also needs to be with the *.h decl
> +int xrt_xclbin_get_section(const struct axlf *buf,
> + enum axlf_section_kind kind,
> + void **data, u64 *len)
> +{
> + const struct axlf *xclbin = (const struct axlf *)buf;
> + void *section = NULL;
> + int err = 0;
> + u64 offset = 0;
> + u64 size = 0;
> +
> + err = xrt_xclbin_section_info(xclbin, kind, &offset, &size);
> + if (err)
> + return err;
> +
> + section = vmalloc(size);
> + if (!section)
> + return -ENOMEM;
> +
> + memcpy(section, ((const char *)xclbin) + offset, size);
> +
> + *data = section;
a general comment
for exported function checking the validity of the inputs in more important.
here you assume **data is valid, really you should check.
> + if (len)
> + *len = size;
len setting being optional, needs to be in the *.h comment
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_xclbin_get_section);
> +
Instead of allocating new memory and making copies of bits of *data
why not have the points reference data ?
The size operations look like translating big endian data to little endian.
This will break on a big endian host.
> +/* parse bitstream header */
> +int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
> + unsigned int size,
> + struct hw_icap_bit_header *header)
> +{
> + unsigned int index;
> + unsigned int len;
> + unsigned int tmp;
> + unsigned int i;
> +
> + memset(header, 0, sizeof(*header));
> + /* Start Index at start of bitstream */
> + index = 0;
> +
> + /* Initialize HeaderLength. If header returned early inidicates
> + * failure.
This side effect should be documented in the *.h comment.
Also the multi line comment is a bit weird, not sure if it is ok
> + */
> + header->header_length = XHI_BIT_HEADER_FAILURE;
> +
> + /* Get "Magic" length */
> + header->magic_length = xhi_data_and_inc(data, &index, size);
> + header->magic_length = (header->magic_length << 8) | xhi_data_and_inc(data, &index, size);
> +
> + /* Read in "magic" */
> + for (i = 0; i < header->magic_length - 1; i++) {
> + tmp = xhi_data_and_inc(data, &index, size);
> + if (i % 2 == 0 && tmp != XHI_EVEN_MAGIC_BYTE)
> + return -1; /* INVALID_FILE_HEADER_ERROR */
> +
> + if (i % 2 == 1 && tmp != XHI_ODD_MAGIC_BYTE)
> + return -1; /* INVALID_FILE_HEADER_ERROR */
> + }
> +
> + /* Read null end of magic data. */
> + tmp = xhi_data_and_inc(data, &index, size);
> +
> + /* Read 0x01 (short) */
> + tmp = xhi_data_and_inc(data, &index, size);
> + tmp = (tmp << 8) | xhi_data_and_inc(data, &index, size);
> +
> + /* Check the "0x01" half word */
> + if (tmp != 0x01)
> + return -1; /* INVALID_FILE_HEADER_ERROR */
> +
> + /* Read 'a' */
> + tmp = xhi_data_and_inc(data, &index, size);
> + if (tmp != 'a')
> + return -1; /* INVALID_FILE_HEADER_ERROR */
> +
> + /* Get Design Name length */
> + len = xhi_data_and_inc(data, &index, size);
> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
> +
> + /* allocate space for design name and final null character. */
> + header->design_name = vmalloc(len);
> + if (!header->design_name)
> + return -ENOMEM;
> +
> + /* Read in Design Name */
> + for (i = 0; i < len; i++)
> + header->design_name[i] = xhi_data_and_inc(data, &index, size);
> +
> + if (header->design_name[len - 1] != '\0')
> + return -1;
> +
> + header->version = strstr(header->design_name, "Version=") + strlen("Version=");
> +
> + /* Read 'b' */
> + tmp = xhi_data_and_inc(data, &index, size);
> + if (tmp != 'b')
> + return -1; /* INVALID_FILE_HEADER_ERROR */
> +
> + /* Get Part Name length */
> + len = xhi_data_and_inc(data, &index, size);
> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
> +
> + /* allocate space for part name and final null character. */
> + header->part_name = vmalloc(len);
> + if (!header->part_name)
> + return -ENOMEM;
> +
> + /* Read in part name */
> + for (i = 0; i < len; i++)
> + header->part_name[i] = xhi_data_and_inc(data, &index, size);
> +
> + if (header->part_name[len - 1] != '\0')
> + return -1;
> +
> + /* Read 'c' */
> + tmp = xhi_data_and_inc(data, &index, size);
> + if (tmp != 'c')
> + return -1; /* INVALID_FILE_HEADER_ERROR */
> +
> + /* Get date length */
> + len = xhi_data_and_inc(data, &index, size);
> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
> +
> + /* allocate space for date and final null character. */
> + header->date = vmalloc(len);
> + if (!header->date)
> + return -ENOMEM;
> +
> + /* Read in date name */
> + for (i = 0; i < len; i++)
> + header->date[i] = xhi_data_and_inc(data, &index, size);
> +
> + if (header->date[len - 1] != '\0')
> + return -1;
generally -EINVAL is more meaningful than -1
> +
> + /* Read 'd' */
> + tmp = xhi_data_and_inc(data, &index, size);
> + if (tmp != 'd')
> + return -1; /* INVALID_FILE_HEADER_ERROR */
> +
> + /* Get time length */
> + len = xhi_data_and_inc(data, &index, size);
> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
> +
> + /* allocate space for time and final null character. */
> + header->time = vmalloc(len);
> + if (!header->time)
> + return -ENOMEM;
> +
> + /* Read in time name */
> + for (i = 0; i < len; i++)
> + header->time[i] = xhi_data_and_inc(data, &index, size);
> +
> + if (header->time[len - 1] != '\0')
> + return -1;
> +
> + /* Read 'e' */
> + tmp = xhi_data_and_inc(data, &index, size);
> + if (tmp != 'e')
> + return -1; /* INVALID_FILE_HEADER_ERROR */
> +
> + /* Get byte length of bitstream */
> + header->bitstream_length = xhi_data_and_inc(data, &index, size);
> + header->bitstream_length = (header->bitstream_length << 8) |
> + xhi_data_and_inc(data, &index, size);
> + header->bitstream_length = (header->bitstream_length << 8) |
> + xhi_data_and_inc(data, &index, size);
> + header->bitstream_length = (header->bitstream_length << 8) |
> + xhi_data_and_inc(data, &index, size);
generally a problem
This is confusing, collect the bytes in a temp[] and construct the header->bitstream_length in on statement.
This is a case where xhi_data_and_inc return is not checked and if it failed could blow up later.
> +
> + header->header_length = index;
index is not a good variable name if it going to be stored as a length.
consider changing it to something like current_length.
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xrt_xclbin_parse_bitstream_header);
> +
> +void xrt_xclbin_free_header(struct hw_icap_bit_header *header)
> +{
> + vfree(header->design_name);
> + vfree(header->part_name);
> + vfree(header->date);
> + vfree(header->time);
missing header->version
> +}
> +EXPORT_SYMBOL_GPL(xrt_xclbin_free_header);
> +
> +struct xrt_clock_desc {
> + char *clock_ep_name;
> + u32 clock_xclbin_type;
> + char *clkfreq_ep_name;
> +} clock_desc[] = {
> + {
> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL1,
> + .clock_xclbin_type = CT_DATA,
> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K1,
> + },
> + {
> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL2,
> + .clock_xclbin_type = CT_KERNEL,
> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K2,
> + },
> + {
> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL3,
> + .clock_xclbin_type = CT_SYSTEM,
> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_HBM,
> + },
> +};
> +
> +const char *xrt_clock_type2epname(enum CLOCK_TYPE type)
> +{
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
> + if (clock_desc[i].clock_xclbin_type == type)
> + return clock_desc[i].clock_ep_name;
> + }
> + return NULL;
> +}
> +EXPORT_SYMBOL_GPL(xrt_clock_type2epname);
What is clock stuff doing in xclbin ?
I think clock needs its own file
> +
> +static const char *clock_type2clkfreq_name(u32 type)
> +{
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
> + if (clock_desc[i].clock_xclbin_type == type)
> + return clock_desc[i].clkfreq_ep_name;
> + }
> + return NULL;
> +}
> +
> +static int xrt_xclbin_add_clock_metadata(struct device *dev,
> + const struct axlf *xclbin,
> + char *dtb)
> +{
> + int i;
> + u16 freq;
> + struct clock_freq_topology *clock_topo;
> + int rc = xrt_xclbin_get_section(xclbin, CLOCK_FREQ_TOPOLOGY,
> + (void **)&clock_topo, NULL);
> +
> + if (rc)
> + return 0;
failing is ok ?
> +
> + for (i = 0; i < clock_topo->m_count; i++) {
> + u8 type = clock_topo->m_clock_freq[i].m_type;
> + const char *ep_name = xrt_clock_type2epname(type);
> + const char *counter_name = clock_type2clkfreq_name(type);
> +
> + if (!ep_name || !counter_name)
> + continue;
> +
> + freq = cpu_to_be16(clock_topo->m_clock_freq[i].m_freq_Mhz);
> + rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_FREQ,
> + &freq, sizeof(freq));
> + if (rc)
> + break;
> +
> + rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_CNT,
> + counter_name, strlen(counter_name) + 1);
> + if (rc)
> + break;
Failing in a loop, why isn't there some cleanup of the *set_prop() calls ?
> + }
> +
> + vfree(clock_topo);
> +
> + return rc;
> +}
> +
> +int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb)
> +{
> + char *md = NULL, *newmd = NULL;
> + u64 len;
*dtb = NULL;
> + int rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA,
> + (void **)&md, &len);
> +
> + if (rc)
> + goto done;
> +
> + /* Sanity check the dtb section. */
> + if (xrt_md_size(dev, md) > len) {
> + rc = -EINVAL;
> + goto done;
> + }
> +
> + newmd = xrt_md_dup(dev, md);
> + if (!newmd) {
> + rc = -EFAULT;
> + goto done;
> + }
> + /* Convert various needed xclbin sections into dtb. */
> + rc = xrt_xclbin_add_clock_metadata(dev, xclbin, newmd);
newmd is only valid here, but the above error handling jump here. change this to
if (!rc)
*dtb = newmd
else
vfree(newmd)
done:
vfree(md)
return rc;
> +
> +done:
> + if (rc == 0)
> + *dtb = newmd;
> + else
> + vfree(newmd);
> + vfree(md);
> + return rc;
> +}
> +EXPORT_SYMBOL_GPL(xrt_xclbin_get_metadata);
> diff --git a/include/uapi/linux/xrt/xclbin.h b/include/uapi/linux/xrt/xclbin.h
> new file mode 100644
> index 000000000000..53f140123ef1
> --- /dev/null
> +++ b/include/uapi/linux/xrt/xclbin.h
> @@ -0,0 +1,408 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Xilinx FPGA compiled binary container format
> + *
> + * Copyright (C) 2015-2021, Xilinx Inc
> + */
> +
> +#ifndef _XCLBIN_H_
> +#define _XCLBIN_H_
> +
> +#ifdef _WIN32
WIN32 ?
Only 1 other header has this ifdef
> + #include <cstdint>
> + #include <algorithm>
c++ is being assumed for windows
> + #include "windows/uuid.h"
thank you for not including windows.h ;)
> +#else
> + #if defined(__KERNEL__)
> + #include <linux/types.h>
> + #include <linux/uuid.h>
> + #include <linux/version.h>
> + #elif defined(__cplusplus)
> + #include <cstdlib>
> + #include <cstdint>
> + #include <algorithm>
> + #include <uuid/uuid.h>
> + #else
> + #include <stdlib.h>
> + #include <stdint.h>
> + #include <uuid/uuid.h>
> + #endif
> +#endif
> +
Review these includes, some could be convenience includes.
ex/ linux/version.h with no obvious use of version macros.
struct axlf_header {
+ uint64_t m_length; /* Total size of the xclbin file */
.. snip ..
+ union {
+ char m_next_axlf[16]; /* Name of next xclbin file */
+ /* in the daisy chain */
+ uuid_t uuid; /* uuid of this xclbin*/
+ };
As mentioned in an earlier patch, if uuid_t is larger than 16 bytes, axlf_header breaks.
while it is convenient to have this type here, it would be better this access was handled in another way.
Maybe a host specific function.
I also do not see a pragma pack, usually this is set of 1 so the compiler does not shuffle elements, increase size etc.
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +/**
> + * DOC: Container format for Xilinx FPGA images
> + * The container stores bitstreams, metadata and firmware images.
> + * xclbin/xsabin is ELF-like binary container format. It is structured
is an ELF-like file format. It is a structured
> + * series of sections. There is a file header followed by several section
> + * headers which is followed by sections. A section header points to an
> + * actual section. There is an optional signature at the end. The
> + * following figure illustrates a typical xclbin:
> + *
> + * +---------------------+
> + * | |
> + * | HEADER |
> + * +---------------------+
> + * | SECTION HEADER |
> + * | |
> + * +---------------------+
> + * | ... |
> + * | |
> + * +---------------------+
> + * | SECTION HEADER |
> + * | |
> + * +---------------------+
> + * | SECTION |
> + * | |
> + * +---------------------+
> + * | ... |
> + * | |
> + * +---------------------+
> + * | SECTION |
> + * | |
> + * +---------------------+
> + * | SIGNATURE |
> + * | (OPTIONAL) |
> + * +---------------------+
This ascii art is a mixture of tabs and spaces, for someone with tab = 2 spaces, this will look messed up.
convert the tabs to spaces
> + */
> +
> +enum XCLBIN_MODE {
> + XCLBIN_FLAT,
generally
all enums used in a file format should be initialized.
This likely should be
XCLBIN_FLAT = 0,
> + XCLBIN_PR,
> + XCLBIN_TANDEM_STAGE2,
> + XCLBIN_TANDEM_STAGE2_WITH_PR,
> + XCLBIN_HW_EMU,
> + XCLBIN_SW_EMU,
> + XCLBIN_MODE_MAX
> +};
> +
> +enum axlf_section_kind {
> + BITSTREAM = 0,
> + CLEARING_BITSTREAM,
> + EMBEDDED_METADATA,
> + FIRMWARE,
> + DEBUG_DATA,
> + SCHED_FIRMWARE,
> + MEM_TOPOLOGY,
> + CONNECTIVITY,
> + IP_LAYOUT,
> + DEBUG_IP_LAYOUT,
> + DESIGN_CHECK_POINT,
> + CLOCK_FREQ_TOPOLOGY,
> + MCS,
> + BMC,
> + BUILD_METADATA,
> + KEYVALUE_METADATA,
> + USER_METADATA,
> + DNA_CERTIFICATE,
> + PDI,
> + BITSTREAM_PARTIAL_PDI,
> + PARTITION_METADATA,
> + EMULATION_DATA,
> + SYSTEM_METADATA,
> + SOFT_KERNEL,
> + ASK_FLASH,
> + AIE_METADATA,
> + ASK_GROUP_TOPOLOGY,
> + ASK_GROUP_CONNECTIVITY
> +};
> +
> +enum MEM_TYPE {
> + MEM_DDR3,
> + MEM_DDR4,
> + MEM_DRAM,
> + MEM_STREAMING,
> + MEM_PREALLOCATED_GLOB,
> + MEM_ARE,
> + MEM_HBM,
> + MEM_BRAM,
> + MEM_URAM,
> + MEM_STREAMING_CONNECTION
> +};
> +
> +enum IP_TYPE {
> + IP_MB = 0,
> + IP_KERNEL,
> + IP_DNASC,
> + IP_DDR4_CONTROLLER,
> + IP_MEM_DDR4,
> + IP_MEM_HBM
> +};
> +
> +struct axlf_section_header {
> + uint32_t m_sectionKind; /* Section type */
> + char m_sectionName[16]; /* Examples: "stage2", "clear1", */
> + /* "clear2", "ocl1", "ocl2, */
> + /* "ublaze", "sched" */
> + uint64_t m_sectionOffset; /* File offset of section data */
> + uint64_t m_sectionSize; /* Size of section data */
> +};
> +
> +struct axlf_header {
> + uint64_t m_length; /* Total size of the xclbin file */
> + uint64_t m_timeStamp; /* Number of seconds since epoch */
> + /* when xclbin was created */
> + uint64_t m_featureRomTimeStamp; /* TimeSinceEpoch of the featureRom */
> + uint16_t m_versionPatch; /* Patch Version */
> + uint8_t m_versionMajor; /* Major Version - Version: 2.1.0*/
i did not see the version checked earlier, which one is expected ?
> + uint8_t m_versionMinor; /* Minor Version */
> + uint32_t m_mode; /* XCLBIN_MODE */
> + union {
> + struct {
> + uint64_t m_platformId; /* 64 bit platform ID: */
> + /* vendor-device-subvendor-subdev */
> + uint64_t m_featureId; /* 64 bit feature id */
> + } rom;
> + unsigned char rom_uuid[16]; /* feature ROM UUID for which */
> + /* this xclbin was generated */
> + };
> + unsigned char m_platformVBNV[64]; /* e.g. */
what is VBNV?
> + /* xilinx:xil-accel-rd-ku115:4ddr-xpr:3.4: null terminated */
> + union {
> + char m_next_axlf[16]; /* Name of next xclbin file */
> + /* in the daisy chain */
> + uuid_t uuid; /* uuid of this xclbin*/
> + };
> + char m_debug_bin[16]; /* Name of binary with debug */
> + /* information */
> + uint32_t m_numSections; /* Number of section headers */
> +};
> +
> +struct axlf {
> + char m_magic[8]; /* Should be "xclbin2\0" */
> + int32_t m_signature_length; /* Length of the signature. */
> + /* -1 indicates no signature */
> + unsigned char reserved[28]; /* Note: Initialized to 0xFFs */
> +
> + unsigned char m_keyBlock[256]; /* Signature for validation */
> + /* of binary */
> + uint64_t m_uniqueId; /* axlf's uniqueId, use it to */
> + /* skip redownload etc */
> + struct axlf_header m_header; /* Inline header */
> + struct axlf_section_header m_sections[1]; /* One or more section */
> + /* headers follow */
> +};
> +
> +/* bitstream information */
> +struct xlnx_bitstream {
> + uint8_t m_freq[8];
> + char bits[1];
> +};
> +
> +/**** MEMORY TOPOLOGY SECTION ****/
> +struct mem_data {
> + uint8_t m_type; /* enum corresponding to mem_type. */
> + uint8_t m_used; /* if 0 this bank is not present */
> + union {
> + uint64_t m_size; /* if mem_type DDR, then size in KB; */
> + uint64_t route_id; /* if streaming then "route_id" */
> + };
> + union {
> + uint64_t m_base_address;/* if DDR then the base address; */
> + uint64_t flow_id; /* if streaming then "flow id" */
> + };
> + unsigned char m_tag[16]; /* DDR: BANK0,1,2,3, has to be null */
> + /* terminated; if streaming then stream0, 1 etc */
> +};
> +
> +struct mem_topology {
> + int32_t m_count; /* Number of mem_data */
> + struct mem_data m_mem_data[1]; /* Should be sorted on mem_type */
> +};
> +
> +/**** CONNECTIVITY SECTION ****/
> +/* Connectivity of each argument of Kernel. It will be in terms of argument
This section does not make sense.
Likely you mean some algorithm kernel, rather than the linux kernel.
> + * index associated. For associating kernel instances with arguments and
> + * banks, start at the connectivity section. Using the m_ip_layout_index
> + * access the ip_data.m_name. Now we can associate this kernel instance
> + * with its original kernel name and get the connectivity as well. This
> + * enables us to form related groups of kernel instances.
> + */
> +
> +struct connection {
> + int32_t arg_index; /* From 0 to n, may not be contiguous as scalars */
> + /* skipped */
> + int32_t m_ip_layout_index; /* index into the ip_layout section. */
> + /* ip_layout.m_ip_data[index].m_type == IP_KERNEL */
> + int32_t mem_data_index; /* index of the m_mem_data . Flag error is */
> + /* m_used false. */
> +};
> +
> +struct connectivity {
> + int32_t m_count;
> + struct connection m_connection[1];
> +};
> +
> +/**** IP_LAYOUT SECTION ****/
> +
> +/* IP Kernel */
> +#define IP_INT_ENABLE_MASK 0x0001
> +#define IP_INTERRUPT_ID_MASK 0x00FE
> +#define IP_INTERRUPT_ID_SHIFT 0x1
> +
> +enum IP_CONTROL {
> + AP_CTRL_HS = 0,
> + AP_CTRL_CHAIN = 1,
> + AP_CTRL_NONE = 2,
> + AP_CTRL_ME = 3,
> + ACCEL_ADAPTER = 4
assigning beyond the first is not necessary unless there are dups or gaps
> +};
> +
> +#define IP_CONTROL_MASK 0xFF00
> +#define IP_CONTROL_SHIFT 0x8
> +
> +/* IPs on AXI lite - their types, names, and base addresses.*/
> +struct ip_data {
> + uint32_t m_type; /* map to IP_TYPE enum */
> + union {
> + uint32_t properties; /* Default: 32-bits to indicate ip */
> + /* specific property. */
> + /* m_type: IP_KERNEL
> + * m_int_enable : Bit - 0x0000_0001;
> + * m_interrupt_id : Bits - 0x0000_00FE;
> + * m_ip_control : Bits = 0x0000_FF00;
> + */
> + struct { /* m_type: IP_MEM_* */
> + uint16_t m_index;
> + uint8_t m_pc_index;
> + uint8_t unused;
> + } indices;
> + };
> + uint64_t m_base_address;
> + uint8_t m_name[64]; /* eg Kernel name corresponding to KERNEL */
> + /* instance, can embed CU name in future. */
> +};
> +
> +struct ip_layout {
> + int32_t m_count;
> + struct ip_data m_ip_data[1]; /* All the ip_data needs to be sorted */
> + /* by m_base_address. */
general
doing the bla[1] for c++ ?
Tom
> +};
> +
> +/*** Debug IP section layout ****/
> +enum DEBUG_IP_TYPE {
> + UNDEFINED = 0,
> + LAPC,
> + ILA,
> + AXI_MM_MONITOR,
> + AXI_TRACE_FUNNEL,
> + AXI_MONITOR_FIFO_LITE,
> + AXI_MONITOR_FIFO_FULL,
> + ACCEL_MONITOR,
> + AXI_STREAM_MONITOR,
> + AXI_STREAM_PROTOCOL_CHECKER,
> + TRACE_S2MM,
> + AXI_DMA,
> + TRACE_S2MM_FULL
> +};
> +
> +struct debug_ip_data {
> + uint8_t m_type; /* type of enum DEBUG_IP_TYPE */
> + uint8_t m_index_lowbyte;
> + uint8_t m_properties;
> + uint8_t m_major;
> + uint8_t m_minor;
> + uint8_t m_index_highbyte;
> + uint8_t m_reserved[2];
> + uint64_t m_base_address;
> + char m_name[128];
> +};
> +
> +struct debug_ip_layout {
> + uint16_t m_count;
> + struct debug_ip_data m_debug_ip_data[1];
> +};
> +
> +/* Supported clock frequency types */
> +enum CLOCK_TYPE {
> + CT_UNUSED = 0, /* Initialized value */
> + CT_DATA = 1, /* Data clock */
> + CT_KERNEL = 2, /* Kernel clock */
> + CT_SYSTEM = 3 /* System Clock */
> +};
> +
> +/* Clock Frequency Entry */
> +struct clock_freq {
> + uint16_t m_freq_Mhz; /* Frequency in MHz */
> + uint8_t m_type; /* Clock type (enum CLOCK_TYPE) */
> + uint8_t m_unused[5]; /* Not used - padding */
> + char m_name[128]; /* Clock Name */
> +};
> +
> +/* Clock frequency section */
> +struct clock_freq_topology {
> + int16_t m_count; /* Number of entries */
> + struct clock_freq m_clock_freq[1]; /* Clock array */
> +};
> +
> +/* Supported MCS file types */
> +enum MCS_TYPE {
> + MCS_UNKNOWN = 0, /* Initialized value */
> + MCS_PRIMARY = 1, /* The primary mcs file data */
> + MCS_SECONDARY = 2, /* The secondary mcs file data */
> +};
> +
> +/* One chunk of MCS data */
> +struct mcs_chunk {
> + uint8_t m_type; /* MCS data type */
> + uint8_t m_unused[7]; /* padding */
> + uint64_t m_offset; /* data offset from the start of */
> + /* the section */
> + uint64_t m_size; /* data size */
> +};
> +
> +/* MCS data section */
> +struct mcs {
> + int8_t m_count; /* Number of chunks */
> + int8_t m_unused[7]; /* padding */
> + struct mcs_chunk m_chunk[1]; /* MCS chunks followed by data */
> +};
> +
> +/* bmc data section */
> +struct bmc {
> + uint64_t m_offset; /* data offset from the start of */
> + /* the section */
> + uint64_t m_size; /* data size (bytes) */
> + char m_image_name[64]; /* Name of the image */
> + /* (e.g., MSP432P401R) */
> + char m_device_name[64]; /* Device ID (e.g., VCU1525) */
> + char m_version[64];
> + char m_md5value[33]; /* MD5 Expected Value */
> + /* (e.g., 56027182079c0bd621761b7dab5a27ca)*/
> + char m_padding[7]; /* Padding */
> +};
> +
> +/* soft kernel data section, used by classic driver */
> +struct soft_kernel {
> + /** Prefix Syntax:
> + * mpo - member, pointer, offset
> + * This variable represents a zero terminated string
> + * that is offseted from the beginning of the section.
> + * The pointer to access the string is initialized as follows:
> + * char * pCharString = (address_of_section) + (mpo value)
> + */
> + uint32_t mpo_name; /* Name of the soft kernel */
> + uint32_t m_image_offset; /* Image offset */
> + uint32_t m_image_size; /* Image size */
> + uint32_t mpo_version; /* Version */
> + uint32_t mpo_md5_value; /* MD5 checksum */
> + uint32_t mpo_symbol_name; /* Symbol name */
> + uint32_t m_num_instances; /* Number of instances */
> + uint8_t padding[36]; /* Reserved for future use */
> + uint8_t reservedExt[16]; /* Reserved for future extended data */
> +};
> +
> +enum CHECKSUM_TYPE {
> + CST_UNKNOWN = 0,
> + CST_SDBM = 1,
> + CST_LAST
> +};
> +
> +#ifdef __cplusplus
> +}
> +#endif
> +
> +#endif
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-02-21 17:12 ` Tom Rix
@ 2021-02-21 18:33 ` Moritz Fischer
2021-03-06 1:13 ` Lizhi Hou
2021-02-26 21:23 ` Lizhi Hou
1 sibling, 1 reply; 87+ messages in thread
From: Moritz Fischer @ 2021-02-21 18:33 UTC (permalink / raw)
To: Tom Rix
Cc: Lizhi Hou, linux-kernel, Lizhi Hou, linux-fpga, maxz,
sonal.santan, michal.simek, stefanos, devicetree, mdf, robh,
Max Zhen
On Sun, Feb 21, 2021 at 09:12:37AM -0800, Tom Rix wrote:
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
> > Alveo FPGA firmware and partial reconfigure file are in xclbin format.
> This code enumerates and extracts
> > Add
> > code to enumerate and extract sections from xclbin files. xclbin.h is cross
> > platform and used across all platforms and OS
> >
> > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> > Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> > ---
> > drivers/fpga/xrt/include/xclbin-helper.h | 52 +++
> > drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++++++++++++++
> > include/uapi/linux/xrt/xclbin.h | 408 +++++++++++++++++++++++
> > 3 files changed, 854 insertions(+)
> > create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
> > create mode 100644 drivers/fpga/xrt/lib/xclbin.c
> > create mode 100644 include/uapi/linux/xrt/xclbin.h
> >
> > diff --git a/drivers/fpga/xrt/include/xclbin-helper.h b/drivers/fpga/xrt/include/xclbin-helper.h
> > new file mode 100644
> > index 000000000000..68218efc9d0b
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/include/xclbin-helper.h
> > @@ -0,0 +1,52 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Header file for Xilinx Runtime (XRT) driver
> > + *
> > + * Copyright (C) 2020-2021 Xilinx, Inc.
> > + *
> > + * Authors:
> > + * David Zhang <davidzha@xilinx.com>
> > + * Sonal Santan <sonal.santan@xilinx.com>
> > + */
> > +
> > +#ifndef _XRT_XCLBIN_H
> > +#define _XRT_XCLBIN_H
>
> The header guard should match the filename.
>
> > +
> > +#include <linux/types.h>
> > +#include <linux/device.h>
> > +#include <linux/xrt/xclbin.h>
> > +
> > +#define ICAP_XCLBIN_V2 "xclbin2"
> > +#define DMA_HWICAP_BITFILE_BUFFER_SIZE 1024
> > +#define MAX_XCLBIN_SIZE (1024 * 1024 * 1024) /* Assuming xclbin <= 1G, always */
> #defines should have a prefix, maybe XRT_ or XCLBIN_
> > +
> > +enum axlf_section_kind;
> > +struct axlf;
> > +
> > +/**
> > + * Bitstream header information as defined by Xilinx tools.
> > + * Please note that this struct definition is not owned by the driver.
> > + */
> > +struct hw_icap_bit_header {
>
> File headers usually have fixed length fields like uint32_t
>
> Is this a structure the real header is converted into ?
>
> > + unsigned int header_length; /* Length of header in 32 bit words */
> > + unsigned int bitstream_length; /* Length of bitstream to read in bytes*/
> > + unsigned char *design_name; /* Design name get from bitstream */
> > + unsigned char *part_name; /* Part name read from bitstream */
> > + unsigned char *date; /* Date read from bitstream header */
> > + unsigned char *time; /* Bitstream creation time */
> > + unsigned int magic_length; /* Length of the magic numbers */
> > + unsigned char *version; /* Version string */
> > +};
> > +
> > +const char *xrt_xclbin_kind_to_string(enum axlf_section_kind kind);
>
> Only add decl's that are using in multiple files.
>
> This is only defined in xclbin.c, why does it need to be in the header ?
>
> > +int xrt_xclbin_get_section(const struct axlf *xclbin,
> > + enum axlf_section_kind kind, void **data,
> > + uint64_t *len);
> > +int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb);
> > +int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
> > + unsigned int size,
> > + struct hw_icap_bit_header *header);
> > +void xrt_xclbin_free_header(struct hw_icap_bit_header *header);
> > +const char *xrt_clock_type2epname(enum CLOCK_TYPE type);
> CLOCK_TYPE needs a prefix, something like XCLBIN_CLOCK_TYPE
> > +
> > +#endif /* _XRT_XCLBIN_H */
> > diff --git a/drivers/fpga/xrt/lib/xclbin.c b/drivers/fpga/xrt/lib/xclbin.c
> > new file mode 100644
> > index 000000000000..47dc6ca25c1b
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/lib/xclbin.c
> > @@ -0,0 +1,394 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Xilinx Alveo FPGA Driver XCLBIN parser
> > + *
> > + * Copyright (C) 2020-2021 Xilinx, Inc.
> > + *
> > + * Authors: David Zhang <davidzha@xilinx.com>
> > + */
> > +
> > +#include <asm/errno.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/device.h>
> > +#include "xclbin-helper.h"
> > +#include "metadata.h"
> > +
> What is XHI ? Maybe expand this, at the lease should comment
> > +/* Used for parsing bitstream header */
> > +#define XHI_EVEN_MAGIC_BYTE 0x0f
> > +#define XHI_ODD_MAGIC_BYTE 0xf0
> > +
> > +/* Extra mode for IDLE */
> > +#define XHI_OP_IDLE -1
> > +#define XHI_BIT_HEADER_FAILURE -1
> > +
> > +/* The imaginary module length register */
> > +#define XHI_MLR 15
> > +
> > +static inline unsigned char xhi_data_and_inc(const unsigned char *d, int *i, int sz)
> could move to the *.h
> > +{_
> > + unsigned char data;
> > +
> > + if (*i >= sz)
> > + return -1;
> The return value of this funtion is not always checked, at the least add a dev_err here
> > +
> > + data = d[*i];
> > + (*i)++;
> > +
> > + return data;
> > +}
> > +
> > +static const struct axlf_section_header *
> > +xrt_xclbin_get_section_hdr(const struct axlf *xclbin,
> > + enum axlf_section_kind kind)
> > +{
> > + int i = 0;
> > +
> > + for (i = 0; i < xclbin->m_header.m_numSections; i++) {
> > + if (xclbin->m_sections[i].m_sectionKind == kind)
> > + return &xclbin->m_sections[i];
> > + }
> > +
> > + return NULL;
> > +}
> > +
> > +static int
> > +xrt_xclbin_check_section_hdr(const struct axlf_section_header *header,
> > + u64 xclbin_len)
> > +{
> > + int ret;
> > +
> > + ret = (header->m_sectionOffset + header->m_sectionSize) > xclbin_len ? -EINVAL : 0;
>
> Tristate is harder to read, consider replacing with if()
>
> int ret = 0
>
> if ()
>
> ret =
Why not just:
if (header->m_section_offset + header->m_section_size)
return -EINVAL;
return 0;
Also please fix the camelCase throughout the entire patchset.
>
>
> > +
> > + return ret;
> > +}
> > +
> > +static int xrt_xclbin_section_info(const struct axlf *xclbin,
> > + enum axlf_section_kind kind,
> > + u64 *offset, u64 *size)
> > +{
> > + const struct axlf_section_header *mem_header = NULL;
> > + u64 xclbin_len;
> > + int err = 0;
> > +
> > + mem_header = xrt_xclbin_get_section_hdr(xclbin, kind);
> > + if (!mem_header)
> > + return -EINVAL;
> > +
> > + xclbin_len = xclbin->m_header.m_length;
> > + if (xclbin_len > MAX_XCLBIN_SIZE)
> > + return -EINVAL;
>
> This check can be added to the function call..
>
> or the sanity checking added to the earier call to *get_section_hdr
>
> There a number of small functions that can be combined.
>
> > +
> > + err = xrt_xclbin_check_section_hdr(mem_header, xclbin_len);
> > + if (err)
> > + return err;
> > +
> > + *offset = mem_header->m_sectionOffset;
> > + *size = mem_header->m_sectionSize;
> > +
> > + return 0;
> > +}
> > +
> > +/* caller should free the allocated memory for **data */
>
> must free
>
> This comment also needs to be with the *.h decl
>
> > +int xrt_xclbin_get_section(const struct axlf *buf,
> > + enum axlf_section_kind kind,
> > + void **data, u64 *len)
> > +{
> > + const struct axlf *xclbin = (const struct axlf *)buf;
> > + void *section = NULL;
> > + int err = 0;
> > + u64 offset = 0;
> > + u64 size = 0;
> > +
> > + err = xrt_xclbin_section_info(xclbin, kind, &offset, &size);
> > + if (err)
> > + return err;
> > +
> > + section = vmalloc(size);
> > + if (!section)
> > + return -ENOMEM;
> > +
> > + memcpy(section, ((const char *)xclbin) + offset, size);
> > +
> > + *data = section;
>
> a general comment
>
> for exported function checking the validity of the inputs in more important.
>
> here you assume **data is valid, really you should check.
>
> > + if (len)
> > + *len = size;
> len setting being optional, needs to be in the *.h comment
> > +
> > + return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(xrt_xclbin_get_section);
> > +
> Instead of allocating new memory and making copies of bits of *data
>
> why not have the points reference data ?
>
> The size operations look like translating big endian data to little endian.
>
> This will break on a big endian host.
>
> > +/* parse bitstream header */
> > +int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
> > + unsigned int size,
> > + struct hw_icap_bit_header *header)
> > +{
> > + unsigned int index;
> > + unsigned int len;
> > + unsigned int tmp;
> > + unsigned int i;
> > +
> > + memset(header, 0, sizeof(*header));
> > + /* Start Index at start of bitstream */
> > + index = 0;
> > +
> > + /* Initialize HeaderLength. If header returned early inidicates
> > + * failure.
>
> This side effect should be documented in the *.h comment.
>
> Also the multi line comment is a bit weird, not sure if it is ok
>
> > + */
> > + header->header_length = XHI_BIT_HEADER_FAILURE;
> > +
> > + /* Get "Magic" length */
> > + header->magic_length = xhi_data_and_inc(data, &index, size);
> > + header->magic_length = (header->magic_length << 8) | xhi_data_and_inc(data, &index, size);
> > +
> > + /* Read in "magic" */
> > + for (i = 0; i < header->magic_length - 1; i++) {
> > + tmp = xhi_data_and_inc(data, &index, size);
> > + if (i % 2 == 0 && tmp != XHI_EVEN_MAGIC_BYTE)
if !(i % 2) ...
> > + return -1; /* INVALID_FILE_HEADER_ERROR */
> > +
> > + if (i % 2 == 1 && tmp != XHI_ODD_MAGIC_BYTE)
> > + return -1; /* INVALID_FILE_HEADER_ERROR */
> > + }
> > +
> > + /* Read null end of magic data. */
> > + tmp = xhi_data_and_inc(data, &index, size);
> > +
> > + /* Read 0x01 (short) */
> > + tmp = xhi_data_and_inc(data, &index, size);
> > + tmp = (tmp << 8) | xhi_data_and_inc(data, &index, size);
> > +
> > + /* Check the "0x01" half word */
> > + if (tmp != 0x01)
> > + return -1; /* INVALID_FILE_HEADER_ERROR */
> > +
> > + /* Read 'a' */
> > + tmp = xhi_data_and_inc(data, &index, size);
> > + if (tmp != 'a')
> > + return -1; /* INVALID_FILE_HEADER_ERROR */
> > +
> > + /* Get Design Name length */
> > + len = xhi_data_and_inc(data, &index, size);
> > + len = (len << 8) | xhi_data_and_inc(data, &index, size);
> > +
> > + /* allocate space for design name and final null character. */
> > + header->design_name = vmalloc(len);
> > + if (!header->design_name)
> > + return -ENOMEM;
> > +
> > + /* Read in Design Name */
> > + for (i = 0; i < len; i++)
> > + header->design_name[i] = xhi_data_and_inc(data, &index, size);
> > +
> > + if (header->design_name[len - 1] != '\0')
> > + return -1;
> > +
> > + header->version = strstr(header->design_name, "Version=") + strlen("Version=");
> > +
> > + /* Read 'b' */
> > + tmp = xhi_data_and_inc(data, &index, size);
> > + if (tmp != 'b')
> > + return -1; /* INVALID_FILE_HEADER_ERROR */
> > +
> > + /* Get Part Name length */
> > + len = xhi_data_and_inc(data, &index, size);
> > + len = (len << 8) | xhi_data_and_inc(data, &index, size);
> > +
> > + /* allocate space for part name and final null character. */
> > + header->part_name = vmalloc(len);
> > + if (!header->part_name)
> > + return -ENOMEM;
> > +
> > + /* Read in part name */
> > + for (i = 0; i < len; i++)
> > + header->part_name[i] = xhi_data_and_inc(data, &index, size);
> > +
> > + if (header->part_name[len - 1] != '\0')
> > + return -1;
> > +
> > + /* Read 'c' */
> > + tmp = xhi_data_and_inc(data, &index, size);
> > + if (tmp != 'c')
> > + return -1; /* INVALID_FILE_HEADER_ERROR */
> > +
> > + /* Get date length */
> > + len = xhi_data_and_inc(data, &index, size);
> > + len = (len << 8) | xhi_data_and_inc(data, &index, size);
> > +
> > + /* allocate space for date and final null character. */
> > + header->date = vmalloc(len);
> > + if (!header->date)
> > + return -ENOMEM;
> > +
> > + /* Read in date name */
> > + for (i = 0; i < len; i++)
> > + header->date[i] = xhi_data_and_inc(data, &index, size);
> > +
> > + if (header->date[len - 1] != '\0')
> > + return -1;
> generally -EINVAL is more meaningful than -1
> > +
> > + /* Read 'd' */
> > + tmp = xhi_data_and_inc(data, &index, size);
> > + if (tmp != 'd')
> > + return -1; /* INVALID_FILE_HEADER_ERROR */
> > +
> > + /* Get time length */
> > + len = xhi_data_and_inc(data, &index, size);
> > + len = (len << 8) | xhi_data_and_inc(data, &index, size);
> > +
> > + /* allocate space for time and final null character. */
> > + header->time = vmalloc(len);
> > + if (!header->time)
> > + return -ENOMEM;
> > +
> > + /* Read in time name */
> > + for (i = 0; i < len; i++)
> > + header->time[i] = xhi_data_and_inc(data, &index, size);
> > +
> > + if (header->time[len - 1] != '\0')
> > + return -1;
> > +
> > + /* Read 'e' */
> > + tmp = xhi_data_and_inc(data, &index, size);
> > + if (tmp != 'e')
> > + return -1; /* INVALID_FILE_HEADER_ERROR */
> > +
> > + /* Get byte length of bitstream */
> > + header->bitstream_length = xhi_data_and_inc(data, &index, size);
> > + header->bitstream_length = (header->bitstream_length << 8) |
> > + xhi_data_and_inc(data, &index, size);
> > + header->bitstream_length = (header->bitstream_length << 8) |
> > + xhi_data_and_inc(data, &index, size);
> > + header->bitstream_length = (header->bitstream_length << 8) |
> > + xhi_data_and_inc(data, &index, size);
>
> generally a problem
>
> This is confusing, collect the bytes in a temp[] and construct the header->bitstream_length in on statement.
>
> This is a case where xhi_data_and_inc return is not checked and if it failed could blow up later.
>
> > +
> > + header->header_length = index;
>
> index is not a good variable name if it going to be stored as a length.
>
> consider changing it to something like current_length.
>
> > +
> > + return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(xrt_xclbin_parse_bitstream_header);
> > +
> > +void xrt_xclbin_free_header(struct hw_icap_bit_header *header)
> > +{
> > + vfree(header->design_name);
> > + vfree(header->part_name);
> > + vfree(header->date);
> > + vfree(header->time);
> missing header->version
> > +}
> > +EXPORT_SYMBOL_GPL(xrt_xclbin_free_header);
> > +
> > +struct xrt_clock_desc {
> > + char *clock_ep_name;
> > + u32 clock_xclbin_type;
> > + char *clkfreq_ep_name;
> > +} clock_desc[] = {
> > + {
> > + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL1,
> > + .clock_xclbin_type = CT_DATA,
> > + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K1,
> > + },
> > + {
> > + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL2,
> > + .clock_xclbin_type = CT_KERNEL,
> > + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K2,
> > + },
> > + {
> > + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL3,
> > + .clock_xclbin_type = CT_SYSTEM,
> > + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_HBM,
> > + },
> > +};
> > +
> > +const char *xrt_clock_type2epname(enum CLOCK_TYPE type)
> > +{
> > + int i;
> > +
> > + for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
> > + if (clock_desc[i].clock_xclbin_type == type)
> > + return clock_desc[i].clock_ep_name;
> > + }
> > + return NULL;
> > +}
> > +EXPORT_SYMBOL_GPL(xrt_clock_type2epname);
>
> What is clock stuff doing in xclbin ?
>
> I think clock needs its own file
>
> > +
> > +static const char *clock_type2clkfreq_name(u32 type)
> > +{
> > + int i;
> > +
> > + for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
> > + if (clock_desc[i].clock_xclbin_type == type)
> > + return clock_desc[i].clkfreq_ep_name;
> > + }
> > + return NULL;
> > +}
> > +
> > +static int xrt_xclbin_add_clock_metadata(struct device *dev,
> > + const struct axlf *xclbin,
> > + char *dtb)
> > +{
> > + int i;
> > + u16 freq;
> > + struct clock_freq_topology *clock_topo;
> > + int rc = xrt_xclbin_get_section(xclbin, CLOCK_FREQ_TOPOLOGY,
> > + (void **)&clock_topo, NULL);
> > +
> > + if (rc)
> > + return 0;
> failing is ok ?
> > +
> > + for (i = 0; i < clock_topo->m_count; i++) {
> > + u8 type = clock_topo->m_clock_freq[i].m_type;
> > + const char *ep_name = xrt_clock_type2epname(type);
> > + const char *counter_name = clock_type2clkfreq_name(type);
> > +
> > + if (!ep_name || !counter_name)
> > + continue;
> > +
> > + freq = cpu_to_be16(clock_topo->m_clock_freq[i].m_freq_Mhz);
> > + rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_FREQ,
> > + &freq, sizeof(freq));
> > + if (rc)
> > + break;
> > +
> > + rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_CNT,
> > + counter_name, strlen(counter_name) + 1);
> > + if (rc)
> > + break;
> Failing in a loop, why isn't there some cleanup of the *set_prop() calls ?
> > + }
> > +
> > + vfree(clock_topo);
> > +
> > + return rc;
> > +}
> > +
> > +int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb)
> > +{
> > + char *md = NULL, *newmd = NULL;
> > + u64 len;
> *dtb = NULL;
> > + int rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA,
> > + (void **)&md, &len);
> > +
> > + if (rc)
> > + goto done;
> > +
> > + /* Sanity check the dtb section. */
> > + if (xrt_md_size(dev, md) > len) {
> > + rc = -EINVAL;
> > + goto done;
> > + }
> > +
> > + newmd = xrt_md_dup(dev, md);
> > + if (!newmd) {
> > + rc = -EFAULT;
> > + goto done;
> > + }
> > + /* Convert various needed xclbin sections into dtb. */
> > + rc = xrt_xclbin_add_clock_metadata(dev, xclbin, newmd);
>
> newmd is only valid here, but the above error handling jump here. change this to
>
> if (!rc)
>
> *dtb = newmd
>
> else
>
> vfree(newmd)
>
> done:
>
> vfree(md)
>
> return rc;
>
> > +
> > +done:
> > + if (rc == 0)
> > + *dtb = newmd;
> > + else
> > + vfree(newmd);
> > + vfree(md);
> > + return rc;
> > +}
> > +EXPORT_SYMBOL_GPL(xrt_xclbin_get_metadata);
> > diff --git a/include/uapi/linux/xrt/xclbin.h b/include/uapi/linux/xrt/xclbin.h
> > new file mode 100644
> > index 000000000000..53f140123ef1
> > --- /dev/null
> > +++ b/include/uapi/linux/xrt/xclbin.h
> > @@ -0,0 +1,408 @@
> > +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> > +/*
> > + * Xilinx FPGA compiled binary container format
> > + *
> > + * Copyright (C) 2015-2021, Xilinx Inc
> > + */
> > +
> > +#ifndef _XCLBIN_H_
> > +#define _XCLBIN_H_
> > +
> > +#ifdef _WIN32
>
> WIN32 ?
>
> Only 1 other header has this ifdef
>
> > + #include <cstdint>
> > + #include <algorithm>
> c++ is being assumed for windows
> > + #include "windows/uuid.h"
> thank you for not including windows.h ;)
> > +#else
> > + #if defined(__KERNEL__)
> > + #include <linux/types.h>
> > + #include <linux/uuid.h>
> > + #include <linux/version.h>
> > + #elif defined(__cplusplus)
> > + #include <cstdlib>
> > + #include <cstdint>
> > + #include <algorithm>
> > + #include <uuid/uuid.h>
> > + #else
> > + #include <stdlib.h>
> > + #include <stdint.h>
> > + #include <uuid/uuid.h>
> > + #endif
> > +#endif
Tbh, not a big fan of this ...
> > +
>
> Review these includes, some could be convenience includes.
>
> ex/ linux/version.h with no obvious use of version macros.
>
> struct axlf_header {
> + uint64_t m_length; /* Total size of the xclbin file */
> .. snip ..
> + union {
> + char m_next_axlf[16]; /* Name of next xclbin file */
> + /* in the daisy chain */
> + uuid_t uuid; /* uuid of this xclbin*/
> + };
>
> As mentioned in an earlier patch, if uuid_t is larger than 16 bytes, axlf_header breaks.
> while it is convenient to have this type here, it would be better this access was handled in another way.
> Maybe a host specific function.
>
> I also do not see a pragma pack, usually this is set of 1 so the compiler does not shuffle elements, increase size etc.
>
> > +#ifdef __cplusplus
> > +extern "C" {
> > +#endif
> > +
> > +/**
> > + * DOC: Container format for Xilinx FPGA images
> > + * The container stores bitstreams, metadata and firmware images.
> > + * xclbin/xsabin is ELF-like binary container format. It is structured
> is an ELF-like file format. It is a structured
> > + * series of sections. There is a file header followed by several section
> > + * headers which is followed by sections. A section header points to an
> > + * actual section. There is an optional signature at the end. The
> > + * following figure illustrates a typical xclbin:
> > + *
> > + * +---------------------+
> > + * | |
> > + * | HEADER |
> > + * +---------------------+
> > + * | SECTION HEADER |
> > + * | |
> > + * +---------------------+
> > + * | ... |
> > + * | |
> > + * +---------------------+
> > + * | SECTION HEADER |
> > + * | |
> > + * +---------------------+
> > + * | SECTION |
> > + * | |
> > + * +---------------------+
> > + * | ... |
> > + * | |
> > + * +---------------------+
> > + * | SECTION |
> > + * | |
> > + * +---------------------+
> > + * | SIGNATURE |
> > + * | (OPTIONAL) |
> > + * +---------------------+
>
> This ascii art is a mixture of tabs and spaces, for someone with tab = 2 spaces, this will look messed up.
>
> convert the tabs to spaces
>
> > + */
> > +
> > +enum XCLBIN_MODE {
> > + XCLBIN_FLAT,
>
> generally
>
> all enums used in a file format should be initialized.
>
> This likely should be
>
> XCLBIN_FLAT = 0,
>
> > + XCLBIN_PR,
> > + XCLBIN_TANDEM_STAGE2,
> > + XCLBIN_TANDEM_STAGE2_WITH_PR,
> > + XCLBIN_HW_EMU,
> > + XCLBIN_SW_EMU,
> > + XCLBIN_MODE_MAX
> > +};
> > +
> > +enum axlf_section_kind {
> > + BITSTREAM = 0,
> > + CLEARING_BITSTREAM,
> > + EMBEDDED_METADATA,
> > + FIRMWARE,
> > + DEBUG_DATA,
> > + SCHED_FIRMWARE,
> > + MEM_TOPOLOGY,
> > + CONNECTIVITY,
> > + IP_LAYOUT,
> > + DEBUG_IP_LAYOUT,
> > + DESIGN_CHECK_POINT,
> > + CLOCK_FREQ_TOPOLOGY,
> > + MCS,
> > + BMC,
> > + BUILD_METADATA,
> > + KEYVALUE_METADATA,
> > + USER_METADATA,
> > + DNA_CERTIFICATE,
> > + PDI,
> > + BITSTREAM_PARTIAL_PDI,
> > + PARTITION_METADATA,
> > + EMULATION_DATA,
> > + SYSTEM_METADATA,
> > + SOFT_KERNEL,
> > + ASK_FLASH,
> > + AIE_METADATA,
> > + ASK_GROUP_TOPOLOGY,
> > + ASK_GROUP_CONNECTIVITY
> > +};
> > +
> > +enum MEM_TYPE {
> > + MEM_DDR3,
> > + MEM_DDR4,
> > + MEM_DRAM,
> > + MEM_STREAMING,
> > + MEM_PREALLOCATED_GLOB,
> > + MEM_ARE,
> > + MEM_HBM,
> > + MEM_BRAM,
> > + MEM_URAM,
> > + MEM_STREAMING_CONNECTION
> > +};
> > +
> > +enum IP_TYPE {
> > + IP_MB = 0,
> > + IP_KERNEL,
> > + IP_DNASC,
> > + IP_DDR4_CONTROLLER,
> > + IP_MEM_DDR4,
> > + IP_MEM_HBM
> > +};
> > +
> > +struct axlf_section_header {
> > + uint32_t m_sectionKind; /* Section type */
> > + char m_sectionName[16]; /* Examples: "stage2", "clear1", */
> > + /* "clear2", "ocl1", "ocl2, */
> > + /* "ublaze", "sched" */
> > + uint64_t m_sectionOffset; /* File offset of section data */
> > + uint64_t m_sectionSize; /* Size of section data */
> > +};
> > +
> > +struct axlf_header {
> > + uint64_t m_length; /* Total size of the xclbin file */
> > + uint64_t m_timeStamp; /* Number of seconds since epoch */
> > + /* when xclbin was created */
> > + uint64_t m_featureRomTimeStamp; /* TimeSinceEpoch of the featureRom */
> > + uint16_t m_versionPatch; /* Patch Version */
> > + uint8_t m_versionMajor; /* Major Version - Version: 2.1.0*/
> i did not see the version checked earlier, which one is expected ?
> > + uint8_t m_versionMinor; /* Minor Version */
> > + uint32_t m_mode; /* XCLBIN_MODE */
> > + union {
> > + struct {
> > + uint64_t m_platformId; /* 64 bit platform ID: */
> > + /* vendor-device-subvendor-subdev */
> > + uint64_t m_featureId; /* 64 bit feature id */
> > + } rom;
> > + unsigned char rom_uuid[16]; /* feature ROM UUID for which */
> > + /* this xclbin was generated */
> > + };
> > + unsigned char m_platformVBNV[64]; /* e.g. */
> what is VBNV?
> > + /* xilinx:xil-accel-rd-ku115:4ddr-xpr:3.4: null terminated */
> > + union {
> > + char m_next_axlf[16]; /* Name of next xclbin file */
> > + /* in the daisy chain */
> > + uuid_t uuid; /* uuid of this xclbin*/
> > + };
> > + char m_debug_bin[16]; /* Name of binary with debug */
> > + /* information */
> > + uint32_t m_numSections; /* Number of section headers */
> > +};
> > +
> > +struct axlf {
> > + char m_magic[8]; /* Should be "xclbin2\0" */
> > + int32_t m_signature_length; /* Length of the signature. */
> > + /* -1 indicates no signature */
> > + unsigned char reserved[28]; /* Note: Initialized to 0xFFs */
> > +
> > + unsigned char m_keyBlock[256]; /* Signature for validation */
> > + /* of binary */
> > + uint64_t m_uniqueId; /* axlf's uniqueId, use it to */
> > + /* skip redownload etc */
> > + struct axlf_header m_header; /* Inline header */
> > + struct axlf_section_header m_sections[1]; /* One or more section */
> > + /* headers follow */
> > +};
> > +
> > +/* bitstream information */
> > +struct xlnx_bitstream {
> > + uint8_t m_freq[8];
> > + char bits[1];
> > +};
> > +
> > +/**** MEMORY TOPOLOGY SECTION ****/
> > +struct mem_data {
> > + uint8_t m_type; /* enum corresponding to mem_type. */
> > + uint8_t m_used; /* if 0 this bank is not present */
> > + union {
> > + uint64_t m_size; /* if mem_type DDR, then size in KB; */
> > + uint64_t route_id; /* if streaming then "route_id" */
> > + };
> > + union {
> > + uint64_t m_base_address;/* if DDR then the base address; */
> > + uint64_t flow_id; /* if streaming then "flow id" */
> > + };
> > + unsigned char m_tag[16]; /* DDR: BANK0,1,2,3, has to be null */
> > + /* terminated; if streaming then stream0, 1 etc */
> > +};
> > +
> > +struct mem_topology {
> > + int32_t m_count; /* Number of mem_data */
> > + struct mem_data m_mem_data[1]; /* Should be sorted on mem_type */
> > +};
> > +
> > +/**** CONNECTIVITY SECTION ****/
> > +/* Connectivity of each argument of Kernel. It will be in terms of argument
>
> This section does not make sense.
>
> Likely you mean some algorithm kernel, rather than the linux kernel.
>
> > + * index associated. For associating kernel instances with arguments and
> > + * banks, start at the connectivity section. Using the m_ip_layout_index
> > + * access the ip_data.m_name. Now we can associate this kernel instance
> > + * with its original kernel name and get the connectivity as well. This
> > + * enables us to form related groups of kernel instances.
> > + */
> > +
> > +struct connection {
> > + int32_t arg_index; /* From 0 to n, may not be contiguous as scalars */
> > + /* skipped */
> > + int32_t m_ip_layout_index; /* index into the ip_layout section. */
> > + /* ip_layout.m_ip_data[index].m_type == IP_KERNEL */
> > + int32_t mem_data_index; /* index of the m_mem_data . Flag error is */
> > + /* m_used false. */
> > +};
> > +
> > +struct connectivity {
> > + int32_t m_count;
> > + struct connection m_connection[1];
> > +};
> > +
> > +/**** IP_LAYOUT SECTION ****/
> > +
> > +/* IP Kernel */
> > +#define IP_INT_ENABLE_MASK 0x0001
> > +#define IP_INTERRUPT_ID_MASK 0x00FE
> > +#define IP_INTERRUPT_ID_SHIFT 0x1
> > +
> > +enum IP_CONTROL {
> > + AP_CTRL_HS = 0,
> > + AP_CTRL_CHAIN = 1,
> > + AP_CTRL_NONE = 2,
> > + AP_CTRL_ME = 3,
> > + ACCEL_ADAPTER = 4
> assigning beyond the first is not necessary unless there are dups or gaps
> > +};
> > +
> > +#define IP_CONTROL_MASK 0xFF00
> > +#define IP_CONTROL_SHIFT 0x8
> > +
> > +/* IPs on AXI lite - their types, names, and base addresses.*/
> > +struct ip_data {
> > + uint32_t m_type; /* map to IP_TYPE enum */
> > + union {
> > + uint32_t properties; /* Default: 32-bits to indicate ip */
> > + /* specific property. */
> > + /* m_type: IP_KERNEL
> > + * m_int_enable : Bit - 0x0000_0001;
> > + * m_interrupt_id : Bits - 0x0000_00FE;
> > + * m_ip_control : Bits = 0x0000_FF00;
> > + */
> > + struct { /* m_type: IP_MEM_* */
> > + uint16_t m_index;
> > + uint8_t m_pc_index;
> > + uint8_t unused;
> > + } indices;
> > + };
> > + uint64_t m_base_address;
> > + uint8_t m_name[64]; /* eg Kernel name corresponding to KERNEL */
> > + /* instance, can embed CU name in future. */
> > +};
> > +
> > +struct ip_layout {
> > + int32_t m_count;
> > + struct ip_data m_ip_data[1]; /* All the ip_data needs to be sorted */
> > + /* by m_base_address. */
>
> general
>
> doing the bla[1] for c++ ?
>
> Tom
>
> > +};
> > +
> > +/*** Debug IP section layout ****/
> > +enum DEBUG_IP_TYPE {
> > + UNDEFINED = 0,
> > + LAPC,
> > + ILA,
> > + AXI_MM_MONITOR,
> > + AXI_TRACE_FUNNEL,
> > + AXI_MONITOR_FIFO_LITE,
> > + AXI_MONITOR_FIFO_FULL,
> > + ACCEL_MONITOR,
> > + AXI_STREAM_MONITOR,
> > + AXI_STREAM_PROTOCOL_CHECKER,
> > + TRACE_S2MM,
> > + AXI_DMA,
> > + TRACE_S2MM_FULL
> > +};
> > +
> > +struct debug_ip_data {
> > + uint8_t m_type; /* type of enum DEBUG_IP_TYPE */
> > + uint8_t m_index_lowbyte;
> > + uint8_t m_properties;
> > + uint8_t m_major;
> > + uint8_t m_minor;
> > + uint8_t m_index_highbyte;
> > + uint8_t m_reserved[2];
> > + uint64_t m_base_address;
> > + char m_name[128];
> > +};
> > +
> > +struct debug_ip_layout {
> > + uint16_t m_count;
> > + struct debug_ip_data m_debug_ip_data[1];
> > +};
> > +
> > +/* Supported clock frequency types */
> > +enum CLOCK_TYPE {
> > + CT_UNUSED = 0, /* Initialized value */
> > + CT_DATA = 1, /* Data clock */
> > + CT_KERNEL = 2, /* Kernel clock */
> > + CT_SYSTEM = 3 /* System Clock */
> > +};
> > +
> > +/* Clock Frequency Entry */
> > +struct clock_freq {
> > + uint16_t m_freq_Mhz; /* Frequency in MHz */
> > + uint8_t m_type; /* Clock type (enum CLOCK_TYPE) */
> > + uint8_t m_unused[5]; /* Not used - padding */
> > + char m_name[128]; /* Clock Name */
> > +};
> > +
> > +/* Clock frequency section */
> > +struct clock_freq_topology {
> > + int16_t m_count; /* Number of entries */
> > + struct clock_freq m_clock_freq[1]; /* Clock array */
> > +};
> > +
> > +/* Supported MCS file types */
> > +enum MCS_TYPE {
> > + MCS_UNKNOWN = 0, /* Initialized value */
> > + MCS_PRIMARY = 1, /* The primary mcs file data */
> > + MCS_SECONDARY = 2, /* The secondary mcs file data */
> > +};
> > +
> > +/* One chunk of MCS data */
> > +struct mcs_chunk {
> > + uint8_t m_type; /* MCS data type */
just call them type, unused, offset. Drop the m_*
> > + uint8_t m_unused[7]; /* padding */
> > + uint64_t m_offset; /* data offset from the start of */
> > + /* the section */
> > + uint64_t m_size; /* data size */
> > +};
> > +
> > +/* MCS data section */
> > +struct mcs {
> > + int8_t m_count; /* Number of chunks */
> > + int8_t m_unused[7]; /* padding */
> > + struct mcs_chunk m_chunk[1]; /* MCS chunks followed by data */
> > +};
> > +
> > +/* bmc data section */
> > +struct bmc {
> > + uint64_t m_offset; /* data offset from the start of */
> > + /* the section */
> > + uint64_t m_size; /* data size (bytes) */
> > + char m_image_name[64]; /* Name of the image */
> > + /* (e.g., MSP432P401R) */
> > + char m_device_name[64]; /* Device ID (e.g., VCU1525) */
> > + char m_version[64];
> > + char m_md5value[33]; /* MD5 Expected Value */
> > + /* (e.g., 56027182079c0bd621761b7dab5a27ca)*/
> > + char m_padding[7]; /* Padding */
> > +};
> > +
> > +/* soft kernel data section, used by classic driver */
> > +struct soft_kernel {
> > + /** Prefix Syntax:
> > + * mpo - member, pointer, offset
> > + * This variable represents a zero terminated string
> > + * that is offseted from the beginning of the section.
> > + * The pointer to access the string is initialized as follows:
> > + * char * pCharString = (address_of_section) + (mpo value)
> > + */
> > + uint32_t mpo_name; /* Name of the soft kernel */
> > + uint32_t m_image_offset; /* Image offset */
> > + uint32_t m_image_size; /* Image size */
> > + uint32_t mpo_version; /* Version */
> > + uint32_t mpo_md5_value; /* MD5 checksum */
> > + uint32_t mpo_symbol_name; /* Symbol name */
> > + uint32_t m_num_instances; /* Number of instances */
> > + uint8_t padding[36]; /* Reserved for future use */
> > + uint8_t reservedExt[16]; /* Reserved for future extended data */
> > +};
> > +
> > +enum CHECKSUM_TYPE {
> > + CST_UNKNOWN = 0,
> > + CST_SDBM = 1,
> > + CST_LAST
> > +};
> > +
> > +#ifdef __cplusplus
> > +}
> > +#endif
> > +
> > +#endif
>
I'll take a closer look, these were just random things I bumped into.
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-02-21 18:33 ` Moritz Fischer
@ 2021-03-06 1:13 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-06 1:13 UTC (permalink / raw)
To: Moritz Fischer, Tom Rix
Cc: Lizhi Hou, linux-kernel, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, robh, Max Zhen
Hi Moritz,
On 02/21/2021 10:33 AM, Moritz Fischer wrote:
> On Sun, Feb 21, 2021 at 09:12:37AM -0800, Tom Rix wrote:
>> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>>> Alveo FPGA firmware and partial reconfigure file are in xclbin format.
>> This code enumerates and extracts
>>> Add
>>> code to enumerate and extract sections from xclbin files. xclbin.h is cross
>>> platform and used across all platforms and OS
>>>
>>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>>> ---
>>> drivers/fpga/xrt/include/xclbin-helper.h | 52 +++
>>> drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++++++++++++++
>>> include/uapi/linux/xrt/xclbin.h | 408 +++++++++++++++++++++++
>>> 3 files changed, 854 insertions(+)
>>> create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>>> create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>>> create mode 100644 include/uapi/linux/xrt/xclbin.h
>>>
>>> diff --git a/drivers/fpga/xrt/include/xclbin-helper.h b/drivers/fpga/xrt/include/xclbin-helper.h
>>> new file mode 100644
>>> index 000000000000..68218efc9d0b
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/include/xclbin-helper.h
>>> @@ -0,0 +1,52 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * Header file for Xilinx Runtime (XRT) driver
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors:
>>> + * David Zhang <davidzha@xilinx.com>
>>> + * Sonal Santan <sonal.santan@xilinx.com>
>>> + */
>>> +
>>> +#ifndef _XRT_XCLBIN_H
>>> +#define _XRT_XCLBIN_H
>> The header guard should match the filename.
>>
>>> +
>>> +#include <linux/types.h>
>>> +#include <linux/device.h>
>>> +#include <linux/xrt/xclbin.h>
>>> +
>>> +#define ICAP_XCLBIN_V2 "xclbin2"
>>> +#define DMA_HWICAP_BITFILE_BUFFER_SIZE 1024
>>> +#define MAX_XCLBIN_SIZE (1024 * 1024 * 1024) /* Assuming xclbin <= 1G, always */
>> #defines should have a prefix, maybe XRT_ or XCLBIN_
>>> +
>>> +enum axlf_section_kind;
>>> +struct axlf;
>>> +
>>> +/**
>>> + * Bitstream header information as defined by Xilinx tools.
>>> + * Please note that this struct definition is not owned by the driver.
>>> + */
>>> +struct hw_icap_bit_header {
>> File headers usually have fixed length fields like uint32_t
>>
>> Is this a structure the real header is converted into ?
>>
>>> + unsigned int header_length; /* Length of header in 32 bit words */
>>> + unsigned int bitstream_length; /* Length of bitstream to read in bytes*/
>>> + unsigned char *design_name; /* Design name get from bitstream */
>>> + unsigned char *part_name; /* Part name read from bitstream */
>>> + unsigned char *date; /* Date read from bitstream header */
>>> + unsigned char *time; /* Bitstream creation time */
>>> + unsigned int magic_length; /* Length of the magic numbers */
>>> + unsigned char *version; /* Version string */
>>> +};
>>> +
>>> +const char *xrt_xclbin_kind_to_string(enum axlf_section_kind kind);
>> Only add decl's that are using in multiple files.
>>
>> This is only defined in xclbin.c, why does it need to be in the header ?
>>
>>> +int xrt_xclbin_get_section(const struct axlf *xclbin,
>>> + enum axlf_section_kind kind, void **data,
>>> + uint64_t *len);
>>> +int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb);
>>> +int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
>>> + unsigned int size,
>>> + struct hw_icap_bit_header *header);
>>> +void xrt_xclbin_free_header(struct hw_icap_bit_header *header);
>>> +const char *xrt_clock_type2epname(enum CLOCK_TYPE type);
>> CLOCK_TYPE needs a prefix, something like XCLBIN_CLOCK_TYPE
>>> +
>>> +#endif /* _XRT_XCLBIN_H */
>>> diff --git a/drivers/fpga/xrt/lib/xclbin.c b/drivers/fpga/xrt/lib/xclbin.c
>>> new file mode 100644
>>> index 000000000000..47dc6ca25c1b
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/lib/xclbin.c
>>> @@ -0,0 +1,394 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * Xilinx Alveo FPGA Driver XCLBIN parser
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors: David Zhang <davidzha@xilinx.com>
>>> + */
>>> +
>>> +#include <asm/errno.h>
>>> +#include <linux/vmalloc.h>
>>> +#include <linux/device.h>
>>> +#include "xclbin-helper.h"
>>> +#include "metadata.h"
>>> +
>> What is XHI ? Maybe expand this, at the lease should comment
>>> +/* Used for parsing bitstream header */
>>> +#define XHI_EVEN_MAGIC_BYTE 0x0f
>>> +#define XHI_ODD_MAGIC_BYTE 0xf0
>>> +
>>> +/* Extra mode for IDLE */
>>> +#define XHI_OP_IDLE -1
>>> +#define XHI_BIT_HEADER_FAILURE -1
>>> +
>>> +/* The imaginary module length register */
>>> +#define XHI_MLR 15
>>> +
>>> +static inline unsigned char xhi_data_and_inc(const unsigned char *d, int *i, int sz)
>> could move to the *.h
>>> +{_
>>> + unsigned char data;
>>> +
>>> + if (*i >= sz)
>>> + return -1;
>> The return value of this funtion is not always checked, at the least add a dev_err here
>>> +
>>> + data = d[*i];
>>> + (*i)++;
>>> +
>>> + return data;
>>> +}
>>> +
>>> +static const struct axlf_section_header *
>>> +xrt_xclbin_get_section_hdr(const struct axlf *xclbin,
>>> + enum axlf_section_kind kind)
>>> +{
>>> + int i = 0;
>>> +
>>> + for (i = 0; i < xclbin->m_header.m_numSections; i++) {
>>> + if (xclbin->m_sections[i].m_sectionKind == kind)
>>> + return &xclbin->m_sections[i];
>>> + }
>>> +
>>> + return NULL;
>>> +}
>>> +
>>> +static int
>>> +xrt_xclbin_check_section_hdr(const struct axlf_section_header *header,
>>> + u64 xclbin_len)
>>> +{
>>> + int ret;
>>> +
>>> + ret = (header->m_sectionOffset + header->m_sectionSize) > xclbin_len ? -EINVAL : 0;
>> Tristate is harder to read, consider replacing with if()
>>
>> int ret = 0
>>
>> if ()
>>
>> ret =
> Why not just:
>
> if (header->m_section_offset + header->m_section_size)
> return -EINVAL;
>
> return 0;
>
> Also please fix the camelCase throughout the entire patchset.
Will fix both.
>>
>>> +
>>> + return ret;
>>> +}
>>> +
>>> +static int xrt_xclbin_section_info(const struct axlf *xclbin,
>>> + enum axlf_section_kind kind,
>>> + u64 *offset, u64 *size)
>>> +{
>>> + const struct axlf_section_header *mem_header = NULL;
>>> + u64 xclbin_len;
>>> + int err = 0;
>>> +
>>> + mem_header = xrt_xclbin_get_section_hdr(xclbin, kind);
>>> + if (!mem_header)
>>> + return -EINVAL;
>>> +
>>> + xclbin_len = xclbin->m_header.m_length;
>>> + if (xclbin_len > MAX_XCLBIN_SIZE)
>>> + return -EINVAL;
>> This check can be added to the function call..
>>
>> or the sanity checking added to the earier call to *get_section_hdr
>>
>> There a number of small functions that can be combined.
>>
>>> +
>>> + err = xrt_xclbin_check_section_hdr(mem_header, xclbin_len);
>>> + if (err)
>>> + return err;
>>> +
>>> + *offset = mem_header->m_sectionOffset;
>>> + *size = mem_header->m_sectionSize;
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +/* caller should free the allocated memory for **data */
>> must free
>>
>> This comment also needs to be with the *.h decl
>>
>>> +int xrt_xclbin_get_section(const struct axlf *buf,
>>> + enum axlf_section_kind kind,
>>> + void **data, u64 *len)
>>> +{
>>> + const struct axlf *xclbin = (const struct axlf *)buf;
>>> + void *section = NULL;
>>> + int err = 0;
>>> + u64 offset = 0;
>>> + u64 size = 0;
>>> +
>>> + err = xrt_xclbin_section_info(xclbin, kind, &offset, &size);
>>> + if (err)
>>> + return err;
>>> +
>>> + section = vmalloc(size);
>>> + if (!section)
>>> + return -ENOMEM;
>>> +
>>> + memcpy(section, ((const char *)xclbin) + offset, size);
>>> +
>>> + *data = section;
>> a general comment
>>
>> for exported function checking the validity of the inputs in more important.
>>
>> here you assume **data is valid, really you should check.
>>
>>> + if (len)
>>> + *len = size;
>> len setting being optional, needs to be in the *.h comment
>>> +
>>> + return 0;
>>> +}
>>> +EXPORT_SYMBOL_GPL(xrt_xclbin_get_section);
>>> +
>> Instead of allocating new memory and making copies of bits of *data
>>
>> why not have the points reference data ?
>>
>> The size operations look like translating big endian data to little endian.
>>
>> This will break on a big endian host.
>>
>>> +/* parse bitstream header */
>>> +int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
>>> + unsigned int size,
>>> + struct hw_icap_bit_header *header)
>>> +{
>>> + unsigned int index;
>>> + unsigned int len;
>>> + unsigned int tmp;
>>> + unsigned int i;
>>> +
>>> + memset(header, 0, sizeof(*header));
>>> + /* Start Index at start of bitstream */
>>> + index = 0;
>>> +
>>> + /* Initialize HeaderLength. If header returned early inidicates
>>> + * failure.
>> This side effect should be documented in the *.h comment.
>>
>> Also the multi line comment is a bit weird, not sure if it is ok
>>
>>> + */
>>> + header->header_length = XHI_BIT_HEADER_FAILURE;
>>> +
>>> + /* Get "Magic" length */
>>> + header->magic_length = xhi_data_and_inc(data, &index, size);
>>> + header->magic_length = (header->magic_length << 8) | xhi_data_and_inc(data, &index, size);
>>> +
>>> + /* Read in "magic" */
>>> + for (i = 0; i < header->magic_length - 1; i++) {
>>> + tmp = xhi_data_and_inc(data, &index, size);
>>> + if (i % 2 == 0 && tmp != XHI_EVEN_MAGIC_BYTE)
> if !(i % 2) ...
Will change it.
>>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>>> +
>>> + if (i % 2 == 1 && tmp != XHI_ODD_MAGIC_BYTE)
>>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>>> + }
>>> +
>>> + /* Read null end of magic data. */
>>> + tmp = xhi_data_and_inc(data, &index, size);
>>> +
>>> + /* Read 0x01 (short) */
>>> + tmp = xhi_data_and_inc(data, &index, size);
>>> + tmp = (tmp << 8) | xhi_data_and_inc(data, &index, size);
>>> +
>>> + /* Check the "0x01" half word */
>>> + if (tmp != 0x01)
>>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>>> +
>>> + /* Read 'a' */
>>> + tmp = xhi_data_and_inc(data, &index, size);
>>> + if (tmp != 'a')
>>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>>> +
>>> + /* Get Design Name length */
>>> + len = xhi_data_and_inc(data, &index, size);
>>> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
>>> +
>>> + /* allocate space for design name and final null character. */
>>> + header->design_name = vmalloc(len);
>>> + if (!header->design_name)
>>> + return -ENOMEM;
>>> +
>>> + /* Read in Design Name */
>>> + for (i = 0; i < len; i++)
>>> + header->design_name[i] = xhi_data_and_inc(data, &index, size);
>>> +
>>> + if (header->design_name[len - 1] != '\0')
>>> + return -1;
>>> +
>>> + header->version = strstr(header->design_name, "Version=") + strlen("Version=");
>>> +
>>> + /* Read 'b' */
>>> + tmp = xhi_data_and_inc(data, &index, size);
>>> + if (tmp != 'b')
>>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>>> +
>>> + /* Get Part Name length */
>>> + len = xhi_data_and_inc(data, &index, size);
>>> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
>>> +
>>> + /* allocate space for part name and final null character. */
>>> + header->part_name = vmalloc(len);
>>> + if (!header->part_name)
>>> + return -ENOMEM;
>>> +
>>> + /* Read in part name */
>>> + for (i = 0; i < len; i++)
>>> + header->part_name[i] = xhi_data_and_inc(data, &index, size);
>>> +
>>> + if (header->part_name[len - 1] != '\0')
>>> + return -1;
>>> +
>>> + /* Read 'c' */
>>> + tmp = xhi_data_and_inc(data, &index, size);
>>> + if (tmp != 'c')
>>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>>> +
>>> + /* Get date length */
>>> + len = xhi_data_and_inc(data, &index, size);
>>> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
>>> +
>>> + /* allocate space for date and final null character. */
>>> + header->date = vmalloc(len);
>>> + if (!header->date)
>>> + return -ENOMEM;
>>> +
>>> + /* Read in date name */
>>> + for (i = 0; i < len; i++)
>>> + header->date[i] = xhi_data_and_inc(data, &index, size);
>>> +
>>> + if (header->date[len - 1] != '\0')
>>> + return -1;
>> generally -EINVAL is more meaningful than -1
>>> +
>>> + /* Read 'd' */
>>> + tmp = xhi_data_and_inc(data, &index, size);
>>> + if (tmp != 'd')
>>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>>> +
>>> + /* Get time length */
>>> + len = xhi_data_and_inc(data, &index, size);
>>> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
>>> +
>>> + /* allocate space for time and final null character. */
>>> + header->time = vmalloc(len);
>>> + if (!header->time)
>>> + return -ENOMEM;
>>> +
>>> + /* Read in time name */
>>> + for (i = 0; i < len; i++)
>>> + header->time[i] = xhi_data_and_inc(data, &index, size);
>>> +
>>> + if (header->time[len - 1] != '\0')
>>> + return -1;
>>> +
>>> + /* Read 'e' */
>>> + tmp = xhi_data_and_inc(data, &index, size);
>>> + if (tmp != 'e')
>>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>>> +
>>> + /* Get byte length of bitstream */
>>> + header->bitstream_length = xhi_data_and_inc(data, &index, size);
>>> + header->bitstream_length = (header->bitstream_length << 8) |
>>> + xhi_data_and_inc(data, &index, size);
>>> + header->bitstream_length = (header->bitstream_length << 8) |
>>> + xhi_data_and_inc(data, &index, size);
>>> + header->bitstream_length = (header->bitstream_length << 8) |
>>> + xhi_data_and_inc(data, &index, size);
>> generally a problem
>>
>> This is confusing, collect the bytes in a temp[] and construct the header->bitstream_length in on statement.
>>
>> This is a case where xhi_data_and_inc return is not checked and if it failed could blow up later.
>>
>>> +
>>> + header->header_length = index;
>> index is not a good variable name if it going to be stored as a length.
>>
>> consider changing it to something like current_length.
>>
>>> +
>>> + return 0;
>>> +}
>>> +EXPORT_SYMBOL_GPL(xrt_xclbin_parse_bitstream_header);
>>> +
>>> +void xrt_xclbin_free_header(struct hw_icap_bit_header *header)
>>> +{
>>> + vfree(header->design_name);
>>> + vfree(header->part_name);
>>> + vfree(header->date);
>>> + vfree(header->time);
>> missing header->version
>>> +}
>>> +EXPORT_SYMBOL_GPL(xrt_xclbin_free_header);
>>> +
>>> +struct xrt_clock_desc {
>>> + char *clock_ep_name;
>>> + u32 clock_xclbin_type;
>>> + char *clkfreq_ep_name;
>>> +} clock_desc[] = {
>>> + {
>>> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL1,
>>> + .clock_xclbin_type = CT_DATA,
>>> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K1,
>>> + },
>>> + {
>>> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL2,
>>> + .clock_xclbin_type = CT_KERNEL,
>>> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K2,
>>> + },
>>> + {
>>> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL3,
>>> + .clock_xclbin_type = CT_SYSTEM,
>>> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_HBM,
>>> + },
>>> +};
>>> +
>>> +const char *xrt_clock_type2epname(enum CLOCK_TYPE type)
>>> +{
>>> + int i;
>>> +
>>> + for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
>>> + if (clock_desc[i].clock_xclbin_type == type)
>>> + return clock_desc[i].clock_ep_name;
>>> + }
>>> + return NULL;
>>> +}
>>> +EXPORT_SYMBOL_GPL(xrt_clock_type2epname);
>> What is clock stuff doing in xclbin ?
>>
>> I think clock needs its own file
>>
>>> +
>>> +static const char *clock_type2clkfreq_name(u32 type)
>>> +{
>>> + int i;
>>> +
>>> + for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
>>> + if (clock_desc[i].clock_xclbin_type == type)
>>> + return clock_desc[i].clkfreq_ep_name;
>>> + }
>>> + return NULL;
>>> +}
>>> +
>>> +static int xrt_xclbin_add_clock_metadata(struct device *dev,
>>> + const struct axlf *xclbin,
>>> + char *dtb)
>>> +{
>>> + int i;
>>> + u16 freq;
>>> + struct clock_freq_topology *clock_topo;
>>> + int rc = xrt_xclbin_get_section(xclbin, CLOCK_FREQ_TOPOLOGY,
>>> + (void **)&clock_topo, NULL);
>>> +
>>> + if (rc)
>>> + return 0;
>> failing is ok ?
>>> +
>>> + for (i = 0; i < clock_topo->m_count; i++) {
>>> + u8 type = clock_topo->m_clock_freq[i].m_type;
>>> + const char *ep_name = xrt_clock_type2epname(type);
>>> + const char *counter_name = clock_type2clkfreq_name(type);
>>> +
>>> + if (!ep_name || !counter_name)
>>> + continue;
>>> +
>>> + freq = cpu_to_be16(clock_topo->m_clock_freq[i].m_freq_Mhz);
>>> + rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_FREQ,
>>> + &freq, sizeof(freq));
>>> + if (rc)
>>> + break;
>>> +
>>> + rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_CNT,
>>> + counter_name, strlen(counter_name) + 1);
>>> + if (rc)
>>> + break;
>> Failing in a loop, why isn't there some cleanup of the *set_prop() calls ?
>>> + }
>>> +
>>> + vfree(clock_topo);
>>> +
>>> + return rc;
>>> +}
>>> +
>>> +int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb)
>>> +{
>>> + char *md = NULL, *newmd = NULL;
>>> + u64 len;
>> *dtb = NULL;
>>> + int rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA,
>>> + (void **)&md, &len);
>>> +
>>> + if (rc)
>>> + goto done;
>>> +
>>> + /* Sanity check the dtb section. */
>>> + if (xrt_md_size(dev, md) > len) {
>>> + rc = -EINVAL;
>>> + goto done;
>>> + }
>>> +
>>> + newmd = xrt_md_dup(dev, md);
>>> + if (!newmd) {
>>> + rc = -EFAULT;
>>> + goto done;
>>> + }
>>> + /* Convert various needed xclbin sections into dtb. */
>>> + rc = xrt_xclbin_add_clock_metadata(dev, xclbin, newmd);
>> newmd is only valid here, but the above error handling jump here. change this to
>>
>> if (!rc)
>>
>> *dtb = newmd
>>
>> else
>>
>> vfree(newmd)
>>
>> done:
>>
>> vfree(md)
>>
>> return rc;
>>
>>> +
>>> +done:
>>> + if (rc == 0)
>>> + *dtb = newmd;
>>> + else
>>> + vfree(newmd);
>>> + vfree(md);
>>> + return rc;
>>> +}
>>> +EXPORT_SYMBOL_GPL(xrt_xclbin_get_metadata);
>>> diff --git a/include/uapi/linux/xrt/xclbin.h b/include/uapi/linux/xrt/xclbin.h
>>> new file mode 100644
>>> index 000000000000..53f140123ef1
>>> --- /dev/null
>>> +++ b/include/uapi/linux/xrt/xclbin.h
>>> @@ -0,0 +1,408 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>>> +/*
>>> + * Xilinx FPGA compiled binary container format
>>> + *
>>> + * Copyright (C) 2015-2021, Xilinx Inc
>>> + */
>>> +
>>> +#ifndef _XCLBIN_H_
>>> +#define _XCLBIN_H_
>>> +
>>> +#ifdef _WIN32
>> WIN32 ?
>>
>> Only 1 other header has this ifdef
>>
>>> + #include <cstdint>
>>> + #include <algorithm>
>> c++ is being assumed for windows
>>> + #include "windows/uuid.h"
>> thank you for not including windows.h ;)
>>> +#else
>>> + #if defined(__KERNEL__)
>>> + #include <linux/types.h>
>>> + #include <linux/uuid.h>
>>> + #include <linux/version.h>
>>> + #elif defined(__cplusplus)
>>> + #include <cstdlib>
>>> + #include <cstdint>
>>> + #include <algorithm>
>>> + #include <uuid/uuid.h>
>>> + #else
>>> + #include <stdlib.h>
>>> + #include <stdint.h>
>>> + #include <uuid/uuid.h>
>>> + #endif
>>> +#endif
> Tbh, not a big fan of this ...
We will remove WIN32 and keep __cplusplus
>>> +
>> Review these includes, some could be convenience includes.
>>
>> ex/ linux/version.h with no obvious use of version macros.
>>
>> struct axlf_header {
>> + uint64_t m_length; /* Total size of the xclbin file */
>> .. snip ..
>> + union {
>> + char m_next_axlf[16]; /* Name of next xclbin file */
>> + /* in the daisy chain */
>> + uuid_t uuid; /* uuid of this xclbin*/
>> + };
>>
>> As mentioned in an earlier patch, if uuid_t is larger than 16 bytes, axlf_header breaks.
>> while it is convenient to have this type here, it would be better this access was handled in another way.
>> Maybe a host specific function.
>>
>> I also do not see a pragma pack, usually this is set of 1 so the compiler does not shuffle elements, increase size etc.
>>
>>> +#ifdef __cplusplus
>>> +extern "C" {
>>> +#endif
>>> +
>>> +/**
>>> + * DOC: Container format for Xilinx FPGA images
>>> + * The container stores bitstreams, metadata and firmware images.
>>> + * xclbin/xsabin is ELF-like binary container format. It is structured
>> is an ELF-like file format. It is a structured
>>> + * series of sections. There is a file header followed by several section
>>> + * headers which is followed by sections. A section header points to an
>>> + * actual section. There is an optional signature at the end. The
>>> + * following figure illustrates a typical xclbin:
>>> + *
>>> + * +---------------------+
>>> + * | |
>>> + * | HEADER |
>>> + * +---------------------+
>>> + * | SECTION HEADER |
>>> + * | |
>>> + * +---------------------+
>>> + * | ... |
>>> + * | |
>>> + * +---------------------+
>>> + * | SECTION HEADER |
>>> + * | |
>>> + * +---------------------+
>>> + * | SECTION |
>>> + * | |
>>> + * +---------------------+
>>> + * | ... |
>>> + * | |
>>> + * +---------------------+
>>> + * | SECTION |
>>> + * | |
>>> + * +---------------------+
>>> + * | SIGNATURE |
>>> + * | (OPTIONAL) |
>>> + * +---------------------+
>> This ascii art is a mixture of tabs and spaces, for someone with tab = 2 spaces, this will look messed up.
>>
>> convert the tabs to spaces
>>
>>> + */
>>> +
>>> +enum XCLBIN_MODE {
>>> + XCLBIN_FLAT,
>> generally
>>
>> all enums used in a file format should be initialized.
>>
>> This likely should be
>>
>> XCLBIN_FLAT = 0,
>>
>>> + XCLBIN_PR,
>>> + XCLBIN_TANDEM_STAGE2,
>>> + XCLBIN_TANDEM_STAGE2_WITH_PR,
>>> + XCLBIN_HW_EMU,
>>> + XCLBIN_SW_EMU,
>>> + XCLBIN_MODE_MAX
>>> +};
>>> +
>>> +enum axlf_section_kind {
>>> + BITSTREAM = 0,
>>> + CLEARING_BITSTREAM,
>>> + EMBEDDED_METADATA,
>>> + FIRMWARE,
>>> + DEBUG_DATA,
>>> + SCHED_FIRMWARE,
>>> + MEM_TOPOLOGY,
>>> + CONNECTIVITY,
>>> + IP_LAYOUT,
>>> + DEBUG_IP_LAYOUT,
>>> + DESIGN_CHECK_POINT,
>>> + CLOCK_FREQ_TOPOLOGY,
>>> + MCS,
>>> + BMC,
>>> + BUILD_METADATA,
>>> + KEYVALUE_METADATA,
>>> + USER_METADATA,
>>> + DNA_CERTIFICATE,
>>> + PDI,
>>> + BITSTREAM_PARTIAL_PDI,
>>> + PARTITION_METADATA,
>>> + EMULATION_DATA,
>>> + SYSTEM_METADATA,
>>> + SOFT_KERNEL,
>>> + ASK_FLASH,
>>> + AIE_METADATA,
>>> + ASK_GROUP_TOPOLOGY,
>>> + ASK_GROUP_CONNECTIVITY
>>> +};
>>> +
>>> +enum MEM_TYPE {
>>> + MEM_DDR3,
>>> + MEM_DDR4,
>>> + MEM_DRAM,
>>> + MEM_STREAMING,
>>> + MEM_PREALLOCATED_GLOB,
>>> + MEM_ARE,
>>> + MEM_HBM,
>>> + MEM_BRAM,
>>> + MEM_URAM,
>>> + MEM_STREAMING_CONNECTION
>>> +};
>>> +
>>> +enum IP_TYPE {
>>> + IP_MB = 0,
>>> + IP_KERNEL,
>>> + IP_DNASC,
>>> + IP_DDR4_CONTROLLER,
>>> + IP_MEM_DDR4,
>>> + IP_MEM_HBM
>>> +};
>>> +
>>> +struct axlf_section_header {
>>> + uint32_t m_sectionKind; /* Section type */
>>> + char m_sectionName[16]; /* Examples: "stage2", "clear1", */
>>> + /* "clear2", "ocl1", "ocl2, */
>>> + /* "ublaze", "sched" */
>>> + uint64_t m_sectionOffset; /* File offset of section data */
>>> + uint64_t m_sectionSize; /* Size of section data */
>>> +};
>>> +
>>> +struct axlf_header {
>>> + uint64_t m_length; /* Total size of the xclbin file */
>>> + uint64_t m_timeStamp; /* Number of seconds since epoch */
>>> + /* when xclbin was created */
>>> + uint64_t m_featureRomTimeStamp; /* TimeSinceEpoch of the featureRom */
>>> + uint16_t m_versionPatch; /* Patch Version */
>>> + uint8_t m_versionMajor; /* Major Version - Version: 2.1.0*/
>> i did not see the version checked earlier, which one is expected ?
>>> + uint8_t m_versionMinor; /* Minor Version */
>>> + uint32_t m_mode; /* XCLBIN_MODE */
>>> + union {
>>> + struct {
>>> + uint64_t m_platformId; /* 64 bit platform ID: */
>>> + /* vendor-device-subvendor-subdev */
>>> + uint64_t m_featureId; /* 64 bit feature id */
>>> + } rom;
>>> + unsigned char rom_uuid[16]; /* feature ROM UUID for which */
>>> + /* this xclbin was generated */
>>> + };
>>> + unsigned char m_platformVBNV[64]; /* e.g. */
>> what is VBNV?
>>> + /* xilinx:xil-accel-rd-ku115:4ddr-xpr:3.4: null terminated */
>>> + union {
>>> + char m_next_axlf[16]; /* Name of next xclbin file */
>>> + /* in the daisy chain */
>>> + uuid_t uuid; /* uuid of this xclbin*/
>>> + };
>>> + char m_debug_bin[16]; /* Name of binary with debug */
>>> + /* information */
>>> + uint32_t m_numSections; /* Number of section headers */
>>> +};
>>> +
>>> +struct axlf {
>>> + char m_magic[8]; /* Should be "xclbin2\0" */
>>> + int32_t m_signature_length; /* Length of the signature. */
>>> + /* -1 indicates no signature */
>>> + unsigned char reserved[28]; /* Note: Initialized to 0xFFs */
>>> +
>>> + unsigned char m_keyBlock[256]; /* Signature for validation */
>>> + /* of binary */
>>> + uint64_t m_uniqueId; /* axlf's uniqueId, use it to */
>>> + /* skip redownload etc */
>>> + struct axlf_header m_header; /* Inline header */
>>> + struct axlf_section_header m_sections[1]; /* One or more section */
>>> + /* headers follow */
>>> +};
>>> +
>>> +/* bitstream information */
>>> +struct xlnx_bitstream {
>>> + uint8_t m_freq[8];
>>> + char bits[1];
>>> +};
>>> +
>>> +/**** MEMORY TOPOLOGY SECTION ****/
>>> +struct mem_data {
>>> + uint8_t m_type; /* enum corresponding to mem_type. */
>>> + uint8_t m_used; /* if 0 this bank is not present */
>>> + union {
>>> + uint64_t m_size; /* if mem_type DDR, then size in KB; */
>>> + uint64_t route_id; /* if streaming then "route_id" */
>>> + };
>>> + union {
>>> + uint64_t m_base_address;/* if DDR then the base address; */
>>> + uint64_t flow_id; /* if streaming then "flow id" */
>>> + };
>>> + unsigned char m_tag[16]; /* DDR: BANK0,1,2,3, has to be null */
>>> + /* terminated; if streaming then stream0, 1 etc */
>>> +};
>>> +
>>> +struct mem_topology {
>>> + int32_t m_count; /* Number of mem_data */
>>> + struct mem_data m_mem_data[1]; /* Should be sorted on mem_type */
>>> +};
>>> +
>>> +/**** CONNECTIVITY SECTION ****/
>>> +/* Connectivity of each argument of Kernel. It will be in terms of argument
>> This section does not make sense.
>>
>> Likely you mean some algorithm kernel, rather than the linux kernel.
>>
>>> + * index associated. For associating kernel instances with arguments and
>>> + * banks, start at the connectivity section. Using the m_ip_layout_index
>>> + * access the ip_data.m_name. Now we can associate this kernel instance
>>> + * with its original kernel name and get the connectivity as well. This
>>> + * enables us to form related groups of kernel instances.
>>> + */
>>> +
>>> +struct connection {
>>> + int32_t arg_index; /* From 0 to n, may not be contiguous as scalars */
>>> + /* skipped */
>>> + int32_t m_ip_layout_index; /* index into the ip_layout section. */
>>> + /* ip_layout.m_ip_data[index].m_type == IP_KERNEL */
>>> + int32_t mem_data_index; /* index of the m_mem_data . Flag error is */
>>> + /* m_used false. */
>>> +};
>>> +
>>> +struct connectivity {
>>> + int32_t m_count;
>>> + struct connection m_connection[1];
>>> +};
>>> +
>>> +/**** IP_LAYOUT SECTION ****/
>>> +
>>> +/* IP Kernel */
>>> +#define IP_INT_ENABLE_MASK 0x0001
>>> +#define IP_INTERRUPT_ID_MASK 0x00FE
>>> +#define IP_INTERRUPT_ID_SHIFT 0x1
>>> +
>>> +enum IP_CONTROL {
>>> + AP_CTRL_HS = 0,
>>> + AP_CTRL_CHAIN = 1,
>>> + AP_CTRL_NONE = 2,
>>> + AP_CTRL_ME = 3,
>>> + ACCEL_ADAPTER = 4
>> assigning beyond the first is not necessary unless there are dups or gaps
>>> +};
>>> +
>>> +#define IP_CONTROL_MASK 0xFF00
>>> +#define IP_CONTROL_SHIFT 0x8
>>> +
>>> +/* IPs on AXI lite - their types, names, and base addresses.*/
>>> +struct ip_data {
>>> + uint32_t m_type; /* map to IP_TYPE enum */
>>> + union {
>>> + uint32_t properties; /* Default: 32-bits to indicate ip */
>>> + /* specific property. */
>>> + /* m_type: IP_KERNEL
>>> + * m_int_enable : Bit - 0x0000_0001;
>>> + * m_interrupt_id : Bits - 0x0000_00FE;
>>> + * m_ip_control : Bits = 0x0000_FF00;
>>> + */
>>> + struct { /* m_type: IP_MEM_* */
>>> + uint16_t m_index;
>>> + uint8_t m_pc_index;
>>> + uint8_t unused;
>>> + } indices;
>>> + };
>>> + uint64_t m_base_address;
>>> + uint8_t m_name[64]; /* eg Kernel name corresponding to KERNEL */
>>> + /* instance, can embed CU name in future. */
>>> +};
>>> +
>>> +struct ip_layout {
>>> + int32_t m_count;
>>> + struct ip_data m_ip_data[1]; /* All the ip_data needs to be sorted */
>>> + /* by m_base_address. */
>> general
>>
>> doing the bla[1] for c++ ?
>>
>> Tom
>>
>>> +};
>>> +
>>> +/*** Debug IP section layout ****/
>>> +enum DEBUG_IP_TYPE {
>>> + UNDEFINED = 0,
>>> + LAPC,
>>> + ILA,
>>> + AXI_MM_MONITOR,
>>> + AXI_TRACE_FUNNEL,
>>> + AXI_MONITOR_FIFO_LITE,
>>> + AXI_MONITOR_FIFO_FULL,
>>> + ACCEL_MONITOR,
>>> + AXI_STREAM_MONITOR,
>>> + AXI_STREAM_PROTOCOL_CHECKER,
>>> + TRACE_S2MM,
>>> + AXI_DMA,
>>> + TRACE_S2MM_FULL
>>> +};
>>> +
>>> +struct debug_ip_data {
>>> + uint8_t m_type; /* type of enum DEBUG_IP_TYPE */
>>> + uint8_t m_index_lowbyte;
>>> + uint8_t m_properties;
>>> + uint8_t m_major;
>>> + uint8_t m_minor;
>>> + uint8_t m_index_highbyte;
>>> + uint8_t m_reserved[2];
>>> + uint64_t m_base_address;
>>> + char m_name[128];
>>> +};
>>> +
>>> +struct debug_ip_layout {
>>> + uint16_t m_count;
>>> + struct debug_ip_data m_debug_ip_data[1];
>>> +};
>>> +
>>> +/* Supported clock frequency types */
>>> +enum CLOCK_TYPE {
>>> + CT_UNUSED = 0, /* Initialized value */
>>> + CT_DATA = 1, /* Data clock */
>>> + CT_KERNEL = 2, /* Kernel clock */
>>> + CT_SYSTEM = 3 /* System Clock */
>>> +};
>>> +
>>> +/* Clock Frequency Entry */
>>> +struct clock_freq {
>>> + uint16_t m_freq_Mhz; /* Frequency in MHz */
>>> + uint8_t m_type; /* Clock type (enum CLOCK_TYPE) */
>>> + uint8_t m_unused[5]; /* Not used - padding */
>>> + char m_name[128]; /* Clock Name */
>>> +};
>>> +
>>> +/* Clock frequency section */
>>> +struct clock_freq_topology {
>>> + int16_t m_count; /* Number of entries */
>>> + struct clock_freq m_clock_freq[1]; /* Clock array */
>>> +};
>>> +
>>> +/* Supported MCS file types */
>>> +enum MCS_TYPE {
>>> + MCS_UNKNOWN = 0, /* Initialized value */
>>> + MCS_PRIMARY = 1, /* The primary mcs file data */
>>> + MCS_SECONDARY = 2, /* The secondary mcs file data */
>>> +};
>>> +
>>> +/* One chunk of MCS data */
>>> +struct mcs_chunk {
>>> + uint8_t m_type; /* MCS data type */
> just call them type, unused, offset. Drop the m_*
Will remove all 'm_'
Thanks,
Lizhi
>>> + uint8_t m_unused[7]; /* padding */
>>> + uint64_t m_offset; /* data offset from the start of */
>>> + /* the section */
>>> + uint64_t m_size; /* data size */
>>> +};
>>> +
>>> +/* MCS data section */
>>> +struct mcs {
>>> + int8_t m_count; /* Number of chunks */
>>> + int8_t m_unused[7]; /* padding */
>>> + struct mcs_chunk m_chunk[1]; /* MCS chunks followed by data */
>>> +};
>>> +
>>> +/* bmc data section */
>>> +struct bmc {
>>> + uint64_t m_offset; /* data offset from the start of */
>>> + /* the section */
>>> + uint64_t m_size; /* data size (bytes) */
>>> + char m_image_name[64]; /* Name of the image */
>>> + /* (e.g., MSP432P401R) */
>>> + char m_device_name[64]; /* Device ID (e.g., VCU1525) */
>>> + char m_version[64];
>>> + char m_md5value[33]; /* MD5 Expected Value */
>>> + /* (e.g., 56027182079c0bd621761b7dab5a27ca)*/
>>> + char m_padding[7]; /* Padding */
>>> +};
>>> +
>>> +/* soft kernel data section, used by classic driver */
>>> +struct soft_kernel {
>>> + /** Prefix Syntax:
>>> + * mpo - member, pointer, offset
>>> + * This variable represents a zero terminated string
>>> + * that is offseted from the beginning of the section.
>>> + * The pointer to access the string is initialized as follows:
>>> + * char * pCharString = (address_of_section) + (mpo value)
>>> + */
>>> + uint32_t mpo_name; /* Name of the soft kernel */
>>> + uint32_t m_image_offset; /* Image offset */
>>> + uint32_t m_image_size; /* Image size */
>>> + uint32_t mpo_version; /* Version */
>>> + uint32_t mpo_md5_value; /* MD5 checksum */
>>> + uint32_t mpo_symbol_name; /* Symbol name */
>>> + uint32_t m_num_instances; /* Number of instances */
>>> + uint8_t padding[36]; /* Reserved for future use */
>>> + uint8_t reservedExt[16]; /* Reserved for future extended data */
>>> +};
>>> +
>>> +enum CHECKSUM_TYPE {
>>> + CST_UNKNOWN = 0,
>>> + CST_SDBM = 1,
>>> + CST_LAST
>>> +};
>>> +
>>> +#ifdef __cplusplus
>>> +}
>>> +#endif
>>> +
>>> +#endif
> I'll take a closer look, these were just random things I bumped into.
>
> - Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-02-21 17:12 ` Tom Rix
2021-02-21 18:33 ` Moritz Fischer
@ 2021-02-26 21:23 ` Lizhi Hou
2021-02-28 16:54 ` Tom Rix
1 sibling, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-26 21:23 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 02/21/2021 09:12 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Alveo FPGA firmware and partial reconfigure file are in xclbin format.
> This code enumerates and extracts
Will change this to
Alveo FPGA firmware and partial reconfigure file are in xclbin format. This
code enumerates and extracts sections from xclbin files. xclbin.h is cross
platform and used across all platforms and OS.
>> Add
>> code to enumerate and extract sections from xclbin files. xclbin.h is cross
>> platform and used across all platforms and OS
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xclbin-helper.h | 52 +++
>> drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++++++++++++++
>> include/uapi/linux/xrt/xclbin.h | 408 +++++++++++++++++++++++
>> 3 files changed, 854 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>> create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>> create mode 100644 include/uapi/linux/xrt/xclbin.h
>>
>> diff --git a/drivers/fpga/xrt/include/xclbin-helper.h b/drivers/fpga/xrt/include/xclbin-helper.h
>> new file mode 100644
>> index 000000000000..68218efc9d0b
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xclbin-helper.h
>> @@ -0,0 +1,52 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for Xilinx Runtime (XRT) driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * David Zhang <davidzha@xilinx.com>
>> + * Sonal Santan <sonal.santan@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_XCLBIN_H
>> +#define _XRT_XCLBIN_H
> The header guard should match the filename.
will fix this.
>
>> +
>> +#include <linux/types.h>
>> +#include <linux/device.h>
>> +#include <linux/xrt/xclbin.h>
>> +
>> +#define ICAP_XCLBIN_V2 "xclbin2"
>> +#define DMA_HWICAP_BITFILE_BUFFER_SIZE 1024
>> +#define MAX_XCLBIN_SIZE (1024 * 1024 * 1024) /* Assuming xclbin <= 1G, always */
> #defines should have a prefix, maybe XRT_ or XCLBIN_
Will add prefix XCLBIN_
>> +
>> +enum axlf_section_kind;
>> +struct axlf;
>> +
>> +/**
>> + * Bitstream header information as defined by Xilinx tools.
>> + * Please note that this struct definition is not owned by the driver.
>> + */
>> +struct hw_icap_bit_header {
> File headers usually have fixed length fields like uint32_t
>
> Is this a structure the real header is converted into ?
This is not real header. This structure saves the information extracted
from bitstream header.
>
>> + unsigned int header_length; /* Length of header in 32 bit words */
>> + unsigned int bitstream_length; /* Length of bitstream to read in bytes*/
>> + unsigned char *design_name; /* Design name get from bitstream */
>> + unsigned char *part_name; /* Part name read from bitstream */
>> + unsigned char *date; /* Date read from bitstream header */
>> + unsigned char *time; /* Bitstream creation time */
>> + unsigned int magic_length; /* Length of the magic numbers */
>> + unsigned char *version; /* Version string */
>> +};
>> +
>> +const char *xrt_xclbin_kind_to_string(enum axlf_section_kind kind);
> Only add decl's that are using in multiple files.
>
> This is only defined in xclbin.c, why does it need to be in the header ?
Will remove this.
>
>> +int xrt_xclbin_get_section(const struct axlf *xclbin,
>> + enum axlf_section_kind kind, void **data,
>> + uint64_t *len);
>> +int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb);
>> +int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
>> + unsigned int size,
>> + struct hw_icap_bit_header *header);
>> +void xrt_xclbin_free_header(struct hw_icap_bit_header *header);
>> +const char *xrt_clock_type2epname(enum CLOCK_TYPE type);
> CLOCK_TYPE needs a prefix, something like XCLBIN_CLOCK_TYPE
Will change to XCLBIN_CLOCK_TYPE.
>> +
>> +#endif /* _XRT_XCLBIN_H */
>> diff --git a/drivers/fpga/xrt/lib/xclbin.c b/drivers/fpga/xrt/lib/xclbin.c
>> new file mode 100644
>> index 000000000000..47dc6ca25c1b
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xclbin.c
>> @@ -0,0 +1,394 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA Driver XCLBIN parser
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors: David Zhang <davidzha@xilinx.com>
>> + */
>> +
>> +#include <asm/errno.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/device.h>
>> +#include "xclbin-helper.h"
>> +#include "metadata.h"
>> +
> What is XHI ? Maybe expand this, at the lease should comment
Will use BITSTREAM_ instead.
>> +/* Used for parsing bitstream header */
>> +#define XHI_EVEN_MAGIC_BYTE 0x0f
>> +#define XHI_ODD_MAGIC_BYTE 0xf0
>> +
>> +/* Extra mode for IDLE */
>> +#define XHI_OP_IDLE -1
>> +#define XHI_BIT_HEADER_FAILURE -1
>> +
>> +/* The imaginary module length register */
>> +#define XHI_MLR 15
>> +
>> +static inline unsigned char xhi_data_and_inc(const unsigned char *d, int *i, int sz)
> could move to the *.h
I will restructure caller function xrt_xclbin_parse_bitstream_header()
and remove xhi_data_and_inc().
>> +{_
>> + unsigned char data;
>> +
>> + if (*i >= sz)
>> + return -1;
> The return value of this funtion is not always checked, at the least add a dev_err here
>> +
>> + data = d[*i];
>> + (*i)++;
>> +
>> + return data;
>> +}
>> +
>> +static const struct axlf_section_header *
>> +xrt_xclbin_get_section_hdr(const struct axlf *xclbin,
>> + enum axlf_section_kind kind)
>> +{
>> + int i = 0;
>> +
>> + for (i = 0; i < xclbin->m_header.m_numSections; i++) {
>> + if (xclbin->m_sections[i].m_sectionKind == kind)
>> + return &xclbin->m_sections[i];
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static int
>> +xrt_xclbin_check_section_hdr(const struct axlf_section_header *header,
>> + u64 xclbin_len)
>> +{
>> + int ret;
>> +
>> + ret = (header->m_sectionOffset + header->m_sectionSize) > xclbin_len ? -EINVAL : 0;
> Tristate is harder to read, consider replacing with if()
>
> int ret = 0
>
> if ()
>
> ret =
Will change to if statement.
>
>> +
>> + return ret;
>> +}
>> +
>> +static int xrt_xclbin_section_info(const struct axlf *xclbin,
>> + enum axlf_section_kind kind,
>> + u64 *offset, u64 *size)
>> +{
>> + const struct axlf_section_header *mem_header = NULL;
>> + u64 xclbin_len;
>> + int err = 0;
>> +
>> + mem_header = xrt_xclbin_get_section_hdr(xclbin, kind);
>> + if (!mem_header)
>> + return -EINVAL;
>> +
>> + xclbin_len = xclbin->m_header.m_length;
>> + if (xclbin_len > MAX_XCLBIN_SIZE)
>> + return -EINVAL;
> This check can be added to the function call..
>
> or the sanity checking added to the earier call to *get_section_hdr
>
> There a number of small functions that can be combined.
Will change this and combine the small functions.
>
>> +
>> + err = xrt_xclbin_check_section_hdr(mem_header, xclbin_len);
>> + if (err)
>> + return err;
>> +
>> + *offset = mem_header->m_sectionOffset;
>> + *size = mem_header->m_sectionSize;
>> +
>> + return 0;
>> +}
>> +
>> +/* caller should free the allocated memory for **data */
> must free
>
> This comment also needs to be with the *.h decl
Will change this and add the same comment to .h
>
>> +int xrt_xclbin_get_section(const struct axlf *buf,
>> + enum axlf_section_kind kind,
>> + void **data, u64 *len)
>> +{
>> + const struct axlf *xclbin = (const struct axlf *)buf;
>> + void *section = NULL;
>> + int err = 0;
>> + u64 offset = 0;
>> + u64 size = 0;
>> +
>> + err = xrt_xclbin_section_info(xclbin, kind, &offset, &size);
>> + if (err)
>> + return err;
>> +
>> + section = vmalloc(size);
>> + if (!section)
>> + return -ENOMEM;
>> +
>> + memcpy(section, ((const char *)xclbin) + offset, size);
>> +
>> + *data = section;
> a general comment
>
> for exported function checking the validity of the inputs in more important.
>
> here you assume **data is valid, really you should check.
Will add check for exported functions.
>
>> + if (len)
>> + *len = size;
> len setting being optional, needs to be in the *.h comment
Will add comment.
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_xclbin_get_section);
>> +
> Instead of allocating new memory and making copies of bits of *data
>
> why not have the points reference data ?
Agree. I will change to use this points reference data.
>
> The size operations look like translating big endian data to little endian.
>
> This will break on a big endian host.
It does not translate to little endian. It converts to host endian. We
can not use existing function because of the alignment.
>
>> +/* parse bitstream header */
>> +int xrt_xclbin_parse_bitstream_header(const unsigned char *data,
>> + unsigned int size,
>> + struct hw_icap_bit_header *header)
>> +{
>> + unsigned int index;
>> + unsigned int len;
>> + unsigned int tmp;
>> + unsigned int i;
>> +
>> + memset(header, 0, sizeof(*header));
>> + /* Start Index at start of bitstream */
>> + index = 0;
>> +
>> + /* Initialize HeaderLength. If header returned early inidicates
>> + * failure.
> This side effect should be documented in the *.h comment.
>
> Also the multi line comment is a bit weird, not sure if it is ok
Will remove this. The function will return -EINVAL for failure cases.
>
>> + */
>> + header->header_length = XHI_BIT_HEADER_FAILURE;
>> +
>> + /* Get "Magic" length */
>> + header->magic_length = xhi_data_and_inc(data, &index, size);
>> + header->magic_length = (header->magic_length << 8) | xhi_data_and_inc(data, &index, size);
>> +
>> + /* Read in "magic" */
>> + for (i = 0; i < header->magic_length - 1; i++) {
>> + tmp = xhi_data_and_inc(data, &index, size);
>> + if (i % 2 == 0 && tmp != XHI_EVEN_MAGIC_BYTE)
>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>> +
>> + if (i % 2 == 1 && tmp != XHI_ODD_MAGIC_BYTE)
>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>> + }
>> +
>> + /* Read null end of magic data. */
>> + tmp = xhi_data_and_inc(data, &index, size);
>> +
>> + /* Read 0x01 (short) */
>> + tmp = xhi_data_and_inc(data, &index, size);
>> + tmp = (tmp << 8) | xhi_data_and_inc(data, &index, size);
>> +
>> + /* Check the "0x01" half word */
>> + if (tmp != 0x01)
>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>> +
>> + /* Read 'a' */
>> + tmp = xhi_data_and_inc(data, &index, size);
>> + if (tmp != 'a')
>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>> +
>> + /* Get Design Name length */
>> + len = xhi_data_and_inc(data, &index, size);
>> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
>> +
>> + /* allocate space for design name and final null character. */
>> + header->design_name = vmalloc(len);
>> + if (!header->design_name)
>> + return -ENOMEM;
>> +
>> + /* Read in Design Name */
>> + for (i = 0; i < len; i++)
>> + header->design_name[i] = xhi_data_and_inc(data, &index, size);
>> +
>> + if (header->design_name[len - 1] != '\0')
>> + return -1;
>> +
>> + header->version = strstr(header->design_name, "Version=") + strlen("Version=");
>> +
>> + /* Read 'b' */
>> + tmp = xhi_data_and_inc(data, &index, size);
>> + if (tmp != 'b')
>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>> +
>> + /* Get Part Name length */
>> + len = xhi_data_and_inc(data, &index, size);
>> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
>> +
>> + /* allocate space for part name and final null character. */
>> + header->part_name = vmalloc(len);
>> + if (!header->part_name)
>> + return -ENOMEM;
>> +
>> + /* Read in part name */
>> + for (i = 0; i < len; i++)
>> + header->part_name[i] = xhi_data_and_inc(data, &index, size);
>> +
>> + if (header->part_name[len - 1] != '\0')
>> + return -1;
>> +
>> + /* Read 'c' */
>> + tmp = xhi_data_and_inc(data, &index, size);
>> + if (tmp != 'c')
>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>> +
>> + /* Get date length */
>> + len = xhi_data_and_inc(data, &index, size);
>> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
>> +
>> + /* allocate space for date and final null character. */
>> + header->date = vmalloc(len);
>> + if (!header->date)
>> + return -ENOMEM;
>> +
>> + /* Read in date name */
>> + for (i = 0; i < len; i++)
>> + header->date[i] = xhi_data_and_inc(data, &index, size);
>> +
>> + if (header->date[len - 1] != '\0')
>> + return -1;
> generally -EINVAL is more meaningful than -1
Will fix this.
>> +
>> + /* Read 'd' */
>> + tmp = xhi_data_and_inc(data, &index, size);
>> + if (tmp != 'd')
>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>> +
>> + /* Get time length */
>> + len = xhi_data_and_inc(data, &index, size);
>> + len = (len << 8) | xhi_data_and_inc(data, &index, size);
>> +
>> + /* allocate space for time and final null character. */
>> + header->time = vmalloc(len);
>> + if (!header->time)
>> + return -ENOMEM;
>> +
>> + /* Read in time name */
>> + for (i = 0; i < len; i++)
>> + header->time[i] = xhi_data_and_inc(data, &index, size);
>> +
>> + if (header->time[len - 1] != '\0')
>> + return -1;
>> +
>> + /* Read 'e' */
>> + tmp = xhi_data_and_inc(data, &index, size);
>> + if (tmp != 'e')
>> + return -1; /* INVALID_FILE_HEADER_ERROR */
>> +
>> + /* Get byte length of bitstream */
>> + header->bitstream_length = xhi_data_and_inc(data, &index, size);
>> + header->bitstream_length = (header->bitstream_length << 8) |
>> + xhi_data_and_inc(data, &index, size);
>> + header->bitstream_length = (header->bitstream_length << 8) |
>> + xhi_data_and_inc(data, &index, size);
>> + header->bitstream_length = (header->bitstream_length << 8) |
>> + xhi_data_and_inc(data, &index, size);
> generally a problem
>
> This is confusing, collect the bytes in a temp[] and construct the header->bitstream_length in on statement.
>
> This is a case where xhi_data_and_inc return is not checked and if it failed could blow up later.
Will restructure this function and check the possible overflow.
>
>> +
>> + header->header_length = index;
> index is not a good variable name if it going to be stored as a length.
>
> consider changing it to something like current_length.
Will change it to offset.
>
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_xclbin_parse_bitstream_header);
>> +
>> +void xrt_xclbin_free_header(struct hw_icap_bit_header *header)
>> +{
>> + vfree(header->design_name);
>> + vfree(header->part_name);
>> + vfree(header->date);
>> + vfree(header->time);
> missing header->version
Will remove xrt_xclbin_free_header(). All the pointers will point to
reference data instead of allocation.
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_xclbin_free_header);
>> +
>> +struct xrt_clock_desc {
>> + char *clock_ep_name;
>> + u32 clock_xclbin_type;
>> + char *clkfreq_ep_name;
>> +} clock_desc[] = {
>> + {
>> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL1,
>> + .clock_xclbin_type = CT_DATA,
>> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K1,
>> + },
>> + {
>> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL2,
>> + .clock_xclbin_type = CT_KERNEL,
>> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_K2,
>> + },
>> + {
>> + .clock_ep_name = XRT_MD_NODE_CLK_KERNEL3,
>> + .clock_xclbin_type = CT_SYSTEM,
>> + .clkfreq_ep_name = XRT_MD_NODE_CLKFREQ_HBM,
>> + },
>> +};
>> +
>> +const char *xrt_clock_type2epname(enum CLOCK_TYPE type)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
>> + if (clock_desc[i].clock_xclbin_type == type)
>> + return clock_desc[i].clock_ep_name;
>> + }
>> + return NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_clock_type2epname);
> What is clock stuff doing in xclbin ?
>
> I think clock needs its own file
xclbin file contains a section which describes the clock settings to
running this xclbin on hardware. The _clock_ functions convert clock
section to XRT driver metadata which is in device tree format.
>
>> +
>> +static const char *clock_type2clkfreq_name(u32 type)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < ARRAY_SIZE(clock_desc); i++) {
>> + if (clock_desc[i].clock_xclbin_type == type)
>> + return clock_desc[i].clkfreq_ep_name;
>> + }
>> + return NULL;
>> +}
>> +
>> +static int xrt_xclbin_add_clock_metadata(struct device *dev,
>> + const struct axlf *xclbin,
>> + char *dtb)
>> +{
>> + int i;
>> + u16 freq;
>> + struct clock_freq_topology *clock_topo;
>> + int rc = xrt_xclbin_get_section(xclbin, CLOCK_FREQ_TOPOLOGY,
>> + (void **)&clock_topo, NULL);
>> +
>> + if (rc)
>> + return 0;
> failing is ok ?
If clock section does not exist in xclbin, that means the xclbin uses
default clock. And this is a valid case. I will add comment
/* if clock section does not exist, add nothing and return success */
>> +
>> + for (i = 0; i < clock_topo->m_count; i++) {
>> + u8 type = clock_topo->m_clock_freq[i].m_type;
>> + const char *ep_name = xrt_clock_type2epname(type);
>> + const char *counter_name = clock_type2clkfreq_name(type);
>> +
>> + if (!ep_name || !counter_name)
>> + continue;
>> +
>> + freq = cpu_to_be16(clock_topo->m_clock_freq[i].m_freq_Mhz);
>> + rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_FREQ,
>> + &freq, sizeof(freq));
>> + if (rc)
>> + break;
>> +
>> + rc = xrt_md_set_prop(dev, dtb, ep_name, NULL, XRT_MD_PROP_CLK_CNT,
>> + counter_name, strlen(counter_name) + 1);
>> + if (rc)
>> + break;
> Failing in a loop, why isn't there some cleanup of the *set_prop() calls ?
The entire blob will be freed outside if this function return failure.
>> + }
>> +
>> + vfree(clock_topo);
>> +
>> + return rc;
>> +}
>> +
>> +int xrt_xclbin_get_metadata(struct device *dev, const struct axlf *xclbin, char **dtb)
>> +{
>> + char *md = NULL, *newmd = NULL;
>> + u64 len;
> *dtb = NULL;
Will fix this.
>> + int rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA,
>> + (void **)&md, &len);
>> +
>> + if (rc)
>> + goto done;
>> +
>> + /* Sanity check the dtb section. */
>> + if (xrt_md_size(dev, md) > len) {
>> + rc = -EINVAL;
>> + goto done;
>> + }
>> +
>> + newmd = xrt_md_dup(dev, md);
>> + if (!newmd) {
>> + rc = -EFAULT;
>> + goto done;
>> + }
>> + /* Convert various needed xclbin sections into dtb. */
>> + rc = xrt_xclbin_add_clock_metadata(dev, xclbin, newmd);
> newmd is only valid here, but the above error handling jump here. change this to
>
> if (!rc)
>
> *dtb = newmd
>
> else
>
> vfree(newmd)
>
> done:
>
> vfree(md)
>
> return rc;
Will change it.
>> +
>> +done:
>> + if (rc == 0)
>> + *dtb = newmd;
>> + else
>> + vfree(newmd);
>> + vfree(md);
>> + return rc;
>> +}
>> +EXPORT_SYMBOL_GPL(xrt_xclbin_get_metadata);
>> diff --git a/include/uapi/linux/xrt/xclbin.h b/include/uapi/linux/xrt/xclbin.h
>> new file mode 100644
>> index 000000000000..53f140123ef1
>> --- /dev/null
>> +++ b/include/uapi/linux/xrt/xclbin.h
>> @@ -0,0 +1,408 @@
>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>> +/*
>> + * Xilinx FPGA compiled binary container format
>> + *
>> + * Copyright (C) 2015-2021, Xilinx Inc
>> + */
>> +
>> +#ifndef _XCLBIN_H_
>> +#define _XCLBIN_H_
>> +
>> +#ifdef _WIN32
> WIN32 ?
>
> Only 1 other header has this ifdef
>
>> + #include <cstdint>
>> + #include <algorithm>
> c++ is being assumed for windows
>> + #include "windows/uuid.h"
> thank you for not including windows.h ;)
>> +#else
>> + #if defined(__KERNEL__)
>> + #include <linux/types.h>
>> + #include <linux/uuid.h>
>> + #include <linux/version.h>
>> + #elif defined(__cplusplus)
>> + #include <cstdlib>
>> + #include <cstdint>
>> + #include <algorithm>
>> + #include <uuid/uuid.h>
>> + #else
>> + #include <stdlib.h>
>> + #include <stdint.h>
>> + #include <uuid/uuid.h>
>> + #endif
>> +#endif
>> +
> Review these includes, some could be convenience includes.
removed uuid.h and version.h.
>
> ex/ linux/version.h with no obvious use of version macros.
>
> struct axlf_header {
> + uint64_t m_length; /* Total size of the xclbin file */
> .. snip ..
> + union {
> + char m_next_axlf[16]; /* Name of next xclbin file */
> + /* in the daisy chain */
> + uuid_t uuid; /* uuid of this xclbin*/
> + };
>
> As mentioned in an earlier patch, if uuid_t is larger than 16 bytes, axlf_header breaks.
> while it is convenient to have this type here, it would be better this access was handled in another way.
> Maybe a host specific function.
Agree. will change to "unsigned char uuid[16]".
>
> I also do not see a pragma pack, usually this is set of 1 so the compiler does not shuffle elements, increase size etc.
This data structure is shared with other tools. And the structure is
well defined with reasonable alignment. It is compatible with all
compilers we have tested. So pragma pack is not necessary.
>
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +/**
>> + * DOC: Container format for Xilinx FPGA images
>> + * The container stores bitstreams, metadata and firmware images.
>> + * xclbin/xsabin is ELF-like binary container format. It is structured
> is an ELF-like file format. It is a structured
Will fix this.
>> + * series of sections. There is a file header followed by several section
>> + * headers which is followed by sections. A section header points to an
>> + * actual section. There is an optional signature at the end. The
>> + * following figure illustrates a typical xclbin:
>> + *
>> + * +---------------------+
>> + * | |
>> + * | HEADER |
>> + * +---------------------+
>> + * | SECTION HEADER |
>> + * | |
>> + * +---------------------+
>> + * | ... |
>> + * | |
>> + * +---------------------+
>> + * | SECTION HEADER |
>> + * | |
>> + * +---------------------+
>> + * | SECTION |
>> + * | |
>> + * +---------------------+
>> + * | ... |
>> + * | |
>> + * +---------------------+
>> + * | SECTION |
>> + * | |
>> + * +---------------------+
>> + * | SIGNATURE |
>> + * | (OPTIONAL) |
>> + * +---------------------+
> This ascii art is a mixture of tabs and spaces, for someone with tab = 2 spaces, this will look messed up.
>
> convert the tabs to spaces
Will fix this.
>
>> + */
>> +
>> +enum XCLBIN_MODE {
>> + XCLBIN_FLAT,
> generally
>
> all enums used in a file format should be initialized.
>
> This likely should be
>
> XCLBIN_FLAT = 0,
Will fix this.
>
>> + XCLBIN_PR,
>> + XCLBIN_TANDEM_STAGE2,
>> + XCLBIN_TANDEM_STAGE2_WITH_PR,
>> + XCLBIN_HW_EMU,
>> + XCLBIN_SW_EMU,
>> + XCLBIN_MODE_MAX
>> +};
>> +
>> +enum axlf_section_kind {
>> + BITSTREAM = 0,
>> + CLEARING_BITSTREAM,
>> + EMBEDDED_METADATA,
>> + FIRMWARE,
>> + DEBUG_DATA,
>> + SCHED_FIRMWARE,
>> + MEM_TOPOLOGY,
>> + CONNECTIVITY,
>> + IP_LAYOUT,
>> + DEBUG_IP_LAYOUT,
>> + DESIGN_CHECK_POINT,
>> + CLOCK_FREQ_TOPOLOGY,
>> + MCS,
>> + BMC,
>> + BUILD_METADATA,
>> + KEYVALUE_METADATA,
>> + USER_METADATA,
>> + DNA_CERTIFICATE,
>> + PDI,
>> + BITSTREAM_PARTIAL_PDI,
>> + PARTITION_METADATA,
>> + EMULATION_DATA,
>> + SYSTEM_METADATA,
>> + SOFT_KERNEL,
>> + ASK_FLASH,
>> + AIE_METADATA,
>> + ASK_GROUP_TOPOLOGY,
>> + ASK_GROUP_CONNECTIVITY
>> +};
>> +
>> +enum MEM_TYPE {
>> + MEM_DDR3,
>> + MEM_DDR4,
>> + MEM_DRAM,
>> + MEM_STREAMING,
>> + MEM_PREALLOCATED_GLOB,
>> + MEM_ARE,
>> + MEM_HBM,
>> + MEM_BRAM,
>> + MEM_URAM,
>> + MEM_STREAMING_CONNECTION
>> +};
>> +
>> +enum IP_TYPE {
>> + IP_MB = 0,
>> + IP_KERNEL,
>> + IP_DNASC,
>> + IP_DDR4_CONTROLLER,
>> + IP_MEM_DDR4,
>> + IP_MEM_HBM
>> +};
>> +
>> +struct axlf_section_header {
>> + uint32_t m_sectionKind; /* Section type */
>> + char m_sectionName[16]; /* Examples: "stage2", "clear1", */
>> + /* "clear2", "ocl1", "ocl2, */
>> + /* "ublaze", "sched" */
>> + uint64_t m_sectionOffset; /* File offset of section data */
>> + uint64_t m_sectionSize; /* Size of section data */
>> +};
>> +
>> +struct axlf_header {
>> + uint64_t m_length; /* Total size of the xclbin file */
>> + uint64_t m_timeStamp; /* Number of seconds since epoch */
>> + /* when xclbin was created */
>> + uint64_t m_featureRomTimeStamp; /* TimeSinceEpoch of the featureRom */
>> + uint16_t m_versionPatch; /* Patch Version */
>> + uint8_t m_versionMajor; /* Major Version - Version: 2.1.0*/
> i did not see the version checked earlier, which one is expected ?
Will check Major.
>> + uint8_t m_versionMinor; /* Minor Version */
>> + uint32_t m_mode; /* XCLBIN_MODE */
>> + union {
>> + struct {
>> + uint64_t m_platformId; /* 64 bit platform ID: */
>> + /* vendor-device-subvendor-subdev */
>> + uint64_t m_featureId; /* 64 bit feature id */
>> + } rom;
>> + unsigned char rom_uuid[16]; /* feature ROM UUID for which */
>> + /* this xclbin was generated */
>> + };
>> + unsigned char m_platformVBNV[64]; /* e.g. */
> what is VBNV?
VBNV stands for Vendor, BoardID, Name, Version. It is a string like
<Vendor>:<BoardID>:<Name>:<Version> or <Vendor>_<BoardID>_<Name>_<Version>
>> + /* xilinx:xil-accel-rd-ku115:4ddr-xpr:3.4: null terminated */
>> + union {
>> + char m_next_axlf[16]; /* Name of next xclbin file */
>> + /* in the daisy chain */
>> + uuid_t uuid; /* uuid of this xclbin*/
>> + };
>> + char m_debug_bin[16]; /* Name of binary with debug */
>> + /* information */
>> + uint32_t m_numSections; /* Number of section headers */
>> +};
>> +
>> +struct axlf {
>> + char m_magic[8]; /* Should be "xclbin2\0" */
>> + int32_t m_signature_length; /* Length of the signature. */
>> + /* -1 indicates no signature */
>> + unsigned char reserved[28]; /* Note: Initialized to 0xFFs */
>> +
>> + unsigned char m_keyBlock[256]; /* Signature for validation */
>> + /* of binary */
>> + uint64_t m_uniqueId; /* axlf's uniqueId, use it to */
>> + /* skip redownload etc */
>> + struct axlf_header m_header; /* Inline header */
>> + struct axlf_section_header m_sections[1]; /* One or more section */
>> + /* headers follow */
>> +};
>> +
>> +/* bitstream information */
>> +struct xlnx_bitstream {
>> + uint8_t m_freq[8];
>> + char bits[1];
>> +};
>> +
>> +/**** MEMORY TOPOLOGY SECTION ****/
>> +struct mem_data {
>> + uint8_t m_type; /* enum corresponding to mem_type. */
>> + uint8_t m_used; /* if 0 this bank is not present */
>> + union {
>> + uint64_t m_size; /* if mem_type DDR, then size in KB; */
>> + uint64_t route_id; /* if streaming then "route_id" */
>> + };
>> + union {
>> + uint64_t m_base_address;/* if DDR then the base address; */
>> + uint64_t flow_id; /* if streaming then "flow id" */
>> + };
>> + unsigned char m_tag[16]; /* DDR: BANK0,1,2,3, has to be null */
>> + /* terminated; if streaming then stream0, 1 etc */
>> +};
>> +
>> +struct mem_topology {
>> + int32_t m_count; /* Number of mem_data */
>> + struct mem_data m_mem_data[1]; /* Should be sorted on mem_type */
>> +};
>> +
>> +/**** CONNECTIVITY SECTION ****/
>> +/* Connectivity of each argument of Kernel. It will be in terms of argument
> This section does not make sense.
>
> Likely you mean some algorithm kernel, rather than the linux kernel.
The kernel here means hardware IP kernel. I will change the term to CU
(Compute Unit).
>
>> + * index associated. For associating kernel instances with arguments and
>> + * banks, start at the connectivity section. Using the m_ip_layout_index
>> + * access the ip_data.m_name. Now we can associate this kernel instance
>> + * with its original kernel name and get the connectivity as well. This
>> + * enables us to form related groups of kernel instances.
>> + */
>> +
>> +struct connection {
>> + int32_t arg_index; /* From 0 to n, may not be contiguous as scalars */
>> + /* skipped */
>> + int32_t m_ip_layout_index; /* index into the ip_layout section. */
>> + /* ip_layout.m_ip_data[index].m_type == IP_KERNEL */
>> + int32_t mem_data_index; /* index of the m_mem_data . Flag error is */
>> + /* m_used false. */
>> +};
>> +
>> +struct connectivity {
>> + int32_t m_count;
>> + struct connection m_connection[1];
>> +};
>> +
>> +/**** IP_LAYOUT SECTION ****/
>> +
>> +/* IP Kernel */
>> +#define IP_INT_ENABLE_MASK 0x0001
>> +#define IP_INTERRUPT_ID_MASK 0x00FE
>> +#define IP_INTERRUPT_ID_SHIFT 0x1
>> +
>> +enum IP_CONTROL {
>> + AP_CTRL_HS = 0,
>> + AP_CTRL_CHAIN = 1,
>> + AP_CTRL_NONE = 2,
>> + AP_CTRL_ME = 3,
>> + ACCEL_ADAPTER = 4
> assigning beyond the first is not necessary unless there are dups or gaps
Will fix this.
>> +};
>> +
>> +#define IP_CONTROL_MASK 0xFF00
>> +#define IP_CONTROL_SHIFT 0x8
>> +
>> +/* IPs on AXI lite - their types, names, and base addresses.*/
>> +struct ip_data {
>> + uint32_t m_type; /* map to IP_TYPE enum */
>> + union {
>> + uint32_t properties; /* Default: 32-bits to indicate ip */
>> + /* specific property. */
>> + /* m_type: IP_KERNEL
>> + * m_int_enable : Bit - 0x0000_0001;
>> + * m_interrupt_id : Bits - 0x0000_00FE;
>> + * m_ip_control : Bits = 0x0000_FF00;
>> + */
>> + struct { /* m_type: IP_MEM_* */
>> + uint16_t m_index;
>> + uint8_t m_pc_index;
>> + uint8_t unused;
>> + } indices;
>> + };
>> + uint64_t m_base_address;
>> + uint8_t m_name[64]; /* eg Kernel name corresponding to KERNEL */
>> + /* instance, can embed CU name in future. */
>> +};
>> +
>> +struct ip_layout {
>> + int32_t m_count;
>> + struct ip_data m_ip_data[1]; /* All the ip_data needs to be sorted */
>> + /* by m_base_address. */
> general
>
> doing the bla[1] for c++ ?
This file is cross platform and having an array with size 0 chokes
certain compilers.
Thanks,
Lizhi
>
> Tom
>
>> +};
>> +
>> +/*** Debug IP section layout ****/
>> +enum DEBUG_IP_TYPE {
>> + UNDEFINED = 0,
>> + LAPC,
>> + ILA,
>> + AXI_MM_MONITOR,
>> + AXI_TRACE_FUNNEL,
>> + AXI_MONITOR_FIFO_LITE,
>> + AXI_MONITOR_FIFO_FULL,
>> + ACCEL_MONITOR,
>> + AXI_STREAM_MONITOR,
>> + AXI_STREAM_PROTOCOL_CHECKER,
>> + TRACE_S2MM,
>> + AXI_DMA,
>> + TRACE_S2MM_FULL
>> +};
>> +
>> +struct debug_ip_data {
>> + uint8_t m_type; /* type of enum DEBUG_IP_TYPE */
>> + uint8_t m_index_lowbyte;
>> + uint8_t m_properties;
>> + uint8_t m_major;
>> + uint8_t m_minor;
>> + uint8_t m_index_highbyte;
>> + uint8_t m_reserved[2];
>> + uint64_t m_base_address;
>> + char m_name[128];
>> +};
>> +
>> +struct debug_ip_layout {
>> + uint16_t m_count;
>> + struct debug_ip_data m_debug_ip_data[1];
>> +};
>> +
>> +/* Supported clock frequency types */
>> +enum CLOCK_TYPE {
>> + CT_UNUSED = 0, /* Initialized value */
>> + CT_DATA = 1, /* Data clock */
>> + CT_KERNEL = 2, /* Kernel clock */
>> + CT_SYSTEM = 3 /* System Clock */
>> +};
>> +
>> +/* Clock Frequency Entry */
>> +struct clock_freq {
>> + uint16_t m_freq_Mhz; /* Frequency in MHz */
>> + uint8_t m_type; /* Clock type (enum CLOCK_TYPE) */
>> + uint8_t m_unused[5]; /* Not used - padding */
>> + char m_name[128]; /* Clock Name */
>> +};
>> +
>> +/* Clock frequency section */
>> +struct clock_freq_topology {
>> + int16_t m_count; /* Number of entries */
>> + struct clock_freq m_clock_freq[1]; /* Clock array */
>> +};
>> +
>> +/* Supported MCS file types */
>> +enum MCS_TYPE {
>> + MCS_UNKNOWN = 0, /* Initialized value */
>> + MCS_PRIMARY = 1, /* The primary mcs file data */
>> + MCS_SECONDARY = 2, /* The secondary mcs file data */
>> +};
>> +
>> +/* One chunk of MCS data */
>> +struct mcs_chunk {
>> + uint8_t m_type; /* MCS data type */
>> + uint8_t m_unused[7]; /* padding */
>> + uint64_t m_offset; /* data offset from the start of */
>> + /* the section */
>> + uint64_t m_size; /* data size */
>> +};
>> +
>> +/* MCS data section */
>> +struct mcs {
>> + int8_t m_count; /* Number of chunks */
>> + int8_t m_unused[7]; /* padding */
>> + struct mcs_chunk m_chunk[1]; /* MCS chunks followed by data */
>> +};
>> +
>> +/* bmc data section */
>> +struct bmc {
>> + uint64_t m_offset; /* data offset from the start of */
>> + /* the section */
>> + uint64_t m_size; /* data size (bytes) */
>> + char m_image_name[64]; /* Name of the image */
>> + /* (e.g., MSP432P401R) */
>> + char m_device_name[64]; /* Device ID (e.g., VCU1525) */
>> + char m_version[64];
>> + char m_md5value[33]; /* MD5 Expected Value */
>> + /* (e.g., 56027182079c0bd621761b7dab5a27ca)*/
>> + char m_padding[7]; /* Padding */
>> +};
>> +
>> +/* soft kernel data section, used by classic driver */
>> +struct soft_kernel {
>> + /** Prefix Syntax:
>> + * mpo - member, pointer, offset
>> + * This variable represents a zero terminated string
>> + * that is offseted from the beginning of the section.
>> + * The pointer to access the string is initialized as follows:
>> + * char * pCharString = (address_of_section) + (mpo value)
>> + */
>> + uint32_t mpo_name; /* Name of the soft kernel */
>> + uint32_t m_image_offset; /* Image offset */
>> + uint32_t m_image_size; /* Image size */
>> + uint32_t mpo_version; /* Version */
>> + uint32_t mpo_md5_value; /* MD5 checksum */
>> + uint32_t mpo_symbol_name; /* Symbol name */
>> + uint32_t m_num_instances; /* Number of instances */
>> + uint8_t padding[36]; /* Reserved for future use */
>> + uint8_t reservedExt[16]; /* Reserved for future extended data */
>> +};
>> +
>> +enum CHECKSUM_TYPE {
>> + CST_UNKNOWN = 0,
>> + CST_SDBM = 1,
>> + CST_LAST
>> +};
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> +
>> +#endif
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-02-26 21:23 ` Lizhi Hou
@ 2021-02-28 16:54 ` Tom Rix
2021-03-02 0:25 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-28 16:54 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
On 2/26/21 1:23 PM, Lizhi Hou wrote:
> Hi Tom,
>
>
snip
>>
>> I also do not see a pragma pack, usually this is set of 1 so the compiler does not shuffle elements, increase size etc.
> This data structure is shared with other tools. And the structure is well defined with reasonable alignment. It is compatible with all compilers we have tested. So pragma pack is not necessary.
You can not have possibly tested all the configurations since the kernel supports many arches and compilers.
If the tested existing alignment is ok, pragma pack should be a noop on your tested configurations.
And help cover the untested configurations.
Tom
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-02-28 16:54 ` Tom Rix
@ 2021-03-02 0:25 ` Lizhi Hou
2021-03-02 15:14 ` Moritz Fischer
0 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-03-02 0:25 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom，
On 02/28/2021 08:54 AM, Tom Rix wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> On 2/26/21 1:23 PM, Lizhi Hou wrote:
>> Hi Tom,
>>
>>
> snip
>
>>> I also do not see a pragma pack, usually this is set of 1 so the compiler does not shuffle elements, increase size etc.
>> This data structure is shared with other tools. And the structure is well defined with reasonable alignment. It is compatible with all compilers we have tested. So pragma pack is not necessary.
> You can not have possibly tested all the configurations since the kernel supports many arches and compilers.
>
> If the tested existing alignment is ok, pragma pack should be a noop on your tested configurations.
>
> And help cover the untested configurations.
Got it. I will add pragma pack(1).
Lizhi
>
> Tom
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-03-02 0:25 ` Lizhi Hou
@ 2021-03-02 15:14 ` Moritz Fischer
2021-03-04 18:53 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Moritz Fischer @ 2021-03-02 15:14 UTC (permalink / raw)
To: Lizhi Hou
Cc: Tom Rix, linux-kernel, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, mdf, robh, Max Zhen
On Mon, Mar 01, 2021 at 04:25:37PM -0800, Lizhi Hou wrote:
> Hi Tom，
>
>
> On 02/28/2021 08:54 AM, Tom Rix wrote:
> > CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
> >
> >
> > On 2/26/21 1:23 PM, Lizhi Hou wrote:
> > > Hi Tom,
> > >
> > >
> > snip
> >
> > > > I also do not see a pragma pack, usually this is set of 1 so the compiler does not shuffle elements, increase size etc.
> > > This data structure is shared with other tools. And the structure is well defined with reasonable alignment. It is compatible with all compilers we have tested. So pragma pack is not necessary.
> > You can not have possibly tested all the configurations since the kernel supports many arches and compilers.
> >
> > If the tested existing alignment is ok, pragma pack should be a noop on your tested configurations.
> >
> > And help cover the untested configurations.
> Got it. I will add pragma pack(1).
Please do not use pragma pack(), add __packed to the structs in
question.
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file helper functions
2021-03-02 15:14 ` Moritz Fischer
@ 2021-03-04 18:53 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-04 18:53 UTC (permalink / raw)
To: Moritz Fischer, Lizhi Hou
Cc: Tom Rix, linux-kernel, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, robh, Max Zhen
Hi Moritz,
On 03/02/2021 07:14 AM, Moritz Fischer wrote:
>
> On Mon, Mar 01, 2021 at 04:25:37PM -0800, Lizhi Hou wrote:
>> Hi Tom，
>>
>>
>> On 02/28/2021 08:54 AM, Tom Rix wrote:
>>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>>>
>>>
>>> On 2/26/21 1:23 PM, Lizhi Hou wrote:
>>>> Hi Tom,
>>>>
>>>>
>>> snip
>>>
>>>>> I also do not see a pragma pack, usually this is set of 1 so the compiler does not shuffle elements, increase size etc.
>>>> This data structure is shared with other tools. And the structure is well defined with reasonable alignment. It is compatible with all compilers we have tested. So pragma pack is not necessary.
>>> You can not have possibly tested all the configurations since the kernel supports many arches and compilers.
>>>
>>> If the tested existing alignment is ok, pragma pack should be a noop on your tested configurations.
>>>
>>> And help cover the untested configurations.
>> Got it. I will add pragma pack(1).
> Please do not use pragma pack(), add __packed to the structs in
> question.
Ok, I will use __packed.
Thanks,
Lizhi
>
> - Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (2 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 03/18] fpga: xrt: xclbin file " Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-21 20:39 ` Moritz Fischer
2021-02-22 15:05 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 05/18] fpga: xrt: group platform driver Lizhi Hou
` (15 subsequent siblings)
19 siblings, 2 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
xrt-lib kernel module infrastructure code to register and manage all
leaf driver modules.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/lib/main.c | 274 ++++++++++++++++++++++++++++++++++++
drivers/fpga/xrt/lib/main.h | 17 +++
2 files changed, 291 insertions(+)
create mode 100644 drivers/fpga/xrt/lib/main.c
create mode 100644 drivers/fpga/xrt/lib/main.h
diff --git a/drivers/fpga/xrt/lib/main.c b/drivers/fpga/xrt/lib/main.c
new file mode 100644
index 000000000000..36fb62710843
--- /dev/null
+++ b/drivers/fpga/xrt/lib/main.c
@@ -0,0 +1,274 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Driver for Xilinx Alveo FPGA Support
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include "xleaf.h"
+#include "xroot.h"
+#include "main.h"
+
+#define XRT_IPLIB_MODULE_NAME "xrt-lib"
+#define XRT_IPLIB_MODULE_VERSION "4.0.0"
+#define XRT_MAX_DEVICE_NODES 128
+#define XRT_DRVNAME(drv) ((drv)->driver.name)
+
+/*
+ * Subdev driver is known by ID to others. We map the ID to it's
+ * struct platform_driver, which contains it's binding name and driver/file ops.
+ * We also map it to the endpoint name in DTB as well, if it's different
+ * than the driver's binding name.
+ */
+struct xrt_drv_map {
+ struct list_head list;
+ enum xrt_subdev_id id;
+ struct platform_driver *drv;
+ struct xrt_subdev_endpoints *eps;
+ struct ida ida; /* manage driver instance and char dev minor */
+};
+
+static DEFINE_MUTEX(xrt_lib_lock); /* global lock protecting xrt_drv_maps list */
+static LIST_HEAD(xrt_drv_maps);
+struct class *xrt_class;
+
+static inline struct xrt_subdev_drvdata *
+xrt_drv_map2drvdata(struct xrt_drv_map *map)
+{
+ return (struct xrt_subdev_drvdata *)map->drv->id_table[0].driver_data;
+}
+
+static struct xrt_drv_map *
+xrt_drv_find_map_by_id_nolock(enum xrt_subdev_id id)
+{
+ const struct list_head *ptr;
+
+ list_for_each(ptr, &xrt_drv_maps) {
+ struct xrt_drv_map *tmap = list_entry(ptr, struct xrt_drv_map, list);
+
+ if (tmap->id == id)
+ return tmap;
+ }
+ return NULL;
+}
+
+static struct xrt_drv_map *
+xrt_drv_find_map_by_id(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map;
+
+ mutex_lock(&xrt_lib_lock);
+ map = xrt_drv_find_map_by_id_nolock(id);
+ mutex_unlock(&xrt_lib_lock);
+ /*
+ * map should remain valid even after lock is dropped since a registered
+ * driver should only be unregistered when driver module is being unloaded,
+ * which means that the driver should not be used by then.
+ */
+ return map;
+}
+
+static int xrt_drv_register_driver(struct xrt_drv_map *map)
+{
+ struct xrt_subdev_drvdata *drvdata;
+ int rc = 0;
+ const char *drvname = XRT_DRVNAME(map->drv);
+
+ rc = platform_driver_register(map->drv);
+ if (rc) {
+ pr_err("register %s platform driver failed\n", drvname);
+ return rc;
+ }
+
+ drvdata = xrt_drv_map2drvdata(map);
+ if (drvdata) {
+ /* Initialize dev_t for char dev node. */
+ if (xleaf_devnode_enabled(drvdata)) {
+ rc = alloc_chrdev_region(&drvdata->xsd_file_ops.xsf_dev_t, 0,
+ XRT_MAX_DEVICE_NODES, drvname);
+ if (rc) {
+ platform_driver_unregister(map->drv);
+ pr_err("failed to alloc dev minor for %s: %d\n", drvname, rc);
+ return rc;
+ }
+ } else {
+ drvdata->xsd_file_ops.xsf_dev_t = (dev_t)-1;
+ }
+ }
+
+ ida_init(&map->ida);
+
+ pr_info("%s registered successfully\n", drvname);
+
+ return 0;
+}
+
+static void xrt_drv_unregister_driver(struct xrt_drv_map *map)
+{
+ const char *drvname = XRT_DRVNAME(map->drv);
+ struct xrt_subdev_drvdata *drvdata;
+
+ ida_destroy(&map->ida);
+
+ drvdata = xrt_drv_map2drvdata(map);
+ if (drvdata && drvdata->xsd_file_ops.xsf_dev_t != (dev_t)-1) {
+ unregister_chrdev_region(drvdata->xsd_file_ops.xsf_dev_t,
+ XRT_MAX_DEVICE_NODES);
+ }
+
+ platform_driver_unregister(map->drv);
+
+ pr_info("%s unregistered successfully\n", drvname);
+}
+
+int xleaf_register_driver(enum xrt_subdev_id id,
+ struct platform_driver *drv,
+ struct xrt_subdev_endpoints *eps)
+{
+ struct xrt_drv_map *map;
+
+ mutex_lock(&xrt_lib_lock);
+
+ map = xrt_drv_find_map_by_id_nolock(id);
+ if (map) {
+ mutex_unlock(&xrt_lib_lock);
+ pr_err("Id %d already has a registered driver, 0x%p\n",
+ id, map->drv);
+ return -EEXIST;
+ }
+
+ map = vzalloc(sizeof(*map));
+ if (!map) {
+ mutex_unlock(&xrt_lib_lock);
+ return -ENOMEM;
+ }
+ map->id = id;
+ map->drv = drv;
+ map->eps = eps;
+
+ xrt_drv_register_driver(map);
+
+ list_add(&map->list, &xrt_drv_maps);
+
+ mutex_unlock(&xrt_lib_lock);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xleaf_register_driver);
+
+void xleaf_unregister_driver(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map;
+
+ mutex_lock(&xrt_lib_lock);
+
+ map = xrt_drv_find_map_by_id_nolock(id);
+ if (!map) {
+ mutex_unlock(&xrt_lib_lock);
+ pr_err("Id %d has no registered driver\n", id);
+ return;
+ }
+
+ list_del(&map->list);
+
+ mutex_unlock(&xrt_lib_lock);
+
+ xrt_drv_unregister_driver(map);
+ vfree(map);
+}
+EXPORT_SYMBOL_GPL(xleaf_unregister_driver);
+
+const char *xrt_drv_name(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+
+ if (map)
+ return XRT_DRVNAME(map->drv);
+ return NULL;
+}
+
+int xrt_drv_get_instance(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+
+ return ida_alloc_range(&map->ida, 0, XRT_MAX_DEVICE_NODES, GFP_KERNEL);
+}
+
+void xrt_drv_put_instance(enum xrt_subdev_id id, int instance)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+
+ ida_free(&map->ida, instance);
+}
+
+struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id)
+{
+ struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
+ struct xrt_subdev_endpoints *eps;
+
+ eps = map ? map->eps : NULL;
+ return eps;
+}
+
+/* Leaf driver's module init/fini callbacks. */
+static void (*leaf_init_fini_cbs[])(bool) = {
+ group_leaf_init_fini,
+ vsec_leaf_init_fini,
+ devctl_leaf_init_fini,
+ axigate_leaf_init_fini,
+ icap_leaf_init_fini,
+ calib_leaf_init_fini,
+ clkfreq_leaf_init_fini,
+ clock_leaf_init_fini,
+ ucs_leaf_init_fini,
+};
+
+static __init int xrt_lib_init(void)
+{
+ int i;
+
+ xrt_class = class_create(THIS_MODULE, XRT_IPLIB_MODULE_NAME);
+ if (IS_ERR(xrt_class))
+ return PTR_ERR(xrt_class);
+
+ for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
+ leaf_init_fini_cbs[i](true);
+ return 0;
+}
+
+static __exit void xrt_lib_fini(void)
+{
+ struct xrt_drv_map *map;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
+ leaf_init_fini_cbs[i](false);
+
+ mutex_lock(&xrt_lib_lock);
+
+ while (!list_empty(&xrt_drv_maps)) {
+ map = list_first_entry_or_null(&xrt_drv_maps, struct xrt_drv_map, list);
+ pr_err("Unloading module with %s still registered\n", XRT_DRVNAME(map->drv));
+ list_del(&map->list);
+ mutex_unlock(&xrt_lib_lock);
+ xrt_drv_unregister_driver(map);
+ vfree(map);
+ mutex_lock(&xrt_lib_lock);
+ }
+
+ mutex_unlock(&xrt_lib_lock);
+
+ class_destroy(xrt_class);
+}
+
+module_init(xrt_lib_init);
+module_exit(xrt_lib_fini);
+
+MODULE_VERSION(XRT_IPLIB_MODULE_VERSION);
+MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
+MODULE_DESCRIPTION("Xilinx Alveo IP Lib driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/fpga/xrt/lib/main.h b/drivers/fpga/xrt/lib/main.h
new file mode 100644
index 000000000000..f3bfc87ee614
--- /dev/null
+++ b/drivers/fpga/xrt/lib/main.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_MAIN_H_
+#define _XRT_MAIN_H_
+
+const char *xrt_drv_name(enum xrt_subdev_id id);
+int xrt_drv_get_instance(enum xrt_subdev_id id);
+void xrt_drv_put_instance(enum xrt_subdev_id id, int instance);
+struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id);
+
+#endif /* _XRT_MAIN_H_ */
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager Lizhi Hou
@ 2021-02-21 20:39 ` Moritz Fischer
2021-03-01 20:34 ` Max Zhen
2021-02-22 15:05 ` Tom Rix
1 sibling, 1 reply; 87+ messages in thread
From: Moritz Fischer @ 2021-02-21 20:39 UTC (permalink / raw)
To: Lizhi Hou
Cc: linux-kernel, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, mdf, robh, Max Zhen
Lizhi,
On Wed, Feb 17, 2021 at 10:40:05PM -0800, Lizhi Hou wrote:
> xrt-lib kernel module infrastructure code to register and manage all
> leaf driver modules.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/lib/main.c | 274 ++++++++++++++++++++++++++++++++++++
> drivers/fpga/xrt/lib/main.h | 17 +++
> 2 files changed, 291 insertions(+)
> create mode 100644 drivers/fpga/xrt/lib/main.c
> create mode 100644 drivers/fpga/xrt/lib/main.h
>
> diff --git a/drivers/fpga/xrt/lib/main.c b/drivers/fpga/xrt/lib/main.c
> new file mode 100644
> index 000000000000..36fb62710843
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/main.c
> @@ -0,0 +1,274 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for Xilinx Alveo FPGA Support
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/module.h>
> +#include "xleaf.h"
> +#include "xroot.h"
> +#include "main.h"
> +
> +#define XRT_IPLIB_MODULE_NAME "xrt-lib"
> +#define XRT_IPLIB_MODULE_VERSION "4.0.0"
> +#define XRT_MAX_DEVICE_NODES 128
> +#define XRT_DRVNAME(drv) ((drv)->driver.name)
> +
> +/*
> + * Subdev driver is known by ID to others. We map the ID to it's
> + * struct platform_driver, which contains it's binding name and driver/file ops.
> + * We also map it to the endpoint name in DTB as well, if it's different
> + * than the driver's binding name.
> + */
> +struct xrt_drv_map {
> + struct list_head list;
> + enum xrt_subdev_id id;
> + struct platform_driver *drv;
> + struct xrt_subdev_endpoints *eps;
> + struct ida ida; /* manage driver instance and char dev minor */
> +};
> +
> +static DEFINE_MUTEX(xrt_lib_lock); /* global lock protecting xrt_drv_maps list */
> +static LIST_HEAD(xrt_drv_maps);
> +struct class *xrt_class;
> +
> +static inline struct xrt_subdev_drvdata *
> +xrt_drv_map2drvdata(struct xrt_drv_map *map)
> +{
> + return (struct xrt_subdev_drvdata *)map->drv->id_table[0].driver_data;
> +}
> +
> +static struct xrt_drv_map *
> +xrt_drv_find_map_by_id_nolock(enum xrt_subdev_id id)
> +{
> + const struct list_head *ptr;
> +
> + list_for_each(ptr, &xrt_drv_maps) {
> + struct xrt_drv_map *tmap = list_entry(ptr, struct xrt_drv_map, list);
list_for_each_entry, maybe?
> +
> + if (tmap->id == id)
> + return tmap;
> + }
> + return NULL;
> +}
> +
> +static struct xrt_drv_map *
> +xrt_drv_find_map_by_id(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map;
> +
> + mutex_lock(&xrt_lib_lock);
> + map = xrt_drv_find_map_by_id_nolock(id);
> + mutex_unlock(&xrt_lib_lock);
> + /*
> + * map should remain valid even after lock is dropped since a registered
> + * driver should only be unregistered when driver module is being unloaded,
> + * which means that the driver should not be used by then.
> + */
> + return map;
> +}
> +
> +static int xrt_drv_register_driver(struct xrt_drv_map *map)
> +{
> + struct xrt_subdev_drvdata *drvdata;
> + int rc = 0;
> + const char *drvname = XRT_DRVNAME(map->drv);
> +
> + rc = platform_driver_register(map->drv);
> + if (rc) {
> + pr_err("register %s platform driver failed\n", drvname);
> + return rc;
> + }
> +
> + drvdata = xrt_drv_map2drvdata(map);
> + if (drvdata) {
> + /* Initialize dev_t for char dev node. */
> + if (xleaf_devnode_enabled(drvdata)) {
> + rc = alloc_chrdev_region(&drvdata->xsd_file_ops.xsf_dev_t, 0,
> + XRT_MAX_DEVICE_NODES, drvname);
> + if (rc) {
> + platform_driver_unregister(map->drv);
> + pr_err("failed to alloc dev minor for %s: %d\n", drvname, rc);
> + return rc;
> + }
> + } else {
> + drvdata->xsd_file_ops.xsf_dev_t = (dev_t)-1;
> + }
> + }
> +
> + ida_init(&map->ida);
> +
> + pr_info("%s registered successfully\n", drvname);
Is this not xrt_info() then?
> +
> + return 0;
> +}
> +
> +static void xrt_drv_unregister_driver(struct xrt_drv_map *map)
> +{
> + const char *drvname = XRT_DRVNAME(map->drv);
> + struct xrt_subdev_drvdata *drvdata;
> +
> + ida_destroy(&map->ida);
> +
> + drvdata = xrt_drv_map2drvdata(map);
> + if (drvdata && drvdata->xsd_file_ops.xsf_dev_t != (dev_t)-1) {
> + unregister_chrdev_region(drvdata->xsd_file_ops.xsf_dev_t,
> + XRT_MAX_DEVICE_NODES);
> + }
> +
> + platform_driver_unregister(map->drv);
> +
> + pr_info("%s unregistered successfully\n", drvname);
> +}
> +
> +int xleaf_register_driver(enum xrt_subdev_id id,
> + struct platform_driver *drv,
> + struct xrt_subdev_endpoints *eps)
> +{
> + struct xrt_drv_map *map;
> +
> + mutex_lock(&xrt_lib_lock);
> +
> + map = xrt_drv_find_map_by_id_nolock(id);
> + if (map) {
> + mutex_unlock(&xrt_lib_lock);
> + pr_err("Id %d already has a registered driver, 0x%p\n",
> + id, map->drv);
> + return -EEXIST;
> + }
> +
> + map = vzalloc(sizeof(*map));
> + if (!map) {
> + mutex_unlock(&xrt_lib_lock);
> + return -ENOMEM;
> + }
> + map->id = id;
> + map->drv = drv;
> + map->eps = eps;
> +
> + xrt_drv_register_driver(map);
> +
> + list_add(&map->list, &xrt_drv_maps);
> +
> + mutex_unlock(&xrt_lib_lock);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xleaf_register_driver);
> +
> +void xleaf_unregister_driver(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map;
> +
> + mutex_lock(&xrt_lib_lock);
> +
> + map = xrt_drv_find_map_by_id_nolock(id);
> + if (!map) {
> + mutex_unlock(&xrt_lib_lock);
> + pr_err("Id %d has no registered driver\n", id);
> + return;
> + }
> +
> + list_del(&map->list);
> +
> + mutex_unlock(&xrt_lib_lock);
> +
> + xrt_drv_unregister_driver(map);
> + vfree(map);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_unregister_driver);
> +
> +const char *xrt_drv_name(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> +
> + if (map)
> + return XRT_DRVNAME(map->drv);
> + return NULL;
> +}
> +
> +int xrt_drv_get_instance(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> +
> + return ida_alloc_range(&map->ida, 0, XRT_MAX_DEVICE_NODES, GFP_KERNEL);
> +}
> +
> +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance)
> +{
> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> +
> + ida_free(&map->ida, instance);
> +}
> +
> +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> + struct xrt_subdev_endpoints *eps;
> +
> + eps = map ? map->eps : NULL;
> + return eps;
> +}
> +
> +/* Leaf driver's module init/fini callbacks. */
> +static void (*leaf_init_fini_cbs[])(bool) = {
> + group_leaf_init_fini,
> + vsec_leaf_init_fini,
> + devctl_leaf_init_fini,
> + axigate_leaf_init_fini,
> + icap_leaf_init_fini,
> + calib_leaf_init_fini,
> + clkfreq_leaf_init_fini,
> + clock_leaf_init_fini,
> + ucs_leaf_init_fini,
> +};
> +
> +static __init int xrt_lib_init(void)
> +{
> + int i;
> +
> + xrt_class = class_create(THIS_MODULE, XRT_IPLIB_MODULE_NAME);
> + if (IS_ERR(xrt_class))
> + return PTR_ERR(xrt_class);
> +
> + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
> + leaf_init_fini_cbs[i](true);
> + return 0;
> +}
> +
> +static __exit void xrt_lib_fini(void)
> +{
> + struct xrt_drv_map *map;
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
> + leaf_init_fini_cbs[i](false);
> +
> + mutex_lock(&xrt_lib_lock);
> +
> + while (!list_empty(&xrt_drv_maps)) {
> + map = list_first_entry_or_null(&xrt_drv_maps, struct xrt_drv_map, list);
> + pr_err("Unloading module with %s still registered\n", XRT_DRVNAME(map->drv));
> + list_del(&map->list);
> + mutex_unlock(&xrt_lib_lock);
> + xrt_drv_unregister_driver(map);
> + vfree(map);
> + mutex_lock(&xrt_lib_lock);
> + }
> +
> + mutex_unlock(&xrt_lib_lock);
> +
> + class_destroy(xrt_class);
> +}
> +
> +module_init(xrt_lib_init);
> +module_exit(xrt_lib_fini);
> +
> +MODULE_VERSION(XRT_IPLIB_MODULE_VERSION);
> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
> +MODULE_DESCRIPTION("Xilinx Alveo IP Lib driver");
> +MODULE_LICENSE("GPL v2");
> diff --git a/drivers/fpga/xrt/lib/main.h b/drivers/fpga/xrt/lib/main.h
> new file mode 100644
> index 000000000000..f3bfc87ee614
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/main.h
> @@ -0,0 +1,17 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_MAIN_H_
> +#define _XRT_MAIN_H_
> +
> +const char *xrt_drv_name(enum xrt_subdev_id id);
> +int xrt_drv_get_instance(enum xrt_subdev_id id);
> +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance);
> +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id);
> +
> +#endif /* _XRT_MAIN_H_ */
> --
> 2.18.4
>
This smells like it should be a bus to me...
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager
2021-02-21 20:39 ` Moritz Fischer
@ 2021-03-01 20:34 ` Max Zhen
0 siblings, 0 replies; 87+ messages in thread
From: Max Zhen @ 2021-03-01 20:34 UTC (permalink / raw)
To: Moritz Fischer, Lizhi Hou
Cc: linux-kernel, Lizhi Hou, linux-fpga, sonal.santan, michal.simek,
stefanos, devicetree, trix, robh, Max Zhen
Hi Moritz,
On 2/21/21 12:39 PM, Moritz Fischer wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> Lizhi,
>
> On Wed, Feb 17, 2021 at 10:40:05PM -0800, Lizhi Hou wrote:
>> xrt-lib kernel module infrastructure code to register and manage all
>> leaf driver modules.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/lib/main.c | 274 ++++++++++++++++++++++++++++++++++++
>> drivers/fpga/xrt/lib/main.h | 17 +++
>> 2 files changed, 291 insertions(+)
>> create mode 100644 drivers/fpga/xrt/lib/main.c
>> create mode 100644 drivers/fpga/xrt/lib/main.h
>>
>> diff --git a/drivers/fpga/xrt/lib/main.c b/drivers/fpga/xrt/lib/main.c
>> new file mode 100644
>> index 000000000000..36fb62710843
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/main.c
>> @@ -0,0 +1,274 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for Xilinx Alveo FPGA Support
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#include <linux/module.h>
>> +#include "xleaf.h"
>> +#include "xroot.h"
>> +#include "main.h"
>> +
>> +#define XRT_IPLIB_MODULE_NAME "xrt-lib"
>> +#define XRT_IPLIB_MODULE_VERSION "4.0.0"
>> +#define XRT_MAX_DEVICE_NODES 128
>> +#define XRT_DRVNAME(drv) ((drv)->driver.name)
>> +
>> +/*
>> + * Subdev driver is known by ID to others. We map the ID to it's
>> + * struct platform_driver, which contains it's binding name and driver/file ops.
>> + * We also map it to the endpoint name in DTB as well, if it's different
>> + * than the driver's binding name.
>> + */
>> +struct xrt_drv_map {
>> + struct list_head list;
>> + enum xrt_subdev_id id;
>> + struct platform_driver *drv;
>> + struct xrt_subdev_endpoints *eps;
>> + struct ida ida; /* manage driver instance and char dev minor */
>> +};
>> +
>> +static DEFINE_MUTEX(xrt_lib_lock); /* global lock protecting xrt_drv_maps list */
>> +static LIST_HEAD(xrt_drv_maps);
>> +struct class *xrt_class;
>> +
>> +static inline struct xrt_subdev_drvdata *
>> +xrt_drv_map2drvdata(struct xrt_drv_map *map)
>> +{
>> + return (struct xrt_subdev_drvdata *)map->drv->id_table[0].driver_data;
>> +}
>> +
>> +static struct xrt_drv_map *
>> +xrt_drv_find_map_by_id_nolock(enum xrt_subdev_id id)
>> +{
>> + const struct list_head *ptr;
>> +
>> + list_for_each(ptr, &xrt_drv_maps) {
>> + struct xrt_drv_map *tmap = list_entry(ptr, struct xrt_drv_map, list);
> list_for_each_entry, maybe?
Yes!
>> +
>> + if (tmap->id == id)
>> + return tmap;
>> + }
>> + return NULL;
>> +}
>> +
>> +static struct xrt_drv_map *
>> +xrt_drv_find_map_by_id(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map;
>> +
>> + mutex_lock(&xrt_lib_lock);
>> + map = xrt_drv_find_map_by_id_nolock(id);
>> + mutex_unlock(&xrt_lib_lock);
>> + /*
>> + * map should remain valid even after lock is dropped since a registered
>> + * driver should only be unregistered when driver module is being unloaded,
>> + * which means that the driver should not be used by then.
>> + */
>> + return map;
>> +}
>> +
>> +static int xrt_drv_register_driver(struct xrt_drv_map *map)
>> +{
>> + struct xrt_subdev_drvdata *drvdata;
>> + int rc = 0;
>> + const char *drvname = XRT_DRVNAME(map->drv);
>> +
>> + rc = platform_driver_register(map->drv);
>> + if (rc) {
>> + pr_err("register %s platform driver failed\n", drvname);
>> + return rc;
>> + }
>> +
>> + drvdata = xrt_drv_map2drvdata(map);
>> + if (drvdata) {
>> + /* Initialize dev_t for char dev node. */
>> + if (xleaf_devnode_enabled(drvdata)) {
>> + rc = alloc_chrdev_region(&drvdata->xsd_file_ops.xsf_dev_t, 0,
>> + XRT_MAX_DEVICE_NODES, drvname);
>> + if (rc) {
>> + platform_driver_unregister(map->drv);
>> + pr_err("failed to alloc dev minor for %s: %d\n", drvname, rc);
>> + return rc;
>> + }
>> + } else {
>> + drvdata->xsd_file_ops.xsf_dev_t = (dev_t)-1;
>> + }
>> + }
>> +
>> + ida_init(&map->ida);
>> +
>> + pr_info("%s registered successfully\n", drvname);
> Is this not xrt_info() then?
xrt_info() is meant for leaf driver to call where struct platform_device
is available. We are initializing the drivers here, so can't call
xrt_info().
>> +
>> + return 0;
>> +}
>> +
>> +static void xrt_drv_unregister_driver(struct xrt_drv_map *map)
>> +{
>> + const char *drvname = XRT_DRVNAME(map->drv);
>> + struct xrt_subdev_drvdata *drvdata;
>> +
>> + ida_destroy(&map->ida);
>> +
>> + drvdata = xrt_drv_map2drvdata(map);
>> + if (drvdata && drvdata->xsd_file_ops.xsf_dev_t != (dev_t)-1) {
>> + unregister_chrdev_region(drvdata->xsd_file_ops.xsf_dev_t,
>> + XRT_MAX_DEVICE_NODES);
>> + }
>> +
>> + platform_driver_unregister(map->drv);
>> +
>> + pr_info("%s unregistered successfully\n", drvname);
>> +}
>> +
>> +int xleaf_register_driver(enum xrt_subdev_id id,
>> + struct platform_driver *drv,
>> + struct xrt_subdev_endpoints *eps)
>> +{
>> + struct xrt_drv_map *map;
>> +
>> + mutex_lock(&xrt_lib_lock);
>> +
>> + map = xrt_drv_find_map_by_id_nolock(id);
>> + if (map) {
>> + mutex_unlock(&xrt_lib_lock);
>> + pr_err("Id %d already has a registered driver, 0x%p\n",
>> + id, map->drv);
>> + return -EEXIST;
>> + }
>> +
>> + map = vzalloc(sizeof(*map));
>> + if (!map) {
>> + mutex_unlock(&xrt_lib_lock);
>> + return -ENOMEM;
>> + }
>> + map->id = id;
>> + map->drv = drv;
>> + map->eps = eps;
>> +
>> + xrt_drv_register_driver(map);
>> +
>> + list_add(&map->list, &xrt_drv_maps);
>> +
>> + mutex_unlock(&xrt_lib_lock);
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xleaf_register_driver);
>> +
>> +void xleaf_unregister_driver(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map;
>> +
>> + mutex_lock(&xrt_lib_lock);
>> +
>> + map = xrt_drv_find_map_by_id_nolock(id);
>> + if (!map) {
>> + mutex_unlock(&xrt_lib_lock);
>> + pr_err("Id %d has no registered driver\n", id);
>> + return;
>> + }
>> +
>> + list_del(&map->list);
>> +
>> + mutex_unlock(&xrt_lib_lock);
>> +
>> + xrt_drv_unregister_driver(map);
>> + vfree(map);
>> +}
>> +EXPORT_SYMBOL_GPL(xleaf_unregister_driver);
>> +
>> +const char *xrt_drv_name(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
>> +
>> + if (map)
>> + return XRT_DRVNAME(map->drv);
>> + return NULL;
>> +}
>> +
>> +int xrt_drv_get_instance(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
>> +
>> + return ida_alloc_range(&map->ida, 0, XRT_MAX_DEVICE_NODES, GFP_KERNEL);
>> +}
>> +
>> +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance)
>> +{
>> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
>> +
>> + ida_free(&map->ida, instance);
>> +}
>> +
>> +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
>> + struct xrt_subdev_endpoints *eps;
>> +
>> + eps = map ? map->eps : NULL;
>> + return eps;
>> +}
>> +
>> +/* Leaf driver's module init/fini callbacks. */
>> +static void (*leaf_init_fini_cbs[])(bool) = {
>> + group_leaf_init_fini,
>> + vsec_leaf_init_fini,
>> + devctl_leaf_init_fini,
>> + axigate_leaf_init_fini,
>> + icap_leaf_init_fini,
>> + calib_leaf_init_fini,
>> + clkfreq_leaf_init_fini,
>> + clock_leaf_init_fini,
>> + ucs_leaf_init_fini,
>> +};
>> +
>> +static __init int xrt_lib_init(void)
>> +{
>> + int i;
>> +
>> + xrt_class = class_create(THIS_MODULE, XRT_IPLIB_MODULE_NAME);
>> + if (IS_ERR(xrt_class))
>> + return PTR_ERR(xrt_class);
>> +
>> + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
>> + leaf_init_fini_cbs[i](true);
>> + return 0;
>> +}
>> +
>> +static __exit void xrt_lib_fini(void)
>> +{
>> + struct xrt_drv_map *map;
>> + int i;
>> +
>> + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
>> + leaf_init_fini_cbs[i](false);
>> +
>> + mutex_lock(&xrt_lib_lock);
>> +
>> + while (!list_empty(&xrt_drv_maps)) {
>> + map = list_first_entry_or_null(&xrt_drv_maps, struct xrt_drv_map, list);
>> + pr_err("Unloading module with %s still registered\n", XRT_DRVNAME(map->drv));
>> + list_del(&map->list);
>> + mutex_unlock(&xrt_lib_lock);
>> + xrt_drv_unregister_driver(map);
>> + vfree(map);
>> + mutex_lock(&xrt_lib_lock);
>> + }
>> +
>> + mutex_unlock(&xrt_lib_lock);
>> +
>> + class_destroy(xrt_class);
>> +}
>> +
>> +module_init(xrt_lib_init);
>> +module_exit(xrt_lib_fini);
>> +
>> +MODULE_VERSION(XRT_IPLIB_MODULE_VERSION);
>> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
>> +MODULE_DESCRIPTION("Xilinx Alveo IP Lib driver");
>> +MODULE_LICENSE("GPL v2");
>> diff --git a/drivers/fpga/xrt/lib/main.h b/drivers/fpga/xrt/lib/main.h
>> new file mode 100644
>> index 000000000000..f3bfc87ee614
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/main.h
>> @@ -0,0 +1,17 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_MAIN_H_
>> +#define _XRT_MAIN_H_
>> +
>> +const char *xrt_drv_name(enum xrt_subdev_id id);
>> +int xrt_drv_get_instance(enum xrt_subdev_id id);
>> +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance);
>> +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id);
>> +
>> +#endif /* _XRT_MAIN_H_ */
>> --
>> 2.18.4
>>
> This smells like it should be a bus to me...
This is discussed in another email thread. I will not add more comment here.
Thanks,
Max
>
> - Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager Lizhi Hou
2021-02-21 20:39 ` Moritz Fischer
@ 2021-02-22 15:05 ` Tom Rix
2021-02-23 3:35 ` Moritz Fischer
2021-03-03 17:20 ` Max Zhen
1 sibling, 2 replies; 87+ messages in thread
From: Tom Rix @ 2021-02-22 15:05 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> xrt-lib kernel module infrastructure code to register and manage all
> leaf driver modules.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/lib/main.c | 274 ++++++++++++++++++++++++++++++++++++
> drivers/fpga/xrt/lib/main.h | 17 +++
> 2 files changed, 291 insertions(+)
> create mode 100644 drivers/fpga/xrt/lib/main.c
> create mode 100644 drivers/fpga/xrt/lib/main.h
Not sure if 'main' is a good base name for something going into a lib.
>
> diff --git a/drivers/fpga/xrt/lib/main.c b/drivers/fpga/xrt/lib/main.c
> new file mode 100644
> index 000000000000..36fb62710843
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/main.c
> @@ -0,0 +1,274 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Driver for Xilinx Alveo FPGA Support
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/module.h>
> +#include "xleaf.h"
> +#include "xroot.h"
> +#include "main.h"
> +
> +#define XRT_IPLIB_MODULE_NAME "xrt-lib"
> +#define XRT_IPLIB_MODULE_VERSION "4.0.0"
> +#define XRT_MAX_DEVICE_NODES 128
> +#define XRT_DRVNAME(drv) ((drv)->driver.name)
> +
> +/*
> + * Subdev driver is known by ID to others. We map the ID to it's
by it's ID
> + * struct platform_driver, which contains it's binding name and driver/file ops.
> + * We also map it to the endpoint name in DTB as well, if it's different
> + * than the driver's binding name.
> + */
> +struct xrt_drv_map {
> + struct list_head list;
> + enum xrt_subdev_id id;
> + struct platform_driver *drv;
> + struct xrt_subdev_endpoints *eps;
> + struct ida ida; /* manage driver instance and char dev minor */
> +};
> +
> +static DEFINE_MUTEX(xrt_lib_lock); /* global lock protecting xrt_drv_maps list */
> +static LIST_HEAD(xrt_drv_maps);
> +struct class *xrt_class;
> +
> +static inline struct xrt_subdev_drvdata *
> +xrt_drv_map2drvdata(struct xrt_drv_map *map)
> +{
> + return (struct xrt_subdev_drvdata *)map->drv->id_table[0].driver_data;
> +}
> +
> +static struct xrt_drv_map *
> +xrt_drv_find_map_by_id_nolock(enum xrt_subdev_id id)
name could be by convention
__xrt_drv_find_map_id
> +{
> + const struct list_head *ptr;
> +
> + list_for_each(ptr, &xrt_drv_maps) {
> + struct xrt_drv_map *tmap = list_entry(ptr, struct xrt_drv_map, list);
> +
> + if (tmap->id == id)
> + return tmap;
> + }
> + return NULL;
> +}
> +
> +static struct xrt_drv_map *
> +xrt_drv_find_map_by_id(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map;
> +
> + mutex_lock(&xrt_lib_lock);
> + map = xrt_drv_find_map_by_id_nolock(id);
> + mutex_unlock(&xrt_lib_lock);
> + /*
> + * map should remain valid even after lock is dropped since a registered
even after the lock
> + * driver should only be unregistered when driver module is being unloaded,
> + * which means that the driver should not be used by then.
> + */
> + return map;
> +}
> +
> +static int xrt_drv_register_driver(struct xrt_drv_map *map)
> +{
> + struct xrt_subdev_drvdata *drvdata;
> + int rc = 0;
> + const char *drvname = XRT_DRVNAME(map->drv);
> +
> + rc = platform_driver_register(map->drv);
> + if (rc) {
> + pr_err("register %s platform driver failed\n", drvname);
> + return rc;
> + }
> +
> + drvdata = xrt_drv_map2drvdata(map);
> + if (drvdata) {
> + /* Initialize dev_t for char dev node. */
> + if (xleaf_devnode_enabled(drvdata)) {
> + rc = alloc_chrdev_region(&drvdata->xsd_file_ops.xsf_dev_t, 0,
> + XRT_MAX_DEVICE_NODES, drvname);
> + if (rc) {
> + platform_driver_unregister(map->drv);
> + pr_err("failed to alloc dev minor for %s: %d\n", drvname, rc);
> + return rc;
> + }
> + } else {
> + drvdata->xsd_file_ops.xsf_dev_t = (dev_t)-1;
> + }
> + }
> +
> + ida_init(&map->ida);
> +
> + pr_info("%s registered successfully\n", drvname);
> +
> + return 0;
> +}
> +
> +static void xrt_drv_unregister_driver(struct xrt_drv_map *map)
> +{
> + const char *drvname = XRT_DRVNAME(map->drv);
> + struct xrt_subdev_drvdata *drvdata;
> +
> + ida_destroy(&map->ida);
> +
> + drvdata = xrt_drv_map2drvdata(map);
> + if (drvdata && drvdata->xsd_file_ops.xsf_dev_t != (dev_t)-1) {
> + unregister_chrdev_region(drvdata->xsd_file_ops.xsf_dev_t,
> + XRT_MAX_DEVICE_NODES);
> + }
> +
> + platform_driver_unregister(map->drv);
> +
> + pr_info("%s unregistered successfully\n", drvname);
> +}
> +
> +int xleaf_register_driver(enum xrt_subdev_id id,
> + struct platform_driver *drv,
> + struct xrt_subdev_endpoints *eps)
> +{
> + struct xrt_drv_map *map;
> +
> + mutex_lock(&xrt_lib_lock);
Trying to minimize length of lock being held.
Could holding this lock be split or the alloc moved above ?
> +
> + map = xrt_drv_find_map_by_id_nolock(id);
> + if (map) {
> + mutex_unlock(&xrt_lib_lock);
> + pr_err("Id %d already has a registered driver, 0x%p\n",
> + id, map->drv);
> + return -EEXIST;
> + }
> +
> + map = vzalloc(sizeof(*map));
general issue
map is small, so kzalloc
> + if (!map) {
> + mutex_unlock(&xrt_lib_lock);
> + return -ENOMEM;
> + }
> + map->id = id;
> + map->drv = drv;
> + map->eps = eps;
> +
> + xrt_drv_register_driver(map);
xrt_drv_register_driver failure is unhandled.
This is the only time xrt_drv_register_driver is called, consider expanding the function here and removing the call.
> +
> + list_add(&map->list, &xrt_drv_maps);
> +
> + mutex_unlock(&xrt_lib_lock);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xleaf_register_driver);
> +
> +void xleaf_unregister_driver(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map;
> +
> + mutex_lock(&xrt_lib_lock);
> +
> + map = xrt_drv_find_map_by_id_nolock(id);
> + if (!map) {
> + mutex_unlock(&xrt_lib_lock);
> + pr_err("Id %d has no registered driver\n", id);
> + return;
> + }
> +
> + list_del(&map->list);
> +
> + mutex_unlock(&xrt_lib_lock);
> +
> + xrt_drv_unregister_driver(map);
> + vfree(map);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_unregister_driver);
> +
> +const char *xrt_drv_name(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> +
> + if (map)
> + return XRT_DRVNAME(map->drv);
> + return NULL;
> +}
> +
> +int xrt_drv_get_instance(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> +
> + return ida_alloc_range(&map->ida, 0, XRT_MAX_DEVICE_NODES, GFP_KERNEL);
> +}
> +
> +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance)
> +{
> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> +
> + ida_free(&map->ida, instance);
> +}
> +
> +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id)
> +{
> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> + struct xrt_subdev_endpoints *eps;
> +
> + eps = map ? map->eps : NULL;
> + return eps;
> +}
> +
> +/* Leaf driver's module init/fini callbacks. */
These constructor/destructor calls needs to be more dynamic.
calls are made even if there are no subdevices to go with the id's.
Also this list can not grow. How would a new id be added by a module ?
> +static void (*leaf_init_fini_cbs[])(bool) = {
> + group_leaf_init_fini,
> + vsec_leaf_init_fini,
> + devctl_leaf_init_fini,
> + axigate_leaf_init_fini,
> + icap_leaf_init_fini,
> + calib_leaf_init_fini,
> + clkfreq_leaf_init_fini,
> + clock_leaf_init_fini,
> + ucs_leaf_init_fini,
> +};
> +
> +static __init int xrt_lib_init(void)
> +{
> + int i;
> +
> + xrt_class = class_create(THIS_MODULE, XRT_IPLIB_MODULE_NAME);
> + if (IS_ERR(xrt_class))
> + return PTR_ERR(xrt_class);
> +
> + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
> + leaf_init_fini_cbs[i](true);
> + return 0;
> +}
> +
> +static __exit void xrt_lib_fini(void)
> +{
> + struct xrt_drv_map *map;
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
> + leaf_init_fini_cbs[i](false);
> +
> + mutex_lock(&xrt_lib_lock);
> +
> + while (!list_empty(&xrt_drv_maps)) {
> + map = list_first_entry_or_null(&xrt_drv_maps, struct xrt_drv_map, list);
> + pr_err("Unloading module with %s still registered\n", XRT_DRVNAME(map->drv));
> + list_del(&map->list);
> + mutex_unlock(&xrt_lib_lock);
> + xrt_drv_unregister_driver(map);
> + vfree(map);
> + mutex_lock(&xrt_lib_lock);
> + }
> +
> + mutex_unlock(&xrt_lib_lock);
> +
> + class_destroy(xrt_class);
> +}
> +
> +module_init(xrt_lib_init);
> +module_exit(xrt_lib_fini);
> +
> +MODULE_VERSION(XRT_IPLIB_MODULE_VERSION);
> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
> +MODULE_DESCRIPTION("Xilinx Alveo IP Lib driver");
> +MODULE_LICENSE("GPL v2");
> diff --git a/drivers/fpga/xrt/lib/main.h b/drivers/fpga/xrt/lib/main.h
> new file mode 100644
> index 000000000000..f3bfc87ee614
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/main.h
> @@ -0,0 +1,17 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_MAIN_H_
> +#define _XRT_MAIN_H_
> +
> +const char *xrt_drv_name(enum xrt_subdev_id id);
To be self contained, the header defining enum xrt_subdev_id should be included.
This is subdev_id.h which comes in with patch 6
A dependency on a future patch breaks bisectablity.
It may make sense to collect these small headers into a single large header for the ip infra lib and bring them all in this patch.
Tom
> +int xrt_drv_get_instance(enum xrt_subdev_id id);
> +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance);
> +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id);
> +
> +#endif /* _XRT_MAIN_H_ */
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager
2021-02-22 15:05 ` Tom Rix
@ 2021-02-23 3:35 ` Moritz Fischer
2021-03-03 17:20 ` Max Zhen
1 sibling, 0 replies; 87+ messages in thread
From: Moritz Fischer @ 2021-02-23 3:35 UTC (permalink / raw)
To: Tom Rix
Cc: Lizhi Hou, linux-kernel, Lizhi Hou, linux-fpga, maxz,
sonal.santan, michal.simek, stefanos, devicetree, mdf, robh,
Max Zhen
On Mon, Feb 22, 2021 at 07:05:29AM -0800, Tom Rix wrote:
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
> > xrt-lib kernel module infrastructure code to register and manage all
> > leaf driver modules.
> >
> > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> > Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> > ---
> > drivers/fpga/xrt/lib/main.c | 274 ++++++++++++++++++++++++++++++++++++
> > drivers/fpga/xrt/lib/main.h | 17 +++
> > 2 files changed, 291 insertions(+)
> > create mode 100644 drivers/fpga/xrt/lib/main.c
> > create mode 100644 drivers/fpga/xrt/lib/main.h
>
> Not sure if 'main' is a good base name for something going into a lib.
>
> >
> > diff --git a/drivers/fpga/xrt/lib/main.c b/drivers/fpga/xrt/lib/main.c
> > new file mode 100644
> > index 000000000000..36fb62710843
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/lib/main.c
> > @@ -0,0 +1,274 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Driver for Xilinx Alveo FPGA Support
> > + *
> > + * Copyright (C) 2020-2021 Xilinx, Inc.
> > + *
> > + * Authors:
> > + * Cheng Zhen <maxz@xilinx.com>
> > + */
> > +
> > +#include <linux/module.h>
> > +#include "xleaf.h"
> > +#include "xroot.h"
> > +#include "main.h"
> > +
> > +#define XRT_IPLIB_MODULE_NAME "xrt-lib"
> > +#define XRT_IPLIB_MODULE_VERSION "4.0.0"
> > +#define XRT_MAX_DEVICE_NODES 128
> > +#define XRT_DRVNAME(drv) ((drv)->driver.name)
> > +
> > +/*
> > + * Subdev driver is known by ID to others. We map the ID to it's
> by it's ID
> > + * struct platform_driver, which contains it's binding name and driver/file ops.
> > + * We also map it to the endpoint name in DTB as well, if it's different
> > + * than the driver's binding name.
> > + */
> > +struct xrt_drv_map {
> > + struct list_head list;
> > + enum xrt_subdev_id id;
> > + struct platform_driver *drv;
> > + struct xrt_subdev_endpoints *eps;
> > + struct ida ida; /* manage driver instance and char dev minor */
> > +};
> > +
> > +static DEFINE_MUTEX(xrt_lib_lock); /* global lock protecting xrt_drv_maps list */
> > +static LIST_HEAD(xrt_drv_maps);
> > +struct class *xrt_class;
> > +
> > +static inline struct xrt_subdev_drvdata *
> > +xrt_drv_map2drvdata(struct xrt_drv_map *map)
> > +{
> > + return (struct xrt_subdev_drvdata *)map->drv->id_table[0].driver_data;
> > +}
> > +
> > +static struct xrt_drv_map *
> > +xrt_drv_find_map_by_id_nolock(enum xrt_subdev_id id)
>
> name could be by convention
>
> __xrt_drv_find_map_id
>
> > +{
> > + const struct list_head *ptr;
> > +
> > + list_for_each(ptr, &xrt_drv_maps) {
> > + struct xrt_drv_map *tmap = list_entry(ptr, struct xrt_drv_map, list);
> > +
> > + if (tmap->id == id)
> > + return tmap;
> > + }
> > + return NULL;
> > +}
> > +
> > +static struct xrt_drv_map *
> > +xrt_drv_find_map_by_id(enum xrt_subdev_id id)
> > +{
> > + struct xrt_drv_map *map;
> > +
> > + mutex_lock(&xrt_lib_lock);
> > + map = xrt_drv_find_map_by_id_nolock(id);
> > + mutex_unlock(&xrt_lib_lock);
> > + /*
> > + * map should remain valid even after lock is dropped since a registered
> even after the lock
> > + * driver should only be unregistered when driver module is being unloaded,
> > + * which means that the driver should not be used by then.
> > + */
> > + return map;
> > +}
> > +
> > +static int xrt_drv_register_driver(struct xrt_drv_map *map)
> > +{
> > + struct xrt_subdev_drvdata *drvdata;
> > + int rc = 0;
> > + const char *drvname = XRT_DRVNAME(map->drv);
> > +
> > + rc = platform_driver_register(map->drv);
> > + if (rc) {
> > + pr_err("register %s platform driver failed\n", drvname);
> > + return rc;
> > + }
> > +
> > + drvdata = xrt_drv_map2drvdata(map);
> > + if (drvdata) {
> > + /* Initialize dev_t for char dev node. */
> > + if (xleaf_devnode_enabled(drvdata)) {
> > + rc = alloc_chrdev_region(&drvdata->xsd_file_ops.xsf_dev_t, 0,
> > + XRT_MAX_DEVICE_NODES, drvname);
> > + if (rc) {
> > + platform_driver_unregister(map->drv);
> > + pr_err("failed to alloc dev minor for %s: %d\n", drvname, rc);
> > + return rc;
> > + }
> > + } else {
> > + drvdata->xsd_file_ops.xsf_dev_t = (dev_t)-1;
> > + }
> > + }
> > +
> > + ida_init(&map->ida);
> > +
> > + pr_info("%s registered successfully\n", drvname);
> > +
> > + return 0;
> > +}
> > +
> > +static void xrt_drv_unregister_driver(struct xrt_drv_map *map)
> > +{
> > + const char *drvname = XRT_DRVNAME(map->drv);
> > + struct xrt_subdev_drvdata *drvdata;
> > +
> > + ida_destroy(&map->ida);
> > +
> > + drvdata = xrt_drv_map2drvdata(map);
> > + if (drvdata && drvdata->xsd_file_ops.xsf_dev_t != (dev_t)-1) {
> > + unregister_chrdev_region(drvdata->xsd_file_ops.xsf_dev_t,
> > + XRT_MAX_DEVICE_NODES);
> > + }
> > +
> > + platform_driver_unregister(map->drv);
> > +
> > + pr_info("%s unregistered successfully\n", drvname);
> > +}
> > +
> > +int xleaf_register_driver(enum xrt_subdev_id id,
> > + struct platform_driver *drv,
> > + struct xrt_subdev_endpoints *eps)
> > +{
> > + struct xrt_drv_map *map;
> > +
> > + mutex_lock(&xrt_lib_lock);
>
> Trying to minimize length of lock being held.
>
> Could holding this lock be split or the alloc moved above ?
>
> > +
> > + map = xrt_drv_find_map_by_id_nolock(id);
> > + if (map) {
> > + mutex_unlock(&xrt_lib_lock);
> > + pr_err("Id %d already has a registered driver, 0x%p\n",
> > + id, map->drv);
> > + return -EEXIST;
> > + }
> > +
> > + map = vzalloc(sizeof(*map));
>
> general issue
>
> map is small, so kzalloc
>
> > + if (!map) {
> > + mutex_unlock(&xrt_lib_lock);
> > + return -ENOMEM;
> > + }
> > + map->id = id;
> > + map->drv = drv;
> > + map->eps = eps;
> > +
> > + xrt_drv_register_driver(map);
>
> xrt_drv_register_driver failure is unhandled.
>
> This is the only time xrt_drv_register_driver is called, consider expanding the function here and removing the call.
>
> > +
> > + list_add(&map->list, &xrt_drv_maps);
> > +
> > + mutex_unlock(&xrt_lib_lock);
> > +
> > + return 0;
> > +}
> > +EXPORT_SYMBOL_GPL(xleaf_register_driver);
> > +
> > +void xleaf_unregister_driver(enum xrt_subdev_id id)
> > +{
> > + struct xrt_drv_map *map;
> > +
> > + mutex_lock(&xrt_lib_lock);
> > +
> > + map = xrt_drv_find_map_by_id_nolock(id);
> > + if (!map) {
> > + mutex_unlock(&xrt_lib_lock);
> > + pr_err("Id %d has no registered driver\n", id);
> > + return;
> > + }
> > +
> > + list_del(&map->list);
> > +
> > + mutex_unlock(&xrt_lib_lock);
> > +
> > + xrt_drv_unregister_driver(map);
> > + vfree(map);
> > +}
> > +EXPORT_SYMBOL_GPL(xleaf_unregister_driver);
> > +
> > +const char *xrt_drv_name(enum xrt_subdev_id id)
> > +{
> > + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> > +
> > + if (map)
> > + return XRT_DRVNAME(map->drv);
> > + return NULL;
> > +}
> > +
> > +int xrt_drv_get_instance(enum xrt_subdev_id id)
> > +{
> > + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> > +
> > + return ida_alloc_range(&map->ida, 0, XRT_MAX_DEVICE_NODES, GFP_KERNEL);
> > +}
> > +
> > +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance)
> > +{
> > + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> > +
> > + ida_free(&map->ida, instance);
> > +}
> > +
> > +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id)
> > +{
> > + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
> > + struct xrt_subdev_endpoints *eps;
> > +
> > + eps = map ? map->eps : NULL;
> > + return eps;
> > +}
> > +
> > +/* Leaf driver's module init/fini callbacks. */
>
> These constructor/destructor calls needs to be more dynamic.
>
> calls are made even if there are no subdevices to go with the id's.
>
> Also this list can not grow. How would a new id be added by a module ?
>
> > +static void (*leaf_init_fini_cbs[])(bool) = {
> > + group_leaf_init_fini,
> > + vsec_leaf_init_fini,
> > + devctl_leaf_init_fini,
> > + axigate_leaf_init_fini,
> > + icap_leaf_init_fini,
> > + calib_leaf_init_fini,
> > + clkfreq_leaf_init_fini,
> > + clock_leaf_init_fini,
> > + ucs_leaf_init_fini,
> > +};
> > +
> > +static __init int xrt_lib_init(void)
> > +{
> > + int i;
> > +
> > + xrt_class = class_create(THIS_MODULE, XRT_IPLIB_MODULE_NAME);
> > + if (IS_ERR(xrt_class))
> > + return PTR_ERR(xrt_class);
> > +
> > + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
> > + leaf_init_fini_cbs[i](true);
> > + return 0;
> > +}
> > +
> > +static __exit void xrt_lib_fini(void)
> > +{
> > + struct xrt_drv_map *map;
> > + int i;
> > +
> > + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
> > + leaf_init_fini_cbs[i](false);
> > +
> > + mutex_lock(&xrt_lib_lock);
> > +
> > + while (!list_empty(&xrt_drv_maps)) {
> > + map = list_first_entry_or_null(&xrt_drv_maps, struct xrt_drv_map, list);
> > + pr_err("Unloading module with %s still registered\n", XRT_DRVNAME(map->drv));
> > + list_del(&map->list);
> > + mutex_unlock(&xrt_lib_lock);
> > + xrt_drv_unregister_driver(map);
> > + vfree(map);
> > + mutex_lock(&xrt_lib_lock);
> > + }
> > +
> > + mutex_unlock(&xrt_lib_lock);
> > +
> > + class_destroy(xrt_class);
> > +}
> > +
> > +module_init(xrt_lib_init);
> > +module_exit(xrt_lib_fini);
> > +
> > +MODULE_VERSION(XRT_IPLIB_MODULE_VERSION);
> > +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
> > +MODULE_DESCRIPTION("Xilinx Alveo IP Lib driver");
> > +MODULE_LICENSE("GPL v2");
> > diff --git a/drivers/fpga/xrt/lib/main.h b/drivers/fpga/xrt/lib/main.h
> > new file mode 100644
> > index 000000000000..f3bfc87ee614
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/lib/main.h
> > @@ -0,0 +1,17 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Copyright (C) 2020-2021 Xilinx, Inc.
> > + *
> > + * Authors:
> > + * Cheng Zhen <maxz@xilinx.com>
> > + */
> > +
> > +#ifndef _XRT_MAIN_H_
> > +#define _XRT_MAIN_H_
> > +
> > +const char *xrt_drv_name(enum xrt_subdev_id id);
>
> To be self contained, the header defining enum xrt_subdev_id should be included.
>
> This is subdev_id.h which comes in with patch 6
>
> A dependency on a future patch breaks bisectablity.
>
> It may make sense to collect these small headers into a single large header for the ip infra lib and bring them all in this patch.
Please add the headers when you use them, do *not* do header only commits.
It's perfectly fine (and desirable) to add things over time to a header
as you use them.
Note *each* commit must compile and work standing on its own, so yes as
Tom pointed out, do not depend on future (later commit) files.
>
> Tom
>
> > +int xrt_drv_get_instance(enum xrt_subdev_id id);
> > +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance);
> > +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id);
> > +
> > +#endif /* _XRT_MAIN_H_ */
>
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager
2021-02-22 15:05 ` Tom Rix
2021-02-23 3:35 ` Moritz Fischer
@ 2021-03-03 17:20 ` Max Zhen
1 sibling, 0 replies; 87+ messages in thread
From: Max Zhen @ 2021-03-03 17:20 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
On 2/22/21 7:05 AM, Tom Rix wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> xrt-lib kernel module infrastructure code to register and manage all
>> leaf driver modules.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/lib/main.c | 274 ++++++++++++++++++++++++++++++++++++
>> drivers/fpga/xrt/lib/main.h | 17 +++
>> 2 files changed, 291 insertions(+)
>> create mode 100644 drivers/fpga/xrt/lib/main.c
>> create mode 100644 drivers/fpga/xrt/lib/main.h
> Not sure if 'main' is a good base name for something going into a lib.
These files are the main file for xrt-lib.ko. I'm not sure what name you
prefer, but I've changed them to lib-drv.[c|h]. Let me know if you don't
like them...
>
>> diff --git a/drivers/fpga/xrt/lib/main.c b/drivers/fpga/xrt/lib/main.c
>> new file mode 100644
>> index 000000000000..36fb62710843
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/main.c
>> @@ -0,0 +1,274 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Driver for Xilinx Alveo FPGA Support
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#include <linux/module.h>
>> +#include "xleaf.h"
>> +#include "xroot.h"
>> +#include "main.h"
>> +
>> +#define XRT_IPLIB_MODULE_NAME "xrt-lib"
>> +#define XRT_IPLIB_MODULE_VERSION "4.0.0"
>> +#define XRT_MAX_DEVICE_NODES 128
>> +#define XRT_DRVNAME(drv) ((drv)->driver.name)
>> +
>> +/*
>> + * Subdev driver is known by ID to others. We map the ID to it's
> by it's ID
Sure.
>> + * struct platform_driver, which contains it's binding name and driver/file ops.
>> + * We also map it to the endpoint name in DTB as well, if it's different
>> + * than the driver's binding name.
>> + */
>> +struct xrt_drv_map {
>> + struct list_head list;
>> + enum xrt_subdev_id id;
>> + struct platform_driver *drv;
>> + struct xrt_subdev_endpoints *eps;
>> + struct ida ida; /* manage driver instance and char dev minor */
>> +};
>> +
>> +static DEFINE_MUTEX(xrt_lib_lock); /* global lock protecting xrt_drv_maps list */
>> +static LIST_HEAD(xrt_drv_maps);
>> +struct class *xrt_class;
>> +
>> +static inline struct xrt_subdev_drvdata *
>> +xrt_drv_map2drvdata(struct xrt_drv_map *map)
>> +{
>> + return (struct xrt_subdev_drvdata *)map->drv->id_table[0].driver_data;
>> +}
>> +
>> +static struct xrt_drv_map *
>> +xrt_drv_find_map_by_id_nolock(enum xrt_subdev_id id)
> name could be by convention
>
> __xrt_drv_find_map_id
Sure.
>
>> +{
>> + const struct list_head *ptr;
>> +
>> + list_for_each(ptr, &xrt_drv_maps) {
>> + struct xrt_drv_map *tmap = list_entry(ptr, struct xrt_drv_map, list);
>> +
>> + if (tmap->id == id)
>> + return tmap;
>> + }
>> + return NULL;
>> +}
>> +
>> +static struct xrt_drv_map *
>> +xrt_drv_find_map_by_id(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map;
>> +
>> + mutex_lock(&xrt_lib_lock);
>> + map = xrt_drv_find_map_by_id_nolock(id);
>> + mutex_unlock(&xrt_lib_lock);
>> + /*
>> + * map should remain valid even after lock is dropped since a registered
> even after the lock
Sure.
>> + * driver should only be unregistered when driver module is being unloaded,
>> + * which means that the driver should not be used by then.
>> + */
>> + return map;
>> +}
>> +
>> +static int xrt_drv_register_driver(struct xrt_drv_map *map)
>> +{
>> + struct xrt_subdev_drvdata *drvdata;
>> + int rc = 0;
>> + const char *drvname = XRT_DRVNAME(map->drv);
>> +
>> + rc = platform_driver_register(map->drv);
>> + if (rc) {
>> + pr_err("register %s platform driver failed\n", drvname);
>> + return rc;
>> + }
>> +
>> + drvdata = xrt_drv_map2drvdata(map);
>> + if (drvdata) {
>> + /* Initialize dev_t for char dev node. */
>> + if (xleaf_devnode_enabled(drvdata)) {
>> + rc = alloc_chrdev_region(&drvdata->xsd_file_ops.xsf_dev_t, 0,
>> + XRT_MAX_DEVICE_NODES, drvname);
>> + if (rc) {
>> + platform_driver_unregister(map->drv);
>> + pr_err("failed to alloc dev minor for %s: %d\n", drvname, rc);
>> + return rc;
>> + }
>> + } else {
>> + drvdata->xsd_file_ops.xsf_dev_t = (dev_t)-1;
>> + }
>> + }
>> +
>> + ida_init(&map->ida);
>> +
>> + pr_info("%s registered successfully\n", drvname);
>> +
>> + return 0;
>> +}
>> +
>> +static void xrt_drv_unregister_driver(struct xrt_drv_map *map)
>> +{
>> + const char *drvname = XRT_DRVNAME(map->drv);
>> + struct xrt_subdev_drvdata *drvdata;
>> +
>> + ida_destroy(&map->ida);
>> +
>> + drvdata = xrt_drv_map2drvdata(map);
>> + if (drvdata && drvdata->xsd_file_ops.xsf_dev_t != (dev_t)-1) {
>> + unregister_chrdev_region(drvdata->xsd_file_ops.xsf_dev_t,
>> + XRT_MAX_DEVICE_NODES);
>> + }
>> +
>> + platform_driver_unregister(map->drv);
>> +
>> + pr_info("%s unregistered successfully\n", drvname);
>> +}
>> +
>> +int xleaf_register_driver(enum xrt_subdev_id id,
>> + struct platform_driver *drv,
>> + struct xrt_subdev_endpoints *eps)
>> +{
>> + struct xrt_drv_map *map;
>> +
>> + mutex_lock(&xrt_lib_lock);
> Trying to minimize length of lock being held.
>
> Could holding this lock be split or the alloc moved above ?
We don't want to do memory allocation unless the mapping can't be found.
So, finding the mapping and allocating memory for new entry is done as
one atomic operation.
The code path here is far from being in critical path, so holding a lock
here for some longer time should not be a problem. I'd like to keep the
code straightforward and easy to follow by holding the lock from start
to end.
>
>> +
>> + map = xrt_drv_find_map_by_id_nolock(id);
>> + if (map) {
>> + mutex_unlock(&xrt_lib_lock);
>> + pr_err("Id %d already has a registered driver, 0x%p\n",
>> + id, map->drv);
>> + return -EEXIST;
>> + }
>> +
>> + map = vzalloc(sizeof(*map));
> general issue
>
> map is small, so kzalloc
Sure.
>> + if (!map) {
>> + mutex_unlock(&xrt_lib_lock);
>> + return -ENOMEM;
>> + }
>> + map->id = id;
>> + map->drv = drv;
>> + map->eps = eps;
>> +
>> + xrt_drv_register_driver(map);
> xrt_drv_register_driver failure is unhandled.
Sure.
>
> This is the only time xrt_drv_register_driver is called, consider expanding the function here and removing the call.
Expanding the function here will result in > 3 level of indentation,
which is bad. I'd like to keep it as a function.
>> +
>> + list_add(&map->list, &xrt_drv_maps);
>> +
>> + mutex_unlock(&xrt_lib_lock);
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(xleaf_register_driver);
>> +
>> +void xleaf_unregister_driver(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map;
>> +
>> + mutex_lock(&xrt_lib_lock);
>> +
>> + map = xrt_drv_find_map_by_id_nolock(id);
>> + if (!map) {
>> + mutex_unlock(&xrt_lib_lock);
>> + pr_err("Id %d has no registered driver\n", id);
>> + return;
>> + }
>> +
>> + list_del(&map->list);
>> +
>> + mutex_unlock(&xrt_lib_lock);
>> +
>> + xrt_drv_unregister_driver(map);
>> + vfree(map);
>> +}
>> +EXPORT_SYMBOL_GPL(xleaf_unregister_driver);
>> +
>> +const char *xrt_drv_name(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
>> +
>> + if (map)
>> + return XRT_DRVNAME(map->drv);
>> + return NULL;
>> +}
>> +
>> +int xrt_drv_get_instance(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
>> +
>> + return ida_alloc_range(&map->ida, 0, XRT_MAX_DEVICE_NODES, GFP_KERNEL);
>> +}
>> +
>> +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance)
>> +{
>> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
>> +
>> + ida_free(&map->ida, instance);
>> +}
>> +
>> +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id)
>> +{
>> + struct xrt_drv_map *map = xrt_drv_find_map_by_id(id);
>> + struct xrt_subdev_endpoints *eps;
>> +
>> + eps = map ? map->eps : NULL;
>> + return eps;
>> +}
>> +
>> +/* Leaf driver's module init/fini callbacks. */
> These constructor/destructor calls needs to be more dynamic.
>
> calls are made even if there are no subdevices to go with the id's.
>
> Also this list can not grow. How would a new id be added by a module ?
We do not support dynamically adding drivers/IDs. All drivers needs to
be added statically after thoroughly tested. We do not intend to build
an open infrastructure to support random hardware and driver anyway.
>
>> +static void (*leaf_init_fini_cbs[])(bool) = {
>> + group_leaf_init_fini,
>> + vsec_leaf_init_fini,
>> + devctl_leaf_init_fini,
>> + axigate_leaf_init_fini,
>> + icap_leaf_init_fini,
>> + calib_leaf_init_fini,
>> + clkfreq_leaf_init_fini,
>> + clock_leaf_init_fini,
>> + ucs_leaf_init_fini,
>> +};
>> +
>> +static __init int xrt_lib_init(void)
>> +{
>> + int i;
>> +
>> + xrt_class = class_create(THIS_MODULE, XRT_IPLIB_MODULE_NAME);
>> + if (IS_ERR(xrt_class))
>> + return PTR_ERR(xrt_class);
>> +
>> + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
>> + leaf_init_fini_cbs[i](true);
>> + return 0;
>> +}
>> +
>> +static __exit void xrt_lib_fini(void)
>> +{
>> + struct xrt_drv_map *map;
>> + int i;
>> +
>> + for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
>> + leaf_init_fini_cbs[i](false);
>> +
>> + mutex_lock(&xrt_lib_lock);
>> +
>> + while (!list_empty(&xrt_drv_maps)) {
>> + map = list_first_entry_or_null(&xrt_drv_maps, struct xrt_drv_map, list);
>> + pr_err("Unloading module with %s still registered\n", XRT_DRVNAME(map->drv));
>> + list_del(&map->list);
>> + mutex_unlock(&xrt_lib_lock);
>> + xrt_drv_unregister_driver(map);
>> + vfree(map);
>> + mutex_lock(&xrt_lib_lock);
>> + }
>> +
>> + mutex_unlock(&xrt_lib_lock);
>> +
>> + class_destroy(xrt_class);
>> +}
>> +
>> +module_init(xrt_lib_init);
>> +module_exit(xrt_lib_fini);
>> +
>> +MODULE_VERSION(XRT_IPLIB_MODULE_VERSION);
>> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
>> +MODULE_DESCRIPTION("Xilinx Alveo IP Lib driver");
>> +MODULE_LICENSE("GPL v2");
>> diff --git a/drivers/fpga/xrt/lib/main.h b/drivers/fpga/xrt/lib/main.h
>> new file mode 100644
>> index 000000000000..f3bfc87ee614
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/main.h
>> @@ -0,0 +1,17 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_MAIN_H_
>> +#define _XRT_MAIN_H_
>> +
>> +const char *xrt_drv_name(enum xrt_subdev_id id);
> To be self contained, the header defining enum xrt_subdev_id should be included.
>
> This is subdev_id.h which comes in with patch 6
>
> A dependency on a future patch breaks bisectablity.
>
> It may make sense to collect these small headers into a single large header for the ip infra lib and bring them all in this patch.
Yes, we will reconsider where to put these headers in next patch set.
Thanks,
Max
>
> Tom
>
>> +int xrt_drv_get_instance(enum xrt_subdev_id id);
>> +void xrt_drv_put_instance(enum xrt_subdev_id id, int instance);
>> +struct xrt_subdev_endpoints *xrt_drv_get_endpoints(enum xrt_subdev_id id);
>> +
>> +#endif /* _XRT_MAIN_H_ */
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 05/18] fpga: xrt: group platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (3 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 04/18] fpga: xrt: xrt-lib platform driver manager Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-22 18:50 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 06/18] fpga: xrt: platform driver infrastructure Lizhi Hou
` (14 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
group driver that manages life cycle of a bunch of leaf driver instances
and bridges them with root.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/group.h | 27 ++++
drivers/fpga/xrt/lib/group.c | 265 +++++++++++++++++++++++++++++++
2 files changed, 292 insertions(+)
create mode 100644 drivers/fpga/xrt/include/group.h
create mode 100644 drivers/fpga/xrt/lib/group.c
diff --git a/drivers/fpga/xrt/include/group.h b/drivers/fpga/xrt/include/group.h
new file mode 100644
index 000000000000..1874cdd5120d
--- /dev/null
+++ b/drivers/fpga/xrt/include/group.h
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_GROUP_H_
+#define _XRT_GROUP_H_
+
+#include "xleaf.h"
+
+/*
+ * Group driver IOCTL calls.
+ */
+enum xrt_group_ioctl_cmd {
+ XRT_GROUP_GET_LEAF = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+ XRT_GROUP_PUT_LEAF,
+ XRT_GROUP_INIT_CHILDREN,
+ XRT_GROUP_FINI_CHILDREN,
+ XRT_GROUP_TRIGGER_EVENT,
+};
+
+#endif /* _XRT_GROUP_H_ */
diff --git a/drivers/fpga/xrt/lib/group.c b/drivers/fpga/xrt/lib/group.c
new file mode 100644
index 000000000000..6ba56eea479b
--- /dev/null
+++ b/drivers/fpga/xrt/lib/group.c
@@ -0,0 +1,265 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Group Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include "xleaf.h"
+#include "subdev_pool.h"
+#include "group.h"
+#include "metadata.h"
+#include "main.h"
+
+#define XRT_GRP "xrt_group"
+
+struct xrt_group {
+ struct platform_device *pdev;
+ struct xrt_subdev_pool leaves;
+ bool leaves_created;
+ struct mutex lock; /* lock for group */
+};
+
+static int xrt_grp_root_cb(struct device *dev, void *parg,
+ u32 cmd, void *arg)
+{
+ int rc;
+ struct platform_device *pdev =
+ container_of(dev, struct platform_device, dev);
+ struct xrt_group *xg = (struct xrt_group *)parg;
+
+ switch (cmd) {
+ case XRT_ROOT_GET_LEAF_HOLDERS: {
+ struct xrt_root_ioctl_get_holders *holders =
+ (struct xrt_root_ioctl_get_holders *)arg;
+ rc = xrt_subdev_pool_get_holders(&xg->leaves,
+ holders->xpigh_pdev,
+ holders->xpigh_holder_buf,
+ holders->xpigh_holder_buf_len);
+ break;
+ }
+ default:
+ /* Forward parent call to root. */
+ rc = xrt_subdev_root_request(pdev, cmd, arg);
+ break;
+ }
+
+ return rc;
+}
+
+static int xrt_grp_create_leaves(struct xrt_group *xg)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(xg->pdev);
+ enum xrt_subdev_id did;
+ struct xrt_subdev_endpoints *eps = NULL;
+ int ep_count = 0, i, ret = 0, failed = 0;
+ unsigned long mlen;
+ char *dtb, *grp_dtb = NULL;
+ const char *ep_name;
+
+ mutex_lock(&xg->lock);
+
+ if (xg->leaves_created) {
+ mutex_unlock(&xg->lock);
+ return -EEXIST;
+ }
+
+ xrt_info(xg->pdev, "bringing up leaves...");
+
+ /* Create all leaves based on dtb. */
+ if (!pdata)
+ goto bail;
+
+ mlen = xrt_md_size(DEV(xg->pdev), pdata->xsp_dtb);
+ if (mlen == XRT_MD_INVALID_LENGTH) {
+ xrt_err(xg->pdev, "invalid dtb, len %ld", mlen);
+ goto bail;
+ }
+
+ grp_dtb = vmalloc(mlen);
+ if (!grp_dtb)
+ goto bail;
+
+ memcpy(grp_dtb, pdata->xsp_dtb, mlen);
+ for (did = 0; did < XRT_SUBDEV_NUM;) {
+ eps = eps ? eps + 1 : xrt_drv_get_endpoints(did);
+ if (!eps || !eps->xse_names) {
+ did++;
+ eps = NULL;
+ continue;
+ }
+ ret = xrt_md_create(DEV(xg->pdev), &dtb);
+ if (ret) {
+ xrt_err(xg->pdev, "create md failed, drv %s",
+ xrt_drv_name(did));
+ failed++;
+ continue;
+ }
+ for (i = 0; eps->xse_names[i].ep_name ||
+ eps->xse_names[i].regmap_name; i++) {
+ ep_name = (char *)eps->xse_names[i].ep_name;
+ if (!ep_name) {
+ xrt_md_get_compatible_endpoint(DEV(xg->pdev),
+ grp_dtb,
+ eps->xse_names[i].regmap_name,
+ &ep_name);
+ }
+ if (!ep_name)
+ continue;
+
+ ret = xrt_md_copy_endpoint(DEV(xg->pdev),
+ dtb, grp_dtb, ep_name,
+ (char *)eps->xse_names[i].regmap_name,
+ NULL);
+ if (ret)
+ continue;
+ xrt_md_del_endpoint(DEV(xg->pdev), grp_dtb, ep_name,
+ (char *)eps->xse_names[i].regmap_name);
+ ep_count++;
+ }
+ if (ep_count >= eps->xse_min_ep) {
+ ret = xrt_subdev_pool_add(&xg->leaves, did,
+ xrt_grp_root_cb, xg, dtb);
+ eps = NULL;
+ if (ret < 0) {
+ failed++;
+ xrt_err(xg->pdev, "failed to create %s: %d",
+ xrt_drv_name(did), ret);
+ }
+ } else if (ep_count > 0) {
+ xrt_md_copy_all_endpoints(DEV(xg->pdev), grp_dtb, dtb);
+ }
+ vfree(dtb);
+ ep_count = 0;
+ }
+
+ xg->leaves_created = true;
+
+bail:
+ vfree(grp_dtb);
+ mutex_unlock(&xg->lock);
+
+ return failed == 0 ? 0 : -ECHILD;
+}
+
+static void xrt_grp_remove_leaves(struct xrt_group *xg)
+{
+ mutex_lock(&xg->lock);
+
+ if (!xg->leaves_created) {
+ mutex_unlock(&xg->lock);
+ return;
+ }
+
+ xrt_info(xg->pdev, "tearing down leaves...");
+ xrt_subdev_pool_fini(&xg->leaves);
+ xg->leaves_created = false;
+
+ mutex_unlock(&xg->lock);
+}
+
+static int xrt_grp_probe(struct platform_device *pdev)
+{
+ struct xrt_group *xg;
+
+ xrt_info(pdev, "probing...");
+
+ xg = devm_kzalloc(&pdev->dev, sizeof(*xg), GFP_KERNEL);
+ if (!xg)
+ return -ENOMEM;
+
+ xg->pdev = pdev;
+ mutex_init(&xg->lock);
+ xrt_subdev_pool_init(DEV(pdev), &xg->leaves);
+ platform_set_drvdata(pdev, xg);
+
+ return 0;
+}
+
+static int xrt_grp_remove(struct platform_device *pdev)
+{
+ struct xrt_group *xg = platform_get_drvdata(pdev);
+
+ xrt_info(pdev, "leaving...");
+ xrt_grp_remove_leaves(xg);
+ return 0;
+}
+
+static int xrt_grp_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ int rc = 0;
+ struct xrt_group *xg = platform_get_drvdata(pdev);
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ /* Simply forward to every child. */
+ xrt_subdev_pool_handle_event(&xg->leaves,
+ (struct xrt_event *)arg);
+ break;
+ case XRT_GROUP_GET_LEAF: {
+ struct xrt_root_ioctl_get_leaf *get_leaf =
+ (struct xrt_root_ioctl_get_leaf *)arg;
+
+ rc = xrt_subdev_pool_get(&xg->leaves, get_leaf->xpigl_match_cb,
+ get_leaf->xpigl_match_arg,
+ DEV(get_leaf->xpigl_pdev),
+ &get_leaf->xpigl_leaf);
+ break;
+ }
+ case XRT_GROUP_PUT_LEAF: {
+ struct xrt_root_ioctl_put_leaf *put_leaf =
+ (struct xrt_root_ioctl_put_leaf *)arg;
+
+ rc = xrt_subdev_pool_put(&xg->leaves, put_leaf->xpipl_leaf,
+ DEV(put_leaf->xpipl_pdev));
+ break;
+ }
+ case XRT_GROUP_INIT_CHILDREN:
+ rc = xrt_grp_create_leaves(xg);
+ break;
+ case XRT_GROUP_FINI_CHILDREN:
+ xrt_grp_remove_leaves(xg);
+ break;
+ case XRT_GROUP_TRIGGER_EVENT:
+ xrt_subdev_pool_trigger_event(&xg->leaves, (enum xrt_events)(uintptr_t)arg);
+ break;
+ default:
+ xrt_err(pdev, "unknown IOCTL cmd %d", cmd);
+ rc = -EINVAL;
+ break;
+ }
+ return rc;
+}
+
+static struct xrt_subdev_drvdata xrt_grp_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_grp_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_grp_id_table[] = {
+ { XRT_GRP, (kernel_ulong_t)&xrt_grp_data },
+ { },
+};
+
+static struct platform_driver xrt_group_driver = {
+ .driver = {
+ .name = XRT_GRP,
+ },
+ .probe = xrt_grp_probe,
+ .remove = xrt_grp_remove,
+ .id_table = xrt_grp_id_table,
+};
+
+void group_leaf_init_fini(bool init)
+{
+ if (init)
+ xleaf_register_driver(XRT_SUBDEV_GRP, &xrt_group_driver, NULL);
+ else
+ xleaf_unregister_driver(XRT_SUBDEV_GRP);
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 05/18] fpga: xrt: group platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 05/18] fpga: xrt: group platform driver Lizhi Hou
@ 2021-02-22 18:50 ` Tom Rix
2021-02-26 21:57 ` Max Zhen
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-22 18:50 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> group driver that manages life cycle of a bunch of leaf driver instances
> and bridges them with root.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/group.h | 27 ++++
> drivers/fpga/xrt/lib/group.c | 265 +++++++++++++++++++++++++++++++
> 2 files changed, 292 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/group.h
> create mode 100644 drivers/fpga/xrt/lib/group.c
>
> diff --git a/drivers/fpga/xrt/include/group.h b/drivers/fpga/xrt/include/group.h
> new file mode 100644
> index 000000000000..1874cdd5120d
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/group.h
> @@ -0,0 +1,27 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
A bit too generic, please add a description or remove.
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_GROUP_H_
> +#define _XRT_GROUP_H_
> +
> +#include "xleaf.h"
This is patch 6, consider comments on patch 4.
> +
> +/*
> + * Group driver IOCTL calls.
Are these really ioctl calls?
Seems more like messages between nodes in a tree.
Consider changing to better jagon, maybe ioctl -> msg
> + */
> +enum xrt_group_ioctl_cmd {
> + XRT_GROUP_GET_LEAF = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
XRT_LEAF_CUSTOM_BASE is a #define, while these are enums. To be consistent, the XRT_LEAF_CUSTOM_BASE should be an enum in xleaf, you can initialize it to 64 there.
> + XRT_GROUP_PUT_LEAF,
> + XRT_GROUP_INIT_CHILDREN,
> + XRT_GROUP_FINI_CHILDREN,
> + XRT_GROUP_TRIGGER_EVENT,
> +};
> +
> +#endif /* _XRT_GROUP_H_ */
> diff --git a/drivers/fpga/xrt/lib/group.c b/drivers/fpga/xrt/lib/group.c
> new file mode 100644
> index 000000000000..6ba56eea479b
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/group.c
> @@ -0,0 +1,265 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA Group Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include "xleaf.h"
> +#include "subdev_pool.h"
> +#include "group.h"
> +#include "metadata.h"
> +#include "main.h"
> +
> +#define XRT_GRP "xrt_group"
> +
> +struct xrt_group {
> + struct platform_device *pdev;
> + struct xrt_subdev_pool leaves;
> + bool leaves_created;
> + struct mutex lock; /* lock for group */
> +};
> +
> +static int xrt_grp_root_cb(struct device *dev, void *parg,
> + u32 cmd, void *arg)
could 'cmd' be some enum type ?
> +{
> + int rc;
> + struct platform_device *pdev =
> + container_of(dev, struct platform_device, dev);
> + struct xrt_group *xg = (struct xrt_group *)parg;
> +
> + switch (cmd) {
> + case XRT_ROOT_GET_LEAF_HOLDERS: {
> + struct xrt_root_ioctl_get_holders *holders =
> + (struct xrt_root_ioctl_get_holders *)arg;
> + rc = xrt_subdev_pool_get_holders(&xg->leaves,
> + holders->xpigh_pdev,
> + holders->xpigh_holder_buf,
> + holders->xpigh_holder_buf_len);
> + break;
> + }
> + default:
> + /* Forward parent call to root. */
> + rc = xrt_subdev_root_request(pdev, cmd, arg);
> + break;
> + }
> +
> + return rc;
> +}
> +
> +static int xrt_grp_create_leaves(struct xrt_group *xg)
> +{
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(xg->pdev);
> + enum xrt_subdev_id did;
> + struct xrt_subdev_endpoints *eps = NULL;
> + int ep_count = 0, i, ret = 0, failed = 0;
> + unsigned long mlen;
> + char *dtb, *grp_dtb = NULL;
> + const char *ep_name;
> +
> + mutex_lock(&xg->lock);
> +
> + if (xg->leaves_created) {
> + mutex_unlock(&xg->lock);
This happens should be programming error, so print out some error message
> + return -EEXIST;
> + }
> +
> + xrt_info(xg->pdev, "bringing up leaves...");
> +
> + /* Create all leaves based on dtb. */
> + if (!pdata)
> + goto bail;
move to above the lock and fail with something like -EINVAL
> +
> + mlen = xrt_md_size(DEV(xg->pdev), pdata->xsp_dtb);
> + if (mlen == XRT_MD_INVALID_LENGTH) {
> + xrt_err(xg->pdev, "invalid dtb, len %ld", mlen);
> + goto bail;
> + }
> +
> + grp_dtb = vmalloc(mlen);
> + if (!grp_dtb)
> + goto bail;
failed is only set in the loop. This is an unreported -ENOMEM
> +
> + memcpy(grp_dtb, pdata->xsp_dtb, mlen);
> + for (did = 0; did < XRT_SUBDEV_NUM;) {
why isn't the did incremented ?
> + eps = eps ? eps + 1 : xrt_drv_get_endpoints(did);
this assumes the enpoints are in an array and accessed serially.
this is fragile.
convert to using just the xrt_drv_get_endpoints() call
> + if (!eps || !eps->xse_names) {
> + did++;
> + eps = NULL;
> + continue;
> + }
> + ret = xrt_md_create(DEV(xg->pdev), &dtb);
> + if (ret) {
> + xrt_err(xg->pdev, "create md failed, drv %s",
> + xrt_drv_name(did));
> + failed++;
failed but no cleanup of earier successes
> + continue;
> + }
> + for (i = 0; eps->xse_names[i].ep_name ||
this assumes that xse_names[] always has a guard.
why not use xse_min_ep ?
> + eps->xse_names[i].regmap_name; i++) {
> + ep_name = (char *)eps->xse_names[i].ep_name;
> + if (!ep_name) {
> + xrt_md_get_compatible_endpoint(DEV(xg->pdev),
> + grp_dtb,
> + eps->xse_names[i].regmap_name,
> + &ep_name);
> + }
> + if (!ep_name)
> + continue;
> +
> + ret = xrt_md_copy_endpoint(DEV(xg->pdev),
> + dtb, grp_dtb, ep_name,
> + (char *)eps->xse_names[i].regmap_name,
> + NULL);
> + if (ret)
> + continue;
> + xrt_md_del_endpoint(DEV(xg->pdev), grp_dtb, ep_name,
> + (char *)eps->xse_names[i].regmap_name);
> + ep_count++;
> + }
> + if (ep_count >= eps->xse_min_ep) {
This only happens if all additions are successful.
> + ret = xrt_subdev_pool_add(&xg->leaves, did,
> + xrt_grp_root_cb, xg, dtb);
> + eps = NULL;
> + if (ret < 0) {
> + failed++;
> + xrt_err(xg->pdev, "failed to create %s: %d",
> + xrt_drv_name(did), ret);
> + }
> + } else if (ep_count > 0) {
> + xrt_md_copy_all_endpoints(DEV(xg->pdev), grp_dtb, dtb);
> + }
> + vfree(dtb);
> + ep_count = 0;
> + }
> +
> + xg->leaves_created = true;
This is true even if some failed ?
> +
> +bail:
> + vfree(grp_dtb);
> + mutex_unlock(&xg->lock);
> +
> + return failed == 0 ? 0 : -ECHILD;
> +}
> +
> +static void xrt_grp_remove_leaves(struct xrt_group *xg)
> +{
> + mutex_lock(&xg->lock);
> +
> + if (!xg->leaves_created) {
> + mutex_unlock(&xg->lock);
> + return;
> + }
> +
> + xrt_info(xg->pdev, "tearing down leaves...");
> + xrt_subdev_pool_fini(&xg->leaves);
partial failure above and the subdev_pool is not created ?
> + xg->leaves_created = false;
> +
> + mutex_unlock(&xg->lock);
> +}
> +
> +static int xrt_grp_probe(struct platform_device *pdev)
> +{
> + struct xrt_group *xg;
> +
> + xrt_info(pdev, "probing...");
> +
> + xg = devm_kzalloc(&pdev->dev, sizeof(*xg), GFP_KERNEL);
> + if (!xg)
> + return -ENOMEM;
> +
> + xg->pdev = pdev;
> + mutex_init(&xg->lock);
> + xrt_subdev_pool_init(DEV(pdev), &xg->leaves);
> + platform_set_drvdata(pdev, xg);
> +
> + return 0;
> +}
> +
> +static int xrt_grp_remove(struct platform_device *pdev)
> +{
> + struct xrt_group *xg = platform_get_drvdata(pdev);
> +
> + xrt_info(pdev, "leaving...");
> + xrt_grp_remove_leaves(xg);
lock ?
Tom
> + return 0;
> +}
> +
> +static int xrt_grp_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + int rc = 0;
> + struct xrt_group *xg = platform_get_drvdata(pdev);
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + /* Simply forward to every child. */
> + xrt_subdev_pool_handle_event(&xg->leaves,
> + (struct xrt_event *)arg);
> + break;
> + case XRT_GROUP_GET_LEAF: {
> + struct xrt_root_ioctl_get_leaf *get_leaf =
> + (struct xrt_root_ioctl_get_leaf *)arg;
> +
> + rc = xrt_subdev_pool_get(&xg->leaves, get_leaf->xpigl_match_cb,
> + get_leaf->xpigl_match_arg,
> + DEV(get_leaf->xpigl_pdev),
> + &get_leaf->xpigl_leaf);
> + break;
> + }
> + case XRT_GROUP_PUT_LEAF: {
> + struct xrt_root_ioctl_put_leaf *put_leaf =
> + (struct xrt_root_ioctl_put_leaf *)arg;
> +
> + rc = xrt_subdev_pool_put(&xg->leaves, put_leaf->xpipl_leaf,
> + DEV(put_leaf->xpipl_pdev));
> + break;
> + }
> + case XRT_GROUP_INIT_CHILDREN:
> + rc = xrt_grp_create_leaves(xg);
> + break;
> + case XRT_GROUP_FINI_CHILDREN:
> + xrt_grp_remove_leaves(xg);
> + break;
> + case XRT_GROUP_TRIGGER_EVENT:
> + xrt_subdev_pool_trigger_event(&xg->leaves, (enum xrt_events)(uintptr_t)arg);
> + break;
> + default:
> + xrt_err(pdev, "unknown IOCTL cmd %d", cmd);
> + rc = -EINVAL;
> + break;
> + }
> + return rc;
> +}
> +
> +static struct xrt_subdev_drvdata xrt_grp_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_grp_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_grp_id_table[] = {
> + { XRT_GRP, (kernel_ulong_t)&xrt_grp_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_group_driver = {
> + .driver = {
> + .name = XRT_GRP,
> + },
> + .probe = xrt_grp_probe,
> + .remove = xrt_grp_remove,
> + .id_table = xrt_grp_id_table,
> +};
> +
> +void group_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_GRP, &xrt_group_driver, NULL);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_GRP);
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 05/18] fpga: xrt: group platform driver
2021-02-22 18:50 ` Tom Rix
@ 2021-02-26 21:57 ` Max Zhen
0 siblings, 0 replies; 87+ messages in thread
From: Max Zhen @ 2021-02-26 21:57 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 2/22/21 10:50 AM, Tom Rix wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> group driver that manages life cycle of a bunch of leaf driver instances
>> and bridges them with root.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/group.h | 27 ++++
>> drivers/fpga/xrt/lib/group.c | 265 +++++++++++++++++++++++++++++++
>> 2 files changed, 292 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/group.h
>> create mode 100644 drivers/fpga/xrt/lib/group.c
>>
>> diff --git a/drivers/fpga/xrt/include/group.h b/drivers/fpga/xrt/include/group.h
>> new file mode 100644
>> index 000000000000..1874cdd5120d
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/group.h
>> @@ -0,0 +1,27 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for Xilinx Runtime (XRT) driver
> A bit too generic, please add a description or remove.
Will remove.
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_GROUP_H_
>> +#define _XRT_GROUP_H_
>> +
>> +#include "xleaf.h"
> This is patch 6, consider comments on patch 4.
Will move this header to other patch.
>> +
>> +/*
>> + * Group driver IOCTL calls.
> Are these really ioctl calls?
>
> Seems more like messages between nodes in a tree.
>
> Consider changing to better jagon, maybe ioctl -> msg
You're right. They are not really ioctl calls. They are just calls b/w
leaf nodes. I changed to leaf calls/commands.
>
>> + */
>> +enum xrt_group_ioctl_cmd {
>> + XRT_GROUP_GET_LEAF = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> XRT_LEAF_CUSTOM_BASE is a #define, while these are enums. To be consistent, the XRT_LEAF_CUSTOM_BASE should be an enum in xleaf, you can initialize it to 64 there.
Will fix.
>> + XRT_GROUP_PUT_LEAF,
>> + XRT_GROUP_INIT_CHILDREN,
>> + XRT_GROUP_FINI_CHILDREN,
>> + XRT_GROUP_TRIGGER_EVENT,
>> +};
>> +
>> +#endif /* _XRT_GROUP_H_ */
>> diff --git a/drivers/fpga/xrt/lib/group.c b/drivers/fpga/xrt/lib/group.c
>> new file mode 100644
>> index 000000000000..6ba56eea479b
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/group.c
>> @@ -0,0 +1,265 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA Group Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include "xleaf.h"
>> +#include "subdev_pool.h"
>> +#include "group.h"
>> +#include "metadata.h"
>> +#include "main.h"
>> +
>> +#define XRT_GRP "xrt_group"
>> +
>> +struct xrt_group {
>> + struct platform_device *pdev;
>> + struct xrt_subdev_pool leaves;
>> + bool leaves_created;
>> + struct mutex lock; /* lock for group */
>> +};
>> +
>> +static int xrt_grp_root_cb(struct device *dev, void *parg,
>> + u32 cmd, void *arg)
> could 'cmd' be some enum type ?
Will fix.
>> +{
>> + int rc;
>> + struct platform_device *pdev =
>> + container_of(dev, struct platform_device, dev);
>> + struct xrt_group *xg = (struct xrt_group *)parg;
>> +
>> + switch (cmd) {
>> + case XRT_ROOT_GET_LEAF_HOLDERS: {
>> + struct xrt_root_ioctl_get_holders *holders =
>> + (struct xrt_root_ioctl_get_holders *)arg;
>> + rc = xrt_subdev_pool_get_holders(&xg->leaves,
>> + holders->xpigh_pdev,
>> + holders->xpigh_holder_buf,
>> + holders->xpigh_holder_buf_len);
>> + break;
>> + }
>> + default:
>> + /* Forward parent call to root. */
>> + rc = xrt_subdev_root_request(pdev, cmd, arg);
>> + break;
>> + }
>> +
>> + return rc;
>> +}
>> +
>> +static int xrt_grp_create_leaves(struct xrt_group *xg)
>> +{
>> + struct xrt_subdev_platdata *pdata = DEV_PDATA(xg->pdev);
>> + enum xrt_subdev_id did;
>> + struct xrt_subdev_endpoints *eps = NULL;
>> + int ep_count = 0, i, ret = 0, failed = 0;
>> + unsigned long mlen;
>> + char *dtb, *grp_dtb = NULL;
>> + const char *ep_name;
>> +
>> + mutex_lock(&xg->lock);
>> +
>> + if (xg->leaves_created) {
>> + mutex_unlock(&xg->lock);
> This happens should be programming error, so print out some error message
The root driver does not remember which group has created leaves or not.
It'll just blindly make the call. The group driver itself should
remember and tell the root that it's done already or not. So, this is
not considered as an error.
>> + return -EEXIST;
>> + }
>> +
>> + xrt_info(xg->pdev, "bringing up leaves...");
>> +
>> + /* Create all leaves based on dtb. */
>> + if (!pdata)
>> + goto bail;
> move to above the lock and fail with something like -EINVAL
Will fix.
>> +
>> + mlen = xrt_md_size(DEV(xg->pdev), pdata->xsp_dtb);
>> + if (mlen == XRT_MD_INVALID_LENGTH) {
>> + xrt_err(xg->pdev, "invalid dtb, len %ld", mlen);
>> + goto bail;
>> + }
>> +
>> + grp_dtb = vmalloc(mlen);
>> + if (!grp_dtb)
>> + goto bail;
> failed is only set in the loop. This is an unreported -ENOMEM
Will fix.
>> +
>> + memcpy(grp_dtb, pdata->xsp_dtb, mlen);
>> + for (did = 0; did < XRT_SUBDEV_NUM;) {
> why isn't the did incremented ?
This is not a bug, but I will fix it by rewriting this function so that
it'll be easier to follow (hopefully :-)).
>> + eps = eps ? eps + 1 : xrt_drv_get_endpoints(did);
> this assumes the enpoints are in an array and accessed serially.
>
> this is fragile.
>
> convert to using just the xrt_drv_get_endpoints() call
It does not seem to be fragile? The endpoint data structure is private
data structure so it has to be an array as defined. It's usage is just
like an ID table as we have in other PCIE drivers.
>
>> + if (!eps || !eps->xse_names) {
>> + did++;
>> + eps = NULL;
>> + continue;
>> + }
>> + ret = xrt_md_create(DEV(xg->pdev), &dtb);
>> + if (ret) {
>> + xrt_err(xg->pdev, "create md failed, drv %s",
>> + xrt_drv_name(did));
>> + failed++;
> failed but no cleanup of earier successes
The leaves already created will still be there until the group is
destroyed. This is not considered as fatal error.
>> + continue;
>> + }
>> + for (i = 0; eps->xse_names[i].ep_name ||
> this assumes that xse_names[] always has a guard.
>
> why not use xse_min_ep ?
xse_min_ep is to tell caller the minimum number of end points needs to
be collected before instantiate the leaf driver. it does not indicate
exactly how many end points this leaf can handle (it might be able to
handle more). So, the guard is necessary to tell caller total number of
end points it can handle.
>
>> + eps->xse_names[i].regmap_name; i++) {
>> + ep_name = (char *)eps->xse_names[i].ep_name;
>> + if (!ep_name) {
>> + xrt_md_get_compatible_endpoint(DEV(xg->pdev),
>> + grp_dtb,
>> + eps->xse_names[i].regmap_name,
>> + &ep_name);
>> + }
>> + if (!ep_name)
>> + continue;
>> +
>> + ret = xrt_md_copy_endpoint(DEV(xg->pdev),
>> + dtb, grp_dtb, ep_name,
>> + (char *)eps->xse_names[i].regmap_name,
>> + NULL);
>> + if (ret)
>> + continue;
>> + xrt_md_del_endpoint(DEV(xg->pdev), grp_dtb, ep_name,
>> + (char *)eps->xse_names[i].regmap_name);
>> + ep_count++;
>> + }
>> + if (ep_count >= eps->xse_min_ep) {
> This only happens if all additions are successful.
This happens if the minimum number of end points has been collected.
Anyway, the function is going to be rewritten.
>> + ret = xrt_subdev_pool_add(&xg->leaves, did,
>> + xrt_grp_root_cb, xg, dtb);
>> + eps = NULL;
>> + if (ret < 0) {
>> + failed++;
>> + xrt_err(xg->pdev, "failed to create %s: %d",
>> + xrt_drv_name(did), ret);
>> + }
>> + } else if (ep_count > 0) {
>> + xrt_md_copy_all_endpoints(DEV(xg->pdev), grp_dtb, dtb);
>> + }
>> + vfree(dtb);
>> + ep_count = 0;
>> + }
>> +
>> + xg->leaves_created = true;
> This is true even if some failed ?
Yes, this is OK. Some functionality may be missing if some leaves can be
instantiated, but the group will be in leaves created state now.
>> +
>> +bail:
>> + vfree(grp_dtb);
>> + mutex_unlock(&xg->lock);
>> +
>> + return failed == 0 ? 0 : -ECHILD;
>> +}
>> +
>> +static void xrt_grp_remove_leaves(struct xrt_group *xg)
>> +{
>> + mutex_lock(&xg->lock);
>> +
>> + if (!xg->leaves_created) {
>> + mutex_unlock(&xg->lock);
>> + return;
>> + }
>> +
>> + xrt_info(xg->pdev, "tearing down leaves...");
>> + xrt_subdev_pool_fini(&xg->leaves);
> partial failure above and the subdev_pool is not created ?
It is still created, just not fully populated.
>> + xg->leaves_created = false;
>> +
>> + mutex_unlock(&xg->lock);
>> +}
>> +
>> +static int xrt_grp_probe(struct platform_device *pdev)
>> +{
>> + struct xrt_group *xg;
>> +
>> + xrt_info(pdev, "probing...");
>> +
>> + xg = devm_kzalloc(&pdev->dev, sizeof(*xg), GFP_KERNEL);
>> + if (!xg)
>> + return -ENOMEM;
>> +
>> + xg->pdev = pdev;
>> + mutex_init(&xg->lock);
>> + xrt_subdev_pool_init(DEV(pdev), &xg->leaves);
>> + platform_set_drvdata(pdev, xg);
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_grp_remove(struct platform_device *pdev)
>> +{
>> + struct xrt_group *xg = platform_get_drvdata(pdev);
>> +
>> + xrt_info(pdev, "leaving...");
>> + xrt_grp_remove_leaves(xg);
> lock ?
xrt_grp_remove_leaves() takes lock internally.
Thanks,
Max
>
> Tom
>
>> + return 0;
>> +}
>> +
>> +static int xrt_grp_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + int rc = 0;
>> + struct xrt_group *xg = platform_get_drvdata(pdev);
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + /* Simply forward to every child. */
>> + xrt_subdev_pool_handle_event(&xg->leaves,
>> + (struct xrt_event *)arg);
>> + break;
>> + case XRT_GROUP_GET_LEAF: {
>> + struct xrt_root_ioctl_get_leaf *get_leaf =
>> + (struct xrt_root_ioctl_get_leaf *)arg;
>> +
>> + rc = xrt_subdev_pool_get(&xg->leaves, get_leaf->xpigl_match_cb,
>> + get_leaf->xpigl_match_arg,
>> + DEV(get_leaf->xpigl_pdev),
>> + &get_leaf->xpigl_leaf);
>> + break;
>> + }
>> + case XRT_GROUP_PUT_LEAF: {
>> + struct xrt_root_ioctl_put_leaf *put_leaf =
>> + (struct xrt_root_ioctl_put_leaf *)arg;
>> +
>> + rc = xrt_subdev_pool_put(&xg->leaves, put_leaf->xpipl_leaf,
>> + DEV(put_leaf->xpipl_pdev));
>> + break;
>> + }
>> + case XRT_GROUP_INIT_CHILDREN:
>> + rc = xrt_grp_create_leaves(xg);
>> + break;
>> + case XRT_GROUP_FINI_CHILDREN:
>> + xrt_grp_remove_leaves(xg);
>> + break;
>> + case XRT_GROUP_TRIGGER_EVENT:
>> + xrt_subdev_pool_trigger_event(&xg->leaves, (enum xrt_events)(uintptr_t)arg);
>> + break;
>> + default:
>> + xrt_err(pdev, "unknown IOCTL cmd %d", cmd);
>> + rc = -EINVAL;
>> + break;
>> + }
>> + return rc;
>> +}
>> +
>> +static struct xrt_subdev_drvdata xrt_grp_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_grp_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_grp_id_table[] = {
>> + { XRT_GRP, (kernel_ulong_t)&xrt_grp_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_group_driver = {
>> + .driver = {
>> + .name = XRT_GRP,
>> + },
>> + .probe = xrt_grp_probe,
>> + .remove = xrt_grp_remove,
>> + .id_table = xrt_grp_id_table,
>> +};
>> +
>> +void group_leaf_init_fini(bool init)
>> +{
>> + if (init)
>> + xleaf_register_driver(XRT_SUBDEV_GRP, &xrt_group_driver, NULL);
>> + else
>> + xleaf_unregister_driver(XRT_SUBDEV_GRP);
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 06/18] fpga: xrt: platform driver infrastructure
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (4 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 05/18] fpga: xrt: group platform driver Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-25 21:59 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root) Lizhi Hou
` (13 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
infrastructure code providing APIs for managing leaf driver instance
groups, facilitating inter-leaf driver calls and root calls, managing leaf
driver device nodes.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/events.h | 48 ++
drivers/fpga/xrt/include/subdev_id.h | 43 ++
drivers/fpga/xrt/include/xleaf.h | 276 +++++++++
drivers/fpga/xrt/lib/cdev.c | 231 +++++++
drivers/fpga/xrt/lib/subdev.c | 871 +++++++++++++++++++++++++++
drivers/fpga/xrt/lib/subdev_pool.h | 53 ++
drivers/fpga/xrt/lib/xroot.c | 598 ++++++++++++++++++
7 files changed, 2120 insertions(+)
create mode 100644 drivers/fpga/xrt/include/events.h
create mode 100644 drivers/fpga/xrt/include/subdev_id.h
create mode 100644 drivers/fpga/xrt/include/xleaf.h
create mode 100644 drivers/fpga/xrt/lib/cdev.c
create mode 100644 drivers/fpga/xrt/lib/subdev.c
create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
create mode 100644 drivers/fpga/xrt/lib/xroot.c
diff --git a/drivers/fpga/xrt/include/events.h b/drivers/fpga/xrt/include/events.h
new file mode 100644
index 000000000000..2a9aae8bceb4
--- /dev/null
+++ b/drivers/fpga/xrt/include/events.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_EVENTS_H_
+#define _XRT_EVENTS_H_
+
+#include <linux/platform_device.h>
+#include "subdev_id.h"
+
+/*
+ * Event notification.
+ */
+enum xrt_events {
+ XRT_EVENT_TEST = 0, /* for testing */
+ /*
+ * Events related to specific subdev
+ * Callback arg: struct xrt_event_arg_subdev
+ */
+ XRT_EVENT_POST_CREATION,
+ XRT_EVENT_PRE_REMOVAL,
+ /*
+ * Events related to change of the whole board
+ * Callback arg: <none>
+ */
+ XRT_EVENT_PRE_HOT_RESET,
+ XRT_EVENT_POST_HOT_RESET,
+ XRT_EVENT_PRE_GATE_CLOSE,
+ XRT_EVENT_POST_GATE_OPEN,
+};
+
+struct xrt_event_arg_subdev {
+ enum xrt_subdev_id xevt_subdev_id;
+ int xevt_subdev_instance;
+};
+
+struct xrt_event {
+ enum xrt_events xe_evt;
+ struct xrt_event_arg_subdev xe_subdev;
+};
+
+#endif /* _XRT_EVENTS_H_ */
diff --git a/drivers/fpga/xrt/include/subdev_id.h b/drivers/fpga/xrt/include/subdev_id.h
new file mode 100644
index 000000000000..6205a9f26196
--- /dev/null
+++ b/drivers/fpga/xrt/include/subdev_id.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_SUBDEV_ID_H_
+#define _XRT_SUBDEV_ID_H_
+
+/*
+ * Every subdev driver should have an ID for others to refer to it.
+ * There can be unlimited number of instances of a subdev driver. A
+ * <subdev_id, subdev_instance> tuple should be a unique identification of
+ * a specific instance of a subdev driver.
+ * NOTE: PLEASE do not change the order of IDs. Sub devices in the same
+ * group are initialized by this order.
+ */
+enum xrt_subdev_id {
+ XRT_SUBDEV_GRP = 0,
+ XRT_SUBDEV_VSEC = 1,
+ XRT_SUBDEV_VSEC_GOLDEN = 2,
+ XRT_SUBDEV_DEVCTL = 3,
+ XRT_SUBDEV_AXIGATE = 4,
+ XRT_SUBDEV_ICAP = 5,
+ XRT_SUBDEV_TEST = 6,
+ XRT_SUBDEV_MGMT_MAIN = 7,
+ XRT_SUBDEV_QSPI = 8,
+ XRT_SUBDEV_MAILBOX = 9,
+ XRT_SUBDEV_CMC = 10,
+ XRT_SUBDEV_CALIB = 11,
+ XRT_SUBDEV_CLKFREQ = 12,
+ XRT_SUBDEV_CLOCK = 13,
+ XRT_SUBDEV_SRSR = 14,
+ XRT_SUBDEV_UCS = 15,
+ XRT_SUBDEV_NUM = 16, /* Total number of subdevs. */
+ XRT_ROOT = -1, /* Special ID for root driver. */
+};
+
+#endif /* _XRT_SUBDEV_ID_H_ */
diff --git a/drivers/fpga/xrt/include/xleaf.h b/drivers/fpga/xrt/include/xleaf.h
new file mode 100644
index 000000000000..10215a75d474
--- /dev/null
+++ b/drivers/fpga/xrt/include/xleaf.h
@@ -0,0 +1,276 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ * Sonal Santan <sonal.santan@xilinx.com>
+ */
+
+#ifndef _XRT_XLEAF_H_
+#define _XRT_XLEAF_H_
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/fs.h>
+#include <linux/cdev.h>
+#include <linux/pci.h>
+#include <linux/libfdt_env.h>
+#include "libfdt.h"
+#include "subdev_id.h"
+#include "xroot.h"
+#include "events.h"
+
+/* All subdev drivers should use below common routines to print out msg. */
+#define DEV(pdev) (&(pdev)->dev)
+#define DEV_PDATA(pdev) \
+ ((struct xrt_subdev_platdata *)dev_get_platdata(DEV(pdev)))
+#define DEV_DRVDATA(pdev) \
+ ((struct xrt_subdev_drvdata *) \
+ platform_get_device_id(pdev)->driver_data)
+#define FMT_PRT(prt_fn, pdev, fmt, args...) \
+ ({typeof(pdev) (_pdev) = (pdev); \
+ prt_fn(DEV(_pdev), "%s %s: " fmt, \
+ DEV_PDATA(_pdev)->xsp_root_name, __func__, ##args); })
+#define xrt_err(pdev, fmt, args...) FMT_PRT(dev_err, pdev, fmt, ##args)
+#define xrt_warn(pdev, fmt, args...) FMT_PRT(dev_warn, pdev, fmt, ##args)
+#define xrt_info(pdev, fmt, args...) FMT_PRT(dev_info, pdev, fmt, ##args)
+#define xrt_dbg(pdev, fmt, args...) FMT_PRT(dev_dbg, pdev, fmt, ##args)
+
+/* Starting IOCTL for common IOCTLs implemented by all leaves. */
+#define XRT_XLEAF_COMMON_BASE 0
+/* Starting IOCTL for leaves' specific IOCTLs. */
+#define XRT_XLEAF_CUSTOM_BASE 64
+enum xrt_xleaf_common_ioctl_cmd {
+ XRT_XLEAF_EVENT = XRT_XLEAF_COMMON_BASE,
+};
+
+/*
+ * If populated by subdev driver, infra will handle the mechanics of
+ * char device (un)registration.
+ */
+enum xrt_subdev_file_mode {
+ /* Infra create cdev, default file name */
+ XRT_SUBDEV_FILE_DEFAULT = 0,
+ /* Infra create cdev, need to encode inst num in file name */
+ XRT_SUBDEV_FILE_MULTI_INST,
+ /* No auto creation of cdev by infra, leaf handles it by itself */
+ XRT_SUBDEV_FILE_NO_AUTO,
+};
+
+struct xrt_subdev_file_ops {
+ const struct file_operations xsf_ops;
+ dev_t xsf_dev_t;
+ const char *xsf_dev_name;
+ enum xrt_subdev_file_mode xsf_mode;
+};
+
+/*
+ * Subdev driver callbacks populated by subdev driver.
+ */
+struct xrt_subdev_drv_ops {
+ /*
+ * Per driver instance callback. The pdev points to the instance.
+ * If defined these are called by other leaf drivers.
+ * Note that root driver may call into xsd_ioctl of a group driver.
+ */
+ int (*xsd_ioctl)(struct platform_device *pdev, u32 cmd, void *arg);
+};
+
+/*
+ * Defined and populated by subdev driver, exported as driver_data in
+ * struct platform_device_id.
+ */
+struct xrt_subdev_drvdata {
+ struct xrt_subdev_file_ops xsd_file_ops;
+ struct xrt_subdev_drv_ops xsd_dev_ops;
+};
+
+/*
+ * Partially initialized by the parent driver, then, passed in as subdev driver's
+ * platform data when creating subdev driver instance by calling platform
+ * device register API (platform_device_register_data() or the likes).
+ *
+ * Once device register API returns, platform driver framework makes a copy of
+ * this buffer and maintains its life cycle. The content of the buffer is
+ * completely owned by subdev driver.
+ *
+ * Thus, parent driver should be very careful when it touches this buffer
+ * again once it's handed over to subdev driver. And the data structure
+ * should not contain pointers pointing to buffers that is managed by
+ * other or parent drivers since it could have been freed before platform
+ * data buffer is freed by platform driver framework.
+ */
+struct xrt_subdev_platdata {
+ /*
+ * Per driver instance callback. The pdev points to the instance.
+ * Should always be defined for subdev driver to get service from root.
+ */
+ xrt_subdev_root_cb_t xsp_root_cb;
+ void *xsp_root_cb_arg;
+
+ /* Something to associate w/ root for msg printing. */
+ const char *xsp_root_name;
+
+ /*
+ * Char dev support for this subdev instance.
+ * Initialized by subdev driver.
+ */
+ struct cdev xsp_cdev;
+ struct device *xsp_sysdev;
+ struct mutex xsp_devnode_lock; /* devnode lock */
+ struct completion xsp_devnode_comp;
+ int xsp_devnode_ref;
+ bool xsp_devnode_online;
+ bool xsp_devnode_excl;
+
+ /*
+ * Subdev driver specific init data. The buffer should be embedded
+ * in this data structure buffer after dtb, so that it can be freed
+ * together with platform data.
+ */
+ loff_t xsp_priv_off; /* Offset into this platform data buffer. */
+ size_t xsp_priv_len;
+
+ /*
+ * Populated by parent driver to describe the device tree for
+ * the subdev driver to handle. Should always be last one since it's
+ * of variable length.
+ */
+ char xsp_dtb[sizeof(struct fdt_header)];
+};
+
+/*
+ * this struct define the endpoints belong to the same subdevice
+ */
+struct xrt_subdev_ep_names {
+ const char *ep_name;
+ const char *regmap_name;
+};
+
+struct xrt_subdev_endpoints {
+ struct xrt_subdev_ep_names *xse_names;
+ /* minimum number of endpoints to support the subdevice */
+ u32 xse_min_ep;
+};
+
+struct subdev_match_arg {
+ enum xrt_subdev_id id;
+ int instance;
+};
+
+bool xleaf_has_endpoint(struct platform_device *pdev, const char *endpoint_name);
+struct platform_device *xleaf_get_leaf(struct platform_device *pdev,
+ xrt_subdev_match_t cb, void *arg);
+
+static inline bool subdev_match(enum xrt_subdev_id id, struct platform_device *pdev, void *arg)
+{
+ const struct subdev_match_arg *a = (struct subdev_match_arg *)arg;
+ bool ret = (id == a->id && (pdev->id == a->instance || PLATFORM_DEVID_NONE == a->instance));
+
+ return ret;
+}
+
+static inline bool xrt_subdev_match_epname(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ return xleaf_has_endpoint(pdev, arg);
+}
+
+static inline struct platform_device *
+xleaf_get_leaf_by_id(struct platform_device *pdev,
+ enum xrt_subdev_id id, int instance)
+{
+ struct subdev_match_arg arg = { id, instance };
+
+ return xleaf_get_leaf(pdev, subdev_match, &arg);
+}
+
+static inline struct platform_device *
+xleaf_get_leaf_by_epname(struct platform_device *pdev, const char *name)
+{
+ return xleaf_get_leaf(pdev, xrt_subdev_match_epname, (void *)name);
+}
+
+static inline int xleaf_ioctl(struct platform_device *tgt, u32 cmd, void *arg)
+{
+ struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(tgt);
+
+ return (*drvdata->xsd_dev_ops.xsd_ioctl)(tgt, cmd, arg);
+}
+
+int xleaf_put_leaf(struct platform_device *pdev,
+ struct platform_device *leaf);
+int xleaf_create_group(struct platform_device *pdev, char *dtb);
+int xleaf_destroy_group(struct platform_device *pdev, int instance);
+int xleaf_wait_for_group_bringup(struct platform_device *pdev);
+void xleaf_hot_reset(struct platform_device *pdev);
+int xleaf_broadcast_event(struct platform_device *pdev,
+ enum xrt_events evt, bool async);
+void xleaf_get_barres(struct platform_device *pdev,
+ struct resource **res, uint bar_idx);
+void xleaf_get_root_id(struct platform_device *pdev,
+ unsigned short *vendor, unsigned short *device,
+ unsigned short *subvendor, unsigned short *subdevice);
+struct device *xleaf_register_hwmon(struct platform_device *pdev,
+ const char *name, void *drvdata,
+ const struct attribute_group **grps);
+void xleaf_unregister_hwmon(struct platform_device *pdev, struct device *hwmon);
+
+/*
+ * Character device helper APIs for use by leaf drivers
+ */
+static inline bool xleaf_devnode_enabled(struct xrt_subdev_drvdata *drvdata)
+{
+ return drvdata && drvdata->xsd_file_ops.xsf_ops.open;
+}
+
+int xleaf_devnode_create(struct platform_device *pdev,
+ const char *file_name, const char *inst_name);
+int xleaf_devnode_destroy(struct platform_device *pdev);
+
+struct platform_device *xleaf_devnode_open_excl(struct inode *inode);
+struct platform_device *xleaf_devnode_open(struct inode *inode);
+void xleaf_devnode_close(struct inode *inode);
+
+/* Helpers. */
+static inline void xrt_memcpy_fromio(void *buf, void __iomem *iomem, u32 size)
+{
+ int i;
+
+ WARN_ON(size & 0x3);
+ for (i = 0; i < size / 4; i++)
+ ((u32 *)buf)[i] = ioread32((char *)(iomem) + sizeof(u32) * i);
+}
+
+static inline void xrt_memcpy_toio(void __iomem *iomem, void *buf, u32 size)
+{
+ int i;
+
+ WARN_ON(size & 0x3);
+ for (i = 0; i < size / 4; i++)
+ iowrite32(((u32 *)buf)[i], ((char *)(iomem) + sizeof(u32) * i));
+}
+
+int xleaf_register_driver(enum xrt_subdev_id id, struct platform_driver *drv,
+ struct xrt_subdev_endpoints *eps);
+void xleaf_unregister_driver(enum xrt_subdev_id id);
+
+/* Module's init/fini routines for leaf driver in xrt-lib module */
+void group_leaf_init_fini(bool init);
+void vsec_leaf_init_fini(bool init);
+void vsec_golden_leaf_init_fini(bool init);
+void devctl_leaf_init_fini(bool init);
+void axigate_leaf_init_fini(bool init);
+void icap_leaf_init_fini(bool init);
+void calib_leaf_init_fini(bool init);
+void qspi_leaf_init_fini(bool init);
+void mailbox_leaf_init_fini(bool init);
+void cmc_leaf_init_fini(bool init);
+void clkfreq_leaf_init_fini(bool init);
+void clock_leaf_init_fini(bool init);
+void ucs_leaf_init_fini(bool init);
+
+#endif /* _XRT_LEAF_H_ */
diff --git a/drivers/fpga/xrt/lib/cdev.c b/drivers/fpga/xrt/lib/cdev.c
new file mode 100644
index 000000000000..7f104ab3d527
--- /dev/null
+++ b/drivers/fpga/xrt/lib/cdev.c
@@ -0,0 +1,231 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA device node helper functions.
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include "xleaf.h"
+
+extern struct class *xrt_class;
+
+#define XRT_CDEV_DIR "xfpga"
+#define INODE2PDATA(inode) \
+ container_of((inode)->i_cdev, struct xrt_subdev_platdata, xsp_cdev)
+#define INODE2PDEV(inode) \
+ to_platform_device(kobj_to_dev((inode)->i_cdev->kobj.parent))
+#define CDEV_NAME(sysdev) (strchr((sysdev)->kobj.name, '!') + 1)
+
+/* Allow it to be accessed from cdev. */
+static void xleaf_devnode_allowed(struct platform_device *pdev)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+
+ /* Allow new opens. */
+ mutex_lock(&pdata->xsp_devnode_lock);
+ pdata->xsp_devnode_online = true;
+ mutex_unlock(&pdata->xsp_devnode_lock);
+}
+
+/* Turn off access from cdev and wait for all existing user to go away. */
+static int xleaf_devnode_disallowed(struct platform_device *pdev)
+{
+ int ret = 0;
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+
+ mutex_lock(&pdata->xsp_devnode_lock);
+
+ /* Prevent new opens. */
+ pdata->xsp_devnode_online = false;
+ /* Wait for existing user to close. */
+ while (!ret && pdata->xsp_devnode_ref) {
+ int rc;
+
+ mutex_unlock(&pdata->xsp_devnode_lock);
+ rc = wait_for_completion_killable(&pdata->xsp_devnode_comp);
+ mutex_lock(&pdata->xsp_devnode_lock);
+
+ if (rc == -ERESTARTSYS) {
+ /* Restore online state. */
+ pdata->xsp_devnode_online = true;
+ xrt_err(pdev, "%s is in use, ref=%d",
+ CDEV_NAME(pdata->xsp_sysdev),
+ pdata->xsp_devnode_ref);
+ ret = -EBUSY;
+ }
+ }
+
+ mutex_unlock(&pdata->xsp_devnode_lock);
+
+ return ret;
+}
+
+static struct platform_device *
+__xleaf_devnode_open(struct inode *inode, bool excl)
+{
+ struct xrt_subdev_platdata *pdata = INODE2PDATA(inode);
+ struct platform_device *pdev = INODE2PDEV(inode);
+ bool opened = false;
+
+ mutex_lock(&pdata->xsp_devnode_lock);
+
+ if (pdata->xsp_devnode_online) {
+ if (excl && pdata->xsp_devnode_ref) {
+ xrt_err(pdev, "%s has already been opened exclusively",
+ CDEV_NAME(pdata->xsp_sysdev));
+ } else if (!excl && pdata->xsp_devnode_excl) {
+ xrt_err(pdev, "%s has been opened exclusively",
+ CDEV_NAME(pdata->xsp_sysdev));
+ } else {
+ pdata->xsp_devnode_ref++;
+ pdata->xsp_devnode_excl = excl;
+ opened = true;
+ xrt_info(pdev, "opened %s, ref=%d",
+ CDEV_NAME(pdata->xsp_sysdev),
+ pdata->xsp_devnode_ref);
+ }
+ } else {
+ xrt_err(pdev, "%s is offline", CDEV_NAME(pdata->xsp_sysdev));
+ }
+
+ mutex_unlock(&pdata->xsp_devnode_lock);
+
+ pdev = opened ? pdev : NULL;
+ return pdev;
+}
+
+struct platform_device *
+xleaf_devnode_open_excl(struct inode *inode)
+{
+ return __xleaf_devnode_open(inode, true);
+}
+
+struct platform_device *
+xleaf_devnode_open(struct inode *inode)
+{
+ return __xleaf_devnode_open(inode, false);
+}
+EXPORT_SYMBOL_GPL(xleaf_devnode_open);
+
+void xleaf_devnode_close(struct inode *inode)
+{
+ struct xrt_subdev_platdata *pdata = INODE2PDATA(inode);
+ struct platform_device *pdev = INODE2PDEV(inode);
+ bool notify = false;
+
+ mutex_lock(&pdata->xsp_devnode_lock);
+
+ pdata->xsp_devnode_ref--;
+ if (pdata->xsp_devnode_ref == 0) {
+ pdata->xsp_devnode_excl = false;
+ notify = true;
+ }
+ if (notify) {
+ xrt_info(pdev, "closed %s, ref=%d",
+ CDEV_NAME(pdata->xsp_sysdev), pdata->xsp_devnode_ref);
+ } else {
+ xrt_info(pdev, "closed %s, notifying waiter",
+ CDEV_NAME(pdata->xsp_sysdev));
+ }
+
+ mutex_unlock(&pdata->xsp_devnode_lock);
+
+ if (notify)
+ complete(&pdata->xsp_devnode_comp);
+}
+EXPORT_SYMBOL_GPL(xleaf_devnode_close);
+
+static inline enum xrt_subdev_file_mode
+devnode_mode(struct xrt_subdev_drvdata *drvdata)
+{
+ return drvdata->xsd_file_ops.xsf_mode;
+}
+
+int xleaf_devnode_create(struct platform_device *pdev, const char *file_name,
+ const char *inst_name)
+{
+ struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(pdev);
+ struct xrt_subdev_file_ops *fops = &drvdata->xsd_file_ops;
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+ struct cdev *cdevp;
+ struct device *sysdev;
+ int ret = 0;
+ char fname[256];
+
+ mutex_init(&pdata->xsp_devnode_lock);
+ init_completion(&pdata->xsp_devnode_comp);
+
+ cdevp = &DEV_PDATA(pdev)->xsp_cdev;
+ cdev_init(cdevp, &fops->xsf_ops);
+ cdevp->owner = fops->xsf_ops.owner;
+ cdevp->dev = MKDEV(MAJOR(fops->xsf_dev_t), pdev->id);
+
+ /*
+ * Set pdev as parent of cdev so that when pdev (and its platform
+ * data) will not be freed when cdev is not freed.
+ */
+ cdev_set_parent(cdevp, &DEV(pdev)->kobj);
+
+ ret = cdev_add(cdevp, cdevp->dev, 1);
+ if (ret) {
+ xrt_err(pdev, "failed to add cdev: %d", ret);
+ goto failed;
+ }
+ if (!file_name)
+ file_name = pdev->name;
+ if (!inst_name) {
+ if (devnode_mode(drvdata) == XRT_SUBDEV_FILE_MULTI_INST) {
+ snprintf(fname, sizeof(fname), "%s/%s/%s.%u",
+ XRT_CDEV_DIR, DEV_PDATA(pdev)->xsp_root_name,
+ file_name, pdev->id);
+ } else {
+ snprintf(fname, sizeof(fname), "%s/%s/%s",
+ XRT_CDEV_DIR, DEV_PDATA(pdev)->xsp_root_name,
+ file_name);
+ }
+ } else {
+ snprintf(fname, sizeof(fname), "%s/%s/%s.%s", XRT_CDEV_DIR,
+ DEV_PDATA(pdev)->xsp_root_name, file_name, inst_name);
+ }
+ sysdev = device_create(xrt_class, NULL, cdevp->dev, NULL, "%s", fname);
+ if (IS_ERR(sysdev)) {
+ ret = PTR_ERR(sysdev);
+ xrt_err(pdev, "failed to create device node: %d", ret);
+ goto failed;
+ }
+ pdata->xsp_sysdev = sysdev;
+
+ xleaf_devnode_allowed(pdev);
+
+ xrt_info(pdev, "created (%d, %d): /dev/%s",
+ MAJOR(cdevp->dev), pdev->id, fname);
+ return 0;
+
+failed:
+ device_destroy(xrt_class, cdevp->dev);
+ cdev_del(cdevp);
+ cdevp->owner = NULL;
+ return ret;
+}
+
+int xleaf_devnode_destroy(struct platform_device *pdev)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+ struct cdev *cdevp = &pdata->xsp_cdev;
+ dev_t dev = cdevp->dev;
+ int rc;
+
+ rc = xleaf_devnode_disallowed(pdev);
+ if (rc)
+ return rc;
+
+ xrt_info(pdev, "removed (%d, %d): /dev/%s/%s", MAJOR(dev), MINOR(dev),
+ XRT_CDEV_DIR, CDEV_NAME(pdata->xsp_sysdev));
+ device_destroy(xrt_class, cdevp->dev);
+ pdata->xsp_sysdev = NULL;
+ cdev_del(cdevp);
+ return 0;
+}
diff --git a/drivers/fpga/xrt/lib/subdev.c b/drivers/fpga/xrt/lib/subdev.c
new file mode 100644
index 000000000000..73552c549bdb
--- /dev/null
+++ b/drivers/fpga/xrt/lib/subdev.c
@@ -0,0 +1,871 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA device helper functions
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/platform_device.h>
+#include <linux/pci.h>
+#include <linux/vmalloc.h>
+#include "xleaf.h"
+#include "subdev_pool.h"
+#include "main.h"
+#include "metadata.h"
+
+#define DEV_IS_PCI(dev) ((dev)->bus == &pci_bus_type)
+static inline struct device *find_root(struct platform_device *pdev)
+{
+ struct device *d = DEV(pdev);
+
+ while (!DEV_IS_PCI(d))
+ d = d->parent;
+ return d;
+}
+
+/*
+ * It represents a holder of a subdev. One holder can repeatedly hold a subdev
+ * as long as there is a unhold corresponding to a hold.
+ */
+struct xrt_subdev_holder {
+ struct list_head xsh_holder_list;
+ struct device *xsh_holder;
+ int xsh_count;
+ struct kref xsh_kref;
+};
+
+/*
+ * It represents a specific instance of platform driver for a subdev, which
+ * provides services to its clients (another subdev driver or root driver).
+ */
+struct xrt_subdev {
+ struct list_head xs_dev_list;
+ struct list_head xs_holder_list;
+ enum xrt_subdev_id xs_id; /* type of subdev */
+ struct platform_device *xs_pdev; /* a particular subdev inst */
+ struct completion xs_holder_comp;
+};
+
+static struct xrt_subdev *xrt_subdev_alloc(void)
+{
+ struct xrt_subdev *sdev = vzalloc(sizeof(*sdev));
+
+ if (!sdev)
+ return NULL;
+
+ INIT_LIST_HEAD(&sdev->xs_dev_list);
+ INIT_LIST_HEAD(&sdev->xs_holder_list);
+ init_completion(&sdev->xs_holder_comp);
+ return sdev;
+}
+
+static void xrt_subdev_free(struct xrt_subdev *sdev)
+{
+ vfree(sdev);
+}
+
+int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void *arg)
+{
+ struct device *dev = DEV(self);
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(self);
+
+ return (*pdata->xsp_root_cb)(dev->parent, pdata->xsp_root_cb_arg, cmd, arg);
+}
+
+/*
+ * Subdev common sysfs nodes.
+ */
+static ssize_t holders_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ ssize_t len;
+ struct platform_device *pdev = to_platform_device(dev);
+ struct xrt_root_ioctl_get_holders holders = { pdev, buf, 1024 };
+
+ len = xrt_subdev_root_request(pdev, XRT_ROOT_GET_LEAF_HOLDERS, &holders);
+ if (len >= holders.xpigh_holder_buf_len)
+ return len;
+ buf[len] = '\n';
+ return len + 1;
+}
+static DEVICE_ATTR_RO(holders);
+
+static struct attribute *xrt_subdev_attrs[] = {
+ &dev_attr_holders.attr,
+ NULL,
+};
+
+static ssize_t metadata_output(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *attr, char *buf, loff_t off, size_t count)
+{
+ struct device *dev = kobj_to_dev(kobj);
+ struct platform_device *pdev = to_platform_device(dev);
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
+ unsigned char *blob;
+ unsigned long size;
+ ssize_t ret = 0;
+
+ blob = pdata->xsp_dtb;
+ size = xrt_md_size(dev, blob);
+ if (size == XRT_MD_INVALID_LENGTH) {
+ ret = -EINVAL;
+ goto failed;
+ }
+
+ if (off >= size)
+ goto failed;
+
+ if (off + count > size)
+ count = size - off;
+ memcpy(buf, blob + off, count);
+
+ ret = count;
+failed:
+ return ret;
+}
+
+static struct bin_attribute meta_data_attr = {
+ .attr = {
+ .name = "metadata",
+ .mode = 0400
+ },
+ .read = metadata_output,
+ .size = 0
+};
+
+static struct bin_attribute *xrt_subdev_bin_attrs[] = {
+ &meta_data_attr,
+ NULL,
+};
+
+static const struct attribute_group xrt_subdev_attrgroup = {
+ .attrs = xrt_subdev_attrs,
+ .bin_attrs = xrt_subdev_bin_attrs,
+};
+
+/*
+ * Given the device metadata, parse it to get IO ranges and construct
+ * resource array.
+ */
+static int
+xrt_subdev_getres(struct device *parent, enum xrt_subdev_id id,
+ char *dtb, struct resource **res, int *res_num)
+{
+ struct xrt_subdev_platdata *pdata;
+ struct resource *pci_res = NULL;
+ const u64 *bar_range;
+ const u32 *bar_idx;
+ char *ep_name = NULL, *regmap = NULL;
+ uint bar;
+ int count1 = 0, count2 = 0, ret;
+
+ if (!dtb)
+ return -EINVAL;
+
+ pdata = DEV_PDATA(to_platform_device(parent));
+
+ /* go through metadata and count endpoints in it */
+ for (xrt_md_get_next_endpoint(parent, dtb, NULL, NULL, &ep_name, ®map); ep_name;
+ xrt_md_get_next_endpoint(parent, dtb, ep_name, regmap, &ep_name, ®map)) {
+ ret = xrt_md_get_prop(parent, dtb, ep_name, regmap,
+ XRT_MD_PROP_IO_OFFSET, (const void **)&bar_range, NULL);
+ if (!ret)
+ count1++;
+ }
+ if (!count1)
+ return 0;
+
+ /* allocate resource array for all endpoints been found in metadata */
+ *res = vzalloc(sizeof(**res) * count1);
+
+ /* go through all endpoints again and get IO range for each endpoint */
+ for (xrt_md_get_next_endpoint(parent, dtb, NULL, NULL, &ep_name, ®map); ep_name;
+ xrt_md_get_next_endpoint(parent, dtb, ep_name, regmap, &ep_name, ®map)) {
+ ret = xrt_md_get_prop(parent, dtb, ep_name, regmap,
+ XRT_MD_PROP_IO_OFFSET, (const void **)&bar_range, NULL);
+ if (ret)
+ continue;
+ xrt_md_get_prop(parent, dtb, ep_name, regmap,
+ XRT_MD_PROP_BAR_IDX, (const void **)&bar_idx, NULL);
+ bar = bar_idx ? be32_to_cpu(*bar_idx) : 0;
+ xleaf_get_barres(to_platform_device(parent), &pci_res, bar);
+ (*res)[count2].start = pci_res->start +
+ be64_to_cpu(bar_range[0]);
+ (*res)[count2].end = pci_res->start +
+ be64_to_cpu(bar_range[0]) +
+ be64_to_cpu(bar_range[1]) - 1;
+ (*res)[count2].flags = IORESOURCE_MEM;
+ /* check if there is conflicted resource */
+ ret = request_resource(pci_res, *res + count2);
+ if (ret) {
+ dev_err(parent, "Conflict resource %pR\n", *res + count2);
+ vfree(*res);
+ *res_num = 0;
+ *res = NULL;
+ return ret;
+ }
+ release_resource(*res + count2);
+
+ (*res)[count2].parent = pci_res;
+
+ xrt_md_find_endpoint(parent, pdata->xsp_dtb, ep_name,
+ regmap, &(*res)[count2].name);
+
+ count2++;
+ }
+
+ WARN_ON(count1 != count2);
+ *res_num = count2;
+
+ return 0;
+}
+
+static inline enum xrt_subdev_file_mode
+xleaf_devnode_mode(struct xrt_subdev_drvdata *drvdata)
+{
+ return drvdata->xsd_file_ops.xsf_mode;
+}
+
+static bool xrt_subdev_cdev_auto_creation(struct platform_device *pdev)
+{
+ struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(pdev);
+
+ if (!drvdata)
+ return false;
+
+ return xleaf_devnode_enabled(drvdata) &&
+ (xleaf_devnode_mode(drvdata) == XRT_SUBDEV_FILE_DEFAULT ||
+ (xleaf_devnode_mode(drvdata) == XRT_SUBDEV_FILE_MULTI_INST));
+}
+
+static struct xrt_subdev *
+xrt_subdev_create(struct device *parent, enum xrt_subdev_id id,
+ xrt_subdev_root_cb_t pcb, void *pcb_arg, char *dtb)
+{
+ struct xrt_subdev *sdev = NULL;
+ struct platform_device *pdev = NULL;
+ struct xrt_subdev_platdata *pdata = NULL;
+ unsigned long dtb_len = 0;
+ size_t pdata_sz;
+ int inst = PLATFORM_DEVID_NONE;
+ struct resource *res = NULL;
+ int res_num = 0;
+
+ sdev = xrt_subdev_alloc();
+ if (!sdev) {
+ dev_err(parent, "failed to alloc subdev for ID %d", id);
+ goto fail;
+ }
+ sdev->xs_id = id;
+
+ if (dtb) {
+ xrt_md_pack(parent, dtb);
+ dtb_len = xrt_md_size(parent, dtb);
+ if (dtb_len == XRT_MD_INVALID_LENGTH) {
+ dev_err(parent, "invalid metadata len %ld", dtb_len);
+ goto fail;
+ }
+ }
+ pdata_sz = sizeof(struct xrt_subdev_platdata) + dtb_len - 1;
+
+ /* Prepare platform data passed to subdev. */
+ pdata = vzalloc(pdata_sz);
+ if (!pdata)
+ goto fail;
+
+ pdata->xsp_root_cb = pcb;
+ pdata->xsp_root_cb_arg = pcb_arg;
+ memcpy(pdata->xsp_dtb, dtb, dtb_len);
+ if (id == XRT_SUBDEV_GRP) {
+ /* Group can only be created by root driver. */
+ pdata->xsp_root_name = dev_name(parent);
+ } else {
+ struct platform_device *grp = to_platform_device(parent);
+ /* Leaf can only be created by group driver. */
+ WARN_ON(strcmp(xrt_drv_name(XRT_SUBDEV_GRP), platform_get_device_id(grp)->name));
+ pdata->xsp_root_name = DEV_PDATA(grp)->xsp_root_name;
+ }
+
+ /* Obtain dev instance number. */
+ inst = xrt_drv_get_instance(id);
+ if (inst < 0) {
+ dev_err(parent, "failed to obtain instance: %d", inst);
+ goto fail;
+ }
+
+ /* Create subdev. */
+ if (id == XRT_SUBDEV_GRP) {
+ pdev = platform_device_register_data(parent, xrt_drv_name(XRT_SUBDEV_GRP),
+ inst, pdata, pdata_sz);
+ } else {
+ int rc = xrt_subdev_getres(parent, id, dtb, &res, &res_num);
+
+ if (rc) {
+ dev_err(parent, "failed to get resource for %s.%d: %d",
+ xrt_drv_name(id), inst, rc);
+ goto fail;
+ }
+ pdev = platform_device_register_resndata(parent, xrt_drv_name(id),
+ inst, res, res_num, pdata, pdata_sz);
+ vfree(res);
+ }
+ if (IS_ERR(pdev)) {
+ dev_err(parent, "failed to create subdev for %s inst %d: %ld",
+ xrt_drv_name(id), inst, PTR_ERR(pdev));
+ goto fail;
+ }
+ sdev->xs_pdev = pdev;
+
+ if (device_attach(DEV(pdev)) != 1) {
+ xrt_err(pdev, "failed to attach");
+ goto fail;
+ }
+
+ if (sysfs_create_group(&DEV(pdev)->kobj, &xrt_subdev_attrgroup))
+ xrt_err(pdev, "failed to create sysfs group");
+
+ /*
+ * Create sysfs sym link under root for leaves
+ * under random groups for easy access to them.
+ */
+ if (id != XRT_SUBDEV_GRP) {
+ if (sysfs_create_link(&find_root(pdev)->kobj,
+ &DEV(pdev)->kobj, dev_name(DEV(pdev)))) {
+ xrt_err(pdev, "failed to create sysfs link");
+ }
+ }
+
+ /* All done, ready to handle req thru cdev. */
+ if (xrt_subdev_cdev_auto_creation(pdev))
+ xleaf_devnode_create(pdev, DEV_DRVDATA(pdev)->xsd_file_ops.xsf_dev_name, NULL);
+
+ vfree(pdata);
+ return sdev;
+
+fail:
+ vfree(pdata);
+ if (sdev && !IS_ERR_OR_NULL(sdev->xs_pdev))
+ platform_device_unregister(sdev->xs_pdev);
+ if (inst >= 0)
+ xrt_drv_put_instance(id, inst);
+ xrt_subdev_free(sdev);
+ return NULL;
+}
+
+static void xrt_subdev_destroy(struct xrt_subdev *sdev)
+{
+ struct platform_device *pdev = sdev->xs_pdev;
+ int inst = pdev->id;
+ struct device *dev = DEV(pdev);
+
+ /* Take down the device node */
+ if (xrt_subdev_cdev_auto_creation(pdev))
+ xleaf_devnode_destroy(pdev);
+ if (sdev->xs_id != XRT_SUBDEV_GRP)
+ sysfs_remove_link(&find_root(pdev)->kobj, dev_name(dev));
+ sysfs_remove_group(&dev->kobj, &xrt_subdev_attrgroup);
+ platform_device_unregister(pdev);
+ xrt_drv_put_instance(sdev->xs_id, inst);
+ xrt_subdev_free(sdev);
+}
+
+struct platform_device *
+xleaf_get_leaf(struct platform_device *pdev, xrt_subdev_match_t match_cb, void *match_arg)
+{
+ int rc;
+ struct xrt_root_ioctl_get_leaf get_leaf = {
+ pdev, match_cb, match_arg, };
+
+ rc = xrt_subdev_root_request(pdev, XRT_ROOT_GET_LEAF, &get_leaf);
+ if (rc)
+ return NULL;
+ return get_leaf.xpigl_leaf;
+}
+EXPORT_SYMBOL_GPL(xleaf_get_leaf);
+
+bool xleaf_has_endpoint(struct platform_device *pdev, const char *endpoint_name)
+{
+ struct resource *res;
+ int i;
+
+ for (i = 0, res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ res;
+ res = platform_get_resource(pdev, IORESOURCE_MEM, ++i)) {
+ if (!strncmp(res->name, endpoint_name, strlen(res->name) + 1))
+ return true;
+ }
+
+ return false;
+}
+EXPORT_SYMBOL_GPL(xleaf_has_endpoint);
+
+int xleaf_put_leaf(struct platform_device *pdev, struct platform_device *leaf)
+{
+ struct xrt_root_ioctl_put_leaf put_leaf = { pdev, leaf };
+
+ return xrt_subdev_root_request(pdev, XRT_ROOT_PUT_LEAF, &put_leaf);
+}
+EXPORT_SYMBOL_GPL(xleaf_put_leaf);
+
+int xleaf_create_group(struct platform_device *pdev, char *dtb)
+{
+ return xrt_subdev_root_request(pdev, XRT_ROOT_CREATE_GROUP, dtb);
+}
+EXPORT_SYMBOL_GPL(xleaf_create_group);
+
+int xleaf_destroy_group(struct platform_device *pdev, int instance)
+{
+ return xrt_subdev_root_request(pdev, XRT_ROOT_REMOVE_GROUP, (void *)(uintptr_t)instance);
+}
+EXPORT_SYMBOL_GPL(xleaf_destroy_group);
+
+int xleaf_wait_for_group_bringup(struct platform_device *pdev)
+{
+ return xrt_subdev_root_request(pdev, XRT_ROOT_WAIT_GROUP_BRINGUP, NULL);
+}
+EXPORT_SYMBOL_GPL(xleaf_wait_for_group_bringup);
+
+static ssize_t
+xrt_subdev_get_holders(struct xrt_subdev *sdev, char *buf, size_t len)
+{
+ const struct list_head *ptr;
+ struct xrt_subdev_holder *h;
+ ssize_t n = 0;
+
+ list_for_each(ptr, &sdev->xs_holder_list) {
+ h = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list);
+ n += snprintf(buf + n, len - n, "%s:%d ",
+ dev_name(h->xsh_holder), kref_read(&h->xsh_kref));
+ if (n >= (len - 1))
+ break;
+ }
+ return n;
+}
+
+void xrt_subdev_pool_init(struct device *dev, struct xrt_subdev_pool *spool)
+{
+ INIT_LIST_HEAD(&spool->xsp_dev_list);
+ spool->xsp_owner = dev;
+ mutex_init(&spool->xsp_lock);
+ spool->xsp_closing = false;
+}
+
+static void xrt_subdev_free_holder(struct xrt_subdev_holder *holder)
+{
+ list_del(&holder->xsh_holder_list);
+ vfree(holder);
+}
+
+static void xrt_subdev_pool_wait_for_holders(struct xrt_subdev_pool *spool, struct xrt_subdev *sdev)
+{
+ const struct list_head *ptr, *next;
+ char holders[128];
+ struct xrt_subdev_holder *holder;
+ struct mutex *lk = &spool->xsp_lock;
+
+ WARN_ON(!mutex_is_locked(lk));
+
+ while (!list_empty(&sdev->xs_holder_list)) {
+ int rc;
+
+ /* It's most likely a bug if we ever enters this loop. */
+ xrt_subdev_get_holders(sdev, holders, sizeof(holders));
+ xrt_err(sdev->xs_pdev, "awaits holders: %s", holders);
+ mutex_unlock(lk);
+ rc = wait_for_completion_killable(&sdev->xs_holder_comp);
+ mutex_lock(lk);
+ if (rc == -ERESTARTSYS) {
+ xrt_err(sdev->xs_pdev, "give up on waiting for holders, clean up now");
+ list_for_each_safe(ptr, next, &sdev->xs_holder_list) {
+ holder = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list);
+ xrt_subdev_free_holder(holder);
+ }
+ }
+ }
+}
+
+void xrt_subdev_pool_fini(struct xrt_subdev_pool *spool)
+{
+ struct list_head *dl = &spool->xsp_dev_list;
+ struct mutex *lk = &spool->xsp_lock;
+
+ mutex_lock(lk);
+
+ if (spool->xsp_closing) {
+ mutex_unlock(lk);
+ return;
+ }
+
+ spool->xsp_closing = true;
+ /* Remove subdev in the reverse order of added. */
+ while (!list_empty(dl)) {
+ struct xrt_subdev *sdev = list_first_entry(dl, struct xrt_subdev, xs_dev_list);
+
+ xrt_subdev_pool_wait_for_holders(spool, sdev);
+ list_del(&sdev->xs_dev_list);
+ mutex_unlock(lk);
+ xrt_subdev_destroy(sdev);
+ mutex_lock(lk);
+ }
+
+ mutex_unlock(lk);
+}
+
+static struct xrt_subdev_holder *xrt_subdev_find_holder(struct xrt_subdev *sdev,
+ struct device *holder_dev)
+{
+ struct list_head *hl = &sdev->xs_holder_list;
+ struct xrt_subdev_holder *holder;
+ const struct list_head *ptr;
+
+ list_for_each(ptr, hl) {
+ holder = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list);
+ if (holder->xsh_holder == holder_dev)
+ return holder;
+ }
+ return NULL;
+}
+
+static int xrt_subdev_hold(struct xrt_subdev *sdev, struct device *holder_dev)
+{
+ struct xrt_subdev_holder *holder = xrt_subdev_find_holder(sdev, holder_dev);
+ struct list_head *hl = &sdev->xs_holder_list;
+
+ if (!holder) {
+ holder = vzalloc(sizeof(*holder));
+ if (!holder)
+ return -ENOMEM;
+ holder->xsh_holder = holder_dev;
+ kref_init(&holder->xsh_kref);
+ list_add_tail(&holder->xsh_holder_list, hl);
+ } else {
+ kref_get(&holder->xsh_kref);
+ }
+
+ return 0;
+}
+
+static void xrt_subdev_free_holder_kref(struct kref *kref)
+{
+ struct xrt_subdev_holder *holder = container_of(kref, struct xrt_subdev_holder, xsh_kref);
+
+ xrt_subdev_free_holder(holder);
+}
+
+static int
+xrt_subdev_release(struct xrt_subdev *sdev, struct device *holder_dev)
+{
+ struct xrt_subdev_holder *holder = xrt_subdev_find_holder(sdev, holder_dev);
+ struct list_head *hl = &sdev->xs_holder_list;
+
+ if (!holder) {
+ dev_err(holder_dev, "can't release, %s did not hold %s",
+ dev_name(holder_dev), dev_name(DEV(sdev->xs_pdev)));
+ return -EINVAL;
+ }
+ kref_put(&holder->xsh_kref, xrt_subdev_free_holder_kref);
+
+ /* kref_put above may remove holder from list. */
+ if (list_empty(hl))
+ complete(&sdev->xs_holder_comp);
+ return 0;
+}
+
+int xrt_subdev_pool_add(struct xrt_subdev_pool *spool, enum xrt_subdev_id id,
+ xrt_subdev_root_cb_t pcb, void *pcb_arg, char *dtb)
+{
+ struct mutex *lk = &spool->xsp_lock;
+ struct list_head *dl = &spool->xsp_dev_list;
+ struct xrt_subdev *sdev;
+ int ret = 0;
+
+ sdev = xrt_subdev_create(spool->xsp_owner, id, pcb, pcb_arg, dtb);
+ if (sdev) {
+ mutex_lock(lk);
+ if (spool->xsp_closing) {
+ /* No new subdev when pool is going away. */
+ xrt_err(sdev->xs_pdev, "pool is closing");
+ ret = -ENODEV;
+ } else {
+ list_add(&sdev->xs_dev_list, dl);
+ }
+ mutex_unlock(lk);
+ if (ret)
+ xrt_subdev_destroy(sdev);
+ } else {
+ ret = -EINVAL;
+ }
+
+ ret = ret ? ret : sdev->xs_pdev->id;
+ return ret;
+}
+
+int xrt_subdev_pool_del(struct xrt_subdev_pool *spool, enum xrt_subdev_id id, int instance)
+{
+ const struct list_head *ptr;
+ struct mutex *lk = &spool->xsp_lock;
+ struct list_head *dl = &spool->xsp_dev_list;
+ struct xrt_subdev *sdev;
+ int ret = -ENOENT;
+
+ mutex_lock(lk);
+ list_for_each(ptr, dl) {
+ sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (sdev->xs_id != id || sdev->xs_pdev->id != instance)
+ continue;
+ xrt_subdev_pool_wait_for_holders(spool, sdev);
+ list_del(&sdev->xs_dev_list);
+ ret = 0;
+ break;
+ }
+ mutex_unlock(lk);
+ if (ret)
+ return ret;
+
+ xrt_subdev_destroy(sdev);
+ return 0;
+}
+
+static int xrt_subdev_pool_get_impl(struct xrt_subdev_pool *spool, xrt_subdev_match_t match,
+ void *arg, struct device *holder_dev, struct xrt_subdev **sdevp)
+{
+ const struct list_head *ptr;
+ struct mutex *lk = &spool->xsp_lock;
+ struct list_head *dl = &spool->xsp_dev_list;
+ struct xrt_subdev *sdev = NULL;
+ int ret = -ENOENT;
+
+ mutex_lock(lk);
+
+ if (match == XRT_SUBDEV_MATCH_PREV) {
+ struct platform_device *pdev = (struct platform_device *)arg;
+ struct xrt_subdev *d = NULL;
+
+ if (!pdev) {
+ sdev = list_empty(dl) ? NULL :
+ list_last_entry(dl, struct xrt_subdev, xs_dev_list);
+ } else {
+ list_for_each(ptr, dl) {
+ d = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (d->xs_pdev != pdev)
+ continue;
+ if (!list_is_first(ptr, dl))
+ sdev = list_prev_entry(d, xs_dev_list);
+ break;
+ }
+ }
+ } else if (match == XRT_SUBDEV_MATCH_NEXT) {
+ struct platform_device *pdev = (struct platform_device *)arg;
+ struct xrt_subdev *d = NULL;
+
+ if (!pdev) {
+ sdev = list_first_entry_or_null(dl, struct xrt_subdev, xs_dev_list);
+ } else {
+ list_for_each(ptr, dl) {
+ d = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (d->xs_pdev != pdev)
+ continue;
+ if (!list_is_last(ptr, dl))
+ sdev = list_next_entry(d, xs_dev_list);
+ break;
+ }
+ }
+ } else {
+ list_for_each(ptr, dl) {
+ struct xrt_subdev *d = NULL;
+
+ d = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (d && !match(d->xs_id, d->xs_pdev, arg))
+ continue;
+ sdev = d;
+ break;
+ }
+ }
+
+ if (sdev)
+ ret = xrt_subdev_hold(sdev, holder_dev);
+
+ mutex_unlock(lk);
+
+ if (!ret)
+ *sdevp = sdev;
+ return ret;
+}
+
+int xrt_subdev_pool_get(struct xrt_subdev_pool *spool, xrt_subdev_match_t match, void *arg,
+ struct device *holder_dev, struct platform_device **pdevp)
+{
+ int rc;
+ struct xrt_subdev *sdev;
+
+ rc = xrt_subdev_pool_get_impl(spool, match, arg, holder_dev, &sdev);
+ if (rc) {
+ if (rc != -ENOENT)
+ dev_err(holder_dev, "failed to hold device: %d", rc);
+ return rc;
+ }
+
+ if (!DEV_IS_PCI(holder_dev)) {
+ xrt_dbg(to_platform_device(holder_dev), "%s <<==== %s",
+ dev_name(holder_dev), dev_name(DEV(sdev->xs_pdev)));
+ }
+
+ *pdevp = sdev->xs_pdev;
+ return 0;
+}
+
+static int xrt_subdev_pool_put_impl(struct xrt_subdev_pool *spool, struct platform_device *pdev,
+ struct device *holder_dev)
+{
+ const struct list_head *ptr;
+ struct mutex *lk = &spool->xsp_lock;
+ struct list_head *dl = &spool->xsp_dev_list;
+ struct xrt_subdev *sdev;
+ int ret = -ENOENT;
+
+ mutex_lock(lk);
+ list_for_each(ptr, dl) {
+ sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (sdev->xs_pdev != pdev)
+ continue;
+ ret = xrt_subdev_release(sdev, holder_dev);
+ break;
+ }
+ mutex_unlock(lk);
+
+ return ret;
+}
+
+int xrt_subdev_pool_put(struct xrt_subdev_pool *spool, struct platform_device *pdev,
+ struct device *holder_dev)
+{
+ int ret = xrt_subdev_pool_put_impl(spool, pdev, holder_dev);
+
+ if (ret)
+ return ret;
+
+ if (!DEV_IS_PCI(holder_dev)) {
+ xrt_dbg(to_platform_device(holder_dev), "%s <<==X== %s",
+ dev_name(holder_dev), dev_name(DEV(pdev)));
+ }
+ return 0;
+}
+
+void xrt_subdev_pool_trigger_event(struct xrt_subdev_pool *spool, enum xrt_events e)
+{
+ struct platform_device *tgt = NULL;
+ struct xrt_subdev *sdev = NULL;
+ struct xrt_event evt;
+
+ while (!xrt_subdev_pool_get_impl(spool, XRT_SUBDEV_MATCH_NEXT,
+ tgt, spool->xsp_owner, &sdev)) {
+ tgt = sdev->xs_pdev;
+ evt.xe_evt = e;
+ evt.xe_subdev.xevt_subdev_id = sdev->xs_id;
+ evt.xe_subdev.xevt_subdev_instance = tgt->id;
+ xrt_subdev_root_request(tgt, XRT_ROOT_EVENT, &evt);
+ xrt_subdev_pool_put_impl(spool, tgt, spool->xsp_owner);
+ }
+}
+
+void xrt_subdev_pool_handle_event(struct xrt_subdev_pool *spool, struct xrt_event *evt)
+{
+ struct platform_device *tgt = NULL;
+ struct xrt_subdev *sdev = NULL;
+
+ while (!xrt_subdev_pool_get_impl(spool, XRT_SUBDEV_MATCH_NEXT,
+ tgt, spool->xsp_owner, &sdev)) {
+ tgt = sdev->xs_pdev;
+ xleaf_ioctl(tgt, XRT_XLEAF_EVENT, evt);
+ xrt_subdev_pool_put_impl(spool, tgt, spool->xsp_owner);
+ }
+}
+
+ssize_t xrt_subdev_pool_get_holders(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev, char *buf, size_t len)
+{
+ const struct list_head *ptr;
+ struct mutex *lk = &spool->xsp_lock;
+ struct list_head *dl = &spool->xsp_dev_list;
+ struct xrt_subdev *sdev;
+ ssize_t ret = 0;
+
+ mutex_lock(lk);
+ list_for_each(ptr, dl) {
+ sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
+ if (sdev->xs_pdev != pdev)
+ continue;
+ ret = xrt_subdev_get_holders(sdev, buf, len);
+ break;
+ }
+ mutex_unlock(lk);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xrt_subdev_pool_get_holders);
+
+int xleaf_broadcast_event(struct platform_device *pdev, enum xrt_events evt, bool async)
+{
+ struct xrt_event e = { evt, };
+ u32 cmd = async ? XRT_ROOT_EVENT_ASYNC : XRT_ROOT_EVENT;
+
+ WARN_ON(evt == XRT_EVENT_POST_CREATION || evt == XRT_EVENT_PRE_REMOVAL);
+ return xrt_subdev_root_request(pdev, cmd, &e);
+}
+EXPORT_SYMBOL_GPL(xleaf_broadcast_event);
+
+void xleaf_hot_reset(struct platform_device *pdev)
+{
+ xrt_subdev_root_request(pdev, XRT_ROOT_HOT_RESET, NULL);
+}
+EXPORT_SYMBOL_GPL(xleaf_hot_reset);
+
+void xleaf_get_barres(struct platform_device *pdev, struct resource **res, uint bar_idx)
+{
+ struct xrt_root_ioctl_get_res arg = { 0 };
+
+ if (bar_idx > PCI_STD_RESOURCE_END) {
+ xrt_err(pdev, "Invalid bar idx %d", bar_idx);
+ *res = NULL;
+ return;
+ }
+
+ xrt_subdev_root_request(pdev, XRT_ROOT_GET_RESOURCE, &arg);
+
+ *res = &arg.xpigr_res[bar_idx];
+}
+
+void xleaf_get_root_id(struct platform_device *pdev, unsigned short *vendor, unsigned short *device,
+ unsigned short *subvendor, unsigned short *subdevice)
+{
+ struct xrt_root_ioctl_get_id id = { 0 };
+
+ xrt_subdev_root_request(pdev, XRT_ROOT_GET_ID, (void *)&id);
+ if (vendor)
+ *vendor = id.xpigi_vendor_id;
+ if (device)
+ *device = id.xpigi_device_id;
+ if (subvendor)
+ *subvendor = id.xpigi_sub_vendor_id;
+ if (subdevice)
+ *subdevice = id.xpigi_sub_device_id;
+}
+
+struct device *xleaf_register_hwmon(struct platform_device *pdev, const char *name, void *drvdata,
+ const struct attribute_group **grps)
+{
+ struct xrt_root_ioctl_hwmon hm = { true, name, drvdata, grps, };
+
+ xrt_subdev_root_request(pdev, XRT_ROOT_HWMON, (void *)&hm);
+ return hm.xpih_hwmon_dev;
+}
+
+void xleaf_unregister_hwmon(struct platform_device *pdev, struct device *hwmon)
+{
+ struct xrt_root_ioctl_hwmon hm = { false, };
+
+ hm.xpih_hwmon_dev = hwmon;
+ xrt_subdev_root_request(pdev, XRT_ROOT_HWMON, (void *)&hm);
+}
diff --git a/drivers/fpga/xrt/lib/subdev_pool.h b/drivers/fpga/xrt/lib/subdev_pool.h
new file mode 100644
index 000000000000..50a8490e0e41
--- /dev/null
+++ b/drivers/fpga/xrt/lib/subdev_pool.h
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_SUBDEV_POOL_H_
+#define _XRT_SUBDEV_POOL_H_
+
+#include "xroot.h"
+
+/*
+ * It manages a list of xrt_subdevs for root and group drivers.
+ */
+struct xrt_subdev_pool {
+ struct list_head xsp_dev_list;
+ struct device *xsp_owner;
+ struct mutex xsp_lock; /* pool lock */
+ bool xsp_closing;
+};
+
+/*
+ * Subdev pool API for root and group drivers only.
+ */
+void xrt_subdev_pool_init(struct device *dev,
+ struct xrt_subdev_pool *spool);
+void xrt_subdev_pool_fini(struct xrt_subdev_pool *spool);
+int xrt_subdev_pool_get(struct xrt_subdev_pool *spool,
+ xrt_subdev_match_t match,
+ void *arg, struct device *holder_dev,
+ struct platform_device **pdevp);
+int xrt_subdev_pool_put(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev,
+ struct device *holder_dev);
+int xrt_subdev_pool_add(struct xrt_subdev_pool *spool,
+ enum xrt_subdev_id id, xrt_subdev_root_cb_t pcb,
+ void *pcb_arg, char *dtb);
+int xrt_subdev_pool_del(struct xrt_subdev_pool *spool,
+ enum xrt_subdev_id id, int instance);
+ssize_t xrt_subdev_pool_get_holders(struct xrt_subdev_pool *spool,
+ struct platform_device *pdev,
+ char *buf, size_t len);
+
+void xrt_subdev_pool_trigger_event(struct xrt_subdev_pool *spool,
+ enum xrt_events evt);
+void xrt_subdev_pool_handle_event(struct xrt_subdev_pool *spool,
+ struct xrt_event *evt);
+
+#endif /* _XRT_SUBDEV_POOL_H_ */
diff --git a/drivers/fpga/xrt/lib/xroot.c b/drivers/fpga/xrt/lib/xroot.c
new file mode 100644
index 000000000000..3dc7b0243277
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xroot.c
@@ -0,0 +1,598 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Root Functions
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/hwmon.h>
+#include "xroot.h"
+#include "subdev_pool.h"
+#include "group.h"
+#include "metadata.h"
+
+#define XROOT_PDEV(xr) ((xr)->pdev)
+#define XROOT_DEV(xr) (&(XROOT_PDEV(xr)->dev))
+#define xroot_err(xr, fmt, args...) \
+ dev_err(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
+#define xroot_warn(xr, fmt, args...) \
+ dev_warn(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
+#define xroot_info(xr, fmt, args...) \
+ dev_info(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
+#define xroot_dbg(xr, fmt, args...) \
+ dev_dbg(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
+
+#define XRT_VSEC_ID 0x20
+
+#define XROOT_GRP_FIRST (-1)
+#define XROOT_GRP_LAST (-2)
+
+static int xroot_root_cb(struct device *, void *, u32, void *);
+
+struct xroot_evt {
+ struct list_head list;
+ struct xrt_event evt;
+ struct completion comp;
+ bool async;
+};
+
+struct xroot_events {
+ struct mutex evt_lock; /* event lock */
+ struct list_head evt_list;
+ struct work_struct evt_work;
+};
+
+struct xroot_grps {
+ struct xrt_subdev_pool pool;
+ struct work_struct bringup_work;
+ atomic_t bringup_pending;
+ atomic_t bringup_failed;
+ struct completion bringup_comp;
+};
+
+struct xroot {
+ struct pci_dev *pdev;
+ struct xroot_events events;
+ struct xroot_grps grps;
+ struct xroot_pf_cb pf_cb;
+};
+
+struct xroot_grp_match_arg {
+ enum xrt_subdev_id id;
+ int instance;
+};
+
+static bool xroot_grp_match(enum xrt_subdev_id id,
+ struct platform_device *pdev, void *arg)
+{
+ struct xroot_grp_match_arg *a = (struct xroot_grp_match_arg *)arg;
+ return id == a->id && pdev->id == a->instance;
+}
+
+static int xroot_get_group(struct xroot *xr, int instance,
+ struct platform_device **grpp)
+{
+ int rc = 0;
+ struct xrt_subdev_pool *grps = &xr->grps.pool;
+ struct device *dev = DEV(xr->pdev);
+ struct xroot_grp_match_arg arg = { XRT_SUBDEV_GRP, instance };
+
+ if (instance == XROOT_GRP_LAST) {
+ rc = xrt_subdev_pool_get(grps, XRT_SUBDEV_MATCH_NEXT,
+ *grpp, dev, grpp);
+ } else if (instance == XROOT_GRP_FIRST) {
+ rc = xrt_subdev_pool_get(grps, XRT_SUBDEV_MATCH_PREV,
+ *grpp, dev, grpp);
+ } else {
+ rc = xrt_subdev_pool_get(grps, xroot_grp_match,
+ &arg, dev, grpp);
+ }
+
+ if (rc && rc != -ENOENT)
+ xroot_err(xr, "failed to hold group %d: %d", instance, rc);
+ return rc;
+}
+
+static void xroot_put_group(struct xroot *xr, struct platform_device *grp)
+{
+ int inst = grp->id;
+ int rc = xrt_subdev_pool_put(&xr->grps.pool, grp, DEV(xr->pdev));
+
+ if (rc)
+ xroot_err(xr, "failed to release group %d: %d", inst, rc);
+}
+
+static int xroot_trigger_event(struct xroot *xr,
+ struct xrt_event *e, bool async)
+{
+ struct xroot_evt *enew = vzalloc(sizeof(*enew));
+
+ if (!enew)
+ return -ENOMEM;
+
+ enew->evt = *e;
+ enew->async = async;
+ init_completion(&enew->comp);
+
+ mutex_lock(&xr->events.evt_lock);
+ list_add(&enew->list, &xr->events.evt_list);
+ mutex_unlock(&xr->events.evt_lock);
+
+ schedule_work(&xr->events.evt_work);
+
+ if (async)
+ return 0;
+
+ wait_for_completion(&enew->comp);
+ vfree(enew);
+ return 0;
+}
+
+static void
+xroot_group_trigger_event(struct xroot *xr, int inst, enum xrt_events e)
+{
+ int ret;
+ struct platform_device *pdev = NULL;
+ struct xrt_event evt = { 0 };
+
+ WARN_ON(inst < 0);
+ /* Only triggers subdev specific events. */
+ if (e != XRT_EVENT_POST_CREATION && e != XRT_EVENT_PRE_REMOVAL) {
+ xroot_err(xr, "invalid event %d", e);
+ return;
+ }
+
+ ret = xroot_get_group(xr, inst, &pdev);
+ if (ret)
+ return;
+
+ /* Triggers event for children, first. */
+ (void)xleaf_ioctl(pdev, XRT_GROUP_TRIGGER_EVENT, (void *)(uintptr_t)e);
+
+ /* Triggers event for itself. */
+ evt.xe_evt = e;
+ evt.xe_subdev.xevt_subdev_id = XRT_SUBDEV_GRP;
+ evt.xe_subdev.xevt_subdev_instance = inst;
+ (void)xroot_trigger_event(xr, &evt, false);
+
+ (void)xroot_put_group(xr, pdev);
+}
+
+int xroot_create_group(void *root, char *dtb)
+{
+ struct xroot *xr = (struct xroot *)root;
+ int ret;
+
+ atomic_inc(&xr->grps.bringup_pending);
+ ret = xrt_subdev_pool_add(&xr->grps.pool, XRT_SUBDEV_GRP,
+ xroot_root_cb, xr, dtb);
+ if (ret >= 0) {
+ schedule_work(&xr->grps.bringup_work);
+ } else {
+ atomic_dec(&xr->grps.bringup_pending);
+ atomic_inc(&xr->grps.bringup_failed);
+ xroot_err(xr, "failed to create group: %d", ret);
+ }
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xroot_create_group);
+
+static int xroot_destroy_single_group(struct xroot *xr, int instance)
+{
+ struct platform_device *pdev = NULL;
+ int ret;
+
+ WARN_ON(instance < 0);
+ ret = xroot_get_group(xr, instance, &pdev);
+ if (ret)
+ return ret;
+
+ xroot_group_trigger_event(xr, instance, XRT_EVENT_PRE_REMOVAL);
+
+ /* Now tear down all children in this group. */
+ ret = xleaf_ioctl(pdev, XRT_GROUP_FINI_CHILDREN, NULL);
+ (void)xroot_put_group(xr, pdev);
+ if (!ret) {
+ ret = xrt_subdev_pool_del(&xr->grps.pool, XRT_SUBDEV_GRP,
+ instance);
+ }
+
+ return ret;
+}
+
+static int xroot_destroy_group(struct xroot *xr, int instance)
+{
+ struct platform_device *target = NULL;
+ struct platform_device *deps = NULL;
+ int ret;
+
+ WARN_ON(instance < 0);
+ /*
+ * Make sure target group exists and can't go away before
+ * we remove it's dependents
+ */
+ ret = xroot_get_group(xr, instance, &target);
+ if (ret)
+ return ret;
+
+ /*
+ * Remove all groups depend on target one.
+ * Assuming subdevs in higher group ID can depend on ones in
+ * lower ID groups, we remove them in the reservse order.
+ */
+ while (xroot_get_group(xr, XROOT_GRP_LAST, &deps) != -ENOENT) {
+ int inst = deps->id;
+
+ xroot_put_group(xr, deps);
+ if (instance == inst)
+ break;
+ (void)xroot_destroy_single_group(xr, inst);
+ deps = NULL;
+ }
+
+ /* Now we can remove the target group. */
+ xroot_put_group(xr, target);
+ return xroot_destroy_single_group(xr, instance);
+}
+
+static int xroot_lookup_group(struct xroot *xr,
+ struct xrt_root_ioctl_lookup_group *arg)
+{
+ int rc = -ENOENT;
+ struct platform_device *grp = NULL;
+
+ while (rc < 0 && xroot_get_group(xr, XROOT_GRP_LAST, &grp) != -ENOENT) {
+ if (arg->xpilp_match_cb(XRT_SUBDEV_GRP, grp,
+ arg->xpilp_match_arg)) {
+ rc = grp->id;
+ }
+ xroot_put_group(xr, grp);
+ }
+ return rc;
+}
+
+static void xroot_event_work(struct work_struct *work)
+{
+ struct xroot_evt *tmp;
+ struct xroot *xr = container_of(work, struct xroot, events.evt_work);
+
+ mutex_lock(&xr->events.evt_lock);
+ while (!list_empty(&xr->events.evt_list)) {
+ tmp = list_first_entry(&xr->events.evt_list,
+ struct xroot_evt, list);
+ list_del(&tmp->list);
+ mutex_unlock(&xr->events.evt_lock);
+
+ (void)xrt_subdev_pool_handle_event(&xr->grps.pool, &tmp->evt);
+
+ if (tmp->async)
+ vfree(tmp);
+ else
+ complete(&tmp->comp);
+
+ mutex_lock(&xr->events.evt_lock);
+ }
+ mutex_unlock(&xr->events.evt_lock);
+}
+
+static void xroot_event_init(struct xroot *xr)
+{
+ INIT_LIST_HEAD(&xr->events.evt_list);
+ mutex_init(&xr->events.evt_lock);
+ INIT_WORK(&xr->events.evt_work, xroot_event_work);
+}
+
+static void xroot_event_fini(struct xroot *xr)
+{
+ flush_scheduled_work();
+ WARN_ON(!list_empty(&xr->events.evt_list));
+}
+
+static int xroot_get_leaf(struct xroot *xr, struct xrt_root_ioctl_get_leaf *arg)
+{
+ int rc = -ENOENT;
+ struct platform_device *grp = NULL;
+
+ while (rc && xroot_get_group(xr, XROOT_GRP_LAST, &grp) != -ENOENT) {
+ rc = xleaf_ioctl(grp, XRT_GROUP_GET_LEAF, arg);
+ xroot_put_group(xr, grp);
+ }
+ return rc;
+}
+
+static int xroot_put_leaf(struct xroot *xr, struct xrt_root_ioctl_put_leaf *arg)
+{
+ int rc = -ENOENT;
+ struct platform_device *grp = NULL;
+
+ while (rc && xroot_get_group(xr, XROOT_GRP_LAST, &grp) != -ENOENT) {
+ rc = xleaf_ioctl(grp, XRT_GROUP_PUT_LEAF, arg);
+ xroot_put_group(xr, grp);
+ }
+ return rc;
+}
+
+static int xroot_root_cb(struct device *dev, void *parg, u32 cmd, void *arg)
+{
+ struct xroot *xr = (struct xroot *)parg;
+ int rc = 0;
+
+ switch (cmd) {
+ /* Leaf actions. */
+ case XRT_ROOT_GET_LEAF: {
+ struct xrt_root_ioctl_get_leaf *getleaf =
+ (struct xrt_root_ioctl_get_leaf *)arg;
+ rc = xroot_get_leaf(xr, getleaf);
+ break;
+ }
+ case XRT_ROOT_PUT_LEAF: {
+ struct xrt_root_ioctl_put_leaf *putleaf =
+ (struct xrt_root_ioctl_put_leaf *)arg;
+ rc = xroot_put_leaf(xr, putleaf);
+ break;
+ }
+ case XRT_ROOT_GET_LEAF_HOLDERS: {
+ struct xrt_root_ioctl_get_holders *holders =
+ (struct xrt_root_ioctl_get_holders *)arg;
+ rc = xrt_subdev_pool_get_holders(&xr->grps.pool,
+ holders->xpigh_pdev,
+ holders->xpigh_holder_buf,
+ holders->xpigh_holder_buf_len);
+ break;
+ }
+
+ /* Group actions. */
+ case XRT_ROOT_CREATE_GROUP:
+ rc = xroot_create_group(xr, (char *)arg);
+ break;
+ case XRT_ROOT_REMOVE_GROUP:
+ rc = xroot_destroy_group(xr, (int)(uintptr_t)arg);
+ break;
+ case XRT_ROOT_LOOKUP_GROUP: {
+ struct xrt_root_ioctl_lookup_group *getgrp =
+ (struct xrt_root_ioctl_lookup_group *)arg;
+ rc = xroot_lookup_group(xr, getgrp);
+ break;
+ }
+ case XRT_ROOT_WAIT_GROUP_BRINGUP:
+ rc = xroot_wait_for_bringup(xr) ? 0 : -EINVAL;
+ break;
+
+ /* Event actions. */
+ case XRT_ROOT_EVENT:
+ case XRT_ROOT_EVENT_ASYNC: {
+ bool async = (cmd == XRT_ROOT_EVENT_ASYNC);
+ struct xrt_event *evt = (struct xrt_event *)arg;
+
+ rc = xroot_trigger_event(xr, evt, async);
+ break;
+ }
+
+ /* Device info. */
+ case XRT_ROOT_GET_RESOURCE: {
+ struct xrt_root_ioctl_get_res *res =
+ (struct xrt_root_ioctl_get_res *)arg;
+ res->xpigr_res = xr->pdev->resource;
+ break;
+ }
+ case XRT_ROOT_GET_ID: {
+ struct xrt_root_ioctl_get_id *id =
+ (struct xrt_root_ioctl_get_id *)arg;
+
+ id->xpigi_vendor_id = xr->pdev->vendor;
+ id->xpigi_device_id = xr->pdev->device;
+ id->xpigi_sub_vendor_id = xr->pdev->subsystem_vendor;
+ id->xpigi_sub_device_id = xr->pdev->subsystem_device;
+ break;
+ }
+
+ /* MISC generic PCIE driver functions. */
+ case XRT_ROOT_HOT_RESET: {
+ xr->pf_cb.xpc_hot_reset(xr->pdev);
+ break;
+ }
+ case XRT_ROOT_HWMON: {
+ struct xrt_root_ioctl_hwmon *hwmon =
+ (struct xrt_root_ioctl_hwmon *)arg;
+
+ if (hwmon->xpih_register) {
+ hwmon->xpih_hwmon_dev =
+ hwmon_device_register_with_info(DEV(xr->pdev),
+ hwmon->xpih_name,
+ hwmon->xpih_drvdata,
+ NULL,
+ hwmon->xpih_groups);
+ } else {
+ (void)hwmon_device_unregister(hwmon->xpih_hwmon_dev);
+ }
+ break;
+ }
+
+ default:
+ xroot_err(xr, "unknown IOCTL cmd %d", cmd);
+ rc = -EINVAL;
+ break;
+ }
+
+ return rc;
+}
+
+static void xroot_bringup_group_work(struct work_struct *work)
+{
+ struct platform_device *pdev = NULL;
+ struct xroot *xr = container_of(work, struct xroot, grps.bringup_work);
+
+ while (xroot_get_group(xr, XROOT_GRP_FIRST, &pdev) != -ENOENT) {
+ int r, i;
+
+ i = pdev->id;
+ r = xleaf_ioctl(pdev, XRT_GROUP_INIT_CHILDREN, NULL);
+ (void)xroot_put_group(xr, pdev);
+ if (r == -EEXIST)
+ continue; /* Already brough up, nothing to do. */
+ if (r)
+ atomic_inc(&xr->grps.bringup_failed);
+
+ xroot_group_trigger_event(xr, i, XRT_EVENT_POST_CREATION);
+
+ if (atomic_dec_and_test(&xr->grps.bringup_pending))
+ complete(&xr->grps.bringup_comp);
+ }
+}
+
+static void xroot_grps_init(struct xroot *xr)
+{
+ xrt_subdev_pool_init(DEV(xr->pdev), &xr->grps.pool);
+ INIT_WORK(&xr->grps.bringup_work, xroot_bringup_group_work);
+ atomic_set(&xr->grps.bringup_pending, 0);
+ atomic_set(&xr->grps.bringup_failed, 0);
+ init_completion(&xr->grps.bringup_comp);
+}
+
+static void xroot_grps_fini(struct xroot *xr)
+{
+ flush_scheduled_work();
+ xrt_subdev_pool_fini(&xr->grps.pool);
+}
+
+int xroot_add_vsec_node(void *root, char *dtb)
+{
+ struct xroot *xr = (struct xroot *)root;
+ struct device *dev = DEV(xr->pdev);
+ struct xrt_md_endpoint ep = { 0 };
+ int cap = 0, ret = 0;
+ u32 off_low, off_high, vsec_bar, header;
+ u64 vsec_off;
+
+ while ((cap = pci_find_next_ext_capability(xr->pdev, cap,
+ PCI_EXT_CAP_ID_VNDR))) {
+ pci_read_config_dword(xr->pdev, cap + PCI_VNDR_HEADER, &header);
+ if (PCI_VNDR_HEADER_ID(header) == XRT_VSEC_ID)
+ break;
+ }
+ if (!cap) {
+ xroot_info(xr, "No Vendor Specific Capability.");
+ return -ENOENT;
+ }
+
+ if (pci_read_config_dword(xr->pdev, cap + 8, &off_low) ||
+ pci_read_config_dword(xr->pdev, cap + 12, &off_high)) {
+ xroot_err(xr, "pci_read vendor specific failed.");
+ return -EINVAL;
+ }
+
+ ep.ep_name = XRT_MD_NODE_VSEC;
+ ret = xrt_md_add_endpoint(dev, dtb, &ep);
+ if (ret) {
+ xroot_err(xr, "add vsec metadata failed, ret %d", ret);
+ goto failed;
+ }
+
+ vsec_bar = cpu_to_be32(off_low & 0xf);
+ ret = xrt_md_set_prop(dev, dtb, XRT_MD_NODE_VSEC, NULL,
+ XRT_MD_PROP_BAR_IDX, &vsec_bar, sizeof(vsec_bar));
+ if (ret) {
+ xroot_err(xr, "add vsec bar idx failed, ret %d", ret);
+ goto failed;
+ }
+
+ vsec_off = cpu_to_be64(((u64)off_high << 32) | (off_low & ~0xfU));
+ ret = xrt_md_set_prop(dev, dtb, XRT_MD_NODE_VSEC, NULL,
+ XRT_MD_PROP_OFFSET, &vsec_off, sizeof(vsec_off));
+ if (ret) {
+ xroot_err(xr, "add vsec offset failed, ret %d", ret);
+ goto failed;
+ }
+
+failed:
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xroot_add_vsec_node);
+
+int xroot_add_simple_node(void *root, char *dtb, const char *endpoint)
+{
+ struct xroot *xr = (struct xroot *)root;
+ struct device *dev = DEV(xr->pdev);
+ struct xrt_md_endpoint ep = { 0 };
+ int ret = 0;
+
+ ep.ep_name = endpoint;
+ ret = xrt_md_add_endpoint(dev, dtb, &ep);
+ if (ret)
+ xroot_err(xr, "add %s failed, ret %d", endpoint, ret);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xroot_add_simple_node);
+
+bool xroot_wait_for_bringup(void *root)
+{
+ struct xroot *xr = (struct xroot *)root;
+
+ wait_for_completion(&xr->grps.bringup_comp);
+ return atomic_xchg(&xr->grps.bringup_failed, 0) == 0;
+}
+EXPORT_SYMBOL_GPL(xroot_wait_for_bringup);
+
+int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root)
+{
+ struct device *dev = DEV(pdev);
+ struct xroot *xr = NULL;
+
+ dev_info(dev, "%s: probing...", __func__);
+
+ xr = devm_kzalloc(dev, sizeof(*xr), GFP_KERNEL);
+ if (!xr)
+ return -ENOMEM;
+
+ xr->pdev = pdev;
+ xr->pf_cb = *cb;
+ xroot_grps_init(xr);
+ xroot_event_init(xr);
+
+ *root = xr;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xroot_probe);
+
+void xroot_remove(void *root)
+{
+ struct xroot *xr = (struct xroot *)root;
+ struct platform_device *grp = NULL;
+
+ xroot_info(xr, "leaving...");
+
+ if (xroot_get_group(xr, XROOT_GRP_FIRST, &grp) == 0) {
+ int instance = grp->id;
+
+ xroot_put_group(xr, grp);
+ (void)xroot_destroy_group(xr, instance);
+ }
+
+ xroot_event_fini(xr);
+ xroot_grps_fini(xr);
+}
+EXPORT_SYMBOL_GPL(xroot_remove);
+
+void xroot_broadcast(void *root, enum xrt_events evt)
+{
+ struct xroot *xr = (struct xroot *)root;
+ struct xrt_event e = { 0 };
+
+ /* Root pf driver only broadcasts below two events. */
+ if (evt != XRT_EVENT_POST_CREATION && evt != XRT_EVENT_PRE_REMOVAL) {
+ xroot_info(xr, "invalid event %d", evt);
+ return;
+ }
+
+ e.xe_evt = evt;
+ e.xe_subdev.xevt_subdev_id = XRT_ROOT;
+ e.xe_subdev.xevt_subdev_instance = 0;
+ (void)xroot_trigger_event(xr, &e, false);
+}
+EXPORT_SYMBOL_GPL(xroot_broadcast);
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 06/18] fpga: xrt: platform driver infrastructure
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 06/18] fpga: xrt: platform driver infrastructure Lizhi Hou
@ 2021-02-25 21:59 ` Tom Rix
[not found] ` <13e9a311-2d04-ba65-3ed2-f9f1834c37de@xilinx.com>
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-25 21:59 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> infrastructure code providing APIs for managing leaf driver instance
> groups, facilitating inter-leaf driver calls and root calls, managing leaf
> driver device nodes.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/events.h | 48 ++
> drivers/fpga/xrt/include/subdev_id.h | 43 ++
> drivers/fpga/xrt/include/xleaf.h | 276 +++++++++
> drivers/fpga/xrt/lib/cdev.c | 231 +++++++
> drivers/fpga/xrt/lib/subdev.c | 871 +++++++++++++++++++++++++++
> drivers/fpga/xrt/lib/subdev_pool.h | 53 ++
> drivers/fpga/xrt/lib/xroot.c | 598 ++++++++++++++++++
> 7 files changed, 2120 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/events.h
> create mode 100644 drivers/fpga/xrt/include/subdev_id.h
> create mode 100644 drivers/fpga/xrt/include/xleaf.h
> create mode 100644 drivers/fpga/xrt/lib/cdev.c
> create mode 100644 drivers/fpga/xrt/lib/subdev.c
> create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
> create mode 100644 drivers/fpga/xrt/lib/xroot.c
>
> diff --git a/drivers/fpga/xrt/include/events.h b/drivers/fpga/xrt/include/events.h
> new file mode 100644
> index 000000000000..2a9aae8bceb4
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/events.h
> @@ -0,0 +1,48 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
general problem with generic, low information comments
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_EVENTS_H_
> +#define _XRT_EVENTS_H_
> +
> +#include <linux/platform_device.h>
why is platform_device.h needed ?
> +#include "subdev_id.h"
> +
> +/*
> + * Event notification.
> + */
> +enum xrt_events {
> + XRT_EVENT_TEST = 0, /* for testing */
> + /*
> + * Events related to specific subdev
> + * Callback arg: struct xrt_event_arg_subdev
> + */
> + XRT_EVENT_POST_CREATION,
> + XRT_EVENT_PRE_REMOVAL,
> + /*
> + * Events related to change of the whole board
> + * Callback arg: <none>
> + */
> + XRT_EVENT_PRE_HOT_RESET,
> + XRT_EVENT_POST_HOT_RESET,
> + XRT_EVENT_PRE_GATE_CLOSE,
> + XRT_EVENT_POST_GATE_OPEN,
> +};
> +
> +struct xrt_event_arg_subdev {
> + enum xrt_subdev_id xevt_subdev_id;
> + int xevt_subdev_instance;
> +};
> +
> +struct xrt_event {
> + enum xrt_events xe_evt;
> + struct xrt_event_arg_subdev xe_subdev;
> +};
> +
> +#endif /* _XRT_EVENTS_H_ */
> diff --git a/drivers/fpga/xrt/include/subdev_id.h b/drivers/fpga/xrt/include/subdev_id.h
> new file mode 100644
> index 000000000000..6205a9f26196
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/subdev_id.h
> @@ -0,0 +1,43 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_SUBDEV_ID_H_
> +#define _XRT_SUBDEV_ID_H_
> +
> +/*
> + * Every subdev driver should have an ID for others to refer to it.
driver has an ID
> + * There can be unlimited number of instances of a subdev driver. A
unlimited? change to 'multiple'
> + * <subdev_id, subdev_instance> tuple should be a unique identification of
tuple is a unique
> + * a specific instance of a subdev driver.
> + * NOTE: PLEASE do not change the order of IDs. Sub devices in the same
> + * group are initialized by this order.
why does the order matter? the enums are all initialized
> + */
> +enum xrt_subdev_id {
> + XRT_SUBDEV_GRP = 0,
> + XRT_SUBDEV_VSEC = 1,
> + XRT_SUBDEV_VSEC_GOLDEN = 2,
> + XRT_SUBDEV_DEVCTL = 3,
> + XRT_SUBDEV_AXIGATE = 4,
> + XRT_SUBDEV_ICAP = 5,
> + XRT_SUBDEV_TEST = 6,
> + XRT_SUBDEV_MGMT_MAIN = 7,
> + XRT_SUBDEV_QSPI = 8,
> + XRT_SUBDEV_MAILBOX = 9,
> + XRT_SUBDEV_CMC = 10,
> + XRT_SUBDEV_CALIB = 11,
> + XRT_SUBDEV_CLKFREQ = 12,
> + XRT_SUBDEV_CLOCK = 13,
> + XRT_SUBDEV_SRSR = 14,
> + XRT_SUBDEV_UCS = 15,
> + XRT_SUBDEV_NUM = 16, /* Total number of subdevs. */
> + XRT_ROOT = -1, /* Special ID for root driver. */
> +};
> +
> +#endif /* _XRT_SUBDEV_ID_H_ */
> diff --git a/drivers/fpga/xrt/include/xleaf.h b/drivers/fpga/xrt/include/xleaf.h
> new file mode 100644
> index 000000000000..10215a75d474
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf.h
> @@ -0,0 +1,276 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + * Sonal Santan <sonal.santan@xilinx.com>
> + */
> +
> +#ifndef _XRT_XLEAF_H_
> +#define _XRT_XLEAF_H_
> +
> +#include <linux/mod_devicetable.h>
not needed
> +#include <linux/platform_device.h>
> +#include <linux/fs.h>
> +#include <linux/cdev.h>
> +#include <linux/pci.h>
not needed
check if includes are actually needed.
> +#include <linux/libfdt_env.h>
> +#include "libfdt.h"
> +#include "subdev_id.h"
> +#include "xroot.h"
> +#include "events.h"
> +
> +/* All subdev drivers should use below common routines to print out msg. */
> +#define DEV(pdev) (&(pdev)->dev)
> +#define DEV_PDATA(pdev) \
> + ((struct xrt_subdev_platdata *)dev_get_platdata(DEV(pdev)))
> +#define DEV_DRVDATA(pdev) \
> + ((struct xrt_subdev_drvdata *) \
> + platform_get_device_id(pdev)->driver_data)
> +#define FMT_PRT(prt_fn, pdev, fmt, args...) \
> + ({typeof(pdev) (_pdev) = (pdev); \
> + prt_fn(DEV(_pdev), "%s %s: " fmt, \
> + DEV_PDATA(_pdev)->xsp_root_name, __func__, ##args); })
> +#define xrt_err(pdev, fmt, args...) FMT_PRT(dev_err, pdev, fmt, ##args)
> +#define xrt_warn(pdev, fmt, args...) FMT_PRT(dev_warn, pdev, fmt, ##args)
> +#define xrt_info(pdev, fmt, args...) FMT_PRT(dev_info, pdev, fmt, ##args)
> +#define xrt_dbg(pdev, fmt, args...) FMT_PRT(dev_dbg, pdev, fmt, ##args)
> +
> +/* Starting IOCTL for common IOCTLs implemented by all leaves. */
> +#define XRT_XLEAF_COMMON_BASE 0
> +/* Starting IOCTL for leaves' specific IOCTLs. */
> +#define XRT_XLEAF_CUSTOM_BASE 64
> +enum xrt_xleaf_common_ioctl_cmd {
> + XRT_XLEAF_EVENT = XRT_XLEAF_COMMON_BASE,
> +};
> +
> +/*
> + * If populated by subdev driver, infra will handle the mechanics of
> + * char device (un)registration.
> + */
> +enum xrt_subdev_file_mode {
> + /* Infra create cdev, default file name */
> + XRT_SUBDEV_FILE_DEFAULT = 0,
> + /* Infra create cdev, need to encode inst num in file name */
> + XRT_SUBDEV_FILE_MULTI_INST,
> + /* No auto creation of cdev by infra, leaf handles it by itself */
> + XRT_SUBDEV_FILE_NO_AUTO,
> +};
> +
> +struct xrt_subdev_file_ops {
> + const struct file_operations xsf_ops;
> + dev_t xsf_dev_t;
> + const char *xsf_dev_name;
> + enum xrt_subdev_file_mode xsf_mode;
> +};
> +
> +/*
> + * Subdev driver callbacks populated by subdev driver.
> + */
> +struct xrt_subdev_drv_ops {
> + /*
> + * Per driver instance callback. The pdev points to the instance.
> + * If defined these are called by other leaf drivers.
If defined,
> + * Note that root driver may call into xsd_ioctl of a group driver.
> + */
> + int (*xsd_ioctl)(struct platform_device *pdev, u32 cmd, void *arg);
> +};
> +
> +/*
> + * Defined and populated by subdev driver, exported as driver_data in
> + * struct platform_device_id.
> + */
> +struct xrt_subdev_drvdata {
> + struct xrt_subdev_file_ops xsd_file_ops;
> + struct xrt_subdev_drv_ops xsd_dev_ops;
> +};
> +
> +/*
> + * Partially initialized by the parent driver, then, passed in as subdev driver's
> + * platform data when creating subdev driver instance by calling platform
> + * device register API (platform_device_register_data() or the likes).
> + *
> + * Once device register API returns, platform driver framework makes a copy of
> + * this buffer and maintains its life cycle. The content of the buffer is
> + * completely owned by subdev driver.
> + *
> + * Thus, parent driver should be very careful when it touches this buffer
> + * again once it's handed over to subdev driver. And the data structure
> + * should not contain pointers pointing to buffers that is managed by
> + * other or parent drivers since it could have been freed before platform
> + * data buffer is freed by platform driver framework.
This sounds complicated and risky, why have two copies ?
> + */
> +struct xrt_subdev_platdata {
> + /*
> + * Per driver instance callback. The pdev points to the instance.
> + * Should always be defined for subdev driver to get service from root.
> + */
> + xrt_subdev_root_cb_t xsp_root_cb;
> + void *xsp_root_cb_arg;
> +
> + /* Something to associate w/ root for msg printing. */
> + const char *xsp_root_name;
> +
> + /*
> + * Char dev support for this subdev instance.
> + * Initialized by subdev driver.
> + */
> + struct cdev xsp_cdev;
> + struct device *xsp_sysdev;
> + struct mutex xsp_devnode_lock; /* devnode lock */
> + struct completion xsp_devnode_comp;
> + int xsp_devnode_ref;
> + bool xsp_devnode_online;
> + bool xsp_devnode_excl;
> +
> + /*
> + * Subdev driver specific init data. The buffer should be embedded
> + * in this data structure buffer after dtb, so that it can be freed
> + * together with platform data.
> + */
> + loff_t xsp_priv_off; /* Offset into this platform data buffer. */
> + size_t xsp_priv_len;
> +
> + /*
> + * Populated by parent driver to describe the device tree for
> + * the subdev driver to handle. Should always be last one since it's
> + * of variable length.
> + */
> + char xsp_dtb[sizeof(struct fdt_header)];
could be xsp_dtb[1] and save including the fdt headers just to get a size that doesn't matter.
> +};
> +
> +/*
> + * this struct define the endpoints belong to the same subdevice
> + */
> +struct xrt_subdev_ep_names {
> + const char *ep_name;
> + const char *regmap_name;
> +};
> +
> +struct xrt_subdev_endpoints {
> + struct xrt_subdev_ep_names *xse_names;
> + /* minimum number of endpoints to support the subdevice */
> + u32 xse_min_ep;
see earlier comment about needed a null entry and checking for it.
a 'size' element would be better here.
> +};
> +
> +struct subdev_match_arg {
> + enum xrt_subdev_id id;
> + int instance;
> +};
> +
> +bool xleaf_has_endpoint(struct platform_device *pdev, const char *endpoint_name);
> +struct platform_device *xleaf_get_leaf(struct platform_device *pdev,
> + xrt_subdev_match_t cb, void *arg);
> +
> +static inline bool subdev_match(enum xrt_subdev_id id, struct platform_device *pdev, void *arg)
> +{
> + const struct subdev_match_arg *a = (struct subdev_match_arg *)arg;
> + bool ret = (id == a->id && (pdev->id == a->instance || PLATFORM_DEVID_NONE == a->instance));
This statement is too complicated, turn this into an if-else
> +
> + return ret;
> +}
> +
> +static inline bool xrt_subdev_match_epname(enum xrt_subdev_id id,
> + struct platform_device *pdev, void *arg)
> +{
> + return xleaf_has_endpoint(pdev, arg);
This function is used only once.
Just inline the function to the caller and remove this function.
> +}
> +
> +static inline struct platform_device *
> +xleaf_get_leaf_by_id(struct platform_device *pdev,
> + enum xrt_subdev_id id, int instance)
> +{
> + struct subdev_match_arg arg = { id, instance };
> +
> + return xleaf_get_leaf(pdev, subdev_match, &arg);
> +}
> +
> +static inline struct platform_device *
> +xleaf_get_leaf_by_epname(struct platform_device *pdev, const char *name)
> +{
> + return xleaf_get_leaf(pdev, xrt_subdev_match_epname, (void *)name);
> +}
> +
> +static inline int xleaf_ioctl(struct platform_device *tgt, u32 cmd, void *arg)
> +{
> + struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(tgt);
> +
> + return (*drvdata->xsd_dev_ops.xsd_ioctl)(tgt, cmd, arg);
> +}
> +
> +int xleaf_put_leaf(struct platform_device *pdev,
> + struct platform_device *leaf);
> +int xleaf_create_group(struct platform_device *pdev, char *dtb);
> +int xleaf_destroy_group(struct platform_device *pdev, int instance);
> +int xleaf_wait_for_group_bringup(struct platform_device *pdev);
> +void xleaf_hot_reset(struct platform_device *pdev);
> +int xleaf_broadcast_event(struct platform_device *pdev,
> + enum xrt_events evt, bool async);
> +void xleaf_get_barres(struct platform_device *pdev,
> + struct resource **res, uint bar_idx);
> +void xleaf_get_root_id(struct platform_device *pdev,
> + unsigned short *vendor, unsigned short *device,
> + unsigned short *subvendor, unsigned short *subdevice);
> +struct device *xleaf_register_hwmon(struct platform_device *pdev,
> + const char *name, void *drvdata,
> + const struct attribute_group **grps);
> +void xleaf_unregister_hwmon(struct platform_device *pdev, struct device *hwmon);
could better organize these decl's alphabetically.
Also not intermix inlines and decls.
> +
> +/*
> + * Character device helper APIs for use by leaf drivers
> + */
> +static inline bool xleaf_devnode_enabled(struct xrt_subdev_drvdata *drvdata)
> +{
> + return drvdata && drvdata->xsd_file_ops.xsf_ops.open;
> +}
> +
> +int xleaf_devnode_create(struct platform_device *pdev,
> + const char *file_name, const char *inst_name);
> +int xleaf_devnode_destroy(struct platform_device *pdev);
> +
> +struct platform_device *xleaf_devnode_open_excl(struct inode *inode);
> +struct platform_device *xleaf_devnode_open(struct inode *inode);
> +void xleaf_devnode_close(struct inode *inode);
> +
> +/* Helpers. */
> +static inline void xrt_memcpy_fromio(void *buf, void __iomem *iomem, u32 size)
> +{
Replace with mmio_insl/outsl
> + int i;
> +
> + WARN_ON(size & 0x3);
> + for (i = 0; i < size / 4; i++)
> + ((u32 *)buf)[i] = ioread32((char *)(iomem) + sizeof(u32) * i);
> +}
> +
> +static inline void xrt_memcpy_toio(void __iomem *iomem, void *buf, u32 size)
> +{
> + int i;
> +
> + WARN_ON(size & 0x3);
> + for (i = 0; i < size / 4; i++)
> + iowrite32(((u32 *)buf)[i], ((char *)(iomem) + sizeof(u32) * i));
> +}
> +
> +int xleaf_register_driver(enum xrt_subdev_id id, struct platform_driver *drv,
> + struct xrt_subdev_endpoints *eps);
> +void xleaf_unregister_driver(enum xrt_subdev_id id);
> +
> +/* Module's init/fini routines for leaf driver in xrt-lib module */
> +void group_leaf_init_fini(bool init);
> +void vsec_leaf_init_fini(bool init);
> +void vsec_golden_leaf_init_fini(bool init);
> +void devctl_leaf_init_fini(bool init);
> +void axigate_leaf_init_fini(bool init);
> +void icap_leaf_init_fini(bool init);
> +void calib_leaf_init_fini(bool init);
> +void qspi_leaf_init_fini(bool init);
> +void mailbox_leaf_init_fini(bool init);
> +void cmc_leaf_init_fini(bool init);
> +void clkfreq_leaf_init_fini(bool init);
> +void clock_leaf_init_fini(bool init);
> +void ucs_leaf_init_fini(bool init);
Shouldn't these be in the specific leaf drv ?
> +
> +#endif /* _XRT_LEAF_H_ */
> diff --git a/drivers/fpga/xrt/lib/cdev.c b/drivers/fpga/xrt/lib/cdev.c
> new file mode 100644
> index 000000000000..7f104ab3d527
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/cdev.c
> @@ -0,0 +1,231 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA device node helper functions.
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include "xleaf.h"
> +
> +extern struct class *xrt_class;
> +
> +#define XRT_CDEV_DIR "xfpga"
'xfpga' is not very unique, maybe 'xrt' ?
> +#define INODE2PDATA(inode) \
> + container_of((inode)->i_cdev, struct xrt_subdev_platdata, xsp_cdev)
> +#define INODE2PDEV(inode) \
> + to_platform_device(kobj_to_dev((inode)->i_cdev->kobj.parent))
> +#define CDEV_NAME(sysdev) (strchr((sysdev)->kobj.name, '!') + 1)
> +
> +/* Allow it to be accessed from cdev. */
> +static void xleaf_devnode_allowed(struct platform_device *pdev)
> +{
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> +
> + /* Allow new opens. */
> + mutex_lock(&pdata->xsp_devnode_lock);
> + pdata->xsp_devnode_online = true;
> + mutex_unlock(&pdata->xsp_devnode_lock);
> +}
> +
> +/* Turn off access from cdev and wait for all existing user to go away. */
> +static int xleaf_devnode_disallowed(struct platform_device *pdev)
> +{
> + int ret = 0;
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> +
> + mutex_lock(&pdata->xsp_devnode_lock);
> +
> + /* Prevent new opens. */
> + pdata->xsp_devnode_online = false;
> + /* Wait for existing user to close. */
> + while (!ret && pdata->xsp_devnode_ref) {
> + int rc;
> +
> + mutex_unlock(&pdata->xsp_devnode_lock);
> + rc = wait_for_completion_killable(&pdata->xsp_devnode_comp);
> + mutex_lock(&pdata->xsp_devnode_lock);
> +
> + if (rc == -ERESTARTSYS) {
> + /* Restore online state. */
> + pdata->xsp_devnode_online = true;
> + xrt_err(pdev, "%s is in use, ref=%d",
> + CDEV_NAME(pdata->xsp_sysdev),
> + pdata->xsp_devnode_ref);
> + ret = -EBUSY;
> + }
> + }
> +
> + mutex_unlock(&pdata->xsp_devnode_lock);
> +
> + return ret;
> +}
> +
> +static struct platform_device *
> +__xleaf_devnode_open(struct inode *inode, bool excl)
> +{
> + struct xrt_subdev_platdata *pdata = INODE2PDATA(inode);
> + struct platform_device *pdev = INODE2PDEV(inode);
> + bool opened = false;
> +
> + mutex_lock(&pdata->xsp_devnode_lock);
> +
> + if (pdata->xsp_devnode_online) {
> + if (excl && pdata->xsp_devnode_ref) {
> + xrt_err(pdev, "%s has already been opened exclusively",
> + CDEV_NAME(pdata->xsp_sysdev));
> + } else if (!excl && pdata->xsp_devnode_excl) {
> + xrt_err(pdev, "%s has been opened exclusively",
> + CDEV_NAME(pdata->xsp_sysdev));
> + } else {
> + pdata->xsp_devnode_ref++;
> + pdata->xsp_devnode_excl = excl;
> + opened = true;
> + xrt_info(pdev, "opened %s, ref=%d",
> + CDEV_NAME(pdata->xsp_sysdev),
> + pdata->xsp_devnode_ref);
> + }
> + } else {
> + xrt_err(pdev, "%s is offline", CDEV_NAME(pdata->xsp_sysdev));
> + }
> +
> + mutex_unlock(&pdata->xsp_devnode_lock);
> +
> + pdev = opened ? pdev : NULL;
> + return pdev;
> +}
> +
> +struct platform_device *
> +xleaf_devnode_open_excl(struct inode *inode)
> +{
> + return __xleaf_devnode_open(inode, true);
> +}
> +
> +struct platform_device *
> +xleaf_devnode_open(struct inode *inode)
> +{
> + return __xleaf_devnode_open(inode, false);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_devnode_open);
generally
exported systems should have their decl's in include/linux/fpga/
These are in drivers/fpga/xrt/include/xleaf.h
as exported, they should have a better than average prefix.
maybe 'xrt_fpga_'
> +
> +void xleaf_devnode_close(struct inode *inode)
> +{
> + struct xrt_subdev_platdata *pdata = INODE2PDATA(inode);
> + struct platform_device *pdev = INODE2PDEV(inode);
> + bool notify = false;
> +
> + mutex_lock(&pdata->xsp_devnode_lock);
> +
> + pdata->xsp_devnode_ref--;
check before dec ? or at least warn if ref is already 0
> + if (pdata->xsp_devnode_ref == 0) {
> + pdata->xsp_devnode_excl = false;
> + notify = true;
> + }
> + if (notify) {
> + xrt_info(pdev, "closed %s, ref=%d",
> + CDEV_NAME(pdata->xsp_sysdev), pdata->xsp_devnode_ref);
> + } else {
> + xrt_info(pdev, "closed %s, notifying waiter",
> + CDEV_NAME(pdata->xsp_sysdev));
> + }
> +
> + mutex_unlock(&pdata->xsp_devnode_lock);
> +
> + if (notify)
> + complete(&pdata->xsp_devnode_comp);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_devnode_close);
> +
> +static inline enum xrt_subdev_file_mode
> +devnode_mode(struct xrt_subdev_drvdata *drvdata)
> +{
> + return drvdata->xsd_file_ops.xsf_mode;
> +}
> +
> +int xleaf_devnode_create(struct platform_device *pdev, const char *file_name,
> + const char *inst_name)
> +{
> + struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(pdev);
> + struct xrt_subdev_file_ops *fops = &drvdata->xsd_file_ops;
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> + struct cdev *cdevp;
> + struct device *sysdev;
> + int ret = 0;
> + char fname[256];
will a /dev/xfpga* created for ever leaf device ?
do they all really need /dev/ support ?
> +
> + mutex_init(&pdata->xsp_devnode_lock);
> + init_completion(&pdata->xsp_devnode_comp);
> +
> + cdevp = &DEV_PDATA(pdev)->xsp_cdev;
no cdev_alloc ?
> + cdev_init(cdevp, &fops->xsf_ops);
> + cdevp->owner = fops->xsf_ops.owner;
> + cdevp->dev = MKDEV(MAJOR(fops->xsf_dev_t), pdev->id);
> +
> + /*
> + * Set pdev as parent of cdev so that when pdev (and its platform
> + * data) will not be freed when cdev is not freed.
> + */
> + cdev_set_parent(cdevp, &DEV(pdev)->kobj);
> +
> + ret = cdev_add(cdevp, cdevp->dev, 1);
> + if (ret) {
> + xrt_err(pdev, "failed to add cdev: %d", ret);
> + goto failed;
> + }
> + if (!file_name)
> + file_name = pdev->name;
> + if (!inst_name) {
> + if (devnode_mode(drvdata) == XRT_SUBDEV_FILE_MULTI_INST) {
> + snprintf(fname, sizeof(fname), "%s/%s/%s.%u",
> + XRT_CDEV_DIR, DEV_PDATA(pdev)->xsp_root_name,
> + file_name, pdev->id);
> + } else {
> + snprintf(fname, sizeof(fname), "%s/%s/%s",
> + XRT_CDEV_DIR, DEV_PDATA(pdev)->xsp_root_name,
> + file_name);
> + }
> + } else {
> + snprintf(fname, sizeof(fname), "%s/%s/%s.%s", XRT_CDEV_DIR,
> + DEV_PDATA(pdev)->xsp_root_name, file_name, inst_name);
> + }
> + sysdev = device_create(xrt_class, NULL, cdevp->dev, NULL, "%s", fname);
> + if (IS_ERR(sysdev)) {
> + ret = PTR_ERR(sysdev);
> + xrt_err(pdev, "failed to create device node: %d", ret);
> + goto failed;
this calls device_destroy, but the create call failed, so is this needed ?
> + }
> + pdata->xsp_sysdev = sysdev;
> +
> + xleaf_devnode_allowed(pdev);
> +
> + xrt_info(pdev, "created (%d, %d): /dev/%s",
> + MAJOR(cdevp->dev), pdev->id, fname);
> + return 0;
> +
> +failed:
> + device_destroy(xrt_class, cdevp->dev);
> + cdev_del(cdevp);
> + cdevp->owner = NULL;
> + return ret;
> +}
> +
> +int xleaf_devnode_destroy(struct platform_device *pdev)
> +{
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> + struct cdev *cdevp = &pdata->xsp_cdev;
> + dev_t dev = cdevp->dev;
> + int rc;
> +
> + rc = xleaf_devnode_disallowed(pdev);
> + if (rc)
> + return rc;
This return is not checked by xrt_subdev_destroy
> +
> + xrt_info(pdev, "removed (%d, %d): /dev/%s/%s", MAJOR(dev), MINOR(dev),
> + XRT_CDEV_DIR, CDEV_NAME(pdata->xsp_sysdev));
> + device_destroy(xrt_class, cdevp->dev);
> + pdata->xsp_sysdev = NULL;
> + cdev_del(cdevp);
> + return 0;
> +}
> diff --git a/drivers/fpga/xrt/lib/subdev.c b/drivers/fpga/xrt/lib/subdev.c
> new file mode 100644
> index 000000000000..73552c549bdb
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/subdev.c
> @@ -0,0 +1,871 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA device helper functions
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/platform_device.h>
> +#include <linux/pci.h>
> +#include <linux/vmalloc.h>
> +#include "xleaf.h"
> +#include "subdev_pool.h"
> +#include "main.h"
> +#include "metadata.h"
> +
> +#define DEV_IS_PCI(dev) ((dev)->bus == &pci_bus_type)
> +static inline struct device *find_root(struct platform_device *pdev)
> +{
> + struct device *d = DEV(pdev);
> +
> + while (!DEV_IS_PCI(d))
> + d = d->parent;
Shouldn't the root have no parent ?
Could then check if d->parent == NULL instead of bus type
> + return d;
> +}
> +
> +/*
> + * It represents a holder of a subdev. One holder can repeatedly hold a subdev
> + * as long as there is a unhold corresponding to a hold.
> + */
> +struct xrt_subdev_holder {
> + struct list_head xsh_holder_list;
> + struct device *xsh_holder;
> + int xsh_count;
> + struct kref xsh_kref;
general, i see this in struct xrt_subdev
guessing 'xsh' is xrt subdev holder.
why is this prefix needed for the elements ? consider removing it.
> +};
> +
> +/*
> + * It represents a specific instance of platform driver for a subdev, which
> + * provides services to its clients (another subdev driver or root driver).
> + */
> +struct xrt_subdev {
> + struct list_head xs_dev_list;
> + struct list_head xs_holder_list;
> + enum xrt_subdev_id xs_id; /* type of subdev */
> + struct platform_device *xs_pdev; /* a particular subdev inst */
> + struct completion xs_holder_comp;
> +};
> +
> +static struct xrt_subdev *xrt_subdev_alloc(void)
> +{
> + struct xrt_subdev *sdev = vzalloc(sizeof(*sdev));
similar kzalloc as another patch.
> +
> + if (!sdev)
> + return NULL;
> +
> + INIT_LIST_HEAD(&sdev->xs_dev_list);
> + INIT_LIST_HEAD(&sdev->xs_holder_list);
> + init_completion(&sdev->xs_holder_comp);
> + return sdev;
> +}
> +
> +static void xrt_subdev_free(struct xrt_subdev *sdev)
> +{
> + vfree(sdev);
> +}
> +
> +int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void *arg)
> +{
> + struct device *dev = DEV(self);
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(self);
> +
> + return (*pdata->xsp_root_cb)(dev->parent, pdata->xsp_root_cb_arg, cmd, arg);
xrt_subdev_create does not check if pcb is valid. is a null is passed in, it will crash.
there should at least be a warn or -INVALID returned there
> +}
> +
> +/*
> + * Subdev common sysfs nodes.
> + */
> +static ssize_t holders_show(struct device *dev, struct device_attribute *attr, char *buf)
> +{
> + ssize_t len;
> + struct platform_device *pdev = to_platform_device(dev);
> + struct xrt_root_ioctl_get_holders holders = { pdev, buf, 1024 };
is 1024 a guess ?
> +
> + len = xrt_subdev_root_request(pdev, XRT_ROOT_GET_LEAF_HOLDERS, &holders);
take a closer look at xrt_subdev_get_holders() it stops after it goes past len.
> + if (len >= holders.xpigh_holder_buf_len)
> + return len;
> + buf[len] = '\n';
> + return len + 1;
> +}
> +static DEVICE_ATTR_RO(holders);
> +
> +static struct attribute *xrt_subdev_attrs[] = {
> + &dev_attr_holders.attr,
> + NULL,
> +};
> +
> +static ssize_t metadata_output(struct file *filp, struct kobject *kobj,
> + struct bin_attribute *attr, char *buf, loff_t off, size_t count)
> +{
> + struct device *dev = kobj_to_dev(kobj);
> + struct platform_device *pdev = to_platform_device(dev);
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(pdev);
> + unsigned char *blob;
> + unsigned long size;
> + ssize_t ret = 0;
> +
> + blob = pdata->xsp_dtb;
> + size = xrt_md_size(dev, blob);
> + if (size == XRT_MD_INVALID_LENGTH) {
> + ret = -EINVAL;
> + goto failed;
> + }
> +
> + if (off >= size)
> + goto failed;
silently failed because ret = 0 ?
> +
> + if (off + count > size)
> + count = size - off;
truncating is ok ?
> + memcpy(buf, blob + off, count);
> +
> + ret = count;
> +failed:
> + return ret;
> +}
> +
> +static struct bin_attribute meta_data_attr = {
> + .attr = {
> + .name = "metadata",
> + .mode = 0400
> + },
> + .read = metadata_output,
> + .size = 0
> +};
> +
> +static struct bin_attribute *xrt_subdev_bin_attrs[] = {
> + &meta_data_attr,
is giving the average user access to the meta data a good idea ?
this seems like a developer only need.
> + NULL,
> +};
> +
> +static const struct attribute_group xrt_subdev_attrgroup = {
> + .attrs = xrt_subdev_attrs,
> + .bin_attrs = xrt_subdev_bin_attrs,
> +};
> +
> +/*
> + * Given the device metadata, parse it to get IO ranges and construct
> + * resource array.
> + */
> +static int
> +xrt_subdev_getres(struct device *parent, enum xrt_subdev_id id,
> + char *dtb, struct resource **res, int *res_num)
> +{
> + struct xrt_subdev_platdata *pdata;
> + struct resource *pci_res = NULL;
> + const u64 *bar_range;
> + const u32 *bar_idx;
> + char *ep_name = NULL, *regmap = NULL;
> + uint bar;
> + int count1 = 0, count2 = 0, ret;
> +
> + if (!dtb)
> + return -EINVAL;
> +
> + pdata = DEV_PDATA(to_platform_device(parent));
> +
> + /* go through metadata and count endpoints in it */
> + for (xrt_md_get_next_endpoint(parent, dtb, NULL, NULL, &ep_name, ®map); ep_name;
Ugly.
Can you preprocess the dtb into a list of end points ?
> + xrt_md_get_next_endpoint(parent, dtb, ep_name, regmap, &ep_name, ®map)) {
> + ret = xrt_md_get_prop(parent, dtb, ep_name, regmap,
> + XRT_MD_PROP_IO_OFFSET, (const void **)&bar_range, NULL);
> + if (!ret)
> + count1++;
> + }
> + if (!count1)
> + return 0;
> +
> + /* allocate resource array for all endpoints been found in metadata */
> + *res = vzalloc(sizeof(**res) * count1);
> +
> + /* go through all endpoints again and get IO range for each endpoint */
> + for (xrt_md_get_next_endpoint(parent, dtb, NULL, NULL, &ep_name, ®map); ep_name;
> + xrt_md_get_next_endpoint(parent, dtb, ep_name, regmap, &ep_name, ®map)) {
> + ret = xrt_md_get_prop(parent, dtb, ep_name, regmap,
> + XRT_MD_PROP_IO_OFFSET, (const void **)&bar_range, NULL);
> + if (ret)
> + continue;
> + xrt_md_get_prop(parent, dtb, ep_name, regmap,
> + XRT_MD_PROP_BAR_IDX, (const void **)&bar_idx, NULL);
bar can fail, but bar idx can not.
should add an assert here
> + bar = bar_idx ? be32_to_cpu(*bar_idx) : 0;
> + xleaf_get_barres(to_platform_device(parent), &pci_res, bar);
> + (*res)[count2].start = pci_res->start +
> + be64_to_cpu(bar_range[0]);
> + (*res)[count2].end = pci_res->start +
> + be64_to_cpu(bar_range[0]) +
> + be64_to_cpu(bar_range[1]) - 1;
> + (*res)[count2].flags = IORESOURCE_MEM;
any irqs need handling?
> + /* check if there is conflicted resource */
> + ret = request_resource(pci_res, *res + count2);
> + if (ret) {
> + dev_err(parent, "Conflict resource %pR\n", *res + count2);
> + vfree(*res);
> + *res_num = 0;
> + *res = NULL;
> + return ret;
> + }
> + release_resource(*res + count2);
> +
> + (*res)[count2].parent = pci_res;
> +
> + xrt_md_find_endpoint(parent, pdata->xsp_dtb, ep_name,
> + regmap, &(*res)[count2].name);
> +
> + count2++;
> + }
> +
> + WARN_ON(count1 != count2);
> + *res_num = count2;
> +
> + return 0;
> +}
> +
> +static inline enum xrt_subdev_file_mode
> +xleaf_devnode_mode(struct xrt_subdev_drvdata *drvdata)
> +{
> + return drvdata->xsd_file_ops.xsf_mode;
> +}
> +
> +static bool xrt_subdev_cdev_auto_creation(struct platform_device *pdev)
> +{
> + struct xrt_subdev_drvdata *drvdata = DEV_DRVDATA(pdev);
> +
> + if (!drvdata)
> + return false;
> +
> + return xleaf_devnode_enabled(drvdata) &&
> + (xleaf_devnode_mode(drvdata) == XRT_SUBDEV_FILE_DEFAULT ||
> + (xleaf_devnode_mode(drvdata) == XRT_SUBDEV_FILE_MULTI_INST));
This is complicated to check, split into checking the call and then checking its side effects.
> +}
> +
> +static struct xrt_subdev *
> +xrt_subdev_create(struct device *parent, enum xrt_subdev_id id,
> + xrt_subdev_root_cb_t pcb, void *pcb_arg, char *dtb)
> +{
> + struct xrt_subdev *sdev = NULL;
> + struct platform_device *pdev = NULL;
> + struct xrt_subdev_platdata *pdata = NULL;
> + unsigned long dtb_len = 0;
> + size_t pdata_sz;
> + int inst = PLATFORM_DEVID_NONE;
> + struct resource *res = NULL;
> + int res_num = 0;
> +
> + sdev = xrt_subdev_alloc();
> + if (!sdev) {
> + dev_err(parent, "failed to alloc subdev for ID %d", id);
> + goto fail;
> + }
> + sdev->xs_id = id;
> +
> + if (dtb) {
> + xrt_md_pack(parent, dtb);
> + dtb_len = xrt_md_size(parent, dtb);
> + if (dtb_len == XRT_MD_INVALID_LENGTH) {
> + dev_err(parent, "invalid metadata len %ld", dtb_len);
> + goto fail;
> + }
> + }
> + pdata_sz = sizeof(struct xrt_subdev_platdata) + dtb_len - 1;
-1 ?
if dtb_len == 0, pdata_sz be too small.
> +
> + /* Prepare platform data passed to subdev. */
> + pdata = vzalloc(pdata_sz);
> + if (!pdata)
> + goto fail;
> +
> + pdata->xsp_root_cb = pcb;
> + pdata->xsp_root_cb_arg = pcb_arg;
> + memcpy(pdata->xsp_dtb, dtb, dtb_len);
> + if (id == XRT_SUBDEV_GRP) {
> + /* Group can only be created by root driver. */
> + pdata->xsp_root_name = dev_name(parent);
> + } else {
> + struct platform_device *grp = to_platform_device(parent);
> + /* Leaf can only be created by group driver. */
> + WARN_ON(strcmp(xrt_drv_name(XRT_SUBDEV_GRP), platform_get_device_id(grp)->name));
> + pdata->xsp_root_name = DEV_PDATA(grp)->xsp_root_name;
> + }
> +
> + /* Obtain dev instance number. */
> + inst = xrt_drv_get_instance(id);
> + if (inst < 0) {
> + dev_err(parent, "failed to obtain instance: %d", inst);
> + goto fail;
> + }
> +
> + /* Create subdev. */
> + if (id == XRT_SUBDEV_GRP) {
> + pdev = platform_device_register_data(parent, xrt_drv_name(XRT_SUBDEV_GRP),
> + inst, pdata, pdata_sz);
> + } else {
> + int rc = xrt_subdev_getres(parent, id, dtb, &res, &res_num);
> +
> + if (rc) {
> + dev_err(parent, "failed to get resource for %s.%d: %d",
> + xrt_drv_name(id), inst, rc);
> + goto fail;
> + }
> + pdev = platform_device_register_resndata(parent, xrt_drv_name(id),
> + inst, res, res_num, pdata, pdata_sz);
> + vfree(res);
> + }
a small optimization
platform_device_register_data is a wrapper to platform_device_register_resndata.
with initial values for res, res_num, just one call need to be made.
> + if (IS_ERR(pdev)) {
> + dev_err(parent, "failed to create subdev for %s inst %d: %ld",
> + xrt_drv_name(id), inst, PTR_ERR(pdev));
> + goto fail;
> + }
> + sdev->xs_pdev = pdev;
> +
> + if (device_attach(DEV(pdev)) != 1) {
> + xrt_err(pdev, "failed to attach");
> + goto fail;
> + }
> +
> + if (sysfs_create_group(&DEV(pdev)->kobj, &xrt_subdev_attrgroup))
> + xrt_err(pdev, "failed to create sysfs group");
no failure ?
> +
> + /*
> + * Create sysfs sym link under root for leaves
> + * under random groups for easy access to them.
> + */
> + if (id != XRT_SUBDEV_GRP) {
> + if (sysfs_create_link(&find_root(pdev)->kobj,
> + &DEV(pdev)->kobj, dev_name(DEV(pdev)))) {
> + xrt_err(pdev, "failed to create sysfs link");
> + }
> + }
> +
> + /* All done, ready to handle req thru cdev. */
> + if (xrt_subdev_cdev_auto_creation(pdev))
> + xleaf_devnode_create(pdev, DEV_DRVDATA(pdev)->xsd_file_ops.xsf_dev_name, NULL);
> +
> + vfree(pdata);
> + return sdev;
> +
> +fail:
Instead of adding checks in the error handling block, add more specific labels and gotos.
I think i have noticed this before, so apply this advice generally.
> + vfree(pdata);
> + if (sdev && !IS_ERR_OR_NULL(sdev->xs_pdev))
> + platform_device_unregister(sdev->xs_pdev);
> + if (inst >= 0)
> + xrt_drv_put_instance(id, inst);
> + xrt_subdev_free(sdev);
> + return NULL;
> +}
> +
> +static void xrt_subdev_destroy(struct xrt_subdev *sdev)
> +{
> + struct platform_device *pdev = sdev->xs_pdev;
> + int inst = pdev->id;
> + struct device *dev = DEV(pdev);
> +
> + /* Take down the device node */
> + if (xrt_subdev_cdev_auto_creation(pdev))
> + xleaf_devnode_destroy(pdev);
> + if (sdev->xs_id != XRT_SUBDEV_GRP)
> + sysfs_remove_link(&find_root(pdev)->kobj, dev_name(dev));
> + sysfs_remove_group(&dev->kobj, &xrt_subdev_attrgroup);
> + platform_device_unregister(pdev);
> + xrt_drv_put_instance(sdev->xs_id, inst);
> + xrt_subdev_free(sdev);
> +}
> +
> +struct platform_device *
> +xleaf_get_leaf(struct platform_device *pdev, xrt_subdev_match_t match_cb, void *match_arg)
> +{
> + int rc;
> + struct xrt_root_ioctl_get_leaf get_leaf = {
> + pdev, match_cb, match_arg, };
> +
> + rc = xrt_subdev_root_request(pdev, XRT_ROOT_GET_LEAF, &get_leaf);
> + if (rc)
> + return NULL;
> + return get_leaf.xpigl_leaf;
> +}
> +EXPORT_SYMBOL_GPL(xleaf_get_leaf);
> +
> +bool xleaf_has_endpoint(struct platform_device *pdev, const char *endpoint_name)
> +{
> + struct resource *res;
> + int i;
whitespace
> +
> + for (i = 0, res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + res;
> + res = platform_get_resource(pdev, IORESOURCE_MEM, ++i)) {
Do not inc i inside the call, do it at the bottom of the loop
> + if (!strncmp(res->name, endpoint_name, strlen(res->name) + 1))
shouldn't you also check the strlen matches ?
> + return true;
> + }
> +
> + return false;
> +}
> +EXPORT_SYMBOL_GPL(xleaf_has_endpoint);
> +
> +int xleaf_put_leaf(struct platform_device *pdev, struct platform_device *leaf)
> +{
> + struct xrt_root_ioctl_put_leaf put_leaf = { pdev, leaf };
> +
> + return xrt_subdev_root_request(pdev, XRT_ROOT_PUT_LEAF, &put_leaf);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_put_leaf);
> +
> +int xleaf_create_group(struct platform_device *pdev, char *dtb)
> +{
> + return xrt_subdev_root_request(pdev, XRT_ROOT_CREATE_GROUP, dtb);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_create_group);
> +
> +int xleaf_destroy_group(struct platform_device *pdev, int instance)
> +{
> + return xrt_subdev_root_request(pdev, XRT_ROOT_REMOVE_GROUP, (void *)(uintptr_t)instance);
Instead of these clunky casts, why not make the type of the args void *
and leave it to the handler to cast.
this would unify the signature of these functions somewhat.
> +}
> +EXPORT_SYMBOL_GPL(xleaf_destroy_group);
> +
> +int xleaf_wait_for_group_bringup(struct platform_device *pdev)
> +{
> + return xrt_subdev_root_request(pdev, XRT_ROOT_WAIT_GROUP_BRINGUP, NULL);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_wait_for_group_bringup);
> +
> +static ssize_t
> +xrt_subdev_get_holders(struct xrt_subdev *sdev, char *buf, size_t len)
> +{
> + const struct list_head *ptr;
> + struct xrt_subdev_holder *h;
> + ssize_t n = 0;
> +
> + list_for_each(ptr, &sdev->xs_holder_list) {
> + h = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list);
> + n += snprintf(buf + n, len - n, "%s:%d ",
> + dev_name(h->xsh_holder), kref_read(&h->xsh_kref));
> + if (n >= (len - 1))
This is the overrun i mentioned above.
> + break;
> + }
> + return n;
> +}
> +
> +void xrt_subdev_pool_init(struct device *dev, struct xrt_subdev_pool *spool)
> +{
> + INIT_LIST_HEAD(&spool->xsp_dev_list);
> + spool->xsp_owner = dev;
> + mutex_init(&spool->xsp_lock);
> + spool->xsp_closing = false;
> +}
> +
> +static void xrt_subdev_free_holder(struct xrt_subdev_holder *holder)
> +{
> + list_del(&holder->xsh_holder_list);
> + vfree(holder);
> +}
> +
> +static void xrt_subdev_pool_wait_for_holders(struct xrt_subdev_pool *spool, struct xrt_subdev *sdev)
> +{
> + const struct list_head *ptr, *next;
> + char holders[128];
> + struct xrt_subdev_holder *holder;
> + struct mutex *lk = &spool->xsp_lock;
> +
> + WARN_ON(!mutex_is_locked(lk));
> +
> + while (!list_empty(&sdev->xs_holder_list)) {
> + int rc;
> +
> + /* It's most likely a bug if we ever enters this loop. */
> + xrt_subdev_get_holders(sdev, holders, sizeof(holders));
will overrun, error not reported.
> + xrt_err(sdev->xs_pdev, "awaits holders: %s", holders);
> + mutex_unlock(lk);
> + rc = wait_for_completion_killable(&sdev->xs_holder_comp);
> + mutex_lock(lk);
> + if (rc == -ERESTARTSYS) {
> + xrt_err(sdev->xs_pdev, "give up on waiting for holders, clean up now");
> + list_for_each_safe(ptr, next, &sdev->xs_holder_list) {
> + holder = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list);
> + xrt_subdev_free_holder(holder);
> + }
> + }
> + }
> +}
> +
> +void xrt_subdev_pool_fini(struct xrt_subdev_pool *spool)
> +{
> + struct list_head *dl = &spool->xsp_dev_list;
> + struct mutex *lk = &spool->xsp_lock;
> +
> + mutex_lock(lk);
> +
i am wondering about the locking here.
xsp_closing is only set to true in this function.
the unlocking then relocking in the loop is strange, why do you need to do this ?
> + if (spool->xsp_closing) {
> + mutex_unlock(lk);
> + return;
> + }
> +
> + spool->xsp_closing = true;
> + /* Remove subdev in the reverse order of added. */
> + while (!list_empty(dl)) {
> + struct xrt_subdev *sdev = list_first_entry(dl, struct xrt_subdev, xs_dev_list);
> +
> + xrt_subdev_pool_wait_for_holders(spool, sdev);
> + list_del(&sdev->xs_dev_list);
> + mutex_unlock(lk);
> + xrt_subdev_destroy(sdev);
> + mutex_lock(lk);
> + }
> +
> + mutex_unlock(lk);
> +}
> +
> +static struct xrt_subdev_holder *xrt_subdev_find_holder(struct xrt_subdev *sdev,
> + struct device *holder_dev)
> +{
> + struct list_head *hl = &sdev->xs_holder_list;
> + struct xrt_subdev_holder *holder;
> + const struct list_head *ptr;
> +
> + list_for_each(ptr, hl) {
> + holder = list_entry(ptr, struct xrt_subdev_holder, xsh_holder_list);
> + if (holder->xsh_holder == holder_dev)
> + return holder;
> + }
> + return NULL;
> +}
> +
> +static int xrt_subdev_hold(struct xrt_subdev *sdev, struct device *holder_dev)
> +{
> + struct xrt_subdev_holder *holder = xrt_subdev_find_holder(sdev, holder_dev);
> + struct list_head *hl = &sdev->xs_holder_list;
> +
> + if (!holder) {
> + holder = vzalloc(sizeof(*holder));
> + if (!holder)
> + return -ENOMEM;
> + holder->xsh_holder = holder_dev;
> + kref_init(&holder->xsh_kref);
> + list_add_tail(&holder->xsh_holder_list, hl);
> + } else {
> + kref_get(&holder->xsh_kref);
> + }
> +
> + return 0;
> +}
> +
> +static void xrt_subdev_free_holder_kref(struct kref *kref)
> +{
> + struct xrt_subdev_holder *holder = container_of(kref, struct xrt_subdev_holder, xsh_kref);
> +
> + xrt_subdev_free_holder(holder);
> +}
> +
> +static int
> +xrt_subdev_release(struct xrt_subdev *sdev, struct device *holder_dev)
> +{
> + struct xrt_subdev_holder *holder = xrt_subdev_find_holder(sdev, holder_dev);
> + struct list_head *hl = &sdev->xs_holder_list;
> +
> + if (!holder) {
> + dev_err(holder_dev, "can't release, %s did not hold %s",
> + dev_name(holder_dev), dev_name(DEV(sdev->xs_pdev)));
> + return -EINVAL;
> + }
> + kref_put(&holder->xsh_kref, xrt_subdev_free_holder_kref);
> +
> + /* kref_put above may remove holder from list. */
> + if (list_empty(hl))
> + complete(&sdev->xs_holder_comp);
> + return 0;
> +}
> +
> +int xrt_subdev_pool_add(struct xrt_subdev_pool *spool, enum xrt_subdev_id id,
> + xrt_subdev_root_cb_t pcb, void *pcb_arg, char *dtb)
> +{
> + struct mutex *lk = &spool->xsp_lock;
> + struct list_head *dl = &spool->xsp_dev_list;
> + struct xrt_subdev *sdev;
> + int ret = 0;
> +
> + sdev = xrt_subdev_create(spool->xsp_owner, id, pcb, pcb_arg, dtb);
> + if (sdev) {
> + mutex_lock(lk);
> + if (spool->xsp_closing) {
> + /* No new subdev when pool is going away. */
> + xrt_err(sdev->xs_pdev, "pool is closing");
> + ret = -ENODEV;
> + } else {
> + list_add(&sdev->xs_dev_list, dl);
> + }
> + mutex_unlock(lk);
> + if (ret)
> + xrt_subdev_destroy(sdev);
> + } else {
> + ret = -EINVAL;
> + }
> +
> + ret = ret ? ret : sdev->xs_pdev->id;
> + return ret;
> +}
> +
> +int xrt_subdev_pool_del(struct xrt_subdev_pool *spool, enum xrt_subdev_id id, int instance)
> +{
> + const struct list_head *ptr;
> + struct mutex *lk = &spool->xsp_lock;
> + struct list_head *dl = &spool->xsp_dev_list;
> + struct xrt_subdev *sdev;
> + int ret = -ENOENT;
> +
> + mutex_lock(lk);
> + list_for_each(ptr, dl) {
> + sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
> + if (sdev->xs_id != id || sdev->xs_pdev->id != instance)
> + continue;
> + xrt_subdev_pool_wait_for_holders(spool, sdev);
> + list_del(&sdev->xs_dev_list);
> + ret = 0;
> + break;
> + }
> + mutex_unlock(lk);
> + if (ret)
> + return ret;
> +
> + xrt_subdev_destroy(sdev);
> + return 0;
> +}
> +
> +static int xrt_subdev_pool_get_impl(struct xrt_subdev_pool *spool, xrt_subdev_match_t match,
> + void *arg, struct device *holder_dev, struct xrt_subdev **sdevp)
> +{
> + const struct list_head *ptr;
> + struct mutex *lk = &spool->xsp_lock;
> + struct list_head *dl = &spool->xsp_dev_list;
> + struct xrt_subdev *sdev = NULL;
> + int ret = -ENOENT;
> +
> + mutex_lock(lk);
> +
> + if (match == XRT_SUBDEV_MATCH_PREV) {
> + struct platform_device *pdev = (struct platform_device *)arg;
> + struct xrt_subdev *d = NULL;
> +
> + if (!pdev) {
> + sdev = list_empty(dl) ? NULL :
> + list_last_entry(dl, struct xrt_subdev, xs_dev_list);
> + } else {
> + list_for_each(ptr, dl) {
> + d = list_entry(ptr, struct xrt_subdev, xs_dev_list);
> + if (d->xs_pdev != pdev)
> + continue;
> + if (!list_is_first(ptr, dl))
> + sdev = list_prev_entry(d, xs_dev_list);
> + break;
> + }
> + }
> + } else if (match == XRT_SUBDEV_MATCH_NEXT) {
> + struct platform_device *pdev = (struct platform_device *)arg;
> + struct xrt_subdev *d = NULL;
> +
> + if (!pdev) {
> + sdev = list_first_entry_or_null(dl, struct xrt_subdev, xs_dev_list);
> + } else {
> + list_for_each(ptr, dl) {
> + d = list_entry(ptr, struct xrt_subdev, xs_dev_list);
> + if (d->xs_pdev != pdev)
> + continue;
> + if (!list_is_last(ptr, dl))
> + sdev = list_next_entry(d, xs_dev_list);
> + break;
> + }
> + }
> + } else {
> + list_for_each(ptr, dl) {
> + struct xrt_subdev *d = NULL;
> +
> + d = list_entry(ptr, struct xrt_subdev, xs_dev_list);
> + if (d && !match(d->xs_id, d->xs_pdev, arg))
> + continue;
> + sdev = d;
> + break;
> + }
> + }
3 similar blocks of code
This looks like it could be refactored into this else case and minor changes for match_next/prev
> +
> + if (sdev)
> + ret = xrt_subdev_hold(sdev, holder_dev);
> +
> + mutex_unlock(lk);
> +
> + if (!ret)
> + *sdevp = sdev;
> + return ret;
> +}
> +
> +int xrt_subdev_pool_get(struct xrt_subdev_pool *spool, xrt_subdev_match_t match, void *arg,
> + struct device *holder_dev, struct platform_device **pdevp)
> +{
> + int rc;
> + struct xrt_subdev *sdev;
> +
> + rc = xrt_subdev_pool_get_impl(spool, match, arg, holder_dev, &sdev);
> + if (rc) {
> + if (rc != -ENOENT)
> + dev_err(holder_dev, "failed to hold device: %d", rc);
> + return rc;
> + }
> +
> + if (!DEV_IS_PCI(holder_dev)) {
! root_dev()
> + xrt_dbg(to_platform_device(holder_dev), "%s <<==== %s",
> + dev_name(holder_dev), dev_name(DEV(sdev->xs_pdev)));
> + }
> +
> + *pdevp = sdev->xs_pdev;
> + return 0;
> +}
> +
> +static int xrt_subdev_pool_put_impl(struct xrt_subdev_pool *spool, struct platform_device *pdev,
> + struct device *holder_dev)
> +{
> + const struct list_head *ptr;
> + struct mutex *lk = &spool->xsp_lock;
> + struct list_head *dl = &spool->xsp_dev_list;
> + struct xrt_subdev *sdev;
> + int ret = -ENOENT;
> +
> + mutex_lock(lk);
> + list_for_each(ptr, dl) {
> + sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
> + if (sdev->xs_pdev != pdev)
> + continue;
Could this and similar looping be avoided by storing sdev in pdev ?
> + ret = xrt_subdev_release(sdev, holder_dev);
> + break;
> + }
> + mutex_unlock(lk);
> +
> + return ret;
> +}
> +
> +int xrt_subdev_pool_put(struct xrt_subdev_pool *spool, struct platform_device *pdev,
> + struct device *holder_dev)
> +{
> + int ret = xrt_subdev_pool_put_impl(spool, pdev, holder_dev);
> +
> + if (ret)
> + return ret;
> +
> + if (!DEV_IS_PCI(holder_dev)) {
! root_dev() or similar.
If you really need to use DEV_IS_PCI, do it only once so when you need to change something you don not have to find all the calls to DEV_IS_PCI.
> + xrt_dbg(to_platform_device(holder_dev), "%s <<==X== %s",
> + dev_name(holder_dev), dev_name(DEV(pdev)));
> + }
> + return 0;
> +}
> +
> +void xrt_subdev_pool_trigger_event(struct xrt_subdev_pool *spool, enum xrt_events e)
> +{
> + struct platform_device *tgt = NULL;
> + struct xrt_subdev *sdev = NULL;
> + struct xrt_event evt;
> +
> + while (!xrt_subdev_pool_get_impl(spool, XRT_SUBDEV_MATCH_NEXT,
> + tgt, spool->xsp_owner, &sdev)) {
> + tgt = sdev->xs_pdev;
> + evt.xe_evt = e;
> + evt.xe_subdev.xevt_subdev_id = sdev->xs_id;
> + evt.xe_subdev.xevt_subdev_instance = tgt->id;
> + xrt_subdev_root_request(tgt, XRT_ROOT_EVENT, &evt);
> + xrt_subdev_pool_put_impl(spool, tgt, spool->xsp_owner);
> + }
> +}
> +
> +void xrt_subdev_pool_handle_event(struct xrt_subdev_pool *spool, struct xrt_event *evt)
> +{
> + struct platform_device *tgt = NULL;
> + struct xrt_subdev *sdev = NULL;
> +
> + while (!xrt_subdev_pool_get_impl(spool, XRT_SUBDEV_MATCH_NEXT,
> + tgt, spool->xsp_owner, &sdev)) {
> + tgt = sdev->xs_pdev;
> + xleaf_ioctl(tgt, XRT_XLEAF_EVENT, evt);
> + xrt_subdev_pool_put_impl(spool, tgt, spool->xsp_owner);
> + }
> +}
> +
> +ssize_t xrt_subdev_pool_get_holders(struct xrt_subdev_pool *spool,
> + struct platform_device *pdev, char *buf, size_t len)
> +{
> + const struct list_head *ptr;
> + struct mutex *lk = &spool->xsp_lock;
> + struct list_head *dl = &spool->xsp_dev_list;
> + struct xrt_subdev *sdev;
> + ssize_t ret = 0;
> +
> + mutex_lock(lk);
> + list_for_each(ptr, dl) {
> + sdev = list_entry(ptr, struct xrt_subdev, xs_dev_list);
> + if (sdev->xs_pdev != pdev)
> + continue;
> + ret = xrt_subdev_get_holders(sdev, buf, len);
> + break;
> + }
> + mutex_unlock(lk);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xrt_subdev_pool_get_holders);
> +
> +int xleaf_broadcast_event(struct platform_device *pdev, enum xrt_events evt, bool async)
> +{
> + struct xrt_event e = { evt, };
> + u32 cmd = async ? XRT_ROOT_EVENT_ASYNC : XRT_ROOT_EVENT;
> +
> + WARN_ON(evt == XRT_EVENT_POST_CREATION || evt == XRT_EVENT_PRE_REMOVAL);
> + return xrt_subdev_root_request(pdev, cmd, &e);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_broadcast_event);
> +
> +void xleaf_hot_reset(struct platform_device *pdev)
> +{
> + xrt_subdev_root_request(pdev, XRT_ROOT_HOT_RESET, NULL);
> +}
> +EXPORT_SYMBOL_GPL(xleaf_hot_reset);
> +
> +void xleaf_get_barres(struct platform_device *pdev, struct resource **res, uint bar_idx)
> +{
> + struct xrt_root_ioctl_get_res arg = { 0 };
> +
> + if (bar_idx > PCI_STD_RESOURCE_END) {
> + xrt_err(pdev, "Invalid bar idx %d", bar_idx);
> + *res = NULL;
> + return;
> + }
> +
> + xrt_subdev_root_request(pdev, XRT_ROOT_GET_RESOURCE, &arg);
> +
> + *res = &arg.xpigr_res[bar_idx];
is this correct ?
do all res need to be found to return a single one ?
> +}
> +
> +void xleaf_get_root_id(struct platform_device *pdev, unsigned short *vendor, unsigned short *device,
> + unsigned short *subvendor, unsigned short *subdevice)
> +{
> + struct xrt_root_ioctl_get_id id = { 0 };
> +
> + xrt_subdev_root_request(pdev, XRT_ROOT_GET_ID, (void *)&id);
> + if (vendor)
> + *vendor = id.xpigi_vendor_id;
> + if (device)
> + *device = id.xpigi_device_id;
> + if (subvendor)
> + *subvendor = id.xpigi_sub_vendor_id;
> + if (subdevice)
> + *subdevice = id.xpigi_sub_device_id;
not setting anything because user passed in all nulls would make this function a noop.
> +}
> +
> +struct device *xleaf_register_hwmon(struct platform_device *pdev, const char *name, void *drvdata,
> + const struct attribute_group **grps)
> +{
> + struct xrt_root_ioctl_hwmon hm = { true, name, drvdata, grps, };
> +
> + xrt_subdev_root_request(pdev, XRT_ROOT_HWMON, (void *)&hm);
> + return hm.xpih_hwmon_dev;
> +}
> +
> +void xleaf_unregister_hwmon(struct platform_device *pdev, struct device *hwmon)
> +{
> + struct xrt_root_ioctl_hwmon hm = { false, };
> +
> + hm.xpih_hwmon_dev = hwmon;
> + xrt_subdev_root_request(pdev, XRT_ROOT_HWMON, (void *)&hm);
> +}
> diff --git a/drivers/fpga/xrt/lib/subdev_pool.h b/drivers/fpga/xrt/lib/subdev_pool.h
> new file mode 100644
> index 000000000000..50a8490e0e41
apologies for delay, was busy.
If it seems like i forgot a train of thought, i did.
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/subdev_pool.h
> @@ -0,0 +1,53 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_SUBDEV_POOL_H_
> +#define _XRT_SUBDEV_POOL_H_
> +
> +#include "xroot.h"
> +
> +/*
> + * It manages a list of xrt_subdevs for root and group drivers.
'It' does not have a lot of context, better would be
The xrt_subdev_pool struct ..
> + */
> +struct xrt_subdev_pool {
> + struct list_head xsp_dev_list;
> + struct device *xsp_owner;
> + struct mutex xsp_lock; /* pool lock */
Header files should be self contained, a quick look at xroot.h makes me suspicious that device and mutex decls assume the includer has added their headers before this one
> + bool xsp_closing;
If you thing additional state will be needed, you could change this to a bitfield. sizewise with compiler padding i don't think the size would change.
> +};
> +
> +/*
> + * Subdev pool API for root and group drivers only.
'API' makes me think these should go in include/linux/fpga
Do/will these functions get called outside of the drivers/fpga ?
> + */
> +void xrt_subdev_pool_init(struct device *dev,
> + struct xrt_subdev_pool *spool);
> +void xrt_subdev_pool_fini(struct xrt_subdev_pool *spool);
> +int xrt_subdev_pool_get(struct xrt_subdev_pool *spool,
> + xrt_subdev_match_t match,
> + void *arg, struct device *holder_dev,
> + struct platform_device **pdevp);
> +int xrt_subdev_pool_put(struct xrt_subdev_pool *spool,
> + struct platform_device *pdev,
> + struct device *holder_dev);
> +int xrt_subdev_pool_add(struct xrt_subdev_pool *spool,
> + enum xrt_subdev_id id, xrt_subdev_root_cb_t pcb,
> + void *pcb_arg, char *dtb);
> +int xrt_subdev_pool_del(struct xrt_subdev_pool *spool,
> + enum xrt_subdev_id id, int instance);
> +ssize_t xrt_subdev_pool_get_holders(struct xrt_subdev_pool *spool,
> + struct platform_device *pdev,
> + char *buf, size_t len);
> +
> +void xrt_subdev_pool_trigger_event(struct xrt_subdev_pool *spool,
> + enum xrt_events evt);
> +void xrt_subdev_pool_handle_event(struct xrt_subdev_pool *spool,
> + struct xrt_event *evt);
> +
> +#endif /* _XRT_SUBDEV_POOL_H_ */
> diff --git a/drivers/fpga/xrt/lib/xroot.c b/drivers/fpga/xrt/lib/xroot.c
> new file mode 100644
> index 000000000000..3dc7b0243277
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xroot.c
> @@ -0,0 +1,598 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA Root Functions
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/hwmon.h>
> +#include "xroot.h"
> +#include "subdev_pool.h"
> +#include "group.h"
> +#include "metadata.h"
> +
> +#define XROOT_PDEV(xr) ((xr)->pdev)
> +#define XROOT_DEV(xr) (&(XROOT_PDEV(xr)->dev))
> +#define xroot_err(xr, fmt, args...) \
> + dev_err(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
> +#define xroot_warn(xr, fmt, args...) \
> + dev_warn(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
> +#define xroot_info(xr, fmt, args...) \
> + dev_info(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
> +#define xroot_dbg(xr, fmt, args...) \
> + dev_dbg(XROOT_DEV(xr), "%s: " fmt, __func__, ##args)
> +
> +#define XRT_VSEC_ID 0x20
Is this the best place to define some pci magic ?
It looks like the xroot is combination of the root of the device tree and the pci setup for the board.
Can the pci-ness be split and the root mostly handling how the subtrees are organized ?
> +
> +#define XROOT_GRP_FIRST (-1)
> +#define XROOT_GRP_LAST (-2)
> +
> +static int xroot_root_cb(struct device *, void *, u32, void *);
> +
> +struct xroot_evt {
> + struct list_head list;
> + struct xrt_event evt;
> + struct completion comp;
> + bool async;
> +};
> +
> +struct xroot_events {
> + struct mutex evt_lock; /* event lock */
> + struct list_head evt_list;
> + struct work_struct evt_work;
> +};
> +
> +struct xroot_grps {
> + struct xrt_subdev_pool pool;
> + struct work_struct bringup_work;
> + atomic_t bringup_pending;
> + atomic_t bringup_failed;
combine with bitfield
> + struct completion bringup_comp;
> +};
> +
> +struct xroot {
> + struct pci_dev *pdev;
> + struct xroot_events events;
> + struct xroot_grps grps;
> + struct xroot_pf_cb pf_cb;
expand pf_cb, maybe 'physical_function_callback' ?
> +};
> +
> +struct xroot_grp_match_arg {
> + enum xrt_subdev_id id;
> + int instance;
> +};
> +
> +static bool xroot_grp_match(enum xrt_subdev_id id,
> + struct platform_device *pdev, void *arg)
> +{
> + struct xroot_grp_match_arg *a = (struct xroot_grp_match_arg *)arg;
> + return id == a->id && pdev->id == a->instance;
scanning the code i expected to see ... && pdev->instance == a->instance
pdev->id == a->instance looks like a bug, a change to pdev->id element name to pdev->instance or in needed of a comment.
> +}
> +
> +static int xroot_get_group(struct xroot *xr, int instance,
> + struct platform_device **grpp)
> +{
> + int rc = 0;
> + struct xrt_subdev_pool *grps = &xr->grps.pool;
> + struct device *dev = DEV(xr->pdev);
> + struct xroot_grp_match_arg arg = { XRT_SUBDEV_GRP, instance };
> +
> + if (instance == XROOT_GRP_LAST) {
> + rc = xrt_subdev_pool_get(grps, XRT_SUBDEV_MATCH_NEXT,
> + *grpp, dev, grpp);
> + } else if (instance == XROOT_GRP_FIRST) {
> + rc = xrt_subdev_pool_get(grps, XRT_SUBDEV_MATCH_PREV,
> + *grpp, dev, grpp);
For consistency, maybe the suffix of ...MATCH_NEXT/PREV should be changed to LAST/FIRST
> + } else {
> + rc = xrt_subdev_pool_get(grps, xroot_grp_match,
> + &arg, dev, grpp);
> + }
> +
> + if (rc && rc != -ENOENT)
> + xroot_err(xr, "failed to hold group %d: %d", instance, rc);
> + return rc;
> +}
> +
> +static void xroot_put_group(struct xroot *xr, struct platform_device *grp)
> +{
> + int inst = grp->id;
> + int rc = xrt_subdev_pool_put(&xr->grps.pool, grp, DEV(xr->pdev));
> +
> + if (rc)
> + xroot_err(xr, "failed to release group %d: %d", inst, rc);
> +}
> +
> +static int xroot_trigger_event(struct xroot *xr,
> + struct xrt_event *e, bool async)
> +{
> + struct xroot_evt *enew = vzalloc(sizeof(*enew));
> +
> + if (!enew)
> + return -ENOMEM;
> +
> + enew->evt = *e;
> + enew->async = async;
> + init_completion(&enew->comp);
> +
> + mutex_lock(&xr->events.evt_lock);
> + list_add(&enew->list, &xr->events.evt_list);
> + mutex_unlock(&xr->events.evt_lock);
> +
> + schedule_work(&xr->events.evt_work);
> +
> + if (async)
> + return 0;
> +
> + wait_for_completion(&enew->comp);
> + vfree(enew);
> + return 0;
> +}
> +
> +static void
> +xroot_group_trigger_event(struct xroot *xr, int inst, enum xrt_events e)
> +{
> + int ret;
> + struct platform_device *pdev = NULL;
> + struct xrt_event evt = { 0 };
> +
> + WARN_ON(inst < 0);
> + /* Only triggers subdev specific events. */
> + if (e != XRT_EVENT_POST_CREATION && e != XRT_EVENT_PRE_REMOVAL) {
> + xroot_err(xr, "invalid event %d", e);
> + return;
> + }
> +
> + ret = xroot_get_group(xr, inst, &pdev);
> + if (ret)
> + return;
> +
> + /* Triggers event for children, first. */
> + (void)xleaf_ioctl(pdev, XRT_GROUP_TRIGGER_EVENT, (void *)(uintptr_t)e);
These voids are not needed, but maybe error checking is.
> +
> + /* Triggers event for itself. */
> + evt.xe_evt = e;
> + evt.xe_subdev.xevt_subdev_id = XRT_SUBDEV_GRP;
> + evt.xe_subdev.xevt_subdev_instance = inst;
> + (void)xroot_trigger_event(xr, &evt, false);
> +
> + (void)xroot_put_group(xr, pdev);
> +}
> +
> +int xroot_create_group(void *root, char *dtb)
> +{
> + struct xroot *xr = (struct xroot *)root;
> + int ret;
> +
> + atomic_inc(&xr->grps.bringup_pending);
could this state and the error be moved to xrt_sbudev_pool_add where locking happens so atomics are not needed ?
> + ret = xrt_subdev_pool_add(&xr->grps.pool, XRT_SUBDEV_GRP,
> + xroot_root_cb, xr, dtb);
> + if (ret >= 0) {
> + schedule_work(&xr->grps.bringup_work);
> + } else {
> + atomic_dec(&xr->grps.bringup_pending);
> + atomic_inc(&xr->grps.bringup_failed);
> + xroot_err(xr, "failed to create group: %d", ret);
> + }
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xroot_create_group);
> +
> +static int xroot_destroy_single_group(struct xroot *xr, int instance)
> +{
A better name would be 'xroot_destroy_group'
> + struct platform_device *pdev = NULL;
> + int ret;
> +
> + WARN_ON(instance < 0);
> + ret = xroot_get_group(xr, instance, &pdev);
> + if (ret)
> + return ret;
> +
> + xroot_group_trigger_event(xr, instance, XRT_EVENT_PRE_REMOVAL);
> +
> + /* Now tear down all children in this group. */
> + ret = xleaf_ioctl(pdev, XRT_GROUP_FINI_CHILDREN, NULL);
> + (void)xroot_put_group(xr, pdev);
> + if (!ret) {
> + ret = xrt_subdev_pool_del(&xr->grps.pool, XRT_SUBDEV_GRP,
> + instance);
> + }
> +
> + return ret;
> +}
> +
> +static int xroot_destroy_group(struct xroot *xr, int instance)
A better name would be 'xroot_destroy_groups'
> +{
> + struct platform_device *target = NULL;
> + struct platform_device *deps = NULL;
> + int ret;
> +
> + WARN_ON(instance < 0);
> + /*
> + * Make sure target group exists and can't go away before
> + * we remove it's dependents
> + */
> + ret = xroot_get_group(xr, instance, &target);
> + if (ret)
> + return ret;
> +
> + /*
> + * Remove all groups depend on target one.
> + * Assuming subdevs in higher group ID can depend on ones in
> + * lower ID groups, we remove them in the reservse order.
> + */
> + while (xroot_get_group(xr, XROOT_GRP_LAST, &deps) != -ENOENT) {
> + int inst = deps->id;
> +
> + xroot_put_group(xr, deps);
> + if (instance == inst)
> + break;
breaking in the middle does not seem correct.
please add a comment
> + (void)xroot_destroy_single_group(xr, inst);
> + deps = NULL;
> + }
> +
> + /* Now we can remove the target group. */
> + xroot_put_group(xr, target);
> + return xroot_destroy_single_group(xr, instance);
> +}
> +
> +static int xroot_lookup_group(struct xroot *xr,
> + struct xrt_root_ioctl_lookup_group *arg)
> +{
> + int rc = -ENOENT;
> + struct platform_device *grp = NULL;
> +
> + while (rc < 0 && xroot_get_group(xr, XROOT_GRP_LAST, &grp) != -ENOENT) {
> + if (arg->xpilp_match_cb(XRT_SUBDEV_GRP, grp,
> + arg->xpilp_match_arg)) {
> + rc = grp->id;
> + }
> + xroot_put_group(xr, grp);
> + }
> + return rc;
> +}
> +
> +static void xroot_event_work(struct work_struct *work)
> +{
> + struct xroot_evt *tmp;
> + struct xroot *xr = container_of(work, struct xroot, events.evt_work);
> +
> + mutex_lock(&xr->events.evt_lock);
> + while (!list_empty(&xr->events.evt_list)) {
> + tmp = list_first_entry(&xr->events.evt_list,
> + struct xroot_evt, list);
> + list_del(&tmp->list);
> + mutex_unlock(&xr->events.evt_lock);
why is unlocking necessary ?
> +
> + (void)xrt_subdev_pool_handle_event(&xr->grps.pool, &tmp->evt);
> +
> + if (tmp->async)
> + vfree(tmp);
> + else
> + complete(&tmp->comp);
> +
> + mutex_lock(&xr->events.evt_lock);
> + }
> + mutex_unlock(&xr->events.evt_lock);
> +}
> +
> +static void xroot_event_init(struct xroot *xr)
> +{
> + INIT_LIST_HEAD(&xr->events.evt_list);
> + mutex_init(&xr->events.evt_lock);
> + INIT_WORK(&xr->events.evt_work, xroot_event_work);
> +}
> +
> +static void xroot_event_fini(struct xroot *xr)
> +{
> + flush_scheduled_work();
> + WARN_ON(!list_empty(&xr->events.evt_list));
> +}
> +
> +static int xroot_get_leaf(struct xroot *xr, struct xrt_root_ioctl_get_leaf *arg)
> +{
> + int rc = -ENOENT;
> + struct platform_device *grp = NULL;
> +
> + while (rc && xroot_get_group(xr, XROOT_GRP_LAST, &grp) != -ENOENT) {
while (rc) ?
while we see an error on xleaf_ioctl, keep going ?
Seems like would rather have !rc
similar below in put_leaf
> + rc = xleaf_ioctl(grp, XRT_GROUP_GET_LEAF, arg);
> + xroot_put_group(xr, grp);
> + }
> + return rc;
> +}
> +
> +static int xroot_put_leaf(struct xroot *xr, struct xrt_root_ioctl_put_leaf *arg)
> +{
> + int rc = -ENOENT;
> + struct platform_device *grp = NULL;
> +
> + while (rc && xroot_get_group(xr, XROOT_GRP_LAST, &grp) != -ENOENT) {
> + rc = xleaf_ioctl(grp, XRT_GROUP_PUT_LEAF, arg);
> + xroot_put_group(xr, grp);
> + }
> + return rc;
> +}
> +
> +static int xroot_root_cb(struct device *dev, void *parg, u32 cmd, void *arg)
> +{
> + struct xroot *xr = (struct xroot *)parg;
> + int rc = 0;
> +
> + switch (cmd) {
> + /* Leaf actions. */
> + case XRT_ROOT_GET_LEAF: {
> + struct xrt_root_ioctl_get_leaf *getleaf =
> + (struct xrt_root_ioctl_get_leaf *)arg;
> + rc = xroot_get_leaf(xr, getleaf);
> + break;
> + }
> + case XRT_ROOT_PUT_LEAF: {
> + struct xrt_root_ioctl_put_leaf *putleaf =
> + (struct xrt_root_ioctl_put_leaf *)arg;
> + rc = xroot_put_leaf(xr, putleaf);
> + break;
> + }
looking at these two cases without any changes to arg but a cast, i think these and the next pass the void * onto the function and have the function manage the cast.
> + case XRT_ROOT_GET_LEAF_HOLDERS: {
> + struct xrt_root_ioctl_get_holders *holders =
> + (struct xrt_root_ioctl_get_holders *)arg;
> + rc = xrt_subdev_pool_get_holders(&xr->grps.pool,
> + holders->xpigh_pdev,
> + holders->xpigh_holder_buf,
> + holders->xpigh_holder_buf_len);
> + break;
> + }
> +
> + /* Group actions. */
> + case XRT_ROOT_CREATE_GROUP:
> + rc = xroot_create_group(xr, (char *)arg);
> + break;
> + case XRT_ROOT_REMOVE_GROUP:
> + rc = xroot_destroy_group(xr, (int)(uintptr_t)arg);
> + break;
> + case XRT_ROOT_LOOKUP_GROUP: {
> + struct xrt_root_ioctl_lookup_group *getgrp =
> + (struct xrt_root_ioctl_lookup_group *)arg;
> + rc = xroot_lookup_group(xr, getgrp);
> + break;
> + }
> + case XRT_ROOT_WAIT_GROUP_BRINGUP:
> + rc = xroot_wait_for_bringup(xr) ? 0 : -EINVAL;
> + break;
> +
> + /* Event actions. */
> + case XRT_ROOT_EVENT:
> + case XRT_ROOT_EVENT_ASYNC: {
> + bool async = (cmd == XRT_ROOT_EVENT_ASYNC);
> + struct xrt_event *evt = (struct xrt_event *)arg;
> +
> + rc = xroot_trigger_event(xr, evt, async);
> + break;
> + }
> +
> + /* Device info. */
> + case XRT_ROOT_GET_RESOURCE: {
> + struct xrt_root_ioctl_get_res *res =
> + (struct xrt_root_ioctl_get_res *)arg;
> + res->xpigr_res = xr->pdev->resource;
> + break;
> + }
> + case XRT_ROOT_GET_ID: {
> + struct xrt_root_ioctl_get_id *id =
> + (struct xrt_root_ioctl_get_id *)arg;
> +
> + id->xpigi_vendor_id = xr->pdev->vendor;
> + id->xpigi_device_id = xr->pdev->device;
> + id->xpigi_sub_vendor_id = xr->pdev->subsystem_vendor;
> + id->xpigi_sub_device_id = xr->pdev->subsystem_device;
> + break;
> + }
> +
> + /* MISC generic PCIE driver functions. */
misc functions may need a need some place else.
Is there a way to extend the cmd with some additional layer of abstraction ?
> + case XRT_ROOT_HOT_RESET: {
> + xr->pf_cb.xpc_hot_reset(xr->pdev);
> + break;
> + }
> + case XRT_ROOT_HWMON: {
> + struct xrt_root_ioctl_hwmon *hwmon =
> + (struct xrt_root_ioctl_hwmon *)arg;
> +
> + if (hwmon->xpih_register) {
> + hwmon->xpih_hwmon_dev =
> + hwmon_device_register_with_info(DEV(xr->pdev),
> + hwmon->xpih_name,
> + hwmon->xpih_drvdata,
> + NULL,
> + hwmon->xpih_groups);
> + } else {
> + (void)hwmon_device_unregister(hwmon->xpih_hwmon_dev);
> + }
> + break;
> + }
> +
> + default:
> + xroot_err(xr, "unknown IOCTL cmd %d", cmd);
> + rc = -EINVAL;
> + break;
> + }
> +
> + return rc;
> +}
> +
> +static void xroot_bringup_group_work(struct work_struct *work)
> +{
> + struct platform_device *pdev = NULL;
> + struct xroot *xr = container_of(work, struct xroot, grps.bringup_work);
> +
> + while (xroot_get_group(xr, XROOT_GRP_FIRST, &pdev) != -ENOENT) {
> + int r, i;
> +
> + i = pdev->id;
> + r = xleaf_ioctl(pdev, XRT_GROUP_INIT_CHILDREN, NULL);
> + (void)xroot_put_group(xr, pdev);
> + if (r == -EEXIST)
> + continue; /* Already brough up, nothing to do. */
> + if (r)
> + atomic_inc(&xr->grps.bringup_failed);
> +
> + xroot_group_trigger_event(xr, i, XRT_EVENT_POST_CREATION);
> +
> + if (atomic_dec_and_test(&xr->grps.bringup_pending))
> + complete(&xr->grps.bringup_comp);
> + }
> +}
> +
> +static void xroot_grps_init(struct xroot *xr)
Consistency in terms is needed. In the last few lines i see
group, grp, grps, my vote is for group(s)
> +{
> + xrt_subdev_pool_init(DEV(xr->pdev), &xr->grps.pool);
> + INIT_WORK(&xr->grps.bringup_work, xroot_bringup_group_work);
> + atomic_set(&xr->grps.bringup_pending, 0);
> + atomic_set(&xr->grps.bringup_failed, 0);
> + init_completion(&xr->grps.bringup_comp);
> +}
> +
> +static void xroot_grps_fini(struct xroot *xr)
> +{
> + flush_scheduled_work();
> + xrt_subdev_pool_fini(&xr->grps.pool);
> +}
> +
> +int xroot_add_vsec_node(void *root, char *dtb)
> +{
This is the pci part i think needs to move to its own file.
> + struct xroot *xr = (struct xroot *)root;
> + struct device *dev = DEV(xr->pdev);
> + struct xrt_md_endpoint ep = { 0 };
> + int cap = 0, ret = 0;
> + u32 off_low, off_high, vsec_bar, header;
> + u64 vsec_off;
> +
> + while ((cap = pci_find_next_ext_capability(xr->pdev, cap,
> + PCI_EXT_CAP_ID_VNDR))) {
> + pci_read_config_dword(xr->pdev, cap + PCI_VNDR_HEADER, &header);
> + if (PCI_VNDR_HEADER_ID(header) == XRT_VSEC_ID)
> + break;
> + }
> + if (!cap) {
> + xroot_info(xr, "No Vendor Specific Capability.");
> + return -ENOENT;
> + }
> +
> + if (pci_read_config_dword(xr->pdev, cap + 8, &off_low) ||
> + pci_read_config_dword(xr->pdev, cap + 12, &off_high)) {
> + xroot_err(xr, "pci_read vendor specific failed.");
> + return -EINVAL;
> + }
> +
> + ep.ep_name = XRT_MD_NODE_VSEC;
> + ret = xrt_md_add_endpoint(dev, dtb, &ep);
> + if (ret) {
> + xroot_err(xr, "add vsec metadata failed, ret %d", ret);
> + goto failed;
> + }
> +
> + vsec_bar = cpu_to_be32(off_low & 0xf);
> + ret = xrt_md_set_prop(dev, dtb, XRT_MD_NODE_VSEC, NULL,
> + XRT_MD_PROP_BAR_IDX, &vsec_bar, sizeof(vsec_bar));
> + if (ret) {
> + xroot_err(xr, "add vsec bar idx failed, ret %d", ret);
> + goto failed;
> + }
> +
> + vsec_off = cpu_to_be64(((u64)off_high << 32) | (off_low & ~0xfU));
> + ret = xrt_md_set_prop(dev, dtb, XRT_MD_NODE_VSEC, NULL,
> + XRT_MD_PROP_OFFSET, &vsec_off, sizeof(vsec_off));
> + if (ret) {
> + xroot_err(xr, "add vsec offset failed, ret %d", ret);
> + goto failed;
> + }
> +
> +failed:
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xroot_add_vsec_node);
> +
> +int xroot_add_simple_node(void *root, char *dtb, const char *endpoint)
> +{
> + struct xroot *xr = (struct xroot *)root;
> + struct device *dev = DEV(xr->pdev);
> + struct xrt_md_endpoint ep = { 0 };
> + int ret = 0;
> +
> + ep.ep_name = endpoint;
> + ret = xrt_md_add_endpoint(dev, dtb, &ep);
> + if (ret)
> + xroot_err(xr, "add %s failed, ret %d", endpoint, ret);
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(xroot_add_simple_node);
> +
> +bool xroot_wait_for_bringup(void *root)
> +{
> + struct xroot *xr = (struct xroot *)root;
> +
> + wait_for_completion(&xr->grps.bringup_comp);
> + return atomic_xchg(&xr->grps.bringup_failed, 0) == 0;
Is there going to a race in intialization ?
> +}
> +EXPORT_SYMBOL_GPL(xroot_wait_for_bringup);
> +
> +int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root)
> +{
> + struct device *dev = DEV(pdev);
> + struct xroot *xr = NULL;
> +
> + dev_info(dev, "%s: probing...", __func__);
> +
> + xr = devm_kzalloc(dev, sizeof(*xr), GFP_KERNEL);
> + if (!xr)
> + return -ENOMEM;
> +
> + xr->pdev = pdev;
> + xr->pf_cb = *cb;
> + xroot_grps_init(xr);
> + xroot_event_init(xr);
> +
> + *root = xr;
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xroot_probe);
> +
> +void xroot_remove(void *root)
> +{
> + struct xroot *xr = (struct xroot *)root;
> + struct platform_device *grp = NULL;
> +
> + xroot_info(xr, "leaving...");
> +
> + if (xroot_get_group(xr, XROOT_GRP_FIRST, &grp) == 0) {
> + int instance = grp->id;
another instance = id, the variable and element names should be consistent.
earlier (id, instance) is used to uniquely determine a node. if that is so then using the names should be kept seperate.
> +
> + xroot_put_group(xr, grp);
> + (void)xroot_destroy_group(xr, instance);
> + }
> +
> + xroot_event_fini(xr);
> + xroot_grps_fini(xr);
> +}
> +EXPORT_SYMBOL_GPL(xroot_remove);
> +
> +void xroot_broadcast(void *root, enum xrt_events evt)
> +{
> + struct xroot *xr = (struct xroot *)root;
> + struct xrt_event e = { 0 };
> +
> + /* Root pf driver only broadcasts below two events. */
> + if (evt != XRT_EVENT_POST_CREATION && evt != XRT_EVENT_PRE_REMOVAL) {
> + xroot_info(xr, "invalid event %d", evt);
> + return;
> + }
> +
> + e.xe_evt = evt;
> + e.xe_subdev.xevt_subdev_id = XRT_ROOT;
> + e.xe_subdev.xevt_subdev_instance = 0;
see..
id =
instance =
Tom
> + (void)xroot_trigger_event(xr, &e, false);
> +}
> +EXPORT_SYMBOL_GPL(xroot_broadcast);
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root)
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (5 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 06/18] fpga: xrt: platform driver infrastructure Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-26 15:01 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 08/18] fpga: xrt: main platform driver for management function device Lizhi Hou
` (12 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
The PCIE device driver which attaches to management function on Alveo
devices. It instantiates one or more partition drivers which in turn
instantiate platform drivers. The instantiation of partition and platform
drivers is completely data driven.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xroot.h | 114 +++++++++++
drivers/fpga/xrt/mgmt/root.c | 342 +++++++++++++++++++++++++++++++
2 files changed, 456 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xroot.h
create mode 100644 drivers/fpga/xrt/mgmt/root.c
diff --git a/drivers/fpga/xrt/include/xroot.h b/drivers/fpga/xrt/include/xroot.h
new file mode 100644
index 000000000000..752e10daa85e
--- /dev/null
+++ b/drivers/fpga/xrt/include/xroot.h
@@ -0,0 +1,114 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_ROOT_H_
+#define _XRT_ROOT_H_
+
+#include <linux/pci.h>
+#include "subdev_id.h"
+#include "events.h"
+
+typedef bool (*xrt_subdev_match_t)(enum xrt_subdev_id,
+ struct platform_device *, void *);
+#define XRT_SUBDEV_MATCH_PREV ((xrt_subdev_match_t)-1)
+#define XRT_SUBDEV_MATCH_NEXT ((xrt_subdev_match_t)-2)
+
+/*
+ * Root IOCTL calls.
+ */
+enum xrt_root_ioctl_cmd {
+ /* Leaf actions. */
+ XRT_ROOT_GET_LEAF = 0,
+ XRT_ROOT_PUT_LEAF,
+ XRT_ROOT_GET_LEAF_HOLDERS,
+
+ /* Group actions. */
+ XRT_ROOT_CREATE_GROUP,
+ XRT_ROOT_REMOVE_GROUP,
+ XRT_ROOT_LOOKUP_GROUP,
+ XRT_ROOT_WAIT_GROUP_BRINGUP,
+
+ /* Event actions. */
+ XRT_ROOT_EVENT,
+ XRT_ROOT_EVENT_ASYNC,
+
+ /* Device info. */
+ XRT_ROOT_GET_RESOURCE,
+ XRT_ROOT_GET_ID,
+
+ /* Misc. */
+ XRT_ROOT_HOT_RESET,
+ XRT_ROOT_HWMON,
+};
+
+struct xrt_root_ioctl_get_leaf {
+ struct platform_device *xpigl_pdev; /* caller's pdev */
+ xrt_subdev_match_t xpigl_match_cb;
+ void *xpigl_match_arg;
+ struct platform_device *xpigl_leaf; /* target leaf pdev */
+};
+
+struct xrt_root_ioctl_put_leaf {
+ struct platform_device *xpipl_pdev; /* caller's pdev */
+ struct platform_device *xpipl_leaf; /* target's pdev */
+};
+
+struct xrt_root_ioctl_lookup_group {
+ struct platform_device *xpilp_pdev; /* caller's pdev */
+ xrt_subdev_match_t xpilp_match_cb;
+ void *xpilp_match_arg;
+ int xpilp_grp_inst;
+};
+
+struct xrt_root_ioctl_get_holders {
+ struct platform_device *xpigh_pdev; /* caller's pdev */
+ char *xpigh_holder_buf;
+ size_t xpigh_holder_buf_len;
+};
+
+struct xrt_root_ioctl_get_res {
+ struct resource *xpigr_res;
+};
+
+struct xrt_root_ioctl_get_id {
+ unsigned short xpigi_vendor_id;
+ unsigned short xpigi_device_id;
+ unsigned short xpigi_sub_vendor_id;
+ unsigned short xpigi_sub_device_id;
+};
+
+struct xrt_root_ioctl_hwmon {
+ bool xpih_register;
+ const char *xpih_name;
+ void *xpih_drvdata;
+ const struct attribute_group **xpih_groups;
+ struct device *xpih_hwmon_dev;
+};
+
+typedef int (*xrt_subdev_root_cb_t)(struct device *, void *, u32, void *);
+int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void *arg);
+
+/*
+ * Defines physical function (MPF / UPF) specific operations
+ * needed in common root driver.
+ */
+struct xroot_pf_cb {
+ void (*xpc_hot_reset)(struct pci_dev *pdev);
+};
+
+int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root);
+void xroot_remove(void *root);
+bool xroot_wait_for_bringup(void *root);
+int xroot_add_vsec_node(void *root, char *dtb);
+int xroot_create_group(void *xr, char *dtb);
+int xroot_add_simple_node(void *root, char *dtb, const char *endpoint);
+void xroot_broadcast(void *root, enum xrt_events evt);
+
+#endif /* _XRT_ROOT_H_ */
diff --git a/drivers/fpga/xrt/mgmt/root.c b/drivers/fpga/xrt/mgmt/root.c
new file mode 100644
index 000000000000..583a37c9d30c
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/root.c
@@ -0,0 +1,342 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/aer.h>
+#include <linux/vmalloc.h>
+#include <linux/delay.h>
+
+#include "xroot.h"
+#include "main-impl.h"
+#include "metadata.h"
+
+#define XMGMT_MODULE_NAME "xmgmt"
+#define XMGMT_DRIVER_VERSION "4.0.0"
+
+#define XMGMT_PDEV(xm) ((xm)->pdev)
+#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
+#define xmgmt_err(xm, fmt, args...) \
+ dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_warn(xm, fmt, args...) \
+ dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_info(xm, fmt, args...) \
+ dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_dbg(xm, fmt, args...) \
+ dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define XMGMT_DEV_ID(_pcidev) \
+ ({ typeof(_pcidev) (pcidev) = (_pcidev); \
+ ((pci_domain_nr((pcidev)->bus) << 16) | \
+ PCI_DEVID((pcidev)->bus->number, 0)); })
+
+static struct class *xmgmt_class;
+static const struct pci_device_id xmgmt_pci_ids[] = {
+ { PCI_DEVICE(0x10EE, 0xd020), }, /* Alveo U50 (golden image) */
+ { PCI_DEVICE(0x10EE, 0x5020), }, /* Alveo U50 */
+ { 0, }
+};
+
+struct xmgmt {
+ struct pci_dev *pdev;
+ void *root;
+
+ bool ready;
+};
+
+static int xmgmt_config_pci(struct xmgmt *xm)
+{
+ struct pci_dev *pdev = XMGMT_PDEV(xm);
+ int rc;
+
+ rc = pcim_enable_device(pdev);
+ if (rc < 0) {
+ xmgmt_err(xm, "failed to enable device: %d", rc);
+ return rc;
+ }
+
+ rc = pci_enable_pcie_error_reporting(pdev);
+ if (rc)
+ xmgmt_warn(xm, "failed to enable AER: %d", rc);
+
+ pci_set_master(pdev);
+
+ rc = pcie_get_readrq(pdev);
+ if (rc < 0) {
+ xmgmt_err(xm, "failed to read mrrs %d", rc);
+ return rc;
+ }
+ if (rc > 512) {
+ rc = pcie_set_readrq(pdev, 512);
+ if (rc) {
+ xmgmt_err(xm, "failed to force mrrs %d", rc);
+ return rc;
+ }
+ }
+
+ return 0;
+}
+
+static int xmgmt_match_slot_and_save(struct device *dev, void *data)
+{
+ struct xmgmt *xm = data;
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
+ pci_cfg_access_lock(pdev);
+ pci_save_state(pdev);
+ }
+
+ return 0;
+}
+
+static void xmgmt_pci_save_config_all(struct xmgmt *xm)
+{
+ bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save);
+}
+
+static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
+{
+ struct xmgmt *xm = data;
+ struct pci_dev *pdev = to_pci_dev(dev);
+
+ if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
+ pci_restore_state(pdev);
+ pci_cfg_access_unlock(pdev);
+ }
+
+ return 0;
+}
+
+static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
+{
+ bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore);
+}
+
+static void xmgmt_root_hot_reset(struct pci_dev *pdev)
+{
+ struct xmgmt *xm = pci_get_drvdata(pdev);
+ struct pci_bus *bus;
+ u8 pci_bctl;
+ u16 pci_cmd, devctl;
+ int i, ret;
+
+ xmgmt_info(xm, "hot reset start");
+
+ xmgmt_pci_save_config_all(xm);
+
+ pci_disable_device(pdev);
+
+ bus = pdev->bus;
+
+ /*
+ * When flipping the SBR bit, device can fall off the bus. This is
+ * usually no problem at all so long as drivers are working properly
+ * after SBR. However, some systems complain bitterly when the device
+ * falls off the bus.
+ * The quick solution is to temporarily disable the SERR reporting of
+ * switch port during SBR.
+ */
+
+ pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
+ pci_write_config_word(bus->self, PCI_COMMAND,
+ (pci_cmd & ~PCI_COMMAND_SERR));
+ pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
+ pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
+ (devctl & ~PCI_EXP_DEVCTL_FERE));
+ pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
+ pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
+ pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
+
+ msleep(100);
+ pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
+ pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
+ ssleep(1);
+
+ pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
+ pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
+
+ ret = pci_enable_device(pdev);
+ if (ret)
+ xmgmt_err(xm, "failed to enable device, ret %d", ret);
+
+ for (i = 0; i < 300; i++) {
+ pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
+ if (pci_cmd != 0xffff)
+ break;
+ msleep(20);
+ }
+
+ xmgmt_info(xm, "waiting for %d ms", i * 20);
+ xmgmt_pci_restore_config_all(xm);
+ xmgmt_config_pci(xm);
+}
+
+static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb)
+{
+ char *dtb = NULL;
+ int ret;
+
+ ret = xrt_md_create(XMGMT_DEV(xm), &dtb);
+ if (ret) {
+ xmgmt_err(xm, "create metadata failed, ret %d", ret);
+ goto failed;
+ }
+
+ ret = xroot_add_vsec_node(xm->root, dtb);
+ if (ret == -ENOENT) {
+ /*
+ * We may be dealing with a MFG board.
+ * Try vsec-golden which will bring up all hard-coded leaves
+ * at hard-coded offsets.
+ */
+ ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_VSEC_GOLDEN);
+ } else if (ret == 0) {
+ ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_MGMT_MAIN);
+ }
+ if (ret)
+ goto failed;
+
+ *root_dtb = dtb;
+ return 0;
+
+failed:
+ vfree(dtb);
+ return ret;
+}
+
+static ssize_t ready_show(struct device *dev,
+ struct device_attribute *da,
+ char *buf)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ struct xmgmt *xm = pci_get_drvdata(pdev);
+
+ return sprintf(buf, "%d\n", xm->ready);
+}
+static DEVICE_ATTR_RO(ready);
+
+static struct attribute *xmgmt_root_attrs[] = {
+ &dev_attr_ready.attr,
+ NULL
+};
+
+static struct attribute_group xmgmt_root_attr_group = {
+ .attrs = xmgmt_root_attrs,
+};
+
+static struct xroot_pf_cb xmgmt_xroot_pf_cb = {
+ .xpc_hot_reset = xmgmt_root_hot_reset,
+};
+
+static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ int ret;
+ struct device *dev = &pdev->dev;
+ struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
+ char *dtb = NULL;
+
+ if (!xm)
+ return -ENOMEM;
+ xm->pdev = pdev;
+ pci_set_drvdata(pdev, xm);
+
+ ret = xmgmt_config_pci(xm);
+ if (ret)
+ goto failed;
+
+ ret = xroot_probe(pdev, &xmgmt_xroot_pf_cb, &xm->root);
+ if (ret)
+ goto failed;
+
+ ret = xmgmt_create_root_metadata(xm, &dtb);
+ if (ret)
+ goto failed_metadata;
+
+ ret = xroot_create_group(xm->root, dtb);
+ vfree(dtb);
+ if (ret)
+ xmgmt_err(xm, "failed to create root group: %d", ret);
+
+ if (!xroot_wait_for_bringup(xm->root))
+ xmgmt_err(xm, "failed to bringup all groups");
+ else
+ xm->ready = true;
+
+ ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
+ if (ret) {
+ /* Warning instead of failing the probe. */
+ xmgmt_warn(xm, "create xmgmt root attrs failed: %d", ret);
+ }
+
+ xroot_broadcast(xm->root, XRT_EVENT_POST_CREATION);
+ xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
+ return 0;
+
+failed_metadata:
+ (void)xroot_remove(xm->root);
+failed:
+ pci_set_drvdata(pdev, NULL);
+ return ret;
+}
+
+static void xmgmt_remove(struct pci_dev *pdev)
+{
+ struct xmgmt *xm = pci_get_drvdata(pdev);
+
+ xroot_broadcast(xm->root, XRT_EVENT_PRE_REMOVAL);
+ sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
+ (void)xroot_remove(xm->root);
+ pci_disable_pcie_error_reporting(xm->pdev);
+ xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
+}
+
+static struct pci_driver xmgmt_driver = {
+ .name = XMGMT_MODULE_NAME,
+ .id_table = xmgmt_pci_ids,
+ .probe = xmgmt_probe,
+ .remove = xmgmt_remove,
+};
+
+static int __init xmgmt_init(void)
+{
+ int res = 0;
+
+ res = xmgmt_main_register_leaf();
+ if (res)
+ return res;
+
+ xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME);
+ if (IS_ERR(xmgmt_class))
+ return PTR_ERR(xmgmt_class);
+
+ res = pci_register_driver(&xmgmt_driver);
+ if (res) {
+ class_destroy(xmgmt_class);
+ return res;
+ }
+
+ return 0;
+}
+
+static __exit void xmgmt_exit(void)
+{
+ pci_unregister_driver(&xmgmt_driver);
+ class_destroy(xmgmt_class);
+ xmgmt_main_unregister_leaf();
+}
+
+module_init(xmgmt_init);
+module_exit(xmgmt_exit);
+
+MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
+MODULE_VERSION(XMGMT_DRIVER_VERSION);
+MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
+MODULE_DESCRIPTION("Xilinx Alveo management function driver");
+MODULE_LICENSE("GPL v2");
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root) Lizhi Hou
@ 2021-02-26 15:01 ` Tom Rix
2021-02-26 17:56 ` Moritz Fischer
2021-03-16 20:29 ` Max Zhen
0 siblings, 2 replies; 87+ messages in thread
From: Tom Rix @ 2021-02-26 15:01 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel, mdf
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
A question i do not know the answer to.
Seems like 'golden' is linked to a manufacturing (diagnostics?) image.
If the public will never see it, should handling it here be done ?
Moritz, do you know ?
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> The PCIE device driver which attaches to management function on Alveo
to the management
> devices. It instantiates one or more partition drivers which in turn
more fpga partition / group ?
> instantiate platform drivers. The instantiation of partition and platform
> drivers is completely data driven.
data driven ? everything is data driven. do you mean dtb driven ?
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xroot.h | 114 +++++++++++
> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++++++++++++++++++++++++
> 2 files changed, 456 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xroot.h
> create mode 100644 drivers/fpga/xrt/mgmt/root.c
>
> diff --git a/drivers/fpga/xrt/include/xroot.h b/drivers/fpga/xrt/include/xroot.h
> new file mode 100644
> index 000000000000..752e10daa85e
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xroot.h
> @@ -0,0 +1,114 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_ROOT_H_
> +#define _XRT_ROOT_H_
> +
> +#include <linux/pci.h>
> +#include "subdev_id.h"
> +#include "events.h"
> +
> +typedef bool (*xrt_subdev_match_t)(enum xrt_subdev_id,
> + struct platform_device *, void *);
> +#define XRT_SUBDEV_MATCH_PREV ((xrt_subdev_match_t)-1)
> +#define XRT_SUBDEV_MATCH_NEXT ((xrt_subdev_match_t)-2)
> +
> +/*
> + * Root IOCTL calls.
> + */
> +enum xrt_root_ioctl_cmd {
> + /* Leaf actions. */
> + XRT_ROOT_GET_LEAF = 0,
> + XRT_ROOT_PUT_LEAF,
> + XRT_ROOT_GET_LEAF_HOLDERS,
> +
> + /* Group actions. */
> + XRT_ROOT_CREATE_GROUP,
> + XRT_ROOT_REMOVE_GROUP,
> + XRT_ROOT_LOOKUP_GROUP,
> + XRT_ROOT_WAIT_GROUP_BRINGUP,
> +
> + /* Event actions. */
> + XRT_ROOT_EVENT,
should this be XRT_ROOT_EVENT_SYNC ?
> + XRT_ROOT_EVENT_ASYNC,
> +
> + /* Device info. */
> + XRT_ROOT_GET_RESOURCE,
> + XRT_ROOT_GET_ID,
> +
> + /* Misc. */
> + XRT_ROOT_HOT_RESET,
> + XRT_ROOT_HWMON,
> +};
> +
> +struct xrt_root_ioctl_get_leaf {
> + struct platform_device *xpigl_pdev; /* caller's pdev */
xpigl_ ? unneeded suffix in element names
> + xrt_subdev_match_t xpigl_match_cb;
> + void *xpigl_match_arg;
> + struct platform_device *xpigl_leaf; /* target leaf pdev */
> +};
> +
> +struct xrt_root_ioctl_put_leaf {
> + struct platform_device *xpipl_pdev; /* caller's pdev */
> + struct platform_device *xpipl_leaf; /* target's pdev */
caller_pdev;
target_pdev;
> +};
> +
> +struct xrt_root_ioctl_lookup_group {
> + struct platform_device *xpilp_pdev; /* caller's pdev */
> + xrt_subdev_match_t xpilp_match_cb;
> + void *xpilp_match_arg;
> + int xpilp_grp_inst;
> +};
> +
> +struct xrt_root_ioctl_get_holders {
> + struct platform_device *xpigh_pdev; /* caller's pdev */
> + char *xpigh_holder_buf;
> + size_t xpigh_holder_buf_len;
> +};
> +
> +struct xrt_root_ioctl_get_res {
> + struct resource *xpigr_res;
> +};
> +
> +struct xrt_root_ioctl_get_id {
> + unsigned short xpigi_vendor_id;
> + unsigned short xpigi_device_id;
> + unsigned short xpigi_sub_vendor_id;
> + unsigned short xpigi_sub_device_id;
> +};
> +
> +struct xrt_root_ioctl_hwmon {
> + bool xpih_register;
> + const char *xpih_name;
> + void *xpih_drvdata;
> + const struct attribute_group **xpih_groups;
> + struct device *xpih_hwmon_dev;
> +};
> +
> +typedef int (*xrt_subdev_root_cb_t)(struct device *, void *, u32, void *);
This function pointer type is important, please add a comment about its use and expected parameters
> +int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void *arg);
> +
> +/*
> + * Defines physical function (MPF / UPF) specific operations
> + * needed in common root driver.
> + */
> +struct xroot_pf_cb {
> + void (*xpc_hot_reset)(struct pci_dev *pdev);
This is only ever set to xmgmt_root_hot_reset, why is this abstraction needed ?
> +};
> +
> +int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root);
> +void xroot_remove(void *root);
> +bool xroot_wait_for_bringup(void *root);
> +int xroot_add_vsec_node(void *root, char *dtb);
> +int xroot_create_group(void *xr, char *dtb);
> +int xroot_add_simple_node(void *root, char *dtb, const char *endpoint);
> +void xroot_broadcast(void *root, enum xrt_events evt);
> +
> +#endif /* _XRT_ROOT_H_ */
> diff --git a/drivers/fpga/xrt/mgmt/root.c b/drivers/fpga/xrt/mgmt/root.c
> new file mode 100644
> index 000000000000..583a37c9d30c
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/root.c
> @@ -0,0 +1,342 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/aer.h>
> +#include <linux/vmalloc.h>
> +#include <linux/delay.h>
> +
> +#include "xroot.h"
> +#include "main-impl.h"
> +#include "metadata.h"
> +
> +#define XMGMT_MODULE_NAME "xmgmt"
The xrt modules would be more easily identified with a 'xrt' prefix instead of 'x'
> +#define XMGMT_DRIVER_VERSION "4.0.0"
> +
> +#define XMGMT_PDEV(xm) ((xm)->pdev)
> +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
> +#define xmgmt_err(xm, fmt, args...) \
> + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> +#define xmgmt_warn(xm, fmt, args...) \
> + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> +#define xmgmt_info(xm, fmt, args...) \
> + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> +#define xmgmt_dbg(xm, fmt, args...) \
> + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> +#define XMGMT_DEV_ID(_pcidev) \
> + ({ typeof(_pcidev) (pcidev) = (_pcidev); \
> + ((pci_domain_nr((pcidev)->bus) << 16) | \
> + PCI_DEVID((pcidev)->bus->number, 0)); })
> +
> +static struct class *xmgmt_class;
> +static const struct pci_device_id xmgmt_pci_ids[] = {
> + { PCI_DEVICE(0x10EE, 0xd020), }, /* Alveo U50 (golden image) */
> + { PCI_DEVICE(0x10EE, 0x5020), }, /* Alveo U50 */
demagic this table, look at dfl-pci for how to use existing #define for the vendor and create a new on for the device. If there are vf's add them at the same time.
What is a golden image ?
> + { 0, }
> +};
> +
> +struct xmgmt {
> + struct pci_dev *pdev;
> + void *root;
> +
> + bool ready;
> +};
> +
> +static int xmgmt_config_pci(struct xmgmt *xm)
> +{
> + struct pci_dev *pdev = XMGMT_PDEV(xm);
> + int rc;
> +
> + rc = pcim_enable_device(pdev);
> + if (rc < 0) {
> + xmgmt_err(xm, "failed to enable device: %d", rc);
> + return rc;
> + }
> +
> + rc = pci_enable_pcie_error_reporting(pdev);
> + if (rc)
> + xmgmt_warn(xm, "failed to enable AER: %d", rc);
> +
> + pci_set_master(pdev);
> +
> + rc = pcie_get_readrq(pdev);
Review this call, it does not go negative
> + if (rc < 0) {
> + xmgmt_err(xm, "failed to read mrrs %d", rc);
> + return rc;
> + }
this is a quirk, add a comment.
> + if (rc > 512) {
> + rc = pcie_set_readrq(pdev, 512);
> + if (rc) {
> + xmgmt_err(xm, "failed to force mrrs %d", rc);
similar calls do not fail here.
> + return rc;
> + }
> + }
> +
> + return 0;
> +}
> +
> +static int xmgmt_match_slot_and_save(struct device *dev, void *data)
> +{
> + struct xmgmt *xm = data;
> + struct pci_dev *pdev = to_pci_dev(dev);
> +
> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
> + pci_cfg_access_lock(pdev);
> + pci_save_state(pdev);
> + }
> +
> + return 0;
> +}
> +
> +static void xmgmt_pci_save_config_all(struct xmgmt *xm)
> +{
> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save);
This is a bus call, not a device call.
Can this be changed into something like what hot reset does ?
> +}
> +
> +static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
> +{
> + struct xmgmt *xm = data;
> + struct pci_dev *pdev = to_pci_dev(dev);
> +
> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
> + pci_restore_state(pdev);
> + pci_cfg_access_unlock(pdev);
> + }
> +
> + return 0;
> +}
> +
> +static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
> +{
> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore);
> +}
> +
> +static void xmgmt_root_hot_reset(struct pci_dev *pdev)
> +{
> + struct xmgmt *xm = pci_get_drvdata(pdev);
> + struct pci_bus *bus;
> + u8 pci_bctl;
> + u16 pci_cmd, devctl;
> + int i, ret;
> +
> + xmgmt_info(xm, "hot reset start");
> +
> + xmgmt_pci_save_config_all(xm);
> +
> + pci_disable_device(pdev);
> +
> + bus = pdev->bus;
> +
> + /*
> + * When flipping the SBR bit, device can fall off the bus. This is
> + * usually no problem at all so long as drivers are working properly
> + * after SBR. However, some systems complain bitterly when the device
> + * falls off the bus.
> + * The quick solution is to temporarily disable the SERR reporting of
> + * switch port during SBR.
> + */
> +
> + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
> + pci_write_config_word(bus->self, PCI_COMMAND,
> + (pci_cmd & ~PCI_COMMAND_SERR));
> + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
> + (devctl & ~PCI_EXP_DEVCTL_FERE));
> + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
> + pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
how the pci config values are set and cleared should be consistent.
this call should be
pci_write_config_byte (... pci_bctl | PCI_BRIDGE_CTL_BUF_RESET )
and the next &= avoided
> +
> + msleep(100);
> + pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
> + ssleep(1);
> +
> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
> + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
> +
> + ret = pci_enable_device(pdev);
> + if (ret)
> + xmgmt_err(xm, "failed to enable device, ret %d", ret);
> +
> + for (i = 0; i < 300; i++) {
> + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
> + if (pci_cmd != 0xffff)
what happens with i == 300 and pci_cmd is still 0xffff ?
> + break;
> + msleep(20);
> + }
> +
> + xmgmt_info(xm, "waiting for %d ms", i * 20);
> + xmgmt_pci_restore_config_all(xm);
> + xmgmt_config_pci(xm);
> +}
> +
> +static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb)
> +{
> + char *dtb = NULL;
> + int ret;
> +
> + ret = xrt_md_create(XMGMT_DEV(xm), &dtb);
> + if (ret) {
> + xmgmt_err(xm, "create metadata failed, ret %d", ret);
> + goto failed;
> + }
> +
> + ret = xroot_add_vsec_node(xm->root, dtb);
> + if (ret == -ENOENT) {
> + /*
> + * We may be dealing with a MFG board.
> + * Try vsec-golden which will bring up all hard-coded leaves
> + * at hard-coded offsets.
> + */
> + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_VSEC_GOLDEN);
Manufacturing diagnostics ?
Tom
> + } else if (ret == 0) {
> + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_MGMT_MAIN);
> + }
> + if (ret)
> + goto failed;
> +
> + *root_dtb = dtb;
> + return 0;
> +
> +failed:
> + vfree(dtb);
> + return ret;
> +}
> +
> +static ssize_t ready_show(struct device *dev,
> + struct device_attribute *da,
> + char *buf)
> +{
> + struct pci_dev *pdev = to_pci_dev(dev);
> + struct xmgmt *xm = pci_get_drvdata(pdev);
> +
> + return sprintf(buf, "%d\n", xm->ready);
> +}
> +static DEVICE_ATTR_RO(ready);
> +
> +static struct attribute *xmgmt_root_attrs[] = {
> + &dev_attr_ready.attr,
> + NULL
> +};
> +
> +static struct attribute_group xmgmt_root_attr_group = {
> + .attrs = xmgmt_root_attrs,
> +};
> +
> +static struct xroot_pf_cb xmgmt_xroot_pf_cb = {
> + .xpc_hot_reset = xmgmt_root_hot_reset,
> +};
> +
> +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> +{
> + int ret;
> + struct device *dev = &pdev->dev;
> + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
> + char *dtb = NULL;
> +
> + if (!xm)
> + return -ENOMEM;
> + xm->pdev = pdev;
> + pci_set_drvdata(pdev, xm);
> +
> + ret = xmgmt_config_pci(xm);
> + if (ret)
> + goto failed;
> +
> + ret = xroot_probe(pdev, &xmgmt_xroot_pf_cb, &xm->root);
> + if (ret)
> + goto failed;
> +
> + ret = xmgmt_create_root_metadata(xm, &dtb);
> + if (ret)
> + goto failed_metadata;
> +
> + ret = xroot_create_group(xm->root, dtb);
> + vfree(dtb);
> + if (ret)
> + xmgmt_err(xm, "failed to create root group: %d", ret);
> +
> + if (!xroot_wait_for_bringup(xm->root))
> + xmgmt_err(xm, "failed to bringup all groups");
> + else
> + xm->ready = true;
> +
> + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
> + if (ret) {
> + /* Warning instead of failing the probe. */
> + xmgmt_warn(xm, "create xmgmt root attrs failed: %d", ret);
> + }
> +
> + xroot_broadcast(xm->root, XRT_EVENT_POST_CREATION);
> + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
> + return 0;
> +
> +failed_metadata:
> + (void)xroot_remove(xm->root);
> +failed:
> + pci_set_drvdata(pdev, NULL);
> + return ret;
> +}
> +
> +static void xmgmt_remove(struct pci_dev *pdev)
> +{
> + struct xmgmt *xm = pci_get_drvdata(pdev);
> +
> + xroot_broadcast(xm->root, XRT_EVENT_PRE_REMOVAL);
> + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
> + (void)xroot_remove(xm->root);
> + pci_disable_pcie_error_reporting(xm->pdev);
> + xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
> +}
> +
> +static struct pci_driver xmgmt_driver = {
> + .name = XMGMT_MODULE_NAME,
> + .id_table = xmgmt_pci_ids,
> + .probe = xmgmt_probe,
> + .remove = xmgmt_remove,
> +};
> +
> +static int __init xmgmt_init(void)
> +{
> + int res = 0;
> +
> + res = xmgmt_main_register_leaf();
> + if (res)
> + return res;
> +
> + xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME);
> + if (IS_ERR(xmgmt_class))
> + return PTR_ERR(xmgmt_class);
> +
> + res = pci_register_driver(&xmgmt_driver);
> + if (res) {
> + class_destroy(xmgmt_class);
> + return res;
> + }
> +
> + return 0;
> +}
> +
> +static __exit void xmgmt_exit(void)
> +{
> + pci_unregister_driver(&xmgmt_driver);
> + class_destroy(xmgmt_class);
> + xmgmt_main_unregister_leaf();
> +}
> +
> +module_init(xmgmt_init);
> +module_exit(xmgmt_exit);
> +
> +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
> +MODULE_VERSION(XMGMT_DRIVER_VERSION);
> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
> +MODULE_DESCRIPTION("Xilinx Alveo management function driver");
> +MODULE_LICENSE("GPL v2");
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root)
2021-02-26 15:01 ` Tom Rix
@ 2021-02-26 17:56 ` Moritz Fischer
2021-03-16 20:29 ` Max Zhen
1 sibling, 0 replies; 87+ messages in thread
From: Moritz Fischer @ 2021-02-26 17:56 UTC (permalink / raw)
To: Tom Rix
Cc: Lizhi Hou, linux-kernel, mdf, Lizhi Hou, linux-fpga, maxz,
sonal.santan, michal.simek, stefanos, devicetree, robh, Max Zhen
On Fri, Feb 26, 2021 at 07:01:05AM -0800, Tom Rix wrote:
> A question i do not know the answer to.
>
> Seems like 'golden' is linked to a manufacturing (diagnostics?) image.
From my brief history with Xilinx Ultrascale+ PCI cards I recall the golden
image being a sort of known good recovery image.
If we can't tell it should probably be explained better :)
>
> If the public will never see it, should handling it here be done ?
Yes. We do want people to run their entire stack using mainline linux,
not just a part of it, if code is needed to get from recovery image to
full image or similar, then we should support that.
> Moritz, do you know ?
>
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
> > The PCIE device driver which attaches to management function on Alveo
> to the management
> > devices. It instantiates one or more partition drivers which in turn
> more fpga partition / group ?
> > instantiate platform drivers. The instantiation of partition and platform
> > drivers is completely data driven.
> data driven ? everything is data driven. do you mean dtb driven ?
> >
> > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> > Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> > ---
> > drivers/fpga/xrt/include/xroot.h | 114 +++++++++++
> > drivers/fpga/xrt/mgmt/root.c | 342 +++++++++++++++++++++++++++++++
> > 2 files changed, 456 insertions(+)
> > create mode 100644 drivers/fpga/xrt/include/xroot.h
> > create mode 100644 drivers/fpga/xrt/mgmt/root.c
> >
> > diff --git a/drivers/fpga/xrt/include/xroot.h b/drivers/fpga/xrt/include/xroot.h
> > new file mode 100644
> > index 000000000000..752e10daa85e
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/include/xroot.h
> > @@ -0,0 +1,114 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * Header file for Xilinx Runtime (XRT) driver
> > + *
> > + * Copyright (C) 2020-2021 Xilinx, Inc.
> > + *
> > + * Authors:
> > + * Cheng Zhen <maxz@xilinx.com>
> > + */
> > +
> > +#ifndef _XRT_ROOT_H_
> > +#define _XRT_ROOT_H_
> > +
> > +#include <linux/pci.h>
> > +#include "subdev_id.h"
> > +#include "events.h"
> > +
> > +typedef bool (*xrt_subdev_match_t)(enum xrt_subdev_id,
> > + struct platform_device *, void *);
> > +#define XRT_SUBDEV_MATCH_PREV ((xrt_subdev_match_t)-1)
> > +#define XRT_SUBDEV_MATCH_NEXT ((xrt_subdev_match_t)-2)
> > +
> > +/*
> > + * Root IOCTL calls.
> > + */
> > +enum xrt_root_ioctl_cmd {
> > + /* Leaf actions. */
> > + XRT_ROOT_GET_LEAF = 0,
> > + XRT_ROOT_PUT_LEAF,
> > + XRT_ROOT_GET_LEAF_HOLDERS,
> > +
> > + /* Group actions. */
> > + XRT_ROOT_CREATE_GROUP,
> > + XRT_ROOT_REMOVE_GROUP,
> > + XRT_ROOT_LOOKUP_GROUP,
> > + XRT_ROOT_WAIT_GROUP_BRINGUP,
> > +
> > + /* Event actions. */
> > + XRT_ROOT_EVENT,
> should this be XRT_ROOT_EVENT_SYNC ?
> > + XRT_ROOT_EVENT_ASYNC,
> > +
> > + /* Device info. */
> > + XRT_ROOT_GET_RESOURCE,
> > + XRT_ROOT_GET_ID,
> > +
> > + /* Misc. */
> > + XRT_ROOT_HOT_RESET,
> > + XRT_ROOT_HWMON,
> > +};
> > +
> > +struct xrt_root_ioctl_get_leaf {
> > + struct platform_device *xpigl_pdev; /* caller's pdev */
> xpigl_ ? unneeded suffix in element names
> > + xrt_subdev_match_t xpigl_match_cb;
> > + void *xpigl_match_arg;
> > + struct platform_device *xpigl_leaf; /* target leaf pdev */
> > +};
> > +
> > +struct xrt_root_ioctl_put_leaf {
> > + struct platform_device *xpipl_pdev; /* caller's pdev */
> > + struct platform_device *xpipl_leaf; /* target's pdev */
>
> caller_pdev;
>
> target_pdev;
>
> > +};
> > +
> > +struct xrt_root_ioctl_lookup_group {
> > + struct platform_device *xpilp_pdev; /* caller's pdev */
> > + xrt_subdev_match_t xpilp_match_cb;
> > + void *xpilp_match_arg;
> > + int xpilp_grp_inst;
> > +};
> > +
> > +struct xrt_root_ioctl_get_holders {
> > + struct platform_device *xpigh_pdev; /* caller's pdev */
> > + char *xpigh_holder_buf;
> > + size_t xpigh_holder_buf_len;
> > +};
> > +
> > +struct xrt_root_ioctl_get_res {
> > + struct resource *xpigr_res;
> > +};
> > +
> > +struct xrt_root_ioctl_get_id {
> > + unsigned short xpigi_vendor_id;
> > + unsigned short xpigi_device_id;
> > + unsigned short xpigi_sub_vendor_id;
> > + unsigned short xpigi_sub_device_id;
> > +};
> > +
> > +struct xrt_root_ioctl_hwmon {
> > + bool xpih_register;
> > + const char *xpih_name;
> > + void *xpih_drvdata;
> > + const struct attribute_group **xpih_groups;
> > + struct device *xpih_hwmon_dev;
> > +};
> > +
> > +typedef int (*xrt_subdev_root_cb_t)(struct device *, void *, u32, void *);
> This function pointer type is important, please add a comment about its use and expected parameters
> > +int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void *arg);
> > +
> > +/*
> > + * Defines physical function (MPF / UPF) specific operations
> > + * needed in common root driver.
> > + */
> > +struct xroot_pf_cb {
> > + void (*xpc_hot_reset)(struct pci_dev *pdev);
> This is only ever set to xmgmt_root_hot_reset, why is this abstraction needed ?
> > +};
> > +
> > +int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root);
> > +void xroot_remove(void *root);
> > +bool xroot_wait_for_bringup(void *root);
> > +int xroot_add_vsec_node(void *root, char *dtb);
> > +int xroot_create_group(void *xr, char *dtb);
> > +int xroot_add_simple_node(void *root, char *dtb, const char *endpoint);
> > +void xroot_broadcast(void *root, enum xrt_events evt);
> > +
> > +#endif /* _XRT_ROOT_H_ */
> > diff --git a/drivers/fpga/xrt/mgmt/root.c b/drivers/fpga/xrt/mgmt/root.c
> > new file mode 100644
> > index 000000000000..583a37c9d30c
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/mgmt/root.c
> > @@ -0,0 +1,342 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Xilinx Alveo Management Function Driver
> > + *
> > + * Copyright (C) 2020-2021 Xilinx, Inc.
> > + *
> > + * Authors:
> > + * Cheng Zhen <maxz@xilinx.com>
> > + */
> > +
> > +#include <linux/module.h>
> > +#include <linux/pci.h>
> > +#include <linux/aer.h>
> > +#include <linux/vmalloc.h>
> > +#include <linux/delay.h>
> > +
> > +#include "xroot.h"
> > +#include "main-impl.h"
> > +#include "metadata.h"
> > +
> > +#define XMGMT_MODULE_NAME "xmgmt"
> The xrt modules would be more easily identified with a 'xrt' prefix instead of 'x'
> > +#define XMGMT_DRIVER_VERSION "4.0.0"
> > +
> > +#define XMGMT_PDEV(xm) ((xm)->pdev)
> > +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
> > +#define xmgmt_err(xm, fmt, args...) \
> > + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> > +#define xmgmt_warn(xm, fmt, args...) \
> > + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> > +#define xmgmt_info(xm, fmt, args...) \
> > + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> > +#define xmgmt_dbg(xm, fmt, args...) \
> > + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
> > +#define XMGMT_DEV_ID(_pcidev) \
> > + ({ typeof(_pcidev) (pcidev) = (_pcidev); \
> > + ((pci_domain_nr((pcidev)->bus) << 16) | \
> > + PCI_DEVID((pcidev)->bus->number, 0)); })
> > +
> > +static struct class *xmgmt_class;
> > +static const struct pci_device_id xmgmt_pci_ids[] = {
> > + { PCI_DEVICE(0x10EE, 0xd020), }, /* Alveo U50 (golden image) */
> > + { PCI_DEVICE(0x10EE, 0x5020), }, /* Alveo U50 */
>
> demagic this table, look at dfl-pci for how to use existing #define for the vendor and create a new on for the device. If there are vf's add them at the same time.
>
> What is a golden image ?
>
> > + { 0, }
> > +};
> > +
> > +struct xmgmt {
> > + struct pci_dev *pdev;
> > + void *root;
> > +
> > + bool ready;
> > +};
> > +
> > +static int xmgmt_config_pci(struct xmgmt *xm)
> > +{
> > + struct pci_dev *pdev = XMGMT_PDEV(xm);
> > + int rc;
> > +
> > + rc = pcim_enable_device(pdev);
> > + if (rc < 0) {
> > + xmgmt_err(xm, "failed to enable device: %d", rc);
> > + return rc;
> > + }
> > +
> > + rc = pci_enable_pcie_error_reporting(pdev);
> > + if (rc)
> > + xmgmt_warn(xm, "failed to enable AER: %d", rc);
> > +
> > + pci_set_master(pdev);
> > +
> > + rc = pcie_get_readrq(pdev);
> Review this call, it does not go negative
> > + if (rc < 0) {
> > + xmgmt_err(xm, "failed to read mrrs %d", rc);
> > + return rc;
> > + }
> this is a quirk, add a comment.
> > + if (rc > 512) {
> > + rc = pcie_set_readrq(pdev, 512);
> > + if (rc) {
> > + xmgmt_err(xm, "failed to force mrrs %d", rc);
> similar calls do not fail here.
> > + return rc;
> > + }
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int xmgmt_match_slot_and_save(struct device *dev, void *data)
> > +{
> > + struct xmgmt *xm = data;
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > +
> > + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
> > + pci_cfg_access_lock(pdev);
> > + pci_save_state(pdev);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void xmgmt_pci_save_config_all(struct xmgmt *xm)
> > +{
> > + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save);
>
> This is a bus call, not a device call.
>
> Can this be changed into something like what hot reset does ?
>
> > +}
> > +
> > +static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
> > +{
> > + struct xmgmt *xm = data;
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > +
> > + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
> > + pci_restore_state(pdev);
> > + pci_cfg_access_unlock(pdev);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
> > +{
> > + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore);
> > +}
> > +
> > +static void xmgmt_root_hot_reset(struct pci_dev *pdev)
> > +{
> > + struct xmgmt *xm = pci_get_drvdata(pdev);
> > + struct pci_bus *bus;
> > + u8 pci_bctl;
> > + u16 pci_cmd, devctl;
> > + int i, ret;
> > +
> > + xmgmt_info(xm, "hot reset start");
> > +
> > + xmgmt_pci_save_config_all(xm);
> > +
> > + pci_disable_device(pdev);
> > +
> > + bus = pdev->bus;
> > +
> > + /*
> > + * When flipping the SBR bit, device can fall off the bus. This is
> > + * usually no problem at all so long as drivers are working properly
> > + * after SBR. However, some systems complain bitterly when the device
> > + * falls off the bus.
> > + * The quick solution is to temporarily disable the SERR reporting of
> > + * switch port during SBR.
> > + */
> > +
> > + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
> > + pci_write_config_word(bus->self, PCI_COMMAND,
> > + (pci_cmd & ~PCI_COMMAND_SERR));
> > + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
> > + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
> > + (devctl & ~PCI_EXP_DEVCTL_FERE));
> > + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
> > + pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
> > + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
>
> how the pci config values are set and cleared should be consistent.
>
> this call should be
>
> pci_write_config_byte (... pci_bctl | PCI_BRIDGE_CTL_BUF_RESET )
>
> and the next &= avoided
>
> > +
> > + msleep(100);
> > + pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
> > + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
> > + ssleep(1);
> > +
> > + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
> > + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
> > +
> > + ret = pci_enable_device(pdev);
> > + if (ret)
> > + xmgmt_err(xm, "failed to enable device, ret %d", ret);
> > +
> > + for (i = 0; i < 300; i++) {
> > + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
> > + if (pci_cmd != 0xffff)
> what happens with i == 300 and pci_cmd is still 0xffff ?
> > + break;
> > + msleep(20);
> > + }
> > +
> > + xmgmt_info(xm, "waiting for %d ms", i * 20);
> > + xmgmt_pci_restore_config_all(xm);
> > + xmgmt_config_pci(xm);
> > +}
> > +
> > +static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb)
> > +{
> > + char *dtb = NULL;
> > + int ret;
> > +
> > + ret = xrt_md_create(XMGMT_DEV(xm), &dtb);
> > + if (ret) {
> > + xmgmt_err(xm, "create metadata failed, ret %d", ret);
> > + goto failed;
> > + }
> > +
> > + ret = xroot_add_vsec_node(xm->root, dtb);
> > + if (ret == -ENOENT) {
> > + /*
> > + * We may be dealing with a MFG board.
> > + * Try vsec-golden which will bring up all hard-coded leaves
> > + * at hard-coded offsets.
> > + */
> > + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_VSEC_GOLDEN);
>
> Manufacturing diagnostics ?
>
> Tom
>
> > + } else if (ret == 0) {
> > + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_MGMT_MAIN);
> > + }
> > + if (ret)
> > + goto failed;
> > +
> > + *root_dtb = dtb;
> > + return 0;
> > +
> > +failed:
> > + vfree(dtb);
> > + return ret;
> > +}
> > +
> > +static ssize_t ready_show(struct device *dev,
> > + struct device_attribute *da,
> > + char *buf)
> > +{
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > + struct xmgmt *xm = pci_get_drvdata(pdev);
> > +
> > + return sprintf(buf, "%d\n", xm->ready);
> > +}
> > +static DEVICE_ATTR_RO(ready);
> > +
> > +static struct attribute *xmgmt_root_attrs[] = {
> > + &dev_attr_ready.attr,
> > + NULL
> > +};
> > +
> > +static struct attribute_group xmgmt_root_attr_group = {
> > + .attrs = xmgmt_root_attrs,
> > +};
> > +
> > +static struct xroot_pf_cb xmgmt_xroot_pf_cb = {
> > + .xpc_hot_reset = xmgmt_root_hot_reset,
> > +};
> > +
> > +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> > +{
> > + int ret;
> > + struct device *dev = &pdev->dev;
> > + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
> > + char *dtb = NULL;
> > +
> > + if (!xm)
> > + return -ENOMEM;
> > + xm->pdev = pdev;
> > + pci_set_drvdata(pdev, xm);
> > +
> > + ret = xmgmt_config_pci(xm);
> > + if (ret)
> > + goto failed;
> > +
> > + ret = xroot_probe(pdev, &xmgmt_xroot_pf_cb, &xm->root);
> > + if (ret)
> > + goto failed;
> > +
> > + ret = xmgmt_create_root_metadata(xm, &dtb);
> > + if (ret)
> > + goto failed_metadata;
> > +
> > + ret = xroot_create_group(xm->root, dtb);
> > + vfree(dtb);
> > + if (ret)
> > + xmgmt_err(xm, "failed to create root group: %d", ret);
> > +
> > + if (!xroot_wait_for_bringup(xm->root))
> > + xmgmt_err(xm, "failed to bringup all groups");
> > + else
> > + xm->ready = true;
> > +
> > + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
> > + if (ret) {
> > + /* Warning instead of failing the probe. */
> > + xmgmt_warn(xm, "create xmgmt root attrs failed: %d", ret);
> > + }
> > +
> > + xroot_broadcast(xm->root, XRT_EVENT_POST_CREATION);
> > + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
> > + return 0;
> > +
> > +failed_metadata:
> > + (void)xroot_remove(xm->root);
> > +failed:
> > + pci_set_drvdata(pdev, NULL);
> > + return ret;
> > +}
> > +
> > +static void xmgmt_remove(struct pci_dev *pdev)
> > +{
> > + struct xmgmt *xm = pci_get_drvdata(pdev);
> > +
> > + xroot_broadcast(xm->root, XRT_EVENT_PRE_REMOVAL);
> > + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
> > + (void)xroot_remove(xm->root);
> > + pci_disable_pcie_error_reporting(xm->pdev);
> > + xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
> > +}
> > +
> > +static struct pci_driver xmgmt_driver = {
> > + .name = XMGMT_MODULE_NAME,
> > + .id_table = xmgmt_pci_ids,
> > + .probe = xmgmt_probe,
> > + .remove = xmgmt_remove,
> > +};
> > +
> > +static int __init xmgmt_init(void)
> > +{
> > + int res = 0;
> > +
> > + res = xmgmt_main_register_leaf();
> > + if (res)
> > + return res;
> > +
> > + xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME);
> > + if (IS_ERR(xmgmt_class))
> > + return PTR_ERR(xmgmt_class);
> > +
> > + res = pci_register_driver(&xmgmt_driver);
> > + if (res) {
> > + class_destroy(xmgmt_class);
> > + return res;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static __exit void xmgmt_exit(void)
> > +{
> > + pci_unregister_driver(&xmgmt_driver);
> > + class_destroy(xmgmt_class);
> > + xmgmt_main_unregister_leaf();
> > +}
> > +
> > +module_init(xmgmt_init);
> > +module_exit(xmgmt_exit);
> > +
> > +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
> > +MODULE_VERSION(XMGMT_DRIVER_VERSION);
> > +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
> > +MODULE_DESCRIPTION("Xilinx Alveo management function driver");
> > +MODULE_LICENSE("GPL v2");
>
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root)
2021-02-26 15:01 ` Tom Rix
2021-02-26 17:56 ` Moritz Fischer
@ 2021-03-16 20:29 ` Max Zhen
2021-03-17 21:08 ` Tom Rix
1 sibling, 1 reply; 87+ messages in thread
From: Max Zhen @ 2021-03-16 20:29 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel, mdf
Cc: Lizhi Hou, linux-fpga, sonal.santan, michal.simek, stefanos,
devicetree, robh, Max Zhen
Hi Tom,
On 2/26/21 7:01 AM, Tom Rix wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> A question i do not know the answer to.
>
> Seems like 'golden' is linked to a manufacturing (diagnostics?) image.
>
> If the public will never see it, should handling it here be done ?
>
> Moritz, do you know ?
Golden image is preloaded on the device when it is shipped to customer.
Then, customer can load other shells (from Xilinx or some other vendor).
If something goes wrong with the shell, customer can always go back to
golden and start over again. So, golden image is going to be used in
public, not just internally by Xilinx.
>
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> The PCIE device driver which attaches to management function on Alveo
> to the management
Sure.
>> devices. It instantiates one or more partition drivers which in turn
> more fpga partition / group ?
Group driver.
>> instantiate platform drivers. The instantiation of partition and platform
>> drivers is completely data driven.
> data driven ? everything is data driven. do you mean dtb driven ?
Data driven means not hard-coded. Here data means meta data which is
presented in device tree format, dtb.
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xroot.h | 114 +++++++++++
>> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++++++++++++++++++++++++
>> 2 files changed, 456 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xroot.h
>> create mode 100644 drivers/fpga/xrt/mgmt/root.c
>>
>> diff --git a/drivers/fpga/xrt/include/xroot.h b/drivers/fpga/xrt/include/xroot.h
>> new file mode 100644
>> index 000000000000..752e10daa85e
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xroot.h
>> @@ -0,0 +1,114 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for Xilinx Runtime (XRT) driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_ROOT_H_
>> +#define _XRT_ROOT_H_
>> +
>> +#include <linux/pci.h>
>> +#include "subdev_id.h"
>> +#include "events.h"
>> +
>> +typedef bool (*xrt_subdev_match_t)(enum xrt_subdev_id,
>> + struct platform_device *, void *);
>> +#define XRT_SUBDEV_MATCH_PREV ((xrt_subdev_match_t)-1)
>> +#define XRT_SUBDEV_MATCH_NEXT ((xrt_subdev_match_t)-2)
>> +
>> +/*
>> + * Root IOCTL calls.
>> + */
>> +enum xrt_root_ioctl_cmd {
>> + /* Leaf actions. */
>> + XRT_ROOT_GET_LEAF = 0,
>> + XRT_ROOT_PUT_LEAF,
>> + XRT_ROOT_GET_LEAF_HOLDERS,
>> +
>> + /* Group actions. */
>> + XRT_ROOT_CREATE_GROUP,
>> + XRT_ROOT_REMOVE_GROUP,
>> + XRT_ROOT_LOOKUP_GROUP,
>> + XRT_ROOT_WAIT_GROUP_BRINGUP,
>> +
>> + /* Event actions. */
>> + XRT_ROOT_EVENT,
> should this be XRT_ROOT_EVENT_SYNC ?
Sure.
>> + XRT_ROOT_EVENT_ASYNC,
>> +
>> + /* Device info. */
>> + XRT_ROOT_GET_RESOURCE,
>> + XRT_ROOT_GET_ID,
>> +
>> + /* Misc. */
>> + XRT_ROOT_HOT_RESET,
>> + XRT_ROOT_HWMON,
>> +};
>> +
>> +struct xrt_root_ioctl_get_leaf {
>> + struct platform_device *xpigl_pdev; /* caller's pdev */
> xpigl_ ? unneeded suffix in element names
It's needed since the it might be included and used in > 1 .c files. I'd
like to keep it's name unique.
>> + xrt_subdev_match_t xpigl_match_cb;
>> + void *xpigl_match_arg;
>> + struct platform_device *xpigl_leaf; /* target leaf pdev */
>> +};
>> +
>> +struct xrt_root_ioctl_put_leaf {
>> + struct platform_device *xpipl_pdev; /* caller's pdev */
>> + struct platform_device *xpipl_leaf; /* target's pdev */
> caller_pdev;
>
> target_pdev;
Sure.
>
>> +};
>> +
>> +struct xrt_root_ioctl_lookup_group {
>> + struct platform_device *xpilp_pdev; /* caller's pdev */
>> + xrt_subdev_match_t xpilp_match_cb;
>> + void *xpilp_match_arg;
>> + int xpilp_grp_inst;
>> +};
>> +
>> +struct xrt_root_ioctl_get_holders {
>> + struct platform_device *xpigh_pdev; /* caller's pdev */
>> + char *xpigh_holder_buf;
>> + size_t xpigh_holder_buf_len;
>> +};
>> +
>> +struct xrt_root_ioctl_get_res {
>> + struct resource *xpigr_res;
>> +};
>> +
>> +struct xrt_root_ioctl_get_id {
>> + unsigned short xpigi_vendor_id;
>> + unsigned short xpigi_device_id;
>> + unsigned short xpigi_sub_vendor_id;
>> + unsigned short xpigi_sub_device_id;
>> +};
>> +
>> +struct xrt_root_ioctl_hwmon {
>> + bool xpih_register;
>> + const char *xpih_name;
>> + void *xpih_drvdata;
>> + const struct attribute_group **xpih_groups;
>> + struct device *xpih_hwmon_dev;
>> +};
>> +
>> +typedef int (*xrt_subdev_root_cb_t)(struct device *, void *, u32, void *);
> This function pointer type is important, please add a comment about its use and expected parameters
Added.
>> +int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void *arg);
>> +
>> +/*
>> + * Defines physical function (MPF / UPF) specific operations
>> + * needed in common root driver.
>> + */
>> +struct xroot_pf_cb {
>> + void (*xpc_hot_reset)(struct pci_dev *pdev);
> This is only ever set to xmgmt_root_hot_reset, why is this abstraction needed ?
As comment says, hot reset is implemented differently in MPF and UPF
driver. So, we need this callback in this common code. Note that we have
not added UPF code in our initial patch yet. It will be added in the future.
>> +};
>> +
>> +int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root);
>> +void xroot_remove(void *root);
>> +bool xroot_wait_for_bringup(void *root);
>> +int xroot_add_vsec_node(void *root, char *dtb);
>> +int xroot_create_group(void *xr, char *dtb);
>> +int xroot_add_simple_node(void *root, char *dtb, const char *endpoint);
>> +void xroot_broadcast(void *root, enum xrt_events evt);
>> +
>> +#endif /* _XRT_ROOT_H_ */
>> diff --git a/drivers/fpga/xrt/mgmt/root.c b/drivers/fpga/xrt/mgmt/root.c
>> new file mode 100644
>> index 000000000000..583a37c9d30c
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/mgmt/root.c
>> @@ -0,0 +1,342 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo Management Function Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#include <linux/module.h>
>> +#include <linux/pci.h>
>> +#include <linux/aer.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/delay.h>
>> +
>> +#include "xroot.h"
>> +#include "main-impl.h"
>> +#include "metadata.h"
>> +
>> +#define XMGMT_MODULE_NAME "xmgmt"
> The xrt modules would be more easily identified with a 'xrt' prefix instead of 'x'
We will change the module name to xrt-mgmt.
>> +#define XMGMT_DRIVER_VERSION "4.0.0"
>> +
>> +#define XMGMT_PDEV(xm) ((xm)->pdev)
>> +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
>> +#define xmgmt_err(xm, fmt, args...) \
>> + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>> +#define xmgmt_warn(xm, fmt, args...) \
>> + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>> +#define xmgmt_info(xm, fmt, args...) \
>> + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>> +#define xmgmt_dbg(xm, fmt, args...) \
>> + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>> +#define XMGMT_DEV_ID(_pcidev) \
>> + ({ typeof(_pcidev) (pcidev) = (_pcidev); \
>> + ((pci_domain_nr((pcidev)->bus) << 16) | \
>> + PCI_DEVID((pcidev)->bus->number, 0)); })
>> +
>> +static struct class *xmgmt_class;
>> +static const struct pci_device_id xmgmt_pci_ids[] = {
>> + { PCI_DEVICE(0x10EE, 0xd020), }, /* Alveo U50 (golden image) */
>> + { PCI_DEVICE(0x10EE, 0x5020), }, /* Alveo U50 */
> demagic this table, look at dfl-pci for how to use existing #define for the vendor and create a new on for the device. If there are vf's add them at the same time.
>
> What is a golden image ?
Fixed. Please see my comments above for golden image.
>
>> + { 0, }
>> +};
>> +
>> +struct xmgmt {
>> + struct pci_dev *pdev;
>> + void *root;
>> +
>> + bool ready;
>> +};
>> +
>> +static int xmgmt_config_pci(struct xmgmt *xm)
>> +{
>> + struct pci_dev *pdev = XMGMT_PDEV(xm);
>> + int rc;
>> +
>> + rc = pcim_enable_device(pdev);
>> + if (rc < 0) {
>> + xmgmt_err(xm, "failed to enable device: %d", rc);
>> + return rc;
>> + }
>> +
>> + rc = pci_enable_pcie_error_reporting(pdev);
>> + if (rc)
>> + xmgmt_warn(xm, "failed to enable AER: %d", rc);
>> +
>> + pci_set_master(pdev);
>> +
>> + rc = pcie_get_readrq(pdev);
> Review this call, it does not go negative
I'll remove the check against negative value.
>> + if (rc < 0) {
>> + xmgmt_err(xm, "failed to read mrrs %d", rc);
>> + return rc;
>> + }
> this is a quirk, add a comment.
Will remove.
>> + if (rc > 512) {
>> + rc = pcie_set_readrq(pdev, 512);
>> + if (rc) {
>> + xmgmt_err(xm, "failed to force mrrs %d", rc);
> similar calls do not fail here.
Will remove.
>> + return rc;
>> + }
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int xmgmt_match_slot_and_save(struct device *dev, void *data)
>> +{
>> + struct xmgmt *xm = data;
>> + struct pci_dev *pdev = to_pci_dev(dev);
>> +
>> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
>> + pci_cfg_access_lock(pdev);
>> + pci_save_state(pdev);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static void xmgmt_pci_save_config_all(struct xmgmt *xm)
>> +{
>> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save);
> This is a bus call, not a device call.
>
> Can this be changed into something like what hot reset does ?
We are working on both mgmt pf and user pf here, so sort of like a bus.
But, it might be better to refactor this when we have our own bus type
implementation. We do not need to make PCIE bus call. We will fix this
in V5 patch set where we'll implement our own bus type.
>
>> +}
>> +
>> +static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
>> +{
>> + struct xmgmt *xm = data;
>> + struct pci_dev *pdev = to_pci_dev(dev);
>> +
>> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
>> + pci_restore_state(pdev);
>> + pci_cfg_access_unlock(pdev);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
>> +{
>> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore);
>> +}
>> +
>> +static void xmgmt_root_hot_reset(struct pci_dev *pdev)
>> +{
>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>> + struct pci_bus *bus;
>> + u8 pci_bctl;
>> + u16 pci_cmd, devctl;
>> + int i, ret;
>> +
>> + xmgmt_info(xm, "hot reset start");
>> +
>> + xmgmt_pci_save_config_all(xm);
>> +
>> + pci_disable_device(pdev);
>> +
>> + bus = pdev->bus;
>> +
>> + /*
>> + * When flipping the SBR bit, device can fall off the bus. This is
>> + * usually no problem at all so long as drivers are working properly
>> + * after SBR. However, some systems complain bitterly when the device
>> + * falls off the bus.
>> + * The quick solution is to temporarily disable the SERR reporting of
>> + * switch port during SBR.
>> + */
>> +
>> + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
>> + pci_write_config_word(bus->self, PCI_COMMAND,
>> + (pci_cmd & ~PCI_COMMAND_SERR));
>> + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
>> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
>> + (devctl & ~PCI_EXP_DEVCTL_FERE));
>> + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
>> + pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
>> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
> how the pci config values are set and cleared should be consistent.
>
> this call should be
>
> pci_write_config_byte (... pci_bctl | PCI_BRIDGE_CTL_BUF_RESET )
>
> and the next &= avoided
Sure.
>
>> +
>> + msleep(100);
>> + pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
>> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
>> + ssleep(1);
>> +
>> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
>> + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
>> +
>> + ret = pci_enable_device(pdev);
>> + if (ret)
>> + xmgmt_err(xm, "failed to enable device, ret %d", ret);
>> +
>> + for (i = 0; i < 300; i++) {
>> + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
>> + if (pci_cmd != 0xffff)
> what happens with i == 300 and pci_cmd is still 0xffff ?
Something wrong happens to the device since it's not coming back after
the reset. In this case, the device cannot be used and the only way to
recover is to power cycle the system so that the shell can be reloaded
from the flash on the device.
>> + break;
>> + msleep(20);
>> + }
>> +
>> + xmgmt_info(xm, "waiting for %d ms", i * 20);
>> + xmgmt_pci_restore_config_all(xm);
>> + xmgmt_config_pci(xm);
>> +}
>> +
>> +static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb)
>> +{
>> + char *dtb = NULL;
>> + int ret;
>> +
>> + ret = xrt_md_create(XMGMT_DEV(xm), &dtb);
>> + if (ret) {
>> + xmgmt_err(xm, "create metadata failed, ret %d", ret);
>> + goto failed;
>> + }
>> +
>> + ret = xroot_add_vsec_node(xm->root, dtb);
>> + if (ret == -ENOENT) {
>> + /*
>> + * We may be dealing with a MFG board.
>> + * Try vsec-golden which will bring up all hard-coded leaves
>> + * at hard-coded offsets.
>> + */
>> + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_VSEC_GOLDEN);
> Manufacturing diagnostics ?
This is for golden image support. Please see my comments above.
Thanks,
Max
>
> Tom
>
>> + } else if (ret == 0) {
>> + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_MGMT_MAIN);
>> + }
>> + if (ret)
>> + goto failed;
>> +
>> + *root_dtb = dtb;
>> + return 0;
>> +
>> +failed:
>> + vfree(dtb);
>> + return ret;
>> +}
>> +
>> +static ssize_t ready_show(struct device *dev,
>> + struct device_attribute *da,
>> + char *buf)
>> +{
>> + struct pci_dev *pdev = to_pci_dev(dev);
>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>> +
>> + return sprintf(buf, "%d\n", xm->ready);
>> +}
>> +static DEVICE_ATTR_RO(ready);
>> +
>> +static struct attribute *xmgmt_root_attrs[] = {
>> + &dev_attr_ready.attr,
>> + NULL
>> +};
>> +
>> +static struct attribute_group xmgmt_root_attr_group = {
>> + .attrs = xmgmt_root_attrs,
>> +};
>> +
>> +static struct xroot_pf_cb xmgmt_xroot_pf_cb = {
>> + .xpc_hot_reset = xmgmt_root_hot_reset,
>> +};
>> +
>> +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>> +{
>> + int ret;
>> + struct device *dev = &pdev->dev;
>> + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
>> + char *dtb = NULL;
>> +
>> + if (!xm)
>> + return -ENOMEM;
>> + xm->pdev = pdev;
>> + pci_set_drvdata(pdev, xm);
>> +
>> + ret = xmgmt_config_pci(xm);
>> + if (ret)
>> + goto failed;
>> +
>> + ret = xroot_probe(pdev, &xmgmt_xroot_pf_cb, &xm->root);
>> + if (ret)
>> + goto failed;
>> +
>> + ret = xmgmt_create_root_metadata(xm, &dtb);
>> + if (ret)
>> + goto failed_metadata;
>> +
>> + ret = xroot_create_group(xm->root, dtb);
>> + vfree(dtb);
>> + if (ret)
>> + xmgmt_err(xm, "failed to create root group: %d", ret);
>> +
>> + if (!xroot_wait_for_bringup(xm->root))
>> + xmgmt_err(xm, "failed to bringup all groups");
>> + else
>> + xm->ready = true;
>> +
>> + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
>> + if (ret) {
>> + /* Warning instead of failing the probe. */
>> + xmgmt_warn(xm, "create xmgmt root attrs failed: %d", ret);
>> + }
>> +
>> + xroot_broadcast(xm->root, XRT_EVENT_POST_CREATION);
>> + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
>> + return 0;
>> +
>> +failed_metadata:
>> + (void)xroot_remove(xm->root);
>> +failed:
>> + pci_set_drvdata(pdev, NULL);
>> + return ret;
>> +}
>> +
>> +static void xmgmt_remove(struct pci_dev *pdev)
>> +{
>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>> +
>> + xroot_broadcast(xm->root, XRT_EVENT_PRE_REMOVAL);
>> + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
>> + (void)xroot_remove(xm->root);
>> + pci_disable_pcie_error_reporting(xm->pdev);
>> + xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
>> +}
>> +
>> +static struct pci_driver xmgmt_driver = {
>> + .name = XMGMT_MODULE_NAME,
>> + .id_table = xmgmt_pci_ids,
>> + .probe = xmgmt_probe,
>> + .remove = xmgmt_remove,
>> +};
>> +
>> +static int __init xmgmt_init(void)
>> +{
>> + int res = 0;
>> +
>> + res = xmgmt_main_register_leaf();
>> + if (res)
>> + return res;
>> +
>> + xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME);
>> + if (IS_ERR(xmgmt_class))
>> + return PTR_ERR(xmgmt_class);
>> +
>> + res = pci_register_driver(&xmgmt_driver);
>> + if (res) {
>> + class_destroy(xmgmt_class);
>> + return res;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static __exit void xmgmt_exit(void)
>> +{
>> + pci_unregister_driver(&xmgmt_driver);
>> + class_destroy(xmgmt_class);
>> + xmgmt_main_unregister_leaf();
>> +}
>> +
>> +module_init(xmgmt_init);
>> +module_exit(xmgmt_exit);
>> +
>> +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
>> +MODULE_VERSION(XMGMT_DRIVER_VERSION);
>> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
>> +MODULE_DESCRIPTION("Xilinx Alveo management function driver");
>> +MODULE_LICENSE("GPL v2");
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root)
2021-03-16 20:29 ` Max Zhen
@ 2021-03-17 21:08 ` Tom Rix
2021-03-18 0:44 ` Max Zhen
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-17 21:08 UTC (permalink / raw)
To: Max Zhen, Lizhi Hou, linux-kernel, mdf
Cc: Lizhi Hou, linux-fpga, sonal.santan, michal.simek, stefanos,
devicetree, robh
On 3/16/21 1:29 PM, Max Zhen wrote:
> Hi Tom,
>
>
> On 2/26/21 7:01 AM, Tom Rix wrote:
>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>>
>>
>> A question i do not know the answer to.
>>
>> Seems like 'golden' is linked to a manufacturing (diagnostics?) image.
>>
>> If the public will never see it, should handling it here be done ?
>>
>> Moritz, do you know ?
>
>
> Golden image is preloaded on the device when it is shipped to customer. Then, customer can load other shells (from Xilinx or some other vendor). If something goes wrong with the shell, customer can always go back to golden and start over again. So, golden image is going to be used in public, not just internally by Xilinx.
>
>
Thanks for the explanation.
>>
>>
>> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>>> The PCIE device driver which attaches to management function on Alveo
>> to the management
>
>
> Sure.
>
>
>>> devices. It instantiates one or more partition drivers which in turn
>> more fpga partition / group ?
>
>
> Group driver.
>
>
>>> instantiate platform drivers. The instantiation of partition and platform
>>> drivers is completely data driven.
>> data driven ? everything is data driven. do you mean dtb driven ?
>
>
> Data driven means not hard-coded. Here data means meta data which is presented in device tree format, dtb.
>
>
>>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>>> ---
>>> drivers/fpga/xrt/include/xroot.h | 114 +++++++++++
>>> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++++++++++++++++++++++++
>>> 2 files changed, 456 insertions(+)
>>> create mode 100644 drivers/fpga/xrt/include/xroot.h
>>> create mode 100644 drivers/fpga/xrt/mgmt/root.c
>>>
>>> diff --git a/drivers/fpga/xrt/include/xroot.h b/drivers/fpga/xrt/include/xroot.h
>>> new file mode 100644
>>> index 000000000000..752e10daa85e
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/include/xroot.h
>>> @@ -0,0 +1,114 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * Header file for Xilinx Runtime (XRT) driver
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors:
>>> + * Cheng Zhen <maxz@xilinx.com>
>>> + */
>>> +
>>> +#ifndef _XRT_ROOT_H_
>>> +#define _XRT_ROOT_H_
>>> +
>>> +#include <linux/pci.h>
>>> +#include "subdev_id.h"
>>> +#include "events.h"
>>> +
>>> +typedef bool (*xrt_subdev_match_t)(enum xrt_subdev_id,
>>> + struct platform_device *, void *);
>>> +#define XRT_SUBDEV_MATCH_PREV ((xrt_subdev_match_t)-1)
>>> +#define XRT_SUBDEV_MATCH_NEXT ((xrt_subdev_match_t)-2)
>>> +
>>> +/*
>>> + * Root IOCTL calls.
>>> + */
>>> +enum xrt_root_ioctl_cmd {
>>> + /* Leaf actions. */
>>> + XRT_ROOT_GET_LEAF = 0,
>>> + XRT_ROOT_PUT_LEAF,
>>> + XRT_ROOT_GET_LEAF_HOLDERS,
>>> +
>>> + /* Group actions. */
>>> + XRT_ROOT_CREATE_GROUP,
>>> + XRT_ROOT_REMOVE_GROUP,
>>> + XRT_ROOT_LOOKUP_GROUP,
>>> + XRT_ROOT_WAIT_GROUP_BRINGUP,
>>> +
>>> + /* Event actions. */
>>> + XRT_ROOT_EVENT,
>> should this be XRT_ROOT_EVENT_SYNC ?
>
>
> Sure.
>
>
>>> + XRT_ROOT_EVENT_ASYNC,
>>> +
>>> + /* Device info. */
>>> + XRT_ROOT_GET_RESOURCE,
>>> + XRT_ROOT_GET_ID,
>>> +
>>> + /* Misc. */
>>> + XRT_ROOT_HOT_RESET,
>>> + XRT_ROOT_HWMON,
>>> +};
>>> +
>>> +struct xrt_root_ioctl_get_leaf {
>>> + struct platform_device *xpigl_pdev; /* caller's pdev */
>> xpigl_ ? unneeded suffix in element names
>
>
> It's needed since the it might be included and used in > 1 .c files. I'd like to keep it's name unique.
This is an element name, the variable name sound be unique enough to make it clear.
This is not a critical issue, ok as-is.
>
>
>>> + xrt_subdev_match_t xpigl_match_cb;
>>> + void *xpigl_match_arg;
>>> + struct platform_device *xpigl_leaf; /* target leaf pdev */
>>> +};
>>> +
>>> +struct xrt_root_ioctl_put_leaf {
>>> + struct platform_device *xpipl_pdev; /* caller's pdev */
>>> + struct platform_device *xpipl_leaf; /* target's pdev */
>> caller_pdev;
>>
>> target_pdev;
>
>
> Sure.
>
>
>>
>>> +};
>>> +
>>> +struct xrt_root_ioctl_lookup_group {
>>> + struct platform_device *xpilp_pdev; /* caller's pdev */
>>> + xrt_subdev_match_t xpilp_match_cb;
>>> + void *xpilp_match_arg;
>>> + int xpilp_grp_inst;
>>> +};
>>> +
>>> +struct xrt_root_ioctl_get_holders {
>>> + struct platform_device *xpigh_pdev; /* caller's pdev */
>>> + char *xpigh_holder_buf;
>>> + size_t xpigh_holder_buf_len;
>>> +};
>>> +
>>> +struct xrt_root_ioctl_get_res {
>>> + struct resource *xpigr_res;
>>> +};
>>> +
>>> +struct xrt_root_ioctl_get_id {
>>> + unsigned short xpigi_vendor_id;
>>> + unsigned short xpigi_device_id;
>>> + unsigned short xpigi_sub_vendor_id;
>>> + unsigned short xpigi_sub_device_id;
>>> +};
>>> +
>>> +struct xrt_root_ioctl_hwmon {
>>> + bool xpih_register;
>>> + const char *xpih_name;
>>> + void *xpih_drvdata;
>>> + const struct attribute_group **xpih_groups;
>>> + struct device *xpih_hwmon_dev;
>>> +};
>>> +
>>> +typedef int (*xrt_subdev_root_cb_t)(struct device *, void *, u32, void *);
>> This function pointer type is important, please add a comment about its use and expected parameters
>
>
> Added.
>
>
>>> +int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void *arg);
>>> +
>>> +/*
>>> + * Defines physical function (MPF / UPF) specific operations
>>> + * needed in common root driver.
>>> + */
>>> +struct xroot_pf_cb {
>>> + void (*xpc_hot_reset)(struct pci_dev *pdev);
>> This is only ever set to xmgmt_root_hot_reset, why is this abstraction needed ?
>
>
> As comment says, hot reset is implemented differently in MPF and UPF driver. So, we need this callback in this common code. Note that we have not added UPF code in our initial patch yet. It will be added in the future.
>
>
>>> +};
>>> +
>>> +int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root);
>>> +void xroot_remove(void *root);
>>> +bool xroot_wait_for_bringup(void *root);
>>> +int xroot_add_vsec_node(void *root, char *dtb);
>>> +int xroot_create_group(void *xr, char *dtb);
>>> +int xroot_add_simple_node(void *root, char *dtb, const char *endpoint);
>>> +void xroot_broadcast(void *root, enum xrt_events evt);
>>> +
>>> +#endif /* _XRT_ROOT_H_ */
>>> diff --git a/drivers/fpga/xrt/mgmt/root.c b/drivers/fpga/xrt/mgmt/root.c
>>> new file mode 100644
>>> index 000000000000..583a37c9d30c
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/mgmt/root.c
>>> @@ -0,0 +1,342 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * Xilinx Alveo Management Function Driver
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors:
>>> + * Cheng Zhen <maxz@xilinx.com>
>>> + */
>>> +
>>> +#include <linux/module.h>
>>> +#include <linux/pci.h>
>>> +#include <linux/aer.h>
>>> +#include <linux/vmalloc.h>
>>> +#include <linux/delay.h>
>>> +
>>> +#include "xroot.h"
>>> +#include "main-impl.h"
>>> +#include "metadata.h"
>>> +
>>> +#define XMGMT_MODULE_NAME "xmgmt"
>> The xrt modules would be more easily identified with a 'xrt' prefix instead of 'x'
>
>
> We will change the module name to xrt-mgmt.
>
>
>>> +#define XMGMT_DRIVER_VERSION "4.0.0"
>>> +
>>> +#define XMGMT_PDEV(xm) ((xm)->pdev)
>>> +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
>>> +#define xmgmt_err(xm, fmt, args...) \
>>> + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>>> +#define xmgmt_warn(xm, fmt, args...) \
>>> + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>>> +#define xmgmt_info(xm, fmt, args...) \
>>> + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>>> +#define xmgmt_dbg(xm, fmt, args...) \
>>> + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>>> +#define XMGMT_DEV_ID(_pcidev) \
>>> + ({ typeof(_pcidev) (pcidev) = (_pcidev); \
>>> + ((pci_domain_nr((pcidev)->bus) << 16) | \
>>> + PCI_DEVID((pcidev)->bus->number, 0)); })
>>> +
>>> +static struct class *xmgmt_class;
>>> +static const struct pci_device_id xmgmt_pci_ids[] = {
>>> + { PCI_DEVICE(0x10EE, 0xd020), }, /* Alveo U50 (golden image) */
>>> + { PCI_DEVICE(0x10EE, 0x5020), }, /* Alveo U50 */
>> demagic this table, look at dfl-pci for how to use existing #define for the vendor and create a new on for the device. If there are vf's add them at the same time.
>>
>> What is a golden image ?
>
>
> Fixed. Please see my comments above for golden image.
>
>
>>
>>> + { 0, }
>>> +};
>>> +
>>> +struct xmgmt {
>>> + struct pci_dev *pdev;
>>> + void *root;
>>> +
>>> + bool ready;
>>> +};
>>> +
>>> +static int xmgmt_config_pci(struct xmgmt *xm)
>>> +{
>>> + struct pci_dev *pdev = XMGMT_PDEV(xm);
>>> + int rc;
>>> +
>>> + rc = pcim_enable_device(pdev);
>>> + if (rc < 0) {
>>> + xmgmt_err(xm, "failed to enable device: %d", rc);
>>> + return rc;
>>> + }
>>> +
>>> + rc = pci_enable_pcie_error_reporting(pdev);
>>> + if (rc)
>>> + xmgmt_warn(xm, "failed to enable AER: %d", rc);
>>> +
>>> + pci_set_master(pdev);
>>> +
>>> + rc = pcie_get_readrq(pdev);
>> Review this call, it does not go negative
>
>
> I'll remove the check against negative value.
>
>
>>> + if (rc < 0) {
>>> + xmgmt_err(xm, "failed to read mrrs %d", rc);
>>> + return rc;
>>> + }
>> this is a quirk, add a comment.
>
>
> Will remove.
>
>
>>> + if (rc > 512) {
>>> + rc = pcie_set_readrq(pdev, 512);
>>> + if (rc) {
>>> + xmgmt_err(xm, "failed to force mrrs %d", rc);
>> similar calls do not fail here.
>
>
> Will remove.
>
>
>>> + return rc;
>>> + }
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static int xmgmt_match_slot_and_save(struct device *dev, void *data)
>>> +{
>>> + struct xmgmt *xm = data;
>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>> +
>>> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
>>> + pci_cfg_access_lock(pdev);
>>> + pci_save_state(pdev);
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static void xmgmt_pci_save_config_all(struct xmgmt *xm)
>>> +{
>>> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save);
>> This is a bus call, not a device call.
>>
>> Can this be changed into something like what hot reset does ?
>
>
> We are working on both mgmt pf and user pf here, so sort of like a bus. But, it might be better to refactor this when we have our own bus type implementation. We do not need to make PCIE bus call. We will fix this in V5 patch set where we'll implement our own bus type.
>
>
ok
>>
>>> +}
>>> +
>>> +static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
>>> +{
>>> + struct xmgmt *xm = data;
>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>> +
>>> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
>>> + pci_restore_state(pdev);
>>> + pci_cfg_access_unlock(pdev);
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
>>> +{
>>> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore);
>>> +}
>>> +
>>> +static void xmgmt_root_hot_reset(struct pci_dev *pdev)
>>> +{
>>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>>> + struct pci_bus *bus;
>>> + u8 pci_bctl;
>>> + u16 pci_cmd, devctl;
>>> + int i, ret;
>>> +
>>> + xmgmt_info(xm, "hot reset start");
>>> +
>>> + xmgmt_pci_save_config_all(xm);
>>> +
>>> + pci_disable_device(pdev);
>>> +
>>> + bus = pdev->bus;
>>> +
>>> + /*
>>> + * When flipping the SBR bit, device can fall off the bus. This is
>>> + * usually no problem at all so long as drivers are working properly
>>> + * after SBR. However, some systems complain bitterly when the device
>>> + * falls off the bus.
>>> + * The quick solution is to temporarily disable the SERR reporting of
>>> + * switch port during SBR.
>>> + */
>>> +
>>> + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
>>> + pci_write_config_word(bus->self, PCI_COMMAND,
>>> + (pci_cmd & ~PCI_COMMAND_SERR));
>>> + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
>>> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
>>> + (devctl & ~PCI_EXP_DEVCTL_FERE));
>>> + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
>>> + pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
>>> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
>> how the pci config values are set and cleared should be consistent.
>>
>> this call should be
>>
>> pci_write_config_byte (... pci_bctl | PCI_BRIDGE_CTL_BUF_RESET )
>>
>> and the next &= avoided
>
>
> Sure.
>
>
>>
>>> +
>>> + msleep(100);
>>> + pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
>>> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
>>> + ssleep(1);
>>> +
>>> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
>>> + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
>>> +
>>> + ret = pci_enable_device(pdev);
>>> + if (ret)
>>> + xmgmt_err(xm, "failed to enable device, ret %d", ret);
>>> +
>>> + for (i = 0; i < 300; i++) {
>>> + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
>>> + if (pci_cmd != 0xffff)
>> what happens with i == 300 and pci_cmd is still 0xffff ?
>
>
> Something wrong happens to the device since it's not coming back after the reset. In this case, the device cannot be used and the only way to recover is to power cycle the system so that the shell can be reloaded from the flash on the device.
>
>
so check and add a dev_crit() to let the user know.
>>> + break;
>>> + msleep(20);
>>> + }
>>> +
>>> + xmgmt_info(xm, "waiting for %d ms", i * 20);
>>> + xmgmt_pci_restore_config_all(xm);
>>> + xmgmt_config_pci(xm);
>>> +}
>>> +
>>> +static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb)
>>> +{
>>> + char *dtb = NULL;
>>> + int ret;
>>> +
>>> + ret = xrt_md_create(XMGMT_DEV(xm), &dtb);
>>> + if (ret) {
>>> + xmgmt_err(xm, "create metadata failed, ret %d", ret);
>>> + goto failed;
>>> + }
>>> +
>>> + ret = xroot_add_vsec_node(xm->root, dtb);
>>> + if (ret == -ENOENT) {
>>> + /*
>>> + * We may be dealing with a MFG board.
>>> + * Try vsec-golden which will bring up all hard-coded leaves
>>> + * at hard-coded offsets.
>>> + */
>>> + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_VSEC_GOLDEN);
>> Manufacturing diagnostics ?
>
>
> This is for golden image support. Please see my comments above.
Ok, i got it :)
Thanks, looking forward next rev
Tom
>
>
> Thanks,
>
> Max
>
>>
>> Tom
>>
>>> + } else if (ret == 0) {
>>> + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_MGMT_MAIN);
>>> + }
>>> + if (ret)
>>> + goto failed;
>>> +
>>> + *root_dtb = dtb;
>>> + return 0;
>>> +
>>> +failed:
>>> + vfree(dtb);
>>> + return ret;
>>> +}
>>> +
>>> +static ssize_t ready_show(struct device *dev,
>>> + struct device_attribute *da,
>>> + char *buf)
>>> +{
>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>>> +
>>> + return sprintf(buf, "%d\n", xm->ready);
>>> +}
>>> +static DEVICE_ATTR_RO(ready);
>>> +
>>> +static struct attribute *xmgmt_root_attrs[] = {
>>> + &dev_attr_ready.attr,
>>> + NULL
>>> +};
>>> +
>>> +static struct attribute_group xmgmt_root_attr_group = {
>>> + .attrs = xmgmt_root_attrs,
>>> +};
>>> +
>>> +static struct xroot_pf_cb xmgmt_xroot_pf_cb = {
>>> + .xpc_hot_reset = xmgmt_root_hot_reset,
>>> +};
>>> +
>>> +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>>> +{
>>> + int ret;
>>> + struct device *dev = &pdev->dev;
>>> + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
>>> + char *dtb = NULL;
>>> +
>>> + if (!xm)
>>> + return -ENOMEM;
>>> + xm->pdev = pdev;
>>> + pci_set_drvdata(pdev, xm);
>>> +
>>> + ret = xmgmt_config_pci(xm);
>>> + if (ret)
>>> + goto failed;
>>> +
>>> + ret = xroot_probe(pdev, &xmgmt_xroot_pf_cb, &xm->root);
>>> + if (ret)
>>> + goto failed;
>>> +
>>> + ret = xmgmt_create_root_metadata(xm, &dtb);
>>> + if (ret)
>>> + goto failed_metadata;
>>> +
>>> + ret = xroot_create_group(xm->root, dtb);
>>> + vfree(dtb);
>>> + if (ret)
>>> + xmgmt_err(xm, "failed to create root group: %d", ret);
>>> +
>>> + if (!xroot_wait_for_bringup(xm->root))
>>> + xmgmt_err(xm, "failed to bringup all groups");
>>> + else
>>> + xm->ready = true;
>>> +
>>> + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
>>> + if (ret) {
>>> + /* Warning instead of failing the probe. */
>>> + xmgmt_warn(xm, "create xmgmt root attrs failed: %d", ret);
>>> + }
>>> +
>>> + xroot_broadcast(xm->root, XRT_EVENT_POST_CREATION);
>>> + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
>>> + return 0;
>>> +
>>> +failed_metadata:
>>> + (void)xroot_remove(xm->root);
>>> +failed:
>>> + pci_set_drvdata(pdev, NULL);
>>> + return ret;
>>> +}
>>> +
>>> +static void xmgmt_remove(struct pci_dev *pdev)
>>> +{
>>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>>> +
>>> + xroot_broadcast(xm->root, XRT_EVENT_PRE_REMOVAL);
>>> + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
>>> + (void)xroot_remove(xm->root);
>>> + pci_disable_pcie_error_reporting(xm->pdev);
>>> + xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
>>> +}
>>> +
>>> +static struct pci_driver xmgmt_driver = {
>>> + .name = XMGMT_MODULE_NAME,
>>> + .id_table = xmgmt_pci_ids,
>>> + .probe = xmgmt_probe,
>>> + .remove = xmgmt_remove,
>>> +};
>>> +
>>> +static int __init xmgmt_init(void)
>>> +{
>>> + int res = 0;
>>> +
>>> + res = xmgmt_main_register_leaf();
>>> + if (res)
>>> + return res;
>>> +
>>> + xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME);
>>> + if (IS_ERR(xmgmt_class))
>>> + return PTR_ERR(xmgmt_class);
>>> +
>>> + res = pci_register_driver(&xmgmt_driver);
>>> + if (res) {
>>> + class_destroy(xmgmt_class);
>>> + return res;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static __exit void xmgmt_exit(void)
>>> +{
>>> + pci_unregister_driver(&xmgmt_driver);
>>> + class_destroy(xmgmt_class);
>>> + xmgmt_main_unregister_leaf();
>>> +}
>>> +
>>> +module_init(xmgmt_init);
>>> +module_exit(xmgmt_exit);
>>> +
>>> +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
>>> +MODULE_VERSION(XMGMT_DRIVER_VERSION);
>>> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
>>> +MODULE_DESCRIPTION("Xilinx Alveo management function driver");
>>> +MODULE_LICENSE("GPL v2");
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root)
2021-03-17 21:08 ` Tom Rix
@ 2021-03-18 0:44 ` Max Zhen
0 siblings, 0 replies; 87+ messages in thread
From: Max Zhen @ 2021-03-18 0:44 UTC (permalink / raw)
To: Tom Rix, Max Zhen, Lizhi Hou, linux-kernel, mdf
Cc: Lizhi Hou, linux-fpga, sonal.santan, michal.simek, stefanos,
devicetree, robh
On 3/17/21 2:08 PM, Tom Rix wrote:
>
> On 3/16/21 1:29 PM, Max Zhen wrote:
>> Hi Tom,
>>
>>
>> On 2/26/21 7:01 AM, Tom Rix wrote:
>>
>>
>>
>> A question i do not know the answer to.
>>
>> Seems like 'golden' is linked to a manufacturing (diagnostics?) image.
>>
>> If the public will never see it, should handling it here be done ?
>>
>> Moritz, do you know ?
>>
>> Golden image is preloaded on the device when it is shipped to customer. Then, customer can load other shells (from Xilinx or some other vendor). If something goes wrong with the shell, customer can always go back to golden and start over again. So, golden image is going to be used in public, not just internally by Xilinx.
>>
>>
> Thanks for the explanation.
>
>
>>>
>>> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>>>> The PCIE device driver which attaches to management function on Alveo
>>> to the management
>>
>> Sure.
>>
>>
>>>> devices. It instantiates one or more partition drivers which in turn
>>> more fpga partition / group ?
>>
>> Group driver.
>>
>>
>>>> instantiate platform drivers. The instantiation of partition and platform
>>>> drivers is completely data driven.
>>> data driven ? everything is data driven. do you mean dtb driven ?
>>
>> Data driven means not hard-coded. Here data means meta data which is presented in device tree format, dtb.
>>
>>
>>>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>>>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>>>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>>>> ---
>>>> drivers/fpga/xrt/include/xroot.h | 114 +++++++++++
>>>> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++++++++++++++++++++++++
>>>> 2 files changed, 456 insertions(+)
>>>> create mode 100644 drivers/fpga/xrt/include/xroot.h
>>>> create mode 100644 drivers/fpga/xrt/mgmt/root.c
>>>>
>>>> diff --git a/drivers/fpga/xrt/include/xroot.h b/drivers/fpga/xrt/include/xroot.h
>>>> new file mode 100644
>>>> index 000000000000..752e10daa85e
>>>> --- /dev/null
>>>> +++ b/drivers/fpga/xrt/include/xroot.h
>>>> @@ -0,0 +1,114 @@
>>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>>> +/*
>>>> + * Header file for Xilinx Runtime (XRT) driver
>>>> + *
>>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>>> + *
>>>> + * Authors:
>>>> + * Cheng Zhen <maxz@xilinx.com>
>>>> + */
>>>> +
>>>> +#ifndef _XRT_ROOT_H_
>>>> +#define _XRT_ROOT_H_
>>>> +
>>>> +#include <linux/pci.h>
>>>> +#include "subdev_id.h"
>>>> +#include "events.h"
>>>> +
>>>> +typedef bool (*xrt_subdev_match_t)(enum xrt_subdev_id,
>>>> + struct platform_device *, void *);
>>>> +#define XRT_SUBDEV_MATCH_PREV ((xrt_subdev_match_t)-1)
>>>> +#define XRT_SUBDEV_MATCH_NEXT ((xrt_subdev_match_t)-2)
>>>> +
>>>> +/*
>>>> + * Root IOCTL calls.
>>>> + */
>>>> +enum xrt_root_ioctl_cmd {
>>>> + /* Leaf actions. */
>>>> + XRT_ROOT_GET_LEAF = 0,
>>>> + XRT_ROOT_PUT_LEAF,
>>>> + XRT_ROOT_GET_LEAF_HOLDERS,
>>>> +
>>>> + /* Group actions. */
>>>> + XRT_ROOT_CREATE_GROUP,
>>>> + XRT_ROOT_REMOVE_GROUP,
>>>> + XRT_ROOT_LOOKUP_GROUP,
>>>> + XRT_ROOT_WAIT_GROUP_BRINGUP,
>>>> +
>>>> + /* Event actions. */
>>>> + XRT_ROOT_EVENT,
>>> should this be XRT_ROOT_EVENT_SYNC ?
>>
>> Sure.
>>
>>
>>>> + XRT_ROOT_EVENT_ASYNC,
>>>> +
>>>> + /* Device info. */
>>>> + XRT_ROOT_GET_RESOURCE,
>>>> + XRT_ROOT_GET_ID,
>>>> +
>>>> + /* Misc. */
>>>> + XRT_ROOT_HOT_RESET,
>>>> + XRT_ROOT_HWMON,
>>>> +};
>>>> +
>>>> +struct xrt_root_ioctl_get_leaf {
>>>> + struct platform_device *xpigl_pdev; /* caller's pdev */
>>> xpigl_ ? unneeded suffix in element names
>>
>> It's needed since the it might be included and used in > 1 .c files. I'd like to keep it's name unique.
> This is an element name, the variable name sound be unique enough to make it clear.
>
> This is not a critical issue, ok as-is.
>
>>
>>>> + xrt_subdev_match_t xpigl_match_cb;
>>>> + void *xpigl_match_arg;
>>>> + struct platform_device *xpigl_leaf; /* target leaf pdev */
>>>> +};
>>>> +
>>>> +struct xrt_root_ioctl_put_leaf {
>>>> + struct platform_device *xpipl_pdev; /* caller's pdev */
>>>> + struct platform_device *xpipl_leaf; /* target's pdev */
>>> caller_pdev;
>>>
>>> target_pdev;
>>
>> Sure.
>>
>>
>>>> +};
>>>> +
>>>> +struct xrt_root_ioctl_lookup_group {
>>>> + struct platform_device *xpilp_pdev; /* caller's pdev */
>>>> + xrt_subdev_match_t xpilp_match_cb;
>>>> + void *xpilp_match_arg;
>>>> + int xpilp_grp_inst;
>>>> +};
>>>> +
>>>> +struct xrt_root_ioctl_get_holders {
>>>> + struct platform_device *xpigh_pdev; /* caller's pdev */
>>>> + char *xpigh_holder_buf;
>>>> + size_t xpigh_holder_buf_len;
>>>> +};
>>>> +
>>>> +struct xrt_root_ioctl_get_res {
>>>> + struct resource *xpigr_res;
>>>> +};
>>>> +
>>>> +struct xrt_root_ioctl_get_id {
>>>> + unsigned short xpigi_vendor_id;
>>>> + unsigned short xpigi_device_id;
>>>> + unsigned short xpigi_sub_vendor_id;
>>>> + unsigned short xpigi_sub_device_id;
>>>> +};
>>>> +
>>>> +struct xrt_root_ioctl_hwmon {
>>>> + bool xpih_register;
>>>> + const char *xpih_name;
>>>> + void *xpih_drvdata;
>>>> + const struct attribute_group **xpih_groups;
>>>> + struct device *xpih_hwmon_dev;
>>>> +};
>>>> +
>>>> +typedef int (*xrt_subdev_root_cb_t)(struct device *, void *, u32, void *);
>>> This function pointer type is important, please add a comment about its use and expected parameters
>>
>> Added.
>>
>>
>>>> +int xrt_subdev_root_request(struct platform_device *self, u32 cmd, void *arg);
>>>> +
>>>> +/*
>>>> + * Defines physical function (MPF / UPF) specific operations
>>>> + * needed in common root driver.
>>>> + */
>>>> +struct xroot_pf_cb {
>>>> + void (*xpc_hot_reset)(struct pci_dev *pdev);
>>> This is only ever set to xmgmt_root_hot_reset, why is this abstraction needed ?
>>
>> As comment says, hot reset is implemented differently in MPF and UPF driver. So, we need this callback in this common code. Note that we have not added UPF code in our initial patch yet. It will be added in the future.
>>
>>
>>>> +};
>>>> +
>>>> +int xroot_probe(struct pci_dev *pdev, struct xroot_pf_cb *cb, void **root);
>>>> +void xroot_remove(void *root);
>>>> +bool xroot_wait_for_bringup(void *root);
>>>> +int xroot_add_vsec_node(void *root, char *dtb);
>>>> +int xroot_create_group(void *xr, char *dtb);
>>>> +int xroot_add_simple_node(void *root, char *dtb, const char *endpoint);
>>>> +void xroot_broadcast(void *root, enum xrt_events evt);
>>>> +
>>>> +#endif /* _XRT_ROOT_H_ */
>>>> diff --git a/drivers/fpga/xrt/mgmt/root.c b/drivers/fpga/xrt/mgmt/root.c
>>>> new file mode 100644
>>>> index 000000000000..583a37c9d30c
>>>> --- /dev/null
>>>> +++ b/drivers/fpga/xrt/mgmt/root.c
>>>> @@ -0,0 +1,342 @@
>>>> +// SPDX-License-Identifier: GPL-2.0
>>>> +/*
>>>> + * Xilinx Alveo Management Function Driver
>>>> + *
>>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>>> + *
>>>> + * Authors:
>>>> + * Cheng Zhen <maxz@xilinx.com>
>>>> + */
>>>> +
>>>> +#include <linux/module.h>
>>>> +#include <linux/pci.h>
>>>> +#include <linux/aer.h>
>>>> +#include <linux/vmalloc.h>
>>>> +#include <linux/delay.h>
>>>> +
>>>> +#include "xroot.h"
>>>> +#include "main-impl.h"
>>>> +#include "metadata.h"
>>>> +
>>>> +#define XMGMT_MODULE_NAME "xmgmt"
>>> The xrt modules would be more easily identified with a 'xrt' prefix instead of 'x'
>>
>> We will change the module name to xrt-mgmt.
>>
>>
>>>> +#define XMGMT_DRIVER_VERSION "4.0.0"
>>>> +
>>>> +#define XMGMT_PDEV(xm) ((xm)->pdev)
>>>> +#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
>>>> +#define xmgmt_err(xm, fmt, args...) \
>>>> + dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>>>> +#define xmgmt_warn(xm, fmt, args...) \
>>>> + dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>>>> +#define xmgmt_info(xm, fmt, args...) \
>>>> + dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>>>> +#define xmgmt_dbg(xm, fmt, args...) \
>>>> + dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
>>>> +#define XMGMT_DEV_ID(_pcidev) \
>>>> + ({ typeof(_pcidev) (pcidev) = (_pcidev); \
>>>> + ((pci_domain_nr((pcidev)->bus) << 16) | \
>>>> + PCI_DEVID((pcidev)->bus->number, 0)); })
>>>> +
>>>> +static struct class *xmgmt_class;
>>>> +static const struct pci_device_id xmgmt_pci_ids[] = {
>>>> + { PCI_DEVICE(0x10EE, 0xd020), }, /* Alveo U50 (golden image) */
>>>> + { PCI_DEVICE(0x10EE, 0x5020), }, /* Alveo U50 */
>>> demagic this table, look at dfl-pci for how to use existing #define for the vendor and create a new on for the device. If there are vf's add them at the same time.
>>>
>>> What is a golden image ?
>>
>> Fixed. Please see my comments above for golden image.
>>
>>
>>>> + { 0, }
>>>> +};
>>>> +
>>>> +struct xmgmt {
>>>> + struct pci_dev *pdev;
>>>> + void *root;
>>>> +
>>>> + bool ready;
>>>> +};
>>>> +
>>>> +static int xmgmt_config_pci(struct xmgmt *xm)
>>>> +{
>>>> + struct pci_dev *pdev = XMGMT_PDEV(xm);
>>>> + int rc;
>>>> +
>>>> + rc = pcim_enable_device(pdev);
>>>> + if (rc < 0) {
>>>> + xmgmt_err(xm, "failed to enable device: %d", rc);
>>>> + return rc;
>>>> + }
>>>> +
>>>> + rc = pci_enable_pcie_error_reporting(pdev);
>>>> + if (rc)
>>>> + xmgmt_warn(xm, "failed to enable AER: %d", rc);
>>>> +
>>>> + pci_set_master(pdev);
>>>> +
>>>> + rc = pcie_get_readrq(pdev);
>>> Review this call, it does not go negative
>>
>> I'll remove the check against negative value.
>>
>>
>>>> + if (rc < 0) {
>>>> + xmgmt_err(xm, "failed to read mrrs %d", rc);
>>>> + return rc;
>>>> + }
>>> this is a quirk, add a comment.
>>
>> Will remove.
>>
>>
>>>> + if (rc > 512) {
>>>> + rc = pcie_set_readrq(pdev, 512);
>>>> + if (rc) {
>>>> + xmgmt_err(xm, "failed to force mrrs %d", rc);
>>> similar calls do not fail here.
>>
>> Will remove.
>>
>>
>>>> + return rc;
>>>> + }
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static int xmgmt_match_slot_and_save(struct device *dev, void *data)
>>>> +{
>>>> + struct xmgmt *xm = data;
>>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>>> +
>>>> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
>>>> + pci_cfg_access_lock(pdev);
>>>> + pci_save_state(pdev);
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static void xmgmt_pci_save_config_all(struct xmgmt *xm)
>>>> +{
>>>> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_save);
>>> This is a bus call, not a device call.
>>>
>>> Can this be changed into something like what hot reset does ?
>>
>> We are working on both mgmt pf and user pf here, so sort of like a bus. But, it might be better to refactor this when we have our own bus type implementation. We do not need to make PCIE bus call. We will fix this in V5 patch set where we'll implement our own bus type.
>>
>>
> ok
>>>> +}
>>>> +
>>>> +static int xmgmt_match_slot_and_restore(struct device *dev, void *data)
>>>> +{
>>>> + struct xmgmt *xm = data;
>>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>>> +
>>>> + if (XMGMT_DEV_ID(pdev) == XMGMT_DEV_ID(xm->pdev)) {
>>>> + pci_restore_state(pdev);
>>>> + pci_cfg_access_unlock(pdev);
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static void xmgmt_pci_restore_config_all(struct xmgmt *xm)
>>>> +{
>>>> + bus_for_each_dev(&pci_bus_type, NULL, xm, xmgmt_match_slot_and_restore);
>>>> +}
>>>> +
>>>> +static void xmgmt_root_hot_reset(struct pci_dev *pdev)
>>>> +{
>>>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>>>> + struct pci_bus *bus;
>>>> + u8 pci_bctl;
>>>> + u16 pci_cmd, devctl;
>>>> + int i, ret;
>>>> +
>>>> + xmgmt_info(xm, "hot reset start");
>>>> +
>>>> + xmgmt_pci_save_config_all(xm);
>>>> +
>>>> + pci_disable_device(pdev);
>>>> +
>>>> + bus = pdev->bus;
>>>> +
>>>> + /*
>>>> + * When flipping the SBR bit, device can fall off the bus. This is
>>>> + * usually no problem at all so long as drivers are working properly
>>>> + * after SBR. However, some systems complain bitterly when the device
>>>> + * falls off the bus.
>>>> + * The quick solution is to temporarily disable the SERR reporting of
>>>> + * switch port during SBR.
>>>> + */
>>>> +
>>>> + pci_read_config_word(bus->self, PCI_COMMAND, &pci_cmd);
>>>> + pci_write_config_word(bus->self, PCI_COMMAND,
>>>> + (pci_cmd & ~PCI_COMMAND_SERR));
>>>> + pcie_capability_read_word(bus->self, PCI_EXP_DEVCTL, &devctl);
>>>> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL,
>>>> + (devctl & ~PCI_EXP_DEVCTL_FERE));
>>>> + pci_read_config_byte(bus->self, PCI_BRIDGE_CONTROL, &pci_bctl);
>>>> + pci_bctl |= PCI_BRIDGE_CTL_BUS_RESET;
>>>> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
>>> how the pci config values are set and cleared should be consistent.
>>>
>>> this call should be
>>>
>>> pci_write_config_byte (... pci_bctl | PCI_BRIDGE_CTL_BUF_RESET )
>>>
>>> and the next &= avoided
>>
>> Sure.
>>
>>
>>>> +
>>>> + msleep(100);
>>>> + pci_bctl &= ~PCI_BRIDGE_CTL_BUS_RESET;
>>>> + pci_write_config_byte(bus->self, PCI_BRIDGE_CONTROL, pci_bctl);
>>>> + ssleep(1);
>>>> +
>>>> + pcie_capability_write_word(bus->self, PCI_EXP_DEVCTL, devctl);
>>>> + pci_write_config_word(bus->self, PCI_COMMAND, pci_cmd);
>>>> +
>>>> + ret = pci_enable_device(pdev);
>>>> + if (ret)
>>>> + xmgmt_err(xm, "failed to enable device, ret %d", ret);
>>>> +
>>>> + for (i = 0; i < 300; i++) {
>>>> + pci_read_config_word(pdev, PCI_COMMAND, &pci_cmd);
>>>> + if (pci_cmd != 0xffff)
>>> what happens with i == 300 and pci_cmd is still 0xffff ?
>>
>> Something wrong happens to the device since it's not coming back after the reset. In this case, the device cannot be used and the only way to recover is to power cycle the system so that the shell can be reloaded from the flash on the device.
>>
>>
> so check and add a dev_crit() to let the user know.
Sure. Will do.
Thanks,
Max
>>>> + break;
>>>> + msleep(20);
>>>> + }
>>>> +
>>>> + xmgmt_info(xm, "waiting for %d ms", i * 20);
>>>> + xmgmt_pci_restore_config_all(xm);
>>>> + xmgmt_config_pci(xm);
>>>> +}
>>>> +
>>>> +static int xmgmt_create_root_metadata(struct xmgmt *xm, char **root_dtb)
>>>> +{
>>>> + char *dtb = NULL;
>>>> + int ret;
>>>> +
>>>> + ret = xrt_md_create(XMGMT_DEV(xm), &dtb);
>>>> + if (ret) {
>>>> + xmgmt_err(xm, "create metadata failed, ret %d", ret);
>>>> + goto failed;
>>>> + }
>>>> +
>>>> + ret = xroot_add_vsec_node(xm->root, dtb);
>>>> + if (ret == -ENOENT) {
>>>> + /*
>>>> + * We may be dealing with a MFG board.
>>>> + * Try vsec-golden which will bring up all hard-coded leaves
>>>> + * at hard-coded offsets.
>>>> + */
>>>> + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_VSEC_GOLDEN);
>>> Manufacturing diagnostics ?
>>
>> This is for golden image support. Please see my comments above.
> Ok, i got it :)
>
> Thanks, looking forward next rev
>
> Tom
>
>>
>> Thanks,
>>
>> Max
>>
>>> Tom
>>>
>>>> + } else if (ret == 0) {
>>>> + ret = xroot_add_simple_node(xm->root, dtb, XRT_MD_NODE_MGMT_MAIN);
>>>> + }
>>>> + if (ret)
>>>> + goto failed;
>>>> +
>>>> + *root_dtb = dtb;
>>>> + return 0;
>>>> +
>>>> +failed:
>>>> + vfree(dtb);
>>>> + return ret;
>>>> +}
>>>> +
>>>> +static ssize_t ready_show(struct device *dev,
>>>> + struct device_attribute *da,
>>>> + char *buf)
>>>> +{
>>>> + struct pci_dev *pdev = to_pci_dev(dev);
>>>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>>>> +
>>>> + return sprintf(buf, "%d\n", xm->ready);
>>>> +}
>>>> +static DEVICE_ATTR_RO(ready);
>>>> +
>>>> +static struct attribute *xmgmt_root_attrs[] = {
>>>> + &dev_attr_ready.attr,
>>>> + NULL
>>>> +};
>>>> +
>>>> +static struct attribute_group xmgmt_root_attr_group = {
>>>> + .attrs = xmgmt_root_attrs,
>>>> +};
>>>> +
>>>> +static struct xroot_pf_cb xmgmt_xroot_pf_cb = {
>>>> + .xpc_hot_reset = xmgmt_root_hot_reset,
>>>> +};
>>>> +
>>>> +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>>>> +{
>>>> + int ret;
>>>> + struct device *dev = &pdev->dev;
>>>> + struct xmgmt *xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
>>>> + char *dtb = NULL;
>>>> +
>>>> + if (!xm)
>>>> + return -ENOMEM;
>>>> + xm->pdev = pdev;
>>>> + pci_set_drvdata(pdev, xm);
>>>> +
>>>> + ret = xmgmt_config_pci(xm);
>>>> + if (ret)
>>>> + goto failed;
>>>> +
>>>> + ret = xroot_probe(pdev, &xmgmt_xroot_pf_cb, &xm->root);
>>>> + if (ret)
>>>> + goto failed;
>>>> +
>>>> + ret = xmgmt_create_root_metadata(xm, &dtb);
>>>> + if (ret)
>>>> + goto failed_metadata;
>>>> +
>>>> + ret = xroot_create_group(xm->root, dtb);
>>>> + vfree(dtb);
>>>> + if (ret)
>>>> + xmgmt_err(xm, "failed to create root group: %d", ret);
>>>> +
>>>> + if (!xroot_wait_for_bringup(xm->root))
>>>> + xmgmt_err(xm, "failed to bringup all groups");
>>>> + else
>>>> + xm->ready = true;
>>>> +
>>>> + ret = sysfs_create_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
>>>> + if (ret) {
>>>> + /* Warning instead of failing the probe. */
>>>> + xmgmt_warn(xm, "create xmgmt root attrs failed: %d", ret);
>>>> + }
>>>> +
>>>> + xroot_broadcast(xm->root, XRT_EVENT_POST_CREATION);
>>>> + xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
>>>> + return 0;
>>>> +
>>>> +failed_metadata:
>>>> + (void)xroot_remove(xm->root);
>>>> +failed:
>>>> + pci_set_drvdata(pdev, NULL);
>>>> + return ret;
>>>> +}
>>>> +
>>>> +static void xmgmt_remove(struct pci_dev *pdev)
>>>> +{
>>>> + struct xmgmt *xm = pci_get_drvdata(pdev);
>>>> +
>>>> + xroot_broadcast(xm->root, XRT_EVENT_PRE_REMOVAL);
>>>> + sysfs_remove_group(&pdev->dev.kobj, &xmgmt_root_attr_group);
>>>> + (void)xroot_remove(xm->root);
>>>> + pci_disable_pcie_error_reporting(xm->pdev);
>>>> + xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
>>>> +}
>>>> +
>>>> +static struct pci_driver xmgmt_driver = {
>>>> + .name = XMGMT_MODULE_NAME,
>>>> + .id_table = xmgmt_pci_ids,
>>>> + .probe = xmgmt_probe,
>>>> + .remove = xmgmt_remove,
>>>> +};
>>>> +
>>>> +static int __init xmgmt_init(void)
>>>> +{
>>>> + int res = 0;
>>>> +
>>>> + res = xmgmt_main_register_leaf();
>>>> + if (res)
>>>> + return res;
>>>> +
>>>> + xmgmt_class = class_create(THIS_MODULE, XMGMT_MODULE_NAME);
>>>> + if (IS_ERR(xmgmt_class))
>>>> + return PTR_ERR(xmgmt_class);
>>>> +
>>>> + res = pci_register_driver(&xmgmt_driver);
>>>> + if (res) {
>>>> + class_destroy(xmgmt_class);
>>>> + return res;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +static __exit void xmgmt_exit(void)
>>>> +{
>>>> + pci_unregister_driver(&xmgmt_driver);
>>>> + class_destroy(xmgmt_class);
>>>> + xmgmt_main_unregister_leaf();
>>>> +}
>>>> +
>>>> +module_init(xmgmt_init);
>>>> +module_exit(xmgmt_exit);
>>>> +
>>>> +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
>>>> +MODULE_VERSION(XMGMT_DRIVER_VERSION);
>>>> +MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
>>>> +MODULE_DESCRIPTION("Xilinx Alveo management function driver");
>>>> +MODULE_LICENSE("GPL v2");
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 08/18] fpga: xrt: main platform driver for management function device
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (6 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 07/18] fpga: xrt: management physical function driver (root) Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-26 17:22 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 09/18] fpga: xrt: fpga-mgr and region implementation for xclbin download Lizhi Hou
` (11 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
platform driver that handles IOCTLs, such as hot reset and xclbin download.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xmgmt-main.h | 37 ++
drivers/fpga/xrt/mgmt/main-impl.h | 37 ++
drivers/fpga/xrt/mgmt/main.c | 693 ++++++++++++++++++++++++++
include/uapi/linux/xrt/xmgmt-ioctl.h | 46 ++
4 files changed, 813 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
create mode 100644 drivers/fpga/xrt/mgmt/main.c
create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
diff --git a/drivers/fpga/xrt/include/xmgmt-main.h b/drivers/fpga/xrt/include/xmgmt-main.h
new file mode 100644
index 000000000000..1216d1881f8e
--- /dev/null
+++ b/drivers/fpga/xrt/include/xmgmt-main.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Runtime (XRT) driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XMGMT_MAIN_H_
+#define _XMGMT_MAIN_H_
+
+#include <linux/xrt/xclbin.h>
+#include "xleaf.h"
+
+enum xrt_mgmt_main_ioctl_cmd {
+ /* section needs to be vfree'd by caller */
+ XRT_MGMT_MAIN_GET_AXLF_SECTION = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+ /* vbnv needs to be kfree'd by caller */
+ XRT_MGMT_MAIN_GET_VBNV,
+};
+
+enum provider_kind {
+ XMGMT_BLP,
+ XMGMT_PLP,
+ XMGMT_ULP,
+};
+
+struct xrt_mgmt_main_ioctl_get_axlf_section {
+ enum provider_kind xmmigas_axlf_kind;
+ enum axlf_section_kind xmmigas_section_kind;
+ void *xmmigas_section;
+ u64 xmmigas_section_size;
+};
+
+#endif /* _XMGMT_MAIN_H_ */
diff --git a/drivers/fpga/xrt/mgmt/main-impl.h b/drivers/fpga/xrt/mgmt/main-impl.h
new file mode 100644
index 000000000000..dd1b3e3773cc
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/main-impl.h
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XMGMT_MAIN_IMPL_H_
+#define _XMGMT_MAIN_IMPL_H_
+
+#include <linux/platform_device.h>
+#include "xmgmt-main.h"
+
+struct fpga_manager;
+int xmgmt_process_xclbin(struct platform_device *pdev,
+ struct fpga_manager *fmgr,
+ const struct axlf *xclbin,
+ enum provider_kind kind);
+void xmgmt_region_cleanup_all(struct platform_device *pdev);
+
+int bitstream_axlf_mailbox(struct platform_device *pdev, const void *xclbin);
+int xmgmt_hot_reset(struct platform_device *pdev);
+
+/* Getting dtb for specified group. Caller should vfree returned dtb .*/
+char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind);
+char *xmgmt_get_vbnv(struct platform_device *pdev);
+int xmgmt_get_provider_uuid(struct platform_device *pdev,
+ enum provider_kind kind, uuid_t *uuid);
+
+int xmgmt_main_register_leaf(void);
+void xmgmt_main_unregister_leaf(void);
+
+#endif /* _XMGMT_MAIN_IMPL_H_ */
diff --git a/drivers/fpga/xrt/mgmt/main.c b/drivers/fpga/xrt/mgmt/main.c
new file mode 100644
index 000000000000..66ffb4e7029d
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/main.c
@@ -0,0 +1,693 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA MGMT PF entry point driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Sonal Santan <sonals@xilinx.com>
+ */
+
+#include <linux/firmware.h>
+#include <linux/uaccess.h>
+#include "xclbin-helper.h"
+#include "metadata.h"
+#include "xleaf.h"
+#include <linux/xrt/xmgmt-ioctl.h>
+#include "xleaf/devctl.h"
+#include "xmgmt-main.h"
+#include "fmgr.h"
+#include "xleaf/icap.h"
+#include "xleaf/axigate.h"
+#include "main-impl.h"
+
+#define XMGMT_MAIN "xmgmt_main"
+
+struct xmgmt_main {
+ struct platform_device *pdev;
+ struct axlf *firmware_blp;
+ struct axlf *firmware_plp;
+ struct axlf *firmware_ulp;
+ bool flash_ready;
+ bool devctl_ready;
+ struct fpga_manager *fmgr;
+ struct mutex busy_mutex; /* busy lock */
+
+ uuid_t *blp_intf_uuids;
+ u32 blp_intf_uuid_num;
+};
+
+/* Caller should be responsible for freeing the returned string. */
+char *xmgmt_get_vbnv(struct platform_device *pdev)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ const char *vbnv;
+ char *ret;
+ int i;
+
+ if (xmm->firmware_plp)
+ vbnv = xmm->firmware_plp->m_header.m_platformVBNV;
+ else if (xmm->firmware_blp)
+ vbnv = xmm->firmware_blp->m_header.m_platformVBNV;
+ else
+ return NULL;
+
+ ret = kstrdup(vbnv, GFP_KERNEL);
+ if (!ret)
+ return NULL;
+
+ for (i = 0; i < strlen(ret); i++) {
+ if (ret[i] == ':' || ret[i] == '.')
+ ret[i] = '_';
+ }
+ return ret;
+}
+
+static int get_dev_uuid(struct platform_device *pdev, char *uuidstr, size_t len)
+{
+ char uuid[16];
+ struct platform_device *devctl_leaf;
+ struct xrt_devctl_ioctl_rw devctl_arg = { 0 };
+ int err, i, count;
+
+ devctl_leaf = xleaf_get_leaf_by_epname(pdev, XRT_MD_NODE_BLP_ROM);
+ if (!devctl_leaf) {
+ xrt_err(pdev, "can not get %s", XRT_MD_NODE_BLP_ROM);
+ return -EINVAL;
+ }
+
+ devctl_arg.xgir_id = XRT_DEVCTL_ROM_UUID;
+ devctl_arg.xgir_buf = uuid;
+ devctl_arg.xgir_len = sizeof(uuid);
+ devctl_arg.xgir_offset = 0;
+ err = xleaf_ioctl(devctl_leaf, XRT_DEVCTL_READ, &devctl_arg);
+ xleaf_put_leaf(pdev, devctl_leaf);
+ if (err) {
+ xrt_err(pdev, "can not get uuid: %d", err);
+ return err;
+ }
+
+ for (count = 0, i = sizeof(uuid) - sizeof(u32);
+ i >= 0 && len > count; i -= sizeof(u32)) {
+ count += snprintf(uuidstr + count, len - count, "%08x", *(u32 *)&uuid[i]);
+ }
+ return 0;
+}
+
+int xmgmt_hot_reset(struct platform_device *pdev)
+{
+ int ret = xleaf_broadcast_event(pdev, XRT_EVENT_PRE_HOT_RESET, false);
+
+ if (ret) {
+ xrt_err(pdev, "offline failed, hot reset is canceled");
+ return ret;
+ }
+
+ xleaf_hot_reset(pdev);
+ xleaf_broadcast_event(pdev, XRT_EVENT_POST_HOT_RESET, false);
+ return 0;
+}
+
+static ssize_t reset_store(struct device *dev, struct device_attribute *da,
+ const char *buf, size_t count)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+
+ xmgmt_hot_reset(pdev);
+ return count;
+}
+static DEVICE_ATTR_WO(reset);
+
+static ssize_t VBNV_show(struct device *dev, struct device_attribute *da, char *buf)
+{
+ ssize_t ret;
+ char *vbnv;
+ struct platform_device *pdev = to_platform_device(dev);
+
+ vbnv = xmgmt_get_vbnv(pdev);
+ ret = sprintf(buf, "%s\n", vbnv);
+ kfree(vbnv);
+ return ret;
+}
+static DEVICE_ATTR_RO(VBNV);
+
+static ssize_t logic_uuids_show(struct device *dev, struct device_attribute *da, char *buf)
+{
+ ssize_t ret;
+ char uuid[80];
+ struct platform_device *pdev = to_platform_device(dev);
+
+ /* Getting UUID pointed to by VSEC, should be the same as logic UUID of BLP. */
+ ret = get_dev_uuid(pdev, uuid, sizeof(uuid));
+ if (ret)
+ return ret;
+ ret = sprintf(buf, "%s\n", uuid);
+ return ret;
+}
+static DEVICE_ATTR_RO(logic_uuids);
+
+static ssize_t interface_uuids_show(struct device *dev, struct device_attribute *da, char *buf)
+{
+ ssize_t ret = 0;
+ struct platform_device *pdev = to_platform_device(dev);
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ u32 i;
+
+ for (i = 0; i < xmm->blp_intf_uuid_num; i++) {
+ char uuidstr[80];
+
+ xrt_md_trans_uuid2str(&xmm->blp_intf_uuids[i], uuidstr);
+ ret += sprintf(buf + ret, "%s\n", uuidstr);
+ }
+ return ret;
+}
+static DEVICE_ATTR_RO(interface_uuids);
+
+static struct attribute *xmgmt_main_attrs[] = {
+ &dev_attr_reset.attr,
+ &dev_attr_VBNV.attr,
+ &dev_attr_logic_uuids.attr,
+ &dev_attr_interface_uuids.attr,
+ NULL,
+};
+
+/*
+ * sysfs hook to load xclbin primarily used for driver debug
+ */
+static ssize_t ulp_image_write(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *attr, char *buffer, loff_t off, size_t count)
+{
+ struct xmgmt_main *xmm = dev_get_drvdata(container_of(kobj, struct device, kobj));
+ struct axlf *xclbin;
+ ulong len;
+
+ if (off == 0) {
+ if (count < sizeof(*xclbin)) {
+ xrt_err(xmm->pdev, "count is too small %zu", count);
+ return -EINVAL;
+ }
+
+ if (xmm->firmware_ulp) {
+ vfree(xmm->firmware_ulp);
+ xmm->firmware_ulp = NULL;
+ }
+ xclbin = (struct axlf *)buffer;
+ xmm->firmware_ulp = vmalloc(xclbin->m_header.m_length);
+ if (!xmm->firmware_ulp)
+ return -ENOMEM;
+ } else {
+ xclbin = xmm->firmware_ulp;
+ }
+
+ len = xclbin->m_header.m_length;
+ if (off + count >= len && off < len) {
+ memcpy(xmm->firmware_ulp + off, buffer, len - off);
+ xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, xmm->firmware_ulp, XMGMT_ULP);
+ } else if (off + count < len) {
+ memcpy(xmm->firmware_ulp + off, buffer, count);
+ }
+
+ return count;
+}
+
+static struct bin_attribute ulp_image_attr = {
+ .attr = {
+ .name = "ulp_image",
+ .mode = 0200
+ },
+ .write = ulp_image_write,
+ .size = 0
+};
+
+static struct bin_attribute *xmgmt_main_bin_attrs[] = {
+ &ulp_image_attr,
+ NULL,
+};
+
+static const struct attribute_group xmgmt_main_attrgroup = {
+ .attrs = xmgmt_main_attrs,
+ .bin_attrs = xmgmt_main_bin_attrs,
+};
+
+static int load_firmware_from_flash(struct platform_device *pdev, struct axlf **fw_buf, size_t *len)
+{
+ return -EOPNOTSUPP;
+}
+
+static int load_firmware_from_disk(struct platform_device *pdev, struct axlf **fw_buf, size_t *len)
+{
+ char uuid[80];
+ int err = 0;
+ char fw_name[256];
+ const struct firmware *fw;
+
+ err = get_dev_uuid(pdev, uuid, sizeof(uuid));
+ if (err)
+ return err;
+
+ (void)snprintf(fw_name, sizeof(fw_name), "xilinx/%s/partition.xsabin", uuid);
+ xrt_info(pdev, "try loading fw: %s", fw_name);
+
+ err = request_firmware(&fw, fw_name, DEV(pdev));
+ if (err)
+ return err;
+
+ *fw_buf = vmalloc(fw->size);
+ *len = fw->size;
+ if (*fw_buf)
+ memcpy(*fw_buf, fw->data, fw->size);
+ else
+ err = -ENOMEM;
+
+ release_firmware(fw);
+ return 0;
+}
+
+static const struct axlf *xmgmt_get_axlf_firmware(struct xmgmt_main *xmm, enum provider_kind kind)
+{
+ switch (kind) {
+ case XMGMT_BLP:
+ return xmm->firmware_blp;
+ case XMGMT_PLP:
+ return xmm->firmware_plp;
+ case XMGMT_ULP:
+ return xmm->firmware_ulp;
+ default:
+ xrt_err(xmm->pdev, "unknown axlf kind: %d", kind);
+ return NULL;
+ }
+}
+
+char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ char *dtb = NULL;
+ const struct axlf *provider = xmgmt_get_axlf_firmware(xmm, kind);
+ int rc;
+
+ if (!provider)
+ return dtb;
+
+ rc = xrt_xclbin_get_metadata(DEV(pdev), provider, &dtb);
+ if (rc)
+ xrt_err(pdev, "failed to find dtb: %d", rc);
+ return dtb;
+}
+
+static const char *get_uuid_from_firmware(struct platform_device *pdev, const struct axlf *xclbin)
+{
+ const void *uuid = NULL;
+ const void *uuiddup = NULL;
+ void *dtb = NULL;
+ int rc;
+
+ rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA, &dtb, NULL);
+ if (rc)
+ return NULL;
+
+ rc = xrt_md_get_prop(DEV(pdev), dtb, NULL, NULL, XRT_MD_PROP_LOGIC_UUID, &uuid, NULL);
+ if (!rc)
+ uuiddup = kstrdup(uuid, GFP_KERNEL);
+ vfree(dtb);
+ return uuiddup;
+}
+
+static bool is_valid_firmware(struct platform_device *pdev,
+ const struct axlf *xclbin, size_t fw_len)
+{
+ const char *fw_buf = (const char *)xclbin;
+ size_t axlflen = xclbin->m_header.m_length;
+ const char *fw_uuid;
+ char dev_uuid[80];
+ int err;
+
+ err = get_dev_uuid(pdev, dev_uuid, sizeof(dev_uuid));
+ if (err)
+ return false;
+
+ if (memcmp(fw_buf, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)) != 0) {
+ xrt_err(pdev, "unknown fw format");
+ return false;
+ }
+
+ if (axlflen > fw_len) {
+ xrt_err(pdev, "truncated fw, length: %zu, expect: %zu", fw_len, axlflen);
+ return false;
+ }
+
+ fw_uuid = get_uuid_from_firmware(pdev, xclbin);
+ if (!fw_uuid || strcmp(fw_uuid, dev_uuid) != 0) {
+ xrt_err(pdev, "bad fw UUID: %s, expect: %s",
+ fw_uuid ? fw_uuid : "<none>", dev_uuid);
+ kfree(fw_uuid);
+ return false;
+ }
+
+ kfree(fw_uuid);
+ return true;
+}
+
+int xmgmt_get_provider_uuid(struct platform_device *pdev, enum provider_kind kind, uuid_t *uuid)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ const struct axlf *fwbuf;
+ const char *fw_uuid;
+ int rc = -ENOENT;
+
+ mutex_lock(&xmm->busy_mutex);
+
+ fwbuf = xmgmt_get_axlf_firmware(xmm, kind);
+ if (!fwbuf)
+ goto done;
+
+ fw_uuid = get_uuid_from_firmware(pdev, fwbuf);
+ if (!fw_uuid)
+ goto done;
+
+ rc = xrt_md_trans_str2uuid(DEV(pdev), fw_uuid, uuid);
+ kfree(fw_uuid);
+
+done:
+ mutex_unlock(&xmm->busy_mutex);
+ return rc;
+}
+
+static int xmgmt_create_blp(struct xmgmt_main *xmm)
+{
+ struct platform_device *pdev = xmm->pdev;
+ int rc = 0;
+ char *dtb = NULL;
+ const struct axlf *provider = xmgmt_get_axlf_firmware(xmm, XMGMT_BLP);
+
+ dtb = xmgmt_get_dtb(pdev, XMGMT_BLP);
+ if (dtb) {
+ rc = xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, provider, XMGMT_BLP);
+ if (rc) {
+ xrt_err(pdev, "failed to process BLP: %d", rc);
+ goto failed;
+ }
+
+ rc = xleaf_create_group(pdev, dtb);
+ if (rc < 0)
+ xrt_err(pdev, "failed to create BLP group: %d", rc);
+ else
+ rc = 0;
+
+ WARN_ON(xmm->blp_intf_uuids);
+ xrt_md_get_intf_uuids(&pdev->dev, dtb, &xmm->blp_intf_uuid_num, NULL);
+ if (xmm->blp_intf_uuid_num > 0) {
+ xmm->blp_intf_uuids = vzalloc(sizeof(uuid_t) * xmm->blp_intf_uuid_num);
+ xrt_md_get_intf_uuids(&pdev->dev, dtb, &xmm->blp_intf_uuid_num,
+ xmm->blp_intf_uuids);
+ }
+ }
+
+failed:
+ vfree(dtb);
+ return rc;
+}
+
+static int xmgmt_load_firmware(struct xmgmt_main *xmm)
+{
+ struct platform_device *pdev = xmm->pdev;
+ int rc;
+ size_t fwlen;
+
+ rc = load_firmware_from_disk(pdev, &xmm->firmware_blp, &fwlen);
+ if (rc != 0)
+ rc = load_firmware_from_flash(pdev, &xmm->firmware_blp, &fwlen);
+ if (rc == 0 && is_valid_firmware(pdev, xmm->firmware_blp, fwlen))
+ (void)xmgmt_create_blp(xmm);
+ else
+ xrt_err(pdev, "failed to find firmware, giving up: %d", rc);
+ return rc;
+}
+
+static void xmgmt_main_event_cb(struct platform_device *pdev, void *arg)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ struct xrt_event *evt = (struct xrt_event *)arg;
+ enum xrt_events e = evt->xe_evt;
+ enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
+ struct platform_device *leaf;
+
+ switch (e) {
+ case XRT_EVENT_POST_CREATION: {
+ if (id == XRT_SUBDEV_DEVCTL && !xmm->devctl_ready) {
+ leaf = xleaf_get_leaf_by_epname(pdev, XRT_MD_NODE_BLP_ROM);
+ if (leaf) {
+ xmm->devctl_ready = true;
+ xleaf_put_leaf(pdev, leaf);
+ }
+ } else if (id == XRT_SUBDEV_QSPI && !xmm->flash_ready) {
+ xmm->flash_ready = true;
+ } else {
+ break;
+ }
+
+ if (xmm->devctl_ready)
+ (void)xmgmt_load_firmware(xmm);
+ break;
+ }
+ case XRT_EVENT_PRE_REMOVAL:
+ break;
+ default:
+ xrt_dbg(pdev, "ignored event %d", e);
+ break;
+ }
+}
+
+static int xmgmt_main_probe(struct platform_device *pdev)
+{
+ struct xmgmt_main *xmm;
+
+ xrt_info(pdev, "probing...");
+
+ xmm = devm_kzalloc(DEV(pdev), sizeof(*xmm), GFP_KERNEL);
+ if (!xmm)
+ return -ENOMEM;
+
+ xmm->pdev = pdev;
+ xmm->fmgr = xmgmt_fmgr_probe(pdev);
+ if (IS_ERR(xmm->fmgr))
+ return PTR_ERR(xmm->fmgr);
+
+ platform_set_drvdata(pdev, xmm);
+ mutex_init(&xmm->busy_mutex);
+
+ /* Ready to handle req thru sysfs nodes. */
+ if (sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup))
+ xrt_err(pdev, "failed to create sysfs group");
+ return 0;
+}
+
+static int xmgmt_main_remove(struct platform_device *pdev)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+
+ /* By now, group driver should prevent any inter-leaf call. */
+
+ xrt_info(pdev, "leaving...");
+
+ vfree(xmm->blp_intf_uuids);
+ vfree(xmm->firmware_blp);
+ vfree(xmm->firmware_plp);
+ vfree(xmm->firmware_ulp);
+ xmgmt_region_cleanup_all(pdev);
+ (void)xmgmt_fmgr_remove(xmm->fmgr);
+ (void)sysfs_remove_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup);
+ return 0;
+}
+
+static int
+xmgmt_main_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ struct xmgmt_main *xmm = platform_get_drvdata(pdev);
+ int ret = 0;
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ xmgmt_main_event_cb(pdev, arg);
+ break;
+ case XRT_MGMT_MAIN_GET_AXLF_SECTION: {
+ struct xrt_mgmt_main_ioctl_get_axlf_section *get =
+ (struct xrt_mgmt_main_ioctl_get_axlf_section *)arg;
+ const struct axlf *firmware = xmgmt_get_axlf_firmware(xmm, get->xmmigas_axlf_kind);
+
+ if (!firmware) {
+ ret = -ENOENT;
+ } else {
+ ret = xrt_xclbin_get_section(firmware, get->xmmigas_section_kind,
+ &get->xmmigas_section,
+ &get->xmmigas_section_size);
+ }
+ break;
+ }
+ case XRT_MGMT_MAIN_GET_VBNV: {
+ char **vbnv_p = (char **)arg;
+
+ *vbnv_p = xmgmt_get_vbnv(pdev);
+ break;
+ }
+ default:
+ xrt_err(pdev, "unknown cmd: %d", cmd);
+ ret = -EINVAL;
+ break;
+ }
+ return ret;
+}
+
+static int xmgmt_main_open(struct inode *inode, struct file *file)
+{
+ struct platform_device *pdev = xleaf_devnode_open(inode);
+
+ /* Device may have gone already when we get here. */
+ if (!pdev)
+ return -ENODEV;
+
+ xrt_info(pdev, "opened");
+ file->private_data = platform_get_drvdata(pdev);
+ return 0;
+}
+
+static int xmgmt_main_close(struct inode *inode, struct file *file)
+{
+ struct xmgmt_main *xmm = file->private_data;
+
+ xleaf_devnode_close(inode);
+
+ xrt_info(xmm->pdev, "closed");
+ return 0;
+}
+
+/*
+ * Called for xclbin download xclbin load ioctl.
+ */
+static int xmgmt_bitstream_axlf_fpga_mgr(struct xmgmt_main *xmm, void *axlf, size_t size)
+{
+ int ret;
+
+ WARN_ON(!mutex_is_locked(&xmm->busy_mutex));
+
+ /*
+ * Should any error happens during download, we can't trust
+ * the cached xclbin any more.
+ */
+ vfree(xmm->firmware_ulp);
+ xmm->firmware_ulp = NULL;
+
+ ret = xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, axlf, XMGMT_ULP);
+ if (ret == 0)
+ xmm->firmware_ulp = axlf;
+
+ return ret;
+}
+
+static int bitstream_axlf_ioctl(struct xmgmt_main *xmm, const void __user *arg)
+{
+ void *copy_buffer = NULL;
+ size_t copy_buffer_size = 0;
+ struct xmgmt_ioc_bitstream_axlf ioc_obj = { 0 };
+ struct axlf xclbin_obj = { {0} };
+ int ret = 0;
+
+ if (copy_from_user((void *)&ioc_obj, arg, sizeof(ioc_obj)))
+ return -EFAULT;
+ if (copy_from_user((void *)&xclbin_obj, ioc_obj.xclbin, sizeof(xclbin_obj)))
+ return -EFAULT;
+ if (memcmp(xclbin_obj.m_magic, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)))
+ return -EINVAL;
+
+ copy_buffer_size = xclbin_obj.m_header.m_length;
+ if (copy_buffer_size > MAX_XCLBIN_SIZE)
+ return -EINVAL;
+ copy_buffer = vmalloc(copy_buffer_size);
+ if (!copy_buffer)
+ return -ENOMEM;
+
+ if (copy_from_user(copy_buffer, ioc_obj.xclbin, copy_buffer_size)) {
+ vfree(copy_buffer);
+ return -EFAULT;
+ }
+
+ ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer, copy_buffer_size);
+ if (ret)
+ vfree(copy_buffer);
+
+ return ret;
+}
+
+static long xmgmt_main_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+ long result = 0;
+ struct xmgmt_main *xmm = filp->private_data;
+
+ if (_IOC_TYPE(cmd) != XMGMT_IOC_MAGIC)
+ return -ENOTTY;
+
+ mutex_lock(&xmm->busy_mutex);
+
+ xrt_info(xmm->pdev, "ioctl cmd %d, arg %ld", cmd, arg);
+ switch (cmd) {
+ case XMGMT_IOCICAPDOWNLOAD_AXLF:
+ result = bitstream_axlf_ioctl(xmm, (const void __user *)arg);
+ break;
+ default:
+ result = -ENOTTY;
+ break;
+ }
+
+ mutex_unlock(&xmm->busy_mutex);
+ return result;
+}
+
+static struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names []){
+ { .ep_name = XRT_MD_NODE_MGMT_MAIN },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xmgmt_main_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xmgmt_main_leaf_ioctl,
+ },
+ .xsd_file_ops = {
+ .xsf_ops = {
+ .owner = THIS_MODULE,
+ .open = xmgmt_main_open,
+ .release = xmgmt_main_close,
+ .unlocked_ioctl = xmgmt_main_ioctl,
+ },
+ .xsf_dev_name = "xmgmt",
+ },
+};
+
+static const struct platform_device_id xmgmt_main_id_table[] = {
+ { XMGMT_MAIN, (kernel_ulong_t)&xmgmt_main_data },
+ { },
+};
+
+static struct platform_driver xmgmt_main_driver = {
+ .driver = {
+ .name = XMGMT_MAIN,
+ },
+ .probe = xmgmt_main_probe,
+ .remove = xmgmt_main_remove,
+ .id_table = xmgmt_main_id_table,
+};
+
+int xmgmt_main_register_leaf(void)
+{
+ return xleaf_register_driver(XRT_SUBDEV_MGMT_MAIN,
+ &xmgmt_main_driver, xrt_mgmt_main_endpoints);
+}
+
+void xmgmt_main_unregister_leaf(void)
+{
+ xleaf_unregister_driver(XRT_SUBDEV_MGMT_MAIN);
+}
diff --git a/include/uapi/linux/xrt/xmgmt-ioctl.h b/include/uapi/linux/xrt/xmgmt-ioctl.h
new file mode 100644
index 000000000000..15834476f3b4
--- /dev/null
+++ b/include/uapi/linux/xrt/xmgmt-ioctl.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright (C) 2015-2021, Xilinx Inc
+ *
+ */
+
+/**
+ * DOC: PCIe Kernel Driver for Managament Physical Function
+ * Interfaces exposed by *xclmgmt* driver are defined in file, *mgmt-ioctl.h*.
+ * Core functionality provided by *xmgmt* driver is described in the following table:
+ *
+ * =========== ============================== ==================================
+ * Functionality ioctl request code data format
+ * =========== ============================== ==================================
+ * 1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
+ * =========== ============================== ==================================
+ */
+
+#ifndef _XMGMT_IOCTL_H_
+#define _XMGMT_IOCTL_H_
+
+#include <linux/ioctl.h>
+
+#define XMGMT_IOC_MAGIC 'X'
+#define XMGMT_IOC_ICAP_DOWNLOAD_AXLF 0x6
+
+/**
+ * struct xmgmt_ioc_bitstream_axlf - load xclbin (AXLF) device image
+ * used with XMGMT_IOCICAPDOWNLOAD_AXLF ioctl
+ *
+ * @xclbin: Pointer to user's xclbin structure in memory
+ */
+struct xmgmt_ioc_bitstream_axlf {
+ struct axlf *xclbin;
+};
+
+#define XMGMT_IOCICAPDOWNLOAD_AXLF \
+ _IOW(XMGMT_IOC_MAGIC, XMGMT_IOC_ICAP_DOWNLOAD_AXLF, struct xmgmt_ioc_bitstream_axlf)
+
+/*
+ * The following definitions are for binary compatibility with classic XRT management driver
+ */
+#define XCLMGMT_IOCICAPDOWNLOAD_AXLF XMGMT_IOCICAPDOWNLOAD_AXLF
+#define xclmgmt_ioc_bitstream_axlf xmgmt_ioc_bitstream_axlf
+
+#endif
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 08/18] fpga: xrt: main platform driver for management function device
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 08/18] fpga: xrt: main platform driver for management function device Lizhi Hou
@ 2021-02-26 17:22 ` Tom Rix
2021-03-16 21:23 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-26 17:22 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> platform driver that handles IOCTLs, such as hot reset and xclbin download.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xmgmt-main.h | 37 ++
> drivers/fpga/xrt/mgmt/main-impl.h | 37 ++
> drivers/fpga/xrt/mgmt/main.c | 693 ++++++++++++++++++++++++++
> include/uapi/linux/xrt/xmgmt-ioctl.h | 46 ++
> 4 files changed, 813 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
> create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
> create mode 100644 drivers/fpga/xrt/mgmt/main.c
> create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>
> diff --git a/drivers/fpga/xrt/include/xmgmt-main.h b/drivers/fpga/xrt/include/xmgmt-main.h
> new file mode 100644
> index 000000000000..1216d1881f8e
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xmgmt-main.h
> @@ -0,0 +1,37 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Runtime (XRT) driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XMGMT_MAIN_H_
> +#define _XMGMT_MAIN_H_
> +
> +#include <linux/xrt/xclbin.h>
> +#include "xleaf.h"
> +
> +enum xrt_mgmt_main_ioctl_cmd {
> + /* section needs to be vfree'd by caller */
> + XRT_MGMT_MAIN_GET_AXLF_SECTION = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
the must free instructions should go with the pointer needing freeing
> + /* vbnv needs to be kfree'd by caller */
> + XRT_MGMT_MAIN_GET_VBNV,
> +};
> +
> +enum provider_kind {
> + XMGMT_BLP,
> + XMGMT_PLP,
> + XMGMT_ULP,
what do these three mean ?
> +};
> +
> +struct xrt_mgmt_main_ioctl_get_axlf_section {
> + enum provider_kind xmmigas_axlf_kind;
> + enum axlf_section_kind xmmigas_section_kind;
> + void *xmmigas_section;
> + u64 xmmigas_section_size;
> +};
> +
> +#endif /* _XMGMT_MAIN_H_ */
> diff --git a/drivers/fpga/xrt/mgmt/main-impl.h b/drivers/fpga/xrt/mgmt/main-impl.h
From prefix used in the functions, a better name for this file would be xmgnt.h
> new file mode 100644
> index 000000000000..dd1b3e3773cc
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/main-impl.h
> @@ -0,0 +1,37 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XMGMT_MAIN_IMPL_H_
> +#define _XMGMT_MAIN_IMPL_H_
> +
> +#include <linux/platform_device.h>
> +#include "xmgmt-main.h"
> +
> +struct fpga_manager;
> +int xmgmt_process_xclbin(struct platform_device *pdev,
> + struct fpga_manager *fmgr,
> + const struct axlf *xclbin,
> + enum provider_kind kind);
> +void xmgmt_region_cleanup_all(struct platform_device *pdev);
> +
> +int bitstream_axlf_mailbox(struct platform_device *pdev, const void *xclbin);
the prefix should be consistent
> +int xmgmt_hot_reset(struct platform_device *pdev);
> +
> +/* Getting dtb for specified group. Caller should vfree returned dtb .*/
> +char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind);
> +char *xmgmt_get_vbnv(struct platform_device *pdev);
> +int xmgmt_get_provider_uuid(struct platform_device *pdev,
> + enum provider_kind kind, uuid_t *uuid);
> +
> +int xmgmt_main_register_leaf(void);
> +void xmgmt_main_unregister_leaf(void);
is _main_ needed ?
> +
> +#endif /* _XMGMT_MAIN_IMPL_H_ */
> diff --git a/drivers/fpga/xrt/mgmt/main.c b/drivers/fpga/xrt/mgmt/main.c
> new file mode 100644
> index 000000000000..66ffb4e7029d
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/main.c
> @@ -0,0 +1,693 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA MGMT PF entry point driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Sonal Santan <sonals@xilinx.com>
> + */
> +
> +#include <linux/firmware.h>
> +#include <linux/uaccess.h>
> +#include "xclbin-helper.h"
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include <linux/xrt/xmgmt-ioctl.h>
> +#include "xleaf/devctl.h"
> +#include "xmgmt-main.h"
> +#include "fmgr.h"
> +#include "xleaf/icap.h"
> +#include "xleaf/axigate.h"
> +#include "main-impl.h"
> +
> +#define XMGMT_MAIN "xmgmt_main"
> +
> +struct xmgmt_main {
> + struct platform_device *pdev;
> + struct axlf *firmware_blp;
> + struct axlf *firmware_plp;
> + struct axlf *firmware_ulp;
> + bool flash_ready;
> + bool devctl_ready;
could combine in a bitfield
> + struct fpga_manager *fmgr;
> + struct mutex busy_mutex; /* busy lock */
busy_mutex ? maybe just call this 'lock'
> +
> + uuid_t *blp_intf_uuids;
> + u32 blp_intf_uuid_num;
expand intf to interface
> +};
> +
> +/* Caller should be responsible for freeing the returned string. */
should be -> is
> +char *xmgmt_get_vbnv(struct platform_device *pdev)
what is 'vbnv' ?
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + const char *vbnv;
> + char *ret;
> + int i;
> +
> + if (xmm->firmware_plp)
> + vbnv = xmm->firmware_plp->m_header.m_platformVBNV;
> + else if (xmm->firmware_blp)
> + vbnv = xmm->firmware_blp->m_header.m_platformVBNV;
> + else
> + return NULL;
check usage in at least VBNV_show, this return is not handled
> +
> + ret = kstrdup(vbnv, GFP_KERNEL);
> + if (!ret)
> + return NULL;
> +
> + for (i = 0; i < strlen(ret); i++) {
> + if (ret[i] == ':' || ret[i] == '.')
> + ret[i] = '_';
> + }
> + return ret;
> +}
> +
> +static int get_dev_uuid(struct platform_device *pdev, char *uuidstr, size_t len)
> +{
> + char uuid[16];
> + struct platform_device *devctl_leaf;
> + struct xrt_devctl_ioctl_rw devctl_arg = { 0 };
> + int err, i, count;
> +
> + devctl_leaf = xleaf_get_leaf_by_epname(pdev, XRT_MD_NODE_BLP_ROM);
> + if (!devctl_leaf) {
> + xrt_err(pdev, "can not get %s", XRT_MD_NODE_BLP_ROM);
> + return -EINVAL;
> + }
> +
> + devctl_arg.xgir_id = XRT_DEVCTL_ROM_UUID;
> + devctl_arg.xgir_buf = uuid;
> + devctl_arg.xgir_len = sizeof(uuid);
> + devctl_arg.xgir_offset = 0;
> + err = xleaf_ioctl(devctl_leaf, XRT_DEVCTL_READ, &devctl_arg);
> + xleaf_put_leaf(pdev, devctl_leaf);
> + if (err) {
> + xrt_err(pdev, "can not get uuid: %d", err);
> + return err;
> + }
> +
This some strange word swapping, add a comment to explain why it is needed.
Consider if this needs to change on a big endian host.
> + for (count = 0, i = sizeof(uuid) - sizeof(u32);
> + i >= 0 && len > count; i -= sizeof(u32)) {
> + count += snprintf(uuidstr + count, len - count, "%08x", *(u32 *)&uuid[i]);
> + }
> + return 0;
> +}
> +
> +int xmgmt_hot_reset(struct platform_device *pdev)
> +{
> + int ret = xleaf_broadcast_event(pdev, XRT_EVENT_PRE_HOT_RESET, false);
> +
> + if (ret) {
> + xrt_err(pdev, "offline failed, hot reset is canceled");
> + return ret;
> + }
> +
> + xleaf_hot_reset(pdev);
> + xleaf_broadcast_event(pdev, XRT_EVENT_POST_HOT_RESET, false);
> + return 0;
> +}
> +
> +static ssize_t reset_store(struct device *dev, struct device_attribute *da,
> + const char *buf, size_t count)
> +{
> + struct platform_device *pdev = to_platform_device(dev);
> +
> + xmgmt_hot_reset(pdev);
> + return count;
> +}
> +static DEVICE_ATTR_WO(reset);
> +
> +static ssize_t VBNV_show(struct device *dev, struct device_attribute *da, char *buf)
> +{
> + ssize_t ret;
> + char *vbnv;
> + struct platform_device *pdev = to_platform_device(dev);
> +
> + vbnv = xmgmt_get_vbnv(pdev);
> + ret = sprintf(buf, "%s\n", vbnv);
null return not handled
> + kfree(vbnv);
> + return ret;
> +}
> +static DEVICE_ATTR_RO(VBNV);
> +
> +static ssize_t logic_uuids_show(struct device *dev, struct device_attribute *da, char *buf)
> +{
what is a logic uuid ?
> + ssize_t ret;
> + char uuid[80];
> + struct platform_device *pdev = to_platform_device(dev);
> +
> + /* Getting UUID pointed to by VSEC, should be the same as logic UUID of BLP. */
> + ret = get_dev_uuid(pdev, uuid, sizeof(uuid));
> + if (ret)
> + return ret;
> + ret = sprintf(buf, "%s\n", uuid);
> + return ret;
> +}
> +static DEVICE_ATTR_RO(logic_uuids);
> +
> +static ssize_t interface_uuids_show(struct device *dev, struct device_attribute *da, char *buf)
> +{
> + ssize_t ret = 0;
> + struct platform_device *pdev = to_platform_device(dev);
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + u32 i;
> +
> + for (i = 0; i < xmm->blp_intf_uuid_num; i++) {
> + char uuidstr[80];
80 is used several places, consider making this a #define
> +
> + xrt_md_trans_uuid2str(&xmm->blp_intf_uuids[i], uuidstr);
> + ret += sprintf(buf + ret, "%s\n", uuidstr);
> + }
> + return ret;
> +}
> +static DEVICE_ATTR_RO(interface_uuids);
> +
> +static struct attribute *xmgmt_main_attrs[] = {
> + &dev_attr_reset.attr,
> + &dev_attr_VBNV.attr,
> + &dev_attr_logic_uuids.attr,
> + &dev_attr_interface_uuids.attr,
> + NULL,
> +};
> +
> +/*
> + * sysfs hook to load xclbin primarily used for driver debug
> + */
> +static ssize_t ulp_image_write(struct file *filp, struct kobject *kobj,
> + struct bin_attribute *attr, char *buffer, loff_t off, size_t count)
> +{
off is signed, and this function assumes it is unsigned.
this will segfault the memcpy
> + struct xmgmt_main *xmm = dev_get_drvdata(container_of(kobj, struct device, kobj));
> + struct axlf *xclbin;
> + ulong len;
> +
> + if (off == 0) {
> + if (count < sizeof(*xclbin)) {
> + xrt_err(xmm->pdev, "count is too small %zu", count);
> + return -EINVAL;
> + }
> +
> + if (xmm->firmware_ulp) {
could check if the current buffer size is less than needed to avoid another alloc
> + vfree(xmm->firmware_ulp);
> + xmm->firmware_ulp = NULL;
> + }
> + xclbin = (struct axlf *)buffer;
> + xmm->firmware_ulp = vmalloc(xclbin->m_header.m_length);
> + if (!xmm->firmware_ulp)
> + return -ENOMEM;
> + } else {
> + xclbin = xmm->firmware_ulp;
> + }
> +
> + len = xclbin->m_header.m_length;
> + if (off + count >= len && off < len) {
off + count > is ok ?
> + memcpy(xmm->firmware_ulp + off, buffer, len - off);
> + xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, xmm->firmware_ulp, XMGMT_ULP);
> + } else if (off + count < len) {
> + memcpy(xmm->firmware_ulp + off, buffer, count);
> + }
> +
> + return count;
> +}
> +
> +static struct bin_attribute ulp_image_attr = {
> + .attr = {
> + .name = "ulp_image",
> + .mode = 0200
> + },
> + .write = ulp_image_write,
> + .size = 0
> +};
> +
> +static struct bin_attribute *xmgmt_main_bin_attrs[] = {
> + &ulp_image_attr,
> + NULL,
> +};
> +
> +static const struct attribute_group xmgmt_main_attrgroup = {
> + .attrs = xmgmt_main_attrs,
> + .bin_attrs = xmgmt_main_bin_attrs,
> +};
> +
> +static int load_firmware_from_flash(struct platform_device *pdev, struct axlf **fw_buf, size_t *len)
> +{
> + return -EOPNOTSUPP;
> +}
this function is not needed, it is used only in a direct call from xmgmt_load_firmware.
looks like it is part of an error hander which will return this NOSUPPORT error instead of the real error from load_firmware_from disk
> +
> +static int load_firmware_from_disk(struct platform_device *pdev, struct axlf **fw_buf, size_t *len)
> +{
> + char uuid[80];
> + int err = 0;
> + char fw_name[256];
> + const struct firmware *fw;
> +
> + err = get_dev_uuid(pdev, uuid, sizeof(uuid));
> + if (err)
> + return err;
> +
> + (void)snprintf(fw_name, sizeof(fw_name), "xilinx/%s/partition.xsabin", uuid);
> + xrt_info(pdev, "try loading fw: %s", fw_name);
> +
> + err = request_firmware(&fw, fw_name, DEV(pdev));
> + if (err)
> + return err;
> +
> + *fw_buf = vmalloc(fw->size);
> + *len = fw->size;
malloc fails but len is set ?
better to set len to 0 on failure
> + if (*fw_buf)
> + memcpy(*fw_buf, fw->data, fw->size);
> + else
> + err = -ENOMEM;
> +
> + release_firmware(fw);
> + return 0;
> +}
> +
> +static const struct axlf *xmgmt_get_axlf_firmware(struct xmgmt_main *xmm, enum provider_kind kind)
> +{
> + switch (kind) {
> + case XMGMT_BLP:
> + return xmm->firmware_blp;
> + case XMGMT_PLP:
> + return xmm->firmware_plp;
> + case XMGMT_ULP:
> + return xmm->firmware_ulp;
> + default:
> + xrt_err(xmm->pdev, "unknown axlf kind: %d", kind);
> + return NULL;
> + }
> +}
> +
needs a comment that user is responsible for freeing return
> +char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + char *dtb = NULL;
> + const struct axlf *provider = xmgmt_get_axlf_firmware(xmm, kind);
> + int rc;
> +
> + if (!provider)
> + return dtb;
> +
> + rc = xrt_xclbin_get_metadata(DEV(pdev), provider, &dtb);
> + if (rc)
> + xrt_err(pdev, "failed to find dtb: %d", rc);
> + return dtb;
> +}
> +
similar caller responsible for freeing
> +static const char *get_uuid_from_firmware(struct platform_device *pdev, const struct axlf *xclbin)
> +{
> + const void *uuid = NULL;
> + const void *uuiddup = NULL;
> + void *dtb = NULL;
> + int rc;
> +
> + rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA, &dtb, NULL);
> + if (rc)
> + return NULL;
> +
> + rc = xrt_md_get_prop(DEV(pdev), dtb, NULL, NULL, XRT_MD_PROP_LOGIC_UUID, &uuid, NULL);
> + if (!rc)
> + uuiddup = kstrdup(uuid, GFP_KERNEL);
> + vfree(dtb);
> + return uuiddup;
> +}
> +
> +static bool is_valid_firmware(struct platform_device *pdev,
> + const struct axlf *xclbin, size_t fw_len)
> +{
> + const char *fw_buf = (const char *)xclbin;
> + size_t axlflen = xclbin->m_header.m_length;
> + const char *fw_uuid;
> + char dev_uuid[80];
> + int err;
> +
> + err = get_dev_uuid(pdev, dev_uuid, sizeof(dev_uuid));
> + if (err)
> + return false;
> +
> + if (memcmp(fw_buf, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)) != 0) {
> + xrt_err(pdev, "unknown fw format");
> + return false;
> + }
> +
> + if (axlflen > fw_len) {
> + xrt_err(pdev, "truncated fw, length: %zu, expect: %zu", fw_len, axlflen);
> + return false;
> + }
> +
> + fw_uuid = get_uuid_from_firmware(pdev, xclbin);
> + if (!fw_uuid || strcmp(fw_uuid, dev_uuid) != 0) {
> + xrt_err(pdev, "bad fw UUID: %s, expect: %s",
> + fw_uuid ? fw_uuid : "<none>", dev_uuid);
> + kfree(fw_uuid);
> + return false;
> + }
> +
> + kfree(fw_uuid);
> + return true;
> +}
> +
> +int xmgmt_get_provider_uuid(struct platform_device *pdev, enum provider_kind kind, uuid_t *uuid)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + const struct axlf *fwbuf;
> + const char *fw_uuid;
> + int rc = -ENOENT;
> +
> + mutex_lock(&xmm->busy_mutex);
> +
> + fwbuf = xmgmt_get_axlf_firmware(xmm, kind);
> + if (!fwbuf)
> + goto done;
> +
> + fw_uuid = get_uuid_from_firmware(pdev, fwbuf);
> + if (!fw_uuid)
> + goto done;
> +
> + rc = xrt_md_trans_str2uuid(DEV(pdev), fw_uuid, uuid);
should this be &fw_uuid ?
> + kfree(fw_uuid);
> +
> +done:
> + mutex_unlock(&xmm->busy_mutex);
> + return rc;
> +}
> +
> +static int xmgmt_create_blp(struct xmgmt_main *xmm)
> +{
> + struct platform_device *pdev = xmm->pdev;
> + int rc = 0;
> + char *dtb = NULL;
> + const struct axlf *provider = xmgmt_get_axlf_firmware(xmm, XMGMT_BLP);
> +
> + dtb = xmgmt_get_dtb(pdev, XMGMT_BLP);
> + if (dtb) {
not doing any work is ok ?
> + rc = xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, provider, XMGMT_BLP);
> + if (rc) {
> + xrt_err(pdev, "failed to process BLP: %d", rc);
> + goto failed;
> + }
> +
> + rc = xleaf_create_group(pdev, dtb);
> + if (rc < 0)
why not (rc) ?
> + xrt_err(pdev, "failed to create BLP group: %d", rc);
> + else
> + rc = 0;
> +
> + WARN_ON(xmm->blp_intf_uuids);
warn but not free ?
> + xrt_md_get_intf_uuids(&pdev->dev, dtb, &xmm->blp_intf_uuid_num, NULL);
> + if (xmm->blp_intf_uuid_num > 0) {
> + xmm->blp_intf_uuids = vzalloc(sizeof(uuid_t) * xmm->blp_intf_uuid_num);
unchecked alloc
> + xrt_md_get_intf_uuids(&pdev->dev, dtb, &xmm->blp_intf_uuid_num,
> + xmm->blp_intf_uuids);
> + }
> + }
> +
> +failed:
> + vfree(dtb);
> + return rc;
> +}
> +
> +static int xmgmt_load_firmware(struct xmgmt_main *xmm)
> +{
> + struct platform_device *pdev = xmm->pdev;
> + int rc;
> + size_t fwlen;
> +
> + rc = load_firmware_from_disk(pdev, &xmm->firmware_blp, &fwlen);
> + if (rc != 0)
> + rc = load_firmware_from_flash(pdev, &xmm->firmware_blp, &fwlen);
this is the function that should be removed
> + if (rc == 0 && is_valid_firmware(pdev, xmm->firmware_blp, fwlen))
> + (void)xmgmt_create_blp(xmm);
> + else
> + xrt_err(pdev, "failed to find firmware, giving up: %d", rc);
> + return rc;
> +}
> +
> +static void xmgmt_main_event_cb(struct platform_device *pdev, void *arg)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + struct xrt_event *evt = (struct xrt_event *)arg;
> + enum xrt_events e = evt->xe_evt;
> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
> + struct platform_device *leaf;
> +
> + switch (e) {
> + case XRT_EVENT_POST_CREATION: {
> + if (id == XRT_SUBDEV_DEVCTL && !xmm->devctl_ready) {
> + leaf = xleaf_get_leaf_by_epname(pdev, XRT_MD_NODE_BLP_ROM);
> + if (leaf) {
> + xmm->devctl_ready = true;
> + xleaf_put_leaf(pdev, leaf);
> + }
> + } else if (id == XRT_SUBDEV_QSPI && !xmm->flash_ready) {
> + xmm->flash_ready = true;
> + } else {
> + break;
> + }
> +
> + if (xmm->devctl_ready)
> + (void)xmgmt_load_firmware(xmm);
> + break;
> + }
> + case XRT_EVENT_PRE_REMOVAL:
> + break;
> + default:
> + xrt_dbg(pdev, "ignored event %d", e);
> + break;
> + }
> +}
> +
> +static int xmgmt_main_probe(struct platform_device *pdev)
> +{
> + struct xmgmt_main *xmm;
> +
> + xrt_info(pdev, "probing...");
> +
> + xmm = devm_kzalloc(DEV(pdev), sizeof(*xmm), GFP_KERNEL);
> + if (!xmm)
> + return -ENOMEM;
> +
> + xmm->pdev = pdev;
> + xmm->fmgr = xmgmt_fmgr_probe(pdev);
> + if (IS_ERR(xmm->fmgr))
> + return PTR_ERR(xmm->fmgr);
> +
> + platform_set_drvdata(pdev, xmm);
> + mutex_init(&xmm->busy_mutex);
> +
> + /* Ready to handle req thru sysfs nodes. */
> + if (sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup))
> + xrt_err(pdev, "failed to create sysfs group");
> + return 0;
> +}
> +
> +static int xmgmt_main_remove(struct platform_device *pdev)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> +
> + /* By now, group driver should prevent any inter-leaf call. */
> +
> + xrt_info(pdev, "leaving...");
> +
> + vfree(xmm->blp_intf_uuids);
> + vfree(xmm->firmware_blp);
> + vfree(xmm->firmware_plp);
> + vfree(xmm->firmware_ulp);
> + xmgmt_region_cleanup_all(pdev);
> + (void)xmgmt_fmgr_remove(xmm->fmgr);
> + (void)sysfs_remove_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup);
> + return 0;
> +}
> +
> +static int
> +xmgmt_main_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
> + int ret = 0;
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + xmgmt_main_event_cb(pdev, arg);
> + break;
> + case XRT_MGMT_MAIN_GET_AXLF_SECTION: {
> + struct xrt_mgmt_main_ioctl_get_axlf_section *get =
> + (struct xrt_mgmt_main_ioctl_get_axlf_section *)arg;
> + const struct axlf *firmware = xmgmt_get_axlf_firmware(xmm, get->xmmigas_axlf_kind);
> +
> + if (!firmware) {
> + ret = -ENOENT;
> + } else {
> + ret = xrt_xclbin_get_section(firmware, get->xmmigas_section_kind,
> + &get->xmmigas_section,
> + &get->xmmigas_section_size);
> + }
> + break;
> + }
> + case XRT_MGMT_MAIN_GET_VBNV: {
> + char **vbnv_p = (char **)arg;
> +
> + *vbnv_p = xmgmt_get_vbnv(pdev);
this can fail
> + break;
> + }
> + default:
> + xrt_err(pdev, "unknown cmd: %d", cmd);
> + ret = -EINVAL;
> + break;
> + }
> + return ret;
> +}
> +
> +static int xmgmt_main_open(struct inode *inode, struct file *file)
> +{
> + struct platform_device *pdev = xleaf_devnode_open(inode);
> +
> + /* Device may have gone already when we get here. */
> + if (!pdev)
> + return -ENODEV;
> +
> + xrt_info(pdev, "opened");
> + file->private_data = platform_get_drvdata(pdev);
> + return 0;
> +}
> +
> +static int xmgmt_main_close(struct inode *inode, struct file *file)
> +{
> + struct xmgmt_main *xmm = file->private_data;
> +
> + xleaf_devnode_close(inode);
> +
> + xrt_info(xmm->pdev, "closed");
> + return 0;
> +}
> +
> +/*
> + * Called for xclbin download xclbin load ioctl.
> + */
> +static int xmgmt_bitstream_axlf_fpga_mgr(struct xmgmt_main *xmm, void *axlf, size_t size)
> +{
> + int ret;
> +
> + WARN_ON(!mutex_is_locked(&xmm->busy_mutex));
> +
> + /*
> + * Should any error happens during download, we can't trust
> + * the cached xclbin any more.
> + */
> + vfree(xmm->firmware_ulp);
> + xmm->firmware_ulp = NULL;
> +
> + ret = xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, axlf, XMGMT_ULP);
> + if (ret == 0)
> + xmm->firmware_ulp = axlf;
> +
> + return ret;
> +}
> +
> +static int bitstream_axlf_ioctl(struct xmgmt_main *xmm, const void __user *arg)
> +{
> + void *copy_buffer = NULL;
> + size_t copy_buffer_size = 0;
> + struct xmgmt_ioc_bitstream_axlf ioc_obj = { 0 };
> + struct axlf xclbin_obj = { {0} };
> + int ret = 0;
> +
> + if (copy_from_user((void *)&ioc_obj, arg, sizeof(ioc_obj)))
> + return -EFAULT;
> + if (copy_from_user((void *)&xclbin_obj, ioc_obj.xclbin, sizeof(xclbin_obj)))
> + return -EFAULT;
> + if (memcmp(xclbin_obj.m_magic, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)))
> + return -EINVAL;
> +
> + copy_buffer_size = xclbin_obj.m_header.m_length;
> + if (copy_buffer_size > MAX_XCLBIN_SIZE)
> + return -EINVAL;
is there a min size ?
> + copy_buffer = vmalloc(copy_buffer_size);
> + if (!copy_buffer)
> + return -ENOMEM;
> +
> + if (copy_from_user(copy_buffer, ioc_obj.xclbin, copy_buffer_size)) {
> + vfree(copy_buffer);
> + return -EFAULT;
> + }
> +
> + ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer, copy_buffer_size);
> + if (ret)
> + vfree(copy_buffer);
> +
> + return ret;
> +}
> +
> +static long xmgmt_main_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> +{
> + long result = 0;
> + struct xmgmt_main *xmm = filp->private_data;
> +
> + if (_IOC_TYPE(cmd) != XMGMT_IOC_MAGIC)
> + return -ENOTTY;
> +
> + mutex_lock(&xmm->busy_mutex);
> +
> + xrt_info(xmm->pdev, "ioctl cmd %d, arg %ld", cmd, arg);
> + switch (cmd) {
> + case XMGMT_IOCICAPDOWNLOAD_AXLF:
> + result = bitstream_axlf_ioctl(xmm, (const void __user *)arg);
> + break;
> + default:
> + result = -ENOTTY;
> + break;
> + }
> +
> + mutex_unlock(&xmm->busy_mutex);
> + return result;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names []){
> + { .ep_name = XRT_MD_NODE_MGMT_MAIN },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xmgmt_main_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xmgmt_main_leaf_ioctl,
> + },
> + .xsd_file_ops = {
> + .xsf_ops = {
> + .owner = THIS_MODULE,
> + .open = xmgmt_main_open,
> + .release = xmgmt_main_close,
> + .unlocked_ioctl = xmgmt_main_ioctl,
> + },
> + .xsf_dev_name = "xmgmt",
> + },
> +};
> +
> +static const struct platform_device_id xmgmt_main_id_table[] = {
> + { XMGMT_MAIN, (kernel_ulong_t)&xmgmt_main_data },
> + { },
> +};
> +
> +static struct platform_driver xmgmt_main_driver = {
> + .driver = {
> + .name = XMGMT_MAIN,
> + },
> + .probe = xmgmt_main_probe,
> + .remove = xmgmt_main_remove,
> + .id_table = xmgmt_main_id_table,
> +};
> +
> +int xmgmt_main_register_leaf(void)
> +{
> + return xleaf_register_driver(XRT_SUBDEV_MGMT_MAIN,
> + &xmgmt_main_driver, xrt_mgmt_main_endpoints);
> +}
> +
> +void xmgmt_main_unregister_leaf(void)
> +{
> + xleaf_unregister_driver(XRT_SUBDEV_MGMT_MAIN);
> +}
> diff --git a/include/uapi/linux/xrt/xmgmt-ioctl.h b/include/uapi/linux/xrt/xmgmt-ioctl.h
> new file mode 100644
> index 000000000000..15834476f3b4
> --- /dev/null
> +++ b/include/uapi/linux/xrt/xmgmt-ioctl.h
> @@ -0,0 +1,46 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Copyright (C) 2015-2021, Xilinx Inc
> + *
> + */
> +
> +/**
> + * DOC: PCIe Kernel Driver for Managament Physical Function
> + * Interfaces exposed by *xclmgmt* driver are defined in file, *mgmt-ioctl.h*.
> + * Core functionality provided by *xmgmt* driver is described in the following table:
> + *
> + * =========== ============================== ==================================
> + * Functionality ioctl request code data format
> + * =========== ============================== ==================================
> + * 1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
> + * =========== ============================== ==================================
> + */
> +
> +#ifndef _XMGMT_IOCTL_H_
> +#define _XMGMT_IOCTL_H_
> +
> +#include <linux/ioctl.h>
> +
> +#define XMGMT_IOC_MAGIC 'X'
> +#define XMGMT_IOC_ICAP_DOWNLOAD_AXLF 0x6
> +
> +/**
> + * struct xmgmt_ioc_bitstream_axlf - load xclbin (AXLF) device image
> + * used with XMGMT_IOCICAPDOWNLOAD_AXLF ioctl
> + *
> + * @xclbin: Pointer to user's xclbin structure in memory
> + */
> +struct xmgmt_ioc_bitstream_axlf {
> + struct axlf *xclbin;
where is struct axlf defined ?
Tom
> +};
> +
> +#define XMGMT_IOCICAPDOWNLOAD_AXLF \
> + _IOW(XMGMT_IOC_MAGIC, XMGMT_IOC_ICAP_DOWNLOAD_AXLF, struct xmgmt_ioc_bitstream_axlf)
> +
> +/*
> + * The following definitions are for binary compatibility with classic XRT management driver
> + */
> +#define XCLMGMT_IOCICAPDOWNLOAD_AXLF XMGMT_IOCICAPDOWNLOAD_AXLF
> +#define xclmgmt_ioc_bitstream_axlf xmgmt_ioc_bitstream_axlf
> +
> +#endif
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 08/18] fpga: xrt: main platform driver for management function device
2021-02-26 17:22 ` Tom Rix
@ 2021-03-16 21:23 ` Lizhi Hou
2021-03-17 21:12 ` Tom Rix
0 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-03-16 21:23 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 02/26/2021 09:22 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> platform driver that handles IOCTLs, such as hot reset and xclbin download.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xmgmt-main.h | 37 ++
>> drivers/fpga/xrt/mgmt/main-impl.h | 37 ++
>> drivers/fpga/xrt/mgmt/main.c | 693 ++++++++++++++++++++++++++
>> include/uapi/linux/xrt/xmgmt-ioctl.h | 46 ++
>> 4 files changed, 813 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>> create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
>> create mode 100644 drivers/fpga/xrt/mgmt/main.c
>> create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>>
>> diff --git a/drivers/fpga/xrt/include/xmgmt-main.h b/drivers/fpga/xrt/include/xmgmt-main.h
>> new file mode 100644
>> index 000000000000..1216d1881f8e
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xmgmt-main.h
>> @@ -0,0 +1,37 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for Xilinx Runtime (XRT) driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#ifndef _XMGMT_MAIN_H_
>> +#define _XMGMT_MAIN_H_
>> +
>> +#include <linux/xrt/xclbin.h>
>> +#include "xleaf.h"
>> +
>> +enum xrt_mgmt_main_ioctl_cmd {
>> + /* section needs to be vfree'd by caller */
>> + XRT_MGMT_MAIN_GET_AXLF_SECTION = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> the must free instructions should go with the pointer needing freeing
Sure. Will move the free instructions.
>> + /* vbnv needs to be kfree'd by caller */
>> + XRT_MGMT_MAIN_GET_VBNV,
>> +};
>> +
>> +enum provider_kind {
>> + XMGMT_BLP,
>> + XMGMT_PLP,
>> + XMGMT_ULP,
> what do these three mean ?
Will add comment
/* There are three kind of partitions. Each of them is programmed
independently. */
enum provider_kind {
XMGMT_BLP, /* Base Logic Partition */
XMGMT_PLP, /* Provider Logic Partition */
XMGMT_ULP, /* User Logic Partition */
};
>> +};
>> +
>> +struct xrt_mgmt_main_ioctl_get_axlf_section {
>> + enum provider_kind xmmigas_axlf_kind;
>> + enum axlf_section_kind xmmigas_section_kind;
>> + void *xmmigas_section;
>> + u64 xmmigas_section_size;
>> +};
>> +
>> +#endif /* _XMGMT_MAIN_H_ */
>> diff --git a/drivers/fpga/xrt/mgmt/main-impl.h b/drivers/fpga/xrt/mgmt/main-impl.h
> From prefix used in the functions, a better name for this file would be xmgnt.h
Will change.
>> new file mode 100644
>> index 000000000000..dd1b3e3773cc
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/mgmt/main-impl.h
>> @@ -0,0 +1,37 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for Xilinx Alveo Management Function Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#ifndef _XMGMT_MAIN_IMPL_H_
>> +#define _XMGMT_MAIN_IMPL_H_
>> +
>> +#include <linux/platform_device.h>
>> +#include "xmgmt-main.h"
>> +
>> +struct fpga_manager;
>> +int xmgmt_process_xclbin(struct platform_device *pdev,
>> + struct fpga_manager *fmgr,
>> + const struct axlf *xclbin,
>> + enum provider_kind kind);
>> +void xmgmt_region_cleanup_all(struct platform_device *pdev);
>> +
>> +int bitstream_axlf_mailbox(struct platform_device *pdev, const void *xclbin);
> the prefix should be consistent
Will fix this.
>> +int xmgmt_hot_reset(struct platform_device *pdev);
>> +
>> +/* Getting dtb for specified group. Caller should vfree returned dtb .*/
>> +char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind);
>> +char *xmgmt_get_vbnv(struct platform_device *pdev);
>> +int xmgmt_get_provider_uuid(struct platform_device *pdev,
>> + enum provider_kind kind, uuid_t *uuid);
>> +
>> +int xmgmt_main_register_leaf(void);
>> +void xmgmt_main_unregister_leaf(void);
> is _main_ needed ?
Will remove.
>> +
>> +#endif /* _XMGMT_MAIN_IMPL_H_ */
>> diff --git a/drivers/fpga/xrt/mgmt/main.c b/drivers/fpga/xrt/mgmt/main.c
>> new file mode 100644
>> index 000000000000..66ffb4e7029d
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/mgmt/main.c
>> @@ -0,0 +1,693 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA MGMT PF entry point driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Sonal Santan <sonals@xilinx.com>
>> + */
>> +
>> +#include <linux/firmware.h>
>> +#include <linux/uaccess.h>
>> +#include "xclbin-helper.h"
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include <linux/xrt/xmgmt-ioctl.h>
>> +#include "xleaf/devctl.h"
>> +#include "xmgmt-main.h"
>> +#include "fmgr.h"
>> +#include "xleaf/icap.h"
>> +#include "xleaf/axigate.h"
>> +#include "main-impl.h"
>> +
>> +#define XMGMT_MAIN "xmgmt_main"
>> +
>> +struct xmgmt_main {
>> + struct platform_device *pdev;
>> + struct axlf *firmware_blp;
>> + struct axlf *firmware_plp;
>> + struct axlf *firmware_ulp;
>> + bool flash_ready;
>> + bool devctl_ready;
> could combine in a bitfield
Will change.
>> + struct fpga_manager *fmgr;
>> + struct mutex busy_mutex; /* busy lock */
> busy_mutex ? maybe just call this 'lock'
Will change.
>> +
>> + uuid_t *blp_intf_uuids;
>> + u32 blp_intf_uuid_num;
> expand intf to interface
Will change.
>> +};
>> +
>> +/* Caller should be responsible for freeing the returned string. */
> should be -> is
Will fix it.
>> +char *xmgmt_get_vbnv(struct platform_device *pdev)
> what is 'vbnv' ?
vbnv stands for Vendor, BoardID, Name, Version. It is a string which
describes board and shell.
>> +{
>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>> + const char *vbnv;
>> + char *ret;
>> + int i;
>> +
>> + if (xmm->firmware_plp)
>> + vbnv = xmm->firmware_plp->m_header.m_platformVBNV;
>> + else if (xmm->firmware_blp)
>> + vbnv = xmm->firmware_blp->m_header.m_platformVBNV;
>> + else
>> + return NULL;
> check usage in at least VBNV_show, this return is not handled
Will add check.
>> +
>> + ret = kstrdup(vbnv, GFP_KERNEL);
>> + if (!ret)
>> + return NULL;
>> +
>> + for (i = 0; i < strlen(ret); i++) {
>> + if (ret[i] == ':' || ret[i] == '.')
>> + ret[i] = '_';
>> + }
>> + return ret;
>> +}
>> +
>> +static int get_dev_uuid(struct platform_device *pdev, char *uuidstr, size_t len)
>> +{
>> + char uuid[16];
>> + struct platform_device *devctl_leaf;
>> + struct xrt_devctl_ioctl_rw devctl_arg = { 0 };
>> + int err, i, count;
>> +
>> + devctl_leaf = xleaf_get_leaf_by_epname(pdev, XRT_MD_NODE_BLP_ROM);
>> + if (!devctl_leaf) {
>> + xrt_err(pdev, "can not get %s", XRT_MD_NODE_BLP_ROM);
>> + return -EINVAL;
>> + }
>> +
>> + devctl_arg.xgir_id = XRT_DEVCTL_ROM_UUID;
>> + devctl_arg.xgir_buf = uuid;
>> + devctl_arg.xgir_len = sizeof(uuid);
>> + devctl_arg.xgir_offset = 0;
>> + err = xleaf_ioctl(devctl_leaf, XRT_DEVCTL_READ, &devctl_arg);
>> + xleaf_put_leaf(pdev, devctl_leaf);
>> + if (err) {
>> + xrt_err(pdev, "can not get uuid: %d", err);
>> + return err;
>> + }
>> +
> This some strange word swapping, add a comment to explain why it is needed.
>
> Consider if this needs to change on a big endian host.
Will change to use import_uuid then convert to string.
>
>> + for (count = 0, i = sizeof(uuid) - sizeof(u32);
>> + i >= 0 && len > count; i -= sizeof(u32)) {
>> + count += snprintf(uuidstr + count, len - count, "%08x", *(u32 *)&uuid[i]);
>> + }
>> + return 0;
>> +}
>> +
>> +int xmgmt_hot_reset(struct platform_device *pdev)
>> +{
>> + int ret = xleaf_broadcast_event(pdev, XRT_EVENT_PRE_HOT_RESET, false);
>> +
>> + if (ret) {
>> + xrt_err(pdev, "offline failed, hot reset is canceled");
>> + return ret;
>> + }
>> +
>> + xleaf_hot_reset(pdev);
>> + xleaf_broadcast_event(pdev, XRT_EVENT_POST_HOT_RESET, false);
>> + return 0;
>> +}
>> +
>> +static ssize_t reset_store(struct device *dev, struct device_attribute *da,
>> + const char *buf, size_t count)
>> +{
>> + struct platform_device *pdev = to_platform_device(dev);
>> +
>> + xmgmt_hot_reset(pdev);
>> + return count;
>> +}
>> +static DEVICE_ATTR_WO(reset);
>> +
>> +static ssize_t VBNV_show(struct device *dev, struct device_attribute *da, char *buf)
>> +{
>> + ssize_t ret;
>> + char *vbnv;
>> + struct platform_device *pdev = to_platform_device(dev);
>> +
>> + vbnv = xmgmt_get_vbnv(pdev);
>> + ret = sprintf(buf, "%s\n", vbnv);
> null return not handled
Will add check.
>> + kfree(vbnv);
>> + return ret;
>> +}
>> +static DEVICE_ATTR_RO(VBNV);
>> +
>> +static ssize_t logic_uuids_show(struct device *dev, struct device_attribute *da, char *buf)
>> +{
> what is a logic uuid ?
logic uuid is a unique id to identify the shell.
>> + ssize_t ret;
>> + char uuid[80];
>> + struct platform_device *pdev = to_platform_device(dev);
>> +
>> + /* Getting UUID pointed to by VSEC, should be the same as logic UUID of BLP. */
>> + ret = get_dev_uuid(pdev, uuid, sizeof(uuid));
>> + if (ret)
>> + return ret;
>> + ret = sprintf(buf, "%s\n", uuid);
>> + return ret;
>> +}
>> +static DEVICE_ATTR_RO(logic_uuids);
>> +
>> +static ssize_t interface_uuids_show(struct device *dev, struct device_attribute *da, char *buf)
>> +{
>> + ssize_t ret = 0;
>> + struct platform_device *pdev = to_platform_device(dev);
>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>> + u32 i;
>> +
>> + for (i = 0; i < xmm->blp_intf_uuid_num; i++) {
>> + char uuidstr[80];
> 80 is used several places, consider making this a #define
Will fix this.
>> +
>> + xrt_md_trans_uuid2str(&xmm->blp_intf_uuids[i], uuidstr);
>> + ret += sprintf(buf + ret, "%s\n", uuidstr);
>> + }
>> + return ret;
>> +}
>> +static DEVICE_ATTR_RO(interface_uuids);
>> +
>> +static struct attribute *xmgmt_main_attrs[] = {
>> + &dev_attr_reset.attr,
>> + &dev_attr_VBNV.attr,
>> + &dev_attr_logic_uuids.attr,
>> + &dev_attr_interface_uuids.attr,
>> + NULL,
>> +};
>> +
>> +/*
>> + * sysfs hook to load xclbin primarily used for driver debug
>> + */
>> +static ssize_t ulp_image_write(struct file *filp, struct kobject *kobj,
>> + struct bin_attribute *attr, char *buffer, loff_t off, size_t count)
>> +{
> off is signed, and this function assumes it is unsigned.
>
> this will segfault the memcpy
Will remove ulp_image_write(). This function is not needed anymore.
>
>> + struct xmgmt_main *xmm = dev_get_drvdata(container_of(kobj, struct device, kobj));
>> + struct axlf *xclbin;
>> + ulong len;
>> +
>> + if (off == 0) {
>> + if (count < sizeof(*xclbin)) {
>> + xrt_err(xmm->pdev, "count is too small %zu", count);
>> + return -EINVAL;
>> + }
>> +
>> + if (xmm->firmware_ulp) {
> could check if the current buffer size is less than needed to avoid another alloc
>> + vfree(xmm->firmware_ulp);
>> + xmm->firmware_ulp = NULL;
>> + }
>> + xclbin = (struct axlf *)buffer;
>> + xmm->firmware_ulp = vmalloc(xclbin->m_header.m_length);
>> + if (!xmm->firmware_ulp)
>> + return -ENOMEM;
>> + } else {
>> + xclbin = xmm->firmware_ulp;
>> + }
>> +
>> + len = xclbin->m_header.m_length;
>> + if (off + count >= len && off < len) {
> off + count > is ok ?
>> + memcpy(xmm->firmware_ulp + off, buffer, len - off);
>> + xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, xmm->firmware_ulp, XMGMT_ULP);
>> + } else if (off + count < len) {
>> + memcpy(xmm->firmware_ulp + off, buffer, count);
>> + }
>> +
>> + return count;
>> +}
>> +
>> +static struct bin_attribute ulp_image_attr = {
>> + .attr = {
>> + .name = "ulp_image",
>> + .mode = 0200
>> + },
>> + .write = ulp_image_write,
>> + .size = 0
>> +};
>> +
>> +static struct bin_attribute *xmgmt_main_bin_attrs[] = {
>> + &ulp_image_attr,
>> + NULL,
>> +};
>> +
>> +static const struct attribute_group xmgmt_main_attrgroup = {
>> + .attrs = xmgmt_main_attrs,
>> + .bin_attrs = xmgmt_main_bin_attrs,
>> +};
>> +
>> +static int load_firmware_from_flash(struct platform_device *pdev, struct axlf **fw_buf, size_t *len)
>> +{
>> + return -EOPNOTSUPP;
>> +}
> this function is not needed, it is used only in a direct call from xmgmt_load_firmware.
>
> looks like it is part of an error hander which will return this NOSUPPORT error instead of the real error from load_firmware_from disk
Will remove it.
>> +
>> +static int load_firmware_from_disk(struct platform_device *pdev, struct axlf **fw_buf, size_t *len)
>> +{
>> + char uuid[80];
>> + int err = 0;
>> + char fw_name[256];
>> + const struct firmware *fw;
>> +
>> + err = get_dev_uuid(pdev, uuid, sizeof(uuid));
>> + if (err)
>> + return err;
>> +
>> + (void)snprintf(fw_name, sizeof(fw_name), "xilinx/%s/partition.xsabin", uuid);
>> + xrt_info(pdev, "try loading fw: %s", fw_name);
>> +
>> + err = request_firmware(&fw, fw_name, DEV(pdev));
>> + if (err)
>> + return err;
>> +
>> + *fw_buf = vmalloc(fw->size);
>> + *len = fw->size;
> malloc fails but len is set ?
>
> better to set len to 0 on failure
Will add check and set len to 0 on failure.
>
>> + if (*fw_buf)
>> + memcpy(*fw_buf, fw->data, fw->size);
>> + else
>> + err = -ENOMEM;
>> +
>> + release_firmware(fw);
>> + return 0;
>> +}
>> +
>> +static const struct axlf *xmgmt_get_axlf_firmware(struct xmgmt_main *xmm, enum provider_kind kind)
>> +{
>> + switch (kind) {
>> + case XMGMT_BLP:
>> + return xmm->firmware_blp;
>> + case XMGMT_PLP:
>> + return xmm->firmware_plp;
>> + case XMGMT_ULP:
>> + return xmm->firmware_ulp;
>> + default:
>> + xrt_err(xmm->pdev, "unknown axlf kind: %d", kind);
>> + return NULL;
>> + }
>> +}
>> +
> needs a comment that user is responsible for freeing return
Will add.
>> +char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind)
>> +{
>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>> + char *dtb = NULL;
>> + const struct axlf *provider = xmgmt_get_axlf_firmware(xmm, kind);
>> + int rc;
>> +
>> + if (!provider)
>> + return dtb;
>> +
>> + rc = xrt_xclbin_get_metadata(DEV(pdev), provider, &dtb);
>> + if (rc)
>> + xrt_err(pdev, "failed to find dtb: %d", rc);
>> + return dtb;
>> +}
>> +
> similar caller responsible for freeing
Will add comment.
>> +static const char *get_uuid_from_firmware(struct platform_device *pdev, const struct axlf *xclbin)
>> +{
>> + const void *uuid = NULL;
>> + const void *uuiddup = NULL;
>> + void *dtb = NULL;
>> + int rc;
>> +
>> + rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA, &dtb, NULL);
>> + if (rc)
>> + return NULL;
>> +
>> + rc = xrt_md_get_prop(DEV(pdev), dtb, NULL, NULL, XRT_MD_PROP_LOGIC_UUID, &uuid, NULL);
>> + if (!rc)
>> + uuiddup = kstrdup(uuid, GFP_KERNEL);
>> + vfree(dtb);
>> + return uuiddup;
>> +}
>> +
>> +static bool is_valid_firmware(struct platform_device *pdev,
>> + const struct axlf *xclbin, size_t fw_len)
>> +{
>> + const char *fw_buf = (const char *)xclbin;
>> + size_t axlflen = xclbin->m_header.m_length;
>> + const char *fw_uuid;
>> + char dev_uuid[80];
>> + int err;
>> +
>> + err = get_dev_uuid(pdev, dev_uuid, sizeof(dev_uuid));
>> + if (err)
>> + return false;
>> +
>> + if (memcmp(fw_buf, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)) != 0) {
>> + xrt_err(pdev, "unknown fw format");
>> + return false;
>> + }
>> +
>> + if (axlflen > fw_len) {
>> + xrt_err(pdev, "truncated fw, length: %zu, expect: %zu", fw_len, axlflen);
>> + return false;
>> + }
>> +
>> + fw_uuid = get_uuid_from_firmware(pdev, xclbin);
>> + if (!fw_uuid || strcmp(fw_uuid, dev_uuid) != 0) {
>> + xrt_err(pdev, "bad fw UUID: %s, expect: %s",
>> + fw_uuid ? fw_uuid : "<none>", dev_uuid);
>> + kfree(fw_uuid);
>> + return false;
>> + }
>> +
>> + kfree(fw_uuid);
>> + return true;
>> +}
>> +
>> +int xmgmt_get_provider_uuid(struct platform_device *pdev, enum provider_kind kind, uuid_t *uuid)
>> +{
>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>> + const struct axlf *fwbuf;
>> + const char *fw_uuid;
>> + int rc = -ENOENT;
>> +
>> + mutex_lock(&xmm->busy_mutex);
>> +
>> + fwbuf = xmgmt_get_axlf_firmware(xmm, kind);
>> + if (!fwbuf)
>> + goto done;
>> +
>> + fw_uuid = get_uuid_from_firmware(pdev, fwbuf);
>> + if (!fw_uuid)
>> + goto done;
>> +
>> + rc = xrt_md_trans_str2uuid(DEV(pdev), fw_uuid, uuid);
> should this be &fw_uuid ?
No. fw_uuid points to the uuid string.
>> + kfree(fw_uuid);
>> +
>> +done:
>> + mutex_unlock(&xmm->busy_mutex);
>> + return rc;
>> +}
>> +
>> +static int xmgmt_create_blp(struct xmgmt_main *xmm)
>> +{
>> + struct platform_device *pdev = xmm->pdev;
>> + int rc = 0;
>> + char *dtb = NULL;
>> + const struct axlf *provider = xmgmt_get_axlf_firmware(xmm, XMGMT_BLP);
>> +
>> + dtb = xmgmt_get_dtb(pdev, XMGMT_BLP);
>> + if (dtb) {
> not doing any work is ok ?
Will add check for dtb.
>> + rc = xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, provider, XMGMT_BLP);
>> + if (rc) {
>> + xrt_err(pdev, "failed to process BLP: %d", rc);
>> + goto failed;
>> + }
>> +
>> + rc = xleaf_create_group(pdev, dtb);
>> + if (rc < 0)
> why not (rc) ?
xleaf_create_group() returns positive group id.
>> + xrt_err(pdev, "failed to create BLP group: %d", rc);
>> + else
>> + rc = 0;
>> +
>> + WARN_ON(xmm->blp_intf_uuids);
> warn but not free ?
non zero means memory leak. That will be a bug need to be fixed.
>> + xrt_md_get_intf_uuids(&pdev->dev, dtb, &xmm->blp_intf_uuid_num, NULL);
>> + if (xmm->blp_intf_uuid_num > 0) {
>> + xmm->blp_intf_uuids = vzalloc(sizeof(uuid_t) * xmm->blp_intf_uuid_num);
> unchecked alloc
Will check.
>> + xrt_md_get_intf_uuids(&pdev->dev, dtb, &xmm->blp_intf_uuid_num,
>> + xmm->blp_intf_uuids);
>> + }
>> + }
>> +
>> +failed:
>> + vfree(dtb);
>> + return rc;
>> +}
>> +
>> +static int xmgmt_load_firmware(struct xmgmt_main *xmm)
>> +{
>> + struct platform_device *pdev = xmm->pdev;
>> + int rc;
>> + size_t fwlen;
>> +
>> + rc = load_firmware_from_disk(pdev, &xmm->firmware_blp, &fwlen);
>> + if (rc != 0)
>> + rc = load_firmware_from_flash(pdev, &xmm->firmware_blp, &fwlen);
> this is the function that should be removed
Sure.
>> + if (rc == 0 && is_valid_firmware(pdev, xmm->firmware_blp, fwlen))
>> + (void)xmgmt_create_blp(xmm);
>> + else
>> + xrt_err(pdev, "failed to find firmware, giving up: %d", rc);
>> + return rc;
>> +}
>> +
>> +static void xmgmt_main_event_cb(struct platform_device *pdev, void *arg)
>> +{
>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>> + struct xrt_event *evt = (struct xrt_event *)arg;
>> + enum xrt_events e = evt->xe_evt;
>> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
>> + struct platform_device *leaf;
>> +
>> + switch (e) {
>> + case XRT_EVENT_POST_CREATION: {
>> + if (id == XRT_SUBDEV_DEVCTL && !xmm->devctl_ready) {
>> + leaf = xleaf_get_leaf_by_epname(pdev, XRT_MD_NODE_BLP_ROM);
>> + if (leaf) {
>> + xmm->devctl_ready = true;
>> + xleaf_put_leaf(pdev, leaf);
>> + }
>> + } else if (id == XRT_SUBDEV_QSPI && !xmm->flash_ready) {
>> + xmm->flash_ready = true;
>> + } else {
>> + break;
>> + }
>> +
>> + if (xmm->devctl_ready)
>> + (void)xmgmt_load_firmware(xmm);
>> + break;
>> + }
>> + case XRT_EVENT_PRE_REMOVAL:
>> + break;
>> + default:
>> + xrt_dbg(pdev, "ignored event %d", e);
>> + break;
>> + }
>> +}
>> +
>> +static int xmgmt_main_probe(struct platform_device *pdev)
>> +{
>> + struct xmgmt_main *xmm;
>> +
>> + xrt_info(pdev, "probing...");
>> +
>> + xmm = devm_kzalloc(DEV(pdev), sizeof(*xmm), GFP_KERNEL);
>> + if (!xmm)
>> + return -ENOMEM;
>> +
>> + xmm->pdev = pdev;
>> + xmm->fmgr = xmgmt_fmgr_probe(pdev);
>> + if (IS_ERR(xmm->fmgr))
>> + return PTR_ERR(xmm->fmgr);
>> +
>> + platform_set_drvdata(pdev, xmm);
>> + mutex_init(&xmm->busy_mutex);
>> +
>> + /* Ready to handle req thru sysfs nodes. */
>> + if (sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup))
>> + xrt_err(pdev, "failed to create sysfs group");
>> + return 0;
>> +}
>> +
>> +static int xmgmt_main_remove(struct platform_device *pdev)
>> +{
>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>> +
>> + /* By now, group driver should prevent any inter-leaf call. */
>> +
>> + xrt_info(pdev, "leaving...");
>> +
>> + vfree(xmm->blp_intf_uuids);
>> + vfree(xmm->firmware_blp);
>> + vfree(xmm->firmware_plp);
>> + vfree(xmm->firmware_ulp);
>> + xmgmt_region_cleanup_all(pdev);
>> + (void)xmgmt_fmgr_remove(xmm->fmgr);
>> + (void)sysfs_remove_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup);
>> + return 0;
>> +}
>> +
>> +static int
>> +xmgmt_main_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>> + int ret = 0;
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + xmgmt_main_event_cb(pdev, arg);
>> + break;
>> + case XRT_MGMT_MAIN_GET_AXLF_SECTION: {
>> + struct xrt_mgmt_main_ioctl_get_axlf_section *get =
>> + (struct xrt_mgmt_main_ioctl_get_axlf_section *)arg;
>> + const struct axlf *firmware = xmgmt_get_axlf_firmware(xmm, get->xmmigas_axlf_kind);
>> +
>> + if (!firmware) {
>> + ret = -ENOENT;
>> + } else {
>> + ret = xrt_xclbin_get_section(firmware, get->xmmigas_section_kind,
>> + &get->xmmigas_section,
>> + &get->xmmigas_section_size);
>> + }
>> + break;
>> + }
>> + case XRT_MGMT_MAIN_GET_VBNV: {
>> + char **vbnv_p = (char **)arg;
>> +
>> + *vbnv_p = xmgmt_get_vbnv(pdev);
> this can fail
Will add check.
>> + break;
>> + }
>> + default:
>> + xrt_err(pdev, "unknown cmd: %d", cmd);
>> + ret = -EINVAL;
>> + break;
>> + }
>> + return ret;
>> +}
>> +
>> +static int xmgmt_main_open(struct inode *inode, struct file *file)
>> +{
>> + struct platform_device *pdev = xleaf_devnode_open(inode);
>> +
>> + /* Device may have gone already when we get here. */
>> + if (!pdev)
>> + return -ENODEV;
>> +
>> + xrt_info(pdev, "opened");
>> + file->private_data = platform_get_drvdata(pdev);
>> + return 0;
>> +}
>> +
>> +static int xmgmt_main_close(struct inode *inode, struct file *file)
>> +{
>> + struct xmgmt_main *xmm = file->private_data;
>> +
>> + xleaf_devnode_close(inode);
>> +
>> + xrt_info(xmm->pdev, "closed");
>> + return 0;
>> +}
>> +
>> +/*
>> + * Called for xclbin download xclbin load ioctl.
>> + */
>> +static int xmgmt_bitstream_axlf_fpga_mgr(struct xmgmt_main *xmm, void *axlf, size_t size)
>> +{
>> + int ret;
>> +
>> + WARN_ON(!mutex_is_locked(&xmm->busy_mutex));
>> +
>> + /*
>> + * Should any error happens during download, we can't trust
>> + * the cached xclbin any more.
>> + */
>> + vfree(xmm->firmware_ulp);
>> + xmm->firmware_ulp = NULL;
>> +
>> + ret = xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, axlf, XMGMT_ULP);
>> + if (ret == 0)
>> + xmm->firmware_ulp = axlf;
>> +
>> + return ret;
>> +}
>> +
>> +static int bitstream_axlf_ioctl(struct xmgmt_main *xmm, const void __user *arg)
>> +{
>> + void *copy_buffer = NULL;
>> + size_t copy_buffer_size = 0;
>> + struct xmgmt_ioc_bitstream_axlf ioc_obj = { 0 };
>> + struct axlf xclbin_obj = { {0} };
>> + int ret = 0;
>> +
>> + if (copy_from_user((void *)&ioc_obj, arg, sizeof(ioc_obj)))
>> + return -EFAULT;
>> + if (copy_from_user((void *)&xclbin_obj, ioc_obj.xclbin, sizeof(xclbin_obj)))
>> + return -EFAULT;
>> + if (memcmp(xclbin_obj.m_magic, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)))
>> + return -EINVAL;
>> +
>> + copy_buffer_size = xclbin_obj.m_header.m_length;
>> + if (copy_buffer_size > MAX_XCLBIN_SIZE)
>> + return -EINVAL;
> is there a min size ?
Will add check.
>> + copy_buffer = vmalloc(copy_buffer_size);
>> + if (!copy_buffer)
>> + return -ENOMEM;
>> +
>> + if (copy_from_user(copy_buffer, ioc_obj.xclbin, copy_buffer_size)) {
>> + vfree(copy_buffer);
>> + return -EFAULT;
>> + }
>> +
>> + ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer, copy_buffer_size);
>> + if (ret)
>> + vfree(copy_buffer);
>> +
>> + return ret;
>> +}
>> +
>> +static long xmgmt_main_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>> +{
>> + long result = 0;
>> + struct xmgmt_main *xmm = filp->private_data;
>> +
>> + if (_IOC_TYPE(cmd) != XMGMT_IOC_MAGIC)
>> + return -ENOTTY;
>> +
>> + mutex_lock(&xmm->busy_mutex);
>> +
>> + xrt_info(xmm->pdev, "ioctl cmd %d, arg %ld", cmd, arg);
>> + switch (cmd) {
>> + case XMGMT_IOCICAPDOWNLOAD_AXLF:
>> + result = bitstream_axlf_ioctl(xmm, (const void __user *)arg);
>> + break;
>> + default:
>> + result = -ENOTTY;
>> + break;
>> + }
>> +
>> + mutex_unlock(&xmm->busy_mutex);
>> + return result;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names []){
>> + { .ep_name = XRT_MD_NODE_MGMT_MAIN },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xmgmt_main_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xmgmt_main_leaf_ioctl,
>> + },
>> + .xsd_file_ops = {
>> + .xsf_ops = {
>> + .owner = THIS_MODULE,
>> + .open = xmgmt_main_open,
>> + .release = xmgmt_main_close,
>> + .unlocked_ioctl = xmgmt_main_ioctl,
>> + },
>> + .xsf_dev_name = "xmgmt",
>> + },
>> +};
>> +
>> +static const struct platform_device_id xmgmt_main_id_table[] = {
>> + { XMGMT_MAIN, (kernel_ulong_t)&xmgmt_main_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xmgmt_main_driver = {
>> + .driver = {
>> + .name = XMGMT_MAIN,
>> + },
>> + .probe = xmgmt_main_probe,
>> + .remove = xmgmt_main_remove,
>> + .id_table = xmgmt_main_id_table,
>> +};
>> +
>> +int xmgmt_main_register_leaf(void)
>> +{
>> + return xleaf_register_driver(XRT_SUBDEV_MGMT_MAIN,
>> + &xmgmt_main_driver, xrt_mgmt_main_endpoints);
>> +}
>> +
>> +void xmgmt_main_unregister_leaf(void)
>> +{
>> + xleaf_unregister_driver(XRT_SUBDEV_MGMT_MAIN);
>> +}
>> diff --git a/include/uapi/linux/xrt/xmgmt-ioctl.h b/include/uapi/linux/xrt/xmgmt-ioctl.h
>> new file mode 100644
>> index 000000000000..15834476f3b4
>> --- /dev/null
>> +++ b/include/uapi/linux/xrt/xmgmt-ioctl.h
>> @@ -0,0 +1,46 @@
>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>> +/*
>> + * Copyright (C) 2015-2021, Xilinx Inc
>> + *
>> + */
>> +
>> +/**
>> + * DOC: PCIe Kernel Driver for Managament Physical Function
>> + * Interfaces exposed by *xclmgmt* driver are defined in file, *mgmt-ioctl.h*.
>> + * Core functionality provided by *xmgmt* driver is described in the following table:
>> + *
>> + * =========== ============================== ==================================
>> + * Functionality ioctl request code data format
>> + * =========== ============================== ==================================
>> + * 1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
>> + * =========== ============================== ==================================
>> + */
>> +
>> +#ifndef _XMGMT_IOCTL_H_
>> +#define _XMGMT_IOCTL_H_
>> +
>> +#include <linux/ioctl.h>
>> +
>> +#define XMGMT_IOC_MAGIC 'X'
>> +#define XMGMT_IOC_ICAP_DOWNLOAD_AXLF 0x6
>> +
>> +/**
>> + * struct xmgmt_ioc_bitstream_axlf - load xclbin (AXLF) device image
>> + * used with XMGMT_IOCICAPDOWNLOAD_AXLF ioctl
>> + *
>> + * @xclbin: Pointer to user's xclbin structure in memory
>> + */
>> +struct xmgmt_ioc_bitstream_axlf {
>> + struct axlf *xclbin;
> where is struct axlf defined ?
It is defined in include/uapi/linux/xrt/xclbin.h
Thanks,
Lizhi
>
> Tom
>
>> +};
>> +
>> +#define XMGMT_IOCICAPDOWNLOAD_AXLF \
>> + _IOW(XMGMT_IOC_MAGIC, XMGMT_IOC_ICAP_DOWNLOAD_AXLF, struct xmgmt_ioc_bitstream_axlf)
>> +
>> +/*
>> + * The following definitions are for binary compatibility with classic XRT management driver
>> + */
>> +#define XCLMGMT_IOCICAPDOWNLOAD_AXLF XMGMT_IOCICAPDOWNLOAD_AXLF
>> +#define xclmgmt_ioc_bitstream_axlf xmgmt_ioc_bitstream_axlf
>> +
>> +#endif
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 08/18] fpga: xrt: main platform driver for management function device
2021-03-16 21:23 ` Lizhi Hou
@ 2021-03-17 21:12 ` Tom Rix
0 siblings, 0 replies; 87+ messages in thread
From: Tom Rix @ 2021-03-17 21:12 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
On 3/16/21 2:23 PM, Lizhi Hou wrote:
> Hi Tom,
>
>
> On 02/26/2021 09:22 AM, Tom Rix wrote:
>> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>>> platform driver that handles IOCTLs, such as hot reset and xclbin download.
>>>
>>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>>> ---
>>> drivers/fpga/xrt/include/xmgmt-main.h | 37 ++
>>> drivers/fpga/xrt/mgmt/main-impl.h | 37 ++
>>> drivers/fpga/xrt/mgmt/main.c | 693 ++++++++++++++++++++++++++
>>> include/uapi/linux/xrt/xmgmt-ioctl.h | 46 ++
>>> 4 files changed, 813 insertions(+)
>>> create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>>> create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
>>> create mode 100644 drivers/fpga/xrt/mgmt/main.c
>>> create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>>>
>>> diff --git a/drivers/fpga/xrt/include/xmgmt-main.h b/drivers/fpga/xrt/include/xmgmt-main.h
>>> new file mode 100644
>>> index 000000000000..1216d1881f8e
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/include/xmgmt-main.h
>>> @@ -0,0 +1,37 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * Header file for Xilinx Runtime (XRT) driver
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors:
>>> + * Cheng Zhen <maxz@xilinx.com>
>>> + */
>>> +
>>> +#ifndef _XMGMT_MAIN_H_
>>> +#define _XMGMT_MAIN_H_
>>> +
>>> +#include <linux/xrt/xclbin.h>
>>> +#include "xleaf.h"
>>> +
>>> +enum xrt_mgmt_main_ioctl_cmd {
>>> + /* section needs to be vfree'd by caller */
>>> + XRT_MGMT_MAIN_GET_AXLF_SECTION = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> the must free instructions should go with the pointer needing freeing
> Sure. Will move the free instructions.
>>> + /* vbnv needs to be kfree'd by caller */
>>> + XRT_MGMT_MAIN_GET_VBNV,
>>> +};
>>> +
>>> +enum provider_kind {
>>> + XMGMT_BLP,
>>> + XMGMT_PLP,
>>> + XMGMT_ULP,
>> what do these three mean ?
> Will add comment
>
> /* There are three kind of partitions. Each of them is programmed independently. */
> enum provider_kind {
> XMGMT_BLP, /* Base Logic Partition */
> XMGMT_PLP, /* Provider Logic Partition */
> XMGMT_ULP, /* User Logic Partition */
> };
>
looks good
>>> +};
>>> +
>>> +struct xrt_mgmt_main_ioctl_get_axlf_section {
>>> + enum provider_kind xmmigas_axlf_kind;
>>> + enum axlf_section_kind xmmigas_section_kind;
>>> + void *xmmigas_section;
>>> + u64 xmmigas_section_size;
>>> +};
>>> +
>>> +#endif /* _XMGMT_MAIN_H_ */
>>> diff --git a/drivers/fpga/xrt/mgmt/main-impl.h b/drivers/fpga/xrt/mgmt/main-impl.h
>> From prefix used in the functions, a better name for this file would be xmgnt.h
> Will change.
>>> new file mode 100644
>>> index 000000000000..dd1b3e3773cc
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/mgmt/main-impl.h
>>> @@ -0,0 +1,37 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * Header file for Xilinx Alveo Management Function Driver
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors:
>>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>>> + * Cheng Zhen <maxz@xilinx.com>
>>> + */
>>> +
>>> +#ifndef _XMGMT_MAIN_IMPL_H_
>>> +#define _XMGMT_MAIN_IMPL_H_
>>> +
>>> +#include <linux/platform_device.h>
>>> +#include "xmgmt-main.h"
>>> +
>>> +struct fpga_manager;
>>> +int xmgmt_process_xclbin(struct platform_device *pdev,
>>> + struct fpga_manager *fmgr,
>>> + const struct axlf *xclbin,
>>> + enum provider_kind kind);
>>> +void xmgmt_region_cleanup_all(struct platform_device *pdev);
>>> +
>>> +int bitstream_axlf_mailbox(struct platform_device *pdev, const void *xclbin);
>> the prefix should be consistent
> Will fix this.
>>> +int xmgmt_hot_reset(struct platform_device *pdev);
>>> +
>>> +/* Getting dtb for specified group. Caller should vfree returned dtb .*/
>>> +char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind);
>>> +char *xmgmt_get_vbnv(struct platform_device *pdev);
>>> +int xmgmt_get_provider_uuid(struct platform_device *pdev,
>>> + enum provider_kind kind, uuid_t *uuid);
>>> +
>>> +int xmgmt_main_register_leaf(void);
>>> +void xmgmt_main_unregister_leaf(void);
>> is _main_ needed ?
> Will remove.
>>> +
>>> +#endif /* _XMGMT_MAIN_IMPL_H_ */
>>> diff --git a/drivers/fpga/xrt/mgmt/main.c b/drivers/fpga/xrt/mgmt/main.c
>>> new file mode 100644
>>> index 000000000000..66ffb4e7029d
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/mgmt/main.c
>>> @@ -0,0 +1,693 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * Xilinx Alveo FPGA MGMT PF entry point driver
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors:
>>> + * Sonal Santan <sonals@xilinx.com>
>>> + */
>>> +
>>> +#include <linux/firmware.h>
>>> +#include <linux/uaccess.h>
>>> +#include "xclbin-helper.h"
>>> +#include "metadata.h"
>>> +#include "xleaf.h"
>>> +#include <linux/xrt/xmgmt-ioctl.h>
>>> +#include "xleaf/devctl.h"
>>> +#include "xmgmt-main.h"
>>> +#include "fmgr.h"
>>> +#include "xleaf/icap.h"
>>> +#include "xleaf/axigate.h"
>>> +#include "main-impl.h"
>>> +
>>> +#define XMGMT_MAIN "xmgmt_main"
>>> +
>>> +struct xmgmt_main {
>>> + struct platform_device *pdev;
>>> + struct axlf *firmware_blp;
>>> + struct axlf *firmware_plp;
>>> + struct axlf *firmware_ulp;
>>> + bool flash_ready;
>>> + bool devctl_ready;
>> could combine in a bitfield
> Will change.
>>> + struct fpga_manager *fmgr;
>>> + struct mutex busy_mutex; /* busy lock */
>> busy_mutex ? maybe just call this 'lock'
> Will change.
>>> +
>>> + uuid_t *blp_intf_uuids;
>>> + u32 blp_intf_uuid_num;
>> expand intf to interface
> Will change.
>>> +};
>>> +
>>> +/* Caller should be responsible for freeing the returned string. */
>> should be -> is
> Will fix it.
>>> +char *xmgmt_get_vbnv(struct platform_device *pdev)
>> what is 'vbnv' ?
> vbnv stands for Vendor, BoardID, Name, Version. It is a string which describes board and shell.
ok, makes sense.
please add a comment
>>> +{
>>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>>> + const char *vbnv;
>>> + char *ret;
>>> + int i;
>>> +
>>> + if (xmm->firmware_plp)
>>> + vbnv = xmm->firmware_plp->m_header.m_platformVBNV;
>>> + else if (xmm->firmware_blp)
>>> + vbnv = xmm->firmware_blp->m_header.m_platformVBNV;
>>> + else
>>> + return NULL;
>> check usage in at least VBNV_show, this return is not handled
> Will add check.
>>> +
>>> + ret = kstrdup(vbnv, GFP_KERNEL);
>>> + if (!ret)
>>> + return NULL;
>>> +
>>> + for (i = 0; i < strlen(ret); i++) {
>>> + if (ret[i] == ':' || ret[i] == '.')
>>> + ret[i] = '_';
>>> + }
>>> + return ret;
>>> +}
>>> +
>>> +static int get_dev_uuid(struct platform_device *pdev, char *uuidstr, size_t len)
>>> +{
>>> + char uuid[16];
>>> + struct platform_device *devctl_leaf;
>>> + struct xrt_devctl_ioctl_rw devctl_arg = { 0 };
>>> + int err, i, count;
>>> +
>>> + devctl_leaf = xleaf_get_leaf_by_epname(pdev, XRT_MD_NODE_BLP_ROM);
>>> + if (!devctl_leaf) {
>>> + xrt_err(pdev, "can not get %s", XRT_MD_NODE_BLP_ROM);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + devctl_arg.xgir_id = XRT_DEVCTL_ROM_UUID;
>>> + devctl_arg.xgir_buf = uuid;
>>> + devctl_arg.xgir_len = sizeof(uuid);
>>> + devctl_arg.xgir_offset = 0;
>>> + err = xleaf_ioctl(devctl_leaf, XRT_DEVCTL_READ, &devctl_arg);
>>> + xleaf_put_leaf(pdev, devctl_leaf);
>>> + if (err) {
>>> + xrt_err(pdev, "can not get uuid: %d", err);
>>> + return err;
>>> + }
>>> +
>> This some strange word swapping, add a comment to explain why it is needed.
>>
>> Consider if this needs to change on a big endian host.
> Will change to use import_uuid then convert to string.
>>
>>> + for (count = 0, i = sizeof(uuid) - sizeof(u32);
>>> + i >= 0 && len > count; i -= sizeof(u32)) {
>>> + count += snprintf(uuidstr + count, len - count, "%08x", *(u32 *)&uuid[i]);
>>> + }
>>> + return 0;
>>> +}
>>> +
>>> +int xmgmt_hot_reset(struct platform_device *pdev)
>>> +{
>>> + int ret = xleaf_broadcast_event(pdev, XRT_EVENT_PRE_HOT_RESET, false);
>>> +
>>> + if (ret) {
>>> + xrt_err(pdev, "offline failed, hot reset is canceled");
>>> + return ret;
>>> + }
>>> +
>>> + xleaf_hot_reset(pdev);
>>> + xleaf_broadcast_event(pdev, XRT_EVENT_POST_HOT_RESET, false);
>>> + return 0;
>>> +}
>>> +
>>> +static ssize_t reset_store(struct device *dev, struct device_attribute *da,
>>> + const char *buf, size_t count)
>>> +{
>>> + struct platform_device *pdev = to_platform_device(dev);
>>> +
>>> + xmgmt_hot_reset(pdev);
>>> + return count;
>>> +}
>>> +static DEVICE_ATTR_WO(reset);
>>> +
>>> +static ssize_t VBNV_show(struct device *dev, struct device_attribute *da, char *buf)
>>> +{
>>> + ssize_t ret;
>>> + char *vbnv;
>>> + struct platform_device *pdev = to_platform_device(dev);
>>> +
>>> + vbnv = xmgmt_get_vbnv(pdev);
>>> + ret = sprintf(buf, "%s\n", vbnv);
>> null return not handled
> Will add check.
>>> + kfree(vbnv);
>>> + return ret;
>>> +}
>>> +static DEVICE_ATTR_RO(VBNV);
>>> +
>>> +static ssize_t logic_uuids_show(struct device *dev, struct device_attribute *da, char *buf)
>>> +{
>> what is a logic uuid ?
> logic uuid is a unique id to identify the shell.
>>> + ssize_t ret;
>>> + char uuid[80];
>>> + struct platform_device *pdev = to_platform_device(dev);
>>> +
>>> + /* Getting UUID pointed to by VSEC, should be the same as logic UUID of BLP. */
>>> + ret = get_dev_uuid(pdev, uuid, sizeof(uuid));
>>> + if (ret)
>>> + return ret;
>>> + ret = sprintf(buf, "%s\n", uuid);
>>> + return ret;
>>> +}
>>> +static DEVICE_ATTR_RO(logic_uuids);
>>> +
>>> +static ssize_t interface_uuids_show(struct device *dev, struct device_attribute *da, char *buf)
>>> +{
>>> + ssize_t ret = 0;
>>> + struct platform_device *pdev = to_platform_device(dev);
>>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>>> + u32 i;
>>> +
>>> + for (i = 0; i < xmm->blp_intf_uuid_num; i++) {
>>> + char uuidstr[80];
>> 80 is used several places, consider making this a #define
> Will fix this.
>>> +
>>> + xrt_md_trans_uuid2str(&xmm->blp_intf_uuids[i], uuidstr);
>>> + ret += sprintf(buf + ret, "%s\n", uuidstr);
>>> + }
>>> + return ret;
>>> +}
>>> +static DEVICE_ATTR_RO(interface_uuids);
>>> +
>>> +static struct attribute *xmgmt_main_attrs[] = {
>>> + &dev_attr_reset.attr,
>>> + &dev_attr_VBNV.attr,
>>> + &dev_attr_logic_uuids.attr,
>>> + &dev_attr_interface_uuids.attr,
>>> + NULL,
>>> +};
>>> +
>>> +/*
>>> + * sysfs hook to load xclbin primarily used for driver debug
>>> + */
>>> +static ssize_t ulp_image_write(struct file *filp, struct kobject *kobj,
>>> + struct bin_attribute *attr, char *buffer, loff_t off, size_t count)
>>> +{
>> off is signed, and this function assumes it is unsigned.
>>
>> this will segfault the memcpy
> Will remove ulp_image_write(). This function is not needed anymore.
>>
>>> + struct xmgmt_main *xmm = dev_get_drvdata(container_of(kobj, struct device, kobj));
>>> + struct axlf *xclbin;
>>> + ulong len;
>>> +
>>> + if (off == 0) {
>>> + if (count < sizeof(*xclbin)) {
>>> + xrt_err(xmm->pdev, "count is too small %zu", count);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + if (xmm->firmware_ulp) {
>> could check if the current buffer size is less than needed to avoid another alloc
>>> + vfree(xmm->firmware_ulp);
>>> + xmm->firmware_ulp = NULL;
>>> + }
>>> + xclbin = (struct axlf *)buffer;
>>> + xmm->firmware_ulp = vmalloc(xclbin->m_header.m_length);
>>> + if (!xmm->firmware_ulp)
>>> + return -ENOMEM;
>>> + } else {
>>> + xclbin = xmm->firmware_ulp;
>>> + }
>>> +
>>> + len = xclbin->m_header.m_length;
>>> + if (off + count >= len && off < len) {
>> off + count > is ok ?
>>> + memcpy(xmm->firmware_ulp + off, buffer, len - off);
>>> + xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, xmm->firmware_ulp, XMGMT_ULP);
>>> + } else if (off + count < len) {
>>> + memcpy(xmm->firmware_ulp + off, buffer, count);
>>> + }
>>> +
>>> + return count;
>>> +}
>>> +
>>> +static struct bin_attribute ulp_image_attr = {
>>> + .attr = {
>>> + .name = "ulp_image",
>>> + .mode = 0200
>>> + },
>>> + .write = ulp_image_write,
>>> + .size = 0
>>> +};
>>> +
>>> +static struct bin_attribute *xmgmt_main_bin_attrs[] = {
>>> + &ulp_image_attr,
>>> + NULL,
>>> +};
>>> +
>>> +static const struct attribute_group xmgmt_main_attrgroup = {
>>> + .attrs = xmgmt_main_attrs,
>>> + .bin_attrs = xmgmt_main_bin_attrs,
>>> +};
>>> +
>>> +static int load_firmware_from_flash(struct platform_device *pdev, struct axlf **fw_buf, size_t *len)
>>> +{
>>> + return -EOPNOTSUPP;
>>> +}
>> this function is not needed, it is used only in a direct call from xmgmt_load_firmware.
>>
>> looks like it is part of an error hander which will return this NOSUPPORT error instead of the real error from load_firmware_from disk
> Will remove it.
>>> +
>>> +static int load_firmware_from_disk(struct platform_device *pdev, struct axlf **fw_buf, size_t *len)
>>> +{
>>> + char uuid[80];
>>> + int err = 0;
>>> + char fw_name[256];
>>> + const struct firmware *fw;
>>> +
>>> + err = get_dev_uuid(pdev, uuid, sizeof(uuid));
>>> + if (err)
>>> + return err;
>>> +
>>> + (void)snprintf(fw_name, sizeof(fw_name), "xilinx/%s/partition.xsabin", uuid);
>>> + xrt_info(pdev, "try loading fw: %s", fw_name);
>>> +
>>> + err = request_firmware(&fw, fw_name, DEV(pdev));
>>> + if (err)
>>> + return err;
>>> +
>>> + *fw_buf = vmalloc(fw->size);
>>> + *len = fw->size;
>> malloc fails but len is set ?
>>
>> better to set len to 0 on failure
> Will add check and set len to 0 on failure.
>>
>>> + if (*fw_buf)
>>> + memcpy(*fw_buf, fw->data, fw->size);
>>> + else
>>> + err = -ENOMEM;
>>> +
>>> + release_firmware(fw);
>>> + return 0;
>>> +}
>>> +
>>> +static const struct axlf *xmgmt_get_axlf_firmware(struct xmgmt_main *xmm, enum provider_kind kind)
>>> +{
>>> + switch (kind) {
>>> + case XMGMT_BLP:
>>> + return xmm->firmware_blp;
>>> + case XMGMT_PLP:
>>> + return xmm->firmware_plp;
>>> + case XMGMT_ULP:
>>> + return xmm->firmware_ulp;
>>> + default:
>>> + xrt_err(xmm->pdev, "unknown axlf kind: %d", kind);
>>> + return NULL;
>>> + }
>>> +}
>>> +
>> needs a comment that user is responsible for freeing return
> Will add.
>>> +char *xmgmt_get_dtb(struct platform_device *pdev, enum provider_kind kind)
>>> +{
>>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>>> + char *dtb = NULL;
>>> + const struct axlf *provider = xmgmt_get_axlf_firmware(xmm, kind);
>>> + int rc;
>>> +
>>> + if (!provider)
>>> + return dtb;
>>> +
>>> + rc = xrt_xclbin_get_metadata(DEV(pdev), provider, &dtb);
>>> + if (rc)
>>> + xrt_err(pdev, "failed to find dtb: %d", rc);
>>> + return dtb;
>>> +}
>>> +
>> similar caller responsible for freeing
> Will add comment.
>>> +static const char *get_uuid_from_firmware(struct platform_device *pdev, const struct axlf *xclbin)
>>> +{
>>> + const void *uuid = NULL;
>>> + const void *uuiddup = NULL;
>>> + void *dtb = NULL;
>>> + int rc;
>>> +
>>> + rc = xrt_xclbin_get_section(xclbin, PARTITION_METADATA, &dtb, NULL);
>>> + if (rc)
>>> + return NULL;
>>> +
>>> + rc = xrt_md_get_prop(DEV(pdev), dtb, NULL, NULL, XRT_MD_PROP_LOGIC_UUID, &uuid, NULL);
>>> + if (!rc)
>>> + uuiddup = kstrdup(uuid, GFP_KERNEL);
>>> + vfree(dtb);
>>> + return uuiddup;
>>> +}
>>> +
>>> +static bool is_valid_firmware(struct platform_device *pdev,
>>> + const struct axlf *xclbin, size_t fw_len)
>>> +{
>>> + const char *fw_buf = (const char *)xclbin;
>>> + size_t axlflen = xclbin->m_header.m_length;
>>> + const char *fw_uuid;
>>> + char dev_uuid[80];
>>> + int err;
>>> +
>>> + err = get_dev_uuid(pdev, dev_uuid, sizeof(dev_uuid));
>>> + if (err)
>>> + return false;
>>> +
>>> + if (memcmp(fw_buf, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)) != 0) {
>>> + xrt_err(pdev, "unknown fw format");
>>> + return false;
>>> + }
>>> +
>>> + if (axlflen > fw_len) {
>>> + xrt_err(pdev, "truncated fw, length: %zu, expect: %zu", fw_len, axlflen);
>>> + return false;
>>> + }
>>> +
>>> + fw_uuid = get_uuid_from_firmware(pdev, xclbin);
>>> + if (!fw_uuid || strcmp(fw_uuid, dev_uuid) != 0) {
>>> + xrt_err(pdev, "bad fw UUID: %s, expect: %s",
>>> + fw_uuid ? fw_uuid : "<none>", dev_uuid);
>>> + kfree(fw_uuid);
>>> + return false;
>>> + }
>>> +
>>> + kfree(fw_uuid);
>>> + return true;
>>> +}
>>> +
>>> +int xmgmt_get_provider_uuid(struct platform_device *pdev, enum provider_kind kind, uuid_t *uuid)
>>> +{
>>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>>> + const struct axlf *fwbuf;
>>> + const char *fw_uuid;
>>> + int rc = -ENOENT;
>>> +
>>> + mutex_lock(&xmm->busy_mutex);
>>> +
>>> + fwbuf = xmgmt_get_axlf_firmware(xmm, kind);
>>> + if (!fwbuf)
>>> + goto done;
>>> +
>>> + fw_uuid = get_uuid_from_firmware(pdev, fwbuf);
>>> + if (!fw_uuid)
>>> + goto done;
>>> +
>>> + rc = xrt_md_trans_str2uuid(DEV(pdev), fw_uuid, uuid);
>> should this be &fw_uuid ?
> No. fw_uuid points to the uuid string.
ok
>>> + kfree(fw_uuid);
>>> +
>>> +done:
>>> + mutex_unlock(&xmm->busy_mutex);
>>> + return rc;
>>> +}
>>> +
>>> +static int xmgmt_create_blp(struct xmgmt_main *xmm)
>>> +{
>>> + struct platform_device *pdev = xmm->pdev;
>>> + int rc = 0;
>>> + char *dtb = NULL;
>>> + const struct axlf *provider = xmgmt_get_axlf_firmware(xmm, XMGMT_BLP);
>>> +
>>> + dtb = xmgmt_get_dtb(pdev, XMGMT_BLP);
>>> + if (dtb) {
>> not doing any work is ok ?
> Will add check for dtb.
>>> + rc = xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, provider, XMGMT_BLP);
>>> + if (rc) {
>>> + xrt_err(pdev, "failed to process BLP: %d", rc);
>>> + goto failed;
>>> + }
>>> +
>>> + rc = xleaf_create_group(pdev, dtb);
>>> + if (rc < 0)
>> why not (rc) ?
> xleaf_create_group() returns positive group id.
ok
>>> + xrt_err(pdev, "failed to create BLP group: %d", rc);
>>> + else
>>> + rc = 0;
>>> +
>>> + WARN_ON(xmm->blp_intf_uuids);
>> warn but not free ?
> non zero means memory leak. That will be a bug need to be fixed.
>>> + xrt_md_get_intf_uuids(&pdev->dev, dtb, &xmm->blp_intf_uuid_num, NULL);
>>> + if (xmm->blp_intf_uuid_num > 0) {
>>> + xmm->blp_intf_uuids = vzalloc(sizeof(uuid_t) * xmm->blp_intf_uuid_num);
>> unchecked alloc
> Will check.
>>> + xrt_md_get_intf_uuids(&pdev->dev, dtb, &xmm->blp_intf_uuid_num,
>>> + xmm->blp_intf_uuids);
>>> + }
>>> + }
>>> +
>>> +failed:
>>> + vfree(dtb);
>>> + return rc;
>>> +}
>>> +
>>> +static int xmgmt_load_firmware(struct xmgmt_main *xmm)
>>> +{
>>> + struct platform_device *pdev = xmm->pdev;
>>> + int rc;
>>> + size_t fwlen;
>>> +
>>> + rc = load_firmware_from_disk(pdev, &xmm->firmware_blp, &fwlen);
>>> + if (rc != 0)
>>> + rc = load_firmware_from_flash(pdev, &xmm->firmware_blp, &fwlen);
>> this is the function that should be removed
> Sure.
>>> + if (rc == 0 && is_valid_firmware(pdev, xmm->firmware_blp, fwlen))
>>> + (void)xmgmt_create_blp(xmm);
>>> + else
>>> + xrt_err(pdev, "failed to find firmware, giving up: %d", rc);
>>> + return rc;
>>> +}
>>> +
>>> +static void xmgmt_main_event_cb(struct platform_device *pdev, void *arg)
>>> +{
>>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>>> + struct xrt_event *evt = (struct xrt_event *)arg;
>>> + enum xrt_events e = evt->xe_evt;
>>> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
>>> + struct platform_device *leaf;
>>> +
>>> + switch (e) {
>>> + case XRT_EVENT_POST_CREATION: {
>>> + if (id == XRT_SUBDEV_DEVCTL && !xmm->devctl_ready) {
>>> + leaf = xleaf_get_leaf_by_epname(pdev, XRT_MD_NODE_BLP_ROM);
>>> + if (leaf) {
>>> + xmm->devctl_ready = true;
>>> + xleaf_put_leaf(pdev, leaf);
>>> + }
>>> + } else if (id == XRT_SUBDEV_QSPI && !xmm->flash_ready) {
>>> + xmm->flash_ready = true;
>>> + } else {
>>> + break;
>>> + }
>>> +
>>> + if (xmm->devctl_ready)
>>> + (void)xmgmt_load_firmware(xmm);
>>> + break;
>>> + }
>>> + case XRT_EVENT_PRE_REMOVAL:
>>> + break;
>>> + default:
>>> + xrt_dbg(pdev, "ignored event %d", e);
>>> + break;
>>> + }
>>> +}
>>> +
>>> +static int xmgmt_main_probe(struct platform_device *pdev)
>>> +{
>>> + struct xmgmt_main *xmm;
>>> +
>>> + xrt_info(pdev, "probing...");
>>> +
>>> + xmm = devm_kzalloc(DEV(pdev), sizeof(*xmm), GFP_KERNEL);
>>> + if (!xmm)
>>> + return -ENOMEM;
>>> +
>>> + xmm->pdev = pdev;
>>> + xmm->fmgr = xmgmt_fmgr_probe(pdev);
>>> + if (IS_ERR(xmm->fmgr))
>>> + return PTR_ERR(xmm->fmgr);
>>> +
>>> + platform_set_drvdata(pdev, xmm);
>>> + mutex_init(&xmm->busy_mutex);
>>> +
>>> + /* Ready to handle req thru sysfs nodes. */
>>> + if (sysfs_create_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup))
>>> + xrt_err(pdev, "failed to create sysfs group");
>>> + return 0;
>>> +}
>>> +
>>> +static int xmgmt_main_remove(struct platform_device *pdev)
>>> +{
>>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>>> +
>>> + /* By now, group driver should prevent any inter-leaf call. */
>>> +
>>> + xrt_info(pdev, "leaving...");
>>> +
>>> + vfree(xmm->blp_intf_uuids);
>>> + vfree(xmm->firmware_blp);
>>> + vfree(xmm->firmware_plp);
>>> + vfree(xmm->firmware_ulp);
>>> + xmgmt_region_cleanup_all(pdev);
>>> + (void)xmgmt_fmgr_remove(xmm->fmgr);
>>> + (void)sysfs_remove_group(&DEV(pdev)->kobj, &xmgmt_main_attrgroup);
>>> + return 0;
>>> +}
>>> +
>>> +static int
>>> +xmgmt_main_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>>> +{
>>> + struct xmgmt_main *xmm = platform_get_drvdata(pdev);
>>> + int ret = 0;
>>> +
>>> + switch (cmd) {
>>> + case XRT_XLEAF_EVENT:
>>> + xmgmt_main_event_cb(pdev, arg);
>>> + break;
>>> + case XRT_MGMT_MAIN_GET_AXLF_SECTION: {
>>> + struct xrt_mgmt_main_ioctl_get_axlf_section *get =
>>> + (struct xrt_mgmt_main_ioctl_get_axlf_section *)arg;
>>> + const struct axlf *firmware = xmgmt_get_axlf_firmware(xmm, get->xmmigas_axlf_kind);
>>> +
>>> + if (!firmware) {
>>> + ret = -ENOENT;
>>> + } else {
>>> + ret = xrt_xclbin_get_section(firmware, get->xmmigas_section_kind,
>>> + &get->xmmigas_section,
>>> + &get->xmmigas_section_size);
>>> + }
>>> + break;
>>> + }
>>> + case XRT_MGMT_MAIN_GET_VBNV: {
>>> + char **vbnv_p = (char **)arg;
>>> +
>>> + *vbnv_p = xmgmt_get_vbnv(pdev);
>> this can fail
> Will add check.
>>> + break;
>>> + }
>>> + default:
>>> + xrt_err(pdev, "unknown cmd: %d", cmd);
>>> + ret = -EINVAL;
>>> + break;
>>> + }
>>> + return ret;
>>> +}
>>> +
>>> +static int xmgmt_main_open(struct inode *inode, struct file *file)
>>> +{
>>> + struct platform_device *pdev = xleaf_devnode_open(inode);
>>> +
>>> + /* Device may have gone already when we get here. */
>>> + if (!pdev)
>>> + return -ENODEV;
>>> +
>>> + xrt_info(pdev, "opened");
>>> + file->private_data = platform_get_drvdata(pdev);
>>> + return 0;
>>> +}
>>> +
>>> +static int xmgmt_main_close(struct inode *inode, struct file *file)
>>> +{
>>> + struct xmgmt_main *xmm = file->private_data;
>>> +
>>> + xleaf_devnode_close(inode);
>>> +
>>> + xrt_info(xmm->pdev, "closed");
>>> + return 0;
>>> +}
>>> +
>>> +/*
>>> + * Called for xclbin download xclbin load ioctl.
>>> + */
>>> +static int xmgmt_bitstream_axlf_fpga_mgr(struct xmgmt_main *xmm, void *axlf, size_t size)
>>> +{
>>> + int ret;
>>> +
>>> + WARN_ON(!mutex_is_locked(&xmm->busy_mutex));
>>> +
>>> + /*
>>> + * Should any error happens during download, we can't trust
>>> + * the cached xclbin any more.
>>> + */
>>> + vfree(xmm->firmware_ulp);
>>> + xmm->firmware_ulp = NULL;
>>> +
>>> + ret = xmgmt_process_xclbin(xmm->pdev, xmm->fmgr, axlf, XMGMT_ULP);
>>> + if (ret == 0)
>>> + xmm->firmware_ulp = axlf;
>>> +
>>> + return ret;
>>> +}
>>> +
>>> +static int bitstream_axlf_ioctl(struct xmgmt_main *xmm, const void __user *arg)
>>> +{
>>> + void *copy_buffer = NULL;
>>> + size_t copy_buffer_size = 0;
>>> + struct xmgmt_ioc_bitstream_axlf ioc_obj = { 0 };
>>> + struct axlf xclbin_obj = { {0} };
>>> + int ret = 0;
>>> +
>>> + if (copy_from_user((void *)&ioc_obj, arg, sizeof(ioc_obj)))
>>> + return -EFAULT;
>>> + if (copy_from_user((void *)&xclbin_obj, ioc_obj.xclbin, sizeof(xclbin_obj)))
>>> + return -EFAULT;
>>> + if (memcmp(xclbin_obj.m_magic, ICAP_XCLBIN_V2, sizeof(ICAP_XCLBIN_V2)))
>>> + return -EINVAL;
>>> +
>>> + copy_buffer_size = xclbin_obj.m_header.m_length;
>>> + if (copy_buffer_size > MAX_XCLBIN_SIZE)
>>> + return -EINVAL;
>> is there a min size ?
> Will add check.
>>> + copy_buffer = vmalloc(copy_buffer_size);
>>> + if (!copy_buffer)
>>> + return -ENOMEM;
>>> +
>>> + if (copy_from_user(copy_buffer, ioc_obj.xclbin, copy_buffer_size)) {
>>> + vfree(copy_buffer);
>>> + return -EFAULT;
>>> + }
>>> +
>>> + ret = xmgmt_bitstream_axlf_fpga_mgr(xmm, copy_buffer, copy_buffer_size);
>>> + if (ret)
>>> + vfree(copy_buffer);
>>> +
>>> + return ret;
>>> +}
>>> +
>>> +static long xmgmt_main_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>>> +{
>>> + long result = 0;
>>> + struct xmgmt_main *xmm = filp->private_data;
>>> +
>>> + if (_IOC_TYPE(cmd) != XMGMT_IOC_MAGIC)
>>> + return -ENOTTY;
>>> +
>>> + mutex_lock(&xmm->busy_mutex);
>>> +
>>> + xrt_info(xmm->pdev, "ioctl cmd %d, arg %ld", cmd, arg);
>>> + switch (cmd) {
>>> + case XMGMT_IOCICAPDOWNLOAD_AXLF:
>>> + result = bitstream_axlf_ioctl(xmm, (const void __user *)arg);
>>> + break;
>>> + default:
>>> + result = -ENOTTY;
>>> + break;
>>> + }
>>> +
>>> + mutex_unlock(&xmm->busy_mutex);
>>> + return result;
>>> +}
>>> +
>>> +static struct xrt_subdev_endpoints xrt_mgmt_main_endpoints[] = {
>>> + {
>>> + .xse_names = (struct xrt_subdev_ep_names []){
>>> + { .ep_name = XRT_MD_NODE_MGMT_MAIN },
>>> + { NULL },
>>> + },
>>> + .xse_min_ep = 1,
>>> + },
>>> + { 0 },
>>> +};
>>> +
>>> +static struct xrt_subdev_drvdata xmgmt_main_data = {
>>> + .xsd_dev_ops = {
>>> + .xsd_ioctl = xmgmt_main_leaf_ioctl,
>>> + },
>>> + .xsd_file_ops = {
>>> + .xsf_ops = {
>>> + .owner = THIS_MODULE,
>>> + .open = xmgmt_main_open,
>>> + .release = xmgmt_main_close,
>>> + .unlocked_ioctl = xmgmt_main_ioctl,
>>> + },
>>> + .xsf_dev_name = "xmgmt",
>>> + },
>>> +};
>>> +
>>> +static const struct platform_device_id xmgmt_main_id_table[] = {
>>> + { XMGMT_MAIN, (kernel_ulong_t)&xmgmt_main_data },
>>> + { },
>>> +};
>>> +
>>> +static struct platform_driver xmgmt_main_driver = {
>>> + .driver = {
>>> + .name = XMGMT_MAIN,
>>> + },
>>> + .probe = xmgmt_main_probe,
>>> + .remove = xmgmt_main_remove,
>>> + .id_table = xmgmt_main_id_table,
>>> +};
>>> +
>>> +int xmgmt_main_register_leaf(void)
>>> +{
>>> + return xleaf_register_driver(XRT_SUBDEV_MGMT_MAIN,
>>> + &xmgmt_main_driver, xrt_mgmt_main_endpoints);
>>> +}
>>> +
>>> +void xmgmt_main_unregister_leaf(void)
>>> +{
>>> + xleaf_unregister_driver(XRT_SUBDEV_MGMT_MAIN);
>>> +}
>>> diff --git a/include/uapi/linux/xrt/xmgmt-ioctl.h b/include/uapi/linux/xrt/xmgmt-ioctl.h
>>> new file mode 100644
>>> index 000000000000..15834476f3b4
>>> --- /dev/null
>>> +++ b/include/uapi/linux/xrt/xmgmt-ioctl.h
>>> @@ -0,0 +1,46 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>>> +/*
>>> + * Copyright (C) 2015-2021, Xilinx Inc
>>> + *
>>> + */
>>> +
>>> +/**
>>> + * DOC: PCIe Kernel Driver for Managament Physical Function
>>> + * Interfaces exposed by *xclmgmt* driver are defined in file, *mgmt-ioctl.h*.
>>> + * Core functionality provided by *xmgmt* driver is described in the following table:
>>> + *
>>> + * =========== ============================== ==================================
>>> + * Functionality ioctl request code data format
>>> + * =========== ============================== ==================================
>>> + * 1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
>>> + * =========== ============================== ==================================
>>> + */
>>> +
>>> +#ifndef _XMGMT_IOCTL_H_
>>> +#define _XMGMT_IOCTL_H_
>>> +
>>> +#include <linux/ioctl.h>
>>> +
>>> +#define XMGMT_IOC_MAGIC 'X'
>>> +#define XMGMT_IOC_ICAP_DOWNLOAD_AXLF 0x6
>>> +
>>> +/**
>>> + * struct xmgmt_ioc_bitstream_axlf - load xclbin (AXLF) device image
>>> + * used with XMGMT_IOCICAPDOWNLOAD_AXLF ioctl
>>> + *
>>> + * @xclbin: Pointer to user's xclbin structure in memory
>>> + */
>>> +struct xmgmt_ioc_bitstream_axlf {
>>> + struct axlf *xclbin;
>> where is struct axlf defined ?
> It is defined in include/uapi/linux/xrt/xclbin.h
ok, thanks.
Tom
>
> Thanks,
> Lizhi
>>
>> Tom
>>
>>> +};
>>> +
>>> +#define XMGMT_IOCICAPDOWNLOAD_AXLF \
>>> + _IOW(XMGMT_IOC_MAGIC, XMGMT_IOC_ICAP_DOWNLOAD_AXLF, struct xmgmt_ioc_bitstream_axlf)
>>> +
>>> +/*
>>> + * The following definitions are for binary compatibility with classic XRT management driver
>>> + */
>>> +#define XCLMGMT_IOCICAPDOWNLOAD_AXLF XMGMT_IOCICAPDOWNLOAD_AXLF
>>> +#define xclmgmt_ioc_bitstream_axlf xmgmt_ioc_bitstream_axlf
>>> +
>>> +#endif
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 09/18] fpga: xrt: fpga-mgr and region implementation for xclbin download
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (7 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 08/18] fpga: xrt: main platform driver for management function device Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-28 16:36 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 10/18] fpga: xrt: VSEC platform driver Lizhi Hou
` (10 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
fpga-mgr and region implementation for xclbin download which will be
called from main platform driver
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/mgmt/fmgr-drv.c | 187 +++++++++++
drivers/fpga/xrt/mgmt/fmgr.h | 28 ++
drivers/fpga/xrt/mgmt/main-region.c | 471 ++++++++++++++++++++++++++++
3 files changed, 686 insertions(+)
create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
diff --git a/drivers/fpga/xrt/mgmt/fmgr-drv.c b/drivers/fpga/xrt/mgmt/fmgr-drv.c
new file mode 100644
index 000000000000..a44d35ecdb60
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/fmgr-drv.c
@@ -0,0 +1,187 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * FPGA Manager Support for Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors: Sonal.Santan@xilinx.com
+ */
+
+#include <linux/cred.h>
+#include <linux/efi.h>
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/platform_device.h>
+#include <linux/module.h>
+#include <linux/vmalloc.h>
+
+#include "xclbin-helper.h"
+#include "xleaf.h"
+#include "fmgr.h"
+#include "xleaf/axigate.h"
+#include "xleaf/icap.h"
+#include "main-impl.h"
+
+struct xfpga_class {
+ const struct platform_device *pdev;
+ char name[64];
+};
+
+/*
+ * xclbin download plumbing -- find the download subsystem, ICAP and
+ * pass the xclbin for heavy lifting
+ */
+static int xmgmt_download_bitstream(struct platform_device *pdev,
+ const struct axlf *xclbin)
+
+{
+ struct hw_icap_bit_header bit_header = { 0 };
+ struct platform_device *icap_leaf = NULL;
+ struct xrt_icap_ioctl_wr arg;
+ char *bitstream = NULL;
+ u64 bit_len;
+ int ret;
+
+ ret = xrt_xclbin_get_section(xclbin, BITSTREAM, (void **)&bitstream, &bit_len);
+ if (ret || !bitstream) {
+ xrt_err(pdev, "bitstream not found");
+ return -ENOENT;
+ }
+ ret = xrt_xclbin_parse_bitstream_header(bitstream,
+ DMA_HWICAP_BITFILE_BUFFER_SIZE,
+ &bit_header);
+ if (ret) {
+ ret = -EINVAL;
+ xrt_err(pdev, "invalid bitstream header");
+ goto done;
+ }
+ if (bit_header.header_length + bit_header.bitstream_length > bit_len) {
+ ret = -EINVAL;
+ xrt_err(pdev, "invalid bitstream length. header %d, bitstream %d, section len %lld",
+ bit_header.header_length, bit_header.bitstream_length, bit_len);
+ goto done;
+ }
+
+ icap_leaf = xleaf_get_leaf_by_id(pdev, XRT_SUBDEV_ICAP, PLATFORM_DEVID_NONE);
+ if (!icap_leaf) {
+ ret = -ENODEV;
+ xrt_err(pdev, "icap does not exist");
+ xrt_xclbin_free_header(&bit_header);
+ goto done;
+ }
+ arg.xiiw_bit_data = bitstream + bit_header.header_length;
+ arg.xiiw_data_len = bit_header.bitstream_length;
+ ret = xleaf_ioctl(icap_leaf, XRT_ICAP_WRITE, &arg);
+ if (ret)
+ xrt_err(pdev, "write bitstream failed, ret = %d", ret);
+
+ xrt_xclbin_free_header(&bit_header);
+done:
+ if (icap_leaf)
+ xleaf_put_leaf(pdev, icap_leaf);
+ vfree(bitstream);
+
+ return ret;
+}
+
+/*
+ * There is no HW prep work we do here since we need the full
+ * xclbin for its sanity check.
+ */
+static int xmgmt_pr_write_init(struct fpga_manager *mgr,
+ struct fpga_image_info *info,
+ const char *buf, size_t count)
+{
+ const struct axlf *bin = (const struct axlf *)buf;
+ struct xfpga_class *obj = mgr->priv;
+
+ if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
+ xrt_info(obj->pdev, "%s only supports partial reconfiguration\n", obj->name);
+ return -EINVAL;
+ }
+
+ if (count < sizeof(struct axlf))
+ return -EINVAL;
+
+ if (count > bin->m_header.m_length)
+ return -EINVAL;
+
+ xrt_info(obj->pdev, "Prepare download of xclbin %pUb of length %lld B",
+ &bin->m_header.uuid, bin->m_header.m_length);
+
+ return 0;
+}
+
+/*
+ * The implementation requries full xclbin image before we can start
+ * programming the hardware via ICAP subsystem. Full image is required
+ * for checking the validity of xclbin and walking the sections to
+ * discover the bitstream.
+ */
+static int xmgmt_pr_write(struct fpga_manager *mgr,
+ const char *buf, size_t count)
+{
+ const struct axlf *bin = (const struct axlf *)buf;
+ struct xfpga_class *obj = mgr->priv;
+
+ if (bin->m_header.m_length != count)
+ return -EINVAL;
+
+ return xmgmt_download_bitstream((void *)obj->pdev, bin);
+}
+
+static int xmgmt_pr_write_complete(struct fpga_manager *mgr,
+ struct fpga_image_info *info)
+{
+ const struct axlf *bin = (const struct axlf *)info->buf;
+ struct xfpga_class *obj = mgr->priv;
+
+ xrt_info(obj->pdev, "Finished download of xclbin %pUb",
+ &bin->m_header.uuid);
+ return 0;
+}
+
+static enum fpga_mgr_states xmgmt_pr_state(struct fpga_manager *mgr)
+{
+ return FPGA_MGR_STATE_UNKNOWN;
+}
+
+static const struct fpga_manager_ops xmgmt_pr_ops = {
+ .initial_header_size = sizeof(struct axlf),
+ .write_init = xmgmt_pr_write_init,
+ .write = xmgmt_pr_write,
+ .write_complete = xmgmt_pr_write_complete,
+ .state = xmgmt_pr_state,
+};
+
+struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev)
+{
+ struct xfpga_class *obj = devm_kzalloc(DEV(pdev), sizeof(struct xfpga_class),
+ GFP_KERNEL);
+ struct fpga_manager *fmgr = NULL;
+ int ret = 0;
+
+ if (!obj)
+ return ERR_PTR(-ENOMEM);
+
+ snprintf(obj->name, sizeof(obj->name), "Xilinx Alveo FPGA Manager");
+ obj->pdev = pdev;
+ fmgr = fpga_mgr_create(&pdev->dev,
+ obj->name,
+ &xmgmt_pr_ops,
+ obj);
+ if (!fmgr)
+ return ERR_PTR(-ENOMEM);
+
+ ret = fpga_mgr_register(fmgr);
+ if (ret) {
+ fpga_mgr_free(fmgr);
+ return ERR_PTR(ret);
+ }
+ return fmgr;
+}
+
+int xmgmt_fmgr_remove(struct fpga_manager *fmgr)
+{
+ fpga_mgr_unregister(fmgr);
+ return 0;
+}
diff --git a/drivers/fpga/xrt/mgmt/fmgr.h b/drivers/fpga/xrt/mgmt/fmgr.h
new file mode 100644
index 000000000000..e1fc033e2542
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/fmgr.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors: Sonal.Santan@xilinx.com
+ */
+
+#ifndef _XMGMT_FMGR_H_
+#define _XMGMT_FMGR_H_
+
+#include <linux/fpga/fpga-mgr.h>
+#include <linux/mutex.h>
+
+#include <linux/xrt/xclbin.h>
+
+enum xfpga_sec_level {
+ XFPGA_SEC_NONE = 0,
+ XFPGA_SEC_DEDICATE,
+ XFPGA_SEC_SYSTEM,
+ XFPGA_SEC_MAX = XFPGA_SEC_SYSTEM,
+};
+
+struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev);
+int xmgmt_fmgr_remove(struct fpga_manager *fmgr);
+
+#endif
diff --git a/drivers/fpga/xrt/mgmt/main-region.c b/drivers/fpga/xrt/mgmt/main-region.c
new file mode 100644
index 000000000000..9779693fe7ae
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/main-region.c
@@ -0,0 +1,471 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * FPGA Region Support for Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
+ *
+ * Authors: Lizhi.Hou@xilinx.com
+ */
+
+#include <linux/uuid.h>
+#include <linux/fpga/fpga-bridge.h>
+#include <linux/fpga/fpga-region.h>
+#include "metadata.h"
+#include "xleaf.h"
+#include "xleaf/axigate.h"
+#include "xclbin-helper.h"
+#include "main-impl.h"
+
+struct xmgmt_bridge {
+ struct platform_device *pdev;
+ const char *axigate_name;
+};
+
+struct xmgmt_region {
+ struct platform_device *pdev;
+ struct fpga_region *fregion;
+ uuid_t intf_uuid;
+ struct fpga_bridge *fbridge;
+ int grp_inst;
+ uuid_t dep_uuid;
+ struct list_head list;
+};
+
+struct xmgmt_region_match_arg {
+ struct platform_device *pdev;
+ uuid_t *uuids;
+ u32 uuid_num;
+};
+
+static int xmgmt_br_enable_set(struct fpga_bridge *bridge, bool enable)
+{
+ struct xmgmt_bridge *br_data = (struct xmgmt_bridge *)bridge->priv;
+ struct platform_device *axigate_leaf;
+ int rc;
+
+ axigate_leaf = xleaf_get_leaf_by_epname(br_data->pdev, br_data->axigate_name);
+ if (!axigate_leaf) {
+ xrt_err(br_data->pdev, "failed to get leaf %s",
+ br_data->axigate_name);
+ return -ENOENT;
+ }
+
+ if (enable)
+ rc = xleaf_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
+ else
+ rc = xleaf_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE, NULL);
+
+ if (rc) {
+ xrt_err(br_data->pdev, "failed to %s gate %s, rc %d",
+ (enable ? "free" : "freeze"), br_data->axigate_name,
+ rc);
+ }
+
+ xleaf_put_leaf(br_data->pdev, axigate_leaf);
+
+ return rc;
+}
+
+const struct fpga_bridge_ops xmgmt_bridge_ops = {
+ .enable_set = xmgmt_br_enable_set
+};
+
+static void xmgmt_destroy_bridge(struct fpga_bridge *br)
+{
+ struct xmgmt_bridge *br_data = br->priv;
+
+ if (!br_data)
+ return;
+
+ xrt_info(br_data->pdev, "destroy fpga bridge %s", br_data->axigate_name);
+ fpga_bridge_unregister(br);
+
+ devm_kfree(DEV(br_data->pdev), br_data);
+
+ fpga_bridge_free(br);
+}
+
+static struct fpga_bridge *xmgmt_create_bridge(struct platform_device *pdev,
+ char *dtb)
+{
+ struct xmgmt_bridge *br_data;
+ struct fpga_bridge *br = NULL;
+ const char *gate;
+ int rc;
+
+ br_data = devm_kzalloc(DEV(pdev), sizeof(*br_data), GFP_KERNEL);
+ if (!br_data)
+ return NULL;
+ br_data->pdev = pdev;
+
+ br_data->axigate_name = XRT_MD_NODE_GATE_ULP;
+ rc = xrt_md_find_endpoint(&pdev->dev, dtb, XRT_MD_NODE_GATE_ULP,
+ NULL, &gate);
+ if (rc) {
+ br_data->axigate_name = XRT_MD_NODE_GATE_PLP;
+ rc = xrt_md_find_endpoint(&pdev->dev, dtb, XRT_MD_NODE_GATE_PLP,
+ NULL, &gate);
+ }
+ if (rc) {
+ xrt_err(pdev, "failed to get axigate, rc %d", rc);
+ goto failed;
+ }
+
+ br = fpga_bridge_create(DEV(pdev), br_data->axigate_name,
+ &xmgmt_bridge_ops, br_data);
+ if (!br) {
+ xrt_err(pdev, "failed to create bridge");
+ goto failed;
+ }
+
+ rc = fpga_bridge_register(br);
+ if (rc) {
+ xrt_err(pdev, "failed to register bridge, rc %d", rc);
+ goto failed;
+ }
+
+ xrt_info(pdev, "created fpga bridge %s", br_data->axigate_name);
+
+ return br;
+
+failed:
+ if (br)
+ fpga_bridge_free(br);
+ if (br_data)
+ devm_kfree(DEV(pdev), br_data);
+
+ return NULL;
+}
+
+static void xmgmt_destroy_region(struct fpga_region *re)
+{
+ struct xmgmt_region *r_data = re->priv;
+
+ xrt_info(r_data->pdev, "destroy fpga region %llx%llx",
+ re->compat_id->id_l, re->compat_id->id_h);
+
+ fpga_region_unregister(re);
+
+ if (r_data->grp_inst > 0)
+ xleaf_destroy_group(r_data->pdev, r_data->grp_inst);
+
+ if (r_data->fbridge)
+ xmgmt_destroy_bridge(r_data->fbridge);
+
+ if (r_data->fregion->info) {
+ fpga_image_info_free(r_data->fregion->info);
+ r_data->fregion->info = NULL;
+ }
+
+ fpga_region_free(re);
+
+ devm_kfree(DEV(r_data->pdev), r_data);
+}
+
+static int xmgmt_region_match(struct device *dev, const void *data)
+{
+ const struct xmgmt_region_match_arg *arg = data;
+ const struct fpga_region *match_re;
+ int i;
+
+ if (dev->parent != &arg->pdev->dev)
+ return false;
+
+ match_re = to_fpga_region(dev);
+ /*
+ * The device tree provides both parent and child uuids for an
+ * xclbin in one array. Here we try both uuids to see if it matches
+ * with target region's compat_id. Strictly speaking we should
+ * only match xclbin's parent uuid with target region's compat_id
+ * but given the uuids by design are unique comparing with both
+ * does not hurt.
+ */
+ for (i = 0; i < arg->uuid_num; i++) {
+ if (!memcmp(match_re->compat_id, &arg->uuids[i],
+ sizeof(*match_re->compat_id)))
+ return true;
+ }
+
+ return false;
+}
+
+static int xmgmt_region_match_base(struct device *dev, const void *data)
+{
+ const struct xmgmt_region_match_arg *arg = data;
+ const struct fpga_region *match_re;
+ const struct xmgmt_region *r_data;
+
+ if (dev->parent != &arg->pdev->dev)
+ return false;
+
+ match_re = to_fpga_region(dev);
+ r_data = match_re->priv;
+ if (uuid_is_null(&r_data->dep_uuid))
+ return true;
+
+ return false;
+}
+
+static int xmgmt_region_match_by_depuuid(struct device *dev, const void *data)
+{
+ const struct xmgmt_region_match_arg *arg = data;
+ const struct fpga_region *match_re;
+ const struct xmgmt_region *r_data;
+
+ if (dev->parent != &arg->pdev->dev)
+ return false;
+
+ match_re = to_fpga_region(dev);
+ r_data = match_re->priv;
+ if (!memcmp(&r_data->dep_uuid, arg->uuids, sizeof(uuid_t)))
+ return true;
+
+ return false;
+}
+
+static void xmgmt_region_cleanup(struct fpga_region *re)
+{
+ struct xmgmt_region *r_data = re->priv, *temp;
+ struct platform_device *pdev = r_data->pdev;
+ struct fpga_region *match_re = NULL;
+ struct device *start_dev = NULL;
+ struct xmgmt_region_match_arg arg;
+ LIST_HEAD(free_list);
+
+ list_add_tail(&r_data->list, &free_list);
+ arg.pdev = pdev;
+ arg.uuid_num = 1;
+
+ while (!r_data) {
+ arg.uuids = (uuid_t *)r_data->fregion->compat_id;
+ match_re = fpga_region_class_find(start_dev, &arg,
+ xmgmt_region_match_by_depuuid);
+ if (match_re) {
+ r_data = match_re->priv;
+ list_add_tail(&r_data->list, &free_list);
+ start_dev = &match_re->dev;
+ put_device(&match_re->dev);
+ continue;
+ }
+
+ r_data = list_is_last(&r_data->list, &free_list) ? NULL :
+ list_next_entry(r_data, list);
+ start_dev = NULL;
+ }
+
+ list_for_each_entry_safe_reverse(r_data, temp, &free_list, list) {
+ if (list_is_first(&r_data->list, &free_list)) {
+ if (r_data->grp_inst > 0) {
+ xleaf_destroy_group(pdev, r_data->grp_inst);
+ r_data->grp_inst = -1;
+ }
+ if (r_data->fregion->info) {
+ fpga_image_info_free(r_data->fregion->info);
+ r_data->fregion->info = NULL;
+ }
+ continue;
+ }
+ xmgmt_destroy_region(r_data->fregion);
+ }
+}
+
+void xmgmt_region_cleanup_all(struct platform_device *pdev)
+{
+ struct fpga_region *base_re;
+ struct xmgmt_region_match_arg arg;
+
+ arg.pdev = pdev;
+
+ for (base_re = fpga_region_class_find(NULL, &arg, xmgmt_region_match_base);
+ base_re;
+ base_re = fpga_region_class_find(NULL, &arg, xmgmt_region_match_base)) {
+ put_device(&base_re->dev);
+
+ xmgmt_region_cleanup(base_re);
+ xmgmt_destroy_region(base_re);
+ }
+}
+
+/*
+ * Program a given region with given xclbin image. Bring up the subdevs and the
+ * group object to contain the subdevs.
+ */
+static int xmgmt_region_program(struct fpga_region *re, const void *xclbin, char *dtb)
+{
+ struct xmgmt_region *r_data = re->priv;
+ struct platform_device *pdev = r_data->pdev;
+ struct fpga_image_info *info;
+ const struct axlf *xclbin_obj = xclbin;
+ int rc;
+
+ info = fpga_image_info_alloc(&pdev->dev);
+ if (!info)
+ return -ENOMEM;
+
+ info->buf = xclbin;
+ info->count = xclbin_obj->m_header.m_length;
+ info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
+ re->info = info;
+ rc = fpga_region_program_fpga(re);
+ if (rc) {
+ xrt_err(pdev, "programming xclbin failed, rc %d", rc);
+ return rc;
+ }
+
+ /* free bridges to allow reprogram */
+ if (re->get_bridges)
+ fpga_bridges_put(&re->bridge_list);
+
+ /*
+ * Next bringup the subdevs for this region which will be managed by
+ * its own group object.
+ */
+ r_data->grp_inst = xleaf_create_group(pdev, dtb);
+ if (r_data->grp_inst < 0) {
+ xrt_err(pdev, "failed to create group, rc %d",
+ r_data->grp_inst);
+ rc = r_data->grp_inst;
+ return rc;
+ }
+
+ rc = xleaf_wait_for_group_bringup(pdev);
+ if (rc)
+ xrt_err(pdev, "group bringup failed, rc %d", rc);
+ return rc;
+}
+
+static int xmgmt_get_bridges(struct fpga_region *re)
+{
+ struct xmgmt_region *r_data = re->priv;
+ struct device *dev = &r_data->pdev->dev;
+
+ return fpga_bridge_get_to_list(dev, re->info, &re->bridge_list);
+}
+
+/*
+ * Program/create FPGA regions based on input xclbin file. This is key function
+ * stitching the flow together:
+ * 1. Identify a matching existing region for this xclbin
+ * 2. Tear down any previous objects for the found region
+ * 3. Program this region with input xclbin
+ * 4. Iterate over this region's interface uuids to determine if it defines any
+ * child region. Create fpga_region for the child region.
+ */
+int xmgmt_process_xclbin(struct platform_device *pdev,
+ struct fpga_manager *fmgr,
+ const struct axlf *xclbin,
+ enum provider_kind kind)
+{
+ struct fpga_region *re, *compat_re = NULL;
+ struct xmgmt_region_match_arg arg;
+ struct xmgmt_region *r_data;
+ char *dtb = NULL;
+ int rc, i;
+
+ rc = xrt_xclbin_get_metadata(DEV(pdev), xclbin, &dtb);
+ if (rc) {
+ xrt_err(pdev, "failed to get dtb: %d", rc);
+ goto failed;
+ }
+
+ xrt_md_get_intf_uuids(DEV(pdev), dtb, &arg.uuid_num, NULL);
+ if (arg.uuid_num == 0) {
+ xrt_err(pdev, "failed to get intf uuid");
+ rc = -EINVAL;
+ goto failed;
+ }
+ arg.uuids = vzalloc(sizeof(uuid_t) * arg.uuid_num);
+ if (!arg.uuids) {
+ rc = -ENOMEM;
+ goto failed;
+ }
+ arg.pdev = pdev;
+
+ xrt_md_get_intf_uuids(DEV(pdev), dtb, &arg.uuid_num, arg.uuids);
+
+ /* if this is not base firmware, search for a compatible region */
+ if (kind != XMGMT_BLP) {
+ compat_re = fpga_region_class_find(NULL, &arg,
+ xmgmt_region_match);
+ if (!compat_re) {
+ xrt_err(pdev, "failed to get compatible region");
+ rc = -ENOENT;
+ goto failed;
+ }
+
+ xmgmt_region_cleanup(compat_re);
+
+ rc = xmgmt_region_program(compat_re, xclbin, dtb);
+ if (rc) {
+ xrt_err(pdev, "failed to program region");
+ goto failed;
+ }
+ }
+
+ /* create all the new regions contained in this xclbin */
+ for (i = 0; i < arg.uuid_num; i++) {
+ if (compat_re && !memcmp(compat_re->compat_id, &arg.uuids[i],
+ sizeof(*compat_re->compat_id)))
+ /* region for this interface already exists */
+ continue;
+ re = fpga_region_create(DEV(pdev), fmgr, xmgmt_get_bridges);
+ if (!re) {
+ xrt_err(pdev, "failed to create fpga region");
+ rc = -EFAULT;
+ goto failed;
+ }
+ r_data = devm_kzalloc(DEV(pdev), sizeof(*r_data), GFP_KERNEL);
+ if (!r_data) {
+ rc = -ENOMEM;
+ fpga_region_free(re);
+ goto failed;
+ }
+ r_data->pdev = pdev;
+ r_data->fregion = re;
+ r_data->grp_inst = -1;
+ memcpy(&r_data->intf_uuid, &arg.uuids[i],
+ sizeof(r_data->intf_uuid));
+ if (compat_re) {
+ memcpy(&r_data->dep_uuid, compat_re->compat_id,
+ sizeof(r_data->intf_uuid));
+ }
+ r_data->fbridge = xmgmt_create_bridge(pdev, dtb);
+ if (!r_data->fbridge) {
+ xrt_err(pdev, "failed to create fpga bridge");
+ rc = -EFAULT;
+ devm_kfree(DEV(pdev), r_data);
+ fpga_region_free(re);
+ goto failed;
+ }
+
+ re->compat_id = (struct fpga_compat_id *)&r_data->intf_uuid;
+ re->priv = r_data;
+
+ rc = fpga_region_register(re);
+ if (rc) {
+ xrt_err(pdev, "failed to register fpga region");
+ xmgmt_destroy_bridge(r_data->fbridge);
+ fpga_region_free(re);
+ devm_kfree(DEV(pdev), r_data);
+ goto failed;
+ }
+
+ xrt_info(pdev, "created fpga region %llx%llx",
+ re->compat_id->id_l, re->compat_id->id_h);
+ }
+
+failed:
+ if (compat_re)
+ put_device(&compat_re->dev);
+
+ if (rc) {
+ if (compat_re)
+ xmgmt_region_cleanup(compat_re);
+ }
+
+ if (dtb)
+ vfree(dtb);
+
+ return rc;
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 09/18] fpga: xrt: fpga-mgr and region implementation for xclbin download
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 09/18] fpga: xrt: fpga-mgr and region implementation for xclbin download Lizhi Hou
@ 2021-02-28 16:36 ` Tom Rix
2021-03-04 17:50 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-28 16:36 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> fpga-mgr and region implementation for xclbin download which will be
> called from main platform driver
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/mgmt/fmgr-drv.c | 187 +++++++++++
> drivers/fpga/xrt/mgmt/fmgr.h | 28 ++
> drivers/fpga/xrt/mgmt/main-region.c | 471 ++++++++++++++++++++++++++++
> 3 files changed, 686 insertions(+)
> create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
> create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
> create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
>
> diff --git a/drivers/fpga/xrt/mgmt/fmgr-drv.c b/drivers/fpga/xrt/mgmt/fmgr-drv.c
> new file mode 100644
> index 000000000000..a44d35ecdb60
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/fmgr-drv.c
> @@ -0,0 +1,187 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * FPGA Manager Support for Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors: Sonal.Santan@xilinx.com
> + */
> +
> +#include <linux/cred.h>
> +#include <linux/efi.h>
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/platform_device.h>
> +#include <linux/module.h>
> +#include <linux/vmalloc.h>
> +
> +#include "xclbin-helper.h"
> +#include "xleaf.h"
> +#include "fmgr.h"
> +#include "xleaf/axigate.h"
> +#include "xleaf/icap.h"
> +#include "main-impl.h"
> +
> +struct xfpga_class {
> + const struct platform_device *pdev;
> + char name[64];
> +};
> +
> +/*
> + * xclbin download plumbing -- find the download subsystem, ICAP and
> + * pass the xclbin for heavy lifting
> + */
> +static int xmgmt_download_bitstream(struct platform_device *pdev,
> + const struct axlf *xclbin)
> +
> +{
> + struct hw_icap_bit_header bit_header = { 0 };
> + struct platform_device *icap_leaf = NULL;
> + struct xrt_icap_ioctl_wr arg;
> + char *bitstream = NULL;
> + u64 bit_len;
> + int ret;
> +
> + ret = xrt_xclbin_get_section(xclbin, BITSTREAM, (void **)&bitstream, &bit_len);
> + if (ret || !bitstream) {
!bitstream check is unneeded
> + xrt_err(pdev, "bitstream not found");
> + return -ENOENT;
> + }
> + ret = xrt_xclbin_parse_bitstream_header(bitstream,
> + DMA_HWICAP_BITFILE_BUFFER_SIZE,
> + &bit_header);
> + if (ret) {
> + ret = -EINVAL;
> + xrt_err(pdev, "invalid bitstream header");
> + goto done;
> + }
> + if (bit_header.header_length + bit_header.bitstream_length > bit_len) {
> + ret = -EINVAL;
> + xrt_err(pdev, "invalid bitstream length. header %d, bitstream %d, section len %lld",
> + bit_header.header_length, bit_header.bitstream_length, bit_len);
> + goto done;
> + }
> +
> + icap_leaf = xleaf_get_leaf_by_id(pdev, XRT_SUBDEV_ICAP, PLATFORM_DEVID_NONE);
> + if (!icap_leaf) {
> + ret = -ENODEV;
> + xrt_err(pdev, "icap does not exist");
> + xrt_xclbin_free_header(&bit_header);
> + goto done;
> + }
> + arg.xiiw_bit_data = bitstream + bit_header.header_length;
> + arg.xiiw_data_len = bit_header.bitstream_length;
> + ret = xleaf_ioctl(icap_leaf, XRT_ICAP_WRITE, &arg);
> + if (ret)
> + xrt_err(pdev, "write bitstream failed, ret = %d", ret);
> +
> + xrt_xclbin_free_header(&bit_header);
memory leak when something fails and goto done's
> +done:
previous general problem, use mutliple label in error handling blocks
> + if (icap_leaf)
> + xleaf_put_leaf(pdev, icap_leaf);
> + vfree(bitstream);
> +
> + return ret;
> +}
> +
> +/*
> + * There is no HW prep work we do here since we need the full
> + * xclbin for its sanity check.
> + */
> +static int xmgmt_pr_write_init(struct fpga_manager *mgr,
> + struct fpga_image_info *info,
> + const char *buf, size_t count)
> +{
> + const struct axlf *bin = (const struct axlf *)buf;
> + struct xfpga_class *obj = mgr->priv;
> +
> + if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
> + xrt_info(obj->pdev, "%s only supports partial reconfiguration\n", obj->name);
> + return -EINVAL;
> + }
> +
> + if (count < sizeof(struct axlf))
> + return -EINVAL;
> +
> + if (count > bin->m_header.m_length)
> + return -EINVAL;
> +
> + xrt_info(obj->pdev, "Prepare download of xclbin %pUb of length %lld B",
> + &bin->m_header.uuid, bin->m_header.m_length);
> +
> + return 0;
> +}
> +
> +/*
> + * The implementation requries full xclbin image before we can start
> + * programming the hardware via ICAP subsystem. Full image is required
The full image
> + * for checking the validity of xclbin and walking the sections to
> + * discover the bitstream.
> + */
> +static int xmgmt_pr_write(struct fpga_manager *mgr,
> + const char *buf, size_t count)
> +{
> + const struct axlf *bin = (const struct axlf *)buf;
> + struct xfpga_class *obj = mgr->priv;
> +
> + if (bin->m_header.m_length != count)
> + return -EINVAL;
> +
> + return xmgmt_download_bitstream((void *)obj->pdev, bin);
> +}
> +
> +static int xmgmt_pr_write_complete(struct fpga_manager *mgr,
> + struct fpga_image_info *info)
> +{
> + const struct axlf *bin = (const struct axlf *)info->buf;
> + struct xfpga_class *obj = mgr->priv;
> +
> + xrt_info(obj->pdev, "Finished download of xclbin %pUb",
> + &bin->m_header.uuid);
> + return 0;
> +}
> +
> +static enum fpga_mgr_states xmgmt_pr_state(struct fpga_manager *mgr)
> +{
> + return FPGA_MGR_STATE_UNKNOWN;
why just this state ?
> +}
> +
> +static const struct fpga_manager_ops xmgmt_pr_ops = {
> + .initial_header_size = sizeof(struct axlf),
> + .write_init = xmgmt_pr_write_init,
> + .write = xmgmt_pr_write,
> + .write_complete = xmgmt_pr_write_complete,
> + .state = xmgmt_pr_state,
> +};
> +
> +struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev)
> +{
> + struct xfpga_class *obj = devm_kzalloc(DEV(pdev), sizeof(struct xfpga_class),
> + GFP_KERNEL);
> + struct fpga_manager *fmgr = NULL;
> + int ret = 0;
> +
> + if (!obj)
> + return ERR_PTR(-ENOMEM);
> +
> + snprintf(obj->name, sizeof(obj->name), "Xilinx Alveo FPGA Manager");
> + obj->pdev = pdev;
> + fmgr = fpga_mgr_create(&pdev->dev,
> + obj->name,
> + &xmgmt_pr_ops,
> + obj);
> + if (!fmgr)
> + return ERR_PTR(-ENOMEM);
> +
> + ret = fpga_mgr_register(fmgr);
> + if (ret) {
> + fpga_mgr_free(fmgr);
> + return ERR_PTR(ret);
> + }
> + return fmgr;
> +}
> +
> +int xmgmt_fmgr_remove(struct fpga_manager *fmgr)
> +{
> + fpga_mgr_unregister(fmgr);
> + return 0;
> +}
> diff --git a/drivers/fpga/xrt/mgmt/fmgr.h b/drivers/fpga/xrt/mgmt/fmgr.h
> new file mode 100644
> index 000000000000..e1fc033e2542
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/fmgr.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors: Sonal.Santan@xilinx.com
> + */
> +
> +#ifndef _XMGMT_FMGR_H_
> +#define _XMGMT_FMGR_H_
> +
> +#include <linux/fpga/fpga-mgr.h>
> +#include <linux/mutex.h>
> +
> +#include <linux/xrt/xclbin.h>
> +
> +enum xfpga_sec_level {
> + XFPGA_SEC_NONE = 0,
> + XFPGA_SEC_DEDICATE,
> + XFPGA_SEC_SYSTEM,
> + XFPGA_SEC_MAX = XFPGA_SEC_SYSTEM,
> +};
This enum is not used, remove.
> +
> +struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev);
> +int xmgmt_fmgr_remove(struct fpga_manager *fmgr);
> +
> +#endif
> diff --git a/drivers/fpga/xrt/mgmt/main-region.c b/drivers/fpga/xrt/mgmt/main-region.c
> new file mode 100644
> index 000000000000..9779693fe7ae
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/main-region.c
> @@ -0,0 +1,471 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * FPGA Region Support for Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
> + *
> + * Authors: Lizhi.Hou@xilinx.com
> + */
> +
> +#include <linux/uuid.h>
> +#include <linux/fpga/fpga-bridge.h>
> +#include <linux/fpga/fpga-region.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/axigate.h"
> +#include "xclbin-helper.h"
> +#include "main-impl.h"
> +
> +struct xmgmt_bridge {
> + struct platform_device *pdev;
> + const char *axigate_name;
axigate ?
this element could just be 'name'
it seems like 'axigate' means 'bridge', why not use 'bridge' ?
> +};
> +
> +struct xmgmt_region {
> + struct platform_device *pdev;
> + struct fpga_region *fregion;
> + uuid_t intf_uuid;
> + struct fpga_bridge *fbridge;
> + int grp_inst;
> + uuid_t dep_uuid;
> + struct list_head list;
clean up element names
fregion -> region
grp_inst -> group_instance
ect.
> +};
> +
> +struct xmgmt_region_match_arg {
> + struct platform_device *pdev;
> + uuid_t *uuids;
> + u32 uuid_num;
> +};
> +
> +static int xmgmt_br_enable_set(struct fpga_bridge *bridge, bool enable)
> +{
> + struct xmgmt_bridge *br_data = (struct xmgmt_bridge *)bridge->priv;
> + struct platform_device *axigate_leaf;
> + int rc;
> +
> + axigate_leaf = xleaf_get_leaf_by_epname(br_data->pdev, br_data->axigate_name);
> + if (!axigate_leaf) {
> + xrt_err(br_data->pdev, "failed to get leaf %s",
> + br_data->axigate_name);
> + return -ENOENT;
> + }
> +
> + if (enable)
> + rc = xleaf_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
> + else
> + rc = xleaf_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE, NULL);
> +
> + if (rc) {
> + xrt_err(br_data->pdev, "failed to %s gate %s, rc %d",
> + (enable ? "free" : "freeze"), br_data->axigate_name,
> + rc);
> + }
> +
> + xleaf_put_leaf(br_data->pdev, axigate_leaf);
> +
> + return rc;
> +}
> +
> +const struct fpga_bridge_ops xmgmt_bridge_ops = {
> + .enable_set = xmgmt_br_enable_set
> +};
> +
> +static void xmgmt_destroy_bridge(struct fpga_bridge *br)
> +{
> + struct xmgmt_bridge *br_data = br->priv;
> +
> + if (!br_data)
> + return;
> +
> + xrt_info(br_data->pdev, "destroy fpga bridge %s", br_data->axigate_name);
> + fpga_bridge_unregister(br);
> +
> + devm_kfree(DEV(br_data->pdev), br_data);
> +
> + fpga_bridge_free(br);
> +}
> +
> +static struct fpga_bridge *xmgmt_create_bridge(struct platform_device *pdev,
> + char *dtb)
> +{
> + struct xmgmt_bridge *br_data;
> + struct fpga_bridge *br = NULL;
> + const char *gate;
> + int rc;
> +
> + br_data = devm_kzalloc(DEV(pdev), sizeof(*br_data), GFP_KERNEL);
> + if (!br_data)
> + return NULL;
> + br_data->pdev = pdev;
> +
> + br_data->axigate_name = XRT_MD_NODE_GATE_ULP;
> + rc = xrt_md_find_endpoint(&pdev->dev, dtb, XRT_MD_NODE_GATE_ULP,
> + NULL, &gate);
> + if (rc) {
> + br_data->axigate_name = XRT_MD_NODE_GATE_PLP;
> + rc = xrt_md_find_endpoint(&pdev->dev, dtb, XRT_MD_NODE_GATE_PLP,
> + NULL, &gate);
> + }
> + if (rc) {
> + xrt_err(pdev, "failed to get axigate, rc %d", rc);
> + goto failed;
> + }
> +
> + br = fpga_bridge_create(DEV(pdev), br_data->axigate_name,
> + &xmgmt_bridge_ops, br_data);
> + if (!br) {
> + xrt_err(pdev, "failed to create bridge");
> + goto failed;
> + }
> +
> + rc = fpga_bridge_register(br);
> + if (rc) {
> + xrt_err(pdev, "failed to register bridge, rc %d", rc);
> + goto failed;
> + }
> +
> + xrt_info(pdev, "created fpga bridge %s", br_data->axigate_name);
> +
> + return br;
> +
> +failed:
> + if (br)
> + fpga_bridge_free(br);
> + if (br_data)
> + devm_kfree(DEV(pdev), br_data);
> +
> + return NULL;
> +}
> +
> +static void xmgmt_destroy_region(struct fpga_region *re)
re, to me means regular expression.
can you use 'region' instead ?
> +{
> + struct xmgmt_region *r_data = re->priv;
> +
> + xrt_info(r_data->pdev, "destroy fpga region %llx%llx",
need a sperator here ? between %llx%llx
> + re->compat_id->id_l, re->compat_id->id_h);
> +
> + fpga_region_unregister(re);
> +
> + if (r_data->grp_inst > 0)
> + xleaf_destroy_group(r_data->pdev, r_data->grp_inst);
> +
> + if (r_data->fbridge)
> + xmgmt_destroy_bridge(r_data->fbridge);
> +
> + if (r_data->fregion->info) {
> + fpga_image_info_free(r_data->fregion->info);
> + r_data->fregion->info = NULL;
> + }
> +
> + fpga_region_free(re);
> +
> + devm_kfree(DEV(r_data->pdev), r_data);
> +}
> +
> +static int xmgmt_region_match(struct device *dev, const void *data)
> +{
> + const struct xmgmt_region_match_arg *arg = data;
> + const struct fpga_region *match_re;
> + int i;
> +
> + if (dev->parent != &arg->pdev->dev)
> + return false;
> +
> + match_re = to_fpga_region(dev);
> + /*
> + * The device tree provides both parent and child uuids for an
> + * xclbin in one array. Here we try both uuids to see if it matches
> + * with target region's compat_id. Strictly speaking we should
> + * only match xclbin's parent uuid with target region's compat_id
> + * but given the uuids by design are unique comparing with both
> + * does not hurt.
> + */
> + for (i = 0; i < arg->uuid_num; i++) {
> + if (!memcmp(match_re->compat_id, &arg->uuids[i],
> + sizeof(*match_re->compat_id)))
> + return true;
> + }
> +
> + return false;
> +}
> +
> +static int xmgmt_region_match_base(struct device *dev, const void *data)
> +{
> + const struct xmgmt_region_match_arg *arg = data;
> + const struct fpga_region *match_re;
> + const struct xmgmt_region *r_data;
> +
> + if (dev->parent != &arg->pdev->dev)
> + return false;
> +
> + match_re = to_fpga_region(dev);
me getting confused thinking match_re is 'match regular expression'
> + r_data = match_re->priv;
> + if (uuid_is_null(&r_data->dep_uuid))
> + return true;
> +
> + return false;
> +}
> +
> +static int xmgmt_region_match_by_depuuid(struct device *dev, const void *data)
what is a dep uuid ?
should add a '_' so function name is xmgmt_region_match_by_dep_uuid()
this is quite long.
> +{
> + const struct xmgmt_region_match_arg *arg = data;
> + const struct fpga_region *match_re;
> + const struct xmgmt_region *r_data;
> +
> + if (dev->parent != &arg->pdev->dev)
> + return false;
> +
> + match_re = to_fpga_region(dev);
> + r_data = match_re->priv;
> + if (!memcmp(&r_data->dep_uuid, arg->uuids, sizeof(uuid_t)))
arg->uuids is an array
arg->uuid_num is its extent.
should be a loop or check on uuid_num == 1 or both.
> + return true;
> +
> + return false;
> +}
> +
> +static void xmgmt_region_cleanup(struct fpga_region *re)
> +{
> + struct xmgmt_region *r_data = re->priv, *temp;
> + struct platform_device *pdev = r_data->pdev;
> + struct fpga_region *match_re = NULL;
> + struct device *start_dev = NULL;
> + struct xmgmt_region_match_arg arg;
> + LIST_HEAD(free_list);
> +
> + list_add_tail(&r_data->list, &free_list);
> + arg.pdev = pdev;
> + arg.uuid_num = 1;
> +
> + while (!r_data) {
> + arg.uuids = (uuid_t *)r_data->fregion->compat_id;
> + match_re = fpga_region_class_find(start_dev, &arg,
> + xmgmt_region_match_by_depuuid);
> + if (match_re) {
> + r_data = match_re->priv;
This setting of r_data and continuing is strange, add a comment.
or if you intend to do a pair of operations, do the pair within the if block and remove the strangeness.
> + list_add_tail(&r_data->list, &free_list);
> + start_dev = &match_re->dev;
> + put_device(&match_re->dev);
> + continue;
> + }
> +
> + r_data = list_is_last(&r_data->list, &free_list) ? NULL :
> + list_next_entry(r_data, list);
> + start_dev = NULL;
> + }
> +
> + list_for_each_entry_safe_reverse(r_data, temp, &free_list, list) {
> + if (list_is_first(&r_data->list, &free_list)) {
> + if (r_data->grp_inst > 0) {
> + xleaf_destroy_group(pdev, r_data->grp_inst);
> + r_data->grp_inst = -1;
> + }
> + if (r_data->fregion->info) {
> + fpga_image_info_free(r_data->fregion->info);
> + r_data->fregion->info = NULL;
> + }
> + continue;
add a comment, why is a continue needed here ?
> + }
> + xmgmt_destroy_region(r_data->fregion);
> + }
> +}
> +
> +void xmgmt_region_cleanup_all(struct platform_device *pdev)
> +{
> + struct fpga_region *base_re;
> + struct xmgmt_region_match_arg arg;
> +
> + arg.pdev = pdev;
> +
> + for (base_re = fpga_region_class_find(NULL, &arg, xmgmt_region_match_base);
> + base_re;
> + base_re = fpga_region_class_find(NULL, &arg, xmgmt_region_match_base)) {
convert to a while
while (base_region = ... ) {
> + put_device(&base_re->dev);
> +
> + xmgmt_region_cleanup(base_re);
> + xmgmt_destroy_region(base_re);
> + }
> +}
> +
> +/*
> + * Program a given region with given xclbin image. Bring up the subdevs and the
Program a region with a xclbin image.
> + * group object to contain the subdevs.
> + */
> +static int xmgmt_region_program(struct fpga_region *re, const void *xclbin, char *dtb)
> +{
> + struct xmgmt_region *r_data = re->priv;
> + struct platform_device *pdev = r_data->pdev;
> + struct fpga_image_info *info;
> + const struct axlf *xclbin_obj = xclbin;
> + int rc;
> +
> + info = fpga_image_info_alloc(&pdev->dev);
> + if (!info)
> + return -ENOMEM;
> +
> + info->buf = xclbin;
> + info->count = xclbin_obj->m_header.m_length;
> + info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
> + re->info = info;
> + rc = fpga_region_program_fpga(re);
> + if (rc) {
> + xrt_err(pdev, "programming xclbin failed, rc %d", rc);
> + return rc;
> + }
free info ?
> +
> + /* free bridges to allow reprogram */
> + if (re->get_bridges)
> + fpga_bridges_put(&re->bridge_list);
> +
> + /*
> + * Next bringup the subdevs for this region which will be managed by
> + * its own group object.
> + */
> + r_data->grp_inst = xleaf_create_group(pdev, dtb);
> + if (r_data->grp_inst < 0) {
> + xrt_err(pdev, "failed to create group, rc %d",
> + r_data->grp_inst);
> + rc = r_data->grp_inst;
> + return rc;
> + }
> +
> + rc = xleaf_wait_for_group_bringup(pdev);
> + if (rc)
> + xrt_err(pdev, "group bringup failed, rc %d", rc);
failed but no error handling, shouldn't the leaves and group be torn down ?
> + return rc;
> +}
> +
> +static int xmgmt_get_bridges(struct fpga_region *re)
> +{
> + struct xmgmt_region *r_data = re->priv;
> + struct device *dev = &r_data->pdev->dev;
> +
> + return fpga_bridge_get_to_list(dev, re->info, &re->bridge_list);
> +}
> +
> +/*
> + * Program/create FPGA regions based on input xclbin file. This is key function
> + * stitching the flow together:
'This is ' .. sentence does not make sense, but is not needed drop it.
>
> + * 1. Identify a matching existing region for this xclbin
> + * 2. Tear down any previous objects for the found region
> + * 3. Program this region with input xclbin
> + * 4. Iterate over this region's interface uuids to determine if it defines any
> + * child region. Create fpga_region for the child region.
> + */
> +int xmgmt_process_xclbin(struct platform_device *pdev,
> + struct fpga_manager *fmgr,
> + const struct axlf *xclbin,
> + enum provider_kind kind)
> +{
> + struct fpga_region *re, *compat_re = NULL;
> + struct xmgmt_region_match_arg arg;
should initialize to { 0 }
> + struct xmgmt_region *r_data;
> + char *dtb = NULL;
> + int rc, i;
> +
> + rc = xrt_xclbin_get_metadata(DEV(pdev), xclbin, &dtb);
> + if (rc) {
> + xrt_err(pdev, "failed to get dtb: %d", rc);
> + goto failed;
> + }
> +
> + xrt_md_get_intf_uuids(DEV(pdev), dtb, &arg.uuid_num, NULL);
should also check return and return that error code, it isn't always -EINVAL
> + if (arg.uuid_num == 0) {
> + xrt_err(pdev, "failed to get intf uuid");
> + rc = -EINVAL;
> + goto failed;
> + }
> + arg.uuids = vzalloc(sizeof(uuid_t) * arg.uuid_num);
> + if (!arg.uuids) {
> + rc = -ENOMEM;
> + goto failed;
> + }
> + arg.pdev = pdev;
> +
> + xrt_md_get_intf_uuids(DEV(pdev), dtb, &arg.uuid_num, arg.uuids);
This can still fail, check this return and check the expected num of uuid's has not changed.
> +
> + /* if this is not base firmware, search for a compatible region */
> + if (kind != XMGMT_BLP) {
> + compat_re = fpga_region_class_find(NULL, &arg,
> + xmgmt_region_match);
> + if (!compat_re) {
> + xrt_err(pdev, "failed to get compatible region");
> + rc = -ENOENT;
> + goto failed;
> + }
> +
> + xmgmt_region_cleanup(compat_re);
> +
> + rc = xmgmt_region_program(compat_re, xclbin, dtb);
> + if (rc) {
> + xrt_err(pdev, "failed to program region");
> + goto failed;
> + }
> + }
> +
> + /* create all the new regions contained in this xclbin */
> + for (i = 0; i < arg.uuid_num; i++) {
> + if (compat_re && !memcmp(compat_re->compat_id, &arg.uuids[i],
> + sizeof(*compat_re->compat_id)))
> + /* region for this interface already exists */
> + continue;
> + re = fpga_region_create(DEV(pdev), fmgr, xmgmt_get_bridges);
> + if (!re) {
> + xrt_err(pdev, "failed to create fpga region");
> + rc = -EFAULT;
> + goto failed;
> + }
> + r_data = devm_kzalloc(DEV(pdev), sizeof(*r_data), GFP_KERNEL);
> + if (!r_data) {
> + rc = -ENOMEM;
> + fpga_region_free(re);
> + goto failed;
Failed in a loop but no cleanup of previous iterations.
Tom
> + }
> + r_data->pdev = pdev;
> + r_data->fregion = re;
> + r_data->grp_inst = -1;
> + memcpy(&r_data->intf_uuid, &arg.uuids[i],
> + sizeof(r_data->intf_uuid));
> + if (compat_re) {
> + memcpy(&r_data->dep_uuid, compat_re->compat_id,
> + sizeof(r_data->intf_uuid));
> + }
> + r_data->fbridge = xmgmt_create_bridge(pdev, dtb);
> + if (!r_data->fbridge) {
> + xrt_err(pdev, "failed to create fpga bridge");
> + rc = -EFAULT;
> + devm_kfree(DEV(pdev), r_data);
> + fpga_region_free(re);
> + goto failed;
> + }
> +
> + re->compat_id = (struct fpga_compat_id *)&r_data->intf_uuid;
> + re->priv = r_data;
> +
> + rc = fpga_region_register(re);
> + if (rc) {
> + xrt_err(pdev, "failed to register fpga region");
> + xmgmt_destroy_bridge(r_data->fbridge);
> + fpga_region_free(re);
> + devm_kfree(DEV(pdev), r_data);
> + goto failed;
> + }
> +
> + xrt_info(pdev, "created fpga region %llx%llx",
> + re->compat_id->id_l, re->compat_id->id_h);
> + }
> +
> +failed:
> + if (compat_re)
> + put_device(&compat_re->dev);
> +
> + if (rc) {
> + if (compat_re)
> + xmgmt_region_cleanup(compat_re);
> + }
> +
> + if (dtb)
> + vfree(dtb);
> +
> + return rc;
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 09/18] fpga: xrt: fpga-mgr and region implementation for xclbin download
2021-02-28 16:36 ` Tom Rix
@ 2021-03-04 17:50 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-04 17:50 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 02/28/2021 08:36 AM, Tom Rix wrote:
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> fpga-mgr and region implementation for xclbin download which will be
>> called from main platform driver
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/mgmt/fmgr-drv.c | 187 +++++++++++
>> drivers/fpga/xrt/mgmt/fmgr.h | 28 ++
>> drivers/fpga/xrt/mgmt/main-region.c | 471 ++++++++++++++++++++++++++++
>> 3 files changed, 686 insertions(+)
>> create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
>> create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
>> create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
>>
>> diff --git a/drivers/fpga/xrt/mgmt/fmgr-drv.c b/drivers/fpga/xrt/mgmt/fmgr-drv.c
>> new file mode 100644
>> index 000000000000..a44d35ecdb60
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/mgmt/fmgr-drv.c
>> @@ -0,0 +1,187 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * FPGA Manager Support for Xilinx Alveo Management Function Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors: Sonal.Santan@xilinx.com
>> + */
>> +
>> +#include <linux/cred.h>
>> +#include <linux/efi.h>
>> +#include <linux/fpga/fpga-mgr.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/module.h>
>> +#include <linux/vmalloc.h>
>> +
>> +#include "xclbin-helper.h"
>> +#include "xleaf.h"
>> +#include "fmgr.h"
>> +#include "xleaf/axigate.h"
>> +#include "xleaf/icap.h"
>> +#include "main-impl.h"
>> +
>> +struct xfpga_class {
>> + const struct platform_device *pdev;
>> + char name[64];
>> +};
>> +
>> +/*
>> + * xclbin download plumbing -- find the download subsystem, ICAP and
>> + * pass the xclbin for heavy lifting
>> + */
>> +static int xmgmt_download_bitstream(struct platform_device *pdev,
>> + const struct axlf *xclbin)
>> +
>> +{
>> + struct hw_icap_bit_header bit_header = { 0 };
>> + struct platform_device *icap_leaf = NULL;
>> + struct xrt_icap_ioctl_wr arg;
>> + char *bitstream = NULL;
>> + u64 bit_len;
>> + int ret;
>> +
>> + ret = xrt_xclbin_get_section(xclbin, BITSTREAM, (void **)&bitstream, &bit_len);
>> + if (ret || !bitstream) {
> !bitstream check is unneeded
Will fix.
>> + xrt_err(pdev, "bitstream not found");
>> + return -ENOENT;
>> + }
>> + ret = xrt_xclbin_parse_bitstream_header(bitstream,
>> + DMA_HWICAP_BITFILE_BUFFER_SIZE,
>> + &bit_header);
>> + if (ret) {
>> + ret = -EINVAL;
>> + xrt_err(pdev, "invalid bitstream header");
>> + goto done;
>> + }
>> + if (bit_header.header_length + bit_header.bitstream_length > bit_len) {
>> + ret = -EINVAL;
>> + xrt_err(pdev, "invalid bitstream length. header %d, bitstream %d, section len %lld",
>> + bit_header.header_length, bit_header.bitstream_length, bit_len);
>> + goto done;
>> + }
>> +
>> + icap_leaf = xleaf_get_leaf_by_id(pdev, XRT_SUBDEV_ICAP, PLATFORM_DEVID_NONE);
>> + if (!icap_leaf) {
>> + ret = -ENODEV;
>> + xrt_err(pdev, "icap does not exist");
>> + xrt_xclbin_free_header(&bit_header);
>> + goto done;
>> + }
>> + arg.xiiw_bit_data = bitstream + bit_header.header_length;
>> + arg.xiiw_data_len = bit_header.bitstream_length;
>> + ret = xleaf_ioctl(icap_leaf, XRT_ICAP_WRITE, &arg);
>> + if (ret)
>> + xrt_err(pdev, "write bitstream failed, ret = %d", ret);
>> +
>> + xrt_xclbin_free_header(&bit_header);
> memory leak when something fails and goto done's
Based on comments of previous patch, it does not need allocating extra
memory in header parsing function. xrt_xclbin_free_header() will be removed.
>> +done:
> previous general problem, use mutliple label in error handling blocks
Will fix this.
>> + if (icap_leaf)
>> + xleaf_put_leaf(pdev, icap_leaf);
>> + vfree(bitstream);
>> +
>> + return ret;
>> +}
>> +
>> +/*
>> + * There is no HW prep work we do here since we need the full
>> + * xclbin for its sanity check.
>> + */
>> +static int xmgmt_pr_write_init(struct fpga_manager *mgr,
>> + struct fpga_image_info *info,
>> + const char *buf, size_t count)
>> +{
>> + const struct axlf *bin = (const struct axlf *)buf;
>> + struct xfpga_class *obj = mgr->priv;
>> +
>> + if (!(info->flags & FPGA_MGR_PARTIAL_RECONFIG)) {
>> + xrt_info(obj->pdev, "%s only supports partial reconfiguration\n", obj->name);
>> + return -EINVAL;
>> + }
>> +
>> + if (count < sizeof(struct axlf))
>> + return -EINVAL;
>> +
>> + if (count > bin->m_header.m_length)
>> + return -EINVAL;
>> +
>> + xrt_info(obj->pdev, "Prepare download of xclbin %pUb of length %lld B",
>> + &bin->m_header.uuid, bin->m_header.m_length);
>> +
>> + return 0;
>> +}
>> +
>> +/*
>> + * The implementation requries full xclbin image before we can start
>> + * programming the hardware via ICAP subsystem. Full image is required
> The full image
Will fix this.
>> + * for checking the validity of xclbin and walking the sections to
>> + * discover the bitstream.
>> + */
>> +static int xmgmt_pr_write(struct fpga_manager *mgr,
>> + const char *buf, size_t count)
>> +{
>> + const struct axlf *bin = (const struct axlf *)buf;
>> + struct xfpga_class *obj = mgr->priv;
>> +
>> + if (bin->m_header.m_length != count)
>> + return -EINVAL;
>> +
>> + return xmgmt_download_bitstream((void *)obj->pdev, bin);
>> +}
>> +
>> +static int xmgmt_pr_write_complete(struct fpga_manager *mgr,
>> + struct fpga_image_info *info)
>> +{
>> + const struct axlf *bin = (const struct axlf *)info->buf;
>> + struct xfpga_class *obj = mgr->priv;
>> +
>> + xrt_info(obj->pdev, "Finished download of xclbin %pUb",
>> + &bin->m_header.uuid);
>> + return 0;
>> +}
>> +
>> +static enum fpga_mgr_states xmgmt_pr_state(struct fpga_manager *mgr)
>> +{
>> + return FPGA_MGR_STATE_UNKNOWN;
> why just this state ?
This is copied from fme_mgr_state(). I think we can defer this and
return more status when we need them?
>> +}
>> +
>> +static const struct fpga_manager_ops xmgmt_pr_ops = {
>> + .initial_header_size = sizeof(struct axlf),
>> + .write_init = xmgmt_pr_write_init,
>> + .write = xmgmt_pr_write,
>> + .write_complete = xmgmt_pr_write_complete,
>> + .state = xmgmt_pr_state,
>> +};
>> +
>> +struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev)
>> +{
>> + struct xfpga_class *obj = devm_kzalloc(DEV(pdev), sizeof(struct xfpga_class),
>> + GFP_KERNEL);
>> + struct fpga_manager *fmgr = NULL;
>> + int ret = 0;
>> +
>> + if (!obj)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + snprintf(obj->name, sizeof(obj->name), "Xilinx Alveo FPGA Manager");
>> + obj->pdev = pdev;
>> + fmgr = fpga_mgr_create(&pdev->dev,
>> + obj->name,
>> + &xmgmt_pr_ops,
>> + obj);
>> + if (!fmgr)
>> + return ERR_PTR(-ENOMEM);
>> +
>> + ret = fpga_mgr_register(fmgr);
>> + if (ret) {
>> + fpga_mgr_free(fmgr);
>> + return ERR_PTR(ret);
>> + }
>> + return fmgr;
>> +}
>> +
>> +int xmgmt_fmgr_remove(struct fpga_manager *fmgr)
>> +{
>> + fpga_mgr_unregister(fmgr);
>> + return 0;
>> +}
>> diff --git a/drivers/fpga/xrt/mgmt/fmgr.h b/drivers/fpga/xrt/mgmt/fmgr.h
>> new file mode 100644
>> index 000000000000..e1fc033e2542
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/mgmt/fmgr.h
>> @@ -0,0 +1,28 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for Xilinx Alveo Management Function Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors: Sonal.Santan@xilinx.com
>> + */
>> +
>> +#ifndef _XMGMT_FMGR_H_
>> +#define _XMGMT_FMGR_H_
>> +
>> +#include <linux/fpga/fpga-mgr.h>
>> +#include <linux/mutex.h>
>> +
>> +#include <linux/xrt/xclbin.h>
>> +
>> +enum xfpga_sec_level {
>> + XFPGA_SEC_NONE = 0,
>> + XFPGA_SEC_DEDICATE,
>> + XFPGA_SEC_SYSTEM,
>> + XFPGA_SEC_MAX = XFPGA_SEC_SYSTEM,
>> +};
> This enum is not used, remove.
Will remove.
>> +
>> +struct fpga_manager *xmgmt_fmgr_probe(struct platform_device *pdev);
>> +int xmgmt_fmgr_remove(struct fpga_manager *fmgr);
>> +
>> +#endif
>> diff --git a/drivers/fpga/xrt/mgmt/main-region.c b/drivers/fpga/xrt/mgmt/main-region.c
>> new file mode 100644
>> index 000000000000..9779693fe7ae
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/mgmt/main-region.c
>> @@ -0,0 +1,471 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * FPGA Region Support for Xilinx Alveo Management Function Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + * Bulk of the code borrowed from XRT mgmt driver file, fmgr.c
>> + *
>> + * Authors: Lizhi.Hou@xilinx.com
>> + */
>> +
>> +#include <linux/uuid.h>
>> +#include <linux/fpga/fpga-bridge.h>
>> +#include <linux/fpga/fpga-region.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/axigate.h"
>> +#include "xclbin-helper.h"
>> +#include "main-impl.h"
>> +
>> +struct xmgmt_bridge {
>> + struct platform_device *pdev;
>> + const char *axigate_name;
> axigate ?
>
> this element could just be 'name'
>
> it seems like 'axigate' means 'bridge', why not use 'bridge' ?
Will change to 'bridge'
>
>> +};
>> +
>> +struct xmgmt_region {
>> + struct platform_device *pdev;
>> + struct fpga_region *fregion;
>> + uuid_t intf_uuid;
>> + struct fpga_bridge *fbridge;
>> + int grp_inst;
>> + uuid_t dep_uuid;
>> + struct list_head list;
> clean up element names
>
> fregion -> region
>
> grp_inst -> group_instance
>
> ect.
Will fix this.
>
>> +};
>> +
>> +struct xmgmt_region_match_arg {
>> + struct platform_device *pdev;
>> + uuid_t *uuids;
>> + u32 uuid_num;
>> +};
>> +
>> +static int xmgmt_br_enable_set(struct fpga_bridge *bridge, bool enable)
>> +{
>> + struct xmgmt_bridge *br_data = (struct xmgmt_bridge *)bridge->priv;
>> + struct platform_device *axigate_leaf;
>> + int rc;
>> +
>> + axigate_leaf = xleaf_get_leaf_by_epname(br_data->pdev, br_data->axigate_name);
>> + if (!axigate_leaf) {
>> + xrt_err(br_data->pdev, "failed to get leaf %s",
>> + br_data->axigate_name);
>> + return -ENOENT;
>> + }
>> +
>> + if (enable)
>> + rc = xleaf_ioctl(axigate_leaf, XRT_AXIGATE_FREE, NULL);
>> + else
>> + rc = xleaf_ioctl(axigate_leaf, XRT_AXIGATE_FREEZE, NULL);
>> +
>> + if (rc) {
>> + xrt_err(br_data->pdev, "failed to %s gate %s, rc %d",
>> + (enable ? "free" : "freeze"), br_data->axigate_name,
>> + rc);
>> + }
>> +
>> + xleaf_put_leaf(br_data->pdev, axigate_leaf);
>> +
>> + return rc;
>> +}
>> +
>> +const struct fpga_bridge_ops xmgmt_bridge_ops = {
>> + .enable_set = xmgmt_br_enable_set
>> +};
>> +
>> +static void xmgmt_destroy_bridge(struct fpga_bridge *br)
>> +{
>> + struct xmgmt_bridge *br_data = br->priv;
>> +
>> + if (!br_data)
>> + return;
>> +
>> + xrt_info(br_data->pdev, "destroy fpga bridge %s", br_data->axigate_name);
>> + fpga_bridge_unregister(br);
>> +
>> + devm_kfree(DEV(br_data->pdev), br_data);
>> +
>> + fpga_bridge_free(br);
>> +}
>> +
>> +static struct fpga_bridge *xmgmt_create_bridge(struct platform_device *pdev,
>> + char *dtb)
>> +{
>> + struct xmgmt_bridge *br_data;
>> + struct fpga_bridge *br = NULL;
>> + const char *gate;
>> + int rc;
>> +
>> + br_data = devm_kzalloc(DEV(pdev), sizeof(*br_data), GFP_KERNEL);
>> + if (!br_data)
>> + return NULL;
>> + br_data->pdev = pdev;
>> +
>> + br_data->axigate_name = XRT_MD_NODE_GATE_ULP;
>> + rc = xrt_md_find_endpoint(&pdev->dev, dtb, XRT_MD_NODE_GATE_ULP,
>> + NULL, &gate);
>> + if (rc) {
>> + br_data->axigate_name = XRT_MD_NODE_GATE_PLP;
>> + rc = xrt_md_find_endpoint(&pdev->dev, dtb, XRT_MD_NODE_GATE_PLP,
>> + NULL, &gate);
>> + }
>> + if (rc) {
>> + xrt_err(pdev, "failed to get axigate, rc %d", rc);
>> + goto failed;
>> + }
>> +
>> + br = fpga_bridge_create(DEV(pdev), br_data->axigate_name,
>> + &xmgmt_bridge_ops, br_data);
>> + if (!br) {
>> + xrt_err(pdev, "failed to create bridge");
>> + goto failed;
>> + }
>> +
>> + rc = fpga_bridge_register(br);
>> + if (rc) {
>> + xrt_err(pdev, "failed to register bridge, rc %d", rc);
>> + goto failed;
>> + }
>> +
>> + xrt_info(pdev, "created fpga bridge %s", br_data->axigate_name);
>> +
>> + return br;
>> +
>> +failed:
>> + if (br)
>> + fpga_bridge_free(br);
>> + if (br_data)
>> + devm_kfree(DEV(pdev), br_data);
>> +
>> + return NULL;
>> +}
>> +
>> +static void xmgmt_destroy_region(struct fpga_region *re)
> re, to me means regular expression.
>
> can you use 'region' instead ?
Will change this.
>
>> +{
>> + struct xmgmt_region *r_data = re->priv;
>> +
>> + xrt_info(r_data->pdev, "destroy fpga region %llx%llx",
> need a sperator here ? between %llx%llx
Will add '.' in between.
>> + re->compat_id->id_l, re->compat_id->id_h);
>> +
>> + fpga_region_unregister(re);
>> +
>> + if (r_data->grp_inst > 0)
>> + xleaf_destroy_group(r_data->pdev, r_data->grp_inst);
>> +
>> + if (r_data->fbridge)
>> + xmgmt_destroy_bridge(r_data->fbridge);
>> +
>> + if (r_data->fregion->info) {
>> + fpga_image_info_free(r_data->fregion->info);
>> + r_data->fregion->info = NULL;
>> + }
>> +
>> + fpga_region_free(re);
>> +
>> + devm_kfree(DEV(r_data->pdev), r_data);
>> +}
>> +
>> +static int xmgmt_region_match(struct device *dev, const void *data)
>> +{
>> + const struct xmgmt_region_match_arg *arg = data;
>> + const struct fpga_region *match_re;
>> + int i;
>> +
>> + if (dev->parent != &arg->pdev->dev)
>> + return false;
>> +
>> + match_re = to_fpga_region(dev);
>> + /*
>> + * The device tree provides both parent and child uuids for an
>> + * xclbin in one array. Here we try both uuids to see if it matches
>> + * with target region's compat_id. Strictly speaking we should
>> + * only match xclbin's parent uuid with target region's compat_id
>> + * but given the uuids by design are unique comparing with both
>> + * does not hurt.
>> + */
>> + for (i = 0; i < arg->uuid_num; i++) {
>> + if (!memcmp(match_re->compat_id, &arg->uuids[i],
>> + sizeof(*match_re->compat_id)))
>> + return true;
>> + }
>> +
>> + return false;
>> +}
>> +
>> +static int xmgmt_region_match_base(struct device *dev, const void *data)
>> +{
>> + const struct xmgmt_region_match_arg *arg = data;
>> + const struct fpga_region *match_re;
>> + const struct xmgmt_region *r_data;
>> +
>> + if (dev->parent != &arg->pdev->dev)
>> + return false;
>> +
>> + match_re = to_fpga_region(dev);
> me getting confused thinking match_re is 'match regular expression'
Will change to 'match_region'.
>> + r_data = match_re->priv;
>> + if (uuid_is_null(&r_data->dep_uuid))
>> + return true;
>> +
>> + return false;
>> +}
>> +
>> +static int xmgmt_region_match_by_depuuid(struct device *dev, const void *data)
> what is a dep uuid ?
A region could be on top of another base region. 'dep uuid' means base
region uuid it depends on.
>
> should add a '_' so function name is xmgmt_region_match_by_dep_uuid()
>
> this is quite long.
Will change to xmgmt_region_match_by_uuid().
>
>> +{
>> + const struct xmgmt_region_match_arg *arg = data;
>> + const struct fpga_region *match_re;
>> + const struct xmgmt_region *r_data;
>> +
>> + if (dev->parent != &arg->pdev->dev)
>> + return false;
>> +
>> + match_re = to_fpga_region(dev);
>> + r_data = match_re->priv;
>> + if (!memcmp(&r_data->dep_uuid, arg->uuids, sizeof(uuid_t)))
> arg->uuids is an array
>
> arg->uuid_num is its extent.
>
> should be a loop or check on uuid_num == 1 or both.
Will add check on uuid_num == 1.
>
>> + return true;
>> +
>> + return false;
>> +}
>> +
>> +static void xmgmt_region_cleanup(struct fpga_region *re)
>> +{
>> + struct xmgmt_region *r_data = re->priv, *temp;
>> + struct platform_device *pdev = r_data->pdev;
>> + struct fpga_region *match_re = NULL;
>> + struct device *start_dev = NULL;
>> + struct xmgmt_region_match_arg arg;
>> + LIST_HEAD(free_list);
>> +
>> + list_add_tail(&r_data->list, &free_list);
>> + arg.pdev = pdev;
>> + arg.uuid_num = 1;
>> +
>> + while (!r_data) {
>> + arg.uuids = (uuid_t *)r_data->fregion->compat_id;
>> + match_re = fpga_region_class_find(start_dev, &arg,
>> + xmgmt_region_match_by_depuuid);
>> + if (match_re) {
>> + r_data = match_re->priv;
> This setting of r_data and continuing is strange, add a comment.
>
> or if you intend to do a pair of operations, do the pair within the if block and remove the strangeness.
Thanks for pointing it out. This is poor implementation. I will
re-implement the loop and add comment.
>
>> + list_add_tail(&r_data->list, &free_list);
>> + start_dev = &match_re->dev;
>> + put_device(&match_re->dev);
>> + continue;
>> + }
>> +
>> + r_data = list_is_last(&r_data->list, &free_list) ? NULL :
>> + list_next_entry(r_data, list);
>> + start_dev = NULL;
>> + }
>> +
>> + list_for_each_entry_safe_reverse(r_data, temp, &free_list, list) {
>> + if (list_is_first(&r_data->list, &free_list)) {
>> + if (r_data->grp_inst > 0) {
>> + xleaf_destroy_group(pdev, r_data->grp_inst);
>> + r_data->grp_inst = -1;
>> + }
>> + if (r_data->fregion->info) {
>> + fpga_image_info_free(r_data->fregion->info);
>> + r_data->fregion->info = NULL;
>> + }
>> + continue;
> add a comment, why is a continue needed here ?
Will remove the continue and pop the first node before the loop.
>> + }
>> + xmgmt_destroy_region(r_data->fregion);
>> + }
>> +}
>> +
>> +void xmgmt_region_cleanup_all(struct platform_device *pdev)
>> +{
>> + struct fpga_region *base_re;
>> + struct xmgmt_region_match_arg arg;
>> +
>> + arg.pdev = pdev;
>> +
>> + for (base_re = fpga_region_class_find(NULL, &arg, xmgmt_region_match_base);
>> + base_re;
>> + base_re = fpga_region_class_find(NULL, &arg, xmgmt_region_match_base)) {
> convert to a while
>
> while (base_region = ... ) {
Will convert.
>
>> + put_device(&base_re->dev);
>> +
>> + xmgmt_region_cleanup(base_re);
>> + xmgmt_destroy_region(base_re);
>> + }
>> +}
>> +
>> +/*
>> + * Program a given region with given xclbin image. Bring up the subdevs and the
> Program a region with a xclbin image.
Will fix this.
>> + * group object to contain the subdevs.
>> + */
>> +static int xmgmt_region_program(struct fpga_region *re, const void *xclbin, char *dtb)
>> +{
>> + struct xmgmt_region *r_data = re->priv;
>> + struct platform_device *pdev = r_data->pdev;
>> + struct fpga_image_info *info;
>> + const struct axlf *xclbin_obj = xclbin;
>> + int rc;
>> +
>> + info = fpga_image_info_alloc(&pdev->dev);
>> + if (!info)
>> + return -ENOMEM;
>> +
>> + info->buf = xclbin;
>> + info->count = xclbin_obj->m_header.m_length;
>> + info->flags |= FPGA_MGR_PARTIAL_RECONFIG;
>> + re->info = info;
>> + rc = fpga_region_program_fpga(re);
>> + if (rc) {
>> + xrt_err(pdev, "programming xclbin failed, rc %d", rc);
>> + return rc;
>> + }
> free info ?
info will be freed outside by xmgmt_destroy_region().
>> +
>> + /* free bridges to allow reprogram */
>> + if (re->get_bridges)
>> + fpga_bridges_put(&re->bridge_list);
>> +
>> + /*
>> + * Next bringup the subdevs for this region which will be managed by
>> + * its own group object.
>> + */
>> + r_data->grp_inst = xleaf_create_group(pdev, dtb);
>> + if (r_data->grp_inst < 0) {
>> + xrt_err(pdev, "failed to create group, rc %d",
>> + r_data->grp_inst);
>> + rc = r_data->grp_inst;
>> + return rc;
>> + }
>> +
>> + rc = xleaf_wait_for_group_bringup(pdev);
>> + if (rc)
>> + xrt_err(pdev, "group bringup failed, rc %d", rc);
> failed but no error handling, shouldn't the leaves and group be torn down ?
Group is torn down outside by xmgmt_destroy_region().
>> + return rc;
>> +}
>> +
>> +static int xmgmt_get_bridges(struct fpga_region *re)
>> +{
>> + struct xmgmt_region *r_data = re->priv;
>> + struct device *dev = &r_data->pdev->dev;
>> +
>> + return fpga_bridge_get_to_list(dev, re->info, &re->bridge_list);
>> +}
>> +
>> +/*
>> + * Program/create FPGA regions based on input xclbin file. This is key function
>> + * stitching the flow together:
> 'This is ' .. sentence does not make sense, but is not needed drop it.
Will drop it.
>> + * 1. Identify a matching existing region for this xclbin
>> + * 2. Tear down any previous objects for the found region
>> + * 3. Program this region with input xclbin
>> + * 4. Iterate over this region's interface uuids to determine if it defines any
>> + * child region. Create fpga_region for the child region.
>> + */
>> +int xmgmt_process_xclbin(struct platform_device *pdev,
>> + struct fpga_manager *fmgr,
>> + const struct axlf *xclbin,
>> + enum provider_kind kind)
>> +{
>> + struct fpga_region *re, *compat_re = NULL;
>> + struct xmgmt_region_match_arg arg;
> should initialize to { 0 }
Will fix this and few other places.
>> + struct xmgmt_region *r_data;
>> + char *dtb = NULL;
>> + int rc, i;
>> +
>> + rc = xrt_xclbin_get_metadata(DEV(pdev), xclbin, &dtb);
>> + if (rc) {
>> + xrt_err(pdev, "failed to get dtb: %d", rc);
>> + goto failed;
>> + }
>> +
>> + xrt_md_get_intf_uuids(DEV(pdev), dtb, &arg.uuid_num, NULL);
> should also check return and return that error code, it isn't always -EINVAL
Will fix this.
>> + if (arg.uuid_num == 0) {
>> + xrt_err(pdev, "failed to get intf uuid");
>> + rc = -EINVAL;
>> + goto failed;
>> + }
>> + arg.uuids = vzalloc(sizeof(uuid_t) * arg.uuid_num);
>> + if (!arg.uuids) {
>> + rc = -ENOMEM;
>> + goto failed;
>> + }
>> + arg.pdev = pdev;
>> +
>> + xrt_md_get_intf_uuids(DEV(pdev), dtb, &arg.uuid_num, arg.uuids);
> This can still fail, check this return and check the expected num of uuid's has not changed.
Will fix this.
>> +
>> + /* if this is not base firmware, search for a compatible region */
>> + if (kind != XMGMT_BLP) {
>> + compat_re = fpga_region_class_find(NULL, &arg,
>> + xmgmt_region_match);
>> + if (!compat_re) {
>> + xrt_err(pdev, "failed to get compatible region");
>> + rc = -ENOENT;
>> + goto failed;
>> + }
>> +
>> + xmgmt_region_cleanup(compat_re);
>> +
>> + rc = xmgmt_region_program(compat_re, xclbin, dtb);
>> + if (rc) {
>> + xrt_err(pdev, "failed to program region");
>> + goto failed;
>> + }
>> + }
>> +
>> + /* create all the new regions contained in this xclbin */
>> + for (i = 0; i < arg.uuid_num; i++) {
>> + if (compat_re && !memcmp(compat_re->compat_id, &arg.uuids[i],
>> + sizeof(*compat_re->compat_id)))
>> + /* region for this interface already exists */
>> + continue;
>> + re = fpga_region_create(DEV(pdev), fmgr, xmgmt_get_bridges);
>> + if (!re) {
>> + xrt_err(pdev, "failed to create fpga region");
>> + rc = -EFAULT;
>> + goto failed;
>> + }
>> + r_data = devm_kzalloc(DEV(pdev), sizeof(*r_data), GFP_KERNEL);
>> + if (!r_data) {
>> + rc = -ENOMEM;
>> + fpga_region_free(re);
>> + goto failed;
> Failed in a loop but no cleanup of previous iterations.
Will fix this.
Thanks,
Lizhi
>
> Tom
>
>> + }
>> + r_data->pdev = pdev;
>> + r_data->fregion = re;
>> + r_data->grp_inst = -1;
>> + memcpy(&r_data->intf_uuid, &arg.uuids[i],
>> + sizeof(r_data->intf_uuid));
>> + if (compat_re) {
>> + memcpy(&r_data->dep_uuid, compat_re->compat_id,
>> + sizeof(r_data->intf_uuid));
>> + }
>> + r_data->fbridge = xmgmt_create_bridge(pdev, dtb);
>> + if (!r_data->fbridge) {
>> + xrt_err(pdev, "failed to create fpga bridge");
>> + rc = -EFAULT;
>> + devm_kfree(DEV(pdev), r_data);
>> + fpga_region_free(re);
>> + goto failed;
>> + }
>> +
>> + re->compat_id = (struct fpga_compat_id *)&r_data->intf_uuid;
>> + re->priv = r_data;
>> +
>> + rc = fpga_region_register(re);
>> + if (rc) {
>> + xrt_err(pdev, "failed to register fpga region");
>> + xmgmt_destroy_bridge(r_data->fbridge);
>> + fpga_region_free(re);
>> + devm_kfree(DEV(pdev), r_data);
>> + goto failed;
>> + }
>> +
>> + xrt_info(pdev, "created fpga region %llx%llx",
>> + re->compat_id->id_l, re->compat_id->id_h);
>> + }
>> +
>> +failed:
>> + if (compat_re)
>> + put_device(&compat_re->dev);
>> +
>> + if (rc) {
>> + if (compat_re)
>> + xmgmt_region_cleanup(compat_re);
>> + }
>> +
>> + if (dtb)
>> + vfree(dtb);
>> +
>> + return rc;
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 10/18] fpga: xrt: VSEC platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (8 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 09/18] fpga: xrt: fpga-mgr and region implementation for xclbin download Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-03-01 19:01 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 11/18] fpga: xrt: UCS " Lizhi Hou
` (9 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Add VSEC driver. VSEC is a hardware function discovered by walking
PCI Express configure space. A platform device node will be created
for it. VSEC provides board logic UUID and few offset of other hardware
functions.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/lib/xleaf/vsec.c | 359 ++++++++++++++++++++++++++++++
1 file changed, 359 insertions(+)
create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
diff --git a/drivers/fpga/xrt/lib/xleaf/vsec.c b/drivers/fpga/xrt/lib/xleaf/vsec.c
new file mode 100644
index 000000000000..8e5cb22522ec
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xleaf/vsec.c
@@ -0,0 +1,359 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA VSEC Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou<Lizhi.Hou@xilinx.com>
+ */
+
+#include <linux/platform_device.h>
+#include "metadata.h"
+#include "xleaf.h"
+
+#define XRT_VSEC "xrt_vsec"
+
+#define VSEC_TYPE_UUID 0x50
+#define VSEC_TYPE_FLASH 0x51
+#define VSEC_TYPE_PLATINFO 0x52
+#define VSEC_TYPE_MAILBOX 0x53
+#define VSEC_TYPE_END 0xff
+
+#define VSEC_UUID_LEN 16
+
+struct xrt_vsec_header {
+ u32 format;
+ u32 length;
+ u32 entry_sz;
+ u32 rsvd;
+} __packed;
+
+#define head_rd(g, r) \
+ ioread32((void *)(g)->base + offsetof(struct xrt_vsec_header, r))
+
+#define GET_BAR(entry) (((entry)->bar_rev >> 4) & 0xf)
+#define GET_BAR_OFF(_entry) \
+ ({ typeof(_entry) entry = (_entry); \
+ ((entry)->off_lo | ((u64)(entry)->off_hi << 16)); })
+#define GET_REV(entry) ((entry)->bar_rev & 0xf)
+
+struct xrt_vsec_entry {
+ u8 type;
+ u8 bar_rev;
+ u16 off_lo;
+ u32 off_hi;
+ u8 ver_type;
+ u8 minor;
+ u8 major;
+ u8 rsvd0;
+ u32 rsvd1;
+} __packed;
+
+#define read_entry(g, i, e) \
+ do { \
+ u32 *p = (u32 *)((g)->base + \
+ sizeof(struct xrt_vsec_header) + \
+ (i) * sizeof(struct xrt_vsec_entry)); \
+ u32 off; \
+ for (off = 0; \
+ off < sizeof(struct xrt_vsec_entry) / 4; \
+ off++) \
+ *((u32 *)(e) + off) = ioread32(p + off);\
+ } while (0)
+
+struct vsec_device {
+ u8 type;
+ char *ep_name;
+ ulong size;
+ char *regmap;
+};
+
+static struct vsec_device vsec_devs[] = {
+ {
+ .type = VSEC_TYPE_UUID,
+ .ep_name = XRT_MD_NODE_BLP_ROM,
+ .size = VSEC_UUID_LEN,
+ .regmap = "vsec-uuid",
+ },
+ {
+ .type = VSEC_TYPE_FLASH,
+ .ep_name = XRT_MD_NODE_FLASH_VSEC,
+ .size = 4096,
+ .regmap = "vsec-flash",
+ },
+ {
+ .type = VSEC_TYPE_PLATINFO,
+ .ep_name = XRT_MD_NODE_PLAT_INFO,
+ .size = 4,
+ .regmap = "vsec-platinfo",
+ },
+ {
+ .type = VSEC_TYPE_MAILBOX,
+ .ep_name = XRT_MD_NODE_MAILBOX_VSEC,
+ .size = 48,
+ .regmap = "vsec-mbx",
+ },
+};
+
+struct xrt_vsec {
+ struct platform_device *pdev;
+ void *base;
+ ulong length;
+
+ char *metadata;
+ char uuid[VSEC_UUID_LEN];
+};
+
+static char *type2epname(u32 type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
+ if (vsec_devs[i].type == type)
+ return (vsec_devs[i].ep_name);
+ }
+
+ return NULL;
+}
+
+static ulong type2size(u32 type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
+ if (vsec_devs[i].type == type)
+ return (vsec_devs[i].size);
+ }
+
+ return 0;
+}
+
+static char *type2regmap(u32 type)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
+ if (vsec_devs[i].type == type)
+ return (vsec_devs[i].regmap);
+ }
+
+ return NULL;
+}
+
+static int xrt_vsec_add_node(struct xrt_vsec *vsec,
+ void *md_blob, struct xrt_vsec_entry *p_entry)
+{
+ struct xrt_md_endpoint ep;
+ char regmap_ver[64];
+ int ret;
+
+ if (!type2epname(p_entry->type))
+ return -EINVAL;
+
+ /*
+ * VSEC may have more than 1 mailbox instance for the card
+ * which has more than 1 physical function.
+ * This is not supported for now. Assuming only one mailbox
+ */
+
+ snprintf(regmap_ver, sizeof(regmap_ver) - 1, "%d-%d.%d.%d",
+ p_entry->ver_type, p_entry->major, p_entry->minor,
+ GET_REV(p_entry));
+ ep.ep_name = type2epname(p_entry->type);
+ ep.bar = GET_BAR(p_entry);
+ ep.bar_off = GET_BAR_OFF(p_entry);
+ ep.size = type2size(p_entry->type);
+ ep.regmap = type2regmap(p_entry->type);
+ ep.regmap_ver = regmap_ver;
+ ret = xrt_md_add_endpoint(DEV(vsec->pdev), vsec->metadata, &ep);
+ if (ret) {
+ xrt_err(vsec->pdev, "add ep failed, ret %d", ret);
+ goto failed;
+ }
+
+failed:
+ return ret;
+}
+
+static int xrt_vsec_create_metadata(struct xrt_vsec *vsec)
+{
+ struct xrt_vsec_entry entry;
+ int i, ret;
+
+ ret = xrt_md_create(&vsec->pdev->dev, &vsec->metadata);
+ if (ret) {
+ xrt_err(vsec->pdev, "create metadata failed");
+ return ret;
+ }
+
+ for (i = 0; i * sizeof(entry) < vsec->length -
+ sizeof(struct xrt_vsec_header); i++) {
+ read_entry(vsec, i, &entry);
+ xrt_vsec_add_node(vsec, vsec->metadata, &entry);
+ }
+
+ return 0;
+}
+
+static int xrt_vsec_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ int ret = 0;
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ /* Does not handle any event. */
+ break;
+ default:
+ ret = -EINVAL;
+ xrt_err(pdev, "should never been called");
+ break;
+ }
+
+ return ret;
+}
+
+static int xrt_vsec_mapio(struct xrt_vsec *vsec)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(vsec->pdev);
+ const u32 *bar;
+ const u64 *bar_off;
+ struct resource *res = NULL;
+ ulong addr;
+ int ret;
+
+ if (!pdata || xrt_md_size(DEV(vsec->pdev), pdata->xsp_dtb) == XRT_MD_INVALID_LENGTH) {
+ xrt_err(vsec->pdev, "empty metadata");
+ return -EINVAL;
+ }
+
+ ret = xrt_md_get_prop(DEV(vsec->pdev), pdata->xsp_dtb, XRT_MD_NODE_VSEC,
+ NULL, XRT_MD_PROP_BAR_IDX, (const void **)&bar, NULL);
+ if (ret) {
+ xrt_err(vsec->pdev, "failed to get bar idx, ret %d", ret);
+ return -EINVAL;
+ }
+
+ ret = xrt_md_get_prop(DEV(vsec->pdev), pdata->xsp_dtb, XRT_MD_NODE_VSEC,
+ NULL, XRT_MD_PROP_OFFSET, (const void **)&bar_off, NULL);
+ if (ret) {
+ xrt_err(vsec->pdev, "failed to get bar off, ret %d", ret);
+ return -EINVAL;
+ }
+
+ xrt_info(vsec->pdev, "Map vsec at bar %d, offset 0x%llx",
+ be32_to_cpu(*bar), be64_to_cpu(*bar_off));
+
+ xleaf_get_barres(vsec->pdev, &res, be32_to_cpu(*bar));
+ if (!res) {
+ xrt_err(vsec->pdev, "failed to get bar addr");
+ return -EINVAL;
+ }
+
+ addr = res->start + (ulong)be64_to_cpu(*bar_off);
+
+ vsec->base = ioremap(addr, sizeof(struct xrt_vsec_header));
+ if (!vsec->base) {
+ xrt_err(vsec->pdev, "Map header failed");
+ return -EIO;
+ }
+
+ vsec->length = head_rd(vsec, length);
+ iounmap(vsec->base);
+ vsec->base = ioremap(addr, vsec->length);
+ if (!vsec->base) {
+ xrt_err(vsec->pdev, "map failed");
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static int xrt_vsec_remove(struct platform_device *pdev)
+{
+ struct xrt_vsec *vsec;
+
+ vsec = platform_get_drvdata(pdev);
+
+ if (vsec->base) {
+ iounmap(vsec->base);
+ vsec->base = NULL;
+ }
+
+ vfree(vsec->metadata);
+
+ return 0;
+}
+
+static int xrt_vsec_probe(struct platform_device *pdev)
+{
+ struct xrt_vsec *vsec;
+ int ret = 0;
+
+ vsec = devm_kzalloc(&pdev->dev, sizeof(*vsec), GFP_KERNEL);
+ if (!vsec)
+ return -ENOMEM;
+
+ vsec->pdev = pdev;
+ platform_set_drvdata(pdev, vsec);
+
+ ret = xrt_vsec_mapio(vsec);
+ if (ret)
+ goto failed;
+
+ ret = xrt_vsec_create_metadata(vsec);
+ if (ret) {
+ xrt_err(pdev, "create metadata failed, ret %d", ret);
+ goto failed;
+ }
+ ret = xleaf_create_group(pdev, vsec->metadata);
+ if (ret < 0)
+ xrt_err(pdev, "create group failed, ret %d", ret);
+ else
+ ret = 0;
+
+failed:
+ if (ret)
+ xrt_vsec_remove(pdev);
+
+ return ret;
+}
+
+static struct xrt_subdev_endpoints xrt_vsec_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names []){
+ { .ep_name = XRT_MD_NODE_VSEC },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xrt_vsec_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_vsec_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_vsec_table[] = {
+ { XRT_VSEC, (kernel_ulong_t)&xrt_vsec_data },
+ { },
+};
+
+static struct platform_driver xrt_vsec_driver = {
+ .driver = {
+ .name = XRT_VSEC,
+ },
+ .probe = xrt_vsec_probe,
+ .remove = xrt_vsec_remove,
+ .id_table = xrt_vsec_table,
+};
+
+void vsec_leaf_init_fini(bool init)
+{
+ if (init)
+ xleaf_register_driver(XRT_SUBDEV_VSEC, &xrt_vsec_driver, xrt_vsec_endpoints);
+ else
+ xleaf_unregister_driver(XRT_SUBDEV_VSEC);
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 10/18] fpga: xrt: VSEC platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 10/18] fpga: xrt: VSEC platform driver Lizhi Hou
@ 2021-03-01 19:01 ` Tom Rix
2021-03-05 19:58 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-01 19:01 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Add VSEC driver. VSEC is a hardware function discovered by walking
> PCI Express configure space. A platform device node will be created
> for it. VSEC provides board logic UUID and few offset of other hardware
> functions.
Is this vsec walking infra or is a general find a list of mmio regions that need to be mapped in and do the mapping in as a set of platform drivers ?
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/lib/xleaf/vsec.c | 359 ++++++++++++++++++++++++++++++
> 1 file changed, 359 insertions(+)
> create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
>
> diff --git a/drivers/fpga/xrt/lib/xleaf/vsec.c b/drivers/fpga/xrt/lib/xleaf/vsec.c
> new file mode 100644
> index 000000000000..8e5cb22522ec
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/vsec.c
> @@ -0,0 +1,359 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA VSEC Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + */
> +
> +#include <linux/platform_device.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +
> +#define XRT_VSEC "xrt_vsec"
> +
> +#define VSEC_TYPE_UUID 0x50
> +#define VSEC_TYPE_FLASH 0x51
> +#define VSEC_TYPE_PLATINFO 0x52
> +#define VSEC_TYPE_MAILBOX 0x53
> +#define VSEC_TYPE_END 0xff
Type of devices, this list can not grow much.
> +
> +#define VSEC_UUID_LEN 16
> +
> +struct xrt_vsec_header {
> + u32 format;
> + u32 length;
> + u32 entry_sz;
> + u32 rsvd;
> +} __packed;
> +
> +#define head_rd(g, r) \
> + ioread32((void *)(g)->base + offsetof(struct xrt_vsec_header, r))
> +
> +#define GET_BAR(entry) (((entry)->bar_rev >> 4) & 0xf)
> +#define GET_BAR_OFF(_entry) \
> + ({ typeof(_entry) entry = (_entry); \
> + ((entry)->off_lo | ((u64)(entry)->off_hi << 16)); })
A 48 bit value stored in xrt_md_endpoint.bar_off (long)
bar_off should be u64
> +#define GET_REV(entry) ((entry)->bar_rev & 0xf)
> +
I prefer functions over macros.
> +struct xrt_vsec_entry {
> + u8 type;
> + u8 bar_rev;
> + u16 off_lo;
> + u32 off_hi;
> + u8 ver_type;
> + u8 minor;
> + u8 major;
> + u8 rsvd0;
> + u32 rsvd1;
> +} __packed;
> +
> +#define read_entry(g, i, e) \
> + do { \
> + u32 *p = (u32 *)((g)->base + \
> + sizeof(struct xrt_vsec_header) + \
> + (i) * sizeof(struct xrt_vsec_entry)); \
> + u32 off; \
> + for (off = 0; \
> + off < sizeof(struct xrt_vsec_entry) / 4; \
> + off++) \
> + *((u32 *)(e) + off) = ioread32(p + off);\
> + } while (0)
This could be a static inline func.
> +
> +struct vsec_device {
> + u8 type;
> + char *ep_name;
> + ulong size;
> + char *regmap;
> +};
> +
> +static struct vsec_device vsec_devs[] = {
> + {
> + .type = VSEC_TYPE_UUID,
> + .ep_name = XRT_MD_NODE_BLP_ROM,
> + .size = VSEC_UUID_LEN,
> + .regmap = "vsec-uuid",
> + },
> + {
> + .type = VSEC_TYPE_FLASH,
> + .ep_name = XRT_MD_NODE_FLASH_VSEC,
> + .size = 4096,
> + .regmap = "vsec-flash",
> + },
> + {
> + .type = VSEC_TYPE_PLATINFO,
> + .ep_name = XRT_MD_NODE_PLAT_INFO,
> + .size = 4,
> + .regmap = "vsec-platinfo",
> + },
> + {
> + .type = VSEC_TYPE_MAILBOX,
> + .ep_name = XRT_MD_NODE_MAILBOX_VSEC,
> + .size = 48,
> + .regmap = "vsec-mbx",
> + },
This is a static list, how would a new type be added to this ?
> +};
> +
> +struct xrt_vsec {
> + struct platform_device *pdev;
> + void *base;
> + ulong length;
> +
> + char *metadata;
> + char uuid[VSEC_UUID_LEN];
> +};
> +
> +static char *type2epname(u32 type)
> +{
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
> + if (vsec_devs[i].type == type)
> + return (vsec_devs[i].ep_name);
> + }
> +
> + return NULL;
> +}
> +
> +static ulong type2size(u32 type)
> +{
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
> + if (vsec_devs[i].type == type)
> + return (vsec_devs[i].size);
> + }
> +
> + return 0;
> +}
> +
> +static char *type2regmap(u32 type)
> +{
> + int i;
> +
> + for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
> + if (vsec_devs[i].type == type)
> + return (vsec_devs[i].regmap);
> + }
> +
> + return NULL;
> +}
> +
> +static int xrt_vsec_add_node(struct xrt_vsec *vsec,
> + void *md_blob, struct xrt_vsec_entry *p_entry)
> +{
> + struct xrt_md_endpoint ep;
> + char regmap_ver[64];
> + int ret;
> +
> + if (!type2epname(p_entry->type))
> + return -EINVAL;
> +
> + /*
> + * VSEC may have more than 1 mailbox instance for the card
> + * which has more than 1 physical function.
> + * This is not supported for now. Assuming only one mailbox
> + */
are multiple uuid types allowed ?
this says assume 1, but logic will recreate 1+
can you check if a mbx ep exists before creating ?
> +
> + snprintf(regmap_ver, sizeof(regmap_ver) - 1, "%d-%d.%d.%d",
> + p_entry->ver_type, p_entry->major, p_entry->minor,
> + GET_REV(p_entry));
> + ep.ep_name = type2epname(p_entry->type);
> + ep.bar = GET_BAR(p_entry);
> + ep.bar_off = GET_BAR_OFF(p_entry);
here is the bar_off type overlow
> + ep.size = type2size(p_entry->type);
> + ep.regmap = type2regmap(p_entry->type);
> + ep.regmap_ver = regmap_ver;
> + ret = xrt_md_add_endpoint(DEV(vsec->pdev), vsec->metadata, &ep);
> + if (ret) {
> + xrt_err(vsec->pdev, "add ep failed, ret %d", ret);
> + goto failed;
> + }
> +
> +failed:
> + return ret;
> +}
> +
> +static int xrt_vsec_create_metadata(struct xrt_vsec *vsec)
> +{
> + struct xrt_vsec_entry entry;
> + int i, ret;
> +
> + ret = xrt_md_create(&vsec->pdev->dev, &vsec->metadata);
> + if (ret) {
> + xrt_err(vsec->pdev, "create metadata failed");
> + return ret;
> + }
> +
> + for (i = 0; i * sizeof(entry) < vsec->length -
> + sizeof(struct xrt_vsec_header); i++) {
> + read_entry(vsec, i, &entry);
> + xrt_vsec_add_node(vsec, vsec->metadata, &entry);
This can fail.
> + }
> +
> + return 0;
> +}
> +
> +static int xrt_vsec_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + int ret = 0;
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + /* Does not handle any event. */
> + break;
> + default:
> + ret = -EINVAL;
> + xrt_err(pdev, "should never been called");
> + break;
> + }
This function looks like a noop. Is anything going to be added to this later ?
> +
> + return ret;
> +}
> +
> +static int xrt_vsec_mapio(struct xrt_vsec *vsec)
> +{
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(vsec->pdev);
> + const u32 *bar;
> + const u64 *bar_off;
> + struct resource *res = NULL;
> + ulong addr;
> + int ret;
> +
> + if (!pdata || xrt_md_size(DEV(vsec->pdev), pdata->xsp_dtb) == XRT_MD_INVALID_LENGTH) {
> + xrt_err(vsec->pdev, "empty metadata");
> + return -EINVAL;
> + }
> +
> + ret = xrt_md_get_prop(DEV(vsec->pdev), pdata->xsp_dtb, XRT_MD_NODE_VSEC,
> + NULL, XRT_MD_PROP_BAR_IDX, (const void **)&bar, NULL);
> + if (ret) {
> + xrt_err(vsec->pdev, "failed to get bar idx, ret %d", ret);
> + return -EINVAL;
> + }
> +
> + ret = xrt_md_get_prop(DEV(vsec->pdev), pdata->xsp_dtb, XRT_MD_NODE_VSEC,
> + NULL, XRT_MD_PROP_OFFSET, (const void **)&bar_off, NULL);
> + if (ret) {
> + xrt_err(vsec->pdev, "failed to get bar off, ret %d", ret);
> + return -EINVAL;
> + }
> +
> + xrt_info(vsec->pdev, "Map vsec at bar %d, offset 0x%llx",
> + be32_to_cpu(*bar), be64_to_cpu(*bar_off));
> +
> + xleaf_get_barres(vsec->pdev, &res, be32_to_cpu(*bar));
> + if (!res) {
> + xrt_err(vsec->pdev, "failed to get bar addr");
> + return -EINVAL;
> + }
> +
> + addr = res->start + (ulong)be64_to_cpu(*bar_off);
review this type, addr is ulong and bar_off is not.
> +
> + vsec->base = ioremap(addr, sizeof(struct xrt_vsec_header));
> + if (!vsec->base) {
> + xrt_err(vsec->pdev, "Map header failed");
> + return -EIO;
> + }
why the double call on ioremap ?
just do the last one.
> +
> + vsec->length = head_rd(vsec, length);
> + iounmap(vsec->base);
> + vsec->base = ioremap(addr, vsec->length);
> + if (!vsec->base) {
> + xrt_err(vsec->pdev, "map failed");
> + return -EIO;
> + }
> +
> + return 0;
> +}
> +
> +static int xrt_vsec_remove(struct platform_device *pdev)
> +{
> + struct xrt_vsec *vsec;
> +
> + vsec = platform_get_drvdata(pdev);
> +
> + if (vsec->base) {
> + iounmap(vsec->base);
> + vsec->base = NULL;
> + }
> +
> + vfree(vsec->metadata);
> +
> + return 0;
> +}
> +
> +static int xrt_vsec_probe(struct platform_device *pdev)
> +{
> + struct xrt_vsec *vsec;
> + int ret = 0;
> +
> + vsec = devm_kzalloc(&pdev->dev, sizeof(*vsec), GFP_KERNEL);
> + if (!vsec)
> + return -ENOMEM;
> +
> + vsec->pdev = pdev;
> + platform_set_drvdata(pdev, vsec);
> +
> + ret = xrt_vsec_mapio(vsec);
> + if (ret)
> + goto failed;
> +
> + ret = xrt_vsec_create_metadata(vsec);
> + if (ret) {
> + xrt_err(pdev, "create metadata failed, ret %d", ret);
> + goto failed;
> + }
> + ret = xleaf_create_group(pdev, vsec->metadata);
> + if (ret < 0)
> + xrt_err(pdev, "create group failed, ret %d", ret);
> + else
> + ret = 0;
why is it just
if (ret)
fail ?
Tom
> +
> +failed:
> + if (ret)
> + xrt_vsec_remove(pdev);
> +
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_vsec_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names []){
> + { .ep_name = XRT_MD_NODE_VSEC },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_vsec_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_vsec_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_vsec_table[] = {
> + { XRT_VSEC, (kernel_ulong_t)&xrt_vsec_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_vsec_driver = {
> + .driver = {
> + .name = XRT_VSEC,
> + },
> + .probe = xrt_vsec_probe,
> + .remove = xrt_vsec_remove,
> + .id_table = xrt_vsec_table,
> +};
> +
> +void vsec_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_VSEC, &xrt_vsec_driver, xrt_vsec_endpoints);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_VSEC);
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 10/18] fpga: xrt: VSEC platform driver
2021-03-01 19:01 ` Tom Rix
@ 2021-03-05 19:58 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-05 19:58 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 03/01/2021 11:01 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Add VSEC driver. VSEC is a hardware function discovered by walking
>> PCI Express configure space. A platform device node will be created
>> for it. VSEC provides board logic UUID and few offset of other hardware
>> functions.
> Is this vsec walking infra or is a general find a list of mmio regions that need to be mapped in and do the mapping in as a set of platform drivers ?
vsec is pointed by PCIe vender-specific capability. And vsec itself
locates on PCI BAR. vsec has a list of minimum IPs (mmio regions)
required for driver to load firmware and communicate with the other pcie
function. After firmware is loaded, xrt will look into the fireware
metadata to get the information of rest IPs.
vsec driver notifies the root driver for the list of minimum IPs been
discovered. Then the root driver will create platform device nodes and
bring up drivers based on vsec's notification.
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/lib/xleaf/vsec.c | 359 ++++++++++++++++++++++++++++++
>> 1 file changed, 359 insertions(+)
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
>>
>> diff --git a/drivers/fpga/xrt/lib/xleaf/vsec.c b/drivers/fpga/xrt/lib/xleaf/vsec.c
>> new file mode 100644
>> index 000000000000..8e5cb22522ec
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/vsec.c
>> @@ -0,0 +1,359 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA VSEC Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#include <linux/platform_device.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +
>> +#define XRT_VSEC "xrt_vsec"
>> +
>> +#define VSEC_TYPE_UUID 0x50
>> +#define VSEC_TYPE_FLASH 0x51
>> +#define VSEC_TYPE_PLATINFO 0x52
>> +#define VSEC_TYPE_MAILBOX 0x53
>> +#define VSEC_TYPE_END 0xff
> Type of devices, this list can not grow much.
Because vsec only contains minimum required IPs for loading firmware and
communication. The list will only change when there is major hardware
change.
>> +
>> +#define VSEC_UUID_LEN 16
>> +
>> +struct xrt_vsec_header {
>> + u32 format;
>> + u32 length;
>> + u32 entry_sz;
>> + u32 rsvd;
>> +} __packed;
>> +
>> +#define head_rd(g, r) \
>> + ioread32((void *)(g)->base + offsetof(struct xrt_vsec_header, r))
>> +
>> +#define GET_BAR(entry) (((entry)->bar_rev >> 4) & 0xf)
>> +#define GET_BAR_OFF(_entry) \
>> + ({ typeof(_entry) entry = (_entry); \
>> + ((entry)->off_lo | ((u64)(entry)->off_hi << 16)); })
> A 48 bit value stored in xrt_md_endpoint.bar_off (long)
>
> bar_off should be u64
Will fix this.
>
>> +#define GET_REV(entry) ((entry)->bar_rev & 0xf)
>> +
> I prefer functions over macros.
Will change to inline function.
>> +struct xrt_vsec_entry {
>> + u8 type;
>> + u8 bar_rev;
>> + u16 off_lo;
>> + u32 off_hi;
>> + u8 ver_type;
>> + u8 minor;
>> + u8 major;
>> + u8 rsvd0;
>> + u32 rsvd1;
>> +} __packed;
>> +
>> +#define read_entry(g, i, e) \
>> + do { \
>> + u32 *p = (u32 *)((g)->base + \
>> + sizeof(struct xrt_vsec_header) + \
>> + (i) * sizeof(struct xrt_vsec_entry)); \
>> + u32 off; \
>> + for (off = 0; \
>> + off < sizeof(struct xrt_vsec_entry) / 4; \
>> + off++) \
>> + *((u32 *)(e) + off) = ioread32(p + off);\
>> + } while (0)
> This could be a static inline func.
Will change to inline function.
>> +
>> +struct vsec_device {
>> + u8 type;
>> + char *ep_name;
>> + ulong size;
>> + char *regmap;
>> +};
>> +
>> +static struct vsec_device vsec_devs[] = {
>> + {
>> + .type = VSEC_TYPE_UUID,
>> + .ep_name = XRT_MD_NODE_BLP_ROM,
>> + .size = VSEC_UUID_LEN,
>> + .regmap = "vsec-uuid",
>> + },
>> + {
>> + .type = VSEC_TYPE_FLASH,
>> + .ep_name = XRT_MD_NODE_FLASH_VSEC,
>> + .size = 4096,
>> + .regmap = "vsec-flash",
>> + },
>> + {
>> + .type = VSEC_TYPE_PLATINFO,
>> + .ep_name = XRT_MD_NODE_PLAT_INFO,
>> + .size = 4,
>> + .regmap = "vsec-platinfo",
>> + },
>> + {
>> + .type = VSEC_TYPE_MAILBOX,
>> + .ep_name = XRT_MD_NODE_MAILBOX_VSEC,
>> + .size = 48,
>> + .regmap = "vsec-mbx",
>> + },
> This is a static list, how would a new type be added to this ?
Because the list will only change when there is major hardware change,
the list will be update manually if hardware introduces a new type.
>> +};
>> +
>> +struct xrt_vsec {
>> + struct platform_device *pdev;
>> + void *base;
>> + ulong length;
>> +
>> + char *metadata;
>> + char uuid[VSEC_UUID_LEN];
>> +};
>> +
>> +static char *type2epname(u32 type)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
>> + if (vsec_devs[i].type == type)
>> + return (vsec_devs[i].ep_name);
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static ulong type2size(u32 type)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
>> + if (vsec_devs[i].type == type)
>> + return (vsec_devs[i].size);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static char *type2regmap(u32 type)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < ARRAY_SIZE(vsec_devs); i++) {
>> + if (vsec_devs[i].type == type)
>> + return (vsec_devs[i].regmap);
>> + }
>> +
>> + return NULL;
>> +}
>> +
>> +static int xrt_vsec_add_node(struct xrt_vsec *vsec,
>> + void *md_blob, struct xrt_vsec_entry *p_entry)
>> +{
>> + struct xrt_md_endpoint ep;
>> + char regmap_ver[64];
>> + int ret;
>> +
>> + if (!type2epname(p_entry->type))
>> + return -EINVAL;
>> +
>> + /*
>> + * VSEC may have more than 1 mailbox instance for the card
>> + * which has more than 1 physical function.
>> + * This is not supported for now. Assuming only one mailbox
>> + */
> are multiple uuid types allowed ?
No. And there will be only one uuid in vsec list.
>
> this says assume 1, but logic will recreate 1+
>
> can you check if a mbx ep exists before creating ?
Maybe the comment is confusing. All current Alveo boards only have one
mailbox in vsec list. In theory, there could be more than 1 mailboxes in
the future. And how it will present in vsec list is undetermined.
>
>> +
>> + snprintf(regmap_ver, sizeof(regmap_ver) - 1, "%d-%d.%d.%d",
>> + p_entry->ver_type, p_entry->major, p_entry->minor,
>> + GET_REV(p_entry));
>> + ep.ep_name = type2epname(p_entry->type);
>> + ep.bar = GET_BAR(p_entry);
>> + ep.bar_off = GET_BAR_OFF(p_entry);
> here is the bar_off type overlow
Will fix it.
>> + ep.size = type2size(p_entry->type);
>> + ep.regmap = type2regmap(p_entry->type);
>> + ep.regmap_ver = regmap_ver;
>> + ret = xrt_md_add_endpoint(DEV(vsec->pdev), vsec->metadata, &ep);
>> + if (ret) {
>> + xrt_err(vsec->pdev, "add ep failed, ret %d", ret);
>> + goto failed;
>> + }
>> +
>> +failed:
>> + return ret;
>> +}
>> +
>> +static int xrt_vsec_create_metadata(struct xrt_vsec *vsec)
>> +{
>> + struct xrt_vsec_entry entry;
>> + int i, ret;
>> +
>> + ret = xrt_md_create(&vsec->pdev->dev, &vsec->metadata);
>> + if (ret) {
>> + xrt_err(vsec->pdev, "create metadata failed");
>> + return ret;
>> + }
>> +
>> + for (i = 0; i * sizeof(entry) < vsec->length -
>> + sizeof(struct xrt_vsec_header); i++) {
>> + read_entry(vsec, i, &entry);
>> + xrt_vsec_add_node(vsec, vsec->metadata, &entry);
> This can fail.
Will add check.
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_vsec_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + int ret = 0;
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + /* Does not handle any event. */
>> + break;
>> + default:
>> + ret = -EINVAL;
>> + xrt_err(pdev, "should never been called");
>> + break;
>> + }
> This function looks like a noop. Is anything going to be added to this later ?
It could be. And there are broadcast events can reach to this handler. I
think it is harmless to ignore and return.
>> +
>> + return ret;
>> +}
>> +
>> +static int xrt_vsec_mapio(struct xrt_vsec *vsec)
>> +{
>> + struct xrt_subdev_platdata *pdata = DEV_PDATA(vsec->pdev);
>> + const u32 *bar;
>> + const u64 *bar_off;
>> + struct resource *res = NULL;
>> + ulong addr;
>> + int ret;
>> +
>> + if (!pdata || xrt_md_size(DEV(vsec->pdev), pdata->xsp_dtb) == XRT_MD_INVALID_LENGTH) {
>> + xrt_err(vsec->pdev, "empty metadata");
>> + return -EINVAL;
>> + }
>> +
>> + ret = xrt_md_get_prop(DEV(vsec->pdev), pdata->xsp_dtb, XRT_MD_NODE_VSEC,
>> + NULL, XRT_MD_PROP_BAR_IDX, (const void **)&bar, NULL);
>> + if (ret) {
>> + xrt_err(vsec->pdev, "failed to get bar idx, ret %d", ret);
>> + return -EINVAL;
>> + }
>> +
>> + ret = xrt_md_get_prop(DEV(vsec->pdev), pdata->xsp_dtb, XRT_MD_NODE_VSEC,
>> + NULL, XRT_MD_PROP_OFFSET, (const void **)&bar_off, NULL);
>> + if (ret) {
>> + xrt_err(vsec->pdev, "failed to get bar off, ret %d", ret);
>> + return -EINVAL;
>> + }
>> +
>> + xrt_info(vsec->pdev, "Map vsec at bar %d, offset 0x%llx",
>> + be32_to_cpu(*bar), be64_to_cpu(*bar_off));
>> +
>> + xleaf_get_barres(vsec->pdev, &res, be32_to_cpu(*bar));
>> + if (!res) {
>> + xrt_err(vsec->pdev, "failed to get bar addr");
>> + return -EINVAL;
>> + }
>> +
>> + addr = res->start + (ulong)be64_to_cpu(*bar_off);
> review this type, addr is ulong and bar_off is not.
Will use u64.
>> +
>> + vsec->base = ioremap(addr, sizeof(struct xrt_vsec_header));
>> + if (!vsec->base) {
>> + xrt_err(vsec->pdev, "Map header failed");
>> + return -EIO;
>> + }
> why the double call on ioremap ?
>
> just do the last one.
The first ioremap only maps in the header and read out the length of the
body (mmio list).
Then the next ioremap maps in the body based on the length.
>
>> +
>> + vsec->length = head_rd(vsec, length);
>> + iounmap(vsec->base);
>> + vsec->base = ioremap(addr, vsec->length);
>> + if (!vsec->base) {
>> + xrt_err(vsec->pdev, "map failed");
>> + return -EIO;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_vsec_remove(struct platform_device *pdev)
>> +{
>> + struct xrt_vsec *vsec;
>> +
>> + vsec = platform_get_drvdata(pdev);
>> +
>> + if (vsec->base) {
>> + iounmap(vsec->base);
>> + vsec->base = NULL;
>> + }
>> +
>> + vfree(vsec->metadata);
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_vsec_probe(struct platform_device *pdev)
>> +{
>> + struct xrt_vsec *vsec;
>> + int ret = 0;
>> +
>> + vsec = devm_kzalloc(&pdev->dev, sizeof(*vsec), GFP_KERNEL);
>> + if (!vsec)
>> + return -ENOMEM;
>> +
>> + vsec->pdev = pdev;
>> + platform_set_drvdata(pdev, vsec);
>> +
>> + ret = xrt_vsec_mapio(vsec);
>> + if (ret)
>> + goto failed;
>> +
>> + ret = xrt_vsec_create_metadata(vsec);
>> + if (ret) {
>> + xrt_err(pdev, "create metadata failed, ret %d", ret);
>> + goto failed;
>> + }
>> + ret = xleaf_create_group(pdev, vsec->metadata);
>> + if (ret < 0)
>> + xrt_err(pdev, "create group failed, ret %d", ret);
>> + else
>> + ret = 0;
> why is it just
>
> if (ret)
>
> fail ?
xleaf_create_group() returns 0 or positive id on success. I will change to
if (ret < 0)
goto fail;
return 0
fail:
Thanks,
Lizhi
>
> Tom
>
>> +
>> +failed:
>> + if (ret)
>> + xrt_vsec_remove(pdev);
>> +
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_vsec_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names []){
>> + { .ep_name = XRT_MD_NODE_VSEC },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_vsec_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_vsec_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_vsec_table[] = {
>> + { XRT_VSEC, (kernel_ulong_t)&xrt_vsec_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_vsec_driver = {
>> + .driver = {
>> + .name = XRT_VSEC,
>> + },
>> + .probe = xrt_vsec_probe,
>> + .remove = xrt_vsec_remove,
>> + .id_table = xrt_vsec_table,
>> +};
>> +
>> +void vsec_leaf_init_fini(bool init)
>> +{
>> + if (init)
>> + xleaf_register_driver(XRT_SUBDEV_VSEC, &xrt_vsec_driver, xrt_vsec_endpoints);
>> + else
>> + xleaf_unregister_driver(XRT_SUBDEV_VSEC);
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 11/18] fpga: xrt: UCS platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (9 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 10/18] fpga: xrt: VSEC platform driver Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-03-02 16:09 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP " Lizhi Hou
` (8 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Add UCS driver. UCS is a hardware function discovered by walking xclbin
metadata. A platform device node will be created for it.
UCS enables/disables the dynamic region clocks.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xleaf/ucs.h | 24 +++
drivers/fpga/xrt/lib/xleaf/ucs.c | 235 +++++++++++++++++++++++++++
2 files changed, 259 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
diff --git a/drivers/fpga/xrt/include/xleaf/ucs.h b/drivers/fpga/xrt/include/xleaf/ucs.h
new file mode 100644
index 000000000000..a5ef0e100e12
--- /dev/null
+++ b/drivers/fpga/xrt/include/xleaf/ucs.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for XRT UCS Leaf Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_UCS_H_
+#define _XRT_UCS_H_
+
+#include "xleaf.h"
+
+/*
+ * UCS driver IOCTL calls.
+ */
+enum xrt_ucs_ioctl_cmd {
+ XRT_UCS_CHECK = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+ XRT_UCS_ENABLE,
+};
+
+#endif /* _XRT_UCS_H_ */
diff --git a/drivers/fpga/xrt/lib/xleaf/ucs.c b/drivers/fpga/xrt/lib/xleaf/ucs.c
new file mode 100644
index 000000000000..ae762c8fddbb
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xleaf/ucs.c
@@ -0,0 +1,235 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA UCS Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou<Lizhi.Hou@xilinx.com>
+ */
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include "metadata.h"
+#include "xleaf.h"
+#include "xleaf/ucs.h"
+#include "xleaf/clock.h"
+
+#define UCS_ERR(ucs, fmt, arg...) \
+ xrt_err((ucs)->pdev, fmt "\n", ##arg)
+#define UCS_WARN(ucs, fmt, arg...) \
+ xrt_warn((ucs)->pdev, fmt "\n", ##arg)
+#define UCS_INFO(ucs, fmt, arg...) \
+ xrt_info((ucs)->pdev, fmt "\n", ##arg)
+#define UCS_DBG(ucs, fmt, arg...) \
+ xrt_dbg((ucs)->pdev, fmt "\n", ##arg)
+
+#define XRT_UCS "xrt_ucs"
+
+#define CHANNEL1_OFFSET 0
+#define CHANNEL2_OFFSET 8
+
+#define CLK_MAX_VALUE 6400
+
+struct ucs_control_status_ch1 {
+ unsigned int shutdown_clocks_latched:1;
+ unsigned int reserved1:15;
+ unsigned int clock_throttling_average:14;
+ unsigned int reserved2:2;
+};
+
+struct xrt_ucs {
+ struct platform_device *pdev;
+ void __iomem *ucs_base;
+ struct mutex ucs_lock; /* ucs dev lock */
+};
+
+static inline u32 reg_rd(struct xrt_ucs *ucs, u32 offset)
+{
+ return ioread32(ucs->ucs_base + offset);
+}
+
+static inline void reg_wr(struct xrt_ucs *ucs, u32 val, u32 offset)
+{
+ iowrite32(val, ucs->ucs_base + offset);
+}
+
+static void xrt_ucs_event_cb(struct platform_device *pdev, void *arg)
+{
+ struct platform_device *leaf;
+ struct xrt_event *evt = (struct xrt_event *)arg;
+ enum xrt_events e = evt->xe_evt;
+ enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
+ int instance = evt->xe_subdev.xevt_subdev_instance;
+
+ switch (e) {
+ case XRT_EVENT_POST_CREATION:
+ break;
+ default:
+ xrt_dbg(pdev, "ignored event %d", e);
+ return;
+ }
+
+ if (id != XRT_SUBDEV_CLOCK)
+ return;
+
+ leaf = xleaf_get_leaf_by_id(pdev, XRT_SUBDEV_CLOCK, instance);
+ if (!leaf) {
+ xrt_err(pdev, "does not get clock subdev");
+ return;
+ }
+
+ xleaf_ioctl(leaf, XRT_CLOCK_VERIFY, NULL);
+ xleaf_put_leaf(pdev, leaf);
+}
+
+static void ucs_check(struct xrt_ucs *ucs, bool *latched)
+{
+ struct ucs_control_status_ch1 *ucs_status_ch1;
+ u32 status;
+
+ mutex_lock(&ucs->ucs_lock);
+ status = reg_rd(ucs, CHANNEL1_OFFSET);
+ ucs_status_ch1 = (struct ucs_control_status_ch1 *)&status;
+ if (ucs_status_ch1->shutdown_clocks_latched) {
+ UCS_ERR(ucs,
+ "Critical temperature or power event, kernel clocks have been stopped.");
+ UCS_ERR(ucs,
+ "run 'xbutil valiate -q' to continue. See AR 73398 for more details.");
+ /* explicitly indicate reset should be latched */
+ *latched = true;
+ } else if (ucs_status_ch1->clock_throttling_average >
+ CLK_MAX_VALUE) {
+ UCS_ERR(ucs, "kernel clocks %d exceeds expected maximum value %d.",
+ ucs_status_ch1->clock_throttling_average,
+ CLK_MAX_VALUE);
+ } else if (ucs_status_ch1->clock_throttling_average) {
+ UCS_ERR(ucs, "kernel clocks throttled at %d%%.",
+ (ucs_status_ch1->clock_throttling_average /
+ (CLK_MAX_VALUE / 100)));
+ }
+ mutex_unlock(&ucs->ucs_lock);
+}
+
+static void ucs_enable(struct xrt_ucs *ucs)
+{
+ reg_wr(ucs, 1, CHANNEL2_OFFSET);
+}
+
+static int
+xrt_ucs_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ struct xrt_ucs *ucs;
+ int ret = 0;
+
+ ucs = platform_get_drvdata(pdev);
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ xrt_ucs_event_cb(pdev, arg);
+ break;
+ case XRT_UCS_CHECK: {
+ ucs_check(ucs, (bool *)arg);
+ break;
+ }
+ case XRT_UCS_ENABLE:
+ ucs_enable(ucs);
+ break;
+ default:
+ xrt_err(pdev, "unsupported cmd %d", cmd);
+ return -EINVAL;
+ }
+
+ return ret;
+}
+
+static int ucs_remove(struct platform_device *pdev)
+{
+ struct xrt_ucs *ucs;
+
+ ucs = platform_get_drvdata(pdev);
+ if (!ucs) {
+ xrt_err(pdev, "driver data is NULL");
+ return -EINVAL;
+ }
+
+ if (ucs->ucs_base)
+ iounmap(ucs->ucs_base);
+
+ platform_set_drvdata(pdev, NULL);
+ devm_kfree(&pdev->dev, ucs);
+
+ return 0;
+}
+
+static int ucs_probe(struct platform_device *pdev)
+{
+ struct xrt_ucs *ucs = NULL;
+ struct resource *res;
+ int ret;
+
+ ucs = devm_kzalloc(&pdev->dev, sizeof(*ucs), GFP_KERNEL);
+ if (!ucs)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, ucs);
+ ucs->pdev = pdev;
+ mutex_init(&ucs->ucs_lock);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ ucs->ucs_base = ioremap(res->start, res->end - res->start + 1);
+ if (!ucs->ucs_base) {
+ UCS_ERR(ucs, "map base %pR failed", res);
+ ret = -EFAULT;
+ goto failed;
+ }
+ ucs_enable(ucs);
+
+ return 0;
+
+failed:
+ ucs_remove(pdev);
+ return ret;
+}
+
+static struct xrt_subdev_endpoints xrt_ucs_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ { .ep_name = XRT_MD_NODE_UCS_CONTROL_STATUS },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xrt_ucs_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_ucs_leaf_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_ucs_table[] = {
+ { XRT_UCS, (kernel_ulong_t)&xrt_ucs_data },
+ { },
+};
+
+static struct platform_driver xrt_ucs_driver = {
+ .driver = {
+ .name = XRT_UCS,
+ },
+ .probe = ucs_probe,
+ .remove = ucs_remove,
+ .id_table = xrt_ucs_table,
+};
+
+void ucs_leaf_init_fini(bool init)
+{
+ if (init)
+ xleaf_register_driver(XRT_SUBDEV_UCS, &xrt_ucs_driver, xrt_ucs_endpoints);
+ else
+ xleaf_unregister_driver(XRT_SUBDEV_UCS);
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 11/18] fpga: xrt: UCS platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 11/18] fpga: xrt: UCS " Lizhi Hou
@ 2021-03-02 16:09 ` Tom Rix
2021-03-10 20:24 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-02 16:09 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Add UCS driver. UCS is a hardware function discovered by walking xclbin
What does UCS stand for ? add to commit log
> metadata. A platform device node will be created for it.
> UCS enables/disables the dynamic region clocks.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/ucs.h | 24 +++
> drivers/fpga/xrt/lib/xleaf/ucs.c | 235 +++++++++++++++++++++++++++
> 2 files changed, 259 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/ucs.h b/drivers/fpga/xrt/include/xleaf/ucs.h
> new file mode 100644
> index 000000000000..a5ef0e100e12
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/ucs.h
This header is only used by ucs.c, so is it needed ?
could the enum be defined in ucs.c ?
> @@ -0,0 +1,24 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT UCS Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_UCS_H_
> +#define _XRT_UCS_H_
> +
> +#include "xleaf.h"
> +
> +/*
> + * UCS driver IOCTL calls.
> + */
> +enum xrt_ucs_ioctl_cmd {
> + XRT_UCS_CHECK = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> + XRT_UCS_ENABLE,
no disable ?
> +};
> +
> +#endif /* _XRT_UCS_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/ucs.c b/drivers/fpga/xrt/lib/xleaf/ucs.c
> new file mode 100644
> index 000000000000..ae762c8fddbb
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/ucs.c
> @@ -0,0 +1,235 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA UCS Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/ucs.h"
> +#include "xleaf/clock.h"
> +
> +#define UCS_ERR(ucs, fmt, arg...) \
> + xrt_err((ucs)->pdev, fmt "\n", ##arg)
> +#define UCS_WARN(ucs, fmt, arg...) \
> + xrt_warn((ucs)->pdev, fmt "\n", ##arg)
> +#define UCS_INFO(ucs, fmt, arg...) \
> + xrt_info((ucs)->pdev, fmt "\n", ##arg)
> +#define UCS_DBG(ucs, fmt, arg...) \
> + xrt_dbg((ucs)->pdev, fmt "\n", ##arg)
> +
> +#define XRT_UCS "xrt_ucs"
> +
> +#define CHANNEL1_OFFSET 0
> +#define CHANNEL2_OFFSET 8
> +
> +#define CLK_MAX_VALUE 6400
> +
> +struct ucs_control_status_ch1 {
> + unsigned int shutdown_clocks_latched:1;
> + unsigned int reserved1:15;
> + unsigned int clock_throttling_average:14;
> + unsigned int reserved2:2;
> +};
Likely needs to be packed and/or the unsigned int changed to u32
> +
> +struct xrt_ucs {
> + struct platform_device *pdev;
> + void __iomem *ucs_base;
> + struct mutex ucs_lock; /* ucs dev lock */
> +};
> +
> +static inline u32 reg_rd(struct xrt_ucs *ucs, u32 offset)
> +{
> + return ioread32(ucs->ucs_base + offset);
> +}
> +
> +static inline void reg_wr(struct xrt_ucs *ucs, u32 val, u32 offset)
> +{
> + iowrite32(val, ucs->ucs_base + offset);
> +}
> +
> +static void xrt_ucs_event_cb(struct platform_device *pdev, void *arg)
> +{
> + struct platform_device *leaf;
> + struct xrt_event *evt = (struct xrt_event *)arg;
> + enum xrt_events e = evt->xe_evt;
> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
> + int instance = evt->xe_subdev.xevt_subdev_instance;
> +
> + switch (e) {
> + case XRT_EVENT_POST_CREATION:
> + break;
> + default:
> + xrt_dbg(pdev, "ignored event %d", e);
> + return;
> + }
this switch is a noop, remove
> +
> + if (id != XRT_SUBDEV_CLOCK)
> + return;
> +
> + leaf = xleaf_get_leaf_by_id(pdev, XRT_SUBDEV_CLOCK, instance);
> + if (!leaf) {
> + xrt_err(pdev, "does not get clock subdev");
> + return;
> + }
> +
> + xleaf_ioctl(leaf, XRT_CLOCK_VERIFY, NULL);
> + xleaf_put_leaf(pdev, leaf);
> +}
> +
> +static void ucs_check(struct xrt_ucs *ucs, bool *latched)
> +{
checking but not returning status, change to returning int.
this function is called but xrt_ucs_leaf_ioctl which does return status.
> + struct ucs_control_status_ch1 *ucs_status_ch1;
> + u32 status;
> +
> + mutex_lock(&ucs->ucs_lock);
> + status = reg_rd(ucs, CHANNEL1_OFFSET);
> + ucs_status_ch1 = (struct ucs_control_status_ch1 *)&status;
> + if (ucs_status_ch1->shutdown_clocks_latched) {
> + UCS_ERR(ucs,
> + "Critical temperature or power event, kernel clocks have been stopped.");
> + UCS_ERR(ucs,
> + "run 'xbutil valiate -q' to continue. See AR 73398 for more details.");
This error message does not seem like it would be useful, please review.
> + /* explicitly indicate reset should be latched */
> + *latched = true;
> + } else if (ucs_status_ch1->clock_throttling_average >
> + CLK_MAX_VALUE) {
> + UCS_ERR(ucs, "kernel clocks %d exceeds expected maximum value %d.",
> + ucs_status_ch1->clock_throttling_average,
> + CLK_MAX_VALUE);
> + } else if (ucs_status_ch1->clock_throttling_average) {
> + UCS_ERR(ucs, "kernel clocks throttled at %d%%.",
> + (ucs_status_ch1->clock_throttling_average /
> + (CLK_MAX_VALUE / 100)));
> + }
> + mutex_unlock(&ucs->ucs_lock);
> +}
> +
> +static void ucs_enable(struct xrt_ucs *ucs)
> +{
> + reg_wr(ucs, 1, CHANNEL2_OFFSET);
lock ?
> +}
> +
> +static int
> +xrt_ucs_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct xrt_ucs *ucs;
> + int ret = 0;
> +
> + ucs = platform_get_drvdata(pdev);
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + xrt_ucs_event_cb(pdev, arg);
> + break;
> + case XRT_UCS_CHECK: {
brace not needed here
> + ucs_check(ucs, (bool *)arg);
> + break;
> + }
> + case XRT_UCS_ENABLE:
> + ucs_enable(ucs);
> + break;
> + default:
> + xrt_err(pdev, "unsupported cmd %d", cmd);
> + return -EINVAL;
> + }
> +
> + return ret;
> +}
> +
> +static int ucs_remove(struct platform_device *pdev)
> +{
> + struct xrt_ucs *ucs;
> +
> + ucs = platform_get_drvdata(pdev);
> + if (!ucs) {
is this possible ?
Tom
> + xrt_err(pdev, "driver data is NULL");
> + return -EINVAL;
> + }
> +
> + if (ucs->ucs_base)
> + iounmap(ucs->ucs_base);
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, ucs);
> +
> + return 0;
> +}
> +
> +static int ucs_probe(struct platform_device *pdev)
> +{
> + struct xrt_ucs *ucs = NULL;
> + struct resource *res;
> + int ret;
> +
> + ucs = devm_kzalloc(&pdev->dev, sizeof(*ucs), GFP_KERNEL);
> + if (!ucs)
> + return -ENOMEM;
> +
> + platform_set_drvdata(pdev, ucs);
> + ucs->pdev = pdev;
> + mutex_init(&ucs->ucs_lock);
> +
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + ucs->ucs_base = ioremap(res->start, res->end - res->start + 1);
> + if (!ucs->ucs_base) {
> + UCS_ERR(ucs, "map base %pR failed", res);
> + ret = -EFAULT;
> + goto failed;
> + }
> + ucs_enable(ucs);
> +
> + return 0;
> +
> +failed:
> + ucs_remove(pdev);
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_ucs_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = XRT_MD_NODE_UCS_CONTROL_STATUS },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_ucs_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_ucs_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_ucs_table[] = {
> + { XRT_UCS, (kernel_ulong_t)&xrt_ucs_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_ucs_driver = {
> + .driver = {
> + .name = XRT_UCS,
> + },
> + .probe = ucs_probe,
> + .remove = ucs_remove,
> + .id_table = xrt_ucs_table,
> +};
> +
> +void ucs_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_UCS, &xrt_ucs_driver, xrt_ucs_endpoints);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_UCS);
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 11/18] fpga: xrt: UCS platform driver
2021-03-02 16:09 ` Tom Rix
@ 2021-03-10 20:24 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-10 20:24 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 03/02/2021 08:09 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Add UCS driver. UCS is a hardware function discovered by walking xclbin
> What does UCS stand for ? add to commit log
UCS stands for User Clock Subsystem. I will add it to log.
>> metadata. A platform device node will be created for it.
>> UCS enables/disables the dynamic region clocks.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/ucs.h | 24 +++
>> drivers/fpga/xrt/lib/xleaf/ucs.c | 235 +++++++++++++++++++++++++++
>> 2 files changed, 259 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
>>
>> diff --git a/drivers/fpga/xrt/include/xleaf/ucs.h b/drivers/fpga/xrt/include/xleaf/ucs.h
>> new file mode 100644
>> index 000000000000..a5ef0e100e12
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/ucs.h
> This header is only used by ucs.c, so is it needed ?
>
> could the enum be defined in ucs.c ?
It will be used in the future. I will remove it.
>
>> @@ -0,0 +1,24 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT UCS Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_UCS_H_
>> +#define _XRT_UCS_H_
>> +
>> +#include "xleaf.h"
>> +
>> +/*
>> + * UCS driver IOCTL calls.
>> + */
>> +enum xrt_ucs_ioctl_cmd {
>> + XRT_UCS_CHECK = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> + XRT_UCS_ENABLE,
> no disable ?
It does not need to disable because reset the fpga bridge will disable
it. I will remove ucs.h because it is not used in this patchset.
>> +};
>> +
>> +#endif /* _XRT_UCS_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/ucs.c b/drivers/fpga/xrt/lib/xleaf/ucs.c
>> new file mode 100644
>> index 000000000000..ae762c8fddbb
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/ucs.c
>> @@ -0,0 +1,235 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA UCS Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/io.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/ucs.h"
>> +#include "xleaf/clock.h"
>> +
>> +#define UCS_ERR(ucs, fmt, arg...) \
>> + xrt_err((ucs)->pdev, fmt "\n", ##arg)
>> +#define UCS_WARN(ucs, fmt, arg...) \
>> + xrt_warn((ucs)->pdev, fmt "\n", ##arg)
>> +#define UCS_INFO(ucs, fmt, arg...) \
>> + xrt_info((ucs)->pdev, fmt "\n", ##arg)
>> +#define UCS_DBG(ucs, fmt, arg...) \
>> + xrt_dbg((ucs)->pdev, fmt "\n", ##arg)
>> +
>> +#define XRT_UCS "xrt_ucs"
>> +
>> +#define CHANNEL1_OFFSET 0
>> +#define CHANNEL2_OFFSET 8
>> +
>> +#define CLK_MAX_VALUE 6400
>> +
>> +struct ucs_control_status_ch1 {
>> + unsigned int shutdown_clocks_latched:1;
>> + unsigned int reserved1:15;
>> + unsigned int clock_throttling_average:14;
>> + unsigned int reserved2:2;
>> +};
> Likely needs to be packed and/or the unsigned int changed to u32
Will remove this structure because it is not used in this patch set.
>> +
>> +struct xrt_ucs {
>> + struct platform_device *pdev;
>> + void __iomem *ucs_base;
>> + struct mutex ucs_lock; /* ucs dev lock */
>> +};
>> +
>> +static inline u32 reg_rd(struct xrt_ucs *ucs, u32 offset)
>> +{
>> + return ioread32(ucs->ucs_base + offset);
>> +}
>> +
>> +static inline void reg_wr(struct xrt_ucs *ucs, u32 val, u32 offset)
>> +{
>> + iowrite32(val, ucs->ucs_base + offset);
>> +}
>> +
>> +static void xrt_ucs_event_cb(struct platform_device *pdev, void *arg)
>> +{
>> + struct platform_device *leaf;
>> + struct xrt_event *evt = (struct xrt_event *)arg;
>> + enum xrt_events e = evt->xe_evt;
>> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
>> + int instance = evt->xe_subdev.xevt_subdev_instance;
>> +
>> + switch (e) {
>> + case XRT_EVENT_POST_CREATION:
>> + break;
>> + default:
>> + xrt_dbg(pdev, "ignored event %d", e);
>> + return;
>> + }
> this switch is a noop, remove
Will change to if (e != XRT_EVENT_POST_CREATION) return -EINVAL
>> +
>> + if (id != XRT_SUBDEV_CLOCK)
>> + return;
>> +
>> + leaf = xleaf_get_leaf_by_id(pdev, XRT_SUBDEV_CLOCK, instance);
>> + if (!leaf) {
>> + xrt_err(pdev, "does not get clock subdev");
>> + return;
>> + }
>> +
>> + xleaf_ioctl(leaf, XRT_CLOCK_VERIFY, NULL);
>> + xleaf_put_leaf(pdev, leaf);
>> +}
>> +
>> +static void ucs_check(struct xrt_ucs *ucs, bool *latched)
>> +{
> checking but not returning status, change to returning int.
>
> this function is called but xrt_ucs_leaf_ioctl which does return status.
Will remove ucs_check() because it is not used in this patch set.
>
>> + struct ucs_control_status_ch1 *ucs_status_ch1;
>> + u32 status;
>> +
>> + mutex_lock(&ucs->ucs_lock);
>> + status = reg_rd(ucs, CHANNEL1_OFFSET);
>> + ucs_status_ch1 = (struct ucs_control_status_ch1 *)&status;
>> + if (ucs_status_ch1->shutdown_clocks_latched) {
>> + UCS_ERR(ucs,
>> + "Critical temperature or power event, kernel clocks have been stopped.");
>> + UCS_ERR(ucs,
>> + "run 'xbutil valiate -q' to continue. See AR 73398 for more details.");
> This error message does not seem like it would be useful, please review.
>> + /* explicitly indicate reset should be latched */
>> + *latched = true;
>> + } else if (ucs_status_ch1->clock_throttling_average >
>> + CLK_MAX_VALUE) {
>> + UCS_ERR(ucs, "kernel clocks %d exceeds expected maximum value %d.",
>> + ucs_status_ch1->clock_throttling_average,
>> + CLK_MAX_VALUE);
>> + } else if (ucs_status_ch1->clock_throttling_average) {
>> + UCS_ERR(ucs, "kernel clocks throttled at %d%%.",
>> + (ucs_status_ch1->clock_throttling_average /
>> + (CLK_MAX_VALUE / 100)));
>> + }
>> + mutex_unlock(&ucs->ucs_lock);
>> +}
>> +
>> +static void ucs_enable(struct xrt_ucs *ucs)
>> +{
>> + reg_wr(ucs, 1, CHANNEL2_OFFSET);
> lock ?
Yes. will add it.
>> +}
>> +
>> +static int
>> +xrt_ucs_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + struct xrt_ucs *ucs;
>> + int ret = 0;
>> +
>> + ucs = platform_get_drvdata(pdev);
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + xrt_ucs_event_cb(pdev, arg);
>> + break;
>> + case XRT_UCS_CHECK: {
> brace not needed here
will remove.
>> + ucs_check(ucs, (bool *)arg);
>> + break;
>> + }
>> + case XRT_UCS_ENABLE:
>> + ucs_enable(ucs);
>> + break;
>> + default:
>> + xrt_err(pdev, "unsupported cmd %d", cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +static int ucs_remove(struct platform_device *pdev)
>> +{
>> + struct xrt_ucs *ucs;
>> +
>> + ucs = platform_get_drvdata(pdev);
>> + if (!ucs) {
> is this possible ?
Will remove.
Thanks,
Lizhi
>
> Tom
>
>> + xrt_err(pdev, "driver data is NULL");
>> + return -EINVAL;
>> + }
>> +
>> + if (ucs->ucs_base)
>> + iounmap(ucs->ucs_base);
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, ucs);
>> +
>> + return 0;
>> +}
>> +
>> +static int ucs_probe(struct platform_device *pdev)
>> +{
>> + struct xrt_ucs *ucs = NULL;
>> + struct resource *res;
>> + int ret;
>> +
>> + ucs = devm_kzalloc(&pdev->dev, sizeof(*ucs), GFP_KERNEL);
>> + if (!ucs)
>> + return -ENOMEM;
>> +
>> + platform_set_drvdata(pdev, ucs);
>> + ucs->pdev = pdev;
>> + mutex_init(&ucs->ucs_lock);
>> +
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + ucs->ucs_base = ioremap(res->start, res->end - res->start + 1);
>> + if (!ucs->ucs_base) {
>> + UCS_ERR(ucs, "map base %pR failed", res);
>> + ret = -EFAULT;
>> + goto failed;
>> + }
>> + ucs_enable(ucs);
>> +
>> + return 0;
>> +
>> +failed:
>> + ucs_remove(pdev);
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_ucs_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = XRT_MD_NODE_UCS_CONTROL_STATUS },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_ucs_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_ucs_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_ucs_table[] = {
>> + { XRT_UCS, (kernel_ulong_t)&xrt_ucs_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_ucs_driver = {
>> + .driver = {
>> + .name = XRT_UCS,
>> + },
>> + .probe = ucs_probe,
>> + .remove = ucs_remove,
>> + .id_table = xrt_ucs_table,
>> +};
>> +
>> +void ucs_leaf_init_fini(bool init)
>> +{
>> + if (init)
>> + xleaf_register_driver(XRT_SUBDEV_UCS, &xrt_ucs_driver, xrt_ucs_endpoints);
>> + else
>> + xleaf_unregister_driver(XRT_SUBDEV_UCS);
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (10 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 11/18] fpga: xrt: UCS " Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-21 20:24 ` Moritz Fischer
2021-03-03 15:12 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 13/18] fpga: xrt: devctl " Lizhi Hou
` (7 subsequent siblings)
19 siblings, 2 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Add ICAP driver. ICAP is a hardware function discovered by walking
firmware metadata. A platform device node will be created for it.
FPGA bitstream is written to hardware through ICAP.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xleaf/icap.h | 29 +++
drivers/fpga/xrt/lib/xleaf/icap.c | 317 ++++++++++++++++++++++++++
2 files changed, 346 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
diff --git a/drivers/fpga/xrt/include/xleaf/icap.h b/drivers/fpga/xrt/include/xleaf/icap.h
new file mode 100644
index 000000000000..a14fc0ffa78f
--- /dev/null
+++ b/drivers/fpga/xrt/include/xleaf/icap.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for XRT ICAP Leaf Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_ICAP_H_
+#define _XRT_ICAP_H_
+
+#include "xleaf.h"
+
+/*
+ * ICAP driver IOCTL calls.
+ */
+enum xrt_icap_ioctl_cmd {
+ XRT_ICAP_WRITE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+ XRT_ICAP_IDCODE,
+};
+
+struct xrt_icap_ioctl_wr {
+ void *xiiw_bit_data;
+ u32 xiiw_data_len;
+};
+
+#endif /* _XRT_ICAP_H_ */
diff --git a/drivers/fpga/xrt/lib/xleaf/icap.c b/drivers/fpga/xrt/lib/xleaf/icap.c
new file mode 100644
index 000000000000..0500a97bdef9
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xleaf/icap.c
@@ -0,0 +1,317 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA ICAP Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou<Lizhi.Hou@xilinx.com>
+ * Sonal Santan <sonals@xilinx.com>
+ * Max Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include "metadata.h"
+#include "xleaf.h"
+#include "xleaf/icap.h"
+#include "xclbin-helper.h"
+
+#define XRT_ICAP "xrt_icap"
+
+#define ICAP_ERR(icap, fmt, arg...) \
+ xrt_err((icap)->pdev, fmt "\n", ##arg)
+#define ICAP_WARN(icap, fmt, arg...) \
+ xrt_warn((icap)->pdev, fmt "\n", ##arg)
+#define ICAP_INFO(icap, fmt, arg...) \
+ xrt_info((icap)->pdev, fmt "\n", ##arg)
+#define ICAP_DBG(icap, fmt, arg...) \
+ xrt_dbg((icap)->pdev, fmt "\n", ##arg)
+
+/*
+ * AXI-HWICAP IP register layout
+ */
+struct icap_reg {
+ u32 ir_rsvd1[7];
+ u32 ir_gier;
+ u32 ir_isr;
+ u32 ir_rsvd2;
+ u32 ir_ier;
+ u32 ir_rsvd3[53];
+ u32 ir_wf;
+ u32 ir_rf;
+ u32 ir_sz;
+ u32 ir_cr;
+ u32 ir_sr;
+ u32 ir_wfv;
+ u32 ir_rfo;
+ u32 ir_asr;
+} __packed;
+
+struct icap {
+ struct platform_device *pdev;
+ struct icap_reg *icap_regs;
+ struct mutex icap_lock; /* icap dev lock */
+
+ unsigned int idcode;
+};
+
+static inline u32 reg_rd(void __iomem *reg)
+{
+ if (!reg)
+ return -1;
+
+ return ioread32(reg);
+}
+
+static inline void reg_wr(void __iomem *reg, u32 val)
+{
+ if (!reg)
+ return;
+
+ iowrite32(val, reg);
+}
+
+static int wait_for_done(struct icap *icap)
+{
+ u32 w;
+ int i = 0;
+
+ WARN_ON(!mutex_is_locked(&icap->icap_lock));
+ for (i = 0; i < 10; i++) {
+ udelay(5);
+ w = reg_rd(&icap->icap_regs->ir_sr);
+ ICAP_INFO(icap, "XHWICAP_SR: %x", w);
+ if (w & 0x5)
+ return 0;
+ }
+
+ ICAP_ERR(icap, "bitstream download timeout");
+ return -ETIMEDOUT;
+}
+
+static int icap_write(struct icap *icap, const u32 *word_buf, int size)
+{
+ int i;
+ u32 value = 0;
+
+ for (i = 0; i < size; i++) {
+ value = be32_to_cpu(word_buf[i]);
+ reg_wr(&icap->icap_regs->ir_wf, value);
+ }
+
+ reg_wr(&icap->icap_regs->ir_cr, 0x1);
+
+ for (i = 0; i < 20; i++) {
+ value = reg_rd(&icap->icap_regs->ir_cr);
+ if ((value & 0x1) == 0)
+ return 0;
+ ndelay(50);
+ }
+
+ ICAP_ERR(icap, "writing %d dwords timeout", size);
+ return -EIO;
+}
+
+static int bitstream_helper(struct icap *icap, const u32 *word_buffer,
+ u32 word_count)
+{
+ u32 remain_word;
+ u32 word_written = 0;
+ int wr_fifo_vacancy = 0;
+ int err = 0;
+
+ WARN_ON(!mutex_is_locked(&icap->icap_lock));
+ for (remain_word = word_count; remain_word > 0;
+ remain_word -= word_written, word_buffer += word_written) {
+ wr_fifo_vacancy = reg_rd(&icap->icap_regs->ir_wfv);
+ if (wr_fifo_vacancy <= 0) {
+ ICAP_ERR(icap, "no vacancy: %d", wr_fifo_vacancy);
+ err = -EIO;
+ break;
+ }
+ word_written = (wr_fifo_vacancy < remain_word) ?
+ wr_fifo_vacancy : remain_word;
+ if (icap_write(icap, word_buffer, word_written) != 0) {
+ ICAP_ERR(icap, "write failed remain %d, written %d",
+ remain_word, word_written);
+ err = -EIO;
+ break;
+ }
+ }
+
+ return err;
+}
+
+static int icap_download(struct icap *icap, const char *buffer,
+ unsigned long length)
+{
+ u32 num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
+ u32 byte_read;
+ int err = 0;
+
+ mutex_lock(&icap->icap_lock);
+ for (byte_read = 0; byte_read < length; byte_read += num_chars_read) {
+ num_chars_read = length - byte_read;
+ if (num_chars_read > DMA_HWICAP_BITFILE_BUFFER_SIZE)
+ num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
+
+ err = bitstream_helper(icap, (u32 *)buffer, num_chars_read / sizeof(u32));
+ if (err)
+ goto failed;
+ buffer += num_chars_read;
+ }
+
+ err = wait_for_done(icap);
+
+failed:
+ mutex_unlock(&icap->icap_lock);
+
+ return err;
+}
+
+/*
+ * Run the following sequence of canned commands to obtain IDCODE of the FPGA
+ */
+static void icap_probe_chip(struct icap *icap)
+{
+ u32 w;
+
+ w = reg_rd(&icap->icap_regs->ir_sr);
+ w = reg_rd(&icap->icap_regs->ir_sr);
+ reg_wr(&icap->icap_regs->ir_gier, 0x0);
+ w = reg_rd(&icap->icap_regs->ir_wfv);
+ reg_wr(&icap->icap_regs->ir_wf, 0xffffffff);
+ reg_wr(&icap->icap_regs->ir_wf, 0xaa995566);
+ reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
+ reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
+ reg_wr(&icap->icap_regs->ir_wf, 0x28018001);
+ reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
+ reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
+ w = reg_rd(&icap->icap_regs->ir_cr);
+ reg_wr(&icap->icap_regs->ir_cr, 0x1);
+ w = reg_rd(&icap->icap_regs->ir_cr);
+ w = reg_rd(&icap->icap_regs->ir_cr);
+ w = reg_rd(&icap->icap_regs->ir_sr);
+ w = reg_rd(&icap->icap_regs->ir_cr);
+ w = reg_rd(&icap->icap_regs->ir_sr);
+ reg_wr(&icap->icap_regs->ir_sz, 0x1);
+ w = reg_rd(&icap->icap_regs->ir_cr);
+ reg_wr(&icap->icap_regs->ir_cr, 0x2);
+ w = reg_rd(&icap->icap_regs->ir_rfo);
+ icap->idcode = reg_rd(&icap->icap_regs->ir_rf);
+ w = reg_rd(&icap->icap_regs->ir_cr);
+ (void)w;
+}
+
+static int
+xrt_icap_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ struct xrt_icap_ioctl_wr *wr_arg = arg;
+ struct icap *icap;
+ int ret = 0;
+
+ icap = platform_get_drvdata(pdev);
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ /* Does not handle any event. */
+ break;
+ case XRT_ICAP_WRITE:
+ ret = icap_download(icap, wr_arg->xiiw_bit_data,
+ wr_arg->xiiw_data_len);
+ break;
+ case XRT_ICAP_IDCODE:
+ *(u64 *)arg = icap->idcode;
+ break;
+ default:
+ ICAP_ERR(icap, "unknown command %d", cmd);
+ return -EINVAL;
+ }
+
+ return ret;
+}
+
+static int xrt_icap_remove(struct platform_device *pdev)
+{
+ struct icap *icap;
+
+ icap = platform_get_drvdata(pdev);
+
+ platform_set_drvdata(pdev, NULL);
+ devm_kfree(&pdev->dev, icap);
+
+ return 0;
+}
+
+static int xrt_icap_probe(struct platform_device *pdev)
+{
+ struct icap *icap;
+ int ret = 0;
+ struct resource *res;
+
+ icap = devm_kzalloc(&pdev->dev, sizeof(*icap), GFP_KERNEL);
+ if (!icap)
+ return -ENOMEM;
+
+ icap->pdev = pdev;
+ platform_set_drvdata(pdev, icap);
+ mutex_init(&icap->icap_lock);
+
+ xrt_info(pdev, "probing");
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (res) {
+ icap->icap_regs = ioremap(res->start, res->end - res->start + 1);
+ if (!icap->icap_regs) {
+ xrt_err(pdev, "map base failed %pR", res);
+ ret = -EIO;
+ goto failed;
+ }
+ }
+
+ icap_probe_chip(icap);
+failed:
+ return ret;
+}
+
+static struct xrt_subdev_endpoints xrt_icap_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ { .ep_name = XRT_MD_NODE_FPGA_CONFIG },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xrt_icap_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_icap_leaf_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_icap_table[] = {
+ { XRT_ICAP, (kernel_ulong_t)&xrt_icap_data },
+ { },
+};
+
+static struct platform_driver xrt_icap_driver = {
+ .driver = {
+ .name = XRT_ICAP,
+ },
+ .probe = xrt_icap_probe,
+ .remove = xrt_icap_remove,
+ .id_table = xrt_icap_table,
+};
+
+void icap_leaf_init_fini(bool init)
+{
+ if (init)
+ xleaf_register_driver(XRT_SUBDEV_ICAP, &xrt_icap_driver, xrt_icap_endpoints);
+ else
+ xleaf_unregister_driver(XRT_SUBDEV_ICAP);
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP " Lizhi Hou
@ 2021-02-21 20:24 ` Moritz Fischer
2021-03-02 18:26 ` Lizhi Hou
2021-03-03 15:12 ` Tom Rix
1 sibling, 1 reply; 87+ messages in thread
From: Moritz Fischer @ 2021-02-21 20:24 UTC (permalink / raw)
To: Lizhi Hou
Cc: linux-kernel, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, mdf, robh, Max Zhen
On Wed, Feb 17, 2021 at 10:40:13PM -0800, Lizhi Hou wrote:
> Add ICAP driver. ICAP is a hardware function discovered by walking
> firmware metadata. A platform device node will be created for it.
> FPGA bitstream is written to hardware through ICAP.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/icap.h | 29 +++
> drivers/fpga/xrt/lib/xleaf/icap.c | 317 ++++++++++++++++++++++++++
> 2 files changed, 346 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/icap.h b/drivers/fpga/xrt/include/xleaf/icap.h
> new file mode 100644
> index 000000000000..a14fc0ffa78f
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/icap.h
> @@ -0,0 +1,29 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT ICAP Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_ICAP_H_
> +#define _XRT_ICAP_H_
> +
> +#include "xleaf.h"
> +
> +/*
> + * ICAP driver IOCTL calls.
> + */
> +enum xrt_icap_ioctl_cmd {
> + XRT_ICAP_WRITE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> + XRT_ICAP_IDCODE,
> +};
> +
> +struct xrt_icap_ioctl_wr {
> + void *xiiw_bit_data;
> + u32 xiiw_data_len;
> +};
> +
> +#endif /* _XRT_ICAP_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/icap.c b/drivers/fpga/xrt/lib/xleaf/icap.c
> new file mode 100644
> index 000000000000..0500a97bdef9
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/icap.c
> @@ -0,0 +1,317 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA ICAP Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + * Sonal Santan <sonals@xilinx.com>
> + * Max Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/icap.h"
> +#include "xclbin-helper.h"
> +
> +#define XRT_ICAP "xrt_icap"
> +
> +#define ICAP_ERR(icap, fmt, arg...) \
> + xrt_err((icap)->pdev, fmt "\n", ##arg)
> +#define ICAP_WARN(icap, fmt, arg...) \
> + xrt_warn((icap)->pdev, fmt "\n", ##arg)
> +#define ICAP_INFO(icap, fmt, arg...) \
> + xrt_info((icap)->pdev, fmt "\n", ##arg)
> +#define ICAP_DBG(icap, fmt, arg...) \
> + xrt_dbg((icap)->pdev, fmt "\n", ##arg)
Do we really need two layers of indirection here? What's wrong with
dev_{info,dbg,...} ?
> +
> +/*
> + * AXI-HWICAP IP register layout
> + */
> +struct icap_reg {
> + u32 ir_rsvd1[7];
> + u32 ir_gier;
> + u32 ir_isr;
> + u32 ir_rsvd2;
> + u32 ir_ier;
> + u32 ir_rsvd3[53];
> + u32 ir_wf;
> + u32 ir_rf;
> + u32 ir_sz;
> + u32 ir_cr;
> + u32 ir_sr;
> + u32 ir_wfv;
> + u32 ir_rfo;
> + u32 ir_asr;
> +} __packed;
Can we make those #define and just use writel/readl() ? If you want more
abstraction, use regmap....
> +
> +struct icap {
> + struct platform_device *pdev;
> + struct icap_reg *icap_regs;
> + struct mutex icap_lock; /* icap dev lock */
> +
> + unsigned int idcode;
> +};
> +
> +static inline u32 reg_rd(void __iomem *reg)
> +{
> + if (!reg)
> + return -1;
> +
> + return ioread32(reg);
> +}
> +
> +static inline void reg_wr(void __iomem *reg, u32 val)
> +{
> + if (!reg)
> + return;
> +
> + iowrite32(val, reg);
> +}
> +
> +static int wait_for_done(struct icap *icap)
> +{
> + u32 w;
> + int i = 0;
> +
> + WARN_ON(!mutex_is_locked(&icap->icap_lock));
> + for (i = 0; i < 10; i++) {
> + udelay(5);
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + ICAP_INFO(icap, "XHWICAP_SR: %x", w);
> + if (w & 0x5)
> + return 0;
> + }
> +
> + ICAP_ERR(icap, "bitstream download timeout");
> + return -ETIMEDOUT;
> +}
> +
> +static int icap_write(struct icap *icap, const u32 *word_buf, int size)
> +{
> + int i;
> + u32 value = 0;
> +
> + for (i = 0; i < size; i++) {
> + value = be32_to_cpu(word_buf[i]);
> + reg_wr(&icap->icap_regs->ir_wf, value);
> + }
> +
> + reg_wr(&icap->icap_regs->ir_cr, 0x1);
> +
> + for (i = 0; i < 20; i++) {
> + value = reg_rd(&icap->icap_regs->ir_cr);
> + if ((value & 0x1) == 0)
> + return 0;
> + ndelay(50);
> + }
> +
> + ICAP_ERR(icap, "writing %d dwords timeout", size);
> + return -EIO;
> +}
> +
> +static int bitstream_helper(struct icap *icap, const u32 *word_buffer,
> + u32 word_count)
> +{
> + u32 remain_word;
> + u32 word_written = 0;
> + int wr_fifo_vacancy = 0;
> + int err = 0;
> +
> + WARN_ON(!mutex_is_locked(&icap->icap_lock));
> + for (remain_word = word_count; remain_word > 0;
> + remain_word -= word_written, word_buffer += word_written) {
> + wr_fifo_vacancy = reg_rd(&icap->icap_regs->ir_wfv);
> + if (wr_fifo_vacancy <= 0) {
> + ICAP_ERR(icap, "no vacancy: %d", wr_fifo_vacancy);
> + err = -EIO;
> + break;
> + }
> + word_written = (wr_fifo_vacancy < remain_word) ?
> + wr_fifo_vacancy : remain_word;
> + if (icap_write(icap, word_buffer, word_written) != 0) {
> + ICAP_ERR(icap, "write failed remain %d, written %d",
> + remain_word, word_written);
> + err = -EIO;
> + break;
> + }
> + }
> +
> + return err;
> +}
> +
> +static int icap_download(struct icap *icap, const char *buffer,
> + unsigned long length)
> +{
> + u32 num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
> + u32 byte_read;
> + int err = 0;
> +
> + mutex_lock(&icap->icap_lock);
> + for (byte_read = 0; byte_read < length; byte_read += num_chars_read) {
> + num_chars_read = length - byte_read;
> + if (num_chars_read > DMA_HWICAP_BITFILE_BUFFER_SIZE)
> + num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
> +
> + err = bitstream_helper(icap, (u32 *)buffer, num_chars_read / sizeof(u32));
> + if (err)
> + goto failed;
> + buffer += num_chars_read;
> + }
> +
> + err = wait_for_done(icap);
> +
> +failed:
> + mutex_unlock(&icap->icap_lock);
> +
> + return err;
> +}
> +
> +/*
> + * Run the following sequence of canned commands to obtain IDCODE of the FPGA
> + */
> +static void icap_probe_chip(struct icap *icap)
> +{
> + u32 w;
> +
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + reg_wr(&icap->icap_regs->ir_gier, 0x0);
> + w = reg_rd(&icap->icap_regs->ir_wfv);
> + reg_wr(&icap->icap_regs->ir_wf, 0xffffffff);
> + reg_wr(&icap->icap_regs->ir_wf, 0xaa995566);
> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
> + reg_wr(&icap->icap_regs->ir_wf, 0x28018001);
> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + reg_wr(&icap->icap_regs->ir_cr, 0x1);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + reg_wr(&icap->icap_regs->ir_sz, 0x1);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + reg_wr(&icap->icap_regs->ir_cr, 0x2);
> + w = reg_rd(&icap->icap_regs->ir_rfo);
> + icap->idcode = reg_rd(&icap->icap_regs->ir_rf);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + (void)w;
?!
> +}
> +
> +static int
> +xrt_icap_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct xrt_icap_ioctl_wr *wr_arg = arg;
> + struct icap *icap;
> + int ret = 0;
> +
> + icap = platform_get_drvdata(pdev);
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + /* Does not handle any event. */
> + break;
> + case XRT_ICAP_WRITE:
> + ret = icap_download(icap, wr_arg->xiiw_bit_data,
> + wr_arg->xiiw_data_len);
> + break;
> + case XRT_ICAP_IDCODE:
> + *(u64 *)arg = icap->idcode;
> + break;
> + default:
> + ICAP_ERR(icap, "unknown command %d", cmd);
> + return -EINVAL;
> + }
> +
> + return ret;
> +}
> +
> +static int xrt_icap_remove(struct platform_device *pdev)
> +{
> + struct icap *icap;
> +
> + icap = platform_get_drvdata(pdev);
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, icap);
> +
> + return 0;
> +}
> +
> +static int xrt_icap_probe(struct platform_device *pdev)
> +{
> + struct icap *icap;
> + int ret = 0;
> + struct resource *res;
> +
> + icap = devm_kzalloc(&pdev->dev, sizeof(*icap), GFP_KERNEL);
> + if (!icap)
> + return -ENOMEM;
> +
> + icap->pdev = pdev;
> + platform_set_drvdata(pdev, icap);
> + mutex_init(&icap->icap_lock);
> +
> + xrt_info(pdev, "probing");
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + if (res) {
> + icap->icap_regs = ioremap(res->start, res->end - res->start + 1);
> + if (!icap->icap_regs) {
> + xrt_err(pdev, "map base failed %pR", res);
> + ret = -EIO;
> + goto failed;
> + }
> + }
> +
> + icap_probe_chip(icap);
> +failed:
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_icap_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = XRT_MD_NODE_FPGA_CONFIG },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_icap_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_icap_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_icap_table[] = {
> + { XRT_ICAP, (kernel_ulong_t)&xrt_icap_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_icap_driver = {
> + .driver = {
> + .name = XRT_ICAP,
> + },
> + .probe = xrt_icap_probe,
> + .remove = xrt_icap_remove,
> + .id_table = xrt_icap_table,
> +};
> +
> +void icap_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_ICAP, &xrt_icap_driver, xrt_icap_endpoints);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_ICAP);
> +}
> --
> 2.18.4
>
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP platform driver
2021-02-21 20:24 ` Moritz Fischer
@ 2021-03-02 18:26 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-02 18:26 UTC (permalink / raw)
To: Moritz Fischer, Lizhi Hou
Cc: linux-kernel, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, robh, Max Zhen
Hi Moritz,
On 02/21/2021 12:24 PM, Moritz Fischer wrote:
> On Wed, Feb 17, 2021 at 10:40:13PM -0800, Lizhi Hou wrote:
>> Add ICAP driver. ICAP is a hardware function discovered by walking
>> firmware metadata. A platform device node will be created for it.
>> FPGA bitstream is written to hardware through ICAP.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/icap.h | 29 +++
>> drivers/fpga/xrt/lib/xleaf/icap.c | 317 ++++++++++++++++++++++++++
>> 2 files changed, 346 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>>
>> diff --git a/drivers/fpga/xrt/include/xleaf/icap.h b/drivers/fpga/xrt/include/xleaf/icap.h
>> new file mode 100644
>> index 000000000000..a14fc0ffa78f
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/icap.h
>> @@ -0,0 +1,29 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT ICAP Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_ICAP_H_
>> +#define _XRT_ICAP_H_
>> +
>> +#include "xleaf.h"
>> +
>> +/*
>> + * ICAP driver IOCTL calls.
>> + */
>> +enum xrt_icap_ioctl_cmd {
>> + XRT_ICAP_WRITE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> + XRT_ICAP_IDCODE,
>> +};
>> +
>> +struct xrt_icap_ioctl_wr {
>> + void *xiiw_bit_data;
>> + u32 xiiw_data_len;
>> +};
>> +
>> +#endif /* _XRT_ICAP_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/icap.c b/drivers/fpga/xrt/lib/xleaf/icap.c
>> new file mode 100644
>> index 000000000000..0500a97bdef9
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/icap.c
>> @@ -0,0 +1,317 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA ICAP Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + * Sonal Santan <sonals@xilinx.com>
>> + * Max Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/io.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/icap.h"
>> +#include "xclbin-helper.h"
>> +
>> +#define XRT_ICAP "xrt_icap"
>> +
>> +#define ICAP_ERR(icap, fmt, arg...) \
>> + xrt_err((icap)->pdev, fmt "\n", ##arg)
>> +#define ICAP_WARN(icap, fmt, arg...) \
>> + xrt_warn((icap)->pdev, fmt "\n", ##arg)
>> +#define ICAP_INFO(icap, fmt, arg...) \
>> + xrt_info((icap)->pdev, fmt "\n", ##arg)
>> +#define ICAP_DBG(icap, fmt, arg...) \
>> + xrt_dbg((icap)->pdev, fmt "\n", ##arg)
> Do we really need two layers of indirection here? What's wrong with
> dev_{info,dbg,...} ?
In case that we would change the massage 'fmt' in the future we can
change it at one place. And it does not expose any interface or
introduce performance issue. Could we just keep these Macros?
>> +
>> +/*
>> + * AXI-HWICAP IP register layout
>> + */
>> +struct icap_reg {
>> + u32 ir_rsvd1[7];
>> + u32 ir_gier;
>> + u32 ir_isr;
>> + u32 ir_rsvd2;
>> + u32 ir_ier;
>> + u32 ir_rsvd3[53];
>> + u32 ir_wf;
>> + u32 ir_rf;
>> + u32 ir_sz;
>> + u32 ir_cr;
>> + u32 ir_sr;
>> + u32 ir_wfv;
>> + u32 ir_rfo;
>> + u32 ir_asr;
>> +} __packed;
> Can we make those #define and just use writel/readl() ? If you want more
> abstraction, use regmap....
Sure, we will use #define as below and use writel/readl to access them.
#define ICAP_REG_ISR(base) ((base) + 0x20)
....
>> +
>> +struct icap {
>> + struct platform_device *pdev;
>> + struct icap_reg *icap_regs;
>> + struct mutex icap_lock; /* icap dev lock */
>> +
>> + unsigned int idcode;
>> +};
>> +
>> +static inline u32 reg_rd(void __iomem *reg)
>> +{
>> + if (!reg)
>> + return -1;
>> +
>> + return ioread32(reg);
>> +}
>> +
>> +static inline void reg_wr(void __iomem *reg, u32 val)
>> +{
>> + if (!reg)
>> + return;
>> +
>> + iowrite32(val, reg);
>> +}
>> +
>> +static int wait_for_done(struct icap *icap)
>> +{
>> + u32 w;
>> + int i = 0;
>> +
>> + WARN_ON(!mutex_is_locked(&icap->icap_lock));
>> + for (i = 0; i < 10; i++) {
>> + udelay(5);
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + ICAP_INFO(icap, "XHWICAP_SR: %x", w);
>> + if (w & 0x5)
>> + return 0;
>> + }
>> +
>> + ICAP_ERR(icap, "bitstream download timeout");
>> + return -ETIMEDOUT;
>> +}
>> +
>> +static int icap_write(struct icap *icap, const u32 *word_buf, int size)
>> +{
>> + int i;
>> + u32 value = 0;
>> +
>> + for (i = 0; i < size; i++) {
>> + value = be32_to_cpu(word_buf[i]);
>> + reg_wr(&icap->icap_regs->ir_wf, value);
>> + }
>> +
>> + reg_wr(&icap->icap_regs->ir_cr, 0x1);
>> +
>> + for (i = 0; i < 20; i++) {
>> + value = reg_rd(&icap->icap_regs->ir_cr);
>> + if ((value & 0x1) == 0)
>> + return 0;
>> + ndelay(50);
>> + }
>> +
>> + ICAP_ERR(icap, "writing %d dwords timeout", size);
>> + return -EIO;
>> +}
>> +
>> +static int bitstream_helper(struct icap *icap, const u32 *word_buffer,
>> + u32 word_count)
>> +{
>> + u32 remain_word;
>> + u32 word_written = 0;
>> + int wr_fifo_vacancy = 0;
>> + int err = 0;
>> +
>> + WARN_ON(!mutex_is_locked(&icap->icap_lock));
>> + for (remain_word = word_count; remain_word > 0;
>> + remain_word -= word_written, word_buffer += word_written) {
>> + wr_fifo_vacancy = reg_rd(&icap->icap_regs->ir_wfv);
>> + if (wr_fifo_vacancy <= 0) {
>> + ICAP_ERR(icap, "no vacancy: %d", wr_fifo_vacancy);
>> + err = -EIO;
>> + break;
>> + }
>> + word_written = (wr_fifo_vacancy < remain_word) ?
>> + wr_fifo_vacancy : remain_word;
>> + if (icap_write(icap, word_buffer, word_written) != 0) {
>> + ICAP_ERR(icap, "write failed remain %d, written %d",
>> + remain_word, word_written);
>> + err = -EIO;
>> + break;
>> + }
>> + }
>> +
>> + return err;
>> +}
>> +
>> +static int icap_download(struct icap *icap, const char *buffer,
>> + unsigned long length)
>> +{
>> + u32 num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
>> + u32 byte_read;
>> + int err = 0;
>> +
>> + mutex_lock(&icap->icap_lock);
>> + for (byte_read = 0; byte_read < length; byte_read += num_chars_read) {
>> + num_chars_read = length - byte_read;
>> + if (num_chars_read > DMA_HWICAP_BITFILE_BUFFER_SIZE)
>> + num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
>> +
>> + err = bitstream_helper(icap, (u32 *)buffer, num_chars_read / sizeof(u32));
>> + if (err)
>> + goto failed;
>> + buffer += num_chars_read;
>> + }
>> +
>> + err = wait_for_done(icap);
>> +
>> +failed:
>> + mutex_unlock(&icap->icap_lock);
>> +
>> + return err;
>> +}
>> +
>> +/*
>> + * Run the following sequence of canned commands to obtain IDCODE of the FPGA
>> + */
>> +static void icap_probe_chip(struct icap *icap)
>> +{
>> + u32 w;
>> +
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + reg_wr(&icap->icap_regs->ir_gier, 0x0);
>> + w = reg_rd(&icap->icap_regs->ir_wfv);
>> + reg_wr(&icap->icap_regs->ir_wf, 0xffffffff);
>> + reg_wr(&icap->icap_regs->ir_wf, 0xaa995566);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x28018001);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + reg_wr(&icap->icap_regs->ir_cr, 0x1);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + reg_wr(&icap->icap_regs->ir_sz, 0x1);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + reg_wr(&icap->icap_regs->ir_cr, 0x2);
>> + w = reg_rd(&icap->icap_regs->ir_rfo);
>> + icap->idcode = reg_rd(&icap->icap_regs->ir_rf);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + (void)w;
> ?!
Right. 'w' does not make sense here. We will remove 'w'.
Thanks,
Lizhi
>> +}
>> +
>> +static int
>> +xrt_icap_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + struct xrt_icap_ioctl_wr *wr_arg = arg;
>> + struct icap *icap;
>> + int ret = 0;
>> +
>> + icap = platform_get_drvdata(pdev);
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + /* Does not handle any event. */
>> + break;
>> + case XRT_ICAP_WRITE:
>> + ret = icap_download(icap, wr_arg->xiiw_bit_data,
>> + wr_arg->xiiw_data_len);
>> + break;
>> + case XRT_ICAP_IDCODE:
>> + *(u64 *)arg = icap->idcode;
>> + break;
>> + default:
>> + ICAP_ERR(icap, "unknown command %d", cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +static int xrt_icap_remove(struct platform_device *pdev)
>> +{
>> + struct icap *icap;
>> +
>> + icap = platform_get_drvdata(pdev);
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, icap);
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_icap_probe(struct platform_device *pdev)
>> +{
>> + struct icap *icap;
>> + int ret = 0;
>> + struct resource *res;
>> +
>> + icap = devm_kzalloc(&pdev->dev, sizeof(*icap), GFP_KERNEL);
>> + if (!icap)
>> + return -ENOMEM;
>> +
>> + icap->pdev = pdev;
>> + platform_set_drvdata(pdev, icap);
>> + mutex_init(&icap->icap_lock);
>> +
>> + xrt_info(pdev, "probing");
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + if (res) {
>> + icap->icap_regs = ioremap(res->start, res->end - res->start + 1);
>> + if (!icap->icap_regs) {
>> + xrt_err(pdev, "map base failed %pR", res);
>> + ret = -EIO;
>> + goto failed;
>> + }
>> + }
>> +
>> + icap_probe_chip(icap);
>> +failed:
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_icap_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = XRT_MD_NODE_FPGA_CONFIG },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_icap_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_icap_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_icap_table[] = {
>> + { XRT_ICAP, (kernel_ulong_t)&xrt_icap_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_icap_driver = {
>> + .driver = {
>> + .name = XRT_ICAP,
>> + },
>> + .probe = xrt_icap_probe,
>> + .remove = xrt_icap_remove,
>> + .id_table = xrt_icap_table,
>> +};
>> +
>> +void icap_leaf_init_fini(bool init)
>> +{
>> + if (init)
>> + xleaf_register_driver(XRT_SUBDEV_ICAP, &xrt_icap_driver, xrt_icap_endpoints);
>> + else
>> + xleaf_unregister_driver(XRT_SUBDEV_ICAP);
>> +}
>> --
>> 2.18.4
>>
> - Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP " Lizhi Hou
2021-02-21 20:24 ` Moritz Fischer
@ 2021-03-03 15:12 ` Tom Rix
2021-03-17 20:56 ` Lizhi Hou
1 sibling, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-03 15:12 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Add ICAP driver. ICAP is a hardware function discovered by walking
What does ICAP stand for ?
> firmware metadata. A platform device node will be created for it.
> FPGA bitstream is written to hardware through ICAP.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/icap.h | 29 +++
> drivers/fpga/xrt/lib/xleaf/icap.c | 317 ++++++++++++++++++++++++++
> 2 files changed, 346 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/icap.h b/drivers/fpga/xrt/include/xleaf/icap.h
> new file mode 100644
> index 000000000000..a14fc0ffa78f
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/icap.h
> @@ -0,0 +1,29 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT ICAP Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_ICAP_H_
> +#define _XRT_ICAP_H_
> +
> +#include "xleaf.h"
> +
> +/*
> + * ICAP driver IOCTL calls.
> + */
> +enum xrt_icap_ioctl_cmd {
> + XRT_ICAP_WRITE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
maybe XRT_ICAP_GET_IDCODE
> + XRT_ICAP_IDCODE,
> +};
> +
> +struct xrt_icap_ioctl_wr {
> + void *xiiw_bit_data;
> + u32 xiiw_data_len;
> +};
> +
> +#endif /* _XRT_ICAP_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/icap.c b/drivers/fpga/xrt/lib/xleaf/icap.c
> new file mode 100644
> index 000000000000..0500a97bdef9
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/icap.c
> @@ -0,0 +1,317 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA ICAP Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + * Sonal Santan <sonals@xilinx.com>
> + * Max Zhen <maxz@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/icap.h"
> +#include "xclbin-helper.h"
> +
> +#define XRT_ICAP "xrt_icap"
> +
> +#define ICAP_ERR(icap, fmt, arg...) \
> + xrt_err((icap)->pdev, fmt "\n", ##arg)
> +#define ICAP_WARN(icap, fmt, arg...) \
> + xrt_warn((icap)->pdev, fmt "\n", ##arg)
> +#define ICAP_INFO(icap, fmt, arg...) \
> + xrt_info((icap)->pdev, fmt "\n", ##arg)
> +#define ICAP_DBG(icap, fmt, arg...) \
> + xrt_dbg((icap)->pdev, fmt "\n", ##arg)
> +
> +/*
> + * AXI-HWICAP IP register layout
> + */
> +struct icap_reg {
> + u32 ir_rsvd1[7];
> + u32 ir_gier;
> + u32 ir_isr;
> + u32 ir_rsvd2;
> + u32 ir_ier;
> + u32 ir_rsvd3[53];
> + u32 ir_wf;
> + u32 ir_rf;
> + u32 ir_sz;
> + u32 ir_cr;
> + u32 ir_sr;
> + u32 ir_wfv;
> + u32 ir_rfo;
> + u32 ir_asr;
> +} __packed;
> +
> +struct icap {
> + struct platform_device *pdev;
> + struct icap_reg *icap_regs;
> + struct mutex icap_lock; /* icap dev lock */
> +
> + unsigned int idcode;
returned as a 64 bit value, but could be stored as 32 bit
> +};
> +
> +static inline u32 reg_rd(void __iomem *reg)
> +{
> + if (!reg)
> + return -1;
> +
> + return ioread32(reg);
Look at converting the io access to using regmap* api
> +}
> +
> +static inline void reg_wr(void __iomem *reg, u32 val)
> +{
> + if (!reg)
> + return;
> +
> + iowrite32(val, reg);
> +}
> +
> +static int wait_for_done(struct icap *icap)
> +{
> + u32 w;
> + int i = 0;
> +
> + WARN_ON(!mutex_is_locked(&icap->icap_lock));
is this needed ? wait_for_done is only called in one place.
> + for (i = 0; i < 10; i++) {
> + udelay(5);
comment on delay.
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + ICAP_INFO(icap, "XHWICAP_SR: %x", w);
> + if (w & 0x5)
0x5 is a magic number, should be #defined
> + return 0;
> + }
> +
> + ICAP_ERR(icap, "bitstream download timeout");
> + return -ETIMEDOUT;
> +}
> +
> +static int icap_write(struct icap *icap, const u32 *word_buf, int size)
> +{
> + int i;
> + u32 value = 0;
> +
> + for (i = 0; i < size; i++) {
> + value = be32_to_cpu(word_buf[i]);
> + reg_wr(&icap->icap_regs->ir_wf, value);
> + }
> +
> + reg_wr(&icap->icap_regs->ir_cr, 0x1);
> +
> + for (i = 0; i < 20; i++) {
> + value = reg_rd(&icap->icap_regs->ir_cr);
> + if ((value & 0x1) == 0)
> + return 0;
> + ndelay(50);
> + }
> +
> + ICAP_ERR(icap, "writing %d dwords timeout", size);
> + return -EIO;
> +}
> +
> +static int bitstream_helper(struct icap *icap, const u32 *word_buffer,
> + u32 word_count)
> +{
> + u32 remain_word;
> + u32 word_written = 0;
> + int wr_fifo_vacancy = 0;
> + int err = 0;
> +
> + WARN_ON(!mutex_is_locked(&icap->icap_lock));
> + for (remain_word = word_count; remain_word > 0;
> + remain_word -= word_written, word_buffer += word_written) {
> + wr_fifo_vacancy = reg_rd(&icap->icap_regs->ir_wfv);
> + if (wr_fifo_vacancy <= 0) {
> + ICAP_ERR(icap, "no vacancy: %d", wr_fifo_vacancy);
> + err = -EIO;
> + break;
> + }
> + word_written = (wr_fifo_vacancy < remain_word) ?
> + wr_fifo_vacancy : remain_word;
> + if (icap_write(icap, word_buffer, word_written) != 0) {
> + ICAP_ERR(icap, "write failed remain %d, written %d",
> + remain_word, word_written);
> + err = -EIO;
> + break;
> + }
> + }
> +
> + return err;
> +}
> +
> +static int icap_download(struct icap *icap, const char *buffer,
> + unsigned long length)
> +{
> + u32 num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
> + u32 byte_read;
> + int err = 0;
> +
> + mutex_lock(&icap->icap_lock);
> + for (byte_read = 0; byte_read < length; byte_read += num_chars_read) {
> + num_chars_read = length - byte_read;
> + if (num_chars_read > DMA_HWICAP_BITFILE_BUFFER_SIZE)
> + num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
> +
> + err = bitstream_helper(icap, (u32 *)buffer, num_chars_read / sizeof(u32));
assumption that num_chars_read % 4 == 0
Add a check, or handle.
> + if (err)
> + goto failed;
> + buffer += num_chars_read;
> + }
> +
> + err = wait_for_done(icap);
timeout is not handled
> +
> +failed:
> + mutex_unlock(&icap->icap_lock);
> +
> + return err;
> +}
> +
> +/*
> + * Run the following sequence of canned commands to obtain IDCODE of the FPGA
> + */
> +static void icap_probe_chip(struct icap *icap)
> +{
> + u32 w;
De magic this.
If this is a documented startup sequence, please add a link to the document.
Else add a comment about what you are doing here.
Where possible, convert the hex values to #defines.
Tom
> +
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + reg_wr(&icap->icap_regs->ir_gier, 0x0);
> + w = reg_rd(&icap->icap_regs->ir_wfv);
> + reg_wr(&icap->icap_regs->ir_wf, 0xffffffff);
> + reg_wr(&icap->icap_regs->ir_wf, 0xaa995566);
> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
> + reg_wr(&icap->icap_regs->ir_wf, 0x28018001);
> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + reg_wr(&icap->icap_regs->ir_cr, 0x1);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + w = reg_rd(&icap->icap_regs->ir_sr);
> + reg_wr(&icap->icap_regs->ir_sz, 0x1);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + reg_wr(&icap->icap_regs->ir_cr, 0x2);
> + w = reg_rd(&icap->icap_regs->ir_rfo);
> + icap->idcode = reg_rd(&icap->icap_regs->ir_rf);
> + w = reg_rd(&icap->icap_regs->ir_cr);
> + (void)w;
> +}
> +
> +static int
> +xrt_icap_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct xrt_icap_ioctl_wr *wr_arg = arg;
> + struct icap *icap;
> + int ret = 0;
> +
> + icap = platform_get_drvdata(pdev);
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + /* Does not handle any event. */
> + break;
> + case XRT_ICAP_WRITE:
> + ret = icap_download(icap, wr_arg->xiiw_bit_data,
> + wr_arg->xiiw_data_len);
> + break;
> + case XRT_ICAP_IDCODE:
> + *(u64 *)arg = icap->idcode;
> + break;
> + default:
> + ICAP_ERR(icap, "unknown command %d", cmd);
> + return -EINVAL;
> + }
> +
> + return ret;
> +}
> +
> +static int xrt_icap_remove(struct platform_device *pdev)
> +{
> + struct icap *icap;
> +
> + icap = platform_get_drvdata(pdev);
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, icap);
> +
> + return 0;
> +}
> +
> +static int xrt_icap_probe(struct platform_device *pdev)
> +{
> + struct icap *icap;
> + int ret = 0;
> + struct resource *res;
> +
> + icap = devm_kzalloc(&pdev->dev, sizeof(*icap), GFP_KERNEL);
> + if (!icap)
> + return -ENOMEM;
> +
> + icap->pdev = pdev;
> + platform_set_drvdata(pdev, icap);
> + mutex_init(&icap->icap_lock);
> +
> + xrt_info(pdev, "probing");
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + if (res) {
> + icap->icap_regs = ioremap(res->start, res->end - res->start + 1);
> + if (!icap->icap_regs) {
> + xrt_err(pdev, "map base failed %pR", res);
> + ret = -EIO;
> + goto failed;
> + }
> + }
> +
> + icap_probe_chip(icap);
> +failed:
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_icap_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = XRT_MD_NODE_FPGA_CONFIG },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_icap_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_icap_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_icap_table[] = {
> + { XRT_ICAP, (kernel_ulong_t)&xrt_icap_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_icap_driver = {
> + .driver = {
> + .name = XRT_ICAP,
> + },
> + .probe = xrt_icap_probe,
> + .remove = xrt_icap_remove,
> + .id_table = xrt_icap_table,
> +};
> +
> +void icap_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_ICAP, &xrt_icap_driver, xrt_icap_endpoints);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_ICAP);
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP platform driver
2021-03-03 15:12 ` Tom Rix
@ 2021-03-17 20:56 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-17 20:56 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 03/03/2021 07:12 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Add ICAP driver. ICAP is a hardware function discovered by walking
> What does ICAP stand for ?
ICAP stands for Hardware Internal Configuration Access Port. I will add
this.
>> firmware metadata. A platform device node will be created for it.
>> FPGA bitstream is written to hardware through ICAP.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/icap.h | 29 +++
>> drivers/fpga/xrt/lib/xleaf/icap.c | 317 ++++++++++++++++++++++++++
>> 2 files changed, 346 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>>
>> diff --git a/drivers/fpga/xrt/include/xleaf/icap.h b/drivers/fpga/xrt/include/xleaf/icap.h
>> new file mode 100644
>> index 000000000000..a14fc0ffa78f
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/icap.h
>> @@ -0,0 +1,29 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT ICAP Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_ICAP_H_
>> +#define _XRT_ICAP_H_
>> +
>> +#include "xleaf.h"
>> +
>> +/*
>> + * ICAP driver IOCTL calls.
>> + */
>> +enum xrt_icap_ioctl_cmd {
>> + XRT_ICAP_WRITE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> maybe XRT_ICAP_GET_IDCODE
Sure.
>> + XRT_ICAP_IDCODE,
>> +};
>> +
>> +struct xrt_icap_ioctl_wr {
>> + void *xiiw_bit_data;
>> + u32 xiiw_data_len;
>> +};
>> +
>> +#endif /* _XRT_ICAP_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/icap.c b/drivers/fpga/xrt/lib/xleaf/icap.c
>> new file mode 100644
>> index 000000000000..0500a97bdef9
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/icap.c
>> @@ -0,0 +1,317 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA ICAP Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + * Sonal Santan <sonals@xilinx.com>
>> + * Max Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/io.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/icap.h"
>> +#include "xclbin-helper.h"
>> +
>> +#define XRT_ICAP "xrt_icap"
>> +
>> +#define ICAP_ERR(icap, fmt, arg...) \
>> + xrt_err((icap)->pdev, fmt "\n", ##arg)
>> +#define ICAP_WARN(icap, fmt, arg...) \
>> + xrt_warn((icap)->pdev, fmt "\n", ##arg)
>> +#define ICAP_INFO(icap, fmt, arg...) \
>> + xrt_info((icap)->pdev, fmt "\n", ##arg)
>> +#define ICAP_DBG(icap, fmt, arg...) \
>> + xrt_dbg((icap)->pdev, fmt "\n", ##arg)
>> +
>> +/*
>> + * AXI-HWICAP IP register layout
>> + */
>> +struct icap_reg {
>> + u32 ir_rsvd1[7];
>> + u32 ir_gier;
>> + u32 ir_isr;
>> + u32 ir_rsvd2;
>> + u32 ir_ier;
>> + u32 ir_rsvd3[53];
>> + u32 ir_wf;
>> + u32 ir_rf;
>> + u32 ir_sz;
>> + u32 ir_cr;
>> + u32 ir_sr;
>> + u32 ir_wfv;
>> + u32 ir_rfo;
>> + u32 ir_asr;
>> +} __packed;
>> +
>> +struct icap {
>> + struct platform_device *pdev;
>> + struct icap_reg *icap_regs;
>> + struct mutex icap_lock; /* icap dev lock */
>> +
>> + unsigned int idcode;
> returned as a 64 bit value, but could be stored as 32 bit
Will change return to u32.
>> +};
>> +
>> +static inline u32 reg_rd(void __iomem *reg)
>> +{
>> + if (!reg)
>> + return -1;
>> +
>> + return ioread32(reg);
> Look at converting the io access to using regmap* api
Will change it.
>> +}
>> +
>> +static inline void reg_wr(void __iomem *reg, u32 val)
>> +{
>> + if (!reg)
>> + return;
>> +
>> + iowrite32(val, reg);
>> +}
>> +
>> +static int wait_for_done(struct icap *icap)
>> +{
>> + u32 w;
>> + int i = 0;
>> +
>> + WARN_ON(!mutex_is_locked(&icap->icap_lock));
> is this needed ? wait_for_done is only called in one place.
Will remove it.
>> + for (i = 0; i < 10; i++) {
>> + udelay(5);
> comment on delay.
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + ICAP_INFO(icap, "XHWICAP_SR: %x", w);
>> + if (w & 0x5)
> 0x5 is a magic number, should be #defined
Sure.
>> + return 0;
>> + }
>> +
>> + ICAP_ERR(icap, "bitstream download timeout");
>> + return -ETIMEDOUT;
>> +}
>> +
>> +static int icap_write(struct icap *icap, const u32 *word_buf, int size)
>> +{
>> + int i;
>> + u32 value = 0;
>> +
>> + for (i = 0; i < size; i++) {
>> + value = be32_to_cpu(word_buf[i]);
>> + reg_wr(&icap->icap_regs->ir_wf, value);
>> + }
>> +
>> + reg_wr(&icap->icap_regs->ir_cr, 0x1);
>> +
>> + for (i = 0; i < 20; i++) {
>> + value = reg_rd(&icap->icap_regs->ir_cr);
>> + if ((value & 0x1) == 0)
>> + return 0;
>> + ndelay(50);
>> + }
>> +
>> + ICAP_ERR(icap, "writing %d dwords timeout", size);
>> + return -EIO;
>> +}
>> +
>> +static int bitstream_helper(struct icap *icap, const u32 *word_buffer,
>> + u32 word_count)
>> +{
>> + u32 remain_word;
>> + u32 word_written = 0;
>> + int wr_fifo_vacancy = 0;
>> + int err = 0;
>> +
>> + WARN_ON(!mutex_is_locked(&icap->icap_lock));
>> + for (remain_word = word_count; remain_word > 0;
>> + remain_word -= word_written, word_buffer += word_written) {
>> + wr_fifo_vacancy = reg_rd(&icap->icap_regs->ir_wfv);
>> + if (wr_fifo_vacancy <= 0) {
>> + ICAP_ERR(icap, "no vacancy: %d", wr_fifo_vacancy);
>> + err = -EIO;
>> + break;
>> + }
>> + word_written = (wr_fifo_vacancy < remain_word) ?
>> + wr_fifo_vacancy : remain_word;
>> + if (icap_write(icap, word_buffer, word_written) != 0) {
>> + ICAP_ERR(icap, "write failed remain %d, written %d",
>> + remain_word, word_written);
>> + err = -EIO;
>> + break;
>> + }
>> + }
>> +
>> + return err;
>> +}
>> +
>> +static int icap_download(struct icap *icap, const char *buffer,
>> + unsigned long length)
>> +{
>> + u32 num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
>> + u32 byte_read;
>> + int err = 0;
>> +
>> + mutex_lock(&icap->icap_lock);
>> + for (byte_read = 0; byte_read < length; byte_read += num_chars_read) {
>> + num_chars_read = length - byte_read;
>> + if (num_chars_read > DMA_HWICAP_BITFILE_BUFFER_SIZE)
>> + num_chars_read = DMA_HWICAP_BITFILE_BUFFER_SIZE;
>> +
>> + err = bitstream_helper(icap, (u32 *)buffer, num_chars_read / sizeof(u32));
> assumption that num_chars_read % 4 == 0
>
> Add a check, or handle.
Sure.
>
>> + if (err)
>> + goto failed;
>> + buffer += num_chars_read;
>> + }
>> +
>> + err = wait_for_done(icap);
> timeout is not handled
>> +
>> +failed:
>> + mutex_unlock(&icap->icap_lock);
>> +
>> + return err;
>> +}
>> +
>> +/*
>> + * Run the following sequence of canned commands to obtain IDCODE of the FPGA
>> + */
>> +static void icap_probe_chip(struct icap *icap)
>> +{
>> + u32 w;
> De magic this.
>
> If this is a documented startup sequence, please add a link to the document.
>
> Else add a comment about what you are doing here.
>
> Where possible, convert the hex values to #defines.
Will add comment for this.
Thanks,
Lizhi
>
> Tom
>
>> +
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + reg_wr(&icap->icap_regs->ir_gier, 0x0);
>> + w = reg_rd(&icap->icap_regs->ir_wfv);
>> + reg_wr(&icap->icap_regs->ir_wf, 0xffffffff);
>> + reg_wr(&icap->icap_regs->ir_wf, 0xaa995566);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x28018001);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
>> + reg_wr(&icap->icap_regs->ir_wf, 0x20000000);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + reg_wr(&icap->icap_regs->ir_cr, 0x1);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + w = reg_rd(&icap->icap_regs->ir_sr);
>> + reg_wr(&icap->icap_regs->ir_sz, 0x1);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + reg_wr(&icap->icap_regs->ir_cr, 0x2);
>> + w = reg_rd(&icap->icap_regs->ir_rfo);
>> + icap->idcode = reg_rd(&icap->icap_regs->ir_rf);
>> + w = reg_rd(&icap->icap_regs->ir_cr);
>> + (void)w;
>> +}
>> +
>> +static int
>> +xrt_icap_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + struct xrt_icap_ioctl_wr *wr_arg = arg;
>> + struct icap *icap;
>> + int ret = 0;
>> +
>> + icap = platform_get_drvdata(pdev);
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + /* Does not handle any event. */
>> + break;
>> + case XRT_ICAP_WRITE:
>> + ret = icap_download(icap, wr_arg->xiiw_bit_data,
>> + wr_arg->xiiw_data_len);
>> + break;
>> + case XRT_ICAP_IDCODE:
>> + *(u64 *)arg = icap->idcode;
>> + break;
>> + default:
>> + ICAP_ERR(icap, "unknown command %d", cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +static int xrt_icap_remove(struct platform_device *pdev)
>> +{
>> + struct icap *icap;
>> +
>> + icap = platform_get_drvdata(pdev);
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, icap);
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_icap_probe(struct platform_device *pdev)
>> +{
>> + struct icap *icap;
>> + int ret = 0;
>> + struct resource *res;
>> +
>> + icap = devm_kzalloc(&pdev->dev, sizeof(*icap), GFP_KERNEL);
>> + if (!icap)
>> + return -ENOMEM;
>> +
>> + icap->pdev = pdev;
>> + platform_set_drvdata(pdev, icap);
>> + mutex_init(&icap->icap_lock);
>> +
>> + xrt_info(pdev, "probing");
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + if (res) {
>> + icap->icap_regs = ioremap(res->start, res->end - res->start + 1);
>> + if (!icap->icap_regs) {
>> + xrt_err(pdev, "map base failed %pR", res);
>> + ret = -EIO;
>> + goto failed;
>> + }
>> + }
>> +
>> + icap_probe_chip(icap);
>> +failed:
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_icap_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = XRT_MD_NODE_FPGA_CONFIG },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_icap_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_icap_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_icap_table[] = {
>> + { XRT_ICAP, (kernel_ulong_t)&xrt_icap_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_icap_driver = {
>> + .driver = {
>> + .name = XRT_ICAP,
>> + },
>> + .probe = xrt_icap_probe,
>> + .remove = xrt_icap_remove,
>> + .id_table = xrt_icap_table,
>> +};
>> +
>> +void icap_leaf_init_fini(bool init)
>> +{
>> + if (init)
>> + xleaf_register_driver(XRT_SUBDEV_ICAP, &xrt_icap_driver, xrt_icap_endpoints);
>> + else
>> + xleaf_unregister_driver(XRT_SUBDEV_ICAP);
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 13/18] fpga: xrt: devctl platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (11 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 12/18] fpga: xrt: ICAP " Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-03-04 13:39 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 14/18] fpga: xrt: clock " Lizhi Hou
` (6 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Add devctl driver. devctl is a type of hardware function which only has
few registers to read or write. They are discovered by walking firmware
metadata. A platform device node will be created for them.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xleaf/devctl.h | 43 +++++
drivers/fpga/xrt/lib/xleaf/devctl.c | 206 ++++++++++++++++++++++++
2 files changed, 249 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
diff --git a/drivers/fpga/xrt/include/xleaf/devctl.h b/drivers/fpga/xrt/include/xleaf/devctl.h
new file mode 100644
index 000000000000..96a40e066f83
--- /dev/null
+++ b/drivers/fpga/xrt/include/xleaf/devctl.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for XRT DEVCTL Leaf Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_DEVCTL_H_
+#define _XRT_DEVCTL_H_
+
+#include "xleaf.h"
+
+/*
+ * DEVCTL driver IOCTL calls.
+ */
+enum xrt_devctl_ioctl_cmd {
+ XRT_DEVCTL_READ = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+ XRT_DEVCTL_WRITE,
+};
+
+enum xrt_devctl_id {
+ XRT_DEVCTL_ROM_UUID,
+ XRT_DEVCTL_DDR_CALIB,
+ XRT_DEVCTL_GOLDEN_VER,
+ XRT_DEVCTL_MAX
+};
+
+struct xrt_devctl_ioctl_rw {
+ u32 xgir_id;
+ void *xgir_buf;
+ u32 xgir_len;
+ u32 xgir_offset;
+};
+
+struct xrt_devctl_ioctl_intf_uuid {
+ u32 xgir_uuid_num;
+ uuid_t *xgir_uuids;
+};
+
+#endif /* _XRT_DEVCTL_H_ */
diff --git a/drivers/fpga/xrt/lib/xleaf/devctl.c b/drivers/fpga/xrt/lib/xleaf/devctl.c
new file mode 100644
index 000000000000..caf8c6569f0f
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xleaf/devctl.c
@@ -0,0 +1,206 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA devctl Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou<Lizhi.Hou@xilinx.com>
+ */
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include "metadata.h"
+#include "xleaf.h"
+#include "xleaf/devctl.h"
+
+#define XRT_DEVCTL "xrt_devctl"
+
+struct xrt_name_id {
+ char *ep_name;
+ int id;
+};
+
+static struct xrt_name_id name_id[XRT_DEVCTL_MAX] = {
+ { XRT_MD_NODE_BLP_ROM, XRT_DEVCTL_ROM_UUID },
+ { XRT_MD_NODE_GOLDEN_VER, XRT_DEVCTL_GOLDEN_VER },
+};
+
+struct xrt_devctl {
+ struct platform_device *pdev;
+ void __iomem *base_addrs[XRT_DEVCTL_MAX];
+ ulong sizes[XRT_DEVCTL_MAX];
+};
+
+static int xrt_devctl_name2id(struct xrt_devctl *devctl, const char *name)
+{
+ int i;
+
+ for (i = 0; i < XRT_DEVCTL_MAX && name_id[i].ep_name; i++) {
+ if (!strncmp(name_id[i].ep_name, name, strlen(name_id[i].ep_name) + 1))
+ return name_id[i].id;
+ }
+
+ return -EINVAL;
+}
+
+static int
+xrt_devctl_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ struct xrt_devctl *devctl;
+ int ret = 0;
+
+ devctl = platform_get_drvdata(pdev);
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ /* Does not handle any event. */
+ break;
+ case XRT_DEVCTL_READ: {
+ struct xrt_devctl_ioctl_rw *rw_arg = arg;
+ u32 *p_src, *p_dst, i;
+
+ if (rw_arg->xgir_len & 0x3) {
+ xrt_err(pdev, "invalid len %d", rw_arg->xgir_len);
+ return -EINVAL;
+ }
+
+ if (rw_arg->xgir_id >= XRT_DEVCTL_MAX) {
+ xrt_err(pdev, "invalid id %d", rw_arg->xgir_id);
+ return -EINVAL;
+ }
+
+ p_src = devctl->base_addrs[rw_arg->xgir_id];
+ if (!p_src) {
+ xrt_err(pdev, "io not found, id %d",
+ rw_arg->xgir_id);
+ return -EINVAL;
+ }
+ if (rw_arg->xgir_offset + rw_arg->xgir_len >
+ devctl->sizes[rw_arg->xgir_id]) {
+ xrt_err(pdev, "invalid argument, off %d, len %d",
+ rw_arg->xgir_offset, rw_arg->xgir_len);
+ return -EINVAL;
+ }
+ p_dst = rw_arg->xgir_buf;
+ for (i = 0; i < rw_arg->xgir_len / sizeof(u32); i++) {
+ u32 val = ioread32(p_src + rw_arg->xgir_offset + i);
+
+ memcpy(p_dst + i, &val, sizeof(u32));
+ }
+ break;
+ }
+ default:
+ xrt_err(pdev, "unsupported cmd %d", cmd);
+ return -EINVAL;
+ }
+
+ return ret;
+}
+
+static int xrt_devctl_remove(struct platform_device *pdev)
+{
+ struct xrt_devctl *devctl;
+ int i;
+
+ devctl = platform_get_drvdata(pdev);
+
+ for (i = 0; i < XRT_DEVCTL_MAX; i++) {
+ if (devctl->base_addrs[i])
+ iounmap(devctl->base_addrs[i]);
+ }
+
+ platform_set_drvdata(pdev, NULL);
+ devm_kfree(&pdev->dev, devctl);
+
+ return 0;
+}
+
+static int xrt_devctl_probe(struct platform_device *pdev)
+{
+ struct xrt_devctl *devctl;
+ int i, id, ret = 0;
+ struct resource *res;
+
+ devctl = devm_kzalloc(&pdev->dev, sizeof(*devctl), GFP_KERNEL);
+ if (!devctl)
+ return -ENOMEM;
+
+ devctl->pdev = pdev;
+ platform_set_drvdata(pdev, devctl);
+
+ xrt_info(pdev, "probing...");
+ for (i = 0, res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ res;
+ res = platform_get_resource(pdev, IORESOURCE_MEM, ++i)) {
+ id = xrt_devctl_name2id(devctl, res->name);
+ if (id < 0) {
+ xrt_err(pdev, "ep %s not found", res->name);
+ continue;
+ }
+ devctl->base_addrs[id] = ioremap(res->start, res->end - res->start + 1);
+ if (!devctl->base_addrs[id]) {
+ xrt_err(pdev, "map base failed %pR", res);
+ ret = -EIO;
+ goto failed;
+ }
+ devctl->sizes[id] = res->end - res->start + 1;
+ }
+
+failed:
+ if (ret)
+ xrt_devctl_remove(pdev);
+
+ return ret;
+}
+
+static struct xrt_subdev_endpoints xrt_devctl_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ /* add name if ep is in same partition */
+ { .ep_name = XRT_MD_NODE_BLP_ROM },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ { .ep_name = XRT_MD_NODE_GOLDEN_VER },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ /* adding ep bundle generates devctl device instance */
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xrt_devctl_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_devctl_leaf_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_devctl_table[] = {
+ { XRT_DEVCTL, (kernel_ulong_t)&xrt_devctl_data },
+ { },
+};
+
+static struct platform_driver xrt_devctl_driver = {
+ .driver = {
+ .name = XRT_DEVCTL,
+ },
+ .probe = xrt_devctl_probe,
+ .remove = xrt_devctl_remove,
+ .id_table = xrt_devctl_table,
+};
+
+void devctl_leaf_init_fini(bool init)
+{
+ if (init)
+ xleaf_register_driver(XRT_SUBDEV_DEVCTL, &xrt_devctl_driver, xrt_devctl_endpoints);
+ else
+ xleaf_unregister_driver(XRT_SUBDEV_DEVCTL);
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 13/18] fpga: xrt: devctl platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 13/18] fpga: xrt: devctl " Lizhi Hou
@ 2021-03-04 13:39 ` Tom Rix
2021-03-16 23:54 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-04 13:39 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Add devctl driver. devctl is a type of hardware function which only has
> few registers to read or write. They are discovered by walking firmware
> metadata. A platform device node will be created for them.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/devctl.h | 43 +++++
> drivers/fpga/xrt/lib/xleaf/devctl.c | 206 ++++++++++++++++++++++++
> 2 files changed, 249 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/devctl.h b/drivers/fpga/xrt/include/xleaf/devctl.h
> new file mode 100644
> index 000000000000..96a40e066f83
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/devctl.h
> @@ -0,0 +1,43 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT DEVCTL Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_DEVCTL_H_
> +#define _XRT_DEVCTL_H_
> +
> +#include "xleaf.h"
> +
> +/*
> + * DEVCTL driver IOCTL calls.
> + */
> +enum xrt_devctl_ioctl_cmd {
> + XRT_DEVCTL_READ = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> + XRT_DEVCTL_WRITE,
> +};
> +
> +enum xrt_devctl_id {
> + XRT_DEVCTL_ROM_UUID,
Assumes 0, should make this explicit and initialize to 0
> + XRT_DEVCTL_DDR_CALIB,
> + XRT_DEVCTL_GOLDEN_VER,
> + XRT_DEVCTL_MAX
> +};
> +
> +struct xrt_devctl_ioctl_rw {
> + u32 xgir_id;
> + void *xgir_buf;
> + u32 xgir_len;
> + u32 xgir_offset;
similar to other patches, the xgir_ prefix is not needed
> +};
> +
> +struct xrt_devctl_ioctl_intf_uuid {
> + u32 xgir_uuid_num;
> + uuid_t *xgir_uuids;
> +};
> +
> +#endif /* _XRT_DEVCTL_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/devctl.c b/drivers/fpga/xrt/lib/xleaf/devctl.c
> new file mode 100644
> index 000000000000..caf8c6569f0f
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/devctl.c
> @@ -0,0 +1,206 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA devctl Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/devctl.h"
> +
> +#define XRT_DEVCTL "xrt_devctl"
> +
> +struct xrt_name_id {
> + char *ep_name;
> + int id;
> +};
> +
> +static struct xrt_name_id name_id[XRT_DEVCTL_MAX] = {
> + { XRT_MD_NODE_BLP_ROM, XRT_DEVCTL_ROM_UUID },
> + { XRT_MD_NODE_GOLDEN_VER, XRT_DEVCTL_GOLDEN_VER },
DDR_CALIB is unused ?
> +};
> +
> +struct xrt_devctl {
> + struct platform_device *pdev;
> + void __iomem *base_addrs[XRT_DEVCTL_MAX];
> + ulong sizes[XRT_DEVCTL_MAX];
> +};
similar to other patches, why not use regmap ?
> +
> +static int xrt_devctl_name2id(struct xrt_devctl *devctl, const char *name)
> +{
> + int i;
> +
> + for (i = 0; i < XRT_DEVCTL_MAX && name_id[i].ep_name; i++) {
> + if (!strncmp(name_id[i].ep_name, name, strlen(name_id[i].ep_name) + 1))
> + return name_id[i].id;
> + }
> +
> + return -EINVAL;
> +}
> +
> +static int
> +xrt_devctl_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct xrt_devctl *devctl;
> + int ret = 0;
> +
> + devctl = platform_get_drvdata(pdev);
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + /* Does not handle any event. */
> + break;
> + case XRT_DEVCTL_READ: {
> + struct xrt_devctl_ioctl_rw *rw_arg = arg;
> + u32 *p_src, *p_dst, i;
> +
> + if (rw_arg->xgir_len & 0x3) {
> + xrt_err(pdev, "invalid len %d", rw_arg->xgir_len);
> + return -EINVAL;
> + }
> +
> + if (rw_arg->xgir_id >= XRT_DEVCTL_MAX) {
> + xrt_err(pdev, "invalid id %d", rw_arg->xgir_id);
> + return -EINVAL;
> + }
needs a < 0 check ?
> +
> + p_src = devctl->base_addrs[rw_arg->xgir_id];
> + if (!p_src) {
> + xrt_err(pdev, "io not found, id %d",
> + rw_arg->xgir_id);
> + return -EINVAL;
> + }
> + if (rw_arg->xgir_offset + rw_arg->xgir_len >
> + devctl->sizes[rw_arg->xgir_id]) {
> + xrt_err(pdev, "invalid argument, off %d, len %d",
> + rw_arg->xgir_offset, rw_arg->xgir_len);
> + return -EINVAL;
> + }
> + p_dst = rw_arg->xgir_buf;
> + for (i = 0; i < rw_arg->xgir_len / sizeof(u32); i++) {
> + u32 val = ioread32(p_src + rw_arg->xgir_offset + i);
> +
> + memcpy(p_dst + i, &val, sizeof(u32));
> + }
> + break;
> + }
The _WRITE msg is not handled Then why have it ?
Tom
> + default:
> + xrt_err(pdev, "unsupported cmd %d", cmd);
> + return -EINVAL;
> + }
> +
> + return ret;
> +}
> +
> +static int xrt_devctl_remove(struct platform_device *pdev)
> +{
> + struct xrt_devctl *devctl;
> + int i;
> +
> + devctl = platform_get_drvdata(pdev);
> +
> + for (i = 0; i < XRT_DEVCTL_MAX; i++) {
> + if (devctl->base_addrs[i])
> + iounmap(devctl->base_addrs[i]);
> + }
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, devctl);
> +
> + return 0;
> +}
> +
> +static int xrt_devctl_probe(struct platform_device *pdev)
> +{
> + struct xrt_devctl *devctl;
> + int i, id, ret = 0;
> + struct resource *res;
> +
> + devctl = devm_kzalloc(&pdev->dev, sizeof(*devctl), GFP_KERNEL);
> + if (!devctl)
> + return -ENOMEM;
> +
> + devctl->pdev = pdev;
> + platform_set_drvdata(pdev, devctl);
> +
> + xrt_info(pdev, "probing...");
> + for (i = 0, res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + res;
> + res = platform_get_resource(pdev, IORESOURCE_MEM, ++i)) {
> + id = xrt_devctl_name2id(devctl, res->name);
> + if (id < 0) {
> + xrt_err(pdev, "ep %s not found", res->name);
> + continue;
> + }
> + devctl->base_addrs[id] = ioremap(res->start, res->end - res->start + 1);
> + if (!devctl->base_addrs[id]) {
> + xrt_err(pdev, "map base failed %pR", res);
> + ret = -EIO;
> + goto failed;
> + }
> + devctl->sizes[id] = res->end - res->start + 1;
> + }
> +
> +failed:
> + if (ret)
> + xrt_devctl_remove(pdev);
> +
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_devctl_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + /* add name if ep is in same partition */
> + { .ep_name = XRT_MD_NODE_BLP_ROM },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = XRT_MD_NODE_GOLDEN_VER },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + /* adding ep bundle generates devctl device instance */
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_devctl_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_devctl_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_devctl_table[] = {
> + { XRT_DEVCTL, (kernel_ulong_t)&xrt_devctl_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_devctl_driver = {
> + .driver = {
> + .name = XRT_DEVCTL,
> + },
> + .probe = xrt_devctl_probe,
> + .remove = xrt_devctl_remove,
> + .id_table = xrt_devctl_table,
> +};
> +
> +void devctl_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_DEVCTL, &xrt_devctl_driver, xrt_devctl_endpoints);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_DEVCTL);
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 13/18] fpga: xrt: devctl platform driver
2021-03-04 13:39 ` Tom Rix
@ 2021-03-16 23:54 ` Lizhi Hou
2021-03-17 21:16 ` Tom Rix
0 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-03-16 23:54 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
On 03/04/2021 05:39 AM, Tom Rix wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Add devctl driver. devctl is a type of hardware function which only has
>> few registers to read or write. They are discovered by walking firmware
>> metadata. A platform device node will be created for them.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/devctl.h | 43 +++++
>> drivers/fpga/xrt/lib/xleaf/devctl.c | 206 ++++++++++++++++++++++++
>> 2 files changed, 249 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>>
>> diff --git a/drivers/fpga/xrt/include/xleaf/devctl.h b/drivers/fpga/xrt/include/xleaf/devctl.h
>> new file mode 100644
>> index 000000000000..96a40e066f83
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/devctl.h
>> @@ -0,0 +1,43 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT DEVCTL Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_DEVCTL_H_
>> +#define _XRT_DEVCTL_H_
>> +
>> +#include "xleaf.h"
>> +
>> +/*
>> + * DEVCTL driver IOCTL calls.
>> + */
>> +enum xrt_devctl_ioctl_cmd {
>> + XRT_DEVCTL_READ = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> + XRT_DEVCTL_WRITE,
>> +};
>> +
>> +enum xrt_devctl_id {
>> + XRT_DEVCTL_ROM_UUID,
> Assumes 0, should make this explicit and initialize to 0
Sure.
>> + XRT_DEVCTL_DDR_CALIB,
>> + XRT_DEVCTL_GOLDEN_VER,
>> + XRT_DEVCTL_MAX
>> +};
>> +
>> +struct xrt_devctl_ioctl_rw {
>> + u32 xgir_id;
>> + void *xgir_buf;
>> + u32 xgir_len;
>> + u32 xgir_offset;
> similar to other patches, the xgir_ prefix is not needed
>> +};
>> +
>> +struct xrt_devctl_ioctl_intf_uuid {
>> + u32 xgir_uuid_num;
>> + uuid_t *xgir_uuids;
>> +};
>> +
>> +#endif /* _XRT_DEVCTL_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/devctl.c b/drivers/fpga/xrt/lib/xleaf/devctl.c
>> new file mode 100644
>> index 000000000000..caf8c6569f0f
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/devctl.c
>> @@ -0,0 +1,206 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA devctl Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/io.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/devctl.h"
>> +
>> +#define XRT_DEVCTL "xrt_devctl"
>> +
>> +struct xrt_name_id {
>> + char *ep_name;
>> + int id;
>> +};
>> +
>> +static struct xrt_name_id name_id[XRT_DEVCTL_MAX] = {
>> + { XRT_MD_NODE_BLP_ROM, XRT_DEVCTL_ROM_UUID },
>> + { XRT_MD_NODE_GOLDEN_VER, XRT_DEVCTL_GOLDEN_VER },
> DDR_CALIB is unused ?
Not sure if I understand the question correctly. ddr_calib will have
more things need to handle other than just read/write.
>> +};
>> +
>> +struct xrt_devctl {
>> + struct platform_device *pdev;
>> + void __iomem *base_addrs[XRT_DEVCTL_MAX];
>> + ulong sizes[XRT_DEVCTL_MAX];
>> +};
> similar to other patches, why not use regmap ?
Will change to regmap.
>> +
>> +static int xrt_devctl_name2id(struct xrt_devctl *devctl, const char *name)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < XRT_DEVCTL_MAX && name_id[i].ep_name; i++) {
>> + if (!strncmp(name_id[i].ep_name, name, strlen(name_id[i].ep_name) + 1))
>> + return name_id[i].id;
>> + }
>> +
>> + return -EINVAL;
>> +}
>> +
>> +static int
>> +xrt_devctl_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + struct xrt_devctl *devctl;
>> + int ret = 0;
>> +
>> + devctl = platform_get_drvdata(pdev);
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + /* Does not handle any event. */
>> + break;
>> + case XRT_DEVCTL_READ: {
>> + struct xrt_devctl_ioctl_rw *rw_arg = arg;
>> + u32 *p_src, *p_dst, i;
>> +
>> + if (rw_arg->xgir_len & 0x3) {
>> + xrt_err(pdev, "invalid len %d", rw_arg->xgir_len);
>> + return -EINVAL;
>> + }
>> +
>> + if (rw_arg->xgir_id >= XRT_DEVCTL_MAX) {
>> + xrt_err(pdev, "invalid id %d", rw_arg->xgir_id);
>> + return -EINVAL;
>> + }
> needs a < 0 check ?
change xgir_id to u32.
>> +
>> + p_src = devctl->base_addrs[rw_arg->xgir_id];
>> + if (!p_src) {
>> + xrt_err(pdev, "io not found, id %d",
>> + rw_arg->xgir_id);
>> + return -EINVAL;
>> + }
>> + if (rw_arg->xgir_offset + rw_arg->xgir_len >
>> + devctl->sizes[rw_arg->xgir_id]) {
>> + xrt_err(pdev, "invalid argument, off %d, len %d",
>> + rw_arg->xgir_offset, rw_arg->xgir_len);
>> + return -EINVAL;
>> + }
>> + p_dst = rw_arg->xgir_buf;
>> + for (i = 0; i < rw_arg->xgir_len / sizeof(u32); i++) {
>> + u32 val = ioread32(p_src + rw_arg->xgir_offset + i);
>> +
>> + memcpy(p_dst + i, &val, sizeof(u32));
>> + }
>> + break;
>> + }
> The _WRITE msg is not handled Then why have it ?
Will remove write msg from this patch set and add it back in future patches.
Thanks,
Lizhi
>
> Tom
>
>> + default:
>> + xrt_err(pdev, "unsupported cmd %d", cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +static int xrt_devctl_remove(struct platform_device *pdev)
>> +{
>> + struct xrt_devctl *devctl;
>> + int i;
>> +
>> + devctl = platform_get_drvdata(pdev);
>> +
>> + for (i = 0; i < XRT_DEVCTL_MAX; i++) {
>> + if (devctl->base_addrs[i])
>> + iounmap(devctl->base_addrs[i]);
>> + }
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, devctl);
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_devctl_probe(struct platform_device *pdev)
>> +{
>> + struct xrt_devctl *devctl;
>> + int i, id, ret = 0;
>> + struct resource *res;
>> +
>> + devctl = devm_kzalloc(&pdev->dev, sizeof(*devctl), GFP_KERNEL);
>> + if (!devctl)
>> + return -ENOMEM;
>> +
>> + devctl->pdev = pdev;
>> + platform_set_drvdata(pdev, devctl);
>> +
>> + xrt_info(pdev, "probing...");
>> + for (i = 0, res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + res;
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, ++i)) {
>> + id = xrt_devctl_name2id(devctl, res->name);
>> + if (id < 0) {
>> + xrt_err(pdev, "ep %s not found", res->name);
>> + continue;
>> + }
>> + devctl->base_addrs[id] = ioremap(res->start, res->end - res->start + 1);
>> + if (!devctl->base_addrs[id]) {
>> + xrt_err(pdev, "map base failed %pR", res);
>> + ret = -EIO;
>> + goto failed;
>> + }
>> + devctl->sizes[id] = res->end - res->start + 1;
>> + }
>> +
>> +failed:
>> + if (ret)
>> + xrt_devctl_remove(pdev);
>> +
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_devctl_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + /* add name if ep is in same partition */
>> + { .ep_name = XRT_MD_NODE_BLP_ROM },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = XRT_MD_NODE_GOLDEN_VER },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + /* adding ep bundle generates devctl device instance */
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_devctl_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_devctl_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_devctl_table[] = {
>> + { XRT_DEVCTL, (kernel_ulong_t)&xrt_devctl_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_devctl_driver = {
>> + .driver = {
>> + .name = XRT_DEVCTL,
>> + },
>> + .probe = xrt_devctl_probe,
>> + .remove = xrt_devctl_remove,
>> + .id_table = xrt_devctl_table,
>> +};
>> +
>> +void devctl_leaf_init_fini(bool init)
>> +{
>> + if (init)
>> + xleaf_register_driver(XRT_SUBDEV_DEVCTL, &xrt_devctl_driver, xrt_devctl_endpoints);
>> + else
>> + xleaf_unregister_driver(XRT_SUBDEV_DEVCTL);
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 13/18] fpga: xrt: devctl platform driver
2021-03-16 23:54 ` Lizhi Hou
@ 2021-03-17 21:16 ` Tom Rix
0 siblings, 0 replies; 87+ messages in thread
From: Tom Rix @ 2021-03-17 21:16 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
On 3/16/21 4:54 PM, Lizhi Hou wrote:
>
>
> On 03/04/2021 05:39 AM, Tom Rix wrote:
>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>>
>>
>> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>>> Add devctl driver. devctl is a type of hardware function which only has
>>> few registers to read or write. They are discovered by walking firmware
>>> metadata. A platform device node will be created for them.
>>>
>>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>>> ---
>>> drivers/fpga/xrt/include/xleaf/devctl.h | 43 +++++
>>> drivers/fpga/xrt/lib/xleaf/devctl.c | 206 ++++++++++++++++++++++++
>>> 2 files changed, 249 insertions(+)
>>> create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>>> create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>>>
>>> diff --git a/drivers/fpga/xrt/include/xleaf/devctl.h b/drivers/fpga/xrt/include/xleaf/devctl.h
>>> new file mode 100644
>>> index 000000000000..96a40e066f83
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/include/xleaf/devctl.h
>>> @@ -0,0 +1,43 @@
>>> +/* SPDX-License-Identifier: GPL-2.0 */
>>> +/*
>>> + * Header file for XRT DEVCTL Leaf Driver
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors:
>>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>>> + */
>>> +
>>> +#ifndef _XRT_DEVCTL_H_
>>> +#define _XRT_DEVCTL_H_
>>> +
>>> +#include "xleaf.h"
>>> +
>>> +/*
>>> + * DEVCTL driver IOCTL calls.
>>> + */
>>> +enum xrt_devctl_ioctl_cmd {
>>> + XRT_DEVCTL_READ = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>>> + XRT_DEVCTL_WRITE,
>>> +};
>>> +
>>> +enum xrt_devctl_id {
>>> + XRT_DEVCTL_ROM_UUID,
>> Assumes 0, should make this explicit and initialize to 0
> Sure.
>>> + XRT_DEVCTL_DDR_CALIB,
>>> + XRT_DEVCTL_GOLDEN_VER,
>>> + XRT_DEVCTL_MAX
>>> +};
>>> +
>>> +struct xrt_devctl_ioctl_rw {
>>> + u32 xgir_id;
>>> + void *xgir_buf;
>>> + u32 xgir_len;
>>> + u32 xgir_offset;
>> similar to other patches, the xgir_ prefix is not needed
>>> +};
>>> +
>>> +struct xrt_devctl_ioctl_intf_uuid {
>>> + u32 xgir_uuid_num;
>>> + uuid_t *xgir_uuids;
>>> +};
>>> +
>>> +#endif /* _XRT_DEVCTL_H_ */
>>> diff --git a/drivers/fpga/xrt/lib/xleaf/devctl.c b/drivers/fpga/xrt/lib/xleaf/devctl.c
>>> new file mode 100644
>>> index 000000000000..caf8c6569f0f
>>> --- /dev/null
>>> +++ b/drivers/fpga/xrt/lib/xleaf/devctl.c
>>> @@ -0,0 +1,206 @@
>>> +// SPDX-License-Identifier: GPL-2.0
>>> +/*
>>> + * Xilinx Alveo FPGA devctl Driver
>>> + *
>>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>>> + *
>>> + * Authors:
>>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>>> + */
>>> +
>>> +#include <linux/mod_devicetable.h>
>>> +#include <linux/platform_device.h>
>>> +#include <linux/delay.h>
>>> +#include <linux/device.h>
>>> +#include <linux/io.h>
>>> +#include "metadata.h"
>>> +#include "xleaf.h"
>>> +#include "xleaf/devctl.h"
>>> +
>>> +#define XRT_DEVCTL "xrt_devctl"
>>> +
>>> +struct xrt_name_id {
>>> + char *ep_name;
>>> + int id;
>>> +};
>>> +
>>> +static struct xrt_name_id name_id[XRT_DEVCTL_MAX] = {
>>> + { XRT_MD_NODE_BLP_ROM, XRT_DEVCTL_ROM_UUID },
>>> + { XRT_MD_NODE_GOLDEN_VER, XRT_DEVCTL_GOLDEN_VER },
>> DDR_CALIB is unused ?
> Not sure if I understand the question correctly. ddr_calib will have more things need to handle other than just read/write.
I do not understand either, ignore this comment.
If it is important, I will bring it up in a later revision's review and do a better job of explaining the issue.
Tom
>>> +};
>>> +
>>> +struct xrt_devctl {
>>> + struct platform_device *pdev;
>>> + void __iomem *base_addrs[XRT_DEVCTL_MAX];
>>> + ulong sizes[XRT_DEVCTL_MAX];
>>> +};
>> similar to other patches, why not use regmap ?
> Will change to regmap.
>>> +
>>> +static int xrt_devctl_name2id(struct xrt_devctl *devctl, const char *name)
>>> +{
>>> + int i;
>>> +
>>> + for (i = 0; i < XRT_DEVCTL_MAX && name_id[i].ep_name; i++) {
>>> + if (!strncmp(name_id[i].ep_name, name, strlen(name_id[i].ep_name) + 1))
>>> + return name_id[i].id;
>>> + }
>>> +
>>> + return -EINVAL;
>>> +}
>>> +
>>> +static int
>>> +xrt_devctl_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>>> +{
>>> + struct xrt_devctl *devctl;
>>> + int ret = 0;
>>> +
>>> + devctl = platform_get_drvdata(pdev);
>>> +
>>> + switch (cmd) {
>>> + case XRT_XLEAF_EVENT:
>>> + /* Does not handle any event. */
>>> + break;
>>> + case XRT_DEVCTL_READ: {
>>> + struct xrt_devctl_ioctl_rw *rw_arg = arg;
>>> + u32 *p_src, *p_dst, i;
>>> +
>>> + if (rw_arg->xgir_len & 0x3) {
>>> + xrt_err(pdev, "invalid len %d", rw_arg->xgir_len);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + if (rw_arg->xgir_id >= XRT_DEVCTL_MAX) {
>>> + xrt_err(pdev, "invalid id %d", rw_arg->xgir_id);
>>> + return -EINVAL;
>>> + }
>> needs a < 0 check ?
> change xgir_id to u32.
>>> +
>>> + p_src = devctl->base_addrs[rw_arg->xgir_id];
>>> + if (!p_src) {
>>> + xrt_err(pdev, "io not found, id %d",
>>> + rw_arg->xgir_id);
>>> + return -EINVAL;
>>> + }
>>> + if (rw_arg->xgir_offset + rw_arg->xgir_len >
>>> + devctl->sizes[rw_arg->xgir_id]) {
>>> + xrt_err(pdev, "invalid argument, off %d, len %d",
>>> + rw_arg->xgir_offset, rw_arg->xgir_len);
>>> + return -EINVAL;
>>> + }
>>> + p_dst = rw_arg->xgir_buf;
>>> + for (i = 0; i < rw_arg->xgir_len / sizeof(u32); i++) {
>>> + u32 val = ioread32(p_src + rw_arg->xgir_offset + i);
>>> +
>>> + memcpy(p_dst + i, &val, sizeof(u32));
>>> + }
>>> + break;
>>> + }
>> The _WRITE msg is not handled Then why have it ?
> Will remove write msg from this patch set and add it back in future patches.
>
> Thanks,
> Lizhi
>>
>> Tom
>>
>>> + default:
>>> + xrt_err(pdev, "unsupported cmd %d", cmd);
>>> + return -EINVAL;
>>> + }
>>> +
>>> + return ret;
>>> +}
>>> +
>>> +static int xrt_devctl_remove(struct platform_device *pdev)
>>> +{
>>> + struct xrt_devctl *devctl;
>>> + int i;
>>> +
>>> + devctl = platform_get_drvdata(pdev);
>>> +
>>> + for (i = 0; i < XRT_DEVCTL_MAX; i++) {
>>> + if (devctl->base_addrs[i])
>>> + iounmap(devctl->base_addrs[i]);
>>> + }
>>> +
>>> + platform_set_drvdata(pdev, NULL);
>>> + devm_kfree(&pdev->dev, devctl);
>>> +
>>> + return 0;
>>> +}
>>> +
>>> +static int xrt_devctl_probe(struct platform_device *pdev)
>>> +{
>>> + struct xrt_devctl *devctl;
>>> + int i, id, ret = 0;
>>> + struct resource *res;
>>> +
>>> + devctl = devm_kzalloc(&pdev->dev, sizeof(*devctl), GFP_KERNEL);
>>> + if (!devctl)
>>> + return -ENOMEM;
>>> +
>>> + devctl->pdev = pdev;
>>> + platform_set_drvdata(pdev, devctl);
>>> +
>>> + xrt_info(pdev, "probing...");
>>> + for (i = 0, res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>>> + res;
>>> + res = platform_get_resource(pdev, IORESOURCE_MEM, ++i)) {
>>> + id = xrt_devctl_name2id(devctl, res->name);
>>> + if (id < 0) {
>>> + xrt_err(pdev, "ep %s not found", res->name);
>>> + continue;
>>> + }
>>> + devctl->base_addrs[id] = ioremap(res->start, res->end - res->start + 1);
>>> + if (!devctl->base_addrs[id]) {
>>> + xrt_err(pdev, "map base failed %pR", res);
>>> + ret = -EIO;
>>> + goto failed;
>>> + }
>>> + devctl->sizes[id] = res->end - res->start + 1;
>>> + }
>>> +
>>> +failed:
>>> + if (ret)
>>> + xrt_devctl_remove(pdev);
>>> +
>>> + return ret;
>>> +}
>>> +
>>> +static struct xrt_subdev_endpoints xrt_devctl_endpoints[] = {
>>> + {
>>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>>> + /* add name if ep is in same partition */
>>> + { .ep_name = XRT_MD_NODE_BLP_ROM },
>>> + { NULL },
>>> + },
>>> + .xse_min_ep = 1,
>>> + },
>>> + {
>>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>>> + { .ep_name = XRT_MD_NODE_GOLDEN_VER },
>>> + { NULL },
>>> + },
>>> + .xse_min_ep = 1,
>>> + },
>>> + /* adding ep bundle generates devctl device instance */
>>> + { 0 },
>>> +};
>>> +
>>> +static struct xrt_subdev_drvdata xrt_devctl_data = {
>>> + .xsd_dev_ops = {
>>> + .xsd_ioctl = xrt_devctl_leaf_ioctl,
>>> + },
>>> +};
>>> +
>>> +static const struct platform_device_id xrt_devctl_table[] = {
>>> + { XRT_DEVCTL, (kernel_ulong_t)&xrt_devctl_data },
>>> + { },
>>> +};
>>> +
>>> +static struct platform_driver xrt_devctl_driver = {
>>> + .driver = {
>>> + .name = XRT_DEVCTL,
>>> + },
>>> + .probe = xrt_devctl_probe,
>>> + .remove = xrt_devctl_remove,
>>> + .id_table = xrt_devctl_table,
>>> +};
>>> +
>>> +void devctl_leaf_init_fini(bool init)
>>> +{
>>> + if (init)
>>> + xleaf_register_driver(XRT_SUBDEV_DEVCTL, &xrt_devctl_driver, xrt_devctl_endpoints);
>>> + else
>>> + xleaf_unregister_driver(XRT_SUBDEV_DEVCTL);
>>> +}
>
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 14/18] fpga: xrt: clock platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (12 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 13/18] fpga: xrt: devctl " Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-03-05 15:23 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 15/18] fpga: xrt: clock frequence counter " Lizhi Hou
` (5 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Add clock driver. Clock is a hardware function discovered by walking
xclbin metadata. A platform device node will be created for it. Other
part of driver configures clock through clock driver.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xleaf/clock.h | 31 ++
drivers/fpga/xrt/lib/xleaf/clock.c | 648 +++++++++++++++++++++++++
2 files changed, 679 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
diff --git a/drivers/fpga/xrt/include/xleaf/clock.h b/drivers/fpga/xrt/include/xleaf/clock.h
new file mode 100644
index 000000000000..a2da59b32551
--- /dev/null
+++ b/drivers/fpga/xrt/include/xleaf/clock.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for XRT Clock Leaf Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_CLOCK_H_
+#define _XRT_CLOCK_H_
+
+#include "xleaf.h"
+#include <linux/xrt/xclbin.h>
+
+/*
+ * CLOCK driver IOCTL calls.
+ */
+enum xrt_clock_ioctl_cmd {
+ XRT_CLOCK_SET = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+ XRT_CLOCK_GET,
+ XRT_CLOCK_VERIFY,
+};
+
+struct xrt_clock_ioctl_get {
+ u16 freq;
+ u32 freq_cnter;
+};
+
+#endif /* _XRT_CLOCK_H_ */
diff --git a/drivers/fpga/xrt/lib/xleaf/clock.c b/drivers/fpga/xrt/lib/xleaf/clock.c
new file mode 100644
index 000000000000..a067b501a607
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xleaf/clock.c
@@ -0,0 +1,648 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Clock Wizard Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou<Lizhi.Hou@xilinx.com>
+ * Sonal Santan <sonals@xilinx.com>
+ * David Zhang <davidzha@xilinx.com>
+ */
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include "metadata.h"
+#include "xleaf.h"
+#include "xleaf/clock.h"
+#include "xleaf/clkfreq.h"
+
+/* CLOCK_MAX_NUM_CLOCKS should be a concept from XCLBIN_ in the future */
+#define CLOCK_MAX_NUM_CLOCKS 4
+#define OCL_CLKWIZ_STATUS_OFFSET 0x4
+#define OCL_CLKWIZ_STATUS_MASK 0xffff
+#define OCL_CLKWIZ_STATUS_MEASURE_START 0x1
+#define OCL_CLKWIZ_STATUS_MEASURE_DONE 0x2
+#define OCL_CLKWIZ_CONFIG_OFFSET(n) (0x200 + 4 * (n))
+#define CLOCK_DEFAULT_EXPIRE_SECS 1
+
+#define CLOCK_ERR(clock, fmt, arg...) \
+ xrt_err((clock)->pdev, fmt "\n", ##arg)
+#define CLOCK_WARN(clock, fmt, arg...) \
+ xrt_warn((clock)->pdev, fmt "\n", ##arg)
+#define CLOCK_INFO(clock, fmt, arg...) \
+ xrt_info((clock)->pdev, fmt "\n", ##arg)
+#define CLOCK_DBG(clock, fmt, arg...) \
+ xrt_dbg((clock)->pdev, fmt "\n", ##arg)
+
+#define XRT_CLOCK "xrt_clock"
+
+struct clock {
+ struct platform_device *pdev;
+ void __iomem *clock_base;
+ struct mutex clock_lock; /* clock dev lock */
+
+ const char *clock_ep_name;
+};
+
+/*
+ * Precomputed table with config0 and config2 register values together with
+ * target frequency. The steps are approximately 5 MHz apart. Table is
+ * generated by wiz.pl.
+ */
+static const struct xmgmt_ocl_clockwiz {
+ /* target frequency */
+ unsigned short ocl;
+ /* config0 register */
+ unsigned long config0;
+ /* config2 register */
+ unsigned int config2;
+} frequency_table[] = {
+ {/*1275.000*/ 10.000, 0x02EE0C01, 0x0001F47F},
+ {/*1575.000*/ 15.000, 0x02EE0F01, 0x00000069},
+ {/*1600.000*/ 20.000, 0x00001001, 0x00000050},
+ {/*1600.000*/ 25.000, 0x00001001, 0x00000040},
+ {/*1575.000*/ 30.000, 0x02EE0F01, 0x0001F434},
+ {/*1575.000*/ 35.000, 0x02EE0F01, 0x0000002D},
+ {/*1600.000*/ 40.000, 0x00001001, 0x00000028},
+ {/*1575.000*/ 45.000, 0x02EE0F01, 0x00000023},
+ {/*1600.000*/ 50.000, 0x00001001, 0x00000020},
+ {/*1512.500*/ 55.000, 0x007D0F01, 0x0001F41B},
+ {/*1575.000*/ 60.000, 0x02EE0F01, 0x0000FA1A},
+ {/*1462.500*/ 65.000, 0x02710E01, 0x0001F416},
+ {/*1575.000*/ 70.000, 0x02EE0F01, 0x0001F416},
+ {/*1575.000*/ 75.000, 0x02EE0F01, 0x00000015},
+ {/*1600.000*/ 80.000, 0x00001001, 0x00000014},
+ {/*1487.500*/ 85.000, 0x036B0E01, 0x0001F411},
+ {/*1575.000*/ 90.000, 0x02EE0F01, 0x0001F411},
+ {/*1425.000*/ 95.000, 0x00FA0E01, 0x0000000F},
+ {/*1600.000*/ 100.000, 0x00001001, 0x00000010},
+ {/*1575.000*/ 105.000, 0x02EE0F01, 0x0000000F},
+ {/*1512.500*/ 110.000, 0x007D0F01, 0x0002EE0D},
+ {/*1437.500*/ 115.000, 0x01770E01, 0x0001F40C},
+ {/*1575.000*/ 120.000, 0x02EE0F01, 0x00007D0D},
+ {/*1562.500*/ 125.000, 0x02710F01, 0x0001F40C},
+ {/*1462.500*/ 130.000, 0x02710E01, 0x0000FA0B},
+ {/*1350.000*/ 135.000, 0x01F40D01, 0x0000000A},
+ {/*1575.000*/ 140.000, 0x02EE0F01, 0x0000FA0B},
+ {/*1450.000*/ 145.000, 0x01F40E01, 0x0000000A},
+ {/*1575.000*/ 150.000, 0x02EE0F01, 0x0001F40A},
+ {/*1550.000*/ 155.000, 0x01F40F01, 0x0000000A},
+ {/*1600.000*/ 160.000, 0x00001001, 0x0000000A},
+ {/*1237.500*/ 165.000, 0x01770C01, 0x0001F407},
+ {/*1487.500*/ 170.000, 0x036B0E01, 0x0002EE08},
+ {/*1575.000*/ 175.000, 0x02EE0F01, 0x00000009},
+ {/*1575.000*/ 180.000, 0x02EE0F01, 0x0002EE08},
+ {/*1387.500*/ 185.000, 0x036B0D01, 0x0001F407},
+ {/*1425.000*/ 190.000, 0x00FA0E01, 0x0001F407},
+ {/*1462.500*/ 195.000, 0x02710E01, 0x0001F407},
+ {/*1600.000*/ 200.000, 0x00001001, 0x00000008},
+ {/*1537.500*/ 205.000, 0x01770F01, 0x0001F407},
+ {/*1575.000*/ 210.000, 0x02EE0F01, 0x0001F407},
+ {/*1075.000*/ 215.000, 0x02EE0A01, 0x00000005},
+ {/*1512.500*/ 220.000, 0x007D0F01, 0x00036B06},
+ {/*1575.000*/ 225.000, 0x02EE0F01, 0x00000007},
+ {/*1437.500*/ 230.000, 0x01770E01, 0x0000FA06},
+ {/*1175.000*/ 235.000, 0x02EE0B01, 0x00000005},
+ {/*1500.000*/ 240.000, 0x00000F01, 0x0000FA06},
+ {/*1225.000*/ 245.000, 0x00FA0C01, 0x00000005},
+ {/*1562.500*/ 250.000, 0x02710F01, 0x0000FA06},
+ {/*1275.000*/ 255.000, 0x02EE0C01, 0x00000005},
+ {/*1462.500*/ 260.000, 0x02710E01, 0x00027105},
+ {/*1325.000*/ 265.000, 0x00FA0D01, 0x00000005},
+ {/*1350.000*/ 270.000, 0x01F40D01, 0x00000005},
+ {/*1512.500*/ 275.000, 0x007D0F01, 0x0001F405},
+ {/*1575.000*/ 280.000, 0x02EE0F01, 0x00027105},
+ {/*1425.000*/ 285.000, 0x00FA0E01, 0x00000005},
+ {/*1450.000*/ 290.000, 0x01F40E01, 0x00000005},
+ {/*1475.000*/ 295.000, 0x02EE0E01, 0x00000005},
+ {/*1575.000*/ 300.000, 0x02EE0F01, 0x0000FA05},
+ {/*1525.000*/ 305.000, 0x00FA0F01, 0x00000005},
+ {/*1550.000*/ 310.000, 0x01F40F01, 0x00000005},
+ {/*1575.000*/ 315.000, 0x02EE0F01, 0x00000005},
+ {/*1600.000*/ 320.000, 0x00001001, 0x00000005},
+ {/*1462.500*/ 325.000, 0x02710E01, 0x0001F404},
+ {/*1237.500*/ 330.000, 0x01770C01, 0x0002EE03},
+ {/*837.500*/ 335.000, 0x01770801, 0x0001F402},
+ {/*1487.500*/ 340.000, 0x036B0E01, 0x00017704},
+ {/*862.500*/ 345.000, 0x02710801, 0x0001F402},
+ {/*1575.000*/ 350.000, 0x02EE0F01, 0x0001F404},
+ {/*887.500*/ 355.000, 0x036B0801, 0x0001F402},
+ {/*1575.000*/ 360.000, 0x02EE0F01, 0x00017704},
+ {/*912.500*/ 365.000, 0x007D0901, 0x0001F402},
+ {/*1387.500*/ 370.000, 0x036B0D01, 0x0002EE03},
+ {/*1500.000*/ 375.000, 0x00000F01, 0x00000004},
+ {/*1425.000*/ 380.000, 0x00FA0E01, 0x0002EE03},
+ {/*962.500*/ 385.000, 0x02710901, 0x0001F402},
+ {/*1462.500*/ 390.000, 0x02710E01, 0x0002EE03},
+ {/*987.500*/ 395.000, 0x036B0901, 0x0001F402},
+ {/*1600.000*/ 400.000, 0x00001001, 0x00000004},
+ {/*1012.500*/ 405.000, 0x007D0A01, 0x0001F402},
+ {/*1537.500*/ 410.000, 0x01770F01, 0x0002EE03},
+ {/*1037.500*/ 415.000, 0x01770A01, 0x0001F402},
+ {/*1575.000*/ 420.000, 0x02EE0F01, 0x0002EE03},
+ {/*1487.500*/ 425.000, 0x036B0E01, 0x0001F403},
+ {/*1075.000*/ 430.000, 0x02EE0A01, 0x0001F402},
+ {/*1087.500*/ 435.000, 0x036B0A01, 0x0001F402},
+ {/*1375.000*/ 440.000, 0x02EE0D01, 0x00007D03},
+ {/*1112.500*/ 445.000, 0x007D0B01, 0x0001F402},
+ {/*1575.000*/ 450.000, 0x02EE0F01, 0x0001F403},
+ {/*1137.500*/ 455.000, 0x01770B01, 0x0001F402},
+ {/*1437.500*/ 460.000, 0x01770E01, 0x00007D03},
+ {/*1162.500*/ 465.000, 0x02710B01, 0x0001F402},
+ {/*1175.000*/ 470.000, 0x02EE0B01, 0x0001F402},
+ {/*1425.000*/ 475.000, 0x00FA0E01, 0x00000003},
+ {/*1500.000*/ 480.000, 0x00000F01, 0x00007D03},
+ {/*1212.500*/ 485.000, 0x007D0C01, 0x0001F402},
+ {/*1225.000*/ 490.000, 0x00FA0C01, 0x0001F402},
+ {/*1237.500*/ 495.000, 0x01770C01, 0x0001F402},
+ {/*1562.500*/ 500.000, 0x02710F01, 0x00007D03},
+ {/*1262.500*/ 505.000, 0x02710C01, 0x0001F402},
+ {/*1275.000*/ 510.000, 0x02EE0C01, 0x0001F402},
+ {/*1287.500*/ 515.000, 0x036B0C01, 0x0001F402},
+ {/*1300.000*/ 520.000, 0x00000D01, 0x0001F402},
+ {/*1575.000*/ 525.000, 0x02EE0F01, 0x00000003},
+ {/*1325.000*/ 530.000, 0x00FA0D01, 0x0001F402},
+ {/*1337.500*/ 535.000, 0x01770D01, 0x0001F402},
+ {/*1350.000*/ 540.000, 0x01F40D01, 0x0001F402},
+ {/*1362.500*/ 545.000, 0x02710D01, 0x0001F402},
+ {/*1512.500*/ 550.000, 0x007D0F01, 0x0002EE02},
+ {/*1387.500*/ 555.000, 0x036B0D01, 0x0001F402},
+ {/*1400.000*/ 560.000, 0x00000E01, 0x0001F402},
+ {/*1412.500*/ 565.000, 0x007D0E01, 0x0001F402},
+ {/*1425.000*/ 570.000, 0x00FA0E01, 0x0001F402},
+ {/*1437.500*/ 575.000, 0x01770E01, 0x0001F402},
+ {/*1450.000*/ 580.000, 0x01F40E01, 0x0001F402},
+ {/*1462.500*/ 585.000, 0x02710E01, 0x0001F402},
+ {/*1475.000*/ 590.000, 0x02EE0E01, 0x0001F402},
+ {/*1487.500*/ 595.000, 0x036B0E01, 0x0001F402},
+ {/*1575.000*/ 600.000, 0x02EE0F01, 0x00027102},
+ {/*1512.500*/ 605.000, 0x007D0F01, 0x0001F402},
+ {/*1525.000*/ 610.000, 0x00FA0F01, 0x0001F402},
+ {/*1537.500*/ 615.000, 0x01770F01, 0x0001F402},
+ {/*1550.000*/ 620.000, 0x01F40F01, 0x0001F402},
+ {/*1562.500*/ 625.000, 0x02710F01, 0x0001F402},
+ {/*1575.000*/ 630.000, 0x02EE0F01, 0x0001F402},
+ {/*1587.500*/ 635.000, 0x036B0F01, 0x0001F402},
+ {/*1600.000*/ 640.000, 0x00001001, 0x0001F402},
+ {/*1290.000*/ 645.000, 0x01F44005, 0x00000002},
+ {/*1462.500*/ 650.000, 0x02710E01, 0x0000FA02}
+};
+
+static inline u32 reg_rd(struct clock *clock, u32 offset)
+{
+ return ioread32(clock->clock_base + offset);
+}
+
+static inline void reg_wr(struct clock *clock, u32 val, u32 offset)
+{
+ iowrite32(val, clock->clock_base + offset);
+}
+
+static u32 find_matching_freq_config(unsigned short freq,
+ const struct xmgmt_ocl_clockwiz *table,
+ int size)
+{
+ u32 start = 0;
+ u32 end = size - 1;
+ u32 idx = size - 1;
+
+ if (freq < table[0].ocl)
+ return 0;
+
+ if (freq > table[size - 1].ocl)
+ return size - 1;
+
+ while (start < end) {
+ if (freq == table[idx].ocl)
+ break;
+ if (freq < table[idx].ocl)
+ end = idx;
+ else
+ start = idx + 1;
+ idx = start + (end - start) / 2;
+ }
+ if (freq < table[idx].ocl)
+ idx--;
+
+ return idx;
+}
+
+static u32 find_matching_freq(u32 freq,
+ const struct xmgmt_ocl_clockwiz *freq_table,
+ int freq_table_size)
+{
+ int idx = find_matching_freq_config(freq, freq_table, freq_table_size);
+
+ return freq_table[idx].ocl;
+}
+
+static inline int clock_wiz_busy(struct clock *clock, int cycle, int interval)
+{
+ u32 val = 0;
+ int count;
+
+ val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
+ for (count = 0; val != 1 && count < cycle; count++) {
+ mdelay(interval);
+ val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
+ }
+ if (val != 1) {
+ CLOCK_ERR(clock, "clockwiz is (%u) busy after %d ms",
+ val, cycle * interval);
+ return -ETIMEDOUT;
+ }
+
+ return 0;
+}
+
+static int get_freq(struct clock *clock, u16 *freq)
+{
+#define XCL_INPUT_FREQ 100
+ const u64 input = XCL_INPUT_FREQ;
+ u32 val;
+ u32 mul0, div0;
+ u32 mul_frac0 = 0;
+ u32 div1;
+ u32 div_frac1 = 0;
+
+ WARN_ON(!mutex_is_locked(&clock->clock_lock));
+
+ val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
+ if ((val & 0x1) == 0) {
+ CLOCK_ERR(clock, "clockwiz is busy %x", val);
+ *freq = 0;
+ return -EBUSY;
+ }
+
+ val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(0));
+
+ div0 = val & 0xff;
+ mul0 = (val & 0xff00) >> 8;
+ if (val & BIT(26)) {
+ mul_frac0 = val >> 16;
+ mul_frac0 &= 0x3ff;
+ }
+
+ /*
+ * Multiply both numerator (mul0) and the denominator (div0) with 1000
+ * to account for fractional portion of multiplier
+ */
+ mul0 *= 1000;
+ mul0 += mul_frac0;
+ div0 *= 1000;
+
+ val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(2));
+
+ div1 = val & 0xff;
+ if (val & BIT(18)) {
+ div_frac1 = val >> 8;
+ div_frac1 &= 0x3ff;
+ }
+
+ /*
+ * Multiply both numerator (mul0) and the denominator (div1) with
+ * 1000 to account for fractional portion of divider
+ */
+
+ div1 *= 1000;
+ div1 += div_frac1;
+ div0 *= div1;
+ mul0 *= 1000;
+ if (div0 == 0) {
+ CLOCK_ERR(clock, "clockwiz 0 divider");
+ return 0;
+ }
+
+ *freq = (u16)((input * mul0) / div0);
+
+ return 0;
+}
+
+static int set_freq(struct clock *clock, u16 freq)
+{
+ u32 config;
+ int err;
+ u32 idx = 0;
+ u32 val;
+
+ WARN_ON(!mutex_is_locked(&clock->clock_lock));
+
+ idx = find_matching_freq_config(freq, frequency_table,
+ ARRAY_SIZE(frequency_table));
+
+ CLOCK_INFO(clock, "New: %d Mhz", freq);
+ err = clock_wiz_busy(clock, 20, 50);
+ if (err)
+ return -EBUSY;
+
+ config = frequency_table[idx].config0;
+ reg_wr(clock, config, OCL_CLKWIZ_CONFIG_OFFSET(0));
+
+ config = frequency_table[idx].config2;
+ reg_wr(clock, config, OCL_CLKWIZ_CONFIG_OFFSET(2));
+
+ mdelay(10);
+ reg_wr(clock, 7, OCL_CLKWIZ_CONFIG_OFFSET(23));
+
+ mdelay(1);
+ reg_wr(clock, 2, OCL_CLKWIZ_CONFIG_OFFSET(23));
+
+ CLOCK_INFO(clock, "clockwiz waiting for locked signal");
+
+ err = clock_wiz_busy(clock, 100, 100);
+ if (err) {
+ CLOCK_ERR(clock, "clockwiz MMCM/PLL did not lock");
+ /* restore */
+ reg_wr(clock, 4, OCL_CLKWIZ_CONFIG_OFFSET(23));
+ mdelay(10);
+ reg_wr(clock, 0, OCL_CLKWIZ_CONFIG_OFFSET(23));
+ return err;
+ }
+ val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(0));
+ CLOCK_INFO(clock, "clockwiz CONFIG(0) 0x%x", val);
+ val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(2));
+ CLOCK_INFO(clock, "clockwiz CONFIG(2) 0x%x", val);
+
+ return 0;
+}
+
+static int get_freq_counter(struct clock *clock, u32 *freq)
+{
+ const void *cnter;
+ struct platform_device *cnter_leaf;
+ struct platform_device *pdev = clock->pdev;
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(clock->pdev);
+ int err = xrt_md_get_prop(DEV(pdev), pdata->xsp_dtb,
+ clock->clock_ep_name, NULL, XRT_MD_PROP_CLK_CNT, &cnter, NULL);
+
+ WARN_ON(!mutex_is_locked(&clock->clock_lock));
+
+ if (err) {
+ xrt_err(pdev, "no counter specified");
+ return err;
+ }
+
+ cnter_leaf = xleaf_get_leaf_by_epname(pdev, cnter);
+ if (!cnter_leaf) {
+ xrt_err(pdev, "can't find counter");
+ return -ENOENT;
+ }
+
+ err = xleaf_ioctl(cnter_leaf, XRT_CLKFREQ_READ, freq);
+ if (err)
+ xrt_err(pdev, "can't read counter");
+ xleaf_put_leaf(clock->pdev, cnter_leaf);
+
+ return err;
+}
+
+static int clock_get_freq(struct clock *clock, u16 *freq, u32 *freq_cnter)
+{
+ int err = 0;
+
+ mutex_lock(&clock->clock_lock);
+
+ if (err == 0 && freq)
+ err = get_freq(clock, freq);
+
+ if (err == 0 && freq_cnter)
+ err = get_freq_counter(clock, freq_cnter);
+
+ mutex_unlock(&clock->clock_lock);
+ return err;
+}
+
+static int clock_set_freq(struct clock *clock, u16 freq)
+{
+ int err;
+
+ mutex_lock(&clock->clock_lock);
+ err = set_freq(clock, freq);
+ mutex_unlock(&clock->clock_lock);
+
+ return err;
+}
+
+static int clock_verify_freq(struct clock *clock)
+{
+ int err = 0;
+ u16 freq;
+ u32 lookup_freq, clock_freq_counter, request_in_khz, tolerance;
+
+ mutex_lock(&clock->clock_lock);
+
+ err = get_freq(clock, &freq);
+ if (err) {
+ xrt_err(clock->pdev, "get freq failed, %d", err);
+ goto end;
+ }
+
+ err = get_freq_counter(clock, &clock_freq_counter);
+ if (err) {
+ xrt_err(clock->pdev, "get freq counter failed, %d", err);
+ goto end;
+ }
+
+ lookup_freq = find_matching_freq(freq, frequency_table,
+ ARRAY_SIZE(frequency_table));
+ request_in_khz = lookup_freq * 1000;
+ tolerance = lookup_freq * 50;
+ if (tolerance < abs(clock_freq_counter - request_in_khz)) {
+ CLOCK_ERR(clock,
+ "set clock(%s) failed, request %ukhz, actual %dkhz",
+ clock->clock_ep_name, request_in_khz, clock_freq_counter);
+ err = -EDOM;
+ } else {
+ CLOCK_INFO(clock, "verified clock (%s)", clock->clock_ep_name);
+ }
+
+end:
+ mutex_unlock(&clock->clock_lock);
+ return err;
+}
+
+static int clock_init(struct clock *clock)
+{
+ struct xrt_subdev_platdata *pdata = DEV_PDATA(clock->pdev);
+ int err = 0;
+ const u16 *freq;
+
+ err = xrt_md_get_prop(DEV(clock->pdev), pdata->xsp_dtb,
+ clock->clock_ep_name, NULL, XRT_MD_PROP_CLK_FREQ,
+ (const void **)&freq, NULL);
+ if (err) {
+ xrt_info(clock->pdev, "no default freq");
+ return 0;
+ }
+
+ mutex_lock(&clock->clock_lock);
+ err = set_freq(clock, be16_to_cpu(*freq));
+ mutex_unlock(&clock->clock_lock);
+
+ return err;
+}
+
+static ssize_t freq_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct clock *clock = platform_get_drvdata(to_platform_device(dev));
+ u16 freq = 0;
+ ssize_t count;
+
+ count = clock_get_freq(clock, &freq, NULL);
+ if (count < 0)
+ return count;
+
+ count = snprintf(buf, 64, "%d\n", freq);
+
+ return count;
+}
+static DEVICE_ATTR_RO(freq);
+
+static struct attribute *clock_attrs[] = {
+ &dev_attr_freq.attr,
+ NULL,
+};
+
+static struct attribute_group clock_attr_group = {
+ .attrs = clock_attrs,
+};
+
+static int
+xrt_clock_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ struct clock *clock;
+ int ret = 0;
+
+ clock = platform_get_drvdata(pdev);
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ /* Does not handle any event. */
+ break;
+ case XRT_CLOCK_SET: {
+ u16 freq = (u16)(uintptr_t)arg;
+
+ ret = clock_set_freq(clock, freq);
+ break;
+ }
+ case XRT_CLOCK_VERIFY: {
+ ret = clock_verify_freq(clock);
+ break;
+ }
+ case XRT_CLOCK_GET: {
+ struct xrt_clock_ioctl_get *get =
+ (struct xrt_clock_ioctl_get *)arg;
+
+ ret = clock_get_freq(clock, &get->freq, &get->freq_cnter);
+ break;
+ }
+ default:
+ xrt_err(pdev, "unsupported cmd %d", cmd);
+ return -EINVAL;
+ }
+
+ return ret;
+}
+
+static int clock_remove(struct platform_device *pdev)
+{
+ struct clock *clock;
+
+ clock = platform_get_drvdata(pdev);
+ if (!clock) {
+ xrt_err(pdev, "driver data is NULL");
+ return -EINVAL;
+ }
+
+ platform_set_drvdata(pdev, NULL);
+ devm_kfree(&pdev->dev, clock);
+
+ CLOCK_INFO(clock, "successfully removed Clock subdev");
+ return 0;
+}
+
+static int clock_probe(struct platform_device *pdev)
+{
+ struct clock *clock = NULL;
+ struct resource *res;
+ int ret;
+
+ clock = devm_kzalloc(&pdev->dev, sizeof(*clock), GFP_KERNEL);
+ if (!clock)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, clock);
+ clock->pdev = pdev;
+ mutex_init(&clock->clock_lock);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ clock->clock_base = ioremap(res->start, res->end - res->start + 1);
+ if (!clock->clock_base) {
+ CLOCK_ERR(clock, "map base %pR failed", res);
+ ret = -EFAULT;
+ goto failed;
+ }
+
+ clock->clock_ep_name = res->name;
+
+ ret = clock_init(clock);
+ if (ret)
+ goto failed;
+
+ ret = sysfs_create_group(&pdev->dev.kobj, &clock_attr_group);
+ if (ret) {
+ CLOCK_ERR(clock, "create clock attrs failed: %d", ret);
+ goto failed;
+ }
+
+ CLOCK_INFO(clock, "successfully initialized Clock subdev");
+
+ return 0;
+
+failed:
+ clock_remove(pdev);
+ return ret;
+}
+
+static struct xrt_subdev_endpoints xrt_clock_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ { .regmap_name = "clkwiz" },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xrt_clock_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_clock_leaf_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_clock_table[] = {
+ { XRT_CLOCK, (kernel_ulong_t)&xrt_clock_data },
+ { },
+};
+
+static struct platform_driver xrt_clock_driver = {
+ .driver = {
+ .name = XRT_CLOCK,
+ },
+ .probe = clock_probe,
+ .remove = clock_remove,
+ .id_table = xrt_clock_table,
+};
+
+void clock_leaf_init_fini(bool init)
+{
+ if (init)
+ xleaf_register_driver(XRT_SUBDEV_CLOCK, &xrt_clock_driver, xrt_clock_endpoints);
+ else
+ xleaf_unregister_driver(XRT_SUBDEV_CLOCK);
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 14/18] fpga: xrt: clock platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 14/18] fpga: xrt: clock " Lizhi Hou
@ 2021-03-05 15:23 ` Tom Rix
2021-03-11 0:12 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-05 15:23 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
why are clock and clkfeq separated ?
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Add clock driver. Clock is a hardware function discovered by walking
> xclbin metadata. A platform device node will be created for it. Other
> part of driver configures clock through clock driver.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/clock.h | 31 ++
> drivers/fpga/xrt/lib/xleaf/clock.c | 648 +++++++++++++++++++++++++
> 2 files changed, 679 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/clock.h b/drivers/fpga/xrt/include/xleaf/clock.h
> new file mode 100644
> index 000000000000..a2da59b32551
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/clock.h
> @@ -0,0 +1,31 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT Clock Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_CLOCK_H_
> +#define _XRT_CLOCK_H_
> +
> +#include "xleaf.h"
> +#include <linux/xrt/xclbin.h>
> +
> +/*
> + * CLOCK driver IOCTL calls.
> + */
> +enum xrt_clock_ioctl_cmd {
> + XRT_CLOCK_SET = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> + XRT_CLOCK_GET,
> + XRT_CLOCK_VERIFY,
> +};
> +
> +struct xrt_clock_ioctl_get {
> + u16 freq;
> + u32 freq_cnter;
> +};
> +
> +#endif /* _XRT_CLOCK_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/clock.c b/drivers/fpga/xrt/lib/xleaf/clock.c
> new file mode 100644
> index 000000000000..a067b501a607
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/clock.c
> @@ -0,0 +1,648 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA Clock Wizard Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + * Sonal Santan <sonals@xilinx.com>
> + * David Zhang <davidzha@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/clock.h"
> +#include "xleaf/clkfreq.h"
> +
> +/* CLOCK_MAX_NUM_CLOCKS should be a concept from XCLBIN_ in the future */
> +#define CLOCK_MAX_NUM_CLOCKS 4
> +#define OCL_CLKWIZ_STATUS_OFFSET 0x4
OCL_CLKWIZ does not match the name of this file, change to something like XRT_CLOCK
> +#define OCL_CLKWIZ_STATUS_MASK 0xffff
> +#define OCL_CLKWIZ_STATUS_MEASURE_START 0x1
> +#define OCL_CLKWIZ_STATUS_MEASURE_DONE 0x2
> +#define OCL_CLKWIZ_CONFIG_OFFSET(n) (0x200 + 4 * (n))
This should be expanded to logical names of the registers.
It's use below has a magic indexes.
> +#define CLOCK_DEFAULT_EXPIRE_SECS 1
> +
> +#define CLOCK_ERR(clock, fmt, arg...) \
> + xrt_err((clock)->pdev, fmt "\n", ##arg)
> +#define CLOCK_WARN(clock, fmt, arg...) \
> + xrt_warn((clock)->pdev, fmt "\n", ##arg)
> +#define CLOCK_INFO(clock, fmt, arg...) \
> + xrt_info((clock)->pdev, fmt "\n", ##arg)
> +#define CLOCK_DBG(clock, fmt, arg...) \
> + xrt_dbg((clock)->pdev, fmt "\n", ##arg)
> +
> +#define XRT_CLOCK "xrt_clock"
> +
> +struct clock {
> + struct platform_device *pdev;
> + void __iomem *clock_base;
> + struct mutex clock_lock; /* clock dev lock */
> +
> + const char *clock_ep_name;
> +};
> +
> +/*
> + * Precomputed table with config0 and config2 register values together with
> + * target frequency. The steps are approximately 5 MHz apart. Table is
> + * generated by wiz.pl.
where is wiz.pl ? include the script
> + */
> +static const struct xmgmt_ocl_clockwiz {
> + /* target frequency */
> + unsigned short ocl;
> + /* config0 register */
> + unsigned long config0;
Should be u32
> + /* config2 register */
> + unsigned int config2;
> +} frequency_table[] = {
> + {/*1275.000*/ 10.000, 0x02EE0C01, 0x0001F47F},
Could clean up this table, to use spaces instead of tabs, move the comment out of branches
/* freq */ { ocl, config0, config2 },
> + {/*1575.000*/ 15.000, 0x02EE0F01, 0x00000069},
ocl is short, the data if float, change the generator output an integer
> + {/*1600.000*/ 20.000, 0x00001001, 0x00000050},
> + {/*1600.000*/ 25.000, 0x00001001, 0x00000040},
> + {/*1575.000*/ 30.000, 0x02EE0F01, 0x0001F434},
> + {/*1575.000*/ 35.000, 0x02EE0F01, 0x0000002D},
> + {/*1600.000*/ 40.000, 0x00001001, 0x00000028},
> + {/*1575.000*/ 45.000, 0x02EE0F01, 0x00000023},
> + {/*1600.000*/ 50.000, 0x00001001, 0x00000020},
> + {/*1512.500*/ 55.000, 0x007D0F01, 0x0001F41B},
> + {/*1575.000*/ 60.000, 0x02EE0F01, 0x0000FA1A},
> + {/*1462.500*/ 65.000, 0x02710E01, 0x0001F416},
> + {/*1575.000*/ 70.000, 0x02EE0F01, 0x0001F416},
> + {/*1575.000*/ 75.000, 0x02EE0F01, 0x00000015},
> + {/*1600.000*/ 80.000, 0x00001001, 0x00000014},
> + {/*1487.500*/ 85.000, 0x036B0E01, 0x0001F411},
> + {/*1575.000*/ 90.000, 0x02EE0F01, 0x0001F411},
> + {/*1425.000*/ 95.000, 0x00FA0E01, 0x0000000F},
> + {/*1600.000*/ 100.000, 0x00001001, 0x00000010},
> + {/*1575.000*/ 105.000, 0x02EE0F01, 0x0000000F},
> + {/*1512.500*/ 110.000, 0x007D0F01, 0x0002EE0D},
> + {/*1437.500*/ 115.000, 0x01770E01, 0x0001F40C},
> + {/*1575.000*/ 120.000, 0x02EE0F01, 0x00007D0D},
> + {/*1562.500*/ 125.000, 0x02710F01, 0x0001F40C},
> + {/*1462.500*/ 130.000, 0x02710E01, 0x0000FA0B},
> + {/*1350.000*/ 135.000, 0x01F40D01, 0x0000000A},
> + {/*1575.000*/ 140.000, 0x02EE0F01, 0x0000FA0B},
> + {/*1450.000*/ 145.000, 0x01F40E01, 0x0000000A},
> + {/*1575.000*/ 150.000, 0x02EE0F01, 0x0001F40A},
> + {/*1550.000*/ 155.000, 0x01F40F01, 0x0000000A},
> + {/*1600.000*/ 160.000, 0x00001001, 0x0000000A},
> + {/*1237.500*/ 165.000, 0x01770C01, 0x0001F407},
> + {/*1487.500*/ 170.000, 0x036B0E01, 0x0002EE08},
> + {/*1575.000*/ 175.000, 0x02EE0F01, 0x00000009},
> + {/*1575.000*/ 180.000, 0x02EE0F01, 0x0002EE08},
> + {/*1387.500*/ 185.000, 0x036B0D01, 0x0001F407},
> + {/*1425.000*/ 190.000, 0x00FA0E01, 0x0001F407},
> + {/*1462.500*/ 195.000, 0x02710E01, 0x0001F407},
> + {/*1600.000*/ 200.000, 0x00001001, 0x00000008},
> + {/*1537.500*/ 205.000, 0x01770F01, 0x0001F407},
> + {/*1575.000*/ 210.000, 0x02EE0F01, 0x0001F407},
> + {/*1075.000*/ 215.000, 0x02EE0A01, 0x00000005},
> + {/*1512.500*/ 220.000, 0x007D0F01, 0x00036B06},
> + {/*1575.000*/ 225.000, 0x02EE0F01, 0x00000007},
> + {/*1437.500*/ 230.000, 0x01770E01, 0x0000FA06},
> + {/*1175.000*/ 235.000, 0x02EE0B01, 0x00000005},
> + {/*1500.000*/ 240.000, 0x00000F01, 0x0000FA06},
> + {/*1225.000*/ 245.000, 0x00FA0C01, 0x00000005},
> + {/*1562.500*/ 250.000, 0x02710F01, 0x0000FA06},
> + {/*1275.000*/ 255.000, 0x02EE0C01, 0x00000005},
> + {/*1462.500*/ 260.000, 0x02710E01, 0x00027105},
> + {/*1325.000*/ 265.000, 0x00FA0D01, 0x00000005},
> + {/*1350.000*/ 270.000, 0x01F40D01, 0x00000005},
> + {/*1512.500*/ 275.000, 0x007D0F01, 0x0001F405},
> + {/*1575.000*/ 280.000, 0x02EE0F01, 0x00027105},
> + {/*1425.000*/ 285.000, 0x00FA0E01, 0x00000005},
> + {/*1450.000*/ 290.000, 0x01F40E01, 0x00000005},
> + {/*1475.000*/ 295.000, 0x02EE0E01, 0x00000005},
> + {/*1575.000*/ 300.000, 0x02EE0F01, 0x0000FA05},
> + {/*1525.000*/ 305.000, 0x00FA0F01, 0x00000005},
> + {/*1550.000*/ 310.000, 0x01F40F01, 0x00000005},
> + {/*1575.000*/ 315.000, 0x02EE0F01, 0x00000005},
> + {/*1600.000*/ 320.000, 0x00001001, 0x00000005},
> + {/*1462.500*/ 325.000, 0x02710E01, 0x0001F404},
> + {/*1237.500*/ 330.000, 0x01770C01, 0x0002EE03},
> + {/*837.500*/ 335.000, 0x01770801, 0x0001F402},
> + {/*1487.500*/ 340.000, 0x036B0E01, 0x00017704},
> + {/*862.500*/ 345.000, 0x02710801, 0x0001F402},
> + {/*1575.000*/ 350.000, 0x02EE0F01, 0x0001F404},
> + {/*887.500*/ 355.000, 0x036B0801, 0x0001F402},
> + {/*1575.000*/ 360.000, 0x02EE0F01, 0x00017704},
> + {/*912.500*/ 365.000, 0x007D0901, 0x0001F402},
> + {/*1387.500*/ 370.000, 0x036B0D01, 0x0002EE03},
> + {/*1500.000*/ 375.000, 0x00000F01, 0x00000004},
> + {/*1425.000*/ 380.000, 0x00FA0E01, 0x0002EE03},
> + {/*962.500*/ 385.000, 0x02710901, 0x0001F402},
> + {/*1462.500*/ 390.000, 0x02710E01, 0x0002EE03},
> + {/*987.500*/ 395.000, 0x036B0901, 0x0001F402},
> + {/*1600.000*/ 400.000, 0x00001001, 0x00000004},
> + {/*1012.500*/ 405.000, 0x007D0A01, 0x0001F402},
> + {/*1537.500*/ 410.000, 0x01770F01, 0x0002EE03},
> + {/*1037.500*/ 415.000, 0x01770A01, 0x0001F402},
> + {/*1575.000*/ 420.000, 0x02EE0F01, 0x0002EE03},
> + {/*1487.500*/ 425.000, 0x036B0E01, 0x0001F403},
> + {/*1075.000*/ 430.000, 0x02EE0A01, 0x0001F402},
> + {/*1087.500*/ 435.000, 0x036B0A01, 0x0001F402},
> + {/*1375.000*/ 440.000, 0x02EE0D01, 0x00007D03},
> + {/*1112.500*/ 445.000, 0x007D0B01, 0x0001F402},
> + {/*1575.000*/ 450.000, 0x02EE0F01, 0x0001F403},
> + {/*1137.500*/ 455.000, 0x01770B01, 0x0001F402},
> + {/*1437.500*/ 460.000, 0x01770E01, 0x00007D03},
> + {/*1162.500*/ 465.000, 0x02710B01, 0x0001F402},
> + {/*1175.000*/ 470.000, 0x02EE0B01, 0x0001F402},
> + {/*1425.000*/ 475.000, 0x00FA0E01, 0x00000003},
> + {/*1500.000*/ 480.000, 0x00000F01, 0x00007D03},
> + {/*1212.500*/ 485.000, 0x007D0C01, 0x0001F402},
> + {/*1225.000*/ 490.000, 0x00FA0C01, 0x0001F402},
> + {/*1237.500*/ 495.000, 0x01770C01, 0x0001F402},
> + {/*1562.500*/ 500.000, 0x02710F01, 0x00007D03},
> + {/*1262.500*/ 505.000, 0x02710C01, 0x0001F402},
> + {/*1275.000*/ 510.000, 0x02EE0C01, 0x0001F402},
> + {/*1287.500*/ 515.000, 0x036B0C01, 0x0001F402},
> + {/*1300.000*/ 520.000, 0x00000D01, 0x0001F402},
> + {/*1575.000*/ 525.000, 0x02EE0F01, 0x00000003},
> + {/*1325.000*/ 530.000, 0x00FA0D01, 0x0001F402},
> + {/*1337.500*/ 535.000, 0x01770D01, 0x0001F402},
> + {/*1350.000*/ 540.000, 0x01F40D01, 0x0001F402},
> + {/*1362.500*/ 545.000, 0x02710D01, 0x0001F402},
> + {/*1512.500*/ 550.000, 0x007D0F01, 0x0002EE02},
> + {/*1387.500*/ 555.000, 0x036B0D01, 0x0001F402},
> + {/*1400.000*/ 560.000, 0x00000E01, 0x0001F402},
> + {/*1412.500*/ 565.000, 0x007D0E01, 0x0001F402},
> + {/*1425.000*/ 570.000, 0x00FA0E01, 0x0001F402},
> + {/*1437.500*/ 575.000, 0x01770E01, 0x0001F402},
> + {/*1450.000*/ 580.000, 0x01F40E01, 0x0001F402},
> + {/*1462.500*/ 585.000, 0x02710E01, 0x0001F402},
> + {/*1475.000*/ 590.000, 0x02EE0E01, 0x0001F402},
> + {/*1487.500*/ 595.000, 0x036B0E01, 0x0001F402},
> + {/*1575.000*/ 600.000, 0x02EE0F01, 0x00027102},
> + {/*1512.500*/ 605.000, 0x007D0F01, 0x0001F402},
> + {/*1525.000*/ 610.000, 0x00FA0F01, 0x0001F402},
> + {/*1537.500*/ 615.000, 0x01770F01, 0x0001F402},
> + {/*1550.000*/ 620.000, 0x01F40F01, 0x0001F402},
> + {/*1562.500*/ 625.000, 0x02710F01, 0x0001F402},
> + {/*1575.000*/ 630.000, 0x02EE0F01, 0x0001F402},
> + {/*1587.500*/ 635.000, 0x036B0F01, 0x0001F402},
> + {/*1600.000*/ 640.000, 0x00001001, 0x0001F402},
> + {/*1290.000*/ 645.000, 0x01F44005, 0x00000002},
> + {/*1462.500*/ 650.000, 0x02710E01, 0x0000FA02}
> +};
> +
> +static inline u32 reg_rd(struct clock *clock, u32 offset)
> +{
> + return ioread32(clock->clock_base + offset);
> +}
> +
> +static inline void reg_wr(struct clock *clock, u32 val, u32 offset)
> +{
> + iowrite32(val, clock->clock_base + offset);
> +}
> +
> +static u32 find_matching_freq_config(unsigned short freq,
> + const struct xmgmt_ocl_clockwiz *table,
> + int size)
> +{
> + u32 start = 0;
> + u32 end = size - 1;
> + u32 idx = size - 1;
looks like you are doing a binary search, why not start idx at size / 2 ?
> +
> + if (freq < table[0].ocl)
> + return 0;
> +
> + if (freq > table[size - 1].ocl)
> + return size - 1;
> +
What aren't these two conditions errors ?
> + while (start < end) {
> + if (freq == table[idx].ocl)
> + break;
> + if (freq < table[idx].ocl)
> + end = idx;
> + else
> + start = idx + 1;
> + idx = start + (end - start) / 2;
> + }
> + if (freq < table[idx].ocl)
> + idx--;
> +
> + return idx;
> +}
> +
> +static u32 find_matching_freq(u32 freq,
> + const struct xmgmt_ocl_clockwiz *freq_table,
> + int freq_table_size)
> +{
> + int idx = find_matching_freq_config(freq, freq_table, freq_table_size);
> +
> + return freq_table[idx].ocl;
> +}
> +
> +static inline int clock_wiz_busy(struct clock *clock, int cycle, int interval)
> +{
> + u32 val = 0;
> + int count;
> +
> + val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
> + for (count = 0; val != 1 && count < cycle; count++) {
> + mdelay(interval);
> + val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
if (val == 1)
break;
Is a bit easier to follow than putting another condition in the for()
> + }
> + if (val != 1) {
> + CLOCK_ERR(clock, "clockwiz is (%u) busy after %d ms",
> + val, cycle * interval);
> + return -ETIMEDOUT;
> + }
> +
> + return 0;
> +}
> +
> +static int get_freq(struct clock *clock, u16 *freq)
> +{
> +#define XCL_INPUT_FREQ 100
??
If this a real parameter, it needs to go in the clock.h
> + const u64 input = XCL_INPUT_FREQ;
> + u32 val;
> + u32 mul0, div0;
> + u32 mul_frac0 = 0;
> + u32 div1;
> + u32 div_frac1 = 0;
> +
> + WARN_ON(!mutex_is_locked(&clock->clock_lock));
> +
> + val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
> + if ((val & 0x1) == 0) {
> + CLOCK_ERR(clock, "clockwiz is busy %x", val);
> + *freq = 0;
> + return -EBUSY;
> + }
> +
> + val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(0));
A general problem.
The '0' is magic, convert this to a logically named #define or enum.
> +
> + div0 = val & 0xff;
> + mul0 = (val & 0xff00) >> 8;
> + if (val & BIT(26)) {
> + mul_frac0 = val >> 16;
> + mul_frac0 &= 0x3ff;
> + }
> +
> + /*
> + * Multiply both numerator (mul0) and the denominator (div0) with 1000
> + * to account for fractional portion of multiplier
> + */
> + mul0 *= 1000;
> + mul0 += mul_frac0;
> + div0 *= 1000;
> +
> + val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(2));
> +
> + div1 = val & 0xff;
> + if (val & BIT(18)) {
> + div_frac1 = val >> 8;
> + div_frac1 &= 0x3ff;
> + }
> +
> + /*
> + * Multiply both numerator (mul0) and the denominator (div1) with
> + * 1000 to account for fractional portion of divider
> + */
> +
> + div1 *= 1000;
> + div1 += div_frac1;
> + div0 *= div1;
> + mul0 *= 1000;
> + if (div0 == 0) {
> + CLOCK_ERR(clock, "clockwiz 0 divider");
> + return 0;
> + }
> +
> + *freq = (u16)((input * mul0) / div0);
> +
> + return 0;
> +}
> +
> +static int set_freq(struct clock *clock, u16 freq)
> +{
> + u32 config;
> + int err;
> + u32 idx = 0;
> + u32 val;
> +
> + WARN_ON(!mutex_is_locked(&clock->clock_lock));
> +
> + idx = find_matching_freq_config(freq, frequency_table,
> + ARRAY_SIZE(frequency_table));
> +
> + CLOCK_INFO(clock, "New: %d Mhz", freq);
> + err = clock_wiz_busy(clock, 20, 50);
> + if (err)
> + return -EBUSY;
> +
> + config = frequency_table[idx].config0;
> + reg_wr(clock, config, OCL_CLKWIZ_CONFIG_OFFSET(0));
> +
> + config = frequency_table[idx].config2;
> + reg_wr(clock, config, OCL_CLKWIZ_CONFIG_OFFSET(2));
> +
> + mdelay(10);
> + reg_wr(clock, 7, OCL_CLKWIZ_CONFIG_OFFSET(23));
> +
> + mdelay(1);
> + reg_wr(clock, 2, OCL_CLKWIZ_CONFIG_OFFSET(23));
> +
> + CLOCK_INFO(clock, "clockwiz waiting for locked signal");
> +
> + err = clock_wiz_busy(clock, 100, 100);
> + if (err) {
> + CLOCK_ERR(clock, "clockwiz MMCM/PLL did not lock");
> + /* restore */
> + reg_wr(clock, 4, OCL_CLKWIZ_CONFIG_OFFSET(23));
> + mdelay(10);
> + reg_wr(clock, 0, OCL_CLKWIZ_CONFIG_OFFSET(23));
> + return err;
> + }
> + val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(0));
> + CLOCK_INFO(clock, "clockwiz CONFIG(0) 0x%x", val);
> + val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(2));
> + CLOCK_INFO(clock, "clockwiz CONFIG(2) 0x%x", val);
> +
> + return 0;
> +}
> +
> +static int get_freq_counter(struct clock *clock, u32 *freq)
> +{
> + const void *cnter;
variable name should be easy to read, use 'counter'
> + struct platform_device *cnter_leaf;
> + struct platform_device *pdev = clock->pdev;
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(clock->pdev);
> + int err = xrt_md_get_prop(DEV(pdev), pdata->xsp_dtb,
> + clock->clock_ep_name, NULL, XRT_MD_PROP_CLK_CNT, &cnter, NULL);
> +
> + WARN_ON(!mutex_is_locked(&clock->clock_lock));
> +
> + if (err) {
> + xrt_err(pdev, "no counter specified");
> + return err;
> + }
> +
> + cnter_leaf = xleaf_get_leaf_by_epname(pdev, cnter);
> + if (!cnter_leaf) {
> + xrt_err(pdev, "can't find counter");
> + return -ENOENT;
> + }
> +
> + err = xleaf_ioctl(cnter_leaf, XRT_CLKFREQ_READ, freq);
> + if (err)
> + xrt_err(pdev, "can't read counter");
> + xleaf_put_leaf(clock->pdev, cnter_leaf);
> +
> + return err;
> +}
> +
> +static int clock_get_freq(struct clock *clock, u16 *freq, u32 *freq_cnter)
> +{
> + int err = 0;
> +
> + mutex_lock(&clock->clock_lock);
> +
> + if (err == 0 && freq)
> + err = get_freq(clock, freq);
> +
> + if (err == 0 && freq_cnter)
> + err = get_freq_counter(clock, freq_cnter);
> +
> + mutex_unlock(&clock->clock_lock);
> + return err;
> +}
> +
> +static int clock_set_freq(struct clock *clock, u16 freq)
> +{
> + int err;
Why is this function needed ?
Move the locking into set_freq.
> +
> + mutex_lock(&clock->clock_lock);
> + err = set_freq(clock, freq);
> + mutex_unlock(&clock->clock_lock);
> +
> + return err;
> +}
> +
> +static int clock_verify_freq(struct clock *clock)
> +{
> + int err = 0;
> + u16 freq;
> + u32 lookup_freq, clock_freq_counter, request_in_khz, tolerance;
> +
> + mutex_lock(&clock->clock_lock);
> +
> + err = get_freq(clock, &freq);
> + if (err) {
> + xrt_err(clock->pdev, "get freq failed, %d", err);
> + goto end;
> + }
> +
> + err = get_freq_counter(clock, &clock_freq_counter);
> + if (err) {
> + xrt_err(clock->pdev, "get freq counter failed, %d", err);
> + goto end;
> + }
> +
> + lookup_freq = find_matching_freq(freq, frequency_table,
> + ARRAY_SIZE(frequency_table));
> + request_in_khz = lookup_freq * 1000;
> + tolerance = lookup_freq * 50;
> + if (tolerance < abs(clock_freq_counter - request_in_khz)) {
> + CLOCK_ERR(clock,
> + "set clock(%s) failed, request %ukhz, actual %dkhz",
> + clock->clock_ep_name, request_in_khz, clock_freq_counter);
> + err = -EDOM;
> + } else {
> + CLOCK_INFO(clock, "verified clock (%s)", clock->clock_ep_name);
> + }
> +
> +end:
> + mutex_unlock(&clock->clock_lock);
> + return err;
> +}
> +
> +static int clock_init(struct clock *clock)
> +{
> + struct xrt_subdev_platdata *pdata = DEV_PDATA(clock->pdev);
> + int err = 0;
> + const u16 *freq;
> +
> + err = xrt_md_get_prop(DEV(clock->pdev), pdata->xsp_dtb,
> + clock->clock_ep_name, NULL, XRT_MD_PROP_CLK_FREQ,
> + (const void **)&freq, NULL);
> + if (err) {
> + xrt_info(clock->pdev, "no default freq");
> + return 0;
> + }
> +
> + mutex_lock(&clock->clock_lock);
> + err = set_freq(clock, be16_to_cpu(*freq));
> + mutex_unlock(&clock->clock_lock);
> +
> + return err;
> +}
> +
> +static ssize_t freq_show(struct device *dev, struct device_attribute *attr, char *buf)
> +{
> + struct clock *clock = platform_get_drvdata(to_platform_device(dev));
> + u16 freq = 0;
> + ssize_t count;
> +
> + count = clock_get_freq(clock, &freq, NULL);
> + if (count < 0)
> + return count;
> +
> + count = snprintf(buf, 64, "%d\n", freq);
%u ?
Tom
> +
> + return count;
> +}
> +static DEVICE_ATTR_RO(freq);
> +
> +static struct attribute *clock_attrs[] = {
> + &dev_attr_freq.attr,
> + NULL,
> +};
> +
> +static struct attribute_group clock_attr_group = {
> + .attrs = clock_attrs,
> +};
> +
> +static int
> +xrt_clock_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct clock *clock;
> + int ret = 0;
> +
> + clock = platform_get_drvdata(pdev);
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + /* Does not handle any event. */
> + break;
> + case XRT_CLOCK_SET: {
> + u16 freq = (u16)(uintptr_t)arg;
> +
> + ret = clock_set_freq(clock, freq);
> + break;
> + }
> + case XRT_CLOCK_VERIFY: {
> + ret = clock_verify_freq(clock);
> + break;
> + }
> + case XRT_CLOCK_GET: {
> + struct xrt_clock_ioctl_get *get =
> + (struct xrt_clock_ioctl_get *)arg;
> +
> + ret = clock_get_freq(clock, &get->freq, &get->freq_cnter);
> + break;
> + }
> + default:
> + xrt_err(pdev, "unsupported cmd %d", cmd);
> + return -EINVAL;
> + }
> +
> + return ret;
> +}
> +
> +static int clock_remove(struct platform_device *pdev)
> +{
> + struct clock *clock;
> +
> + clock = platform_get_drvdata(pdev);
> + if (!clock) {
> + xrt_err(pdev, "driver data is NULL");
> + return -EINVAL;
> + }
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, clock);
> +
> + CLOCK_INFO(clock, "successfully removed Clock subdev");
> + return 0;
> +}
> +
> +static int clock_probe(struct platform_device *pdev)
> +{
> + struct clock *clock = NULL;
> + struct resource *res;
> + int ret;
> +
> + clock = devm_kzalloc(&pdev->dev, sizeof(*clock), GFP_KERNEL);
> + if (!clock)
> + return -ENOMEM;
> +
> + platform_set_drvdata(pdev, clock);
> + clock->pdev = pdev;
> + mutex_init(&clock->clock_lock);
> +
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + clock->clock_base = ioremap(res->start, res->end - res->start + 1);
> + if (!clock->clock_base) {
> + CLOCK_ERR(clock, "map base %pR failed", res);
> + ret = -EFAULT;
> + goto failed;
> + }
> +
> + clock->clock_ep_name = res->name;
> +
> + ret = clock_init(clock);
> + if (ret)
> + goto failed;
> +
> + ret = sysfs_create_group(&pdev->dev.kobj, &clock_attr_group);
> + if (ret) {
> + CLOCK_ERR(clock, "create clock attrs failed: %d", ret);
> + goto failed;
> + }
> +
> + CLOCK_INFO(clock, "successfully initialized Clock subdev");
> +
> + return 0;
> +
> +failed:
> + clock_remove(pdev);
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_clock_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .regmap_name = "clkwiz" },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_clock_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_clock_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_clock_table[] = {
> + { XRT_CLOCK, (kernel_ulong_t)&xrt_clock_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_clock_driver = {
> + .driver = {
> + .name = XRT_CLOCK,
> + },
> + .probe = clock_probe,
> + .remove = clock_remove,
> + .id_table = xrt_clock_table,
> +};
> +
> +void clock_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_CLOCK, &xrt_clock_driver, xrt_clock_endpoints);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_CLOCK);
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 14/18] fpga: xrt: clock platform driver
2021-03-05 15:23 ` Tom Rix
@ 2021-03-11 0:12 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-11 0:12 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 03/05/2021 07:23 AM, Tom Rix wrote:
> why are clock and clkfeq separated ?
clock and clkfreq are two different IPs. clkfreq is a simple counter to
confirm the clock output is expected.
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Add clock driver. Clock is a hardware function discovered by walking
>> xclbin metadata. A platform device node will be created for it. Other
>> part of driver configures clock through clock driver.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/clock.h | 31 ++
>> drivers/fpga/xrt/lib/xleaf/clock.c | 648 +++++++++++++++++++++++++
>> 2 files changed, 679 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
>>
>> diff --git a/drivers/fpga/xrt/include/xleaf/clock.h b/drivers/fpga/xrt/include/xleaf/clock.h
>> new file mode 100644
>> index 000000000000..a2da59b32551
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/clock.h
>> @@ -0,0 +1,31 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT Clock Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_CLOCK_H_
>> +#define _XRT_CLOCK_H_
>> +
>> +#include "xleaf.h"
>> +#include <linux/xrt/xclbin.h>
>> +
>> +/*
>> + * CLOCK driver IOCTL calls.
>> + */
>> +enum xrt_clock_ioctl_cmd {
>> + XRT_CLOCK_SET = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> + XRT_CLOCK_GET,
>> + XRT_CLOCK_VERIFY,
>> +};
>> +
>> +struct xrt_clock_ioctl_get {
>> + u16 freq;
>> + u32 freq_cnter;
>> +};
>> +
>> +#endif /* _XRT_CLOCK_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/clock.c b/drivers/fpga/xrt/lib/xleaf/clock.c
>> new file mode 100644
>> index 000000000000..a067b501a607
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/clock.c
>> @@ -0,0 +1,648 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA Clock Wizard Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + * Sonal Santan <sonals@xilinx.com>
>> + * David Zhang <davidzha@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/io.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/clock.h"
>> +#include "xleaf/clkfreq.h"
>> +
>> +/* CLOCK_MAX_NUM_CLOCKS should be a concept from XCLBIN_ in the future */
>> +#define CLOCK_MAX_NUM_CLOCKS 4
>> +#define OCL_CLKWIZ_STATUS_OFFSET 0x4
> OCL_CLKWIZ does not match the name of this file, change to something like XRT_CLOCK
Will change to XRT_CLOCK_
>> +#define OCL_CLKWIZ_STATUS_MASK 0xffff
>> +#define OCL_CLKWIZ_STATUS_MEASURE_START 0x1
>> +#define OCL_CLKWIZ_STATUS_MEASURE_DONE 0x2
>> +#define OCL_CLKWIZ_CONFIG_OFFSET(n) (0x200 + 4 * (n))
> This should be expanded to logical names of the registers.
>
> It's use below has a magic indexes.
Will change it.
>
>> +#define CLOCK_DEFAULT_EXPIRE_SECS 1
>> +
>> +#define CLOCK_ERR(clock, fmt, arg...) \
>> + xrt_err((clock)->pdev, fmt "\n", ##arg)
>> +#define CLOCK_WARN(clock, fmt, arg...) \
>> + xrt_warn((clock)->pdev, fmt "\n", ##arg)
>> +#define CLOCK_INFO(clock, fmt, arg...) \
>> + xrt_info((clock)->pdev, fmt "\n", ##arg)
>> +#define CLOCK_DBG(clock, fmt, arg...) \
>> + xrt_dbg((clock)->pdev, fmt "\n", ##arg)
>> +
>> +#define XRT_CLOCK "xrt_clock"
>> +
>> +struct clock {
>> + struct platform_device *pdev;
>> + void __iomem *clock_base;
>> + struct mutex clock_lock; /* clock dev lock */
>> +
>> + const char *clock_ep_name;
>> +};
>> +
>> +/*
>> + * Precomputed table with config0 and config2 register values together with
>> + * target frequency. The steps are approximately 5 MHz apart. Table is
>> + * generated by wiz.pl.
> where is wiz.pl ? include the script
wiz.pl may not be a accurate anymore. I will change the comment to "
Table is generated by platform creation tool. These are the only known
good values which have been validated on real hardware hence a fixed
table. Please see
https://www.xilinx.com/support/documentation/ip_documentation/clk_wiz/v6_0/pg065-clk-wiz.pdf
for programming guide.
>> + */
>> +static const struct xmgmt_ocl_clockwiz {
>> + /* target frequency */
>> + unsigned short ocl;
>> + /* config0 register */
>> + unsigned long config0;
> Should be u32
Will change.
>> + /* config2 register */
>> + unsigned int config2;
>> +} frequency_table[] = {
>> + {/*1275.000*/ 10.000, 0x02EE0C01, 0x0001F47F},
> Could clean up this table, to use spaces instead of tabs, move the comment out of branches
>
> /* freq */ { ocl, config0, config2 },
Sure
>
>> + {/*1575.000*/ 15.000, 0x02EE0F01, 0x00000069},
> ocl is short, the data if float, change the generator output an integer
Will fix.
>> + {/*1600.000*/ 20.000, 0x00001001, 0x00000050},
>> + {/*1600.000*/ 25.000, 0x00001001, 0x00000040},
>> + {/*1575.000*/ 30.000, 0x02EE0F01, 0x0001F434},
>> + {/*1575.000*/ 35.000, 0x02EE0F01, 0x0000002D},
>> + {/*1600.000*/ 40.000, 0x00001001, 0x00000028},
>> + {/*1575.000*/ 45.000, 0x02EE0F01, 0x00000023},
>> + {/*1600.000*/ 50.000, 0x00001001, 0x00000020},
>> + {/*1512.500*/ 55.000, 0x007D0F01, 0x0001F41B},
>> + {/*1575.000*/ 60.000, 0x02EE0F01, 0x0000FA1A},
>> + {/*1462.500*/ 65.000, 0x02710E01, 0x0001F416},
>> + {/*1575.000*/ 70.000, 0x02EE0F01, 0x0001F416},
>> + {/*1575.000*/ 75.000, 0x02EE0F01, 0x00000015},
>> + {/*1600.000*/ 80.000, 0x00001001, 0x00000014},
>> + {/*1487.500*/ 85.000, 0x036B0E01, 0x0001F411},
>> + {/*1575.000*/ 90.000, 0x02EE0F01, 0x0001F411},
>> + {/*1425.000*/ 95.000, 0x00FA0E01, 0x0000000F},
>> + {/*1600.000*/ 100.000, 0x00001001, 0x00000010},
>> + {/*1575.000*/ 105.000, 0x02EE0F01, 0x0000000F},
>> + {/*1512.500*/ 110.000, 0x007D0F01, 0x0002EE0D},
>> + {/*1437.500*/ 115.000, 0x01770E01, 0x0001F40C},
>> + {/*1575.000*/ 120.000, 0x02EE0F01, 0x00007D0D},
>> + {/*1562.500*/ 125.000, 0x02710F01, 0x0001F40C},
>> + {/*1462.500*/ 130.000, 0x02710E01, 0x0000FA0B},
>> + {/*1350.000*/ 135.000, 0x01F40D01, 0x0000000A},
>> + {/*1575.000*/ 140.000, 0x02EE0F01, 0x0000FA0B},
>> + {/*1450.000*/ 145.000, 0x01F40E01, 0x0000000A},
>> + {/*1575.000*/ 150.000, 0x02EE0F01, 0x0001F40A},
>> + {/*1550.000*/ 155.000, 0x01F40F01, 0x0000000A},
>> + {/*1600.000*/ 160.000, 0x00001001, 0x0000000A},
>> + {/*1237.500*/ 165.000, 0x01770C01, 0x0001F407},
>> + {/*1487.500*/ 170.000, 0x036B0E01, 0x0002EE08},
>> + {/*1575.000*/ 175.000, 0x02EE0F01, 0x00000009},
>> + {/*1575.000*/ 180.000, 0x02EE0F01, 0x0002EE08},
>> + {/*1387.500*/ 185.000, 0x036B0D01, 0x0001F407},
>> + {/*1425.000*/ 190.000, 0x00FA0E01, 0x0001F407},
>> + {/*1462.500*/ 195.000, 0x02710E01, 0x0001F407},
>> + {/*1600.000*/ 200.000, 0x00001001, 0x00000008},
>> + {/*1537.500*/ 205.000, 0x01770F01, 0x0001F407},
>> + {/*1575.000*/ 210.000, 0x02EE0F01, 0x0001F407},
>> + {/*1075.000*/ 215.000, 0x02EE0A01, 0x00000005},
>> + {/*1512.500*/ 220.000, 0x007D0F01, 0x00036B06},
>> + {/*1575.000*/ 225.000, 0x02EE0F01, 0x00000007},
>> + {/*1437.500*/ 230.000, 0x01770E01, 0x0000FA06},
>> + {/*1175.000*/ 235.000, 0x02EE0B01, 0x00000005},
>> + {/*1500.000*/ 240.000, 0x00000F01, 0x0000FA06},
>> + {/*1225.000*/ 245.000, 0x00FA0C01, 0x00000005},
>> + {/*1562.500*/ 250.000, 0x02710F01, 0x0000FA06},
>> + {/*1275.000*/ 255.000, 0x02EE0C01, 0x00000005},
>> + {/*1462.500*/ 260.000, 0x02710E01, 0x00027105},
>> + {/*1325.000*/ 265.000, 0x00FA0D01, 0x00000005},
>> + {/*1350.000*/ 270.000, 0x01F40D01, 0x00000005},
>> + {/*1512.500*/ 275.000, 0x007D0F01, 0x0001F405},
>> + {/*1575.000*/ 280.000, 0x02EE0F01, 0x00027105},
>> + {/*1425.000*/ 285.000, 0x00FA0E01, 0x00000005},
>> + {/*1450.000*/ 290.000, 0x01F40E01, 0x00000005},
>> + {/*1475.000*/ 295.000, 0x02EE0E01, 0x00000005},
>> + {/*1575.000*/ 300.000, 0x02EE0F01, 0x0000FA05},
>> + {/*1525.000*/ 305.000, 0x00FA0F01, 0x00000005},
>> + {/*1550.000*/ 310.000, 0x01F40F01, 0x00000005},
>> + {/*1575.000*/ 315.000, 0x02EE0F01, 0x00000005},
>> + {/*1600.000*/ 320.000, 0x00001001, 0x00000005},
>> + {/*1462.500*/ 325.000, 0x02710E01, 0x0001F404},
>> + {/*1237.500*/ 330.000, 0x01770C01, 0x0002EE03},
>> + {/*837.500*/ 335.000, 0x01770801, 0x0001F402},
>> + {/*1487.500*/ 340.000, 0x036B0E01, 0x00017704},
>> + {/*862.500*/ 345.000, 0x02710801, 0x0001F402},
>> + {/*1575.000*/ 350.000, 0x02EE0F01, 0x0001F404},
>> + {/*887.500*/ 355.000, 0x036B0801, 0x0001F402},
>> + {/*1575.000*/ 360.000, 0x02EE0F01, 0x00017704},
>> + {/*912.500*/ 365.000, 0x007D0901, 0x0001F402},
>> + {/*1387.500*/ 370.000, 0x036B0D01, 0x0002EE03},
>> + {/*1500.000*/ 375.000, 0x00000F01, 0x00000004},
>> + {/*1425.000*/ 380.000, 0x00FA0E01, 0x0002EE03},
>> + {/*962.500*/ 385.000, 0x02710901, 0x0001F402},
>> + {/*1462.500*/ 390.000, 0x02710E01, 0x0002EE03},
>> + {/*987.500*/ 395.000, 0x036B0901, 0x0001F402},
>> + {/*1600.000*/ 400.000, 0x00001001, 0x00000004},
>> + {/*1012.500*/ 405.000, 0x007D0A01, 0x0001F402},
>> + {/*1537.500*/ 410.000, 0x01770F01, 0x0002EE03},
>> + {/*1037.500*/ 415.000, 0x01770A01, 0x0001F402},
>> + {/*1575.000*/ 420.000, 0x02EE0F01, 0x0002EE03},
>> + {/*1487.500*/ 425.000, 0x036B0E01, 0x0001F403},
>> + {/*1075.000*/ 430.000, 0x02EE0A01, 0x0001F402},
>> + {/*1087.500*/ 435.000, 0x036B0A01, 0x0001F402},
>> + {/*1375.000*/ 440.000, 0x02EE0D01, 0x00007D03},
>> + {/*1112.500*/ 445.000, 0x007D0B01, 0x0001F402},
>> + {/*1575.000*/ 450.000, 0x02EE0F01, 0x0001F403},
>> + {/*1137.500*/ 455.000, 0x01770B01, 0x0001F402},
>> + {/*1437.500*/ 460.000, 0x01770E01, 0x00007D03},
>> + {/*1162.500*/ 465.000, 0x02710B01, 0x0001F402},
>> + {/*1175.000*/ 470.000, 0x02EE0B01, 0x0001F402},
>> + {/*1425.000*/ 475.000, 0x00FA0E01, 0x00000003},
>> + {/*1500.000*/ 480.000, 0x00000F01, 0x00007D03},
>> + {/*1212.500*/ 485.000, 0x007D0C01, 0x0001F402},
>> + {/*1225.000*/ 490.000, 0x00FA0C01, 0x0001F402},
>> + {/*1237.500*/ 495.000, 0x01770C01, 0x0001F402},
>> + {/*1562.500*/ 500.000, 0x02710F01, 0x00007D03},
>> + {/*1262.500*/ 505.000, 0x02710C01, 0x0001F402},
>> + {/*1275.000*/ 510.000, 0x02EE0C01, 0x0001F402},
>> + {/*1287.500*/ 515.000, 0x036B0C01, 0x0001F402},
>> + {/*1300.000*/ 520.000, 0x00000D01, 0x0001F402},
>> + {/*1575.000*/ 525.000, 0x02EE0F01, 0x00000003},
>> + {/*1325.000*/ 530.000, 0x00FA0D01, 0x0001F402},
>> + {/*1337.500*/ 535.000, 0x01770D01, 0x0001F402},
>> + {/*1350.000*/ 540.000, 0x01F40D01, 0x0001F402},
>> + {/*1362.500*/ 545.000, 0x02710D01, 0x0001F402},
>> + {/*1512.500*/ 550.000, 0x007D0F01, 0x0002EE02},
>> + {/*1387.500*/ 555.000, 0x036B0D01, 0x0001F402},
>> + {/*1400.000*/ 560.000, 0x00000E01, 0x0001F402},
>> + {/*1412.500*/ 565.000, 0x007D0E01, 0x0001F402},
>> + {/*1425.000*/ 570.000, 0x00FA0E01, 0x0001F402},
>> + {/*1437.500*/ 575.000, 0x01770E01, 0x0001F402},
>> + {/*1450.000*/ 580.000, 0x01F40E01, 0x0001F402},
>> + {/*1462.500*/ 585.000, 0x02710E01, 0x0001F402},
>> + {/*1475.000*/ 590.000, 0x02EE0E01, 0x0001F402},
>> + {/*1487.500*/ 595.000, 0x036B0E01, 0x0001F402},
>> + {/*1575.000*/ 600.000, 0x02EE0F01, 0x00027102},
>> + {/*1512.500*/ 605.000, 0x007D0F01, 0x0001F402},
>> + {/*1525.000*/ 610.000, 0x00FA0F01, 0x0001F402},
>> + {/*1537.500*/ 615.000, 0x01770F01, 0x0001F402},
>> + {/*1550.000*/ 620.000, 0x01F40F01, 0x0001F402},
>> + {/*1562.500*/ 625.000, 0x02710F01, 0x0001F402},
>> + {/*1575.000*/ 630.000, 0x02EE0F01, 0x0001F402},
>> + {/*1587.500*/ 635.000, 0x036B0F01, 0x0001F402},
>> + {/*1600.000*/ 640.000, 0x00001001, 0x0001F402},
>> + {/*1290.000*/ 645.000, 0x01F44005, 0x00000002},
>> + {/*1462.500*/ 650.000, 0x02710E01, 0x0000FA02}
>> +};
>> +
>> +static inline u32 reg_rd(struct clock *clock, u32 offset)
>> +{
>> + return ioread32(clock->clock_base + offset);
>> +}
>> +
>> +static inline void reg_wr(struct clock *clock, u32 val, u32 offset)
>> +{
>> + iowrite32(val, clock->clock_base + offset);
>> +}
>> +
>> +static u32 find_matching_freq_config(unsigned short freq,
>> + const struct xmgmt_ocl_clockwiz *table,
>> + int size)
>> +{
>> + u32 start = 0;
>> + u32 end = size - 1;
>> + u32 idx = size - 1;
> looks like you are doing a binary search, why not start idx at size / 2 ?
Will change.
>> +
>> + if (freq < table[0].ocl)
>> + return 0;
>> +
>> + if (freq > table[size - 1].ocl)
>> + return size - 1;
>> +
> What aren't these two conditions errors ?
If user specified the frequency out of scope, we will set to min/max
supported frequency.
>> + while (start < end) {
>> + if (freq == table[idx].ocl)
>> + break;
>> + if (freq < table[idx].ocl)
>> + end = idx;
>> + else
>> + start = idx + 1;
>> + idx = start + (end - start) / 2;
>> + }
>> + if (freq < table[idx].ocl)
>> + idx--;
>> +
>> + return idx;
>> +}
>> +
>> +static u32 find_matching_freq(u32 freq,
>> + const struct xmgmt_ocl_clockwiz *freq_table,
>> + int freq_table_size)
>> +{
>> + int idx = find_matching_freq_config(freq, freq_table, freq_table_size);
>> +
>> + return freq_table[idx].ocl;
>> +}
>> +
>> +static inline int clock_wiz_busy(struct clock *clock, int cycle, int interval)
>> +{
>> + u32 val = 0;
>> + int count;
>> +
>> + val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
>> + for (count = 0; val != 1 && count < cycle; count++) {
>> + mdelay(interval);
>> + val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
> if (val == 1)
>
> break;
>
> Is a bit easier to follow than putting another condition in the for()
Will change.
>
>> + }
>> + if (val != 1) {
>> + CLOCK_ERR(clock, "clockwiz is (%u) busy after %d ms",
>> + val, cycle * interval);
>> + return -ETIMEDOUT;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int get_freq(struct clock *clock, u16 *freq)
>> +{
>> +#define XCL_INPUT_FREQ 100
> ??
>
> If this a real parameter, it needs to go in the clock.h
Will remove this define.
>
>> + const u64 input = XCL_INPUT_FREQ;
>> + u32 val;
>> + u32 mul0, div0;
>> + u32 mul_frac0 = 0;
>> + u32 div1;
>> + u32 div_frac1 = 0;
>> +
>> + WARN_ON(!mutex_is_locked(&clock->clock_lock));
>> +
>> + val = reg_rd(clock, OCL_CLKWIZ_STATUS_OFFSET);
>> + if ((val & 0x1) == 0) {
>> + CLOCK_ERR(clock, "clockwiz is busy %x", val);
>> + *freq = 0;
>> + return -EBUSY;
>> + }
>> +
>> + val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(0));
> A general problem.
>
> The '0' is magic, convert this to a logically named #define or enum.
Will #define.
>
>> +
>> + div0 = val & 0xff;
>> + mul0 = (val & 0xff00) >> 8;
>> + if (val & BIT(26)) {
>> + mul_frac0 = val >> 16;
>> + mul_frac0 &= 0x3ff;
>> + }
>> +
>> + /*
>> + * Multiply both numerator (mul0) and the denominator (div0) with 1000
>> + * to account for fractional portion of multiplier
>> + */
>> + mul0 *= 1000;
>> + mul0 += mul_frac0;
>> + div0 *= 1000;
>> +
>> + val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(2));
>> +
>> + div1 = val & 0xff;
>> + if (val & BIT(18)) {
>> + div_frac1 = val >> 8;
>> + div_frac1 &= 0x3ff;
>> + }
>> +
>> + /*
>> + * Multiply both numerator (mul0) and the denominator (div1) with
>> + * 1000 to account for fractional portion of divider
>> + */
>> +
>> + div1 *= 1000;
>> + div1 += div_frac1;
>> + div0 *= div1;
>> + mul0 *= 1000;
>> + if (div0 == 0) {
>> + CLOCK_ERR(clock, "clockwiz 0 divider");
>> + return 0;
>> + }
>> +
>> + *freq = (u16)((input * mul0) / div0);
>> +
>> + return 0;
>> +}
>> +
>> +static int set_freq(struct clock *clock, u16 freq)
>> +{
>> + u32 config;
>> + int err;
>> + u32 idx = 0;
>> + u32 val;
>> +
>> + WARN_ON(!mutex_is_locked(&clock->clock_lock));
>> +
>> + idx = find_matching_freq_config(freq, frequency_table,
>> + ARRAY_SIZE(frequency_table));
>> +
>> + CLOCK_INFO(clock, "New: %d Mhz", freq);
>> + err = clock_wiz_busy(clock, 20, 50);
>> + if (err)
>> + return -EBUSY;
>> +
>> + config = frequency_table[idx].config0;
>> + reg_wr(clock, config, OCL_CLKWIZ_CONFIG_OFFSET(0));
>> +
>> + config = frequency_table[idx].config2;
>> + reg_wr(clock, config, OCL_CLKWIZ_CONFIG_OFFSET(2));
>> +
>> + mdelay(10);
>> + reg_wr(clock, 7, OCL_CLKWIZ_CONFIG_OFFSET(23));
>> +
>> + mdelay(1);
>> + reg_wr(clock, 2, OCL_CLKWIZ_CONFIG_OFFSET(23));
>> +
>> + CLOCK_INFO(clock, "clockwiz waiting for locked signal");
>> +
>> + err = clock_wiz_busy(clock, 100, 100);
>> + if (err) {
>> + CLOCK_ERR(clock, "clockwiz MMCM/PLL did not lock");
>> + /* restore */
>> + reg_wr(clock, 4, OCL_CLKWIZ_CONFIG_OFFSET(23));
>> + mdelay(10);
>> + reg_wr(clock, 0, OCL_CLKWIZ_CONFIG_OFFSET(23));
>> + return err;
>> + }
>> + val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(0));
>> + CLOCK_INFO(clock, "clockwiz CONFIG(0) 0x%x", val);
>> + val = reg_rd(clock, OCL_CLKWIZ_CONFIG_OFFSET(2));
>> + CLOCK_INFO(clock, "clockwiz CONFIG(2) 0x%x", val);
>> +
>> + return 0;
>> +}
>> +
>> +static int get_freq_counter(struct clock *clock, u32 *freq)
>> +{
>> + const void *cnter;
> variable name should be easy to read, use 'counter'
Will fix.
>> + struct platform_device *cnter_leaf;
>> + struct platform_device *pdev = clock->pdev;
>> + struct xrt_subdev_platdata *pdata = DEV_PDATA(clock->pdev);
>> + int err = xrt_md_get_prop(DEV(pdev), pdata->xsp_dtb,
>> + clock->clock_ep_name, NULL, XRT_MD_PROP_CLK_CNT, &cnter, NULL);
>> +
>> + WARN_ON(!mutex_is_locked(&clock->clock_lock));
>> +
>> + if (err) {
>> + xrt_err(pdev, "no counter specified");
>> + return err;
>> + }
>> +
>> + cnter_leaf = xleaf_get_leaf_by_epname(pdev, cnter);
>> + if (!cnter_leaf) {
>> + xrt_err(pdev, "can't find counter");
>> + return -ENOENT;
>> + }
>> +
>> + err = xleaf_ioctl(cnter_leaf, XRT_CLKFREQ_READ, freq);
>> + if (err)
>> + xrt_err(pdev, "can't read counter");
>> + xleaf_put_leaf(clock->pdev, cnter_leaf);
>> +
>> + return err;
>> +}
>> +
>> +static int clock_get_freq(struct clock *clock, u16 *freq, u32 *freq_cnter)
>> +{
>> + int err = 0;
>> +
>> + mutex_lock(&clock->clock_lock);
>> +
>> + if (err == 0 && freq)
>> + err = get_freq(clock, freq);
>> +
>> + if (err == 0 && freq_cnter)
>> + err = get_freq_counter(clock, freq_cnter);
>> +
>> + mutex_unlock(&clock->clock_lock);
>> + return err;
>> +}
>> +
>> +static int clock_set_freq(struct clock *clock, u16 freq)
>> +{
>> + int err;
> Why is this function needed ?
>
> Move the locking into set_freq.
Sure.
>
>> +
>> + mutex_lock(&clock->clock_lock);
>> + err = set_freq(clock, freq);
>> + mutex_unlock(&clock->clock_lock);
>> +
>> + return err;
>> +}
>> +
>> +static int clock_verify_freq(struct clock *clock)
>> +{
>> + int err = 0;
>> + u16 freq;
>> + u32 lookup_freq, clock_freq_counter, request_in_khz, tolerance;
>> +
>> + mutex_lock(&clock->clock_lock);
>> +
>> + err = get_freq(clock, &freq);
>> + if (err) {
>> + xrt_err(clock->pdev, "get freq failed, %d", err);
>> + goto end;
>> + }
>> +
>> + err = get_freq_counter(clock, &clock_freq_counter);
>> + if (err) {
>> + xrt_err(clock->pdev, "get freq counter failed, %d", err);
>> + goto end;
>> + }
>> +
>> + lookup_freq = find_matching_freq(freq, frequency_table,
>> + ARRAY_SIZE(frequency_table));
>> + request_in_khz = lookup_freq * 1000;
>> + tolerance = lookup_freq * 50;
>> + if (tolerance < abs(clock_freq_counter - request_in_khz)) {
>> + CLOCK_ERR(clock,
>> + "set clock(%s) failed, request %ukhz, actual %dkhz",
>> + clock->clock_ep_name, request_in_khz, clock_freq_counter);
>> + err = -EDOM;
>> + } else {
>> + CLOCK_INFO(clock, "verified clock (%s)", clock->clock_ep_name);
>> + }
>> +
>> +end:
>> + mutex_unlock(&clock->clock_lock);
>> + return err;
>> +}
>> +
>> +static int clock_init(struct clock *clock)
>> +{
>> + struct xrt_subdev_platdata *pdata = DEV_PDATA(clock->pdev);
>> + int err = 0;
>> + const u16 *freq;
>> +
>> + err = xrt_md_get_prop(DEV(clock->pdev), pdata->xsp_dtb,
>> + clock->clock_ep_name, NULL, XRT_MD_PROP_CLK_FREQ,
>> + (const void **)&freq, NULL);
>> + if (err) {
>> + xrt_info(clock->pdev, "no default freq");
>> + return 0;
>> + }
>> +
>> + mutex_lock(&clock->clock_lock);
>> + err = set_freq(clock, be16_to_cpu(*freq));
>> + mutex_unlock(&clock->clock_lock);
>> +
>> + return err;
>> +}
>> +
>> +static ssize_t freq_show(struct device *dev, struct device_attribute *attr, char *buf)
>> +{
>> + struct clock *clock = platform_get_drvdata(to_platform_device(dev));
>> + u16 freq = 0;
>> + ssize_t count;
>> +
>> + count = clock_get_freq(clock, &freq, NULL);
>> + if (count < 0)
>> + return count;
>> +
>> + count = snprintf(buf, 64, "%d\n", freq);
> %u ?
Sure.
Thanks,
Lizhi
>
> Tom
>
>> +
>> + return count;
>> +}
>> +static DEVICE_ATTR_RO(freq);
>> +
>> +static struct attribute *clock_attrs[] = {
>> + &dev_attr_freq.attr,
>> + NULL,
>> +};
>> +
>> +static struct attribute_group clock_attr_group = {
>> + .attrs = clock_attrs,
>> +};
>> +
>> +static int
>> +xrt_clock_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + struct clock *clock;
>> + int ret = 0;
>> +
>> + clock = platform_get_drvdata(pdev);
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + /* Does not handle any event. */
>> + break;
>> + case XRT_CLOCK_SET: {
>> + u16 freq = (u16)(uintptr_t)arg;
>> +
>> + ret = clock_set_freq(clock, freq);
>> + break;
>> + }
>> + case XRT_CLOCK_VERIFY: {
>> + ret = clock_verify_freq(clock);
>> + break;
>> + }
>> + case XRT_CLOCK_GET: {
>> + struct xrt_clock_ioctl_get *get =
>> + (struct xrt_clock_ioctl_get *)arg;
>> +
>> + ret = clock_get_freq(clock, &get->freq, &get->freq_cnter);
>> + break;
>> + }
>> + default:
>> + xrt_err(pdev, "unsupported cmd %d", cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +static int clock_remove(struct platform_device *pdev)
>> +{
>> + struct clock *clock;
>> +
>> + clock = platform_get_drvdata(pdev);
>> + if (!clock) {
>> + xrt_err(pdev, "driver data is NULL");
>> + return -EINVAL;
>> + }
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, clock);
>> +
>> + CLOCK_INFO(clock, "successfully removed Clock subdev");
>> + return 0;
>> +}
>> +
>> +static int clock_probe(struct platform_device *pdev)
>> +{
>> + struct clock *clock = NULL;
>> + struct resource *res;
>> + int ret;
>> +
>> + clock = devm_kzalloc(&pdev->dev, sizeof(*clock), GFP_KERNEL);
>> + if (!clock)
>> + return -ENOMEM;
>> +
>> + platform_set_drvdata(pdev, clock);
>> + clock->pdev = pdev;
>> + mutex_init(&clock->clock_lock);
>> +
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + clock->clock_base = ioremap(res->start, res->end - res->start + 1);
>> + if (!clock->clock_base) {
>> + CLOCK_ERR(clock, "map base %pR failed", res);
>> + ret = -EFAULT;
>> + goto failed;
>> + }
>> +
>> + clock->clock_ep_name = res->name;
>> +
>> + ret = clock_init(clock);
>> + if (ret)
>> + goto failed;
>> +
>> + ret = sysfs_create_group(&pdev->dev.kobj, &clock_attr_group);
>> + if (ret) {
>> + CLOCK_ERR(clock, "create clock attrs failed: %d", ret);
>> + goto failed;
>> + }
>> +
>> + CLOCK_INFO(clock, "successfully initialized Clock subdev");
>> +
>> + return 0;
>> +
>> +failed:
>> + clock_remove(pdev);
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_clock_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .regmap_name = "clkwiz" },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_clock_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_clock_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_clock_table[] = {
>> + { XRT_CLOCK, (kernel_ulong_t)&xrt_clock_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_clock_driver = {
>> + .driver = {
>> + .name = XRT_CLOCK,
>> + },
>> + .probe = clock_probe,
>> + .remove = clock_remove,
>> + .id_table = xrt_clock_table,
>> +};
>> +
>> +void clock_leaf_init_fini(bool init)
>> +{
>> + if (init)
>> + xleaf_register_driver(XRT_SUBDEV_CLOCK, &xrt_clock_driver, xrt_clock_endpoints);
>> + else
>> + xleaf_unregister_driver(XRT_SUBDEV_CLOCK);
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 15/18] fpga: xrt: clock frequence counter platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (13 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 14/18] fpga: xrt: clock " Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-03-06 15:25 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 16/18] fpga: xrt: DDR calibration " Lizhi Hou
` (4 subsequent siblings)
19 siblings, 1 reply; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Add clock frequence counter driver. Clock frequence counter is
a hardware function discovered by walking xclbin metadata. A platform
device node will be created for it. Other part of driver can read the
actual clock frequence through clock frequence counter driver.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xleaf/clkfreq.h | 23 +++
drivers/fpga/xrt/lib/xleaf/clkfreq.c | 221 +++++++++++++++++++++++
2 files changed, 244 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
diff --git a/drivers/fpga/xrt/include/xleaf/clkfreq.h b/drivers/fpga/xrt/include/xleaf/clkfreq.h
new file mode 100644
index 000000000000..29fc45e8a31b
--- /dev/null
+++ b/drivers/fpga/xrt/include/xleaf/clkfreq.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for XRT Clock Counter Leaf Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_CLKFREQ_H_
+#define _XRT_CLKFREQ_H_
+
+#include "xleaf.h"
+
+/*
+ * CLKFREQ driver IOCTL calls.
+ */
+enum xrt_clkfreq_ioctl_cmd {
+ XRT_CLKFREQ_READ = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+};
+
+#endif /* _XRT_CLKFREQ_H_ */
diff --git a/drivers/fpga/xrt/lib/xleaf/clkfreq.c b/drivers/fpga/xrt/lib/xleaf/clkfreq.c
new file mode 100644
index 000000000000..2482dd2cff47
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xleaf/clkfreq.c
@@ -0,0 +1,221 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Clock Frequency Counter Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou<Lizhi.Hou@xilinx.com>
+ */
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include "metadata.h"
+#include "xleaf.h"
+#include "xleaf/clkfreq.h"
+
+#define CLKFREQ_ERR(clkfreq, fmt, arg...) \
+ xrt_err((clkfreq)->pdev, fmt "\n", ##arg)
+#define CLKFREQ_WARN(clkfreq, fmt, arg...) \
+ xrt_warn((clkfreq)->pdev, fmt "\n", ##arg)
+#define CLKFREQ_INFO(clkfreq, fmt, arg...) \
+ xrt_info((clkfreq)->pdev, fmt "\n", ##arg)
+#define CLKFREQ_DBG(clkfreq, fmt, arg...) \
+ xrt_dbg((clkfreq)->pdev, fmt "\n", ##arg)
+
+#define XRT_CLKFREQ "xrt_clkfreq"
+
+#define OCL_CLKWIZ_STATUS_MASK 0xffff
+
+#define OCL_CLKWIZ_STATUS_MEASURE_START 0x1
+#define OCL_CLKWIZ_STATUS_MEASURE_DONE 0x2
+#define OCL_CLK_FREQ_COUNTER_OFFSET 0x8
+#define OCL_CLK_FREQ_V5_COUNTER_OFFSET 0x10
+#define OCL_CLK_FREQ_V5_CLK0_ENABLED 0x10000
+
+struct clkfreq {
+ struct platform_device *pdev;
+ void __iomem *clkfreq_base;
+ const char *clkfreq_ep_name;
+ struct mutex clkfreq_lock; /* clock counter dev lock */
+};
+
+static inline u32 reg_rd(struct clkfreq *clkfreq, u32 offset)
+{
+ return ioread32(clkfreq->clkfreq_base + offset);
+}
+
+static inline void reg_wr(struct clkfreq *clkfreq, u32 val, u32 offset)
+{
+ iowrite32(val, clkfreq->clkfreq_base + offset);
+}
+
+static u32 clkfreq_read(struct clkfreq *clkfreq)
+{
+ u32 freq = 0, status;
+ int times = 10;
+
+ mutex_lock(&clkfreq->clkfreq_lock);
+ reg_wr(clkfreq, OCL_CLKWIZ_STATUS_MEASURE_START, 0);
+ while (times != 0) {
+ status = reg_rd(clkfreq, 0);
+ if ((status & OCL_CLKWIZ_STATUS_MASK) ==
+ OCL_CLKWIZ_STATUS_MEASURE_DONE)
+ break;
+ mdelay(1);
+ times--;
+ };
+ if (times > 0) {
+ freq = (status & OCL_CLK_FREQ_V5_CLK0_ENABLED) ?
+ reg_rd(clkfreq, OCL_CLK_FREQ_V5_COUNTER_OFFSET) :
+ reg_rd(clkfreq, OCL_CLK_FREQ_COUNTER_OFFSET);
+ }
+ mutex_unlock(&clkfreq->clkfreq_lock);
+
+ return freq;
+}
+
+static ssize_t freq_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct clkfreq *clkfreq = platform_get_drvdata(to_platform_device(dev));
+ u32 freq;
+ ssize_t count;
+
+ freq = clkfreq_read(clkfreq);
+ count = snprintf(buf, 64, "%d\n", freq);
+
+ return count;
+}
+static DEVICE_ATTR_RO(freq);
+
+static struct attribute *clkfreq_attrs[] = {
+ &dev_attr_freq.attr,
+ NULL,
+};
+
+static struct attribute_group clkfreq_attr_group = {
+ .attrs = clkfreq_attrs,
+};
+
+static int
+xrt_clkfreq_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ struct clkfreq *clkfreq;
+ int ret = 0;
+
+ clkfreq = platform_get_drvdata(pdev);
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ /* Does not handle any event. */
+ break;
+ case XRT_CLKFREQ_READ: {
+ *(u32 *)arg = clkfreq_read(clkfreq);
+ break;
+ }
+ default:
+ xrt_err(pdev, "unsupported cmd %d", cmd);
+ return -EINVAL;
+ }
+
+ return ret;
+}
+
+static int clkfreq_remove(struct platform_device *pdev)
+{
+ struct clkfreq *clkfreq;
+
+ clkfreq = platform_get_drvdata(pdev);
+ if (!clkfreq) {
+ xrt_err(pdev, "driver data is NULL");
+ return -EINVAL;
+ }
+
+ platform_set_drvdata(pdev, NULL);
+ devm_kfree(&pdev->dev, clkfreq);
+
+ CLKFREQ_INFO(clkfreq, "successfully removed clkfreq subdev");
+ return 0;
+}
+
+static int clkfreq_probe(struct platform_device *pdev)
+{
+ struct clkfreq *clkfreq = NULL;
+ struct resource *res;
+ int ret;
+
+ clkfreq = devm_kzalloc(&pdev->dev, sizeof(*clkfreq), GFP_KERNEL);
+ if (!clkfreq)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, clkfreq);
+ clkfreq->pdev = pdev;
+ mutex_init(&clkfreq->clkfreq_lock);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ clkfreq->clkfreq_base = ioremap(res->start, res->end - res->start + 1);
+ if (!clkfreq->clkfreq_base) {
+ CLKFREQ_ERR(clkfreq, "map base %pR failed", res);
+ ret = -EFAULT;
+ goto failed;
+ }
+ clkfreq->clkfreq_ep_name = res->name;
+
+ ret = sysfs_create_group(&pdev->dev.kobj, &clkfreq_attr_group);
+ if (ret) {
+ CLKFREQ_ERR(clkfreq, "create clkfreq attrs failed: %d", ret);
+ goto failed;
+ }
+
+ CLKFREQ_INFO(clkfreq, "successfully initialized clkfreq subdev");
+
+ return 0;
+
+failed:
+ clkfreq_remove(pdev);
+ return ret;
+}
+
+static struct xrt_subdev_endpoints xrt_clkfreq_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ { .regmap_name = "freq_cnt" },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xrt_clkfreq_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_clkfreq_leaf_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_clkfreq_table[] = {
+ { XRT_CLKFREQ, (kernel_ulong_t)&xrt_clkfreq_data },
+ { },
+};
+
+static struct platform_driver xrt_clkfreq_driver = {
+ .driver = {
+ .name = XRT_CLKFREQ,
+ },
+ .probe = clkfreq_probe,
+ .remove = clkfreq_remove,
+ .id_table = xrt_clkfreq_table,
+};
+
+void clkfreq_leaf_init_fini(bool init)
+{
+ if (init) {
+ xleaf_register_driver(XRT_SUBDEV_CLKFREQ,
+ &xrt_clkfreq_driver, xrt_clkfreq_endpoints);
+ } else {
+ xleaf_unregister_driver(XRT_SUBDEV_CLKFREQ);
+ }
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 15/18] fpga: xrt: clock frequence counter platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 15/18] fpga: xrt: clock frequence counter " Lizhi Hou
@ 2021-03-06 15:25 ` Tom Rix
2021-03-12 23:43 ` Lizhi Hou
0 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-06 15:25 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Add clock frequence counter driver. Clock frequence counter is
> a hardware function discovered by walking xclbin metadata. A platform
> device node will be created for it. Other part of driver can read the
> actual clock frequence through clock frequence counter driver.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/clkfreq.h | 23 +++
> drivers/fpga/xrt/lib/xleaf/clkfreq.c | 221 +++++++++++++++++++++++
> 2 files changed, 244 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/clkfreq.h b/drivers/fpga/xrt/include/xleaf/clkfreq.h
> new file mode 100644
> index 000000000000..29fc45e8a31b
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/clkfreq.h
> @@ -0,0 +1,23 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT Clock Counter Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_CLKFREQ_H_
> +#define _XRT_CLKFREQ_H_
> +
> +#include "xleaf.h"
> +
> +/*
> + * CLKFREQ driver IOCTL calls.
> + */
> +enum xrt_clkfreq_ioctl_cmd {
> + XRT_CLKFREQ_READ = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> +};
> +
> +#endif /* _XRT_CLKFREQ_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/clkfreq.c b/drivers/fpga/xrt/lib/xleaf/clkfreq.c
> new file mode 100644
> index 000000000000..2482dd2cff47
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/clkfreq.c
> @@ -0,0 +1,221 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA Clock Frequency Counter Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/clkfreq.h"
> +
> +#define CLKFREQ_ERR(clkfreq, fmt, arg...) \
> + xrt_err((clkfreq)->pdev, fmt "\n", ##arg)
> +#define CLKFREQ_WARN(clkfreq, fmt, arg...) \
> + xrt_warn((clkfreq)->pdev, fmt "\n", ##arg)
> +#define CLKFREQ_INFO(clkfreq, fmt, arg...) \
> + xrt_info((clkfreq)->pdev, fmt "\n", ##arg)
> +#define CLKFREQ_DBG(clkfreq, fmt, arg...) \
> + xrt_dbg((clkfreq)->pdev, fmt "\n", ##arg)
> +
> +#define XRT_CLKFREQ "xrt_clkfreq"
> +
> +#define OCL_CLKWIZ_STATUS_MASK 0xffff
> +
> +#define OCL_CLKWIZ_STATUS_MEASURE_START 0x1
> +#define OCL_CLKWIZ_STATUS_MEASURE_DONE 0x2
> +#define OCL_CLK_FREQ_COUNTER_OFFSET 0x8
> +#define OCL_CLK_FREQ_V5_COUNTER_OFFSET 0x10
> +#define OCL_CLK_FREQ_V5_CLK0_ENABLED 0x10000
Similar to earlier, OCL -> XRT_CLKFREQ
Use regmap
> +
> +struct clkfreq {
> + struct platform_device *pdev;
> + void __iomem *clkfreq_base;
> + const char *clkfreq_ep_name;
> + struct mutex clkfreq_lock; /* clock counter dev lock */
> +};
> +
> +static inline u32 reg_rd(struct clkfreq *clkfreq, u32 offset)
> +{
> + return ioread32(clkfreq->clkfreq_base + offset);
> +}
> +
> +static inline void reg_wr(struct clkfreq *clkfreq, u32 val, u32 offset)
> +{
> + iowrite32(val, clkfreq->clkfreq_base + offset);
> +}
> +
> +static u32 clkfreq_read(struct clkfreq *clkfreq)
> +{
failure returns 0, it would be better if -EINVAL or similar was returned.
and u32 *freq added as a function parameter
> + u32 freq = 0, status;
> + int times = 10;
10 is a config parameter, should be a #define
> +
> + mutex_lock(&clkfreq->clkfreq_lock);
> + reg_wr(clkfreq, OCL_CLKWIZ_STATUS_MEASURE_START, 0);
> + while (times != 0) {
> + status = reg_rd(clkfreq, 0);
> + if ((status & OCL_CLKWIZ_STATUS_MASK) ==
> + OCL_CLKWIZ_STATUS_MEASURE_DONE)
> + break;
> + mdelay(1);
> + times--;
> + };
> + if (times > 0) {
I do not like tristate setting, convert to if-else
> + freq = (status & OCL_CLK_FREQ_V5_CLK0_ENABLED) ?
> + reg_rd(clkfreq, OCL_CLK_FREQ_V5_COUNTER_OFFSET) :
> + reg_rd(clkfreq, OCL_CLK_FREQ_COUNTER_OFFSET);
> + }
> + mutex_unlock(&clkfreq->clkfreq_lock);
> +
> + return freq;
> +}
> +
> +static ssize_t freq_show(struct device *dev, struct device_attribute *attr, char *buf)
> +{
> + struct clkfreq *clkfreq = platform_get_drvdata(to_platform_device(dev));
> + u32 freq;
> + ssize_t count;
> +
> + freq = clkfreq_read(clkfreq);
unchecked error
> + count = snprintf(buf, 64, "%d\n", freq);
%u
> +
> + return count;
> +}
> +static DEVICE_ATTR_RO(freq);
> +
> +static struct attribute *clkfreq_attrs[] = {
> + &dev_attr_freq.attr,
> + NULL,
> +};
> +
> +static struct attribute_group clkfreq_attr_group = {
> + .attrs = clkfreq_attrs,
> +};
> +
> +static int
> +xrt_clkfreq_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct clkfreq *clkfreq;
> + int ret = 0;
> +
> + clkfreq = platform_get_drvdata(pdev);
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + /* Does not handle any event. */
> + break;
> + case XRT_CLKFREQ_READ: {
brace not needed
> + *(u32 *)arg = clkfreq_read(clkfreq);
Unchecked error
> + break;
> + }
> + default:
> + xrt_err(pdev, "unsupported cmd %d", cmd);
> + return -EINVAL;
> + }
> +
> + return ret;
> +}
> +
> +static int clkfreq_remove(struct platform_device *pdev)
> +{
> + struct clkfreq *clkfreq;
> +
> + clkfreq = platform_get_drvdata(pdev);
> + if (!clkfreq) {
> + xrt_err(pdev, "driver data is NULL");
> + return -EINVAL;
> + }
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, clkfreq);
> +
> + CLKFREQ_INFO(clkfreq, "successfully removed clkfreq subdev");
> + return 0;
> +}
> +
> +static int clkfreq_probe(struct platform_device *pdev)
> +{
> + struct clkfreq *clkfreq = NULL;
> + struct resource *res;
> + int ret;
> +
> + clkfreq = devm_kzalloc(&pdev->dev, sizeof(*clkfreq), GFP_KERNEL);
> + if (!clkfreq)
> + return -ENOMEM;
> +
> + platform_set_drvdata(pdev, clkfreq);
> + clkfreq->pdev = pdev;
> + mutex_init(&clkfreq->clkfreq_lock);
> +
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + clkfreq->clkfreq_base = ioremap(res->start, res->end - res->start + 1);
> + if (!clkfreq->clkfreq_base) {
> + CLKFREQ_ERR(clkfreq, "map base %pR failed", res);
> + ret = -EFAULT;
> + goto failed;
> + }
> + clkfreq->clkfreq_ep_name = res->name;
> +
> + ret = sysfs_create_group(&pdev->dev.kobj, &clkfreq_attr_group);
> + if (ret) {
> + CLKFREQ_ERR(clkfreq, "create clkfreq attrs failed: %d", ret);
> + goto failed;
> + }
> +
> + CLKFREQ_INFO(clkfreq, "successfully initialized clkfreq subdev");
> +
> + return 0;
> +
> +failed:
> + clkfreq_remove(pdev);
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_clkfreq_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .regmap_name = "freq_cnt" },
name should be closer to filename, maybe 'clock_frequency' or 'clkfreq'
Tom
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_clkfreq_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_clkfreq_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_clkfreq_table[] = {
> + { XRT_CLKFREQ, (kernel_ulong_t)&xrt_clkfreq_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_clkfreq_driver = {
> + .driver = {
> + .name = XRT_CLKFREQ,
> + },
> + .probe = clkfreq_probe,
> + .remove = clkfreq_remove,
> + .id_table = xrt_clkfreq_table,
> +};
> +
> +void clkfreq_leaf_init_fini(bool init)
> +{
> + if (init) {
> + xleaf_register_driver(XRT_SUBDEV_CLKFREQ,
> + &xrt_clkfreq_driver, xrt_clkfreq_endpoints);
> + } else {
> + xleaf_unregister_driver(XRT_SUBDEV_CLKFREQ);
> + }
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 15/18] fpga: xrt: clock frequence counter platform driver
2021-03-06 15:25 ` Tom Rix
@ 2021-03-12 23:43 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-12 23:43 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 03/06/2021 07:25 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Add clock frequence counter driver. Clock frequence counter is
>> a hardware function discovered by walking xclbin metadata. A platform
>> device node will be created for it. Other part of driver can read the
>> actual clock frequence through clock frequence counter driver.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/clkfreq.h | 23 +++
>> drivers/fpga/xrt/lib/xleaf/clkfreq.c | 221 +++++++++++++++++++++++
>> 2 files changed, 244 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
>>
>> diff --git a/drivers/fpga/xrt/include/xleaf/clkfreq.h b/drivers/fpga/xrt/include/xleaf/clkfreq.h
>> new file mode 100644
>> index 000000000000..29fc45e8a31b
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/clkfreq.h
>> @@ -0,0 +1,23 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT Clock Counter Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_CLKFREQ_H_
>> +#define _XRT_CLKFREQ_H_
>> +
>> +#include "xleaf.h"
>> +
>> +/*
>> + * CLKFREQ driver IOCTL calls.
>> + */
>> +enum xrt_clkfreq_ioctl_cmd {
>> + XRT_CLKFREQ_READ = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> +};
>> +
>> +#endif /* _XRT_CLKFREQ_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/clkfreq.c b/drivers/fpga/xrt/lib/xleaf/clkfreq.c
>> new file mode 100644
>> index 000000000000..2482dd2cff47
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/clkfreq.c
>> @@ -0,0 +1,221 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA Clock Frequency Counter Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/io.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/clkfreq.h"
>> +
>> +#define CLKFREQ_ERR(clkfreq, fmt, arg...) \
>> + xrt_err((clkfreq)->pdev, fmt "\n", ##arg)
>> +#define CLKFREQ_WARN(clkfreq, fmt, arg...) \
>> + xrt_warn((clkfreq)->pdev, fmt "\n", ##arg)
>> +#define CLKFREQ_INFO(clkfreq, fmt, arg...) \
>> + xrt_info((clkfreq)->pdev, fmt "\n", ##arg)
>> +#define CLKFREQ_DBG(clkfreq, fmt, arg...) \
>> + xrt_dbg((clkfreq)->pdev, fmt "\n", ##arg)
>> +
>> +#define XRT_CLKFREQ "xrt_clkfreq"
>> +
>> +#define OCL_CLKWIZ_STATUS_MASK 0xffff
>> +
>> +#define OCL_CLKWIZ_STATUS_MEASURE_START 0x1
>> +#define OCL_CLKWIZ_STATUS_MEASURE_DONE 0x2
>> +#define OCL_CLK_FREQ_COUNTER_OFFSET 0x8
>> +#define OCL_CLK_FREQ_V5_COUNTER_OFFSET 0x10
>> +#define OCL_CLK_FREQ_V5_CLK0_ENABLED 0x10000
> Similar to earlier, OCL -> XRT_CLKFREQ
>
> Use regmap
Will change this.
>
>> +
>> +struct clkfreq {
>> + struct platform_device *pdev;
>> + void __iomem *clkfreq_base;
>> + const char *clkfreq_ep_name;
>> + struct mutex clkfreq_lock; /* clock counter dev lock */
>> +};
>> +
>> +static inline u32 reg_rd(struct clkfreq *clkfreq, u32 offset)
>> +{
>> + return ioread32(clkfreq->clkfreq_base + offset);
>> +}
>> +
>> +static inline void reg_wr(struct clkfreq *clkfreq, u32 val, u32 offset)
>> +{
>> + iowrite32(val, clkfreq->clkfreq_base + offset);
>> +}
>> +
>> +static u32 clkfreq_read(struct clkfreq *clkfreq)
>> +{
> failure returns 0, it would be better if -EINVAL or similar was returned.
>
> and u32 *freq added as a function parameter
Will change this.
>
>> + u32 freq = 0, status;
>> + int times = 10;
> 10 is a config parameter, should be a #define
Sure.
>> +
>> + mutex_lock(&clkfreq->clkfreq_lock);
>> + reg_wr(clkfreq, OCL_CLKWIZ_STATUS_MEASURE_START, 0);
>> + while (times != 0) {
>> + status = reg_rd(clkfreq, 0);
>> + if ((status & OCL_CLKWIZ_STATUS_MASK) ==
>> + OCL_CLKWIZ_STATUS_MEASURE_DONE)
>> + break;
>> + mdelay(1);
>> + times--;
>> + };
>> + if (times > 0) {
> I do not like tristate setting, convert to if-else
Will change this.
>> + freq = (status & OCL_CLK_FREQ_V5_CLK0_ENABLED) ?
>> + reg_rd(clkfreq, OCL_CLK_FREQ_V5_COUNTER_OFFSET) :
>> + reg_rd(clkfreq, OCL_CLK_FREQ_COUNTER_OFFSET);
>> + }
>> + mutex_unlock(&clkfreq->clkfreq_lock);
>> +
>> + return freq;
>> +}
>> +
>> +static ssize_t freq_show(struct device *dev, struct device_attribute *attr, char *buf)
>> +{
>> + struct clkfreq *clkfreq = platform_get_drvdata(to_platform_device(dev));
>> + u32 freq;
>> + ssize_t count;
>> +
>> + freq = clkfreq_read(clkfreq);
> unchecked error
Will add check.
>> + count = snprintf(buf, 64, "%d\n", freq);
> %u
Sure.
>> +
>> + return count;
>> +}
>> +static DEVICE_ATTR_RO(freq);
>> +
>> +static struct attribute *clkfreq_attrs[] = {
>> + &dev_attr_freq.attr,
>> + NULL,
>> +};
>> +
>> +static struct attribute_group clkfreq_attr_group = {
>> + .attrs = clkfreq_attrs,
>> +};
>> +
>> +static int
>> +xrt_clkfreq_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + struct clkfreq *clkfreq;
>> + int ret = 0;
>> +
>> + clkfreq = platform_get_drvdata(pdev);
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + /* Does not handle any event. */
>> + break;
>> + case XRT_CLKFREQ_READ: {
> brace not needed
Will remove.
>> + *(u32 *)arg = clkfreq_read(clkfreq);
> Unchecked error
Will add check.
>> + break;
>> + }
>> + default:
>> + xrt_err(pdev, "unsupported cmd %d", cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +static int clkfreq_remove(struct platform_device *pdev)
>> +{
>> + struct clkfreq *clkfreq;
>> +
>> + clkfreq = platform_get_drvdata(pdev);
>> + if (!clkfreq) {
>> + xrt_err(pdev, "driver data is NULL");
>> + return -EINVAL;
>> + }
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, clkfreq);
>> +
>> + CLKFREQ_INFO(clkfreq, "successfully removed clkfreq subdev");
>> + return 0;
>> +}
>> +
>> +static int clkfreq_probe(struct platform_device *pdev)
>> +{
>> + struct clkfreq *clkfreq = NULL;
>> + struct resource *res;
>> + int ret;
>> +
>> + clkfreq = devm_kzalloc(&pdev->dev, sizeof(*clkfreq), GFP_KERNEL);
>> + if (!clkfreq)
>> + return -ENOMEM;
>> +
>> + platform_set_drvdata(pdev, clkfreq);
>> + clkfreq->pdev = pdev;
>> + mutex_init(&clkfreq->clkfreq_lock);
>> +
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + clkfreq->clkfreq_base = ioremap(res->start, res->end - res->start + 1);
>> + if (!clkfreq->clkfreq_base) {
>> + CLKFREQ_ERR(clkfreq, "map base %pR failed", res);
>> + ret = -EFAULT;
>> + goto failed;
>> + }
>> + clkfreq->clkfreq_ep_name = res->name;
>> +
>> + ret = sysfs_create_group(&pdev->dev.kobj, &clkfreq_attr_group);
>> + if (ret) {
>> + CLKFREQ_ERR(clkfreq, "create clkfreq attrs failed: %d", ret);
>> + goto failed;
>> + }
>> +
>> + CLKFREQ_INFO(clkfreq, "successfully initialized clkfreq subdev");
>> +
>> + return 0;
>> +
>> +failed:
>> + clkfreq_remove(pdev);
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_clkfreq_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .regmap_name = "freq_cnt" },
> name should be closer to filename, maybe 'clock_frequency' or 'clkfreq'
'freq_cnt' is from firmware metadata. I will add #define in metadata.h.
Thanks,
Lizhi
>
> Tom
>
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_clkfreq_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_clkfreq_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_clkfreq_table[] = {
>> + { XRT_CLKFREQ, (kernel_ulong_t)&xrt_clkfreq_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_clkfreq_driver = {
>> + .driver = {
>> + .name = XRT_CLKFREQ,
>> + },
>> + .probe = clkfreq_probe,
>> + .remove = clkfreq_remove,
>> + .id_table = xrt_clkfreq_table,
>> +};
>> +
>> +void clkfreq_leaf_init_fini(bool init)
>> +{
>> + if (init) {
>> + xleaf_register_driver(XRT_SUBDEV_CLKFREQ,
>> + &xrt_clkfreq_driver, xrt_clkfreq_endpoints);
>> + } else {
>> + xleaf_unregister_driver(XRT_SUBDEV_CLKFREQ);
>> + }
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 16/18] fpga: xrt: DDR calibration platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (14 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 15/18] fpga: xrt: clock frequence counter " Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-21 20:21 ` Moritz Fischer
2021-03-06 15:34 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation " Lizhi Hou
` (3 subsequent siblings)
19 siblings, 2 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Add DDR calibration driver. DDR calibration is a hardware function
discovered by walking firmware metadata. A platform device node will
be created for it. Hardware provides DDR calibration status through
this function.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xleaf/calib.h | 30 ++++
drivers/fpga/xrt/lib/xleaf/calib.c | 226 +++++++++++++++++++++++++
2 files changed, 256 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
diff --git a/drivers/fpga/xrt/include/xleaf/calib.h b/drivers/fpga/xrt/include/xleaf/calib.h
new file mode 100644
index 000000000000..f8aba4594c58
--- /dev/null
+++ b/drivers/fpga/xrt/include/xleaf/calib.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for XRT DDR Calibration Leaf Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_CALIB_H_
+#define _XRT_CALIB_H_
+
+#include "xleaf.h"
+#include <linux/xrt/xclbin.h>
+
+/*
+ * Memory calibration driver IOCTL calls.
+ */
+enum xrt_calib_results {
+ XRT_CALIB_UNKNOWN,
+ XRT_CALIB_SUCCEEDED,
+ XRT_CALIB_FAILED,
+};
+
+enum xrt_calib_ioctl_cmd {
+ XRT_CALIB_RESULT = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+};
+
+#endif /* _XRT_CALIB_H_ */
diff --git a/drivers/fpga/xrt/lib/xleaf/calib.c b/drivers/fpga/xrt/lib/xleaf/calib.c
new file mode 100644
index 000000000000..fbb813636e76
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xleaf/calib.c
@@ -0,0 +1,226 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA memory calibration driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * memory calibration
+ *
+ * Authors:
+ * Lizhi Hou<Lizhi.Hou@xilinx.com>
+ */
+#include <linux/delay.h>
+#include "xclbin-helper.h"
+#include "metadata.h"
+#include "xleaf/calib.h"
+
+#define XRT_CALIB "xrt_calib"
+
+struct calib_cache {
+ struct list_head link;
+ const char *ep_name;
+ char *data;
+ u32 data_size;
+};
+
+struct calib {
+ struct platform_device *pdev;
+ void *calib_base;
+ struct mutex lock; /* calibration dev lock */
+ struct list_head cache_list;
+ u32 cache_num;
+ enum xrt_calib_results result;
+};
+
+#define CALIB_DONE(calib) \
+ (ioread32((calib)->calib_base) & BIT(0))
+
+static void calib_cache_clean_nolock(struct calib *calib)
+{
+ struct calib_cache *cache, *temp;
+
+ list_for_each_entry_safe(cache, temp, &calib->cache_list, link) {
+ vfree(cache->data);
+ list_del(&cache->link);
+ vfree(cache);
+ }
+ calib->cache_num = 0;
+}
+
+static void calib_cache_clean(struct calib *calib)
+{
+ mutex_lock(&calib->lock);
+ calib_cache_clean_nolock(calib);
+ mutex_unlock(&calib->lock);
+}
+
+static int calib_srsr(struct calib *calib, struct platform_device *srsr_leaf)
+{
+ return -EOPNOTSUPP;
+}
+
+static int calib_calibration(struct calib *calib)
+{
+ int i;
+
+ for (i = 0; i < 20; i++) {
+ if (CALIB_DONE(calib))
+ break;
+ msleep(500);
+ }
+
+ if (i == 20) {
+ xrt_err(calib->pdev,
+ "MIG calibration timeout after bitstream download");
+ return -ETIMEDOUT;
+ }
+
+ xrt_info(calib->pdev, "took %dms", i * 500);
+ return 0;
+}
+
+static void xrt_calib_event_cb(struct platform_device *pdev, void *arg)
+{
+ struct calib *calib = platform_get_drvdata(pdev);
+ struct xrt_event *evt = (struct xrt_event *)arg;
+ enum xrt_events e = evt->xe_evt;
+ enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
+ int instance = evt->xe_subdev.xevt_subdev_instance;
+ struct platform_device *leaf;
+ int ret;
+
+ switch (e) {
+ case XRT_EVENT_POST_CREATION: {
+ if (id == XRT_SUBDEV_SRSR) {
+ leaf = xleaf_get_leaf_by_id(pdev,
+ XRT_SUBDEV_SRSR,
+ instance);
+ if (!leaf) {
+ xrt_err(pdev, "does not get SRSR subdev");
+ return;
+ }
+ ret = calib_srsr(calib, leaf);
+ xleaf_put_leaf(pdev, leaf);
+ calib->result =
+ ret ? XRT_CALIB_FAILED : XRT_CALIB_SUCCEEDED;
+ } else if (id == XRT_SUBDEV_UCS) {
+ ret = calib_calibration(calib);
+ calib->result =
+ ret ? XRT_CALIB_FAILED : XRT_CALIB_SUCCEEDED;
+ }
+ break;
+ }
+ default:
+ break;
+ }
+}
+
+static int xrt_calib_remove(struct platform_device *pdev)
+{
+ struct calib *calib = platform_get_drvdata(pdev);
+
+ calib_cache_clean(calib);
+
+ if (calib->calib_base)
+ iounmap(calib->calib_base);
+
+ platform_set_drvdata(pdev, NULL);
+ devm_kfree(&pdev->dev, calib);
+
+ return 0;
+}
+
+static int xrt_calib_probe(struct platform_device *pdev)
+{
+ struct calib *calib;
+ struct resource *res;
+ int err = 0;
+
+ calib = devm_kzalloc(&pdev->dev, sizeof(*calib), GFP_KERNEL);
+ if (!calib)
+ return -ENOMEM;
+
+ calib->pdev = pdev;
+ platform_set_drvdata(pdev, calib);
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res)
+ goto failed;
+
+ calib->calib_base = ioremap(res->start, res->end - res->start + 1);
+ if (!calib->calib_base) {
+ err = -EIO;
+ xrt_err(pdev, "Map iomem failed");
+ goto failed;
+ }
+
+ mutex_init(&calib->lock);
+ INIT_LIST_HEAD(&calib->cache_list);
+
+ return 0;
+
+failed:
+ xrt_calib_remove(pdev);
+ return err;
+}
+
+static int
+xrt_calib_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ struct calib *calib = platform_get_drvdata(pdev);
+ int ret = 0;
+
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ xrt_calib_event_cb(pdev, arg);
+ break;
+ case XRT_CALIB_RESULT: {
+ enum xrt_calib_results *r = (enum xrt_calib_results *)arg;
+ *r = calib->result;
+ break;
+ }
+ default:
+ xrt_err(pdev, "unsupported cmd %d", cmd);
+ ret = -EINVAL;
+ }
+ return ret;
+}
+
+static struct xrt_subdev_endpoints xrt_calib_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ { .ep_name = XRT_MD_NODE_DDR_CALIB },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xrt_calib_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_calib_leaf_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_calib_table[] = {
+ { XRT_CALIB, (kernel_ulong_t)&xrt_calib_data },
+ { },
+};
+
+static struct platform_driver xrt_calib_driver = {
+ .driver = {
+ .name = XRT_CALIB,
+ },
+ .probe = xrt_calib_probe,
+ .remove = xrt_calib_remove,
+ .id_table = xrt_calib_table,
+};
+
+void calib_leaf_init_fini(bool init)
+{
+ if (init)
+ xleaf_register_driver(XRT_SUBDEV_CALIB, &xrt_calib_driver, xrt_calib_endpoints);
+ else
+ xleaf_unregister_driver(XRT_SUBDEV_CALIB);
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 16/18] fpga: xrt: DDR calibration platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 16/18] fpga: xrt: DDR calibration " Lizhi Hou
@ 2021-02-21 20:21 ` Moritz Fischer
2021-03-06 15:34 ` Tom Rix
1 sibling, 0 replies; 87+ messages in thread
From: Moritz Fischer @ 2021-02-21 20:21 UTC (permalink / raw)
To: Lizhi Hou
Cc: linux-kernel, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, mdf, robh, Max Zhen
Lizhi,
On Wed, Feb 17, 2021 at 10:40:17PM -0800, Lizhi Hou wrote:
> Add DDR calibration driver. DDR calibration is a hardware function
> discovered by walking firmware metadata. A platform device node will
> be created for it. Hardware provides DDR calibration status through
> this function.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/calib.h | 30 ++++
> drivers/fpga/xrt/lib/xleaf/calib.c | 226 +++++++++++++++++++++++++
> 2 files changed, 256 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/calib.h b/drivers/fpga/xrt/include/xleaf/calib.h
> new file mode 100644
> index 000000000000..f8aba4594c58
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/calib.h
> @@ -0,0 +1,30 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT DDR Calibration Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_CALIB_H_
> +#define _XRT_CALIB_H_
> +
> +#include "xleaf.h"
> +#include <linux/xrt/xclbin.h>
> +
> +/*
> + * Memory calibration driver IOCTL calls.
> + */
> +enum xrt_calib_results {
> + XRT_CALIB_UNKNOWN,
> + XRT_CALIB_SUCCEEDED,
> + XRT_CALIB_FAILED,
> +};
> +
> +enum xrt_calib_ioctl_cmd {
> + XRT_CALIB_RESULT = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> +};
Could this be using sysfs instead of an IOCTL?
> +
> +#endif /* _XRT_CALIB_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/calib.c b/drivers/fpga/xrt/lib/xleaf/calib.c
> new file mode 100644
> index 000000000000..fbb813636e76
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/calib.c
> @@ -0,0 +1,226 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA memory calibration driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * memory calibration
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + */
> +#include <linux/delay.h>
> +#include "xclbin-helper.h"
> +#include "metadata.h"
> +#include "xleaf/calib.h"
> +
> +#define XRT_CALIB "xrt_calib"
> +
> +struct calib_cache {
> + struct list_head link;
> + const char *ep_name;
> + char *data;
> + u32 data_size;
> +};
> +
> +struct calib {
> + struct platform_device *pdev;
> + void *calib_base;
> + struct mutex lock; /* calibration dev lock */
> + struct list_head cache_list;
> + u32 cache_num;
> + enum xrt_calib_results result;
> +};
> +
> +#define CALIB_DONE(calib) \
> + (ioread32((calib)->calib_base) & BIT(0))
> +
> +static void calib_cache_clean_nolock(struct calib *calib)
> +{
> + struct calib_cache *cache, *temp;
> +
> + list_for_each_entry_safe(cache, temp, &calib->cache_list, link) {
> + vfree(cache->data);
> + list_del(&cache->link);
> + vfree(cache);
> + }
> + calib->cache_num = 0;
> +}
> +
> +static void calib_cache_clean(struct calib *calib)
> +{
> + mutex_lock(&calib->lock);
> + calib_cache_clean_nolock(calib);
> + mutex_unlock(&calib->lock);
> +}
> +
> +static int calib_srsr(struct calib *calib, struct platform_device *srsr_leaf)
> +{
> + return -EOPNOTSUPP;
> +}
> +
> +static int calib_calibration(struct calib *calib)
> +{
> + int i;
> +
> + for (i = 0; i < 20; i++) {
> + if (CALIB_DONE(calib))
> + break;
> + msleep(500);
> + }
> +
> + if (i == 20) {
> + xrt_err(calib->pdev,
> + "MIG calibration timeout after bitstream download");
> + return -ETIMEDOUT;
> + }
> +
> + xrt_info(calib->pdev, "took %dms", i * 500);
> + return 0;
> +}
> +
> +static void xrt_calib_event_cb(struct platform_device *pdev, void *arg)
> +{
> + struct calib *calib = platform_get_drvdata(pdev);
> + struct xrt_event *evt = (struct xrt_event *)arg;
> + enum xrt_events e = evt->xe_evt;
> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
> + int instance = evt->xe_subdev.xevt_subdev_instance;
> + struct platform_device *leaf;
> + int ret;
> +
> + switch (e) {
> + case XRT_EVENT_POST_CREATION: {
> + if (id == XRT_SUBDEV_SRSR) {
> + leaf = xleaf_get_leaf_by_id(pdev,
> + XRT_SUBDEV_SRSR,
> + instance);
> + if (!leaf) {
> + xrt_err(pdev, "does not get SRSR subdev");
> + return;
> + }
> + ret = calib_srsr(calib, leaf);
> + xleaf_put_leaf(pdev, leaf);
> + calib->result =
> + ret ? XRT_CALIB_FAILED : XRT_CALIB_SUCCEEDED;
> + } else if (id == XRT_SUBDEV_UCS) {
> + ret = calib_calibration(calib);
> + calib->result =
> + ret ? XRT_CALIB_FAILED : XRT_CALIB_SUCCEEDED;
> + }
> + break;
> + }
> + default:
> + break;
> + }
> +}
> +
> +static int xrt_calib_remove(struct platform_device *pdev)
> +{
> + struct calib *calib = platform_get_drvdata(pdev);
> +
> + calib_cache_clean(calib);
> +
> + if (calib->calib_base)
> + iounmap(calib->calib_base);
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, calib);
> +
> + return 0;
> +}
> +
> +static int xrt_calib_probe(struct platform_device *pdev)
> +{
> + struct calib *calib;
> + struct resource *res;
> + int err = 0;
> +
> + calib = devm_kzalloc(&pdev->dev, sizeof(*calib), GFP_KERNEL);
> + if (!calib)
> + return -ENOMEM;
> +
> + calib->pdev = pdev;
> + platform_set_drvdata(pdev, calib);
> +
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + if (!res)
> + goto failed;
> +
> + calib->calib_base = ioremap(res->start, res->end - res->start + 1);
> + if (!calib->calib_base) {
> + err = -EIO;
> + xrt_err(pdev, "Map iomem failed");
> + goto failed;
> + }
> +
> + mutex_init(&calib->lock);
> + INIT_LIST_HEAD(&calib->cache_list);
> +
> + return 0;
> +
> +failed:
> + xrt_calib_remove(pdev);
> + return err;
> +}
This is extremly weird imho, you have a platform driver that essentially
does not register with a subsystem or anything else.
Do you intend to use this from the outside through
platform_get_drvdata(), or how does this tie together with the rest?
> +
> +static int
> +xrt_calib_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct calib *calib = platform_get_drvdata(pdev);
> + int ret = 0;
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + xrt_calib_event_cb(pdev, arg);
> + break;
> + case XRT_CALIB_RESULT: {
> + enum xrt_calib_results *r = (enum xrt_calib_results *)arg;
> + *r = calib->result;
> + break;
> + }
> + default:
> + xrt_err(pdev, "unsupported cmd %d", cmd);
> + ret = -EINVAL;
> + }
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_calib_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = XRT_MD_NODE_DDR_CALIB },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_calib_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_calib_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_calib_table[] = {
> + { XRT_CALIB, (kernel_ulong_t)&xrt_calib_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_calib_driver = {
> + .driver = {
> + .name = XRT_CALIB,
> + },
> + .probe = xrt_calib_probe,
> + .remove = xrt_calib_remove,
> + .id_table = xrt_calib_table,
> +};
> +
> +void calib_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_CALIB, &xrt_calib_driver, xrt_calib_endpoints);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_CALIB);
> +}
> --
> 2.18.4
>
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 16/18] fpga: xrt: DDR calibration platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 16/18] fpga: xrt: DDR calibration " Lizhi Hou
2021-02-21 20:21 ` Moritz Fischer
@ 2021-03-06 15:34 ` Tom Rix
2021-03-13 0:45 ` Lizhi Hou
1 sibling, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-06 15:34 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Add DDR calibration driver. DDR calibration is a hardware function
> discovered by walking firmware metadata. A platform device node will
> be created for it. Hardware provides DDR calibration status through
> this function.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/calib.h | 30 ++++
> drivers/fpga/xrt/lib/xleaf/calib.c | 226 +++++++++++++++++++++++++
> 2 files changed, 256 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
calib is not descriptive, change filename to ddr_calibration
>
> diff --git a/drivers/fpga/xrt/include/xleaf/calib.h b/drivers/fpga/xrt/include/xleaf/calib.h
> new file mode 100644
> index 000000000000..f8aba4594c58
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/calib.h
> @@ -0,0 +1,30 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT DDR Calibration Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <maxz@xilinx.com>
> + */
> +
> +#ifndef _XRT_CALIB_H_
> +#define _XRT_CALIB_H_
> +
> +#include "xleaf.h"
> +#include <linux/xrt/xclbin.h>
> +
> +/*
> + * Memory calibration driver IOCTL calls.
> + */
> +enum xrt_calib_results {
> + XRT_CALIB_UNKNOWN,
Initialize ?
> + XRT_CALIB_SUCCEEDED,
> + XRT_CALIB_FAILED,
> +};
> +
> +enum xrt_calib_ioctl_cmd {
> + XRT_CALIB_RESULT = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> +};
> +
> +#endif /* _XRT_CALIB_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/calib.c b/drivers/fpga/xrt/lib/xleaf/calib.c
> new file mode 100644
> index 000000000000..fbb813636e76
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/calib.c
> @@ -0,0 +1,226 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA memory calibration driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * memory calibration
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + */
> +#include <linux/delay.h>
> +#include "xclbin-helper.h"
> +#include "metadata.h"
> +#include "xleaf/calib.h"
> +
> +#define XRT_CALIB "xrt_calib"
> +
> +struct calib_cache {
> + struct list_head link;
> + const char *ep_name;
> + char *data;
> + u32 data_size;
> +};
> +
> +struct calib {
> + struct platform_device *pdev;
> + void *calib_base;
> + struct mutex lock; /* calibration dev lock */
> + struct list_head cache_list;
> + u32 cache_num;
> + enum xrt_calib_results result;
> +};
> +
> +#define CALIB_DONE(calib) \
> + (ioread32((calib)->calib_base) & BIT(0))
> +
> +static void calib_cache_clean_nolock(struct calib *calib)
> +{
> + struct calib_cache *cache, *temp;
> +
> + list_for_each_entry_safe(cache, temp, &calib->cache_list, link) {
> + vfree(cache->data);
> + list_del(&cache->link);
> + vfree(cache);
> + }
> + calib->cache_num = 0;
> +}
> +
> +static void calib_cache_clean(struct calib *calib)
> +{
> + mutex_lock(&calib->lock);
> + calib_cache_clean_nolock(calib);
No lock functions (i believe) should be prefixed with '__'
> + mutex_unlock(&calib->lock);
> +}
> +
> +static int calib_srsr(struct calib *calib, struct platform_device *srsr_leaf)
what is srsr ?
Why a noop function ?
> +{
> + return -EOPNOTSUPP;
> +}
> +
> +static int calib_calibration(struct calib *calib)
> +{
> + int i;
> +
> + for (i = 0; i < 20; i++) {
20 is a config parameter so should have a #define
There a couple of busy wait blocks in xrt/ some count up, some count down.
It would be good if they were consistent.
> + if (CALIB_DONE(calib))
> + break;
> + msleep(500);
500 is another config
Tom
> + }
> +
> + if (i == 20) {
> + xrt_err(calib->pdev,
> + "MIG calibration timeout after bitstream download");
> + return -ETIMEDOUT;
> + }
> +
> + xrt_info(calib->pdev, "took %dms", i * 500);
> + return 0;
> +}
> +
> +static void xrt_calib_event_cb(struct platform_device *pdev, void *arg)
> +{
> + struct calib *calib = platform_get_drvdata(pdev);
> + struct xrt_event *evt = (struct xrt_event *)arg;
> + enum xrt_events e = evt->xe_evt;
> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
> + int instance = evt->xe_subdev.xevt_subdev_instance;
> + struct platform_device *leaf;
> + int ret;
> +
> + switch (e) {
> + case XRT_EVENT_POST_CREATION: {
> + if (id == XRT_SUBDEV_SRSR) {
> + leaf = xleaf_get_leaf_by_id(pdev,
> + XRT_SUBDEV_SRSR,
> + instance);
> + if (!leaf) {
> + xrt_err(pdev, "does not get SRSR subdev");
> + return;
> + }
> + ret = calib_srsr(calib, leaf);
> + xleaf_put_leaf(pdev, leaf);
> + calib->result =
> + ret ? XRT_CALIB_FAILED : XRT_CALIB_SUCCEEDED;
> + } else if (id == XRT_SUBDEV_UCS) {
> + ret = calib_calibration(calib);
> + calib->result =
> + ret ? XRT_CALIB_FAILED : XRT_CALIB_SUCCEEDED;
> + }
> + break;
> + }
> + default:
> + break;
> + }
> +}
> +
> +static int xrt_calib_remove(struct platform_device *pdev)
> +{
> + struct calib *calib = platform_get_drvdata(pdev);
> +
> + calib_cache_clean(calib);
> +
> + if (calib->calib_base)
> + iounmap(calib->calib_base);
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, calib);
> +
> + return 0;
> +}
> +
> +static int xrt_calib_probe(struct platform_device *pdev)
> +{
> + struct calib *calib;
> + struct resource *res;
> + int err = 0;
> +
> + calib = devm_kzalloc(&pdev->dev, sizeof(*calib), GFP_KERNEL);
> + if (!calib)
> + return -ENOMEM;
> +
> + calib->pdev = pdev;
> + platform_set_drvdata(pdev, calib);
> +
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + if (!res)
> + goto failed;
> +
> + calib->calib_base = ioremap(res->start, res->end - res->start + 1);
> + if (!calib->calib_base) {
> + err = -EIO;
> + xrt_err(pdev, "Map iomem failed");
> + goto failed;
> + }
> +
> + mutex_init(&calib->lock);
> + INIT_LIST_HEAD(&calib->cache_list);
> +
> + return 0;
> +
> +failed:
> + xrt_calib_remove(pdev);
> + return err;
> +}
> +
> +static int
> +xrt_calib_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + struct calib *calib = platform_get_drvdata(pdev);
> + int ret = 0;
> +
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + xrt_calib_event_cb(pdev, arg);
> + break;
> + case XRT_CALIB_RESULT: {
> + enum xrt_calib_results *r = (enum xrt_calib_results *)arg;
> + *r = calib->result;
> + break;
> + }
> + default:
> + xrt_err(pdev, "unsupported cmd %d", cmd);
> + ret = -EINVAL;
> + }
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_calib_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = XRT_MD_NODE_DDR_CALIB },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_calib_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_calib_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_calib_table[] = {
> + { XRT_CALIB, (kernel_ulong_t)&xrt_calib_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_calib_driver = {
> + .driver = {
> + .name = XRT_CALIB,
> + },
> + .probe = xrt_calib_probe,
> + .remove = xrt_calib_remove,
> + .id_table = xrt_calib_table,
> +};
> +
> +void calib_leaf_init_fini(bool init)
> +{
> + if (init)
> + xleaf_register_driver(XRT_SUBDEV_CALIB, &xrt_calib_driver, xrt_calib_endpoints);
> + else
> + xleaf_unregister_driver(XRT_SUBDEV_CALIB);
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 16/18] fpga: xrt: DDR calibration platform driver
2021-03-06 15:34 ` Tom Rix
@ 2021-03-13 0:45 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-13 0:45 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 03/06/2021 07:34 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Add DDR calibration driver. DDR calibration is a hardware function
>> discovered by walking firmware metadata. A platform device node will
>> be created for it. Hardware provides DDR calibration status through
>> this function.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/calib.h | 30 ++++
>> drivers/fpga/xrt/lib/xleaf/calib.c | 226 +++++++++++++++++++++++++
>> 2 files changed, 256 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
> calib is not descriptive, change filename to ddr_calibration
Sure.
>> diff --git a/drivers/fpga/xrt/include/xleaf/calib.h b/drivers/fpga/xrt/include/xleaf/calib.h
>> new file mode 100644
>> index 000000000000..f8aba4594c58
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/calib.h
>> @@ -0,0 +1,30 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT DDR Calibration Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Cheng Zhen <maxz@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_CALIB_H_
>> +#define _XRT_CALIB_H_
>> +
>> +#include "xleaf.h"
>> +#include <linux/xrt/xclbin.h>
>> +
>> +/*
>> + * Memory calibration driver IOCTL calls.
>> + */
>> +enum xrt_calib_results {
>> + XRT_CALIB_UNKNOWN,
> Initialize ?
Will fix.
>> + XRT_CALIB_SUCCEEDED,
>> + XRT_CALIB_FAILED,
>> +};
>> +
>> +enum xrt_calib_ioctl_cmd {
>> + XRT_CALIB_RESULT = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> +};
>> +
>> +#endif /* _XRT_CALIB_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/calib.c b/drivers/fpga/xrt/lib/xleaf/calib.c
>> new file mode 100644
>> index 000000000000..fbb813636e76
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/calib.c
>> @@ -0,0 +1,226 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA memory calibration driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * memory calibration
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + */
>> +#include <linux/delay.h>
>> +#include "xclbin-helper.h"
>> +#include "metadata.h"
>> +#include "xleaf/calib.h"
>> +
>> +#define XRT_CALIB "xrt_calib"
>> +
>> +struct calib_cache {
>> + struct list_head link;
>> + const char *ep_name;
>> + char *data;
>> + u32 data_size;
>> +};
>> +
>> +struct calib {
>> + struct platform_device *pdev;
>> + void *calib_base;
>> + struct mutex lock; /* calibration dev lock */
>> + struct list_head cache_list;
>> + u32 cache_num;
>> + enum xrt_calib_results result;
>> +};
>> +
>> +#define CALIB_DONE(calib) \
>> + (ioread32((calib)->calib_base) & BIT(0))
>> +
>> +static void calib_cache_clean_nolock(struct calib *calib)
>> +{
>> + struct calib_cache *cache, *temp;
>> +
>> + list_for_each_entry_safe(cache, temp, &calib->cache_list, link) {
>> + vfree(cache->data);
>> + list_del(&cache->link);
>> + vfree(cache);
>> + }
>> + calib->cache_num = 0;
>> +}
>> +
>> +static void calib_cache_clean(struct calib *calib)
>> +{
>> + mutex_lock(&calib->lock);
>> + calib_cache_clean_nolock(calib);
> No lock functions (i believe) should be prefixed with '__'
Will change.
>> + mutex_unlock(&calib->lock);
>> +}
>> +
>> +static int calib_srsr(struct calib *calib, struct platform_device *srsr_leaf)
> what is srsr ?
>
> Why a noop function ?
srsr is save-restore and self-refresh. It will not be supported in this
patch set. I will remove this function.
>
>> +{
>> + return -EOPNOTSUPP;
>> +}
>> +
>> +static int calib_calibration(struct calib *calib)
>> +{
>> + int i;
>> +
>> + for (i = 0; i < 20; i++) {
> 20 is a config parameter so should have a #define
>
> There a couple of busy wait blocks in xrt/ some count up, some count down.
>
> It would be good if they were consistent.
Will change these.
>
>> + if (CALIB_DONE(calib))
>> + break;
>> + msleep(500);
> 500 is another config
Will define.
Thanks,
Lizhi
>
> Tom
>
>> + }
>> +
>> + if (i == 20) {
>> + xrt_err(calib->pdev,
>> + "MIG calibration timeout after bitstream download");
>> + return -ETIMEDOUT;
>> + }
>> +
>> + xrt_info(calib->pdev, "took %dms", i * 500);
>> + return 0;
>> +}
>> +
>> +static void xrt_calib_event_cb(struct platform_device *pdev, void *arg)
>> +{
>> + struct calib *calib = platform_get_drvdata(pdev);
>> + struct xrt_event *evt = (struct xrt_event *)arg;
>> + enum xrt_events e = evt->xe_evt;
>> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
>> + int instance = evt->xe_subdev.xevt_subdev_instance;
>> + struct platform_device *leaf;
>> + int ret;
>> +
>> + switch (e) {
>> + case XRT_EVENT_POST_CREATION: {
>> + if (id == XRT_SUBDEV_SRSR) {
>> + leaf = xleaf_get_leaf_by_id(pdev,
>> + XRT_SUBDEV_SRSR,
>> + instance);
>> + if (!leaf) {
>> + xrt_err(pdev, "does not get SRSR subdev");
>> + return;
>> + }
>> + ret = calib_srsr(calib, leaf);
>> + xleaf_put_leaf(pdev, leaf);
>> + calib->result =
>> + ret ? XRT_CALIB_FAILED : XRT_CALIB_SUCCEEDED;
>> + } else if (id == XRT_SUBDEV_UCS) {
>> + ret = calib_calibration(calib);
>> + calib->result =
>> + ret ? XRT_CALIB_FAILED : XRT_CALIB_SUCCEEDED;
>> + }
>> + break;
>> + }
>> + default:
>> + break;
>> + }
>> +}
>> +
>> +static int xrt_calib_remove(struct platform_device *pdev)
>> +{
>> + struct calib *calib = platform_get_drvdata(pdev);
>> +
>> + calib_cache_clean(calib);
>> +
>> + if (calib->calib_base)
>> + iounmap(calib->calib_base);
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, calib);
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_calib_probe(struct platform_device *pdev)
>> +{
>> + struct calib *calib;
>> + struct resource *res;
>> + int err = 0;
>> +
>> + calib = devm_kzalloc(&pdev->dev, sizeof(*calib), GFP_KERNEL);
>> + if (!calib)
>> + return -ENOMEM;
>> +
>> + calib->pdev = pdev;
>> + platform_set_drvdata(pdev, calib);
>> +
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + if (!res)
>> + goto failed;
>> +
>> + calib->calib_base = ioremap(res->start, res->end - res->start + 1);
>> + if (!calib->calib_base) {
>> + err = -EIO;
>> + xrt_err(pdev, "Map iomem failed");
>> + goto failed;
>> + }
>> +
>> + mutex_init(&calib->lock);
>> + INIT_LIST_HEAD(&calib->cache_list);
>> +
>> + return 0;
>> +
>> +failed:
>> + xrt_calib_remove(pdev);
>> + return err;
>> +}
>> +
>> +static int
>> +xrt_calib_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + struct calib *calib = platform_get_drvdata(pdev);
>> + int ret = 0;
>> +
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + xrt_calib_event_cb(pdev, arg);
>> + break;
>> + case XRT_CALIB_RESULT: {
>> + enum xrt_calib_results *r = (enum xrt_calib_results *)arg;
>> + *r = calib->result;
>> + break;
>> + }
>> + default:
>> + xrt_err(pdev, "unsupported cmd %d", cmd);
>> + ret = -EINVAL;
>> + }
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_calib_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = XRT_MD_NODE_DDR_CALIB },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_calib_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_calib_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_calib_table[] = {
>> + { XRT_CALIB, (kernel_ulong_t)&xrt_calib_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_calib_driver = {
>> + .driver = {
>> + .name = XRT_CALIB,
>> + },
>> + .probe = xrt_calib_probe,
>> + .remove = xrt_calib_remove,
>> + .id_table = xrt_calib_table,
>> +};
>> +
>> +void calib_leaf_init_fini(bool init)
>> +{
>> + if (init)
>> + xleaf_register_driver(XRT_SUBDEV_CALIB, &xrt_calib_driver, xrt_calib_endpoints);
>> + else
>> + xleaf_unregister_driver(XRT_SUBDEV_CALIB);
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation platform driver
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (15 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 16/18] fpga: xrt: DDR calibration " Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-21 20:36 ` Moritz Fischer
2021-03-06 15:54 ` Tom Rix
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers Lizhi Hou
` (2 subsequent siblings)
19 siblings, 2 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Add partition isolation platform driver. partition isolation is
a hardware function discovered by walking firmware metadata.
A platform device node will be created for it. Partition isolation
function isolate the different fpga regions
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
drivers/fpga/xrt/include/xleaf/axigate.h | 25 ++
drivers/fpga/xrt/lib/xleaf/axigate.c | 298 +++++++++++++++++++++++
2 files changed, 323 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
diff --git a/drivers/fpga/xrt/include/xleaf/axigate.h b/drivers/fpga/xrt/include/xleaf/axigate.h
new file mode 100644
index 000000000000..2cef71e13b30
--- /dev/null
+++ b/drivers/fpga/xrt/include/xleaf/axigate.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Header file for XRT Axigate Leaf Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <Lizhi.Hou@xilinx.com>
+ */
+
+#ifndef _XRT_AXIGATE_H_
+#define _XRT_AXIGATE_H_
+
+#include "xleaf.h"
+#include "metadata.h"
+
+/*
+ * AXIGATE driver IOCTL calls.
+ */
+enum xrt_axigate_ioctl_cmd {
+ XRT_AXIGATE_FREEZE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
+ XRT_AXIGATE_FREE,
+};
+
+#endif /* _XRT_AXIGATE_H_ */
diff --git a/drivers/fpga/xrt/lib/xleaf/axigate.c b/drivers/fpga/xrt/lib/xleaf/axigate.c
new file mode 100644
index 000000000000..382969f9925f
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xleaf/axigate.c
@@ -0,0 +1,298 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA AXI Gate Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou<Lizhi.Hou@xilinx.com>
+ */
+
+#include <linux/mod_devicetable.h>
+#include <linux/platform_device.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/io.h>
+#include "metadata.h"
+#include "xleaf.h"
+#include "xleaf/axigate.h"
+
+#define XRT_AXIGATE "xrt_axigate"
+
+struct axigate_regs {
+ u32 iag_wr;
+ u32 iag_rvsd;
+ u32 iag_rd;
+} __packed;
+
+struct xrt_axigate {
+ struct platform_device *pdev;
+ void *base;
+ struct mutex gate_lock; /* gate dev lock */
+
+ void *evt_hdl;
+ const char *ep_name;
+
+ bool gate_freezed;
+};
+
+/* the ep names are in the order of hardware layers */
+static const char * const xrt_axigate_epnames[] = {
+ XRT_MD_NODE_GATE_PLP,
+ XRT_MD_NODE_GATE_ULP,
+ NULL
+};
+
+#define reg_rd(g, r) \
+ ioread32((void *)(g)->base + offsetof(struct axigate_regs, r))
+#define reg_wr(g, v, r) \
+ iowrite32(v, (void *)(g)->base + offsetof(struct axigate_regs, r))
+
+static inline void freeze_gate(struct xrt_axigate *gate)
+{
+ reg_wr(gate, 0, iag_wr);
+ ndelay(500);
+ reg_rd(gate, iag_rd);
+}
+
+static inline void free_gate(struct xrt_axigate *gate)
+{
+ reg_wr(gate, 0x2, iag_wr);
+ ndelay(500);
+ (void)reg_rd(gate, iag_rd);
+ reg_wr(gate, 0x3, iag_wr);
+ ndelay(500);
+ reg_rd(gate, iag_rd);
+}
+
+static int xrt_axigate_epname_idx(struct platform_device *pdev)
+{
+ int i;
+ int ret;
+ struct resource *res;
+
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ xrt_err(pdev, "Empty Resource!");
+ return -EINVAL;
+ }
+
+ for (i = 0; xrt_axigate_epnames[i]; i++) {
+ ret = strncmp(xrt_axigate_epnames[i], res->name,
+ strlen(xrt_axigate_epnames[i]) + 1);
+ if (!ret)
+ break;
+ }
+
+ ret = (xrt_axigate_epnames[i]) ? i : -EINVAL;
+ return ret;
+}
+
+static void xrt_axigate_freeze(struct platform_device *pdev)
+{
+ struct xrt_axigate *gate;
+ u32 freeze = 0;
+
+ gate = platform_get_drvdata(pdev);
+
+ mutex_lock(&gate->gate_lock);
+ freeze = reg_rd(gate, iag_rd);
+ if (freeze) { /* gate is opened */
+ xleaf_broadcast_event(pdev, XRT_EVENT_PRE_GATE_CLOSE, false);
+ freeze_gate(gate);
+ }
+
+ gate->gate_freezed = true;
+ mutex_unlock(&gate->gate_lock);
+
+ xrt_info(pdev, "freeze gate %s", gate->ep_name);
+}
+
+static void xrt_axigate_free(struct platform_device *pdev)
+{
+ struct xrt_axigate *gate;
+ u32 freeze;
+
+ gate = platform_get_drvdata(pdev);
+
+ mutex_lock(&gate->gate_lock);
+ freeze = reg_rd(gate, iag_rd);
+ if (!freeze) { /* gate is closed */
+ free_gate(gate);
+ xleaf_broadcast_event(pdev, XRT_EVENT_POST_GATE_OPEN, true);
+ /* xrt_axigate_free() could be called in event cb, thus
+ * we can not wait for the completes
+ */
+ }
+
+ gate->gate_freezed = false;
+ mutex_unlock(&gate->gate_lock);
+
+ xrt_info(pdev, "free gate %s", gate->ep_name);
+}
+
+static void xrt_axigate_event_cb(struct platform_device *pdev, void *arg)
+{
+ struct platform_device *leaf;
+ struct xrt_event *evt = (struct xrt_event *)arg;
+ enum xrt_events e = evt->xe_evt;
+ enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
+ int instance = evt->xe_subdev.xevt_subdev_instance;
+ struct xrt_axigate *gate = platform_get_drvdata(pdev);
+ struct resource *res;
+
+ switch (e) {
+ case XRT_EVENT_POST_CREATION:
+ break;
+ default:
+ return;
+ }
+
+ if (id != XRT_SUBDEV_AXIGATE)
+ return;
+
+ leaf = xleaf_get_leaf_by_id(pdev, id, instance);
+ if (!leaf)
+ return;
+
+ res = platform_get_resource(leaf, IORESOURCE_MEM, 0);
+ if (!res || !strncmp(res->name, gate->ep_name, strlen(res->name) + 1)) {
+ (void)xleaf_put_leaf(pdev, leaf);
+ return;
+ }
+
+ /*
+ * higher level axigate instance created,
+ * make sure the gate is openned. This covers 1RP flow which
+ * has plp gate as well.
+ */
+ if (xrt_axigate_epname_idx(leaf) > xrt_axigate_epname_idx(pdev))
+ xrt_axigate_free(pdev);
+ else
+ xleaf_ioctl(leaf, XRT_AXIGATE_FREE, NULL);
+
+ (void)xleaf_put_leaf(pdev, leaf);
+}
+
+static int
+xrt_axigate_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
+{
+ switch (cmd) {
+ case XRT_XLEAF_EVENT:
+ xrt_axigate_event_cb(pdev, arg);
+ break;
+ case XRT_AXIGATE_FREEZE:
+ xrt_axigate_freeze(pdev);
+ break;
+ case XRT_AXIGATE_FREE:
+ xrt_axigate_free(pdev);
+ break;
+ default:
+ xrt_err(pdev, "unsupported cmd %d", cmd);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int xrt_axigate_remove(struct platform_device *pdev)
+{
+ struct xrt_axigate *gate;
+
+ gate = platform_get_drvdata(pdev);
+
+ if (gate->base)
+ iounmap(gate->base);
+
+ platform_set_drvdata(pdev, NULL);
+ devm_kfree(&pdev->dev, gate);
+
+ return 0;
+}
+
+static int xrt_axigate_probe(struct platform_device *pdev)
+{
+ struct xrt_axigate *gate;
+ struct resource *res;
+ int ret;
+
+ gate = devm_kzalloc(&pdev->dev, sizeof(*gate), GFP_KERNEL);
+ if (!gate)
+ return -ENOMEM;
+
+ gate->pdev = pdev;
+ platform_set_drvdata(pdev, gate);
+
+ xrt_info(pdev, "probing...");
+ res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res) {
+ xrt_err(pdev, "Empty resource 0");
+ ret = -EINVAL;
+ goto failed;
+ }
+
+ gate->base = ioremap(res->start, res->end - res->start + 1);
+ if (!gate->base) {
+ xrt_err(pdev, "map base iomem failed");
+ ret = -EFAULT;
+ goto failed;
+ }
+
+ gate->ep_name = res->name;
+
+ mutex_init(&gate->gate_lock);
+
+ return 0;
+
+failed:
+ xrt_axigate_remove(pdev);
+ return ret;
+}
+
+static struct xrt_subdev_endpoints xrt_axigate_endpoints[] = {
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ { .ep_name = "ep_pr_isolate_ulp_00" },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ {
+ .xse_names = (struct xrt_subdev_ep_names[]) {
+ { .ep_name = "ep_pr_isolate_plp_00" },
+ { NULL },
+ },
+ .xse_min_ep = 1,
+ },
+ { 0 },
+};
+
+static struct xrt_subdev_drvdata xrt_axigate_data = {
+ .xsd_dev_ops = {
+ .xsd_ioctl = xrt_axigate_leaf_ioctl,
+ },
+};
+
+static const struct platform_device_id xrt_axigate_table[] = {
+ { XRT_AXIGATE, (kernel_ulong_t)&xrt_axigate_data },
+ { },
+};
+
+static struct platform_driver xrt_axigate_driver = {
+ .driver = {
+ .name = XRT_AXIGATE,
+ },
+ .probe = xrt_axigate_probe,
+ .remove = xrt_axigate_remove,
+ .id_table = xrt_axigate_table,
+};
+
+void axigate_leaf_init_fini(bool init)
+{
+ if (init) {
+ xleaf_register_driver(XRT_SUBDEV_AXIGATE,
+ &xrt_axigate_driver, xrt_axigate_endpoints);
+ } else {
+ xleaf_unregister_driver(XRT_SUBDEV_AXIGATE);
+ }
+}
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation " Lizhi Hou
@ 2021-02-21 20:36 ` Moritz Fischer
2021-03-16 20:38 ` Lizhi Hou
2021-03-06 15:54 ` Tom Rix
1 sibling, 1 reply; 87+ messages in thread
From: Moritz Fischer @ 2021-02-21 20:36 UTC (permalink / raw)
To: Lizhi Hou
Cc: linux-kernel, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, mdf, robh, Max Zhen
On Wed, Feb 17, 2021 at 10:40:18PM -0800, Lizhi Hou wrote:
> Add partition isolation platform driver. partition isolation is
> a hardware function discovered by walking firmware metadata.
> A platform device node will be created for it. Partition isolation
> function isolate the different fpga regions
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/axigate.h | 25 ++
> drivers/fpga/xrt/lib/xleaf/axigate.c | 298 +++++++++++++++++++++++
> 2 files changed, 323 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/axigate.h b/drivers/fpga/xrt/include/xleaf/axigate.h
> new file mode 100644
> index 000000000000..2cef71e13b30
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/axigate.h
> @@ -0,0 +1,25 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT Axigate Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_AXIGATE_H_
> +#define _XRT_AXIGATE_H_
> +
> +#include "xleaf.h"
> +#include "metadata.h"
> +
> +/*
> + * AXIGATE driver IOCTL calls.
> + */
> +enum xrt_axigate_ioctl_cmd {
> + XRT_AXIGATE_FREEZE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> + XRT_AXIGATE_FREE,
> +};
> +
> +#endif /* _XRT_AXIGATE_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/axigate.c b/drivers/fpga/xrt/lib/xleaf/axigate.c
> new file mode 100644
> index 000000000000..382969f9925f
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/axigate.c
> @@ -0,0 +1,298 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA AXI Gate Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/axigate.h"
> +
> +#define XRT_AXIGATE "xrt_axigate"
> +
> +struct axigate_regs {
> + u32 iag_wr;
> + u32 iag_rvsd;
> + u32 iag_rd;
> +} __packed;
Just make them #defines, even more so if there are only 3 of them.
> +
> +struct xrt_axigate {
> + struct platform_device *pdev;
> + void *base;
> + struct mutex gate_lock; /* gate dev lock */
> +
> + void *evt_hdl;
> + const char *ep_name;
> +
> + bool gate_freezed;
> +};
> +
> +/* the ep names are in the order of hardware layers */
> +static const char * const xrt_axigate_epnames[] = {
> + XRT_MD_NODE_GATE_PLP,
> + XRT_MD_NODE_GATE_ULP,
> + NULL
> +};
> +
> +#define reg_rd(g, r) \
> + ioread32((void *)(g)->base + offsetof(struct axigate_regs, r))
> +#define reg_wr(g, v, r) \
> + iowrite32(v, (void *)(g)->base + offsetof(struct axigate_regs, r))
> +
> +static inline void freeze_gate(struct xrt_axigate *gate)
> +{
> + reg_wr(gate, 0, iag_wr);
> + ndelay(500);
> + reg_rd(gate, iag_rd);
> +}
> +
> +static inline void free_gate(struct xrt_axigate *gate)
> +{
> + reg_wr(gate, 0x2, iag_wr);
> + ndelay(500);
Magic constants?
> + (void)reg_rd(gate, iag_rd);
At the very least add a comment on why? Is this for PCI synchronization
reasons?
> + reg_wr(gate, 0x3, iag_wr);
> + ndelay(500);
Magic constants?
> + reg_rd(gate, iag_rd);
Does it nead a (void) or not? Be consistent, again, why do we read here
at all?
> +}
> +
> +static int xrt_axigate_epname_idx(struct platform_device *pdev)
> +{
> + int i;
> + int ret;
> + struct resource *res;
Nope. Indents:
struct resource *res;
int, i, ret;
> +
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + if (!res) {
> + xrt_err(pdev, "Empty Resource!");
> + return -EINVAL;
> + }
> +
> + for (i = 0; xrt_axigate_epnames[i]; i++) {
> + ret = strncmp(xrt_axigate_epnames[i], res->name,
> + strlen(xrt_axigate_epnames[i]) + 1);
> + if (!ret)
> + break;
> + }
> +
> + ret = (xrt_axigate_epnames[i]) ? i : -EINVAL;
Why not just:
if (xrt_axigate_epnames[i])
return i;
return -EINVAL;
> + return ret;
> +}
> +
> +static void xrt_axigate_freeze(struct platform_device *pdev)
> +{
> + struct xrt_axigate *gate;
> + u32 freeze = 0;
Indents. Fix everywhere.
> +
> + gate = platform_get_drvdata(pdev);
> +
> + mutex_lock(&gate->gate_lock);
> + freeze = reg_rd(gate, iag_rd);
> + if (freeze) { /* gate is opened */
> + xleaf_broadcast_event(pdev, XRT_EVENT_PRE_GATE_CLOSE, false);
> + freeze_gate(gate);
> + }
> +
> + gate->gate_freezed = true;
s/freezed/frozen
> + mutex_unlock(&gate->gate_lock);
> +
> + xrt_info(pdev, "freeze gate %s", gate->ep_name);
debug?
> +}
> +
> +static void xrt_axigate_free(struct platform_device *pdev)
> +{
> + struct xrt_axigate *gate;
> + u32 freeze;
> +
> + gate = platform_get_drvdata(pdev);
> +
> + mutex_lock(&gate->gate_lock);
> + freeze = reg_rd(gate, iag_rd);
> + if (!freeze) { /* gate is closed */
> + free_gate(gate);
> + xleaf_broadcast_event(pdev, XRT_EVENT_POST_GATE_OPEN, true);
> + /* xrt_axigate_free() could be called in event cb, thus
> + * we can not wait for the completes
> + */
> + }
> +
> + gate->gate_freezed = false;
> + mutex_unlock(&gate->gate_lock);
> +
> + xrt_info(pdev, "free gate %s", gate->ep_name);
> +}
> +
> +static void xrt_axigate_event_cb(struct platform_device *pdev, void *arg)
> +{
> + struct platform_device *leaf;
> + struct xrt_event *evt = (struct xrt_event *)arg;
> + enum xrt_events e = evt->xe_evt;
> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
> + int instance = evt->xe_subdev.xevt_subdev_instance;
> + struct xrt_axigate *gate = platform_get_drvdata(pdev);
> + struct resource *res;
Reverse x-mas tree;
xxxxxxxxxx
xxxxxxxxx
xxxxxxxx
xxxxxx
> +
> + switch (e) {
> + case XRT_EVENT_POST_CREATION:
> + break;
> + default:
> + return;
> + }
> +
> + if (id != XRT_SUBDEV_AXIGATE)
> + return;
> +
> + leaf = xleaf_get_leaf_by_id(pdev, id, instance);
> + if (!leaf)
> + return;
> +
> + res = platform_get_resource(leaf, IORESOURCE_MEM, 0);
> + if (!res || !strncmp(res->name, gate->ep_name, strlen(res->name) + 1)) {
> + (void)xleaf_put_leaf(pdev, leaf);
> + return;
> + }
> +
> + /*
> + * higher level axigate instance created,
> + * make sure the gate is openned. This covers 1RP flow which
> + * has plp gate as well.
> + */
> + if (xrt_axigate_epname_idx(leaf) > xrt_axigate_epname_idx(pdev))
> + xrt_axigate_free(pdev);
> + else
> + xleaf_ioctl(leaf, XRT_AXIGATE_FREE, NULL);
> +
> + (void)xleaf_put_leaf(pdev, leaf);
> +}
> +
> +static int
> +xrt_axigate_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + xrt_axigate_event_cb(pdev, arg);
> + break;
> + case XRT_AXIGATE_FREEZE:
> + xrt_axigate_freeze(pdev);
> + break;
> + case XRT_AXIGATE_FREE:
> + xrt_axigate_free(pdev);
> + break;
> + default:
> + xrt_err(pdev, "unsupported cmd %d", cmd);
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static int xrt_axigate_remove(struct platform_device *pdev)
> +{
> + struct xrt_axigate *gate;
> +
> + gate = platform_get_drvdata(pdev);
> +
> + if (gate->base)
> + iounmap(gate->base);
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, gate);
No! The point of using devres is so cleanup happens on removal.
While you're at it, if you move the ioremap to a devres version, this
function can basically go away entirely.
> +
> + return 0;
> +}
> +
> +static int xrt_axigate_probe(struct platform_device *pdev)
> +{
> + struct xrt_axigate *gate;
> + struct resource *res;
> + int ret;
> +
> + gate = devm_kzalloc(&pdev->dev, sizeof(*gate), GFP_KERNEL);
> + if (!gate)
> + return -ENOMEM;
> +
> + gate->pdev = pdev;
> + platform_set_drvdata(pdev, gate);
> +
> + xrt_info(pdev, "probing...");
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + if (!res) {
> + xrt_err(pdev, "Empty resource 0");
> + ret = -EINVAL;
> + goto failed;
> + }
> +
> + gate->base = ioremap(res->start, res->end - res->start + 1);
> + if (!gate->base) {
> + xrt_err(pdev, "map base iomem failed");
> + ret = -EFAULT;
> + goto failed;
> + }
> +
> + gate->ep_name = res->name;
> +
> + mutex_init(&gate->gate_lock);
> +
> + return 0;
> +
> +failed:
> + xrt_axigate_remove(pdev);
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_axigate_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = "ep_pr_isolate_ulp_00" },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = "ep_pr_isolate_plp_00" },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_axigate_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_axigate_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_axigate_table[] = {
> + { XRT_AXIGATE, (kernel_ulong_t)&xrt_axigate_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_axigate_driver = {
> + .driver = {
> + .name = XRT_AXIGATE,
> + },
> + .probe = xrt_axigate_probe,
> + .remove = xrt_axigate_remove,
> + .id_table = xrt_axigate_table,
> +};
> +
> +void axigate_leaf_init_fini(bool init)
> +{
> + if (init) {
> + xleaf_register_driver(XRT_SUBDEV_AXIGATE,
> + &xrt_axigate_driver, xrt_axigate_endpoints);
> + } else {
> + xleaf_unregister_driver(XRT_SUBDEV_AXIGATE);
> + }
> +}
This thing is duplicated in every file, maybe a macro would be an idea.
> --
> 2.18.4
>
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation platform driver
2021-02-21 20:36 ` Moritz Fischer
@ 2021-03-16 20:38 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-16 20:38 UTC (permalink / raw)
To: Moritz Fischer, Lizhi Hou
Cc: linux-kernel, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, robh, Max Zhen
Hi Moritz,
On 02/21/2021 12:36 PM, Moritz Fischer wrote:
>
> On Wed, Feb 17, 2021 at 10:40:18PM -0800, Lizhi Hou wrote:
>> Add partition isolation platform driver. partition isolation is
>> a hardware function discovered by walking firmware metadata.
>> A platform device node will be created for it. Partition isolation
>> function isolate the different fpga regions
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/axigate.h | 25 ++
>> drivers/fpga/xrt/lib/xleaf/axigate.c | 298 +++++++++++++++++++++++
>> 2 files changed, 323 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>>
>> diff --git a/drivers/fpga/xrt/include/xleaf/axigate.h b/drivers/fpga/xrt/include/xleaf/axigate.h
>> new file mode 100644
>> index 000000000000..2cef71e13b30
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/axigate.h
>> @@ -0,0 +1,25 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT Axigate Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_AXIGATE_H_
>> +#define _XRT_AXIGATE_H_
>> +
>> +#include "xleaf.h"
>> +#include "metadata.h"
>> +
>> +/*
>> + * AXIGATE driver IOCTL calls.
>> + */
>> +enum xrt_axigate_ioctl_cmd {
>> + XRT_AXIGATE_FREEZE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> + XRT_AXIGATE_FREE,
>> +};
>> +
>> +#endif /* _XRT_AXIGATE_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/axigate.c b/drivers/fpga/xrt/lib/xleaf/axigate.c
>> new file mode 100644
>> index 000000000000..382969f9925f
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/axigate.c
>> @@ -0,0 +1,298 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA AXI Gate Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/io.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/axigate.h"
>> +
>> +#define XRT_AXIGATE "xrt_axigate"
>> +
>> +struct axigate_regs {
>> + u32 iag_wr;
>> + u32 iag_rvsd;
>> + u32 iag_rd;
>> +} __packed;
> Just make them #defines, even more so if there are only 3 of them.
We will use #define and regmap.
>> +
>> +struct xrt_axigate {
>> + struct platform_device *pdev;
>> + void *base;
>> + struct mutex gate_lock; /* gate dev lock */
>> +
>> + void *evt_hdl;
>> + const char *ep_name;
>> +
>> + bool gate_freezed;
>> +};
>> +
>> +/* the ep names are in the order of hardware layers */
>> +static const char * const xrt_axigate_epnames[] = {
>> + XRT_MD_NODE_GATE_PLP,
>> + XRT_MD_NODE_GATE_ULP,
>> + NULL
>> +};
>> +
>> +#define reg_rd(g, r) \
>> + ioread32((void *)(g)->base + offsetof(struct axigate_regs, r))
>> +#define reg_wr(g, v, r) \
>> + iowrite32(v, (void *)(g)->base + offsetof(struct axigate_regs, r))
>> +
>> +static inline void freeze_gate(struct xrt_axigate *gate)
>> +{
>> + reg_wr(gate, 0, iag_wr);
>> + ndelay(500);
>> + reg_rd(gate, iag_rd);
>> +}
>> +
>> +static inline void free_gate(struct xrt_axigate *gate)
>> +{
>> + reg_wr(gate, 0x2, iag_wr);
>> + ndelay(500);
> Magic constants?
Will use #define for 500
>> + (void)reg_rd(gate, iag_rd);
> At the very least add a comment on why? Is this for PCI synchronization
> reasons?
Will add comment. Some old board requires this extra read and it will
not hurt performance.
>
>> + reg_wr(gate, 0x3, iag_wr);
>> + ndelay(500);
> Magic constants?
>> + reg_rd(gate, iag_rd);
> Does it nead a (void) or not? Be consistent, again, why do we read here
> at all?
>> +}
>> +
>> +static int xrt_axigate_epname_idx(struct platform_device *pdev)
>> +{
>> + int i;
>> + int ret;
>> + struct resource *res;
> Nope. Indents:
>
> struct resource *res;
> int, i, ret;
Will change this.
>
>> +
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + if (!res) {
>> + xrt_err(pdev, "Empty Resource!");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; xrt_axigate_epnames[i]; i++) {
>> + ret = strncmp(xrt_axigate_epnames[i], res->name,
>> + strlen(xrt_axigate_epnames[i]) + 1);
>> + if (!ret)
>> + break;
>> + }
>> +
>> + ret = (xrt_axigate_epnames[i]) ? i : -EINVAL;
> Why not just:
>
> if (xrt_axigate_epnames[i])
> return i;
>
> return -EINVAL;
Will change this.
>> + return ret;
>> +}
>> +
>> +static void xrt_axigate_freeze(struct platform_device *pdev)
>> +{
>> + struct xrt_axigate *gate;
>> + u32 freeze = 0;
> Indents. Fix everywhere.
Will fix this.
>> +
>> + gate = platform_get_drvdata(pdev);
>> +
>> + mutex_lock(&gate->gate_lock);
>> + freeze = reg_rd(gate, iag_rd);
>> + if (freeze) { /* gate is opened */
>> + xleaf_broadcast_event(pdev, XRT_EVENT_PRE_GATE_CLOSE, false);
>> + freeze_gate(gate);
>> + }
>> +
>> + gate->gate_freezed = true;
> s/freezed/frozen
Will change terms to open / close.
>> + mutex_unlock(&gate->gate_lock);
>> +
>> + xrt_info(pdev, "freeze gate %s", gate->ep_name);
> debug?
axigate is a very critical part for programming FPGA. We hope to have an
explicit printk to indicate the axigate open/close.
>> +}
>> +
>> +static void xrt_axigate_free(struct platform_device *pdev)
>> +{
>> + struct xrt_axigate *gate;
>> + u32 freeze;
>> +
>> + gate = platform_get_drvdata(pdev);
>> +
>> + mutex_lock(&gate->gate_lock);
>> + freeze = reg_rd(gate, iag_rd);
>> + if (!freeze) { /* gate is closed */
>> + free_gate(gate);
>> + xleaf_broadcast_event(pdev, XRT_EVENT_POST_GATE_OPEN, true);
>> + /* xrt_axigate_free() could be called in event cb, thus
>> + * we can not wait for the completes
>> + */
>> + }
>> +
>> + gate->gate_freezed = false;
>> + mutex_unlock(&gate->gate_lock);
>> +
>> + xrt_info(pdev, "free gate %s", gate->ep_name);
>> +}
>> +
>> +static void xrt_axigate_event_cb(struct platform_device *pdev, void *arg)
>> +{
>> + struct platform_device *leaf;
>> + struct xrt_event *evt = (struct xrt_event *)arg;
>> + enum xrt_events e = evt->xe_evt;
>> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
>> + int instance = evt->xe_subdev.xevt_subdev_instance;
>> + struct xrt_axigate *gate = platform_get_drvdata(pdev);
>> + struct resource *res;
> Reverse x-mas tree;
> xxxxxxxxxx
> xxxxxxxxx
> xxxxxxxx
> xxxxxx
Will fix this.
>> +
>> + switch (e) {
>> + case XRT_EVENT_POST_CREATION:
>> + break;
>> + default:
>> + return;
>> + }
>> +
>> + if (id != XRT_SUBDEV_AXIGATE)
>> + return;
>> +
>> + leaf = xleaf_get_leaf_by_id(pdev, id, instance);
>> + if (!leaf)
>> + return;
>> +
>> + res = platform_get_resource(leaf, IORESOURCE_MEM, 0);
>> + if (!res || !strncmp(res->name, gate->ep_name, strlen(res->name) + 1)) {
>> + (void)xleaf_put_leaf(pdev, leaf);
>> + return;
>> + }
>> +
>> + /*
>> + * higher level axigate instance created,
>> + * make sure the gate is openned. This covers 1RP flow which
>> + * has plp gate as well.
>> + */
>> + if (xrt_axigate_epname_idx(leaf) > xrt_axigate_epname_idx(pdev))
>> + xrt_axigate_free(pdev);
>> + else
>> + xleaf_ioctl(leaf, XRT_AXIGATE_FREE, NULL);
>> +
>> + (void)xleaf_put_leaf(pdev, leaf);
>> +}
>> +
>> +static int
>> +xrt_axigate_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + xrt_axigate_event_cb(pdev, arg);
>> + break;
>> + case XRT_AXIGATE_FREEZE:
>> + xrt_axigate_freeze(pdev);
>> + break;
>> + case XRT_AXIGATE_FREE:
>> + xrt_axigate_free(pdev);
>> + break;
>> + default:
>> + xrt_err(pdev, "unsupported cmd %d", cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_axigate_remove(struct platform_device *pdev)
>> +{
>> + struct xrt_axigate *gate;
>> +
>> + gate = platform_get_drvdata(pdev);
>> +
>> + if (gate->base)
>> + iounmap(gate->base);
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, gate);
> No! The point of using devres is so cleanup happens on removal.
> While you're at it, if you move the ioremap to a devres version, this
> function can basically go away entirely.
Will fix this.
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_axigate_probe(struct platform_device *pdev)
>> +{
>> + struct xrt_axigate *gate;
>> + struct resource *res;
>> + int ret;
>> +
>> + gate = devm_kzalloc(&pdev->dev, sizeof(*gate), GFP_KERNEL);
>> + if (!gate)
>> + return -ENOMEM;
>> +
>> + gate->pdev = pdev;
>> + platform_set_drvdata(pdev, gate);
>> +
>> + xrt_info(pdev, "probing...");
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + if (!res) {
>> + xrt_err(pdev, "Empty resource 0");
>> + ret = -EINVAL;
>> + goto failed;
>> + }
>> +
>> + gate->base = ioremap(res->start, res->end - res->start + 1);
>> + if (!gate->base) {
>> + xrt_err(pdev, "map base iomem failed");
>> + ret = -EFAULT;
>> + goto failed;
>> + }
>> +
>> + gate->ep_name = res->name;
>> +
>> + mutex_init(&gate->gate_lock);
>> +
>> + return 0;
>> +
>> +failed:
>> + xrt_axigate_remove(pdev);
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_axigate_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = "ep_pr_isolate_ulp_00" },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = "ep_pr_isolate_plp_00" },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_axigate_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_axigate_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_axigate_table[] = {
>> + { XRT_AXIGATE, (kernel_ulong_t)&xrt_axigate_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_axigate_driver = {
>> + .driver = {
>> + .name = XRT_AXIGATE,
>> + },
>> + .probe = xrt_axigate_probe,
>> + .remove = xrt_axigate_remove,
>> + .id_table = xrt_axigate_table,
>> +};
>> +
>> +void axigate_leaf_init_fini(bool init)
>> +{
>> + if (init) {
>> + xleaf_register_driver(XRT_SUBDEV_AXIGATE,
>> + &xrt_axigate_driver, xrt_axigate_endpoints);
>> + } else {
>> + xleaf_unregister_driver(XRT_SUBDEV_AXIGATE);
>> + }
>> +}
> This thing is duplicated in every file, maybe a macro would be an idea.
Will define a macro.
Thanks,
Lizhi
>> --
>> 2.18.4
>>
> - Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation platform driver
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation " Lizhi Hou
2021-02-21 20:36 ` Moritz Fischer
@ 2021-03-06 15:54 ` Tom Rix
2021-03-13 6:53 ` Lizhi Hou
1 sibling, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-03-06 15:54 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Add partition isolation platform driver. partition isolation is
> a hardware function discovered by walking firmware metadata.
> A platform device node will be created for it. Partition isolation
> function isolate the different fpga regions
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> drivers/fpga/xrt/include/xleaf/axigate.h | 25 ++
> drivers/fpga/xrt/lib/xleaf/axigate.c | 298 +++++++++++++++++++++++
> 2 files changed, 323 insertions(+)
> create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
> create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>
> diff --git a/drivers/fpga/xrt/include/xleaf/axigate.h b/drivers/fpga/xrt/include/xleaf/axigate.h
> new file mode 100644
> index 000000000000..2cef71e13b30
> --- /dev/null
> +++ b/drivers/fpga/xrt/include/xleaf/axigate.h
> @@ -0,0 +1,25 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Header file for XRT Axigate Leaf Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
> + */
> +
> +#ifndef _XRT_AXIGATE_H_
> +#define _XRT_AXIGATE_H_
> +
> +#include "xleaf.h"
> +#include "metadata.h"
> +
> +/*
> + * AXIGATE driver IOCTL calls.
> + */
> +enum xrt_axigate_ioctl_cmd {
> + XRT_AXIGATE_FREEZE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
> + XRT_AXIGATE_FREE,
These are substrings, could change suffix to make it harder for developer to mix up.
> +};
> +
> +#endif /* _XRT_AXIGATE_H_ */
> diff --git a/drivers/fpga/xrt/lib/xleaf/axigate.c b/drivers/fpga/xrt/lib/xleaf/axigate.c
> new file mode 100644
> index 000000000000..382969f9925f
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/xleaf/axigate.c
> @@ -0,0 +1,298 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo FPGA AXI Gate Driver
> + *
> + * Copyright (C) 2020-2021 Xilinx, Inc.
> + *
> + * Authors:
> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
> + */
> +
> +#include <linux/mod_devicetable.h>
> +#include <linux/platform_device.h>
> +#include <linux/delay.h>
> +#include <linux/device.h>
> +#include <linux/io.h>
> +#include "metadata.h"
> +#include "xleaf.h"
> +#include "xleaf/axigate.h"
> +
> +#define XRT_AXIGATE "xrt_axigate"
> +
> +struct axigate_regs {
> + u32 iag_wr;
> + u32 iag_rvsd;
> + u32 iag_rd;
> +} __packed;
similar to other patches, prefix of element is not needed.
> +
> +struct xrt_axigate {
> + struct platform_device *pdev;
> + void *base;
> + struct mutex gate_lock; /* gate dev lock */
> +
> + void *evt_hdl;
> + const char *ep_name;
> +
> + bool gate_freezed;
> +};
> +
> +/* the ep names are in the order of hardware layers */
> +static const char * const xrt_axigate_epnames[] = {
> + XRT_MD_NODE_GATE_PLP,
> + XRT_MD_NODE_GATE_ULP,
what are plp, ulp ? it is helpful to comment or expand acronyms
> + NULL
> +};
> +
> +#define reg_rd(g, r) \
> + ioread32((void *)(g)->base + offsetof(struct axigate_regs, r))
> +#define reg_wr(g, v, r) \
> + iowrite32(v, (void *)(g)->base + offsetof(struct axigate_regs, r))
> +
> +static inline void freeze_gate(struct xrt_axigate *gate)
> +{
> + reg_wr(gate, 0, iag_wr);
The values written here and below are magic, the need to have #defines
> + ndelay(500);
> + reg_rd(gate, iag_rd);
> +}
> +
> +static inline void free_gate(struct xrt_axigate *gate)
> +{
> + reg_wr(gate, 0x2, iag_wr);
> + ndelay(500);
> + (void)reg_rd(gate, iag_rd);
> + reg_wr(gate, 0x3, iag_wr);
> + ndelay(500);
> + reg_rd(gate, iag_rd);
> +}
> +
> +static int xrt_axigate_epname_idx(struct platform_device *pdev)
> +{
> + int i;
> + int ret;
int i, ret;
> + struct resource *res;
> +
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + if (!res) {
> + xrt_err(pdev, "Empty Resource!");
> + return -EINVAL;
> + }
> +
> + for (i = 0; xrt_axigate_epnames[i]; i++) {
null guarded array is useful with the size isn't know,
in this case it is, so covert loop to using ARRAY_SIZE
> + ret = strncmp(xrt_axigate_epnames[i], res->name,
> + strlen(xrt_axigate_epnames[i]) + 1);
needs a strlen check in case res->name is just a substring
> + if (!ret)
> + break;
> + }
> +
> + ret = (xrt_axigate_epnames[i]) ? i : -EINVAL;
> + return ret;
> +}
> +
> +static void xrt_axigate_freeze(struct platform_device *pdev)
> +{
> + struct xrt_axigate *gate;
> + u32 freeze = 0;
> +
> + gate = platform_get_drvdata(pdev);
> +
> + mutex_lock(&gate->gate_lock);
> + freeze = reg_rd(gate, iag_rd);
> + if (freeze) { /* gate is opened */
> + xleaf_broadcast_event(pdev, XRT_EVENT_PRE_GATE_CLOSE, false);
> + freeze_gate(gate);
> + }
> +
> + gate->gate_freezed = true;
Looks like freeze could be 0, so is setting gate_freeze = true correct all the time ?
> + mutex_unlock(&gate->gate_lock);
> +
> + xrt_info(pdev, "freeze gate %s", gate->ep_name);
> +}
> +
> +static void xrt_axigate_free(struct platform_device *pdev)
> +{
> + struct xrt_axigate *gate;
> + u32 freeze;
> +
> + gate = platform_get_drvdata(pdev);
> +
> + mutex_lock(&gate->gate_lock);
> + freeze = reg_rd(gate, iag_rd);
> + if (!freeze) { /* gate is closed */
> + free_gate(gate);
> + xleaf_broadcast_event(pdev, XRT_EVENT_POST_GATE_OPEN, true);
> + /* xrt_axigate_free() could be called in event cb, thus
> + * we can not wait for the completes
> + */
> + }
> +
> + gate->gate_freezed = false;
freezed is not a word, the element name should be 'gate_frozen'
> + mutex_unlock(&gate->gate_lock);
> +
> + xrt_info(pdev, "free gate %s", gate->ep_name);
> +}
> +
> +static void xrt_axigate_event_cb(struct platform_device *pdev, void *arg)
> +{
> + struct platform_device *leaf;
> + struct xrt_event *evt = (struct xrt_event *)arg;
> + enum xrt_events e = evt->xe_evt;
> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
> + int instance = evt->xe_subdev.xevt_subdev_instance;
> + struct xrt_axigate *gate = platform_get_drvdata(pdev);
> + struct resource *res;
> +
> + switch (e) {
> + case XRT_EVENT_POST_CREATION:
> + break;
> + default:
> + return;
> + }
convert switch() to if ()
> +
> + if (id != XRT_SUBDEV_AXIGATE)
> + return;
> +
> + leaf = xleaf_get_leaf_by_id(pdev, id, instance);
> + if (!leaf)
> + return;
> +
> + res = platform_get_resource(leaf, IORESOURCE_MEM, 0);
> + if (!res || !strncmp(res->name, gate->ep_name, strlen(res->name) + 1)) {
> + (void)xleaf_put_leaf(pdev, leaf);
> + return;
> + }
> +
> + /*
> + * higher level axigate instance created,
> + * make sure the gate is openned. This covers 1RP flow which
is openned -> is opened
what is 1RP ?
Tom
> + * has plp gate as well.
> + */
> + if (xrt_axigate_epname_idx(leaf) > xrt_axigate_epname_idx(pdev))
> + xrt_axigate_free(pdev);
> + else
> + xleaf_ioctl(leaf, XRT_AXIGATE_FREE, NULL);
> +
> + (void)xleaf_put_leaf(pdev, leaf);
> +}
> +
> +static int
> +xrt_axigate_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
> +{
> + switch (cmd) {
> + case XRT_XLEAF_EVENT:
> + xrt_axigate_event_cb(pdev, arg);
> + break;
> + case XRT_AXIGATE_FREEZE:
> + xrt_axigate_freeze(pdev);
> + break;
> + case XRT_AXIGATE_FREE:
> + xrt_axigate_free(pdev);
> + break;
> + default:
> + xrt_err(pdev, "unsupported cmd %d", cmd);
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static int xrt_axigate_remove(struct platform_device *pdev)
> +{
> + struct xrt_axigate *gate;
> +
> + gate = platform_get_drvdata(pdev);
> +
> + if (gate->base)
> + iounmap(gate->base);
> +
> + platform_set_drvdata(pdev, NULL);
> + devm_kfree(&pdev->dev, gate);
> +
> + return 0;
> +}
> +
> +static int xrt_axigate_probe(struct platform_device *pdev)
> +{
> + struct xrt_axigate *gate;
> + struct resource *res;
> + int ret;
> +
> + gate = devm_kzalloc(&pdev->dev, sizeof(*gate), GFP_KERNEL);
> + if (!gate)
> + return -ENOMEM;
> +
> + gate->pdev = pdev;
> + platform_set_drvdata(pdev, gate);
> +
> + xrt_info(pdev, "probing...");
> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
> + if (!res) {
> + xrt_err(pdev, "Empty resource 0");
> + ret = -EINVAL;
> + goto failed;
> + }
> +
> + gate->base = ioremap(res->start, res->end - res->start + 1);
> + if (!gate->base) {
> + xrt_err(pdev, "map base iomem failed");
> + ret = -EFAULT;
> + goto failed;
> + }
> +
> + gate->ep_name = res->name;
> +
> + mutex_init(&gate->gate_lock);
> +
> + return 0;
> +
> +failed:
> + xrt_axigate_remove(pdev);
> + return ret;
> +}
> +
> +static struct xrt_subdev_endpoints xrt_axigate_endpoints[] = {
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = "ep_pr_isolate_ulp_00" },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + {
> + .xse_names = (struct xrt_subdev_ep_names[]) {
> + { .ep_name = "ep_pr_isolate_plp_00" },
> + { NULL },
> + },
> + .xse_min_ep = 1,
> + },
> + { 0 },
> +};
> +
> +static struct xrt_subdev_drvdata xrt_axigate_data = {
> + .xsd_dev_ops = {
> + .xsd_ioctl = xrt_axigate_leaf_ioctl,
> + },
> +};
> +
> +static const struct platform_device_id xrt_axigate_table[] = {
> + { XRT_AXIGATE, (kernel_ulong_t)&xrt_axigate_data },
> + { },
> +};
> +
> +static struct platform_driver xrt_axigate_driver = {
> + .driver = {
> + .name = XRT_AXIGATE,
> + },
> + .probe = xrt_axigate_probe,
> + .remove = xrt_axigate_remove,
> + .id_table = xrt_axigate_table,
> +};
> +
> +void axigate_leaf_init_fini(bool init)
> +{
> + if (init) {
> + xleaf_register_driver(XRT_SUBDEV_AXIGATE,
> + &xrt_axigate_driver, xrt_axigate_endpoints);
> + } else {
> + xleaf_unregister_driver(XRT_SUBDEV_AXIGATE);
> + }
> +}
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation platform driver
2021-03-06 15:54 ` Tom Rix
@ 2021-03-13 6:53 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-13 6:53 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh, Max Zhen
Hi Tom,
On 03/06/2021 07:54 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Add partition isolation platform driver. partition isolation is
>> a hardware function discovered by walking firmware metadata.
>> A platform device node will be created for it. Partition isolation
>> function isolate the different fpga regions
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
>> ---
>> drivers/fpga/xrt/include/xleaf/axigate.h | 25 ++
>> drivers/fpga/xrt/lib/xleaf/axigate.c | 298 +++++++++++++++++++++++
>> 2 files changed, 323 insertions(+)
>> create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>>
>> diff --git a/drivers/fpga/xrt/include/xleaf/axigate.h b/drivers/fpga/xrt/include/xleaf/axigate.h
>> new file mode 100644
>> index 000000000000..2cef71e13b30
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/include/xleaf/axigate.h
>> @@ -0,0 +1,25 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +/*
>> + * Header file for XRT Axigate Leaf Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou <Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#ifndef _XRT_AXIGATE_H_
>> +#define _XRT_AXIGATE_H_
>> +
>> +#include "xleaf.h"
>> +#include "metadata.h"
>> +
>> +/*
>> + * AXIGATE driver IOCTL calls.
>> + */
>> +enum xrt_axigate_ioctl_cmd {
>> + XRT_AXIGATE_FREEZE = XRT_XLEAF_CUSTOM_BASE, /* See comments in xleaf.h */
>> + XRT_AXIGATE_FREE,
> These are substrings, could change suffix to make it harder for developer to mix up.
Will change 'freeze' to 'close', 'free' to 'open'.
>> +};
>> +
>> +#endif /* _XRT_AXIGATE_H_ */
>> diff --git a/drivers/fpga/xrt/lib/xleaf/axigate.c b/drivers/fpga/xrt/lib/xleaf/axigate.c
>> new file mode 100644
>> index 000000000000..382969f9925f
>> --- /dev/null
>> +++ b/drivers/fpga/xrt/lib/xleaf/axigate.c
>> @@ -0,0 +1,298 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Xilinx Alveo FPGA AXI Gate Driver
>> + *
>> + * Copyright (C) 2020-2021 Xilinx, Inc.
>> + *
>> + * Authors:
>> + * Lizhi Hou<Lizhi.Hou@xilinx.com>
>> + */
>> +
>> +#include <linux/mod_devicetable.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/delay.h>
>> +#include <linux/device.h>
>> +#include <linux/io.h>
>> +#include "metadata.h"
>> +#include "xleaf.h"
>> +#include "xleaf/axigate.h"
>> +
>> +#define XRT_AXIGATE "xrt_axigate"
>> +
>> +struct axigate_regs {
>> + u32 iag_wr;
>> + u32 iag_rvsd;
>> + u32 iag_rd;
>> +} __packed;
> similar to other patches, prefix of element is not needed.
Will remove this and use #define and regmap.
>> +
>> +struct xrt_axigate {
>> + struct platform_device *pdev;
>> + void *base;
>> + struct mutex gate_lock; /* gate dev lock */
>> +
>> + void *evt_hdl;
>> + const char *ep_name;
>> +
>> + bool gate_freezed;
>> +};
>> +
>> +/* the ep names are in the order of hardware layers */
>> +static const char * const xrt_axigate_epnames[] = {
>> + XRT_MD_NODE_GATE_PLP,
>> + XRT_MD_NODE_GATE_ULP,
> what are plp, ulp ? it is helpful to comment or expand acronyms
plp stands for provider logic partition and ulp stands for user logic
partition. Will add comment.
>> + NULL
>> +};
>> +
>> +#define reg_rd(g, r) \
>> + ioread32((void *)(g)->base + offsetof(struct axigate_regs, r))
>> +#define reg_wr(g, v, r) \
>> + iowrite32(v, (void *)(g)->base + offsetof(struct axigate_regs, r))
>> +
>> +static inline void freeze_gate(struct xrt_axigate *gate)
>> +{
>> + reg_wr(gate, 0, iag_wr);
> The values written here and below are magic, the need to have #defines
Will add #defines
>> + ndelay(500);
>> + reg_rd(gate, iag_rd);
>> +}
>> +
>> +static inline void free_gate(struct xrt_axigate *gate)
>> +{
>> + reg_wr(gate, 0x2, iag_wr);
>> + ndelay(500);
>> + (void)reg_rd(gate, iag_rd);
>> + reg_wr(gate, 0x3, iag_wr);
>> + ndelay(500);
>> + reg_rd(gate, iag_rd);
>> +}
>> +
>> +static int xrt_axigate_epname_idx(struct platform_device *pdev)
>> +{
>> + int i;
>> + int ret;
> int i, ret;
sure.
>> + struct resource *res;
>> +
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + if (!res) {
>> + xrt_err(pdev, "Empty Resource!");
>> + return -EINVAL;
>> + }
>> +
>> + for (i = 0; xrt_axigate_epnames[i]; i++) {
> null guarded array is useful with the size isn't know,
>
> in this case it is, so covert loop to using ARRAY_SIZE
Will use ARRAY_SIZE.
>
>> + ret = strncmp(xrt_axigate_epnames[i], res->name,
>> + strlen(xrt_axigate_epnames[i]) + 1);
> needs a strlen check in case res->name is just a substring
'strlen() + 1' is used, thus the comparing covers substring as well.
>> + if (!ret)
>> + break;
>> + }
>> +
>> + ret = (xrt_axigate_epnames[i]) ? i : -EINVAL;
>> + return ret;
>> +}
>> +
>> +static void xrt_axigate_freeze(struct platform_device *pdev)
>> +{
>> + struct xrt_axigate *gate;
>> + u32 freeze = 0;
>> +
>> + gate = platform_get_drvdata(pdev);
>> +
>> + mutex_lock(&gate->gate_lock);
>> + freeze = reg_rd(gate, iag_rd);
>> + if (freeze) { /* gate is opened */
>> + xleaf_broadcast_event(pdev, XRT_EVENT_PRE_GATE_CLOSE, false);
>> + freeze_gate(gate);
>> + }
>> +
>> + gate->gate_freezed = true;
> Looks like freeze could be 0, so is setting gate_freeze = true correct all the time ?
added checking for freeze_gate(). if it succeed, gate will be frozen
(closed).
>> + mutex_unlock(&gate->gate_lock);
>> +
>> + xrt_info(pdev, "freeze gate %s", gate->ep_name);
>> +}
>> +
>> +static void xrt_axigate_free(struct platform_device *pdev)
>> +{
>> + struct xrt_axigate *gate;
>> + u32 freeze;
>> +
>> + gate = platform_get_drvdata(pdev);
>> +
>> + mutex_lock(&gate->gate_lock);
>> + freeze = reg_rd(gate, iag_rd);
>> + if (!freeze) { /* gate is closed */
>> + free_gate(gate);
>> + xleaf_broadcast_event(pdev, XRT_EVENT_POST_GATE_OPEN, true);
>> + /* xrt_axigate_free() could be called in event cb, thus
>> + * we can not wait for the completes
>> + */
>> + }
>> +
>> + gate->gate_freezed = false;
> freezed is not a word, the element name should be 'gate_frozen'
Will change to gate_closed.
>> + mutex_unlock(&gate->gate_lock);
>> +
>> + xrt_info(pdev, "free gate %s", gate->ep_name);
>> +}
>> +
>> +static void xrt_axigate_event_cb(struct platform_device *pdev, void *arg)
>> +{
>> + struct platform_device *leaf;
>> + struct xrt_event *evt = (struct xrt_event *)arg;
>> + enum xrt_events e = evt->xe_evt;
>> + enum xrt_subdev_id id = evt->xe_subdev.xevt_subdev_id;
>> + int instance = evt->xe_subdev.xevt_subdev_instance;
>> + struct xrt_axigate *gate = platform_get_drvdata(pdev);
>> + struct resource *res;
>> +
>> + switch (e) {
>> + case XRT_EVENT_POST_CREATION:
>> + break;
>> + default:
>> + return;
>> + }
> convert switch() to if ()
Sure.
>> +
>> + if (id != XRT_SUBDEV_AXIGATE)
>> + return;
>> +
>> + leaf = xleaf_get_leaf_by_id(pdev, id, instance);
>> + if (!leaf)
>> + return;
>> +
>> + res = platform_get_resource(leaf, IORESOURCE_MEM, 0);
>> + if (!res || !strncmp(res->name, gate->ep_name, strlen(res->name) + 1)) {
>> + (void)xleaf_put_leaf(pdev, leaf);
>> + return;
>> + }
>> +
>> + /*
>> + * higher level axigate instance created,
>> + * make sure the gate is openned. This covers 1RP flow which
> is openned -> is opened
sure.
>
> what is 1RP ?
1RP flow is one of hardware shell build flow. It is xilinx internal term
I will remove this sentence.
Thanks,
Lizhi
>
> Tom
>
>> + * has plp gate as well.
>> + */
>> + if (xrt_axigate_epname_idx(leaf) > xrt_axigate_epname_idx(pdev))
>> + xrt_axigate_free(pdev);
>> + else
>> + xleaf_ioctl(leaf, XRT_AXIGATE_FREE, NULL);
>> +
>> + (void)xleaf_put_leaf(pdev, leaf);
>> +}
>> +
>> +static int
>> +xrt_axigate_leaf_ioctl(struct platform_device *pdev, u32 cmd, void *arg)
>> +{
>> + switch (cmd) {
>> + case XRT_XLEAF_EVENT:
>> + xrt_axigate_event_cb(pdev, arg);
>> + break;
>> + case XRT_AXIGATE_FREEZE:
>> + xrt_axigate_freeze(pdev);
>> + break;
>> + case XRT_AXIGATE_FREE:
>> + xrt_axigate_free(pdev);
>> + break;
>> + default:
>> + xrt_err(pdev, "unsupported cmd %d", cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_axigate_remove(struct platform_device *pdev)
>> +{
>> + struct xrt_axigate *gate;
>> +
>> + gate = platform_get_drvdata(pdev);
>> +
>> + if (gate->base)
>> + iounmap(gate->base);
>> +
>> + platform_set_drvdata(pdev, NULL);
>> + devm_kfree(&pdev->dev, gate);
>> +
>> + return 0;
>> +}
>> +
>> +static int xrt_axigate_probe(struct platform_device *pdev)
>> +{
>> + struct xrt_axigate *gate;
>> + struct resource *res;
>> + int ret;
>> +
>> + gate = devm_kzalloc(&pdev->dev, sizeof(*gate), GFP_KERNEL);
>> + if (!gate)
>> + return -ENOMEM;
>> +
>> + gate->pdev = pdev;
>> + platform_set_drvdata(pdev, gate);
>> +
>> + xrt_info(pdev, "probing...");
>> + res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
>> + if (!res) {
>> + xrt_err(pdev, "Empty resource 0");
>> + ret = -EINVAL;
>> + goto failed;
>> + }
>> +
>> + gate->base = ioremap(res->start, res->end - res->start + 1);
>> + if (!gate->base) {
>> + xrt_err(pdev, "map base iomem failed");
>> + ret = -EFAULT;
>> + goto failed;
>> + }
>> +
>> + gate->ep_name = res->name;
>> +
>> + mutex_init(&gate->gate_lock);
>> +
>> + return 0;
>> +
>> +failed:
>> + xrt_axigate_remove(pdev);
>> + return ret;
>> +}
>> +
>> +static struct xrt_subdev_endpoints xrt_axigate_endpoints[] = {
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = "ep_pr_isolate_ulp_00" },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + {
>> + .xse_names = (struct xrt_subdev_ep_names[]) {
>> + { .ep_name = "ep_pr_isolate_plp_00" },
>> + { NULL },
>> + },
>> + .xse_min_ep = 1,
>> + },
>> + { 0 },
>> +};
>> +
>> +static struct xrt_subdev_drvdata xrt_axigate_data = {
>> + .xsd_dev_ops = {
>> + .xsd_ioctl = xrt_axigate_leaf_ioctl,
>> + },
>> +};
>> +
>> +static const struct platform_device_id xrt_axigate_table[] = {
>> + { XRT_AXIGATE, (kernel_ulong_t)&xrt_axigate_data },
>> + { },
>> +};
>> +
>> +static struct platform_driver xrt_axigate_driver = {
>> + .driver = {
>> + .name = XRT_AXIGATE,
>> + },
>> + .probe = xrt_axigate_probe,
>> + .remove = xrt_axigate_remove,
>> + .id_table = xrt_axigate_table,
>> +};
>> +
>> +void axigate_leaf_init_fini(bool init)
>> +{
>> + if (init) {
>> + xleaf_register_driver(XRT_SUBDEV_AXIGATE,
>> + &xrt_axigate_driver, xrt_axigate_endpoints);
>> + } else {
>> + xleaf_unregister_driver(XRT_SUBDEV_AXIGATE);
>> + }
>> +}
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (16 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 17/18] fpga: xrt: partition isolation " Lizhi Hou
@ 2021-02-18 6:40 ` Lizhi Hou
2021-02-18 9:02 ` kernel test robot
` (2 more replies)
2021-02-18 13:52 ` [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Tom Rix
2021-02-21 20:43 ` Moritz Fischer
19 siblings, 3 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-18 6:40 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, mdf, robh, Max Zhen
Update fpga Kconfig/Makefile and add Kconfig/Makefile for new drivers.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
---
MAINTAINERS | 11 +++++++++++
drivers/Makefile | 1 +
drivers/fpga/Kconfig | 2 ++
drivers/fpga/Makefile | 4 ++++
drivers/fpga/xrt/Kconfig | 8 ++++++++
drivers/fpga/xrt/lib/Kconfig | 16 ++++++++++++++++
drivers/fpga/xrt/lib/Makefile | 30 ++++++++++++++++++++++++++++++
drivers/fpga/xrt/metadata/Kconfig | 12 ++++++++++++
drivers/fpga/xrt/metadata/Makefile | 16 ++++++++++++++++
drivers/fpga/xrt/mgmt/Kconfig | 15 +++++++++++++++
drivers/fpga/xrt/mgmt/Makefile | 19 +++++++++++++++++++
11 files changed, 134 insertions(+)
create mode 100644 drivers/fpga/xrt/Kconfig
create mode 100644 drivers/fpga/xrt/lib/Kconfig
create mode 100644 drivers/fpga/xrt/lib/Makefile
create mode 100644 drivers/fpga/xrt/metadata/Kconfig
create mode 100644 drivers/fpga/xrt/metadata/Makefile
create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
create mode 100644 drivers/fpga/xrt/mgmt/Makefile
diff --git a/MAINTAINERS b/MAINTAINERS
index d3e847f7f3dc..e6e147c2454c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6973,6 +6973,17 @@ F: Documentation/fpga/
F: drivers/fpga/
F: include/linux/fpga/
+FPGA XRT DRIVERS
+M: Lizhi Hou <lizhi.hou@xilinx.com>
+R: Max Zhen <max.zhen@xilinx.com>
+R: Sonal Santan <sonal.santan@xilinx.com>
+L: linux-fpga@vger.kernel.org
+S: Maintained
+W: https://github.com/Xilinx/XRT
+F: Documentation/fpga/xrt.rst
+F: drivers/fpga/xrt/
+F: include/uapi/linux/xrt/
+
FPU EMULATOR
M: Bill Metzenthen <billm@melbpc.org.au>
S: Maintained
diff --git a/drivers/Makefile b/drivers/Makefile
index fd11b9ac4cc3..e03912af8e48 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -178,6 +178,7 @@ obj-$(CONFIG_STM) += hwtracing/stm/
obj-$(CONFIG_ANDROID) += android/
obj-$(CONFIG_NVMEM) += nvmem/
obj-$(CONFIG_FPGA) += fpga/
+obj-y += fpga/xrt/metadata/
obj-$(CONFIG_FSI) += fsi/
obj-$(CONFIG_TEE) += tee/
obj-$(CONFIG_MULTIPLEXER) += mux/
diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 5645226ca3ce..aeca635b1f25 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -216,4 +216,6 @@ config FPGA_MGR_ZYNQMP_FPGA
to configure the programmable logic(PL) through PS
on ZynqMP SoC.
+source "drivers/fpga/xrt/Kconfig"
+
endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index d8e21dfc6778..2b4453ff7c52 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -46,3 +46,7 @@ dfl-afu-objs += dfl-afu-error.o
# Drivers for FPGAs which implement DFL
obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
+
+# XRT drivers for Alveo
+obj-$(CONFIG_FPGA_XRT_LIB) += xrt/lib/
+obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt/mgmt/
diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig
new file mode 100644
index 000000000000..0e2c59589ddd
--- /dev/null
+++ b/drivers/fpga/xrt/Kconfig
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Xilinx Alveo FPGA device configuration
+#
+
+source "drivers/fpga/xrt/metadata/Kconfig"
+source "drivers/fpga/xrt/lib/Kconfig"
+source "drivers/fpga/xrt/mgmt/Kconfig"
diff --git a/drivers/fpga/xrt/lib/Kconfig b/drivers/fpga/xrt/lib/Kconfig
new file mode 100644
index 000000000000..eed5cb73f5e2
--- /dev/null
+++ b/drivers/fpga/xrt/lib/Kconfig
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# XRT Alveo FPGA device configuration
+#
+
+config FPGA_XRT_LIB
+ tristate "XRT Alveo Driver Library"
+ depends on HWMON && PCI && HAS_IOMEM
+ select FPGA_XRT_METADATA
+ help
+ Select this option to enable Xilinx XRT Alveo driver library. This
+ library is core infrastructure of XRT Alveo FPGA drivers which
+ provides functions for working with device nodes, iteration and
+ lookup of platform devices, common interfaces for platform devices,
+ plumbing of function call and ioctls between platform devices and
+ parent partitions.
diff --git a/drivers/fpga/xrt/lib/Makefile b/drivers/fpga/xrt/lib/Makefile
new file mode 100644
index 000000000000..5641231b2a36
--- /dev/null
+++ b/drivers/fpga/xrt/lib/Makefile
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
+#
+# Authors: Sonal.Santan@xilinx.com
+#
+
+FULL_XRT_PATH=$(srctree)/$(src)/..
+FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
+
+obj-$(CONFIG_FPGA_XRT_LIB) += xrt-lib.o
+
+xrt-lib-objs := \
+ main.o \
+ xroot.o \
+ xclbin.o \
+ subdev.o \
+ cdev.o \
+ group.o \
+ xleaf/vsec.o \
+ xleaf/axigate.o \
+ xleaf/devctl.o \
+ xleaf/icap.o \
+ xleaf/clock.o \
+ xleaf/clkfreq.o \
+ xleaf/ucs.o \
+ xleaf/calib.o \
+
+ccflags-y := -I$(FULL_XRT_PATH)/include \
+ -I$(FULL_DTC_PATH)
diff --git a/drivers/fpga/xrt/metadata/Kconfig b/drivers/fpga/xrt/metadata/Kconfig
new file mode 100644
index 000000000000..5012c9c6584d
--- /dev/null
+++ b/drivers/fpga/xrt/metadata/Kconfig
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# XRT Alveo FPGA device configuration
+#
+
+config FPGA_XRT_METADATA
+ bool "XRT Alveo Driver Metadata Parser"
+ select LIBFDT
+ help
+ This option provides helper functions to parse Xilinx Alveo FPGA
+ firmware metadata. The metadata is in device tree format and XRT
+ driver uses it to discover HW subsystems behind PCIe BAR.
diff --git a/drivers/fpga/xrt/metadata/Makefile b/drivers/fpga/xrt/metadata/Makefile
new file mode 100644
index 000000000000..14f65ef1595c
--- /dev/null
+++ b/drivers/fpga/xrt/metadata/Makefile
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
+#
+# Authors: Sonal.Santan@xilinx.com
+#
+
+FULL_XRT_PATH=$(srctree)/$(src)/..
+FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
+
+obj-$(CONFIG_FPGA_XRT_METADATA) += xrt-md.o
+
+xrt-md-objs := metadata.o
+
+ccflags-y := -I$(FULL_XRT_PATH)/include \
+ -I$(FULL_DTC_PATH)
diff --git a/drivers/fpga/xrt/mgmt/Kconfig b/drivers/fpga/xrt/mgmt/Kconfig
new file mode 100644
index 000000000000..2b2a2c34685c
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/Kconfig
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Xilinx XRT FPGA device configuration
+#
+
+config FPGA_XRT_XMGMT
+ tristate "Xilinx Alveo Management Driver"
+ depends on HWMON && PCI && FPGA_XRT_LIB
+ select FPGA_XRT_METADATA
+ select FPGA_BRIDGE
+ select FPGA_REGION
+ help
+ Select this option to enable XRT PCIe driver for Xilinx Alveo FPGA.
+ This driver provides interfaces for userspace application to access
+ Alveo FPGA device.
diff --git a/drivers/fpga/xrt/mgmt/Makefile b/drivers/fpga/xrt/mgmt/Makefile
new file mode 100644
index 000000000000..8051708c361c
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/Makefile
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
+#
+# Authors: Sonal.Santan@xilinx.com
+#
+
+FULL_XRT_PATH=$(srctree)/$(src)/..
+FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
+
+obj-$(CONFIG_FPGA_XRT_XMGMT) += xmgmt.o
+
+xmgmt-objs := root.o \
+ main.o \
+ fmgr-drv.o \
+ main-region.o
+
+ccflags-y := -I$(FULL_XRT_PATH)/include \
+ -I$(FULL_DTC_PATH)
--
2.18.4
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers Lizhi Hou
@ 2021-02-18 9:02 ` kernel test robot
2021-02-18 19:50 ` kernel test robot
2021-02-21 14:57 ` Tom Rix
2 siblings, 0 replies; 87+ messages in thread
From: kernel test robot @ 2021-02-18 9:02 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: kbuild-all, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, mdf
[-- Attachment #1: Type: text/plain, Size: 5979 bytes --]
Hi Lizhi,
I love your patch! Yet something to improve:
[auto build test ERROR on linux/master]
[also build test ERROR on linus/master v5.11 next-20210217]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Lizhi-Hou/XRT-Alveo-driver-overview/20210218-151631
base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 2ab38c17aac10bf55ab3efde4c4db3893d8691d2
config: parisc-randconfig-r025-20210218 (attached as .config)
compiler: hppa64-linux-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/9069f42b5fcafe471ceb1047ee53983098b2edba
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Lizhi-Hou/XRT-Alveo-driver-overview/20210218-151631
git checkout 9069f42b5fcafe471ceb1047ee53983098b2edba
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=parisc
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All error/warnings (new ones prefixed by >>):
drivers/fpga/xrt/lib/main.c: In function 'xleaf_register_driver':
>> drivers/fpga/xrt/lib/main.c:144:8: error: implicit declaration of function 'vzalloc'; did you mean 'kvzalloc'? [-Werror=implicit-function-declaration]
144 | map = vzalloc(sizeof(*map));
| ^~~~~~~
| kvzalloc
>> drivers/fpga/xrt/lib/main.c:144:6: warning: assignment to 'struct xrt_drv_map *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
144 | map = vzalloc(sizeof(*map));
| ^
drivers/fpga/xrt/lib/main.c: In function 'xleaf_unregister_driver':
>> drivers/fpga/xrt/lib/main.c:181:2: error: implicit declaration of function 'vfree'; did you mean 'kvfree'? [-Werror=implicit-function-declaration]
181 | vfree(map);
| ^~~~~
| kvfree
cc1: some warnings being treated as errors
vim +144 drivers/fpga/xrt/lib/main.c
898da78383bc51 Lizhi Hou 2021-02-17 127
898da78383bc51 Lizhi Hou 2021-02-17 128 int xleaf_register_driver(enum xrt_subdev_id id,
898da78383bc51 Lizhi Hou 2021-02-17 129 struct platform_driver *drv,
898da78383bc51 Lizhi Hou 2021-02-17 130 struct xrt_subdev_endpoints *eps)
898da78383bc51 Lizhi Hou 2021-02-17 131 {
898da78383bc51 Lizhi Hou 2021-02-17 132 struct xrt_drv_map *map;
898da78383bc51 Lizhi Hou 2021-02-17 133
898da78383bc51 Lizhi Hou 2021-02-17 134 mutex_lock(&xrt_lib_lock);
898da78383bc51 Lizhi Hou 2021-02-17 135
898da78383bc51 Lizhi Hou 2021-02-17 136 map = xrt_drv_find_map_by_id_nolock(id);
898da78383bc51 Lizhi Hou 2021-02-17 137 if (map) {
898da78383bc51 Lizhi Hou 2021-02-17 138 mutex_unlock(&xrt_lib_lock);
898da78383bc51 Lizhi Hou 2021-02-17 139 pr_err("Id %d already has a registered driver, 0x%p\n",
898da78383bc51 Lizhi Hou 2021-02-17 140 id, map->drv);
898da78383bc51 Lizhi Hou 2021-02-17 141 return -EEXIST;
898da78383bc51 Lizhi Hou 2021-02-17 142 }
898da78383bc51 Lizhi Hou 2021-02-17 143
898da78383bc51 Lizhi Hou 2021-02-17 @144 map = vzalloc(sizeof(*map));
898da78383bc51 Lizhi Hou 2021-02-17 145 if (!map) {
898da78383bc51 Lizhi Hou 2021-02-17 146 mutex_unlock(&xrt_lib_lock);
898da78383bc51 Lizhi Hou 2021-02-17 147 return -ENOMEM;
898da78383bc51 Lizhi Hou 2021-02-17 148 }
898da78383bc51 Lizhi Hou 2021-02-17 149 map->id = id;
898da78383bc51 Lizhi Hou 2021-02-17 150 map->drv = drv;
898da78383bc51 Lizhi Hou 2021-02-17 151 map->eps = eps;
898da78383bc51 Lizhi Hou 2021-02-17 152
898da78383bc51 Lizhi Hou 2021-02-17 153 xrt_drv_register_driver(map);
898da78383bc51 Lizhi Hou 2021-02-17 154
898da78383bc51 Lizhi Hou 2021-02-17 155 list_add(&map->list, &xrt_drv_maps);
898da78383bc51 Lizhi Hou 2021-02-17 156
898da78383bc51 Lizhi Hou 2021-02-17 157 mutex_unlock(&xrt_lib_lock);
898da78383bc51 Lizhi Hou 2021-02-17 158
898da78383bc51 Lizhi Hou 2021-02-17 159 return 0;
898da78383bc51 Lizhi Hou 2021-02-17 160 }
898da78383bc51 Lizhi Hou 2021-02-17 161 EXPORT_SYMBOL_GPL(xleaf_register_driver);
898da78383bc51 Lizhi Hou 2021-02-17 162
898da78383bc51 Lizhi Hou 2021-02-17 163 void xleaf_unregister_driver(enum xrt_subdev_id id)
898da78383bc51 Lizhi Hou 2021-02-17 164 {
898da78383bc51 Lizhi Hou 2021-02-17 165 struct xrt_drv_map *map;
898da78383bc51 Lizhi Hou 2021-02-17 166
898da78383bc51 Lizhi Hou 2021-02-17 167 mutex_lock(&xrt_lib_lock);
898da78383bc51 Lizhi Hou 2021-02-17 168
898da78383bc51 Lizhi Hou 2021-02-17 169 map = xrt_drv_find_map_by_id_nolock(id);
898da78383bc51 Lizhi Hou 2021-02-17 170 if (!map) {
898da78383bc51 Lizhi Hou 2021-02-17 171 mutex_unlock(&xrt_lib_lock);
898da78383bc51 Lizhi Hou 2021-02-17 172 pr_err("Id %d has no registered driver\n", id);
898da78383bc51 Lizhi Hou 2021-02-17 173 return;
898da78383bc51 Lizhi Hou 2021-02-17 174 }
898da78383bc51 Lizhi Hou 2021-02-17 175
898da78383bc51 Lizhi Hou 2021-02-17 176 list_del(&map->list);
898da78383bc51 Lizhi Hou 2021-02-17 177
898da78383bc51 Lizhi Hou 2021-02-17 178 mutex_unlock(&xrt_lib_lock);
898da78383bc51 Lizhi Hou 2021-02-17 179
898da78383bc51 Lizhi Hou 2021-02-17 180 xrt_drv_unregister_driver(map);
898da78383bc51 Lizhi Hou 2021-02-17 @181 vfree(map);
898da78383bc51 Lizhi Hou 2021-02-17 182 }
898da78383bc51 Lizhi Hou 2021-02-17 183 EXPORT_SYMBOL_GPL(xleaf_unregister_driver);
898da78383bc51 Lizhi Hou 2021-02-17 184
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 37024 bytes --]
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers Lizhi Hou
2021-02-18 9:02 ` kernel test robot
@ 2021-02-18 19:50 ` kernel test robot
2021-02-21 14:57 ` Tom Rix
2 siblings, 0 replies; 87+ messages in thread
From: kernel test robot @ 2021-02-18 19:50 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: kbuild-all, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, mdf
[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]
Hi Lizhi,
I love your patch! Yet something to improve:
[auto build test ERROR on linux/master]
[also build test ERROR on linus/master v5.11 next-20210218]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Lizhi-Hou/XRT-Alveo-driver-overview/20210218-151631
base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 2ab38c17aac10bf55ab3efde4c4db3893d8691d2
config: arm-allyesconfig (attached as .config)
compiler: arm-linux-gnueabi-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/9069f42b5fcafe471ceb1047ee53983098b2edba
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Lizhi-Hou/XRT-Alveo-driver-overview/20210218-151631
git checkout 9069f42b5fcafe471ceb1047ee53983098b2edba
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arm
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
arm-linux-gnueabi-ld: drivers/fpga/xrt/lib/xleaf/clock.o: in function `get_freq':
>> clock.c:(.text+0x228): undefined reference to `__aeabi_uldivmod'
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 77942 bytes --]
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers Lizhi Hou
2021-02-18 9:02 ` kernel test robot
2021-02-18 19:50 ` kernel test robot
@ 2021-02-21 14:57 ` Tom Rix
2021-02-21 18:39 ` Moritz Fischer
2 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-21 14:57 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel, mdf
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh, Max Zhen
As I am looking through the files, I have this comment.
fpga/ is currently a single directory, while files could be organized in subdirectories like
dfl/pci.c
instead have the possible subdir name as a prefix to the filename.
dfl-pci.c
For consistency,
xrt/metadata/metadata.c
should be
xrt-metadata.c
Likewise the build infra needs to integrated within the existing files fpga/Kconfig,Makefile
This is a bigish refactor, so let's get a second opinion.
Moritz ?
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Update fpga Kconfig/Makefile and add Kconfig/Makefile for new drivers.
Expand the comment, there are several new configs that could use an explanation
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> ---
> MAINTAINERS | 11 +++++++++++
> drivers/Makefile | 1 +
> drivers/fpga/Kconfig | 2 ++
> drivers/fpga/Makefile | 4 ++++
> drivers/fpga/xrt/Kconfig | 8 ++++++++
> drivers/fpga/xrt/lib/Kconfig | 16 ++++++++++++++++
> drivers/fpga/xrt/lib/Makefile | 30 ++++++++++++++++++++++++++++++
> drivers/fpga/xrt/metadata/Kconfig | 12 ++++++++++++
> drivers/fpga/xrt/metadata/Makefile | 16 ++++++++++++++++
> drivers/fpga/xrt/mgmt/Kconfig | 15 +++++++++++++++
> drivers/fpga/xrt/mgmt/Makefile | 19 +++++++++++++++++++
> 11 files changed, 134 insertions(+)
> create mode 100644 drivers/fpga/xrt/Kconfig
> create mode 100644 drivers/fpga/xrt/lib/Kconfig
> create mode 100644 drivers/fpga/xrt/lib/Makefile
> create mode 100644 drivers/fpga/xrt/metadata/Kconfig
> create mode 100644 drivers/fpga/xrt/metadata/Makefile
> create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
> create mode 100644 drivers/fpga/xrt/mgmt/Makefile
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index d3e847f7f3dc..e6e147c2454c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6973,6 +6973,17 @@ F: Documentation/fpga/
> F: drivers/fpga/
> F: include/linux/fpga/
>
> +FPGA XRT DRIVERS
> +M: Lizhi Hou <lizhi.hou@xilinx.com>
> +R: Max Zhen <max.zhen@xilinx.com>
> +R: Sonal Santan <sonal.santan@xilinx.com>
> +L: linux-fpga@vger.kernel.org
> +S: Maintained
> +W: https://github.com/Xilinx/XRT
> +F: Documentation/fpga/xrt.rst
> +F: drivers/fpga/xrt/
> +F: include/uapi/linux/xrt/
> +
> FPU EMULATOR
> M: Bill Metzenthen <billm@melbpc.org.au>
> S: Maintained
> diff --git a/drivers/Makefile b/drivers/Makefile
> index fd11b9ac4cc3..e03912af8e48 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -178,6 +178,7 @@ obj-$(CONFIG_STM) += hwtracing/stm/
> obj-$(CONFIG_ANDROID) += android/
> obj-$(CONFIG_NVMEM) += nvmem/
> obj-$(CONFIG_FPGA) += fpga/
> +obj-y += fpga/xrt/metadata/
This is wrong.
Move metadata building to fpga/ Makefile and pick an appropriate config, not just 'obj-y'
> obj-$(CONFIG_FSI) += fsi/
> obj-$(CONFIG_TEE) += tee/
> obj-$(CONFIG_MULTIPLEXER) += mux/
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index 5645226ca3ce..aeca635b1f25 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -216,4 +216,6 @@ config FPGA_MGR_ZYNQMP_FPGA
> to configure the programmable logic(PL) through PS
> on ZynqMP SoC.
>
> +source "drivers/fpga/xrt/Kconfig"
> +
> endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index d8e21dfc6778..2b4453ff7c52 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -46,3 +46,7 @@ dfl-afu-objs += dfl-afu-error.o
>
> # Drivers for FPGAs which implement DFL
> obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
> +
> +# XRT drivers for Alveo
> +obj-$(CONFIG_FPGA_XRT_LIB) += xrt/lib/
> +obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt/mgmt/
I don't see how mgmnt would work without lib. If that is so
these configs could collapse to CONFIG_FPGA_XRT
> diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig
> new file mode 100644
> index 000000000000..0e2c59589ddd
> --- /dev/null
> +++ b/drivers/fpga/xrt/Kconfig
> @@ -0,0 +1,8 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# Xilinx Alveo FPGA device configuration
> +#
> +
> +source "drivers/fpga/xrt/metadata/Kconfig"
> +source "drivers/fpga/xrt/lib/Kconfig"
> +source "drivers/fpga/xrt/mgmt/Kconfig"
> diff --git a/drivers/fpga/xrt/lib/Kconfig b/drivers/fpga/xrt/lib/Kconfig
> new file mode 100644
> index 000000000000..eed5cb73f5e2
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/Kconfig
> @@ -0,0 +1,16 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# XRT Alveo FPGA device configuration
> +#
> +
> +config FPGA_XRT_LIB
> + tristate "XRT Alveo Driver Library"
> + depends on HWMON && PCI && HAS_IOMEM
> + select FPGA_XRT_METADATA
> + help
> + Select this option to enable Xilinx XRT Alveo driver library. This
> + library is core infrastructure of XRT Alveo FPGA drivers which
> + provides functions for working with device nodes, iteration and
> + lookup of platform devices, common interfaces for platform devices,
> + plumbing of function call and ioctls between platform devices and
> + parent partitions.
> diff --git a/drivers/fpga/xrt/lib/Makefile b/drivers/fpga/xrt/lib/Makefile
> new file mode 100644
> index 000000000000..5641231b2a36
> --- /dev/null
> +++ b/drivers/fpga/xrt/lib/Makefile
> @@ -0,0 +1,30 @@
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> +#
> +# Authors: Sonal.Santan@xilinx.com
> +#
> +
> +FULL_XRT_PATH=$(srctree)/$(src)/..
> +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> +
> +obj-$(CONFIG_FPGA_XRT_LIB) += xrt-lib.o
> +
> +xrt-lib-objs := \
> + main.o \
> + xroot.o \
> + xclbin.o \
> + subdev.o \
> + cdev.o \
> + group.o \
> + xleaf/vsec.o \
> + xleaf/axigate.o \
> + xleaf/devctl.o \
> + xleaf/icap.o \
> + xleaf/clock.o \
> + xleaf/clkfreq.o \
> + xleaf/ucs.o \
> + xleaf/calib.o \
> +
> +ccflags-y := -I$(FULL_XRT_PATH)/include \
> + -I$(FULL_DTC_PATH)
> diff --git a/drivers/fpga/xrt/metadata/Kconfig b/drivers/fpga/xrt/metadata/Kconfig
> new file mode 100644
> index 000000000000..5012c9c6584d
> --- /dev/null
> +++ b/drivers/fpga/xrt/metadata/Kconfig
> @@ -0,0 +1,12 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# XRT Alveo FPGA device configuration
> +#
> +
> +config FPGA_XRT_METADATA
> + bool "XRT Alveo Driver Metadata Parser"
> + select LIBFDT
> + help
> + This option provides helper functions to parse Xilinx Alveo FPGA
> + firmware metadata. The metadata is in device tree format and XRT
and the XRT
> + driver uses it to discover HW subsystems behind PCIe BAR.
the HW
> diff --git a/drivers/fpga/xrt/metadata/Makefile b/drivers/fpga/xrt/metadata/Makefile
> new file mode 100644
> index 000000000000..14f65ef1595c
> --- /dev/null
> +++ b/drivers/fpga/xrt/metadata/Makefile
> @@ -0,0 +1,16 @@
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> +#
> +# Authors: Sonal.Santan@xilinx.com
> +#
> +
> +FULL_XRT_PATH=$(srctree)/$(src)/..
> +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> +
> +obj-$(CONFIG_FPGA_XRT_METADATA) += xrt-md.o
> +
> +xrt-md-objs := metadata.o
> +
> +ccflags-y := -I$(FULL_XRT_PATH)/include \
> + -I$(FULL_DTC_PATH)
> diff --git a/drivers/fpga/xrt/mgmt/Kconfig b/drivers/fpga/xrt/mgmt/Kconfig
> new file mode 100644
> index 000000000000..2b2a2c34685c
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/Kconfig
> @@ -0,0 +1,15 @@
> +# SPDX-License-Identifier: GPL-2.0-only
> +#
> +# Xilinx XRT FPGA device configuration
> +#
> +
> +config FPGA_XRT_XMGMT
> + tristate "Xilinx Alveo Management Driver"
> + depends on HWMON && PCI && FPGA_XRT_LIB
FPGA_XRT_LIB also depends on HWMON and PCI, so this could be minimized.
Tom
> + select FPGA_XRT_METADATA
> + select FPGA_BRIDGE
> + select FPGA_REGION
> + help
> + Select this option to enable XRT PCIe driver for Xilinx Alveo FPGA.
> + This driver provides interfaces for userspace application to access
> + Alveo FPGA device.
> diff --git a/drivers/fpga/xrt/mgmt/Makefile b/drivers/fpga/xrt/mgmt/Makefile
> new file mode 100644
> index 000000000000..8051708c361c
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/Makefile
> @@ -0,0 +1,19 @@
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> +#
> +# Authors: Sonal.Santan@xilinx.com
> +#
> +
> +FULL_XRT_PATH=$(srctree)/$(src)/..
> +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> +
> +obj-$(CONFIG_FPGA_XRT_XMGMT) += xmgmt.o
> +
> +xmgmt-objs := root.o \
> + main.o \
> + fmgr-drv.o \
> + main-region.o
> +
> +ccflags-y := -I$(FULL_XRT_PATH)/include \
> + -I$(FULL_DTC_PATH)
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers
2021-02-21 14:57 ` Tom Rix
@ 2021-02-21 18:39 ` Moritz Fischer
2021-02-28 20:52 ` Sonal Santan
0 siblings, 1 reply; 87+ messages in thread
From: Moritz Fischer @ 2021-02-21 18:39 UTC (permalink / raw)
To: Tom Rix
Cc: Lizhi Hou, linux-kernel, mdf, Lizhi Hou, linux-fpga, maxz,
sonal.santan, michal.simek, stefanos, devicetree, robh, Max Zhen
On Sun, Feb 21, 2021 at 06:57:31AM -0800, Tom Rix wrote:
> As I am looking through the files, I have this comment.
>
> fpga/ is currently a single directory, while files could be organized in subdirectories like
>
> dfl/pci.c
>
> instead have the possible subdir name as a prefix to the filename.
>
> dfl-pci.c
>
> For consistency,
>
> xrt/metadata/metadata.c
>
> should be
>
> xrt-metadata.c
Agreed. Keep the prefix.
>
> Likewise the build infra needs to integrated within the existing files fpga/Kconfig,Makefile
>
> This is a bigish refactor, so let's get a second opinion.
In what sense? You mean adding a subdirectory? Maybe something like this
drivers/fpga
- dfl/
- xilinx/
- intel/
- lattice/
- xrt/
...
would generally make sense.
We didn't have enough drivers to prioritize that yet, but we can look
into it.
>
> Moritz ?
>
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
> > Update fpga Kconfig/Makefile and add Kconfig/Makefile for new drivers.
> Expand the comment, there are several new configs that could use an explanation
> >
> > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> > Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> > ---
> > MAINTAINERS | 11 +++++++++++
> > drivers/Makefile | 1 +
> > drivers/fpga/Kconfig | 2 ++
> > drivers/fpga/Makefile | 4 ++++
> > drivers/fpga/xrt/Kconfig | 8 ++++++++
> > drivers/fpga/xrt/lib/Kconfig | 16 ++++++++++++++++
> > drivers/fpga/xrt/lib/Makefile | 30 ++++++++++++++++++++++++++++++
> > drivers/fpga/xrt/metadata/Kconfig | 12 ++++++++++++
> > drivers/fpga/xrt/metadata/Makefile | 16 ++++++++++++++++
> > drivers/fpga/xrt/mgmt/Kconfig | 15 +++++++++++++++
> > drivers/fpga/xrt/mgmt/Makefile | 19 +++++++++++++++++++
> > 11 files changed, 134 insertions(+)
> > create mode 100644 drivers/fpga/xrt/Kconfig
> > create mode 100644 drivers/fpga/xrt/lib/Kconfig
> > create mode 100644 drivers/fpga/xrt/lib/Makefile
> > create mode 100644 drivers/fpga/xrt/metadata/Kconfig
> > create mode 100644 drivers/fpga/xrt/metadata/Makefile
> > create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
> > create mode 100644 drivers/fpga/xrt/mgmt/Makefile
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index d3e847f7f3dc..e6e147c2454c 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -6973,6 +6973,17 @@ F: Documentation/fpga/
> > F: drivers/fpga/
> > F: include/linux/fpga/
> >
> > +FPGA XRT DRIVERS
> > +M: Lizhi Hou <lizhi.hou@xilinx.com>
> > +R: Max Zhen <max.zhen@xilinx.com>
> > +R: Sonal Santan <sonal.santan@xilinx.com>
> > +L: linux-fpga@vger.kernel.org
> > +S: Maintained
> > +W: https://github.com/Xilinx/XRT
> > +F: Documentation/fpga/xrt.rst
> > +F: drivers/fpga/xrt/
> > +F: include/uapi/linux/xrt/
> > +
> > FPU EMULATOR
> > M: Bill Metzenthen <billm@melbpc.org.au>
> > S: Maintained
> > diff --git a/drivers/Makefile b/drivers/Makefile
> > index fd11b9ac4cc3..e03912af8e48 100644
> > --- a/drivers/Makefile
> > +++ b/drivers/Makefile
> > @@ -178,6 +178,7 @@ obj-$(CONFIG_STM) += hwtracing/stm/
> > obj-$(CONFIG_ANDROID) += android/
> > obj-$(CONFIG_NVMEM) += nvmem/
> > obj-$(CONFIG_FPGA) += fpga/
> > +obj-y += fpga/xrt/metadata/
>
> This is wrong.
>
> Move metadata building to fpga/ Makefile and pick an appropriate config, not just 'obj-y'
>
> > obj-$(CONFIG_FSI) += fsi/
> > obj-$(CONFIG_TEE) += tee/
> > obj-$(CONFIG_MULTIPLEXER) += mux/
> > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> > index 5645226ca3ce..aeca635b1f25 100644
> > --- a/drivers/fpga/Kconfig
> > +++ b/drivers/fpga/Kconfig
> > @@ -216,4 +216,6 @@ config FPGA_MGR_ZYNQMP_FPGA
> > to configure the programmable logic(PL) through PS
> > on ZynqMP SoC.
> >
> > +source "drivers/fpga/xrt/Kconfig"
> > +
> > endif # FPGA
> > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> > index d8e21dfc6778..2b4453ff7c52 100644
> > --- a/drivers/fpga/Makefile
> > +++ b/drivers/fpga/Makefile
> > @@ -46,3 +46,7 @@ dfl-afu-objs += dfl-afu-error.o
> >
> > # Drivers for FPGAs which implement DFL
> > obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
> > +
> > +# XRT drivers for Alveo
> > +obj-$(CONFIG_FPGA_XRT_LIB) += xrt/lib/
> > +obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt/mgmt/
>
> I don't see how mgmnt would work without lib. If that is so
>
> these configs could collapse to CONFIG_FPGA_XRT
>
> > diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig
> > new file mode 100644
> > index 000000000000..0e2c59589ddd
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/Kconfig
> > @@ -0,0 +1,8 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +# Xilinx Alveo FPGA device configuration
> > +#
> > +
> > +source "drivers/fpga/xrt/metadata/Kconfig"
> > +source "drivers/fpga/xrt/lib/Kconfig"
> > +source "drivers/fpga/xrt/mgmt/Kconfig"
> > diff --git a/drivers/fpga/xrt/lib/Kconfig b/drivers/fpga/xrt/lib/Kconfig
> > new file mode 100644
> > index 000000000000..eed5cb73f5e2
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/lib/Kconfig
> > @@ -0,0 +1,16 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +# XRT Alveo FPGA device configuration
> > +#
> > +
> > +config FPGA_XRT_LIB
> > + tristate "XRT Alveo Driver Library"
> > + depends on HWMON && PCI && HAS_IOMEM
> > + select FPGA_XRT_METADATA
> > + help
> > + Select this option to enable Xilinx XRT Alveo driver library. This
> > + library is core infrastructure of XRT Alveo FPGA drivers which
> > + provides functions for working with device nodes, iteration and
> > + lookup of platform devices, common interfaces for platform devices,
> > + plumbing of function call and ioctls between platform devices and
> > + parent partitions.
> > diff --git a/drivers/fpga/xrt/lib/Makefile b/drivers/fpga/xrt/lib/Makefile
> > new file mode 100644
> > index 000000000000..5641231b2a36
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/lib/Makefile
> > @@ -0,0 +1,30 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> > +#
> > +# Authors: Sonal.Santan@xilinx.com
> > +#
> > +
> > +FULL_XRT_PATH=$(srctree)/$(src)/..
> > +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> > +
> > +obj-$(CONFIG_FPGA_XRT_LIB) += xrt-lib.o
> > +
> > +xrt-lib-objs := \
> > + main.o \
> > + xroot.o \
> > + xclbin.o \
> > + subdev.o \
> > + cdev.o \
> > + group.o \
> > + xleaf/vsec.o \
> > + xleaf/axigate.o \
> > + xleaf/devctl.o \
> > + xleaf/icap.o \
> > + xleaf/clock.o \
> > + xleaf/clkfreq.o \
> > + xleaf/ucs.o \
> > + xleaf/calib.o \
> > +
> > +ccflags-y := -I$(FULL_XRT_PATH)/include \
> > + -I$(FULL_DTC_PATH)
> > diff --git a/drivers/fpga/xrt/metadata/Kconfig b/drivers/fpga/xrt/metadata/Kconfig
> > new file mode 100644
> > index 000000000000..5012c9c6584d
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/metadata/Kconfig
> > @@ -0,0 +1,12 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +# XRT Alveo FPGA device configuration
> > +#
> > +
> > +config FPGA_XRT_METADATA
> > + bool "XRT Alveo Driver Metadata Parser"
> > + select LIBFDT
> > + help
> > + This option provides helper functions to parse Xilinx Alveo FPGA
> > + firmware metadata. The metadata is in device tree format and XRT
> and the XRT
> > + driver uses it to discover HW subsystems behind PCIe BAR.
> the HW
> > diff --git a/drivers/fpga/xrt/metadata/Makefile b/drivers/fpga/xrt/metadata/Makefile
> > new file mode 100644
> > index 000000000000..14f65ef1595c
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/metadata/Makefile
> > @@ -0,0 +1,16 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> > +#
> > +# Authors: Sonal.Santan@xilinx.com
> > +#
> > +
> > +FULL_XRT_PATH=$(srctree)/$(src)/..
> > +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> > +
> > +obj-$(CONFIG_FPGA_XRT_METADATA) += xrt-md.o
> > +
> > +xrt-md-objs := metadata.o
> > +
> > +ccflags-y := -I$(FULL_XRT_PATH)/include \
> > + -I$(FULL_DTC_PATH)
> > diff --git a/drivers/fpga/xrt/mgmt/Kconfig b/drivers/fpga/xrt/mgmt/Kconfig
> > new file mode 100644
> > index 000000000000..2b2a2c34685c
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/mgmt/Kconfig
> > @@ -0,0 +1,15 @@
> > +# SPDX-License-Identifier: GPL-2.0-only
> > +#
> > +# Xilinx XRT FPGA device configuration
> > +#
> > +
> > +config FPGA_XRT_XMGMT
> > + tristate "Xilinx Alveo Management Driver"
> > + depends on HWMON && PCI && FPGA_XRT_LIB
>
> FPGA_XRT_LIB also depends on HWMON and PCI, so this could be minimized.
>
> Tom
>
> > + select FPGA_XRT_METADATA
> > + select FPGA_BRIDGE
> > + select FPGA_REGION
> > + help
> > + Select this option to enable XRT PCIe driver for Xilinx Alveo FPGA.
> > + This driver provides interfaces for userspace application to access
> > + Alveo FPGA device.
> > diff --git a/drivers/fpga/xrt/mgmt/Makefile b/drivers/fpga/xrt/mgmt/Makefile
> > new file mode 100644
> > index 000000000000..8051708c361c
> > --- /dev/null
> > +++ b/drivers/fpga/xrt/mgmt/Makefile
> > @@ -0,0 +1,19 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +#
> > +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> > +#
> > +# Authors: Sonal.Santan@xilinx.com
> > +#
> > +
> > +FULL_XRT_PATH=$(srctree)/$(src)/..
> > +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> > +
> > +obj-$(CONFIG_FPGA_XRT_XMGMT) += xmgmt.o
> > +
> > +xmgmt-objs := root.o \
> > + main.o \
> > + fmgr-drv.o \
> > + main-region.o
> > +
> > +ccflags-y := -I$(FULL_XRT_PATH)/include \
> > + -I$(FULL_DTC_PATH)
>
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread
* RE: [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers
2021-02-21 18:39 ` Moritz Fischer
@ 2021-02-28 20:52 ` Sonal Santan
0 siblings, 0 replies; 87+ messages in thread
From: Sonal Santan @ 2021-02-28 20:52 UTC (permalink / raw)
To: Moritz Fischer, Tom Rix
Cc: Lizhi Hou, linux-kernel, Lizhi Hou, linux-fpga, Max Zhen,
Michal Simek, Stefano Stabellini, devicetree, robh, Max Zhen
Hello Moritz,
> -----Original Message-----
> From: Moritz Fischer <mdf@kernel.org>
> Sent: Sunday, February 21, 2021 10:40 AM
> To: Tom Rix <trix@redhat.com>
> Cc: Lizhi Hou <lizhih@xilinx.com>; linux-kernel@vger.kernel.org;
> mdf@kernel.org; Lizhi Hou <lizhih@xilinx.com>; linux-fpga@vger.kernel.org;
> Max Zhen <maxz@xilinx.com>; Sonal Santan <sonals@xilinx.com>; Michal
> Simek <michals@xilinx.com>; Stefano Stabellini <stefanos@xilinx.com>;
> devicetree@vger.kernel.org; robh@kernel.org; Max Zhen <maxz@xilinx.com>
> Subject: Re: [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile
> updates for XRT drivers
>
> On Sun, Feb 21, 2021 at 06:57:31AM -0800, Tom Rix wrote:
> > As I am looking through the files, I have this comment.
> >
> > fpga/ is currently a single directory, while files could be organized
> > in subdirectories like
> >
> > dfl/pci.c
> >
> > instead have the possible subdir name as a prefix to the filename.
> >
> > dfl-pci.c
> >
> > For consistency,
> >
> > xrt/metadata/metadata.c
> >
> > should be
> >
> > xrt-metadata.c
>
> Agreed. Keep the prefix.
> >
> > Likewise the build infra needs to integrated within the existing files
> > fpga/Kconfig,Makefile
> >
> > This is a bigish refactor, so let's get a second opinion.
>
> In what sense? You mean adding a subdirectory? Maybe something like this
>
> drivers/fpga
> - dfl/
> - xilinx/
> - intel/
> - lattice/
> - xrt/
> ...
>
> would generally make sense.
>
> We didn't have enough drivers to prioritize that yet, but we can look into it.
If longer term we would like to reorganize the drivers/fpga directory structure
should we keep the current directory structure for XRT as in the patch series?
> >
> > Moritz ?
> >
> > On 2/17/21 10:40 PM, Lizhi Hou wrote:
> > > Update fpga Kconfig/Makefile and add Kconfig/Makefile for new drivers.
> > Expand the comment, there are several new configs that could use an
> > explanation
> > >
> > > Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> > > Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> > > Signed-off-by: Lizhi Hou <lizhih@xilinx.com>
> > > ---
> > > MAINTAINERS | 11 +++++++++++
> > > drivers/Makefile | 1 +
> > > drivers/fpga/Kconfig | 2 ++
> > > drivers/fpga/Makefile | 4 ++++
> > > drivers/fpga/xrt/Kconfig | 8 ++++++++
> > > drivers/fpga/xrt/lib/Kconfig | 16 ++++++++++++++++
> > > drivers/fpga/xrt/lib/Makefile | 30 ++++++++++++++++++++++++++++++
> > > drivers/fpga/xrt/metadata/Kconfig | 12 ++++++++++++
> > > drivers/fpga/xrt/metadata/Makefile | 16 ++++++++++++++++
> > > drivers/fpga/xrt/mgmt/Kconfig | 15 +++++++++++++++
> > > drivers/fpga/xrt/mgmt/Makefile | 19 +++++++++++++++++++
> > > 11 files changed, 134 insertions(+) create mode 100644
> > > drivers/fpga/xrt/Kconfig create mode 100644
> > > drivers/fpga/xrt/lib/Kconfig create mode 100644
> > > drivers/fpga/xrt/lib/Makefile create mode 100644
> > > drivers/fpga/xrt/metadata/Kconfig create mode 100644
> > > drivers/fpga/xrt/metadata/Makefile
> > > create mode 100644 drivers/fpga/xrt/mgmt/Kconfig create mode
> > > 100644 drivers/fpga/xrt/mgmt/Makefile
> > >
> > > diff --git a/MAINTAINERS b/MAINTAINERS index
> > > d3e847f7f3dc..e6e147c2454c 100644
> > > --- a/MAINTAINERS
> > > +++ b/MAINTAINERS
> > > @@ -6973,6 +6973,17 @@ F: Documentation/fpga/
> > > F: drivers/fpga/
> > > F: include/linux/fpga/
> > >
> > > +FPGA XRT DRIVERS
> > > +M: Lizhi Hou <lizhi.hou@xilinx.com>
> > > +R: Max Zhen <max.zhen@xilinx.com>
> > > +R: Sonal Santan <sonal.santan@xilinx.com>
> > > +L: linux-fpga@vger.kernel.org
> > > +S: Maintained
> > > +W: https://github.com/Xilinx/XRT
> > > +F: Documentation/fpga/xrt.rst
> > > +F: drivers/fpga/xrt/
> > > +F: include/uapi/linux/xrt/
> > > +
> > > FPU EMULATOR
> > > M: Bill Metzenthen <billm@melbpc.org.au>
> > > S: Maintained
> > > diff --git a/drivers/Makefile b/drivers/Makefile index
> > > fd11b9ac4cc3..e03912af8e48 100644
> > > --- a/drivers/Makefile
> > > +++ b/drivers/Makefile
> > > @@ -178,6 +178,7 @@ obj-$(CONFIG_STM) += hwtracing/stm/
> > > obj-$(CONFIG_ANDROID) += android/
> > > obj-$(CONFIG_NVMEM) += nvmem/
> > > obj-$(CONFIG_FPGA) += fpga/
> > > +obj-y += fpga/xrt/metadata/
> >
> > This is wrong.
> >
> > Move metadata building to fpga/ Makefile and pick an appropriate config,
> not just 'obj-y'
Will update.
> >
> > > obj-$(CONFIG_FSI) += fsi/
> > > obj-$(CONFIG_TEE) += tee/
> > > obj-$(CONFIG_MULTIPLEXER) += mux/
> > > diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig index
> > > 5645226ca3ce..aeca635b1f25 100644
> > > --- a/drivers/fpga/Kconfig
> > > +++ b/drivers/fpga/Kconfig
> > > @@ -216,4 +216,6 @@ config FPGA_MGR_ZYNQMP_FPGA
> > > to configure the programmable logic(PL) through PS
> > > on ZynqMP SoC.
> > >
> > > +source "drivers/fpga/xrt/Kconfig"
> > > +
> > > endif # FPGA
> > > diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile index
> > > d8e21dfc6778..2b4453ff7c52 100644
> > > --- a/drivers/fpga/Makefile
> > > +++ b/drivers/fpga/Makefile
> > > @@ -46,3 +46,7 @@ dfl-afu-objs += dfl-afu-error.o
> > >
> > > # Drivers for FPGAs which implement DFL
> > > obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
> > > +
> > > +# XRT drivers for Alveo
> > > +obj-$(CONFIG_FPGA_XRT_LIB) += xrt/lib/
> > > +obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt/mgmt/
> >
> > I don't see how mgmnt would work without lib. If that is so
> >
> > these configs could collapse to CONFIG_FPGA_XRT
> >
> > > diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig new
> > > file mode 100644 index 000000000000..0e2c59589ddd
> > > --- /dev/null
> > > +++ b/drivers/fpga/xrt/Kconfig
> > > @@ -0,0 +1,8 @@
> > > +# SPDX-License-Identifier: GPL-2.0-only # # Xilinx Alveo FPGA
> > > +device configuration #
> > > +
> > > +source "drivers/fpga/xrt/metadata/Kconfig"
> > > +source "drivers/fpga/xrt/lib/Kconfig"
> > > +source "drivers/fpga/xrt/mgmt/Kconfig"
> > > diff --git a/drivers/fpga/xrt/lib/Kconfig
> > > b/drivers/fpga/xrt/lib/Kconfig new file mode 100644 index
> > > 000000000000..eed5cb73f5e2
> > > --- /dev/null
> > > +++ b/drivers/fpga/xrt/lib/Kconfig
> > > @@ -0,0 +1,16 @@
> > > +# SPDX-License-Identifier: GPL-2.0-only # # XRT Alveo FPGA device
> > > +configuration #
> > > +
> > > +config FPGA_XRT_LIB
> > > + tristate "XRT Alveo Driver Library"
> > > + depends on HWMON && PCI && HAS_IOMEM
> > > + select FPGA_XRT_METADATA
> > > + help
> > > + Select this option to enable Xilinx XRT Alveo driver library. This
> > > + library is core infrastructure of XRT Alveo FPGA drivers which
> > > + provides functions for working with device nodes, iteration and
> > > + lookup of platform devices, common interfaces for platform devices,
> > > + plumbing of function call and ioctls between platform devices and
> > > + parent partitions.
> > > diff --git a/drivers/fpga/xrt/lib/Makefile
> > > b/drivers/fpga/xrt/lib/Makefile new file mode 100644 index
> > > 000000000000..5641231b2a36
> > > --- /dev/null
> > > +++ b/drivers/fpga/xrt/lib/Makefile
> > > @@ -0,0 +1,30 @@
> > > +# SPDX-License-Identifier: GPL-2.0
> > > +#
> > > +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> > > +#
> > > +# Authors: Sonal.Santan@xilinx.com
> > > +#
> > > +
> > > +FULL_XRT_PATH=$(srctree)/$(src)/..
> > > +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> > > +
> > > +obj-$(CONFIG_FPGA_XRT_LIB) += xrt-lib.o
> > > +
> > > +xrt-lib-objs := \
> > > + main.o \
> > > + xroot.o \
> > > + xclbin.o \
> > > + subdev.o \
> > > + cdev.o \
> > > + group.o \
> > > + xleaf/vsec.o \
> > > + xleaf/axigate.o \
> > > + xleaf/devctl.o \
> > > + xleaf/icap.o \
> > > + xleaf/clock.o \
> > > + xleaf/clkfreq.o \
> > > + xleaf/ucs.o \
> > > + xleaf/calib.o \
> > > +
> > > +ccflags-y := -I$(FULL_XRT_PATH)/include \
> > > + -I$(FULL_DTC_PATH)
> > > diff --git a/drivers/fpga/xrt/metadata/Kconfig
> > > b/drivers/fpga/xrt/metadata/Kconfig
> > > new file mode 100644
> > > index 000000000000..5012c9c6584d
> > > --- /dev/null
> > > +++ b/drivers/fpga/xrt/metadata/Kconfig
> > > @@ -0,0 +1,12 @@
> > > +# SPDX-License-Identifier: GPL-2.0-only # # XRT Alveo FPGA device
> > > +configuration #
> > > +
> > > +config FPGA_XRT_METADATA
> > > + bool "XRT Alveo Driver Metadata Parser"
> > > + select LIBFDT
> > > + help
> > > + This option provides helper functions to parse Xilinx Alveo FPGA
> > > + firmware metadata. The metadata is in device tree format and XRT
> > and the XRT
> > > + driver uses it to discover HW subsystems behind PCIe BAR.
> > the HW
> > > diff --git a/drivers/fpga/xrt/metadata/Makefile
> > > b/drivers/fpga/xrt/metadata/Makefile
> > > new file mode 100644
> > > index 000000000000..14f65ef1595c
> > > --- /dev/null
> > > +++ b/drivers/fpga/xrt/metadata/Makefile
> > > @@ -0,0 +1,16 @@
> > > +# SPDX-License-Identifier: GPL-2.0
> > > +#
> > > +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> > > +#
> > > +# Authors: Sonal.Santan@xilinx.com
> > > +#
> > > +
> > > +FULL_XRT_PATH=$(srctree)/$(src)/..
> > > +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> > > +
> > > +obj-$(CONFIG_FPGA_XRT_METADATA) += xrt-md.o
> > > +
> > > +xrt-md-objs := metadata.o
> > > +
> > > +ccflags-y := -I$(FULL_XRT_PATH)/include \
> > > + -I$(FULL_DTC_PATH)
> > > diff --git a/drivers/fpga/xrt/mgmt/Kconfig
> > > b/drivers/fpga/xrt/mgmt/Kconfig new file mode 100644 index
> > > 000000000000..2b2a2c34685c
> > > --- /dev/null
> > > +++ b/drivers/fpga/xrt/mgmt/Kconfig
> > > @@ -0,0 +1,15 @@
> > > +# SPDX-License-Identifier: GPL-2.0-only # # Xilinx XRT FPGA device
> > > +configuration #
> > > +
> > > +config FPGA_XRT_XMGMT
> > > + tristate "Xilinx Alveo Management Driver"
> > > + depends on HWMON && PCI && FPGA_XRT_LIB
> >
> > FPGA_XRT_LIB also depends on HWMON and PCI, so this could be minimized.
> >
> > Tom
> >
> > > + select FPGA_XRT_METADATA
> > > + select FPGA_BRIDGE
> > > + select FPGA_REGION
> > > + help
> > > + Select this option to enable XRT PCIe driver for Xilinx Alveo FPGA.
> > > + This driver provides interfaces for userspace application to access
> > > + Alveo FPGA device.
> > > diff --git a/drivers/fpga/xrt/mgmt/Makefile
> > > b/drivers/fpga/xrt/mgmt/Makefile new file mode 100644 index
> > > 000000000000..8051708c361c
> > > --- /dev/null
> > > +++ b/drivers/fpga/xrt/mgmt/Makefile
> > > @@ -0,0 +1,19 @@
> > > +# SPDX-License-Identifier: GPL-2.0
> > > +#
> > > +# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
> > > +#
> > > +# Authors: Sonal.Santan@xilinx.com
> > > +#
> > > +
> > > +FULL_XRT_PATH=$(srctree)/$(src)/..
> > > +FULL_DTC_PATH=$(srctree)/scripts/dtc/libfdt
> > > +
> > > +obj-$(CONFIG_FPGA_XRT_XMGMT) += xmgmt.o
> > > +
> > > +xmgmt-objs := root.o \
> > > + main.o \
> > > + fmgr-drv.o \
> > > + main-region.o
> > > +
> > > +ccflags-y := -I$(FULL_XRT_PATH)/include \
> > > + -I$(FULL_DTC_PATH)
> >
>
> - Moritz
-Sonal
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (17 preceding siblings ...)
2021-02-18 6:40 ` [PATCH V3 XRT Alveo 18/18] fpga: xrt: Kconfig and Makefile updates for XRT drivers Lizhi Hou
@ 2021-02-18 13:52 ` Tom Rix
2021-02-19 5:15 ` Lizhi Hou
2021-02-21 20:43 ` Moritz Fischer
19 siblings, 1 reply; 87+ messages in thread
From: Tom Rix @ 2021-02-18 13:52 UTC (permalink / raw)
To: Lizhi Hou, linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, mdf, robh
On 2/17/21 10:40 PM, Lizhi Hou wrote:
> Hello,
>
> This is V3 of patch series which adds management physical function driver for Xilinx
> Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> This driver is part of Xilinx Runtime (XRT) open source stack.
>
> XILINX ALVEO PLATFORM ARCHITECTURE
Thanks for refreshing this patchset.
It will take me a while to do the full review, so I thought I would give some early feed back.
It applies to char-misc-next, but will have conflicts with in-flight patches around the MAINTAINERS file. This is not a big deal.
The checkpatch is much better over v2, the complaints are
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#21:
new file mode 100644
WARNING: From:/Signed-off-by: email address mismatch: 'From: Lizhi Hou <lizhi.hou@xilinx.com>' != 'Signed-off-by: Lizhi Hou <lizhih@xilinx.com>'
MAINTAINERS warning i believe you address in the last patch.
In the next revisions, please fix the signoff.
The test robot is complaining about hppa64. While it may be an unlikely config, it would be best to fix it.
Tom
>
> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
> re-configurable *user* partition. The shell partition is automatically loaded from
> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
> till the next cold reboot. The shell exposes two PCIe physical functions:
>
> 1. management physical function
> 2. user physical function
>
> The patch series includes Documentation/xrt.rst which describes Alveo platform,
> XRT driver architecture and deployment model in more detail.
>
> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
> Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> tools. The compiled image is packaged as xclbin which contains partial bitstream
> for the user partition and necessary metadata. Users can dynamically swap the image
> running on the user partition in order to switch between different workloads by
> loading different xclbins.
>
> XRT DRIVERS FOR XILINX ALVEO
>
> XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
> platform. The modular driver framework is organized into several platform drivers
> which primarily handle the following functionality:
>
> 1. Loading firmware container also called xsabin at driver attach time
> 2. Loading of user compiled xclbin with FPGA Manager integration
> 3. Clock scaling of image running on user partition
> 4. In-band sensors: temp, voltage, power, etc.
> 5. Device reset and rescan
>
> The platform drivers are packaged into *xrt-lib* helper module with well
> defined interfaces. The module provides a pseudo-bus implementation for the
> platform drivers. More details on the driver model can be found in
> Documentation/xrt.rst.
>
> User physical function driver is not included in this patch series.
>
> LIBFDT REQUIREMENT
>
> XRT driver infrastructure uses Device Tree as a metadata format to discover
> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
> is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
> of patch series, V3 version does not require export of libfdt symbols.
>
> TESTING AND VALIDATION
>
> xmgmt driver can be tested with full XRT open source stack which includes user
> space libraries, board utilities and (out of tree) first generation user physical
> function driver xocl. XRT open source runtime stack is available at
> https://github.com/Xilinx/XRT
>
> Complete documentation for XRT open source stack including sections on Alveo/XRT
> security and platform architecture can be found here:
>
> https://xilinx.github.io/XRT/master/html/index.html
> https://xilinx.github.io/XRT/master/html/security.html
> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>
> Changes since v2:
> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
> - Updated documentation to show the driver model with examples
> - Addressed kernel test robot errors
> - Added a selftest for basic driver framework
> - Documented device tree schema
> - Removed need to export libfdt symbols
>
> Changes since v1:
> - Updated the driver to use fpga_region and fpga_bridge for FPGA
> programming
> - Dropped platform drivers not related to PR programming to focus on XRT
> core framework
> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
> - Addressed checkpatch issues
> - Dropped xrt- prefix from some header files
>
> For reference V1 version of patch series can be found here:
>
> https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/
>
> Lizhi Hou (18):
> Documentation: fpga: Add a document describing XRT Alveo drivers
> fpga: xrt: driver metadata helper functions
> fpga: xrt: xclbin file helper functions
> fpga: xrt: xrt-lib platform driver manager
> fpga: xrt: group platform driver
> fpga: xrt: platform driver infrastructure
> fpga: xrt: management physical function driver (root)
> fpga: xrt: main platform driver for management function device
> fpga: xrt: fpga-mgr and region implementation for xclbin download
> fpga: xrt: VSEC platform driver
> fpga: xrt: UCS platform driver
> fpga: xrt: ICAP platform driver
> fpga: xrt: devctl platform driver
> fpga: xrt: clock platform driver
> fpga: xrt: clock frequence counter platform driver
> fpga: xrt: DDR calibration platform driver
> fpga: xrt: partition isolation platform driver
> fpga: xrt: Kconfig and Makefile updates for XRT drivers
>
> Documentation/fpga/index.rst | 1 +
> Documentation/fpga/xrt.rst | 842 ++++++++++++++++++++++
> MAINTAINERS | 11 +
> drivers/Makefile | 1 +
> drivers/fpga/Kconfig | 2 +
> drivers/fpga/Makefile | 4 +
> drivers/fpga/xrt/Kconfig | 8 +
> drivers/fpga/xrt/include/events.h | 48 ++
> drivers/fpga/xrt/include/group.h | 27 +
> drivers/fpga/xrt/include/metadata.h | 229 ++++++
> drivers/fpga/xrt/include/subdev_id.h | 43 ++
> drivers/fpga/xrt/include/xclbin-helper.h | 52 ++
> drivers/fpga/xrt/include/xleaf.h | 276 +++++++
> drivers/fpga/xrt/include/xleaf/axigate.h | 25 +
> drivers/fpga/xrt/include/xleaf/calib.h | 30 +
> drivers/fpga/xrt/include/xleaf/clkfreq.h | 23 +
> drivers/fpga/xrt/include/xleaf/clock.h | 31 +
> drivers/fpga/xrt/include/xleaf/devctl.h | 43 ++
> drivers/fpga/xrt/include/xleaf/icap.h | 29 +
> drivers/fpga/xrt/include/xleaf/ucs.h | 24 +
> drivers/fpga/xrt/include/xmgmt-main.h | 37 +
> drivers/fpga/xrt/include/xroot.h | 114 +++
> drivers/fpga/xrt/lib/Kconfig | 16 +
> drivers/fpga/xrt/lib/Makefile | 30 +
> drivers/fpga/xrt/lib/cdev.c | 231 ++++++
> drivers/fpga/xrt/lib/group.c | 265 +++++++
> drivers/fpga/xrt/lib/main.c | 274 +++++++
> drivers/fpga/xrt/lib/main.h | 17 +
> drivers/fpga/xrt/lib/subdev.c | 871 +++++++++++++++++++++++
> drivers/fpga/xrt/lib/subdev_pool.h | 53 ++
> drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++
> drivers/fpga/xrt/lib/xleaf/axigate.c | 298 ++++++++
> drivers/fpga/xrt/lib/xleaf/calib.c | 226 ++++++
> drivers/fpga/xrt/lib/xleaf/clkfreq.c | 221 ++++++
> drivers/fpga/xrt/lib/xleaf/clock.c | 648 +++++++++++++++++
> drivers/fpga/xrt/lib/xleaf/devctl.c | 206 ++++++
> drivers/fpga/xrt/lib/xleaf/icap.c | 317 +++++++++
> drivers/fpga/xrt/lib/xleaf/ucs.c | 235 ++++++
> drivers/fpga/xrt/lib/xleaf/vsec.c | 359 ++++++++++
> drivers/fpga/xrt/lib/xroot.c | 598 ++++++++++++++++
> drivers/fpga/xrt/metadata/Kconfig | 12 +
> drivers/fpga/xrt/metadata/Makefile | 16 +
> drivers/fpga/xrt/metadata/metadata.c | 524 ++++++++++++++
> drivers/fpga/xrt/mgmt/Kconfig | 15 +
> drivers/fpga/xrt/mgmt/Makefile | 19 +
> drivers/fpga/xrt/mgmt/fmgr-drv.c | 187 +++++
> drivers/fpga/xrt/mgmt/fmgr.h | 28 +
> drivers/fpga/xrt/mgmt/main-impl.h | 37 +
> drivers/fpga/xrt/mgmt/main-region.c | 471 ++++++++++++
> drivers/fpga/xrt/mgmt/main.c | 693 ++++++++++++++++++
> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++
> include/uapi/linux/xrt/xclbin.h | 408 +++++++++++
> include/uapi/linux/xrt/xmgmt-ioctl.h | 46 ++
> 53 files changed, 9957 insertions(+)
> create mode 100644 Documentation/fpga/xrt.rst
> create mode 100644 drivers/fpga/xrt/Kconfig
> create mode 100644 drivers/fpga/xrt/include/events.h
> create mode 100644 drivers/fpga/xrt/include/group.h
> create mode 100644 drivers/fpga/xrt/include/metadata.h
> create mode 100644 drivers/fpga/xrt/include/subdev_id.h
> create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
> create mode 100644 drivers/fpga/xrt/include/xleaf.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
> create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
> create mode 100644 drivers/fpga/xrt/include/xroot.h
> create mode 100644 drivers/fpga/xrt/lib/Kconfig
> create mode 100644 drivers/fpga/xrt/lib/Makefile
> create mode 100644 drivers/fpga/xrt/lib/cdev.c
> create mode 100644 drivers/fpga/xrt/lib/group.c
> create mode 100644 drivers/fpga/xrt/lib/main.c
> create mode 100644 drivers/fpga/xrt/lib/main.h
> create mode 100644 drivers/fpga/xrt/lib/subdev.c
> create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
> create mode 100644 drivers/fpga/xrt/lib/xclbin.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
> create mode 100644 drivers/fpga/xrt/lib/xroot.c
> create mode 100644 drivers/fpga/xrt/metadata/Kconfig
> create mode 100644 drivers/fpga/xrt/metadata/Makefile
> create mode 100644 drivers/fpga/xrt/metadata/metadata.c
> create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
> create mode 100644 drivers/fpga/xrt/mgmt/Makefile
> create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
> create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
> create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
> create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
> create mode 100644 drivers/fpga/xrt/mgmt/main.c
> create mode 100644 drivers/fpga/xrt/mgmt/root.c
> create mode 100644 include/uapi/linux/xrt/xclbin.h
> create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview
2021-02-18 13:52 ` [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Tom Rix
@ 2021-02-19 5:15 ` Lizhi Hou
0 siblings, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-02-19 5:15 UTC (permalink / raw)
To: Tom Rix, Lizhi Hou, linux-kernel
Cc: linux-fpga, maxz, sonal.santan, michal.simek, stefanos,
devicetree, mdf, robh
On 02/18/2021 05:52 AM, Tom Rix wrote:
> On 2/17/21 10:40 PM, Lizhi Hou wrote:
>> Hello,
>>
>> This is V3 of patch series which adds management physical function driver for Xilinx
>> Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
>> This driver is part of Xilinx Runtime (XRT) open source stack.
>>
>> XILINX ALVEO PLATFORM ARCHITECTURE
> Thanks for refreshing this patchset.
>
> It will take me a while to do the full review, so I thought I would give some early feed back.
>
> It applies to char-misc-next, but will have conflicts with in-flight patches around the MAINTAINERS file. This is not a big deal.
>
> The checkpatch is much better over v2, the complaints are
>
> WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
> #21:
> new file mode 100644
>
> WARNING: From:/Signed-off-by: email address mismatch: 'From: Lizhi Hou <lizhi.hou@xilinx.com>' != 'Signed-off-by: Lizhi Hou <lizhih@xilinx.com>'
>
> MAINTAINERS warning i believe you address in the last patch.
>
> In the next revisions, please fix the signoff.
>
> The test robot is complaining about hppa64. While it may be an unlikely config, it would be best to fix it.
Thanks for reviewing. I will fix signoff, hppa64 and arm build issue
reported by robot in next revision.
>
> Tom
>
>> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
>> re-configurable *user* partition. The shell partition is automatically loaded from
>> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
>> till the next cold reboot. The shell exposes two PCIe physical functions:
>>
>> 1. management physical function
>> 2. user physical function
>>
>> The patch series includes Documentation/xrt.rst which describes Alveo platform,
>> XRT driver architecture and deployment model in more detail.
>>
>> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
>> Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
>> tools. The compiled image is packaged as xclbin which contains partial bitstream
>> for the user partition and necessary metadata. Users can dynamically swap the image
>> running on the user partition in order to switch between different workloads by
>> loading different xclbins.
>>
>> XRT DRIVERS FOR XILINX ALVEO
>>
>> XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
>> platform. The modular driver framework is organized into several platform drivers
>> which primarily handle the following functionality:
>>
>> 1. Loading firmware container also called xsabin at driver attach time
>> 2. Loading of user compiled xclbin with FPGA Manager integration
>> 3. Clock scaling of image running on user partition
>> 4. In-band sensors: temp, voltage, power, etc.
>> 5. Device reset and rescan
>>
>> The platform drivers are packaged into *xrt-lib* helper module with well
>> defined interfaces. The module provides a pseudo-bus implementation for the
>> platform drivers. More details on the driver model can be found in
>> Documentation/xrt.rst.
>>
>> User physical function driver is not included in this patch series.
>>
>> LIBFDT REQUIREMENT
>>
>> XRT driver infrastructure uses Device Tree as a metadata format to discover
>> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
>> is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
>> of patch series, V3 version does not require export of libfdt symbols.
>>
>> TESTING AND VALIDATION
>>
>> xmgmt driver can be tested with full XRT open source stack which includes user
>> space libraries, board utilities and (out of tree) first generation user physical
>> function driver xocl. XRT open source runtime stack is available at
>> https://github.com/Xilinx/XRT
>>
>> Complete documentation for XRT open source stack including sections on Alveo/XRT
>> security and platform architecture can be found here:
>>
>> https://xilinx.github.io/XRT/master/html/index.html
>> https://xilinx.github.io/XRT/master/html/security.html
>> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>>
>> Changes since v2:
>> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
>> - Updated documentation to show the driver model with examples
>> - Addressed kernel test robot errors
>> - Added a selftest for basic driver framework
>> - Documented device tree schema
>> - Removed need to export libfdt symbols
>>
>> Changes since v1:
>> - Updated the driver to use fpga_region and fpga_bridge for FPGA
>> programming
>> - Dropped platform drivers not related to PR programming to focus on XRT
>> core framework
>> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
>> - Addressed checkpatch issues
>> - Dropped xrt- prefix from some header files
>>
>> For reference V1 version of patch series can be found here:
>>
>> https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/
>>
>> Lizhi Hou (18):
>> Documentation: fpga: Add a document describing XRT Alveo drivers
>> fpga: xrt: driver metadata helper functions
>> fpga: xrt: xclbin file helper functions
>> fpga: xrt: xrt-lib platform driver manager
>> fpga: xrt: group platform driver
>> fpga: xrt: platform driver infrastructure
>> fpga: xrt: management physical function driver (root)
>> fpga: xrt: main platform driver for management function device
>> fpga: xrt: fpga-mgr and region implementation for xclbin download
>> fpga: xrt: VSEC platform driver
>> fpga: xrt: UCS platform driver
>> fpga: xrt: ICAP platform driver
>> fpga: xrt: devctl platform driver
>> fpga: xrt: clock platform driver
>> fpga: xrt: clock frequence counter platform driver
>> fpga: xrt: DDR calibration platform driver
>> fpga: xrt: partition isolation platform driver
>> fpga: xrt: Kconfig and Makefile updates for XRT drivers
>>
>> Documentation/fpga/index.rst | 1 +
>> Documentation/fpga/xrt.rst | 842 ++++++++++++++++++++++
>> MAINTAINERS | 11 +
>> drivers/Makefile | 1 +
>> drivers/fpga/Kconfig | 2 +
>> drivers/fpga/Makefile | 4 +
>> drivers/fpga/xrt/Kconfig | 8 +
>> drivers/fpga/xrt/include/events.h | 48 ++
>> drivers/fpga/xrt/include/group.h | 27 +
>> drivers/fpga/xrt/include/metadata.h | 229 ++++++
>> drivers/fpga/xrt/include/subdev_id.h | 43 ++
>> drivers/fpga/xrt/include/xclbin-helper.h | 52 ++
>> drivers/fpga/xrt/include/xleaf.h | 276 +++++++
>> drivers/fpga/xrt/include/xleaf/axigate.h | 25 +
>> drivers/fpga/xrt/include/xleaf/calib.h | 30 +
>> drivers/fpga/xrt/include/xleaf/clkfreq.h | 23 +
>> drivers/fpga/xrt/include/xleaf/clock.h | 31 +
>> drivers/fpga/xrt/include/xleaf/devctl.h | 43 ++
>> drivers/fpga/xrt/include/xleaf/icap.h | 29 +
>> drivers/fpga/xrt/include/xleaf/ucs.h | 24 +
>> drivers/fpga/xrt/include/xmgmt-main.h | 37 +
>> drivers/fpga/xrt/include/xroot.h | 114 +++
>> drivers/fpga/xrt/lib/Kconfig | 16 +
>> drivers/fpga/xrt/lib/Makefile | 30 +
>> drivers/fpga/xrt/lib/cdev.c | 231 ++++++
>> drivers/fpga/xrt/lib/group.c | 265 +++++++
>> drivers/fpga/xrt/lib/main.c | 274 +++++++
>> drivers/fpga/xrt/lib/main.h | 17 +
>> drivers/fpga/xrt/lib/subdev.c | 871 +++++++++++++++++++++++
>> drivers/fpga/xrt/lib/subdev_pool.h | 53 ++
>> drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++
>> drivers/fpga/xrt/lib/xleaf/axigate.c | 298 ++++++++
>> drivers/fpga/xrt/lib/xleaf/calib.c | 226 ++++++
>> drivers/fpga/xrt/lib/xleaf/clkfreq.c | 221 ++++++
>> drivers/fpga/xrt/lib/xleaf/clock.c | 648 +++++++++++++++++
>> drivers/fpga/xrt/lib/xleaf/devctl.c | 206 ++++++
>> drivers/fpga/xrt/lib/xleaf/icap.c | 317 +++++++++
>> drivers/fpga/xrt/lib/xleaf/ucs.c | 235 ++++++
>> drivers/fpga/xrt/lib/xleaf/vsec.c | 359 ++++++++++
>> drivers/fpga/xrt/lib/xroot.c | 598 ++++++++++++++++
>> drivers/fpga/xrt/metadata/Kconfig | 12 +
>> drivers/fpga/xrt/metadata/Makefile | 16 +
>> drivers/fpga/xrt/metadata/metadata.c | 524 ++++++++++++++
>> drivers/fpga/xrt/mgmt/Kconfig | 15 +
>> drivers/fpga/xrt/mgmt/Makefile | 19 +
>> drivers/fpga/xrt/mgmt/fmgr-drv.c | 187 +++++
>> drivers/fpga/xrt/mgmt/fmgr.h | 28 +
>> drivers/fpga/xrt/mgmt/main-impl.h | 37 +
>> drivers/fpga/xrt/mgmt/main-region.c | 471 ++++++++++++
>> drivers/fpga/xrt/mgmt/main.c | 693 ++++++++++++++++++
>> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++
>> include/uapi/linux/xrt/xclbin.h | 408 +++++++++++
>> include/uapi/linux/xrt/xmgmt-ioctl.h | 46 ++
>> 53 files changed, 9957 insertions(+)
>> create mode 100644 Documentation/fpga/xrt.rst
>> create mode 100644 drivers/fpga/xrt/Kconfig
>> create mode 100644 drivers/fpga/xrt/include/events.h
>> create mode 100644 drivers/fpga/xrt/include/group.h
>> create mode 100644 drivers/fpga/xrt/include/metadata.h
>> create mode 100644 drivers/fpga/xrt/include/subdev_id.h
>> create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
>> create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>> create mode 100644 drivers/fpga/xrt/include/xroot.h
>> create mode 100644 drivers/fpga/xrt/lib/Kconfig
>> create mode 100644 drivers/fpga/xrt/lib/Makefile
>> create mode 100644 drivers/fpga/xrt/lib/cdev.c
>> create mode 100644 drivers/fpga/xrt/lib/group.c
>> create mode 100644 drivers/fpga/xrt/lib/main.c
>> create mode 100644 drivers/fpga/xrt/lib/main.h
>> create mode 100644 drivers/fpga/xrt/lib/subdev.c
>> create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
>> create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
>> create mode 100644 drivers/fpga/xrt/lib/xroot.c
>> create mode 100644 drivers/fpga/xrt/metadata/Kconfig
>> create mode 100644 drivers/fpga/xrt/metadata/Makefile
>> create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>> create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
>> create mode 100644 drivers/fpga/xrt/mgmt/Makefile
>> create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
>> create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
>> create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
>> create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
>> create mode 100644 drivers/fpga/xrt/mgmt/main.c
>> create mode 100644 drivers/fpga/xrt/mgmt/root.c
>> create mode 100644 include/uapi/linux/xrt/xclbin.h
>> create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>>
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview
2021-02-18 6:40 [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Lizhi Hou
` (18 preceding siblings ...)
2021-02-18 13:52 ` [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview Tom Rix
@ 2021-02-21 20:43 ` Moritz Fischer
2021-03-01 18:29 ` Lizhi Hou
2021-03-03 6:49 ` Joe Perches
19 siblings, 2 replies; 87+ messages in thread
From: Moritz Fischer @ 2021-02-21 20:43 UTC (permalink / raw)
To: Lizhi Hou
Cc: linux-kernel, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, mdf, robh
Lizhi,
On Wed, Feb 17, 2021 at 10:40:01PM -0800, Lizhi Hou wrote:
> Hello,
>
> This is V3 of patch series which adds management physical function driver for Xilinx
> Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> This driver is part of Xilinx Runtime (XRT) open source stack.
>
> XILINX ALVEO PLATFORM ARCHITECTURE
>
> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
> re-configurable *user* partition. The shell partition is automatically loaded from
> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
> till the next cold reboot. The shell exposes two PCIe physical functions:
>
> 1. management physical function
> 2. user physical function
>
> The patch series includes Documentation/xrt.rst which describes Alveo platform,
> XRT driver architecture and deployment model in more detail.
>
> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
> Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
> tools. The compiled image is packaged as xclbin which contains partial bitstream
> for the user partition and necessary metadata. Users can dynamically swap the image
> running on the user partition in order to switch between different workloads by
> loading different xclbins.
>
> XRT DRIVERS FOR XILINX ALVEO
>
> XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
> platform. The modular driver framework is organized into several platform drivers
> which primarily handle the following functionality:
>
> 1. Loading firmware container also called xsabin at driver attach time
> 2. Loading of user compiled xclbin with FPGA Manager integration
> 3. Clock scaling of image running on user partition
> 4. In-band sensors: temp, voltage, power, etc.
> 5. Device reset and rescan
>
> The platform drivers are packaged into *xrt-lib* helper module with well
> defined interfaces. The module provides a pseudo-bus implementation for the
> platform drivers. More details on the driver model can be found in
> Documentation/xrt.rst.
>
> User physical function driver is not included in this patch series.
>
> LIBFDT REQUIREMENT
>
> XRT driver infrastructure uses Device Tree as a metadata format to discover
> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
> is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
> of patch series, V3 version does not require export of libfdt symbols.
>
> TESTING AND VALIDATION
>
> xmgmt driver can be tested with full XRT open source stack which includes user
> space libraries, board utilities and (out of tree) first generation user physical
> function driver xocl. XRT open source runtime stack is available at
> https://github.com/Xilinx/XRT
>
> Complete documentation for XRT open source stack including sections on Alveo/XRT
> security and platform architecture can be found here:
>
> https://xilinx.github.io/XRT/master/html/index.html
> https://xilinx.github.io/XRT/master/html/security.html
> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>
> Changes since v2:
> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
> - Updated documentation to show the driver model with examples
> - Addressed kernel test robot errors
> - Added a selftest for basic driver framework
> - Documented device tree schema
> - Removed need to export libfdt symbols
>
> Changes since v1:
> - Updated the driver to use fpga_region and fpga_bridge for FPGA
> programming
> - Dropped platform drivers not related to PR programming to focus on XRT
> core framework
> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
> - Addressed checkpatch issues
> - Dropped xrt- prefix from some header files
>
> For reference V1 version of patch series can be found here:
>
> https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
> https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/
>
> Lizhi Hou (18):
> Documentation: fpga: Add a document describing XRT Alveo drivers
> fpga: xrt: driver metadata helper functions
> fpga: xrt: xclbin file helper functions
> fpga: xrt: xrt-lib platform driver manager
> fpga: xrt: group platform driver
> fpga: xrt: platform driver infrastructure
> fpga: xrt: management physical function driver (root)
> fpga: xrt: main platform driver for management function device
> fpga: xrt: fpga-mgr and region implementation for xclbin download
> fpga: xrt: VSEC platform driver
> fpga: xrt: UCS platform driver
> fpga: xrt: ICAP platform driver
> fpga: xrt: devctl platform driver
> fpga: xrt: clock platform driver
> fpga: xrt: clock frequence counter platform driver
> fpga: xrt: DDR calibration platform driver
> fpga: xrt: partition isolation platform driver
> fpga: xrt: Kconfig and Makefile updates for XRT drivers
>
> Documentation/fpga/index.rst | 1 +
> Documentation/fpga/xrt.rst | 842 ++++++++++++++++++++++
> MAINTAINERS | 11 +
> drivers/Makefile | 1 +
> drivers/fpga/Kconfig | 2 +
> drivers/fpga/Makefile | 4 +
> drivers/fpga/xrt/Kconfig | 8 +
> drivers/fpga/xrt/include/events.h | 48 ++
> drivers/fpga/xrt/include/group.h | 27 +
> drivers/fpga/xrt/include/metadata.h | 229 ++++++
> drivers/fpga/xrt/include/subdev_id.h | 43 ++
> drivers/fpga/xrt/include/xclbin-helper.h | 52 ++
> drivers/fpga/xrt/include/xleaf.h | 276 +++++++
> drivers/fpga/xrt/include/xleaf/axigate.h | 25 +
> drivers/fpga/xrt/include/xleaf/calib.h | 30 +
> drivers/fpga/xrt/include/xleaf/clkfreq.h | 23 +
> drivers/fpga/xrt/include/xleaf/clock.h | 31 +
> drivers/fpga/xrt/include/xleaf/devctl.h | 43 ++
> drivers/fpga/xrt/include/xleaf/icap.h | 29 +
> drivers/fpga/xrt/include/xleaf/ucs.h | 24 +
> drivers/fpga/xrt/include/xmgmt-main.h | 37 +
> drivers/fpga/xrt/include/xroot.h | 114 +++
> drivers/fpga/xrt/lib/Kconfig | 16 +
> drivers/fpga/xrt/lib/Makefile | 30 +
> drivers/fpga/xrt/lib/cdev.c | 231 ++++++
> drivers/fpga/xrt/lib/group.c | 265 +++++++
> drivers/fpga/xrt/lib/main.c | 274 +++++++
> drivers/fpga/xrt/lib/main.h | 17 +
> drivers/fpga/xrt/lib/subdev.c | 871 +++++++++++++++++++++++
> drivers/fpga/xrt/lib/subdev_pool.h | 53 ++
> drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++
> drivers/fpga/xrt/lib/xleaf/axigate.c | 298 ++++++++
> drivers/fpga/xrt/lib/xleaf/calib.c | 226 ++++++
> drivers/fpga/xrt/lib/xleaf/clkfreq.c | 221 ++++++
> drivers/fpga/xrt/lib/xleaf/clock.c | 648 +++++++++++++++++
> drivers/fpga/xrt/lib/xleaf/devctl.c | 206 ++++++
> drivers/fpga/xrt/lib/xleaf/icap.c | 317 +++++++++
> drivers/fpga/xrt/lib/xleaf/ucs.c | 235 ++++++
> drivers/fpga/xrt/lib/xleaf/vsec.c | 359 ++++++++++
> drivers/fpga/xrt/lib/xroot.c | 598 ++++++++++++++++
> drivers/fpga/xrt/metadata/Kconfig | 12 +
> drivers/fpga/xrt/metadata/Makefile | 16 +
> drivers/fpga/xrt/metadata/metadata.c | 524 ++++++++++++++
> drivers/fpga/xrt/mgmt/Kconfig | 15 +
> drivers/fpga/xrt/mgmt/Makefile | 19 +
> drivers/fpga/xrt/mgmt/fmgr-drv.c | 187 +++++
> drivers/fpga/xrt/mgmt/fmgr.h | 28 +
> drivers/fpga/xrt/mgmt/main-impl.h | 37 +
> drivers/fpga/xrt/mgmt/main-region.c | 471 ++++++++++++
> drivers/fpga/xrt/mgmt/main.c | 693 ++++++++++++++++++
> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++
> include/uapi/linux/xrt/xclbin.h | 408 +++++++++++
> include/uapi/linux/xrt/xmgmt-ioctl.h | 46 ++
> 53 files changed, 9957 insertions(+)
> create mode 100644 Documentation/fpga/xrt.rst
> create mode 100644 drivers/fpga/xrt/Kconfig
> create mode 100644 drivers/fpga/xrt/include/events.h
> create mode 100644 drivers/fpga/xrt/include/group.h
> create mode 100644 drivers/fpga/xrt/include/metadata.h
> create mode 100644 drivers/fpga/xrt/include/subdev_id.h
> create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
> create mode 100644 drivers/fpga/xrt/include/xleaf.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
> create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
> create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
> create mode 100644 drivers/fpga/xrt/include/xroot.h
> create mode 100644 drivers/fpga/xrt/lib/Kconfig
> create mode 100644 drivers/fpga/xrt/lib/Makefile
> create mode 100644 drivers/fpga/xrt/lib/cdev.c
> create mode 100644 drivers/fpga/xrt/lib/group.c
> create mode 100644 drivers/fpga/xrt/lib/main.c
> create mode 100644 drivers/fpga/xrt/lib/main.h
> create mode 100644 drivers/fpga/xrt/lib/subdev.c
> create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
> create mode 100644 drivers/fpga/xrt/lib/xclbin.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
> create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
> create mode 100644 drivers/fpga/xrt/lib/xroot.c
> create mode 100644 drivers/fpga/xrt/metadata/Kconfig
> create mode 100644 drivers/fpga/xrt/metadata/Makefile
> create mode 100644 drivers/fpga/xrt/metadata/metadata.c
> create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
> create mode 100644 drivers/fpga/xrt/mgmt/Makefile
> create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
> create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
> create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
> create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
> create mode 100644 drivers/fpga/xrt/mgmt/main.c
> create mode 100644 drivers/fpga/xrt/mgmt/root.c
> create mode 100644 include/uapi/linux/xrt/xclbin.h
> create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>
> --
> 2.18.4
>
Please fix the indents all across this patchset. Doesn't checkpatch with
--strict complain about this?
Also more generally this looks like it should be a bus. Look at DFL for
reference.
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview
2021-02-21 20:43 ` Moritz Fischer
@ 2021-03-01 18:29 ` Lizhi Hou
2021-03-03 6:49 ` Joe Perches
1 sibling, 0 replies; 87+ messages in thread
From: Lizhi Hou @ 2021-03-01 18:29 UTC (permalink / raw)
To: Moritz Fischer, Lizhi Hou
Cc: linux-kernel, linux-fpga, maxz, sonal.santan, michal.simek,
stefanos, devicetree, trix, robh
Hi Moritz,
On 02/21/2021 12:43 PM, Moritz Fischer wrote:
> Lizhi,
>
> On Wed, Feb 17, 2021 at 10:40:01PM -0800, Lizhi Hou wrote:
>> Hello,
>>
>> This is V3 of patch series which adds management physical function driver for Xilinx
>> Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
>> This driver is part of Xilinx Runtime (XRT) open source stack.
>>
>> XILINX ALVEO PLATFORM ARCHITECTURE
>>
>> Alveo PCIe FPGA based platforms have a static *shell* partition and a partial
>> re-configurable *user* partition. The shell partition is automatically loaded from
>> flash when host is booted and PCIe is enumerated by BIOS. Shell cannot be changed
>> till the next cold reboot. The shell exposes two PCIe physical functions:
>>
>> 1. management physical function
>> 2. user physical function
>>
>> The patch series includes Documentation/xrt.rst which describes Alveo platform,
>> XRT driver architecture and deployment model in more detail.
>>
>> Users compile their high level design in C/C++/OpenCL or RTL into FPGA image using
>> Vitis https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html
>> tools. The compiled image is packaged as xclbin which contains partial bitstream
>> for the user partition and necessary metadata. Users can dynamically swap the image
>> running on the user partition in order to switch between different workloads by
>> loading different xclbins.
>>
>> XRT DRIVERS FOR XILINX ALVEO
>>
>> XRT Linux kernel driver *xmgmt* binds to management physical function of Alveo
>> platform. The modular driver framework is organized into several platform drivers
>> which primarily handle the following functionality:
>>
>> 1. Loading firmware container also called xsabin at driver attach time
>> 2. Loading of user compiled xclbin with FPGA Manager integration
>> 3. Clock scaling of image running on user partition
>> 4. In-band sensors: temp, voltage, power, etc.
>> 5. Device reset and rescan
>>
>> The platform drivers are packaged into *xrt-lib* helper module with well
>> defined interfaces. The module provides a pseudo-bus implementation for the
>> platform drivers. More details on the driver model can be found in
>> Documentation/xrt.rst.
>>
>> User physical function driver is not included in this patch series.
>>
>> LIBFDT REQUIREMENT
>>
>> XRT driver infrastructure uses Device Tree as a metadata format to discover
>> HW subsystems in the Alveo PCIe device. The Device Tree schema used by XRT
>> is documented in Documentation/xrt.rst. Unlike previous V1 and V2 version
>> of patch series, V3 version does not require export of libfdt symbols.
>>
>> TESTING AND VALIDATION
>>
>> xmgmt driver can be tested with full XRT open source stack which includes user
>> space libraries, board utilities and (out of tree) first generation user physical
>> function driver xocl. XRT open source runtime stack is available at
>> https://github.com/Xilinx/XRT
>>
>> Complete documentation for XRT open source stack including sections on Alveo/XRT
>> security and platform architecture can be found here:
>>
>> https://xilinx.github.io/XRT/master/html/index.html
>> https://xilinx.github.io/XRT/master/html/security.html
>> https://xilinx.github.io/XRT/master/html/platforms_partitions.html
>>
>> Changes since v2:
>> - Streamlined the driver framework into *xleaf*, *group* and *xroot*
>> - Updated documentation to show the driver model with examples
>> - Addressed kernel test robot errors
>> - Added a selftest for basic driver framework
>> - Documented device tree schema
>> - Removed need to export libfdt symbols
>>
>> Changes since v1:
>> - Updated the driver to use fpga_region and fpga_bridge for FPGA
>> programming
>> - Dropped platform drivers not related to PR programming to focus on XRT
>> core framework
>> - Updated Documentation/fpga/xrt.rst with information on XRT core framework
>> - Addressed checkpatch issues
>> - Dropped xrt- prefix from some header files
>>
>> For reference V1 version of patch series can be found here:
>>
>> https://lore.kernel.org/lkml/20201217075046.28553-1-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-2-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-3-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-4-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-5-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-6-sonals@xilinx.com/
>> https://lore.kernel.org/lkml/20201217075046.28553-7-sonals@xilinx.com/
>>
>> Lizhi Hou (18):
>> Documentation: fpga: Add a document describing XRT Alveo drivers
>> fpga: xrt: driver metadata helper functions
>> fpga: xrt: xclbin file helper functions
>> fpga: xrt: xrt-lib platform driver manager
>> fpga: xrt: group platform driver
>> fpga: xrt: platform driver infrastructure
>> fpga: xrt: management physical function driver (root)
>> fpga: xrt: main platform driver for management function device
>> fpga: xrt: fpga-mgr and region implementation for xclbin download
>> fpga: xrt: VSEC platform driver
>> fpga: xrt: UCS platform driver
>> fpga: xrt: ICAP platform driver
>> fpga: xrt: devctl platform driver
>> fpga: xrt: clock platform driver
>> fpga: xrt: clock frequence counter platform driver
>> fpga: xrt: DDR calibration platform driver
>> fpga: xrt: partition isolation platform driver
>> fpga: xrt: Kconfig and Makefile updates for XRT drivers
>>
>> Documentation/fpga/index.rst | 1 +
>> Documentation/fpga/xrt.rst | 842 ++++++++++++++++++++++
>> MAINTAINERS | 11 +
>> drivers/Makefile | 1 +
>> drivers/fpga/Kconfig | 2 +
>> drivers/fpga/Makefile | 4 +
>> drivers/fpga/xrt/Kconfig | 8 +
>> drivers/fpga/xrt/include/events.h | 48 ++
>> drivers/fpga/xrt/include/group.h | 27 +
>> drivers/fpga/xrt/include/metadata.h | 229 ++++++
>> drivers/fpga/xrt/include/subdev_id.h | 43 ++
>> drivers/fpga/xrt/include/xclbin-helper.h | 52 ++
>> drivers/fpga/xrt/include/xleaf.h | 276 +++++++
>> drivers/fpga/xrt/include/xleaf/axigate.h | 25 +
>> drivers/fpga/xrt/include/xleaf/calib.h | 30 +
>> drivers/fpga/xrt/include/xleaf/clkfreq.h | 23 +
>> drivers/fpga/xrt/include/xleaf/clock.h | 31 +
>> drivers/fpga/xrt/include/xleaf/devctl.h | 43 ++
>> drivers/fpga/xrt/include/xleaf/icap.h | 29 +
>> drivers/fpga/xrt/include/xleaf/ucs.h | 24 +
>> drivers/fpga/xrt/include/xmgmt-main.h | 37 +
>> drivers/fpga/xrt/include/xroot.h | 114 +++
>> drivers/fpga/xrt/lib/Kconfig | 16 +
>> drivers/fpga/xrt/lib/Makefile | 30 +
>> drivers/fpga/xrt/lib/cdev.c | 231 ++++++
>> drivers/fpga/xrt/lib/group.c | 265 +++++++
>> drivers/fpga/xrt/lib/main.c | 274 +++++++
>> drivers/fpga/xrt/lib/main.h | 17 +
>> drivers/fpga/xrt/lib/subdev.c | 871 +++++++++++++++++++++++
>> drivers/fpga/xrt/lib/subdev_pool.h | 53 ++
>> drivers/fpga/xrt/lib/xclbin.c | 394 ++++++++++
>> drivers/fpga/xrt/lib/xleaf/axigate.c | 298 ++++++++
>> drivers/fpga/xrt/lib/xleaf/calib.c | 226 ++++++
>> drivers/fpga/xrt/lib/xleaf/clkfreq.c | 221 ++++++
>> drivers/fpga/xrt/lib/xleaf/clock.c | 648 +++++++++++++++++
>> drivers/fpga/xrt/lib/xleaf/devctl.c | 206 ++++++
>> drivers/fpga/xrt/lib/xleaf/icap.c | 317 +++++++++
>> drivers/fpga/xrt/lib/xleaf/ucs.c | 235 ++++++
>> drivers/fpga/xrt/lib/xleaf/vsec.c | 359 ++++++++++
>> drivers/fpga/xrt/lib/xroot.c | 598 ++++++++++++++++
>> drivers/fpga/xrt/metadata/Kconfig | 12 +
>> drivers/fpga/xrt/metadata/Makefile | 16 +
>> drivers/fpga/xrt/metadata/metadata.c | 524 ++++++++++++++
>> drivers/fpga/xrt/mgmt/Kconfig | 15 +
>> drivers/fpga/xrt/mgmt/Makefile | 19 +
>> drivers/fpga/xrt/mgmt/fmgr-drv.c | 187 +++++
>> drivers/fpga/xrt/mgmt/fmgr.h | 28 +
>> drivers/fpga/xrt/mgmt/main-impl.h | 37 +
>> drivers/fpga/xrt/mgmt/main-region.c | 471 ++++++++++++
>> drivers/fpga/xrt/mgmt/main.c | 693 ++++++++++++++++++
>> drivers/fpga/xrt/mgmt/root.c | 342 +++++++++
>> include/uapi/linux/xrt/xclbin.h | 408 +++++++++++
>> include/uapi/linux/xrt/xmgmt-ioctl.h | 46 ++
>> 53 files changed, 9957 insertions(+)
>> create mode 100644 Documentation/fpga/xrt.rst
>> create mode 100644 drivers/fpga/xrt/Kconfig
>> create mode 100644 drivers/fpga/xrt/include/events.h
>> create mode 100644 drivers/fpga/xrt/include/group.h
>> create mode 100644 drivers/fpga/xrt/include/metadata.h
>> create mode 100644 drivers/fpga/xrt/include/subdev_id.h
>> create mode 100644 drivers/fpga/xrt/include/xclbin-helper.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/axigate.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/calib.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/clkfreq.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/clock.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/devctl.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/icap.h
>> create mode 100644 drivers/fpga/xrt/include/xleaf/ucs.h
>> create mode 100644 drivers/fpga/xrt/include/xmgmt-main.h
>> create mode 100644 drivers/fpga/xrt/include/xroot.h
>> create mode 100644 drivers/fpga/xrt/lib/Kconfig
>> create mode 100644 drivers/fpga/xrt/lib/Makefile
>> create mode 100644 drivers/fpga/xrt/lib/cdev.c
>> create mode 100644 drivers/fpga/xrt/lib/group.c
>> create mode 100644 drivers/fpga/xrt/lib/main.c
>> create mode 100644 drivers/fpga/xrt/lib/main.h
>> create mode 100644 drivers/fpga/xrt/lib/subdev.c
>> create mode 100644 drivers/fpga/xrt/lib/subdev_pool.h
>> create mode 100644 drivers/fpga/xrt/lib/xclbin.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/axigate.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/calib.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/clkfreq.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/clock.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/devctl.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/icap.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/ucs.c
>> create mode 100644 drivers/fpga/xrt/lib/xleaf/vsec.c
>> create mode 100644 drivers/fpga/xrt/lib/xroot.c
>> create mode 100644 drivers/fpga/xrt/metadata/Kconfig
>> create mode 100644 drivers/fpga/xrt/metadata/Makefile
>> create mode 100644 drivers/fpga/xrt/metadata/metadata.c
>> create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
>> create mode 100644 drivers/fpga/xrt/mgmt/Makefile
>> create mode 100644 drivers/fpga/xrt/mgmt/fmgr-drv.c
>> create mode 100644 drivers/fpga/xrt/mgmt/fmgr.h
>> create mode 100644 drivers/fpga/xrt/mgmt/main-impl.h
>> create mode 100644 drivers/fpga/xrt/mgmt/main-region.c
>> create mode 100644 drivers/fpga/xrt/mgmt/main.c
>> create mode 100644 drivers/fpga/xrt/mgmt/root.c
>> create mode 100644 include/uapi/linux/xrt/xclbin.h
>> create mode 100644 include/uapi/linux/xrt/xmgmt-ioctl.h
>>
>> --
>> 2.18.4
>>
> Please fix the indents all across this patchset. Doesn't checkpatch with
> --strict complain about this?
checkpatch --strict did not complain. And we will fix this.
>
> Also more generally this looks like it should be a bus. Look at DFL for
> reference.
Tom asked the same question in 01/18 and Sonal replied. We may discuss
with that thread.
Thanks,
Lizhi
>
> - Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview
2021-02-21 20:43 ` Moritz Fischer
2021-03-01 18:29 ` Lizhi Hou
@ 2021-03-03 6:49 ` Joe Perches
2021-03-03 23:15 ` Moritz Fischer
1 sibling, 1 reply; 87+ messages in thread
From: Joe Perches @ 2021-03-03 6:49 UTC (permalink / raw)
To: Moritz Fischer, Lizhi Hou
Cc: linux-kernel, Lizhi Hou, linux-fpga, maxz, sonal.santan,
michal.simek, stefanos, devicetree, trix, robh
On Sun, 2021-02-21 at 12:43 -0800, Moritz Fischer wrote:
> On Wed, Feb 17, 2021 at 10:40:01PM -0800, Lizhi Hou wrote:
> > This is V3 of patch series which adds management physical function driver for Xilinx
> > Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> > This driver is part of Xilinx Runtime (XRT) open source stack.
[]
> Please fix the indents all across this patchset. Doesn't checkpatch with
> --strict complain about this?
I glanced at a couple bits of these patches and didn't
notice any of what I consider poor indentation style.
What indent is wrong here?
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V3 XRT Alveo 00/18] XRT Alveo driver overview
2021-03-03 6:49 ` Joe Perches
@ 2021-03-03 23:15 ` Moritz Fischer
0 siblings, 0 replies; 87+ messages in thread
From: Moritz Fischer @ 2021-03-03 23:15 UTC (permalink / raw)
To: Joe Perches
Cc: Moritz Fischer, Lizhi Hou, linux-kernel, Lizhi Hou, linux-fpga,
maxz, sonal.santan, michal.simek, stefanos, devicetree, trix,
robh
On Tue, Mar 02, 2021 at 10:49:43PM -0800, Joe Perches wrote:
> On Sun, 2021-02-21 at 12:43 -0800, Moritz Fischer wrote:
> > On Wed, Feb 17, 2021 at 10:40:01PM -0800, Lizhi Hou wrote:
> > > This is V3 of patch series which adds management physical function driver for Xilinx
> > > Alveo PCIe accelerator cards, https://www.xilinx.com/products/boards-and-kits/alveo.html
> > > This driver is part of Xilinx Runtime (XRT) open source stack.
> []
> > Please fix the indents all across this patchset. Doesn't checkpatch with
> > --strict complain about this?
>
> I glanced at a couple bits of these patches and didn't
> notice any of what I consider poor indentation style.
>
> What indent is wrong here?
Maybe I dreamed it, or confused it with the CamelCase issues instead?
Sorry for the noise in that case,
- Moritz
^ permalink raw reply [flat|nested] 87+ messages in thread