* [PATCH V1 XRT Alveo Infrastructure 1/9] Documentation: fpga: Add a document describing XRT Alveo driver infrastructure
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 2/9] Documentation: devicetree: bindings: add xrt group binding Lizhi Hou
` (7 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
Describe XRT driver architecture and provide basic overview of
Xilinx Alveo platform.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
Documentation/fpga/index.rst | 1 +
Documentation/fpga/xrt.rst | 510 +++++++++++++++++++++++++++++++++++
MAINTAINERS | 10 +
3 files changed, 521 insertions(+)
create mode 100644 Documentation/fpga/xrt.rst
diff --git a/Documentation/fpga/index.rst b/Documentation/fpga/index.rst
index f80f95667ca2..30134357b70d 100644
--- a/Documentation/fpga/index.rst
+++ b/Documentation/fpga/index.rst
@@ -8,6 +8,7 @@ fpga
:maxdepth: 1
dfl
+ xrt
.. only:: subproject and html
diff --git a/Documentation/fpga/xrt.rst b/Documentation/fpga/xrt.rst
new file mode 100644
index 000000000000..323ded5c0f4a
--- /dev/null
+++ b/Documentation/fpga/xrt.rst
@@ -0,0 +1,510 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================================
+XRTV2 Linux Kernel Driver Overview
+==================================
+
+Authors:
+
+* Sonal Santan <sonal.santan@xilinx.com>
+* Max Zhen <max.zhen@xilinx.com>
+* Lizhi Hou <lizhi.hou@xilinx.com>
+
+XRTV2 drivers are second generation `XRT <https://github.com/Xilinx/XRT>`_
+drivers which support `Alveo <https://www.xilinx.com/products/boards-and-kits/alveo.html>`_
+PCIe platforms from Xilinx.
+
+XRTV2 drivers support *subsystem* style data driven platforms where driver's
+configuration and behavior are determined by metadata provided by the platform
+(in *device tree* format). Primary management physical function (MPF) driver
+is called **xrt-mgmt**. Primary user physical function (UPF) driver is called
+**xrt-user** and is under development. xrt_driver framework and FPGA subsystem
+drivers are packaged into a library module called **xrt-lib**, which is shared
+by **xrt-mgmt** and **xrt-user** (under development). The xrt_driver framework
+implements a ``bus_type`` called **xrt_bus_type** which is used to discover HW
+subsystems and facilitate inter HW subsystem interaction.
+
+Driver Modules
+==============
+
+xrt-lib.ko
+----------
+
+xrt-lib is the repository of xrt drivers and pure software modules that can
+potentially be shared between xrt-mgmt and xrt-user. All these drivers are
+structured as **xrt_driver** and are instantiated by xrt-mgmt (or xrt-user under
+development) based on the metadata associated with the hardware. The metadata
+is in the form of a device tree as mentioned before.
+
+xrt-lib relies on OF kernel APIs to unflatten the metadata and overlay the
+unflattened device tree nodes to system device tree. In xrt-lib module initialization
+routine, "/xrt-bus" is created in system device tree, all XRT device
+tree nodes and properties will be under "/xrt-bus".
+
+The xrt-lib infrastructure provides hooks to xrt_drivers for device node
+management, user file operations and ioctl callbacks. The core infrastructure also
+provides a bus functionality called **xrt_bus_type** for xrt_driver registration,
+discovery and inter xrt_driver calls. xrt-lib does not have any dependency on PCIe
+subsystem.
+
+xrt-mgmt.ko
+------------
+
+The xrt-mgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
+PCIe device. It creates one or more *group* device and one or more *xleaf* device.
+The group and xleaf drivers are in xrt-lib and instantiations of the xrt_driver but
+are called group and xleaf to symbolize the logical operation performed by them.
+
+The xrt-mgmt driver uses xrt-lib APIs to manages the life cycle of multiple group
+drivers, which, in turn, manages multiple xleaf drivers. This flexibility allows
+xrt-mgmt.ko and xrt-lib.ko to support various HW subsystems exposed by different
+Alveo shells. The differences among these Alveo shells is handled in xleaf drivers.
+The group driver is part of the infrastructure which provides common services to xleaf
+drivers found on various Alveo shells. See :ref:`alveo_platform_overview`.
+
+The instantiation of specific group driver or xleaf drivers is completely data
+driven based on metadata (mostly in device tree format) found through VSEC
+capability and inside the firmware files, such as platform xsabin or user xclbin
+file.
+
+
+Driver Object Model
+===================
+
+The driver object model looks like the following::
+
+ +-----------+
+ | of root |
+ +-----+-----+
+ |
+ +---------+---------+
+ | |
+ +-----------+ +----------+
+ | xrt-bus | | ... |
+ +-----+-----+ +----------+
+ |
+ +-----------+-----------+
+ | |
+ v v
+ +-----------+ +-----------+
+ | group | ... | group |
+ +-----+-----+ +------+----+
+ | |
+ | |
+ +-----+----+ +-----+----+
+ | | | |
+ v v v v
+ +-------+ +-------+ +-------+ +-------+
+ | xleaf |..| xleaf | | xleaf |..| xleaf |
+ +-------+ +-------+ +-------+ +-------+
+
+As an example, for Xilinx Alveo U50 before user xclbin download, the tree
+looks like the following::
+
+ +-----------+
+ | xrt-bus |
+ +-----+-----+
+ |
+ +-------------------------+--------------------+
+ | | |
+ v v v
+ +--------+ +--------+ +--------+
+ | group0 | | group1 | | group2 |
+ +----+---+ +----+---+ +---+----+
+ | | |
+ | | |
+ +-----+-----+ +----+-----+---+ +-----+-----+----+--------+
+ | | | | | | | | |
+ v v | v v | v v |
+ +------------+ +------+ | +------+ +------+ | +------+ +-----------+ |
+ | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | |
+ +------------+ +------+ | +------+ +------+ | +------+ +-----------+ |
+ | +---------+ | +------+ +-----------+ |
+ +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+
+ +---------+ | +------+ +-----------+
+ | +-------+
+ +->| CALIB |
+ +-------+
+
+After a xclbin is downloaded, group3 will be added and the tree looks like the
+following::
+
+ +-----------+
+ | xrt-bus |
+ +-----+-----+
+ |
+ +-------------------------+--------------------+-----------------+
+ | | | |
+ v v v |
+ +--------+ +--------+ +--------+ |
+ | group0 | | group1 | | group2 | |
+ +----+---+ +----+---+ +---+----+ |
+ | | | |
+ | | | |
+ +-----+-----+ +-----+-----+---+ +-----+-----+----+--------+ |
+ | | | | | | | | | |
+ v v | v v | v v | |
+ +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
+ | xmgmt_main | | VSEC | | | GPIO | | QSPI | | | CMC | | AXI-GATE0 | | |
+ +------------+ +------+ | +------+ +------+ | +------+ +-----------+ | |
+ | +---------+ | +------+ +-----------+ | |
+ +>| MAILBOX | +->| ICAP | | AXI-GATE1 |<+ |
+ +---------+ | +------+ +-----------+ |
+ | +-------+ |
+ +->| CALIB | |
+ +-------+ |
+ +---+----+ |
+ | group3 |<--------------------------------------------+
+ +--------+
+ |
+ |
+ +-------+--------+---+--+--------+------+-------+
+ | | | | | | |
+ v | v | v | v
+ +--------+ | +--------+ | +--------+ | +-----+
+ | CLOCK0 | | | CLOCK1 | | | CLOCK2 | | | UCS |
+ +--------+ v +--------+ v +--------+ v +-----+
+ +-------------+ +-------------+ +-------------+
+ | CLOCK-FREQ0 | | CLOCK-FREQ1 | | CLOCK-FREQ2 |
+ +-------------+ +-------------+ +-------------+
+
+
+group
+-----
+
+The group driver represents a pseudo device whose life cycle is managed by
+root and does not have real IO mem or IRQ resources. It's part of the
+infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
+
+* manages one or more xleaf drivers
+* handle requests from xleaf drivers. For example event notifications and
+ inter xleaf calls.
+
+In xrt-mgmt, an initial group driver instance will be created by the PCIe driver.
+This instance contains xleaf drivers that will trigger group instances to be
+created to manage groups of xleaf drivers found on different partitions of
+hardware, such as VSEC, Shell, and User.
+
+xleaf
+-----
+
+The xleaf driver is a xrt_driver whose life cycle is managed by
+a group driver and may or may not have real IO mem or IRQ resources. They
+manage HW subsystems they are attached to.
+
+A xleaf driver without real hardware resources manages in-memory states for
+xrt-mgmt. These states are shareable by other xleaf drivers.
+
+Xleaf drivers assigned to specific hardware resources drive a specific subsystem
+in the device. To manipulate the subsystem or carry out a task, a xleaf driver
+may ask for help from the root via root calls and/or from other leaves via
+inter xleaf calls.
+
+A xleaf can also broadcast events through infrastructure code for other leaves
+to process. It can also receive event notification from infrastructure about
+certain events, such as post-creation or pre-exit of a particular xleaf.
+
+xrt_bus_type
+------------
+
+xrt_bus_type defines a virtual bus which handles xrt_driver probe, remove and match
+operations. All xrt_drivers register with xrt_bus_type as part of xrt-lib driver
+``module_init`` and un-register as part of xrt-lib driver ``module_exit``.
+
+FPGA Manager Interaction
+========================
+
+fpga_manager
+------------
+
+An instance of fpga_manager is created by xmgmt_main and is used for xclbin
+image download. fpga_manager requires the full xclbin image before it can
+start programming the FPGA configuration engine via Internal Configuration
+Access Port (ICAP) xrt_driver.
+
+fpga_region
+-----------
+
+For every interface exposed by the currently loaded xclbin/xsabin in the
+*parent* fpga_region a new instance of fpga_region is created like a *child*
+fpga_region. The device tree of the *parent* fpga_region defines the
+resources for a new instance of fpga_bridge which isolates the parent from
+child fpga_region. This new instance of fpga_bridge will be used when a
+xclbin image is loaded on the child fpga_region. After the xclbin image is
+downloaded to the fpga_region, an instance of a group is created for the
+fpga_region using the device tree obtained as part of the xclbin. If this
+device tree defines any child interfaces, it can trigger the creation of
+fpga_bridge and fpga_region for the next region in the chain.
+
+fpga_bridge
+-----------
+
+Like the fpga_region, an fpga_bridge is created by walking the device tree
+of the parent group. The bridge is used for isolation between a parent and
+its child.
+
+Driver Interfaces
+=================
+
+xrt-mgmt Driver Ioctls
+----------------------
+
+Ioctls exposed by the xrt-mgmt driver to user space are enumerated in the
+following table:
+
+== ===================== ============================ ==========================
+# Functionality ioctl request code data format
+== ===================== ============================ ==========================
+1 FPGA image download XMGMT_IOCICAPDOWNLOAD_AXLF xmgmt_ioc_bitstream_axlf
+== ===================== ============================ ==========================
+
+A user xclbin can be downloaded by using the xbmgmt tool from the XRT open source
+suite. See example usage below::
+
+ xbmgmt partition --program --path /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/test/verify.xclbin --force
+
+.. _alveo_platform_overview:
+
+Alveo Platform Overview
+=======================
+
+Alveo platforms are architected as two physical FPGA partitions: *Shell* and
+*User*. The Shell provides basic infrastructure for the Alveo platform like
+PCIe connectivity, board management, Dynamic Function Exchange (DFX), sensors,
+clocking, reset, and security. DFX, partial reconfiguration, is responsible for
+loading the user compiled FPGA binary.
+
+For DFX to work properly, physical partitions require strict HW compatibility
+with each other. Every physical partition has two interface UUIDs: the *parent*
+UUID and the *child* UUID. For simple single stage platforms, Shell → User forms
+the parent child relationship.
+
+.. note::
+ Partition compatibility matching is a key design component of the Alveo platforms
+ and XRT. Partitions have child and parent relationship. A loaded partition
+ exposes child partition UUID to advertise its compatibility requirement. When
+ loading a child partition, the xrt-mgmt driver matches the parent
+ UUID of the child partition against the child UUID exported by the parent.
+ The parent and child partition UUIDs are stored in the *xclbin* (for the user)
+ and the *xsabin* (for the shell). Except for the root UUID exported by VSEC,
+ the hardware itself does not know about the UUIDs. The UUIDs are stored in
+ xsabin and xclbin. The image format has a special node called Partition UUIDs
+ which define the compatibility UUIDs.
+
+
+The physical partitions and their loading are illustrated below::
+
+ SHELL USER
+ +-----------+ +-------------------+
+ | | | |
+ | VSEC UUID | CHILD PARENT | LOGIC UUID |
+ | o------->|<--------o |
+ | | UUID UUID | |
+ +-----+-----+ +--------+----------+
+ | |
+ . .
+ | |
+ +---+---+ +------+--------+
+ | POR | | USER COMPILED |
+ | FLASH | | XCLBIN |
+ +-------+ +---------------+
+
+
+Loading Sequence
+----------------
+
+The Shell partition is loaded from flash at system boot time. It establishes the
+PCIe link and exposes two physical functions to the BIOS. After the OS boots,
+the xrt-mgmt driver attaches to the PCIe physical function 0 exposed by the Shell
+and then looks for VSEC in the PCIe extended configuration space. Using VSEC, it
+determines the logic UUID of the Shell and uses the UUID to load matching *xsabin*
+file from Linux firmware directory. The xsabin file contains the metadata to
+discover the peripherals that are part of the Shell and the firmware for any
+embedded soft processors in the Shell. The xsabin file also contains Partition
+UUIDs.
+
+The Shell exports a child interface UUID which is used for the compatibility
+check when loading the user compiled xclbin over the User partition as part of DFX.
+When a user requests loading of a specific xclbin, the xrt-mgmt driver reads
+the parent interface UUID specified in the xclbin and matches it with the child
+interface UUID exported by the Shell to determine if the xclbin is compatible with
+the Shell. If the match fails, loading of xclbin is denied.
+
+xclbin loading is requested using the ICAP_DOWNLOAD_AXLF ioctl command. When loading
+a xclbin, the xrt-mgmt driver performs the following *logical* operations:
+
+1. Copy xclbin from user to kernel memory
+2. Sanity check the xclbin contents
+3. Isolate the User partition
+4. Download the bitstream using the FPGA config engine (ICAP)
+5. De-isolate the User partition
+6. Program the clocks (ClockWiz) driving the User partition
+7. Wait for the memory controller (MIG) calibration
+8. Return the loading status back to the caller
+
+`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
+provides more detailed information on platform loading.
+
+
+xsabin
+------
+
+Each Alveo platform comes packaged with its own xsabin. The xsabin is a trusted
+component of the platform. For format details refer to :ref:`xsabin_xclbin_container_format`
+below. xsabin contains basic information like UUIDs, platform name and metadata in the
+form of device tree. See :ref:`device_tree_usage` below for details and example.
+
+xclbin
+------
+
+xclbin is compiled by end user using
+`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_
+tool set from Xilinx. The xclbin contains sections describing user compiled
+acceleration engines/kernels, memory subsystems, clocking information etc. It also
+contains an FPGA bitstream for the user partition, UUIDs, platform name, etc.
+
+
+.. _xsabin_xclbin_container_format:
+
+xsabin/xclbin Container Format
+------------------------------
+
+xclbin/xsabin is ELF-like binary container format. It is structured as series of
+sections. There is a file header followed by several section headers which is
+followed by sections. A section header points to an actual section. There is an
+optional signature at the end. The format is defined by the header file ``xclbin.h``.
+The following figure illustrates a typical xclbin::
+
+
+ +---------------------+
+ | |
+ | HEADER |
+ +---------------------+
+ | SECTION HEADER |
+ | |
+ +---------------------+
+ | ... |
+ | |
+ +---------------------+
+ | SECTION HEADER |
+ | |
+ +---------------------+
+ | SECTION |
+ | |
+ +---------------------+
+ | ... |
+ | |
+ +---------------------+
+ | SECTION |
+ | |
+ +---------------------+
+ | SIGNATURE |
+ | (OPTIONAL) |
+ +---------------------+
+
+
+xclbin/xsabin files can be packaged, un-packaged and inspected using an XRT
+utility called **xclbinutil**. xclbinutil is part of the XRT open source
+software stack. The source code for xclbinutil can be found at
+https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
+
+For example, to enumerate the contents of a xclbin/xsabin use the *--info* switch
+as shown below::
+
+
+ xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
+ xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
+
+
+.. _device_tree_usage:
+
+Device Tree Usage
+-----------------
+
+The xsabin file stores metadata which advertise HW subsystems present in a
+partition. The metadata is stored in device tree format with a well defined
+schema. XRT management driver uses this information to create *xrt_devices* and
+bind *xrt_drivers* to them. The xrt_drivers could be independent modules or
+found in **xrt-lib.ko** kernel module.
+
+Deployment Models
+=================
+
+Baremetal
+---------
+
+In bare-metal deployments, both MPF and UPF are visible and accessible. The
+xrt-mgmt driver binds to MPF. The xrt-mgmt driver operations are privileged and
+available to system administrator. The full stack is illustrated below::
+
+ HOST
+
+ [XRT-MGMT] [XRT-USER]
+ | |
+ | |
+ +-----+ +-----+
+ | MPF | | UPF |
+ | | | |
+ | PF0 | | PF1 |
+ +--+--+ +--+--+
+ ......... ^................. ^..........
+ | |
+ | PCIe DEVICE |
+ | |
+ +--+------------------+--+
+ | SHELL |
+ | |
+ +------------------------+
+ | USER |
+ | |
+ | |
+ | |
+ | |
+ +------------------------+
+
+
+
+Virtualized
+-----------
+
+In virtualized deployments, the privileged MPF is assigned to the host but the
+unprivileged UPF is assigned to a guest VM via PCIe pass-through. The xrt-mgmt
+driver in host binds to MPF. The xrt-mgmt driver operations are privileged and
+only accessible to the MPF. The full stack is illustrated below::
+
+
+ ..............
+ HOST . VM .
+ . .
+ [XRT-MGMT] . [XRT-USER] .
+ | . | .
+ | . | .
+ +-----+ . +-----+ .
+ | MPF | . | UPF | .
+ | | . | | .
+ | PF0 | . | PF1 | .
+ +--+--+ . +--+--+ .
+ ......... ^................. ^..........
+ | |
+ | PCIe DEVICE |
+ | |
+ +--+------------------+--+
+ | SHELL |
+ | |
+ +------------------------+
+ | USER |
+ | |
+ | |
+ | |
+ | |
+ +------------------------+
+
+
+
+
+
+Platform Security Considerations
+================================
+
+`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
+discusses the deployment options and security implications in great detail.
diff --git a/MAINTAINERS b/MAINTAINERS
index 80eebc1d9ed5..fd7053bcfdb0 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7369,6 +7369,16 @@ F: Documentation/fpga/
F: drivers/fpga/
F: include/linux/fpga/
+FPGA XRT DRIVERS
+M: Lizhi Hou <lizhi.hou@xilinx.com>
+R: Max Zhen <max.zhen@xilinx.com>
+R: Sonal Santan <sonal.santan@xilinx.com>
+L: linux-fpga@vger.kernel.org
+S: Supported
+W: https://github.com/Xilinx/XRT
+F: Documentation/fpga/xrt.rst
+F: drivers/fpga/xrt/
+
FPU EMULATOR
M: Bill Metzenthen <billm@melbpc.org.au>
S: Maintained
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH V1 XRT Alveo Infrastructure 2/9] Documentation: devicetree: bindings: add xrt group binding
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 1/9] Documentation: fpga: Add a document describing XRT Alveo driver infrastructure Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
2021-11-18 22:09 ` Rob Herring
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 3/9] of: handle fdt buffer alignment inside unflatten function Lizhi Hou
` (6 subsequent siblings)
8 siblings, 1 reply; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
Create device tree binding document for xrt group device.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
.../bindings/xrt/xlnx,xrt-group.yaml | 59 +++++++++++++++++++
1 file changed, 59 insertions(+)
create mode 100644 Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml
diff --git a/Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml b/Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml
new file mode 100644
index 000000000000..112c493eb2d2
--- /dev/null
+++ b/Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml
@@ -0,0 +1,59 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/xrt/xlnx,xrt-group.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Xilinx XRT group for Alveo platforms
+
+description: |
+ The xrt group is a pseudo device which is used to manage and
+ support xrt devices in the same Alveo partition. It is part
+ of XRT infrastructure.
+
+maintainers:
+ - Lizhi Hou <lizhi.hou@xilinx.com>
+
+properties:
+ compatible:
+ const: xlnx,xrt-group
+
+ "#address-cells":
+ const: 3
+
+ "#size-cells":
+ const: 2
+
+ ranges: true
+
+patternProperties:
+ "^.*@[0-5],[0-9a-f]+,[0-9a-f]+$":
+ description: xrt devices belongs to this group
+ type: object
+
+required:
+ - compatible
+ - "#address-cells"
+ - "#size-cells"
+ - ranges
+
+additionalProperties: false
+
+examples:
+ - |
+ xrt-bus {
+ #address-cells = <2>;
+ #size-cells = <2>;
+ xrt-group@48,0 {
+ compatible = "xlnx,xrt-group";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ ranges = <0 0 0 0 0xe0000000 0 0x2000000
+ 2 0 0 0 0xe4200000 0 0x40000>;
+ ep_fpga_configuration_00@0,0,1e88000 {
+ reg = <0 0 0x1e88000 0 0x8000>;
+ compatible = "xilinx.com,reg_abs-axi_hwicap-1.0",
+ "axi_hwicap";
+ };
+ };
+ };
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH V1 XRT Alveo Infrastructure 2/9] Documentation: devicetree: bindings: add xrt group binding
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 2/9] Documentation: devicetree: bindings: add xrt group binding Lizhi Hou
@ 2021-11-18 22:09 ` Rob Herring
2021-11-19 1:06 ` Lizhi Hou
0 siblings, 1 reply; 12+ messages in thread
From: Rob Herring @ 2021-11-18 22:09 UTC (permalink / raw)
To: Lizhi Hou
Cc: linux-fpga, yliu, dwmw2, devicetree, stefanos, sonal.santan,
michal.simek, mdf, linux-kernel, Max Zhen, trix, maxz
On Thu, 18 Nov 2021 13:03:16 -0800, Lizhi Hou wrote:
> Create device tree binding document for xrt group device.
>
> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
> Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
> ---
> .../bindings/xrt/xlnx,xrt-group.yaml | 59 +++++++++++++++++++
> 1 file changed, 59 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml
>
My bot found errors running 'make DT_CHECKER_FLAGS=-m dt_binding_check'
on your patch (DT_CHECKER_FLAGS is new in v5.13):
yamllint warnings/errors:
./Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml:22:6: [warning] wrong indentation: expected 4 but found 5 (indentation)
./Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml:25:6: [warning] wrong indentation: expected 4 but found 5 (indentation)
dtschema/dtc warnings/errors:
Documentation/devicetree/bindings/xrt/xlnx,xrt-group.example.dt.yaml:0:0: /example-0/xrt-bus/xrt-group@48,0/ep_fpga_configuration_00@0,0,1e88000: failed to match any schema with compatible: ['xilinx.com,reg_abs-axi_hwicap-1.0', 'axi_hwicap']
Documentation/devicetree/bindings/xrt/xlnx,xrt-group.example.dt.yaml:0:0: /example-0/xrt-bus/xrt-group@48,0/ep_fpga_configuration_00@0,0,1e88000: failed to match any schema with compatible: ['xilinx.com,reg_abs-axi_hwicap-1.0', 'axi_hwicap']
doc reference errors (make refcheckdocs):
See https://patchwork.ozlabs.org/patch/1556809
This check can fail if there are any dependencies. The base for a patch
series is generally the most recent rc1.
If you already ran 'make dt_binding_check' and didn't see the above
error(s), then make sure 'yamllint' is installed and dt-schema is up to
date:
pip3 install dtschema --upgrade
Please check and re-submit.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH V1 XRT Alveo Infrastructure 2/9] Documentation: devicetree: bindings: add xrt group binding
2021-11-18 22:09 ` Rob Herring
@ 2021-11-19 1:06 ` Lizhi Hou
0 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-19 1:06 UTC (permalink / raw)
To: Rob Herring, Lizhi Hou
Cc: linux-fpga, yliu, dwmw2, devicetree, stefanos, sonal.santan,
michal.simek, mdf, linux-kernel, Max Zhen, trix, maxz
I was able to reproduce the error after following the instructions. I
will include the fix with next version of this patch set.
Thanks,
Lizhi
On 11/18/21 2:09 PM, Rob Herring wrote
>
> On Thu, 18 Nov 2021 13:03:16 -0800, Lizhi Hou wrote:
>> Create device tree binding document for xrt group device.
>>
>> Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
>> Signed-off-by: Max Zhen <max.zhen@xilinx.com>
>> Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
>> ---
>> .../bindings/xrt/xlnx,xrt-group.yaml | 59 +++++++++++++++++++
>> 1 file changed, 59 insertions(+)
>> create mode 100644 Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml
>>
> My bot found errors running 'make DT_CHECKER_FLAGS=-m dt_binding_check'
> on your patch (DT_CHECKER_FLAGS is new in v5.13):
>
> yamllint warnings/errors:
> ./Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml:22:6: [warning] wrong indentation: expected 4 but found 5 (indentation)
> ./Documentation/devicetree/bindings/xrt/xlnx,xrt-group.yaml:25:6: [warning] wrong indentation: expected 4 but found 5 (indentation)
>
> dtschema/dtc warnings/errors:
> Documentation/devicetree/bindings/xrt/xlnx,xrt-group.example.dt.yaml:0:0: /example-0/xrt-bus/xrt-group@48,0/ep_fpga_configuration_00@0,0,1e88000: failed to match any schema with compatible: ['xilinx.com,reg_abs-axi_hwicap-1.0', 'axi_hwicap']
> Documentation/devicetree/bindings/xrt/xlnx,xrt-group.example.dt.yaml:0:0: /example-0/xrt-bus/xrt-group@48,0/ep_fpga_configuration_00@0,0,1e88000: failed to match any schema with compatible: ['xilinx.com,reg_abs-axi_hwicap-1.0', 'axi_hwicap']
>
> doc reference errors (make refcheckdocs):
>
> See https://patchwork.ozlabs.org/patch/1556809
>
> This check can fail if there are any dependencies. The base for a patch
> series is generally the most recent rc1.
>
> If you already ran 'make dt_binding_check' and didn't see the above
> error(s), then make sure 'yamllint' is installed and dt-schema is up to
> date:
>
> pip3 install dtschema --upgrade
>
> Please check and re-submit.
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH V1 XRT Alveo Infrastructure 3/9] of: handle fdt buffer alignment inside unflatten function
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 1/9] Documentation: fpga: Add a document describing XRT Alveo driver infrastructure Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 2/9] Documentation: devicetree: bindings: add xrt group binding Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 4/9] of: create empty of root Lizhi Hou
` (5 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
Add alignment check to of_fdt_unflatten_tree(). If it is not aligned,
allocate a aligned buffer and copy the fdt blob. So the caller does not
have to deal with the buffer alignment before calling this function.
XRT uses this function to unflatten fdt which is from Alveo firmware.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
drivers/of/fdt.c | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
index 4546572af24b..d64445e43ceb 100644
--- a/drivers/of/fdt.c
+++ b/drivers/of/fdt.c
@@ -455,13 +455,28 @@ void *of_fdt_unflatten_tree(const unsigned long *blob,
struct device_node *dad,
struct device_node **mynodes)
{
+ void *new_fdt = NULL, *fdt_align;
void *mem;
+ if (fdt_check_header(blob)) {
+ pr_err("Invalid fdt blob\n");
+ return NULL;
+ }
+ fdt_align = (void *)PTR_ALIGN(blob, FDT_ALIGN_SIZE);
+ if (fdt_align != blob) {
+ new_fdt = kmalloc(fdt_totalsize(blob) + FDT_ALIGN_SIZE, GFP_KERNEL);
+ if (!new_fdt)
+ return NULL;
+ fdt_align = PTR_ALIGN(new_fdt, FDT_ALIGN_SIZE);
+ }
+
mutex_lock(&of_fdt_unflatten_mutex);
- mem = __unflatten_device_tree(blob, dad, mynodes, &kernel_tree_alloc,
+ mem = __unflatten_device_tree(fdt_align, dad, mynodes, &kernel_tree_alloc,
true);
mutex_unlock(&of_fdt_unflatten_mutex);
+ kfree(new_fdt);
+
return mem;
}
EXPORT_SYMBOL_GPL(of_fdt_unflatten_tree);
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH V1 XRT Alveo Infrastructure 4/9] of: create empty of root
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
` (2 preceding siblings ...)
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 3/9] of: handle fdt buffer alignment inside unflatten function Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 5/9] fpga: xrt: xrt-lib initialization Lizhi Hou
` (4 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
When OF_FLATTREE is selected and there is not a device tree, create an
empty device tree root node. of/unittest.c code is referenced.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
drivers/of/Makefile | 2 +-
drivers/of/fdt.c | 20 ++++++++++++++++++++
drivers/of/fdt_default.dts | 5 +++++
drivers/of/of_private.h | 5 +++++
4 files changed, 31 insertions(+), 1 deletion(-)
create mode 100644 drivers/of/fdt_default.dts
diff --git a/drivers/of/Makefile b/drivers/of/Makefile
index c13b982084a3..815f5220465b 100644
--- a/drivers/of/Makefile
+++ b/drivers/of/Makefile
@@ -2,7 +2,7 @@
obj-y = base.o device.o platform.o property.o
obj-$(CONFIG_OF_KOBJ) += kobj.o
obj-$(CONFIG_OF_DYNAMIC) += dynamic.o
-obj-$(CONFIG_OF_FLATTREE) += fdt.o
+obj-$(CONFIG_OF_FLATTREE) += fdt.o fdt_default.dtb.o
obj-$(CONFIG_OF_EARLY_FLATTREE) += fdt_address.o
obj-$(CONFIG_OF_PROMTREE) += pdt.o
obj-$(CONFIG_OF_ADDRESS) += address.o
diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
index d64445e43ceb..3d6e4543419e 100644
--- a/drivers/of/fdt.c
+++ b/drivers/of/fdt.c
@@ -481,6 +481,26 @@ void *of_fdt_unflatten_tree(const unsigned long *blob,
}
EXPORT_SYMBOL_GPL(of_fdt_unflatten_tree);
+static int __init of_fdt_root_init(void)
+{
+ struct device_node *np;
+
+ if (of_root)
+ return 0;
+
+ if (!of_fdt_unflatten_tree((const unsigned long *)__dtb_fdt_default_begin,
+ NULL, &of_root)) {
+ pr_warn("%s: unflatten default tree failed\n", __func__);
+ return -ENODATA;
+ }
+
+ for_each_of_allnodes(np)
+ __of_attach_node_sysfs(np);
+
+ return 0;
+}
+late_initcall(of_fdt_root_init);
+
/* Everything below here references initial_boot_params directly. */
int __initdata dt_root_addr_cells;
int __initdata dt_root_size_cells;
diff --git a/drivers/of/fdt_default.dts b/drivers/of/fdt_default.dts
new file mode 100644
index 000000000000..d1f12a76dfc6
--- /dev/null
+++ b/drivers/of/fdt_default.dts
@@ -0,0 +1,5 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+
+/ {
+};
diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
index 631489f7f8c0..47c6bb47ef25 100644
--- a/drivers/of/of_private.h
+++ b/drivers/of/of_private.h
@@ -41,6 +41,11 @@ extern struct mutex of_mutex;
extern struct list_head aliases_lookup;
extern struct kset *of_kset;
+#if defined(CONFIG_OF_FLATTREE)
+extern u8 __dtb_fdt_default_begin[];
+extern u8 __dtb_fdt_default_end[];
+#endif
+
#if defined(CONFIG_OF_DYNAMIC)
extern int of_property_notify(int action, struct device_node *np,
struct property *prop, struct property *old_prop);
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH V1 XRT Alveo Infrastructure 5/9] fpga: xrt: xrt-lib initialization
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
` (3 preceding siblings ...)
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 4/9] of: create empty of root Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 6/9] fpga: xrt: xrt bus and device Lizhi Hou
` (3 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
xrt-lib module initialization routine creates /xrt-bus device tree node
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
drivers/fpga/Kconfig | 3 +++
drivers/fpga/Makefile | 3 +++
drivers/fpga/xrt/Kconfig | 6 +++++
drivers/fpga/xrt/lib/Kconfig | 16 +++++++++++++
drivers/fpga/xrt/lib/Makefile | 16 +++++++++++++
drivers/fpga/xrt/lib/lib-drv.c | 41 ++++++++++++++++++++++++++++++++
drivers/fpga/xrt/lib/lib-drv.h | 15 ++++++++++++
drivers/fpga/xrt/lib/xrt-bus.dts | 13 ++++++++++
8 files changed, 113 insertions(+)
create mode 100644 drivers/fpga/xrt/Kconfig
create mode 100644 drivers/fpga/xrt/lib/Kconfig
create mode 100644 drivers/fpga/xrt/lib/Makefile
create mode 100644 drivers/fpga/xrt/lib/lib-drv.c
create mode 100644 drivers/fpga/xrt/lib/lib-drv.h
create mode 100644 drivers/fpga/xrt/lib/xrt-bus.dts
diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 991b3f361ec9..93ae387c97c5 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -243,4 +243,7 @@ config FPGA_MGR_VERSAL_FPGA
configure the programmable logic(PL).
To compile this as a module, choose M here.
+
+source "drivers/fpga/xrt/Kconfig"
+
endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 0bff783d1b61..5bd41cf4c7ec 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -49,3 +49,6 @@ obj-$(CONFIG_FPGA_DFL_NIOS_INTEL_PAC_N3000) += dfl-n3000-nios.o
# Drivers for FPGAs which implement DFL
obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
+
+# XRT drivers for Alveo
+obj-$(CONFIG_FPGA_XRT_LIB) += xrt/lib/
diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig
new file mode 100644
index 000000000000..04c3bb5aaf4f
--- /dev/null
+++ b/drivers/fpga/xrt/Kconfig
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Xilinx Alveo FPGA device configuration
+#
+
+source "drivers/fpga/xrt/lib/Kconfig"
diff --git a/drivers/fpga/xrt/lib/Kconfig b/drivers/fpga/xrt/lib/Kconfig
new file mode 100644
index 000000000000..bb44956d9f94
--- /dev/null
+++ b/drivers/fpga/xrt/lib/Kconfig
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# XRT Alveo FPGA device configuration
+#
+
+config FPGA_XRT_LIB
+ tristate "XRT Alveo Driver Library"
+ depends on HWMON && PCI && HAS_IOMEM && OF_FLATTREE && OF_OVERLAY
+ select REGMAP_MMIO
+ help
+ Select this option to enable Xilinx XRT Alveo driver library. This
+ library is core infrastructure of XRT Alveo FPGA drivers which
+ provides functions for working with device nodes, iteration and
+ lookup of platform devices, common interfaces for platform devices,
+ plumbing of function call and ioctls between platform devices and
+ parent partitions.
diff --git a/drivers/fpga/xrt/lib/Makefile b/drivers/fpga/xrt/lib/Makefile
new file mode 100644
index 000000000000..f67bb19ef20a
--- /dev/null
+++ b/drivers/fpga/xrt/lib/Makefile
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
+#
+# Authors: Sonal.Santan@xilinx.com
+#
+
+FULL_XRT_PATH=$(srctree)/$(src)/..
+
+obj-$(CONFIG_FPGA_XRT_LIB) += xrt-lib.o
+
+xrt-lib-objs := \
+ lib-drv.o \
+ xrt-bus.dtb.o
+
+ccflags-y := -I$(FULL_XRT_PATH)/include
diff --git a/drivers/fpga/xrt/lib/lib-drv.c b/drivers/fpga/xrt/lib/lib-drv.c
new file mode 100644
index 000000000000..d4597cd4767f
--- /dev/null
+++ b/drivers/fpga/xrt/lib/lib-drv.c
@@ -0,0 +1,41 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ * Lizhi Hou <lizhi.hou@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/of_device.h>
+#include "lib-drv.h"
+
+static int xrt_bus_ovcs_id;
+
+static __init int xrt_lib_init(void)
+{
+ int ret;
+
+ ret = of_overlay_fdt_apply(__dtb_xrt_bus_begin,
+ __dtb_xrt_bus_end - __dtb_xrt_bus_begin,
+ &xrt_bus_ovcs_id);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static __exit void xrt_lib_fini(void)
+{
+ of_overlay_remove(&xrt_bus_ovcs_id);
+}
+
+module_init(xrt_lib_init);
+module_exit(xrt_lib_fini);
+
+MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
+MODULE_DESCRIPTION("Xilinx Alveo IP Lib driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/fpga/xrt/lib/lib-drv.h b/drivers/fpga/xrt/lib/lib-drv.h
new file mode 100644
index 000000000000..4bf8a32c7ec5
--- /dev/null
+++ b/drivers/fpga/xrt/lib/lib-drv.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _LIB_DRV_H_
+#define _LIB_DRV_H_
+
+extern u8 __dtb_xrt_bus_begin[];
+extern u8 __dtb_xrt_bus_end[];
+
+#endif /* _LIB_DRV_H_ */
diff --git a/drivers/fpga/xrt/lib/xrt-bus.dts b/drivers/fpga/xrt/lib/xrt-bus.dts
new file mode 100644
index 000000000000..0720de26851b
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xrt-bus.dts
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+/plugin/;
+
+/*
+ * xrt bus node which is overlayed dynamically when xrt-lib is loaded.
+ */
+&{/} {
+ xrt-bus {
+ #address-cells=<2>;
+ #size-cells=<2>;
+ };
+};
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH V1 XRT Alveo Infrastructure 6/9] fpga: xrt: xrt bus and device
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
` (4 preceding siblings ...)
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 5/9] fpga: xrt: xrt-lib initialization Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 7/9] fpga: xrt: lib-xrt xroot APIs Lizhi Hou
` (2 subsequent siblings)
8 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
xrt-lib kernel module infrastructure code to add xrt_bus_type, xrt_driver
and xrt_device
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
drivers/fpga/xrt/lib/lib-drv.c | 192 +++++++++++++++++++++++++++++++++
include/linux/xrt/xdevice.h | 128 ++++++++++++++++++++++
2 files changed, 320 insertions(+)
create mode 100644 include/linux/xrt/xdevice.h
diff --git a/drivers/fpga/xrt/lib/lib-drv.c b/drivers/fpga/xrt/lib/lib-drv.c
index d4597cd4767f..3ad02a7c2aac 100644
--- a/drivers/fpga/xrt/lib/lib-drv.c
+++ b/drivers/fpga/xrt/lib/lib-drv.c
@@ -11,10 +11,195 @@
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <linux/of_device.h>
+#include <linux/xrt/xdevice.h>
#include "lib-drv.h"
static int xrt_bus_ovcs_id;
+#define XRT_DRVNAME(drv) ((drv)->driver.name)
+
+static DEFINE_IDA(xrt_device_ida);
+
+static int xrt_bus_match(struct device *dev, struct device_driver *drv)
+{
+ if (of_driver_match_device(dev, drv))
+ return 1;
+
+ return 0;
+}
+
+static int xrt_bus_probe(struct device *dev)
+{
+ struct xrt_driver *xdrv = to_xrt_drv(dev->driver);
+ struct xrt_device *xdev = to_xrt_dev(dev);
+
+ return xdrv->probe(xdev);
+}
+
+static void xrt_bus_remove(struct device *dev)
+{
+ struct xrt_driver *xdrv = to_xrt_drv(dev->driver);
+ struct xrt_device *xdev = to_xrt_dev(dev);
+
+ if (xdrv->remove)
+ xdrv->remove(xdev);
+}
+
+struct bus_type xrt_bus_type = {
+ .name = "xrt",
+ .match = xrt_bus_match,
+ .probe = xrt_bus_probe,
+ .remove = xrt_bus_remove,
+};
+
+int xrt_register_driver(struct xrt_driver *drv)
+{
+ const char *drvname = XRT_DRVNAME(drv);
+ int rc = 0;
+
+ /* Initialize dev_t for char dev node. */
+ if (drv->file_ops.xsf_ops.open) {
+ rc = alloc_chrdev_region(&drv->file_ops.xsf_dev_t, 0,
+ XRT_MAX_DEVICE_NODES, drvname);
+ if (rc) {
+ pr_err("failed to alloc dev minor for %s: %d\n", drvname, rc);
+ return rc;
+ }
+ } else {
+ drv->file_ops.xsf_dev_t = (dev_t)-1;
+ }
+
+ drv->driver.bus = &xrt_bus_type;
+
+ rc = driver_register(&drv->driver);
+ if (rc) {
+ pr_err("register %s xrt driver failed\n", drvname);
+ if (drv->file_ops.xsf_dev_t != (dev_t)-1) {
+ unregister_chrdev_region(drv->file_ops.xsf_dev_t,
+ XRT_MAX_DEVICE_NODES);
+ }
+ return rc;
+ }
+
+ pr_info("%s registered successfully\n", drvname);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xrt_register_driver);
+
+void xrt_unregister_driver(struct xrt_driver *drv)
+{
+ driver_unregister(&drv->driver);
+
+ if (drv->file_ops.xsf_dev_t != (dev_t)-1)
+ unregister_chrdev_region(drv->file_ops.xsf_dev_t, XRT_MAX_DEVICE_NODES);
+
+ pr_info("%s unregistered successfully\n", XRT_DRVNAME(drv));
+}
+EXPORT_SYMBOL_GPL(xrt_unregister_driver);
+
+static int xrt_dev_get_instance(void)
+{
+ int ret;
+
+ ret = ida_alloc_range(&xrt_device_ida, 0, 0x7fffffff, GFP_KERNEL);
+ if (ret < 0)
+ return ret;
+
+ return ret;
+}
+
+static void xrt_dev_put_instance(int instance)
+{
+ ida_free(&xrt_device_ida, instance);
+}
+
+static void xrt_device_release(struct device *dev)
+{
+ struct xrt_device *xdev = container_of(dev, struct xrt_device, dev);
+
+ kfree(xdev);
+}
+
+void xrt_device_unregister(struct xrt_device *xdev)
+{
+ if (xdev->state == XRT_DEVICE_STATE_ADDED)
+ device_del(&xdev->dev);
+
+ vfree(xdev->sdev_data);
+ kfree(xdev->resource);
+
+ if (xdev->instance != XRT_INVALID_DEVICE_INST)
+ xrt_dev_put_instance(xdev->instance);
+
+ if (xdev->dev.of_node)
+ of_node_put(xdev->dev.of_node);
+
+ if (xdev->dev.release == xrt_device_release)
+ put_device(&xdev->dev);
+}
+
+struct xrt_device *
+xrt_device_register(struct device *parent, struct device_node *dn,
+ struct resource *res, u32 res_num,
+ void *pdata, size_t data_sz)
+{
+ struct xrt_device *xdev = NULL;
+ int ret;
+
+ xdev = kzalloc(sizeof(*xdev), GFP_KERNEL);
+ if (!xdev)
+ return NULL;
+ xdev->instance = XRT_INVALID_DEVICE_INST;
+
+ /* Obtain dev instance number. */
+ ret = xrt_dev_get_instance();
+ if (ret < 0) {
+ dev_err(parent, "failed get instance, ret %d", ret);
+ goto fail;
+ }
+
+ xdev->instance = ret;
+ xdev->name = dn->full_name;
+ device_initialize(&xdev->dev);
+ xdev->dev.release = xrt_device_release;
+ xdev->dev.parent = parent;
+
+ xdev->dev.bus = &xrt_bus_type;
+ dev_set_name(&xdev->dev, "%s.%d", xdev->name, xdev->instance);
+
+ if (res_num > 0) {
+ xdev->num_resources = res_num;
+ xdev->resource = kmemdup(res, sizeof(*res) * res_num, GFP_KERNEL);
+ if (!xdev->resource)
+ goto fail;
+ }
+
+ if (data_sz > 0) {
+ xdev->sdev_data = vzalloc(data_sz);
+ if (!xdev->sdev_data)
+ goto fail;
+
+ memcpy(xdev->sdev_data, pdata, data_sz);
+ }
+
+ ret = device_add(&xdev->dev);
+ if (ret) {
+ dev_err(parent, "failed add device, ret %d", ret);
+ goto fail;
+ }
+ xdev->state = XRT_DEVICE_STATE_ADDED;
+ xdev->dev.of_node = of_node_get(dn);
+
+ return xdev;
+
+fail:
+ xrt_device_unregister(xdev);
+ kfree(xdev);
+
+ return NULL;
+}
+
static __init int xrt_lib_init(void)
{
int ret;
@@ -25,11 +210,18 @@ static __init int xrt_lib_init(void)
if (ret)
return ret;
+ ret = bus_register(&xrt_bus_type);
+ if (ret) {
+ of_overlay_remove(&xrt_bus_ovcs_id);
+ return ret;
+ }
+
return 0;
}
static __exit void xrt_lib_fini(void)
{
+ bus_unregister(&xrt_bus_type);
of_overlay_remove(&xrt_bus_ovcs_id);
}
diff --git a/include/linux/xrt/xdevice.h b/include/linux/xrt/xdevice.h
new file mode 100644
index 000000000000..0b3fec9ae598
--- /dev/null
+++ b/include/linux/xrt/xdevice.h
@@ -0,0 +1,128 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <lizhi.hou@xilinx.com>
+ */
+
+#ifndef _XRT_DEVICE_H_
+#define _XRT_DEVICE_H_
+
+#include <linux/fs.h>
+#include <linux/cdev.h>
+#include <linux/of.h>
+
+#define XRT_MAX_DEVICE_NODES 128
+#define XRT_INVALID_DEVICE_INST (XRT_MAX_DEVICE_NODES + 1)
+
+enum {
+ XRT_DEVICE_STATE_NONE = 0,
+ XRT_DEVICE_STATE_ADDED
+};
+
+/*
+ * struct xrt_device - represent an xrt device on xrt bus
+ *
+ * dev: generic device interface.
+ * id: id of the xrt device.
+ */
+struct xrt_device {
+ struct device dev;
+ const char *name;
+ u32 instance;
+ u32 state;
+ u32 num_resources;
+ struct resource *resource;
+ void *sdev_data;
+};
+
+/*
+ * If populated by xrt device driver, infra will handle the mechanics of
+ * char device (un)registration.
+ */
+enum xrt_dev_file_mode {
+ /* Infra create cdev, default file name */
+ XRT_DEV_FILE_DEFAULT = 0,
+ /* Infra create cdev, need to encode inst num in file name */
+ XRT_DEV_FILE_MULTI_INST,
+ /* No auto creation of cdev by infra, leaf handles it by itself */
+ XRT_DEV_FILE_NO_AUTO,
+};
+
+struct xrt_dev_file_ops {
+ const struct file_operations xsf_ops;
+ dev_t xsf_dev_t;
+ const char *xsf_dev_name;
+ enum xrt_dev_file_mode xsf_mode;
+};
+
+/*
+ * this struct define the endpoints belong to the same xrt device
+ */
+struct xrt_dev_ep_names {
+ const char *ep_name;
+ const char *compat;
+};
+
+/*
+ * struct xrt_driver - represent a xrt device driver
+ *
+ * driver: driver model structure.
+ * id_table: pointer to table of device IDs the driver is interested in.
+ * { } member terminated.
+ * probe: mandatory callback for device binding.
+ * remove: callback for device unbinding.
+ */
+struct xrt_driver {
+ struct device_driver driver;
+ struct xrt_dev_file_ops file_ops;
+
+ /*
+ * Subdev driver callbacks populated by subdev driver.
+ */
+ int (*probe)(struct xrt_device *xrt_dev);
+ void (*remove)(struct xrt_device *xrt_dev);
+ /*
+ * If leaf_call is defined, these are called by other leaf drivers.
+ * Note that root driver may call into leaf_call of a group driver.
+ */
+ int (*leaf_call)(struct xrt_device *xrt_dev, u32 cmd, void *arg);
+};
+
+#define to_xrt_dev(d) container_of(d, struct xrt_device, dev)
+#define to_xrt_drv(d) container_of(d, struct xrt_driver, driver)
+/*
+ * module_xrt_driver() - Helper macro for drivers that don't do
+ * anything special in module init/exit.
+ */
+#define module_xrt_driver(__xrt_driver) \
+ module_driver(__xrt_driver, xrt_register_driver, \
+ xrt_unregister_driver)
+
+static inline void *xrt_get_drvdata(const struct xrt_device *xdev)
+{
+ return dev_get_drvdata(&xdev->dev);
+}
+
+static inline void xrt_set_drvdata(struct xrt_device *xdev, void *data)
+{
+ dev_set_drvdata(&xdev->dev, data);
+}
+
+static inline void *xrt_get_xdev_data(struct device *dev)
+{
+ struct xrt_device *xdev = to_xrt_dev(dev);
+
+ return xdev->sdev_data;
+}
+
+struct xrt_device *
+xrt_device_register(struct device *parent, struct device_node *dn,
+ struct resource *res, u32 res_num,
+ void *pdata, size_t data_sz);
+void xrt_device_unregister(struct xrt_device *xdev);
+int xrt_register_driver(struct xrt_driver *drv);
+void xrt_unregister_driver(struct xrt_driver *drv);
+
+#endif /* _XRT_DEVICE_H_ */
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH V1 XRT Alveo Infrastructure 7/9] fpga: xrt: lib-xrt xroot APIs
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
` (5 preceding siblings ...)
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 6/9] fpga: xrt: xrt bus and device Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 8/9] fpga: xrt: xrt group device driver Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 9/9] fpga: xrt: management physical function driver Lizhi Hou
8 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
lib-xrt infrastructure code to create xrt group device.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
drivers/fpga/xrt/include/xroot.h | 30 +++++
drivers/fpga/xrt/lib/Makefile | 1 +
drivers/fpga/xrt/lib/xroot.c | 210 +++++++++++++++++++++++++++++++
3 files changed, 241 insertions(+)
create mode 100644 drivers/fpga/xrt/include/xroot.h
create mode 100644 drivers/fpga/xrt/lib/xroot.c
diff --git a/drivers/fpga/xrt/include/xroot.h b/drivers/fpga/xrt/include/xroot.h
new file mode 100644
index 000000000000..6ea1b64b3ae3
--- /dev/null
+++ b/drivers/fpga/xrt/include/xroot.h
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#ifndef _XRT_ROOT_H_
+#define _XRT_ROOT_H_
+
+struct xroot_range {
+ __be32 child_addr[3];
+ __be32 parent_addr[2];
+ __be32 child_size[2];
+};
+
+struct xroot_info {
+ u32 addr;
+ int num_range;
+ struct xroot_range *ranges;
+};
+
+int xroot_probe(struct device *dev, struct xroot_info *info, void **root);
+void xroot_remove(void *root);
+
+int xroot_create_group(void *xr, void *dtb);
+void xroot_destroy_group(void *xr, u32 grp_id);
+
+#endif /* _XRT_ROOT_H_ */
diff --git a/drivers/fpga/xrt/lib/Makefile b/drivers/fpga/xrt/lib/Makefile
index f67bb19ef20a..fd2af2cbd1da 100644
--- a/drivers/fpga/xrt/lib/Makefile
+++ b/drivers/fpga/xrt/lib/Makefile
@@ -11,6 +11,7 @@ obj-$(CONFIG_FPGA_XRT_LIB) += xrt-lib.o
xrt-lib-objs := \
lib-drv.o \
+ xroot.o \
xrt-bus.dtb.o
ccflags-y := -I$(FULL_XRT_PATH)/include
diff --git a/drivers/fpga/xrt/lib/xroot.c b/drivers/fpga/xrt/lib/xroot.c
new file mode 100644
index 000000000000..34489a0cc10b
--- /dev/null
+++ b/drivers/fpga/xrt/lib/xroot.c
@@ -0,0 +1,210 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Root Functions
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ * Lizhi Hou <lizhih@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/of.h>
+#include <linux/of_fdt.h>
+#include <linux/xrt/xdevice.h>
+#include "xroot.h"
+
+#define xroot_err(xr, fmt, args...) dev_err((xr)->dev, "%s: " fmt, __func__, ##args)
+#define xroot_warn(xr, fmt, args...) dev_warn((xr)->dev, "%s: " fmt, __func__, ##args)
+#define xroot_info(xr, fmt, args...) dev_info((xr)->dev, "%s: " fmt, __func__, ##args)
+#define xroot_dbg(xr, fmt, args...) dev_dbg((xr)->dev, "%s: " fmt, __func__, ##args)
+
+struct xroot {
+ struct device *dev;
+ struct list_head groups;
+ u32 addr;
+ struct xroot_range *ranges;
+ int num_range;
+ struct ida grp_ida;
+};
+
+struct xroot_group {
+ struct list_head node;
+ struct xroot *xr;
+ struct xrt_device *grp_dev;
+ struct property ranges;
+ struct of_changeset chgset;
+ bool chgset_applied;
+ void *dn_mem;
+ char *name;
+ int id;
+};
+
+#define XRT_GROUP "xrt-group"
+#define MAX_GRP_NAME_LEN 64
+
+static void xroot_cleanup_group(struct xroot_group *grp)
+{
+ if (grp->grp_dev)
+ xrt_device_unregister(grp->grp_dev);
+ if (grp->chgset_applied)
+ of_changeset_revert(&grp->chgset);
+ of_changeset_destroy(&grp->chgset);
+
+ if (grp->id >= 0)
+ ida_free(&grp->xr->grp_ida, grp->id);
+ kfree(grp->dn_mem);
+ kfree(grp->name);
+}
+
+void xroot_destroy_group(void *root, u32 grp_id)
+{
+ struct xroot *xr = root;
+ struct xroot_group *grp;
+
+ list_for_each_entry(grp, &xr->groups, node) {
+ if (grp->id == grp_id)
+ break;
+ }
+ if (list_entry_is_head(grp, &xr->groups, node))
+ return;
+
+ list_del(&grp->node);
+
+ xroot_cleanup_group(grp);
+ kfree(grp);
+}
+EXPORT_SYMBOL_GPL(xroot_destroy_group);
+
+/*
+ * Create XRT group device.
+ *
+ * Unflatten the device tree blob attached to the group and
+ * overlay the device nodes under /xrt-bus. Then create group device
+ * and link it to device node.
+ */
+int xroot_create_group(void *root, void *dtb)
+{
+ struct device_node *dn, *bus, *grp_dn;
+ struct xroot *xr = root;
+ struct xroot_group *grp;
+ int ret;
+
+ grp = kzalloc(sizeof(*grp), GFP_KERNEL);
+ if (!grp)
+ return -ENOMEM;
+
+ bus = of_find_node_by_path("/xrt-bus");
+ if (!bus) {
+ kfree(grp);
+ return -EINVAL;
+ }
+ grp->xr = xr;
+ of_changeset_init(&grp->chgset);
+
+ ret = ida_alloc(&xr->grp_ida, GFP_KERNEL);
+ if (ret < 0)
+ goto failed;
+
+ grp->id = ret;
+
+ grp->name = kzalloc(MAX_GRP_NAME_LEN, GFP_KERNEL);
+ if (!grp->name) {
+ ret = -ENOMEM;
+ goto failed;
+ }
+ snprintf(grp->name, MAX_GRP_NAME_LEN, "%s@%x,%x", XRT_GROUP, xr->addr, grp->id);
+
+ grp->dn_mem = of_fdt_unflatten_tree(dtb, NULL, &grp_dn);
+ if (!grp->dn_mem) {
+ ret = -EINVAL;
+ goto failed;
+ }
+
+ of_node_get(grp_dn);
+ grp_dn->full_name = grp->name;
+ grp_dn->parent = bus;
+ for (dn = grp_dn; dn; dn = of_find_all_nodes(dn))
+ of_changeset_attach_node(&grp->chgset, dn);
+
+ grp->ranges.name = "ranges";
+ grp->ranges.length = xr->num_range * sizeof(*xr->ranges);
+ grp->ranges.value = xr->ranges;
+ ret = of_changeset_add_property(&grp->chgset, grp_dn, &grp->ranges);
+ if (ret)
+ goto failed;
+
+ ret = of_changeset_apply(&grp->chgset);
+ if (ret)
+ goto failed;
+ grp->chgset_applied = true;
+
+ of_node_put(bus);
+ bus = NULL;
+
+ grp->grp_dev = xrt_device_register(xr->dev, grp_dn, NULL, 0, NULL, 0);
+ if (!grp->grp_dev) {
+ ret = -EFAULT;
+ goto failed;
+ }
+
+ if (device_attach(&grp->grp_dev->dev) != 1) {
+ ret = -EFAULT;
+ xroot_err(xr, "failed to attach");
+ goto failed;
+ }
+
+ list_add(&grp->node, &xr->groups);
+
+ return grp->id;
+
+failed:
+ if (bus)
+ of_node_put(bus);
+ xroot_cleanup_group(grp);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(xroot_create_group);
+
+int xroot_probe(struct device *dev, struct xroot_info *info, void **root)
+{
+ struct xroot *xr = NULL;
+
+ dev_info(dev, "%s: probing...", __func__);
+
+ xr = devm_kzalloc(dev, sizeof(*xr), GFP_KERNEL);
+ if (!xr)
+ return -ENOMEM;
+
+ xr->dev = dev;
+ INIT_LIST_HEAD(&xr->groups);
+
+ xr->addr = info->addr;
+ xr->num_range = info->num_range;
+ xr->ranges = devm_kzalloc(dev, sizeof(*info->ranges) * info->num_range, GFP_KERNEL);
+ if (!xr->ranges)
+ return -ENOMEM;
+
+ memcpy(&xr->ranges, info->ranges, sizeof(*info->ranges) * info->num_range);
+ ida_init(&xr->grp_ida);
+
+ *root = xr;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xroot_probe);
+
+void xroot_remove(void *root)
+{
+ struct xroot *xr = (struct xroot *)root;
+ struct xroot_group *grp, *tmp;
+
+ xroot_info(xr, "leaving...");
+ list_for_each_entry_safe(grp, tmp, &xr->groups, node) {
+ list_del(&grp->node);
+ xroot_cleanup_group(grp);
+ kfree(grp);
+ }
+}
+EXPORT_SYMBOL_GPL(xroot_remove);
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH V1 XRT Alveo Infrastructure 8/9] fpga: xrt: xrt group device driver
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
` (6 preceding siblings ...)
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 7/9] fpga: xrt: lib-xrt xroot APIs Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 9/9] fpga: xrt: management physical function driver Lizhi Hou
8 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
group driver that manages life cycle of a bunch of leaf driver instances.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
drivers/fpga/xrt/lib/Makefile | 1 +
drivers/fpga/xrt/lib/group.c | 94 ++++++++++++++++++++++++++++++++++
drivers/fpga/xrt/lib/lib-drv.c | 18 ++++++-
drivers/fpga/xrt/lib/lib-drv.h | 13 +++++
4 files changed, 125 insertions(+), 1 deletion(-)
create mode 100644 drivers/fpga/xrt/lib/group.c
diff --git a/drivers/fpga/xrt/lib/Makefile b/drivers/fpga/xrt/lib/Makefile
index fd2af2cbd1da..abf7d5341a69 100644
--- a/drivers/fpga/xrt/lib/Makefile
+++ b/drivers/fpga/xrt/lib/Makefile
@@ -12,6 +12,7 @@ obj-$(CONFIG_FPGA_XRT_LIB) += xrt-lib.o
xrt-lib-objs := \
lib-drv.o \
xroot.o \
+ group.o \
xrt-bus.dtb.o
ccflags-y := -I$(FULL_XRT_PATH)/include
diff --git a/drivers/fpga/xrt/lib/group.c b/drivers/fpga/xrt/lib/group.c
new file mode 100644
index 000000000000..feafc45ddb52
--- /dev/null
+++ b/drivers/fpga/xrt/lib/group.c
@@ -0,0 +1,94 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo FPGA Group Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ */
+
+#include <linux/kmod.h>
+#include <linux/slab.h>
+#include <linux/of.h>
+#include <linux/xrt/xdevice.h>
+#include "lib-drv.h"
+#include "xroot.h"
+
+#define XRT_GRP "xrt_group"
+
+struct xgroup_leaf {
+ struct list_head node;
+ struct xrt_device *leaf_dev;
+};
+
+struct xgroup {
+ struct xrt_device *xdev;
+ struct list_head leaves;
+};
+
+static int xrt_grp_probe(struct xrt_device *xdev)
+{
+ struct xgroup_leaf *leaf;
+ struct xgroup *xg = NULL;
+ struct device_node *dn;
+
+ dev_info(&xdev->dev, "probing\n");
+
+ xg = devm_kzalloc(&xdev->dev, sizeof(*xg), GFP_KERNEL);
+ if (!xg)
+ return -ENOMEM;
+
+ xg->xdev = xdev;
+ INIT_LIST_HEAD(&xg->leaves);
+ xrt_set_drvdata(xdev, xg);
+
+ for_each_child_of_node(xdev->dev.of_node, dn) {
+ leaf = kmalloc(sizeof(*leaf), GFP_KERNEL);
+ if (!leaf)
+ break;
+
+ leaf->leaf_dev = xrt_device_register(&xdev->dev, dn, NULL, 0, NULL, 0);
+ if (!leaf->leaf_dev) {
+ kfree(leaf);
+ continue;
+ }
+ list_add(&leaf->node, &xg->leaves);
+ }
+
+ return 0;
+}
+
+static void xrt_grp_remove(struct xrt_device *xdev)
+{
+ struct xgroup *xg = xrt_get_drvdata(xdev);
+ struct xgroup_leaf *leaf, *tmp;
+
+ list_for_each_entry_safe(leaf, tmp, &xg->leaves, node) {
+ list_del(&leaf->node);
+ xrt_device_unregister(leaf->leaf_dev);
+ kfree(leaf);
+ }
+}
+
+static int xrt_grp_leaf_call(struct xrt_device *xdev, u32 cmd, void *arg)
+{
+ return 0;
+}
+
+static const struct of_device_id group_match[] = {
+ { .compatible = "xlnx,xrt-group" },
+ { }
+};
+
+static struct xrt_driver xrt_group_driver = {
+ .driver = {
+ .name = XRT_GRP,
+ .of_match_table = group_match,
+ },
+ .probe = xrt_grp_probe,
+ .remove = xrt_grp_remove,
+ .leaf_call = xrt_grp_leaf_call,
+};
+
+XRT_LEAF_INIT_FINI_FUNC(group);
diff --git a/drivers/fpga/xrt/lib/lib-drv.c b/drivers/fpga/xrt/lib/lib-drv.c
index 3ad02a7c2aac..c9c654f692d4 100644
--- a/drivers/fpga/xrt/lib/lib-drv.c
+++ b/drivers/fpga/xrt/lib/lib-drv.c
@@ -200,9 +200,17 @@ xrt_device_register(struct device *parent, struct device_node *dn,
return NULL;
}
+/*
+ * Leaf driver's module init/fini callbacks. This is not a open infrastructure for dynamic
+ * plugging in drivers. All drivers should be statically added.
+ */
+static void (*leaf_init_fini_cbs[])(bool) = {
+ group_leaf_init_fini,
+};
+
static __init int xrt_lib_init(void)
{
- int ret;
+ int ret, i;
ret = of_overlay_fdt_apply(__dtb_xrt_bus_begin,
__dtb_xrt_bus_end - __dtb_xrt_bus_begin,
@@ -216,11 +224,19 @@ static __init int xrt_lib_init(void)
return ret;
}
+ for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
+ leaf_init_fini_cbs[i](true);
+
return 0;
}
static __exit void xrt_lib_fini(void)
{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(leaf_init_fini_cbs); i++)
+ leaf_init_fini_cbs[i](false);
+
bus_unregister(&xrt_bus_type);
of_overlay_remove(&xrt_bus_ovcs_id);
}
diff --git a/drivers/fpga/xrt/lib/lib-drv.h b/drivers/fpga/xrt/lib/lib-drv.h
index 4bf8a32c7ec5..02d9b3731351 100644
--- a/drivers/fpga/xrt/lib/lib-drv.h
+++ b/drivers/fpga/xrt/lib/lib-drv.h
@@ -9,6 +9,19 @@
#ifndef _LIB_DRV_H_
#define _LIB_DRV_H_
+/* Module's init/fini routines for leaf driver in xrt-lib module */
+#define XRT_LEAF_INIT_FINI_FUNC(name) \
+void name##_leaf_init_fini(bool init) \
+{ \
+ if (init) \
+ xrt_register_driver(&xrt_##name##_driver); \
+ else \
+ xrt_unregister_driver(&xrt_##name##_driver); \
+}
+
+/* Module's init/fini routines for leaf driver in xrt-lib module */
+void group_leaf_init_fini(bool init);
+
extern u8 __dtb_xrt_bus_begin[];
extern u8 __dtb_xrt_bus_end[];
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH V1 XRT Alveo Infrastructure 9/9] fpga: xrt: management physical function driver
2021-11-18 21:03 [PATCH V1 XRT Alveo Infrastructure 0/9] XRT Alveo driver infrastructure overview Lizhi Hou
` (7 preceding siblings ...)
2021-11-18 21:03 ` [PATCH V1 XRT Alveo Infrastructure 8/9] fpga: xrt: xrt group device driver Lizhi Hou
@ 2021-11-18 21:03 ` Lizhi Hou
8 siblings, 0 replies; 12+ messages in thread
From: Lizhi Hou @ 2021-11-18 21:03 UTC (permalink / raw)
To: linux-kernel
Cc: Lizhi Hou, linux-fpga, maxz, sonal.santan, yliu, michal.simek,
stefanos, devicetree, trix, mdf, robh, dwmw2, Max Zhen
The PCIE device driver which attaches to management function on Alveo
devices. It instantiates one or more group drivers which, in turn,
instantiate xrt drivers. The instantiation of group and xrt drivers is
completely dtb driven. A test dtb is used for this patchset.
Signed-off-by: Sonal Santan <sonal.santan@xilinx.com>
Signed-off-by: Max Zhen <max.zhen@xilinx.com>
Signed-off-by: Lizhi Hou <lizhi.hou@xilinx.com>
---
drivers/fpga/Makefile | 1 +
drivers/fpga/xrt/Kconfig | 1 +
drivers/fpga/xrt/mgmt/Kconfig | 14 +++
drivers/fpga/xrt/mgmt/Makefile | 16 +++
drivers/fpga/xrt/mgmt/dt-test.dts | 13 +++
drivers/fpga/xrt/mgmt/dt-test.h | 15 +++
drivers/fpga/xrt/mgmt/xmgmt-drv.c | 166 ++++++++++++++++++++++++++++++
7 files changed, 226 insertions(+)
create mode 100644 drivers/fpga/xrt/mgmt/Kconfig
create mode 100644 drivers/fpga/xrt/mgmt/Makefile
create mode 100644 drivers/fpga/xrt/mgmt/dt-test.dts
create mode 100644 drivers/fpga/xrt/mgmt/dt-test.h
create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-drv.c
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 5bd41cf4c7ec..544e2144878f 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -52,3 +52,4 @@ obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
# XRT drivers for Alveo
obj-$(CONFIG_FPGA_XRT_LIB) += xrt/lib/
+obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt/mgmt/
diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig
index 04c3bb5aaf4f..50422f77c6df 100644
--- a/drivers/fpga/xrt/Kconfig
+++ b/drivers/fpga/xrt/Kconfig
@@ -4,3 +4,4 @@
#
source "drivers/fpga/xrt/lib/Kconfig"
+source "drivers/fpga/xrt/mgmt/Kconfig"
diff --git a/drivers/fpga/xrt/mgmt/Kconfig b/drivers/fpga/xrt/mgmt/Kconfig
new file mode 100644
index 000000000000..a978747482be
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/Kconfig
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Xilinx XRT FPGA device configuration
+#
+
+config FPGA_XRT_XMGMT
+ tristate "Xilinx Alveo Management Driver"
+ depends on FPGA_XRT_LIB
+ select FPGA_BRIDGE
+ select FPGA_REGION
+ help
+ Select this option to enable XRT PCIe driver for Xilinx Alveo FPGA.
+ This driver provides interfaces for userspace application to access
+ Alveo FPGA device.
diff --git a/drivers/fpga/xrt/mgmt/Makefile b/drivers/fpga/xrt/mgmt/Makefile
new file mode 100644
index 000000000000..fd2d47fed0c5
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/Makefile
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2020-2021 Xilinx, Inc. All rights reserved.
+#
+# Authors: Sonal.Santan@xilinx.com
+#
+
+FULL_XRT_PATH=$(srctree)/$(src)/..
+
+obj-$(CONFIG_FPGA_XRT_LIB) += xrt-mgmt.o
+
+xrt-mgmt-objs := \
+ xmgmt-drv.o \
+ dt-test.dtb.o
+
+ccflags-y := -I$(FULL_XRT_PATH)/include
diff --git a/drivers/fpga/xrt/mgmt/dt-test.dts b/drivers/fpga/xrt/mgmt/dt-test.dts
new file mode 100644
index 000000000000..e335fd61632c
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/dt-test.dts
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0
+/dts-v1/;
+
+/ {
+ compatible = "xlnx,xrt-group";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ ep_fpga_configuration_00@0,0,1e88000 {
+ reg = <0 0 0x1e88000 0 0x8000>;
+ compatible = "xilinx.com,reg_abs-axi_hwicap-1.0",
+ "axi_hwicap";
+ };
+};
diff --git a/drivers/fpga/xrt/mgmt/dt-test.h b/drivers/fpga/xrt/mgmt/dt-test.h
new file mode 100644
index 000000000000..25a8b86c381b
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/dt-test.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Lizhi Hou <lizhih@xilinx.com>
+ */
+
+#ifndef _DT_TEST_H_
+#define _DT_TEST_H_
+
+extern u8 __dtb_dt_test_begin[];
+extern u8 __dtb_dt_test_end[];
+
+#endif /* _DT_TEST_H_ */
diff --git a/drivers/fpga/xrt/mgmt/xmgmt-drv.c b/drivers/fpga/xrt/mgmt/xmgmt-drv.c
new file mode 100644
index 000000000000..0782543cf39b
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/xmgmt-drv.c
@@ -0,0 +1,166 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2020-2021 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <maxz@xilinx.com>
+ * Lizhi Hou <lizhih@xilinx.com>
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/aer.h>
+#include <linux/vmalloc.h>
+#include <linux/delay.h>
+#include "xroot.h"
+#include "dt-test.h"
+
+#define XMGMT_MODULE_NAME "xrt-mgmt"
+
+#define XMGMT_PDEV(xm) ((xm)->pdev)
+#define XMGMT_DEV(xm) (&(XMGMT_PDEV(xm)->dev))
+#define xmgmt_err(xm, fmt, args...) \
+ dev_err(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_warn(xm, fmt, args...) \
+ dev_warn(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_info(xm, fmt, args...) \
+ dev_info(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define xmgmt_dbg(xm, fmt, args...) \
+ dev_dbg(XMGMT_DEV(xm), "%s: " fmt, __func__, ##args)
+#define XMGMT_DEV_ID(_pcidev) \
+ ({ typeof(_pcidev) (pcidev) = (_pcidev); \
+ ((pci_domain_nr((pcidev)->bus) << 16) | \
+ PCI_DEVID((pcidev)->bus->number, (pcidev)->devfn)); })
+
+#define XRT_MAX_READRQ 512
+
+/* PCI Device IDs */
+#define PCI_DEVICE_ID_U50 0x5020
+static const struct pci_device_id xmgmt_pci_ids[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_XILINX, PCI_DEVICE_ID_U50), }, /* Alveo U50 */
+ { 0, }
+};
+
+struct xmgmt {
+ struct pci_dev *pdev;
+ void *root;
+
+ bool ready;
+};
+
+static int xmgmt_config_pci(struct xmgmt *xm)
+{
+ struct pci_dev *pdev = XMGMT_PDEV(xm);
+ int rc;
+
+ rc = pcim_enable_device(pdev);
+ if (rc < 0) {
+ xmgmt_err(xm, "failed to enable device: %d", rc);
+ return rc;
+ }
+
+ rc = pci_enable_pcie_error_reporting(pdev);
+ if (rc)
+ xmgmt_warn(xm, "failed to enable AER: %d", rc);
+
+ pci_set_master(pdev);
+
+ rc = pcie_get_readrq(pdev);
+ if (rc > XRT_MAX_READRQ)
+ pcie_set_readrq(pdev, XRT_MAX_READRQ);
+ return 0;
+}
+
+static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ struct xroot_range ranges[PCI_NUM_RESOURCES];
+ struct xroot_info xr_info = { 0 };
+ struct device *dev = &pdev->dev;
+ int ret, i, idx = 0;
+ struct xmgmt *xm;
+ __be64 tmp;
+
+ xm = devm_kzalloc(dev, sizeof(*xm), GFP_KERNEL);
+ if (!xm)
+ return -ENOMEM;
+ xm->pdev = pdev;
+ pci_set_drvdata(pdev, xm);
+
+ ret = xmgmt_config_pci(xm);
+ if (ret)
+ goto failed;
+
+ for (i = 0; i < PCI_NUM_RESOURCES; i++) {
+ if (pci_resource_len(pdev, i) > 0) {
+ ranges[idx].child_addr[0] = cpu_to_be32(i);
+ tmp = cpu_to_be64(pci_resource_start(pdev, i));
+ memcpy(ranges[idx].parent_addr, &tmp, sizeof(tmp));
+ tmp = cpu_to_be64(pci_resource_len(pdev, i));
+ memcpy(ranges[idx].child_size, &tmp, sizeof(tmp));
+ idx++;
+ }
+ }
+ xr_info.num_range = idx;
+ xr_info.ranges = ranges;
+ xr_info.addr = XMGMT_DEV_ID(pdev);
+ ret = xroot_probe(&pdev->dev, &xr_info, &xm->root);
+ if (ret)
+ goto failed;
+
+ ret = xroot_create_group(xm->root, __dtb_dt_test_begin);
+ if (ret) {
+ xmgmt_err(xm, "failed to create root group: %d", ret);
+ goto failed;
+ }
+
+ xmgmt_info(xm, "%s started successfully", XMGMT_MODULE_NAME);
+ return 0;
+
+failed:
+ if (xm->root)
+ xroot_remove(xm->root);
+ pci_set_drvdata(pdev, NULL);
+ return ret;
+}
+
+static void xmgmt_remove(struct pci_dev *pdev)
+{
+ struct xmgmt *xm = pci_get_drvdata(pdev);
+
+ xroot_remove(xm->root);
+ pci_disable_pcie_error_reporting(xm->pdev);
+ xmgmt_info(xm, "%s cleaned up successfully", XMGMT_MODULE_NAME);
+}
+
+static struct pci_driver xmgmt_driver = {
+ .name = XMGMT_MODULE_NAME,
+ .id_table = xmgmt_pci_ids,
+ .probe = xmgmt_probe,
+ .remove = xmgmt_remove,
+};
+
+static int __init xmgmt_init(void)
+{
+ int res = 0;
+
+ res = pci_register_driver(&xmgmt_driver);
+ if (res)
+ return res;
+
+ return 0;
+}
+
+static __exit void xmgmt_exit(void)
+{
+ pci_unregister_driver(&xmgmt_driver);
+}
+
+module_init(xmgmt_init);
+module_exit(xmgmt_exit);
+
+MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
+MODULE_AUTHOR("XRT Team <runtime@xilinx.com>");
+MODULE_DESCRIPTION("Xilinx Alveo management function driver");
+MODULE_LICENSE("GPL v2");
--
2.27.0
^ permalink raw reply related [flat|nested] 12+ messages in thread