From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3114C432C0 for ; Fri, 22 Nov 2019 16:16:13 +0000 (UTC) Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7F78F2071C for ; Fri, 22 Nov 2019 16:16:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="r6E4DMv1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7F78F2071C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 3DA0988D81; Fri, 22 Nov 2019 16:16:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bzVDFDD6bh+f; Fri, 22 Nov 2019 16:16:12 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by hemlock.osuosl.org (Postfix) with ESMTP id 7E1DE88D64; Fri, 22 Nov 2019 16:16:12 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 589B8C1D74; Fri, 22 Nov 2019 16:16:12 +0000 (UTC) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id CF31BC18DA for ; Fri, 22 Nov 2019 16:16:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id CB14D87469 for ; Fri, 22 Nov 2019 16:16:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9cxoSGdbyOV8 for ; Fri, 22 Nov 2019 16:16:10 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 0269787448 for ; Fri, 22 Nov 2019 16:16:10 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id h27so3561986pgn.0 for ; Fri, 22 Nov 2019 08:16:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+pVRv0zmwM6r6qh3u7jefx+roN9IpRfJikQs29Fs7Mc=; b=r6E4DMv1Ew1POCiZziHs0YwoBfyHUTq3KYUrRgtfb4vl0UK6dNOFfiH6gUaxO4D+Si nAKdspmLW+9QlY4Dohyh1Y82LJgbTL0/J+mKWoqN6tCCVfh66MKivVt6hmQuVvIqEEjr ycvmCeYBRQQ4BnbJ9FFQzrsZ9+nU3Hfhf57i6taLJTOwMXrUiGLUZgn9OLRcdhzXVGhV kOMfXA52pUudqEQZhIJNepSok+DmAQGPD9MLwvVpKOf2srwXLYtceFOIueJOR75AMNjj 0C/TK8wEk3A4oDUYhjl8zavOjysg3uzc3fAFPCObboqKbmjcJFey4/qAn6W8vVKxstfY XZ7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+pVRv0zmwM6r6qh3u7jefx+roN9IpRfJikQs29Fs7Mc=; b=qctVDP82HnPDnkQ6IM2nPN/xX5gU9Tpt+cdv8ntBBBVlWIi4K/Ljfr6d6utrqHGJzs fq5TJNg+fsVTzs5wpLnZFWqgaBHTVow5Wvt3t7ql1m8lbUJ/hk66KDNS17hAXhO13vfu 02V79r6vBBHKxDzt9Gx/54K1LwZHt3ItvnyCl1ltIKhEqDl7sIjBMqmSeB/GMf/QNEqJ LCLTDVBLjOyiyoF9NVkvCPrmv8/Wn7Zs5a9Ke+Zst/XZhZhk1BNkWn98uoTB6Z3DBl26 trF9gg5gWHDc16MVvuYigQNOE7UQnDi+rWtJ+WidfK1Ya4ymCwqYWf+t5LUkYnuRfWSB 3wVQ== X-Gm-Message-State: APjAAAX+SqUbkQwdlELWoPq7PYO1ibdW3cMxhvvcmWEdv+NCIRrAxaeV KYfz7XPNprC6HckgGZ5AZRAazmY+SRAAWw== X-Google-Smtp-Source: APXvYqwe1jPYdLfnbYJnZ7Jm1S+hvm/1eSFcyEKVujxjm7ccMO9vt6z4GOxkwjZGiWr/k4y4nLji/w== X-Received: by 2002:a63:c18:: with SMTP id b24mr11367250pgl.291.1574435333618; Fri, 22 Nov 2019 07:08:53 -0800 (PST) Received: from localhost.localdomain ([240e:362:496:8600:f5af:2744:25c3:d01a]) by smtp.gmail.com with ESMTPSA id a19sm8066021pfn.144.2019.11.22.07.08.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 22 Nov 2019 07:08:52 -0800 (PST) From: Zhangfei Gao To: Greg Kroah-Hartman , Arnd Bergmann , Herbert Xu , jonathan.cameron@huawei.com, grant.likely@arm.com, jean-philippe , Jerome Glisse , ilias.apalodimas@linaro.org, francois.ozog@linaro.org, kenneth-lee-2012@foxmail.com, Wangzhou , "haojian . zhuang" , guodong.xu@linaro.org Subject: [PATCH v9 1/4] uacce: Add documents for uacce Date: Fri, 22 Nov 2019 23:07:38 +0800 Message-Id: <1574435261-6031-2-git-send-email-zhangfei.gao@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1574435261-6031-1-git-send-email-zhangfei.gao@linaro.org> References: <1574435261-6031-1-git-send-email-zhangfei.gao@linaro.org> Cc: linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-crypto@vger.kernel.org, Zhangfei Gao , Kenneth Lee , linux-accelerators@lists.ozlabs.org X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" From: Kenneth Lee Uacce (Unified/User-space-access-intended Accelerator Framework) is a kernel module targets to provide Shared Virtual Addressing (SVA) between the accelerator and process. This patch add document to explain how it works. Signed-off-by: Kenneth Lee Signed-off-by: Zaibo Xu Signed-off-by: Zhou Wang Signed-off-by: Zhangfei Gao --- Documentation/misc-devices/uacce.rst | 176 +++++++++++++++++++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 Documentation/misc-devices/uacce.rst diff --git a/Documentation/misc-devices/uacce.rst b/Documentation/misc-devices/uacce.rst new file mode 100644 index 0000000..1db412e --- /dev/null +++ b/Documentation/misc-devices/uacce.rst @@ -0,0 +1,176 @@ +.. SPDX-License-Identifier: GPL-2.0 + +Introduction of Uacce +--------------------- + +Uacce (Unified/User-space-access-intended Accelerator Framework) targets to +provide Shared Virtual Addressing (SVA) between accelerators and processes. +So accelerator can access any data structure of the main cpu. +This differs from the data sharing between cpu and io device, which share +only data content rather than address. +Because of the unified address, hardware and user space of process can +share the same virtual address in the communication. +Uacce takes the hardware accelerator as a heterogeneous processor, while +IOMMU share the same CPU page tables and as a result the same translation +from va to pa. + +:: + + __________________________ __________________________ + | | | | + | User application (CPU) | | Hardware Accelerator | + |__________________________| |__________________________| + + | | + | va | va + V V + __________ __________ + | | | | + | MMU | | IOMMU | + |__________| |__________| + | | + | | + V pa V pa + _______________________________________ + | | + | Memory | + |_______________________________________| + + + +Architecture +------------ + +Uacce is the kernel module, taking charge of iommu and address sharing. +The user drivers and libraries are called WarpDrive. + +The uacce device, built around the IOMMU SVA API, can access multiple +address spaces, including the one without PASID. + +A virtual concept, queue, is used for the communication. It provides a +FIFO-like interface. And it maintains a unified address space between the +application and all involved hardware. + +:: + + ___________________ ________________ + | | user API | | + | WarpDrive library | ------------> | user driver | + |___________________| |________________| + | | + | | + | queue fd | + | | + | | + v | + ___________________ _________ | + | | | | | mmap memory + | Other framework | | uacce | | r/w interface + | crypto/nic/others | |_________| | + |___________________| | + | | | + | register | register | + | | | + | | | + | _________________ __________ | + | | | | | | + ------------- | Device Driver | | IOMMU | | + |_________________| |__________| | + | | + | V + | ___________________ + | | | + -------------------------- | Device(Hardware) | + |___________________| + + +How does it work +---------------- + +Uacce uses mmap and IOMMU to play the trick. + +Uacce creates a chrdev for every device registered to it. New queue is +created when user application open the chrdev. The file descriptor is used +as the user handle of the queue. +The accelerator device present itself as an Uacce object, which exports as +a chrdev to the user space. The user application communicates with the +hardware by ioctl (as control path) or share memory (as data path). + +The control path to the hardware is via file operation, while data path is +via mmap space of the queue fd. + +The queue file address space: + +:: + + /** + * enum uacce_qfrt: qfrt type + * @UACCE_QFRT_MMIO: device mmio region + * @UACCE_QFRT_DUS: device user share region + */ + enum uacce_qfrt { + UACCE_QFRT_MMIO = 0, + UACCE_QFRT_DUS = 1, + }; + +All regions are optional and differ from device type to type. +Each region can be mmapped only once, otherwise -EEXIST returns. + +The device mmio region is mapped to the hardware mmio space. It is generally +used for doorbell or other notification to the hardware. It is not fast enough +as data channel. + +The device user share region is used for share data buffer between user process +and device. + + +The Uacce register API +---------------------- + +The register API is defined in uacce.h. + +:: + + struct uacce_interface { + char name[UACCE_MAX_NAME_SIZE]; + unsigned int flags; + const struct uacce_ops *ops; + }; + +According to the IOMMU capability, uacce_interface flags can be: + +:: + + /** + * UACCE Device flags: + * UACCE_DEV_SVA: Shared Virtual Addresses + * Support PASID + * Support device page faults (PCI PRI or SMMU Stall) + */ + #define UACCE_DEV_SVA BIT(0) + + struct uacce_device *uacce_alloc(struct device *parent, + struct uacce_interface *interface); + int uacce_register(struct uacce_device *uacce); + void uacce_remove(struct uacce_device *uacce); + +uacce_register results can be: + +a. If uacce module is not compiled, ERR_PTR(-ENODEV) + +b. Succeed with the desired flags + +c. Succeed with the negotiated flags, for example + + uacce_interface.flags = UACCE_DEV_SVA but uacce->flags = ~UACCE_DEV_SVA + + So user driver need check return value as well as the negotiated uacce->flags. + + +The user driver +--------------- + +The queue file mmap space will need a user driver to wrap the communication +protocol. Uacce provides some attributes in sysfs for the user driver to +match the right accelerator accordingly. +More details in Documentation/ABI/testing/sysfs-driver-uacce. -- 2.7.4 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu