Linux-HyperV Archive on lore.kernel.org
 help / color / Atom feed
From: Thomas Zimmermann <tzimmermann@suse.de>
To: Sasha Levin <sashal@kernel.org>,
	alexander.deucher@amd.com, chris@chris-wilson.co.uk,
	ville.syrjala@linux.intel.com, Hawking.Zhang@amd.com,
	tvrtko.ursulin@intel.com
Cc: linux-kernel@vger.kernel.org, linux-hyperv@vger.kernel.org,
	kys@microsoft.com, haiyangz@microsoft.com,
	sthemmin@microsoft.com, wei.liu@kernel.org,
	spronovo@microsoft.com, iourit@microsoft.com,
	dri-devel@lists.freedesktop.org, linux-fbdev@vger.kernel.org,
	gregkh@linuxfoundation.org
Subject: Re: [RFC PATCH 0/4] DirectX on Linux
Date: Wed, 20 May 2020 09:10:46 +0200
Message-ID: <55c57049-1869-7421-aa0f-3ce0b6a133cf@suse.de> (raw)
In-Reply-To: <20200519163234.226513-1-sashal@kernel.org>

[-- Attachment #1.1: Type: text/plain, Size: 9368 bytes --]

Hi

Am 19.05.20 um 18:32 schrieb Sasha Levin:
> There is a blog post that goes into more detail about the bigger
> picture, and walks through all the required pieces to make this work. It
> is available here:
> https://devblogs.microsoft.com/directx/directx-heart-linux . The rest of
> this cover letter will focus on the Linux Kernel bits.

That's quite a surprise. Thanks for your efforts to contribute.

> 
> Overview
> ========
> 
> This is the first draft of the Microsoft Virtual GPU (vGPU) driver. The
> driver exposes a paravirtualized GPU to user mode applications running
> in a virtual machine on a Windows host. This enables hardware
> acceleration in environment such as WSL (Windows Subsystem for Linux)
> where the Linux virtual machine is able to share the GPU with the
> Windows host.
> 
> The projection is accomplished by exposing the WDDM (Windows Display
> Driver Model) interface as a set of IOCTL. This allows APIs and user
> mode driver written against the WDDM GPU abstraction on Windows to be
> ported to run within a Linux environment. This enables the port of the
> D3D12 and DirectML APIs as well as their associated user mode driver to
> Linux. This also enables third party APIs, such as the popular NVIDIA
> Cuda compute API, to be hardware accelerated within a WSL environment.
> 
> Only the rendering/compute aspect of the GPU are projected to the
> virtual machine, no display functionality is exposed. Further, at this
> time there are no presentation integration. So although the D3D12 API
> can be use to render graphics offscreen, there is no path (yet) for
> pixel to flow from the Linux environment back onto the Windows host
> desktop. This GPU stack is effectively side-by-side with the native
> Linux graphics stack.
> 
> The driver creates the /dev/dxg device, which can be opened by user mode
> application and handles their ioctls. The IOCTL interface to the driver
> is defined in dxgkmthk.h (Dxgkrnl Graphics Port Driver ioctl
> definitions). The interface matches the D3DKMT interface on Windows.
> Ioctls are implemented in ioctl.c.

Echoing what others said, you're not making a DRM driver. The driver
should live outside of the DRM code.

I have one question about the driver API: on Windows, DirectX versions
are loosly tied to Windows releases. So I guess you can change the
kernel interface among DirectX versions?

If so, how would this work on Linux in the long term? If there ever is a
DirectX 13 or 14 with incompatible kernel interfaces, how would you plan
to update the Linux driver?

Best regards
Thomas

> 
> When a VM starts, hyper-v on the host adds virtual GPU devices to the VM
> via the hyper-v driver. The host offers several VM bus channels to the
> VM: the global channel and one channel per virtual GPU, assigned to the
> VM.
> 
> The driver registers with the hyper-v driver (hv_driver) for the arrival
> of VM bus channels. dxg_probe_device recognizes the vGPU channels and
> creates the corresponding objects (dxgadapter for vGPUs and dxgglobal
> for the global channel).
> 
> The driver uses the hyper-V VM bus interface to communicate with the
> host. dxgvmbus.c implements the communication interface.
> 
> The global channel has 8GB of IO space assigned by the host. This space
> is managed by the host and used to give the guest direct CPU access to
> some allocations. Video memory is allocated on the host except in the
> case of existing_sysmem allocations. The Windows host allocates memory
> for the GPU on behalf of the guest. The Linux guest can access that
> memory by mapping GPU virtual address to allocations and then
> referencing those GPU virtual address from within GPU command buffers
> submitted to the GPU. For allocations which require CPU access, the
> allocation is mapped by the host into a location in the 8GB of IO space
> reserved in the guest for that purpose. The Windows host uses the nested
> CPU page table to ensure that this guest IO space always map to the
> correct location for the allocation as it may migrate between dedicated
> GPU memory (e.g. VRAM, firmware reserved DDR) and shared system memory
> (regular DDR) over its lifetime. The Linux guest maps a user mode CPU
> virtual address to an allocation IO space range for direct access by
> user mode APIs and drivers.
> 
>  
> 
> Implementation of LX_DXLOCK2 ioctl
> ==================================
> 
> We would appreciate your feedback on the implementation of the
> LX_DXLOCK2 ioctl.
> 
> This ioctl is used to get a CPU address to an allocation, which is
> resident in video/system memory on the host. The way it works:
> 
> 1. The driver sends the Lock message to the host
> 
> 2. The host allocates space in the VM IO space and maps it to the
> allocation memory
> 
> 3. The host returns the address in IO space for the mapped allocation
> 
> 4. The driver (in dxg_map_iospace) allocates a user mode virtual address
> range using vm_mmap and maps it to the IO space using
> io_remap_ofn_range)
> 
> 5. The VA is returned to the application
> 
>  
> 
> Internal objects
> ================
> 
> The following objects are created by the driver (defined in dxgkrnl.h):
> 
> - dxgadapter - represents a virtual GPU
> 
> - dxgprocess - tracks per process state (handle table of created
>   objects, list of objects, etc.)
> 
> - dxgdevice - a container for other objects (contexts, paging queues,
>   allocations, GPU synchronization objects)
> 
> - dxgcontext - represents thread of GPU execution for packet
>   scheduling.
> 
> - dxghwqueue - represents thread of GPU execution of hardware scheduling
> 
> - dxgallocation - represents a GPU accessible allocation
> 
> - dxgsyncobject - represents a GPU synchronization object
> 
> - dxgresource - collection of dxgalloction objects
> 
> - dxgsharedresource, dxgsharedsyncobj - helper objects to share objects
>   between different dxgdevice objects, which can belong to different
> processes
> 
> 
>  
> Object handles
> ==============
> 
> All GPU objects, created by the driver, are accessible by a handle
> (d3dkmt_handle). Each process has its own handle table, which is
> implemented in hmgr.c. For each API visible object, created by the
> driver, there is an object, created on the host. For example, the is a
> dxgprocess object on the host for each dxgprocess object in the VM, etc.
> The object handles have the same value in the host and the VM, which is
> done to avoid translation from the guest handles to the host handles.
>  
> 
> 
> Signaling CPU events by the host
> ================================
> 
> The WDDM interface provides a way to signal CPU event objects when
> execution of a context reached certain point. The way it is implemented:
> 
> - application sends an event_fd via ioctl to the driver
> 
> - eventfd_ctx_get is used to get a pointer to the file object
>   (eventfd_ctx)
> 
> - the pointer to sent the host via a VM bus message
> 
> - when GPU execution reaches a certain point, the host sends a message
>   to the VM with the event pointer
> 
> - signal_guest_event() handles the messages and eventually
>   eventfd_signal() is called.
> 
> 
> Sasha Levin (4):
>   gpu: dxgkrnl: core code
>   gpu: dxgkrnl: hook up dxgkrnl
>   Drivers: hv: vmbus: hook up dxgkrnl
>   gpu: dxgkrnl: create a MAINTAINERS entry
> 
>  MAINTAINERS                      |    7 +
>  drivers/gpu/Makefile             |    2 +-
>  drivers/gpu/dxgkrnl/Kconfig      |   10 +
>  drivers/gpu/dxgkrnl/Makefile     |   12 +
>  drivers/gpu/dxgkrnl/d3dkmthk.h   | 1635 +++++++++
>  drivers/gpu/dxgkrnl/dxgadapter.c | 1399 ++++++++
>  drivers/gpu/dxgkrnl/dxgkrnl.h    |  913 ++++++
>  drivers/gpu/dxgkrnl/dxgmodule.c  |  692 ++++
>  drivers/gpu/dxgkrnl/dxgprocess.c |  355 ++
>  drivers/gpu/dxgkrnl/dxgvmbus.c   | 2955 +++++++++++++++++
>  drivers/gpu/dxgkrnl/dxgvmbus.h   |  859 +++++
>  drivers/gpu/dxgkrnl/hmgr.c       |  593 ++++
>  drivers/gpu/dxgkrnl/hmgr.h       |  107 +
>  drivers/gpu/dxgkrnl/ioctl.c      | 5269 ++++++++++++++++++++++++++++++
>  drivers/gpu/dxgkrnl/misc.c       |  280 ++
>  drivers/gpu/dxgkrnl/misc.h       |  288 ++
>  drivers/video/Kconfig            |    2 +
>  include/linux/hyperv.h           |   16 +
>  18 files changed, 15393 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/gpu/dxgkrnl/Kconfig
>  create mode 100644 drivers/gpu/dxgkrnl/Makefile
>  create mode 100644 drivers/gpu/dxgkrnl/d3dkmthk.h
>  create mode 100644 drivers/gpu/dxgkrnl/dxgadapter.c
>  create mode 100644 drivers/gpu/dxgkrnl/dxgkrnl.h
>  create mode 100644 drivers/gpu/dxgkrnl/dxgmodule.c
>  create mode 100644 drivers/gpu/dxgkrnl/dxgprocess.c
>  create mode 100644 drivers/gpu/dxgkrnl/dxgvmbus.c
>  create mode 100644 drivers/gpu/dxgkrnl/dxgvmbus.h
>  create mode 100644 drivers/gpu/dxgkrnl/hmgr.c
>  create mode 100644 drivers/gpu/dxgkrnl/hmgr.h
>  create mode 100644 drivers/gpu/dxgkrnl/ioctl.c
>  create mode 100644 drivers/gpu/dxgkrnl/misc.c
>  create mode 100644 drivers/gpu/dxgkrnl/misc.h
> 

-- 
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Felix Imendörffer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  parent reply index

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-19 16:32 Sasha Levin
2020-05-19 16:32 ` [RFC PATCH 2/4] gpu: dxgkrnl: hook up dxgkrnl Sasha Levin
2020-05-19 16:32 ` [RFC PATCH 3/4] Drivers: hv: vmbus: " Sasha Levin
2020-05-19 16:32 ` [RFC PATCH 4/4] gpu: dxgkrnl: create a MAINTAINERS entry Sasha Levin
     [not found] ` <20200519163234.226513-2-sashal@kernel.org>
2020-05-19 17:19   ` [RFC PATCH 1/4] gpu: dxgkrnl: core code Greg KH
2020-05-19 17:21   ` Greg KH
2020-05-19 17:45     ` Sasha Levin
2020-05-20  6:13       ` Greg KH
2020-05-19 17:27   ` Greg KH
2020-05-19 19:21 ` [RFC PATCH 0/4] DirectX on Linux Daniel Vetter
2020-05-19 20:36   ` Sasha Levin
2020-05-20 10:37     ` Jan Engelhardt
2020-05-19 22:42 ` Dave Airlie
2020-05-19 23:01   ` Daniel Vetter
2020-05-20  3:47     ` [EXTERNAL] " Steve Pronovost
     [not found]       ` <CAKMK7uFubAxtMEeCOYtvgjGYtmDVJeXcPFzmRD7t5BUm_GPP0w@mail.gmail.com>
     [not found]         ` <MWHPR21MB02870909F08EBA08EB903635C7B60@MWHPR21MB0287.namprd21.prod.outlook.com>
2020-05-20 15:34           ` Steve Pronovost
2020-05-19 23:12   ` Dave Airlie
2020-05-20  7:10 ` Thomas Zimmermann [this message]
2020-05-20  7:42   ` [EXTERNAL] " Steve Pronovost
2020-05-20 11:06     ` Thomas Zimmermann

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=55c57049-1869-7421-aa0f-3ce0b6a133cf@suse.de \
    --to=tzimmermann@suse.de \
    --cc=Hawking.Zhang@amd.com \
    --cc=alexander.deucher@amd.com \
    --cc=chris@chris-wilson.co.uk \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=haiyangz@microsoft.com \
    --cc=iourit@microsoft.com \
    --cc=kys@microsoft.com \
    --cc=linux-fbdev@vger.kernel.org \
    --cc=linux-hyperv@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=sashal@kernel.org \
    --cc=spronovo@microsoft.com \
    --cc=sthemmin@microsoft.com \
    --cc=tvrtko.ursulin@intel.com \
    --cc=ville.syrjala@linux.intel.com \
    --cc=wei.liu@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-HyperV Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-hyperv/0 linux-hyperv/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-hyperv linux-hyperv/ https://lore.kernel.org/linux-hyperv \
		linux-hyperv@vger.kernel.org
	public-inbox-index linux-hyperv

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-hyperv


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git