From: "Christian König" <christian.koenig@amd.com>
To: Daniel Vetter <daniel.vetter@ffwll.ch>,
DRI Development <dri-devel@lists.freedesktop.org>
Cc: linux-rdma@vger.kernel.org,
Intel Graphics Development <intel-gfx@lists.freedesktop.org>,
amd-gfx@lists.freedesktop.org,
Chris Wilson <chris@chris-wilson.co.uk>,
linaro-mm-sig@lists.linaro.org,
Steve Pronovost <spronovo@microsoft.com>,
Jason Ekstrand <jason@jlekstrand.net>,
Jesse Natalie <jenatali@microsoft.com>,
Daniel Vetter <daniel.vetter@intel.com>,
Thomas Hellstrom <thomas.hellstrom@intel.com>,
linux-media@vger.kernel.org,
Felix Kuehling <Felix.Kuehling@amd.com>,
Mika Kuoppala <mika.kuoppala@intel.com>
Subject: Re: [PATCH 03/25] dma-buf.rst: Document why idenfinite fences are a bad idea
Date: Thu, 9 Jul 2020 13:53:14 +0200 [thread overview]
Message-ID: <aede585d-6c3f-4559-c612-e8cb8a9e1f92@amd.com> (raw)
In-Reply-To: <20200707201229.472834-4-daniel.vetter@ffwll.ch>
Am 07.07.20 um 22:12 schrieb Daniel Vetter:
> Comes up every few years, gets somewhat tedious to discuss, let's
> write this down once and for all.
>
> What I'm not sure about is whether the text should be more explicit in
> flat out mandating the amdkfd eviction fences for long running compute
> workloads or workloads where userspace fencing is allowed.
>
> v2: Now with dot graph!
>
> Cc: Jesse Natalie <jenatali@microsoft.com>
> Cc: Steve Pronovost <spronovo@microsoft.com>
> Cc: Jason Ekstrand <jason@jlekstrand.net>
> Cc: Felix Kuehling <Felix.Kuehling@amd.com>
> Cc: Mika Kuoppala <mika.kuoppala@intel.com>
> Cc: Thomas Hellstrom <thomas.hellstrom@intel.com>
> Cc: linux-media@vger.kernel.org
> Cc: linaro-mm-sig@lists.linaro.org
> Cc: linux-rdma@vger.kernel.org
> Cc: amd-gfx@lists.freedesktop.org
> Cc: intel-gfx@lists.freedesktop.org
> Cc: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Acked-by: Christian König <christian.koenig@amd.com>
> ---
> Documentation/driver-api/dma-buf.rst | 70 ++++++++++++++++++++++++
> drivers/gpu/drm/virtio/virtgpu_display.c | 20 -------
> 2 files changed, 70 insertions(+), 20 deletions(-)
>
> diff --git a/Documentation/driver-api/dma-buf.rst b/Documentation/driver-api/dma-buf.rst
> index f8f6decde359..037ba0078bb4 100644
> --- a/Documentation/driver-api/dma-buf.rst
> +++ b/Documentation/driver-api/dma-buf.rst
> @@ -178,3 +178,73 @@ DMA Fence uABI/Sync File
> .. kernel-doc:: include/linux/sync_file.h
> :internal:
>
> +Idefinite DMA Fences
> +~~~~~~~~~~~~~~~~~~~~
> +
> +At various times &dma_fence with an indefinite time until dma_fence_wait()
> +finishes have been proposed. Examples include:
> +
> +* Future fences, used in HWC1 to signal when a buffer isn't used by the display
> + any longer, and created with the screen update that makes the buffer visible.
> + The time this fence completes is entirely under userspace's control.
> +
> +* Proxy fences, proposed to handle &drm_syncobj for which the fence has not yet
> + been set. Used to asynchronously delay command submission.
> +
> +* Userspace fences or gpu futexes, fine-grained locking within a command buffer
> + that userspace uses for synchronization across engines or with the CPU, which
> + are then imported as a DMA fence for integration into existing winsys
> + protocols.
> +
> +* Long-running compute command buffers, while still using traditional end of
> + batch DMA fences for memory management instead of context preemption DMA
> + fences which get reattached when the compute job is rescheduled.
> +
> +Common to all these schemes is that userspace controls the dependencies of these
> +fences and controls when they fire. Mixing indefinite fences with normal
> +in-kernel DMA fences does not work, even when a fallback timeout is included to
> +protect against malicious userspace:
> +
> +* Only the kernel knows about all DMA fence dependencies, userspace is not aware
> + of dependencies injected due to memory management or scheduler decisions.
> +
> +* Only userspace knows about all dependencies in indefinite fences and when
> + exactly they will complete, the kernel has no visibility.
> +
> +Furthermore the kernel has to be able to hold up userspace command submission
> +for memory management needs, which means we must support indefinite fences being
> +dependent upon DMA fences. If the kernel also support indefinite fences in the
> +kernel like a DMA fence, like any of the above proposal would, there is the
> +potential for deadlocks.
> +
> +.. kernel-render:: DOT
> + :alt: Indefinite Fencing Dependency Cycle
> + :caption: Indefinite Fencing Dependency Cycle
> +
> + digraph "Fencing Cycle" {
> + node [shape=box bgcolor=grey style=filled]
> + kernel [label="Kernel DMA Fences"]
> + userspace [label="userspace controlled fences"]
> + kernel -> userspace [label="memory management"]
> + userspace -> kernel [label="Future fence, fence proxy, ..."]
> +
> + { rank=same; kernel userspace }
> + }
> +
> +This means that the kernel might accidentally create deadlocks
> +through memory management dependencies which userspace is unaware of, which
> +randomly hangs workloads until the timeout kicks in. Workloads, which from
> +userspace's perspective, do not contain a deadlock. In such a mixed fencing
> +architecture there is no single entity with knowledge of all dependencies.
> +Thefore preventing such deadlocks from within the kernel is not possible.
> +
> +The only solution to avoid dependencies loops is by not allowing indefinite
> +fences in the kernel. This means:
> +
> +* No future fences, proxy fences or userspace fences imported as DMA fences,
> + with or without a timeout.
> +
> +* No DMA fences that signal end of batchbuffer for command submission where
> + userspace is allowed to use userspace fencing or long running compute
> + workloads. This also means no implicit fencing for shared buffers in these
> + cases.
> diff --git a/drivers/gpu/drm/virtio/virtgpu_display.c b/drivers/gpu/drm/virtio/virtgpu_display.c
> index f3ce49c5a34c..af55b334be2f 100644
> --- a/drivers/gpu/drm/virtio/virtgpu_display.c
> +++ b/drivers/gpu/drm/virtio/virtgpu_display.c
> @@ -314,25 +314,6 @@ virtio_gpu_user_framebuffer_create(struct drm_device *dev,
> return &virtio_gpu_fb->base;
> }
>
> -static void vgdev_atomic_commit_tail(struct drm_atomic_state *state)
> -{
> - struct drm_device *dev = state->dev;
> -
> - drm_atomic_helper_commit_modeset_disables(dev, state);
> - drm_atomic_helper_commit_modeset_enables(dev, state);
> - drm_atomic_helper_commit_planes(dev, state, 0);
> -
> - drm_atomic_helper_fake_vblank(state);
> - drm_atomic_helper_commit_hw_done(state);
> -
> - drm_atomic_helper_wait_for_vblanks(dev, state);
> - drm_atomic_helper_cleanup_planes(dev, state);
> -}
> -
> -static const struct drm_mode_config_helper_funcs virtio_mode_config_helpers = {
> - .atomic_commit_tail = vgdev_atomic_commit_tail,
> -};
> -
> static const struct drm_mode_config_funcs virtio_gpu_mode_funcs = {
> .fb_create = virtio_gpu_user_framebuffer_create,
> .atomic_check = drm_atomic_helper_check,
> @@ -346,7 +327,6 @@ void virtio_gpu_modeset_init(struct virtio_gpu_device *vgdev)
> drm_mode_config_init(vgdev->ddev);
> vgdev->ddev->mode_config.quirk_addfb_prefer_host_byte_order = true;
> vgdev->ddev->mode_config.funcs = &virtio_gpu_mode_funcs;
> - vgdev->ddev->mode_config.helper_private = &virtio_mode_config_helpers;
>
> /* modes will be validated against the framebuffer size */
> vgdev->ddev->mode_config.min_width = XRES_MIN;
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
next prev parent reply other threads:[~2020-07-09 11:53 UTC|newest]
Thread overview: 119+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-07 20:12 [PATCH 00/25] dma-fence annotations, round 3 Daniel Vetter
2020-07-07 20:12 ` [PATCH 01/25] dma-fence: basic lockdep annotations Daniel Vetter
2020-07-08 14:57 ` Christian König
2020-07-08 15:12 ` Daniel Vetter
2020-07-08 15:19 ` Alex Deucher
2020-07-08 15:37 ` Daniel Vetter
2020-07-14 11:09 ` Daniel Vetter
2020-07-09 7:32 ` [Intel-gfx] " Daniel Stone
2020-07-09 7:52 ` Daniel Vetter
2020-07-13 16:26 ` Daniel Vetter
2020-07-13 16:39 ` Christian König
2020-07-13 20:31 ` Dave Airlie
2020-07-07 20:12 ` [PATCH 02/25] dma-fence: prime " Daniel Vetter
2020-07-09 8:09 ` Daniel Vetter
2020-07-10 12:43 ` Jason Gunthorpe
2020-07-10 12:48 ` Christian König
2020-07-10 12:54 ` Jason Gunthorpe
2020-07-10 13:01 ` Christian König
2020-07-10 13:48 ` Jason Gunthorpe
2020-07-10 14:02 ` Daniel Vetter
2020-07-10 14:23 ` Jason Gunthorpe
2020-07-10 20:02 ` Daniel Vetter
2020-07-07 20:12 ` [PATCH 03/25] dma-buf.rst: Document why idenfinite fences are a bad idea Daniel Vetter
2020-07-09 7:36 ` [Intel-gfx] " Daniel Stone
2020-07-09 8:04 ` Daniel Vetter
2020-07-09 12:11 ` Daniel Stone
2020-07-09 12:31 ` Daniel Vetter
2020-07-09 14:28 ` Christian König
2020-07-09 11:53 ` Christian König [this message]
2020-07-09 12:33 ` [PATCH 1/2] dma-buf.rst: Document why indefinite " Daniel Vetter
2020-07-09 12:33 ` [PATCH 2/2] drm/virtio: Remove open-coded commit-tail function Daniel Vetter
2020-07-09 12:48 ` Gerd Hoffmann
2020-07-09 14:05 ` Sam Ravnborg
2020-07-14 9:13 ` Daniel Vetter
2020-08-19 12:43 ` Jiri Slaby
2020-08-19 12:47 ` Jiri Slaby
2020-08-19 13:24 ` Gerd Hoffmann
2020-08-20 6:32 ` Jiri Slaby
2020-08-21 7:01 ` Gerd Hoffmann
2020-07-10 12:30 ` [PATCH 1/2] dma-buf.rst: Document why indefinite fences are a bad idea Maarten Lankhorst
2020-07-14 17:46 ` Jason Ekstrand
2020-07-20 11:15 ` [Linaro-mm-sig] " Thomas Hellström (Intel)
2020-07-21 7:41 ` Daniel Vetter
2020-07-21 7:45 ` Christian König
2020-07-21 8:47 ` Thomas Hellström (Intel)
2020-07-21 8:55 ` Christian König
2020-07-21 9:16 ` Daniel Vetter
2020-07-21 9:24 ` Daniel Vetter
2020-07-21 9:37 ` Thomas Hellström (Intel)
2020-07-21 9:50 ` Daniel Vetter
2020-07-21 10:47 ` Thomas Hellström (Intel)
2020-07-21 13:59 ` Christian König
2020-07-21 17:46 ` Thomas Hellström (Intel)
2020-07-21 18:18 ` Daniel Vetter
2020-07-21 21:42 ` Dave Airlie
2020-07-21 22:45 ` Dave Airlie
2020-07-22 6:45 ` Thomas Hellström (Intel)
2020-07-22 7:11 ` Daniel Vetter
2020-07-22 8:05 ` Thomas Hellström (Intel)
2020-07-22 9:45 ` Daniel Vetter
2020-07-22 10:31 ` Thomas Hellström (Intel)
2020-07-22 11:39 ` Daniel Vetter
2020-07-22 12:22 ` Thomas Hellström (Intel)
2020-07-22 12:41 ` Daniel Vetter
2020-07-22 13:12 ` Thomas Hellström (Intel)
2020-07-22 14:07 ` Daniel Vetter
2020-07-22 14:23 ` Christian König
2020-07-22 14:30 ` Thomas Hellström (Intel)
2020-07-22 14:35 ` Christian König
2020-07-07 20:12 ` [PATCH 04/25] drm/vkms: Annotate vblank timer Daniel Vetter
2020-07-12 22:27 ` Rodrigo Siqueira
2020-07-14 9:57 ` Melissa Wen
2020-07-14 9:59 ` Daniel Vetter
2020-07-14 14:55 ` Melissa Wen
2020-07-14 15:23 ` Daniel Vetter
2020-07-07 20:12 ` [PATCH 05/25] drm/vblank: Annotate with dma-fence signalling section Daniel Vetter
2020-07-07 20:12 ` [PATCH 06/25] drm/amdgpu: add dma-fence annotations to atomic commit path Daniel Vetter
2020-07-07 20:12 ` [PATCH 07/25] drm/komdea: Annotate dma-fence critical section in " Daniel Vetter
2020-07-08 5:17 ` james qian wang (Arm Technology China)
2020-07-14 8:34 ` Daniel Vetter
2020-07-07 20:12 ` [PATCH 08/25] drm/malidp: " Daniel Vetter
2020-07-15 12:53 ` Liviu Dudau
2020-07-15 13:51 ` Daniel Vetter
2020-07-07 20:12 ` [PATCH 09/25] drm/atmel: Use drm_atomic_helper_commit Daniel Vetter
2020-07-07 20:37 ` Sam Ravnborg
2020-07-07 21:31 ` [PATCH] " Daniel Vetter
2020-07-14 9:55 ` Sam Ravnborg
2020-07-07 20:12 ` [PATCH 10/25] drm/imx: Annotate dma-fence critical section in commit path Daniel Vetter
2020-07-07 20:12 ` [PATCH 11/25] drm/omapdrm: " Daniel Vetter
2020-07-07 20:12 ` [PATCH 12/25] drm/rcar-du: " Daniel Vetter
2020-07-07 23:32 ` Laurent Pinchart
2020-07-14 8:39 ` Daniel Vetter
2020-07-07 20:12 ` [PATCH 13/25] drm/tegra: " Daniel Vetter
2020-07-07 20:12 ` [PATCH 14/25] drm/tidss: " Daniel Vetter
2020-07-08 9:01 ` Jyri Sarha
2020-07-07 20:12 ` [PATCH 15/25] drm/tilcdc: Use standard drm_atomic_helper_commit Daniel Vetter
2020-07-08 9:17 ` Jyri Sarha
2020-07-08 9:27 ` Daniel Vetter
2020-07-08 9:44 ` [PATCH] " Daniel Vetter
2020-07-08 10:21 ` Jyri Sarha
2020-07-08 14:20 ` Daniel Vetter
2020-07-10 11:16 ` Jyri Sarha
2020-07-14 8:32 ` Daniel Vetter
2020-07-07 20:12 ` [PATCH 16/25] drm/atomic-helper: Add dma-fence annotations Daniel Vetter
2020-07-07 20:12 ` [PATCH 17/25] drm/scheduler: use dma-fence annotations in main thread Daniel Vetter
2020-07-07 20:12 ` [PATCH 18/25] drm/amdgpu: use dma-fence annotations in cs_submit() Daniel Vetter
2020-07-07 20:12 ` [PATCH 19/25] drm/amdgpu: s/GFP_KERNEL/GFP_ATOMIC in scheduler code Daniel Vetter
2020-07-14 10:49 ` Daniel Vetter
2020-07-14 11:40 ` Christian König
2020-07-14 14:31 ` Daniel Vetter
2020-07-15 9:17 ` Christian König
2020-07-15 11:53 ` Daniel Vetter
2020-07-07 20:12 ` [PATCH 20/25] drm/amdgpu: DC also loves to allocate stuff where it shouldn't Daniel Vetter
2020-07-14 11:12 ` Daniel Vetter
2020-07-07 20:12 ` [PATCH 21/25] drm/amdgpu/dc: Stop dma_resv_lock inversion in commit_tail Daniel Vetter
2020-07-07 20:12 ` [PATCH 22/25] drm/scheduler: use dma-fence annotations in tdr work Daniel Vetter
2020-07-07 20:12 ` [PATCH 23/25] drm/amdgpu: use dma-fence annotations for gpu reset code Daniel Vetter
2020-07-07 20:12 ` [PATCH 24/25] Revert "drm/amdgpu: add fbdev suspend/resume on gpu reset" Daniel Vetter
2020-07-07 20:12 ` [PATCH 25/25] drm/amdgpu: gpu recovery does full modesets Daniel Vetter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=aede585d-6c3f-4559-c612-e8cb8a9e1f92@amd.com \
--to=christian.koenig@amd.com \
--cc=Felix.Kuehling@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=chris@chris-wilson.co.uk \
--cc=daniel.vetter@ffwll.ch \
--cc=daniel.vetter@intel.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=intel-gfx@lists.freedesktop.org \
--cc=jason@jlekstrand.net \
--cc=jenatali@microsoft.com \
--cc=linaro-mm-sig@lists.linaro.org \
--cc=linux-media@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=mika.kuoppala@intel.com \
--cc=spronovo@microsoft.com \
--cc=thomas.hellstrom@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).