From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D4CBC0044D for ; Mon, 16 Mar 2020 13:01:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F361820770 for ; Mon, 16 Mar 2020 13:01:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="cXFzTWMZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731069AbgCPNBd (ORCPT ); Mon, 16 Mar 2020 09:01:33 -0400 Received: from perceval.ideasonboard.com ([213.167.242.64]:36046 "EHLO perceval.ideasonboard.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731062AbgCPNBd (ORCPT ); Mon, 16 Mar 2020 09:01:33 -0400 Received: from pendragon.ideasonboard.com (81-175-216-236.bb.dnainternet.fi [81.175.216.236]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7D796A3B; Mon, 16 Mar 2020 14:01:30 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1584363690; bh=1V1sKsfdxrhZLhq7sVC0NonWtNyOzyBM3s3HfRmhCvU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=cXFzTWMZxFg/g9Y0jXJHkK1tDgLjh4IiWsYVysO1SnnrCEbAt7/e4OW5IfzeFL/Md ztuEz+KoJUEMpC46Ic5z7El5yC6ve8JpKk7zUuRV/HUT4IjeqbJBoZwDJ5uYq6twi6 mFraoaRezdE50aUgAYrFP53SrmsJTI9ddqiiOfbs= Date: Mon, 16 Mar 2020 15:01:25 +0200 From: Laurent Pinchart To: Tomek Bury Cc: Nicolas Dufresne , Daniel Vetter , xorg-devel , Maling list - DRI developers , "wayland-devel @ lists . freedesktop . org" , Discussion of the development of and with GStreamer , Jason Ekstrand , Bas Nieuwenhuizen , ML mesa-dev , Daniel Stone , Dave Airlie , linux-media@vger.kernel.org Subject: Re: Plumbing explicit synchronization through the Linux ecosystem Message-ID: <20200316130125.GK4732@pendragon.ideasonboard.com> References: <33d1749d876a83416c44671efcb37c74f87d1bd4.camel@ndufresne.ca> <20200316102034.GA30883@pendragon.ideasonboard.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-media-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Hi Tomek, On Mon, Mar 16, 2020 at 12:55:27PM +0000, Tomek Bury wrote: > Hi Jason, > > I've been wrestling with the sync problems in Wayland some time ago, but only > with regards to 3D drivers. > > The guarantee given by the GL/GLES spec is limited to a single graphics > context. If the same buffer is accessed by 2 contexts the outcome is > unspecified. The cross-context and cross-process synchronisation is not > guaranteed. It happens to work on Mesa, because the read/write locking is > implemented in the kernel space, but it didn't work on Broadcom driver, which > has read-write interlocks in user space. > >  A Vulkan client makes it even worse because of conflicting requirements: > Vulkan's vkQueuePresentKHR() passes in a number of semaphores but disallows > waiting. Wayland WSI requires wl_surface_commit() to be called from > vkQueuePresentKHR() which does require a wait, unless a synchronisation > primitive representing Vulkan samaphores is passed between Vulkan client and > the compositor. > > The most troublesome part was Wayland buffer release mechanism, as it only > involves a CPU signalling over Wayland IPC, without any 3D driver involvement. > The choices were: explicit synchronisation extension or a buffer copy in the > compositor (i.e. compositor textures from the copy, so the client can re-write > the original), or some implicit synchronisation in kernel space (but that > wasn't an option in Broadcom driver). > > With regards to V4L2, I believe it could easily work the same way as 3D > drivers, i.e. pass a buffer+fence pair to the next stage. The encode always > succeeds, but for capture or decode, the main problem is the uncertain outcome, > I believe? If we're fine with rendering or displaying an occasional broken > frame, then buffer+fence pair would work too. The broken frame will go into the > pipeline, but application can drain the pipeline and start over once the > capture works again. > > To answer some points raised by Laurent (although I'm unfamiliar with the > camera drivers): > > > you don't know until capture complete in which buffer the frame has > > been captured > > Surely you do, you only don't know in advance if the capture will be successful You do in kernelspace, but not in userspace at the moment, due to buffer recycling. > > but if an error occurs during capture, they can be recycled internally and > > put to the back of the queue. > > That would have to change in order to use explicit synchronisation. Every > started capture becomes immediately available as a buffer+fence pair. Fence is > signalled once the capture is finished (successfully or otherwise). The buffer > must not be reused until it's released, possibly with another fence - in that > case the buffer must not be reused until the release fence is signalled. We could certainly change this at least in some cases, but it would break existing userspace that doesn't expect incorrect frames. I'm however not sure we could change this behaviour in every case, there may be hardware that can't provide a guarantee on the order in which buffers will be used. I'm aware this wouldn't be compatible with explicit synchronization, and that's my point: camera hardware may not always support explicit synchronization. As long as we can fall back to not using fences then we should be fine. -- Regards, Laurent Pinchart From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.1 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C818C18E5B for ; Mon, 16 Mar 2020 13:01:41 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F20332076A for ; Mon, 16 Mar 2020 13:01:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ideasonboard.com header.i=@ideasonboard.com header.b="cXFzTWMZ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F20332076A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ideasonboard.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1A65A89804; Mon, 16 Mar 2020 13:01:34 +0000 (UTC) X-Greylist: delayed 9652 seconds by postgrey-1.36 at gabe; Mon, 16 Mar 2020 13:01:32 UTC Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by gabe.freedesktop.org (Postfix) with ESMTPS id DEC5389803; Mon, 16 Mar 2020 13:01:32 +0000 (UTC) Received: from pendragon.ideasonboard.com (81-175-216-236.bb.dnainternet.fi [81.175.216.236]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 7D796A3B; Mon, 16 Mar 2020 14:01:30 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1584363690; bh=1V1sKsfdxrhZLhq7sVC0NonWtNyOzyBM3s3HfRmhCvU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=cXFzTWMZxFg/g9Y0jXJHkK1tDgLjh4IiWsYVysO1SnnrCEbAt7/e4OW5IfzeFL/Md ztuEz+KoJUEMpC46Ic5z7El5yC6ve8JpKk7zUuRV/HUT4IjeqbJBoZwDJ5uYq6twi6 mFraoaRezdE50aUgAYrFP53SrmsJTI9ddqiiOfbs= Date: Mon, 16 Mar 2020 15:01:25 +0200 From: Laurent Pinchart To: Tomek Bury Subject: Re: Plumbing explicit synchronization through the Linux ecosystem Message-ID: <20200316130125.GK4732@pendragon.ideasonboard.com> References: <33d1749d876a83416c44671efcb37c74f87d1bd4.camel@ndufresne.ca> <20200316102034.GA30883@pendragon.ideasonboard.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Daniel Vetter , xorg-devel , Maling list - DRI developers , "wayland-devel @ lists . freedesktop . org" , Discussion of the development of and with GStreamer , Jason Ekstrand , ML mesa-dev , Nicolas Dufresne , linux-media@vger.kernel.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" SGkgVG9tZWssCgpPbiBNb24sIE1hciAxNiwgMjAyMCBhdCAxMjo1NToyN1BNICswMDAwLCBUb21l ayBCdXJ5IHdyb3RlOgo+IEhpIEphc29uLAo+IAo+IEkndmUgYmVlbiB3cmVzdGxpbmcgd2l0aCB0 aGUgc3luYyBwcm9ibGVtcyBpbiBXYXlsYW5kIHNvbWUgdGltZSBhZ28swqBidXQgb25seQo+IHdp dGggcmVnYXJkcyB0byAzRCBkcml2ZXJzLgo+IAo+IFRoZSBndWFyYW50ZWUgZ2l2ZW4gYnkgdGhl IEdML0dMRVMgc3BlYyBpcyBsaW1pdGVkIHRvIGEgc2luZ2xlIGdyYXBoaWNzCj4gY29udGV4dC4g SWYgdGhlIHNhbWUgYnVmZmVyIGlzIGFjY2Vzc2VkIGJ5IDIgY29udGV4dHMgdGhlIG91dGNvbWUg aXMKPiB1bnNwZWNpZmllZC4gVGhlIGNyb3NzLWNvbnRleHQgYW5kIGNyb3NzLXByb2Nlc3Mgc3lu Y2hyb25pc2F0aW9uIGlzIG5vdAo+IGd1YXJhbnRlZWQuwqBJdCBoYXBwZW5zIHRvIHdvcmsgb24g TWVzYSwgYmVjYXVzZSB0aGUgcmVhZC93cml0ZSBsb2NraW5nIGlzCj4gaW1wbGVtZW50ZWQgaW4g dGhlIGtlcm5lbCBzcGFjZSwgYnV0IGl0IGRpZG4ndCB3b3JrIG9uIEJyb2FkY29tIGRyaXZlciwg d2hpY2gKPiBoYXMgcmVhZC13cml0ZSBpbnRlcmxvY2tzIGluIHVzZXIgc3BhY2UuCj4gCj4gwqBB IFZ1bGthbiBjbGllbnQgbWFrZXMgaXQgZXZlbiB3b3JzZSBiZWNhdXNlIG9mIGNvbmZsaWN0aW5n IHJlcXVpcmVtZW50czoKPiBWdWxrYW4nc8KgdmtRdWV1ZVByZXNlbnRLSFIoKSBwYXNzZXMgaW4g YSBudW1iZXIgb2Ygc2VtYXBob3JlcyBidXQgZGlzYWxsb3dzCj4gd2FpdGluZy4gV2F5bGFuZCBX U0kgcmVxdWlyZXPCoHdsX3N1cmZhY2VfY29tbWl0KCkgdG8gYmUgY2FsbGVkIGZyb20KPiB2a1F1 ZXVlUHJlc2VudEtIUigpIHdoaWNoIGRvZXMgcmVxdWlyZSBhIHdhaXQsIHVubGVzcyBhIHN5bmNo cm9uaXNhdGlvbgo+IHByaW1pdGl2ZSByZXByZXNlbnRpbmcgVnVsa2FuIHNhbWFwaG9yZXPCoGlz IHBhc3NlZCBiZXR3ZWVuIFZ1bGthbiBjbGllbnQgYW5kCj4gdGhlIGNvbXBvc2l0b3IuCj4gCj4g VGhlIG1vc3QgdHJvdWJsZXNvbWUgcGFydCB3YXMgV2F5bGFuZCBidWZmZXIgcmVsZWFzZSBtZWNo YW5pc20sIGFzIGl0IG9ubHkKPiBpbnZvbHZlcyBhIENQVSBzaWduYWxsaW5nIG92ZXIgV2F5bGFu ZCBJUEMsIHdpdGhvdXQgYW55IDNEIGRyaXZlciBpbnZvbHZlbWVudC4KPiBUaGUgY2hvaWNlcyB3 ZXJlOiBleHBsaWNpdCBzeW5jaHJvbmlzYXRpb24gZXh0ZW5zaW9uIG9yIGEgYnVmZmVyIGNvcHkg aW4gdGhlCj4gY29tcG9zaXRvciAoaS5lLiBjb21wb3NpdG9yIHRleHR1cmVzIGZyb20gdGhlIGNv cHksIHNvIHRoZSBjbGllbnQgY2FuIHJlLXdyaXRlCj4gdGhlIG9yaWdpbmFsKSwgb3Igc29tZcKg aW1wbGljaXQgc3luY2hyb25pc2F0aW9uIGluIGtlcm5lbCBzcGFjZSAoYnV0IHRoYXQKPiB3YXNu J3QgYW4gb3B0aW9uIGluIEJyb2FkY29tIGRyaXZlcikuCj4gCj4gV2l0aCByZWdhcmRzIHRvIFY0 TDIsIEkgYmVsaWV2ZSBpdCBjb3VsZCBlYXNpbHkgd29yayB0aGUgc2FtZSB3YXkgYXMgM0QKPiBk cml2ZXJzLCBpLmUuIHBhc3MgYSBidWZmZXIrZmVuY2UgcGFpciB0byB0aGUgbmV4dCBzdGFnZS4g VGhlIGVuY29kZSBhbHdheXMKPiBzdWNjZWVkcywgYnV0IGZvciBjYXB0dXJlIG9yIGRlY29kZSwg dGhlIG1haW4gcHJvYmxlbSBpcyB0aGUgdW5jZXJ0YWluIG91dGNvbWUsCj4gSSBiZWxpZXZlPyBJ ZiB3ZSdyZSBmaW5lIHdpdGggcmVuZGVyaW5nIG9yIGRpc3BsYXlpbmcgYW4gb2NjYXNpb25hbCBi cm9rZW4KPiBmcmFtZSwgdGhlbiBidWZmZXIrZmVuY2UgcGFpciB3b3VsZCB3b3JrIHRvby4gVGhl IGJyb2tlbiBmcmFtZSB3aWxsIGdvIGludG8gdGhlCj4gcGlwZWxpbmUsIGJ1dCBhcHBsaWNhdGlv biBjYW4gZHJhaW4gdGhlIHBpcGVsaW5lIGFuZCBzdGFydCBvdmVyIG9uY2UgdGhlCj4gY2FwdHVy ZSB3b3JrcyBhZ2Fpbi4KPiAKPiBUbyBhbnN3ZXIgc29tZSBwb2ludHMgcmFpc2VkIGJ5IExhdXJl bnQgKGFsdGhvdWdoIEknbSB1bmZhbWlsaWFyIHdpdGggdGhlCj4gY2FtZXJhIGRyaXZlcnMpOgo+ IAo+ID4geW91wqBkb24ndCBrbm93IHVudGlsIGNhcHR1cmUgY29tcGxldGUgaW4gd2hpY2ggYnVm ZmVyIHRoZSBmcmFtZSBoYXMKPiA+IGJlZW7CoGNhcHR1cmVkCj4KPiBTdXJlbHkgeW91IGRvLCB5 b3Ugb25seSBkb24ndCBrbm93IGluIGFkdmFuY2UgaWYgdGhlIGNhcHR1cmUgd2lsbCBiZSBzdWNj ZXNzZnVsCgpZb3UgZG8gaW4ga2VybmVsc3BhY2UsIGJ1dCBub3QgaW4gdXNlcnNwYWNlIGF0IHRo ZSBtb21lbnQsIGR1ZSB0byBidWZmZXIKcmVjeWNsaW5nLgoKPiA+IGJ1dCBpZsKgYW4gZXJyb3Ig b2NjdXJzIGR1cmluZyBjYXB0dXJlLCB0aGV5IGNhbiBiZSByZWN5Y2xlZCBpbnRlcm5hbGx5IGFu ZAo+ID4gcHV0IHRvIHRoZSBiYWNrIG9mIHRoZSBxdWV1ZS4KPgo+IFRoYXQgd291bGQgaGF2ZSB0 byBjaGFuZ2UgaW4gb3JkZXIgdG8gdXNlIGV4cGxpY2l0IHN5bmNocm9uaXNhdGlvbi4gRXZlcnkK PiBzdGFydGVkIGNhcHR1cmUgYmVjb21lcyBpbW1lZGlhdGVseSBhdmFpbGFibGUgYXMgYSBidWZm ZXIrZmVuY2UgcGFpci4gRmVuY2UgaXMKPiBzaWduYWxsZWQgb25jZSB0aGUgY2FwdHVyZSBpcyBm aW5pc2hlZCAoc3VjY2Vzc2Z1bGx5IG9yIG90aGVyd2lzZSkuIFRoZSBidWZmZXIKPiBtdXN0IG5v dCBiZSByZXVzZWQgdW50aWwgaXQncyByZWxlYXNlZCwgcG9zc2libHkgd2l0aCBhbm90aGVyIGZl bmNlIC0gaW4gdGhhdAo+IGNhc2UgdGhlIGJ1ZmZlciBtdXN0IG5vdCBiZSByZXVzZWQgdW50aWwg dGhlIHJlbGVhc2UgZmVuY2UgaXMgc2lnbmFsbGVkLgoKV2UgY291bGQgY2VydGFpbmx5IGNoYW5n ZSB0aGlzIGF0IGxlYXN0IGluIHNvbWUgY2FzZXMsIGJ1dCBpdCB3b3VsZApicmVhayBleGlzdGlu ZyB1c2Vyc3BhY2UgdGhhdCBkb2Vzbid0IGV4cGVjdCBpbmNvcnJlY3QgZnJhbWVzLgoKSSdtIGhv d2V2ZXIgbm90IHN1cmUgd2UgY291bGQgY2hhbmdlIHRoaXMgYmVoYXZpb3VyIGluIGV2ZXJ5IGNh c2UsIHRoZXJlCm1heSBiZSBoYXJkd2FyZSB0aGF0IGNhbid0IHByb3ZpZGUgYSBndWFyYW50ZWUg b24gdGhlIG9yZGVyIGluIHdoaWNoCmJ1ZmZlcnMgd2lsbCBiZSB1c2VkLiBJJ20gYXdhcmUgdGhp cyB3b3VsZG4ndCBiZSBjb21wYXRpYmxlIHdpdGgKZXhwbGljaXQgc3luY2hyb25pemF0aW9uLCBh bmQgdGhhdCdzIG15IHBvaW50OiBjYW1lcmEgaGFyZHdhcmUgbWF5IG5vdAphbHdheXMgc3VwcG9y dCBleHBsaWNpdCBzeW5jaHJvbml6YXRpb24uIEFzIGxvbmcgYXMgd2UgY2FuIGZhbGwgYmFjayB0 bwpub3QgdXNpbmcgZmVuY2VzIHRoZW4gd2Ugc2hvdWxkIGJlIGZpbmUuCgotLSAKUmVnYXJkcywK CkxhdXJlbnQgUGluY2hhcnQKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18KZHJpLWRldmVsIG1haWxpbmcgbGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0 b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJp LWRldmVsCg==