From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FEF6C433EF for ; Wed, 1 Jun 2022 12:36:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352936AbiFAMgH (ORCPT ); Wed, 1 Jun 2022 08:36:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1352990AbiFAMfm (ORCPT ); Wed, 1 Jun 2022 08:35:42 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 743ABA0D02 for ; Wed, 1 Jun 2022 05:35:40 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id e25so2142918wra.11 for ; Wed, 01 Jun 2022 05:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:in-reply-to; bh=geWAW2xPBu0qLpWdiPyBgNivVlwU5h/jMqQrbUnnVfs=; b=BdiXXMzt8D06LixSdY84NDu9Ost8vce1JU5Ly7m8KGGcWhLpUhElYtn+kEPfDuaH/W ND88PNMAtKQj+HfoxEaDWhcrBetkIBFbdqglb7wVKPfHO9j+2zS1NmYUaejOw1aou2eM 2E2nTMfjIdKfDATqam2LNbA4ivHfMMsaofjKI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to; bh=geWAW2xPBu0qLpWdiPyBgNivVlwU5h/jMqQrbUnnVfs=; b=B93XBtQP1onLysE8Hq+bha8ygp69MxqS1OPSRLU7lXPyIVsHP9/nofK2mac0QBI4O5 6uGcyEWzqvnNOTZuaj6XhsaQN1a2iC/sx3zas6umtxlMSsyAW+dgnOcMYNPUXYU82XdH nw5PKFVuZ1KxdslXnTpycAkLpx9ctCTHpFncK3b6UTjgbZ6OJvFK/0RYPGS/KZHizzLN sAMRstz+AzeBmbkagirQDOPoHIpc738RvrErdfoPCa86/Vsb44EA79Ed3V75zFoIhYAB HRsM6hixjDGr6YrzP0QrKVUPXgSZCYoFBu5igOqCe2lM8snkByUi/Fg4Tlmwc2LivMTt 9oaw== X-Gm-Message-State: AOAM531gp+QFItZgffaqhsv3zRopTjK+E5hOmjpZSoMyTENqxKxSeE4i laFsYubww2slI9bvm7XamKFMrA== X-Google-Smtp-Source: ABdhPJzRlsH0LJwGkcjJA/JqoeBghm0kDNaIwYUbkjauNpnH8gIwKHdrawPBSHH7gN216H+LRzl0Uw== X-Received: by 2002:a5d:43cd:0:b0:20c:fdbd:8c11 with SMTP id v13-20020a5d43cd000000b0020cfdbd8c11mr53732471wrr.7.1654086938960; Wed, 01 Jun 2022 05:35:38 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id d12-20020a5d4f8c000000b00210346cd0b7sm1537054wru.101.2022.06.01.05.35.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 05:35:38 -0700 (PDT) Date: Wed, 1 Jun 2022 14:35:35 +0200 From: Daniel Vetter To: Maxime Ripard Cc: Samuel Holland , Heiko =?iso-8859-1?Q?St=FCbner?= , Sandy Huang , dri-devel@lists.freedesktop.org, linux-rockchip@lists.infradead.org, Alistair Francis , =?utf-8?Q?Ond=C5=99ej?= Jirman , Andreas Kemnade , David Airlie , Geert Uytterhoeven , Krzysztof Kozlowski , Liang Chen , Maarten Lankhorst , Michael Riesch , Nicolas Frattaroli , Peter Geis , Rob Herring , Sam Ravnborg , Thierry Reding , Thomas Zimmermann , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 00/16] drm/rockchip: Rockchip EBC ("E-Book Controller") display driver Message-ID: Mail-Followup-To: Maxime Ripard , Samuel Holland , Heiko =?iso-8859-1?Q?St=FCbner?= , Sandy Huang , dri-devel@lists.freedesktop.org, linux-rockchip@lists.infradead.org, Alistair Francis , =?utf-8?Q?Ond=C5=99ej?= Jirman , Andreas Kemnade , David Airlie , Geert Uytterhoeven , Krzysztof Kozlowski , Liang Chen , Maarten Lankhorst , Michael Riesch , Nicolas Frattaroli , Peter Geis , Rob Herring , Sam Ravnborg , Thierry Reding , Thomas Zimmermann , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: <20220413221916.50995-1-samuel@sholland.org> <20220414085018.ayjvscgdkoen5nw5@houat> <20220531085835.grw5nt4vyofis3po@penduick> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220531085835.grw5nt4vyofis3po@penduick> X-Operating-System: Linux phenom 5.10.0-8-amd64 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, May 31, 2022 at 10:58:35AM +0200, Maxime Ripard wrote: > Hi Daniel, > > Thanks for your feedback > > On Wed, May 25, 2022 at 07:18:07PM +0200, Daniel Vetter wrote: > > > > VBLANK Events and Asynchronous Commits > > > > ====================================== > > > > When should the VBLANK event complete? When the pixels have been blitted > > > > to the kernel's shadow buffer? When the first frame of the waveform is > > > > sent to the panel? When the last frame is sent to the panel? > > > > > > > > Currently, the driver is taking the first option, letting > > > > drm_atomic_helper_fake_vblank() send the VBLANK event without waiting on > > > > the refresh thread. This is the only way I was able to get good > > > > performance with existing userspace. > > > > > > I've been having the same kind of discussions in private lately, so I'm > > > interested by the answer as well :) > > > > > > It would be worth looking into the SPI/I2C panels for this, since it's > > > basically the same case. > > > > So it's maybe a bit misnamed and maybe kerneldocs aren't super clear (pls > > help improve them), but there's two modes: > > > > - drivers which have vblank, which might be somewhat variable (VRR) or > > become simulated (self-refresh panels), but otherwise is a more-or-less > > regular clock. For this case the atomic commit event must match the > > vblank events exactly (frame count and timestamp) > > Part of my interrogation there is do we have any kind of expectation > on whether or not, when we commit, the next vblank is going to be the > one matching that commit or we're allowed to defer it by an arbitrary > number of frames (provided that the frame count and timestamps are > correct) ? In general yes, but there's no guarantee. The only guarante we give for drivers with vblank counters is that if you receive a vblank event (flip complete or vblank event) for frame #n, then an immediate flip/atomic ioctl call will display earliest for frame #n+1. Also usually you should be able to hit #n+1, but even today with fun stuff like self refresh panels getting out of self refresh mode might take a bit more than a few frames, and so you might end up being late. But otoh if you just do a page flip loop then on average (after the crtc is fully resumed) you should be able to update at vrefresh rate exactly. > > - drivers which don't have vblank at all, mostly these are i2c/spi panels > > or virtual hw and stuff like that. In this case the event simply happens > > when the driver is done with refresh/upload, and the frame count should > > be zero (since it's meaningless). > > > > Unfortuantely the helper to dtrt has fake_vblank in it's name, maybe > > should be renamed to no_vblank or so (the various flags that control it > > are a bit better named). > > > > Again the docs should explain it all, but maybe we should clarify them or > > perhaps rename that helper to be more meaningful. > > > > > > Blitting/Blending in Software > > > > ============================= > > > > There are multiple layers to this topic (pun slightly intended): > > > > 1) Today's userspace does not expect a grayscale framebuffer. > > > > Currently, the driver advertises XRGB8888 and converts to Y4 > > > > in software. This seems to match other drivers (e.g. repaper). > > > > > > > > 2) Ignoring what userspace "wants", the closest existing format is > > > > DRM_FORMAT_R8. Geert sent a series[4] adding DRM_FORMAT_R1 through > > > > DRM_FORMAT_R4 (patch 9), which I believe are the "correct" formats > > > > to use. > > > > > > > > 3) The RK356x SoCs have an "RGA" hardware block that can do the > > > > RGB-to-grayscale conversion, and also RGB-to-dithered-monochrome > > > > which is needed for animation/video. Currently this is exposed with > > > > a V4L2 platform driver. Can this be inserted into the pipeline in a > > > > way that is transparent to userspace? Or must some userspace library > > > > be responsible for setting up the RGA => EBC pipeline? > > > > > > I'm very interested in this answer as well :) > > > > > > I think the current consensus is that it's up to userspace to set this > > > up though. > > > > Yeah I think v4l mem2mem device is the answer for these, and then > > userspace gets to set it all up. > > I think the question wasn't really about where that driver should be, > but more about who gets to set it up, and if the kernel could have > some component to expose the formats supported by the converter, but > whenever a commit is being done pipe that to the v4l2 device before > doing a page flip. > > We have a similar use-case for the RaspberryPi where the hardware > codec will produce a framebuffer format that isn't standard. That > format is understood by the display pipeline, and it can do > writeback. > > However, some people are using a separate display (like a SPI display > supported by tinydrm) and we would still like to be able to output the > decoded frames there. > > Is there some way we could plumb things to "route" that buffer through > the writeback engine to perform a format conversion before sending it > over to the SPI display automatically? Currently not transparently. Or at least no one has done that, and I'm not sure that's really a great idea. With big gpus all that stuff is done with separate command submission to the render side of things, and you can fully pipeline all that with in/out-fences. Doing that in the kms driver side in the kernel feels very wrong to me :-/ -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B3450C433EF for ; Wed, 1 Jun 2022 12:36:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=j+bCEa+jQJD4ZH0TCIYV4QYSvYW0xpE8Vts+7XDqzrE=; b=sHwHBKYZFWLINM OHVvivFwuoLYNIw+w8Tkv/6b56ytzxvHIINlz53rX+C3g69KGQCJYgKWuHpqWKit6HhgEQt4S3ugR SdA1n3bmR/xJUwqJz7Jg9jDCnupilm+GLoaiMczoe/4dx6yAOJA4fjGmphwvtVyl4xxkxXfoBPk8u 13GpuD+iCnNIzOzTjm4+wQLc76yBAr74rwLgROW0CGe+RW2xsVWCDQyWAT6Ubq5yb+ldudH++yUnG 1s8PStRwTWAWBBLNClRuy4t4jz2etm1nl1CtqyoIysWvgkQlLgk7ggkuUKFXH2r/WylA+7K1C3aWH bef/eeULsDDd73p2sraA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nwNZk-00G3rz-05; Wed, 01 Jun 2022 12:35:48 +0000 Received: from mail-wr1-x432.google.com ([2a00:1450:4864:20::432]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nwNZf-00G3me-DD for linux-rockchip@lists.infradead.org; Wed, 01 Jun 2022 12:35:46 +0000 Received: by mail-wr1-x432.google.com with SMTP id q7so2163021wrg.5 for ; Wed, 01 Jun 2022 05:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:in-reply-to; bh=geWAW2xPBu0qLpWdiPyBgNivVlwU5h/jMqQrbUnnVfs=; b=BdiXXMzt8D06LixSdY84NDu9Ost8vce1JU5Ly7m8KGGcWhLpUhElYtn+kEPfDuaH/W ND88PNMAtKQj+HfoxEaDWhcrBetkIBFbdqglb7wVKPfHO9j+2zS1NmYUaejOw1aou2eM 2E2nTMfjIdKfDATqam2LNbA4ivHfMMsaofjKI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to; bh=geWAW2xPBu0qLpWdiPyBgNivVlwU5h/jMqQrbUnnVfs=; b=RHqMaKAacncwh0hm9qxfpjdG4tKi5D72hc2ywin42LkgHKel9m+gx/ZbVdxhPukeG6 xFaZ2i05+EbIBE6sipte2t1zkQxqPesGczVNwt0KXrgB1O7fUyHQ3NCYh5kaS9rW1ArB RwgE7j0SnAU3DCTW116nzqS+jfgYQZ4QUIA7k35ZJTwgtjUfw3t7YYvpdSkcD4dWrHpP 52WQYf977dHZPGxRMS+OnGBb/YevoDONWJWLfCmB+aAIbAOcCGExFbcAZ24giE+qsj9I ilOngI6pl6sVGPf1g9HueRlUJag/cMJmwY6zfdchrE0+EOPMeUA3sHWdxeYvy4zLm8Xk dksA== X-Gm-Message-State: AOAM533REwrHN4+5GJw54eQA63ycVcqxniNspJE7HES++cEHvFoolMLy zm+bQ6Zu3OoMcQsaf/eRfWIJYg== X-Google-Smtp-Source: ABdhPJzRlsH0LJwGkcjJA/JqoeBghm0kDNaIwYUbkjauNpnH8gIwKHdrawPBSHH7gN216H+LRzl0Uw== X-Received: by 2002:a5d:43cd:0:b0:20c:fdbd:8c11 with SMTP id v13-20020a5d43cd000000b0020cfdbd8c11mr53732471wrr.7.1654086938960; Wed, 01 Jun 2022 05:35:38 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id d12-20020a5d4f8c000000b00210346cd0b7sm1537054wru.101.2022.06.01.05.35.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 05:35:38 -0700 (PDT) Date: Wed, 1 Jun 2022 14:35:35 +0200 From: Daniel Vetter To: Maxime Ripard Cc: Samuel Holland , Heiko =?iso-8859-1?Q?St=FCbner?= , Sandy Huang , dri-devel@lists.freedesktop.org, linux-rockchip@lists.infradead.org, Alistair Francis , =?utf-8?Q?Ond=C5=99ej?= Jirman , Andreas Kemnade , David Airlie , Geert Uytterhoeven , Krzysztof Kozlowski , Liang Chen , Maarten Lankhorst , Michael Riesch , Nicolas Frattaroli , Peter Geis , Rob Herring , Sam Ravnborg , Thierry Reding , Thomas Zimmermann , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 00/16] drm/rockchip: Rockchip EBC ("E-Book Controller") display driver Message-ID: Mail-Followup-To: Maxime Ripard , Samuel Holland , Heiko =?iso-8859-1?Q?St=FCbner?= , Sandy Huang , dri-devel@lists.freedesktop.org, linux-rockchip@lists.infradead.org, Alistair Francis , =?utf-8?Q?Ond=C5=99ej?= Jirman , Andreas Kemnade , David Airlie , Geert Uytterhoeven , Krzysztof Kozlowski , Liang Chen , Maarten Lankhorst , Michael Riesch , Nicolas Frattaroli , Peter Geis , Rob Herring , Sam Ravnborg , Thierry Reding , Thomas Zimmermann , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: <20220413221916.50995-1-samuel@sholland.org> <20220414085018.ayjvscgdkoen5nw5@houat> <20220531085835.grw5nt4vyofis3po@penduick> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220531085835.grw5nt4vyofis3po@penduick> X-Operating-System: Linux phenom 5.10.0-8-amd64 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220601_053543_596494_79BD0BE9 X-CRM114-Status: GOOD ( 61.26 ) X-BeenThere: linux-rockchip@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Upstream kernel work for Rockchip platforms List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Linux-rockchip" Errors-To: linux-rockchip-bounces+linux-rockchip=archiver.kernel.org@lists.infradead.org On Tue, May 31, 2022 at 10:58:35AM +0200, Maxime Ripard wrote: > Hi Daniel, > > Thanks for your feedback > > On Wed, May 25, 2022 at 07:18:07PM +0200, Daniel Vetter wrote: > > > > VBLANK Events and Asynchronous Commits > > > > ====================================== > > > > When should the VBLANK event complete? When the pixels have been blitted > > > > to the kernel's shadow buffer? When the first frame of the waveform is > > > > sent to the panel? When the last frame is sent to the panel? > > > > > > > > Currently, the driver is taking the first option, letting > > > > drm_atomic_helper_fake_vblank() send the VBLANK event without waiting on > > > > the refresh thread. This is the only way I was able to get good > > > > performance with existing userspace. > > > > > > I've been having the same kind of discussions in private lately, so I'm > > > interested by the answer as well :) > > > > > > It would be worth looking into the SPI/I2C panels for this, since it's > > > basically the same case. > > > > So it's maybe a bit misnamed and maybe kerneldocs aren't super clear (pls > > help improve them), but there's two modes: > > > > - drivers which have vblank, which might be somewhat variable (VRR) or > > become simulated (self-refresh panels), but otherwise is a more-or-less > > regular clock. For this case the atomic commit event must match the > > vblank events exactly (frame count and timestamp) > > Part of my interrogation there is do we have any kind of expectation > on whether or not, when we commit, the next vblank is going to be the > one matching that commit or we're allowed to defer it by an arbitrary > number of frames (provided that the frame count and timestamps are > correct) ? In general yes, but there's no guarantee. The only guarante we give for drivers with vblank counters is that if you receive a vblank event (flip complete or vblank event) for frame #n, then an immediate flip/atomic ioctl call will display earliest for frame #n+1. Also usually you should be able to hit #n+1, but even today with fun stuff like self refresh panels getting out of self refresh mode might take a bit more than a few frames, and so you might end up being late. But otoh if you just do a page flip loop then on average (after the crtc is fully resumed) you should be able to update at vrefresh rate exactly. > > - drivers which don't have vblank at all, mostly these are i2c/spi panels > > or virtual hw and stuff like that. In this case the event simply happens > > when the driver is done with refresh/upload, and the frame count should > > be zero (since it's meaningless). > > > > Unfortuantely the helper to dtrt has fake_vblank in it's name, maybe > > should be renamed to no_vblank or so (the various flags that control it > > are a bit better named). > > > > Again the docs should explain it all, but maybe we should clarify them or > > perhaps rename that helper to be more meaningful. > > > > > > Blitting/Blending in Software > > > > ============================= > > > > There are multiple layers to this topic (pun slightly intended): > > > > 1) Today's userspace does not expect a grayscale framebuffer. > > > > Currently, the driver advertises XRGB8888 and converts to Y4 > > > > in software. This seems to match other drivers (e.g. repaper). > > > > > > > > 2) Ignoring what userspace "wants", the closest existing format is > > > > DRM_FORMAT_R8. Geert sent a series[4] adding DRM_FORMAT_R1 through > > > > DRM_FORMAT_R4 (patch 9), which I believe are the "correct" formats > > > > to use. > > > > > > > > 3) The RK356x SoCs have an "RGA" hardware block that can do the > > > > RGB-to-grayscale conversion, and also RGB-to-dithered-monochrome > > > > which is needed for animation/video. Currently this is exposed with > > > > a V4L2 platform driver. Can this be inserted into the pipeline in a > > > > way that is transparent to userspace? Or must some userspace library > > > > be responsible for setting up the RGA => EBC pipeline? > > > > > > I'm very interested in this answer as well :) > > > > > > I think the current consensus is that it's up to userspace to set this > > > up though. > > > > Yeah I think v4l mem2mem device is the answer for these, and then > > userspace gets to set it all up. > > I think the question wasn't really about where that driver should be, > but more about who gets to set it up, and if the kernel could have > some component to expose the formats supported by the converter, but > whenever a commit is being done pipe that to the v4l2 device before > doing a page flip. > > We have a similar use-case for the RaspberryPi where the hardware > codec will produce a framebuffer format that isn't standard. That > format is understood by the display pipeline, and it can do > writeback. > > However, some people are using a separate display (like a SPI display > supported by tinydrm) and we would still like to be able to output the > decoded frames there. > > Is there some way we could plumb things to "route" that buffer through > the writeback engine to perform a format conversion before sending it > over to the SPI display automatically? Currently not transparently. Or at least no one has done that, and I'm not sure that's really a great idea. With big gpus all that stuff is done with separate command submission to the render side of things, and you can fully pipeline all that with in/out-fences. Doing that in the kms driver side in the kernel feels very wrong to me :-/ -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ Linux-rockchip mailing list Linux-rockchip@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-rockchip From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C103EC433EF for ; Wed, 1 Jun 2022 12:35:42 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 10EB610E06A; Wed, 1 Jun 2022 12:35:42 +0000 (UTC) Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6C37810E06A for ; Wed, 1 Jun 2022 12:35:40 +0000 (UTC) Received: by mail-wr1-x42b.google.com with SMTP id q21so2179544wra.2 for ; Wed, 01 Jun 2022 05:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:in-reply-to; bh=geWAW2xPBu0qLpWdiPyBgNivVlwU5h/jMqQrbUnnVfs=; b=BdiXXMzt8D06LixSdY84NDu9Ost8vce1JU5Ly7m8KGGcWhLpUhElYtn+kEPfDuaH/W ND88PNMAtKQj+HfoxEaDWhcrBetkIBFbdqglb7wVKPfHO9j+2zS1NmYUaejOw1aou2eM 2E2nTMfjIdKfDATqam2LNbA4ivHfMMsaofjKI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to; bh=geWAW2xPBu0qLpWdiPyBgNivVlwU5h/jMqQrbUnnVfs=; b=FmpZ08uwq7/2/PRRkhX45k/iClCgZnjqAdPv0bY2p/IKYLq+oBzLcUP1c9p6xUwtim 970U3KYOI8TYck27/MXbwnnSAVOpNuecOoRS329njf4mobZm4yAVwtKkY7Io6xgwU6Hw /hcl45ONWCIRJPl2ai5vn0DpxTngzEKmR7G93WgwDILJ+QCKtE04K9SoJ45/3ldDlNF0 LVbXgGWrxgh1nlGlDKMjBsRe3l5pRLRsJs06iQNzeONTWKsrDhsnohh6YgEBRVlOvc/4 M0hdpZJBVK7J+6hX0oTJ0Mitc/GYCi/QfLUBuHlM30jleOAWPv7tyrWmFJcXc+Ap4iU6 PAPw== X-Gm-Message-State: AOAM5319ta+DpUfyZmx9I8pbKo7xX9iHZrMgw2PZ9D1HaJrIFQOCxQqj uhL/bAizJtHiS1Fsq2p034lWbg== X-Google-Smtp-Source: ABdhPJzRlsH0LJwGkcjJA/JqoeBghm0kDNaIwYUbkjauNpnH8gIwKHdrawPBSHH7gN216H+LRzl0Uw== X-Received: by 2002:a5d:43cd:0:b0:20c:fdbd:8c11 with SMTP id v13-20020a5d43cd000000b0020cfdbd8c11mr53732471wrr.7.1654086938960; Wed, 01 Jun 2022 05:35:38 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id d12-20020a5d4f8c000000b00210346cd0b7sm1537054wru.101.2022.06.01.05.35.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 05:35:38 -0700 (PDT) Date: Wed, 1 Jun 2022 14:35:35 +0200 From: Daniel Vetter To: Maxime Ripard Subject: Re: [RFC PATCH 00/16] drm/rockchip: Rockchip EBC ("E-Book Controller") display driver Message-ID: Mail-Followup-To: Maxime Ripard , Samuel Holland , Heiko =?iso-8859-1?Q?St=FCbner?= , Sandy Huang , dri-devel@lists.freedesktop.org, linux-rockchip@lists.infradead.org, Alistair Francis , =?utf-8?Q?Ond=C5=99ej?= Jirman , Andreas Kemnade , David Airlie , Geert Uytterhoeven , Krzysztof Kozlowski , Liang Chen , Maarten Lankhorst , Michael Riesch , Nicolas Frattaroli , Peter Geis , Rob Herring , Sam Ravnborg , Thierry Reding , Thomas Zimmermann , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: <20220413221916.50995-1-samuel@sholland.org> <20220414085018.ayjvscgdkoen5nw5@houat> <20220531085835.grw5nt4vyofis3po@penduick> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220531085835.grw5nt4vyofis3po@penduick> X-Operating-System: Linux phenom 5.10.0-8-amd64 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: David Airlie , =?utf-8?Q?Ond=C5=99ej?= Jirman , dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, Thierry Reding , Michael Riesch , Sam Ravnborg , Samuel Holland , Nicolas Frattaroli , linux-rockchip@lists.infradead.org, Andreas Kemnade , Geert Uytterhoeven , Liang Chen , devicetree@vger.kernel.org, Peter Geis , Alistair Francis , Rob Herring , linux-arm-kernel@lists.infradead.org, Sandy Huang , Thomas Zimmermann , Krzysztof Kozlowski Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Tue, May 31, 2022 at 10:58:35AM +0200, Maxime Ripard wrote: > Hi Daniel, > > Thanks for your feedback > > On Wed, May 25, 2022 at 07:18:07PM +0200, Daniel Vetter wrote: > > > > VBLANK Events and Asynchronous Commits > > > > ====================================== > > > > When should the VBLANK event complete? When the pixels have been blitted > > > > to the kernel's shadow buffer? When the first frame of the waveform is > > > > sent to the panel? When the last frame is sent to the panel? > > > > > > > > Currently, the driver is taking the first option, letting > > > > drm_atomic_helper_fake_vblank() send the VBLANK event without waiting on > > > > the refresh thread. This is the only way I was able to get good > > > > performance with existing userspace. > > > > > > I've been having the same kind of discussions in private lately, so I'm > > > interested by the answer as well :) > > > > > > It would be worth looking into the SPI/I2C panels for this, since it's > > > basically the same case. > > > > So it's maybe a bit misnamed and maybe kerneldocs aren't super clear (pls > > help improve them), but there's two modes: > > > > - drivers which have vblank, which might be somewhat variable (VRR) or > > become simulated (self-refresh panels), but otherwise is a more-or-less > > regular clock. For this case the atomic commit event must match the > > vblank events exactly (frame count and timestamp) > > Part of my interrogation there is do we have any kind of expectation > on whether or not, when we commit, the next vblank is going to be the > one matching that commit or we're allowed to defer it by an arbitrary > number of frames (provided that the frame count and timestamps are > correct) ? In general yes, but there's no guarantee. The only guarante we give for drivers with vblank counters is that if you receive a vblank event (flip complete or vblank event) for frame #n, then an immediate flip/atomic ioctl call will display earliest for frame #n+1. Also usually you should be able to hit #n+1, but even today with fun stuff like self refresh panels getting out of self refresh mode might take a bit more than a few frames, and so you might end up being late. But otoh if you just do a page flip loop then on average (after the crtc is fully resumed) you should be able to update at vrefresh rate exactly. > > - drivers which don't have vblank at all, mostly these are i2c/spi panels > > or virtual hw and stuff like that. In this case the event simply happens > > when the driver is done with refresh/upload, and the frame count should > > be zero (since it's meaningless). > > > > Unfortuantely the helper to dtrt has fake_vblank in it's name, maybe > > should be renamed to no_vblank or so (the various flags that control it > > are a bit better named). > > > > Again the docs should explain it all, but maybe we should clarify them or > > perhaps rename that helper to be more meaningful. > > > > > > Blitting/Blending in Software > > > > ============================= > > > > There are multiple layers to this topic (pun slightly intended): > > > > 1) Today's userspace does not expect a grayscale framebuffer. > > > > Currently, the driver advertises XRGB8888 and converts to Y4 > > > > in software. This seems to match other drivers (e.g. repaper). > > > > > > > > 2) Ignoring what userspace "wants", the closest existing format is > > > > DRM_FORMAT_R8. Geert sent a series[4] adding DRM_FORMAT_R1 through > > > > DRM_FORMAT_R4 (patch 9), which I believe are the "correct" formats > > > > to use. > > > > > > > > 3) The RK356x SoCs have an "RGA" hardware block that can do the > > > > RGB-to-grayscale conversion, and also RGB-to-dithered-monochrome > > > > which is needed for animation/video. Currently this is exposed with > > > > a V4L2 platform driver. Can this be inserted into the pipeline in a > > > > way that is transparent to userspace? Or must some userspace library > > > > be responsible for setting up the RGA => EBC pipeline? > > > > > > I'm very interested in this answer as well :) > > > > > > I think the current consensus is that it's up to userspace to set this > > > up though. > > > > Yeah I think v4l mem2mem device is the answer for these, and then > > userspace gets to set it all up. > > I think the question wasn't really about where that driver should be, > but more about who gets to set it up, and if the kernel could have > some component to expose the formats supported by the converter, but > whenever a commit is being done pipe that to the v4l2 device before > doing a page flip. > > We have a similar use-case for the RaspberryPi where the hardware > codec will produce a framebuffer format that isn't standard. That > format is understood by the display pipeline, and it can do > writeback. > > However, some people are using a separate display (like a SPI display > supported by tinydrm) and we would still like to be able to output the > decoded frames there. > > Is there some way we could plumb things to "route" that buffer through > the writeback engine to perform a format conversion before sending it > over to the SPI display automatically? Currently not transparently. Or at least no one has done that, and I'm not sure that's really a great idea. With big gpus all that stuff is done with separate command submission to the render side of things, and you can fully pipeline all that with in/out-fences. Doing that in the kms driver side in the kernel feels very wrong to me :-/ -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4CF52C433F5 for ; Wed, 1 Jun 2022 12:37:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BZw2M62wDHvIN0Ytggk6tArUDwiTAJjJAAvayUC3p/8=; b=MgKVJP7ztnVcXC k34LAYDSLr/MgTfPo6hsafalo1TU2zikMj2blKPTZVQROhHzAH/CxSlbHmI38hqHA/obzap9zmiFk t3HMKBFUs2jqkj/b0pTzLqzoPKv+NAI6x4avmqNvuI17s7tshJFcmfg1NHBEmhBy/XRCdvXvktgDb NXczD2GkT7NOvCS8nv00IMHe8milKRrBO0GLskYp+cBeX61hKqWeOAfVPqJoqEsXLH6zah9T+8JB3 x4JqSz/FQsKrDDnHf8O6s80lJ6c5ztVs13D8FnMH2Zx+PLbUIylnZtwaNuVg6wqTDrEVXhDnjFagv yn8YO5MDlPu0L9NO8qqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nwNZl-00G3sK-5N; Wed, 01 Jun 2022 12:35:49 +0000 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nwNZf-00G3mf-B3 for linux-arm-kernel@lists.infradead.org; Wed, 01 Jun 2022 12:35:46 +0000 Received: by mail-wr1-x434.google.com with SMTP id p10so2146644wrg.12 for ; Wed, 01 Jun 2022 05:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ffwll.ch; s=google; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-disposition:in-reply-to; bh=geWAW2xPBu0qLpWdiPyBgNivVlwU5h/jMqQrbUnnVfs=; b=BdiXXMzt8D06LixSdY84NDu9Ost8vce1JU5Ly7m8KGGcWhLpUhElYtn+kEPfDuaH/W ND88PNMAtKQj+HfoxEaDWhcrBetkIBFbdqglb7wVKPfHO9j+2zS1NmYUaejOw1aou2eM 2E2nTMfjIdKfDATqam2LNbA4ivHfMMsaofjKI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id :mail-followup-to:references:mime-version:content-disposition :in-reply-to; bh=geWAW2xPBu0qLpWdiPyBgNivVlwU5h/jMqQrbUnnVfs=; b=JcvHBjOY7/NzuTpysG/n5kEBsnjf2P8BCJHOlkeRQvjSdCmoO6uq47oX0+Wie+Wr9E LO2aaungNx+7qmmb094yIRqEXNzQsVr/W7p3xYMgc2f0oxgqMEg1yDZVasxSSn54YHlH 52S7BWwU4QSRvuoTWWpVlthJ7eI+NBI3MgAcqypuzh8vRP22GmVstnx1LwiJ501uia2D zmDac5JLtymgDs+leDeKwfPL7UauUCJWeSklYebKc33x8PWji7KXaCIR83AlxLWI8uCE QbNn9rRsvHhfQcVD0ni4Ms+hJYwTkCrqvT+TfiI6iBy+thiOfa/GQLoJkdp22kz0rPe/ 0dSw== X-Gm-Message-State: AOAM531OZJPrvZvYkfb/PVZKKGbviq6rn2GMjPBYrEO2oEnQZyjkbp0O fRddUQFGWjRyvQE5kGZ48W50JzTFN/ehgg== X-Google-Smtp-Source: ABdhPJzRlsH0LJwGkcjJA/JqoeBghm0kDNaIwYUbkjauNpnH8gIwKHdrawPBSHH7gN216H+LRzl0Uw== X-Received: by 2002:a5d:43cd:0:b0:20c:fdbd:8c11 with SMTP id v13-20020a5d43cd000000b0020cfdbd8c11mr53732471wrr.7.1654086938960; Wed, 01 Jun 2022 05:35:38 -0700 (PDT) Received: from phenom.ffwll.local ([2a02:168:57f4:0:efd0:b9e5:5ae6:c2fa]) by smtp.gmail.com with ESMTPSA id d12-20020a5d4f8c000000b00210346cd0b7sm1537054wru.101.2022.06.01.05.35.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 05:35:38 -0700 (PDT) Date: Wed, 1 Jun 2022 14:35:35 +0200 From: Daniel Vetter To: Maxime Ripard Cc: Samuel Holland , Heiko =?iso-8859-1?Q?St=FCbner?= , Sandy Huang , dri-devel@lists.freedesktop.org, linux-rockchip@lists.infradead.org, Alistair Francis , =?utf-8?Q?Ond=C5=99ej?= Jirman , Andreas Kemnade , David Airlie , Geert Uytterhoeven , Krzysztof Kozlowski , Liang Chen , Maarten Lankhorst , Michael Riesch , Nicolas Frattaroli , Peter Geis , Rob Herring , Sam Ravnborg , Thierry Reding , Thomas Zimmermann , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 00/16] drm/rockchip: Rockchip EBC ("E-Book Controller") display driver Message-ID: Mail-Followup-To: Maxime Ripard , Samuel Holland , Heiko =?iso-8859-1?Q?St=FCbner?= , Sandy Huang , dri-devel@lists.freedesktop.org, linux-rockchip@lists.infradead.org, Alistair Francis , =?utf-8?Q?Ond=C5=99ej?= Jirman , Andreas Kemnade , David Airlie , Geert Uytterhoeven , Krzysztof Kozlowski , Liang Chen , Maarten Lankhorst , Michael Riesch , Nicolas Frattaroli , Peter Geis , Rob Herring , Sam Ravnborg , Thierry Reding , Thomas Zimmermann , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org References: <20220413221916.50995-1-samuel@sholland.org> <20220414085018.ayjvscgdkoen5nw5@houat> <20220531085835.grw5nt4vyofis3po@penduick> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220531085835.grw5nt4vyofis3po@penduick> X-Operating-System: Linux phenom 5.10.0-8-amd64 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220601_053543_610901_E0AED91E X-CRM114-Status: GOOD ( 62.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, May 31, 2022 at 10:58:35AM +0200, Maxime Ripard wrote: > Hi Daniel, > > Thanks for your feedback > > On Wed, May 25, 2022 at 07:18:07PM +0200, Daniel Vetter wrote: > > > > VBLANK Events and Asynchronous Commits > > > > ====================================== > > > > When should the VBLANK event complete? When the pixels have been blitted > > > > to the kernel's shadow buffer? When the first frame of the waveform is > > > > sent to the panel? When the last frame is sent to the panel? > > > > > > > > Currently, the driver is taking the first option, letting > > > > drm_atomic_helper_fake_vblank() send the VBLANK event without waiting on > > > > the refresh thread. This is the only way I was able to get good > > > > performance with existing userspace. > > > > > > I've been having the same kind of discussions in private lately, so I'm > > > interested by the answer as well :) > > > > > > It would be worth looking into the SPI/I2C panels for this, since it's > > > basically the same case. > > > > So it's maybe a bit misnamed and maybe kerneldocs aren't super clear (pls > > help improve them), but there's two modes: > > > > - drivers which have vblank, which might be somewhat variable (VRR) or > > become simulated (self-refresh panels), but otherwise is a more-or-less > > regular clock. For this case the atomic commit event must match the > > vblank events exactly (frame count and timestamp) > > Part of my interrogation there is do we have any kind of expectation > on whether or not, when we commit, the next vblank is going to be the > one matching that commit or we're allowed to defer it by an arbitrary > number of frames (provided that the frame count and timestamps are > correct) ? In general yes, but there's no guarantee. The only guarante we give for drivers with vblank counters is that if you receive a vblank event (flip complete or vblank event) for frame #n, then an immediate flip/atomic ioctl call will display earliest for frame #n+1. Also usually you should be able to hit #n+1, but even today with fun stuff like self refresh panels getting out of self refresh mode might take a bit more than a few frames, and so you might end up being late. But otoh if you just do a page flip loop then on average (after the crtc is fully resumed) you should be able to update at vrefresh rate exactly. > > - drivers which don't have vblank at all, mostly these are i2c/spi panels > > or virtual hw and stuff like that. In this case the event simply happens > > when the driver is done with refresh/upload, and the frame count should > > be zero (since it's meaningless). > > > > Unfortuantely the helper to dtrt has fake_vblank in it's name, maybe > > should be renamed to no_vblank or so (the various flags that control it > > are a bit better named). > > > > Again the docs should explain it all, but maybe we should clarify them or > > perhaps rename that helper to be more meaningful. > > > > > > Blitting/Blending in Software > > > > ============================= > > > > There are multiple layers to this topic (pun slightly intended): > > > > 1) Today's userspace does not expect a grayscale framebuffer. > > > > Currently, the driver advertises XRGB8888 and converts to Y4 > > > > in software. This seems to match other drivers (e.g. repaper). > > > > > > > > 2) Ignoring what userspace "wants", the closest existing format is > > > > DRM_FORMAT_R8. Geert sent a series[4] adding DRM_FORMAT_R1 through > > > > DRM_FORMAT_R4 (patch 9), which I believe are the "correct" formats > > > > to use. > > > > > > > > 3) The RK356x SoCs have an "RGA" hardware block that can do the > > > > RGB-to-grayscale conversion, and also RGB-to-dithered-monochrome > > > > which is needed for animation/video. Currently this is exposed with > > > > a V4L2 platform driver. Can this be inserted into the pipeline in a > > > > way that is transparent to userspace? Or must some userspace library > > > > be responsible for setting up the RGA => EBC pipeline? > > > > > > I'm very interested in this answer as well :) > > > > > > I think the current consensus is that it's up to userspace to set this > > > up though. > > > > Yeah I think v4l mem2mem device is the answer for these, and then > > userspace gets to set it all up. > > I think the question wasn't really about where that driver should be, > but more about who gets to set it up, and if the kernel could have > some component to expose the formats supported by the converter, but > whenever a commit is being done pipe that to the v4l2 device before > doing a page flip. > > We have a similar use-case for the RaspberryPi where the hardware > codec will produce a framebuffer format that isn't standard. That > format is understood by the display pipeline, and it can do > writeback. > > However, some people are using a separate display (like a SPI display > supported by tinydrm) and we would still like to be able to output the > decoded frames there. > > Is there some way we could plumb things to "route" that buffer through > the writeback engine to perform a format conversion before sending it > over to the SPI display automatically? Currently not transparently. Or at least no one has done that, and I'm not sure that's really a great idea. With big gpus all that stuff is done with separate command submission to the render side of things, and you can fully pipeline all that with in/out-fences. Doing that in the kms driver side in the kernel feels very wrong to me :-/ -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel