From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BA30C05027 for ; Wed, 8 Feb 2023 19:44:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231726AbjBHToq (ORCPT ); Wed, 8 Feb 2023 14:44:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229515AbjBHToo (ORCPT ); Wed, 8 Feb 2023 14:44:44 -0500 Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C3E6869E for ; Wed, 8 Feb 2023 11:44:43 -0800 (PST) Received: from ptx.hi.pengutronix.de ([2001:67c:670:100:1d::c0]) by metis.ext.pengutronix.de with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pPqMk-0000MT-Df; Wed, 08 Feb 2023 20:44:26 +0100 Received: from mfe by ptx.hi.pengutronix.de with local (Exim 4.92) (envelope-from ) id 1pPqMj-0003FP-6D; Wed, 08 Feb 2023 20:44:25 +0100 Date: Wed, 8 Feb 2023 20:44:25 +0100 From: Marco Felsch To: Laurent Pinchart Cc: Mauro Carvalho Chehab , p.zabel@pengutronix.de, slongerbeam@gmail.com, hverkuil-cisco@xs4all.nl, sakari.ailus@linux.intel.com, linux-arm-kernel@lists.infradead.org, linux-media@vger.kernel.org, kernel@pengutronix.de Subject: Re: [PATCH 1/6] media: uapi: Add MEDIA_BUS_FMT_SGRGB_IGIG_GBGR_IGIG media bus formats Message-ID: <20230208194425.anis36f6jlzmsjwp@pengutronix.de> References: <20210427120701.21809-1-m.felsch@pengutronix.de> <20210427120701.21809-2-m.felsch@pengutronix.de> <20210429074903.cc5gohn52cgv4i5z@pengutronix.de> <20210430065134.jwscxlv3sydo4zgy@pengutronix.de> <20210430145146.359a9b90@coco.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::c0 X-SA-Exim-Mail-From: mfe@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-media@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Hi all, sorry for the long absence on this topic. Only a few years later I'm back on this topic :) On 21-04-30, Laurent Pinchart wrote: > On Fri, Apr 30, 2021 at 02:51:46PM +0200, Mauro Carvalho Chehab wrote: > > Em Fri, 30 Apr 2021 15:18:52 +0300 > > Laurent Pinchart escreveu: > > > > > Hi Marco, > > > > > > On Fri, Apr 30, 2021 at 08:51:34AM +0200, Marco Felsch wrote: > > > > On 21-04-30 01:14, Laurent Pinchart wrote: > > > > > On Thu, Apr 29, 2021 at 09:49:03AM +0200, Marco Felsch wrote: > > > > > > On 21-04-29 04:51, Laurent Pinchart wrote: > > > > > > > On Tue, Apr 27, 2021 at 02:06:56PM +0200, Marco Felsch wrote: > > > > > > > > Add special 8/12bit bayer media bus format for the OnSemi AR0237IR > > > > > > > > camera sensor [1]. OnSemi calls this format RGB-IR, the pixel array > > > > > > > > with the interleaved IR pixels looks like: > > > > > > > > > > > > > > > > | G | R | G | B | ... > > > > > > > > +----+----+----+----+--- > > > > > > > > | IR | G | IR | G | ... > > > > > > > > +----+----+----+----+--- > > > > > > > > | G | B | G | R | ... > > > > > > > > +----+----+----+----+--- > > > > > > > > | IR | G | IR | G | ... > > > > > > > > +----+----+----+----+--- > > > > > > > > | .. | .. | .. | .. | .. > > > > > > > > > > > > > > > > [1] https://www.framos.com/media/pdf/96/ac/8f/AR0237CS-D-PDF-framos.pdf > > > > > > > > > > > > > > I think we're reaching a limit of the media bus codes model here, due to > > > > > > > a historical mistake. The four possible Bayer patterns, times the > > > > > > > different number of bits per pixel, creates a lot of media bus codes, > > > > > > > and drivers for CSI-2 receivers and IP cores further down the pipeline > > > > > > > have to support them all. > > > > > > > > > > > > That's correct but it is not bayer related. Currently it is what it is, > > > > > > if a new code is added it must be propagated through all the involved > > > > > > subdevs. On the other hand I wouldn't say that it is better to support > > > > > > new codes per default for all drivers. Since this would add a lot of > > > > > > untested code paths. > > > > > > > > > > It's not an issue limited to Bayer patterns, but they make the issue > > > > > worse given the number of possible combinations (think about adding DPCM > > > > > and ALAW compression on top of that...). > > > > > > > > You're right and again this will apply to all mbus formats... > > > > > > > > > > > This is already painful, and if we had a > > > > > > > non-Bayer pattern such as this one, > > > > > > > > > > > > That's not exactly true since it is a bayer pattern but instead of using > > > > > > 4x4 it uses 8x8 and it as some special pixels. > > > > > > > > > > > > > we'll open the door to an explosion > > > > > > > of the number of media bus codes (imagine all the different possible > > > > > > > arrangements of this pattern, for instance if you enable horizontal > > > > > > > and/or vertical flipping on the sensor). All drivers would need to be > > > > > > > updated to support these new bus codes, and this really kills the > > > > > > > current model. > > > > > > > > > > > > Yep, I know what you mean but as I said above I think that adding it > > > > > > explicite is the better abbroach since it involves somone who add _and_ > > > > > > test the new code on the particular platform. > > > > > > > > > > > > > The historical mistake was to tie the Bayer pattern with the media bus > > > > > > > code. We should really have specified raw 8/10/12/14/16 media bus codes, > > > > > > > and conveyed the pattern separately. Most IP cores in the pipeline don't > > > > > > > need to care about the pattern at all, and those who do (that's mostly > > > > > > > ISPs) could be programmed explicitly by userspace as long as we have an > > > > > > > API to retrieve the pattern from the sensor. I believe it's time to bite > > > > > > > the bullet and go in that direction. I'm sorry for this case of yak > > > > > > > shaving, but it really wouldn't be manageable anymore :-( > > > > > > > > > > > > I got all your points and would agree but this is not a bayer only > > > > > > related problem. You will have this problem with all new other formats > > > > > > as well. I'm with you, most IP cores don't care but I wouldn't > > > > > > guarantee that. > > > > > > > > > > Sorry, but adding new media bus formats like this one will just not > > > > > scale. We have two options, either fixing the issue, or considering that > > > > > V4L2 is a barely alive API with no future, and merging this without > > > > > caring anymore. > > > > > > > > Hm.. you're right that it doesn't scale, as I said I'm absolute on your > > > > side. So let us consider a new approach @Mauro, @Hans, @Sailus what do > > > > you think about? > > > > > > Starting brainstorming, how about new media bus codes for > > > raw{8,10,12,14,16}, > > > > By "raw", are you meaning vendor-specific formats? If so, that sounds > > a bad idea. Different vendor-specific formats should use different > > media bus codes (and fourccs) as otherwise there won't be an easy way > > to distinguish them and to describe the raw formats at the media specs. > > I mean what the CSI-2 spec means. The exact interpretation of the format > will be a combination of the media bus code and the CFA pattern control. > The whole point of this discussion is to not have different media bus > codes for all possible combinations of formats, as that clearly doesn't > scale. You mean that we should propagate the value as raw{size} through the entire pipeline, if I got this correctly. How the picture should be interpreted is up to the user-space by calling a new read-only CFA v4l2-control. This way we don't need to patch each subdev driver and take the user-space into account of interpreting the data. For the CFA control we could use a global-unique list so the control returns an enum. Regards, Marco From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9203EC05027 for ; Wed, 8 Feb 2023 19:45:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ksfEOexemRP0tKLifyYVz631UcQFt3rw80qkHCwnTFc=; b=bKl3LBDUcJIgb4 C9Og5leS158Nx6BwDcRVQRlVPSvYdsXByeDEwZ+aAi2Q9P56v7s+CGIE8ZyDSmP8c4+60blrzlt24 aWmrgu+tylndTYuWiv45gu01fing24w0fHoqJE2YqrgqpD2GWdJeaIyU2UuqdZAelHzx90CyHvk4Y t2TqAUy3Cbx8Mh5VpiiOE0VQJddnMrgfGZW6Nls8Pef+v2q5aZ5EqXk8WK/OdM9BwRrY9477aVOs7 b0ltbn4goxf9H6RjnB686ZvQ4x5va4voZy6H4QN1fvKzAVZ7LAQjBK1mk+Ty14gyNHRA7KOW0mCGf IxYad+dkp5/fuD1QtS8w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pPqN1-00GmRz-I5; Wed, 08 Feb 2023 19:44:43 +0000 Received: from metis.ext.pengutronix.de ([2001:67c:670:201:290:27ff:fe1d:cc33]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pPqMx-00GmQo-Tz for linux-arm-kernel@lists.infradead.org; Wed, 08 Feb 2023 19:44:41 +0000 Received: from ptx.hi.pengutronix.de ([2001:67c:670:100:1d::c0]) by metis.ext.pengutronix.de with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pPqMk-0000MT-Df; Wed, 08 Feb 2023 20:44:26 +0100 Received: from mfe by ptx.hi.pengutronix.de with local (Exim 4.92) (envelope-from ) id 1pPqMj-0003FP-6D; Wed, 08 Feb 2023 20:44:25 +0100 Date: Wed, 8 Feb 2023 20:44:25 +0100 From: Marco Felsch To: Laurent Pinchart Cc: Mauro Carvalho Chehab , p.zabel@pengutronix.de, slongerbeam@gmail.com, hverkuil-cisco@xs4all.nl, sakari.ailus@linux.intel.com, linux-arm-kernel@lists.infradead.org, linux-media@vger.kernel.org, kernel@pengutronix.de Subject: Re: [PATCH 1/6] media: uapi: Add MEDIA_BUS_FMT_SGRGB_IGIG_GBGR_IGIG media bus formats Message-ID: <20230208194425.anis36f6jlzmsjwp@pengutronix.de> References: <20210427120701.21809-1-m.felsch@pengutronix.de> <20210427120701.21809-2-m.felsch@pengutronix.de> <20210429074903.cc5gohn52cgv4i5z@pengutronix.de> <20210430065134.jwscxlv3sydo4zgy@pengutronix.de> <20210430145146.359a9b90@coco.lan> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20180716 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::c0 X-SA-Exim-Mail-From: mfe@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-arm-kernel@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230208_114439_975366_91A9B30C X-CRM114-Status: GOOD ( 56.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi all, sorry for the long absence on this topic. Only a few years later I'm back on this topic :) On 21-04-30, Laurent Pinchart wrote: > On Fri, Apr 30, 2021 at 02:51:46PM +0200, Mauro Carvalho Chehab wrote: > > Em Fri, 30 Apr 2021 15:18:52 +0300 > > Laurent Pinchart escreveu: > > > > > Hi Marco, > > > > > > On Fri, Apr 30, 2021 at 08:51:34AM +0200, Marco Felsch wrote: > > > > On 21-04-30 01:14, Laurent Pinchart wrote: > > > > > On Thu, Apr 29, 2021 at 09:49:03AM +0200, Marco Felsch wrote: > > > > > > On 21-04-29 04:51, Laurent Pinchart wrote: > > > > > > > On Tue, Apr 27, 2021 at 02:06:56PM +0200, Marco Felsch wrote: > > > > > > > > Add special 8/12bit bayer media bus format for the OnSemi AR0237IR > > > > > > > > camera sensor [1]. OnSemi calls this format RGB-IR, the pixel array > > > > > > > > with the interleaved IR pixels looks like: > > > > > > > > > > > > > > > > | G | R | G | B | ... > > > > > > > > +----+----+----+----+--- > > > > > > > > | IR | G | IR | G | ... > > > > > > > > +----+----+----+----+--- > > > > > > > > | G | B | G | R | ... > > > > > > > > +----+----+----+----+--- > > > > > > > > | IR | G | IR | G | ... > > > > > > > > +----+----+----+----+--- > > > > > > > > | .. | .. | .. | .. | .. > > > > > > > > > > > > > > > > [1] https://www.framos.com/media/pdf/96/ac/8f/AR0237CS-D-PDF-framos.pdf > > > > > > > > > > > > > > I think we're reaching a limit of the media bus codes model here, due to > > > > > > > a historical mistake. The four possible Bayer patterns, times the > > > > > > > different number of bits per pixel, creates a lot of media bus codes, > > > > > > > and drivers for CSI-2 receivers and IP cores further down the pipeline > > > > > > > have to support them all. > > > > > > > > > > > > That's correct but it is not bayer related. Currently it is what it is, > > > > > > if a new code is added it must be propagated through all the involved > > > > > > subdevs. On the other hand I wouldn't say that it is better to support > > > > > > new codes per default for all drivers. Since this would add a lot of > > > > > > untested code paths. > > > > > > > > > > It's not an issue limited to Bayer patterns, but they make the issue > > > > > worse given the number of possible combinations (think about adding DPCM > > > > > and ALAW compression on top of that...). > > > > > > > > You're right and again this will apply to all mbus formats... > > > > > > > > > > > This is already painful, and if we had a > > > > > > > non-Bayer pattern such as this one, > > > > > > > > > > > > That's not exactly true since it is a bayer pattern but instead of using > > > > > > 4x4 it uses 8x8 and it as some special pixels. > > > > > > > > > > > > > we'll open the door to an explosion > > > > > > > of the number of media bus codes (imagine all the different possible > > > > > > > arrangements of this pattern, for instance if you enable horizontal > > > > > > > and/or vertical flipping on the sensor). All drivers would need to be > > > > > > > updated to support these new bus codes, and this really kills the > > > > > > > current model. > > > > > > > > > > > > Yep, I know what you mean but as I said above I think that adding it > > > > > > explicite is the better abbroach since it involves somone who add _and_ > > > > > > test the new code on the particular platform. > > > > > > > > > > > > > The historical mistake was to tie the Bayer pattern with the media bus > > > > > > > code. We should really have specified raw 8/10/12/14/16 media bus codes, > > > > > > > and conveyed the pattern separately. Most IP cores in the pipeline don't > > > > > > > need to care about the pattern at all, and those who do (that's mostly > > > > > > > ISPs) could be programmed explicitly by userspace as long as we have an > > > > > > > API to retrieve the pattern from the sensor. I believe it's time to bite > > > > > > > the bullet and go in that direction. I'm sorry for this case of yak > > > > > > > shaving, but it really wouldn't be manageable anymore :-( > > > > > > > > > > > > I got all your points and would agree but this is not a bayer only > > > > > > related problem. You will have this problem with all new other formats > > > > > > as well. I'm with you, most IP cores don't care but I wouldn't > > > > > > guarantee that. > > > > > > > > > > Sorry, but adding new media bus formats like this one will just not > > > > > scale. We have two options, either fixing the issue, or considering that > > > > > V4L2 is a barely alive API with no future, and merging this without > > > > > caring anymore. > > > > > > > > Hm.. you're right that it doesn't scale, as I said I'm absolute on your > > > > side. So let us consider a new approach @Mauro, @Hans, @Sailus what do > > > > you think about? > > > > > > Starting brainstorming, how about new media bus codes for > > > raw{8,10,12,14,16}, > > > > By "raw", are you meaning vendor-specific formats? If so, that sounds > > a bad idea. Different vendor-specific formats should use different > > media bus codes (and fourccs) as otherwise there won't be an easy way > > to distinguish them and to describe the raw formats at the media specs. > > I mean what the CSI-2 spec means. The exact interpretation of the format > will be a combination of the media bus code and the CFA pattern control. > The whole point of this discussion is to not have different media bus > codes for all possible combinations of formats, as that clearly doesn't > scale. You mean that we should propagate the value as raw{size} through the entire pipeline, if I got this correctly. How the picture should be interpreted is up to the user-space by calling a new read-only CFA v4l2-control. This way we don't need to patch each subdev driver and take the user-space into account of interpreting the data. For the CFA control we could use a global-unique list so the control returns an enum. Regards, Marco _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel