From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 875B8C433ED for ; Fri, 30 Apr 2021 06:51:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 69C0F6141C for ; Fri, 30 Apr 2021 06:51:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229741AbhD3Gwd (ORCPT ); Fri, 30 Apr 2021 02:52:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58734 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229508AbhD3Gwd (ORCPT ); Fri, 30 Apr 2021 02:52:33 -0400 Received: from metis.ext.pengutronix.de (metis.ext.pengutronix.de [IPv6:2001:67c:670:201:290:27ff:fe1d:cc33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59BEBC06174A for ; Thu, 29 Apr 2021 23:51:45 -0700 (PDT) Received: from pty.hi.pengutronix.de ([2001:67c:670:100:1d::c5]) by metis.ext.pengutronix.de with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lcMzv-0006bL-LK; Fri, 30 Apr 2021 08:51:35 +0200 Received: from mfe by pty.hi.pengutronix.de with local (Exim 4.89) (envelope-from ) id 1lcMzv-0004Km-0C; Fri, 30 Apr 2021 08:51:35 +0200 Date: Fri, 30 Apr 2021 08:51:34 +0200 From: Marco Felsch To: Laurent Pinchart Cc: p.zabel@pengutronix.de, mchehab@kernel.org, slongerbeam@gmail.com, hverkuil-cisco@xs4all.nl, sakari.ailus@linux.intel.com, linux-arm-kernel@lists.infradead.org, linux-media@vger.kernel.org, kernel@pengutronix.de Subject: Re: [PATCH 1/6] media: uapi: Add MEDIA_BUS_FMT_SGRGB_IGIG_GBGR_IGIG media bus formats Message-ID: <20210430065134.jwscxlv3sydo4zgy@pengutronix.de> References: <20210427120701.21809-1-m.felsch@pengutronix.de> <20210427120701.21809-2-m.felsch@pengutronix.de> <20210429074903.cc5gohn52cgv4i5z@pengutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Sent-From: Pengutronix Hildesheim X-URL: http://www.pengutronix.de/ X-IRC: #ptxdist @freenode X-Accept-Language: de,en X-Accept-Content-Type: text/plain X-Uptime: 08:41:46 up 148 days, 20:48, 43 users, load average: 0.21, 0.24, 0.10 User-Agent: NeoMutt/20170113 (1.7.2) X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::c5 X-SA-Exim-Mail-From: mfe@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-media@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Hi Laurent, On 21-04-30 01:14, Laurent Pinchart wrote: > Hi Marco, > > On Thu, Apr 29, 2021 at 09:49:03AM +0200, Marco Felsch wrote: > > On 21-04-29 04:51, Laurent Pinchart wrote: > > > On Tue, Apr 27, 2021 at 02:06:56PM +0200, Marco Felsch wrote: > > > > Add special 8/12bit bayer media bus format for the OnSemi AR0237IR > > > > camera sensor [1]. OnSemi calls this format RGB-IR, the pixel array > > > > with the interleaved IR pixels looks like: > > > > > > > > | G | R | G | B | ... > > > > +----+----+----+----+--- > > > > | IR | G | IR | G | ... > > > > +----+----+----+----+--- > > > > | G | B | G | R | ... > > > > +----+----+----+----+--- > > > > | IR | G | IR | G | ... > > > > +----+----+----+----+--- > > > > | .. | .. | .. | .. | .. > > > > > > > > [1] https://www.framos.com/media/pdf/96/ac/8f/AR0237CS-D-PDF-framos.pdf > > > > > > I think we're reaching a limit of the media bus codes model here, due to > > > a historical mistake. The four possible Bayer patterns, times the > > > different number of bits per pixel, creates a lot of media bus codes, > > > and drivers for CSI-2 receivers and IP cores further down the pipeline > > > have to support them all. > > > > That's correct but it is not bayer related. Currently it is what it is, > > if a new code is added it must be propagated through all the involved > > subdevs. On the other hand I wouldn't say that it is better to support > > new codes per default for all drivers. Since this would add a lot of > > untested code paths. > > It's not an issue limited to Bayer patterns, but they make the issue > worse given the number of possible combinations (think about adding DPCM > and ALAW compression on top of that...). You're right and again this will apply to all mbus formats... > > > This is already painful, and if we had a > > > non-Bayer pattern such as this one, > > > > That's not exactly true since it is a bayer pattern but instead of using > > 4x4 it uses 8x8 and it as some special pixels. > > > > > we'll open the door to an explosion > > > of the number of media bus codes (imagine all the different possible > > > arrangements of this pattern, for instance if you enable horizontal > > > and/or vertical flipping on the sensor). All drivers would need to be > > > updated to support these new bus codes, and this really kills the > > > current model. > > > > Yep, I know what you mean but as I said above I think that adding it > > explicite is the better abbroach since it involves somone who add _and_ > > test the new code on the particular platform. > > > > > The historical mistake was to tie the Bayer pattern with the media bus > > > code. We should really have specified raw 8/10/12/14/16 media bus codes, > > > and conveyed the pattern separately. Most IP cores in the pipeline don't > > > need to care about the pattern at all, and those who do (that's mostly > > > ISPs) could be programmed explicitly by userspace as long as we have an > > > API to retrieve the pattern from the sensor. I believe it's time to bite > > > the bullet and go in that direction. I'm sorry for this case of yak > > > shaving, but it really wouldn't be manageable anymore :-( > > > > I got all your points and would agree but this is not a bayer only > > related problem. You will have this problem with all new other formats > > as well. I'm with you, most IP cores don't care but I wouldn't > > guarantee that. > > Sorry, but adding new media bus formats like this one will just not > scale. We have two options, either fixing the issue, or considering that > V4L2 is a barely alive API with no future, and merging this without > caring anymore. Hm.. you're right that it doesn't scale, as I said I'm absolute on your side. So let us consider a new approach @Mauro, @Hans, @Sailus what do you think about? BTW: IMHO videobuf2 interface isn't that good as well, since you are blaming ;) Regards, Marco > > -- > Regards, > > Laurent Pinchart > -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E901C433B4 for ; Fri, 30 Apr 2021 06:53:33 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AE55E613E8 for ; Fri, 30 Apr 2021 06:53:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AE55E613E8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=pengutronix.de Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=JMZO2sRScXLO7mTW7kGUkbaOWMXFno/wiiMI4gd9W7w=; b=qyY5Moqj2TSfjFJkz1bY0Axcq SE1idFYtDymF4O+LH4RGY1E3cBWjU0yHF7Suki480e+yRUONxMsstOKzaG5p9nRWR4WzolquRums3 ezPZMU0rLQ3z4N29M1P/YzK6inx9g3PZM9p4x7tOJZFtbqjLpMtrkZoSfIKtrGtK3q/ORK85guBie QseODgV5+HAEEUKQRDmX2TLgcaiatzlytd/1ENnG4bcoFAOoFcKOQlly9PIaI9SKcXic7/rbUahOs MW+9kZbO+FbkkwzrZqOZVniPFW/5eTNLoAicQzjaKhnttZYpL8Mj556Oklq3bO1YIZyDbgJkJqzS8 okWt1bEig==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lcN0B-007Nfv-T3; Fri, 30 Apr 2021 06:51:52 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lcN07-007NfI-TX for linux-arm-kernel@desiato.infradead.org; Fri, 30 Apr 2021 06:51:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=In-Reply-To:Content-Type:MIME-Version :References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=Cc94XravCg4PsxGA09KuPzP43z9P86sc436YJISI2BM=; b=0rmAer8q2GIWNCMuaN5XR+QzQB TLkpmK3DV+QZdsVoXO0pLNCRFbRc024RhhM93orfKFmT2U4j9VggxIVP8gblx/NPlupzC/51JJ8wT Aqk5+CQLWjAfBnE/nSEkL0VgDXV2rnb3DfvYr8h4PVhyGyqKSHpEFV9umE3ezCNhqtKLB/aUG95Ee CK9UUOSU57CCIT1Z5gNRRdE4ISOXeyuvp+awaP7C+Vpg+eDKy5BM4Qd4/JCSnFM0ioFKYtPRONkTG HNiFfime1pS/miD+TMQgfzGTSyPJp7DOoVeFoiNxEbCNxy9TTaY4+aCpvS5YZ1F9+q8mUZ3G9VR3/ hH1NPGNg==; Received: from metis.ext.pengutronix.de ([2001:67c:670:201:290:27ff:fe1d:cc33]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lcN05-001BJE-2w for linux-arm-kernel@lists.infradead.org; Fri, 30 Apr 2021 06:51:46 +0000 Received: from pty.hi.pengutronix.de ([2001:67c:670:100:1d::c5]) by metis.ext.pengutronix.de with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lcMzv-0006bL-LK; Fri, 30 Apr 2021 08:51:35 +0200 Received: from mfe by pty.hi.pengutronix.de with local (Exim 4.89) (envelope-from ) id 1lcMzv-0004Km-0C; Fri, 30 Apr 2021 08:51:35 +0200 Date: Fri, 30 Apr 2021 08:51:34 +0200 From: Marco Felsch To: Laurent Pinchart Cc: p.zabel@pengutronix.de, mchehab@kernel.org, slongerbeam@gmail.com, hverkuil-cisco@xs4all.nl, sakari.ailus@linux.intel.com, linux-arm-kernel@lists.infradead.org, linux-media@vger.kernel.org, kernel@pengutronix.de Subject: Re: [PATCH 1/6] media: uapi: Add MEDIA_BUS_FMT_SGRGB_IGIG_GBGR_IGIG media bus formats Message-ID: <20210430065134.jwscxlv3sydo4zgy@pengutronix.de> References: <20210427120701.21809-1-m.felsch@pengutronix.de> <20210427120701.21809-2-m.felsch@pengutronix.de> <20210429074903.cc5gohn52cgv4i5z@pengutronix.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Sent-From: Pengutronix Hildesheim X-URL: http://www.pengutronix.de/ X-IRC: #ptxdist @freenode X-Accept-Language: de,en X-Accept-Content-Type: text/plain X-Uptime: 08:41:46 up 148 days, 20:48, 43 users, load average: 0.21, 0.24, 0.10 User-Agent: NeoMutt/20170113 (1.7.2) X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::c5 X-SA-Exim-Mail-From: mfe@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-arm-kernel@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210429_235145_139595_99F9B12C X-CRM114-Status: GOOD ( 50.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Laurent, On 21-04-30 01:14, Laurent Pinchart wrote: > Hi Marco, > > On Thu, Apr 29, 2021 at 09:49:03AM +0200, Marco Felsch wrote: > > On 21-04-29 04:51, Laurent Pinchart wrote: > > > On Tue, Apr 27, 2021 at 02:06:56PM +0200, Marco Felsch wrote: > > > > Add special 8/12bit bayer media bus format for the OnSemi AR0237IR > > > > camera sensor [1]. OnSemi calls this format RGB-IR, the pixel array > > > > with the interleaved IR pixels looks like: > > > > > > > > | G | R | G | B | ... > > > > +----+----+----+----+--- > > > > | IR | G | IR | G | ... > > > > +----+----+----+----+--- > > > > | G | B | G | R | ... > > > > +----+----+----+----+--- > > > > | IR | G | IR | G | ... > > > > +----+----+----+----+--- > > > > | .. | .. | .. | .. | .. > > > > > > > > [1] https://www.framos.com/media/pdf/96/ac/8f/AR0237CS-D-PDF-framos.pdf > > > > > > I think we're reaching a limit of the media bus codes model here, due to > > > a historical mistake. The four possible Bayer patterns, times the > > > different number of bits per pixel, creates a lot of media bus codes, > > > and drivers for CSI-2 receivers and IP cores further down the pipeline > > > have to support them all. > > > > That's correct but it is not bayer related. Currently it is what it is, > > if a new code is added it must be propagated through all the involved > > subdevs. On the other hand I wouldn't say that it is better to support > > new codes per default for all drivers. Since this would add a lot of > > untested code paths. > > It's not an issue limited to Bayer patterns, but they make the issue > worse given the number of possible combinations (think about adding DPCM > and ALAW compression on top of that...). You're right and again this will apply to all mbus formats... > > > This is already painful, and if we had a > > > non-Bayer pattern such as this one, > > > > That's not exactly true since it is a bayer pattern but instead of using > > 4x4 it uses 8x8 and it as some special pixels. > > > > > we'll open the door to an explosion > > > of the number of media bus codes (imagine all the different possible > > > arrangements of this pattern, for instance if you enable horizontal > > > and/or vertical flipping on the sensor). All drivers would need to be > > > updated to support these new bus codes, and this really kills the > > > current model. > > > > Yep, I know what you mean but as I said above I think that adding it > > explicite is the better abbroach since it involves somone who add _and_ > > test the new code on the particular platform. > > > > > The historical mistake was to tie the Bayer pattern with the media bus > > > code. We should really have specified raw 8/10/12/14/16 media bus codes, > > > and conveyed the pattern separately. Most IP cores in the pipeline don't > > > need to care about the pattern at all, and those who do (that's mostly > > > ISPs) could be programmed explicitly by userspace as long as we have an > > > API to retrieve the pattern from the sensor. I believe it's time to bite > > > the bullet and go in that direction. I'm sorry for this case of yak > > > shaving, but it really wouldn't be manageable anymore :-( > > > > I got all your points and would agree but this is not a bayer only > > related problem. You will have this problem with all new other formats > > as well. I'm with you, most IP cores don't care but I wouldn't > > guarantee that. > > Sorry, but adding new media bus formats like this one will just not > scale. We have two options, either fixing the issue, or considering that > V4L2 is a barely alive API with no future, and merging this without > caring anymore. Hm.. you're right that it doesn't scale, as I said I'm absolute on your side. So let us consider a new approach @Mauro, @Hans, @Sailus what do you think about? BTW: IMHO videobuf2 interface isn't that good as well, since you are blaming ;) Regards, Marco > > -- > Regards, > > Laurent Pinchart > -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel