From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D204C04A6B for ; Sun, 12 May 2019 11:36:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 68F8D20C01 for ; Sun, 12 May 2019 11:36:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726664AbfELLgN (ORCPT ); Sun, 12 May 2019 07:36:13 -0400 Received: from relay6-d.mail.gandi.net ([217.70.183.198]:53147 "EHLO relay6-d.mail.gandi.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726307AbfELLgM (ORCPT ); Sun, 12 May 2019 07:36:12 -0400 X-Originating-IP: 93.29.109.196 Received: from collins (196.109.29.93.rev.sfr.net [93.29.109.196]) (Authenticated sender: paul.kocialkowski@bootlin.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 73671C0004; Sun, 12 May 2019 11:36:08 +0000 (UTC) Message-ID: <19804f842e80f1f8bd85b8eb49e75e149abf0062.camel@bootlin.com> Subject: Hardware-accelerated video decoders used through a firmware instead of hardware registers From: Paul Kocialkowski To: Hans Verkuil , Sakari Ailus , Mauro Carvalho Chehab Cc: Laurent Pinchart , Maxime Ripard , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, Linus Torvalds , Thierry Reding , Tiffany Lin , Andrew-CT Chen Date: Sun, 12 May 2019 13:35:55 +0200 Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.32.1 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, With the work done on the media request API and the cedrus driver for Allwinner ARM SoCs, we now have a kernel interface for exposing fixed- hardware video decoding pipelines (currently MPEG-2 and H.264, with H.265 on the way). Some work remains on the per-format interface and we are looking to improve latency-related aspects, but we are all set to have a nice interface here, that plays well with e.g. ffmpeg. A specific situation came to my interest, which is apparently quite common: some platforms have general-purpose microcontrollers embedded, which can help with video decoding. They are however rarely to never used to do the decoding itself (since they are general-purpose, not DSPs) and just coordinate the decoding with the fixed-pipeline decoding hardware block. The advantage is that the interface is just a simple mailbox and the raw video bitstream from the file can be passed directly without the need for userspace to do any parsing that the codec requires. One side-effect from this setup is that the actual hardware register layout of the decoder is hidden away in a non-free piece of microcontroller firmware, that's usually loaded at run-time. With the recent developments on the media interface, we could interface with these hardware decoders directly, which offers various advantages: - we no longer need a 3rd party external non-free firmware, which just makes distribution easier for everyone and allows support in fully- free setups; - all the usual advantages of having free code that can be fixed and updated instead of an obscure binary that many not always be doing the right thing; - parsing of the slices is probably best done in userspace, and I heard that ffmpeg does this threaded, so there could be a latency advantage there as well, not to mention that it avoids the drag of a mailbox interface altogether; - the general-purpose micro-controller can then be reused for something useful where it could actually make a performance difference. As far as I understand, it seems that the video decoder for MT8173 fails in that category, where a MD32 general-purpose micro-controller is used to only do the parsing. We even have device-tree nodes about the decoder and encoder, but no register layout. So I was wondering if the linux-media community should set some boundaries here and push towards native implementations instead of firmware-based ones. My opininon is that it definitely should. It seems that other platforms (e.g. Tegra K1 and onwards) are in the same situation, and I think the ChromiumOS downstream kernel uses an obscure firmware on a general-purpose auxiliary ARM core (that's also used at boot time IIRC). What do you think? Cheers, Paul -- Paul Kocialkowski, Bootlin Embedded Linux and kernel engineering https://bootlin.com