[PATCH v4 0/2] Enable JPEG encoding on rk3588

[PATCH v4 0/2] Enable JPEG encoding on rk3588

[Emmanuel Gil Peyrot]

Only the JPEG encoder is available for now, although there are patches
for the undocumented VP8 encoder floating around[0].

This has been tested on a rock-5b, resulting in four /dev/video*
encoders.  The userspace program I’ve been using to test them is
Onix[1], using the jpeg-encoder example, it will pick one of these four
at random (but displays the one it picked):
% ffmpeg -i <input image> -pix_fmt yuvj420p temp.yuv
% jpeg-encoder temp.yuv <width> <height> NV12 <quality> output.jpeg

[0] https://patchwork.kernel.org/project/linux-rockchip/list/?series=789885
[1] https://crates.io/crates/onix

Changes since v3:
- Keep all the previous tags, sorry I forgot to do that in the previous
  version.
- Remove an extra media: in the first patch.

Changes since v2:
- Only expose a single VEPU121, since the driver doesn’t yet support
  exposing them all as a single video node to userspace.

Changes since v1:
- Dropped patches 1 and 4.
- Use the proper compatible form, since this device should be fully
  compatible with the VEPU of rk356x.
- Describe where the VEPU121 name comes from, and list other encoders
  and decoders present in this SoC.
- Properly test the device tree changes, I previously couldn’t since I
  was using a too recent version of python-jsonschema…

Emmanuel Gil Peyrot (2):
  media: dt-binding: Document rk3588’s VEPU121
  arm64: dts: rockchip: Add one VEPU121 to rk3588

 .../bindings/media/rockchip,rk3568-vepu.yaml  |  8 +++--
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi     | 29 +++++++++++++++++++
 2 files changed, 35 insertions(+), 2 deletions(-)

-- 
2.44.0

[PATCH v4 1/2] media: dt-binding: Document rk3588’s VEPU121

[Emmanuel Gil Peyrot]

This encoder-only device is present four times on this SoC, and should
support everything the rk3568 vepu supports (so JPEG, H.264 and VP8
encoding).

According to the TRM[1], there is also the VEPU580 encoder which
supports H.264 and H.265, and various VDPU* decoders, of which only the
VDPU981 is currently supported.  This patch describes only the VEPU121.

[1] https://github.com/FanX-Tek/rk3588-TRM-and-Datasheet

Signed-off-by: Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
Reviewed-by: Sebastian Reichel <sebastian.reichel@collabora.com>
Acked-by: Conor Dooley <conor.dooley@microchip.com>
---
 .../devicetree/bindings/media/rockchip,rk3568-vepu.yaml   | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
index 9d90d8d0565a..4c6cb21da041 100644
--- a/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
+++ b/Documentation/devicetree/bindings/media/rockchip,rk3568-vepu.yaml
@@ -15,8 +15,12 @@ description:
 
 properties:
   compatible:
-    enum:
-      - rockchip,rk3568-vepu
+    oneOf:
+      - const: rockchip,rk3568-vepu
+      - items:
+          - enum:
+              - rockchip,rk3588-vepu121
+          - const: rockchip,rk3568-vepu
 
   reg:
     maxItems: 1
-- 
2.44.0

[PATCH v4 2/2] arm64: dts: rockchip: Add one VEPU121 to rk3588

[Emmanuel Gil Peyrot]

The TRM (version 1.0 page 385) lists five VEPU121 cores, but only four
interrupts are listed (on page 24), and the driver would expose them all
as different video nodes so only one of them is exposed for now.

Signed-off-by: Emmanuel Gil Peyrot <linkmauve@linkmauve.fr>
Formerly-Reviewed-by: Sebastian Reichel <sebastian.reichel@collabora.com>
---
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 29 +++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
index 87b83c87bd55..5c6cc4cd81df 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
@@ -2488,6 +2488,35 @@ gpio4: gpio@fec50000 {
 		};
 	};
 
+	/*
+	 * Currently only one of the four JPEG encoders is exposed, this
+	 * reduces the optimal throughput by four.
+	 *
+	 * Once the driver is made to expose all four devices as a single video
+	 * node, the rest can be enabled again, see the full patch here:
+	 * https://lore.kernel.org/linux-media/20240327134115.424846-1-linkmauve@linkmauve.fr/T/#m3a6df0ba15e4af40b998b0ff2a02b0dd0d730c8e
+	 */
+
+	jpeg_enc0: video-codec@fdba0000 {
+		compatible = "rockchip,rk3588-vepu121", "rockchip,rk3568-vepu";
+		reg = <0x0 0xfdba0000 0x0 0x800>;
+		interrupts = <GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
+		clock-names = "aclk", "hclk";
+		iommus = <&jpeg_enc0_mmu>;
+		power-domains = <&power RK3588_PD_VDPU>;
+	};
+
+	jpeg_enc0_mmu: iommu@fdba0800 {
+		compatible = "rockchip,rk3588-iommu", "rockchip,rk3568-iommu";
+		reg = <0x0 0xfdba0800 0x0 0x40>;
+		interrupts = <GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH 0>;
+		clocks = <&cru ACLK_JPEG_ENCODER0>, <&cru HCLK_JPEG_ENCODER0>;
+		clock-names = "aclk", "iface";
+		power-domains = <&power RK3588_PD_VDPU>;
+		#iommu-cells = <0>;
+	};
+
 	av1d: video-codec@fdc70000 {
 		compatible = "rockchip,rk3588-av1-vpu";
 		reg = <0x0 0xfdc70000 0x0 0x800>;
-- 
2.44.0

Re: [PATCH v4 0/2] Enable JPEG encoding on rk3588

[Jianfeng Liu]

Hi Emmanuel,

On Thu, 18 Apr 2024 16:15:04 +0200, Emmanuel Gil Peyrot wrote:
>- Only expose a single VEPU121, since the driver doesn’t yet support
>  exposing them all as a single video node to userspace.
I'm also doing work with the hantro node on rk3588 which has both vdpu
and vepu. Discussions can be found here[1]. If I enable the jpeg encoder
at feb50000, would it cause any side effects if other jpeg encoders in
your patch are enabled? And what's the disadvantage of enabling multi
jpeg endoers in devicetree?

[1] https://patchwork.kernel.org/project/linux-rockchip/patch/20240418111002.83015-3-liujianfeng1994@gmail.com/

Best regards,
Jianfeng

Re: [PATCH v4 0/2] Enable JPEG encoding on rk3588

[Heiko Stübner]

Hi Jianfeng,

Am Samstag, 27. April 2024, 09:16:38 CEST schrieb Jianfeng Liu:
> Hi Emmanuel,
> 
> On Thu, 18 Apr 2024 16:15:04 +0200, Emmanuel Gil Peyrot wrote:
> >- Only expose a single VEPU121, since the driver doesn’t yet support
> >  exposing them all as a single video node to userspace.
> I'm also doing work with the hantro node on rk3588 which has both vdpu
> and vepu. Discussions can be found here[1]. If I enable the jpeg encoder
> at feb50000, would it cause any side effects if other jpeg encoders in
> your patch are enabled? And what's the disadvantage of enabling multi
> jpeg endoers in devicetree?

the basic problem is that exposing multiple jpeg encoders would require
userspace to do the scheduling. Which is bad.
I.e. all userspace programms would need to know the existence of
all jpeg encoders and then somehow negotiate how to use all of them
most efficiently.

Think multiple different programs that would need to negotiate to
spread across all of them in the best way.

Doing this in the kernel, we just expose one encoder (and queue) all
programs would just pile onto that one encoder and the kernel then
would be on the hook to do the scheduling - which the kernel can do
better (with relevant code implemented)

Re: [PATCH v4 0/2] Enable JPEG encoding on rk3588

[Jianfeng Liu]

Hi Heiko,

On Sun, 28 Apr 2024 19:01:38 +0200, Heiko Stübner wrote:
>the basic problem is that exposing multiple jpeg encoders would require
>userspace to do the scheduling. Which is bad.
>I.e. all userspace programms would need to know the existence of
>all jpeg encoders and then somehow negotiate how to use all of them
>most efficiently.
>
>Think multiple different programs that would need to negotiate to
>spread across all of them in the best way.
>
>Doing this in the kernel, we just expose one encoder (and queue) all
>programs would just pile onto that one encoder and the kernel then
>would be on the hook to do the scheduling - which the kernel can do
>better (with relevant code implemented)

Yeah let kernel do the scheduling is indeed better. And I'm happy to
hear this method.

So I will keep the vpu at feb50000 with jpeg endoder disabled until
multi encoder scheduling is implemented.

Best regards,
Jianfeng