* [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 @ 2022-04-27 22:44 Nicolas Frattaroli 2022-04-27 22:44 ` [PATCH 1/3] dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible Nicolas Frattaroli ` (3 more replies) 0 siblings, 4 replies; 13+ messages in thread From: Nicolas Frattaroli @ 2022-04-27 22:44 UTC (permalink / raw) To: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski, Heiko Stuebner, Greg Kroah-Hartman, Peter Geis, Michael Riesch, Liang Chen Cc: Nicolas Frattaroli, linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel, linux-staging Hello, the following series adds support for and enables the hardware JPEG encoder on the RK3566 and RK3568 line of SoCs by Rockchip. The JPEG encoder is its own little Hantro instance with seemingly just the encode functionality. The first patch modifies the bindings with a new compatible, and adds the ability to just have a vepu interrupt without a vdpu interrupt. The second patch makes the actual driver changes to support this variant. The third and final patch makes the necessary device tree changes for the rk356x device tree file to add both the node for the encoder and its MMU. The series has been tested on a PINE64 Quartz64 Model A with an RK3566 SoC using GStreamer. Regards, Nicolas Frattaroli Nicolas Frattaroli (3): dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible media: hantro: Add support for RK356x JPEG encoder arm64: dts: rockchip: Add JPEG encoder node to rk356x .../bindings/media/rockchip-vpu.yaml | 2 + arch/arm64/boot/dts/rockchip/rk356x.dtsi | 21 +++++++ drivers/staging/media/hantro/hantro_drv.c | 1 + drivers/staging/media/hantro/hantro_hw.h | 1 + .../staging/media/hantro/rockchip_vpu_hw.c | 62 +++++++++++++++++++ 5 files changed, 87 insertions(+) -- 2.36.0 ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 1/3] dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible 2022-04-27 22:44 [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Frattaroli @ 2022-04-27 22:44 ` Nicolas Frattaroli 2022-05-04 20:23 ` Rob Herring 2022-04-27 22:44 ` [PATCH 2/3] media: hantro: Add support for RK356x JPEG encoder Nicolas Frattaroli ` (2 subsequent siblings) 3 siblings, 1 reply; 13+ messages in thread From: Nicolas Frattaroli @ 2022-04-27 22:44 UTC (permalink / raw) To: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski, Heiko Stuebner Cc: Nicolas Frattaroli, Ezequiel Garcia, linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel The RK3568 and RK3566 have a VPU node solely dedicated to JPEG encoding. This patch adds a compatible for it, and also allows the bindings to only come with a vepu interrupt. Signed-off-by: Nicolas Frattaroli <frattaroli.nicolas@gmail.com> --- Documentation/devicetree/bindings/media/rockchip-vpu.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/devicetree/bindings/media/rockchip-vpu.yaml b/Documentation/devicetree/bindings/media/rockchip-vpu.yaml index bacb60a34989..cd62b44c34c3 100644 --- a/Documentation/devicetree/bindings/media/rockchip-vpu.yaml +++ b/Documentation/devicetree/bindings/media/rockchip-vpu.yaml @@ -22,6 +22,7 @@ properties: - rockchip,rk3288-vpu - rockchip,rk3328-vpu - rockchip,rk3399-vpu + - rockchip,rk3568-jpeg-vepu - rockchip,px30-vpu - items: - const: rockchip,rk3188-vpu @@ -40,6 +41,7 @@ properties: interrupt-names: oneOf: - const: vdpu + - const: vepu - items: - const: vepu - const: vdpu -- 2.36.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 1/3] dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible 2022-04-27 22:44 ` [PATCH 1/3] dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible Nicolas Frattaroli @ 2022-05-04 20:23 ` Rob Herring 0 siblings, 0 replies; 13+ messages in thread From: Rob Herring @ 2022-05-04 20:23 UTC (permalink / raw) To: Nicolas Frattaroli Cc: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Krzysztof Kozlowski, Heiko Stuebner, Ezequiel Garcia, linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel On Thu, Apr 28, 2022 at 12:44:36AM +0200, Nicolas Frattaroli wrote: > The RK3568 and RK3566 have a VPU node solely dedicated to JPEG > encoding. This patch adds a compatible for it, and also allows > the bindings to only come with a vepu interrupt. Sounds like this might be more than just JPEG, so I'm assuming this will get sent again. > Signed-off-by: Nicolas Frattaroli <frattaroli.nicolas@gmail.com> > --- > Documentation/devicetree/bindings/media/rockchip-vpu.yaml | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/Documentation/devicetree/bindings/media/rockchip-vpu.yaml b/Documentation/devicetree/bindings/media/rockchip-vpu.yaml > index bacb60a34989..cd62b44c34c3 100644 > --- a/Documentation/devicetree/bindings/media/rockchip-vpu.yaml > +++ b/Documentation/devicetree/bindings/media/rockchip-vpu.yaml > @@ -22,6 +22,7 @@ properties: > - rockchip,rk3288-vpu > - rockchip,rk3328-vpu > - rockchip,rk3399-vpu > + - rockchip,rk3568-jpeg-vepu > - rockchip,px30-vpu > - items: > - const: rockchip,rk3188-vpu > @@ -40,6 +41,7 @@ properties: > interrupt-names: > oneOf: > - const: vdpu > + - const: vepu > - items: > - const: vepu > - const: vdpu > -- > 2.36.0 > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 2/3] media: hantro: Add support for RK356x JPEG encoder 2022-04-27 22:44 [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Frattaroli 2022-04-27 22:44 ` [PATCH 1/3] dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible Nicolas Frattaroli @ 2022-04-27 22:44 ` Nicolas Frattaroli 2022-08-09 12:45 ` Michael Grzeschik 2022-04-27 22:44 ` [PATCH 3/3] arm64: dts: rockchip: Add JPEG encoder node to rk356x Nicolas Frattaroli 2022-04-29 14:46 ` [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Dufresne 3 siblings, 1 reply; 13+ messages in thread From: Nicolas Frattaroli @ 2022-04-27 22:44 UTC (permalink / raw) To: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Greg Kroah-Hartman, Heiko Stuebner Cc: Nicolas Frattaroli, linux-media, linux-rockchip, linux-staging, linux-kernel, linux-arm-kernel The RK3566 and RK3568 SoCs come with a small Hantro instance which is solely dedicated to encoding JPEG. This patch adds the necessary structs to the Hantro driver to allow it to function. Tested with the following GStreamer command: gst-launch-1.0 videotestsrc ! v4l2jpegenc ! matroskamux ! \ filesink location=foo.mkv Signed-off-by: Nicolas Frattaroli <frattaroli.nicolas@gmail.com> --- drivers/staging/media/hantro/hantro_drv.c | 1 + drivers/staging/media/hantro/hantro_hw.h | 1 + .../staging/media/hantro/rockchip_vpu_hw.c | 62 +++++++++++++++++++ 3 files changed, 64 insertions(+) diff --git a/drivers/staging/media/hantro/hantro_drv.c b/drivers/staging/media/hantro/hantro_drv.c index dc768884cb79..3add9babd7bb 100644 --- a/drivers/staging/media/hantro/hantro_drv.c +++ b/drivers/staging/media/hantro/hantro_drv.c @@ -628,6 +628,7 @@ static const struct of_device_id of_hantro_match[] = { { .compatible = "rockchip,rk3288-vpu", .data = &rk3288_vpu_variant, }, { .compatible = "rockchip,rk3328-vpu", .data = &rk3328_vpu_variant, }, { .compatible = "rockchip,rk3399-vpu", .data = &rk3399_vpu_variant, }, + { .compatible = "rockchip,rk3568-jpeg-vepu", .data = &rk3568_jpeg_vepu_variant, }, #endif #ifdef CONFIG_VIDEO_HANTRO_IMX8M { .compatible = "nxp,imx8mm-vpu-g1", .data = &imx8mm_vpu_g1_variant, }, diff --git a/drivers/staging/media/hantro/hantro_hw.h b/drivers/staging/media/hantro/hantro_hw.h index ed018e293ba0..dd7f1edfacf2 100644 --- a/drivers/staging/media/hantro/hantro_hw.h +++ b/drivers/staging/media/hantro/hantro_hw.h @@ -300,6 +300,7 @@ extern const struct hantro_variant rk3066_vpu_variant; extern const struct hantro_variant rk3288_vpu_variant; extern const struct hantro_variant rk3328_vpu_variant; extern const struct hantro_variant rk3399_vpu_variant; +extern const struct hantro_variant rk3568_jpeg_vepu_variant; extern const struct hantro_variant sama5d4_vdec_variant; extern const struct hantro_variant sunxi_vpu_variant; diff --git a/drivers/staging/media/hantro/rockchip_vpu_hw.c b/drivers/staging/media/hantro/rockchip_vpu_hw.c index 163cf92eafca..10d3ea92a954 100644 --- a/drivers/staging/media/hantro/rockchip_vpu_hw.c +++ b/drivers/staging/media/hantro/rockchip_vpu_hw.c @@ -204,6 +204,43 @@ static const struct hantro_fmt rk3399_vpu_dec_fmts[] = { }, }; +static const struct hantro_fmt rk3568_jpeg_vepu_enc_fmts[] = { + { + .fourcc = V4L2_PIX_FMT_YUV420M, + .codec_mode = HANTRO_MODE_NONE, + .enc_fmt = ROCKCHIP_VPU_ENC_FMT_YUV420P, + }, + { + .fourcc = V4L2_PIX_FMT_NV12M, + .codec_mode = HANTRO_MODE_NONE, + .enc_fmt = ROCKCHIP_VPU_ENC_FMT_YUV420SP, + }, + { + .fourcc = V4L2_PIX_FMT_YUYV, + .codec_mode = HANTRO_MODE_NONE, + .enc_fmt = ROCKCHIP_VPU_ENC_FMT_YUYV422, + }, + { + .fourcc = V4L2_PIX_FMT_UYVY, + .codec_mode = HANTRO_MODE_NONE, + .enc_fmt = ROCKCHIP_VPU_ENC_FMT_UYVY422, + }, + { + .fourcc = V4L2_PIX_FMT_JPEG, + .codec_mode = HANTRO_MODE_JPEG_ENC, + .max_depth = 2, + .header_size = JPEG_HEADER_SIZE, + .frmsize = { + .min_width = 96, + .max_width = 8192, + .step_width = MB_DIM, + .min_height = 32, + .max_height = 8192, + .step_height = MB_DIM, + }, + }, +}; + static irqreturn_t rockchip_vpu1_vepu_irq(int irq, void *dev_id) { struct hantro_dev *vpu = dev_id; @@ -417,6 +454,14 @@ static const struct hantro_codec_ops rk3399_vpu_codec_ops[] = { }, }; +static const struct hantro_codec_ops rk3568_jpeg_enc_codec_ops[] = { + [HANTRO_MODE_JPEG_ENC] = { + .run = rockchip_vpu2_jpeg_enc_run, + .reset = rockchip_vpu2_enc_reset, + .done = rockchip_vpu2_jpeg_enc_done, + }, +}; + /* * VPU variant. */ @@ -439,6 +484,10 @@ static const struct hantro_irq rockchip_vpu2_irqs[] = { { "vdpu", rockchip_vpu2_vdpu_irq }, }; +static const struct hantro_irq rk3568_jpeg_vepu_irqs[] = { + { "vepu", rockchip_vpu2_vepu_irq }, +}; + static const char * const rk3066_vpu_clk_names[] = { "aclk_vdpu", "hclk_vdpu", "aclk_vepu", "hclk_vepu" @@ -545,6 +594,19 @@ const struct hantro_variant rk3399_vpu_variant = { .num_clocks = ARRAY_SIZE(rockchip_vpu_clk_names) }; +const struct hantro_variant rk3568_jpeg_vepu_variant = { + .enc_offset = 0x0, + .enc_fmts = rk3568_jpeg_vepu_enc_fmts, + .num_enc_fmts = ARRAY_SIZE(rk3568_jpeg_vepu_enc_fmts), + .codec = HANTRO_JPEG_ENCODER, + .codec_ops = rk3568_jpeg_enc_codec_ops, + .irqs = rk3568_jpeg_vepu_irqs, + .num_irqs = ARRAY_SIZE(rk3568_jpeg_vepu_irqs), + .init = rockchip_vpu_hw_init, + .clk_names = rockchip_vpu_clk_names, + .num_clocks = ARRAY_SIZE(rockchip_vpu_clk_names) +}; + const struct hantro_variant px30_vpu_variant = { .enc_offset = 0x0, .enc_fmts = rockchip_vpu_enc_fmts, -- 2.36.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 2/3] media: hantro: Add support for RK356x JPEG encoder 2022-04-27 22:44 ` [PATCH 2/3] media: hantro: Add support for RK356x JPEG encoder Nicolas Frattaroli @ 2022-08-09 12:45 ` Michael Grzeschik 2022-08-09 18:43 ` Michael Grzeschik 0 siblings, 1 reply; 13+ messages in thread From: Michael Grzeschik @ 2022-08-09 12:45 UTC (permalink / raw) To: Nicolas Frattaroli Cc: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Greg Kroah-Hartman, Heiko Stuebner, linux-media, linux-rockchip, linux-staging, linux-kernel, linux-arm-kernel [-- Attachment #1: Type: text/plain, Size: 1636 bytes --] Hi Nicolas, On Thu, Apr 28, 2022 at 12:44:37AM +0200, Nicolas Frattaroli wrote: >The RK3566 and RK3568 SoCs come with a small Hantro instance which >is solely dedicated to encoding JPEG. This patch adds the necessary >structs to the Hantro driver to allow it to function. > >Tested with the following GStreamer command: > >gst-launch-1.0 videotestsrc ! v4l2jpegenc ! matroskamux ! \ > filesink location=foo.mkv I just tried this pipeline on v5.19. The filesize generated is increasing and data is generated. However the content seems to be unusable. When playing foo.mkv with e.g. vlc the following errors, that no JPEG data was found in the images, is generated: [mjpeg @ 0x7f2f08c1f780] No JPEG data found in image [mjpeg @ 0x7f2f08c1f780] No JPEG data found in image [mjpeg @ 0x7f2f08c1f780] No JPEG data found in image ... The same with gstreamer: gst-launch-1.0 filesrc location=foo.mkv ! decodebin ! glimagesink ../ext/jpeg/gstjpegdec.c(1418): gst_jpeg_dec_handle_frame (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstJpegDec:jpegdec0: Decode error #53: Not a JPEG file: starts with 0x00 0x00 .. Do you have an idea why that is? Did you test the pipeline with an earlier kernelversion and validated the generated content of foo.mkv? Regards, Michael -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/3] media: hantro: Add support for RK356x JPEG encoder 2022-08-09 12:45 ` Michael Grzeschik @ 2022-08-09 18:43 ` Michael Grzeschik 0 siblings, 0 replies; 13+ messages in thread From: Michael Grzeschik @ 2022-08-09 18:43 UTC (permalink / raw) To: Nicolas Frattaroli Cc: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Greg Kroah-Hartman, Heiko Stuebner, linux-media, linux-rockchip, linux-staging, linux-kernel, linux-arm-kernel [-- Attachment #1: Type: text/plain, Size: 2174 bytes --] On Tue, Aug 09, 2022 at 02:45:08PM +0200, Michael Grzeschik wrote: >On Thu, Apr 28, 2022 at 12:44:37AM +0200, Nicolas Frattaroli wrote: >>The RK3566 and RK3568 SoCs come with a small Hantro instance which >>is solely dedicated to encoding JPEG. This patch adds the necessary >>structs to the Hantro driver to allow it to function. >> >>Tested with the following GStreamer command: >> >>gst-launch-1.0 videotestsrc ! v4l2jpegenc ! matroskamux ! \ >> filesink location=foo.mkv > >I just tried this pipeline on v5.19. The filesize generated is increasing and >data is generated. However the content seems to be unusable. > >When playing foo.mkv with e.g. vlc the following errors, that no JPEG data was found >in the images, is generated: > > >[mjpeg @ 0x7f2f08c1f780] No JPEG data found in image >[mjpeg @ 0x7f2f08c1f780] No JPEG data found in image >[mjpeg @ 0x7f2f08c1f780] No JPEG data found in image >... > > >The same with gstreamer: > >gst-launch-1.0 filesrc location=foo.mkv ! decodebin ! glimagesink >../ext/jpeg/gstjpegdec.c(1418): gst_jpeg_dec_handle_frame (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstJpegDec:jpegdec0: >Decode error #53: Not a JPEG file: starts with 0x00 0x00 >.. > > >Do you have an idea why that is? Did you test the pipeline with an earlier >kernelversion and validated the generated content of foo.mkv? After further debugging, I found out that my latest port of the bootlin patches to support h264 encoding broke the hantro driver somehow. https://git.pengutronix.de/cgit/mgr/linux/log/?h=v5.19/topic/rk3568-vepu-h264-stateless-bootlin The good news about that is; it is possible that the same issue was the cause that I thought the h264 encoding is not supported on the VEPU121. So the support is still unknown I will come back to that. Regards, Michael -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 3/3] arm64: dts: rockchip: Add JPEG encoder node to rk356x 2022-04-27 22:44 [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Frattaroli 2022-04-27 22:44 ` [PATCH 1/3] dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible Nicolas Frattaroli 2022-04-27 22:44 ` [PATCH 2/3] media: hantro: Add support for RK356x JPEG encoder Nicolas Frattaroli @ 2022-04-27 22:44 ` Nicolas Frattaroli 2022-04-29 14:46 ` [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Dufresne 3 siblings, 0 replies; 13+ messages in thread From: Nicolas Frattaroli @ 2022-04-27 22:44 UTC (permalink / raw) To: Rob Herring, Krzysztof Kozlowski, Heiko Stuebner Cc: Nicolas Frattaroli, devicetree, linux-arm-kernel, linux-rockchip, linux-kernel The RK3566 and RK3568 come with a dedicated Hantro instance solely for JPEG encoding. This patch adds a node for this to the device tree, along with a node for its MMU. Signed-off-by: Nicolas Frattaroli <frattaroli.nicolas@gmail.com> --- arch/arm64/boot/dts/rockchip/rk356x.dtsi | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/arm64/boot/dts/rockchip/rk356x.dtsi b/arch/arm64/boot/dts/rockchip/rk356x.dtsi index 7cdef800cb3c..276b76d5f3fb 100644 --- a/arch/arm64/boot/dts/rockchip/rk356x.dtsi +++ b/arch/arm64/boot/dts/rockchip/rk356x.dtsi @@ -508,6 +508,27 @@ gpu: gpu@fde60000 { status = "disabled"; }; + vepu_jpeg: video-codec@fdee0000 { + compatible = "rockchip,rk3568-jpeg-vepu"; + reg = <0x0 0xfdee0000 0x0 0x800>; + interrupts = <GIC_SPI 64 IRQ_TYPE_LEVEL_HIGH>; + interrupt-names = "vepu"; + clocks = <&cru ACLK_JENC>, <&cru HCLK_JENC>; + clock-names = "aclk", "hclk"; + iommus = <&vepu_jpeg_mmu>; + power-domains = <&power RK3568_PD_RGA>; + }; + + vepu_jpeg_mmu: iommu@fdee0800 { + compatible = "rockchip,rk3568-iommu"; + reg = <0x0 0xfdee0800 0x0 0x40>; + interrupts = <GIC_SPI 63 IRQ_TYPE_LEVEL_HIGH>; + clocks = <&cru ACLK_JENC>, <&cru HCLK_JENC>; + clock-names = "aclk", "iface"; + power-domains = <&power RK3568_PD_RGA>; + #iommu-cells = <0>; + }; + sdmmc2: mmc@fe000000 { compatible = "rockchip,rk3568-dw-mshc", "rockchip,rk3288-dw-mshc"; reg = <0x0 0xfe000000 0x0 0x4000>; -- 2.36.0 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 2022-04-27 22:44 [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Frattaroli ` (2 preceding siblings ...) 2022-04-27 22:44 ` [PATCH 3/3] arm64: dts: rockchip: Add JPEG encoder node to rk356x Nicolas Frattaroli @ 2022-04-29 14:46 ` Nicolas Dufresne 2022-04-29 15:28 ` Nicolas Frattaroli 3 siblings, 1 reply; 13+ messages in thread From: Nicolas Dufresne @ 2022-04-29 14:46 UTC (permalink / raw) To: Nicolas Frattaroli, Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski, Heiko Stuebner, Greg Kroah-Hartman, Peter Geis, Michael Riesch, Liang Chen Cc: linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel, linux-staging Le jeudi 28 avril 2022 à 00:44 +0200, Nicolas Frattaroli a écrit : > Hello, > > the following series adds support for and enables the hardware JPEG > encoder on the RK3566 and RK3568 line of SoCs by Rockchip. > > The JPEG encoder is its own little Hantro instance with seemingly just > the encode functionality. I'm a little suspicious about this statement. I believe the Hantro combo is identical to RK3399 and that you are confusing with Rockchip JPEG encoder here. Here's the source of my suspicion: https://github.com/JeffyCN/rockchip_mirrors/blob/mpp/osal/mpp_soc.cpp#L637 As this get burnt into DT, we really need to get this right. Perhaps we need to run the reference software to verify ? Ping me if you need help with that. > > The first patch modifies the bindings with a new compatible, and adds > the ability to just have a vepu interrupt without a vdpu interrupt. > > The second patch makes the actual driver changes to support this variant. > > The third and final patch makes the necessary device tree changes for > the rk356x device tree file to add both the node for the encoder and > its MMU. > > The series has been tested on a PINE64 Quartz64 Model A with an RK3566 > SoC using GStreamer. > > Regards, > Nicolas Frattaroli > > Nicolas Frattaroli (3): > dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible > media: hantro: Add support for RK356x JPEG encoder > arm64: dts: rockchip: Add JPEG encoder node to rk356x > > .../bindings/media/rockchip-vpu.yaml | 2 + > arch/arm64/boot/dts/rockchip/rk356x.dtsi | 21 +++++++ > drivers/staging/media/hantro/hantro_drv.c | 1 + > drivers/staging/media/hantro/hantro_hw.h | 1 + > .../staging/media/hantro/rockchip_vpu_hw.c | 62 +++++++++++++++++++ > 5 files changed, 87 insertions(+) > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 2022-04-29 14:46 ` [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Dufresne @ 2022-04-29 15:28 ` Nicolas Frattaroli 2022-04-29 23:20 ` Michael Grzeschik 0 siblings, 1 reply; 13+ messages in thread From: Nicolas Frattaroli @ 2022-04-29 15:28 UTC (permalink / raw) To: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski, Heiko Stuebner, Greg Kroah-Hartman, Peter Geis, Michael Riesch, Liang Chen, Nicolas Dufresne Cc: linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel, linux-staging, Kever Yang On Freitag, 29. April 2022 16:46:01 CEST Nicolas Dufresne wrote: > Le jeudi 28 avril 2022 à 00:44 +0200, Nicolas Frattaroli a écrit : > > Hello, > > > > the following series adds support for and enables the hardware JPEG > > encoder on the RK3566 and RK3568 line of SoCs by Rockchip. > > > > The JPEG encoder is its own little Hantro instance with seemingly just > > the encode functionality. > > I'm a little suspicious about this statement. I believe the Hantro combo is > identical to RK3399 and that you are confusing with Rockchip JPEG encoder here. > Here's the source of my suspicion: > > https://github.com/JeffyCN/rockchip_mirrors/blob/mpp/osal/mpp_soc.cpp#L637 > > As this get burnt into DT, we really need to get this right. Perhaps we need to > run the reference software to verify ? Ping me if you need help with that. I believe there's three separate questions here: A) is this a Hantro instance with more than just JPEG encoding? B) is this the same as the other[1] Hantro instance, just with a bigger offset for the encoder? C) if A is true and B is false, does this also have decoders? I think I can only answer B). I do not think fdee0000 is the same hardware instance as fdea0400. They are in different power domains (yes, this one being in the RGA PD is correct according to downstream code, TRM and my own testing). They also have their own MMU each. Your findings point towards A) being true. I'd love to be able to just throw a mainline VP8 or H.264 driver at it to check, but that sadly does not exist yet. Upon checking the TRM again, it seems it documents registers for H.264 as well. However I can do this to answer C), and I will once I get the time. It would be somewhat surprising if there was a hidden third H.264 decoder on this chip. I'll also CC Kever Yang from rockchip, maybe he can shine some light on this. Regards, Nicolas Frattaroli [1]: https://patchwork.kernel.org/project/linux-rockchip/patch/20220214212955.1178947-2-piotr.oniszczuk@gmail.com/ > > > > > The first patch modifies the bindings with a new compatible, and adds > > the ability to just have a vepu interrupt without a vdpu interrupt. > > > > The second patch makes the actual driver changes to support this variant. > > > > The third and final patch makes the necessary device tree changes for > > the rk356x device tree file to add both the node for the encoder and > > its MMU. > > > > The series has been tested on a PINE64 Quartz64 Model A with an RK3566 > > SoC using GStreamer. > > > > Regards, > > Nicolas Frattaroli > > > > Nicolas Frattaroli (3): > > dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible > > media: hantro: Add support for RK356x JPEG encoder > > arm64: dts: rockchip: Add JPEG encoder node to rk356x > > > > .../bindings/media/rockchip-vpu.yaml | 2 + > > arch/arm64/boot/dts/rockchip/rk356x.dtsi | 21 +++++++ > > drivers/staging/media/hantro/hantro_drv.c | 1 + > > drivers/staging/media/hantro/hantro_hw.h | 1 + > > .../staging/media/hantro/rockchip_vpu_hw.c | 62 +++++++++++++++++++ > > 5 files changed, 87 insertions(+) > > > > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 2022-04-29 15:28 ` Nicolas Frattaroli @ 2022-04-29 23:20 ` Michael Grzeschik 2022-05-02 12:20 ` Nicolas Dufresne 0 siblings, 1 reply; 13+ messages in thread From: Michael Grzeschik @ 2022-04-29 23:20 UTC (permalink / raw) To: Nicolas Frattaroli Cc: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski, Heiko Stuebner, Greg Kroah-Hartman, Peter Geis, Michael Riesch, Liang Chen, Nicolas Dufresne, linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel, linux-staging, Kever Yang [-- Attachment #1: Type: text/plain, Size: 5137 bytes --] Hi, On Fri, Apr 29, 2022 at 05:28:35PM +0200, Nicolas Frattaroli wrote: >On Freitag, 29. April 2022 16:46:01 CEST Nicolas Dufresne wrote: >> Le jeudi 28 avril 2022 à 00:44 +0200, Nicolas Frattaroli a écrit : >> > the following series adds support for and enables the hardware JPEG >> > encoder on the RK3566 and RK3568 line of SoCs by Rockchip. >> > >> > The JPEG encoder is its own little Hantro instance with seemingly just >> > the encode functionality. >> >> I'm a little suspicious about this statement. I believe the Hantro combo is >> identical to RK3399 and that you are confusing with Rockchip JPEG encoder here. >> Here's the source of my suspicion: >> >> https://github.com/JeffyCN/rockchip_mirrors/blob/mpp/osal/mpp_soc.cpp#L637 >> >> As this get burnt into DT, we really need to get this right. Perhaps we need to >> run the reference software to verify ? Ping me if you need help with that. > >I believe there's three separate questions here: > >A) is this a Hantro instance with more than just JPEG encoding? >B) is this the same as the other[1] Hantro instance, just with a bigger > offset for the encoder? >C) if A is true and B is false, does this also have decoders? > >I think I can only answer B). I do not think fdee0000 is the same hardware >instance as fdea0400. They are in different power domains (yes, this one >being in the RGA PD is correct according to downstream code, TRM and my >own testing). They also have their own MMU each. >Your findings point towards A) being true. I'd love to be able to just >throw a mainline VP8 or H.264 driver at it to check, but that sadly does >not exist yet. Not Mainline, but for the the RK3399 Encoder there is some initial work available made by bootlin. https://github.com/bootlin/linux bootlin/hantro/h264-encoding-v5.11 They also implemented some initial userspace tool to test this. https://github.com/bootlin/v4l2-hantro-h264-encoder If you have some time on your hands, it would be great if we could test this on the RK3568 with the VEPU121 core. >Upon checking the TRM again, it seems it documents registers for H.264 as well. I would like to describe the whole sitation a little bit more in detail. In the Datasheet of the RK3568 there are several decoder and encoder entities documented. This encoder instance is called VEPU121 in the TRM. RK3568: ------- Look into "Chapter 10 Multi-format Video Encoder and Decoder" (Page 463) of RK3568 TRM Part2. https://dl.radxa.com/rock3/docs/hw/datasheet/Rockchip%20RK3568%20TRM%20Part2%20V1.1-20210301.pdf We also find some mentionioning of JPEG encoding in the RK3399 datasheet. RK3399: ------- Look into "Chapter 5 Multi-format Video Decoder And Encoder" (Page 297) of RK3399 TRM Part 3. http://www.netbsd.org/~mrg/rk3399/Rockchip%20RK3399TRM%20V1.1%20Part3%2020160728.pdf * Supports encoding of the following standards: * H.264: up to HP level 4.1 * JPEG: Baseline (DCT sequential) Which probably refers to the core that is also available on the RK3399. Comparing the register layout of VEPU121 of the RK3568 with the encoder part in RK3399 we find nearly the same register description. 10.5.21 VEPU121 Detail Registers Description (Page 704 of RK3568 TRM Part2) vs. 5.5.4 VEPU Registers Summary (Page 443 of RK3399 TRM Part3) However this is just the documentation, this could mean nothing to the real world. Getting the core tested with the bootlin stack or getting feedback from Kever would probably tell the whole story. >However I can do this to answer C), and I will once I get the time. >It would be somewhat surprising if there was a hidden third H.264 >decoder on this chip. The corresponding decoder world on the RK3568 is also very unclear. It supports RKVDEC (referd as VDPU346 in the TRM), that is probably also supported by the mainline code you can find in drivers/staging/media/rkvdec/. Then there is VDPU121 which seems to be just the same G1 as in RK3399, regarding to these patches: https://lore.kernel.org/all/20220213195316.4149106-3-piotr.oniszczuk@gmail.com/T/ The last Core you probably refer to is called VDPU720 JPEG Decoder located at 0xFDED0000. I also really don't know where to put this in at the moment. >I'll also CC Kever Yang from rockchip, maybe he can shine some light on >this. Since the RK3568 seems to get more and more of attention at the moment I would like to ask if somebody is planning to write support the RKVENC in mainline. That is the VEPU540 core refered to in the RK3568 TRM. I would start with that in the next weeks, taking the bootlin H1 support as an reference to handle the request_api for the encoder part. Which is currently completely untouched in mainline AFAIK. Regards, Michael -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 2022-04-29 23:20 ` Michael Grzeschik @ 2022-05-02 12:20 ` Nicolas Dufresne 2022-06-20 13:40 ` Michael Grzeschik 0 siblings, 1 reply; 13+ messages in thread From: Nicolas Dufresne @ 2022-05-02 12:20 UTC (permalink / raw) To: Michael Grzeschik, Nicolas Frattaroli Cc: Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski, Heiko Stuebner, Greg Kroah-Hartman, Peter Geis, Michael Riesch, Liang Chen, linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel, linux-staging, Kever Yang, Andrzej Pietrasiewicz Hi Michael, Le samedi 30 avril 2022 à 01:20 +0200, Michael Grzeschik a écrit : > Since the RK3568 seems to get more and more of attention at the moment > I would like to ask if somebody is planning to write support the RKVENC > in mainline. That is the VEPU540 core refered to in the RK3568 TRM. > > I would start with that in the next weeks, taking the bootlin H1 support > as an reference to handle the request_api for the encoder part. Which is > currently completely untouched in mainline AFAIK. I'm very happy to ear you'd be interest in helping with this. I'm adding Andrzej in CC, as he started some base work using VP8 encoder on RK3399 (same HW found on other RK SoC) to make this possible. Note that these are a new type of encoders and a specification is needed to ensure they all have a coherent work- flow. The first step was obviously to have working prototype, Bootlin started with H264 (but never posted anything on the mailing list). I'm under the impression they have abandoned it. Andrzej have decided to use VP8 as it is simpler. The prototype should help understand the basic flow of an encoder and produce a specification for this. From there, the "easy" part is to make per codec controls, to configure the encoder. Demonstrating fixed QP, this is about were I believe Bootlin stopped. And I personally believe some OSS userland (not just tests) that can handle multiple reference, perhaps more advance GOP pattern and some basic rate control would help build confidence in the uAPI. Let us know how we can help, we should avoid doing the same thing, as this is already quite a large project that can easily take over a year to become mainline ready. regards, Nicolas > > Regards, > Michael > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 2022-05-02 12:20 ` Nicolas Dufresne @ 2022-06-20 13:40 ` Michael Grzeschik 2022-06-21 14:44 ` Nicolas Dufresne 0 siblings, 1 reply; 13+ messages in thread From: Michael Grzeschik @ 2022-06-20 13:40 UTC (permalink / raw) To: Nicolas Dufresne Cc: Nicolas Frattaroli, Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski, Heiko Stuebner, Greg Kroah-Hartman, Peter Geis, Michael Riesch, Liang Chen, linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel, linux-staging, Kever Yang, Andrzej Pietrasiewicz [-- Attachment #1: Type: text/plain, Size: 3597 bytes --] Hi Nicolas, Hi Andrzej, On Mon, May 02, 2022 at 08:20:50AM -0400, Nicolas Dufresne wrote: >Le samedi 30 avril 2022 à 01:20 +0200, Michael Grzeschik a écrit : >> Since the RK3568 seems to get more and more of attention at the moment >> I would like to ask if somebody is planning to write support the RKVENC >> in mainline. That is the VEPU540 core refered to in the RK3568 TRM. >> >> I would start with that in the next weeks, taking the bootlin H1 support >> as an reference to handle the request_api for the encoder part. Which is >> currently completely untouched in mainline AFAIK. > >I'm very happy to ear you'd be interest in helping with this. I'm adding Andrzej >in CC, as he started some base work using VP8 encoder on RK3399 (same HW found >on other RK SoC) to make this possible. Note that these are a new type of >encoders and a specification is needed to ensure they all have a coherent work- >flow. The first step was obviously to have working prototype, Bootlin started >with H264 (but never posted anything on the mailing list). I'm under the >impression they have abandoned it. Andrzej have decided to use VP8 as it is >simpler. The prototype should help understand the basic flow of an encoder and >produce a specification for this. I just have tested the bootlin v4l2-h264-encoder stack on the rk3399 with v5.19-rc2. It is working as expected. Since there have been some rumours about using the VEPU121 core on the rk3568, which could have h264 support as well. I have tested this stack on that core as well. The findings are; It is producing some data, but the stream created is not possible to be decoded. Tested with vlc and gstreamer. Be it due to misconfiguration or the core is really not h264 capable. Find the latest code here: https://git.pengutronix.de/cgit/mgr/linux/log/?h=v5.19/topic/rk3568-vepu-h264-stateless-bootlin >From there, the "easy" part is to make per codec controls, to configure the >encoder. Demonstrating fixed QP, this is about were I believe Bootlin stopped. I have seen that the userspace tool is filling in some initial values in the PPS and SPS headers, which are not adjustable. I bet you refer to that. >And I personally believe some OSS userland (not just tests) that can handle >multiple reference, perhaps more advance GOP pattern and some basic rate control >would help build confidence in the uAPI. I found the following CTRL Types being defined in the bootlin stack for the request API. V4L2_CID_STATELESS_H264_ENCODE_PARAMS V4L2_CID_STATELESS_H264_ENCODE_RC V4L2_CID_STATELESS_H264_ENCODE_FEEDBACK The FEEDBACK CID is the information we gain after one frame was encoded. With this data the RC (Rate Control) parameter is prepared for the next frame. You mean to have some user interface to make the rate control user configurable? >Let us know how we can help, we should >void doing the same thing, as >this is already quite a large project that can easily take over a year >to become mainline ready. It would probably help to share some codebase. If you already have something for that rkvenc codec, be it VP8 support. It would probably make sense for me to build up on that. Or to discuss common code patterns on structures. Michael -- Pengutronix e.K. | | Steuerwalder Str. 21 | http://www.pengutronix.de/ | 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 | [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 2022-06-20 13:40 ` Michael Grzeschik @ 2022-06-21 14:44 ` Nicolas Dufresne 0 siblings, 0 replies; 13+ messages in thread From: Nicolas Dufresne @ 2022-06-21 14:44 UTC (permalink / raw) To: Michael Grzeschik Cc: Nicolas Frattaroli, Ezequiel Garcia, Philipp Zabel, Mauro Carvalho Chehab, Rob Herring, Krzysztof Kozlowski, Heiko Stuebner, Greg Kroah-Hartman, Peter Geis, Michael Riesch, Liang Chen, linux-media, linux-rockchip, devicetree, linux-arm-kernel, linux-kernel, linux-staging, Kever Yang, Andrzej Pietrasiewicz Le lundi 20 juin 2022 à 15:40 +0200, Michael Grzeschik a écrit : > Hi Nicolas, > Hi Andrzej, > > On Mon, May 02, 2022 at 08:20:50AM -0400, Nicolas Dufresne wrote: > > Le samedi 30 avril 2022 à 01:20 +0200, Michael Grzeschik a écrit : > > > Since the RK3568 seems to get more and more of attention at the moment > > > I would like to ask if somebody is planning to write support the RKVENC > > > in mainline. That is the VEPU540 core refered to in the RK3568 TRM. > > > > > > I would start with that in the next weeks, taking the bootlin H1 support > > > as an reference to handle the request_api for the encoder part. Which is > > > currently completely untouched in mainline AFAIK. > > > > I'm very happy to ear you'd be interest in helping with this. I'm adding Andrzej > > in CC, as he started some base work using VP8 encoder on RK3399 (same HW found > > on other RK SoC) to make this possible. Note that these are a new type of > > encoders and a specification is needed to ensure they all have a coherent work- > > flow. The first step was obviously to have working prototype, Bootlin started > > with H264 (but never posted anything on the mailing list). I'm under the > > impression they have abandoned it. Andrzej have decided to use VP8 as it is > > simpler. The prototype should help understand the basic flow of an encoder and > > produce a specification for this. > > I just have tested the bootlin v4l2-h264-encoder stack on the rk3399 > with v5.19-rc2. It is working as expected. > > Since there have been some rumours about using the VEPU121 core on the > rk3568, which could have h264 support as well. I have tested this > stack on that core as well. > > The findings are; It is producing some data, but the stream created > is not possible to be decoded. Tested with vlc and gstreamer. Oh well, I'd say unless vendor document it, we'll not going to just enable it there. There is another (and faster) encoder there with an vendor but fully Open Source stack anyway. Thanks for the extra step here. > > Be it due to misconfiguration or the core is really not h264 capable. > > Find the latest code here: > > https://git.pengutronix.de/cgit/mgr/linux/log/?h=v5.19/topic/rk3568-vepu-h264-stateless-bootlin > > > From there, the "easy" part is to make per codec controls, to configure the > > encoder. Demonstrating fixed QP, this is about were I believe Bootlin stopped. > > I have seen that the userspace tool is filling in some initial values > in the PPS and SPS headers, which are not adjustable. I bet you refer to that. > > > And I personally believe some OSS userland (not just tests) that can handle > > multiple reference, perhaps more advance GOP pattern and some basic rate control > > would help build confidence in the uAPI. > > I found the following CTRL Types being defined in the bootlin stack > for the request API. > > V4L2_CID_STATELESS_H264_ENCODE_PARAMS > V4L2_CID_STATELESS_H264_ENCODE_RC > V4L2_CID_STATELESS_H264_ENCODE_FEEDBACK > > The FEEDBACK CID is the information we gain after one frame was > encoded. With this data the RC (Rate Control) parameter is prepared for > the next frame. > > You mean to have some user interface to make the rate control user > configurable? RC feedback controls are going to all be vendor specific. This one cannot be called with a generic name like "ENCODE_FEEDBACK" since it is not generic. In fact, news Hantro chip don't have this feedback, they have something else but it is not used by the vendor software (not open source, but you can find leaks I suppose). They only use the size of the compressed frames, and do RC in software there. In addition to that, with the current trend, "feedback" controls would have to use RO Request, so that one can still queue multiple encode and be able to match the feedback to a specific request/frame. > > > Let us know how we can help, we should >void doing the same thing, as > > this is already quite a large project that can easily take over a year > > to become mainline ready. > > It would probably help to share some codebase. If you already have > something for that rkvenc codec, be it VP8 support. It would probably > make sense for me to build up on that. Or to discuss common code > patterns on structures. We are doing the exercise with VP8 using the original Hantro G1 design, as fond on IMX8MQ and RK3288. Though we only test on NXP boards at the moment. Andrzej is on vacation this week, I'll check with him why hist draft isn't accessible to the public. Meanwhile, the userland / GStreamer part is drafted (careful its a draft) here: https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer/-/tree/WIP_V4L2_VP8_STATELESS_ENCODER Currently we are still working on very basic bit of the puzzle, understanding the problem in order to produce a proper Stateless Encoder specification. As an example, the Hantro VP8 encoder brought interesting aspect, as an example it produces 2 buffers. Shall we offset these into 1 buffer, shall we use planes to zero copy that to user, or shall we memcpy in the driver .. Shall the VP8 frame header be produced by the driver (even though the HW don't produce it) or shall we have more stateless VP8 format for both cases. Its a big series of tiny things like this that we need to study and weight, and compare against other Hardware designs. regards, Nicolas ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2022-08-09 19:08 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-27 22:44 [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Frattaroli 2022-04-27 22:44 ` [PATCH 1/3] dt-bindings: media: rockchip-vpu: Add RK3568 JPEG compatible Nicolas Frattaroli 2022-05-04 20:23 ` Rob Herring 2022-04-27 22:44 ` [PATCH 2/3] media: hantro: Add support for RK356x JPEG encoder Nicolas Frattaroli 2022-08-09 12:45 ` Michael Grzeschik 2022-08-09 18:43 ` Michael Grzeschik 2022-04-27 22:44 ` [PATCH 3/3] arm64: dts: rockchip: Add JPEG encoder node to rk356x Nicolas Frattaroli 2022-04-29 14:46 ` [PATCH 0/3] Enable JPEG Encoder on RK3566/RK3568 Nicolas Dufresne 2022-04-29 15:28 ` Nicolas Frattaroli 2022-04-29 23:20 ` Michael Grzeschik 2022-05-02 12:20 ` Nicolas Dufresne 2022-06-20 13:40 ` Michael Grzeschik 2022-06-21 14:44 ` Nicolas Dufresne
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).