* [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
independent subset of earlier series [2] posted to upport CSI-2/CCP2
receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
SoCs (namely BCM2711). Unicam is still under active development to work
with multistream support to get into mainline. Hence only the ISP driver
will remain the primary area of this series.
Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
with the VC4 VPU.
Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
dma_buf instead of a file-descriptor.
Patch (04-05)/14 adds a couple of improvements/support for
bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
and format defintions.
Patch (10-11)/14 deals with the colorspace support.
Note: This is still WIP since the implementation of colorspace is still
getting ironed out (especially around setting of colorspace flags handling).
Patch 12/14 allows multiple instances of the ISP.
Patch 13/14 adds a admin-guide document on bcm2835-isp.
Patch 14/14 wires all this up with the vchiq-mmal driver.
Testing:
Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
bindings from [1]). Once done, functional testing was conducted with
libcamera[3] and its utility tools.
Also note: Reviews given on [2] for the relevant ISP driver patches have
been incorporated in this version.
Known issues:
- Colorspace handling
- vc-sm-cma spamming kernel log with
- pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
[1]: https://github.com/raspberrypi/linux
[2]: https://lore.kernel.org/linux-media/20200504092611.9798-1-laurent.pinchart@ideasonboard.com/
[3]: https://libcamera.org/getting-started.html
Dave Stevenson (7):
staging: vc04_services: Add new vc-sm-cma driver
staging: vchiq_arm: Register vcsm-cma as a platform driver
media: videobuf2: Allow exporting of a struct dmabuf
staging: mmal-vchiq: Add support for event callbacks
staging: mmal-vchiq: Use vc-sm-cma to support zero copy
staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
uapi: bcm2835-isp: Add bcm2835-isp uapi header file
David Plowman (2):
vc04_services: bcm2835-isp: Allow formats with different colour spaces
vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
outputs
Naushir Patuck (5):
media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
component
staging: vc04_services: bcm2835_isp: Allow multiple users
docs: admin-guide: media: bcm2835-isp: Add documentation for
bcm2835-isp
staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
.../admin-guide/media/bcm2835-isp.rst | 127 ++
.../userspace-api/media/drivers/index.rst | 1 +
.../userspace-api/media/v4l/meta-formats.rst | 1 +
.../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
MAINTAINERS | 9 +
.../media/common/videobuf2/videobuf2-core.c | 36 +-
drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
drivers/staging/vc04_services/Kconfig | 4 +
drivers/staging/vc04_services/Makefile | 2 +
.../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
.../vc04_services/bcm2835-isp/Makefile | 8 +
.../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
.../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
.../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
.../interface/vchiq_arm/vchiq_arm.c | 6 +
.../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
.../staging/vc04_services/vc-sm-cma/Makefile | 12 +
.../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
.../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
.../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
.../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
.../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
.../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
.../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
.../vc04_services/vchiq-mmal/mmal-common.h | 5 +
.../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
.../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
.../vchiq-mmal/mmal-parameters.h | 165 +-
.../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
.../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
include/media/videobuf2-core.h | 15 +
include/uapi/linux/bcm2835-isp.h | 347 ++++
include/uapi/linux/v4l2-controls.h | 6 +
include/uapi/linux/videodev2.h | 1 +
34 files changed, 5225 insertions(+), 33 deletions(-)
create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
create mode 100644 include/uapi/linux/bcm2835-isp.h
--
2.37.3
^ permalink raw reply [flat|nested] 95+ messages in thread
* [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
independent subset of earlier series [2] posted to upport CSI-2/CCP2
receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
SoCs (namely BCM2711). Unicam is still under active development to work
with multistream support to get into mainline. Hence only the ISP driver
will remain the primary area of this series.
Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
with the VC4 VPU.
Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
dma_buf instead of a file-descriptor.
Patch (04-05)/14 adds a couple of improvements/support for
bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
and format defintions.
Patch (10-11)/14 deals with the colorspace support.
Note: This is still WIP since the implementation of colorspace is still
getting ironed out (especially around setting of colorspace flags handling).
Patch 12/14 allows multiple instances of the ISP.
Patch 13/14 adds a admin-guide document on bcm2835-isp.
Patch 14/14 wires all this up with the vchiq-mmal driver.
Testing:
Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
bindings from [1]). Once done, functional testing was conducted with
libcamera[3] and its utility tools.
Also note: Reviews given on [2] for the relevant ISP driver patches have
been incorporated in this version.
Known issues:
- Colorspace handling
- vc-sm-cma spamming kernel log with
- pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
[1]: https://github.com/raspberrypi/linux
[2]: https://lore.kernel.org/linux-media/20200504092611.9798-1-laurent.pinchart@ideasonboard.com/
[3]: https://libcamera.org/getting-started.html
Dave Stevenson (7):
staging: vc04_services: Add new vc-sm-cma driver
staging: vchiq_arm: Register vcsm-cma as a platform driver
media: videobuf2: Allow exporting of a struct dmabuf
staging: mmal-vchiq: Add support for event callbacks
staging: mmal-vchiq: Use vc-sm-cma to support zero copy
staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
uapi: bcm2835-isp: Add bcm2835-isp uapi header file
David Plowman (2):
vc04_services: bcm2835-isp: Allow formats with different colour spaces
vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
outputs
Naushir Patuck (5):
media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
component
staging: vc04_services: bcm2835_isp: Allow multiple users
docs: admin-guide: media: bcm2835-isp: Add documentation for
bcm2835-isp
staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
.../admin-guide/media/bcm2835-isp.rst | 127 ++
.../userspace-api/media/drivers/index.rst | 1 +
.../userspace-api/media/v4l/meta-formats.rst | 1 +
.../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
MAINTAINERS | 9 +
.../media/common/videobuf2/videobuf2-core.c | 36 +-
drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
drivers/staging/vc04_services/Kconfig | 4 +
drivers/staging/vc04_services/Makefile | 2 +
.../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
.../vc04_services/bcm2835-isp/Makefile | 8 +
.../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
.../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
.../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
.../interface/vchiq_arm/vchiq_arm.c | 6 +
.../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
.../staging/vc04_services/vc-sm-cma/Makefile | 12 +
.../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
.../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
.../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
.../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
.../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
.../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
.../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
.../vc04_services/vchiq-mmal/mmal-common.h | 5 +
.../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
.../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
.../vchiq-mmal/mmal-parameters.h | 165 +-
.../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
.../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
include/media/videobuf2-core.h | 15 +
include/uapi/linux/bcm2835-isp.h | 347 ++++
include/uapi/linux/v4l2-controls.h | 6 +
include/uapi/linux/videodev2.h | 1 +
34 files changed, 5225 insertions(+), 33 deletions(-)
create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
create mode 100644 include/uapi/linux/bcm2835-isp.h
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* [PATCH 01/14] staging: vc04_services: Add new vc-sm-cma driver
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
Add Broadcom VideoCore Shared Memory support.
This new driver allows contiguous memory blocks to be imported
into the VideoCore VPU memory map, and manages the lifetime of
those objects, only releasing the source dmabuf once the VPU has
confirmed it has finished with it.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
drivers/staging/vc04_services/Kconfig | 2 +
drivers/staging/vc04_services/Makefile | 1 +
.../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
.../staging/vc04_services/vc-sm-cma/Makefile | 12 +
.../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++++++++++++
.../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 ++
.../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++++++++
.../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 ++
.../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++++
.../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
10 files changed, 1665 insertions(+)
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig
index 31e58c9d1a11..6c0e77d64376 100644
--- a/drivers/staging/vc04_services/Kconfig
+++ b/drivers/staging/vc04_services/Kconfig
@@ -46,5 +46,7 @@ source "drivers/staging/vc04_services/bcm2835-camera/Kconfig"
source "drivers/staging/vc04_services/vchiq-mmal/Kconfig"
+source "drivers/staging/vc04_services/vc-sm-cma/Kconfig"
+
endif
diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile
index 1fd191e2e2a5..01089f369fb4 100644
--- a/drivers/staging/vc04_services/Makefile
+++ b/drivers/staging/vc04_services/Makefile
@@ -14,6 +14,7 @@ endif
obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/
obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/
obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/
+obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/
ccflags-y += -I $(srctree)/$(src)/include
diff --git a/drivers/staging/vc04_services/vc-sm-cma/Kconfig b/drivers/staging/vc04_services/vc-sm-cma/Kconfig
new file mode 100644
index 000000000000..bbd296f5826b
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/Kconfig
@@ -0,0 +1,10 @@
+config BCM_VC_SM_CMA
+ tristate "VideoCore Shared Memory (CMA) driver"
+ depends on BCM2835_VCHIQ
+ select RBTREE
+ select DMA_SHARED_BUFFER
+ help
+ Say Y here to enable the shared memory interface that
+ supports sharing dmabufs with VideoCore.
+ This operates over the VCHIQ interface to a service
+ running on VideoCore.
diff --git a/drivers/staging/vc04_services/vc-sm-cma/Makefile b/drivers/staging/vc04_services/vc-sm-cma/Makefile
new file mode 100644
index 000000000000..c92a5775c62e
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/Makefile
@@ -0,0 +1,12 @@
+ccflags-y += \
+ -I$(srctree)/$(src)/../ \
+ -I$(srctree)/$(src)/../interface/vchiq_arm\
+ -I$(srctree)/$(src)/../include
+
+ccflags-y += \
+ -D__VCCOREVER__=0
+
+vc-sm-cma-$(CONFIG_BCM_VC_SM_CMA) := \
+ vc_sm.o vc_sm_cma_vchi.o
+
+obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma.o
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
new file mode 100644
index 000000000000..7fe81d259c7b
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
@@ -0,0 +1,801 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * VideoCore Shared Memory driver using CMA.
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ * Dave Stevenson <dave.stevenson@raspberrypi.com>
+ *
+ * Based on vmcs_sm driver from Broadcom Corporation for some API,
+ * and taking some code for buffer allocation and dmabuf handling from
+ * videobuf2.
+ *
+ * This driver handles taking a dmabuf, mapping it into the VPU address space,
+ * and returning a handle that the VPU can use.
+ */
+
+/* ---- Include Files ----------------------------------------------------- */
+#include <linux/debugfs.h>
+#include <linux/dma-buf.h>
+#include <linux/errno.h>
+#include <linux/miscdevice.h>
+#include <linux/platform_device.h>
+#include <linux/syscalls.h>
+
+#include "vchiq_connected.h"
+#include "vc_sm_cma_vchi.h"
+
+#include "vc_sm.h"
+#include "vc_sm_knl.h"
+
+MODULE_IMPORT_NS(DMA_BUF);
+
+/* ---- Private Constants and Types --------------------------------------- */
+
+#define VC_SM_RESOURCE_NAME_DEFAULT "sm-host-resource"
+
+#define VC_SM_DIR_ROOT_NAME "vcsm-cma"
+#define VC_SM_STATE "state"
+
+/* Private file data associated with each opened device. */
+struct vc_sm_privdata_t {
+};
+
+typedef int (*VC_SM_SHOW) (struct seq_file *s, void *v);
+struct sm_pde_t {
+ VC_SM_SHOW show; /* Debug fs function hookup. */
+ struct dentry *dir_entry; /* Debug fs directory entry. */
+ void *priv_data; /* Private data */
+};
+
+/* Global state information. */
+struct sm_state_t {
+ struct platform_device *pdev;
+
+ struct miscdevice misc_dev;
+
+ struct sm_instance *sm_handle; /* Handle for videocore service. */
+
+ spinlock_t kernelid_map_lock; /* Spinlock protecting kernelid_map */
+ struct idr kernelid_map;
+
+ struct mutex map_lock; /* Global map lock. */
+ struct list_head buffer_list; /* List of buffer. */
+
+ struct dentry *dir_root; /* Debug fs entries root. */
+ struct sm_pde_t dir_state; /* Debug fs entries state sub-tree. */
+
+ bool require_released_callback; /* VPU will send a released msg when it
+ * has finished with a resource.
+ */
+ /* State for transactions */
+ int restart_sys; /* Tracks restart on interrupt. */
+ enum vc_sm_msg_type int_action; /* Interrupted action. */
+
+ u32 int_trans_id; /* Interrupted transaction. */
+ struct vchiq_instance *vchiq_instance;
+};
+
+struct vc_sm_dma_buf_attachment {
+ struct device *dev;
+ struct sg_table sg_table;
+ struct list_head list;
+ enum dma_data_direction dma_dir;
+};
+
+/* ---- Private Variables ----------------------------------------------- */
+
+static struct sm_state_t *sm_state;
+static int sm_inited;
+
+/* ---- Private Function Prototypes -------------------------------------- */
+
+/* ---- Private Functions ------------------------------------------------ */
+
+static int get_kernel_id(struct vc_sm_buffer *buffer)
+{
+ int handle;
+
+ spin_lock(&sm_state->kernelid_map_lock);
+ handle = idr_alloc(&sm_state->kernelid_map, buffer, 0, 0, GFP_KERNEL);
+ spin_unlock(&sm_state->kernelid_map_lock);
+
+ return handle;
+}
+
+static struct vc_sm_buffer *lookup_kernel_id(int handle)
+{
+ return idr_find(&sm_state->kernelid_map, handle);
+}
+
+static void free_kernel_id(int handle)
+{
+ spin_lock(&sm_state->kernelid_map_lock);
+ idr_remove(&sm_state->kernelid_map, handle);
+ spin_unlock(&sm_state->kernelid_map_lock);
+}
+
+static int vc_sm_cma_seq_file_show(struct seq_file *s, void *v)
+{
+ struct sm_pde_t *sm_pde;
+
+ sm_pde = (struct sm_pde_t *)(s->private);
+
+ if (sm_pde && sm_pde->show)
+ sm_pde->show(s, v);
+
+ return 0;
+}
+
+static int vc_sm_cma_single_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, vc_sm_cma_seq_file_show, inode->i_private);
+}
+
+static const struct file_operations vc_sm_cma_debug_fs_fops = {
+ .open = vc_sm_cma_single_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static int vc_sm_cma_global_state_show(struct seq_file *s, void *v)
+{
+ struct vc_sm_buffer *resource = NULL;
+ int resource_count = 0;
+
+ if (!sm_state)
+ return 0;
+
+ seq_printf(s, "\nVC-ServiceHandle %p\n", sm_state->sm_handle);
+
+ /* Log all applicable mapping(s). */
+
+ mutex_lock(&sm_state->map_lock);
+ seq_puts(s, "\nResources\n");
+ if (!list_empty(&sm_state->buffer_list)) {
+ list_for_each_entry(resource, &sm_state->buffer_list,
+ global_buffer_list) {
+ resource_count++;
+
+ seq_printf(s, "\nResource %p\n",
+ resource);
+ seq_printf(s, " NAME %s\n",
+ resource->name);
+ seq_printf(s, " SIZE %zu\n",
+ resource->size);
+ seq_printf(s, " DMABUF %p\n",
+ resource->dma_buf);
+ seq_printf(s, " IMPORTED_DMABUF %p\n",
+ resource->imported_dma_buf);
+ seq_printf(s, " ATTACH %p\n",
+ resource->attach);
+ seq_printf(s, " SGT %p\n",
+ resource->sgt);
+ seq_printf(s, " DMA_ADDR %pad\n",
+ &resource->dma_addr);
+ seq_printf(s, " VC_HANDLE %08x\n",
+ resource->vc_handle);
+ seq_printf(s, " VC_MAPPING %d\n",
+ resource->vpu_state);
+ }
+ }
+ seq_printf(s, "\n\nTotal resource count: %d\n\n", resource_count);
+
+ mutex_unlock(&sm_state->map_lock);
+
+ return 0;
+}
+
+/*
+ * Adds a buffer to the private data list which tracks all the allocated
+ * data.
+ */
+static void vc_sm_add_resource(struct vc_sm_buffer *buffer)
+{
+ mutex_lock(&sm_state->map_lock);
+ list_add(&buffer->global_buffer_list, &sm_state->buffer_list);
+ mutex_unlock(&sm_state->map_lock);
+
+ pr_debug("[%s]: added buffer %p (name %s, size %zu)\n",
+ __func__, buffer, buffer->name, buffer->size);
+}
+
+/*
+ * Cleans up imported dmabuf.
+ * Should be called with mutex held.
+ */
+static void vc_sm_clean_up_dmabuf(struct vc_sm_buffer *buffer)
+{
+ /* Handle cleaning up imported dmabufs */
+ if (buffer->sgt) {
+ dma_buf_unmap_attachment(buffer->attach,
+ buffer->sgt,
+ DMA_BIDIRECTIONAL);
+ buffer->sgt = NULL;
+ }
+ if (buffer->attach) {
+ dma_buf_detach(buffer->dma_buf, buffer->attach);
+ buffer->attach = NULL;
+ }
+}
+
+/*
+ * Instructs VPU to decrement the refcount on a buffer.
+ */
+static void vc_sm_vpu_free(struct vc_sm_buffer *buffer)
+{
+ if (buffer->vc_handle && buffer->vpu_state == VPU_MAPPED) {
+ struct vc_sm_free_t free = { buffer->vc_handle, 0 };
+ int status = vc_sm_cma_vchi_free(sm_state->sm_handle, &free,
+ &sm_state->int_trans_id);
+ if (status != 0 && status != -EINTR) {
+ pr_err("[%s]: failed to free memory on videocore (status: %u, trans_id: %u)\n",
+ __func__, status, sm_state->int_trans_id);
+ }
+
+ if (sm_state->require_released_callback) {
+ /* Need to wait for the VPU to confirm the free. */
+
+ /* Retain a reference on this until the VPU has
+ * released it
+ */
+ buffer->vpu_state = VPU_UNMAPPING;
+ } else {
+ buffer->vpu_state = VPU_NOT_MAPPED;
+ buffer->vc_handle = 0;
+ }
+ }
+}
+
+/*
+ * Release an allocation.
+ * All refcounting is done via the dma buf object.
+ *
+ * Must be called with the mutex held. The function will either release the
+ * mutex (if defering the release) or destroy it. The caller must therefore not
+ * reuse the buffer on return.
+ */
+static void vc_sm_release_resource(struct vc_sm_buffer *buffer)
+{
+ pr_debug("[%s]: buffer %p (name %s, size %zu)\n",
+ __func__, buffer, buffer->name, buffer->size);
+
+ if (buffer->vc_handle) {
+ /* We've sent the unmap request but not had the response. */
+ pr_debug("[%s]: Waiting for VPU unmap response on %p\n",
+ __func__, buffer);
+ goto defer;
+ }
+ if (buffer->in_use) {
+ /* dmabuf still in use - we await the release */
+ pr_debug("[%s]: buffer %p is still in use\n", __func__, buffer);
+ goto defer;
+ }
+
+ /* Release the allocation */
+ if (buffer->imported_dma_buf)
+ dma_buf_put(buffer->imported_dma_buf);
+ else
+ pr_err("%s: Imported dmabuf already been put for buf %p\n",
+ __func__, buffer);
+ buffer->imported_dma_buf = NULL;
+
+ /* Free our buffer. Start by removing it from the list */
+ mutex_lock(&sm_state->map_lock);
+ list_del(&buffer->global_buffer_list);
+ mutex_unlock(&sm_state->map_lock);
+
+ pr_debug("%s: Release our allocation - done\n", __func__);
+ mutex_unlock(&buffer->lock);
+
+ mutex_destroy(&buffer->lock);
+
+ kfree(buffer);
+ return;
+
+defer:
+ mutex_unlock(&buffer->lock);
+}
+
+/* Dma_buf operations for chaining through to an imported dma_buf */
+
+static void vc_sm_dma_buf_release(struct dma_buf *dmabuf)
+{
+ struct vc_sm_buffer *buffer;
+
+ if (!dmabuf)
+ return;
+
+ buffer = (struct vc_sm_buffer *)dmabuf->priv;
+
+ mutex_lock(&buffer->lock);
+
+ pr_debug("%s dmabuf %p, buffer %p\n", __func__, dmabuf, buffer);
+
+ buffer->in_use = 0;
+
+ /* Unmap on the VPU */
+ vc_sm_vpu_free(buffer);
+ pr_debug("%s vpu_free done\n", __func__);
+
+ /* Unmap our dma_buf object (the vc_sm_buffer remains until released
+ * on the VPU).
+ */
+ vc_sm_clean_up_dmabuf(buffer);
+ pr_debug("%s clean_up dmabuf done\n", __func__);
+
+ /* buffer->lock will be destroyed by vc_sm_release_resource if finished
+ * with, otherwise unlocked. Do NOT unlock here.
+ */
+ vc_sm_release_resource(buffer);
+ pr_debug("%s done\n", __func__);
+}
+
+static
+int vc_sm_import_dma_buf_attach(struct dma_buf *dmabuf,
+ struct dma_buf_attachment *attachment)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ return buf->imported_dma_buf->ops->attach(buf->imported_dma_buf,
+ attachment);
+}
+
+static
+void vc_sm_import_dma_buf_detatch(struct dma_buf *dmabuf,
+ struct dma_buf_attachment *attachment)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ buf->imported_dma_buf->ops->detach(buf->imported_dma_buf, attachment);
+}
+
+static
+struct sg_table *vc_sm_import_map_dma_buf(struct dma_buf_attachment *attachment,
+ enum dma_data_direction direction)
+{
+ struct vc_sm_buffer *buf = attachment->dmabuf->priv;
+
+ return buf->imported_dma_buf->ops->map_dma_buf(attachment,
+ direction);
+}
+
+static
+void vc_sm_import_unmap_dma_buf(struct dma_buf_attachment *attachment,
+ struct sg_table *table,
+ enum dma_data_direction direction)
+{
+ struct vc_sm_buffer *buf = attachment->dmabuf->priv;
+
+ buf->imported_dma_buf->ops->unmap_dma_buf(attachment, table, direction);
+}
+
+static
+int vc_sm_import_dmabuf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ pr_debug("%s: mmap dma_buf %p, buf %p, imported db %p\n", __func__,
+ dmabuf, buf, buf->imported_dma_buf);
+
+ return buf->imported_dma_buf->ops->mmap(buf->imported_dma_buf, vma);
+}
+
+static
+int vc_sm_import_dma_buf_begin_cpu_access(struct dma_buf *dmabuf,
+ enum dma_data_direction direction)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ return buf->imported_dma_buf->ops->begin_cpu_access(buf->imported_dma_buf,
+ direction);
+}
+
+static
+int vc_sm_import_dma_buf_end_cpu_access(struct dma_buf *dmabuf,
+ enum dma_data_direction direction)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ return buf->imported_dma_buf->ops->end_cpu_access(buf->imported_dma_buf,
+ direction);
+}
+
+static const struct dma_buf_ops dma_buf_import_ops = {
+ .map_dma_buf = vc_sm_import_map_dma_buf,
+ .unmap_dma_buf = vc_sm_import_unmap_dma_buf,
+ .mmap = vc_sm_import_dmabuf_mmap,
+ .release = vc_sm_dma_buf_release,
+ .attach = vc_sm_import_dma_buf_attach,
+ .detach = vc_sm_import_dma_buf_detatch,
+ .begin_cpu_access = vc_sm_import_dma_buf_begin_cpu_access,
+ .end_cpu_access = vc_sm_import_dma_buf_end_cpu_access,
+};
+
+/* Import a dma_buf to be shared with VC. */
+int
+vc_sm_cma_import_dmabuf_internal(struct sm_state_t *state,
+ struct dma_buf *dma_buf,
+ int fd,
+ struct dma_buf **imported_buf)
+{
+ DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
+ struct vc_sm_buffer *buffer = NULL;
+ struct vc_sm_import import = { };
+ struct vc_sm_import_result result = { };
+ struct dma_buf_attachment *attach = NULL;
+ struct sg_table *sgt = NULL;
+ dma_addr_t dma_addr;
+ int ret = 0;
+ int status;
+
+ /* Setup our allocation parameters */
+ pr_debug("%s: importing dma_buf %p/fd %d\n", __func__, dma_buf, fd);
+
+ if (fd < 0)
+ get_dma_buf(dma_buf);
+ else
+ dma_buf = dma_buf_get(fd);
+
+ if (!dma_buf)
+ return -EINVAL;
+
+ attach = dma_buf_attach(dma_buf, &sm_state->pdev->dev);
+ if (IS_ERR(attach)) {
+ ret = PTR_ERR(attach);
+ goto error;
+ }
+
+ sgt = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL);
+ if (IS_ERR(sgt)) {
+ ret = PTR_ERR(sgt);
+ goto error;
+ }
+
+ /* Verify that the address block is contiguous */
+ if (sgt->nents != 1) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ /* Allocate local buffer to track this allocation. */
+ buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
+ if (!buffer) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ import.type = VC_SM_ALLOC_NON_CACHED;
+ dma_addr = sg_dma_address(sgt->sgl);
+ import.addr = (u32)dma_addr;
+ if ((import.addr & 0xC0000000) != 0xC0000000) {
+ pr_err("%s: Expecting an uncached alias for dma_addr %pad\n",
+ __func__, &dma_addr);
+ import.addr |= 0xC0000000;
+ }
+ import.size = sg_dma_len(sgt->sgl);
+ import.allocator = current->tgid;
+ import.kernel_id = get_kernel_id(buffer);
+
+ memcpy(import.name, VC_SM_RESOURCE_NAME_DEFAULT,
+ sizeof(VC_SM_RESOURCE_NAME_DEFAULT));
+
+ pr_debug("[%s]: attempt to import \"%s\" data - type %u, addr %pad, size %u.\n",
+ __func__, import.name, import.type, &dma_addr, import.size);
+
+ /* Allocate the videocore buffer. */
+ status = vc_sm_cma_vchi_import(sm_state->sm_handle, &import, &result,
+ &sm_state->int_trans_id);
+ if (status == -EINTR) {
+ pr_debug("[%s]: requesting import memory action restart (trans_id: %u)\n",
+ __func__, sm_state->int_trans_id);
+ ret = -ERESTARTSYS;
+ sm_state->restart_sys = -EINTR;
+ sm_state->int_action = VC_SM_MSG_TYPE_IMPORT;
+ goto error;
+ } else if (status || !result.res_handle) {
+ pr_debug("[%s]: failed to import memory on videocore (status: %u, trans_id: %u)\n",
+ __func__, status, sm_state->int_trans_id);
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ mutex_init(&buffer->lock);
+ INIT_LIST_HEAD(&buffer->attachments);
+ memcpy(buffer->name, import.name,
+ min(sizeof(buffer->name), sizeof(import.name) - 1));
+
+ /* Keep track of the buffer we created. */
+ buffer->vc_handle = result.res_handle;
+ buffer->size = import.size;
+ buffer->vpu_state = VPU_MAPPED;
+
+ buffer->imported_dma_buf = dma_buf;
+
+ buffer->attach = attach;
+ buffer->sgt = sgt;
+ buffer->dma_addr = dma_addr;
+ buffer->in_use = 1;
+ buffer->kernel_id = import.kernel_id;
+
+ /*
+ * We're done - we need to export a new dmabuf chaining through most
+ * functions, but enabling us to release our own internal references
+ * here.
+ */
+ exp_info.ops = &dma_buf_import_ops;
+ exp_info.size = import.size;
+ exp_info.flags = O_RDWR;
+ exp_info.priv = buffer;
+
+ buffer->dma_buf = dma_buf_export(&exp_info);
+ if (IS_ERR(buffer->dma_buf)) {
+ ret = PTR_ERR(buffer->dma_buf);
+ goto error;
+ }
+
+ vc_sm_add_resource(buffer);
+
+ *imported_buf = buffer->dma_buf;
+
+ return 0;
+
+error:
+ if (result.res_handle) {
+ struct vc_sm_free_t free = { result.res_handle, 0 };
+
+ vc_sm_cma_vchi_free(sm_state->sm_handle, &free,
+ &sm_state->int_trans_id);
+ }
+ free_kernel_id(import.kernel_id);
+ kfree(buffer);
+ if (sgt)
+ dma_buf_unmap_attachment(attach, sgt, DMA_BIDIRECTIONAL);
+ if (attach)
+ dma_buf_detach(dma_buf, attach);
+ dma_buf_put(dma_buf);
+ return ret;
+}
+
+static void
+vc_sm_vpu_event(struct sm_instance *instance, struct vc_sm_result_t *reply,
+ int reply_len)
+{
+ switch (reply->trans_id & ~0x80000000) {
+ case VC_SM_MSG_TYPE_CLIENT_VERSION:
+ {
+ /* Acknowledge that the firmware supports the version command */
+ pr_debug("%s: firmware acked version msg. Require release cb\n",
+ __func__);
+ sm_state->require_released_callback = true;
+ }
+ break;
+ case VC_SM_MSG_TYPE_RELEASED:
+ {
+ struct vc_sm_released *release = (struct vc_sm_released *)reply;
+ struct vc_sm_buffer *buffer =
+ lookup_kernel_id(release->kernel_id);
+ if (!buffer) {
+ pr_err("%s: VC released a buffer that is already released, kernel_id %d\n",
+ __func__, release->kernel_id);
+ break;
+ }
+ mutex_lock(&buffer->lock);
+
+ pr_debug("%s: Released addr %08x, size %u, id %08x, mem_handle %08x\n",
+ __func__, release->addr, release->size,
+ release->kernel_id, release->vc_handle);
+
+ buffer->vc_handle = 0;
+ buffer->vpu_state = VPU_NOT_MAPPED;
+ free_kernel_id(release->kernel_id);
+
+ vc_sm_release_resource(buffer);
+ }
+ break;
+ default:
+ pr_err("%s: Unknown vpu cmd %x\n", __func__, reply->trans_id);
+ break;
+ }
+}
+
+/* Driver load/unload functions */
+/* Videocore connected. */
+static void vc_sm_connected_init(void)
+{
+ int ret;
+ struct vc_sm_version version;
+ struct vc_sm_result_t version_result;
+
+ pr_info("[%s]: start\n", __func__);
+
+ /*
+ * Initialize and create a VCHI connection for the shared memory service
+ * running on videocore.
+ */
+ ret = vchiq_initialise(&sm_state->vchiq_instance);
+ if (ret) {
+ pr_err("[%s]: failed to initialise VCHI instance (ret=%d)\n",
+ __func__, ret);
+
+ return;
+ }
+
+ ret = vchiq_connect(sm_state->vchiq_instance);
+ if (ret) {
+ pr_err("[%s]: failed to connect VCHI instance (ret=%d)\n",
+ __func__, ret);
+
+ return;
+ }
+
+ /* Initialize an instance of the shared memory service. */
+ sm_state->sm_handle = vc_sm_cma_vchi_init(sm_state->vchiq_instance, 1,
+ vc_sm_vpu_event);
+ if (!sm_state->sm_handle) {
+ pr_err("[%s]: failed to initialize shared memory service\n",
+ __func__);
+
+ return;
+ }
+
+ /* Create a debug fs directory entry (root). */
+ sm_state->dir_root = debugfs_create_dir(VC_SM_DIR_ROOT_NAME, NULL);
+
+ sm_state->dir_state.show = &vc_sm_cma_global_state_show;
+ sm_state->dir_state.dir_entry =
+ debugfs_create_file(VC_SM_STATE, 0444, sm_state->dir_root,
+ &sm_state->dir_state,
+ &vc_sm_cma_debug_fs_fops);
+
+ INIT_LIST_HEAD(&sm_state->buffer_list);
+
+ version.version = 2;
+ ret = vc_sm_cma_vchi_client_version(sm_state->sm_handle, &version,
+ &version_result,
+ &sm_state->int_trans_id);
+ if (ret) {
+ pr_err("[%s]: Failed to send version request %d\n", __func__,
+ ret);
+ }
+
+ /* Done! */
+ sm_inited = 1;
+ pr_info("[%s]: installed successfully\n", __func__);
+}
+
+/* Driver loading. */
+static int bcm2835_vc_sm_cma_probe(struct platform_device *pdev)
+{
+ pr_info("%s: Videocore shared memory driver\n", __func__);
+
+ sm_state = devm_kzalloc(&pdev->dev, sizeof(*sm_state), GFP_KERNEL);
+ if (!sm_state)
+ return -ENOMEM;
+ sm_state->pdev = pdev;
+ mutex_init(&sm_state->map_lock);
+
+ spin_lock_init(&sm_state->kernelid_map_lock);
+ idr_init_base(&sm_state->kernelid_map, 1);
+
+ pdev->dev.dma_parms = devm_kzalloc(&pdev->dev,
+ sizeof(*pdev->dev.dma_parms),
+ GFP_KERNEL);
+ /* dma_set_max_seg_size checks if dma_parms is NULL. */
+ dma_set_max_seg_size(&pdev->dev, 0x3FFFFFFF);
+
+ vchiq_add_connected_callback(vc_sm_connected_init);
+ return 0;
+}
+
+/* Driver unloading. */
+static int bcm2835_vc_sm_cma_remove(struct platform_device *pdev)
+{
+ pr_debug("[%s]: start\n", __func__);
+ if (sm_inited) {
+ misc_deregister(&sm_state->misc_dev);
+
+ /* Remove all proc entries. */
+ debugfs_remove_recursive(sm_state->dir_root);
+
+ /* Stop the videocore shared memory service. */
+ vc_sm_cma_vchi_stop(sm_state->vchiq_instance, &sm_state->sm_handle);
+ }
+
+ if (sm_state) {
+ idr_destroy(&sm_state->kernelid_map);
+
+ /* Free the memory for the state structure. */
+ mutex_destroy(&sm_state->map_lock);
+ }
+
+ pr_debug("[%s]: end\n", __func__);
+ return 0;
+}
+
+/* Kernel API calls */
+/* Get an internal resource handle mapped from the external one. */
+int vc_sm_cma_int_handle(void *handle)
+{
+ struct dma_buf *dma_buf = (struct dma_buf *)handle;
+ struct vc_sm_buffer *buf;
+
+ /* Validate we can work with this device. */
+ if (!sm_state || !handle) {
+ pr_err("[%s]: invalid input\n", __func__);
+ return 0;
+ }
+
+ buf = (struct vc_sm_buffer *)dma_buf->priv;
+ return buf->vc_handle;
+}
+EXPORT_SYMBOL_GPL(vc_sm_cma_int_handle);
+
+/* Free a previously allocated shared memory handle and block. */
+int vc_sm_cma_free(void *handle)
+{
+ struct dma_buf *dma_buf = (struct dma_buf *)handle;
+
+ /* Validate we can work with this device. */
+ if (!sm_state || !handle) {
+ pr_err("[%s]: invalid input\n", __func__);
+ return -EPERM;
+ }
+
+ pr_debug("%s: handle %p/dmabuf %p\n", __func__, handle, dma_buf);
+
+ dma_buf_put(dma_buf);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(vc_sm_cma_free);
+
+/* Import a dmabuf to be shared with VC. */
+int vc_sm_cma_import_dmabuf(struct dma_buf *src_dmabuf, void **handle)
+{
+ struct dma_buf *new_dma_buf;
+ int ret;
+
+ /* Validate we can work with this device. */
+ if (!sm_state || !src_dmabuf || !handle) {
+ pr_err("[%s]: invalid input\n", __func__);
+ return -EPERM;
+ }
+
+ ret = vc_sm_cma_import_dmabuf_internal(sm_state, src_dmabuf,
+ -1, &new_dma_buf);
+
+ if (!ret) {
+ pr_debug("%s: imported to ptr %p\n", __func__, new_dma_buf);
+
+ /* Assign valid handle at this time.*/
+ *handle = new_dma_buf;
+ } else {
+ /*
+ * succeeded in importing the dma_buf, but then
+ * failed to look it up again. How?
+ * Release the fd again.
+ */
+ pr_err("%s: imported vc_sm_cma_get_buffer failed %d\n",
+ __func__, ret);
+ }
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(vc_sm_cma_import_dmabuf);
+
+static struct platform_driver bcm2835_vcsm_cma_driver = {
+ .probe = bcm2835_vc_sm_cma_probe,
+ .remove = bcm2835_vc_sm_cma_remove,
+ .driver = {
+ .name = "vcsm-cma",
+ .owner = THIS_MODULE,
+ },
+};
+
+module_platform_driver(bcm2835_vcsm_cma_driver);
+
+MODULE_AUTHOR("Dave Stevenson");
+MODULE_DESCRIPTION("VideoCore CMA Shared Memory Driver");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:vcsm-cma");
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
new file mode 100644
index 000000000000..61e110ec414d
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * VideoCore Shared Memory driver using CMA.
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ *
+ */
+
+#ifndef VC_SM_H
+#define VC_SM_H
+
+#define VC_SM_MAX_NAME_LEN 32
+
+enum vc_sm_vpu_mapping_state {
+ VPU_NOT_MAPPED,
+ VPU_MAPPED,
+ VPU_UNMAPPING
+};
+
+struct vc_sm_buffer {
+ struct list_head global_buffer_list; /* Global list of buffers. */
+
+ /* Index in the kernel_id idr so that we can find the
+ * mmal_msg_context again when servicing the VCHI reply.
+ */
+ int kernel_id;
+
+ size_t size;
+
+ /* Lock over all the following state for this buffer */
+ struct mutex lock;
+ struct list_head attachments;
+
+ char name[VC_SM_MAX_NAME_LEN];
+
+ int in_use:1; /* Kernel is still using this resource */
+
+ enum vc_sm_vpu_mapping_state vpu_state;
+ u32 vc_handle; /* VideoCore handle for this buffer */
+
+ /* DMABUF related fields */
+ struct dma_buf *dma_buf;
+ dma_addr_t dma_addr;
+ void *cookie;
+
+ struct vc_sm_privdata_t *private;
+
+ struct dma_buf *imported_dma_buf;
+ struct dma_buf_attachment *attach;
+ struct sg_table *sgt;
+};
+
+#endif
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
new file mode 100644
index 000000000000..c77ef0998a31
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
@@ -0,0 +1,507 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * VideoCore Shared Memory CMA allocator
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ * Copyright 2011-2012 Broadcom Corporation. All rights reserved.
+ *
+ * Based on vmcs_sm driver from Broadcom Corporation.
+ *
+ */
+
+/* ---- Include Files ----------------------------------------------------- */
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/semaphore.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+
+#include "vc_sm_cma_vchi.h"
+
+#define VC_SM_VER 1
+#define VC_SM_MIN_VER 0
+
+/* ---- Private Constants and Types -------------------------------------- */
+
+/* Command blocks come from a pool */
+#define SM_MAX_NUM_CMD_RSP_BLKS 32
+
+/* The number of supported connections */
+#define SM_MAX_NUM_CONNECTIONS 3
+
+struct sm_cmd_rsp_blk {
+ struct list_head head; /* To create lists */
+ /* To be signaled when the response is there */
+ struct completion cmplt;
+
+ u32 id;
+ u16 length;
+
+ u8 msg[VC_SM_MAX_MSG_LEN];
+
+ uint32_t wait:1;
+ uint32_t sent:1;
+ uint32_t alloc:1;
+
+};
+
+struct sm_instance {
+ u32 num_connections;
+ unsigned int service_handle[SM_MAX_NUM_CONNECTIONS];
+ struct task_struct *io_thread;
+ struct completion io_cmplt;
+
+ vpu_event_cb vpu_event;
+
+ /* Mutex over the following lists */
+ struct mutex lock;
+ u32 trans_id;
+ struct list_head cmd_list;
+ struct list_head rsp_list;
+ struct list_head dead_list;
+
+ struct sm_cmd_rsp_blk free_blk[SM_MAX_NUM_CMD_RSP_BLKS];
+
+ /* Mutex over the free_list */
+ struct mutex free_lock;
+ struct list_head free_list;
+
+ struct semaphore free_sema;
+ struct vchiq_instance *vchiq_instance;
+};
+
+/* ---- Private Variables ------------------------------------------------ */
+
+/* ---- Private Function Prototypes -------------------------------------- */
+
+/* ---- Private Functions ------------------------------------------------ */
+static int
+bcm2835_vchi_msg_queue(struct vchiq_instance *vchiq_instance, unsigned int handle,
+ void *data,
+ unsigned int size)
+{
+ return vchiq_queue_kernel_message(vchiq_instance, handle, data, size);
+}
+
+static struct
+sm_cmd_rsp_blk *vc_vchi_cmd_create(struct sm_instance *instance,
+ enum vc_sm_msg_type id, void *msg,
+ u32 size, int wait)
+{
+ struct sm_cmd_rsp_blk *blk;
+ struct vc_sm_msg_hdr_t *hdr;
+
+ if (down_interruptible(&instance->free_sema)) {
+ blk = kmalloc(sizeof(*blk), GFP_KERNEL);
+ if (!blk)
+ return NULL;
+
+ blk->alloc = 1;
+ init_completion(&blk->cmplt);
+ } else {
+ mutex_lock(&instance->free_lock);
+ blk =
+ list_first_entry(&instance->free_list,
+ struct sm_cmd_rsp_blk, head);
+ list_del(&blk->head);
+ mutex_unlock(&instance->free_lock);
+ }
+
+ blk->sent = 0;
+ blk->wait = wait;
+ blk->length = sizeof(*hdr) + size;
+
+ hdr = (struct vc_sm_msg_hdr_t *)blk->msg;
+ hdr->type = id;
+ mutex_lock(&instance->lock);
+ instance->trans_id++;
+ /*
+ * Retain the top bit for identifying asynchronous events, or VPU cmds.
+ */
+ instance->trans_id &= ~0x80000000;
+ hdr->trans_id = instance->trans_id;
+ blk->id = instance->trans_id;
+ mutex_unlock(&instance->lock);
+
+ if (size)
+ memcpy(hdr->body, msg, size);
+
+ return blk;
+}
+
+static void
+vc_vchi_cmd_delete(struct sm_instance *instance, struct sm_cmd_rsp_blk *blk)
+{
+ if (blk->alloc) {
+ kfree(blk);
+ return;
+ }
+
+ mutex_lock(&instance->free_lock);
+ list_add(&blk->head, &instance->free_list);
+ mutex_unlock(&instance->free_lock);
+ up(&instance->free_sema);
+}
+
+static void vc_sm_cma_vchi_rx_ack(struct sm_instance *instance,
+ struct sm_cmd_rsp_blk *cmd,
+ struct vc_sm_result_t *reply,
+ u32 reply_len)
+{
+ mutex_lock(&instance->lock);
+ list_for_each_entry(cmd,
+ &instance->rsp_list,
+ head) {
+ if (cmd->id == reply->trans_id)
+ break;
+ }
+ mutex_unlock(&instance->lock);
+
+ if (&cmd->head == &instance->rsp_list) {
+ //pr_debug("%s: received response %u, throw away...",
+ pr_err("%s: received response %u, throw away...",
+ __func__,
+ reply->trans_id);
+ } else if (reply_len > sizeof(cmd->msg)) {
+ pr_err("%s: reply too big (%u) %u, throw away...",
+ __func__, reply_len,
+ reply->trans_id);
+ } else {
+ memcpy(cmd->msg, reply,
+ reply_len);
+ complete(&cmd->cmplt);
+ }
+}
+
+static int vc_sm_cma_vchi_videocore_io(void *arg)
+{
+ struct sm_instance *instance = arg;
+ struct sm_cmd_rsp_blk *cmd = NULL, *cmd_tmp;
+ struct vc_sm_result_t *reply;
+ struct vchiq_header *header;
+ s32 status;
+ int svc_use = 1;
+
+ while (1) {
+ if (svc_use)
+ vchiq_release_service(instance->vchiq_instance,
+ instance->service_handle[0]);
+ svc_use = 0;
+
+ if (wait_for_completion_interruptible(&instance->io_cmplt))
+ continue;
+ vchiq_use_service(instance->vchiq_instance, instance->service_handle[0]);
+ svc_use = 1;
+
+ do {
+ /*
+ * Get new command and move it to response list
+ */
+ mutex_lock(&instance->lock);
+ if (list_empty(&instance->cmd_list)) {
+ /* no more commands to process */
+ mutex_unlock(&instance->lock);
+ break;
+ }
+ cmd = list_first_entry(&instance->cmd_list,
+ struct sm_cmd_rsp_blk, head);
+ list_move(&cmd->head, &instance->rsp_list);
+ cmd->sent = 1;
+ mutex_unlock(&instance->lock);
+ /* Send the command */
+ status =
+ bcm2835_vchi_msg_queue(instance->vchiq_instance,
+ instance->service_handle[0],
+ cmd->msg, cmd->length);
+ if (status) {
+ pr_err("%s: failed to queue message (%d)",
+ __func__, status);
+ }
+
+ /* If no reply is needed then we're done */
+ if (!cmd->wait) {
+ mutex_lock(&instance->lock);
+ list_del(&cmd->head);
+ mutex_unlock(&instance->lock);
+ vc_vchi_cmd_delete(instance, cmd);
+ continue;
+ }
+
+ if (status) {
+ complete(&cmd->cmplt);
+ continue;
+ }
+
+ } while (1);
+
+ while ((header = vchiq_msg_hold(instance->vchiq_instance,
+ instance->service_handle[0]))) {
+ reply = (struct vc_sm_result_t *)header->data;
+ if (reply->trans_id & 0x80000000) {
+ /* Async event or cmd from the VPU */
+ if (instance->vpu_event)
+ instance->vpu_event(instance, reply,
+ header->size);
+ } else {
+ vc_sm_cma_vchi_rx_ack(instance, cmd, reply,
+ header->size);
+ }
+
+ vchiq_release_message(instance->vchiq_instance,
+ instance->service_handle[0],
+ header);
+ }
+
+ /* Go through the dead list and free them */
+ mutex_lock(&instance->lock);
+ list_for_each_entry_safe(cmd, cmd_tmp, &instance->dead_list,
+ head) {
+ list_del(&cmd->head);
+ vc_vchi_cmd_delete(instance, cmd);
+ }
+ mutex_unlock(&instance->lock);
+ }
+
+ return 0;
+}
+
+static enum vchiq_status vc_sm_cma_vchi_callback(struct vchiq_instance *vchiq_instance,
+ enum vchiq_reason reason,
+ struct vchiq_header *header,
+ unsigned int handle, void *userdata)
+{
+ struct sm_instance *instance = vchiq_get_service_userdata(vchiq_instance, handle);
+
+ switch (reason) {
+ case VCHIQ_MESSAGE_AVAILABLE:
+ vchiq_msg_queue_push(vchiq_instance, handle, header);
+ complete(&instance->io_cmplt);
+ break;
+
+ case VCHIQ_SERVICE_CLOSED:
+ pr_info("%s: service CLOSED!!", __func__);
+ default:
+ break;
+ }
+
+ return VCHIQ_SUCCESS;
+}
+
+struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchiq_instance,
+ unsigned int num_connections,
+ vpu_event_cb vpu_event)
+{
+ u32 i;
+ struct sm_instance *instance;
+ int status;
+
+ pr_debug("%s: start", __func__);
+
+ if (num_connections > SM_MAX_NUM_CONNECTIONS) {
+ pr_err("%s: unsupported number of connections %u (max=%u)",
+ __func__, num_connections, SM_MAX_NUM_CONNECTIONS);
+
+ goto err_null;
+ }
+ /* Allocate memory for this instance */
+ instance = kzalloc(sizeof(*instance), GFP_KERNEL);
+
+ /* Misc initialisations */
+ mutex_init(&instance->lock);
+ init_completion(&instance->io_cmplt);
+ INIT_LIST_HEAD(&instance->cmd_list);
+ INIT_LIST_HEAD(&instance->rsp_list);
+ INIT_LIST_HEAD(&instance->dead_list);
+ INIT_LIST_HEAD(&instance->free_list);
+ sema_init(&instance->free_sema, SM_MAX_NUM_CMD_RSP_BLKS);
+ mutex_init(&instance->free_lock);
+ for (i = 0; i < SM_MAX_NUM_CMD_RSP_BLKS; i++) {
+ init_completion(&instance->free_blk[i].cmplt);
+ list_add(&instance->free_blk[i].head, &instance->free_list);
+ }
+
+ instance->vchiq_instance = vchiq_instance;
+
+ /* Open the VCHI service connections */
+ instance->num_connections = num_connections;
+ for (i = 0; i < num_connections; i++) {
+ struct vchiq_service_params_kernel params = {
+ .version = VC_SM_VER,
+ .version_min = VC_SM_MIN_VER,
+ .fourcc = VCHIQ_MAKE_FOURCC('S', 'M', 'E', 'M'),
+ .callback = vc_sm_cma_vchi_callback,
+ .userdata = instance,
+ };
+
+ status = vchiq_open_service(vchiq_instance, ¶ms,
+ &instance->service_handle[i]);
+ if (status) {
+ pr_err("%s: failed to open VCHI service (%d)",
+ __func__, status);
+
+ goto err_close_services;
+ }
+ }
+ /* Create the thread which takes care of all io to/from videoocore. */
+ instance->io_thread = kthread_create(&vc_sm_cma_vchi_videocore_io,
+ (void *)instance, "SMIO");
+ if (!instance->io_thread) {
+ pr_err("%s: failed to create SMIO thread", __func__);
+
+ goto err_close_services;
+ }
+ instance->vpu_event = vpu_event;
+ set_user_nice(instance->io_thread, -10);
+ wake_up_process(instance->io_thread);
+
+ pr_debug("%s: success - instance %p", __func__, instance);
+ return instance;
+
+err_close_services:
+ for (i = 0; i < instance->num_connections; i++) {
+ if (instance->service_handle[i])
+ vchiq_close_service(vchiq_instance, instance->service_handle[i]);
+ }
+ kfree(instance);
+err_null:
+ pr_debug("%s: FAILED", __func__);
+ return NULL;
+}
+
+int vc_sm_cma_vchi_stop(struct vchiq_instance *vchiq_instance, struct sm_instance **handle)
+{
+ struct sm_instance *instance;
+ u32 i;
+
+ if (!handle) {
+ pr_err("%s: invalid pointer to handle %p", __func__, handle);
+ goto lock;
+ }
+
+ if (!*handle) {
+ pr_err("%s: invalid handle %p", __func__, *handle);
+ goto lock;
+ }
+
+ instance = *handle;
+
+ /* Close all VCHI service connections */
+ for (i = 0; i < instance->num_connections; i++) {
+ vchiq_use_service(vchiq_instance, instance->service_handle[i]);
+ vchiq_close_service(vchiq_instance, instance->service_handle[i]);
+ }
+
+ kfree(instance);
+
+ *handle = NULL;
+ return 0;
+
+lock:
+ return -EINVAL;
+}
+
+static int vc_sm_cma_vchi_send_msg(struct sm_instance *handle,
+ enum vc_sm_msg_type msg_id, void *msg,
+ u32 msg_size, void *result, u32 result_size,
+ u32 *cur_trans_id, u8 wait_reply)
+{
+ int status = 0;
+ struct sm_instance *instance = handle;
+ struct sm_cmd_rsp_blk *cmd_blk;
+
+ if (!handle) {
+ pr_err("%s: invalid handle", __func__);
+ return -EINVAL;
+ }
+ if (!msg) {
+ pr_err("%s: invalid msg pointer", __func__);
+ return -EINVAL;
+ }
+
+ cmd_blk =
+ vc_vchi_cmd_create(instance, msg_id, msg, msg_size, wait_reply);
+ if (!cmd_blk) {
+ pr_err("[%s]: failed to allocate global tracking resource",
+ __func__);
+ return -ENOMEM;
+ }
+
+ if (cur_trans_id)
+ *cur_trans_id = cmd_blk->id;
+
+ mutex_lock(&instance->lock);
+ list_add_tail(&cmd_blk->head, &instance->cmd_list);
+ mutex_unlock(&instance->lock);
+ complete(&instance->io_cmplt);
+
+ if (!wait_reply)
+ /* We're done */
+ return 0;
+
+ /* Wait for the response */
+ if (wait_for_completion_interruptible(&cmd_blk->cmplt)) {
+ mutex_lock(&instance->lock);
+ if (!cmd_blk->sent) {
+ list_del(&cmd_blk->head);
+ mutex_unlock(&instance->lock);
+ vc_vchi_cmd_delete(instance, cmd_blk);
+ return -ENXIO;
+ }
+
+ list_move(&cmd_blk->head, &instance->dead_list);
+ mutex_unlock(&instance->lock);
+ complete(&instance->io_cmplt);
+ return -EINTR; /* We're done */
+ }
+
+ if (result && result_size) {
+ memcpy(result, cmd_blk->msg, result_size);
+ } else {
+ struct vc_sm_result_t *res =
+ (struct vc_sm_result_t *)cmd_blk->msg;
+ status = (res->success == 0) ? 0 : -ENXIO;
+ }
+
+ mutex_lock(&instance->lock);
+ list_del(&cmd_blk->head);
+ mutex_unlock(&instance->lock);
+ vc_vchi_cmd_delete(instance, cmd_blk);
+ return status;
+}
+
+int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg,
+ u32 *cur_trans_id)
+{
+ return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_FREE,
+ msg, sizeof(*msg), 0, 0, cur_trans_id, 0);
+}
+
+int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg,
+ struct vc_sm_import_result *result, u32 *cur_trans_id)
+{
+ return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_IMPORT,
+ msg, sizeof(*msg), result, sizeof(*result),
+ cur_trans_id, 1);
+}
+
+int vc_sm_cma_vchi_client_version(struct sm_instance *handle,
+ struct vc_sm_version *msg,
+ struct vc_sm_result_t *result,
+ u32 *cur_trans_id)
+{
+ return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_CLIENT_VERSION,
+ //msg, sizeof(*msg), result, sizeof(*result),
+ //cur_trans_id, 1);
+ msg, sizeof(*msg), NULL, 0,
+ cur_trans_id, 0);
+}
+
+int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle,
+ struct vc_sm_vc_mem_request_result *msg,
+ uint32_t *cur_trans_id)
+{
+ return vc_sm_cma_vchi_send_msg(handle,
+ VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY,
+ msg, sizeof(*msg), 0, 0, cur_trans_id,
+ 0);
+}
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
new file mode 100644
index 000000000000..a4f40d4cef05
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * VideoCore Shared Memory CMA allocator
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ * Copyright 2011-2012 Broadcom Corporation. All rights reserved.
+ *
+ * Based on vmcs_sm driver from Broadcom Corporation.
+ *
+ */
+
+#ifndef __VC_SM_CMA_VCHI_H__INCLUDED__
+#define __VC_SM_CMA_VCHI_H__INCLUDED__
+
+#include <linux/raspberrypi/vchiq.h>
+
+#include "vc_sm_defs.h"
+
+/*
+ * Forward declare.
+ */
+struct sm_instance;
+
+typedef void (*vpu_event_cb)(struct sm_instance *instance,
+ struct vc_sm_result_t *reply, int reply_len);
+
+/*
+ * Initialize the shared memory service, opens up vchi connection to talk to it.
+ */
+struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchi_instance,
+ unsigned int num_connections,
+ vpu_event_cb vpu_event);
+
+/*
+ * Terminates the shared memory service.
+ */
+int vc_sm_cma_vchi_stop(struct vchiq_instance *vchi_instance, struct sm_instance **handle);
+
+/*
+ * Ask the shared memory service to free up some memory that was previously
+ * allocated by the vc_sm_cma_vchi_alloc function call.
+ */
+int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg,
+ u32 *cur_trans_id);
+
+/*
+ * Import a contiguous block of memory and wrap it in a GPU MEM_HANDLE_T.
+ */
+int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg,
+ struct vc_sm_import_result *result,
+ u32 *cur_trans_id);
+
+int vc_sm_cma_vchi_client_version(struct sm_instance *handle,
+ struct vc_sm_version *msg,
+ struct vc_sm_result_t *result,
+ u32 *cur_trans_id);
+
+int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle,
+ struct vc_sm_vc_mem_request_result *msg,
+ uint32_t *cur_trans_id);
+
+#endif /* __VC_SM_CMA_VCHI_H__INCLUDED__ */
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
new file mode 100644
index 000000000000..ad4a3ec194d3
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
@@ -0,0 +1,187 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * VideoCore Shared Memory CMA allocator
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ *
+ * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation.
+ * All IPC messages are copied across to this file, even if the vc-sm-cma
+ * driver is not currently using them.
+ *
+ ****************************************************************************
+ */
+
+#ifndef __VC_SM_DEFS_H__INCLUDED__
+#define __VC_SM_DEFS_H__INCLUDED__
+
+/* Maximum message length */
+#define VC_SM_MAX_MSG_LEN (sizeof(union vc_sm_msg_union_t) + \
+ sizeof(struct vc_sm_msg_hdr_t))
+#define VC_SM_MAX_RSP_LEN (sizeof(union vc_sm_msg_union_t))
+
+/* Resource name maximum size */
+#define VC_SM_RESOURCE_NAME 32
+
+/*
+ * Version to be reported to the VPU
+ * VPU assumes 0 (aka 1) which does not require the released callback, nor
+ * expect the client to handle VC_MEM_REQUESTS.
+ * Version 2 requires the released callback, and must support VC_MEM_REQUESTS.
+ */
+#define VC_SM_PROTOCOL_VERSION 2
+
+enum vc_sm_msg_type {
+ /* Message types supported for HOST->VC direction */
+
+ /* Allocate shared memory block */
+ VC_SM_MSG_TYPE_ALLOC,
+ /* Lock allocated shared memory block */
+ VC_SM_MSG_TYPE_LOCK,
+ /* Unlock allocated shared memory block */
+ VC_SM_MSG_TYPE_UNLOCK,
+ /* Unlock allocated shared memory block, do not answer command */
+ VC_SM_MSG_TYPE_UNLOCK_NOANS,
+ /* Free shared memory block */
+ VC_SM_MSG_TYPE_FREE,
+ /* Resize a shared memory block */
+ VC_SM_MSG_TYPE_RESIZE,
+ /* Walk the allocated shared memory block(s) */
+ VC_SM_MSG_TYPE_WALK_ALLOC,
+
+ /* A previously applied action will need to be reverted */
+ VC_SM_MSG_TYPE_ACTION_CLEAN,
+
+ /*
+ * Import a physical address and wrap into a MEM_HANDLE_T.
+ * Release with VC_SM_MSG_TYPE_FREE.
+ */
+ VC_SM_MSG_TYPE_IMPORT,
+ /*
+ *Tells VC the protocol version supported by this client.
+ * 2 supports the async/cmd messages from the VPU for final release
+ * of memory, and for VC allocations.
+ */
+ VC_SM_MSG_TYPE_CLIENT_VERSION,
+ /* Response to VC request for memory */
+ VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY,
+
+ /*
+ * Asynchronous/cmd messages supported for VC->HOST direction.
+ * Signalled by setting the top bit in vc_sm_result_t trans_id.
+ */
+
+ /*
+ * VC has finished with an imported memory allocation.
+ * Release any Linux reference counts on the underlying block.
+ */
+ VC_SM_MSG_TYPE_RELEASED,
+ /* VC request for memory */
+ VC_SM_MSG_TYPE_VC_MEM_REQUEST,
+
+ VC_SM_MSG_TYPE_MAX
+};
+
+/* Type of memory to be allocated */
+enum vc_sm_alloc_type_t {
+ VC_SM_ALLOC_CACHED,
+ VC_SM_ALLOC_NON_CACHED,
+};
+
+/* Message header for all messages in HOST->VC direction */
+struct vc_sm_msg_hdr_t {
+ u32 type;
+ u32 trans_id;
+ u8 body[0];
+
+};
+
+/* Request to free a previously allocated memory (HOST->VC) */
+struct vc_sm_free_t {
+ /* Resource handle (returned from alloc) */
+ u32 res_handle;
+ /* Resource buffer (returned from alloc) */
+ u32 res_mem;
+
+};
+
+/* Generic result for a request (VC->HOST) */
+struct vc_sm_result_t {
+ /* Transaction identifier */
+ u32 trans_id;
+
+ s32 success;
+
+};
+
+/* Request to import memory (HOST->VC) */
+struct vc_sm_import {
+ /* type of memory to allocate */
+ enum vc_sm_alloc_type_t type;
+ /* pointer to the VC (ie physical) address of the allocated memory */
+ u32 addr;
+ /* size of buffer */
+ u32 size;
+ /* opaque handle returned in RELEASED messages */
+ u32 kernel_id;
+ /* Allocator identifier */
+ u32 allocator;
+ /* resource name (for easier tracking on vc side) */
+ char name[VC_SM_RESOURCE_NAME];
+};
+
+/* Result of a requested memory import (VC->HOST) */
+struct vc_sm_import_result {
+ /* Transaction identifier */
+ u32 trans_id;
+
+ /* Resource handle */
+ u32 res_handle;
+};
+
+/* Notification that VC has finished with an allocation (VC->HOST) */
+struct vc_sm_released {
+ /* cmd type / trans_id */
+ u32 cmd;
+
+ /* pointer to the VC (ie physical) address of the allocated memory */
+ u32 addr;
+ /* size of buffer */
+ u32 size;
+ /* opaque handle returned in RELEASED messages */
+ u32 kernel_id;
+ u32 vc_handle;
+};
+
+/*
+ * Client informing VC as to the protocol version it supports.
+ * >=2 requires the released callback, and supports VC asking for memory.
+ * Failure means that the firmware doesn't support this call, and therefore the
+ * client should either fail, or NOT rely on getting the released callback.
+ */
+struct vc_sm_version {
+ u32 version;
+};
+
+/* Response from the kernel to provide the VPU with some memory */
+struct vc_sm_vc_mem_request_result {
+ /* Transaction identifier for the VPU */
+ u32 trans_id;
+ /* pointer to the physical address of the allocated memory */
+ u32 addr;
+ /* opaque handle returned in RELEASED messages */
+ u32 kernel_id;
+};
+
+/* Union of ALL messages */
+union vc_sm_msg_union_t {
+ struct vc_sm_free_t free;
+ struct vc_sm_result_t result;
+ struct vc_sm_import import;
+ struct vc_sm_import_result import_result;
+ struct vc_sm_version version;
+ struct vc_sm_released released;
+ struct vc_sm_vc_mem_request_result vc_request_result;
+};
+
+#endif /* __VC_SM_DEFS_H__INCLUDED__ */
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
new file mode 100644
index 000000000000..988fdd967922
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * VideoCore Shared Memory CMA allocator
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ *
+ * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation.
+ *
+ */
+
+#ifndef __VC_SM_KNL_H__INCLUDED__
+#define __VC_SM_KNL_H__INCLUDED__
+
+#if !defined(__KERNEL__)
+#error "This interface is for kernel use only..."
+#endif
+
+/* Free a previously allocated or imported shared memory handle and block. */
+int vc_sm_cma_free(void *handle);
+
+/* Get an internal resource handle mapped from the external one. */
+int vc_sm_cma_int_handle(void *handle);
+
+/* Import a block of memory into the GPU space. */
+int vc_sm_cma_import_dmabuf(struct dma_buf *dmabuf, void **handle);
+
+#endif /* __VC_SM_KNL_H__INCLUDED__ */
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 01/14] staging: vc04_services: Add new vc-sm-cma driver
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
Add Broadcom VideoCore Shared Memory support.
This new driver allows contiguous memory blocks to be imported
into the VideoCore VPU memory map, and manages the lifetime of
those objects, only releasing the source dmabuf once the VPU has
confirmed it has finished with it.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
drivers/staging/vc04_services/Kconfig | 2 +
drivers/staging/vc04_services/Makefile | 1 +
.../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
.../staging/vc04_services/vc-sm-cma/Makefile | 12 +
.../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++++++++++++
.../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 ++
.../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++++++++
.../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 ++
.../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++++
.../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
10 files changed, 1665 insertions(+)
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig
index 31e58c9d1a11..6c0e77d64376 100644
--- a/drivers/staging/vc04_services/Kconfig
+++ b/drivers/staging/vc04_services/Kconfig
@@ -46,5 +46,7 @@ source "drivers/staging/vc04_services/bcm2835-camera/Kconfig"
source "drivers/staging/vc04_services/vchiq-mmal/Kconfig"
+source "drivers/staging/vc04_services/vc-sm-cma/Kconfig"
+
endif
diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile
index 1fd191e2e2a5..01089f369fb4 100644
--- a/drivers/staging/vc04_services/Makefile
+++ b/drivers/staging/vc04_services/Makefile
@@ -14,6 +14,7 @@ endif
obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/
obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/
obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/
+obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/
ccflags-y += -I $(srctree)/$(src)/include
diff --git a/drivers/staging/vc04_services/vc-sm-cma/Kconfig b/drivers/staging/vc04_services/vc-sm-cma/Kconfig
new file mode 100644
index 000000000000..bbd296f5826b
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/Kconfig
@@ -0,0 +1,10 @@
+config BCM_VC_SM_CMA
+ tristate "VideoCore Shared Memory (CMA) driver"
+ depends on BCM2835_VCHIQ
+ select RBTREE
+ select DMA_SHARED_BUFFER
+ help
+ Say Y here to enable the shared memory interface that
+ supports sharing dmabufs with VideoCore.
+ This operates over the VCHIQ interface to a service
+ running on VideoCore.
diff --git a/drivers/staging/vc04_services/vc-sm-cma/Makefile b/drivers/staging/vc04_services/vc-sm-cma/Makefile
new file mode 100644
index 000000000000..c92a5775c62e
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/Makefile
@@ -0,0 +1,12 @@
+ccflags-y += \
+ -I$(srctree)/$(src)/../ \
+ -I$(srctree)/$(src)/../interface/vchiq_arm\
+ -I$(srctree)/$(src)/../include
+
+ccflags-y += \
+ -D__VCCOREVER__=0
+
+vc-sm-cma-$(CONFIG_BCM_VC_SM_CMA) := \
+ vc_sm.o vc_sm_cma_vchi.o
+
+obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma.o
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
new file mode 100644
index 000000000000..7fe81d259c7b
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
@@ -0,0 +1,801 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * VideoCore Shared Memory driver using CMA.
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ * Dave Stevenson <dave.stevenson@raspberrypi.com>
+ *
+ * Based on vmcs_sm driver from Broadcom Corporation for some API,
+ * and taking some code for buffer allocation and dmabuf handling from
+ * videobuf2.
+ *
+ * This driver handles taking a dmabuf, mapping it into the VPU address space,
+ * and returning a handle that the VPU can use.
+ */
+
+/* ---- Include Files ----------------------------------------------------- */
+#include <linux/debugfs.h>
+#include <linux/dma-buf.h>
+#include <linux/errno.h>
+#include <linux/miscdevice.h>
+#include <linux/platform_device.h>
+#include <linux/syscalls.h>
+
+#include "vchiq_connected.h"
+#include "vc_sm_cma_vchi.h"
+
+#include "vc_sm.h"
+#include "vc_sm_knl.h"
+
+MODULE_IMPORT_NS(DMA_BUF);
+
+/* ---- Private Constants and Types --------------------------------------- */
+
+#define VC_SM_RESOURCE_NAME_DEFAULT "sm-host-resource"
+
+#define VC_SM_DIR_ROOT_NAME "vcsm-cma"
+#define VC_SM_STATE "state"
+
+/* Private file data associated with each opened device. */
+struct vc_sm_privdata_t {
+};
+
+typedef int (*VC_SM_SHOW) (struct seq_file *s, void *v);
+struct sm_pde_t {
+ VC_SM_SHOW show; /* Debug fs function hookup. */
+ struct dentry *dir_entry; /* Debug fs directory entry. */
+ void *priv_data; /* Private data */
+};
+
+/* Global state information. */
+struct sm_state_t {
+ struct platform_device *pdev;
+
+ struct miscdevice misc_dev;
+
+ struct sm_instance *sm_handle; /* Handle for videocore service. */
+
+ spinlock_t kernelid_map_lock; /* Spinlock protecting kernelid_map */
+ struct idr kernelid_map;
+
+ struct mutex map_lock; /* Global map lock. */
+ struct list_head buffer_list; /* List of buffer. */
+
+ struct dentry *dir_root; /* Debug fs entries root. */
+ struct sm_pde_t dir_state; /* Debug fs entries state sub-tree. */
+
+ bool require_released_callback; /* VPU will send a released msg when it
+ * has finished with a resource.
+ */
+ /* State for transactions */
+ int restart_sys; /* Tracks restart on interrupt. */
+ enum vc_sm_msg_type int_action; /* Interrupted action. */
+
+ u32 int_trans_id; /* Interrupted transaction. */
+ struct vchiq_instance *vchiq_instance;
+};
+
+struct vc_sm_dma_buf_attachment {
+ struct device *dev;
+ struct sg_table sg_table;
+ struct list_head list;
+ enum dma_data_direction dma_dir;
+};
+
+/* ---- Private Variables ----------------------------------------------- */
+
+static struct sm_state_t *sm_state;
+static int sm_inited;
+
+/* ---- Private Function Prototypes -------------------------------------- */
+
+/* ---- Private Functions ------------------------------------------------ */
+
+static int get_kernel_id(struct vc_sm_buffer *buffer)
+{
+ int handle;
+
+ spin_lock(&sm_state->kernelid_map_lock);
+ handle = idr_alloc(&sm_state->kernelid_map, buffer, 0, 0, GFP_KERNEL);
+ spin_unlock(&sm_state->kernelid_map_lock);
+
+ return handle;
+}
+
+static struct vc_sm_buffer *lookup_kernel_id(int handle)
+{
+ return idr_find(&sm_state->kernelid_map, handle);
+}
+
+static void free_kernel_id(int handle)
+{
+ spin_lock(&sm_state->kernelid_map_lock);
+ idr_remove(&sm_state->kernelid_map, handle);
+ spin_unlock(&sm_state->kernelid_map_lock);
+}
+
+static int vc_sm_cma_seq_file_show(struct seq_file *s, void *v)
+{
+ struct sm_pde_t *sm_pde;
+
+ sm_pde = (struct sm_pde_t *)(s->private);
+
+ if (sm_pde && sm_pde->show)
+ sm_pde->show(s, v);
+
+ return 0;
+}
+
+static int vc_sm_cma_single_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, vc_sm_cma_seq_file_show, inode->i_private);
+}
+
+static const struct file_operations vc_sm_cma_debug_fs_fops = {
+ .open = vc_sm_cma_single_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static int vc_sm_cma_global_state_show(struct seq_file *s, void *v)
+{
+ struct vc_sm_buffer *resource = NULL;
+ int resource_count = 0;
+
+ if (!sm_state)
+ return 0;
+
+ seq_printf(s, "\nVC-ServiceHandle %p\n", sm_state->sm_handle);
+
+ /* Log all applicable mapping(s). */
+
+ mutex_lock(&sm_state->map_lock);
+ seq_puts(s, "\nResources\n");
+ if (!list_empty(&sm_state->buffer_list)) {
+ list_for_each_entry(resource, &sm_state->buffer_list,
+ global_buffer_list) {
+ resource_count++;
+
+ seq_printf(s, "\nResource %p\n",
+ resource);
+ seq_printf(s, " NAME %s\n",
+ resource->name);
+ seq_printf(s, " SIZE %zu\n",
+ resource->size);
+ seq_printf(s, " DMABUF %p\n",
+ resource->dma_buf);
+ seq_printf(s, " IMPORTED_DMABUF %p\n",
+ resource->imported_dma_buf);
+ seq_printf(s, " ATTACH %p\n",
+ resource->attach);
+ seq_printf(s, " SGT %p\n",
+ resource->sgt);
+ seq_printf(s, " DMA_ADDR %pad\n",
+ &resource->dma_addr);
+ seq_printf(s, " VC_HANDLE %08x\n",
+ resource->vc_handle);
+ seq_printf(s, " VC_MAPPING %d\n",
+ resource->vpu_state);
+ }
+ }
+ seq_printf(s, "\n\nTotal resource count: %d\n\n", resource_count);
+
+ mutex_unlock(&sm_state->map_lock);
+
+ return 0;
+}
+
+/*
+ * Adds a buffer to the private data list which tracks all the allocated
+ * data.
+ */
+static void vc_sm_add_resource(struct vc_sm_buffer *buffer)
+{
+ mutex_lock(&sm_state->map_lock);
+ list_add(&buffer->global_buffer_list, &sm_state->buffer_list);
+ mutex_unlock(&sm_state->map_lock);
+
+ pr_debug("[%s]: added buffer %p (name %s, size %zu)\n",
+ __func__, buffer, buffer->name, buffer->size);
+}
+
+/*
+ * Cleans up imported dmabuf.
+ * Should be called with mutex held.
+ */
+static void vc_sm_clean_up_dmabuf(struct vc_sm_buffer *buffer)
+{
+ /* Handle cleaning up imported dmabufs */
+ if (buffer->sgt) {
+ dma_buf_unmap_attachment(buffer->attach,
+ buffer->sgt,
+ DMA_BIDIRECTIONAL);
+ buffer->sgt = NULL;
+ }
+ if (buffer->attach) {
+ dma_buf_detach(buffer->dma_buf, buffer->attach);
+ buffer->attach = NULL;
+ }
+}
+
+/*
+ * Instructs VPU to decrement the refcount on a buffer.
+ */
+static void vc_sm_vpu_free(struct vc_sm_buffer *buffer)
+{
+ if (buffer->vc_handle && buffer->vpu_state == VPU_MAPPED) {
+ struct vc_sm_free_t free = { buffer->vc_handle, 0 };
+ int status = vc_sm_cma_vchi_free(sm_state->sm_handle, &free,
+ &sm_state->int_trans_id);
+ if (status != 0 && status != -EINTR) {
+ pr_err("[%s]: failed to free memory on videocore (status: %u, trans_id: %u)\n",
+ __func__, status, sm_state->int_trans_id);
+ }
+
+ if (sm_state->require_released_callback) {
+ /* Need to wait for the VPU to confirm the free. */
+
+ /* Retain a reference on this until the VPU has
+ * released it
+ */
+ buffer->vpu_state = VPU_UNMAPPING;
+ } else {
+ buffer->vpu_state = VPU_NOT_MAPPED;
+ buffer->vc_handle = 0;
+ }
+ }
+}
+
+/*
+ * Release an allocation.
+ * All refcounting is done via the dma buf object.
+ *
+ * Must be called with the mutex held. The function will either release the
+ * mutex (if defering the release) or destroy it. The caller must therefore not
+ * reuse the buffer on return.
+ */
+static void vc_sm_release_resource(struct vc_sm_buffer *buffer)
+{
+ pr_debug("[%s]: buffer %p (name %s, size %zu)\n",
+ __func__, buffer, buffer->name, buffer->size);
+
+ if (buffer->vc_handle) {
+ /* We've sent the unmap request but not had the response. */
+ pr_debug("[%s]: Waiting for VPU unmap response on %p\n",
+ __func__, buffer);
+ goto defer;
+ }
+ if (buffer->in_use) {
+ /* dmabuf still in use - we await the release */
+ pr_debug("[%s]: buffer %p is still in use\n", __func__, buffer);
+ goto defer;
+ }
+
+ /* Release the allocation */
+ if (buffer->imported_dma_buf)
+ dma_buf_put(buffer->imported_dma_buf);
+ else
+ pr_err("%s: Imported dmabuf already been put for buf %p\n",
+ __func__, buffer);
+ buffer->imported_dma_buf = NULL;
+
+ /* Free our buffer. Start by removing it from the list */
+ mutex_lock(&sm_state->map_lock);
+ list_del(&buffer->global_buffer_list);
+ mutex_unlock(&sm_state->map_lock);
+
+ pr_debug("%s: Release our allocation - done\n", __func__);
+ mutex_unlock(&buffer->lock);
+
+ mutex_destroy(&buffer->lock);
+
+ kfree(buffer);
+ return;
+
+defer:
+ mutex_unlock(&buffer->lock);
+}
+
+/* Dma_buf operations for chaining through to an imported dma_buf */
+
+static void vc_sm_dma_buf_release(struct dma_buf *dmabuf)
+{
+ struct vc_sm_buffer *buffer;
+
+ if (!dmabuf)
+ return;
+
+ buffer = (struct vc_sm_buffer *)dmabuf->priv;
+
+ mutex_lock(&buffer->lock);
+
+ pr_debug("%s dmabuf %p, buffer %p\n", __func__, dmabuf, buffer);
+
+ buffer->in_use = 0;
+
+ /* Unmap on the VPU */
+ vc_sm_vpu_free(buffer);
+ pr_debug("%s vpu_free done\n", __func__);
+
+ /* Unmap our dma_buf object (the vc_sm_buffer remains until released
+ * on the VPU).
+ */
+ vc_sm_clean_up_dmabuf(buffer);
+ pr_debug("%s clean_up dmabuf done\n", __func__);
+
+ /* buffer->lock will be destroyed by vc_sm_release_resource if finished
+ * with, otherwise unlocked. Do NOT unlock here.
+ */
+ vc_sm_release_resource(buffer);
+ pr_debug("%s done\n", __func__);
+}
+
+static
+int vc_sm_import_dma_buf_attach(struct dma_buf *dmabuf,
+ struct dma_buf_attachment *attachment)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ return buf->imported_dma_buf->ops->attach(buf->imported_dma_buf,
+ attachment);
+}
+
+static
+void vc_sm_import_dma_buf_detatch(struct dma_buf *dmabuf,
+ struct dma_buf_attachment *attachment)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ buf->imported_dma_buf->ops->detach(buf->imported_dma_buf, attachment);
+}
+
+static
+struct sg_table *vc_sm_import_map_dma_buf(struct dma_buf_attachment *attachment,
+ enum dma_data_direction direction)
+{
+ struct vc_sm_buffer *buf = attachment->dmabuf->priv;
+
+ return buf->imported_dma_buf->ops->map_dma_buf(attachment,
+ direction);
+}
+
+static
+void vc_sm_import_unmap_dma_buf(struct dma_buf_attachment *attachment,
+ struct sg_table *table,
+ enum dma_data_direction direction)
+{
+ struct vc_sm_buffer *buf = attachment->dmabuf->priv;
+
+ buf->imported_dma_buf->ops->unmap_dma_buf(attachment, table, direction);
+}
+
+static
+int vc_sm_import_dmabuf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ pr_debug("%s: mmap dma_buf %p, buf %p, imported db %p\n", __func__,
+ dmabuf, buf, buf->imported_dma_buf);
+
+ return buf->imported_dma_buf->ops->mmap(buf->imported_dma_buf, vma);
+}
+
+static
+int vc_sm_import_dma_buf_begin_cpu_access(struct dma_buf *dmabuf,
+ enum dma_data_direction direction)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ return buf->imported_dma_buf->ops->begin_cpu_access(buf->imported_dma_buf,
+ direction);
+}
+
+static
+int vc_sm_import_dma_buf_end_cpu_access(struct dma_buf *dmabuf,
+ enum dma_data_direction direction)
+{
+ struct vc_sm_buffer *buf = dmabuf->priv;
+
+ return buf->imported_dma_buf->ops->end_cpu_access(buf->imported_dma_buf,
+ direction);
+}
+
+static const struct dma_buf_ops dma_buf_import_ops = {
+ .map_dma_buf = vc_sm_import_map_dma_buf,
+ .unmap_dma_buf = vc_sm_import_unmap_dma_buf,
+ .mmap = vc_sm_import_dmabuf_mmap,
+ .release = vc_sm_dma_buf_release,
+ .attach = vc_sm_import_dma_buf_attach,
+ .detach = vc_sm_import_dma_buf_detatch,
+ .begin_cpu_access = vc_sm_import_dma_buf_begin_cpu_access,
+ .end_cpu_access = vc_sm_import_dma_buf_end_cpu_access,
+};
+
+/* Import a dma_buf to be shared with VC. */
+int
+vc_sm_cma_import_dmabuf_internal(struct sm_state_t *state,
+ struct dma_buf *dma_buf,
+ int fd,
+ struct dma_buf **imported_buf)
+{
+ DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
+ struct vc_sm_buffer *buffer = NULL;
+ struct vc_sm_import import = { };
+ struct vc_sm_import_result result = { };
+ struct dma_buf_attachment *attach = NULL;
+ struct sg_table *sgt = NULL;
+ dma_addr_t dma_addr;
+ int ret = 0;
+ int status;
+
+ /* Setup our allocation parameters */
+ pr_debug("%s: importing dma_buf %p/fd %d\n", __func__, dma_buf, fd);
+
+ if (fd < 0)
+ get_dma_buf(dma_buf);
+ else
+ dma_buf = dma_buf_get(fd);
+
+ if (!dma_buf)
+ return -EINVAL;
+
+ attach = dma_buf_attach(dma_buf, &sm_state->pdev->dev);
+ if (IS_ERR(attach)) {
+ ret = PTR_ERR(attach);
+ goto error;
+ }
+
+ sgt = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL);
+ if (IS_ERR(sgt)) {
+ ret = PTR_ERR(sgt);
+ goto error;
+ }
+
+ /* Verify that the address block is contiguous */
+ if (sgt->nents != 1) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ /* Allocate local buffer to track this allocation. */
+ buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
+ if (!buffer) {
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ import.type = VC_SM_ALLOC_NON_CACHED;
+ dma_addr = sg_dma_address(sgt->sgl);
+ import.addr = (u32)dma_addr;
+ if ((import.addr & 0xC0000000) != 0xC0000000) {
+ pr_err("%s: Expecting an uncached alias for dma_addr %pad\n",
+ __func__, &dma_addr);
+ import.addr |= 0xC0000000;
+ }
+ import.size = sg_dma_len(sgt->sgl);
+ import.allocator = current->tgid;
+ import.kernel_id = get_kernel_id(buffer);
+
+ memcpy(import.name, VC_SM_RESOURCE_NAME_DEFAULT,
+ sizeof(VC_SM_RESOURCE_NAME_DEFAULT));
+
+ pr_debug("[%s]: attempt to import \"%s\" data - type %u, addr %pad, size %u.\n",
+ __func__, import.name, import.type, &dma_addr, import.size);
+
+ /* Allocate the videocore buffer. */
+ status = vc_sm_cma_vchi_import(sm_state->sm_handle, &import, &result,
+ &sm_state->int_trans_id);
+ if (status == -EINTR) {
+ pr_debug("[%s]: requesting import memory action restart (trans_id: %u)\n",
+ __func__, sm_state->int_trans_id);
+ ret = -ERESTARTSYS;
+ sm_state->restart_sys = -EINTR;
+ sm_state->int_action = VC_SM_MSG_TYPE_IMPORT;
+ goto error;
+ } else if (status || !result.res_handle) {
+ pr_debug("[%s]: failed to import memory on videocore (status: %u, trans_id: %u)\n",
+ __func__, status, sm_state->int_trans_id);
+ ret = -ENOMEM;
+ goto error;
+ }
+
+ mutex_init(&buffer->lock);
+ INIT_LIST_HEAD(&buffer->attachments);
+ memcpy(buffer->name, import.name,
+ min(sizeof(buffer->name), sizeof(import.name) - 1));
+
+ /* Keep track of the buffer we created. */
+ buffer->vc_handle = result.res_handle;
+ buffer->size = import.size;
+ buffer->vpu_state = VPU_MAPPED;
+
+ buffer->imported_dma_buf = dma_buf;
+
+ buffer->attach = attach;
+ buffer->sgt = sgt;
+ buffer->dma_addr = dma_addr;
+ buffer->in_use = 1;
+ buffer->kernel_id = import.kernel_id;
+
+ /*
+ * We're done - we need to export a new dmabuf chaining through most
+ * functions, but enabling us to release our own internal references
+ * here.
+ */
+ exp_info.ops = &dma_buf_import_ops;
+ exp_info.size = import.size;
+ exp_info.flags = O_RDWR;
+ exp_info.priv = buffer;
+
+ buffer->dma_buf = dma_buf_export(&exp_info);
+ if (IS_ERR(buffer->dma_buf)) {
+ ret = PTR_ERR(buffer->dma_buf);
+ goto error;
+ }
+
+ vc_sm_add_resource(buffer);
+
+ *imported_buf = buffer->dma_buf;
+
+ return 0;
+
+error:
+ if (result.res_handle) {
+ struct vc_sm_free_t free = { result.res_handle, 0 };
+
+ vc_sm_cma_vchi_free(sm_state->sm_handle, &free,
+ &sm_state->int_trans_id);
+ }
+ free_kernel_id(import.kernel_id);
+ kfree(buffer);
+ if (sgt)
+ dma_buf_unmap_attachment(attach, sgt, DMA_BIDIRECTIONAL);
+ if (attach)
+ dma_buf_detach(dma_buf, attach);
+ dma_buf_put(dma_buf);
+ return ret;
+}
+
+static void
+vc_sm_vpu_event(struct sm_instance *instance, struct vc_sm_result_t *reply,
+ int reply_len)
+{
+ switch (reply->trans_id & ~0x80000000) {
+ case VC_SM_MSG_TYPE_CLIENT_VERSION:
+ {
+ /* Acknowledge that the firmware supports the version command */
+ pr_debug("%s: firmware acked version msg. Require release cb\n",
+ __func__);
+ sm_state->require_released_callback = true;
+ }
+ break;
+ case VC_SM_MSG_TYPE_RELEASED:
+ {
+ struct vc_sm_released *release = (struct vc_sm_released *)reply;
+ struct vc_sm_buffer *buffer =
+ lookup_kernel_id(release->kernel_id);
+ if (!buffer) {
+ pr_err("%s: VC released a buffer that is already released, kernel_id %d\n",
+ __func__, release->kernel_id);
+ break;
+ }
+ mutex_lock(&buffer->lock);
+
+ pr_debug("%s: Released addr %08x, size %u, id %08x, mem_handle %08x\n",
+ __func__, release->addr, release->size,
+ release->kernel_id, release->vc_handle);
+
+ buffer->vc_handle = 0;
+ buffer->vpu_state = VPU_NOT_MAPPED;
+ free_kernel_id(release->kernel_id);
+
+ vc_sm_release_resource(buffer);
+ }
+ break;
+ default:
+ pr_err("%s: Unknown vpu cmd %x\n", __func__, reply->trans_id);
+ break;
+ }
+}
+
+/* Driver load/unload functions */
+/* Videocore connected. */
+static void vc_sm_connected_init(void)
+{
+ int ret;
+ struct vc_sm_version version;
+ struct vc_sm_result_t version_result;
+
+ pr_info("[%s]: start\n", __func__);
+
+ /*
+ * Initialize and create a VCHI connection for the shared memory service
+ * running on videocore.
+ */
+ ret = vchiq_initialise(&sm_state->vchiq_instance);
+ if (ret) {
+ pr_err("[%s]: failed to initialise VCHI instance (ret=%d)\n",
+ __func__, ret);
+
+ return;
+ }
+
+ ret = vchiq_connect(sm_state->vchiq_instance);
+ if (ret) {
+ pr_err("[%s]: failed to connect VCHI instance (ret=%d)\n",
+ __func__, ret);
+
+ return;
+ }
+
+ /* Initialize an instance of the shared memory service. */
+ sm_state->sm_handle = vc_sm_cma_vchi_init(sm_state->vchiq_instance, 1,
+ vc_sm_vpu_event);
+ if (!sm_state->sm_handle) {
+ pr_err("[%s]: failed to initialize shared memory service\n",
+ __func__);
+
+ return;
+ }
+
+ /* Create a debug fs directory entry (root). */
+ sm_state->dir_root = debugfs_create_dir(VC_SM_DIR_ROOT_NAME, NULL);
+
+ sm_state->dir_state.show = &vc_sm_cma_global_state_show;
+ sm_state->dir_state.dir_entry =
+ debugfs_create_file(VC_SM_STATE, 0444, sm_state->dir_root,
+ &sm_state->dir_state,
+ &vc_sm_cma_debug_fs_fops);
+
+ INIT_LIST_HEAD(&sm_state->buffer_list);
+
+ version.version = 2;
+ ret = vc_sm_cma_vchi_client_version(sm_state->sm_handle, &version,
+ &version_result,
+ &sm_state->int_trans_id);
+ if (ret) {
+ pr_err("[%s]: Failed to send version request %d\n", __func__,
+ ret);
+ }
+
+ /* Done! */
+ sm_inited = 1;
+ pr_info("[%s]: installed successfully\n", __func__);
+}
+
+/* Driver loading. */
+static int bcm2835_vc_sm_cma_probe(struct platform_device *pdev)
+{
+ pr_info("%s: Videocore shared memory driver\n", __func__);
+
+ sm_state = devm_kzalloc(&pdev->dev, sizeof(*sm_state), GFP_KERNEL);
+ if (!sm_state)
+ return -ENOMEM;
+ sm_state->pdev = pdev;
+ mutex_init(&sm_state->map_lock);
+
+ spin_lock_init(&sm_state->kernelid_map_lock);
+ idr_init_base(&sm_state->kernelid_map, 1);
+
+ pdev->dev.dma_parms = devm_kzalloc(&pdev->dev,
+ sizeof(*pdev->dev.dma_parms),
+ GFP_KERNEL);
+ /* dma_set_max_seg_size checks if dma_parms is NULL. */
+ dma_set_max_seg_size(&pdev->dev, 0x3FFFFFFF);
+
+ vchiq_add_connected_callback(vc_sm_connected_init);
+ return 0;
+}
+
+/* Driver unloading. */
+static int bcm2835_vc_sm_cma_remove(struct platform_device *pdev)
+{
+ pr_debug("[%s]: start\n", __func__);
+ if (sm_inited) {
+ misc_deregister(&sm_state->misc_dev);
+
+ /* Remove all proc entries. */
+ debugfs_remove_recursive(sm_state->dir_root);
+
+ /* Stop the videocore shared memory service. */
+ vc_sm_cma_vchi_stop(sm_state->vchiq_instance, &sm_state->sm_handle);
+ }
+
+ if (sm_state) {
+ idr_destroy(&sm_state->kernelid_map);
+
+ /* Free the memory for the state structure. */
+ mutex_destroy(&sm_state->map_lock);
+ }
+
+ pr_debug("[%s]: end\n", __func__);
+ return 0;
+}
+
+/* Kernel API calls */
+/* Get an internal resource handle mapped from the external one. */
+int vc_sm_cma_int_handle(void *handle)
+{
+ struct dma_buf *dma_buf = (struct dma_buf *)handle;
+ struct vc_sm_buffer *buf;
+
+ /* Validate we can work with this device. */
+ if (!sm_state || !handle) {
+ pr_err("[%s]: invalid input\n", __func__);
+ return 0;
+ }
+
+ buf = (struct vc_sm_buffer *)dma_buf->priv;
+ return buf->vc_handle;
+}
+EXPORT_SYMBOL_GPL(vc_sm_cma_int_handle);
+
+/* Free a previously allocated shared memory handle and block. */
+int vc_sm_cma_free(void *handle)
+{
+ struct dma_buf *dma_buf = (struct dma_buf *)handle;
+
+ /* Validate we can work with this device. */
+ if (!sm_state || !handle) {
+ pr_err("[%s]: invalid input\n", __func__);
+ return -EPERM;
+ }
+
+ pr_debug("%s: handle %p/dmabuf %p\n", __func__, handle, dma_buf);
+
+ dma_buf_put(dma_buf);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(vc_sm_cma_free);
+
+/* Import a dmabuf to be shared with VC. */
+int vc_sm_cma_import_dmabuf(struct dma_buf *src_dmabuf, void **handle)
+{
+ struct dma_buf *new_dma_buf;
+ int ret;
+
+ /* Validate we can work with this device. */
+ if (!sm_state || !src_dmabuf || !handle) {
+ pr_err("[%s]: invalid input\n", __func__);
+ return -EPERM;
+ }
+
+ ret = vc_sm_cma_import_dmabuf_internal(sm_state, src_dmabuf,
+ -1, &new_dma_buf);
+
+ if (!ret) {
+ pr_debug("%s: imported to ptr %p\n", __func__, new_dma_buf);
+
+ /* Assign valid handle at this time.*/
+ *handle = new_dma_buf;
+ } else {
+ /*
+ * succeeded in importing the dma_buf, but then
+ * failed to look it up again. How?
+ * Release the fd again.
+ */
+ pr_err("%s: imported vc_sm_cma_get_buffer failed %d\n",
+ __func__, ret);
+ }
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(vc_sm_cma_import_dmabuf);
+
+static struct platform_driver bcm2835_vcsm_cma_driver = {
+ .probe = bcm2835_vc_sm_cma_probe,
+ .remove = bcm2835_vc_sm_cma_remove,
+ .driver = {
+ .name = "vcsm-cma",
+ .owner = THIS_MODULE,
+ },
+};
+
+module_platform_driver(bcm2835_vcsm_cma_driver);
+
+MODULE_AUTHOR("Dave Stevenson");
+MODULE_DESCRIPTION("VideoCore CMA Shared Memory Driver");
+MODULE_LICENSE("GPL");
+MODULE_ALIAS("platform:vcsm-cma");
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
new file mode 100644
index 000000000000..61e110ec414d
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
@@ -0,0 +1,54 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * VideoCore Shared Memory driver using CMA.
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ *
+ */
+
+#ifndef VC_SM_H
+#define VC_SM_H
+
+#define VC_SM_MAX_NAME_LEN 32
+
+enum vc_sm_vpu_mapping_state {
+ VPU_NOT_MAPPED,
+ VPU_MAPPED,
+ VPU_UNMAPPING
+};
+
+struct vc_sm_buffer {
+ struct list_head global_buffer_list; /* Global list of buffers. */
+
+ /* Index in the kernel_id idr so that we can find the
+ * mmal_msg_context again when servicing the VCHI reply.
+ */
+ int kernel_id;
+
+ size_t size;
+
+ /* Lock over all the following state for this buffer */
+ struct mutex lock;
+ struct list_head attachments;
+
+ char name[VC_SM_MAX_NAME_LEN];
+
+ int in_use:1; /* Kernel is still using this resource */
+
+ enum vc_sm_vpu_mapping_state vpu_state;
+ u32 vc_handle; /* VideoCore handle for this buffer */
+
+ /* DMABUF related fields */
+ struct dma_buf *dma_buf;
+ dma_addr_t dma_addr;
+ void *cookie;
+
+ struct vc_sm_privdata_t *private;
+
+ struct dma_buf *imported_dma_buf;
+ struct dma_buf_attachment *attach;
+ struct sg_table *sgt;
+};
+
+#endif
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
new file mode 100644
index 000000000000..c77ef0998a31
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
@@ -0,0 +1,507 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * VideoCore Shared Memory CMA allocator
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ * Copyright 2011-2012 Broadcom Corporation. All rights reserved.
+ *
+ * Based on vmcs_sm driver from Broadcom Corporation.
+ *
+ */
+
+/* ---- Include Files ----------------------------------------------------- */
+#include <linux/kthread.h>
+#include <linux/list.h>
+#include <linux/semaphore.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+
+#include "vc_sm_cma_vchi.h"
+
+#define VC_SM_VER 1
+#define VC_SM_MIN_VER 0
+
+/* ---- Private Constants and Types -------------------------------------- */
+
+/* Command blocks come from a pool */
+#define SM_MAX_NUM_CMD_RSP_BLKS 32
+
+/* The number of supported connections */
+#define SM_MAX_NUM_CONNECTIONS 3
+
+struct sm_cmd_rsp_blk {
+ struct list_head head; /* To create lists */
+ /* To be signaled when the response is there */
+ struct completion cmplt;
+
+ u32 id;
+ u16 length;
+
+ u8 msg[VC_SM_MAX_MSG_LEN];
+
+ uint32_t wait:1;
+ uint32_t sent:1;
+ uint32_t alloc:1;
+
+};
+
+struct sm_instance {
+ u32 num_connections;
+ unsigned int service_handle[SM_MAX_NUM_CONNECTIONS];
+ struct task_struct *io_thread;
+ struct completion io_cmplt;
+
+ vpu_event_cb vpu_event;
+
+ /* Mutex over the following lists */
+ struct mutex lock;
+ u32 trans_id;
+ struct list_head cmd_list;
+ struct list_head rsp_list;
+ struct list_head dead_list;
+
+ struct sm_cmd_rsp_blk free_blk[SM_MAX_NUM_CMD_RSP_BLKS];
+
+ /* Mutex over the free_list */
+ struct mutex free_lock;
+ struct list_head free_list;
+
+ struct semaphore free_sema;
+ struct vchiq_instance *vchiq_instance;
+};
+
+/* ---- Private Variables ------------------------------------------------ */
+
+/* ---- Private Function Prototypes -------------------------------------- */
+
+/* ---- Private Functions ------------------------------------------------ */
+static int
+bcm2835_vchi_msg_queue(struct vchiq_instance *vchiq_instance, unsigned int handle,
+ void *data,
+ unsigned int size)
+{
+ return vchiq_queue_kernel_message(vchiq_instance, handle, data, size);
+}
+
+static struct
+sm_cmd_rsp_blk *vc_vchi_cmd_create(struct sm_instance *instance,
+ enum vc_sm_msg_type id, void *msg,
+ u32 size, int wait)
+{
+ struct sm_cmd_rsp_blk *blk;
+ struct vc_sm_msg_hdr_t *hdr;
+
+ if (down_interruptible(&instance->free_sema)) {
+ blk = kmalloc(sizeof(*blk), GFP_KERNEL);
+ if (!blk)
+ return NULL;
+
+ blk->alloc = 1;
+ init_completion(&blk->cmplt);
+ } else {
+ mutex_lock(&instance->free_lock);
+ blk =
+ list_first_entry(&instance->free_list,
+ struct sm_cmd_rsp_blk, head);
+ list_del(&blk->head);
+ mutex_unlock(&instance->free_lock);
+ }
+
+ blk->sent = 0;
+ blk->wait = wait;
+ blk->length = sizeof(*hdr) + size;
+
+ hdr = (struct vc_sm_msg_hdr_t *)blk->msg;
+ hdr->type = id;
+ mutex_lock(&instance->lock);
+ instance->trans_id++;
+ /*
+ * Retain the top bit for identifying asynchronous events, or VPU cmds.
+ */
+ instance->trans_id &= ~0x80000000;
+ hdr->trans_id = instance->trans_id;
+ blk->id = instance->trans_id;
+ mutex_unlock(&instance->lock);
+
+ if (size)
+ memcpy(hdr->body, msg, size);
+
+ return blk;
+}
+
+static void
+vc_vchi_cmd_delete(struct sm_instance *instance, struct sm_cmd_rsp_blk *blk)
+{
+ if (blk->alloc) {
+ kfree(blk);
+ return;
+ }
+
+ mutex_lock(&instance->free_lock);
+ list_add(&blk->head, &instance->free_list);
+ mutex_unlock(&instance->free_lock);
+ up(&instance->free_sema);
+}
+
+static void vc_sm_cma_vchi_rx_ack(struct sm_instance *instance,
+ struct sm_cmd_rsp_blk *cmd,
+ struct vc_sm_result_t *reply,
+ u32 reply_len)
+{
+ mutex_lock(&instance->lock);
+ list_for_each_entry(cmd,
+ &instance->rsp_list,
+ head) {
+ if (cmd->id == reply->trans_id)
+ break;
+ }
+ mutex_unlock(&instance->lock);
+
+ if (&cmd->head == &instance->rsp_list) {
+ //pr_debug("%s: received response %u, throw away...",
+ pr_err("%s: received response %u, throw away...",
+ __func__,
+ reply->trans_id);
+ } else if (reply_len > sizeof(cmd->msg)) {
+ pr_err("%s: reply too big (%u) %u, throw away...",
+ __func__, reply_len,
+ reply->trans_id);
+ } else {
+ memcpy(cmd->msg, reply,
+ reply_len);
+ complete(&cmd->cmplt);
+ }
+}
+
+static int vc_sm_cma_vchi_videocore_io(void *arg)
+{
+ struct sm_instance *instance = arg;
+ struct sm_cmd_rsp_blk *cmd = NULL, *cmd_tmp;
+ struct vc_sm_result_t *reply;
+ struct vchiq_header *header;
+ s32 status;
+ int svc_use = 1;
+
+ while (1) {
+ if (svc_use)
+ vchiq_release_service(instance->vchiq_instance,
+ instance->service_handle[0]);
+ svc_use = 0;
+
+ if (wait_for_completion_interruptible(&instance->io_cmplt))
+ continue;
+ vchiq_use_service(instance->vchiq_instance, instance->service_handle[0]);
+ svc_use = 1;
+
+ do {
+ /*
+ * Get new command and move it to response list
+ */
+ mutex_lock(&instance->lock);
+ if (list_empty(&instance->cmd_list)) {
+ /* no more commands to process */
+ mutex_unlock(&instance->lock);
+ break;
+ }
+ cmd = list_first_entry(&instance->cmd_list,
+ struct sm_cmd_rsp_blk, head);
+ list_move(&cmd->head, &instance->rsp_list);
+ cmd->sent = 1;
+ mutex_unlock(&instance->lock);
+ /* Send the command */
+ status =
+ bcm2835_vchi_msg_queue(instance->vchiq_instance,
+ instance->service_handle[0],
+ cmd->msg, cmd->length);
+ if (status) {
+ pr_err("%s: failed to queue message (%d)",
+ __func__, status);
+ }
+
+ /* If no reply is needed then we're done */
+ if (!cmd->wait) {
+ mutex_lock(&instance->lock);
+ list_del(&cmd->head);
+ mutex_unlock(&instance->lock);
+ vc_vchi_cmd_delete(instance, cmd);
+ continue;
+ }
+
+ if (status) {
+ complete(&cmd->cmplt);
+ continue;
+ }
+
+ } while (1);
+
+ while ((header = vchiq_msg_hold(instance->vchiq_instance,
+ instance->service_handle[0]))) {
+ reply = (struct vc_sm_result_t *)header->data;
+ if (reply->trans_id & 0x80000000) {
+ /* Async event or cmd from the VPU */
+ if (instance->vpu_event)
+ instance->vpu_event(instance, reply,
+ header->size);
+ } else {
+ vc_sm_cma_vchi_rx_ack(instance, cmd, reply,
+ header->size);
+ }
+
+ vchiq_release_message(instance->vchiq_instance,
+ instance->service_handle[0],
+ header);
+ }
+
+ /* Go through the dead list and free them */
+ mutex_lock(&instance->lock);
+ list_for_each_entry_safe(cmd, cmd_tmp, &instance->dead_list,
+ head) {
+ list_del(&cmd->head);
+ vc_vchi_cmd_delete(instance, cmd);
+ }
+ mutex_unlock(&instance->lock);
+ }
+
+ return 0;
+}
+
+static enum vchiq_status vc_sm_cma_vchi_callback(struct vchiq_instance *vchiq_instance,
+ enum vchiq_reason reason,
+ struct vchiq_header *header,
+ unsigned int handle, void *userdata)
+{
+ struct sm_instance *instance = vchiq_get_service_userdata(vchiq_instance, handle);
+
+ switch (reason) {
+ case VCHIQ_MESSAGE_AVAILABLE:
+ vchiq_msg_queue_push(vchiq_instance, handle, header);
+ complete(&instance->io_cmplt);
+ break;
+
+ case VCHIQ_SERVICE_CLOSED:
+ pr_info("%s: service CLOSED!!", __func__);
+ default:
+ break;
+ }
+
+ return VCHIQ_SUCCESS;
+}
+
+struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchiq_instance,
+ unsigned int num_connections,
+ vpu_event_cb vpu_event)
+{
+ u32 i;
+ struct sm_instance *instance;
+ int status;
+
+ pr_debug("%s: start", __func__);
+
+ if (num_connections > SM_MAX_NUM_CONNECTIONS) {
+ pr_err("%s: unsupported number of connections %u (max=%u)",
+ __func__, num_connections, SM_MAX_NUM_CONNECTIONS);
+
+ goto err_null;
+ }
+ /* Allocate memory for this instance */
+ instance = kzalloc(sizeof(*instance), GFP_KERNEL);
+
+ /* Misc initialisations */
+ mutex_init(&instance->lock);
+ init_completion(&instance->io_cmplt);
+ INIT_LIST_HEAD(&instance->cmd_list);
+ INIT_LIST_HEAD(&instance->rsp_list);
+ INIT_LIST_HEAD(&instance->dead_list);
+ INIT_LIST_HEAD(&instance->free_list);
+ sema_init(&instance->free_sema, SM_MAX_NUM_CMD_RSP_BLKS);
+ mutex_init(&instance->free_lock);
+ for (i = 0; i < SM_MAX_NUM_CMD_RSP_BLKS; i++) {
+ init_completion(&instance->free_blk[i].cmplt);
+ list_add(&instance->free_blk[i].head, &instance->free_list);
+ }
+
+ instance->vchiq_instance = vchiq_instance;
+
+ /* Open the VCHI service connections */
+ instance->num_connections = num_connections;
+ for (i = 0; i < num_connections; i++) {
+ struct vchiq_service_params_kernel params = {
+ .version = VC_SM_VER,
+ .version_min = VC_SM_MIN_VER,
+ .fourcc = VCHIQ_MAKE_FOURCC('S', 'M', 'E', 'M'),
+ .callback = vc_sm_cma_vchi_callback,
+ .userdata = instance,
+ };
+
+ status = vchiq_open_service(vchiq_instance, ¶ms,
+ &instance->service_handle[i]);
+ if (status) {
+ pr_err("%s: failed to open VCHI service (%d)",
+ __func__, status);
+
+ goto err_close_services;
+ }
+ }
+ /* Create the thread which takes care of all io to/from videoocore. */
+ instance->io_thread = kthread_create(&vc_sm_cma_vchi_videocore_io,
+ (void *)instance, "SMIO");
+ if (!instance->io_thread) {
+ pr_err("%s: failed to create SMIO thread", __func__);
+
+ goto err_close_services;
+ }
+ instance->vpu_event = vpu_event;
+ set_user_nice(instance->io_thread, -10);
+ wake_up_process(instance->io_thread);
+
+ pr_debug("%s: success - instance %p", __func__, instance);
+ return instance;
+
+err_close_services:
+ for (i = 0; i < instance->num_connections; i++) {
+ if (instance->service_handle[i])
+ vchiq_close_service(vchiq_instance, instance->service_handle[i]);
+ }
+ kfree(instance);
+err_null:
+ pr_debug("%s: FAILED", __func__);
+ return NULL;
+}
+
+int vc_sm_cma_vchi_stop(struct vchiq_instance *vchiq_instance, struct sm_instance **handle)
+{
+ struct sm_instance *instance;
+ u32 i;
+
+ if (!handle) {
+ pr_err("%s: invalid pointer to handle %p", __func__, handle);
+ goto lock;
+ }
+
+ if (!*handle) {
+ pr_err("%s: invalid handle %p", __func__, *handle);
+ goto lock;
+ }
+
+ instance = *handle;
+
+ /* Close all VCHI service connections */
+ for (i = 0; i < instance->num_connections; i++) {
+ vchiq_use_service(vchiq_instance, instance->service_handle[i]);
+ vchiq_close_service(vchiq_instance, instance->service_handle[i]);
+ }
+
+ kfree(instance);
+
+ *handle = NULL;
+ return 0;
+
+lock:
+ return -EINVAL;
+}
+
+static int vc_sm_cma_vchi_send_msg(struct sm_instance *handle,
+ enum vc_sm_msg_type msg_id, void *msg,
+ u32 msg_size, void *result, u32 result_size,
+ u32 *cur_trans_id, u8 wait_reply)
+{
+ int status = 0;
+ struct sm_instance *instance = handle;
+ struct sm_cmd_rsp_blk *cmd_blk;
+
+ if (!handle) {
+ pr_err("%s: invalid handle", __func__);
+ return -EINVAL;
+ }
+ if (!msg) {
+ pr_err("%s: invalid msg pointer", __func__);
+ return -EINVAL;
+ }
+
+ cmd_blk =
+ vc_vchi_cmd_create(instance, msg_id, msg, msg_size, wait_reply);
+ if (!cmd_blk) {
+ pr_err("[%s]: failed to allocate global tracking resource",
+ __func__);
+ return -ENOMEM;
+ }
+
+ if (cur_trans_id)
+ *cur_trans_id = cmd_blk->id;
+
+ mutex_lock(&instance->lock);
+ list_add_tail(&cmd_blk->head, &instance->cmd_list);
+ mutex_unlock(&instance->lock);
+ complete(&instance->io_cmplt);
+
+ if (!wait_reply)
+ /* We're done */
+ return 0;
+
+ /* Wait for the response */
+ if (wait_for_completion_interruptible(&cmd_blk->cmplt)) {
+ mutex_lock(&instance->lock);
+ if (!cmd_blk->sent) {
+ list_del(&cmd_blk->head);
+ mutex_unlock(&instance->lock);
+ vc_vchi_cmd_delete(instance, cmd_blk);
+ return -ENXIO;
+ }
+
+ list_move(&cmd_blk->head, &instance->dead_list);
+ mutex_unlock(&instance->lock);
+ complete(&instance->io_cmplt);
+ return -EINTR; /* We're done */
+ }
+
+ if (result && result_size) {
+ memcpy(result, cmd_blk->msg, result_size);
+ } else {
+ struct vc_sm_result_t *res =
+ (struct vc_sm_result_t *)cmd_blk->msg;
+ status = (res->success == 0) ? 0 : -ENXIO;
+ }
+
+ mutex_lock(&instance->lock);
+ list_del(&cmd_blk->head);
+ mutex_unlock(&instance->lock);
+ vc_vchi_cmd_delete(instance, cmd_blk);
+ return status;
+}
+
+int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg,
+ u32 *cur_trans_id)
+{
+ return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_FREE,
+ msg, sizeof(*msg), 0, 0, cur_trans_id, 0);
+}
+
+int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg,
+ struct vc_sm_import_result *result, u32 *cur_trans_id)
+{
+ return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_IMPORT,
+ msg, sizeof(*msg), result, sizeof(*result),
+ cur_trans_id, 1);
+}
+
+int vc_sm_cma_vchi_client_version(struct sm_instance *handle,
+ struct vc_sm_version *msg,
+ struct vc_sm_result_t *result,
+ u32 *cur_trans_id)
+{
+ return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_CLIENT_VERSION,
+ //msg, sizeof(*msg), result, sizeof(*result),
+ //cur_trans_id, 1);
+ msg, sizeof(*msg), NULL, 0,
+ cur_trans_id, 0);
+}
+
+int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle,
+ struct vc_sm_vc_mem_request_result *msg,
+ uint32_t *cur_trans_id)
+{
+ return vc_sm_cma_vchi_send_msg(handle,
+ VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY,
+ msg, sizeof(*msg), 0, 0, cur_trans_id,
+ 0);
+}
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
new file mode 100644
index 000000000000..a4f40d4cef05
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * VideoCore Shared Memory CMA allocator
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ * Copyright 2011-2012 Broadcom Corporation. All rights reserved.
+ *
+ * Based on vmcs_sm driver from Broadcom Corporation.
+ *
+ */
+
+#ifndef __VC_SM_CMA_VCHI_H__INCLUDED__
+#define __VC_SM_CMA_VCHI_H__INCLUDED__
+
+#include <linux/raspberrypi/vchiq.h>
+
+#include "vc_sm_defs.h"
+
+/*
+ * Forward declare.
+ */
+struct sm_instance;
+
+typedef void (*vpu_event_cb)(struct sm_instance *instance,
+ struct vc_sm_result_t *reply, int reply_len);
+
+/*
+ * Initialize the shared memory service, opens up vchi connection to talk to it.
+ */
+struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchi_instance,
+ unsigned int num_connections,
+ vpu_event_cb vpu_event);
+
+/*
+ * Terminates the shared memory service.
+ */
+int vc_sm_cma_vchi_stop(struct vchiq_instance *vchi_instance, struct sm_instance **handle);
+
+/*
+ * Ask the shared memory service to free up some memory that was previously
+ * allocated by the vc_sm_cma_vchi_alloc function call.
+ */
+int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg,
+ u32 *cur_trans_id);
+
+/*
+ * Import a contiguous block of memory and wrap it in a GPU MEM_HANDLE_T.
+ */
+int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg,
+ struct vc_sm_import_result *result,
+ u32 *cur_trans_id);
+
+int vc_sm_cma_vchi_client_version(struct sm_instance *handle,
+ struct vc_sm_version *msg,
+ struct vc_sm_result_t *result,
+ u32 *cur_trans_id);
+
+int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle,
+ struct vc_sm_vc_mem_request_result *msg,
+ uint32_t *cur_trans_id);
+
+#endif /* __VC_SM_CMA_VCHI_H__INCLUDED__ */
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
new file mode 100644
index 000000000000..ad4a3ec194d3
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
@@ -0,0 +1,187 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * VideoCore Shared Memory CMA allocator
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ *
+ * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation.
+ * All IPC messages are copied across to this file, even if the vc-sm-cma
+ * driver is not currently using them.
+ *
+ ****************************************************************************
+ */
+
+#ifndef __VC_SM_DEFS_H__INCLUDED__
+#define __VC_SM_DEFS_H__INCLUDED__
+
+/* Maximum message length */
+#define VC_SM_MAX_MSG_LEN (sizeof(union vc_sm_msg_union_t) + \
+ sizeof(struct vc_sm_msg_hdr_t))
+#define VC_SM_MAX_RSP_LEN (sizeof(union vc_sm_msg_union_t))
+
+/* Resource name maximum size */
+#define VC_SM_RESOURCE_NAME 32
+
+/*
+ * Version to be reported to the VPU
+ * VPU assumes 0 (aka 1) which does not require the released callback, nor
+ * expect the client to handle VC_MEM_REQUESTS.
+ * Version 2 requires the released callback, and must support VC_MEM_REQUESTS.
+ */
+#define VC_SM_PROTOCOL_VERSION 2
+
+enum vc_sm_msg_type {
+ /* Message types supported for HOST->VC direction */
+
+ /* Allocate shared memory block */
+ VC_SM_MSG_TYPE_ALLOC,
+ /* Lock allocated shared memory block */
+ VC_SM_MSG_TYPE_LOCK,
+ /* Unlock allocated shared memory block */
+ VC_SM_MSG_TYPE_UNLOCK,
+ /* Unlock allocated shared memory block, do not answer command */
+ VC_SM_MSG_TYPE_UNLOCK_NOANS,
+ /* Free shared memory block */
+ VC_SM_MSG_TYPE_FREE,
+ /* Resize a shared memory block */
+ VC_SM_MSG_TYPE_RESIZE,
+ /* Walk the allocated shared memory block(s) */
+ VC_SM_MSG_TYPE_WALK_ALLOC,
+
+ /* A previously applied action will need to be reverted */
+ VC_SM_MSG_TYPE_ACTION_CLEAN,
+
+ /*
+ * Import a physical address and wrap into a MEM_HANDLE_T.
+ * Release with VC_SM_MSG_TYPE_FREE.
+ */
+ VC_SM_MSG_TYPE_IMPORT,
+ /*
+ *Tells VC the protocol version supported by this client.
+ * 2 supports the async/cmd messages from the VPU for final release
+ * of memory, and for VC allocations.
+ */
+ VC_SM_MSG_TYPE_CLIENT_VERSION,
+ /* Response to VC request for memory */
+ VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY,
+
+ /*
+ * Asynchronous/cmd messages supported for VC->HOST direction.
+ * Signalled by setting the top bit in vc_sm_result_t trans_id.
+ */
+
+ /*
+ * VC has finished with an imported memory allocation.
+ * Release any Linux reference counts on the underlying block.
+ */
+ VC_SM_MSG_TYPE_RELEASED,
+ /* VC request for memory */
+ VC_SM_MSG_TYPE_VC_MEM_REQUEST,
+
+ VC_SM_MSG_TYPE_MAX
+};
+
+/* Type of memory to be allocated */
+enum vc_sm_alloc_type_t {
+ VC_SM_ALLOC_CACHED,
+ VC_SM_ALLOC_NON_CACHED,
+};
+
+/* Message header for all messages in HOST->VC direction */
+struct vc_sm_msg_hdr_t {
+ u32 type;
+ u32 trans_id;
+ u8 body[0];
+
+};
+
+/* Request to free a previously allocated memory (HOST->VC) */
+struct vc_sm_free_t {
+ /* Resource handle (returned from alloc) */
+ u32 res_handle;
+ /* Resource buffer (returned from alloc) */
+ u32 res_mem;
+
+};
+
+/* Generic result for a request (VC->HOST) */
+struct vc_sm_result_t {
+ /* Transaction identifier */
+ u32 trans_id;
+
+ s32 success;
+
+};
+
+/* Request to import memory (HOST->VC) */
+struct vc_sm_import {
+ /* type of memory to allocate */
+ enum vc_sm_alloc_type_t type;
+ /* pointer to the VC (ie physical) address of the allocated memory */
+ u32 addr;
+ /* size of buffer */
+ u32 size;
+ /* opaque handle returned in RELEASED messages */
+ u32 kernel_id;
+ /* Allocator identifier */
+ u32 allocator;
+ /* resource name (for easier tracking on vc side) */
+ char name[VC_SM_RESOURCE_NAME];
+};
+
+/* Result of a requested memory import (VC->HOST) */
+struct vc_sm_import_result {
+ /* Transaction identifier */
+ u32 trans_id;
+
+ /* Resource handle */
+ u32 res_handle;
+};
+
+/* Notification that VC has finished with an allocation (VC->HOST) */
+struct vc_sm_released {
+ /* cmd type / trans_id */
+ u32 cmd;
+
+ /* pointer to the VC (ie physical) address of the allocated memory */
+ u32 addr;
+ /* size of buffer */
+ u32 size;
+ /* opaque handle returned in RELEASED messages */
+ u32 kernel_id;
+ u32 vc_handle;
+};
+
+/*
+ * Client informing VC as to the protocol version it supports.
+ * >=2 requires the released callback, and supports VC asking for memory.
+ * Failure means that the firmware doesn't support this call, and therefore the
+ * client should either fail, or NOT rely on getting the released callback.
+ */
+struct vc_sm_version {
+ u32 version;
+};
+
+/* Response from the kernel to provide the VPU with some memory */
+struct vc_sm_vc_mem_request_result {
+ /* Transaction identifier for the VPU */
+ u32 trans_id;
+ /* pointer to the physical address of the allocated memory */
+ u32 addr;
+ /* opaque handle returned in RELEASED messages */
+ u32 kernel_id;
+};
+
+/* Union of ALL messages */
+union vc_sm_msg_union_t {
+ struct vc_sm_free_t free;
+ struct vc_sm_result_t result;
+ struct vc_sm_import import;
+ struct vc_sm_import_result import_result;
+ struct vc_sm_version version;
+ struct vc_sm_released released;
+ struct vc_sm_vc_mem_request_result vc_request_result;
+};
+
+#endif /* __VC_SM_DEFS_H__INCLUDED__ */
diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
new file mode 100644
index 000000000000..988fdd967922
--- /dev/null
+++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+/*
+ * VideoCore Shared Memory CMA allocator
+ *
+ * Copyright: 2018, Raspberry Pi (Trading) Ltd
+ *
+ * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation.
+ *
+ */
+
+#ifndef __VC_SM_KNL_H__INCLUDED__
+#define __VC_SM_KNL_H__INCLUDED__
+
+#if !defined(__KERNEL__)
+#error "This interface is for kernel use only..."
+#endif
+
+/* Free a previously allocated or imported shared memory handle and block. */
+int vc_sm_cma_free(void *handle);
+
+/* Get an internal resource handle mapped from the external one. */
+int vc_sm_cma_int_handle(void *handle);
+
+/* Import a block of memory into the GPU space. */
+int vc_sm_cma_import_dmabuf(struct dma_buf *dmabuf, void **handle);
+
+#endif /* __VC_SM_KNL_H__INCLUDED__ */
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 02/14] staging: vchiq_arm: Register vcsm-cma as a platform driver
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
Following the same pattern as bcm2835-camera and bcm2835-audio,
register the vcsm-cma driver as a platform driver.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
index dc33490ba7fb..642fdbc0d654 100644
--- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
+++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
@@ -67,6 +67,7 @@ struct vchiq_state g_state;
static struct platform_device *bcm2835_camera;
static struct platform_device *bcm2835_audio;
+static struct platform_device *vcsm_cma;
struct vchiq_drvdata {
const unsigned int cache_line_size;
@@ -1832,6 +1833,7 @@ static int vchiq_probe(struct platform_device *pdev)
goto error_exit;
}
+ vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
@@ -1847,6 +1849,7 @@ static int vchiq_remove(struct platform_device *pdev)
{
platform_device_unregister(bcm2835_audio);
platform_device_unregister(bcm2835_camera);
+ platform_device_unregister(vcsm_cma);
vchiq_debugfs_deinit();
vchiq_deregister_chrdev();
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 02/14] staging: vchiq_arm: Register vcsm-cma as a platform driver
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
Following the same pattern as bcm2835-camera and bcm2835-audio,
register the vcsm-cma driver as a platform driver.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
index dc33490ba7fb..642fdbc0d654 100644
--- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
+++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
@@ -67,6 +67,7 @@ struct vchiq_state g_state;
static struct platform_device *bcm2835_camera;
static struct platform_device *bcm2835_audio;
+static struct platform_device *vcsm_cma;
struct vchiq_drvdata {
const unsigned int cache_line_size;
@@ -1832,6 +1833,7 @@ static int vchiq_probe(struct platform_device *pdev)
goto error_exit;
}
+ vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
@@ -1847,6 +1849,7 @@ static int vchiq_remove(struct platform_device *pdev)
{
platform_device_unregister(bcm2835_audio);
platform_device_unregister(bcm2835_camera);
+ platform_device_unregister(vcsm_cma);
vchiq_debugfs_deinit();
vchiq_deregister_chrdev();
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Dave Stevenson, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.org>
videobuf2 only allowed exporting a dmabuf as a file descriptor,
but there are instances where having the struct dma_buf is
useful within the kernel.
Split the current implementation into two, one step which
exports a struct dma_buf, and the second which converts that
into an fd.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------
include/media/videobuf2-core.h | 15 ++++++++
2 files changed, 40 insertions(+), 11 deletions(-)
diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index ab9697f3b5f1..32b26737cac4 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -2184,49 +2184,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off,
return -EINVAL;
}
-int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
- unsigned int index, unsigned int plane, unsigned int flags)
+struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
+ unsigned int index, unsigned int plane,
+ unsigned int flags)
{
struct vb2_buffer *vb = NULL;
struct vb2_plane *vb_plane;
- int ret;
struct dma_buf *dbuf;
if (q->memory != VB2_MEMORY_MMAP) {
dprintk(q, 1, "queue is not currently set up for mmap\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (!q->mem_ops->get_dmabuf) {
dprintk(q, 1, "queue does not support DMA buffer exporting\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (flags & ~(O_CLOEXEC | O_ACCMODE)) {
dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (type != q->type) {
dprintk(q, 1, "invalid buffer type\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (index >= q->num_buffers) {
dprintk(q, 1, "buffer index out of range\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
vb = q->bufs[index];
if (plane >= vb->num_planes) {
dprintk(q, 1, "buffer plane out of range\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (vb2_fileio_is_active(q)) {
dprintk(q, 1, "expbuf: file io in progress\n");
- return -EBUSY;
+ return ERR_PTR(-EBUSY);
}
vb_plane = &vb->planes[plane];
@@ -2238,9 +2238,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
if (IS_ERR_OR_NULL(dbuf)) {
dprintk(q, 1, "failed to export buffer %d, plane %d\n",
index, plane);
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
+ return dbuf;
+}
+EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf);
+
+int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
+ unsigned int index, unsigned int plane, unsigned int flags)
+{
+ struct dma_buf *dbuf;
+ int ret;
+
+ dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags);
+ if (IS_ERR(dbuf))
+ return PTR_ERR(dbuf);
+
ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE);
if (ret < 0) {
dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n",
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 3253bd2f6fee..33629ed2b64f 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -911,6 +911,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
*/
int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
+/**
+ * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure
+ * @q: videobuf2 queue
+ * @type: buffer type
+ * @index: id number of the buffer
+ * @plane: index of the plane to be exported, 0 for single plane queues
+ * @flags: flags for newly created file, currently only O_CLOEXEC is
+ * supported, refer to manual of open syscall for more details
+ *
+ * Return: Returns the dmabuf pointer
+ */
+struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
+ unsigned int index, unsigned int plane,
+ unsigned int flags);
+
/**
* vb2_core_expbuf() - Export a buffer as a file descriptor.
* @q: pointer to &struct vb2_queue with videobuf2 queue.
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Dave Stevenson, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.org>
videobuf2 only allowed exporting a dmabuf as a file descriptor,
but there are instances where having the struct dma_buf is
useful within the kernel.
Split the current implementation into two, one step which
exports a struct dma_buf, and the second which converts that
into an fd.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------
include/media/videobuf2-core.h | 15 ++++++++
2 files changed, 40 insertions(+), 11 deletions(-)
diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index ab9697f3b5f1..32b26737cac4 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -2184,49 +2184,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off,
return -EINVAL;
}
-int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
- unsigned int index, unsigned int plane, unsigned int flags)
+struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
+ unsigned int index, unsigned int plane,
+ unsigned int flags)
{
struct vb2_buffer *vb = NULL;
struct vb2_plane *vb_plane;
- int ret;
struct dma_buf *dbuf;
if (q->memory != VB2_MEMORY_MMAP) {
dprintk(q, 1, "queue is not currently set up for mmap\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (!q->mem_ops->get_dmabuf) {
dprintk(q, 1, "queue does not support DMA buffer exporting\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (flags & ~(O_CLOEXEC | O_ACCMODE)) {
dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (type != q->type) {
dprintk(q, 1, "invalid buffer type\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (index >= q->num_buffers) {
dprintk(q, 1, "buffer index out of range\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
vb = q->bufs[index];
if (plane >= vb->num_planes) {
dprintk(q, 1, "buffer plane out of range\n");
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
if (vb2_fileio_is_active(q)) {
dprintk(q, 1, "expbuf: file io in progress\n");
- return -EBUSY;
+ return ERR_PTR(-EBUSY);
}
vb_plane = &vb->planes[plane];
@@ -2238,9 +2238,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
if (IS_ERR_OR_NULL(dbuf)) {
dprintk(q, 1, "failed to export buffer %d, plane %d\n",
index, plane);
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
}
+ return dbuf;
+}
+EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf);
+
+int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
+ unsigned int index, unsigned int plane, unsigned int flags)
+{
+ struct dma_buf *dbuf;
+ int ret;
+
+ dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags);
+ if (IS_ERR(dbuf))
+ return PTR_ERR(dbuf);
+
ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE);
if (ret < 0) {
dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n",
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 3253bd2f6fee..33629ed2b64f 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -911,6 +911,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
*/
int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
+/**
+ * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure
+ * @q: videobuf2 queue
+ * @type: buffer type
+ * @index: id number of the buffer
+ * @plane: index of the plane to be exported, 0 for single plane queues
+ * @flags: flags for newly created file, currently only O_CLOEXEC is
+ * supported, refer to manual of open syscall for more details
+ *
+ * Return: Returns the dmabuf pointer
+ */
+struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
+ unsigned int index, unsigned int plane,
+ unsigned int flags);
+
/**
* vb2_core_expbuf() - Export a buffer as a file descriptor.
* @q: pointer to &struct vb2_queue with videobuf2 queue.
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 04/14] staging: mmal-vchiq: Add support for event callbacks
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Dave Stevenson, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.org>
The bcm2835-isp will use the event mechanism to report things such as
stream start event. It is signalled by the cmd field of the buffer
being non-zero.
Add support for passing this information out to the client.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../vc04_services/vchiq-mmal/mmal-common.h | 1 +
.../vc04_services/vchiq-mmal/mmal-msg.h | 35 ++++
.../vc04_services/vchiq-mmal/mmal-vchiq.c | 170 ++++++++++++++++--
.../vc04_services/vchiq-mmal/mmal-vchiq.h | 4 +
4 files changed, 196 insertions(+), 14 deletions(-)
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
index b33129403a30..0443be8198ea 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
@@ -50,6 +50,7 @@ struct mmal_buffer {
struct mmal_msg_context *msg_context;
+ u32 cmd; /* MMAL command. 0=data. */
unsigned long length;
u32 mmal_flags;
s64 dts;
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h
index 471413248a14..ef84b4e35608 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h
@@ -346,6 +346,41 @@ struct mmal_msg_port_parameter_get_reply {
/* event messages */
#define MMAL_WORKER_EVENT_SPACE 256
+/* Four CC's for events */
+#define MMAL_FOURCC(a, b, c, d) ((a) | (b << 8) | (c << 16) | (d << 24))
+
+#define MMAL_EVENT_ERROR MMAL_FOURCC('E', 'R', 'R', 'O')
+#define MMAL_EVENT_EOS MMAL_FOURCC('E', 'E', 'O', 'S')
+#define MMAL_EVENT_FORMAT_CHANGED MMAL_FOURCC('E', 'F', 'C', 'H')
+#define MMAL_EVENT_PARAMETER_CHANGED MMAL_FOURCC('E', 'P', 'C', 'H')
+
+/* Structs for each of the event message payloads */
+struct mmal_msg_event_eos {
+ u32 port_type; /**< Type of port that received the end of stream */
+ u32 port_index; /**< Index of port that received the end of stream */
+};
+
+/** Format changed event data. */
+struct mmal_msg_event_format_changed {
+ /* Minimum size of buffers the port requires */
+ u32 buffer_size_min;
+ /* Minimum number of buffers the port requires */
+ u32 buffer_num_min;
+ /* Size of buffers the port recommends for optimal performance.
+ * A value of zero means no special recommendation.
+ */
+ u32 buffer_size_recommended;
+ /* Number of buffers the port recommends for optimal
+ * performance. A value of zero means no special recommendation.
+ */
+ u32 buffer_num_recommended;
+
+ u32 es_ptr;
+ struct mmal_es_format format;
+ union mmal_es_specific_format es;
+ u8 extradata[MMAL_FORMAT_EXTRADATA_MAX_SIZE];
+};
+
struct mmal_msg_event_to_host {
u32 client_component; /* component context */
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
index cb921c94996a..1c0047eead7e 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
@@ -143,6 +143,8 @@ struct mmal_msg_context {
/* Presentation and Decode timestamps */
s64 pts;
s64 dts;
+ /* MMAL buffer command flag */
+ u32 cmd;
int status; /* context status */
@@ -230,18 +232,6 @@ release_msg_context(struct mmal_msg_context *msg_context)
kfree(msg_context);
}
-/* deals with receipt of event to host message */
-static void event_to_host_cb(struct vchiq_mmal_instance *instance,
- struct mmal_msg *msg, u32 msg_len)
-{
- pr_debug("unhandled event\n");
- pr_debug("component:%u port type:%d num:%d cmd:0x%x length:%d\n",
- msg->u.event_to_host.client_component,
- msg->u.event_to_host.port_type,
- msg->u.event_to_host.port_num,
- msg->u.event_to_host.cmd, msg->u.event_to_host.length);
-}
-
/* workqueue scheduled callback
*
* we do this because it is important we do not call any other vchiq
@@ -263,13 +253,18 @@ static void buffer_work_cb(struct work_struct *work)
buffer->mmal_flags = msg_context->u.bulk.mmal_flags;
buffer->dts = msg_context->u.bulk.dts;
buffer->pts = msg_context->u.bulk.pts;
+ buffer->cmd = msg_context->u.bulk.cmd;
- atomic_dec(&msg_context->u.bulk.port->buffers_with_vpu);
+ if (!buffer->cmd)
+ atomic_dec(&msg_context->u.bulk.port->buffers_with_vpu);
msg_context->u.bulk.port->buffer_cb(msg_context->u.bulk.instance,
msg_context->u.bulk.port,
msg_context->u.bulk.status,
msg_context->u.bulk.buffer);
+
+ if (buffer->cmd)
+ mutex_unlock(&msg_context->u.bulk.port->event_context_mutex);
}
/* workqueue scheduled callback to handle receiving buffers
@@ -347,6 +342,7 @@ static int bulk_receive(struct vchiq_mmal_instance *instance,
msg_context->u.bulk.buffer_used = rd_len;
msg_context->u.bulk.dts = msg->u.buffer_from_host.buffer_header.dts;
msg_context->u.bulk.pts = msg->u.buffer_from_host.buffer_header.pts;
+ msg_context->u.bulk.cmd = msg->u.buffer_from_host.buffer_header.cmd;
queue_work(msg_context->instance->bulk_wq,
&msg_context->u.bulk.buffer_to_host_work);
@@ -449,6 +445,103 @@ buffer_from_host(struct vchiq_mmal_instance *instance,
return ret;
}
+/* deals with receipt of event to host message */
+static void event_to_host_cb(struct vchiq_mmal_instance *instance,
+ struct mmal_msg *msg, u32 msg_len)
+{
+ /* FIXME: Not going to work on 64 bit */
+ struct vchiq_mmal_component *component =
+ (struct vchiq_mmal_component *)msg->u.event_to_host.client_component;
+ struct vchiq_mmal_port *port = NULL;
+ struct mmal_msg_context *msg_context;
+ u32 port_num = msg->u.event_to_host.port_num;
+
+ if (msg->u.buffer_from_host.drvbuf.magic == MMAL_MAGIC) {
+ pr_err("%s: MMAL_MSG_TYPE_BUFFER_TO_HOST with bad magic\n",
+ __func__);
+ return;
+ }
+
+ switch (msg->u.event_to_host.port_type) {
+ case MMAL_PORT_TYPE_CONTROL:
+ if (port_num) {
+ pr_err("%s: port_num of %u >= number of ports 1",
+ __func__, port_num);
+ return;
+ }
+ port = &component->control;
+ break;
+ case MMAL_PORT_TYPE_INPUT:
+ if (port_num >= component->inputs) {
+ pr_err("%s: port_num of %u >= number of ports %u",
+ __func__, port_num,
+ port_num >= component->inputs);
+ return;
+ }
+ port = &component->input[port_num];
+ break;
+ case MMAL_PORT_TYPE_OUTPUT:
+ if (port_num >= component->outputs) {
+ pr_err("%s: port_num of %u >= number of ports %u",
+ __func__, port_num,
+ port_num >= component->outputs);
+ return;
+ }
+ port = &component->output[port_num];
+ break;
+ case MMAL_PORT_TYPE_CLOCK:
+ if (port_num >= component->clocks) {
+ pr_err("%s: port_num of %u >= number of ports %u",
+ __func__, port_num,
+ port_num >= component->clocks);
+ return;
+ }
+ port = &component->clock[port_num];
+ break;
+ default:
+ break;
+ }
+
+ if (!mutex_trylock(&port->event_context_mutex)) {
+ pr_err("dropping event 0x%x\n", msg->u.event_to_host.cmd);
+ return;
+ }
+ msg_context = port->event_context;
+
+ if (msg->h.status != MMAL_MSG_STATUS_SUCCESS) {
+ /* message reception had an error */
+ //pr_warn
+ pr_err("%s: error %d in reply\n", __func__, msg->h.status);
+
+ msg_context->u.bulk.status = msg->h.status;
+ } else if (msg->u.event_to_host.length > MMAL_WORKER_EVENT_SPACE) {
+ /* data is not in message, queue a bulk receive */
+ pr_err("%s: payload not in message - bulk receive??! NOT SUPPORTED\n",
+ __func__);
+ msg_context->u.bulk.status = -1;
+ } else {
+ memcpy(msg_context->u.bulk.buffer->buffer,
+ msg->u.event_to_host.data,
+ msg->u.event_to_host.length);
+
+ msg_context->u.bulk.buffer_used =
+ msg->u.event_to_host.length;
+
+ msg_context->u.bulk.mmal_flags = 0;
+ msg_context->u.bulk.dts = MMAL_TIME_UNKNOWN;
+ msg_context->u.bulk.pts = MMAL_TIME_UNKNOWN;
+ msg_context->u.bulk.cmd = msg->u.event_to_host.cmd;
+
+ pr_debug("event component:%u port type:%d num:%d cmd:0x%x length:%d\n",
+ msg->u.event_to_host.client_component,
+ msg->u.event_to_host.port_type,
+ msg->u.event_to_host.port_num,
+ msg->u.event_to_host.cmd, msg->u.event_to_host.length);
+ }
+
+ schedule_work(&msg_context->u.bulk.work);
+}
+
/* deals with receipt of buffer to host message */
static void buffer_to_host_cb(struct vchiq_mmal_instance *instance,
struct mmal_msg *msg, u32 msg_len)
@@ -1330,6 +1423,7 @@ static int port_disable(struct vchiq_mmal_instance *instance,
mmalbuf->mmal_flags = 0;
mmalbuf->dts = MMAL_TIME_UNKNOWN;
mmalbuf->pts = MMAL_TIME_UNKNOWN;
+ mmalbuf->cmd = 0;
port->buffer_cb(instance,
port, 0, mmalbuf);
}
@@ -1631,6 +1725,43 @@ int mmal_vchi_buffer_cleanup(struct mmal_buffer *buf)
}
EXPORT_SYMBOL_GPL(mmal_vchi_buffer_cleanup);
+static void init_event_context(struct vchiq_mmal_instance *instance,
+ struct vchiq_mmal_port *port)
+{
+ struct mmal_msg_context *ctx = get_msg_context(instance);
+
+ mutex_init(&port->event_context_mutex);
+
+ port->event_context = ctx;
+ ctx->u.bulk.instance = instance;
+ ctx->u.bulk.port = port;
+ ctx->u.bulk.buffer =
+ kzalloc(sizeof(*ctx->u.bulk.buffer), GFP_KERNEL);
+ if (!ctx->u.bulk.buffer)
+ goto release_msg_context;
+ ctx->u.bulk.buffer->buffer = kzalloc(MMAL_WORKER_EVENT_SPACE,
+ GFP_KERNEL);
+ if (!ctx->u.bulk.buffer->buffer)
+ goto release_buffer;
+
+ INIT_WORK(&ctx->u.bulk.work, buffer_work_cb);
+ return;
+
+release_buffer:
+ kfree(ctx->u.bulk.buffer);
+release_msg_context:
+ release_msg_context(ctx);
+}
+
+static void free_event_context(struct vchiq_mmal_port *port)
+{
+ struct mmal_msg_context *ctx = port->event_context;
+
+ kfree(ctx->u.bulk.buffer->buffer);
+ kfree(ctx->u.bulk.buffer);
+ release_msg_context(ctx);
+}
+
/* Initialise a mmal component and its ports
*
*/
@@ -1680,6 +1811,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance,
ret = port_info_get(instance, &component->control);
if (ret < 0)
goto release_component;
+ init_event_context(instance, &component->control);
for (idx = 0; idx < component->inputs; idx++) {
component->input[idx].type = MMAL_PORT_TYPE_INPUT;
@@ -1690,6 +1822,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance,
ret = port_info_get(instance, &component->input[idx]);
if (ret < 0)
goto release_component;
+ init_event_context(instance, &component->input[idx]);
}
for (idx = 0; idx < component->outputs; idx++) {
@@ -1701,6 +1834,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance,
ret = port_info_get(instance, &component->output[idx]);
if (ret < 0)
goto release_component;
+ init_event_context(instance, &component->output[idx]);
}
for (idx = 0; idx < component->clocks; idx++) {
@@ -1712,6 +1846,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance,
ret = port_info_get(instance, &component->clock[idx]);
if (ret < 0)
goto release_component;
+ init_event_context(instance, &component->clock[idx]);
}
*component_out = component;
@@ -1737,7 +1872,7 @@ EXPORT_SYMBOL_GPL(vchiq_mmal_component_init);
int vchiq_mmal_component_finalise(struct vchiq_mmal_instance *instance,
struct vchiq_mmal_component *component)
{
- int ret;
+ int ret, idx;
if (mutex_lock_interruptible(&instance->vchiq_mutex))
return -EINTR;
@@ -1749,6 +1884,13 @@ int vchiq_mmal_component_finalise(struct vchiq_mmal_instance *instance,
component->in_use = 0;
+ for (idx = 0; idx < component->inputs; idx++)
+ free_event_context(&component->input[idx]);
+ for (idx = 0; idx < component->outputs; idx++)
+ free_event_context(&component->output[idx]);
+ for (idx = 0; idx < component->clocks; idx++)
+ free_event_context(&component->clock[idx]);
+
mutex_unlock(&instance->vchiq_mutex);
return ret;
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
index 6006e29232b3..e3ecd70b9951 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
@@ -79,6 +79,10 @@ struct vchiq_mmal_port {
vchiq_mmal_buffer_cb buffer_cb;
/* callback context */
void *cb_ctx;
+
+ /* ensure serialised use of the one event context structure */
+ struct mutex event_context_mutex;
+ struct mmal_msg_context *event_context;
};
struct vchiq_mmal_component {
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 04/14] staging: mmal-vchiq: Add support for event callbacks
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Dave Stevenson, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.org>
The bcm2835-isp will use the event mechanism to report things such as
stream start event. It is signalled by the cmd field of the buffer
being non-zero.
Add support for passing this information out to the client.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../vc04_services/vchiq-mmal/mmal-common.h | 1 +
.../vc04_services/vchiq-mmal/mmal-msg.h | 35 ++++
.../vc04_services/vchiq-mmal/mmal-vchiq.c | 170 ++++++++++++++++--
.../vc04_services/vchiq-mmal/mmal-vchiq.h | 4 +
4 files changed, 196 insertions(+), 14 deletions(-)
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
index b33129403a30..0443be8198ea 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
@@ -50,6 +50,7 @@ struct mmal_buffer {
struct mmal_msg_context *msg_context;
+ u32 cmd; /* MMAL command. 0=data. */
unsigned long length;
u32 mmal_flags;
s64 dts;
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h
index 471413248a14..ef84b4e35608 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h
@@ -346,6 +346,41 @@ struct mmal_msg_port_parameter_get_reply {
/* event messages */
#define MMAL_WORKER_EVENT_SPACE 256
+/* Four CC's for events */
+#define MMAL_FOURCC(a, b, c, d) ((a) | (b << 8) | (c << 16) | (d << 24))
+
+#define MMAL_EVENT_ERROR MMAL_FOURCC('E', 'R', 'R', 'O')
+#define MMAL_EVENT_EOS MMAL_FOURCC('E', 'E', 'O', 'S')
+#define MMAL_EVENT_FORMAT_CHANGED MMAL_FOURCC('E', 'F', 'C', 'H')
+#define MMAL_EVENT_PARAMETER_CHANGED MMAL_FOURCC('E', 'P', 'C', 'H')
+
+/* Structs for each of the event message payloads */
+struct mmal_msg_event_eos {
+ u32 port_type; /**< Type of port that received the end of stream */
+ u32 port_index; /**< Index of port that received the end of stream */
+};
+
+/** Format changed event data. */
+struct mmal_msg_event_format_changed {
+ /* Minimum size of buffers the port requires */
+ u32 buffer_size_min;
+ /* Minimum number of buffers the port requires */
+ u32 buffer_num_min;
+ /* Size of buffers the port recommends for optimal performance.
+ * A value of zero means no special recommendation.
+ */
+ u32 buffer_size_recommended;
+ /* Number of buffers the port recommends for optimal
+ * performance. A value of zero means no special recommendation.
+ */
+ u32 buffer_num_recommended;
+
+ u32 es_ptr;
+ struct mmal_es_format format;
+ union mmal_es_specific_format es;
+ u8 extradata[MMAL_FORMAT_EXTRADATA_MAX_SIZE];
+};
+
struct mmal_msg_event_to_host {
u32 client_component; /* component context */
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
index cb921c94996a..1c0047eead7e 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
@@ -143,6 +143,8 @@ struct mmal_msg_context {
/* Presentation and Decode timestamps */
s64 pts;
s64 dts;
+ /* MMAL buffer command flag */
+ u32 cmd;
int status; /* context status */
@@ -230,18 +232,6 @@ release_msg_context(struct mmal_msg_context *msg_context)
kfree(msg_context);
}
-/* deals with receipt of event to host message */
-static void event_to_host_cb(struct vchiq_mmal_instance *instance,
- struct mmal_msg *msg, u32 msg_len)
-{
- pr_debug("unhandled event\n");
- pr_debug("component:%u port type:%d num:%d cmd:0x%x length:%d\n",
- msg->u.event_to_host.client_component,
- msg->u.event_to_host.port_type,
- msg->u.event_to_host.port_num,
- msg->u.event_to_host.cmd, msg->u.event_to_host.length);
-}
-
/* workqueue scheduled callback
*
* we do this because it is important we do not call any other vchiq
@@ -263,13 +253,18 @@ static void buffer_work_cb(struct work_struct *work)
buffer->mmal_flags = msg_context->u.bulk.mmal_flags;
buffer->dts = msg_context->u.bulk.dts;
buffer->pts = msg_context->u.bulk.pts;
+ buffer->cmd = msg_context->u.bulk.cmd;
- atomic_dec(&msg_context->u.bulk.port->buffers_with_vpu);
+ if (!buffer->cmd)
+ atomic_dec(&msg_context->u.bulk.port->buffers_with_vpu);
msg_context->u.bulk.port->buffer_cb(msg_context->u.bulk.instance,
msg_context->u.bulk.port,
msg_context->u.bulk.status,
msg_context->u.bulk.buffer);
+
+ if (buffer->cmd)
+ mutex_unlock(&msg_context->u.bulk.port->event_context_mutex);
}
/* workqueue scheduled callback to handle receiving buffers
@@ -347,6 +342,7 @@ static int bulk_receive(struct vchiq_mmal_instance *instance,
msg_context->u.bulk.buffer_used = rd_len;
msg_context->u.bulk.dts = msg->u.buffer_from_host.buffer_header.dts;
msg_context->u.bulk.pts = msg->u.buffer_from_host.buffer_header.pts;
+ msg_context->u.bulk.cmd = msg->u.buffer_from_host.buffer_header.cmd;
queue_work(msg_context->instance->bulk_wq,
&msg_context->u.bulk.buffer_to_host_work);
@@ -449,6 +445,103 @@ buffer_from_host(struct vchiq_mmal_instance *instance,
return ret;
}
+/* deals with receipt of event to host message */
+static void event_to_host_cb(struct vchiq_mmal_instance *instance,
+ struct mmal_msg *msg, u32 msg_len)
+{
+ /* FIXME: Not going to work on 64 bit */
+ struct vchiq_mmal_component *component =
+ (struct vchiq_mmal_component *)msg->u.event_to_host.client_component;
+ struct vchiq_mmal_port *port = NULL;
+ struct mmal_msg_context *msg_context;
+ u32 port_num = msg->u.event_to_host.port_num;
+
+ if (msg->u.buffer_from_host.drvbuf.magic == MMAL_MAGIC) {
+ pr_err("%s: MMAL_MSG_TYPE_BUFFER_TO_HOST with bad magic\n",
+ __func__);
+ return;
+ }
+
+ switch (msg->u.event_to_host.port_type) {
+ case MMAL_PORT_TYPE_CONTROL:
+ if (port_num) {
+ pr_err("%s: port_num of %u >= number of ports 1",
+ __func__, port_num);
+ return;
+ }
+ port = &component->control;
+ break;
+ case MMAL_PORT_TYPE_INPUT:
+ if (port_num >= component->inputs) {
+ pr_err("%s: port_num of %u >= number of ports %u",
+ __func__, port_num,
+ port_num >= component->inputs);
+ return;
+ }
+ port = &component->input[port_num];
+ break;
+ case MMAL_PORT_TYPE_OUTPUT:
+ if (port_num >= component->outputs) {
+ pr_err("%s: port_num of %u >= number of ports %u",
+ __func__, port_num,
+ port_num >= component->outputs);
+ return;
+ }
+ port = &component->output[port_num];
+ break;
+ case MMAL_PORT_TYPE_CLOCK:
+ if (port_num >= component->clocks) {
+ pr_err("%s: port_num of %u >= number of ports %u",
+ __func__, port_num,
+ port_num >= component->clocks);
+ return;
+ }
+ port = &component->clock[port_num];
+ break;
+ default:
+ break;
+ }
+
+ if (!mutex_trylock(&port->event_context_mutex)) {
+ pr_err("dropping event 0x%x\n", msg->u.event_to_host.cmd);
+ return;
+ }
+ msg_context = port->event_context;
+
+ if (msg->h.status != MMAL_MSG_STATUS_SUCCESS) {
+ /* message reception had an error */
+ //pr_warn
+ pr_err("%s: error %d in reply\n", __func__, msg->h.status);
+
+ msg_context->u.bulk.status = msg->h.status;
+ } else if (msg->u.event_to_host.length > MMAL_WORKER_EVENT_SPACE) {
+ /* data is not in message, queue a bulk receive */
+ pr_err("%s: payload not in message - bulk receive??! NOT SUPPORTED\n",
+ __func__);
+ msg_context->u.bulk.status = -1;
+ } else {
+ memcpy(msg_context->u.bulk.buffer->buffer,
+ msg->u.event_to_host.data,
+ msg->u.event_to_host.length);
+
+ msg_context->u.bulk.buffer_used =
+ msg->u.event_to_host.length;
+
+ msg_context->u.bulk.mmal_flags = 0;
+ msg_context->u.bulk.dts = MMAL_TIME_UNKNOWN;
+ msg_context->u.bulk.pts = MMAL_TIME_UNKNOWN;
+ msg_context->u.bulk.cmd = msg->u.event_to_host.cmd;
+
+ pr_debug("event component:%u port type:%d num:%d cmd:0x%x length:%d\n",
+ msg->u.event_to_host.client_component,
+ msg->u.event_to_host.port_type,
+ msg->u.event_to_host.port_num,
+ msg->u.event_to_host.cmd, msg->u.event_to_host.length);
+ }
+
+ schedule_work(&msg_context->u.bulk.work);
+}
+
/* deals with receipt of buffer to host message */
static void buffer_to_host_cb(struct vchiq_mmal_instance *instance,
struct mmal_msg *msg, u32 msg_len)
@@ -1330,6 +1423,7 @@ static int port_disable(struct vchiq_mmal_instance *instance,
mmalbuf->mmal_flags = 0;
mmalbuf->dts = MMAL_TIME_UNKNOWN;
mmalbuf->pts = MMAL_TIME_UNKNOWN;
+ mmalbuf->cmd = 0;
port->buffer_cb(instance,
port, 0, mmalbuf);
}
@@ -1631,6 +1725,43 @@ int mmal_vchi_buffer_cleanup(struct mmal_buffer *buf)
}
EXPORT_SYMBOL_GPL(mmal_vchi_buffer_cleanup);
+static void init_event_context(struct vchiq_mmal_instance *instance,
+ struct vchiq_mmal_port *port)
+{
+ struct mmal_msg_context *ctx = get_msg_context(instance);
+
+ mutex_init(&port->event_context_mutex);
+
+ port->event_context = ctx;
+ ctx->u.bulk.instance = instance;
+ ctx->u.bulk.port = port;
+ ctx->u.bulk.buffer =
+ kzalloc(sizeof(*ctx->u.bulk.buffer), GFP_KERNEL);
+ if (!ctx->u.bulk.buffer)
+ goto release_msg_context;
+ ctx->u.bulk.buffer->buffer = kzalloc(MMAL_WORKER_EVENT_SPACE,
+ GFP_KERNEL);
+ if (!ctx->u.bulk.buffer->buffer)
+ goto release_buffer;
+
+ INIT_WORK(&ctx->u.bulk.work, buffer_work_cb);
+ return;
+
+release_buffer:
+ kfree(ctx->u.bulk.buffer);
+release_msg_context:
+ release_msg_context(ctx);
+}
+
+static void free_event_context(struct vchiq_mmal_port *port)
+{
+ struct mmal_msg_context *ctx = port->event_context;
+
+ kfree(ctx->u.bulk.buffer->buffer);
+ kfree(ctx->u.bulk.buffer);
+ release_msg_context(ctx);
+}
+
/* Initialise a mmal component and its ports
*
*/
@@ -1680,6 +1811,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance,
ret = port_info_get(instance, &component->control);
if (ret < 0)
goto release_component;
+ init_event_context(instance, &component->control);
for (idx = 0; idx < component->inputs; idx++) {
component->input[idx].type = MMAL_PORT_TYPE_INPUT;
@@ -1690,6 +1822,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance,
ret = port_info_get(instance, &component->input[idx]);
if (ret < 0)
goto release_component;
+ init_event_context(instance, &component->input[idx]);
}
for (idx = 0; idx < component->outputs; idx++) {
@@ -1701,6 +1834,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance,
ret = port_info_get(instance, &component->output[idx]);
if (ret < 0)
goto release_component;
+ init_event_context(instance, &component->output[idx]);
}
for (idx = 0; idx < component->clocks; idx++) {
@@ -1712,6 +1846,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance,
ret = port_info_get(instance, &component->clock[idx]);
if (ret < 0)
goto release_component;
+ init_event_context(instance, &component->clock[idx]);
}
*component_out = component;
@@ -1737,7 +1872,7 @@ EXPORT_SYMBOL_GPL(vchiq_mmal_component_init);
int vchiq_mmal_component_finalise(struct vchiq_mmal_instance *instance,
struct vchiq_mmal_component *component)
{
- int ret;
+ int ret, idx;
if (mutex_lock_interruptible(&instance->vchiq_mutex))
return -EINTR;
@@ -1749,6 +1884,13 @@ int vchiq_mmal_component_finalise(struct vchiq_mmal_instance *instance,
component->in_use = 0;
+ for (idx = 0; idx < component->inputs; idx++)
+ free_event_context(&component->input[idx]);
+ for (idx = 0; idx < component->outputs; idx++)
+ free_event_context(&component->output[idx]);
+ for (idx = 0; idx < component->clocks; idx++)
+ free_event_context(&component->clock[idx]);
+
mutex_unlock(&instance->vchiq_mutex);
return ret;
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
index 6006e29232b3..e3ecd70b9951 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
@@ -79,6 +79,10 @@ struct vchiq_mmal_port {
vchiq_mmal_buffer_cb buffer_cb;
/* callback context */
void *cb_ctx;
+
+ /* ensure serialised use of the one event context structure */
+ struct mutex event_context_mutex;
+ struct mmal_msg_context *event_context;
};
struct vchiq_mmal_component {
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 05/14] staging: mmal-vchiq: Use vc-sm-cma to support zero copy
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
With the vc-sm-cma driver we can support zero copy of buffers between
the kernel and VPU. Add this support to mmal-vchiq.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
.../vc04_services/vchiq-mmal/mmal-common.h | 4 +
.../vc04_services/vchiq-mmal/mmal-vchiq.c | 83 +++++++++++++++++--
.../vc04_services/vchiq-mmal/mmal-vchiq.h | 1 +
4 files changed, 82 insertions(+), 7 deletions(-)
diff --git a/drivers/staging/vc04_services/vchiq-mmal/Kconfig b/drivers/staging/vc04_services/vchiq-mmal/Kconfig
index c99525a0bb45..a7c1a7bf516e 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/Kconfig
+++ b/drivers/staging/vc04_services/vchiq-mmal/Kconfig
@@ -1,6 +1,7 @@
config BCM2835_VCHIQ_MMAL
tristate "BCM2835 MMAL VCHIQ service"
depends on BCM2835_VCHIQ
+ select BCM_VC_SM_CMA
help
Enables the MMAL API over VCHIQ interface as used for the
majority of the multimedia services on VideoCore.
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
index 0443be8198ea..a643cad54b12 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
@@ -50,6 +50,10 @@ struct mmal_buffer {
struct mmal_msg_context *msg_context;
+ struct dma_buf *dma_buf;/* Exported dmabuf fd from videobuf2 */
+ void *vcsm_handle; /* VCSM handle having imported the dmabuf */
+ u32 vc_handle; /* VC handle to that dmabuf */
+
u32 cmd; /* MMAL command. 0=data. */
unsigned long length;
u32 mmal_flags;
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
index 1c0047eead7e..f26fb64a2785 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
@@ -27,9 +27,11 @@
#include <media/videobuf2-vmalloc.h>
#include "mmal-common.h"
+#include "mmal-parameters.h"
#include "mmal-vchiq.h"
#include "mmal-msg.h"
+#include "vc-sm-cma/vc_sm_knl.h"
/*
* maximum number of components supported.
* This matches the maximum permitted by default on the VPU
@@ -416,14 +418,27 @@ buffer_from_host(struct vchiq_mmal_instance *instance,
/* buffer header */
m.u.buffer_from_host.buffer_header.cmd = 0;
- m.u.buffer_from_host.buffer_header.data =
- (u32)(unsigned long)buf->buffer;
+ if (port->zero_copy) {
+ m.u.buffer_from_host.buffer_header.data = buf->vc_handle;
+ } else {
+ m.u.buffer_from_host.buffer_header.data =
+ (u32)(unsigned long)buf->buffer;
+ }
+
m.u.buffer_from_host.buffer_header.alloc_size = buf->buffer_size;
- m.u.buffer_from_host.buffer_header.length = 0; /* nothing used yet */
- m.u.buffer_from_host.buffer_header.offset = 0; /* no offset */
- m.u.buffer_from_host.buffer_header.flags = 0; /* no flags */
- m.u.buffer_from_host.buffer_header.pts = MMAL_TIME_UNKNOWN;
- m.u.buffer_from_host.buffer_header.dts = MMAL_TIME_UNKNOWN;
+ if (port->type == MMAL_PORT_TYPE_OUTPUT) {
+ m.u.buffer_from_host.buffer_header.length = 0;
+ m.u.buffer_from_host.buffer_header.offset = 0;
+ m.u.buffer_from_host.buffer_header.flags = 0;
+ m.u.buffer_from_host.buffer_header.pts = MMAL_TIME_UNKNOWN;
+ m.u.buffer_from_host.buffer_header.dts = MMAL_TIME_UNKNOWN;
+ } else {
+ m.u.buffer_from_host.buffer_header.length = buf->length;
+ m.u.buffer_from_host.buffer_header.offset = 0;
+ m.u.buffer_from_host.buffer_header.flags = buf->mmal_flags;
+ m.u.buffer_from_host.buffer_header.pts = buf->pts;
+ m.u.buffer_from_host.buffer_header.dts = buf->dts;
+ }
/* clear buffer type specific data */
memset(&m.u.buffer_from_host.buffer_header_type_specific, 0,
@@ -575,6 +590,22 @@ static void buffer_to_host_cb(struct vchiq_mmal_instance *instance,
msg_context->u.bulk.status = msg->h.status;
+ } else if (msg->u.buffer_from_host.is_zero_copy) {
+ /*
+ * Zero copy buffer, so nothing to do.
+ * Copy buffer info and make callback.
+ */
+ msg_context->u.bulk.buffer_used =
+ msg->u.buffer_from_host.buffer_header.length;
+ msg_context->u.bulk.mmal_flags =
+ msg->u.buffer_from_host.buffer_header.flags;
+ msg_context->u.bulk.dts =
+ msg->u.buffer_from_host.buffer_header.dts;
+ msg_context->u.bulk.pts =
+ msg->u.buffer_from_host.buffer_header.pts;
+ msg_context->u.bulk.cmd =
+ msg->u.buffer_from_host.buffer_header.cmd;
+
} else if (msg->u.buffer_from_host.buffer_header.length == 0) {
/* empty buffer */
if (msg->u.buffer_from_host.buffer_header.flags &
@@ -1520,6 +1551,9 @@ int vchiq_mmal_port_parameter_set(struct vchiq_mmal_instance *instance,
mutex_unlock(&instance->vchiq_mutex);
+ if (parameter == MMAL_PARAMETER_ZERO_COPY && !ret)
+ port->zero_copy = !!(*(bool *)value);
+
return ret;
}
EXPORT_SYMBOL_GPL(vchiq_mmal_port_parameter_set);
@@ -1688,6 +1722,31 @@ int vchiq_mmal_submit_buffer(struct vchiq_mmal_instance *instance,
unsigned long flags = 0;
int ret;
+ /*
+ * We really want to do this in mmal_vchi_buffer_init but can't as
+ * videobuf2 won't let us have the dmabuf there.
+ */
+ if (port->zero_copy && buffer->dma_buf && !buffer->vcsm_handle) {
+ pr_debug("%s: import dmabuf %p\n", __func__, buffer->dma_buf);
+ ret = vc_sm_cma_import_dmabuf(buffer->dma_buf,
+ &buffer->vcsm_handle);
+ if (ret) {
+ pr_err("%s: vc_sm_import_dmabuf_fd failed, ret %d\n",
+ __func__, ret);
+ return ret;
+ }
+
+ buffer->vc_handle = vc_sm_cma_int_handle(buffer->vcsm_handle);
+ if (!buffer->vc_handle) {
+ pr_err("%s: vc_sm_int_handle failed %d\n",
+ __func__, ret);
+ vc_sm_cma_free(buffer->vcsm_handle);
+ return ret;
+ }
+ pr_debug("%s: import dmabuf %p - got vc handle %08X\n",
+ __func__, buffer->dma_buf, buffer->vc_handle);
+ }
+
ret = buffer_from_host(instance, port, buffer);
if (ret == -EINVAL) {
/* Port is disabled. Queue for when it is enabled. */
@@ -1721,6 +1780,16 @@ int mmal_vchi_buffer_cleanup(struct mmal_buffer *buf)
release_msg_context(msg_context);
buf->msg_context = NULL;
+ if (buf->vcsm_handle) {
+ int ret;
+
+ pr_debug("%s: vc_sm_cma_free on handle %p\n", __func__,
+ buf->vcsm_handle);
+ ret = vc_sm_cma_free(buf->vcsm_handle);
+ if (ret)
+ pr_err("%s: vcsm_free failed, ret %d\n", __func__, ret);
+ buf->vcsm_handle = 0;
+ }
return 0;
}
EXPORT_SYMBOL_GPL(mmal_vchi_buffer_cleanup);
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
index e3ecd70b9951..9addd627ff5b 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
@@ -49,6 +49,7 @@ typedef void (*vchiq_mmal_buffer_cb)(
struct vchiq_mmal_port {
u32 enabled:1;
+ u32 zero_copy:1;
u32 handle;
u32 type; /* port type, cached to use on port info set */
u32 index; /* port index, cached to use on port info set */
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 05/14] staging: mmal-vchiq: Use vc-sm-cma to support zero copy
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
With the vc-sm-cma driver we can support zero copy of buffers between
the kernel and VPU. Add this support to mmal-vchiq.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
.../vc04_services/vchiq-mmal/mmal-common.h | 4 +
.../vc04_services/vchiq-mmal/mmal-vchiq.c | 83 +++++++++++++++++--
.../vc04_services/vchiq-mmal/mmal-vchiq.h | 1 +
4 files changed, 82 insertions(+), 7 deletions(-)
diff --git a/drivers/staging/vc04_services/vchiq-mmal/Kconfig b/drivers/staging/vc04_services/vchiq-mmal/Kconfig
index c99525a0bb45..a7c1a7bf516e 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/Kconfig
+++ b/drivers/staging/vc04_services/vchiq-mmal/Kconfig
@@ -1,6 +1,7 @@
config BCM2835_VCHIQ_MMAL
tristate "BCM2835 MMAL VCHIQ service"
depends on BCM2835_VCHIQ
+ select BCM_VC_SM_CMA
help
Enables the MMAL API over VCHIQ interface as used for the
majority of the multimedia services on VideoCore.
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
index 0443be8198ea..a643cad54b12 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h
@@ -50,6 +50,10 @@ struct mmal_buffer {
struct mmal_msg_context *msg_context;
+ struct dma_buf *dma_buf;/* Exported dmabuf fd from videobuf2 */
+ void *vcsm_handle; /* VCSM handle having imported the dmabuf */
+ u32 vc_handle; /* VC handle to that dmabuf */
+
u32 cmd; /* MMAL command. 0=data. */
unsigned long length;
u32 mmal_flags;
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
index 1c0047eead7e..f26fb64a2785 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c
@@ -27,9 +27,11 @@
#include <media/videobuf2-vmalloc.h>
#include "mmal-common.h"
+#include "mmal-parameters.h"
#include "mmal-vchiq.h"
#include "mmal-msg.h"
+#include "vc-sm-cma/vc_sm_knl.h"
/*
* maximum number of components supported.
* This matches the maximum permitted by default on the VPU
@@ -416,14 +418,27 @@ buffer_from_host(struct vchiq_mmal_instance *instance,
/* buffer header */
m.u.buffer_from_host.buffer_header.cmd = 0;
- m.u.buffer_from_host.buffer_header.data =
- (u32)(unsigned long)buf->buffer;
+ if (port->zero_copy) {
+ m.u.buffer_from_host.buffer_header.data = buf->vc_handle;
+ } else {
+ m.u.buffer_from_host.buffer_header.data =
+ (u32)(unsigned long)buf->buffer;
+ }
+
m.u.buffer_from_host.buffer_header.alloc_size = buf->buffer_size;
- m.u.buffer_from_host.buffer_header.length = 0; /* nothing used yet */
- m.u.buffer_from_host.buffer_header.offset = 0; /* no offset */
- m.u.buffer_from_host.buffer_header.flags = 0; /* no flags */
- m.u.buffer_from_host.buffer_header.pts = MMAL_TIME_UNKNOWN;
- m.u.buffer_from_host.buffer_header.dts = MMAL_TIME_UNKNOWN;
+ if (port->type == MMAL_PORT_TYPE_OUTPUT) {
+ m.u.buffer_from_host.buffer_header.length = 0;
+ m.u.buffer_from_host.buffer_header.offset = 0;
+ m.u.buffer_from_host.buffer_header.flags = 0;
+ m.u.buffer_from_host.buffer_header.pts = MMAL_TIME_UNKNOWN;
+ m.u.buffer_from_host.buffer_header.dts = MMAL_TIME_UNKNOWN;
+ } else {
+ m.u.buffer_from_host.buffer_header.length = buf->length;
+ m.u.buffer_from_host.buffer_header.offset = 0;
+ m.u.buffer_from_host.buffer_header.flags = buf->mmal_flags;
+ m.u.buffer_from_host.buffer_header.pts = buf->pts;
+ m.u.buffer_from_host.buffer_header.dts = buf->dts;
+ }
/* clear buffer type specific data */
memset(&m.u.buffer_from_host.buffer_header_type_specific, 0,
@@ -575,6 +590,22 @@ static void buffer_to_host_cb(struct vchiq_mmal_instance *instance,
msg_context->u.bulk.status = msg->h.status;
+ } else if (msg->u.buffer_from_host.is_zero_copy) {
+ /*
+ * Zero copy buffer, so nothing to do.
+ * Copy buffer info and make callback.
+ */
+ msg_context->u.bulk.buffer_used =
+ msg->u.buffer_from_host.buffer_header.length;
+ msg_context->u.bulk.mmal_flags =
+ msg->u.buffer_from_host.buffer_header.flags;
+ msg_context->u.bulk.dts =
+ msg->u.buffer_from_host.buffer_header.dts;
+ msg_context->u.bulk.pts =
+ msg->u.buffer_from_host.buffer_header.pts;
+ msg_context->u.bulk.cmd =
+ msg->u.buffer_from_host.buffer_header.cmd;
+
} else if (msg->u.buffer_from_host.buffer_header.length == 0) {
/* empty buffer */
if (msg->u.buffer_from_host.buffer_header.flags &
@@ -1520,6 +1551,9 @@ int vchiq_mmal_port_parameter_set(struct vchiq_mmal_instance *instance,
mutex_unlock(&instance->vchiq_mutex);
+ if (parameter == MMAL_PARAMETER_ZERO_COPY && !ret)
+ port->zero_copy = !!(*(bool *)value);
+
return ret;
}
EXPORT_SYMBOL_GPL(vchiq_mmal_port_parameter_set);
@@ -1688,6 +1722,31 @@ int vchiq_mmal_submit_buffer(struct vchiq_mmal_instance *instance,
unsigned long flags = 0;
int ret;
+ /*
+ * We really want to do this in mmal_vchi_buffer_init but can't as
+ * videobuf2 won't let us have the dmabuf there.
+ */
+ if (port->zero_copy && buffer->dma_buf && !buffer->vcsm_handle) {
+ pr_debug("%s: import dmabuf %p\n", __func__, buffer->dma_buf);
+ ret = vc_sm_cma_import_dmabuf(buffer->dma_buf,
+ &buffer->vcsm_handle);
+ if (ret) {
+ pr_err("%s: vc_sm_import_dmabuf_fd failed, ret %d\n",
+ __func__, ret);
+ return ret;
+ }
+
+ buffer->vc_handle = vc_sm_cma_int_handle(buffer->vcsm_handle);
+ if (!buffer->vc_handle) {
+ pr_err("%s: vc_sm_int_handle failed %d\n",
+ __func__, ret);
+ vc_sm_cma_free(buffer->vcsm_handle);
+ return ret;
+ }
+ pr_debug("%s: import dmabuf %p - got vc handle %08X\n",
+ __func__, buffer->dma_buf, buffer->vc_handle);
+ }
+
ret = buffer_from_host(instance, port, buffer);
if (ret == -EINVAL) {
/* Port is disabled. Queue for when it is enabled. */
@@ -1721,6 +1780,16 @@ int mmal_vchi_buffer_cleanup(struct mmal_buffer *buf)
release_msg_context(msg_context);
buf->msg_context = NULL;
+ if (buf->vcsm_handle) {
+ int ret;
+
+ pr_debug("%s: vc_sm_cma_free on handle %p\n", __func__,
+ buf->vcsm_handle);
+ ret = vc_sm_cma_free(buf->vcsm_handle);
+ if (ret)
+ pr_err("%s: vcsm_free failed, ret %d\n", __func__, ret);
+ buf->vcsm_handle = 0;
+ }
return 0;
}
EXPORT_SYMBOL_GPL(mmal_vchi_buffer_cleanup);
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
index e3ecd70b9951..9addd627ff5b 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h
@@ -49,6 +49,7 @@ typedef void (*vchiq_mmal_buffer_cb)(
struct vchiq_mmal_port {
u32 enabled:1;
+ u32 zero_copy:1;
u32 handle;
u32 type; /* port type, cached to use on port info set */
u32 index; /* port index, cached to use on port info set */
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 06/14] staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
The bcm2835-isp supports Bayer, so add in the encodings for them.
Also, Add support for monochrome image formats in the various MIPI
packings.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../vc04_services/vchiq-mmal/mmal-encodings.h | 62 +++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
index e15ae7b24f73..4711877a9711 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
@@ -69,6 +69,68 @@
*/
#define MMAL_ENCODING_OPAQUE MMAL_FOURCC('O', 'P', 'Q', 'V')
+/* Bayer formats
+ * FourCC values copied from V4L2 where defined.
+ */
+/* 8 bit per pixel Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR8 MMAL_FOURCC('B', 'A', '8', '1')
+#define MMAL_ENCODING_BAYER_SGBRG8 MMAL_FOURCC('G', 'B', 'R', 'G')
+#define MMAL_ENCODING_BAYER_SGRBG8 MMAL_FOURCC('G', 'R', 'B', 'G')
+#define MMAL_ENCODING_BAYER_SRGGB8 MMAL_FOURCC('R', 'G', 'G', 'B')
+
+/* 10 bit per pixel packed Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR10P MMAL_FOURCC('p', 'B', 'A', 'A')
+#define MMAL_ENCODING_BAYER_SGRBG10P MMAL_FOURCC('p', 'g', 'A', 'A')
+#define MMAL_ENCODING_BAYER_SGBRG10P MMAL_FOURCC('p', 'G', 'A', 'A')
+#define MMAL_ENCODING_BAYER_SRGGB10P MMAL_FOURCC('p', 'R', 'A', 'A')
+
+/* 12 bit per pixel packed Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR12P MMAL_FOURCC('p', 'B', '1', '2')
+#define MMAL_ENCODING_BAYER_SGRBG12P MMAL_FOURCC('p', 'g', '1', '2')
+#define MMAL_ENCODING_BAYER_SGBRG12P MMAL_FOURCC('p', 'G', '1', '2')
+#define MMAL_ENCODING_BAYER_SRGGB12P MMAL_FOURCC('p', 'R', '1', '2')
+
+/* 14 bit per pixel Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR14P MMAL_FOURCC('p', 'B', 'E', 'E')
+#define MMAL_ENCODING_BAYER_SGBRG14P MMAL_FOURCC('p', 'G', 'E', 'E')
+#define MMAL_ENCODING_BAYER_SGRBG14P MMAL_FOURCC('p', 'g', 'E', 'E')
+#define MMAL_ENCODING_BAYER_SRGGB14P MMAL_FOURCC('p', 'R', 'E', 'E')
+
+/* 16 bit per pixel Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR16 MMAL_FOURCC('B', 'G', '1', '6')
+#define MMAL_ENCODING_BAYER_SGBRG16 MMAL_FOURCC('G', 'B', '1', '6')
+#define MMAL_ENCODING_BAYER_SGRBG16 MMAL_FOURCC('G', 'R', '1', '6')
+#define MMAL_ENCODING_BAYER_SRGGB16 MMAL_FOURCC('R', 'G', '1', '6')
+
+/* 10 bit per pixel unpacked (16bit) Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR10 MMAL_FOURCC('B', 'G', '1', '0')
+#define MMAL_ENCODING_BAYER_SGRBG10 MMAL_FOURCC('B', 'A', '1', '0')
+#define MMAL_ENCODING_BAYER_SGBRG10 MMAL_FOURCC('G', 'B', '1', '0')
+#define MMAL_ENCODING_BAYER_SRGGB10 MMAL_FOURCC('R', 'G', '1', '0')
+
+/* 12 bit per pixel unpacked (16bit) Bayer formats */
+#define MMAL_ENCODING_BAYER_SBGGR12 MMAL_FOURCC('B', 'G', '1', '2')
+#define MMAL_ENCODING_BAYER_SGRBG12 MMAL_FOURCC('B', 'A', '1', '2')
+#define MMAL_ENCODING_BAYER_SGBRG12 MMAL_FOURCC('G', 'B', '1', '2')
+#define MMAL_ENCODING_BAYER_SRGGB12 MMAL_FOURCC('R', 'G', '1', '2')
+
+/* 14 bit per pixel unpacked (16bit) Bayer formats */
+#define MMAL_ENCODING_BAYER_SBGGR14 MMAL_FOURCC('B', 'G', '1', '4')
+#define MMAL_ENCODING_BAYER_SGBRG14 MMAL_FOURCC('G', 'B', '1', '4')
+#define MMAL_ENCODING_BAYER_SGRBG14 MMAL_FOURCC('G', 'R', '1', '4')
+#define MMAL_ENCODING_BAYER_SRGGB14 MMAL_FOURCC('R', 'G', '1', '4')
+
+/* MIPI packed monochrome images */
+#define MMAL_ENCODING_GREY MMAL_FOURCC('G', 'R', 'E', 'Y')
+#define MMAL_ENCODING_Y10P MMAL_FOURCC('Y', '1', '0', 'P')
+#define MMAL_ENCODING_Y12P MMAL_FOURCC('Y', '1', '2', 'P')
+#define MMAL_ENCODING_Y14P MMAL_FOURCC('Y', '1', '4', 'P')
+#define MMAL_ENCODING_Y16 MMAL_FOURCC('Y', '1', '6', ' ')
+/* Unpacked monochrome formats (16bit per sample, but only N LSBs used) */
+#define MMAL_ENCODING_Y10 MMAL_FOURCC('Y', '1', '0', ' ')
+#define MMAL_ENCODING_Y12 MMAL_FOURCC('Y', '1', '2', ' ')
+#define MMAL_ENCODING_Y14 MMAL_FOURCC('Y', '1', '4', ' ')
+
/** An EGL image handle
*/
#define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I')
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 06/14] staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
The bcm2835-isp supports Bayer, so add in the encodings for them.
Also, Add support for monochrome image formats in the various MIPI
packings.
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../vc04_services/vchiq-mmal/mmal-encodings.h | 62 +++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
index e15ae7b24f73..4711877a9711 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
@@ -69,6 +69,68 @@
*/
#define MMAL_ENCODING_OPAQUE MMAL_FOURCC('O', 'P', 'Q', 'V')
+/* Bayer formats
+ * FourCC values copied from V4L2 where defined.
+ */
+/* 8 bit per pixel Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR8 MMAL_FOURCC('B', 'A', '8', '1')
+#define MMAL_ENCODING_BAYER_SGBRG8 MMAL_FOURCC('G', 'B', 'R', 'G')
+#define MMAL_ENCODING_BAYER_SGRBG8 MMAL_FOURCC('G', 'R', 'B', 'G')
+#define MMAL_ENCODING_BAYER_SRGGB8 MMAL_FOURCC('R', 'G', 'G', 'B')
+
+/* 10 bit per pixel packed Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR10P MMAL_FOURCC('p', 'B', 'A', 'A')
+#define MMAL_ENCODING_BAYER_SGRBG10P MMAL_FOURCC('p', 'g', 'A', 'A')
+#define MMAL_ENCODING_BAYER_SGBRG10P MMAL_FOURCC('p', 'G', 'A', 'A')
+#define MMAL_ENCODING_BAYER_SRGGB10P MMAL_FOURCC('p', 'R', 'A', 'A')
+
+/* 12 bit per pixel packed Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR12P MMAL_FOURCC('p', 'B', '1', '2')
+#define MMAL_ENCODING_BAYER_SGRBG12P MMAL_FOURCC('p', 'g', '1', '2')
+#define MMAL_ENCODING_BAYER_SGBRG12P MMAL_FOURCC('p', 'G', '1', '2')
+#define MMAL_ENCODING_BAYER_SRGGB12P MMAL_FOURCC('p', 'R', '1', '2')
+
+/* 14 bit per pixel Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR14P MMAL_FOURCC('p', 'B', 'E', 'E')
+#define MMAL_ENCODING_BAYER_SGBRG14P MMAL_FOURCC('p', 'G', 'E', 'E')
+#define MMAL_ENCODING_BAYER_SGRBG14P MMAL_FOURCC('p', 'g', 'E', 'E')
+#define MMAL_ENCODING_BAYER_SRGGB14P MMAL_FOURCC('p', 'R', 'E', 'E')
+
+/* 16 bit per pixel Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR16 MMAL_FOURCC('B', 'G', '1', '6')
+#define MMAL_ENCODING_BAYER_SGBRG16 MMAL_FOURCC('G', 'B', '1', '6')
+#define MMAL_ENCODING_BAYER_SGRBG16 MMAL_FOURCC('G', 'R', '1', '6')
+#define MMAL_ENCODING_BAYER_SRGGB16 MMAL_FOURCC('R', 'G', '1', '6')
+
+/* 10 bit per pixel unpacked (16bit) Bayer formats. */
+#define MMAL_ENCODING_BAYER_SBGGR10 MMAL_FOURCC('B', 'G', '1', '0')
+#define MMAL_ENCODING_BAYER_SGRBG10 MMAL_FOURCC('B', 'A', '1', '0')
+#define MMAL_ENCODING_BAYER_SGBRG10 MMAL_FOURCC('G', 'B', '1', '0')
+#define MMAL_ENCODING_BAYER_SRGGB10 MMAL_FOURCC('R', 'G', '1', '0')
+
+/* 12 bit per pixel unpacked (16bit) Bayer formats */
+#define MMAL_ENCODING_BAYER_SBGGR12 MMAL_FOURCC('B', 'G', '1', '2')
+#define MMAL_ENCODING_BAYER_SGRBG12 MMAL_FOURCC('B', 'A', '1', '2')
+#define MMAL_ENCODING_BAYER_SGBRG12 MMAL_FOURCC('G', 'B', '1', '2')
+#define MMAL_ENCODING_BAYER_SRGGB12 MMAL_FOURCC('R', 'G', '1', '2')
+
+/* 14 bit per pixel unpacked (16bit) Bayer formats */
+#define MMAL_ENCODING_BAYER_SBGGR14 MMAL_FOURCC('B', 'G', '1', '4')
+#define MMAL_ENCODING_BAYER_SGBRG14 MMAL_FOURCC('G', 'B', '1', '4')
+#define MMAL_ENCODING_BAYER_SGRBG14 MMAL_FOURCC('G', 'R', '1', '4')
+#define MMAL_ENCODING_BAYER_SRGGB14 MMAL_FOURCC('R', 'G', '1', '4')
+
+/* MIPI packed monochrome images */
+#define MMAL_ENCODING_GREY MMAL_FOURCC('G', 'R', 'E', 'Y')
+#define MMAL_ENCODING_Y10P MMAL_FOURCC('Y', '1', '0', 'P')
+#define MMAL_ENCODING_Y12P MMAL_FOURCC('Y', '1', '2', 'P')
+#define MMAL_ENCODING_Y14P MMAL_FOURCC('Y', '1', '4', 'P')
+#define MMAL_ENCODING_Y16 MMAL_FOURCC('Y', '1', '6', ' ')
+/* Unpacked monochrome formats (16bit per sample, but only N LSBs used) */
+#define MMAL_ENCODING_Y10 MMAL_FOURCC('Y', '1', '0', ' ')
+#define MMAL_ENCODING_Y12 MMAL_FOURCC('Y', '1', '2', ' ')
+#define MMAL_ENCODING_Y14 MMAL_FOURCC('Y', '1', '4', ' ')
+
/** An EGL image handle
*/
#define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I')
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 07/14] media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
Add V4L2_META_FMT_BCM2835_ISP_STATS V4L2 format type.
This new format will be used by the bcm2835-isp device to return
out ISP statistics for 3A.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../userspace-api/media/v4l/meta-formats.rst | 1 +
.../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +++++++++++++++++++
drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
include/uapi/linux/videodev2.h | 1 +
4 files changed, 44 insertions(+)
create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index d1ac3f3a2d8f..0c52f3df1576 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
.. toctree::
:maxdepth: 1
+ pixfmt-meta-bcm2835-isp-stats
pixfmt-meta-d4xx
pixfmt-meta-intel-ipu3
pixfmt-meta-sensor-data
diff --git a/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
new file mode 100644
index 000000000000..f974774c8252
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
@@ -0,0 +1,41 @@
+.. Permission is granted to copy, distribute and/or modify this
+.. document under the terms of the GNU Free Documentation License,
+.. Version 1.1 or any later version published by the Free Software
+.. Foundation, with no Invariant Sections, no Front-Cover Texts
+.. and no Back-Cover Texts. A copy of the license is included at
+.. Documentation/media/uapi/fdl-appendix.rst.
+..
+.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
+
+.. _v4l2-meta-fmt-bcm2835-isp-stats:
+
+*****************************************
+V4L2_META_FMT_BCM2835_ISP_STATS ('BSTA')
+*****************************************
+
+BCM2835 ISP Statistics
+
+Description
+===========
+
+The BCM2835 ISP hardware calculate image statistics for an input Bayer frame.
+These statistics are obtained from the "bcm2835-isp0-capture3" device node
+using the :c:type:`v4l2_meta_format` interface. They are formatted as described
+by the :c:type:`bcm2835_isp_stats` structure below.
+
+.. code-block:: c
+
+ #define DEFAULT_AWB_REGIONS_X 16
+ #define DEFAULT_AWB_REGIONS_Y 12
+
+ #define NUM_HISTOGRAMS 2
+ #define NUM_HISTOGRAM_BINS 128
+ #define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
+ #define FLOATING_REGIONS 16
+ #define AGC_REGIONS 16
+ #define FOCUS_REGIONS 12
+
+.. kernel-doc:: include/uapi/linux/bcm2835-isp.h
+ :functions: bcm2835_isp_stats_hist bcm2835_isp_stats_region
+ bcm2835_isp_stats_focus bcm2835_isp_stats
+
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 2680bc33f911..361e47ad6480 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1449,6 +1449,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
case V4L2_PIX_FMT_NV12M_8L128: descr = "NV12M (8x128 Linear)"; break;
case V4L2_PIX_FMT_NV12M_10BE_8L128: descr = "10-bit NV12M (8x128 Linear, BE)"; break;
case V4L2_META_FMT_SENSOR_DATA: descr = "Sensor Ancillary Metadata"; break;
+ case V4L2_META_FMT_BCM2835_ISP_STATS: descr = "BCM2835 ISP Image Statistics"; break;
default:
/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 026fbc7af034..9269b45b3b64 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -807,6 +807,7 @@ struct v4l2_pix_format {
#define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
#define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D') /* Vivid Metadata */
#define V4L2_META_FMT_SENSOR_DATA v4l2_fourcc('S', 'E', 'N', 'S') /* Sensor Ancillary metadata */
+#define V4L2_META_FMT_BCM2835_ISP_STATS v4l2_fourcc('B', 'S', 'T', 'A') /* BCM2835 ISP image statistics output */
/* Vendor specific - used for RK_ISP1 camera sub-system */
#define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 07/14] media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
Add V4L2_META_FMT_BCM2835_ISP_STATS V4L2 format type.
This new format will be used by the bcm2835-isp device to return
out ISP statistics for 3A.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../userspace-api/media/v4l/meta-formats.rst | 1 +
.../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +++++++++++++++++++
drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
include/uapi/linux/videodev2.h | 1 +
4 files changed, 44 insertions(+)
create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
index d1ac3f3a2d8f..0c52f3df1576 100644
--- a/Documentation/userspace-api/media/v4l/meta-formats.rst
+++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
@@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
.. toctree::
:maxdepth: 1
+ pixfmt-meta-bcm2835-isp-stats
pixfmt-meta-d4xx
pixfmt-meta-intel-ipu3
pixfmt-meta-sensor-data
diff --git a/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
new file mode 100644
index 000000000000..f974774c8252
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
@@ -0,0 +1,41 @@
+.. Permission is granted to copy, distribute and/or modify this
+.. document under the terms of the GNU Free Documentation License,
+.. Version 1.1 or any later version published by the Free Software
+.. Foundation, with no Invariant Sections, no Front-Cover Texts
+.. and no Back-Cover Texts. A copy of the license is included at
+.. Documentation/media/uapi/fdl-appendix.rst.
+..
+.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
+
+.. _v4l2-meta-fmt-bcm2835-isp-stats:
+
+*****************************************
+V4L2_META_FMT_BCM2835_ISP_STATS ('BSTA')
+*****************************************
+
+BCM2835 ISP Statistics
+
+Description
+===========
+
+The BCM2835 ISP hardware calculate image statistics for an input Bayer frame.
+These statistics are obtained from the "bcm2835-isp0-capture3" device node
+using the :c:type:`v4l2_meta_format` interface. They are formatted as described
+by the :c:type:`bcm2835_isp_stats` structure below.
+
+.. code-block:: c
+
+ #define DEFAULT_AWB_REGIONS_X 16
+ #define DEFAULT_AWB_REGIONS_Y 12
+
+ #define NUM_HISTOGRAMS 2
+ #define NUM_HISTOGRAM_BINS 128
+ #define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
+ #define FLOATING_REGIONS 16
+ #define AGC_REGIONS 16
+ #define FOCUS_REGIONS 12
+
+.. kernel-doc:: include/uapi/linux/bcm2835-isp.h
+ :functions: bcm2835_isp_stats_hist bcm2835_isp_stats_region
+ bcm2835_isp_stats_focus bcm2835_isp_stats
+
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 2680bc33f911..361e47ad6480 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1449,6 +1449,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
case V4L2_PIX_FMT_NV12M_8L128: descr = "NV12M (8x128 Linear)"; break;
case V4L2_PIX_FMT_NV12M_10BE_8L128: descr = "10-bit NV12M (8x128 Linear, BE)"; break;
case V4L2_META_FMT_SENSOR_DATA: descr = "Sensor Ancillary Metadata"; break;
+ case V4L2_META_FMT_BCM2835_ISP_STATS: descr = "BCM2835 ISP Image Statistics"; break;
default:
/* Compressed formats */
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 026fbc7af034..9269b45b3b64 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -807,6 +807,7 @@ struct v4l2_pix_format {
#define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
#define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D') /* Vivid Metadata */
#define V4L2_META_FMT_SENSOR_DATA v4l2_fourcc('S', 'E', 'N', 'S') /* Sensor Ancillary metadata */
+#define V4L2_META_FMT_BCM2835_ISP_STATS v4l2_fourcc('B', 'S', 'T', 'A') /* BCM2835 ISP image statistics output */
/* Vendor specific - used for RK_ISP1 camera sub-system */
#define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 08/14] uapi: bcm2835-isp: Add bcm2835-isp uapi header file
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
This file defines the userland interface to the bcm2835-isp driver
that will follow in a separate commit.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
include/uapi/linux/bcm2835-isp.h | 320 +++++++++++++++++++++++++++++
include/uapi/linux/v4l2-controls.h | 6 +
2 files changed, 326 insertions(+)
create mode 100644 include/uapi/linux/bcm2835-isp.h
diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
new file mode 100644
index 000000000000..cf8c0437f159
--- /dev/null
+++ b/include/uapi/linux/bcm2835-isp.h
@@ -0,0 +1,320 @@
+/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * bcm2835-isp.h
+ *
+ * BCM2835 ISP driver - user space header file.
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck (naush@raspberrypi.com)
+ *
+ */
+
+#ifndef __BCM2835_ISP_H_
+#define __BCM2835_ISP_H_
+
+#include <linux/v4l2-controls.h>
+
+#define V4L2_CID_USER_BCM2835_ISP_CC_MATRIX \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0001)
+#define V4L2_CID_USER_BCM2835_ISP_LENS_SHADING \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0002)
+#define V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0003)
+#define V4L2_CID_USER_BCM2835_ISP_GEQ \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0004)
+#define V4L2_CID_USER_BCM2835_ISP_GAMMA \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0005)
+#define V4L2_CID_USER_BCM2835_ISP_DENOISE \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0006)
+#define V4L2_CID_USER_BCM2835_ISP_SHARPEN \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
+#define V4L2_CID_USER_BCM2835_ISP_DPC \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
+
+/*
+ * All structs below are directly mapped onto the equivalent structs in
+ * drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
+ * for convenience.
+ */
+
+/**
+ * struct bcm2835_isp_rational - Rational value type.
+ *
+ * @num: Numerator.
+ * @den: Denominator.
+ */
+struct bcm2835_isp_rational {
+ __s32 num;
+ __u32 den;
+};
+
+/**
+ * struct bcm2835_isp_ccm - Colour correction matrix.
+ *
+ * @ccm: 3x3 correction matrix coefficients.
+ * @offsets: 1x3 correction offsets.
+ */
+struct bcm2835_isp_ccm {
+ struct bcm2835_isp_rational ccm[3][3];
+ __s32 offsets[3];
+};
+
+/**
+ * struct bcm2835_isp_custom_ccm - Custom CCM applied with the
+ * V4L2_CID_USER_BCM2835_ISP_CC_MATRIX ctrl.
+ *
+ * @enabled: Enable custom CCM.
+ * @ccm: Custom CCM coefficients and offsets.
+ */
+struct bcm2835_isp_custom_ccm {
+ __u32 enabled;
+ struct bcm2835_isp_ccm ccm;
+};
+
+/**
+ * enum bcm2835_isp_gain_format - format of the gains in the lens shading
+ * tables used with the
+ * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING ctrl.
+ *
+ * @GAIN_FORMAT_U0P8_1: Gains are u0.8 format, starting at 1.0
+ * @GAIN_FORMAT_U1P7_0: Gains are u1.7 format, starting at 0.0
+ * @GAIN_FORMAT_U1P7_1: Gains are u1.7 format, starting at 1.0
+ * @GAIN_FORMAT_U2P6_0: Gains are u2.6 format, starting at 0.0
+ * @GAIN_FORMAT_U2P6_1: Gains are u2.6 format, starting at 1.0
+ * @GAIN_FORMAT_U3P5_0: Gains are u3.5 format, starting at 0.0
+ * @GAIN_FORMAT_U3P5_1: Gains are u3.5 format, starting at 1.0
+ * @GAIN_FORMAT_U4P10: Gains are u4.10 format, starting at 0.0
+ */
+enum bcm2835_isp_gain_format {
+ GAIN_FORMAT_U0P8_1 = 0,
+ GAIN_FORMAT_U1P7_0 = 1,
+ GAIN_FORMAT_U1P7_1 = 2,
+ GAIN_FORMAT_U2P6_0 = 3,
+ GAIN_FORMAT_U2P6_1 = 4,
+ GAIN_FORMAT_U3P5_0 = 5,
+ GAIN_FORMAT_U3P5_1 = 6,
+ GAIN_FORMAT_U4P10 = 7,
+};
+
+/**
+ * struct bcm2835_isp_lens_shading - Lens shading tables supplied with the
+ * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING
+ * ctrl.
+ *
+ * @enabled: Enable lens shading.
+ * @grid_cell_size: Size of grid cells in samples (16, 32, 64, 128 or 256).
+ * @grid_width: Width of lens shading tables in grid cells.
+ * @grid_stride: Row to row distance (in grid cells) between grid cells
+ * in the same horizontal location.
+ * @grid_height: Height of lens shading tables in grid cells.
+ * @dmabuf: dmabuf file handle containing the table.
+ * @ref_transform: Reference transform - unsupported, please pass zero.
+ * @corner_sampled: Whether the gains are sampled at the corner points
+ * of the grid cells or in the cell centres.
+ * @gain_format: Format of the gains (see enum &bcm2835_isp_gain_format).
+ */
+struct bcm2835_isp_lens_shading {
+ __u32 enabled;
+ __u32 grid_cell_size;
+ __u32 grid_width;
+ __u32 grid_stride;
+ __u32 grid_height;
+ __s32 dmabuf;
+ __u32 ref_transform;
+ __u32 corner_sampled;
+ __u32 gain_format;
+};
+
+/**
+ * struct bcm2835_isp_black_level - Sensor black level set with the
+ * V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL ctrl.
+ *
+ * @enabled: Enable black level.
+ * @black_level_r: Black level for red channel.
+ * @black_level_g: Black level for green channels.
+ * @black_level_b: Black level for blue channel.
+ */
+struct bcm2835_isp_black_level {
+ __u32 enabled;
+ __u16 black_level_r;
+ __u16 black_level_g;
+ __u16 black_level_b;
+ __u8 padding[2]; /* Unused */
+};
+
+/**
+ * struct bcm2835_isp_geq - Green equalisation parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_GEQ ctrl.
+ *
+ * @enabled: Enable green equalisation.
+ * @offset: Fixed offset of the green equalisation threshold.
+ * @slope: Slope of the green equalisation threshold.
+ */
+struct bcm2835_isp_geq {
+ __u32 enabled;
+ __u32 offset;
+ struct bcm2835_isp_rational slope;
+};
+
+#define BCM2835_NUM_GAMMA_PTS 33
+
+/**
+ * struct bcm2835_isp_gamma - Gamma parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_GAMMA ctrl.
+ *
+ * @enabled: Enable gamma adjustment.
+ * @X: X values of the points defining the gamma curve.
+ * Values should be scaled to 16 bits.
+ * @Y: Y values of the points defining the gamma curve.
+ * Values should be scaled to 16 bits.
+ */
+struct bcm2835_isp_gamma {
+ __u32 enabled;
+ __u16 x[BCM2835_NUM_GAMMA_PTS];
+ __u16 y[BCM2835_NUM_GAMMA_PTS];
+};
+
+/**
+ * struct bcm2835_isp_denoise - Denoise parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
+ *
+ * @enabled: Enable denoise.
+ * @constant: Fixed offset of the noise threshold.
+ * @slope: Slope of the noise threshold.
+ * @strength: Denoise strength between 0.0 (off) and 1.0 (maximum).
+ */
+struct bcm2835_isp_denoise {
+ __u32 enabled;
+ __u32 constant;
+ struct bcm2835_isp_rational slope;
+ struct bcm2835_isp_rational strength;
+};
+
+/**
+ * struct bcm2835_isp_sharpen - Sharpen parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_SHARPEN ctrl.
+ *
+ * @enabled: Enable sharpening.
+ * @threshold: Threshold at which to start sharpening pixels.
+ * @strength: Strength with which pixel sharpening increases.
+ * @limit: Limit to the amount of sharpening applied.
+ */
+struct bcm2835_isp_sharpen {
+ __u32 enabled;
+ struct bcm2835_isp_rational threshold;
+ struct bcm2835_isp_rational strength;
+ struct bcm2835_isp_rational limit;
+};
+
+/**
+ * enum bcm2835_isp_dpc_mode - defective pixel correction (DPC) strength.
+ *
+ * @DPC_MODE_OFF: No DPC.
+ * @DPC_MODE_NORMAL: Normal DPC.
+ * @DPC_MODE_STRONG: Strong DPC.
+ */
+enum bcm2835_isp_dpc_mode {
+ DPC_MODE_OFF = 0,
+ DPC_MODE_NORMAL = 1,
+ DPC_MODE_STRONG = 2,
+};
+
+/**
+ * struct bcm2835_isp_dpc - Defective pixel correction (DPC) parameters set
+ * with the V4L2_CID_USER_BCM2835_ISP_DPC ctrl.
+ *
+ * @enabled: Enable DPC.
+ * @strength: DPC strength (see enum &bcm2835_isp_dpc_mode).
+ */
+struct bcm2835_isp_dpc {
+ __u32 enabled;
+ __u32 strength;
+};
+
+/*
+ * ISP statistics structures.
+ *
+ * The bcm2835_isp_stats structure is generated at the output of the
+ * statistics node. Note that this does not directly map onto the statistics
+ * output of the ISP HW. Instead, the MMAL firmware code maps the HW statistics
+ * to the bcm2835_isp_stats structure.
+ */
+#define DEFAULT_AWB_REGIONS_X 16
+#define DEFAULT_AWB_REGIONS_Y 12
+
+#define NUM_HISTOGRAMS 2
+#define NUM_HISTOGRAM_BINS 128
+#define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
+#define FLOATING_REGIONS 16
+#define AGC_REGIONS 16
+#define FOCUS_REGIONS 12
+
+/**
+ * struct bcm2835_isp_stats_hist - Histogram statistics
+ *
+ * @r_hist: Red channel histogram.
+ * @g_hist: Combined green channel histogram.
+ * @b_hist: Blue channel histogram.
+ */
+struct bcm2835_isp_stats_hist {
+ __u32 r_hist[NUM_HISTOGRAM_BINS];
+ __u32 g_hist[NUM_HISTOGRAM_BINS];
+ __u32 b_hist[NUM_HISTOGRAM_BINS];
+};
+
+/**
+ * struct bcm2835_isp_stats_region - Region sums.
+ *
+ * @counted: The number of 2x2 bayer tiles accumulated.
+ * @notcounted: The number of 2x2 bayer tiles not accumulated.
+ * @r_sum: Total sum of counted pixels in the red channel for a region.
+ * @g_sum: Total sum of counted pixels in the green channel for a region.
+ * @b_sum: Total sum of counted pixels in the blue channel for a region.
+ */
+struct bcm2835_isp_stats_region {
+ __u32 counted;
+ __u32 notcounted;
+ __u64 r_sum;
+ __u64 g_sum;
+ __u64 b_sum;
+};
+
+/**
+ * struct bcm2835_isp_stats_focus - Focus statistics.
+ *
+ * @contrast_val: Focus measure - accumulated output of the focus filter.
+ * In the first dimension, index [0] counts pixels below a
+ * preset threshold, and index [1] counts pixels above the
+ * threshold. In the second dimension, index [0] uses the
+ * first predefined filter, and index [1] uses the second
+ * predefined filter.
+ * @contrast_val_num: The number of counted pixels in the above accumulation.
+ */
+struct bcm2835_isp_stats_focus {
+ __u64 contrast_val[2][2];
+ __u32 contrast_val_num[2][2];
+};
+
+/**
+ * struct bcm2835_isp_stats - ISP statistics.
+ *
+ * @version: Version of the bcm2835_isp_stats structure.
+ * @size: Size of the bcm2835_isp_stats structure.
+ * @hist: Histogram statistics for the entire image.
+ * @awb_stats: Statistics for the regions defined for AWB calculations.
+ * @floating_stats: Statistics for arbitrarily placed (floating) regions.
+ * @agc_stats: Statistics for the regions defined for AGC calculations.
+ * @focus_stats: Focus filter statistics for the focus regions.
+ */
+struct bcm2835_isp_stats {
+ __u32 version;
+ __u32 size;
+ struct bcm2835_isp_stats_hist hist[NUM_HISTOGRAMS];
+ struct bcm2835_isp_stats_region awb_stats[AWB_REGIONS];
+ struct bcm2835_isp_stats_region floating_stats[FLOATING_REGIONS];
+ struct bcm2835_isp_stats_region agc_stats[AGC_REGIONS];
+ struct bcm2835_isp_stats_focus focus_stats[FOCUS_REGIONS];
+};
+
+#endif /* __BCM2835_ISP_H_ */
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index b5e7d082b8ad..31606d42ec58 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -217,6 +217,7 @@ enum v4l2_colorfx {
* The base for Allegro driver controls.
* We reserve 16 controls for this driver.
*/
+
#define V4L2_CID_USER_ALLEGRO_BASE (V4L2_CID_USER_BASE + 0x1170)
/*
@@ -231,6 +232,11 @@ enum v4l2_colorfx {
*/
#define V4L2_CID_USER_DW100_BASE (V4L2_CID_USER_BASE + 0x1190)
+/* The base for the bcm2835-isp driver controls.
+ * We reserve 16 controls for this driver.
+ */
+#define V4L2_CID_USER_BCM2835_ISP_BASE (V4L2_CID_USER_BASE + 0x11a0)
+
/* MPEG-class control IDs */
/* The MPEG controls are applicable to all codec controls
* and the 'MPEG' part of the define is historical */
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 08/14] uapi: bcm2835-isp: Add bcm2835-isp uapi header file
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Dave Stevenson <dave.stevenson@raspberrypi.com>
This file defines the userland interface to the bcm2835-isp driver
that will follow in a separate commit.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
include/uapi/linux/bcm2835-isp.h | 320 +++++++++++++++++++++++++++++
include/uapi/linux/v4l2-controls.h | 6 +
2 files changed, 326 insertions(+)
create mode 100644 include/uapi/linux/bcm2835-isp.h
diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
new file mode 100644
index 000000000000..cf8c0437f159
--- /dev/null
+++ b/include/uapi/linux/bcm2835-isp.h
@@ -0,0 +1,320 @@
+/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */
+/*
+ * bcm2835-isp.h
+ *
+ * BCM2835 ISP driver - user space header file.
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck (naush@raspberrypi.com)
+ *
+ */
+
+#ifndef __BCM2835_ISP_H_
+#define __BCM2835_ISP_H_
+
+#include <linux/v4l2-controls.h>
+
+#define V4L2_CID_USER_BCM2835_ISP_CC_MATRIX \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0001)
+#define V4L2_CID_USER_BCM2835_ISP_LENS_SHADING \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0002)
+#define V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0003)
+#define V4L2_CID_USER_BCM2835_ISP_GEQ \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0004)
+#define V4L2_CID_USER_BCM2835_ISP_GAMMA \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0005)
+#define V4L2_CID_USER_BCM2835_ISP_DENOISE \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0006)
+#define V4L2_CID_USER_BCM2835_ISP_SHARPEN \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
+#define V4L2_CID_USER_BCM2835_ISP_DPC \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
+
+/*
+ * All structs below are directly mapped onto the equivalent structs in
+ * drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
+ * for convenience.
+ */
+
+/**
+ * struct bcm2835_isp_rational - Rational value type.
+ *
+ * @num: Numerator.
+ * @den: Denominator.
+ */
+struct bcm2835_isp_rational {
+ __s32 num;
+ __u32 den;
+};
+
+/**
+ * struct bcm2835_isp_ccm - Colour correction matrix.
+ *
+ * @ccm: 3x3 correction matrix coefficients.
+ * @offsets: 1x3 correction offsets.
+ */
+struct bcm2835_isp_ccm {
+ struct bcm2835_isp_rational ccm[3][3];
+ __s32 offsets[3];
+};
+
+/**
+ * struct bcm2835_isp_custom_ccm - Custom CCM applied with the
+ * V4L2_CID_USER_BCM2835_ISP_CC_MATRIX ctrl.
+ *
+ * @enabled: Enable custom CCM.
+ * @ccm: Custom CCM coefficients and offsets.
+ */
+struct bcm2835_isp_custom_ccm {
+ __u32 enabled;
+ struct bcm2835_isp_ccm ccm;
+};
+
+/**
+ * enum bcm2835_isp_gain_format - format of the gains in the lens shading
+ * tables used with the
+ * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING ctrl.
+ *
+ * @GAIN_FORMAT_U0P8_1: Gains are u0.8 format, starting at 1.0
+ * @GAIN_FORMAT_U1P7_0: Gains are u1.7 format, starting at 0.0
+ * @GAIN_FORMAT_U1P7_1: Gains are u1.7 format, starting at 1.0
+ * @GAIN_FORMAT_U2P6_0: Gains are u2.6 format, starting at 0.0
+ * @GAIN_FORMAT_U2P6_1: Gains are u2.6 format, starting at 1.0
+ * @GAIN_FORMAT_U3P5_0: Gains are u3.5 format, starting at 0.0
+ * @GAIN_FORMAT_U3P5_1: Gains are u3.5 format, starting at 1.0
+ * @GAIN_FORMAT_U4P10: Gains are u4.10 format, starting at 0.0
+ */
+enum bcm2835_isp_gain_format {
+ GAIN_FORMAT_U0P8_1 = 0,
+ GAIN_FORMAT_U1P7_0 = 1,
+ GAIN_FORMAT_U1P7_1 = 2,
+ GAIN_FORMAT_U2P6_0 = 3,
+ GAIN_FORMAT_U2P6_1 = 4,
+ GAIN_FORMAT_U3P5_0 = 5,
+ GAIN_FORMAT_U3P5_1 = 6,
+ GAIN_FORMAT_U4P10 = 7,
+};
+
+/**
+ * struct bcm2835_isp_lens_shading - Lens shading tables supplied with the
+ * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING
+ * ctrl.
+ *
+ * @enabled: Enable lens shading.
+ * @grid_cell_size: Size of grid cells in samples (16, 32, 64, 128 or 256).
+ * @grid_width: Width of lens shading tables in grid cells.
+ * @grid_stride: Row to row distance (in grid cells) between grid cells
+ * in the same horizontal location.
+ * @grid_height: Height of lens shading tables in grid cells.
+ * @dmabuf: dmabuf file handle containing the table.
+ * @ref_transform: Reference transform - unsupported, please pass zero.
+ * @corner_sampled: Whether the gains are sampled at the corner points
+ * of the grid cells or in the cell centres.
+ * @gain_format: Format of the gains (see enum &bcm2835_isp_gain_format).
+ */
+struct bcm2835_isp_lens_shading {
+ __u32 enabled;
+ __u32 grid_cell_size;
+ __u32 grid_width;
+ __u32 grid_stride;
+ __u32 grid_height;
+ __s32 dmabuf;
+ __u32 ref_transform;
+ __u32 corner_sampled;
+ __u32 gain_format;
+};
+
+/**
+ * struct bcm2835_isp_black_level - Sensor black level set with the
+ * V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL ctrl.
+ *
+ * @enabled: Enable black level.
+ * @black_level_r: Black level for red channel.
+ * @black_level_g: Black level for green channels.
+ * @black_level_b: Black level for blue channel.
+ */
+struct bcm2835_isp_black_level {
+ __u32 enabled;
+ __u16 black_level_r;
+ __u16 black_level_g;
+ __u16 black_level_b;
+ __u8 padding[2]; /* Unused */
+};
+
+/**
+ * struct bcm2835_isp_geq - Green equalisation parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_GEQ ctrl.
+ *
+ * @enabled: Enable green equalisation.
+ * @offset: Fixed offset of the green equalisation threshold.
+ * @slope: Slope of the green equalisation threshold.
+ */
+struct bcm2835_isp_geq {
+ __u32 enabled;
+ __u32 offset;
+ struct bcm2835_isp_rational slope;
+};
+
+#define BCM2835_NUM_GAMMA_PTS 33
+
+/**
+ * struct bcm2835_isp_gamma - Gamma parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_GAMMA ctrl.
+ *
+ * @enabled: Enable gamma adjustment.
+ * @X: X values of the points defining the gamma curve.
+ * Values should be scaled to 16 bits.
+ * @Y: Y values of the points defining the gamma curve.
+ * Values should be scaled to 16 bits.
+ */
+struct bcm2835_isp_gamma {
+ __u32 enabled;
+ __u16 x[BCM2835_NUM_GAMMA_PTS];
+ __u16 y[BCM2835_NUM_GAMMA_PTS];
+};
+
+/**
+ * struct bcm2835_isp_denoise - Denoise parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
+ *
+ * @enabled: Enable denoise.
+ * @constant: Fixed offset of the noise threshold.
+ * @slope: Slope of the noise threshold.
+ * @strength: Denoise strength between 0.0 (off) and 1.0 (maximum).
+ */
+struct bcm2835_isp_denoise {
+ __u32 enabled;
+ __u32 constant;
+ struct bcm2835_isp_rational slope;
+ struct bcm2835_isp_rational strength;
+};
+
+/**
+ * struct bcm2835_isp_sharpen - Sharpen parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_SHARPEN ctrl.
+ *
+ * @enabled: Enable sharpening.
+ * @threshold: Threshold at which to start sharpening pixels.
+ * @strength: Strength with which pixel sharpening increases.
+ * @limit: Limit to the amount of sharpening applied.
+ */
+struct bcm2835_isp_sharpen {
+ __u32 enabled;
+ struct bcm2835_isp_rational threshold;
+ struct bcm2835_isp_rational strength;
+ struct bcm2835_isp_rational limit;
+};
+
+/**
+ * enum bcm2835_isp_dpc_mode - defective pixel correction (DPC) strength.
+ *
+ * @DPC_MODE_OFF: No DPC.
+ * @DPC_MODE_NORMAL: Normal DPC.
+ * @DPC_MODE_STRONG: Strong DPC.
+ */
+enum bcm2835_isp_dpc_mode {
+ DPC_MODE_OFF = 0,
+ DPC_MODE_NORMAL = 1,
+ DPC_MODE_STRONG = 2,
+};
+
+/**
+ * struct bcm2835_isp_dpc - Defective pixel correction (DPC) parameters set
+ * with the V4L2_CID_USER_BCM2835_ISP_DPC ctrl.
+ *
+ * @enabled: Enable DPC.
+ * @strength: DPC strength (see enum &bcm2835_isp_dpc_mode).
+ */
+struct bcm2835_isp_dpc {
+ __u32 enabled;
+ __u32 strength;
+};
+
+/*
+ * ISP statistics structures.
+ *
+ * The bcm2835_isp_stats structure is generated at the output of the
+ * statistics node. Note that this does not directly map onto the statistics
+ * output of the ISP HW. Instead, the MMAL firmware code maps the HW statistics
+ * to the bcm2835_isp_stats structure.
+ */
+#define DEFAULT_AWB_REGIONS_X 16
+#define DEFAULT_AWB_REGIONS_Y 12
+
+#define NUM_HISTOGRAMS 2
+#define NUM_HISTOGRAM_BINS 128
+#define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
+#define FLOATING_REGIONS 16
+#define AGC_REGIONS 16
+#define FOCUS_REGIONS 12
+
+/**
+ * struct bcm2835_isp_stats_hist - Histogram statistics
+ *
+ * @r_hist: Red channel histogram.
+ * @g_hist: Combined green channel histogram.
+ * @b_hist: Blue channel histogram.
+ */
+struct bcm2835_isp_stats_hist {
+ __u32 r_hist[NUM_HISTOGRAM_BINS];
+ __u32 g_hist[NUM_HISTOGRAM_BINS];
+ __u32 b_hist[NUM_HISTOGRAM_BINS];
+};
+
+/**
+ * struct bcm2835_isp_stats_region - Region sums.
+ *
+ * @counted: The number of 2x2 bayer tiles accumulated.
+ * @notcounted: The number of 2x2 bayer tiles not accumulated.
+ * @r_sum: Total sum of counted pixels in the red channel for a region.
+ * @g_sum: Total sum of counted pixels in the green channel for a region.
+ * @b_sum: Total sum of counted pixels in the blue channel for a region.
+ */
+struct bcm2835_isp_stats_region {
+ __u32 counted;
+ __u32 notcounted;
+ __u64 r_sum;
+ __u64 g_sum;
+ __u64 b_sum;
+};
+
+/**
+ * struct bcm2835_isp_stats_focus - Focus statistics.
+ *
+ * @contrast_val: Focus measure - accumulated output of the focus filter.
+ * In the first dimension, index [0] counts pixels below a
+ * preset threshold, and index [1] counts pixels above the
+ * threshold. In the second dimension, index [0] uses the
+ * first predefined filter, and index [1] uses the second
+ * predefined filter.
+ * @contrast_val_num: The number of counted pixels in the above accumulation.
+ */
+struct bcm2835_isp_stats_focus {
+ __u64 contrast_val[2][2];
+ __u32 contrast_val_num[2][2];
+};
+
+/**
+ * struct bcm2835_isp_stats - ISP statistics.
+ *
+ * @version: Version of the bcm2835_isp_stats structure.
+ * @size: Size of the bcm2835_isp_stats structure.
+ * @hist: Histogram statistics for the entire image.
+ * @awb_stats: Statistics for the regions defined for AWB calculations.
+ * @floating_stats: Statistics for arbitrarily placed (floating) regions.
+ * @agc_stats: Statistics for the regions defined for AGC calculations.
+ * @focus_stats: Focus filter statistics for the focus regions.
+ */
+struct bcm2835_isp_stats {
+ __u32 version;
+ __u32 size;
+ struct bcm2835_isp_stats_hist hist[NUM_HISTOGRAMS];
+ struct bcm2835_isp_stats_region awb_stats[AWB_REGIONS];
+ struct bcm2835_isp_stats_region floating_stats[FLOATING_REGIONS];
+ struct bcm2835_isp_stats_region agc_stats[AGC_REGIONS];
+ struct bcm2835_isp_stats_focus focus_stats[FOCUS_REGIONS];
+};
+
+#endif /* __BCM2835_ISP_H_ */
diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
index b5e7d082b8ad..31606d42ec58 100644
--- a/include/uapi/linux/v4l2-controls.h
+++ b/include/uapi/linux/v4l2-controls.h
@@ -217,6 +217,7 @@ enum v4l2_colorfx {
* The base for Allegro driver controls.
* We reserve 16 controls for this driver.
*/
+
#define V4L2_CID_USER_ALLEGRO_BASE (V4L2_CID_USER_BASE + 0x1170)
/*
@@ -231,6 +232,11 @@ enum v4l2_colorfx {
*/
#define V4L2_CID_USER_DW100_BASE (V4L2_CID_USER_BASE + 0x1190)
+/* The base for the bcm2835-isp driver controls.
+ * We reserve 16 controls for this driver.
+ */
+#define V4L2_CID_USER_BCM2835_ISP_BASE (V4L2_CID_USER_BASE + 0x11a0)
+
/* MPEG-class control IDs */
/* The MPEG controls are applicable to all codec controls
* and the 'MPEG' part of the define is historical */
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 09/14] staging: vc04_services: bcm2835-isp: Add a more complex ISP processing component
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
Driver for the BCM2835 ISP hardware block. This driver uses the MMAL
component to program the ISP hardware through the VC firmware.
The ISP component can produce two video stream outputs, and Bayer
image statistics. This can't be encompassed in a simple V4L2
M2M device, so create a new device that registers 4 video nodes.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
MAINTAINERS | 9 +
drivers/staging/vc04_services/Kconfig | 2 +
drivers/staging/vc04_services/Makefile | 1 +
.../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
.../vc04_services/bcm2835-isp/Makefile | 8 +
.../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
.../bcm2835-isp/bcm2835-isp-fmts.h | 481 +++++
.../bcm2835-isp/bcm2835-v4l2-isp.c | 1708 +++++++++++++++++
.../vc04_services/vchiq-mmal/mmal-encodings.h | 4 +
.../vchiq-mmal/mmal-parameters.h | 165 +-
include/uapi/linux/bcm2835-isp.h | 27 +
11 files changed, 2490 insertions(+), 1 deletion(-)
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 6466dd5ef1fa..14e162ca0b58 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4037,6 +4037,15 @@ S: Maintained
F: drivers/media/platform/bcm2835/
F: Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
+BROADCOM BCM2835 ISP DRIVER
+M: Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
+L: linux-media@vger.kernel.org
+S: Maintained
+F: Documentation/media/uapi/v4l/pixfmt-meta-bcm2835-isp-stats.rst
+F: Documentation/media/v4l-drivers/bcm2835-isp.rst
+F: drivers/staging/vc04_services/bcm2835-isp
+F: include/uapi/linux/bcm2835-isp.h
+
BROADCOM BCM47XX MIPS ARCHITECTURE
M: Hauke Mehrtens <hauke@hauke-m.de>
M: Rafał Miłecki <zajec5@gmail.com>
diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig
index 6c0e77d64376..e71e5a8b820a 100644
--- a/drivers/staging/vc04_services/Kconfig
+++ b/drivers/staging/vc04_services/Kconfig
@@ -44,6 +44,8 @@ source "drivers/staging/vc04_services/bcm2835-audio/Kconfig"
source "drivers/staging/vc04_services/bcm2835-camera/Kconfig"
+source "drivers/staging/vc04_services/bcm2835-isp/Kconfig"
+
source "drivers/staging/vc04_services/vchiq-mmal/Kconfig"
source "drivers/staging/vc04_services/vc-sm-cma/Kconfig"
diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile
index 01089f369fb4..055cf28fc19c 100644
--- a/drivers/staging/vc04_services/Makefile
+++ b/drivers/staging/vc04_services/Makefile
@@ -15,6 +15,7 @@ obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/
obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/
obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/
obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/
+obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp/
ccflags-y += -I $(srctree)/$(src)/include
diff --git a/drivers/staging/vc04_services/bcm2835-isp/Kconfig b/drivers/staging/vc04_services/bcm2835-isp/Kconfig
new file mode 100644
index 000000000000..6222799ebe16
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/Kconfig
@@ -0,0 +1,14 @@
+config VIDEO_ISP_BCM2835
+ tristate "BCM2835 ISP support"
+ depends on MEDIA_SUPPORT
+ depends on VIDEO_DEV && (ARCH_BCM2835 || COMPILE_TEST)
+ depends on MEDIA_CONTROLLER
+ select BCM2835_VCHIQ_MMAL
+ select VIDEOBUF2_DMA_CONTIG
+ help
+ This is the V4L2 driver for the Broadcom BCM2835 ISP hardware.
+ This operates over the VCHIQ interface to a service running on
+ VideoCore.
+
+ To compile this driver as a module, choose M here: the module
+ will be called bcm2835-isp.
diff --git a/drivers/staging/vc04_services/bcm2835-isp/Makefile b/drivers/staging/vc04_services/bcm2835-isp/Makefile
new file mode 100644
index 000000000000..42d3081d342f
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+bcm2835-isp-objs := bcm2835-v4l2-isp.o
+
+obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp.o
+
+ccflags-y += \
+ -I$(srctree)/drivers/staging/vc04_services \
+ -D__VCCOREVER__=0x04000000
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
new file mode 100644
index 000000000000..172605718cdf
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck (naush@raspberrypi.com)
+ *
+ */
+
+#ifndef BCM2835_ISP_CTRLS
+#define BCM2835_ISP_CTRLS
+
+#include <linux/bcm2835-isp.h>
+
+struct bcm2835_isp_custom_ctrl {
+ const char *name;
+ u32 id;
+ u32 size;
+ u32 flags;
+};
+
+static const struct bcm2835_isp_custom_ctrl custom_ctrls[] = {
+ {
+ .name = "Colour Correction Matrix",
+ .id = V4L2_CID_USER_BCM2835_ISP_CC_MATRIX,
+ .size = sizeof(struct bcm2835_isp_custom_ccm),
+ .flags = 0
+ }, {
+ .name = "Lens Shading",
+ .id = V4L2_CID_USER_BCM2835_ISP_LENS_SHADING,
+ .size = sizeof(struct bcm2835_isp_lens_shading),
+ .flags = V4L2_CTRL_FLAG_EXECUTE_ON_WRITE
+ }, {
+ .name = "Black Level",
+ .id = V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL,
+ .size = sizeof(struct bcm2835_isp_black_level),
+ .flags = 0
+ }, {
+ .name = "Green Equalisation",
+ .id = V4L2_CID_USER_BCM2835_ISP_GEQ,
+ .size = sizeof(struct bcm2835_isp_geq),
+ .flags = 0
+ }, {
+ .name = "Gamma",
+ .id = V4L2_CID_USER_BCM2835_ISP_GAMMA,
+ .size = sizeof(struct bcm2835_isp_gamma),
+ .flags = 0
+ }, {
+ .name = "Sharpen",
+ .id = V4L2_CID_USER_BCM2835_ISP_SHARPEN,
+ .size = sizeof(struct bcm2835_isp_sharpen),
+ .flags = 0
+ }, {
+ .name = "Denoise",
+ .id = V4L2_CID_USER_BCM2835_ISP_DENOISE,
+ .size = sizeof(struct bcm2835_isp_denoise),
+ .flags = 0
+ }, {
+ .name = "Colour Denoise",
+ .id = V4L2_CID_USER_BCM2835_ISP_CDN,
+ .size = sizeof(struct bcm2835_isp_cdn),
+ .flags = 0
+ }, {
+ .name = "Defective Pixel Correction",
+ .id = V4L2_CID_USER_BCM2835_ISP_DPC,
+ .size = sizeof(struct bcm2835_isp_dpc),
+ .flags = 0
+ }
+};
+
+#endif
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
new file mode 100644
index 000000000000..65913a4ff203
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
@@ -0,0 +1,481 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck (naush@raspberrypi.com)
+ *
+ */
+
+#ifndef BCM2835_ISP_FMTS
+#define BCM2835_ISP_FMTS
+
+#include <linux/videodev2.h>
+#include "vchiq-mmal/mmal-encodings.h"
+
+struct bcm2835_isp_fmt {
+ u32 fourcc;
+ int depth;
+ int bytesperline_align;
+ u32 mmal_fmt;
+ int size_multiplier_x2;
+ enum v4l2_colorspace colorspace;
+ unsigned int step_size;
+};
+
+static const struct bcm2835_isp_fmt supported_formats[] = {
+ {
+ /* YUV formats */
+ .fourcc = V4L2_PIX_FMT_YUV420,
+ .depth = 8,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_I420,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YVU420,
+ .depth = 8,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YV12,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_NV12,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_NV12,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_NV21,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_NV21,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YUYV,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YUYV,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_UYVY,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_UYVY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YVYU,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YVYU,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_VYUY,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_VYUY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ /* RGB formats */
+ .fourcc = V4L2_PIX_FMT_RGB24,
+ .depth = 24,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_RGB24,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGB565,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_RGB16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_BGR24,
+ .depth = 24,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BGR24,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_XBGR32,
+ .depth = 32,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_BGRA,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGBX32,
+ .depth = 32,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_RGBA,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ /* Bayer formats */
+ /* 8 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 16 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* Bayer formats unpacked to 16bpp */
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* Monochrome MIPI formats */
+ /* 8 bit */
+ .fourcc = V4L2_PIX_FMT_GREY,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_GREY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_Y10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_Y12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_Y14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 16 bit */
+ .fourcc = V4L2_PIX_FMT_Y16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_META_FMT_BCM2835_ISP_STATS,
+ .depth = 8,
+ .mmal_fmt = MMAL_ENCODING_BRCM_STATS,
+ /* The rest are not valid fields for stats. */
+ }
+};
+
+#endif
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
new file mode 100644
index 000000000000..eb5f076fc6dc
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
@@ -0,0 +1,1708 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck (naush@raspberrypi.com)
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/platform_device.h>
+
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "vchiq-mmal/mmal-msg.h"
+#include "vchiq-mmal/mmal-parameters.h"
+#include "vchiq-mmal/mmal-vchiq.h"
+
+#include "vc-sm-cma/vc_sm_knl.h"
+
+#include "bcm2835-isp-ctrls.h"
+#include "bcm2835-isp-fmts.h"
+
+MODULE_IMPORT_NS(DMA_BUF);
+
+static unsigned int debug;
+module_param(debug, uint, 0644);
+MODULE_PARM_DESC(debug, "activates debug info");
+
+static unsigned int video_nr = 13;
+module_param(video_nr, uint, 0644);
+MODULE_PARM_DESC(video_nr, "base video device number");
+
+#define BCM2835_ISP_NAME "bcm2835-isp"
+#define BCM2835_ISP_ENTITY_NAME_LEN 32
+
+#define BCM2835_ISP_NUM_OUTPUTS 1
+#define BCM2835_ISP_NUM_CAPTURES 2
+#define BCM2835_ISP_NUM_METADATA 1
+
+#define BCM2835_ISP_NUM_NODES \
+ (BCM2835_ISP_NUM_OUTPUTS + BCM2835_ISP_NUM_CAPTURES + \
+ BCM2835_ISP_NUM_METADATA)
+
+/* Default frame dimension of 1280 pixels. */
+#define DEFAULT_DIM 1280U
+/*
+ * Maximum frame dimension of 16384 pixels. Even though the ISP runs in tiles,
+ * have a sensible limit so that we do not create an excessive number of tiles
+ * to process.
+ */
+#define MAX_DIM 16384U
+/*
+ * Minimum frame dimension of 64 pixels. Anything lower, and the tiling
+ * algorithm may not be able to cope when applying filter context.
+ */
+#define MIN_DIM 64U
+
+/* Timeout for stop_streaming to allow all buffers to return */
+#define COMPLETE_TIMEOUT (2 * HZ)
+
+/* Per-queue, driver-specific private data */
+struct bcm2835_isp_q_data {
+ /*
+ * These parameters should be treated as gospel, with everything else
+ * being determined from them.
+ */
+ unsigned int bytesperline;
+ unsigned int width;
+ unsigned int height;
+ unsigned int sizeimage;
+ const struct bcm2835_isp_fmt *fmt;
+};
+
+/*
+ * Structure to describe a single node /dev/video<N> which represents a single
+ * input or output queue to the ISP device.
+ */
+struct bcm2835_isp_node {
+ int vfl_dir;
+ unsigned int id;
+ const char *name;
+ struct vchiq_mmal_port *port;
+ struct video_device vfd;
+ struct media_pad pad;
+ struct media_intf_devnode *intf_devnode;
+ struct media_link *intf_link;
+ struct mutex lock; /* top level device node lock */
+ struct mutex queue_lock;
+
+ struct vb2_queue queue;
+ unsigned int sequence;
+
+ /* The list of formats supported on the node. */
+ struct bcm2835_isp_fmt const **supported_fmts;
+ unsigned int num_supported_fmts;
+
+ struct bcm2835_isp_q_data q_data;
+
+ /* Parent device structure */
+ struct bcm2835_isp_dev *dev;
+
+ bool registered;
+ bool media_node_registered;
+};
+
+/*
+ * Structure representing the entire ISP device, comprising several input and
+ * output nodes /dev/video<N>.
+ */
+struct bcm2835_isp_dev {
+ struct v4l2_device v4l2_dev;
+ struct device *dev;
+ struct v4l2_ctrl_handler ctrl_handler;
+ struct media_device mdev;
+ struct media_entity entity;
+ bool media_device_registered;
+ bool media_entity_registered;
+ struct vchiq_mmal_instance *mmal_instance;
+ struct vchiq_mmal_component *component;
+ struct completion frame_cmplt;
+
+ struct bcm2835_isp_node node[BCM2835_ISP_NUM_NODES];
+ struct media_pad pad[BCM2835_ISP_NUM_NODES];
+ atomic_t num_streaming;
+
+ /* Image pipeline controls. */
+ int r_gain;
+ int b_gain;
+};
+
+struct bcm2835_isp_buffer {
+ struct vb2_v4l2_buffer vb;
+ struct mmal_buffer mmal;
+};
+
+static
+inline struct bcm2835_isp_dev *node_get_dev(struct bcm2835_isp_node *node)
+{
+ return node->dev;
+}
+
+static inline bool node_is_output(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_VIDEO_OUTPUT;
+}
+
+static inline bool node_is_capture(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE;
+}
+
+static inline bool node_is_stats(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_META_CAPTURE;
+}
+
+static inline enum v4l2_buf_type index_to_queue_type(int index)
+{
+ if (index < BCM2835_ISP_NUM_OUTPUTS)
+ return V4L2_BUF_TYPE_VIDEO_OUTPUT;
+ else if (index < BCM2835_ISP_NUM_OUTPUTS + BCM2835_ISP_NUM_CAPTURES)
+ return V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ else
+ return V4L2_BUF_TYPE_META_CAPTURE;
+}
+
+static int set_isp_param(struct bcm2835_isp_node *node, u32 parameter,
+ void *value, u32 value_size)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ return vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ parameter, value, value_size);
+}
+
+static int set_wb_gains(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct mmal_parameter_awbgains gains = {
+ .r_gain = { dev->r_gain, 1000 },
+ .b_gain = { dev->b_gain, 1000 }
+ };
+
+ return set_isp_param(node, MMAL_PARAMETER_CUSTOM_AWB_GAINS,
+ &gains, sizeof(gains));
+}
+
+static int set_digital_gain(struct bcm2835_isp_node *node, uint32_t gain)
+{
+ struct s32_fract digital_gain = {
+ .numerator = gain,
+ .denominator = 1000
+ };
+
+ return set_isp_param(node, MMAL_PARAMETER_DIGITAL_GAIN,
+ &digital_gain, sizeof(digital_gain));
+}
+
+static const struct bcm2835_isp_fmt *get_fmt(u32 mmal_fmt)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(supported_formats); i++) {
+ if (supported_formats[i].mmal_fmt == mmal_fmt)
+ return &supported_formats[i];
+ }
+ return NULL;
+}
+
+static const
+struct bcm2835_isp_fmt *find_format_by_fourcc(unsigned int fourcc,
+ struct bcm2835_isp_node *node)
+{
+ const struct bcm2835_isp_fmt *fmt;
+ unsigned int i;
+
+ for (i = 0; i < node->num_supported_fmts; i++) {
+ fmt = node->supported_fmts[i];
+ if (fmt->fourcc == fourcc)
+ return fmt;
+ }
+
+ return NULL;
+}
+
+static const
+struct bcm2835_isp_fmt *find_format(struct v4l2_format *f,
+ struct bcm2835_isp_node *node)
+{
+ return find_format_by_fourcc(node_is_stats(node) ?
+ f->fmt.meta.dataformat :
+ f->fmt.pix.pixelformat,
+ node);
+}
+
+/* vb2_to_mmal_buffer() - converts vb2 buffer header to MMAL
+ *
+ * Copies all the required fields from a VB2 buffer to the MMAL buffer header,
+ * ready for sending to the VPU.
+ */
+static void vb2_to_mmal_buffer(struct mmal_buffer *buf,
+ struct vb2_v4l2_buffer *vb2)
+{
+ u64 pts;
+
+ buf->mmal_flags = 0;
+ if (vb2->flags & V4L2_BUF_FLAG_KEYFRAME)
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_KEYFRAME;
+
+ /* Data must be framed correctly as one frame per buffer. */
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_FRAME_END;
+
+ buf->length = vb2->vb2_buf.planes[0].bytesused;
+ /*
+ * Minor ambiguity in the V4L2 spec as to whether passing in a 0 length
+ * buffer, or one with V4L2_BUF_FLAG_LAST set denotes end of stream.
+ * Handle either.
+ */
+ if (!buf->length || vb2->flags & V4L2_BUF_FLAG_LAST)
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_EOS;
+
+ /* vb2 timestamps in nsecs, mmal in usecs */
+ pts = vb2->vb2_buf.timestamp;
+ do_div(pts, 1000);
+ buf->pts = pts;
+ buf->dts = MMAL_TIME_UNKNOWN;
+}
+
+static void mmal_buffer_cb(struct vchiq_mmal_instance *instance,
+ struct vchiq_mmal_port *port, int status,
+ struct mmal_buffer *mmal_buf)
+{
+ struct bcm2835_isp_buffer *q_buf;
+ struct bcm2835_isp_node *node = port->cb_ctx;
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2;
+
+ q_buf = container_of(mmal_buf, struct bcm2835_isp_buffer, mmal);
+ vb2 = &q_buf->vb;
+ v4l2_dbg(2, debug, &dev->v4l2_dev,
+ "%s: port:%s[%d], status:%d, buf:%p, dmabuf:%p, length:%lu, flags %u, pts %lld\n",
+ __func__, node_is_output(node) ? "input" : "output", node->id,
+ status, mmal_buf, mmal_buf->dma_buf, mmal_buf->length,
+ mmal_buf->mmal_flags, mmal_buf->pts);
+
+ if (mmal_buf->cmd)
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Unexpected event on output callback - %08x\n",
+ __func__, mmal_buf->cmd);
+
+ if (status) {
+ /* error in transfer */
+ if (vb2) {
+ /* there was a buffer with the error so return it */
+ vb2_buffer_done(&vb2->vb2_buf, VB2_BUF_STATE_ERROR);
+ }
+ return;
+ }
+
+ /* vb2 timestamps in nsecs, mmal in usecs */
+ vb2->vb2_buf.timestamp = mmal_buf->pts * 1000;
+ vb2->sequence = node->sequence++;
+ vb2_set_plane_payload(&vb2->vb2_buf, 0, mmal_buf->length);
+ vb2_buffer_done(&vb2->vb2_buf, VB2_BUF_STATE_DONE);
+
+ if (!port->enabled)
+ complete(&dev->frame_cmplt);
+}
+
+static void setup_mmal_port_format(struct bcm2835_isp_node *node,
+ struct vchiq_mmal_port *port)
+{
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+
+ port->format.encoding = q_data->fmt->mmal_fmt;
+ /* Raw image format - set width/height */
+ port->es.video.width = (q_data->bytesperline << 3) / q_data->fmt->depth;
+ port->es.video.height = q_data->height;
+ port->es.video.crop.width = q_data->width;
+ port->es.video.crop.height = q_data->height;
+ port->es.video.crop.x = 0;
+ port->es.video.crop.y = 0;
+};
+
+static int setup_mmal_port(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ unsigned int enable = 1;
+ int ret;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "%s: setup %s[%d]\n", __func__,
+ node->name, node->id);
+
+ vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_ZERO_COPY, &enable,
+ sizeof(enable));
+ setup_mmal_port_format(node, node->port);
+ ret = vchiq_mmal_port_set_format(dev->mmal_instance, node->port);
+ if (ret < 0) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: vchiq_mmal_port_set_format failed\n",
+ __func__);
+ return ret;
+ }
+
+ if (node->q_data.sizeimage < node->port->minimum_buffer.size) {
+ v4l2_err(&dev->v4l2_dev,
+ "buffer size mismatch sizeimage %u < min size %u\n",
+ node->q_data.sizeimage,
+ node->port->minimum_buffer.size);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_mmal_buf_cleanup(struct mmal_buffer *mmal_buf)
+{
+ mmal_vchi_buffer_cleanup(mmal_buf);
+
+ if (mmal_buf->dma_buf) {
+ dma_buf_put(mmal_buf->dma_buf);
+ mmal_buf->dma_buf = NULL;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_queue_setup(struct vb2_queue *q,
+ unsigned int *nbuffers,
+ unsigned int *nplanes,
+ unsigned int sizes[],
+ struct device *alloc_devs[])
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ unsigned int size;
+
+ if (setup_mmal_port(node))
+ return -EINVAL;
+
+ size = node->q_data.sizeimage;
+ if (size == 0) {
+ v4l2_info(&node_get_dev(node)->v4l2_dev,
+ "%s: Image size unset in queue_setup for node %s[%d]\n",
+ __func__, node->name, node->id);
+ return -EINVAL;
+ }
+
+ if (*nplanes)
+ return sizes[0] < size ? -EINVAL : 0;
+
+ *nplanes = 1;
+ sizes[0] = size;
+
+ node->port->current_buffer.size = size;
+
+ if (*nbuffers < node->port->minimum_buffer.num)
+ *nbuffers = node->port->minimum_buffer.num;
+
+ node->port->current_buffer.num = *nbuffers;
+
+ v4l2_dbg(2, debug, &node_get_dev(node)->v4l2_dev,
+ "%s: Image size %u, nbuffers %u for node %s[%d]\n",
+ __func__, sizes[0], *nbuffers, node->name, node->id);
+ return 0;
+}
+
+static int bcm2835_isp_buf_init(struct vb2_buffer *vb)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(vb->vb2_queue);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buf =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: vb %p\n", __func__, vb);
+
+ buf->mmal.buffer = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
+ buf->mmal.buffer_size = vb2_plane_size(&buf->vb.vb2_buf, 0);
+ mmal_vchi_buffer_init(dev->mmal_instance, &buf->mmal);
+ return 0;
+}
+
+static int bcm2835_isp_buf_prepare(struct vb2_buffer *vb)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(vb->vb2_queue);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buf =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+ struct dma_buf *dma_buf;
+ int ret;
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: type: %d ptr %p\n",
+ __func__, vb->vb2_queue->type, vb);
+
+ if (V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type)) {
+ if (vb2->field == V4L2_FIELD_ANY)
+ vb2->field = V4L2_FIELD_NONE;
+ if (vb2->field != V4L2_FIELD_NONE) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s field isn't supported\n", __func__);
+ return -EINVAL;
+ }
+ }
+
+ if (vb2_plane_size(vb, 0) < node->q_data.sizeimage) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s data will not fit into plane (%lu < %lu)\n",
+ __func__, vb2_plane_size(vb, 0),
+ (long)node->q_data.sizeimage);
+ return -EINVAL;
+ }
+
+ if (!V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type))
+ vb2_set_plane_payload(vb, 0, node->q_data.sizeimage);
+
+ switch (vb->memory) {
+ case VB2_MEMORY_DMABUF:
+ dma_buf = dma_buf_get(vb->planes[0].m.fd);
+
+ if (dma_buf != buf->mmal.dma_buf) {
+ /*
+ * dmabuf either hasn't already been mapped, or it has
+ * changed.
+ */
+ if (buf->mmal.dma_buf) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s Buffer changed - why did the core not call cleanup?\n",
+ __func__);
+ bcm2835_isp_mmal_buf_cleanup(&buf->mmal);
+ }
+
+ buf->mmal.dma_buf = dma_buf;
+ } else {
+ /*
+ * Already have a reference to the buffer, so release it
+ * here.
+ */
+ dma_buf_put(dma_buf);
+ }
+ ret = 0;
+ break;
+ case VB2_MEMORY_MMAP:
+ /*
+ * We want to do this at init, but vb2_core_expbuf checks that
+ * the index < q->num_buffers, and q->num_buffers only gets
+ * updated once all the buffers are allocated.
+ */
+ if (!buf->mmal.dma_buf) {
+ buf->mmal.dma_buf = vb2_core_expbuf_dmabuf(vb->vb2_queue,
+ vb->vb2_queue->type,
+ vb->index, 0, O_CLOEXEC);
+ v4l2_dbg(3, debug, &dev->v4l2_dev,
+ "%s: exporting ptr %p to dmabuf %p\n",
+ __func__, vb, buf->mmal.dma_buf);
+ if (IS_ERR(buf->mmal.dma_buf)) {
+ ret = PTR_ERR(buf->mmal.dma_buf);
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed to expbuf idx %d, ret %d\n",
+ __func__, vb->index, ret);
+ }
+ } else {
+ ret = 0;
+ }
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void bcm2835_isp_node_buffer_queue(struct vb2_buffer *buf)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(buf->vb2_queue);
+ struct vb2_v4l2_buffer *vbuf =
+ container_of(buf, struct vb2_v4l2_buffer, vb2_buf);
+ struct bcm2835_isp_buffer *buffer =
+ container_of(vbuf, struct bcm2835_isp_buffer, vb);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: node %s[%d], buffer %p\n",
+ __func__, node->name, node->id, buffer);
+
+ vb2_to_mmal_buffer(&buffer->mmal, &buffer->vb);
+ v4l2_dbg(3, debug, &dev->v4l2_dev,
+ "%s: node %s[%d] - submitting mmal dmabuf %p\n", __func__,
+ node->name, node->id, buffer->mmal.dma_buf);
+ vchiq_mmal_submit_buffer(dev->mmal_instance, node->port, &buffer->mmal);
+}
+
+static void bcm2835_isp_buffer_cleanup(struct vb2_buffer *vb)
+{
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buffer =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+
+ bcm2835_isp_mmal_buf_cleanup(&buffer->mmal);
+}
+
+static int bcm2835_isp_node_start_streaming(struct vb2_queue *q,
+ unsigned int count)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ int ret;
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: node %s[%d] (count %u)\n",
+ __func__, node->name, node->id, count);
+
+ ret = vchiq_mmal_component_enable(dev->mmal_instance, dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "%s: Failed enabling component, ret %d\n",
+ __func__, ret);
+ return -EIO;
+ }
+
+ node->sequence = 0;
+ node->port->cb_ctx = node;
+ ret = vchiq_mmal_port_enable(dev->mmal_instance, node->port,
+ mmal_buffer_cb);
+ if (!ret)
+ atomic_inc(&dev->num_streaming);
+ else
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed enabling port, ret %d\n", __func__, ret);
+
+ return ret;
+}
+
+static void bcm2835_isp_node_stop_streaming(struct vb2_queue *q)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ int ret;
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: node %s[%d], mmal port %p\n",
+ __func__, node->name, node->id, node->port);
+
+ init_completion(&dev->frame_cmplt);
+
+ /* Disable MMAL port - this will flush buffers back */
+ ret = vchiq_mmal_port_disable(dev->mmal_instance, node->port);
+ if (ret)
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed disabling %s port, ret %d\n", __func__,
+ node_is_output(node) ? "i/p" : "o/p",
+ ret);
+
+ while (atomic_read(&node->port->buffers_with_vpu)) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Waiting for buffers to be returned - %d outstanding\n",
+ __func__, atomic_read(&node->port->buffers_with_vpu));
+ ret = wait_for_completion_timeout(&dev->frame_cmplt,
+ COMPLETE_TIMEOUT);
+ if (ret <= 0) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Timeout waiting for buffers to be returned - %d outstanding\n",
+ __func__,
+ atomic_read(&node->port->buffers_with_vpu));
+ break;
+ }
+ }
+
+ atomic_dec(&dev->num_streaming);
+ /* If all ports disabled, then disable the component */
+ if (atomic_read(&dev->num_streaming) == 0) {
+ struct bcm2835_isp_lens_shading ls;
+ /*
+ * The ISP component on the firmware has a reference to the
+ * dmabuf handle for the lens shading table. Pass a null handle
+ * to remove that reference now.
+ */
+ memset(&ls, 0, sizeof(ls));
+ /* Must set a valid grid size for the FW */
+ ls.grid_cell_size = 16;
+ set_isp_param(&dev->node[0],
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ &ls, sizeof(ls));
+
+ ret = vchiq_mmal_component_disable(dev->mmal_instance,
+ dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed disabling component, ret %d\n",
+ __func__, ret);
+ }
+ }
+
+ /*
+ * Simply wait for any vb2 buffers to finish. We could take steps to
+ * make them complete more quickly if we care, or even return them
+ * ourselves.
+ */
+ vb2_wait_for_all_buffers(&node->queue);
+}
+
+static const struct vb2_ops bcm2835_isp_node_queue_ops = {
+ .queue_setup = bcm2835_isp_node_queue_setup,
+ .buf_init = bcm2835_isp_buf_init,
+ .buf_prepare = bcm2835_isp_buf_prepare,
+ .buf_queue = bcm2835_isp_node_buffer_queue,
+ .buf_cleanup = bcm2835_isp_buffer_cleanup,
+ .start_streaming = bcm2835_isp_node_start_streaming,
+ .stop_streaming = bcm2835_isp_node_stop_streaming,
+};
+
+static const
+struct bcm2835_isp_fmt *get_default_format(struct bcm2835_isp_node *node)
+{
+ return node->supported_fmts[0];
+}
+
+static inline unsigned int get_bytesperline(int width,
+ const struct bcm2835_isp_fmt *fmt)
+{
+ /* GPU aligns 24bpp images to a multiple of 32 pixels (not bytes). */
+ if (fmt->depth == 24)
+ return ALIGN(width, 32) * 3;
+ else
+ return ALIGN((width * fmt->depth) >> 3, fmt->bytesperline_align);
+}
+
+static inline unsigned int get_sizeimage(int bpl, int width, int height,
+ const struct bcm2835_isp_fmt *fmt)
+{
+ return (bpl * height * fmt->size_multiplier_x2) >> 1;
+}
+
+static int bcm2835_isp_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct bcm2835_isp_dev *dev =
+ container_of(ctrl->handler, struct bcm2835_isp_dev, ctrl_handler);
+ struct bcm2835_isp_node *node = &dev->node[0];
+ int ret = 0;
+
+ /*
+ * The ISP firmware driver will ensure these settings are applied on
+ * a frame boundary, so we are safe to write them as they come in.
+ *
+ * Note that the bcm2835_isp_* param structures are identical to the
+ * mmal-parameters.h definitions. This avoids the need for unnecessary
+ * field-by-field copying between structures.
+ */
+ switch (ctrl->id) {
+ case V4L2_CID_RED_BALANCE:
+ dev->r_gain = ctrl->val;
+ ret = set_wb_gains(node);
+ break;
+ case V4L2_CID_BLUE_BALANCE:
+ dev->b_gain = ctrl->val;
+ ret = set_wb_gains(node);
+ break;
+ case V4L2_CID_DIGITAL_GAIN:
+ ret = set_digital_gain(node, ctrl->val);
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_CC_MATRIX:
+ ret = set_isp_param(node, MMAL_PARAMETER_CUSTOM_CCM,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_custom_ccm));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_LENS_SHADING:
+ {
+ struct bcm2835_isp_lens_shading *v4l2_ls;
+ struct mmal_parameter_lens_shading_v2 ls;
+ struct dma_buf *dmabuf;
+ void *vcsm_handle;
+
+ v4l2_ls = (struct bcm2835_isp_lens_shading *)ctrl->p_new.p_u8;
+ /*
+ * struct bcm2835_isp_lens_shading and struct
+ * mmal_parameter_lens_shading_v2 match so that we can do a
+ * simple memcpy here.
+ * Only the dmabuf to the actual table needs any manipulation.
+ */
+ memcpy(&ls, v4l2_ls, sizeof(ls));
+
+ dmabuf = dma_buf_get(v4l2_ls->dmabuf);
+ if (IS_ERR_OR_NULL(dmabuf))
+ return -EINVAL;
+
+ ret = vc_sm_cma_import_dmabuf(dmabuf, &vcsm_handle);
+ if (ret) {
+ dma_buf_put(dmabuf);
+ return -EINVAL;
+ }
+
+ ls.mem_handle_table = vc_sm_cma_int_handle(vcsm_handle);
+ if (ls.mem_handle_table)
+ /* The VPU will take a reference on the vcsm handle,
+ * which in turn will retain a reference on the dmabuf.
+ * This code can therefore safely release all
+ * references to the buffer.
+ */
+ ret = set_isp_param(node,
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ &ls,
+ sizeof(ls));
+ else
+ ret = -EINVAL;
+
+ vc_sm_cma_free(vcsm_handle);
+ dma_buf_put(dmabuf);
+ break;
+ }
+ case V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL:
+ ret = set_isp_param(node, MMAL_PARAMETER_BLACK_LEVEL,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_black_level));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_GEQ:
+ ret = set_isp_param(node, MMAL_PARAMETER_GEQ,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_geq));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_GAMMA:
+ ret = set_isp_param(node, MMAL_PARAMETER_GAMMA,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_gamma));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_DENOISE:
+ ret = set_isp_param(node, MMAL_PARAMETER_DENOISE,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_denoise));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_CDN:
+ ret = set_isp_param(node, MMAL_PARAMETER_CDN,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_cdn));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_SHARPEN:
+ ret = set_isp_param(node, MMAL_PARAMETER_SHARPEN,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_sharpen));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_DPC:
+ ret = set_isp_param(node, MMAL_PARAMETER_DPC,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_dpc));
+ break;
+ default:
+ v4l2_info(&dev->v4l2_dev, "Unrecognised control\n");
+ ret = -EINVAL;
+ }
+
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "%s: Failed setting ctrl \"%s\" (%08x), err %d\n",
+ __func__, ctrl->name, ctrl->id, ret);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+static const struct v4l2_ctrl_ops bcm2835_isp_ctrl_ops = {
+ .s_ctrl = bcm2835_isp_s_ctrl,
+};
+
+static const struct v4l2_file_operations bcm2835_isp_fops = {
+ .owner = THIS_MODULE,
+ .open = v4l2_fh_open,
+ .release = vb2_fop_release,
+ .poll = vb2_fop_poll,
+ .unlocked_ioctl = video_ioctl2,
+ .mmap = vb2_fop_mmap
+};
+
+static int populate_qdata_fmt(struct v4l2_format *f,
+ struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+ int ret;
+
+ if (!node_is_stats(node)) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Setting pix format for type %d, wxh: %ux%u, fmt: %08x, size %u\n",
+ __func__, f->type, f->fmt.pix.width, f->fmt.pix.height,
+ f->fmt.pix.pixelformat, f->fmt.pix.sizeimage);
+
+ q_data->fmt = find_format(f, node);
+ q_data->width = f->fmt.pix.width;
+ q_data->height = f->fmt.pix.height;
+ q_data->height = f->fmt.pix.height;
+
+ /* All parameters should have been set correctly by try_fmt */
+ q_data->bytesperline = f->fmt.pix.bytesperline;
+ q_data->sizeimage = f->fmt.pix.sizeimage;
+ } else {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Setting meta format for fmt: %08x, size %u\n",
+ __func__, f->fmt.meta.dataformat,
+ f->fmt.meta.buffersize);
+
+ q_data->fmt = find_format(f, node);
+ q_data->width = 0;
+ q_data->height = 0;
+ q_data->bytesperline = 0;
+ q_data->sizeimage = f->fmt.meta.buffersize;
+ }
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Calculated bpl as %u, size %u\n", __func__,
+ q_data->bytesperline, q_data->sizeimage);
+
+ setup_mmal_port_format(node, node->port);
+ ret = vchiq_mmal_port_set_format(dev->mmal_instance, node->port);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed vchiq_mmal_port_set_format on port, ret %d\n",
+ __func__, ret);
+ ret = -EINVAL;
+ }
+
+ if (q_data->sizeimage < node->port->minimum_buffer.size) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Current buffer size of %u < min buf size %u - driver mismatch to MMAL\n",
+ __func__,
+ q_data->sizeimage,
+ node->port->minimum_buffer.size);
+ }
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Set format for type %d, wxh: %dx%d, fmt: %08x, size %u\n",
+ __func__, f->type, q_data->width, q_data->height,
+ q_data->fmt->fourcc, q_data->sizeimage);
+
+ return ret;
+}
+
+static int bcm2835_isp_node_querycap(struct file *file, void *priv,
+ struct v4l2_capability *cap)
+{
+ strscpy(cap->driver, BCM2835_ISP_NAME, sizeof(cap->driver));
+ strscpy(cap->card, BCM2835_ISP_NAME, sizeof(cap->card));
+ snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
+ BCM2835_ISP_NAME);
+
+ return 0;
+}
+
+static int bcm2835_isp_node_g_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ if (node_is_stats(node)) {
+ f->fmt.meta.dataformat = V4L2_META_FMT_BCM2835_ISP_STATS;
+ f->fmt.meta.buffersize =
+ node->port->minimum_buffer.size;
+ } else {
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+
+ f->fmt.pix.width = q_data->width;
+ f->fmt.pix.height = q_data->height;
+ f->fmt.pix.field = V4L2_FIELD_NONE;
+ f->fmt.pix.pixelformat = q_data->fmt->fourcc;
+ f->fmt.pix.bytesperline = q_data->bytesperline;
+ f->fmt.pix.sizeimage = q_data->sizeimage;
+ f->fmt.pix.colorspace = q_data->fmt->colorspace;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_enum_fmt(struct file *file, void *priv,
+ struct v4l2_fmtdesc *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ if (f->index < node->num_supported_fmts) {
+ /* Format found */
+ f->pixelformat = node->supported_fmts[f->index]->fourcc;
+ f->flags = 0;
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+static int bcm2835_isp_enum_framesizes(struct file *file, void *priv,
+ struct v4l2_frmsizeenum *fsize)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ const struct bcm2835_isp_fmt *fmt;
+
+ if (node_is_stats(node) || fsize->index)
+ return -EINVAL;
+
+ fmt = find_format_by_fourcc(fsize->pixel_format, node);
+ if (!fmt) {
+ v4l2_err(&dev->v4l2_dev, "Invalid pixel code: %x\n",
+ fsize->pixel_format);
+ return -EINVAL;
+ }
+
+ fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
+ fsize->stepwise.min_width = MIN_DIM;
+ fsize->stepwise.max_width = MAX_DIM;
+ fsize->stepwise.step_width = fmt->step_size;
+
+ fsize->stepwise.min_height = MIN_DIM;
+ fsize->stepwise.max_height = MAX_DIM;
+ fsize->stepwise.step_height = fmt->step_size;
+
+ return 0;
+}
+
+static int bcm2835_isp_node_try_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ const struct bcm2835_isp_fmt *fmt;
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ fmt = find_format(f, node);
+ if (!fmt)
+ fmt = get_default_format(node);
+
+ if (!node_is_stats(node)) {
+ f->fmt.pix.width = max(min(f->fmt.pix.width, MAX_DIM),
+ MIN_DIM);
+ f->fmt.pix.height = max(min(f->fmt.pix.height, MAX_DIM),
+ MIN_DIM);
+
+ f->fmt.pix.pixelformat = fmt->fourcc;
+ f->fmt.pix.colorspace = fmt->colorspace;
+ f->fmt.pix.bytesperline = get_bytesperline(f->fmt.pix.width,
+ fmt);
+ f->fmt.pix.field = V4L2_FIELD_NONE;
+ f->fmt.pix.sizeimage =
+ get_sizeimage(f->fmt.pix.bytesperline, f->fmt.pix.width,
+ f->fmt.pix.height, fmt);
+ } else {
+ f->fmt.meta.dataformat = fmt->fourcc;
+ f->fmt.meta.buffersize = node->port->minimum_buffer.size;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_s_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ int ret;
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ ret = bcm2835_isp_node_try_fmt(file, priv, f);
+ if (ret)
+ return ret;
+
+ v4l2_dbg(1, debug, &node_get_dev(node)->v4l2_dev,
+ "%s: Set format for node %s[%d]\n",
+ __func__, node->name, node->id);
+
+ return populate_qdata_fmt(f, node);
+}
+
+static int bcm2835_isp_node_s_selection(struct file *file, void *fh,
+ struct v4l2_selection *s)
+{
+ struct mmal_parameter_crop crop;
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ /* This return value is required fro V4L2 compliance. */
+ if (node_is_stats(node))
+ return -ENOTTY;
+
+ if (!s->r.width || !s->r.height)
+ return -EINVAL;
+
+ /* We can only set crop on the input. */
+ switch (s->target) {
+ case V4L2_SEL_TGT_CROP:
+ /*
+ * Adjust the crop window if it goes outside of the frame
+ * dimensions.
+ */
+ s->r.left = min((unsigned int)max(s->r.left, 0),
+ node->q_data.width - MIN_DIM);
+ s->r.top = min((unsigned int)max(s->r.top, 0),
+ node->q_data.height - MIN_DIM);
+ s->r.width = max(min(s->r.width,
+ node->q_data.width - s->r.left), MIN_DIM);
+ s->r.height = max(min(s->r.height,
+ node->q_data.height - s->r.top), MIN_DIM);
+ break;
+ case V4L2_SEL_TGT_CROP_DEFAULT:
+ /* Default (i.e. no) crop window. */
+ s->r.left = 0;
+ s->r.top = 0;
+ s->r.width = node->q_data.width;
+ s->r.height = node->q_data.height;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ crop.rect.x = s->r.left;
+ crop.rect.y = s->r.top;
+ crop.rect.width = s->r.width;
+ crop.rect.height = s->r.height;
+
+ return vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_CROP,
+ &crop, sizeof(crop));
+}
+
+static int bcm2835_isp_node_g_selection(struct file *file, void *fh,
+ struct v4l2_selection *s)
+{
+ struct mmal_parameter_crop crop;
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ u32 crop_size = sizeof(crop);
+ int ret;
+
+ /* We can only return out an input crop. */
+ switch (s->target) {
+ case V4L2_SEL_TGT_CROP:
+ ret = vchiq_mmal_port_parameter_get(dev->mmal_instance,
+ node->port,
+ MMAL_PARAMETER_CROP,
+ &crop, &crop_size);
+ if (!ret) {
+ s->r.left = crop.rect.x;
+ s->r.top = crop.rect.y;
+ s->r.width = crop.rect.width;
+ s->r.height = crop.rect.height;
+ }
+ break;
+ case V4L2_SEL_TGT_CROP_DEFAULT:
+ case V4L2_SEL_TGT_CROP_BOUNDS:
+ /* Default (i.e. no) crop window. */
+ s->r.left = 0;
+ s->r.top = 0;
+ s->r.width = node->q_data.width;
+ s->r.height = node->q_data.height;
+ ret = 0;
+ break;
+ default:
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
+static int bcm3285_isp_subscribe_event(struct v4l2_fh *fh,
+ const struct v4l2_event_subscription *s)
+{
+ switch (s->type) {
+ /* Cannot change source parameters dynamically at runtime. */
+ case V4L2_EVENT_SOURCE_CHANGE:
+ return -EINVAL;
+ case V4L2_EVENT_CTRL:
+ return v4l2_ctrl_subscribe_event(fh, s);
+ default:
+ return v4l2_event_subscribe(fh, s, 4, NULL);
+ }
+}
+
+static const struct v4l2_ioctl_ops bcm2835_isp_node_ioctl_ops = {
+ .vidioc_querycap = bcm2835_isp_node_querycap,
+ .vidioc_g_fmt_vid_cap = bcm2835_isp_node_g_fmt,
+ .vidioc_g_fmt_vid_out = bcm2835_isp_node_g_fmt,
+ .vidioc_g_fmt_meta_cap = bcm2835_isp_node_g_fmt,
+ .vidioc_s_fmt_vid_cap = bcm2835_isp_node_s_fmt,
+ .vidioc_s_fmt_vid_out = bcm2835_isp_node_s_fmt,
+ .vidioc_s_fmt_meta_cap = bcm2835_isp_node_s_fmt,
+ .vidioc_try_fmt_vid_cap = bcm2835_isp_node_try_fmt,
+ .vidioc_try_fmt_vid_out = bcm2835_isp_node_try_fmt,
+ .vidioc_try_fmt_meta_cap = bcm2835_isp_node_try_fmt,
+ .vidioc_s_selection = bcm2835_isp_node_s_selection,
+ .vidioc_g_selection = bcm2835_isp_node_g_selection,
+
+ .vidioc_enum_fmt_vid_cap = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_fmt_vid_out = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_fmt_meta_cap = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_framesizes = bcm2835_isp_enum_framesizes,
+
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+
+ .vidioc_subscribe_event = bcm3285_isp_subscribe_event,
+ .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+/*
+ * Size of the array to provide to the VPU when asking for the list of supported
+ * formats.
+ *
+ * The ISP component currently advertises 62 input formats, so add a small
+ * overhead on that. Should the component advertise more formats then the excess
+ * will be dropped and a warning logged.
+ */
+#define MAX_SUPPORTED_ENCODINGS 70
+
+/* Populate node->supported_fmts with the formats supported by those ports. */
+static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct bcm2835_isp_fmt const **list;
+ unsigned int i, j, num_encodings;
+ u32 fourccs[MAX_SUPPORTED_ENCODINGS];
+ u32 param_size = sizeof(fourccs);
+ int ret;
+
+ ret = vchiq_mmal_port_parameter_get(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_SUPPORTED_ENCODINGS,
+ &fourccs, ¶m_size);
+
+ if (ret) {
+ if (ret == MMAL_MSG_STATUS_ENOSPC) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: port has more encodings than we provided space for. Some are dropped (%zu vs %u).\n",
+ __func__, param_size / sizeof(u32),
+ MAX_SUPPORTED_ENCODINGS);
+ num_encodings = MAX_SUPPORTED_ENCODINGS;
+ } else {
+ v4l2_err(&dev->v4l2_dev, "%s: get_param ret %u.\n",
+ __func__, ret);
+ return -EINVAL;
+ }
+ } else {
+ num_encodings = param_size / sizeof(u32);
+ }
+
+ /*
+ * Assume at this stage that all encodings will be supported in V4L2.
+ * Any that aren't supported will waste a very small amount of memory.
+ */
+ list = devm_kzalloc(dev->dev,
+ sizeof(struct bcm2835_isp_fmt *) * num_encodings,
+ GFP_KERNEL);
+ if (!list)
+ return -ENOMEM;
+ node->supported_fmts = list;
+
+ for (i = 0, j = 0; i < num_encodings; i++) {
+ const struct bcm2835_isp_fmt *fmt = get_fmt(fourccs[i]);
+
+ if (fmt) {
+ list[j] = fmt;
+ j++;
+ }
+ }
+ node->num_supported_fmts = j;
+
+ return 0;
+}
+
+/*
+ * Register a device node /dev/video<N> to go along with one of the ISP's input
+ * or output nodes.
+ */
+static int register_node(struct bcm2835_isp_dev *dev,
+ struct bcm2835_isp_node *node,
+ int index)
+{
+ struct video_device *vfd;
+ struct vb2_queue *queue;
+ int ret;
+
+ mutex_init(&node->lock);
+ mutex_init(&node->queue_lock);
+
+ node->dev = dev;
+ vfd = &node->vfd;
+ queue = &node->queue;
+ queue->type = index_to_queue_type(index);
+ /*
+ * Setup the node type-specific params.
+ *
+ * Only the OUTPUT node can set controls and crop windows. However,
+ * we must allow the s/g_selection ioctl on the stats node as v4l2
+ * compliance expects it to return a -ENOTTY, and the framework
+ * does not handle it if the ioctl is disabled.
+ */
+ switch (queue->type) {
+ case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+ vfd->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+ node->id = index;
+ node->vfl_dir = VFL_DIR_TX;
+ node->name = "output";
+ node->port = &dev->component->input[node->id];
+ break;
+ case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+ vfd->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+ /* First Capture node starts at id 0, etc. */
+ node->id = index - BCM2835_ISP_NUM_OUTPUTS;
+ node->vfl_dir = VFL_DIR_RX;
+ node->name = "capture";
+ node->port = &dev->component->output[node->id];
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_CTRL);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_SELECTION);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_G_SELECTION);
+ break;
+ case V4L2_BUF_TYPE_META_CAPTURE:
+ vfd->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
+ node->id = index - BCM2835_ISP_NUM_OUTPUTS;
+ node->vfl_dir = VFL_DIR_RX;
+ node->name = "stats";
+ node->port = &dev->component->output[node->id];
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_CTRL);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_SELECTION);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_G_SELECTION);
+ break;
+ }
+
+ /* We use the selection API instead of the old crop API. */
+ v4l2_disable_ioctl(vfd, VIDIOC_CROPCAP);
+ v4l2_disable_ioctl(vfd, VIDIOC_G_CROP);
+ v4l2_disable_ioctl(vfd, VIDIOC_S_CROP);
+
+ ret = bcm2835_isp_get_supported_fmts(node);
+ if (ret)
+ return ret;
+
+ /* Initialise the video node. */
+ vfd->vfl_type = VFL_TYPE_VIDEO;
+ vfd->fops = &bcm2835_isp_fops,
+ vfd->ioctl_ops = &bcm2835_isp_node_ioctl_ops,
+ vfd->minor = -1,
+ vfd->release = video_device_release_empty,
+ vfd->queue = &node->queue;
+ vfd->lock = &node->lock;
+ vfd->v4l2_dev = &dev->v4l2_dev;
+ vfd->vfl_dir = node->vfl_dir;
+
+ node->q_data.fmt = get_default_format(node);
+ node->q_data.width = DEFAULT_DIM;
+ node->q_data.height = DEFAULT_DIM;
+ node->q_data.bytesperline =
+ get_bytesperline(DEFAULT_DIM, node->q_data.fmt);
+ node->q_data.sizeimage = node_is_stats(node) ?
+ node->port->recommended_buffer.size :
+ get_sizeimage(node->q_data.bytesperline,
+ node->q_data.width,
+ node->q_data.height,
+ node->q_data.fmt);
+
+ queue->io_modes = VB2_MMAP | VB2_DMABUF;
+ queue->drv_priv = node;
+ queue->ops = &bcm2835_isp_node_queue_ops;
+ queue->mem_ops = &vb2_dma_contig_memops;
+ queue->buf_struct_size = sizeof(struct bcm2835_isp_buffer);
+ queue->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+ queue->dev = dev->dev;
+ queue->lock = &node->queue_lock;
+
+ ret = vb2_queue_init(queue);
+ if (ret < 0) {
+ v4l2_info(&dev->v4l2_dev, "vb2_queue_init failed\n");
+ return ret;
+ }
+
+ /* Set some controls and defaults, but only on the VIDEO_OUTPUT node. */
+ if (node_is_output(node)) {
+ unsigned int i;
+
+ /* Use this ctrl template to assign custom ISP ctrls. */
+ struct v4l2_ctrl_config ctrl_template = {
+ .ops = &bcm2835_isp_ctrl_ops,
+ .type = V4L2_CTRL_TYPE_U8,
+ .def = 0,
+ .min = 0x00,
+ .max = 0xff,
+ .step = 1,
+ };
+
+ /* 3 standard controls, and an array of custom controls */
+ ret = v4l2_ctrl_handler_init(&dev->ctrl_handler,
+ 3 + ARRAY_SIZE(custom_ctrls));
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "ctrl_handler init failed (%d)\n",
+ ret);
+ goto queue_cleanup;
+ }
+
+ dev->r_gain = 1000;
+ dev->b_gain = 1000;
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_RED_BALANCE, 1, 0xffff, 1,
+ dev->r_gain);
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_BLUE_BALANCE, 1, 0xffff, 1,
+ dev->b_gain);
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_DIGITAL_GAIN, 1, 0xffff, 1, 1000);
+
+ for (i = 0; i < ARRAY_SIZE(custom_ctrls); i++) {
+ ctrl_template.name = custom_ctrls[i].name;
+ ctrl_template.id = custom_ctrls[i].id;
+ ctrl_template.dims[0] = custom_ctrls[i].size;
+ ctrl_template.flags = custom_ctrls[i].flags;
+ v4l2_ctrl_new_custom(&dev->ctrl_handler,
+ &ctrl_template, NULL);
+ }
+
+ node->vfd.ctrl_handler = &dev->ctrl_handler;
+ if (dev->ctrl_handler.error) {
+ ret = dev->ctrl_handler.error;
+ v4l2_err(&dev->v4l2_dev, "controls init failed (%d)\n",
+ ret);
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+ goto ctrl_cleanup;
+ }
+ }
+
+ /* Define the device names */
+ snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
+ node->name, node->id);
+
+ ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "Failed to register video %s[%d] device node\n",
+ node->name, node->id);
+ goto ctrl_cleanup;
+ }
+
+ node->registered = true;
+ video_set_drvdata(vfd, node);
+
+ v4l2_info(&dev->v4l2_dev,
+ "Device node %s[%d] registered as /dev/video%d\n",
+ node->name, node->id, vfd->num);
+
+ return 0;
+
+ctrl_cleanup:
+ if (node_is_output(node))
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+queue_cleanup:
+ vb2_queue_release(&node->queue);
+ return ret;
+}
+
+/* Unregister one of the /dev/video<N> nodes associated with the ISP. */
+static void bcm2835_isp_unregister_node(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ v4l2_info(&dev->v4l2_dev,
+ "Unregistering node %s[%d] device node /dev/video%d\n",
+ node->name, node->id, node->vfd.num);
+
+ if (node->registered) {
+ video_unregister_device(&node->vfd);
+ if (node_is_output(node))
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+ vb2_queue_release(&node->queue);
+ }
+
+ /*
+ * node->supported_fmts.list is free'd automatically
+ * as a managed resource.
+ */
+ node->supported_fmts = NULL;
+ node->num_supported_fmts = 0;
+ node->vfd.ctrl_handler = NULL;
+ node->registered = false;
+}
+
+static void media_controller_unregister(struct bcm2835_isp_dev *dev)
+{
+ unsigned int i;
+
+ v4l2_info(&dev->v4l2_dev, "Unregister from media controller\n");
+
+ if (dev->media_device_registered) {
+ media_device_unregister(&dev->mdev);
+ media_device_cleanup(&dev->mdev);
+ dev->media_device_registered = false;
+ }
+
+ kfree(dev->entity.name);
+ dev->entity.name = NULL;
+
+ if (dev->media_entity_registered) {
+ media_device_unregister_entity(&dev->entity);
+ dev->media_entity_registered = false;
+ }
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ struct bcm2835_isp_node *node = &dev->node[i];
+
+ if (node->media_node_registered) {
+ media_remove_intf_links(node->intf_link->intf);
+ media_entity_remove_links(&dev->node[i].vfd.entity);
+ media_devnode_remove(node->intf_devnode);
+ media_device_unregister_entity(&node->vfd.entity);
+ kfree(node->vfd.entity.name);
+ }
+ node->media_node_registered = false;
+ }
+
+ dev->v4l2_dev.mdev = NULL;
+}
+
+static int media_controller_register_node(struct bcm2835_isp_dev *dev, int num)
+{
+ struct bcm2835_isp_node *node = &dev->node[num];
+ struct media_entity *entity = &node->vfd.entity;
+ int output = node_is_output(node);
+ char *name;
+ int ret;
+
+ v4l2_info(&dev->v4l2_dev,
+ "Register %s node %d with media controller\n",
+ output ? "output" : "capture", num);
+ entity->obj_type = MEDIA_ENTITY_TYPE_VIDEO_DEVICE;
+ entity->function = MEDIA_ENT_F_IO_V4L;
+ entity->info.dev.major = VIDEO_MAJOR;
+ entity->info.dev.minor = node->vfd.minor;
+ name = kmalloc(BCM2835_ISP_ENTITY_NAME_LEN, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto error_no_mem;
+ }
+ snprintf(name, BCM2835_ISP_ENTITY_NAME_LEN, "%s0-%s%d",
+ BCM2835_ISP_NAME, output ? "output" : "capture", num);
+ entity->name = name;
+ node->pad.flags = output ? MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
+ ret = media_entity_pads_init(entity, 1, &node->pad);
+ if (ret)
+ goto error_pads_init;
+ ret = media_device_register_entity(&dev->mdev, entity);
+ if (ret)
+ goto error_register_entity;
+
+ node->intf_devnode = media_devnode_create(&dev->mdev,
+ MEDIA_INTF_T_V4L_VIDEO, 0,
+ VIDEO_MAJOR, node->vfd.minor);
+ if (!node->intf_devnode) {
+ ret = -ENOMEM;
+ goto error_devnode_create;
+ }
+
+ node->intf_link = media_create_intf_link(entity,
+ &node->intf_devnode->intf,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ if (!node->intf_link) {
+ ret = -ENOMEM;
+ goto error_create_intf_link;
+ }
+
+ if (output)
+ ret = media_create_pad_link(entity, 0, &dev->entity, num,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ else
+ ret = media_create_pad_link(&dev->entity, num, entity, 0,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ if (ret)
+ goto error_create_pad_link;
+
+ dev->node[num].media_node_registered = true;
+ return 0;
+
+error_create_pad_link:
+ media_remove_intf_links(&node->intf_devnode->intf);
+error_create_intf_link:
+ media_devnode_remove(node->intf_devnode);
+error_devnode_create:
+ media_device_unregister_entity(&node->vfd.entity);
+error_register_entity:
+error_pads_init:
+ kfree(entity->name);
+ entity->name = NULL;
+error_no_mem:
+ if (ret)
+ v4l2_info(&dev->v4l2_dev, "Error registering node\n");
+
+ return ret;
+}
+
+static int media_controller_register(struct bcm2835_isp_dev *dev)
+{
+ char *name;
+ unsigned int i;
+ int ret;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "Registering with media controller\n");
+ dev->mdev.dev = dev->dev;
+ strscpy(dev->mdev.model, "bcm2835-isp",
+ sizeof(dev->mdev.model));
+ strscpy(dev->mdev.bus_info, "platform:bcm2835-isp",
+ sizeof(dev->mdev.bus_info));
+ media_device_init(&dev->mdev);
+ dev->v4l2_dev.mdev = &dev->mdev;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "Register entity for nodes\n");
+
+ name = kmalloc(BCM2835_ISP_ENTITY_NAME_LEN, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto done;
+ }
+ snprintf(name, BCM2835_ISP_ENTITY_NAME_LEN, "bcm2835_isp0");
+ dev->entity.name = name;
+ dev->entity.obj_type = MEDIA_ENTITY_TYPE_BASE;
+ dev->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ dev->pad[i].flags = node_is_output(&dev->node[i]) ?
+ MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
+ }
+
+ ret = media_entity_pads_init(&dev->entity, BCM2835_ISP_NUM_NODES,
+ dev->pad);
+ if (ret)
+ goto done;
+
+ ret = media_device_register_entity(&dev->mdev, &dev->entity);
+ if (ret)
+ goto done;
+
+ dev->media_entity_registered = true;
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ ret = media_controller_register_node(dev, i);
+ if (ret)
+ goto done;
+ }
+
+ ret = media_device_register(&dev->mdev);
+ if (!ret)
+ dev->media_device_registered = true;
+done:
+ return ret;
+}
+
+static int bcm2835_isp_remove(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
+ unsigned int i;
+
+ media_controller_unregister(dev);
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++)
+ bcm2835_isp_unregister_node(&dev->node[i]);
+
+ v4l2_device_unregister(&dev->v4l2_dev);
+
+ if (dev->component)
+ vchiq_mmal_component_finalise(dev->mmal_instance,
+ dev->component);
+
+ vchiq_mmal_finalise(dev->mmal_instance);
+
+ return 0;
+}
+
+static int bcm2835_isp_probe(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev *dev;
+ unsigned int i;
+ int ret;
+
+ dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->dev = &pdev->dev;
+
+ ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
+ if (ret)
+ return ret;
+
+ ret = vchiq_mmal_init(&dev->mmal_instance);
+ if (ret) {
+ v4l2_device_unregister(&dev->v4l2_dev);
+ return ret;
+ }
+
+ ret = vchiq_mmal_component_init(dev->mmal_instance, "ril.isp",
+ &dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: failed to create ril.isp component\n", __func__);
+ goto error;
+ }
+
+ if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
+ dev->component->outputs < BCM2835_ISP_NUM_CAPTURES +
+ BCM2835_ISP_NUM_METADATA) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: ril.isp returned %d i/p (%d expected), %d o/p (%d expected) ports\n",
+ __func__, dev->component->inputs,
+ BCM2835_ISP_NUM_OUTPUTS,
+ dev->component->outputs,
+ BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
+ goto error;
+ }
+
+ atomic_set(&dev->num_streaming, 0);
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ struct bcm2835_isp_node *node = &dev->node[i];
+
+ ret = register_node(dev, node, i);
+ if (ret)
+ goto error;
+ }
+
+ ret = media_controller_register(dev);
+ if (ret)
+ goto error;
+
+ platform_set_drvdata(pdev, dev);
+ v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
+ return 0;
+
+error:
+ bcm2835_isp_remove(pdev);
+
+ return ret;
+}
+
+static struct platform_driver bcm2835_isp_pdrv = {
+ .probe = bcm2835_isp_probe,
+ .remove = bcm2835_isp_remove,
+ .driver = {
+ .name = BCM2835_ISP_NAME,
+ },
+};
+
+module_platform_driver(bcm2835_isp_pdrv);
+
+MODULE_DESCRIPTION("BCM2835 ISP driver");
+MODULE_AUTHOR("Naushir Patuck <naush@raspberrypi.com>");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
+MODULE_ALIAS("platform:bcm2835-isp");
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
index 4711877a9711..538c347db433 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
@@ -135,6 +135,10 @@
*/
#define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I')
+/** ISP image statistics format
+ */
+#define MMAL_ENCODING_BRCM_STATS MMAL_FOURCC('S', 'T', 'A', 'T')
+
/* }@ */
/** \name Pre-defined audio encodings */
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
index a0cdd28101f2..5e6276aca143 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
@@ -223,6 +223,63 @@ enum mmal_parameter_camera_type {
MMAL_PARAMETER_SHUTTER_SPEED,
/**< Takes a @ref MMAL_PARAMETER_AWB_GAINS_T */
MMAL_PARAMETER_CUSTOM_AWB_GAINS,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_SETTINGS_T */
+ MMAL_PARAMETER_CAMERA_SETTINGS,
+ /**< Takes a @ref MMAL_PARAMETER_PRIVACY_INDICATOR_T */
+ MMAL_PARAMETER_PRIVACY_INDICATOR,
+ /**< Takes a @ref MMAL_PARAMETER_BOOLEAN_T */
+ MMAL_PARAMETER_VIDEO_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_BOOLEAN_T */
+ MMAL_PARAMETER_STILLS_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_ANNOTATE_T */
+ MMAL_PARAMETER_ANNOTATE,
+ /**< Takes a @ref MMAL_PARAMETER_STEREOSCOPIC_MODE_T */
+ MMAL_PARAMETER_STEREOSCOPIC_MODE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_INTERFACE_T */
+ MMAL_PARAMETER_CAMERA_INTERFACE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_CLOCKING_MODE_T */
+ MMAL_PARAMETER_CAMERA_CLOCKING_MODE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_CONFIG_T */
+ MMAL_PARAMETER_CAMERA_RX_CONFIG,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_TIMING_T */
+ MMAL_PARAMETER_CAMERA_RX_TIMING,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_DPF_CONFIG,
+ /* 0x50 */
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_JPEG_RESTART_INTERVAL,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE,
+ /**< Takes a @ref MMAL_PARAMETER_LENS_SHADING_T */
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_BLACK_LEVEL,
+ /**< Takes a @ref MMAL_PARAMETER_RESIZE_T */
+ MMAL_PARAMETER_RESIZE_PARAMS,
+ /**< Takes a @ref MMAL_PARAMETER_CROP_T */
+ MMAL_PARAMETER_CROP,
+ /**< Takes a @ref MMAL_PARAMETER_INT32_T */
+ MMAL_PARAMETER_OUTPUT_SHIFT,
+ /**< Takes a @ref MMAL_PARAMETER_INT32_T */
+ MMAL_PARAMETER_CCM_SHIFT,
+ /**< Takes a @ref MMAL_PARAMETER_CUSTOM_CCM_T */
+ MMAL_PARAMETER_CUSTOM_CCM,
+ /**< Takes a @ref MMAL_PARAMETER_RATIONAL_T */
+ MMAL_PARAMETER_ANALOG_GAIN,
+ /**< Takes a @ref MMAL_PARAMETER_RATIONAL_T */
+ MMAL_PARAMETER_DIGITAL_GAIN,
+ /**< Takes a @ref MMAL_PARAMETER_DENOISE_T */
+ MMAL_PARAMETER_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_SHARPEN_T */
+ MMAL_PARAMETER_SHARPEN,
+ /**< Takes a @ref MMAL_PARAMETER_GEQ_T */
+ MMAL_PARAMETER_GEQ,
+ /**< Tales a @ref MMAP_PARAMETER_DPC_T */
+ MMAL_PARAMETER_DPC,
+ /**< Tales a @ref MMAP_PARAMETER_GAMMA_T */
+ MMAL_PARAMETER_GAMMA,
+ /**< Takes a @ref MMAL_PARAMETER_CDN_T */
+ MMAL_PARAMETER_CDN,
};
enum mmal_parameter_camera_config_timestamp_mode {
@@ -746,7 +803,113 @@ struct mmal_parameter_camera_info {
struct mmal_parameter_camera_info_camera
cameras[MMAL_PARAMETER_CAMERA_INFO_MAX_CAMERAS];
struct mmal_parameter_camera_info_flash
- flashes[MMAL_PARAMETER_CAMERA_INFO_MAX_FLASHES];
+ flashes[MMAL_PARAMETER_CAMERA_INFO_MAX_FLASHES];
+};
+
+struct mmal_parameter_ccm {
+ struct s32_fract cm[3][3];
+ s32 offsets[3];
+};
+
+struct mmal_parameter_custom_ccm {
+ u32 enabled; /**< Enable the custom CCM. */
+ struct mmal_parameter_ccm ccm; /**< CCM to be used. */
+};
+
+struct mmal_parameter_lens_shading {
+ u32 enabled;
+ u32 grid_cell_size;
+ u32 grid_width;
+ u32 grid_stride;
+ u32 grid_height;
+ u32 mem_handle_table;
+ u32 ref_transform;
+};
+
+enum mmal_parameter_ls_gain_format_type {
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U0P8_1 = 0,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U1P7_0 = 1,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U1P7_1 = 2,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U2P6_0 = 3,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U2P6_1 = 4,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U3P5_0 = 5,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U3P5_1 = 6,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U4P10 = 7,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_DUMMY = 0x7FFFFFFF
+};
+
+struct mmal_parameter_lens_shading_v2 {
+ u32 enabled;
+ u32 grid_cell_size;
+ u32 grid_width;
+ u32 grid_stride;
+ u32 grid_height;
+ u32 mem_handle_table;
+ u32 ref_transform;
+ u32 corner_sampled;
+ enum mmal_parameter_ls_gain_format_type gain_format;
+};
+
+struct mmal_parameter_black_level {
+ u32 enabled;
+ u16 black_level_r;
+ u16 black_level_g;
+ u16 black_level_b;
+ u8 pad_[2]; /* Unused */
+};
+
+struct mmal_parameter_geq {
+ u32 enabled;
+ u32 offset;
+ struct s32_fract slope;
+};
+
+#define MMAL_NUM_GAMMA_PTS 33
+struct mmal_parameter_gamma {
+ u32 enabled;
+ u16 x[MMAL_NUM_GAMMA_PTS];
+ u16 y[MMAL_NUM_GAMMA_PTS];
+};
+
+enum mmal_parameter_cdn_mode {
+ MMAL_PARAM_CDN_FAST = 0,
+ MMAL_PARAM_CDN_HIGH_QUALITY = 1,
+ MMAL_PARAM_CDN_DUMMY = 0x7FFFFFFF
+};
+
+struct mmal_parameter_colour_denoise {
+ u32 enabled;
+ enum mmal_parameter_cdn_mode mode;
+};
+
+struct mmal_parameter_denoise {
+ u32 enabled;
+ u32 constant;
+ struct s32_fract slope;
+ struct s32_fract strength;
+};
+
+struct mmal_parameter_sharpen {
+ u32 enabled;
+ struct s32_fract threshold;
+ struct s32_fract strength;
+ struct s32_fract limit;
+};
+
+enum mmal_dpc_mode {
+ MMAL_DPC_MODE_OFF = 0,
+ MMAL_DPC_MODE_NORMAL = 1,
+ MMAL_DPC_MODE_STRONG = 2,
+ MMAL_DPC_MODE_MAX = 0x7FFFFFFF,
+};
+
+struct mmal_parameter_dpc {
+ u32 enabled;
+ u32 strength;
+};
+
+struct mmal_parameter_crop {
+ struct vchiq_mmal_rect rect;
};
#endif
diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
index cf8c0437f159..c50e3ca81565 100644
--- a/include/uapi/linux/bcm2835-isp.h
+++ b/include/uapi/linux/bcm2835-isp.h
@@ -31,6 +31,8 @@
(V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
#define V4L2_CID_USER_BCM2835_ISP_DPC \
(V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
+#define V4L2_CID_USER_BCM2835_ISP_CDN \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0009)
/*
* All structs below are directly mapped onto the equivalent structs in
@@ -175,6 +177,31 @@ struct bcm2835_isp_gamma {
__u16 y[BCM2835_NUM_GAMMA_PTS];
};
+/**
+ * enum bcm2835_isp_cdn_mode - Mode of operation for colour denoise.
+ *
+ * @CDN_MODE_FAST: Fast (but lower quality) colour denoise
+ * algorithm, typically used for video recording.
+ * @CDN_HIGH_QUALITY: High quality (but slower) colour denoise
+ * algorithm, typically used for stills capture.
+ */
+enum bcm2835_isp_cdn_mode {
+ CDN_MODE_FAST = 0,
+ CDN_MODE_HIGH_QUALITY = 1,
+};
+
+/**
+ * struct bcm2835_isp_cdn - Colour denoise parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_CDN ctrl.
+ *
+ * @enabled: Enable colour denoise.
+ * @mode: Colour denoise operating mode (see enum &bcm2835_isp_cdn_mode)
+ */
+struct bcm2835_isp_cdn {
+ __u32 enabled;
+ __u32 mode;
+};
+
/**
* struct bcm2835_isp_denoise - Denoise parameters set with the
* V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 09/14] staging: vc04_services: bcm2835-isp: Add a more complex ISP processing component
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
Driver for the BCM2835 ISP hardware block. This driver uses the MMAL
component to program the ISP hardware through the VC firmware.
The ISP component can produce two video stream outputs, and Bayer
image statistics. This can't be encompassed in a simple V4L2
M2M device, so create a new device that registers 4 video nodes.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
MAINTAINERS | 9 +
drivers/staging/vc04_services/Kconfig | 2 +
drivers/staging/vc04_services/Makefile | 1 +
.../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
.../vc04_services/bcm2835-isp/Makefile | 8 +
.../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
.../bcm2835-isp/bcm2835-isp-fmts.h | 481 +++++
.../bcm2835-isp/bcm2835-v4l2-isp.c | 1708 +++++++++++++++++
.../vc04_services/vchiq-mmal/mmal-encodings.h | 4 +
.../vchiq-mmal/mmal-parameters.h | 165 +-
include/uapi/linux/bcm2835-isp.h | 27 +
11 files changed, 2490 insertions(+), 1 deletion(-)
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 6466dd5ef1fa..14e162ca0b58 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4037,6 +4037,15 @@ S: Maintained
F: drivers/media/platform/bcm2835/
F: Documentation/devicetree/bindings/media/brcm,bcm2835-unicam.yaml
+BROADCOM BCM2835 ISP DRIVER
+M: Raspberry Pi Kernel Maintenance <kernel-list@raspberrypi.com>
+L: linux-media@vger.kernel.org
+S: Maintained
+F: Documentation/media/uapi/v4l/pixfmt-meta-bcm2835-isp-stats.rst
+F: Documentation/media/v4l-drivers/bcm2835-isp.rst
+F: drivers/staging/vc04_services/bcm2835-isp
+F: include/uapi/linux/bcm2835-isp.h
+
BROADCOM BCM47XX MIPS ARCHITECTURE
M: Hauke Mehrtens <hauke@hauke-m.de>
M: Rafał Miłecki <zajec5@gmail.com>
diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig
index 6c0e77d64376..e71e5a8b820a 100644
--- a/drivers/staging/vc04_services/Kconfig
+++ b/drivers/staging/vc04_services/Kconfig
@@ -44,6 +44,8 @@ source "drivers/staging/vc04_services/bcm2835-audio/Kconfig"
source "drivers/staging/vc04_services/bcm2835-camera/Kconfig"
+source "drivers/staging/vc04_services/bcm2835-isp/Kconfig"
+
source "drivers/staging/vc04_services/vchiq-mmal/Kconfig"
source "drivers/staging/vc04_services/vc-sm-cma/Kconfig"
diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile
index 01089f369fb4..055cf28fc19c 100644
--- a/drivers/staging/vc04_services/Makefile
+++ b/drivers/staging/vc04_services/Makefile
@@ -15,6 +15,7 @@ obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/
obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/
obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/
obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/
+obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp/
ccflags-y += -I $(srctree)/$(src)/include
diff --git a/drivers/staging/vc04_services/bcm2835-isp/Kconfig b/drivers/staging/vc04_services/bcm2835-isp/Kconfig
new file mode 100644
index 000000000000..6222799ebe16
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/Kconfig
@@ -0,0 +1,14 @@
+config VIDEO_ISP_BCM2835
+ tristate "BCM2835 ISP support"
+ depends on MEDIA_SUPPORT
+ depends on VIDEO_DEV && (ARCH_BCM2835 || COMPILE_TEST)
+ depends on MEDIA_CONTROLLER
+ select BCM2835_VCHIQ_MMAL
+ select VIDEOBUF2_DMA_CONTIG
+ help
+ This is the V4L2 driver for the Broadcom BCM2835 ISP hardware.
+ This operates over the VCHIQ interface to a service running on
+ VideoCore.
+
+ To compile this driver as a module, choose M here: the module
+ will be called bcm2835-isp.
diff --git a/drivers/staging/vc04_services/bcm2835-isp/Makefile b/drivers/staging/vc04_services/bcm2835-isp/Makefile
new file mode 100644
index 000000000000..42d3081d342f
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+bcm2835-isp-objs := bcm2835-v4l2-isp.o
+
+obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp.o
+
+ccflags-y += \
+ -I$(srctree)/drivers/staging/vc04_services \
+ -D__VCCOREVER__=0x04000000
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
new file mode 100644
index 000000000000..172605718cdf
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck (naush@raspberrypi.com)
+ *
+ */
+
+#ifndef BCM2835_ISP_CTRLS
+#define BCM2835_ISP_CTRLS
+
+#include <linux/bcm2835-isp.h>
+
+struct bcm2835_isp_custom_ctrl {
+ const char *name;
+ u32 id;
+ u32 size;
+ u32 flags;
+};
+
+static const struct bcm2835_isp_custom_ctrl custom_ctrls[] = {
+ {
+ .name = "Colour Correction Matrix",
+ .id = V4L2_CID_USER_BCM2835_ISP_CC_MATRIX,
+ .size = sizeof(struct bcm2835_isp_custom_ccm),
+ .flags = 0
+ }, {
+ .name = "Lens Shading",
+ .id = V4L2_CID_USER_BCM2835_ISP_LENS_SHADING,
+ .size = sizeof(struct bcm2835_isp_lens_shading),
+ .flags = V4L2_CTRL_FLAG_EXECUTE_ON_WRITE
+ }, {
+ .name = "Black Level",
+ .id = V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL,
+ .size = sizeof(struct bcm2835_isp_black_level),
+ .flags = 0
+ }, {
+ .name = "Green Equalisation",
+ .id = V4L2_CID_USER_BCM2835_ISP_GEQ,
+ .size = sizeof(struct bcm2835_isp_geq),
+ .flags = 0
+ }, {
+ .name = "Gamma",
+ .id = V4L2_CID_USER_BCM2835_ISP_GAMMA,
+ .size = sizeof(struct bcm2835_isp_gamma),
+ .flags = 0
+ }, {
+ .name = "Sharpen",
+ .id = V4L2_CID_USER_BCM2835_ISP_SHARPEN,
+ .size = sizeof(struct bcm2835_isp_sharpen),
+ .flags = 0
+ }, {
+ .name = "Denoise",
+ .id = V4L2_CID_USER_BCM2835_ISP_DENOISE,
+ .size = sizeof(struct bcm2835_isp_denoise),
+ .flags = 0
+ }, {
+ .name = "Colour Denoise",
+ .id = V4L2_CID_USER_BCM2835_ISP_CDN,
+ .size = sizeof(struct bcm2835_isp_cdn),
+ .flags = 0
+ }, {
+ .name = "Defective Pixel Correction",
+ .id = V4L2_CID_USER_BCM2835_ISP_DPC,
+ .size = sizeof(struct bcm2835_isp_dpc),
+ .flags = 0
+ }
+};
+
+#endif
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
new file mode 100644
index 000000000000..65913a4ff203
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
@@ -0,0 +1,481 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck (naush@raspberrypi.com)
+ *
+ */
+
+#ifndef BCM2835_ISP_FMTS
+#define BCM2835_ISP_FMTS
+
+#include <linux/videodev2.h>
+#include "vchiq-mmal/mmal-encodings.h"
+
+struct bcm2835_isp_fmt {
+ u32 fourcc;
+ int depth;
+ int bytesperline_align;
+ u32 mmal_fmt;
+ int size_multiplier_x2;
+ enum v4l2_colorspace colorspace;
+ unsigned int step_size;
+};
+
+static const struct bcm2835_isp_fmt supported_formats[] = {
+ {
+ /* YUV formats */
+ .fourcc = V4L2_PIX_FMT_YUV420,
+ .depth = 8,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_I420,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YVU420,
+ .depth = 8,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YV12,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_NV12,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_NV12,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_NV21,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_NV21,
+ .size_multiplier_x2 = 3,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YUYV,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YUYV,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_UYVY,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_UYVY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_YVYU,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_YVYU,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_VYUY,
+ .depth = 16,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_VYUY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .step_size = 2,
+ }, {
+ /* RGB formats */
+ .fourcc = V4L2_PIX_FMT_RGB24,
+ .depth = 24,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_RGB24,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGB565,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_RGB16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_BGR24,
+ .depth = 24,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BGR24,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_XBGR32,
+ .depth = 32,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_BGRA,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ .fourcc = V4L2_PIX_FMT_RGBX32,
+ .depth = 32,
+ .bytesperline_align = 64,
+ .mmal_fmt = MMAL_ENCODING_RGBA,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_SRGB,
+ .step_size = 1,
+ }, {
+ /* Bayer formats */
+ /* 8 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG8,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG8,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 16 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* Bayer formats unpacked to 16bpp */
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_SRGGB14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SBGGR14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGRBG14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_PIX_FMT_SGBRG14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* Monochrome MIPI formats */
+ /* 8 bit */
+ .fourcc = V4L2_PIX_FMT_GREY,
+ .depth = 8,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_GREY,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit */
+ .fourcc = V4L2_PIX_FMT_Y10P,
+ .depth = 10,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y10P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit */
+ .fourcc = V4L2_PIX_FMT_Y12P,
+ .depth = 12,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y12P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit */
+ .fourcc = V4L2_PIX_FMT_Y14P,
+ .depth = 14,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y14P,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 16 bit */
+ .fourcc = V4L2_PIX_FMT_Y16,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y16,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 10 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y10,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y10,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 12 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y12,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y12,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ /* 14 bit as 16bpp */
+ .fourcc = V4L2_PIX_FMT_Y14,
+ .depth = 16,
+ .bytesperline_align = 32,
+ .mmal_fmt = MMAL_ENCODING_Y14,
+ .size_multiplier_x2 = 2,
+ .colorspace = V4L2_COLORSPACE_RAW,
+ .step_size = 2,
+ }, {
+ .fourcc = V4L2_META_FMT_BCM2835_ISP_STATS,
+ .depth = 8,
+ .mmal_fmt = MMAL_ENCODING_BRCM_STATS,
+ /* The rest are not valid fields for stats. */
+ }
+};
+
+#endif
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
new file mode 100644
index 000000000000..eb5f076fc6dc
--- /dev/null
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
@@ -0,0 +1,1708 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Broadcom BCM2835 ISP driver
+ *
+ * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
+ *
+ * Author: Naushir Patuck (naush@raspberrypi.com)
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/platform_device.h>
+
+#include <media/v4l2-ctrls.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-event.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-dma-contig.h>
+
+#include "vchiq-mmal/mmal-msg.h"
+#include "vchiq-mmal/mmal-parameters.h"
+#include "vchiq-mmal/mmal-vchiq.h"
+
+#include "vc-sm-cma/vc_sm_knl.h"
+
+#include "bcm2835-isp-ctrls.h"
+#include "bcm2835-isp-fmts.h"
+
+MODULE_IMPORT_NS(DMA_BUF);
+
+static unsigned int debug;
+module_param(debug, uint, 0644);
+MODULE_PARM_DESC(debug, "activates debug info");
+
+static unsigned int video_nr = 13;
+module_param(video_nr, uint, 0644);
+MODULE_PARM_DESC(video_nr, "base video device number");
+
+#define BCM2835_ISP_NAME "bcm2835-isp"
+#define BCM2835_ISP_ENTITY_NAME_LEN 32
+
+#define BCM2835_ISP_NUM_OUTPUTS 1
+#define BCM2835_ISP_NUM_CAPTURES 2
+#define BCM2835_ISP_NUM_METADATA 1
+
+#define BCM2835_ISP_NUM_NODES \
+ (BCM2835_ISP_NUM_OUTPUTS + BCM2835_ISP_NUM_CAPTURES + \
+ BCM2835_ISP_NUM_METADATA)
+
+/* Default frame dimension of 1280 pixels. */
+#define DEFAULT_DIM 1280U
+/*
+ * Maximum frame dimension of 16384 pixels. Even though the ISP runs in tiles,
+ * have a sensible limit so that we do not create an excessive number of tiles
+ * to process.
+ */
+#define MAX_DIM 16384U
+/*
+ * Minimum frame dimension of 64 pixels. Anything lower, and the tiling
+ * algorithm may not be able to cope when applying filter context.
+ */
+#define MIN_DIM 64U
+
+/* Timeout for stop_streaming to allow all buffers to return */
+#define COMPLETE_TIMEOUT (2 * HZ)
+
+/* Per-queue, driver-specific private data */
+struct bcm2835_isp_q_data {
+ /*
+ * These parameters should be treated as gospel, with everything else
+ * being determined from them.
+ */
+ unsigned int bytesperline;
+ unsigned int width;
+ unsigned int height;
+ unsigned int sizeimage;
+ const struct bcm2835_isp_fmt *fmt;
+};
+
+/*
+ * Structure to describe a single node /dev/video<N> which represents a single
+ * input or output queue to the ISP device.
+ */
+struct bcm2835_isp_node {
+ int vfl_dir;
+ unsigned int id;
+ const char *name;
+ struct vchiq_mmal_port *port;
+ struct video_device vfd;
+ struct media_pad pad;
+ struct media_intf_devnode *intf_devnode;
+ struct media_link *intf_link;
+ struct mutex lock; /* top level device node lock */
+ struct mutex queue_lock;
+
+ struct vb2_queue queue;
+ unsigned int sequence;
+
+ /* The list of formats supported on the node. */
+ struct bcm2835_isp_fmt const **supported_fmts;
+ unsigned int num_supported_fmts;
+
+ struct bcm2835_isp_q_data q_data;
+
+ /* Parent device structure */
+ struct bcm2835_isp_dev *dev;
+
+ bool registered;
+ bool media_node_registered;
+};
+
+/*
+ * Structure representing the entire ISP device, comprising several input and
+ * output nodes /dev/video<N>.
+ */
+struct bcm2835_isp_dev {
+ struct v4l2_device v4l2_dev;
+ struct device *dev;
+ struct v4l2_ctrl_handler ctrl_handler;
+ struct media_device mdev;
+ struct media_entity entity;
+ bool media_device_registered;
+ bool media_entity_registered;
+ struct vchiq_mmal_instance *mmal_instance;
+ struct vchiq_mmal_component *component;
+ struct completion frame_cmplt;
+
+ struct bcm2835_isp_node node[BCM2835_ISP_NUM_NODES];
+ struct media_pad pad[BCM2835_ISP_NUM_NODES];
+ atomic_t num_streaming;
+
+ /* Image pipeline controls. */
+ int r_gain;
+ int b_gain;
+};
+
+struct bcm2835_isp_buffer {
+ struct vb2_v4l2_buffer vb;
+ struct mmal_buffer mmal;
+};
+
+static
+inline struct bcm2835_isp_dev *node_get_dev(struct bcm2835_isp_node *node)
+{
+ return node->dev;
+}
+
+static inline bool node_is_output(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_VIDEO_OUTPUT;
+}
+
+static inline bool node_is_capture(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE;
+}
+
+static inline bool node_is_stats(struct bcm2835_isp_node *node)
+{
+ return node->queue.type == V4L2_BUF_TYPE_META_CAPTURE;
+}
+
+static inline enum v4l2_buf_type index_to_queue_type(int index)
+{
+ if (index < BCM2835_ISP_NUM_OUTPUTS)
+ return V4L2_BUF_TYPE_VIDEO_OUTPUT;
+ else if (index < BCM2835_ISP_NUM_OUTPUTS + BCM2835_ISP_NUM_CAPTURES)
+ return V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ else
+ return V4L2_BUF_TYPE_META_CAPTURE;
+}
+
+static int set_isp_param(struct bcm2835_isp_node *node, u32 parameter,
+ void *value, u32 value_size)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ return vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ parameter, value, value_size);
+}
+
+static int set_wb_gains(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct mmal_parameter_awbgains gains = {
+ .r_gain = { dev->r_gain, 1000 },
+ .b_gain = { dev->b_gain, 1000 }
+ };
+
+ return set_isp_param(node, MMAL_PARAMETER_CUSTOM_AWB_GAINS,
+ &gains, sizeof(gains));
+}
+
+static int set_digital_gain(struct bcm2835_isp_node *node, uint32_t gain)
+{
+ struct s32_fract digital_gain = {
+ .numerator = gain,
+ .denominator = 1000
+ };
+
+ return set_isp_param(node, MMAL_PARAMETER_DIGITAL_GAIN,
+ &digital_gain, sizeof(digital_gain));
+}
+
+static const struct bcm2835_isp_fmt *get_fmt(u32 mmal_fmt)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(supported_formats); i++) {
+ if (supported_formats[i].mmal_fmt == mmal_fmt)
+ return &supported_formats[i];
+ }
+ return NULL;
+}
+
+static const
+struct bcm2835_isp_fmt *find_format_by_fourcc(unsigned int fourcc,
+ struct bcm2835_isp_node *node)
+{
+ const struct bcm2835_isp_fmt *fmt;
+ unsigned int i;
+
+ for (i = 0; i < node->num_supported_fmts; i++) {
+ fmt = node->supported_fmts[i];
+ if (fmt->fourcc == fourcc)
+ return fmt;
+ }
+
+ return NULL;
+}
+
+static const
+struct bcm2835_isp_fmt *find_format(struct v4l2_format *f,
+ struct bcm2835_isp_node *node)
+{
+ return find_format_by_fourcc(node_is_stats(node) ?
+ f->fmt.meta.dataformat :
+ f->fmt.pix.pixelformat,
+ node);
+}
+
+/* vb2_to_mmal_buffer() - converts vb2 buffer header to MMAL
+ *
+ * Copies all the required fields from a VB2 buffer to the MMAL buffer header,
+ * ready for sending to the VPU.
+ */
+static void vb2_to_mmal_buffer(struct mmal_buffer *buf,
+ struct vb2_v4l2_buffer *vb2)
+{
+ u64 pts;
+
+ buf->mmal_flags = 0;
+ if (vb2->flags & V4L2_BUF_FLAG_KEYFRAME)
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_KEYFRAME;
+
+ /* Data must be framed correctly as one frame per buffer. */
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_FRAME_END;
+
+ buf->length = vb2->vb2_buf.planes[0].bytesused;
+ /*
+ * Minor ambiguity in the V4L2 spec as to whether passing in a 0 length
+ * buffer, or one with V4L2_BUF_FLAG_LAST set denotes end of stream.
+ * Handle either.
+ */
+ if (!buf->length || vb2->flags & V4L2_BUF_FLAG_LAST)
+ buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_EOS;
+
+ /* vb2 timestamps in nsecs, mmal in usecs */
+ pts = vb2->vb2_buf.timestamp;
+ do_div(pts, 1000);
+ buf->pts = pts;
+ buf->dts = MMAL_TIME_UNKNOWN;
+}
+
+static void mmal_buffer_cb(struct vchiq_mmal_instance *instance,
+ struct vchiq_mmal_port *port, int status,
+ struct mmal_buffer *mmal_buf)
+{
+ struct bcm2835_isp_buffer *q_buf;
+ struct bcm2835_isp_node *node = port->cb_ctx;
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2;
+
+ q_buf = container_of(mmal_buf, struct bcm2835_isp_buffer, mmal);
+ vb2 = &q_buf->vb;
+ v4l2_dbg(2, debug, &dev->v4l2_dev,
+ "%s: port:%s[%d], status:%d, buf:%p, dmabuf:%p, length:%lu, flags %u, pts %lld\n",
+ __func__, node_is_output(node) ? "input" : "output", node->id,
+ status, mmal_buf, mmal_buf->dma_buf, mmal_buf->length,
+ mmal_buf->mmal_flags, mmal_buf->pts);
+
+ if (mmal_buf->cmd)
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Unexpected event on output callback - %08x\n",
+ __func__, mmal_buf->cmd);
+
+ if (status) {
+ /* error in transfer */
+ if (vb2) {
+ /* there was a buffer with the error so return it */
+ vb2_buffer_done(&vb2->vb2_buf, VB2_BUF_STATE_ERROR);
+ }
+ return;
+ }
+
+ /* vb2 timestamps in nsecs, mmal in usecs */
+ vb2->vb2_buf.timestamp = mmal_buf->pts * 1000;
+ vb2->sequence = node->sequence++;
+ vb2_set_plane_payload(&vb2->vb2_buf, 0, mmal_buf->length);
+ vb2_buffer_done(&vb2->vb2_buf, VB2_BUF_STATE_DONE);
+
+ if (!port->enabled)
+ complete(&dev->frame_cmplt);
+}
+
+static void setup_mmal_port_format(struct bcm2835_isp_node *node,
+ struct vchiq_mmal_port *port)
+{
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+
+ port->format.encoding = q_data->fmt->mmal_fmt;
+ /* Raw image format - set width/height */
+ port->es.video.width = (q_data->bytesperline << 3) / q_data->fmt->depth;
+ port->es.video.height = q_data->height;
+ port->es.video.crop.width = q_data->width;
+ port->es.video.crop.height = q_data->height;
+ port->es.video.crop.x = 0;
+ port->es.video.crop.y = 0;
+};
+
+static int setup_mmal_port(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ unsigned int enable = 1;
+ int ret;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "%s: setup %s[%d]\n", __func__,
+ node->name, node->id);
+
+ vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_ZERO_COPY, &enable,
+ sizeof(enable));
+ setup_mmal_port_format(node, node->port);
+ ret = vchiq_mmal_port_set_format(dev->mmal_instance, node->port);
+ if (ret < 0) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: vchiq_mmal_port_set_format failed\n",
+ __func__);
+ return ret;
+ }
+
+ if (node->q_data.sizeimage < node->port->minimum_buffer.size) {
+ v4l2_err(&dev->v4l2_dev,
+ "buffer size mismatch sizeimage %u < min size %u\n",
+ node->q_data.sizeimage,
+ node->port->minimum_buffer.size);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_mmal_buf_cleanup(struct mmal_buffer *mmal_buf)
+{
+ mmal_vchi_buffer_cleanup(mmal_buf);
+
+ if (mmal_buf->dma_buf) {
+ dma_buf_put(mmal_buf->dma_buf);
+ mmal_buf->dma_buf = NULL;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_queue_setup(struct vb2_queue *q,
+ unsigned int *nbuffers,
+ unsigned int *nplanes,
+ unsigned int sizes[],
+ struct device *alloc_devs[])
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ unsigned int size;
+
+ if (setup_mmal_port(node))
+ return -EINVAL;
+
+ size = node->q_data.sizeimage;
+ if (size == 0) {
+ v4l2_info(&node_get_dev(node)->v4l2_dev,
+ "%s: Image size unset in queue_setup for node %s[%d]\n",
+ __func__, node->name, node->id);
+ return -EINVAL;
+ }
+
+ if (*nplanes)
+ return sizes[0] < size ? -EINVAL : 0;
+
+ *nplanes = 1;
+ sizes[0] = size;
+
+ node->port->current_buffer.size = size;
+
+ if (*nbuffers < node->port->minimum_buffer.num)
+ *nbuffers = node->port->minimum_buffer.num;
+
+ node->port->current_buffer.num = *nbuffers;
+
+ v4l2_dbg(2, debug, &node_get_dev(node)->v4l2_dev,
+ "%s: Image size %u, nbuffers %u for node %s[%d]\n",
+ __func__, sizes[0], *nbuffers, node->name, node->id);
+ return 0;
+}
+
+static int bcm2835_isp_buf_init(struct vb2_buffer *vb)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(vb->vb2_queue);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buf =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: vb %p\n", __func__, vb);
+
+ buf->mmal.buffer = vb2_plane_vaddr(&buf->vb.vb2_buf, 0);
+ buf->mmal.buffer_size = vb2_plane_size(&buf->vb.vb2_buf, 0);
+ mmal_vchi_buffer_init(dev->mmal_instance, &buf->mmal);
+ return 0;
+}
+
+static int bcm2835_isp_buf_prepare(struct vb2_buffer *vb)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(vb->vb2_queue);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buf =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+ struct dma_buf *dma_buf;
+ int ret;
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: type: %d ptr %p\n",
+ __func__, vb->vb2_queue->type, vb);
+
+ if (V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type)) {
+ if (vb2->field == V4L2_FIELD_ANY)
+ vb2->field = V4L2_FIELD_NONE;
+ if (vb2->field != V4L2_FIELD_NONE) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s field isn't supported\n", __func__);
+ return -EINVAL;
+ }
+ }
+
+ if (vb2_plane_size(vb, 0) < node->q_data.sizeimage) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s data will not fit into plane (%lu < %lu)\n",
+ __func__, vb2_plane_size(vb, 0),
+ (long)node->q_data.sizeimage);
+ return -EINVAL;
+ }
+
+ if (!V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type))
+ vb2_set_plane_payload(vb, 0, node->q_data.sizeimage);
+
+ switch (vb->memory) {
+ case VB2_MEMORY_DMABUF:
+ dma_buf = dma_buf_get(vb->planes[0].m.fd);
+
+ if (dma_buf != buf->mmal.dma_buf) {
+ /*
+ * dmabuf either hasn't already been mapped, or it has
+ * changed.
+ */
+ if (buf->mmal.dma_buf) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s Buffer changed - why did the core not call cleanup?\n",
+ __func__);
+ bcm2835_isp_mmal_buf_cleanup(&buf->mmal);
+ }
+
+ buf->mmal.dma_buf = dma_buf;
+ } else {
+ /*
+ * Already have a reference to the buffer, so release it
+ * here.
+ */
+ dma_buf_put(dma_buf);
+ }
+ ret = 0;
+ break;
+ case VB2_MEMORY_MMAP:
+ /*
+ * We want to do this at init, but vb2_core_expbuf checks that
+ * the index < q->num_buffers, and q->num_buffers only gets
+ * updated once all the buffers are allocated.
+ */
+ if (!buf->mmal.dma_buf) {
+ buf->mmal.dma_buf = vb2_core_expbuf_dmabuf(vb->vb2_queue,
+ vb->vb2_queue->type,
+ vb->index, 0, O_CLOEXEC);
+ v4l2_dbg(3, debug, &dev->v4l2_dev,
+ "%s: exporting ptr %p to dmabuf %p\n",
+ __func__, vb, buf->mmal.dma_buf);
+ if (IS_ERR(buf->mmal.dma_buf)) {
+ ret = PTR_ERR(buf->mmal.dma_buf);
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed to expbuf idx %d, ret %d\n",
+ __func__, vb->index, ret);
+ }
+ } else {
+ ret = 0;
+ }
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static void bcm2835_isp_node_buffer_queue(struct vb2_buffer *buf)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(buf->vb2_queue);
+ struct vb2_v4l2_buffer *vbuf =
+ container_of(buf, struct vb2_v4l2_buffer, vb2_buf);
+ struct bcm2835_isp_buffer *buffer =
+ container_of(vbuf, struct bcm2835_isp_buffer, vb);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: node %s[%d], buffer %p\n",
+ __func__, node->name, node->id, buffer);
+
+ vb2_to_mmal_buffer(&buffer->mmal, &buffer->vb);
+ v4l2_dbg(3, debug, &dev->v4l2_dev,
+ "%s: node %s[%d] - submitting mmal dmabuf %p\n", __func__,
+ node->name, node->id, buffer->mmal.dma_buf);
+ vchiq_mmal_submit_buffer(dev->mmal_instance, node->port, &buffer->mmal);
+}
+
+static void bcm2835_isp_buffer_cleanup(struct vb2_buffer *vb)
+{
+ struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb);
+ struct bcm2835_isp_buffer *buffer =
+ container_of(vb2, struct bcm2835_isp_buffer, vb);
+
+ bcm2835_isp_mmal_buf_cleanup(&buffer->mmal);
+}
+
+static int bcm2835_isp_node_start_streaming(struct vb2_queue *q,
+ unsigned int count)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ int ret;
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: node %s[%d] (count %u)\n",
+ __func__, node->name, node->id, count);
+
+ ret = vchiq_mmal_component_enable(dev->mmal_instance, dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "%s: Failed enabling component, ret %d\n",
+ __func__, ret);
+ return -EIO;
+ }
+
+ node->sequence = 0;
+ node->port->cb_ctx = node;
+ ret = vchiq_mmal_port_enable(dev->mmal_instance, node->port,
+ mmal_buffer_cb);
+ if (!ret)
+ atomic_inc(&dev->num_streaming);
+ else
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed enabling port, ret %d\n", __func__, ret);
+
+ return ret;
+}
+
+static void bcm2835_isp_node_stop_streaming(struct vb2_queue *q)
+{
+ struct bcm2835_isp_node *node = vb2_get_drv_priv(q);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ int ret;
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: node %s[%d], mmal port %p\n",
+ __func__, node->name, node->id, node->port);
+
+ init_completion(&dev->frame_cmplt);
+
+ /* Disable MMAL port - this will flush buffers back */
+ ret = vchiq_mmal_port_disable(dev->mmal_instance, node->port);
+ if (ret)
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed disabling %s port, ret %d\n", __func__,
+ node_is_output(node) ? "i/p" : "o/p",
+ ret);
+
+ while (atomic_read(&node->port->buffers_with_vpu)) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Waiting for buffers to be returned - %d outstanding\n",
+ __func__, atomic_read(&node->port->buffers_with_vpu));
+ ret = wait_for_completion_timeout(&dev->frame_cmplt,
+ COMPLETE_TIMEOUT);
+ if (ret <= 0) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Timeout waiting for buffers to be returned - %d outstanding\n",
+ __func__,
+ atomic_read(&node->port->buffers_with_vpu));
+ break;
+ }
+ }
+
+ atomic_dec(&dev->num_streaming);
+ /* If all ports disabled, then disable the component */
+ if (atomic_read(&dev->num_streaming) == 0) {
+ struct bcm2835_isp_lens_shading ls;
+ /*
+ * The ISP component on the firmware has a reference to the
+ * dmabuf handle for the lens shading table. Pass a null handle
+ * to remove that reference now.
+ */
+ memset(&ls, 0, sizeof(ls));
+ /* Must set a valid grid size for the FW */
+ ls.grid_cell_size = 16;
+ set_isp_param(&dev->node[0],
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ &ls, sizeof(ls));
+
+ ret = vchiq_mmal_component_disable(dev->mmal_instance,
+ dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed disabling component, ret %d\n",
+ __func__, ret);
+ }
+ }
+
+ /*
+ * Simply wait for any vb2 buffers to finish. We could take steps to
+ * make them complete more quickly if we care, or even return them
+ * ourselves.
+ */
+ vb2_wait_for_all_buffers(&node->queue);
+}
+
+static const struct vb2_ops bcm2835_isp_node_queue_ops = {
+ .queue_setup = bcm2835_isp_node_queue_setup,
+ .buf_init = bcm2835_isp_buf_init,
+ .buf_prepare = bcm2835_isp_buf_prepare,
+ .buf_queue = bcm2835_isp_node_buffer_queue,
+ .buf_cleanup = bcm2835_isp_buffer_cleanup,
+ .start_streaming = bcm2835_isp_node_start_streaming,
+ .stop_streaming = bcm2835_isp_node_stop_streaming,
+};
+
+static const
+struct bcm2835_isp_fmt *get_default_format(struct bcm2835_isp_node *node)
+{
+ return node->supported_fmts[0];
+}
+
+static inline unsigned int get_bytesperline(int width,
+ const struct bcm2835_isp_fmt *fmt)
+{
+ /* GPU aligns 24bpp images to a multiple of 32 pixels (not bytes). */
+ if (fmt->depth == 24)
+ return ALIGN(width, 32) * 3;
+ else
+ return ALIGN((width * fmt->depth) >> 3, fmt->bytesperline_align);
+}
+
+static inline unsigned int get_sizeimage(int bpl, int width, int height,
+ const struct bcm2835_isp_fmt *fmt)
+{
+ return (bpl * height * fmt->size_multiplier_x2) >> 1;
+}
+
+static int bcm2835_isp_s_ctrl(struct v4l2_ctrl *ctrl)
+{
+ struct bcm2835_isp_dev *dev =
+ container_of(ctrl->handler, struct bcm2835_isp_dev, ctrl_handler);
+ struct bcm2835_isp_node *node = &dev->node[0];
+ int ret = 0;
+
+ /*
+ * The ISP firmware driver will ensure these settings are applied on
+ * a frame boundary, so we are safe to write them as they come in.
+ *
+ * Note that the bcm2835_isp_* param structures are identical to the
+ * mmal-parameters.h definitions. This avoids the need for unnecessary
+ * field-by-field copying between structures.
+ */
+ switch (ctrl->id) {
+ case V4L2_CID_RED_BALANCE:
+ dev->r_gain = ctrl->val;
+ ret = set_wb_gains(node);
+ break;
+ case V4L2_CID_BLUE_BALANCE:
+ dev->b_gain = ctrl->val;
+ ret = set_wb_gains(node);
+ break;
+ case V4L2_CID_DIGITAL_GAIN:
+ ret = set_digital_gain(node, ctrl->val);
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_CC_MATRIX:
+ ret = set_isp_param(node, MMAL_PARAMETER_CUSTOM_CCM,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_custom_ccm));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_LENS_SHADING:
+ {
+ struct bcm2835_isp_lens_shading *v4l2_ls;
+ struct mmal_parameter_lens_shading_v2 ls;
+ struct dma_buf *dmabuf;
+ void *vcsm_handle;
+
+ v4l2_ls = (struct bcm2835_isp_lens_shading *)ctrl->p_new.p_u8;
+ /*
+ * struct bcm2835_isp_lens_shading and struct
+ * mmal_parameter_lens_shading_v2 match so that we can do a
+ * simple memcpy here.
+ * Only the dmabuf to the actual table needs any manipulation.
+ */
+ memcpy(&ls, v4l2_ls, sizeof(ls));
+
+ dmabuf = dma_buf_get(v4l2_ls->dmabuf);
+ if (IS_ERR_OR_NULL(dmabuf))
+ return -EINVAL;
+
+ ret = vc_sm_cma_import_dmabuf(dmabuf, &vcsm_handle);
+ if (ret) {
+ dma_buf_put(dmabuf);
+ return -EINVAL;
+ }
+
+ ls.mem_handle_table = vc_sm_cma_int_handle(vcsm_handle);
+ if (ls.mem_handle_table)
+ /* The VPU will take a reference on the vcsm handle,
+ * which in turn will retain a reference on the dmabuf.
+ * This code can therefore safely release all
+ * references to the buffer.
+ */
+ ret = set_isp_param(node,
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ &ls,
+ sizeof(ls));
+ else
+ ret = -EINVAL;
+
+ vc_sm_cma_free(vcsm_handle);
+ dma_buf_put(dmabuf);
+ break;
+ }
+ case V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL:
+ ret = set_isp_param(node, MMAL_PARAMETER_BLACK_LEVEL,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_black_level));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_GEQ:
+ ret = set_isp_param(node, MMAL_PARAMETER_GEQ,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_geq));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_GAMMA:
+ ret = set_isp_param(node, MMAL_PARAMETER_GAMMA,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_gamma));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_DENOISE:
+ ret = set_isp_param(node, MMAL_PARAMETER_DENOISE,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_denoise));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_CDN:
+ ret = set_isp_param(node, MMAL_PARAMETER_CDN,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_cdn));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_SHARPEN:
+ ret = set_isp_param(node, MMAL_PARAMETER_SHARPEN,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_sharpen));
+ break;
+ case V4L2_CID_USER_BCM2835_ISP_DPC:
+ ret = set_isp_param(node, MMAL_PARAMETER_DPC,
+ ctrl->p_new.p_u8,
+ sizeof(struct bcm2835_isp_dpc));
+ break;
+ default:
+ v4l2_info(&dev->v4l2_dev, "Unrecognised control\n");
+ ret = -EINVAL;
+ }
+
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "%s: Failed setting ctrl \"%s\" (%08x), err %d\n",
+ __func__, ctrl->name, ctrl->id, ret);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+static const struct v4l2_ctrl_ops bcm2835_isp_ctrl_ops = {
+ .s_ctrl = bcm2835_isp_s_ctrl,
+};
+
+static const struct v4l2_file_operations bcm2835_isp_fops = {
+ .owner = THIS_MODULE,
+ .open = v4l2_fh_open,
+ .release = vb2_fop_release,
+ .poll = vb2_fop_poll,
+ .unlocked_ioctl = video_ioctl2,
+ .mmap = vb2_fop_mmap
+};
+
+static int populate_qdata_fmt(struct v4l2_format *f,
+ struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+ int ret;
+
+ if (!node_is_stats(node)) {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Setting pix format for type %d, wxh: %ux%u, fmt: %08x, size %u\n",
+ __func__, f->type, f->fmt.pix.width, f->fmt.pix.height,
+ f->fmt.pix.pixelformat, f->fmt.pix.sizeimage);
+
+ q_data->fmt = find_format(f, node);
+ q_data->width = f->fmt.pix.width;
+ q_data->height = f->fmt.pix.height;
+ q_data->height = f->fmt.pix.height;
+
+ /* All parameters should have been set correctly by try_fmt */
+ q_data->bytesperline = f->fmt.pix.bytesperline;
+ q_data->sizeimage = f->fmt.pix.sizeimage;
+ } else {
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Setting meta format for fmt: %08x, size %u\n",
+ __func__, f->fmt.meta.dataformat,
+ f->fmt.meta.buffersize);
+
+ q_data->fmt = find_format(f, node);
+ q_data->width = 0;
+ q_data->height = 0;
+ q_data->bytesperline = 0;
+ q_data->sizeimage = f->fmt.meta.buffersize;
+ }
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Calculated bpl as %u, size %u\n", __func__,
+ q_data->bytesperline, q_data->sizeimage);
+
+ setup_mmal_port_format(node, node->port);
+ ret = vchiq_mmal_port_set_format(dev->mmal_instance, node->port);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Failed vchiq_mmal_port_set_format on port, ret %d\n",
+ __func__, ret);
+ ret = -EINVAL;
+ }
+
+ if (q_data->sizeimage < node->port->minimum_buffer.size) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: Current buffer size of %u < min buf size %u - driver mismatch to MMAL\n",
+ __func__,
+ q_data->sizeimage,
+ node->port->minimum_buffer.size);
+ }
+
+ v4l2_dbg(1, debug, &dev->v4l2_dev,
+ "%s: Set format for type %d, wxh: %dx%d, fmt: %08x, size %u\n",
+ __func__, f->type, q_data->width, q_data->height,
+ q_data->fmt->fourcc, q_data->sizeimage);
+
+ return ret;
+}
+
+static int bcm2835_isp_node_querycap(struct file *file, void *priv,
+ struct v4l2_capability *cap)
+{
+ strscpy(cap->driver, BCM2835_ISP_NAME, sizeof(cap->driver));
+ strscpy(cap->card, BCM2835_ISP_NAME, sizeof(cap->card));
+ snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
+ BCM2835_ISP_NAME);
+
+ return 0;
+}
+
+static int bcm2835_isp_node_g_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ if (node_is_stats(node)) {
+ f->fmt.meta.dataformat = V4L2_META_FMT_BCM2835_ISP_STATS;
+ f->fmt.meta.buffersize =
+ node->port->minimum_buffer.size;
+ } else {
+ struct bcm2835_isp_q_data *q_data = &node->q_data;
+
+ f->fmt.pix.width = q_data->width;
+ f->fmt.pix.height = q_data->height;
+ f->fmt.pix.field = V4L2_FIELD_NONE;
+ f->fmt.pix.pixelformat = q_data->fmt->fourcc;
+ f->fmt.pix.bytesperline = q_data->bytesperline;
+ f->fmt.pix.sizeimage = q_data->sizeimage;
+ f->fmt.pix.colorspace = q_data->fmt->colorspace;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_enum_fmt(struct file *file, void *priv,
+ struct v4l2_fmtdesc *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ if (f->index < node->num_supported_fmts) {
+ /* Format found */
+ f->pixelformat = node->supported_fmts[f->index]->fourcc;
+ f->flags = 0;
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+static int bcm2835_isp_enum_framesizes(struct file *file, void *priv,
+ struct v4l2_frmsizeenum *fsize)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ const struct bcm2835_isp_fmt *fmt;
+
+ if (node_is_stats(node) || fsize->index)
+ return -EINVAL;
+
+ fmt = find_format_by_fourcc(fsize->pixel_format, node);
+ if (!fmt) {
+ v4l2_err(&dev->v4l2_dev, "Invalid pixel code: %x\n",
+ fsize->pixel_format);
+ return -EINVAL;
+ }
+
+ fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE;
+ fsize->stepwise.min_width = MIN_DIM;
+ fsize->stepwise.max_width = MAX_DIM;
+ fsize->stepwise.step_width = fmt->step_size;
+
+ fsize->stepwise.min_height = MIN_DIM;
+ fsize->stepwise.max_height = MAX_DIM;
+ fsize->stepwise.step_height = fmt->step_size;
+
+ return 0;
+}
+
+static int bcm2835_isp_node_try_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ const struct bcm2835_isp_fmt *fmt;
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ fmt = find_format(f, node);
+ if (!fmt)
+ fmt = get_default_format(node);
+
+ if (!node_is_stats(node)) {
+ f->fmt.pix.width = max(min(f->fmt.pix.width, MAX_DIM),
+ MIN_DIM);
+ f->fmt.pix.height = max(min(f->fmt.pix.height, MAX_DIM),
+ MIN_DIM);
+
+ f->fmt.pix.pixelformat = fmt->fourcc;
+ f->fmt.pix.colorspace = fmt->colorspace;
+ f->fmt.pix.bytesperline = get_bytesperline(f->fmt.pix.width,
+ fmt);
+ f->fmt.pix.field = V4L2_FIELD_NONE;
+ f->fmt.pix.sizeimage =
+ get_sizeimage(f->fmt.pix.bytesperline, f->fmt.pix.width,
+ f->fmt.pix.height, fmt);
+ } else {
+ f->fmt.meta.dataformat = fmt->fourcc;
+ f->fmt.meta.buffersize = node->port->minimum_buffer.size;
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_node_s_fmt(struct file *file, void *priv,
+ struct v4l2_format *f)
+{
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ int ret;
+
+ if (f->type != node->queue.type)
+ return -EINVAL;
+
+ ret = bcm2835_isp_node_try_fmt(file, priv, f);
+ if (ret)
+ return ret;
+
+ v4l2_dbg(1, debug, &node_get_dev(node)->v4l2_dev,
+ "%s: Set format for node %s[%d]\n",
+ __func__, node->name, node->id);
+
+ return populate_qdata_fmt(f, node);
+}
+
+static int bcm2835_isp_node_s_selection(struct file *file, void *fh,
+ struct v4l2_selection *s)
+{
+ struct mmal_parameter_crop crop;
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ /* This return value is required fro V4L2 compliance. */
+ if (node_is_stats(node))
+ return -ENOTTY;
+
+ if (!s->r.width || !s->r.height)
+ return -EINVAL;
+
+ /* We can only set crop on the input. */
+ switch (s->target) {
+ case V4L2_SEL_TGT_CROP:
+ /*
+ * Adjust the crop window if it goes outside of the frame
+ * dimensions.
+ */
+ s->r.left = min((unsigned int)max(s->r.left, 0),
+ node->q_data.width - MIN_DIM);
+ s->r.top = min((unsigned int)max(s->r.top, 0),
+ node->q_data.height - MIN_DIM);
+ s->r.width = max(min(s->r.width,
+ node->q_data.width - s->r.left), MIN_DIM);
+ s->r.height = max(min(s->r.height,
+ node->q_data.height - s->r.top), MIN_DIM);
+ break;
+ case V4L2_SEL_TGT_CROP_DEFAULT:
+ /* Default (i.e. no) crop window. */
+ s->r.left = 0;
+ s->r.top = 0;
+ s->r.width = node->q_data.width;
+ s->r.height = node->q_data.height;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ crop.rect.x = s->r.left;
+ crop.rect.y = s->r.top;
+ crop.rect.width = s->r.width;
+ crop.rect.height = s->r.height;
+
+ return vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_CROP,
+ &crop, sizeof(crop));
+}
+
+static int bcm2835_isp_node_g_selection(struct file *file, void *fh,
+ struct v4l2_selection *s)
+{
+ struct mmal_parameter_crop crop;
+ struct bcm2835_isp_node *node = video_drvdata(file);
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ u32 crop_size = sizeof(crop);
+ int ret;
+
+ /* We can only return out an input crop. */
+ switch (s->target) {
+ case V4L2_SEL_TGT_CROP:
+ ret = vchiq_mmal_port_parameter_get(dev->mmal_instance,
+ node->port,
+ MMAL_PARAMETER_CROP,
+ &crop, &crop_size);
+ if (!ret) {
+ s->r.left = crop.rect.x;
+ s->r.top = crop.rect.y;
+ s->r.width = crop.rect.width;
+ s->r.height = crop.rect.height;
+ }
+ break;
+ case V4L2_SEL_TGT_CROP_DEFAULT:
+ case V4L2_SEL_TGT_CROP_BOUNDS:
+ /* Default (i.e. no) crop window. */
+ s->r.left = 0;
+ s->r.top = 0;
+ s->r.width = node->q_data.width;
+ s->r.height = node->q_data.height;
+ ret = 0;
+ break;
+ default:
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
+static int bcm3285_isp_subscribe_event(struct v4l2_fh *fh,
+ const struct v4l2_event_subscription *s)
+{
+ switch (s->type) {
+ /* Cannot change source parameters dynamically at runtime. */
+ case V4L2_EVENT_SOURCE_CHANGE:
+ return -EINVAL;
+ case V4L2_EVENT_CTRL:
+ return v4l2_ctrl_subscribe_event(fh, s);
+ default:
+ return v4l2_event_subscribe(fh, s, 4, NULL);
+ }
+}
+
+static const struct v4l2_ioctl_ops bcm2835_isp_node_ioctl_ops = {
+ .vidioc_querycap = bcm2835_isp_node_querycap,
+ .vidioc_g_fmt_vid_cap = bcm2835_isp_node_g_fmt,
+ .vidioc_g_fmt_vid_out = bcm2835_isp_node_g_fmt,
+ .vidioc_g_fmt_meta_cap = bcm2835_isp_node_g_fmt,
+ .vidioc_s_fmt_vid_cap = bcm2835_isp_node_s_fmt,
+ .vidioc_s_fmt_vid_out = bcm2835_isp_node_s_fmt,
+ .vidioc_s_fmt_meta_cap = bcm2835_isp_node_s_fmt,
+ .vidioc_try_fmt_vid_cap = bcm2835_isp_node_try_fmt,
+ .vidioc_try_fmt_vid_out = bcm2835_isp_node_try_fmt,
+ .vidioc_try_fmt_meta_cap = bcm2835_isp_node_try_fmt,
+ .vidioc_s_selection = bcm2835_isp_node_s_selection,
+ .vidioc_g_selection = bcm2835_isp_node_g_selection,
+
+ .vidioc_enum_fmt_vid_cap = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_fmt_vid_out = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_fmt_meta_cap = bcm2835_isp_node_enum_fmt,
+ .vidioc_enum_framesizes = bcm2835_isp_enum_framesizes,
+
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_prepare_buf = vb2_ioctl_prepare_buf,
+
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+
+ .vidioc_subscribe_event = bcm3285_isp_subscribe_event,
+ .vidioc_unsubscribe_event = v4l2_event_unsubscribe,
+};
+
+/*
+ * Size of the array to provide to the VPU when asking for the list of supported
+ * formats.
+ *
+ * The ISP component currently advertises 62 input formats, so add a small
+ * overhead on that. Should the component advertise more formats then the excess
+ * will be dropped and a warning logged.
+ */
+#define MAX_SUPPORTED_ENCODINGS 70
+
+/* Populate node->supported_fmts with the formats supported by those ports. */
+static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+ struct bcm2835_isp_fmt const **list;
+ unsigned int i, j, num_encodings;
+ u32 fourccs[MAX_SUPPORTED_ENCODINGS];
+ u32 param_size = sizeof(fourccs);
+ int ret;
+
+ ret = vchiq_mmal_port_parameter_get(dev->mmal_instance, node->port,
+ MMAL_PARAMETER_SUPPORTED_ENCODINGS,
+ &fourccs, ¶m_size);
+
+ if (ret) {
+ if (ret == MMAL_MSG_STATUS_ENOSPC) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: port has more encodings than we provided space for. Some are dropped (%zu vs %u).\n",
+ __func__, param_size / sizeof(u32),
+ MAX_SUPPORTED_ENCODINGS);
+ num_encodings = MAX_SUPPORTED_ENCODINGS;
+ } else {
+ v4l2_err(&dev->v4l2_dev, "%s: get_param ret %u.\n",
+ __func__, ret);
+ return -EINVAL;
+ }
+ } else {
+ num_encodings = param_size / sizeof(u32);
+ }
+
+ /*
+ * Assume at this stage that all encodings will be supported in V4L2.
+ * Any that aren't supported will waste a very small amount of memory.
+ */
+ list = devm_kzalloc(dev->dev,
+ sizeof(struct bcm2835_isp_fmt *) * num_encodings,
+ GFP_KERNEL);
+ if (!list)
+ return -ENOMEM;
+ node->supported_fmts = list;
+
+ for (i = 0, j = 0; i < num_encodings; i++) {
+ const struct bcm2835_isp_fmt *fmt = get_fmt(fourccs[i]);
+
+ if (fmt) {
+ list[j] = fmt;
+ j++;
+ }
+ }
+ node->num_supported_fmts = j;
+
+ return 0;
+}
+
+/*
+ * Register a device node /dev/video<N> to go along with one of the ISP's input
+ * or output nodes.
+ */
+static int register_node(struct bcm2835_isp_dev *dev,
+ struct bcm2835_isp_node *node,
+ int index)
+{
+ struct video_device *vfd;
+ struct vb2_queue *queue;
+ int ret;
+
+ mutex_init(&node->lock);
+ mutex_init(&node->queue_lock);
+
+ node->dev = dev;
+ vfd = &node->vfd;
+ queue = &node->queue;
+ queue->type = index_to_queue_type(index);
+ /*
+ * Setup the node type-specific params.
+ *
+ * Only the OUTPUT node can set controls and crop windows. However,
+ * we must allow the s/g_selection ioctl on the stats node as v4l2
+ * compliance expects it to return a -ENOTTY, and the framework
+ * does not handle it if the ioctl is disabled.
+ */
+ switch (queue->type) {
+ case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+ vfd->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+ node->id = index;
+ node->vfl_dir = VFL_DIR_TX;
+ node->name = "output";
+ node->port = &dev->component->input[node->id];
+ break;
+ case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+ vfd->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+ /* First Capture node starts at id 0, etc. */
+ node->id = index - BCM2835_ISP_NUM_OUTPUTS;
+ node->vfl_dir = VFL_DIR_RX;
+ node->name = "capture";
+ node->port = &dev->component->output[node->id];
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_CTRL);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_SELECTION);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_G_SELECTION);
+ break;
+ case V4L2_BUF_TYPE_META_CAPTURE:
+ vfd->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING;
+ node->id = index - BCM2835_ISP_NUM_OUTPUTS;
+ node->vfl_dir = VFL_DIR_RX;
+ node->name = "stats";
+ node->port = &dev->component->output[node->id];
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_CTRL);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_S_SELECTION);
+ v4l2_disable_ioctl(&node->vfd, VIDIOC_G_SELECTION);
+ break;
+ }
+
+ /* We use the selection API instead of the old crop API. */
+ v4l2_disable_ioctl(vfd, VIDIOC_CROPCAP);
+ v4l2_disable_ioctl(vfd, VIDIOC_G_CROP);
+ v4l2_disable_ioctl(vfd, VIDIOC_S_CROP);
+
+ ret = bcm2835_isp_get_supported_fmts(node);
+ if (ret)
+ return ret;
+
+ /* Initialise the video node. */
+ vfd->vfl_type = VFL_TYPE_VIDEO;
+ vfd->fops = &bcm2835_isp_fops,
+ vfd->ioctl_ops = &bcm2835_isp_node_ioctl_ops,
+ vfd->minor = -1,
+ vfd->release = video_device_release_empty,
+ vfd->queue = &node->queue;
+ vfd->lock = &node->lock;
+ vfd->v4l2_dev = &dev->v4l2_dev;
+ vfd->vfl_dir = node->vfl_dir;
+
+ node->q_data.fmt = get_default_format(node);
+ node->q_data.width = DEFAULT_DIM;
+ node->q_data.height = DEFAULT_DIM;
+ node->q_data.bytesperline =
+ get_bytesperline(DEFAULT_DIM, node->q_data.fmt);
+ node->q_data.sizeimage = node_is_stats(node) ?
+ node->port->recommended_buffer.size :
+ get_sizeimage(node->q_data.bytesperline,
+ node->q_data.width,
+ node->q_data.height,
+ node->q_data.fmt);
+
+ queue->io_modes = VB2_MMAP | VB2_DMABUF;
+ queue->drv_priv = node;
+ queue->ops = &bcm2835_isp_node_queue_ops;
+ queue->mem_ops = &vb2_dma_contig_memops;
+ queue->buf_struct_size = sizeof(struct bcm2835_isp_buffer);
+ queue->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+ queue->dev = dev->dev;
+ queue->lock = &node->queue_lock;
+
+ ret = vb2_queue_init(queue);
+ if (ret < 0) {
+ v4l2_info(&dev->v4l2_dev, "vb2_queue_init failed\n");
+ return ret;
+ }
+
+ /* Set some controls and defaults, but only on the VIDEO_OUTPUT node. */
+ if (node_is_output(node)) {
+ unsigned int i;
+
+ /* Use this ctrl template to assign custom ISP ctrls. */
+ struct v4l2_ctrl_config ctrl_template = {
+ .ops = &bcm2835_isp_ctrl_ops,
+ .type = V4L2_CTRL_TYPE_U8,
+ .def = 0,
+ .min = 0x00,
+ .max = 0xff,
+ .step = 1,
+ };
+
+ /* 3 standard controls, and an array of custom controls */
+ ret = v4l2_ctrl_handler_init(&dev->ctrl_handler,
+ 3 + ARRAY_SIZE(custom_ctrls));
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev, "ctrl_handler init failed (%d)\n",
+ ret);
+ goto queue_cleanup;
+ }
+
+ dev->r_gain = 1000;
+ dev->b_gain = 1000;
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_RED_BALANCE, 1, 0xffff, 1,
+ dev->r_gain);
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_BLUE_BALANCE, 1, 0xffff, 1,
+ dev->b_gain);
+
+ v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops,
+ V4L2_CID_DIGITAL_GAIN, 1, 0xffff, 1, 1000);
+
+ for (i = 0; i < ARRAY_SIZE(custom_ctrls); i++) {
+ ctrl_template.name = custom_ctrls[i].name;
+ ctrl_template.id = custom_ctrls[i].id;
+ ctrl_template.dims[0] = custom_ctrls[i].size;
+ ctrl_template.flags = custom_ctrls[i].flags;
+ v4l2_ctrl_new_custom(&dev->ctrl_handler,
+ &ctrl_template, NULL);
+ }
+
+ node->vfd.ctrl_handler = &dev->ctrl_handler;
+ if (dev->ctrl_handler.error) {
+ ret = dev->ctrl_handler.error;
+ v4l2_err(&dev->v4l2_dev, "controls init failed (%d)\n",
+ ret);
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+ goto ctrl_cleanup;
+ }
+ }
+
+ /* Define the device names */
+ snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
+ node->name, node->id);
+
+ ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "Failed to register video %s[%d] device node\n",
+ node->name, node->id);
+ goto ctrl_cleanup;
+ }
+
+ node->registered = true;
+ video_set_drvdata(vfd, node);
+
+ v4l2_info(&dev->v4l2_dev,
+ "Device node %s[%d] registered as /dev/video%d\n",
+ node->name, node->id, vfd->num);
+
+ return 0;
+
+ctrl_cleanup:
+ if (node_is_output(node))
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+queue_cleanup:
+ vb2_queue_release(&node->queue);
+ return ret;
+}
+
+/* Unregister one of the /dev/video<N> nodes associated with the ISP. */
+static void bcm2835_isp_unregister_node(struct bcm2835_isp_node *node)
+{
+ struct bcm2835_isp_dev *dev = node_get_dev(node);
+
+ v4l2_info(&dev->v4l2_dev,
+ "Unregistering node %s[%d] device node /dev/video%d\n",
+ node->name, node->id, node->vfd.num);
+
+ if (node->registered) {
+ video_unregister_device(&node->vfd);
+ if (node_is_output(node))
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+ vb2_queue_release(&node->queue);
+ }
+
+ /*
+ * node->supported_fmts.list is free'd automatically
+ * as a managed resource.
+ */
+ node->supported_fmts = NULL;
+ node->num_supported_fmts = 0;
+ node->vfd.ctrl_handler = NULL;
+ node->registered = false;
+}
+
+static void media_controller_unregister(struct bcm2835_isp_dev *dev)
+{
+ unsigned int i;
+
+ v4l2_info(&dev->v4l2_dev, "Unregister from media controller\n");
+
+ if (dev->media_device_registered) {
+ media_device_unregister(&dev->mdev);
+ media_device_cleanup(&dev->mdev);
+ dev->media_device_registered = false;
+ }
+
+ kfree(dev->entity.name);
+ dev->entity.name = NULL;
+
+ if (dev->media_entity_registered) {
+ media_device_unregister_entity(&dev->entity);
+ dev->media_entity_registered = false;
+ }
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ struct bcm2835_isp_node *node = &dev->node[i];
+
+ if (node->media_node_registered) {
+ media_remove_intf_links(node->intf_link->intf);
+ media_entity_remove_links(&dev->node[i].vfd.entity);
+ media_devnode_remove(node->intf_devnode);
+ media_device_unregister_entity(&node->vfd.entity);
+ kfree(node->vfd.entity.name);
+ }
+ node->media_node_registered = false;
+ }
+
+ dev->v4l2_dev.mdev = NULL;
+}
+
+static int media_controller_register_node(struct bcm2835_isp_dev *dev, int num)
+{
+ struct bcm2835_isp_node *node = &dev->node[num];
+ struct media_entity *entity = &node->vfd.entity;
+ int output = node_is_output(node);
+ char *name;
+ int ret;
+
+ v4l2_info(&dev->v4l2_dev,
+ "Register %s node %d with media controller\n",
+ output ? "output" : "capture", num);
+ entity->obj_type = MEDIA_ENTITY_TYPE_VIDEO_DEVICE;
+ entity->function = MEDIA_ENT_F_IO_V4L;
+ entity->info.dev.major = VIDEO_MAJOR;
+ entity->info.dev.minor = node->vfd.minor;
+ name = kmalloc(BCM2835_ISP_ENTITY_NAME_LEN, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto error_no_mem;
+ }
+ snprintf(name, BCM2835_ISP_ENTITY_NAME_LEN, "%s0-%s%d",
+ BCM2835_ISP_NAME, output ? "output" : "capture", num);
+ entity->name = name;
+ node->pad.flags = output ? MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK;
+ ret = media_entity_pads_init(entity, 1, &node->pad);
+ if (ret)
+ goto error_pads_init;
+ ret = media_device_register_entity(&dev->mdev, entity);
+ if (ret)
+ goto error_register_entity;
+
+ node->intf_devnode = media_devnode_create(&dev->mdev,
+ MEDIA_INTF_T_V4L_VIDEO, 0,
+ VIDEO_MAJOR, node->vfd.minor);
+ if (!node->intf_devnode) {
+ ret = -ENOMEM;
+ goto error_devnode_create;
+ }
+
+ node->intf_link = media_create_intf_link(entity,
+ &node->intf_devnode->intf,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ if (!node->intf_link) {
+ ret = -ENOMEM;
+ goto error_create_intf_link;
+ }
+
+ if (output)
+ ret = media_create_pad_link(entity, 0, &dev->entity, num,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ else
+ ret = media_create_pad_link(&dev->entity, num, entity, 0,
+ MEDIA_LNK_FL_IMMUTABLE |
+ MEDIA_LNK_FL_ENABLED);
+ if (ret)
+ goto error_create_pad_link;
+
+ dev->node[num].media_node_registered = true;
+ return 0;
+
+error_create_pad_link:
+ media_remove_intf_links(&node->intf_devnode->intf);
+error_create_intf_link:
+ media_devnode_remove(node->intf_devnode);
+error_devnode_create:
+ media_device_unregister_entity(&node->vfd.entity);
+error_register_entity:
+error_pads_init:
+ kfree(entity->name);
+ entity->name = NULL;
+error_no_mem:
+ if (ret)
+ v4l2_info(&dev->v4l2_dev, "Error registering node\n");
+
+ return ret;
+}
+
+static int media_controller_register(struct bcm2835_isp_dev *dev)
+{
+ char *name;
+ unsigned int i;
+ int ret;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "Registering with media controller\n");
+ dev->mdev.dev = dev->dev;
+ strscpy(dev->mdev.model, "bcm2835-isp",
+ sizeof(dev->mdev.model));
+ strscpy(dev->mdev.bus_info, "platform:bcm2835-isp",
+ sizeof(dev->mdev.bus_info));
+ media_device_init(&dev->mdev);
+ dev->v4l2_dev.mdev = &dev->mdev;
+
+ v4l2_dbg(2, debug, &dev->v4l2_dev, "Register entity for nodes\n");
+
+ name = kmalloc(BCM2835_ISP_ENTITY_NAME_LEN, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto done;
+ }
+ snprintf(name, BCM2835_ISP_ENTITY_NAME_LEN, "bcm2835_isp0");
+ dev->entity.name = name;
+ dev->entity.obj_type = MEDIA_ENTITY_TYPE_BASE;
+ dev->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER;
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ dev->pad[i].flags = node_is_output(&dev->node[i]) ?
+ MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE;
+ }
+
+ ret = media_entity_pads_init(&dev->entity, BCM2835_ISP_NUM_NODES,
+ dev->pad);
+ if (ret)
+ goto done;
+
+ ret = media_device_register_entity(&dev->mdev, &dev->entity);
+ if (ret)
+ goto done;
+
+ dev->media_entity_registered = true;
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ ret = media_controller_register_node(dev, i);
+ if (ret)
+ goto done;
+ }
+
+ ret = media_device_register(&dev->mdev);
+ if (!ret)
+ dev->media_device_registered = true;
+done:
+ return ret;
+}
+
+static int bcm2835_isp_remove(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
+ unsigned int i;
+
+ media_controller_unregister(dev);
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++)
+ bcm2835_isp_unregister_node(&dev->node[i]);
+
+ v4l2_device_unregister(&dev->v4l2_dev);
+
+ if (dev->component)
+ vchiq_mmal_component_finalise(dev->mmal_instance,
+ dev->component);
+
+ vchiq_mmal_finalise(dev->mmal_instance);
+
+ return 0;
+}
+
+static int bcm2835_isp_probe(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev *dev;
+ unsigned int i;
+ int ret;
+
+ dev = devm_kzalloc(&pdev->dev, sizeof(*dev), GFP_KERNEL);
+ if (!dev)
+ return -ENOMEM;
+
+ dev->dev = &pdev->dev;
+
+ ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
+ if (ret)
+ return ret;
+
+ ret = vchiq_mmal_init(&dev->mmal_instance);
+ if (ret) {
+ v4l2_device_unregister(&dev->v4l2_dev);
+ return ret;
+ }
+
+ ret = vchiq_mmal_component_init(dev->mmal_instance, "ril.isp",
+ &dev->component);
+ if (ret) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: failed to create ril.isp component\n", __func__);
+ goto error;
+ }
+
+ if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
+ dev->component->outputs < BCM2835_ISP_NUM_CAPTURES +
+ BCM2835_ISP_NUM_METADATA) {
+ v4l2_err(&dev->v4l2_dev,
+ "%s: ril.isp returned %d i/p (%d expected), %d o/p (%d expected) ports\n",
+ __func__, dev->component->inputs,
+ BCM2835_ISP_NUM_OUTPUTS,
+ dev->component->outputs,
+ BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
+ goto error;
+ }
+
+ atomic_set(&dev->num_streaming, 0);
+
+ for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
+ struct bcm2835_isp_node *node = &dev->node[i];
+
+ ret = register_node(dev, node, i);
+ if (ret)
+ goto error;
+ }
+
+ ret = media_controller_register(dev);
+ if (ret)
+ goto error;
+
+ platform_set_drvdata(pdev, dev);
+ v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
+ return 0;
+
+error:
+ bcm2835_isp_remove(pdev);
+
+ return ret;
+}
+
+static struct platform_driver bcm2835_isp_pdrv = {
+ .probe = bcm2835_isp_probe,
+ .remove = bcm2835_isp_remove,
+ .driver = {
+ .name = BCM2835_ISP_NAME,
+ },
+};
+
+module_platform_driver(bcm2835_isp_pdrv);
+
+MODULE_DESCRIPTION("BCM2835 ISP driver");
+MODULE_AUTHOR("Naushir Patuck <naush@raspberrypi.com>");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1.0");
+MODULE_ALIAS("platform:bcm2835-isp");
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
index 4711877a9711..538c347db433 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
@@ -135,6 +135,10 @@
*/
#define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I')
+/** ISP image statistics format
+ */
+#define MMAL_ENCODING_BRCM_STATS MMAL_FOURCC('S', 'T', 'A', 'T')
+
/* }@ */
/** \name Pre-defined audio encodings */
diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
index a0cdd28101f2..5e6276aca143 100644
--- a/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
+++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
@@ -223,6 +223,63 @@ enum mmal_parameter_camera_type {
MMAL_PARAMETER_SHUTTER_SPEED,
/**< Takes a @ref MMAL_PARAMETER_AWB_GAINS_T */
MMAL_PARAMETER_CUSTOM_AWB_GAINS,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_SETTINGS_T */
+ MMAL_PARAMETER_CAMERA_SETTINGS,
+ /**< Takes a @ref MMAL_PARAMETER_PRIVACY_INDICATOR_T */
+ MMAL_PARAMETER_PRIVACY_INDICATOR,
+ /**< Takes a @ref MMAL_PARAMETER_BOOLEAN_T */
+ MMAL_PARAMETER_VIDEO_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_BOOLEAN_T */
+ MMAL_PARAMETER_STILLS_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_ANNOTATE_T */
+ MMAL_PARAMETER_ANNOTATE,
+ /**< Takes a @ref MMAL_PARAMETER_STEREOSCOPIC_MODE_T */
+ MMAL_PARAMETER_STEREOSCOPIC_MODE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_INTERFACE_T */
+ MMAL_PARAMETER_CAMERA_INTERFACE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_CLOCKING_MODE_T */
+ MMAL_PARAMETER_CAMERA_CLOCKING_MODE,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_CONFIG_T */
+ MMAL_PARAMETER_CAMERA_RX_CONFIG,
+ /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_TIMING_T */
+ MMAL_PARAMETER_CAMERA_RX_TIMING,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_DPF_CONFIG,
+ /* 0x50 */
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_JPEG_RESTART_INTERVAL,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE,
+ /**< Takes a @ref MMAL_PARAMETER_LENS_SHADING_T */
+ MMAL_PARAMETER_LENS_SHADING_OVERRIDE,
+ /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+ MMAL_PARAMETER_BLACK_LEVEL,
+ /**< Takes a @ref MMAL_PARAMETER_RESIZE_T */
+ MMAL_PARAMETER_RESIZE_PARAMS,
+ /**< Takes a @ref MMAL_PARAMETER_CROP_T */
+ MMAL_PARAMETER_CROP,
+ /**< Takes a @ref MMAL_PARAMETER_INT32_T */
+ MMAL_PARAMETER_OUTPUT_SHIFT,
+ /**< Takes a @ref MMAL_PARAMETER_INT32_T */
+ MMAL_PARAMETER_CCM_SHIFT,
+ /**< Takes a @ref MMAL_PARAMETER_CUSTOM_CCM_T */
+ MMAL_PARAMETER_CUSTOM_CCM,
+ /**< Takes a @ref MMAL_PARAMETER_RATIONAL_T */
+ MMAL_PARAMETER_ANALOG_GAIN,
+ /**< Takes a @ref MMAL_PARAMETER_RATIONAL_T */
+ MMAL_PARAMETER_DIGITAL_GAIN,
+ /**< Takes a @ref MMAL_PARAMETER_DENOISE_T */
+ MMAL_PARAMETER_DENOISE,
+ /**< Takes a @ref MMAL_PARAMETER_SHARPEN_T */
+ MMAL_PARAMETER_SHARPEN,
+ /**< Takes a @ref MMAL_PARAMETER_GEQ_T */
+ MMAL_PARAMETER_GEQ,
+ /**< Tales a @ref MMAP_PARAMETER_DPC_T */
+ MMAL_PARAMETER_DPC,
+ /**< Tales a @ref MMAP_PARAMETER_GAMMA_T */
+ MMAL_PARAMETER_GAMMA,
+ /**< Takes a @ref MMAL_PARAMETER_CDN_T */
+ MMAL_PARAMETER_CDN,
};
enum mmal_parameter_camera_config_timestamp_mode {
@@ -746,7 +803,113 @@ struct mmal_parameter_camera_info {
struct mmal_parameter_camera_info_camera
cameras[MMAL_PARAMETER_CAMERA_INFO_MAX_CAMERAS];
struct mmal_parameter_camera_info_flash
- flashes[MMAL_PARAMETER_CAMERA_INFO_MAX_FLASHES];
+ flashes[MMAL_PARAMETER_CAMERA_INFO_MAX_FLASHES];
+};
+
+struct mmal_parameter_ccm {
+ struct s32_fract cm[3][3];
+ s32 offsets[3];
+};
+
+struct mmal_parameter_custom_ccm {
+ u32 enabled; /**< Enable the custom CCM. */
+ struct mmal_parameter_ccm ccm; /**< CCM to be used. */
+};
+
+struct mmal_parameter_lens_shading {
+ u32 enabled;
+ u32 grid_cell_size;
+ u32 grid_width;
+ u32 grid_stride;
+ u32 grid_height;
+ u32 mem_handle_table;
+ u32 ref_transform;
+};
+
+enum mmal_parameter_ls_gain_format_type {
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U0P8_1 = 0,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U1P7_0 = 1,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U1P7_1 = 2,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U2P6_0 = 3,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U2P6_1 = 4,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U3P5_0 = 5,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U3P5_1 = 6,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U4P10 = 7,
+ MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_DUMMY = 0x7FFFFFFF
+};
+
+struct mmal_parameter_lens_shading_v2 {
+ u32 enabled;
+ u32 grid_cell_size;
+ u32 grid_width;
+ u32 grid_stride;
+ u32 grid_height;
+ u32 mem_handle_table;
+ u32 ref_transform;
+ u32 corner_sampled;
+ enum mmal_parameter_ls_gain_format_type gain_format;
+};
+
+struct mmal_parameter_black_level {
+ u32 enabled;
+ u16 black_level_r;
+ u16 black_level_g;
+ u16 black_level_b;
+ u8 pad_[2]; /* Unused */
+};
+
+struct mmal_parameter_geq {
+ u32 enabled;
+ u32 offset;
+ struct s32_fract slope;
+};
+
+#define MMAL_NUM_GAMMA_PTS 33
+struct mmal_parameter_gamma {
+ u32 enabled;
+ u16 x[MMAL_NUM_GAMMA_PTS];
+ u16 y[MMAL_NUM_GAMMA_PTS];
+};
+
+enum mmal_parameter_cdn_mode {
+ MMAL_PARAM_CDN_FAST = 0,
+ MMAL_PARAM_CDN_HIGH_QUALITY = 1,
+ MMAL_PARAM_CDN_DUMMY = 0x7FFFFFFF
+};
+
+struct mmal_parameter_colour_denoise {
+ u32 enabled;
+ enum mmal_parameter_cdn_mode mode;
+};
+
+struct mmal_parameter_denoise {
+ u32 enabled;
+ u32 constant;
+ struct s32_fract slope;
+ struct s32_fract strength;
+};
+
+struct mmal_parameter_sharpen {
+ u32 enabled;
+ struct s32_fract threshold;
+ struct s32_fract strength;
+ struct s32_fract limit;
+};
+
+enum mmal_dpc_mode {
+ MMAL_DPC_MODE_OFF = 0,
+ MMAL_DPC_MODE_NORMAL = 1,
+ MMAL_DPC_MODE_STRONG = 2,
+ MMAL_DPC_MODE_MAX = 0x7FFFFFFF,
+};
+
+struct mmal_parameter_dpc {
+ u32 enabled;
+ u32 strength;
+};
+
+struct mmal_parameter_crop {
+ struct vchiq_mmal_rect rect;
};
#endif
diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
index cf8c0437f159..c50e3ca81565 100644
--- a/include/uapi/linux/bcm2835-isp.h
+++ b/include/uapi/linux/bcm2835-isp.h
@@ -31,6 +31,8 @@
(V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
#define V4L2_CID_USER_BCM2835_ISP_DPC \
(V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
+#define V4L2_CID_USER_BCM2835_ISP_CDN \
+ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0009)
/*
* All structs below are directly mapped onto the equivalent structs in
@@ -175,6 +177,31 @@ struct bcm2835_isp_gamma {
__u16 y[BCM2835_NUM_GAMMA_PTS];
};
+/**
+ * enum bcm2835_isp_cdn_mode - Mode of operation for colour denoise.
+ *
+ * @CDN_MODE_FAST: Fast (but lower quality) colour denoise
+ * algorithm, typically used for video recording.
+ * @CDN_HIGH_QUALITY: High quality (but slower) colour denoise
+ * algorithm, typically used for stills capture.
+ */
+enum bcm2835_isp_cdn_mode {
+ CDN_MODE_FAST = 0,
+ CDN_MODE_HIGH_QUALITY = 1,
+};
+
+/**
+ * struct bcm2835_isp_cdn - Colour denoise parameters set with the
+ * V4L2_CID_USER_BCM2835_ISP_CDN ctrl.
+ *
+ * @enabled: Enable colour denoise.
+ * @mode: Colour denoise operating mode (see enum &bcm2835_isp_cdn_mode)
+ */
+struct bcm2835_isp_cdn {
+ __u32 enabled;
+ __u32 mode;
+};
+
/**
* struct bcm2835_isp_denoise - Denoise parameters set with the
* V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 10/14] WIP: vc04_services: bcm2835-isp: Allow formats with different colour spaces
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: David Plowman <david.plowman@raspberrypi.com>
Each supported format now includes a mask showing the allowed colour
spaces, as well as a default colour space for when one was not
specified.
Additionally we translate the colour space to mmal format and pass it
over to the VideoCore.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../bcm2835-isp/bcm2835-isp-fmts.h | 180 ++++++++++++------
.../bcm2835-isp/bcm2835-v4l2-isp.c | 68 ++++++-
2 files changed, 192 insertions(+), 56 deletions(-)
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
index 65913a4ff203..a545dbf2b5dd 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
@@ -20,10 +20,29 @@ struct bcm2835_isp_fmt {
int bytesperline_align;
u32 mmal_fmt;
int size_multiplier_x2;
- enum v4l2_colorspace colorspace;
+ u32 colorspace_mask;
+ enum v4l2_colorspace colorspace_default;
unsigned int step_size;
};
+#define V4L2_COLORSPACE_MASK(colorspace) BIT(colorspace)
+
+#define V4L2_COLORSPACE_MASK_JPEG V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_JPEG)
+#define V4L2_COLORSPACE_MASK_SMPTE170M V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_SMPTE170M)
+#define V4L2_COLORSPACE_MASK_REC709 V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_REC709)
+#define V4L2_COLORSPACE_MASK_SRGB V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_SRGB)
+#define V4L2_COLORSPACE_MASK_RAW V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_RAW)
+
+/*
+ * The colour spaces we support for YUV outputs. SRGB features here because,
+ * once you assign the default transfer func and so on, it and JPEG effectively
+ * mean the same.
+ */
+#define V4L2_COLORSPACE_MASK_YUV (V4L2_COLORSPACE_MASK_JPEG | \
+ V4L2_COLORSPACE_MASK_SRGB | \
+ V4L2_COLORSPACE_MASK_SMPTE170M | \
+ V4L2_COLORSPACE_MASK_REC709)
+
static const struct bcm2835_isp_fmt supported_formats[] = {
{
/* YUV formats */
@@ -32,7 +51,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_I420,
.size_multiplier_x2 = 3,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_JPEG,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_YVU420,
@@ -40,7 +60,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YV12,
.size_multiplier_x2 = 3,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_NV12,
@@ -48,7 +69,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV12,
.size_multiplier_x2 = 3,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_NV21,
@@ -56,7 +78,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV21,
.size_multiplier_x2 = 3,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_YUYV,
@@ -64,7 +87,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YUYV,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_UYVY,
@@ -72,7 +96,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_UYVY,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_YVYU,
@@ -80,7 +105,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YVYU,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_VYUY,
@@ -88,7 +114,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_VYUY,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
/* RGB formats */
@@ -97,7 +124,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB24,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
.fourcc = V4L2_PIX_FMT_RGB565,
@@ -105,7 +133,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
.fourcc = V4L2_PIX_FMT_BGR24,
@@ -113,7 +142,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BGR24,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
.fourcc = V4L2_PIX_FMT_XBGR32,
@@ -121,7 +151,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_BGRA,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
.fourcc = V4L2_PIX_FMT_RGBX32,
@@ -129,7 +160,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_RGBA,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
/* Bayer formats */
@@ -139,7 +171,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB8,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR8,
@@ -147,7 +180,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR8,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG8,
@@ -155,7 +189,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG8,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG8,
@@ -163,7 +198,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG8,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 10 bit */
@@ -172,7 +208,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR10P,
@@ -180,7 +217,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG10P,
@@ -188,7 +226,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG10P,
@@ -196,7 +235,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 12 bit */
@@ -205,7 +245,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR12P,
@@ -213,7 +254,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG12P,
@@ -221,7 +263,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG12P,
@@ -229,7 +272,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 14 bit */
@@ -238,7 +282,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR14P,
@@ -246,7 +291,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG14P,
@@ -254,7 +300,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG14P,
@@ -262,7 +309,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 16 bit */
@@ -271,7 +319,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR16,
@@ -279,7 +328,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG16,
@@ -287,7 +337,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG16,
@@ -295,7 +346,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* Bayer formats unpacked to 16bpp */
@@ -305,7 +357,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR10,
@@ -313,7 +366,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG10,
@@ -321,7 +375,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG10,
@@ -329,7 +384,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 12 bit */
@@ -338,7 +394,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR12,
@@ -346,7 +403,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG12,
@@ -354,7 +412,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG12,
@@ -362,7 +421,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 14 bit */
@@ -371,7 +431,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR14,
@@ -379,7 +440,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG14,
@@ -387,7 +449,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG14,
@@ -395,7 +458,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* Monochrome MIPI formats */
@@ -405,7 +469,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_GREY,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 10 bit */
@@ -414,7 +479,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 12 bit */
@@ -423,7 +489,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 14 bit */
@@ -432,7 +499,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 16 bit */
@@ -441,7 +509,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 10 bit as 16bpp */
@@ -450,7 +519,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 12 bit as 16bpp */
@@ -459,7 +529,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 14 bit as 16bpp */
@@ -468,7 +539,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_META_FMT_BCM2835_ISP_STATS,
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
index eb5f076fc6dc..cb7cdba76682 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
@@ -74,6 +74,7 @@ struct bcm2835_isp_q_data {
unsigned int width;
unsigned int height;
unsigned int sizeimage;
+ enum v4l2_colorspace colorspace;
const struct bcm2835_isp_fmt *fmt;
};
@@ -313,6 +314,43 @@ static void mmal_buffer_cb(struct vchiq_mmal_instance *instance,
complete(&dev->frame_cmplt);
}
+struct colorspace_translation {
+ enum v4l2_colorspace v4l2_value;
+ u32 mmal_value;
+};
+
+static u32 translate_color_space(enum v4l2_colorspace color_space)
+{
+ static const struct colorspace_translation translations[] = {
+ { V4L2_COLORSPACE_DEFAULT, MMAL_COLOR_SPACE_UNKNOWN },
+ { V4L2_COLORSPACE_SMPTE170M, MMAL_COLOR_SPACE_ITUR_BT601 },
+ { V4L2_COLORSPACE_SMPTE240M, MMAL_COLOR_SPACE_SMPTE240M },
+ { V4L2_COLORSPACE_REC709, MMAL_COLOR_SPACE_ITUR_BT709 },
+ /* V4L2_COLORSPACE_BT878 unavailable */
+ { V4L2_COLORSPACE_470_SYSTEM_M, MMAL_COLOR_SPACE_BT470_2_M },
+ { V4L2_COLORSPACE_470_SYSTEM_BG, MMAL_COLOR_SPACE_BT470_2_BG },
+ { V4L2_COLORSPACE_JPEG, MMAL_COLOR_SPACE_JPEG_JFIF },
+ /*
+ * We don't have an encoding for SRGB as such, but VideoCore
+ * will do the right thing if it gets "unknown".
+ */
+ { V4L2_COLORSPACE_SRGB, MMAL_COLOR_SPACE_UNKNOWN },
+ /* V4L2_COLORSPACE_OPRGB unavailable */
+ /* V4L2_COLORSPACE_BT2020 unavailable */
+ /* V4L2_COLORSPACE_RAW unavailable */
+ /* V4L2_COLORSPACE_DCI_P3 unavailable */
+ };
+
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(translations); i++) {
+ if (color_space == translations[i].v4l2_value)
+ return translations[i].mmal_value;
+ }
+
+ return MMAL_COLOR_SPACE_UNKNOWN;
+}
+
static void setup_mmal_port_format(struct bcm2835_isp_node *node,
struct vchiq_mmal_port *port)
{
@@ -326,6 +364,7 @@ static void setup_mmal_port_format(struct bcm2835_isp_node *node,
port->es.video.crop.height = q_data->height;
port->es.video.crop.x = 0;
port->es.video.crop.y = 0;
+ port->es.video.color_space = translate_color_space(q_data->colorspace);
};
static int setup_mmal_port(struct bcm2835_isp_node *node)
@@ -834,6 +873,9 @@ static int populate_qdata_fmt(struct v4l2_format *f,
/* All parameters should have been set correctly by try_fmt */
q_data->bytesperline = f->fmt.pix.bytesperline;
q_data->sizeimage = f->fmt.pix.sizeimage;
+
+ /* We must indicate which of the allowed colour spaces we have. */
+ q_data->colorspace = f->fmt.pix.colorspace;
} else {
v4l2_dbg(1, debug, &dev->v4l2_dev,
"%s: Setting meta format for fmt: %08x, size %u\n",
@@ -845,6 +887,9 @@ static int populate_qdata_fmt(struct v4l2_format *f,
q_data->height = 0;
q_data->bytesperline = 0;
q_data->sizeimage = f->fmt.meta.buffersize;
+
+ /* This won't mean anything for metadata, but may as well fill it in. */
+ q_data->colorspace = V4L2_COLORSPACE_DEFAULT;
}
v4l2_dbg(1, debug, &dev->v4l2_dev,
@@ -908,7 +953,7 @@ static int bcm2835_isp_node_g_fmt(struct file *file, void *priv,
f->fmt.pix.pixelformat = q_data->fmt->fourcc;
f->fmt.pix.bytesperline = q_data->bytesperline;
f->fmt.pix.sizeimage = q_data->sizeimage;
- f->fmt.pix.colorspace = q_data->fmt->colorspace;
+ f->fmt.pix.colorspace = q_data->colorspace;
}
return 0;
@@ -975,13 +1020,31 @@ static int bcm2835_isp_node_try_fmt(struct file *file, void *priv,
fmt = get_default_format(node);
if (!node_is_stats(node)) {
+ int is_rgb;
+
f->fmt.pix.width = max(min(f->fmt.pix.width, MAX_DIM),
MIN_DIM);
f->fmt.pix.height = max(min(f->fmt.pix.height, MAX_DIM),
MIN_DIM);
f->fmt.pix.pixelformat = fmt->fourcc;
- f->fmt.pix.colorspace = fmt->colorspace;
+
+ /*
+ * Fill in the actual colour space when the requested one was
+ * not supported. This also catches the case when the "default"
+ * colour space was requested (as that's never in the mask).
+ */
+ if (!(V4L2_COLORSPACE_MASK(f->fmt.pix.colorspace) & fmt->colorspace_mask))
+ f->fmt.pix.colorspace = fmt->colorspace_default;
+ /* In all cases, we only support the defaults for these: */
+ f->fmt.pix.ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(f->fmt.pix.colorspace);
+ f->fmt.pix.xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(f->fmt.pix.colorspace);
+ /* RAW counts as sRGB here so that we get full range. */
+ is_rgb = f->fmt.pix.colorspace == V4L2_COLORSPACE_SRGB ||
+ f->fmt.pix.colorspace == V4L2_COLORSPACE_RAW;
+ f->fmt.pix.quantization = V4L2_MAP_QUANTIZATION_DEFAULT(is_rgb, f->fmt.pix.colorspace,
+ f->fmt.pix.ycbcr_enc);
+
f->fmt.pix.bytesperline = get_bytesperline(f->fmt.pix.width,
fmt);
f->fmt.pix.field = V4L2_FIELD_NONE;
@@ -1307,6 +1370,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
node->q_data.width,
node->q_data.height,
node->q_data.fmt);
+ node->q_data.colorspace = node->q_data.fmt->colorspace_default;
queue->io_modes = VB2_MMAP | VB2_DMABUF;
queue->drv_priv = node;
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 10/14] WIP: vc04_services: bcm2835-isp: Allow formats with different colour spaces
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: David Plowman <david.plowman@raspberrypi.com>
Each supported format now includes a mask showing the allowed colour
spaces, as well as a default colour space for when one was not
specified.
Additionally we translate the colour space to mmal format and pass it
over to the VideoCore.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../bcm2835-isp/bcm2835-isp-fmts.h | 180 ++++++++++++------
.../bcm2835-isp/bcm2835-v4l2-isp.c | 68 ++++++-
2 files changed, 192 insertions(+), 56 deletions(-)
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
index 65913a4ff203..a545dbf2b5dd 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
@@ -20,10 +20,29 @@ struct bcm2835_isp_fmt {
int bytesperline_align;
u32 mmal_fmt;
int size_multiplier_x2;
- enum v4l2_colorspace colorspace;
+ u32 colorspace_mask;
+ enum v4l2_colorspace colorspace_default;
unsigned int step_size;
};
+#define V4L2_COLORSPACE_MASK(colorspace) BIT(colorspace)
+
+#define V4L2_COLORSPACE_MASK_JPEG V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_JPEG)
+#define V4L2_COLORSPACE_MASK_SMPTE170M V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_SMPTE170M)
+#define V4L2_COLORSPACE_MASK_REC709 V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_REC709)
+#define V4L2_COLORSPACE_MASK_SRGB V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_SRGB)
+#define V4L2_COLORSPACE_MASK_RAW V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_RAW)
+
+/*
+ * The colour spaces we support for YUV outputs. SRGB features here because,
+ * once you assign the default transfer func and so on, it and JPEG effectively
+ * mean the same.
+ */
+#define V4L2_COLORSPACE_MASK_YUV (V4L2_COLORSPACE_MASK_JPEG | \
+ V4L2_COLORSPACE_MASK_SRGB | \
+ V4L2_COLORSPACE_MASK_SMPTE170M | \
+ V4L2_COLORSPACE_MASK_REC709)
+
static const struct bcm2835_isp_fmt supported_formats[] = {
{
/* YUV formats */
@@ -32,7 +51,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_I420,
.size_multiplier_x2 = 3,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_JPEG,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_YVU420,
@@ -40,7 +60,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YV12,
.size_multiplier_x2 = 3,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_NV12,
@@ -48,7 +69,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV12,
.size_multiplier_x2 = 3,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_NV21,
@@ -56,7 +78,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV21,
.size_multiplier_x2 = 3,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_YUYV,
@@ -64,7 +87,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YUYV,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_UYVY,
@@ -72,7 +96,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_UYVY,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_YVYU,
@@ -80,7 +105,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YVYU,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_VYUY,
@@ -88,7 +114,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_VYUY,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SMPTE170M,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
/* RGB formats */
@@ -97,7 +124,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB24,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
.fourcc = V4L2_PIX_FMT_RGB565,
@@ -105,7 +133,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
.fourcc = V4L2_PIX_FMT_BGR24,
@@ -113,7 +142,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BGR24,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
.fourcc = V4L2_PIX_FMT_XBGR32,
@@ -121,7 +151,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_BGRA,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
.fourcc = V4L2_PIX_FMT_RGBX32,
@@ -129,7 +160,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_RGBA,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
/* Bayer formats */
@@ -139,7 +171,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB8,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR8,
@@ -147,7 +180,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR8,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG8,
@@ -155,7 +189,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG8,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG8,
@@ -163,7 +198,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG8,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 10 bit */
@@ -172,7 +208,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR10P,
@@ -180,7 +217,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG10P,
@@ -188,7 +226,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG10P,
@@ -196,7 +235,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 12 bit */
@@ -205,7 +245,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR12P,
@@ -213,7 +254,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG12P,
@@ -221,7 +263,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG12P,
@@ -229,7 +272,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 14 bit */
@@ -238,7 +282,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR14P,
@@ -246,7 +291,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG14P,
@@ -254,7 +300,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG14P,
@@ -262,7 +309,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 16 bit */
@@ -271,7 +319,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR16,
@@ -279,7 +328,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG16,
@@ -287,7 +337,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG16,
@@ -295,7 +346,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* Bayer formats unpacked to 16bpp */
@@ -305,7 +357,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR10,
@@ -313,7 +366,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG10,
@@ -321,7 +375,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG10,
@@ -329,7 +384,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 12 bit */
@@ -338,7 +394,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR12,
@@ -346,7 +403,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG12,
@@ -354,7 +412,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG12,
@@ -362,7 +421,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 14 bit */
@@ -371,7 +431,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SBGGR14,
@@ -379,7 +440,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGRBG14,
@@ -387,7 +449,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_PIX_FMT_SGBRG14,
@@ -395,7 +458,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* Monochrome MIPI formats */
@@ -405,7 +469,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_GREY,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 10 bit */
@@ -414,7 +479,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y10P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 12 bit */
@@ -423,7 +489,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y12P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 14 bit */
@@ -432,7 +499,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y14P,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 16 bit */
@@ -441,7 +509,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y16,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 10 bit as 16bpp */
@@ -450,7 +519,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y10,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 12 bit as 16bpp */
@@ -459,7 +529,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y12,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
/* 14 bit as 16bpp */
@@ -468,7 +539,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_Y14,
.size_multiplier_x2 = 2,
- .colorspace = V4L2_COLORSPACE_RAW,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_RAW,
+ .colorspace_default = V4L2_COLORSPACE_RAW,
.step_size = 2,
}, {
.fourcc = V4L2_META_FMT_BCM2835_ISP_STATS,
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
index eb5f076fc6dc..cb7cdba76682 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
@@ -74,6 +74,7 @@ struct bcm2835_isp_q_data {
unsigned int width;
unsigned int height;
unsigned int sizeimage;
+ enum v4l2_colorspace colorspace;
const struct bcm2835_isp_fmt *fmt;
};
@@ -313,6 +314,43 @@ static void mmal_buffer_cb(struct vchiq_mmal_instance *instance,
complete(&dev->frame_cmplt);
}
+struct colorspace_translation {
+ enum v4l2_colorspace v4l2_value;
+ u32 mmal_value;
+};
+
+static u32 translate_color_space(enum v4l2_colorspace color_space)
+{
+ static const struct colorspace_translation translations[] = {
+ { V4L2_COLORSPACE_DEFAULT, MMAL_COLOR_SPACE_UNKNOWN },
+ { V4L2_COLORSPACE_SMPTE170M, MMAL_COLOR_SPACE_ITUR_BT601 },
+ { V4L2_COLORSPACE_SMPTE240M, MMAL_COLOR_SPACE_SMPTE240M },
+ { V4L2_COLORSPACE_REC709, MMAL_COLOR_SPACE_ITUR_BT709 },
+ /* V4L2_COLORSPACE_BT878 unavailable */
+ { V4L2_COLORSPACE_470_SYSTEM_M, MMAL_COLOR_SPACE_BT470_2_M },
+ { V4L2_COLORSPACE_470_SYSTEM_BG, MMAL_COLOR_SPACE_BT470_2_BG },
+ { V4L2_COLORSPACE_JPEG, MMAL_COLOR_SPACE_JPEG_JFIF },
+ /*
+ * We don't have an encoding for SRGB as such, but VideoCore
+ * will do the right thing if it gets "unknown".
+ */
+ { V4L2_COLORSPACE_SRGB, MMAL_COLOR_SPACE_UNKNOWN },
+ /* V4L2_COLORSPACE_OPRGB unavailable */
+ /* V4L2_COLORSPACE_BT2020 unavailable */
+ /* V4L2_COLORSPACE_RAW unavailable */
+ /* V4L2_COLORSPACE_DCI_P3 unavailable */
+ };
+
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(translations); i++) {
+ if (color_space == translations[i].v4l2_value)
+ return translations[i].mmal_value;
+ }
+
+ return MMAL_COLOR_SPACE_UNKNOWN;
+}
+
static void setup_mmal_port_format(struct bcm2835_isp_node *node,
struct vchiq_mmal_port *port)
{
@@ -326,6 +364,7 @@ static void setup_mmal_port_format(struct bcm2835_isp_node *node,
port->es.video.crop.height = q_data->height;
port->es.video.crop.x = 0;
port->es.video.crop.y = 0;
+ port->es.video.color_space = translate_color_space(q_data->colorspace);
};
static int setup_mmal_port(struct bcm2835_isp_node *node)
@@ -834,6 +873,9 @@ static int populate_qdata_fmt(struct v4l2_format *f,
/* All parameters should have been set correctly by try_fmt */
q_data->bytesperline = f->fmt.pix.bytesperline;
q_data->sizeimage = f->fmt.pix.sizeimage;
+
+ /* We must indicate which of the allowed colour spaces we have. */
+ q_data->colorspace = f->fmt.pix.colorspace;
} else {
v4l2_dbg(1, debug, &dev->v4l2_dev,
"%s: Setting meta format for fmt: %08x, size %u\n",
@@ -845,6 +887,9 @@ static int populate_qdata_fmt(struct v4l2_format *f,
q_data->height = 0;
q_data->bytesperline = 0;
q_data->sizeimage = f->fmt.meta.buffersize;
+
+ /* This won't mean anything for metadata, but may as well fill it in. */
+ q_data->colorspace = V4L2_COLORSPACE_DEFAULT;
}
v4l2_dbg(1, debug, &dev->v4l2_dev,
@@ -908,7 +953,7 @@ static int bcm2835_isp_node_g_fmt(struct file *file, void *priv,
f->fmt.pix.pixelformat = q_data->fmt->fourcc;
f->fmt.pix.bytesperline = q_data->bytesperline;
f->fmt.pix.sizeimage = q_data->sizeimage;
- f->fmt.pix.colorspace = q_data->fmt->colorspace;
+ f->fmt.pix.colorspace = q_data->colorspace;
}
return 0;
@@ -975,13 +1020,31 @@ static int bcm2835_isp_node_try_fmt(struct file *file, void *priv,
fmt = get_default_format(node);
if (!node_is_stats(node)) {
+ int is_rgb;
+
f->fmt.pix.width = max(min(f->fmt.pix.width, MAX_DIM),
MIN_DIM);
f->fmt.pix.height = max(min(f->fmt.pix.height, MAX_DIM),
MIN_DIM);
f->fmt.pix.pixelformat = fmt->fourcc;
- f->fmt.pix.colorspace = fmt->colorspace;
+
+ /*
+ * Fill in the actual colour space when the requested one was
+ * not supported. This also catches the case when the "default"
+ * colour space was requested (as that's never in the mask).
+ */
+ if (!(V4L2_COLORSPACE_MASK(f->fmt.pix.colorspace) & fmt->colorspace_mask))
+ f->fmt.pix.colorspace = fmt->colorspace_default;
+ /* In all cases, we only support the defaults for these: */
+ f->fmt.pix.ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(f->fmt.pix.colorspace);
+ f->fmt.pix.xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(f->fmt.pix.colorspace);
+ /* RAW counts as sRGB here so that we get full range. */
+ is_rgb = f->fmt.pix.colorspace == V4L2_COLORSPACE_SRGB ||
+ f->fmt.pix.colorspace == V4L2_COLORSPACE_RAW;
+ f->fmt.pix.quantization = V4L2_MAP_QUANTIZATION_DEFAULT(is_rgb, f->fmt.pix.colorspace,
+ f->fmt.pix.ycbcr_enc);
+
f->fmt.pix.bytesperline = get_bytesperline(f->fmt.pix.width,
fmt);
f->fmt.pix.field = V4L2_FIELD_NONE;
@@ -1307,6 +1370,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
node->q_data.width,
node->q_data.height,
node->q_data.fmt);
+ node->q_data.colorspace = node->q_data.fmt->colorspace_default;
queue->io_modes = VB2_MMAP | VB2_DMABUF;
queue->drv_priv = node;
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 11/14] WIP: vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP outputs
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: David Plowman <david.plowman@raspberrypi.com>
bcm2835-isp outputs actually support all colour spaces that are
fundamentally sRGB underneath, regardless of whether an RGB or YUV output
format is actually requested.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../bcm2835-isp/bcm2835-isp-fmts.h | 45 ++++++++++---------
1 file changed, 25 insertions(+), 20 deletions(-)
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
index a545dbf2b5dd..5ab232ff9bd9 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
@@ -34,14 +34,19 @@ struct bcm2835_isp_fmt {
#define V4L2_COLORSPACE_MASK_RAW V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_RAW)
/*
- * The colour spaces we support for YUV outputs. SRGB features here because,
- * once you assign the default transfer func and so on, it and JPEG effectively
- * mean the same.
+ * All three colour spaces JPEG, SMPTE170M and REC709 are fundamentally sRGB
+ * underneath (as near as makes no difference to us), just with different YCbCr
+ * encodings. Therefore the ISP can generate sRGB on its main output and any of
+ * the others on its low resolution output. Applications should, when using both
+ * outputs, program the colour spaces on them to be the same, matching whatever
+ * is requested for the low resolution output, even if the main output is
+ * producing an RGB format. In turn this requires us to allow all these colour
+ * spaces for every YUV/RGB output format.
*/
-#define V4L2_COLORSPACE_MASK_YUV (V4L2_COLORSPACE_MASK_JPEG | \
- V4L2_COLORSPACE_MASK_SRGB | \
- V4L2_COLORSPACE_MASK_SMPTE170M | \
- V4L2_COLORSPACE_MASK_REC709)
+#define V4L2_COLORSPACE_MASK_ALL_SRGB (V4L2_COLORSPACE_MASK_JPEG | \
+ V4L2_COLORSPACE_MASK_SRGB | \
+ V4L2_COLORSPACE_MASK_SMPTE170M | \
+ V4L2_COLORSPACE_MASK_REC709)
static const struct bcm2835_isp_fmt supported_formats[] = {
{
@@ -51,7 +56,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_I420,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_JPEG,
.step_size = 2,
}, {
@@ -60,7 +65,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YV12,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -69,7 +74,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV12,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -78,7 +83,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV21,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -87,7 +92,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YUYV,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -96,7 +101,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_UYVY,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -105,7 +110,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YVYU,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -114,7 +119,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_VYUY,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -124,7 +129,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB24,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -133,7 +138,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB16,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -142,7 +147,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BGR24,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -151,7 +156,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_BGRA,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -160,7 +165,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_RGBA,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 11/14] WIP: vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP outputs
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: David Plowman <david.plowman@raspberrypi.com>
bcm2835-isp outputs actually support all colour spaces that are
fundamentally sRGB underneath, regardless of whether an RGB or YUV output
format is actually requested.
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../bcm2835-isp/bcm2835-isp-fmts.h | 45 ++++++++++---------
1 file changed, 25 insertions(+), 20 deletions(-)
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
index a545dbf2b5dd..5ab232ff9bd9 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
@@ -34,14 +34,19 @@ struct bcm2835_isp_fmt {
#define V4L2_COLORSPACE_MASK_RAW V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_RAW)
/*
- * The colour spaces we support for YUV outputs. SRGB features here because,
- * once you assign the default transfer func and so on, it and JPEG effectively
- * mean the same.
+ * All three colour spaces JPEG, SMPTE170M and REC709 are fundamentally sRGB
+ * underneath (as near as makes no difference to us), just with different YCbCr
+ * encodings. Therefore the ISP can generate sRGB on its main output and any of
+ * the others on its low resolution output. Applications should, when using both
+ * outputs, program the colour spaces on them to be the same, matching whatever
+ * is requested for the low resolution output, even if the main output is
+ * producing an RGB format. In turn this requires us to allow all these colour
+ * spaces for every YUV/RGB output format.
*/
-#define V4L2_COLORSPACE_MASK_YUV (V4L2_COLORSPACE_MASK_JPEG | \
- V4L2_COLORSPACE_MASK_SRGB | \
- V4L2_COLORSPACE_MASK_SMPTE170M | \
- V4L2_COLORSPACE_MASK_REC709)
+#define V4L2_COLORSPACE_MASK_ALL_SRGB (V4L2_COLORSPACE_MASK_JPEG | \
+ V4L2_COLORSPACE_MASK_SRGB | \
+ V4L2_COLORSPACE_MASK_SMPTE170M | \
+ V4L2_COLORSPACE_MASK_REC709)
static const struct bcm2835_isp_fmt supported_formats[] = {
{
@@ -51,7 +56,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_I420,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_JPEG,
.step_size = 2,
}, {
@@ -60,7 +65,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YV12,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -69,7 +74,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV12,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -78,7 +83,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_NV21,
.size_multiplier_x2 = 3,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -87,7 +92,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YUYV,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -96,7 +101,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_UYVY,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -105,7 +110,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_YVYU,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -114,7 +119,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_VYUY,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_YUV,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SMPTE170M,
.step_size = 2,
}, {
@@ -124,7 +129,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB24,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -133,7 +138,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_RGB16,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -142,7 +147,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 32,
.mmal_fmt = MMAL_ENCODING_BGR24,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -151,7 +156,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_BGRA,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
@@ -160,7 +165,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = {
.bytesperline_align = 64,
.mmal_fmt = MMAL_ENCODING_RGBA,
.size_multiplier_x2 = 2,
- .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB,
+ .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB,
.colorspace_default = V4L2_COLORSPACE_SRGB,
.step_size = 1,
}, {
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
Add a second (identical) set of device nodes to allow concurrent use of
the bcm2835-isp hardware by another user. This change effectively
creates a second state structure (struct bcm2835_isp_dev) to maintain
independent state for the second user. Node and media entity names are
appened with the instance index appropriately.
Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
define.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
1 file changed, 61 insertions(+), 16 deletions(-)
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
index cb7cdba76682..0dbcb25595e7 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
@@ -28,13 +28,19 @@
MODULE_IMPORT_NS(DMA_BUF);
+/*
+ * We want to instantiate 2 independent instances allowing 2 simultaneous users
+ * of the ISP hardware.
+ */
+#define BCM2835_ISP_NUM_INSTANCES 2
+
static unsigned int debug;
module_param(debug, uint, 0644);
MODULE_PARM_DESC(debug, "activates debug info");
-static unsigned int video_nr = 13;
-module_param(video_nr, uint, 0644);
-MODULE_PARM_DESC(video_nr, "base video device number");
+static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
+module_param_array(video_nr, uint, NULL, 0644);
+MODULE_PARM_DESC(video_nr, "base video device numbers");
#define BCM2835_ISP_NAME "bcm2835-isp"
#define BCM2835_ISP_ENTITY_NAME_LEN 32
@@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
* or output nodes.
*/
static int register_node(struct bcm2835_isp_dev *dev,
+ unsigned int instance,
struct bcm2835_isp_node *node,
int index)
{
@@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
node->name, node->id);
- ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
+ ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
if (ret) {
v4l2_err(&dev->v4l2_dev,
"Failed to register video %s[%d] device node\n",
@@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
return ret;
}
-static int bcm2835_isp_remove(struct platform_device *pdev)
+static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
{
- struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
unsigned int i;
media_controller_unregister(dev);
@@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
dev->component);
vchiq_mmal_finalise(dev->mmal_instance);
-
- return 0;
}
-static int bcm2835_isp_probe(struct platform_device *pdev)
+static int bcm2835_isp_probe_instance(struct platform_device *pdev,
+ struct bcm2835_isp_dev **dev_int,
+ unsigned int instance)
{
struct bcm2835_isp_dev *dev;
unsigned int i;
@@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
if (!dev)
return -ENOMEM;
+ *dev_int = dev;
dev->dev = &pdev->dev;
ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
@@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
if (ret) {
v4l2_err(&dev->v4l2_dev,
"%s: failed to create ril.isp component\n", __func__);
- goto error;
+ return ret;
}
if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
@@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
BCM2835_ISP_NUM_OUTPUTS,
dev->component->outputs,
BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
- goto error;
+ return -EINVAL;
}
atomic_set(&dev->num_streaming, 0);
@@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
struct bcm2835_isp_node *node = &dev->node[i];
- ret = register_node(dev, node, i);
+ ret = register_node(dev, instance, node, i);
if (ret)
- goto error;
+ return ret;
}
ret = media_controller_register(dev);
if (ret)
- goto error;
+ return ret;
+
+ return 0;
+}
+
+static int bcm2835_isp_remove(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev **bcm2835_isp_instances;
+ unsigned int i;
+
+ bcm2835_isp_instances = platform_get_drvdata(pdev);
+ for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
+ if (bcm2835_isp_instances[i])
+ bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_probe(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev **bcm2835_isp_instances;
+ unsigned int i;
+ int ret;
+
+ bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
+ sizeof(bcm2835_isp_instances) *
+ BCM2835_ISP_NUM_INSTANCES,
+ GFP_KERNEL);
+ if (!bcm2835_isp_instances)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, bcm2835_isp_instances);
+
+ for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
+ ret = bcm2835_isp_probe_instance(pdev,
+ &bcm2835_isp_instances[i], i);
+ if (ret)
+ goto error;
+ }
- platform_set_drvdata(pdev, dev);
- v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
+ dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
return 0;
error:
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
Add a second (identical) set of device nodes to allow concurrent use of
the bcm2835-isp hardware by another user. This change effectively
creates a second state structure (struct bcm2835_isp_dev) to maintain
independent state for the second user. Node and media entity names are
appened with the instance index appropriately.
Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
define.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
1 file changed, 61 insertions(+), 16 deletions(-)
diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
index cb7cdba76682..0dbcb25595e7 100644
--- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
+++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
@@ -28,13 +28,19 @@
MODULE_IMPORT_NS(DMA_BUF);
+/*
+ * We want to instantiate 2 independent instances allowing 2 simultaneous users
+ * of the ISP hardware.
+ */
+#define BCM2835_ISP_NUM_INSTANCES 2
+
static unsigned int debug;
module_param(debug, uint, 0644);
MODULE_PARM_DESC(debug, "activates debug info");
-static unsigned int video_nr = 13;
-module_param(video_nr, uint, 0644);
-MODULE_PARM_DESC(video_nr, "base video device number");
+static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
+module_param_array(video_nr, uint, NULL, 0644);
+MODULE_PARM_DESC(video_nr, "base video device numbers");
#define BCM2835_ISP_NAME "bcm2835-isp"
#define BCM2835_ISP_ENTITY_NAME_LEN 32
@@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
* or output nodes.
*/
static int register_node(struct bcm2835_isp_dev *dev,
+ unsigned int instance,
struct bcm2835_isp_node *node,
int index)
{
@@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
node->name, node->id);
- ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
+ ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
if (ret) {
v4l2_err(&dev->v4l2_dev,
"Failed to register video %s[%d] device node\n",
@@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
return ret;
}
-static int bcm2835_isp_remove(struct platform_device *pdev)
+static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
{
- struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
unsigned int i;
media_controller_unregister(dev);
@@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
dev->component);
vchiq_mmal_finalise(dev->mmal_instance);
-
- return 0;
}
-static int bcm2835_isp_probe(struct platform_device *pdev)
+static int bcm2835_isp_probe_instance(struct platform_device *pdev,
+ struct bcm2835_isp_dev **dev_int,
+ unsigned int instance)
{
struct bcm2835_isp_dev *dev;
unsigned int i;
@@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
if (!dev)
return -ENOMEM;
+ *dev_int = dev;
dev->dev = &pdev->dev;
ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
@@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
if (ret) {
v4l2_err(&dev->v4l2_dev,
"%s: failed to create ril.isp component\n", __func__);
- goto error;
+ return ret;
}
if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
@@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
BCM2835_ISP_NUM_OUTPUTS,
dev->component->outputs,
BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
- goto error;
+ return -EINVAL;
}
atomic_set(&dev->num_streaming, 0);
@@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
struct bcm2835_isp_node *node = &dev->node[i];
- ret = register_node(dev, node, i);
+ ret = register_node(dev, instance, node, i);
if (ret)
- goto error;
+ return ret;
}
ret = media_controller_register(dev);
if (ret)
- goto error;
+ return ret;
+
+ return 0;
+}
+
+static int bcm2835_isp_remove(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev **bcm2835_isp_instances;
+ unsigned int i;
+
+ bcm2835_isp_instances = platform_get_drvdata(pdev);
+ for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
+ if (bcm2835_isp_instances[i])
+ bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
+ }
+
+ return 0;
+}
+
+static int bcm2835_isp_probe(struct platform_device *pdev)
+{
+ struct bcm2835_isp_dev **bcm2835_isp_instances;
+ unsigned int i;
+ int ret;
+
+ bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
+ sizeof(bcm2835_isp_instances) *
+ BCM2835_ISP_NUM_INSTANCES,
+ GFP_KERNEL);
+ if (!bcm2835_isp_instances)
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, bcm2835_isp_instances);
+
+ for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
+ ret = bcm2835_isp_probe_instance(pdev,
+ &bcm2835_isp_instances[i], i);
+ if (ret)
+ goto error;
+ }
- platform_set_drvdata(pdev, dev);
- v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
+ dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
return 0;
error:
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 13/14] docs: admin-guide: media: bcm2835-isp: Add documentation for bcm2835-isp
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
Document device nodes, frame buffer formats and initial set of
configuraiton that can be set from userspace to configure the pipeline.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../admin-guide/media/bcm2835-isp.rst | 127 ++++++++++++++++++
.../userspace-api/media/drivers/index.rst | 1 +
2 files changed, 128 insertions(+)
create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
diff --git a/Documentation/admin-guide/media/bcm2835-isp.rst b/Documentation/admin-guide/media/bcm2835-isp.rst
new file mode 100644
index 000000000000..e1c19f78435e
--- /dev/null
+++ b/Documentation/admin-guide/media/bcm2835-isp.rst
@@ -0,0 +1,127 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+BCM2835 ISP Driver
+==================
+
+Introduction
+------------
+
+The BCM2835 Image Sensor Pipeline (ISP) is a fixed function hardware pipeline
+for performing image processing operations. Images are fed to the input
+of the ISP through memory frame buffers. These images may be in various YUV,
+RGB, or Bayer formats. A typical use case would have Bayer images obtained from
+an image sensor by the BCM2835 Unicam peripheral, written to a memory
+frame buffer, and finally fed into the input of the ISP. Two concurrent output
+images may be generated in YUV or RGB format at different resolutions.
+Statistics output is also generated for Bayer input images.
+
+The bcm2835-isp driver exposes the following media pads as V4L2 device nodes:
+
+.. tabularcolumns:: |l|l|l|l|
+
+.. cssclass: longtable
+
+.. flat-table::
+
+ * - *Pad*
+ - *Direction*
+ - *Purpose*
+ - *Formats*
+
+ * - "bcm2835-isp0-output0"
+ - sink
+ - Accepts Bayer, RGB or YUV format frame buffers as input to the ISP HW
+ pipeline.
+ - :ref:`RAW8 <V4L2-PIX-FMT-SRGGB8>`,
+ :ref:`RAW10P <V4L2-PIX-FMT-SRGGB10P>`,
+ :ref:`RAW12P <V4L2-PIX-FMT-SRGGB12P>`,
+ :ref:`RAW14P <V4L2-PIX-FMT-SRGGB14P>`,
+ :ref:`RAW16 <V4L2-PIX-FMT-SRGGB16>`,
+ :ref:`RGB24/BGR24 <V4L2-PIX-FMT-RGB24>`,
+ :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
+ :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
+ :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
+ :ref:`VYUY <V4L2-PIX-FMT-VYUY>`,
+ :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`
+
+ * - "bcm2835-isp0-capture1"
+ - source
+ - High resolution YUV or RGB processed output from the ISP.
+ - :ref:`RGB565 <V4L2-PIX-FMT-RGB565>`,
+ :ref:`RGB24/BGR24 <V4L2-PIX-FMT-RGB24>`,
+ :ref:`ABGR32 <V4L2-PIX-FMT-ABGR32>`,
+ :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
+ :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
+ :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
+ :ref:`VYUY <V4L2-PIX-FMT-VYUY>`.
+ :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`,
+ :ref:`NV12/NV21 <V4L2-PIX-FMT-NV12>`,
+
+ * - "bcm2835-isp0-capture2"
+ - source
+ - Low resolution YUV processed output from the ISP. The output of
+ this pad cannot have a resolution larger than the "bcm2835-isp0-capture1" pad in any dimension.
+ - :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
+ :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
+ :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
+ :ref:`VYUY <V4L2-PIX-FMT-VYUY>`.
+ :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`,
+ :ref:`NV12/NV21 <V4L2-PIX-FMT-NV12>`,
+
+ * - "bcm2835-isp0-capture1"
+ - source
+ - Image statistics calculated from the input image provided on the
+ "bcm2835-isp0-output0" pad. Statistics are only available for Bayer
+ format input images.
+ - :ref:`v4l2-meta-fmt-bcm2835-isp-stats`.
+
+Pipeline Configuration
+----------------------
+
+The ISP pipeline can be configure through user-space by calling
+:ref:`VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` on the “bcm2835-isp0-output0”
+node with the appropriate parameters as shown in the table below.
+
+.. tabularcolumns:: |p{2cm}|p{5.0cm}|
+
+.. cssclass: longtable
+
+.. flat-table::
+
+ * - *id*
+ - *Parameter*
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_CC_MATRIX``
+ - struct :c:type:`bcm2835_isp_custom_ccm`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_LENS_SHADING``
+ - struct :c:type:`bcm2835_isp_lens_shading`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL``
+ - struct :c:type:`bcm2835_isp_black_level`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_GEQ``
+ - struct :c:type:`bcm2835_isp_geq`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_GAMMA``
+ - struct :c:type:`bcm2835_isp_gamma`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_DENOISE``
+ - struct :c:type:`bcm2835_isp_denoise`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_SHARPEN``
+ - struct :c:type:`bcm2835_isp_sharpen`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_DPC``
+ - struct :c:type:`bcm2835_isp_dpc`
+
+++++++++++++++++++++++++
+Configuration Parameters
+++++++++++++++++++++++++
+
+.. kernel-doc:: include/uapi/linux/bcm2835-isp.h
+ :functions: bcm2835_isp_rational bcm2835_isp_ccm bcm2835_isp_custom_ccm
+ bcm2835_isp_gain_format bcm2835_isp_lens_shading
+ bcm2835_isp_black_level bcm2835_isp_geq bcm2835_isp_gamma
+ bcm2835_isp_denoise bcm2835_isp_sharpen
+ bcm2835_isp_dpc_mode bcm2835_isp_dpc
diff --git a/Documentation/userspace-api/media/drivers/index.rst b/Documentation/userspace-api/media/drivers/index.rst
index 32f82aed47d9..34e0d7102ef0 100644
--- a/Documentation/userspace-api/media/drivers/index.rst
+++ b/Documentation/userspace-api/media/drivers/index.rst
@@ -31,6 +31,7 @@ For more details see the file COPYING in the source distribution of Linux.
:maxdepth: 5
:numbered:
+ bcm2835-isp
ccs
cx2341x-uapi
dw100
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 13/14] docs: admin-guide: media: bcm2835-isp: Add documentation for bcm2835-isp
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
Document device nodes, frame buffer formats and initial set of
configuraiton that can be set from userspace to configure the pipeline.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
.../admin-guide/media/bcm2835-isp.rst | 127 ++++++++++++++++++
.../userspace-api/media/drivers/index.rst | 1 +
2 files changed, 128 insertions(+)
create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
diff --git a/Documentation/admin-guide/media/bcm2835-isp.rst b/Documentation/admin-guide/media/bcm2835-isp.rst
new file mode 100644
index 000000000000..e1c19f78435e
--- /dev/null
+++ b/Documentation/admin-guide/media/bcm2835-isp.rst
@@ -0,0 +1,127 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+BCM2835 ISP Driver
+==================
+
+Introduction
+------------
+
+The BCM2835 Image Sensor Pipeline (ISP) is a fixed function hardware pipeline
+for performing image processing operations. Images are fed to the input
+of the ISP through memory frame buffers. These images may be in various YUV,
+RGB, or Bayer formats. A typical use case would have Bayer images obtained from
+an image sensor by the BCM2835 Unicam peripheral, written to a memory
+frame buffer, and finally fed into the input of the ISP. Two concurrent output
+images may be generated in YUV or RGB format at different resolutions.
+Statistics output is also generated for Bayer input images.
+
+The bcm2835-isp driver exposes the following media pads as V4L2 device nodes:
+
+.. tabularcolumns:: |l|l|l|l|
+
+.. cssclass: longtable
+
+.. flat-table::
+
+ * - *Pad*
+ - *Direction*
+ - *Purpose*
+ - *Formats*
+
+ * - "bcm2835-isp0-output0"
+ - sink
+ - Accepts Bayer, RGB or YUV format frame buffers as input to the ISP HW
+ pipeline.
+ - :ref:`RAW8 <V4L2-PIX-FMT-SRGGB8>`,
+ :ref:`RAW10P <V4L2-PIX-FMT-SRGGB10P>`,
+ :ref:`RAW12P <V4L2-PIX-FMT-SRGGB12P>`,
+ :ref:`RAW14P <V4L2-PIX-FMT-SRGGB14P>`,
+ :ref:`RAW16 <V4L2-PIX-FMT-SRGGB16>`,
+ :ref:`RGB24/BGR24 <V4L2-PIX-FMT-RGB24>`,
+ :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
+ :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
+ :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
+ :ref:`VYUY <V4L2-PIX-FMT-VYUY>`,
+ :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`
+
+ * - "bcm2835-isp0-capture1"
+ - source
+ - High resolution YUV or RGB processed output from the ISP.
+ - :ref:`RGB565 <V4L2-PIX-FMT-RGB565>`,
+ :ref:`RGB24/BGR24 <V4L2-PIX-FMT-RGB24>`,
+ :ref:`ABGR32 <V4L2-PIX-FMT-ABGR32>`,
+ :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
+ :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
+ :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
+ :ref:`VYUY <V4L2-PIX-FMT-VYUY>`.
+ :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`,
+ :ref:`NV12/NV21 <V4L2-PIX-FMT-NV12>`,
+
+ * - "bcm2835-isp0-capture2"
+ - source
+ - Low resolution YUV processed output from the ISP. The output of
+ this pad cannot have a resolution larger than the "bcm2835-isp0-capture1" pad in any dimension.
+ - :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
+ :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
+ :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
+ :ref:`VYUY <V4L2-PIX-FMT-VYUY>`.
+ :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`,
+ :ref:`NV12/NV21 <V4L2-PIX-FMT-NV12>`,
+
+ * - "bcm2835-isp0-capture1"
+ - source
+ - Image statistics calculated from the input image provided on the
+ "bcm2835-isp0-output0" pad. Statistics are only available for Bayer
+ format input images.
+ - :ref:`v4l2-meta-fmt-bcm2835-isp-stats`.
+
+Pipeline Configuration
+----------------------
+
+The ISP pipeline can be configure through user-space by calling
+:ref:`VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` on the “bcm2835-isp0-output0”
+node with the appropriate parameters as shown in the table below.
+
+.. tabularcolumns:: |p{2cm}|p{5.0cm}|
+
+.. cssclass: longtable
+
+.. flat-table::
+
+ * - *id*
+ - *Parameter*
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_CC_MATRIX``
+ - struct :c:type:`bcm2835_isp_custom_ccm`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_LENS_SHADING``
+ - struct :c:type:`bcm2835_isp_lens_shading`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL``
+ - struct :c:type:`bcm2835_isp_black_level`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_GEQ``
+ - struct :c:type:`bcm2835_isp_geq`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_GAMMA``
+ - struct :c:type:`bcm2835_isp_gamma`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_DENOISE``
+ - struct :c:type:`bcm2835_isp_denoise`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_SHARPEN``
+ - struct :c:type:`bcm2835_isp_sharpen`
+
+ * - ``V4L2_CID_USER_BCM2835_ISP_DPC``
+ - struct :c:type:`bcm2835_isp_dpc`
+
+++++++++++++++++++++++++
+Configuration Parameters
+++++++++++++++++++++++++
+
+.. kernel-doc:: include/uapi/linux/bcm2835-isp.h
+ :functions: bcm2835_isp_rational bcm2835_isp_ccm bcm2835_isp_custom_ccm
+ bcm2835_isp_gain_format bcm2835_isp_lens_shading
+ bcm2835_isp_black_level bcm2835_isp_geq bcm2835_isp_gamma
+ bcm2835_isp_denoise bcm2835_isp_sharpen
+ bcm2835_isp_dpc_mode bcm2835_isp_dpc
diff --git a/Documentation/userspace-api/media/drivers/index.rst b/Documentation/userspace-api/media/drivers/index.rst
index 32f82aed47d9..34e0d7102ef0 100644
--- a/Documentation/userspace-api/media/drivers/index.rst
+++ b/Documentation/userspace-api/media/drivers/index.rst
@@ -31,6 +31,7 @@ For more details see the file COPYING in the source distribution of Linux.
:maxdepth: 5
:numbered:
+ bcm2835-isp
ccs
cx2341x-uapi
dw100
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 14/14] staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 21:47 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
bcmn2835_isp is a platform driver dependent on vchiq,
therefore add the load/unload functions for it to vchiq.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
index 642fdbc0d654..2d070f80a170 100644
--- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
+++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
@@ -67,6 +67,7 @@ struct vchiq_state g_state;
static struct platform_device *bcm2835_camera;
static struct platform_device *bcm2835_audio;
+static struct platform_device *bcm2835_isp;
static struct platform_device *vcsm_cma;
struct vchiq_drvdata {
@@ -1836,6 +1837,7 @@ static int vchiq_probe(struct platform_device *pdev)
vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
+ bcm2835_isp = vchiq_register_child(pdev, "bcm2835-isp");
return 0;
@@ -1847,6 +1849,7 @@ static int vchiq_probe(struct platform_device *pdev)
static int vchiq_remove(struct platform_device *pdev)
{
+ platform_device_unregister(bcm2835_isp);
platform_device_unregister(bcm2835_audio);
platform_device_unregister(bcm2835_camera);
platform_device_unregister(vcsm_cma);
--
2.37.3
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [PATCH 14/14] staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
@ 2022-11-21 21:47 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-21 21:47 UTC (permalink / raw)
To: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart, Umang Jain
From: Naushir Patuck <naush@raspberrypi.com>
bcmn2835_isp is a platform driver dependent on vchiq,
therefore add the load/unload functions for it to vchiq.
Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
---
drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
index 642fdbc0d654..2d070f80a170 100644
--- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
+++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
@@ -67,6 +67,7 @@ struct vchiq_state g_state;
static struct platform_device *bcm2835_camera;
static struct platform_device *bcm2835_audio;
+static struct platform_device *bcm2835_isp;
static struct platform_device *vcsm_cma;
struct vchiq_drvdata {
@@ -1836,6 +1837,7 @@ static int vchiq_probe(struct platform_device *pdev)
vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
+ bcm2835_isp = vchiq_register_child(pdev, "bcm2835-isp");
return 0;
@@ -1847,6 +1849,7 @@ static int vchiq_probe(struct platform_device *pdev)
static int vchiq_remove(struct platform_device *pdev)
{
+ platform_device_unregister(bcm2835_isp);
platform_device_unregister(bcm2835_audio);
platform_device_unregister(bcm2835_camera);
platform_device_unregister(vcsm_cma);
--
2.37.3
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 22:16 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 22:16 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang,
Nice to see this series on the list !
On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> SoCs (namely BCM2711). Unicam is still under active development to work
> with multistream support to get into mainline. Hence only the ISP driver
> will remain the primary area of this series.
>
> Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
> with the VC4 VPU.
>
> Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
> dma_buf instead of a file-descriptor.
>
> Patch (04-05)/14 adds a couple of improvements/support for
> bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
>
> Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
> and format defintions.
>
> Patch (10-11)/14 deals with the colorspace support.
> Note: This is still WIP since the implementation of colorspace is still
> getting ironed out (especially around setting of colorspace flags handling).
>
> Patch 12/14 allows multiple instances of the ISP.
>
> Patch 13/14 adds a admin-guide document on bcm2835-isp.
>
> Patch 14/14 wires all this up with the vchiq-mmal driver.
>
> Testing:
> Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
> end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
> bindings from [1]). Once done, functional testing was conducted with
> libcamera[3] and its utility tools.
>
> Also note: Reviews given on [2] for the relevant ISP driver patches have
> been incorporated in this version.
>
> Known issues:
> - Colorspace handling
This will require further discussions, I'll try to comment on this topic
in the review of the ISP driver patch.
> - vc-sm-cma spamming kernel log with
> - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
Do you have any plan to address this ? Is the root cause known ?
> [1]: https://github.com/raspberrypi/linux
> [2]: https://lore.kernel.org/linux-media/20200504092611.9798-1-laurent.pinchart@ideasonboard.com/
> [3]: https://libcamera.org/getting-started.html
>
> Dave Stevenson (7):
> staging: vc04_services: Add new vc-sm-cma driver
> staging: vchiq_arm: Register vcsm-cma as a platform driver
> media: videobuf2: Allow exporting of a struct dmabuf
> staging: mmal-vchiq: Add support for event callbacks
> staging: mmal-vchiq: Use vc-sm-cma to support zero copy
> staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
> uapi: bcm2835-isp: Add bcm2835-isp uapi header file
>
> David Plowman (2):
> vc04_services: bcm2835-isp: Allow formats with different colour spaces
> vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
> outputs
>
> Naushir Patuck (5):
> media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
> staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
> component
> staging: vc04_services: bcm2835_isp: Allow multiple users
> docs: admin-guide: media: bcm2835-isp: Add documentation for
> bcm2835-isp
> staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
>
> .../admin-guide/media/bcm2835-isp.rst | 127 ++
> .../userspace-api/media/drivers/index.rst | 1 +
> .../userspace-api/media/v4l/meta-formats.rst | 1 +
> .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
> MAINTAINERS | 9 +
> .../media/common/videobuf2/videobuf2-core.c | 36 +-
> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> drivers/staging/vc04_services/Kconfig | 4 +
> drivers/staging/vc04_services/Makefile | 2 +
> .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
> .../vc04_services/bcm2835-isp/Makefile | 8 +
> .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
> .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
> .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
> .../interface/vchiq_arm/vchiq_arm.c | 6 +
> .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
> .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
> .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
> .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
> .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
> .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
> .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
> .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
> .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
> .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
> .../vchiq-mmal/mmal-parameters.h | 165 +-
> .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
> .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
> include/media/videobuf2-core.h | 15 +
> include/uapi/linux/bcm2835-isp.h | 347 ++++
> include/uapi/linux/v4l2-controls.h | 6 +
> include/uapi/linux/videodev2.h | 1 +
> 34 files changed, 5225 insertions(+), 33 deletions(-)
> create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> create mode 100644 include/uapi/linux/bcm2835-isp.h
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-21 22:16 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 22:16 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang,
Nice to see this series on the list !
On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> SoCs (namely BCM2711). Unicam is still under active development to work
> with multistream support to get into mainline. Hence only the ISP driver
> will remain the primary area of this series.
>
> Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
> with the VC4 VPU.
>
> Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
> dma_buf instead of a file-descriptor.
>
> Patch (04-05)/14 adds a couple of improvements/support for
> bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
>
> Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
> and format defintions.
>
> Patch (10-11)/14 deals with the colorspace support.
> Note: This is still WIP since the implementation of colorspace is still
> getting ironed out (especially around setting of colorspace flags handling).
>
> Patch 12/14 allows multiple instances of the ISP.
>
> Patch 13/14 adds a admin-guide document on bcm2835-isp.
>
> Patch 14/14 wires all this up with the vchiq-mmal driver.
>
> Testing:
> Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
> end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
> bindings from [1]). Once done, functional testing was conducted with
> libcamera[3] and its utility tools.
>
> Also note: Reviews given on [2] for the relevant ISP driver patches have
> been incorporated in this version.
>
> Known issues:
> - Colorspace handling
This will require further discussions, I'll try to comment on this topic
in the review of the ISP driver patch.
> - vc-sm-cma spamming kernel log with
> - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
Do you have any plan to address this ? Is the root cause known ?
> [1]: https://github.com/raspberrypi/linux
> [2]: https://lore.kernel.org/linux-media/20200504092611.9798-1-laurent.pinchart@ideasonboard.com/
> [3]: https://libcamera.org/getting-started.html
>
> Dave Stevenson (7):
> staging: vc04_services: Add new vc-sm-cma driver
> staging: vchiq_arm: Register vcsm-cma as a platform driver
> media: videobuf2: Allow exporting of a struct dmabuf
> staging: mmal-vchiq: Add support for event callbacks
> staging: mmal-vchiq: Use vc-sm-cma to support zero copy
> staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
> uapi: bcm2835-isp: Add bcm2835-isp uapi header file
>
> David Plowman (2):
> vc04_services: bcm2835-isp: Allow formats with different colour spaces
> vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
> outputs
>
> Naushir Patuck (5):
> media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
> staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
> component
> staging: vc04_services: bcm2835_isp: Allow multiple users
> docs: admin-guide: media: bcm2835-isp: Add documentation for
> bcm2835-isp
> staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
>
> .../admin-guide/media/bcm2835-isp.rst | 127 ++
> .../userspace-api/media/drivers/index.rst | 1 +
> .../userspace-api/media/v4l/meta-formats.rst | 1 +
> .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
> MAINTAINERS | 9 +
> .../media/common/videobuf2/videobuf2-core.c | 36 +-
> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> drivers/staging/vc04_services/Kconfig | 4 +
> drivers/staging/vc04_services/Makefile | 2 +
> .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
> .../vc04_services/bcm2835-isp/Makefile | 8 +
> .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
> .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
> .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
> .../interface/vchiq_arm/vchiq_arm.c | 6 +
> .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
> .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
> .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
> .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
> .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
> .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
> .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
> .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
> .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
> .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
> .../vchiq-mmal/mmal-parameters.h | 165 +-
> .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
> .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
> include/media/videobuf2-core.h | 15 +
> include/uapi/linux/bcm2835-isp.h | 347 ++++
> include/uapi/linux/v4l2-controls.h | 6 +
> include/uapi/linux/videodev2.h | 1 +
> 34 files changed, 5225 insertions(+), 33 deletions(-)
> create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> create mode 100644 include/uapi/linux/bcm2835-isp.h
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 01/14] staging: vc04_services: Add new vc-sm-cma driver
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:04 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:04 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang (and Dave),
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:09AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> Add Broadcom VideoCore Shared Memory support.
>
> This new driver allows contiguous memory blocks to be imported
> into the VideoCore VPU memory map, and manages the lifetime of
> those objects, only releasing the source dmabuf once the VPU has
> confirmed it has finished with it.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> drivers/staging/vc04_services/Kconfig | 2 +
> drivers/staging/vc04_services/Makefile | 1 +
> .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
> .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
> .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++++++++++++
> .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 ++
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++++++++
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 ++
> .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++++
> .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
> 10 files changed, 1665 insertions(+)
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
>
> diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig
> index 31e58c9d1a11..6c0e77d64376 100644
> --- a/drivers/staging/vc04_services/Kconfig
> +++ b/drivers/staging/vc04_services/Kconfig
> @@ -46,5 +46,7 @@ source "drivers/staging/vc04_services/bcm2835-camera/Kconfig"
>
> source "drivers/staging/vc04_services/vchiq-mmal/Kconfig"
>
> +source "drivers/staging/vc04_services/vc-sm-cma/Kconfig"
> +
> endif
>
> diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile
> index 1fd191e2e2a5..01089f369fb4 100644
> --- a/drivers/staging/vc04_services/Makefile
> +++ b/drivers/staging/vc04_services/Makefile
> @@ -14,6 +14,7 @@ endif
> obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/
> obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/
> obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/
> +obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/
>
> ccflags-y += -I $(srctree)/$(src)/include
>
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/Kconfig b/drivers/staging/vc04_services/vc-sm-cma/Kconfig
> new file mode 100644
> index 000000000000..bbd296f5826b
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/Kconfig
> @@ -0,0 +1,10 @@
> +config BCM_VC_SM_CMA
> + tristate "VideoCore Shared Memory (CMA) driver"
> + depends on BCM2835_VCHIQ
> + select RBTREE
I don't see an RBTREE config option in mainline.
> + select DMA_SHARED_BUFFER
> + help
> + Say Y here to enable the shared memory interface that
> + supports sharing dmabufs with VideoCore.
> + This operates over the VCHIQ interface to a service
> + running on VideoCore.
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/Makefile b/drivers/staging/vc04_services/vc-sm-cma/Makefile
> new file mode 100644
> index 000000000000..c92a5775c62e
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/Makefile
> @@ -0,0 +1,12 @@
> +ccflags-y += \
> + -I$(srctree)/$(src)/../ \
> + -I$(srctree)/$(src)/../interface/vchiq_arm\
Missing space before \
> + -I$(srctree)/$(src)/../include
> +
> +ccflags-y += \
> + -D__VCCOREVER__=0
Is this needed ?
> +
> +vc-sm-cma-$(CONFIG_BCM_VC_SM_CMA) := \
> + vc_sm.o vc_sm_cma_vchi.o
> +
> +obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma.o
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> new file mode 100644
> index 000000000000..7fe81d259c7b
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> @@ -0,0 +1,801 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * VideoCore Shared Memory driver using CMA.
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + * Dave Stevenson <dave.stevenson@raspberrypi.com>
> + *
> + * Based on vmcs_sm driver from Broadcom Corporation for some API,
> + * and taking some code for buffer allocation and dmabuf handling from
> + * videobuf2.
> + *
> + * This driver handles taking a dmabuf, mapping it into the VPU address space,
> + * and returning a handle that the VPU can use.
> + */
> +
> +/* ---- Include Files ----------------------------------------------------- */
I'd drop this comment (same below), it feels a bit alien.
> +#include <linux/debugfs.h>
> +#include <linux/dma-buf.h>
> +#include <linux/errno.h>
> +#include <linux/miscdevice.h>
> +#include <linux/platform_device.h>
> +#include <linux/syscalls.h>
> +
> +#include "vchiq_connected.h"
> +#include "vc_sm_cma_vchi.h"
> +
> +#include "vc_sm.h"
> +#include "vc_sm_knl.h"
> +
> +MODULE_IMPORT_NS(DMA_BUF);
> +
> +/* ---- Private Constants and Types --------------------------------------- */
> +
> +#define VC_SM_RESOURCE_NAME_DEFAULT "sm-host-resource"
> +
> +#define VC_SM_DIR_ROOT_NAME "vcsm-cma"
> +#define VC_SM_STATE "state"
> +
> +/* Private file data associated with each opened device. */
> +struct vc_sm_privdata_t {
> +};
> +
> +typedef int (*VC_SM_SHOW) (struct seq_file *s, void *v);
> +struct sm_pde_t {
Should this structure (and the other ones below) have a vc_ prefix for
consistency ? Also, drop the _t suffix.
> + VC_SM_SHOW show; /* Debug fs function hookup. */
> + struct dentry *dir_entry; /* Debug fs directory entry. */
> + void *priv_data; /* Private data */
Never used.
> +};
> +
> +/* Global state information. */
> +struct sm_state_t {
> + struct platform_device *pdev;
> +
> + struct miscdevice misc_dev;
> +
> + struct sm_instance *sm_handle; /* Handle for videocore service. */
> +
> + spinlock_t kernelid_map_lock; /* Spinlock protecting kernelid_map */
> + struct idr kernelid_map;
> +
> + struct mutex map_lock; /* Global map lock. */
> + struct list_head buffer_list; /* List of buffer. */
> +
> + struct dentry *dir_root; /* Debug fs entries root. */
> + struct sm_pde_t dir_state; /* Debug fs entries state sub-tree. */
> +
> + bool require_released_callback; /* VPU will send a released msg when it
> + * has finished with a resource.
> + */
> + /* State for transactions */
> + int restart_sys; /* Tracks restart on interrupt. */
> + enum vc_sm_msg_type int_action; /* Interrupted action. */
Both of these are set but never used. Either there's dead code below, or
something is wrong.
> +
> + u32 int_trans_id; /* Interrupted transaction. */
> + struct vchiq_instance *vchiq_instance;
> +};
> +
> +struct vc_sm_dma_buf_attachment {
Never used.
> + struct device *dev;
> + struct sg_table sg_table;
> + struct list_head list;
> + enum dma_data_direction dma_dir;
> +};
> +
> +/* ---- Private Variables ----------------------------------------------- */
> +
> +static struct sm_state_t *sm_state;
> +static int sm_inited;
Can we avoid global variables please ?
> +
> +/* ---- Private Function Prototypes -------------------------------------- */
> +
> +/* ---- Private Functions ------------------------------------------------ */
If you want to split the driver in sections (which I personally like for
large files), I would group functions by purpose and add a section
header for each group.
> +
> +static int get_kernel_id(struct vc_sm_buffer *buffer)
> +{
> + int handle;
> +
> + spin_lock(&sm_state->kernelid_map_lock);
> + handle = idr_alloc(&sm_state->kernelid_map, buffer, 0, 0, GFP_KERNEL);
> + spin_unlock(&sm_state->kernelid_map_lock);
> +
> + return handle;
> +}
> +
> +static struct vc_sm_buffer *lookup_kernel_id(int handle)
> +{
> + return idr_find(&sm_state->kernelid_map, handle);
> +}
> +
> +static void free_kernel_id(int handle)
> +{
> + spin_lock(&sm_state->kernelid_map_lock);
> + idr_remove(&sm_state->kernelid_map, handle);
> + spin_unlock(&sm_state->kernelid_map_lock);
> +}
For instance those three functions could be in the IDR section. I would
probably inline them in their respective caller though, but if you wait
to keep them, you should give them better names. vc_sm_cma_idr_ sounds
like a good prefix, the functions could be vc_sm_cma_idr_alloc(),
vc_sm_cma_idr_find() and vc_sm_cma_idr_remove(). I would then move the
IDR init and cleanup code to two separate functions too.
> +
> +static int vc_sm_cma_seq_file_show(struct seq_file *s, void *v)
> +{
> + struct sm_pde_t *sm_pde;
> +
> + sm_pde = (struct sm_pde_t *)(s->private);
> +
> + if (sm_pde && sm_pde->show)
> + sm_pde->show(s, v);
The only show handler is vc_sm_cma_global_state_show(), you can simplify
all this by dropping this function and passing
vc_sm_cma_global_state_show to single_open(). The show field in sm_pde_t
can also be dropped. You can actually drop the sm_pde_t structure and
embed the dir_entry field directly in sm_state_t.
> +
> + return 0;
> +}
> +
> +static int vc_sm_cma_single_open(struct inode *inode, struct file *file)
> +{
> + return single_open(file, vc_sm_cma_seq_file_show, inode->i_private);
> +}
> +
> +static const struct file_operations vc_sm_cma_debug_fs_fops = {
> + .open = vc_sm_cma_single_open,
> + .read = seq_read,
> + .llseek = seq_lseek,
> + .release = single_release,
> +};
> +
> +static int vc_sm_cma_global_state_show(struct seq_file *s, void *v)
> +{
> + struct vc_sm_buffer *resource = NULL;
> + int resource_count = 0;
> +
> + if (!sm_state)
> + return 0;
> +
> + seq_printf(s, "\nVC-ServiceHandle %p\n", sm_state->sm_handle);
> +
> + /* Log all applicable mapping(s). */
> +
> + mutex_lock(&sm_state->map_lock);
> + seq_puts(s, "\nResources\n");
> + if (!list_empty(&sm_state->buffer_list)) {
> + list_for_each_entry(resource, &sm_state->buffer_list,
> + global_buffer_list) {
> + resource_count++;
> +
> + seq_printf(s, "\nResource %p\n",
> + resource);
> + seq_printf(s, " NAME %s\n",
> + resource->name);
> + seq_printf(s, " SIZE %zu\n",
> + resource->size);
> + seq_printf(s, " DMABUF %p\n",
> + resource->dma_buf);
> + seq_printf(s, " IMPORTED_DMABUF %p\n",
> + resource->imported_dma_buf);
> + seq_printf(s, " ATTACH %p\n",
> + resource->attach);
> + seq_printf(s, " SGT %p\n",
> + resource->sgt);
> + seq_printf(s, " DMA_ADDR %pad\n",
> + &resource->dma_addr);
> + seq_printf(s, " VC_HANDLE %08x\n",
> + resource->vc_handle);
> + seq_printf(s, " VC_MAPPING %d\n",
> + resource->vpu_state);
> + }
> + }
> + seq_printf(s, "\n\nTotal resource count: %d\n\n", resource_count);
> +
> + mutex_unlock(&sm_state->map_lock);
> +
> + return 0;
> +}
> +
> +/*
> + * Adds a buffer to the private data list which tracks all the allocated
> + * data.
> + */
> +static void vc_sm_add_resource(struct vc_sm_buffer *buffer)
> +{
> + mutex_lock(&sm_state->map_lock);
> + list_add(&buffer->global_buffer_list, &sm_state->buffer_list);
> + mutex_unlock(&sm_state->map_lock);
> +
> + pr_debug("[%s]: added buffer %p (name %s, size %zu)\n",
> + __func__, buffer, buffer->name, buffer->size);
Let's use dev_* instead of pr_* where possible.
> +}
> +
> +/*
> + * Cleans up imported dmabuf.
> + * Should be called with mutex held.
> + */
> +static void vc_sm_clean_up_dmabuf(struct vc_sm_buffer *buffer)
> +{
> + /* Handle cleaning up imported dmabufs */
> + if (buffer->sgt) {
> + dma_buf_unmap_attachment(buffer->attach,
> + buffer->sgt,
> + DMA_BIDIRECTIONAL);
> + buffer->sgt = NULL;
> + }
> + if (buffer->attach) {
> + dma_buf_detach(buffer->dma_buf, buffer->attach);
> + buffer->attach = NULL;
> + }
> +}
> +
> +/*
> + * Instructs VPU to decrement the refcount on a buffer.
> + */
> +static void vc_sm_vpu_free(struct vc_sm_buffer *buffer)
> +{
> + if (buffer->vc_handle && buffer->vpu_state == VPU_MAPPED) {
> + struct vc_sm_free_t free = { buffer->vc_handle, 0 };
> + int status = vc_sm_cma_vchi_free(sm_state->sm_handle, &free,
> + &sm_state->int_trans_id);
> + if (status != 0 && status != -EINTR) {
> + pr_err("[%s]: failed to free memory on videocore (status: %u, trans_id: %u)\n",
> + __func__, status, sm_state->int_trans_id);
> + }
> +
> + if (sm_state->require_released_callback) {
> + /* Need to wait for the VPU to confirm the free. */
> +
> + /* Retain a reference on this until the VPU has
> + * released it
> + */
> + buffer->vpu_state = VPU_UNMAPPING;
> + } else {
> + buffer->vpu_state = VPU_NOT_MAPPED;
> + buffer->vc_handle = 0;
> + }
> + }
> +}
> +
> +/*
> + * Release an allocation.
> + * All refcounting is done via the dma buf object.
> + *
> + * Must be called with the mutex held. The function will either release the
> + * mutex (if defering the release) or destroy it. The caller must therefore not
> + * reuse the buffer on return.
> + */
> +static void vc_sm_release_resource(struct vc_sm_buffer *buffer)
> +{
> + pr_debug("[%s]: buffer %p (name %s, size %zu)\n",
> + __func__, buffer, buffer->name, buffer->size);
> +
> + if (buffer->vc_handle) {
> + /* We've sent the unmap request but not had the response. */
> + pr_debug("[%s]: Waiting for VPU unmap response on %p\n",
> + __func__, buffer);
> + goto defer;
> + }
> + if (buffer->in_use) {
> + /* dmabuf still in use - we await the release */
> + pr_debug("[%s]: buffer %p is still in use\n", __func__, buffer);
> + goto defer;
> + }
> +
> + /* Release the allocation */
> + if (buffer->imported_dma_buf)
> + dma_buf_put(buffer->imported_dma_buf);
> + else
> + pr_err("%s: Imported dmabuf already been put for buf %p\n",
> + __func__, buffer);
> + buffer->imported_dma_buf = NULL;
> +
> + /* Free our buffer. Start by removing it from the list */
> + mutex_lock(&sm_state->map_lock);
> + list_del(&buffer->global_buffer_list);
> + mutex_unlock(&sm_state->map_lock);
> +
> + pr_debug("%s: Release our allocation - done\n", __func__);
> + mutex_unlock(&buffer->lock);
> +
> + mutex_destroy(&buffer->lock);
> +
> + kfree(buffer);
> + return;
> +
> +defer:
> + mutex_unlock(&buffer->lock);
> +}
> +
> +/* Dma_buf operations for chaining through to an imported dma_buf */
> +
> +static void vc_sm_dma_buf_release(struct dma_buf *dmabuf)
> +{
> + struct vc_sm_buffer *buffer;
> +
> + if (!dmabuf)
> + return;
> +
> + buffer = (struct vc_sm_buffer *)dmabuf->priv;
> +
> + mutex_lock(&buffer->lock);
> +
> + pr_debug("%s dmabuf %p, buffer %p\n", __func__, dmabuf, buffer);
> +
> + buffer->in_use = 0;
> +
> + /* Unmap on the VPU */
> + vc_sm_vpu_free(buffer);
> + pr_debug("%s vpu_free done\n", __func__);
> +
> + /* Unmap our dma_buf object (the vc_sm_buffer remains until released
> + * on the VPU).
> + */
> + vc_sm_clean_up_dmabuf(buffer);
> + pr_debug("%s clean_up dmabuf done\n", __func__);
> +
> + /* buffer->lock will be destroyed by vc_sm_release_resource if finished
> + * with, otherwise unlocked. Do NOT unlock here.
> + */
> + vc_sm_release_resource(buffer);
> + pr_debug("%s done\n", __func__);
> +}
> +
> +static
> +int vc_sm_import_dma_buf_attach(struct dma_buf *dmabuf,
> + struct dma_buf_attachment *attachment)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + return buf->imported_dma_buf->ops->attach(buf->imported_dma_buf,
> + attachment);
> +}
> +
> +static
> +void vc_sm_import_dma_buf_detatch(struct dma_buf *dmabuf,
> + struct dma_buf_attachment *attachment)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + buf->imported_dma_buf->ops->detach(buf->imported_dma_buf, attachment);
> +}
> +
> +static
> +struct sg_table *vc_sm_import_map_dma_buf(struct dma_buf_attachment *attachment,
> + enum dma_data_direction direction)
> +{
> + struct vc_sm_buffer *buf = attachment->dmabuf->priv;
> +
> + return buf->imported_dma_buf->ops->map_dma_buf(attachment,
> + direction);
> +}
> +
> +static
> +void vc_sm_import_unmap_dma_buf(struct dma_buf_attachment *attachment,
> + struct sg_table *table,
> + enum dma_data_direction direction)
> +{
> + struct vc_sm_buffer *buf = attachment->dmabuf->priv;
> +
> + buf->imported_dma_buf->ops->unmap_dma_buf(attachment, table, direction);
> +}
> +
> +static
> +int vc_sm_import_dmabuf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + pr_debug("%s: mmap dma_buf %p, buf %p, imported db %p\n", __func__,
> + dmabuf, buf, buf->imported_dma_buf);
> +
> + return buf->imported_dma_buf->ops->mmap(buf->imported_dma_buf, vma);
> +}
> +
> +static
> +int vc_sm_import_dma_buf_begin_cpu_access(struct dma_buf *dmabuf,
> + enum dma_data_direction direction)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + return buf->imported_dma_buf->ops->begin_cpu_access(buf->imported_dma_buf,
> + direction);
> +}
> +
> +static
> +int vc_sm_import_dma_buf_end_cpu_access(struct dma_buf *dmabuf,
> + enum dma_data_direction direction)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + return buf->imported_dma_buf->ops->end_cpu_access(buf->imported_dma_buf,
> + direction);
> +}
> +
> +static const struct dma_buf_ops dma_buf_import_ops = {
> + .map_dma_buf = vc_sm_import_map_dma_buf,
> + .unmap_dma_buf = vc_sm_import_unmap_dma_buf,
> + .mmap = vc_sm_import_dmabuf_mmap,
> + .release = vc_sm_dma_buf_release,
> + .attach = vc_sm_import_dma_buf_attach,
> + .detach = vc_sm_import_dma_buf_detatch,
> + .begin_cpu_access = vc_sm_import_dma_buf_begin_cpu_access,
> + .end_cpu_access = vc_sm_import_dma_buf_end_cpu_access,
> +};
> +
> +/* Import a dma_buf to be shared with VC. */
> +int
> +vc_sm_cma_import_dmabuf_internal(struct sm_state_t *state,
> + struct dma_buf *dma_buf,
> + int fd,
> + struct dma_buf **imported_buf)
This can be static.
> +{
> + DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
> + struct vc_sm_buffer *buffer = NULL;
> + struct vc_sm_import import = { };
> + struct vc_sm_import_result result = { };
> + struct dma_buf_attachment *attach = NULL;
> + struct sg_table *sgt = NULL;
> + dma_addr_t dma_addr;
> + int ret = 0;
> + int status;
> +
> + /* Setup our allocation parameters */
> + pr_debug("%s: importing dma_buf %p/fd %d\n", __func__, dma_buf, fd);
> +
> + if (fd < 0)
> + get_dma_buf(dma_buf);
> + else
> + dma_buf = dma_buf_get(fd);
This function is always called with fd == -1. There seems to be quite a
bit of dead code in this driver, could you double-check everything and
trim it down ?
> +
> + if (!dma_buf)
> + return -EINVAL;
> +
> + attach = dma_buf_attach(dma_buf, &sm_state->pdev->dev);
> + if (IS_ERR(attach)) {
> + ret = PTR_ERR(attach);
> + goto error;
> + }
> +
> + sgt = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL);
> + if (IS_ERR(sgt)) {
> + ret = PTR_ERR(sgt);
> + goto error;
> + }
> +
> + /* Verify that the address block is contiguous */
> + if (sgt->nents != 1) {
> + ret = -ENOMEM;
> + goto error;
> + }
> +
> + /* Allocate local buffer to track this allocation. */
> + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
> + if (!buffer) {
> + ret = -ENOMEM;
> + goto error;
> + }
> +
> + import.type = VC_SM_ALLOC_NON_CACHED;
> + dma_addr = sg_dma_address(sgt->sgl);
> + import.addr = (u32)dma_addr;
> + if ((import.addr & 0xC0000000) != 0xC0000000) {
> + pr_err("%s: Expecting an uncached alias for dma_addr %pad\n",
> + __func__, &dma_addr);
> + import.addr |= 0xC0000000;
> + }
> + import.size = sg_dma_len(sgt->sgl);
> + import.allocator = current->tgid;
> + import.kernel_id = get_kernel_id(buffer);
> +
> + memcpy(import.name, VC_SM_RESOURCE_NAME_DEFAULT,
> + sizeof(VC_SM_RESOURCE_NAME_DEFAULT));
> +
> + pr_debug("[%s]: attempt to import \"%s\" data - type %u, addr %pad, size %u.\n",
> + __func__, import.name, import.type, &dma_addr, import.size);
> +
> + /* Allocate the videocore buffer. */
> + status = vc_sm_cma_vchi_import(sm_state->sm_handle, &import, &result,
> + &sm_state->int_trans_id);
> + if (status == -EINTR) {
> + pr_debug("[%s]: requesting import memory action restart (trans_id: %u)\n",
> + __func__, sm_state->int_trans_id);
> + ret = -ERESTARTSYS;
> + sm_state->restart_sys = -EINTR;
> + sm_state->int_action = VC_SM_MSG_TYPE_IMPORT;
> + goto error;
> + } else if (status || !result.res_handle) {
> + pr_debug("[%s]: failed to import memory on videocore (status: %u, trans_id: %u)\n",
> + __func__, status, sm_state->int_trans_id);
> + ret = -ENOMEM;
> + goto error;
> + }
> +
> + mutex_init(&buffer->lock);
> + INIT_LIST_HEAD(&buffer->attachments);
> + memcpy(buffer->name, import.name,
> + min(sizeof(buffer->name), sizeof(import.name) - 1));
> +
> + /* Keep track of the buffer we created. */
> + buffer->vc_handle = result.res_handle;
> + buffer->size = import.size;
> + buffer->vpu_state = VPU_MAPPED;
> +
> + buffer->imported_dma_buf = dma_buf;
> +
> + buffer->attach = attach;
> + buffer->sgt = sgt;
> + buffer->dma_addr = dma_addr;
> + buffer->in_use = 1;
> + buffer->kernel_id = import.kernel_id;
> +
> + /*
> + * We're done - we need to export a new dmabuf chaining through most
> + * functions, but enabling us to release our own internal references
> + * here.
> + */
Why do we need to export a new dmabuf ?
> + exp_info.ops = &dma_buf_import_ops;
> + exp_info.size = import.size;
> + exp_info.flags = O_RDWR;
> + exp_info.priv = buffer;
> +
> + buffer->dma_buf = dma_buf_export(&exp_info);
> + if (IS_ERR(buffer->dma_buf)) {
> + ret = PTR_ERR(buffer->dma_buf);
> + goto error;
> + }
> +
> + vc_sm_add_resource(buffer);
> +
> + *imported_buf = buffer->dma_buf;
> +
> + return 0;
> +
> +error:
> + if (result.res_handle) {
> + struct vc_sm_free_t free = { result.res_handle, 0 };
> +
> + vc_sm_cma_vchi_free(sm_state->sm_handle, &free,
> + &sm_state->int_trans_id);
> + }
> + free_kernel_id(import.kernel_id);
> + kfree(buffer);
> + if (sgt)
> + dma_buf_unmap_attachment(attach, sgt, DMA_BIDIRECTIONAL);
> + if (attach)
> + dma_buf_detach(dma_buf, attach);
> + dma_buf_put(dma_buf);
> + return ret;
> +}
> +
> +static void
> +vc_sm_vpu_event(struct sm_instance *instance, struct vc_sm_result_t *reply,
> + int reply_len)
> +{
> + switch (reply->trans_id & ~0x80000000) {
> + case VC_SM_MSG_TYPE_CLIENT_VERSION:
> + {
> + /* Acknowledge that the firmware supports the version command */
> + pr_debug("%s: firmware acked version msg. Require release cb\n",
> + __func__);
> + sm_state->require_released_callback = true;
> + }
> + break;
No need for braces.
> + case VC_SM_MSG_TYPE_RELEASED:
> + {
> + struct vc_sm_released *release = (struct vc_sm_released *)reply;
> + struct vc_sm_buffer *buffer =
> + lookup_kernel_id(release->kernel_id);
> + if (!buffer) {
> + pr_err("%s: VC released a buffer that is already released, kernel_id %d\n",
> + __func__, release->kernel_id);
> + break;
> + }
> + mutex_lock(&buffer->lock);
> +
> + pr_debug("%s: Released addr %08x, size %u, id %08x, mem_handle %08x\n",
> + __func__, release->addr, release->size,
> + release->kernel_id, release->vc_handle);
> +
> + buffer->vc_handle = 0;
> + buffer->vpu_state = VPU_NOT_MAPPED;
> + free_kernel_id(release->kernel_id);
> +
> + vc_sm_release_resource(buffer);
> + }
> + break;
> + default:
> + pr_err("%s: Unknown vpu cmd %x\n", __func__, reply->trans_id);
> + break;
> + }
> +}
> +
> +/* Driver load/unload functions */
> +/* Videocore connected. */
> +static void vc_sm_connected_init(void)
> +{
> + int ret;
> + struct vc_sm_version version;
> + struct vc_sm_result_t version_result;
> +
> + pr_info("[%s]: start\n", __func__);
> +
> + /*
> + * Initialize and create a VCHI connection for the shared memory service
> + * running on videocore.
> + */
> + ret = vchiq_initialise(&sm_state->vchiq_instance);
> + if (ret) {
> + pr_err("[%s]: failed to initialise VCHI instance (ret=%d)\n",
> + __func__, ret);
> +
> + return;
> + }
> +
> + ret = vchiq_connect(sm_state->vchiq_instance);
> + if (ret) {
> + pr_err("[%s]: failed to connect VCHI instance (ret=%d)\n",
> + __func__, ret);
> +
> + return;
> + }
> +
> + /* Initialize an instance of the shared memory service. */
> + sm_state->sm_handle = vc_sm_cma_vchi_init(sm_state->vchiq_instance, 1,
> + vc_sm_vpu_event);
> + if (!sm_state->sm_handle) {
> + pr_err("[%s]: failed to initialize shared memory service\n",
> + __func__);
> +
> + return;
> + }
> +
> + /* Create a debug fs directory entry (root). */
> + sm_state->dir_root = debugfs_create_dir(VC_SM_DIR_ROOT_NAME, NULL);
> +
> + sm_state->dir_state.show = &vc_sm_cma_global_state_show;
> + sm_state->dir_state.dir_entry =
> + debugfs_create_file(VC_SM_STATE, 0444, sm_state->dir_root,
> + &sm_state->dir_state,
> + &vc_sm_cma_debug_fs_fops);
Move this to a separate debugfs init function, grouped with the rest of
the debugfs code. debugfs is conditioned by CONFIG_DEBUG_FS, so you'll
need conditional compilation. You will also need to handle cleanup, you
never destroy the file and directory.
> +
> + INIT_LIST_HEAD(&sm_state->buffer_list);
> +
> + version.version = 2;
> + ret = vc_sm_cma_vchi_client_version(sm_state->sm_handle, &version,
> + &version_result,
> + &sm_state->int_trans_id);
> + if (ret) {
> + pr_err("[%s]: Failed to send version request %d\n", __func__,
> + ret);
> + }
> +
> + /* Done! */
> + sm_inited = 1;
> + pr_info("[%s]: installed successfully\n", __func__);
> +}
> +
> +/* Driver loading. */
> +static int bcm2835_vc_sm_cma_probe(struct platform_device *pdev)
> +{
> + pr_info("%s: Videocore shared memory driver\n", __func__);
Drop this, it only clutters the kernel log.
> +
> + sm_state = devm_kzalloc(&pdev->dev, sizeof(*sm_state), GFP_KERNEL);
> + if (!sm_state)
> + return -ENOMEM;
> + sm_state->pdev = pdev;
> + mutex_init(&sm_state->map_lock);
> +
> + spin_lock_init(&sm_state->kernelid_map_lock);
> + idr_init_base(&sm_state->kernelid_map, 1);
> +
> + pdev->dev.dma_parms = devm_kzalloc(&pdev->dev,
> + sizeof(*pdev->dev.dma_parms),
> + GFP_KERNEL);
> + /* dma_set_max_seg_size checks if dma_parms is NULL. */
> + dma_set_max_seg_size(&pdev->dev, 0x3FFFFFFF);
> +
> + vchiq_add_connected_callback(vc_sm_connected_init);
> + return 0;
> +}
> +
> +/* Driver unloading. */
> +static int bcm2835_vc_sm_cma_remove(struct platform_device *pdev)
> +{
> + pr_debug("[%s]: start\n", __func__);
You can drop this message and the one at the end of the function.
> + if (sm_inited) {
> + misc_deregister(&sm_state->misc_dev);
There's no misc_register() call, something seems wrong. Probably
leftover code ?
> +
> + /* Remove all proc entries. */
> + debugfs_remove_recursive(sm_state->dir_root);
> +
> + /* Stop the videocore shared memory service. */
> + vc_sm_cma_vchi_stop(sm_state->vchiq_instance, &sm_state->sm_handle);
> + }
> +
> + if (sm_state) {
Drop the condition, sm_state can never be NULL if probe() succeeded, and
if it hasn't succeeded, then remove() will not be called.
> + idr_destroy(&sm_state->kernelid_map);
> +
> + /* Free the memory for the state structure. */
> + mutex_destroy(&sm_state->map_lock);
> + }
> +
> + pr_debug("[%s]: end\n", __func__);
> + return 0;
> +}
> +
> +/* Kernel API calls */
> +/* Get an internal resource handle mapped from the external one. */
> +int vc_sm_cma_int_handle(void *handle)
> +{
> + struct dma_buf *dma_buf = (struct dma_buf *)handle;
> + struct vc_sm_buffer *buf;
> +
> + /* Validate we can work with this device. */
> + if (!sm_state || !handle) {
> + pr_err("[%s]: invalid input\n", __func__);
> + return 0;
> + }
> +
> + buf = (struct vc_sm_buffer *)dma_buf->priv;
> + return buf->vc_handle;
> +}
> +EXPORT_SYMBOL_GPL(vc_sm_cma_int_handle);
> +
> +/* Free a previously allocated shared memory handle and block. */
> +int vc_sm_cma_free(void *handle)
> +{
> + struct dma_buf *dma_buf = (struct dma_buf *)handle;
> +
> + /* Validate we can work with this device. */
> + if (!sm_state || !handle) {
> + pr_err("[%s]: invalid input\n", __func__);
> + return -EPERM;
That's a weird error code for an invalid input.
> + }
> +
> + pr_debug("%s: handle %p/dmabuf %p\n", __func__, handle, dma_buf);
> +
> + dma_buf_put(dma_buf);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(vc_sm_cma_free);
> +
> +/* Import a dmabuf to be shared with VC. */
> +int vc_sm_cma_import_dmabuf(struct dma_buf *src_dmabuf, void **handle)
> +{
> + struct dma_buf *new_dma_buf;
> + int ret;
> +
> + /* Validate we can work with this device. */
> + if (!sm_state || !src_dmabuf || !handle) {
> + pr_err("[%s]: invalid input\n", __func__);
> + return -EPERM;
Same here.
> + }
> +
> + ret = vc_sm_cma_import_dmabuf_internal(sm_state, src_dmabuf,
> + -1, &new_dma_buf);
> +
> + if (!ret) {
> + pr_debug("%s: imported to ptr %p\n", __func__, new_dma_buf);
> +
> + /* Assign valid handle at this time.*/
> + *handle = new_dma_buf;
> + } else {
> + /*
> + * succeeded in importing the dma_buf, but then
> + * failed to look it up again. How?
> + * Release the fd again.
> + */
> + pr_err("%s: imported vc_sm_cma_get_buffer failed %d\n",
> + __func__, ret);
> + }
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(vc_sm_cma_import_dmabuf);
Those three functions are a bit ill-placed between the probe/remove
functions and the platform_driver. I'd move them just above
vc_sm_vpu_event(), and order them as mentioned below in the header file.
> +
> +static struct platform_driver bcm2835_vcsm_cma_driver = {
> + .probe = bcm2835_vc_sm_cma_probe,
> + .remove = bcm2835_vc_sm_cma_remove,
> + .driver = {
> + .name = "vcsm-cma",
> + .owner = THIS_MODULE,
> + },
Wrong indentation, should be
.driver = {
.name = "vcsm-cma",
.owner = THIS_MODULE,
},
> +};
> +
> +module_platform_driver(bcm2835_vcsm_cma_driver);
> +
> +MODULE_AUTHOR("Dave Stevenson");
> +MODULE_DESCRIPTION("VideoCore CMA Shared Memory Driver");
> +MODULE_LICENSE("GPL");
> +MODULE_ALIAS("platform:vcsm-cma");
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> new file mode 100644
> index 000000000000..61e110ec414d
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> @@ -0,0 +1,54 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * VideoCore Shared Memory driver using CMA.
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + *
> + */
> +
> +#ifndef VC_SM_H
> +#define VC_SM_H
> +
> +#define VC_SM_MAX_NAME_LEN 32
> +
> +enum vc_sm_vpu_mapping_state {
> + VPU_NOT_MAPPED,
> + VPU_MAPPED,
> + VPU_UNMAPPING
> +};
> +
> +struct vc_sm_buffer {
> + struct list_head global_buffer_list; /* Global list of buffers. */
> +
> + /* Index in the kernel_id idr so that we can find the
> + * mmal_msg_context again when servicing the VCHI reply.
> + */
> + int kernel_id;
> +
> + size_t size;
> +
> + /* Lock over all the following state for this buffer */
> + struct mutex lock;
> + struct list_head attachments;
> +
> + char name[VC_SM_MAX_NAME_LEN];
> +
> + int in_use:1; /* Kernel is still using this resource */
> +
> + enum vc_sm_vpu_mapping_state vpu_state;
> + u32 vc_handle; /* VideoCore handle for this buffer */
> +
> + /* DMABUF related fields */
> + struct dma_buf *dma_buf;
> + dma_addr_t dma_addr;
> + void *cookie;
> +
> + struct vc_sm_privdata_t *private;
> +
> + struct dma_buf *imported_dma_buf;
> + struct dma_buf_attachment *attach;
> + struct sg_table *sgt;
> +};
> +
> +#endif
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> new file mode 100644
> index 000000000000..c77ef0998a31
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> @@ -0,0 +1,507 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * VideoCore Shared Memory CMA allocator
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + * Copyright 2011-2012 Broadcom Corporation. All rights reserved.
> + *
> + * Based on vmcs_sm driver from Broadcom Corporation.
> + *
> + */
> +
> +/* ---- Include Files ----------------------------------------------------- */
> +#include <linux/kthread.h>
> +#include <linux/list.h>
> +#include <linux/semaphore.h>
> +#include <linux/slab.h>
> +#include <linux/types.h>
> +
> +#include "vc_sm_cma_vchi.h"
> +
> +#define VC_SM_VER 1
> +#define VC_SM_MIN_VER 0
> +
> +/* ---- Private Constants and Types -------------------------------------- */
> +
> +/* Command blocks come from a pool */
> +#define SM_MAX_NUM_CMD_RSP_BLKS 32
> +
> +/* The number of supported connections */
> +#define SM_MAX_NUM_CONNECTIONS 3
> +
> +struct sm_cmd_rsp_blk {
> + struct list_head head; /* To create lists */
> + /* To be signaled when the response is there */
> + struct completion cmplt;
> +
> + u32 id;
> + u16 length;
> +
> + u8 msg[VC_SM_MAX_MSG_LEN];
> +
> + uint32_t wait:1;
> + uint32_t sent:1;
> + uint32_t alloc:1;
> +
> +};
> +
> +struct sm_instance {
> + u32 num_connections;
> + unsigned int service_handle[SM_MAX_NUM_CONNECTIONS];
> + struct task_struct *io_thread;
> + struct completion io_cmplt;
> +
> + vpu_event_cb vpu_event;
> +
> + /* Mutex over the following lists */
> + struct mutex lock;
> + u32 trans_id;
> + struct list_head cmd_list;
> + struct list_head rsp_list;
> + struct list_head dead_list;
> +
> + struct sm_cmd_rsp_blk free_blk[SM_MAX_NUM_CMD_RSP_BLKS];
> +
> + /* Mutex over the free_list */
> + struct mutex free_lock;
> + struct list_head free_list;
> +
> + struct semaphore free_sema;
> + struct vchiq_instance *vchiq_instance;
> +};
> +
> +/* ---- Private Variables ------------------------------------------------ */
> +
> +/* ---- Private Function Prototypes -------------------------------------- */
> +
> +/* ---- Private Functions ------------------------------------------------ */
> +static int
> +bcm2835_vchi_msg_queue(struct vchiq_instance *vchiq_instance, unsigned int handle,
> + void *data,
> + unsigned int size)
> +{
> + return vchiq_queue_kernel_message(vchiq_instance, handle, data, size);
> +}
> +
> +static struct
> +sm_cmd_rsp_blk *vc_vchi_cmd_create(struct sm_instance *instance,
> + enum vc_sm_msg_type id, void *msg,
> + u32 size, int wait)
> +{
> + struct sm_cmd_rsp_blk *blk;
> + struct vc_sm_msg_hdr_t *hdr;
> +
> + if (down_interruptible(&instance->free_sema)) {
> + blk = kmalloc(sizeof(*blk), GFP_KERNEL);
> + if (!blk)
> + return NULL;
> +
> + blk->alloc = 1;
> + init_completion(&blk->cmplt);
> + } else {
> + mutex_lock(&instance->free_lock);
> + blk =
> + list_first_entry(&instance->free_list,
> + struct sm_cmd_rsp_blk, head);
> + list_del(&blk->head);
> + mutex_unlock(&instance->free_lock);
> + }
> +
> + blk->sent = 0;
> + blk->wait = wait;
> + blk->length = sizeof(*hdr) + size;
> +
> + hdr = (struct vc_sm_msg_hdr_t *)blk->msg;
> + hdr->type = id;
> + mutex_lock(&instance->lock);
> + instance->trans_id++;
> + /*
> + * Retain the top bit for identifying asynchronous events, or VPU cmds.
> + */
> + instance->trans_id &= ~0x80000000;
> + hdr->trans_id = instance->trans_id;
> + blk->id = instance->trans_id;
> + mutex_unlock(&instance->lock);
> +
> + if (size)
> + memcpy(hdr->body, msg, size);
> +
> + return blk;
> +}
> +
> +static void
> +vc_vchi_cmd_delete(struct sm_instance *instance, struct sm_cmd_rsp_blk *blk)
> +{
> + if (blk->alloc) {
> + kfree(blk);
> + return;
> + }
> +
> + mutex_lock(&instance->free_lock);
> + list_add(&blk->head, &instance->free_list);
> + mutex_unlock(&instance->free_lock);
> + up(&instance->free_sema);
> +}
> +
> +static void vc_sm_cma_vchi_rx_ack(struct sm_instance *instance,
> + struct sm_cmd_rsp_blk *cmd,
> + struct vc_sm_result_t *reply,
> + u32 reply_len)
> +{
> + mutex_lock(&instance->lock);
> + list_for_each_entry(cmd,
> + &instance->rsp_list,
> + head) {
> + if (cmd->id == reply->trans_id)
> + break;
> + }
> + mutex_unlock(&instance->lock);
> +
> + if (&cmd->head == &instance->rsp_list) {
> + //pr_debug("%s: received response %u, throw away...",
> + pr_err("%s: received response %u, throw away...",
> + __func__,
> + reply->trans_id);
> + } else if (reply_len > sizeof(cmd->msg)) {
> + pr_err("%s: reply too big (%u) %u, throw away...",
> + __func__, reply_len,
> + reply->trans_id);
> + } else {
> + memcpy(cmd->msg, reply,
> + reply_len);
> + complete(&cmd->cmplt);
> + }
> +}
> +
> +static int vc_sm_cma_vchi_videocore_io(void *arg)
> +{
> + struct sm_instance *instance = arg;
> + struct sm_cmd_rsp_blk *cmd = NULL, *cmd_tmp;
> + struct vc_sm_result_t *reply;
> + struct vchiq_header *header;
> + s32 status;
> + int svc_use = 1;
> +
> + while (1) {
> + if (svc_use)
> + vchiq_release_service(instance->vchiq_instance,
> + instance->service_handle[0]);
> + svc_use = 0;
> +
> + if (wait_for_completion_interruptible(&instance->io_cmplt))
> + continue;
Isn't it a bit overkill to use a thread, wouldn't a workqueue do ?
> + vchiq_use_service(instance->vchiq_instance, instance->service_handle[0]);
> + svc_use = 1;
> +
> + do {
> + /*
> + * Get new command and move it to response list
> + */
> + mutex_lock(&instance->lock);
> + if (list_empty(&instance->cmd_list)) {
> + /* no more commands to process */
> + mutex_unlock(&instance->lock);
> + break;
> + }
> + cmd = list_first_entry(&instance->cmd_list,
> + struct sm_cmd_rsp_blk, head);
> + list_move(&cmd->head, &instance->rsp_list);
> + cmd->sent = 1;
> + mutex_unlock(&instance->lock);
> + /* Send the command */
> + status =
> + bcm2835_vchi_msg_queue(instance->vchiq_instance,
> + instance->service_handle[0],
> + cmd->msg, cmd->length);
> + if (status) {
> + pr_err("%s: failed to queue message (%d)",
> + __func__, status);
> + }
> +
> + /* If no reply is needed then we're done */
> + if (!cmd->wait) {
> + mutex_lock(&instance->lock);
> + list_del(&cmd->head);
> + mutex_unlock(&instance->lock);
> + vc_vchi_cmd_delete(instance, cmd);
> + continue;
> + }
> +
> + if (status) {
> + complete(&cmd->cmplt);
> + continue;
> + }
> +
> + } while (1);
> +
> + while ((header = vchiq_msg_hold(instance->vchiq_instance,
> + instance->service_handle[0]))) {
> + reply = (struct vc_sm_result_t *)header->data;
> + if (reply->trans_id & 0x80000000) {
> + /* Async event or cmd from the VPU */
> + if (instance->vpu_event)
> + instance->vpu_event(instance, reply,
> + header->size);
> + } else {
> + vc_sm_cma_vchi_rx_ack(instance, cmd, reply,
> + header->size);
> + }
> +
> + vchiq_release_message(instance->vchiq_instance,
> + instance->service_handle[0],
> + header);
> + }
> +
> + /* Go through the dead list and free them */
> + mutex_lock(&instance->lock);
> + list_for_each_entry_safe(cmd, cmd_tmp, &instance->dead_list,
> + head) {
> + list_del(&cmd->head);
> + vc_vchi_cmd_delete(instance, cmd);
> + }
> + mutex_unlock(&instance->lock);
> + }
> +
> + return 0;
> +}
> +
> +static enum vchiq_status vc_sm_cma_vchi_callback(struct vchiq_instance *vchiq_instance,
> + enum vchiq_reason reason,
> + struct vchiq_header *header,
> + unsigned int handle, void *userdata)
> +{
> + struct sm_instance *instance = vchiq_get_service_userdata(vchiq_instance, handle);
> +
> + switch (reason) {
> + case VCHIQ_MESSAGE_AVAILABLE:
> + vchiq_msg_queue_push(vchiq_instance, handle, header);
> + complete(&instance->io_cmplt);
> + break;
> +
> + case VCHIQ_SERVICE_CLOSED:
> + pr_info("%s: service CLOSED!!", __func__);
> + default:
> + break;
> + }
> +
> + return VCHIQ_SUCCESS;
> +}
> +
> +struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchiq_instance,
> + unsigned int num_connections,
> + vpu_event_cb vpu_event)
> +{
> + u32 i;
> + struct sm_instance *instance;
> + int status;
> +
> + pr_debug("%s: start", __func__);
> +
> + if (num_connections > SM_MAX_NUM_CONNECTIONS) {
> + pr_err("%s: unsupported number of connections %u (max=%u)",
> + __func__, num_connections, SM_MAX_NUM_CONNECTIONS);
> +
> + goto err_null;
> + }
> + /* Allocate memory for this instance */
> + instance = kzalloc(sizeof(*instance), GFP_KERNEL);
> +
> + /* Misc initialisations */
> + mutex_init(&instance->lock);
> + init_completion(&instance->io_cmplt);
> + INIT_LIST_HEAD(&instance->cmd_list);
> + INIT_LIST_HEAD(&instance->rsp_list);
> + INIT_LIST_HEAD(&instance->dead_list);
> + INIT_LIST_HEAD(&instance->free_list);
> + sema_init(&instance->free_sema, SM_MAX_NUM_CMD_RSP_BLKS);
> + mutex_init(&instance->free_lock);
> + for (i = 0; i < SM_MAX_NUM_CMD_RSP_BLKS; i++) {
> + init_completion(&instance->free_blk[i].cmplt);
> + list_add(&instance->free_blk[i].head, &instance->free_list);
> + }
> +
> + instance->vchiq_instance = vchiq_instance;
> +
> + /* Open the VCHI service connections */
> + instance->num_connections = num_connections;
> + for (i = 0; i < num_connections; i++) {
> + struct vchiq_service_params_kernel params = {
> + .version = VC_SM_VER,
> + .version_min = VC_SM_MIN_VER,
> + .fourcc = VCHIQ_MAKE_FOURCC('S', 'M', 'E', 'M'),
> + .callback = vc_sm_cma_vchi_callback,
> + .userdata = instance,
> + };
> +
> + status = vchiq_open_service(vchiq_instance, ¶ms,
> + &instance->service_handle[i]);
> + if (status) {
> + pr_err("%s: failed to open VCHI service (%d)",
> + __func__, status);
> +
> + goto err_close_services;
> + }
> + }
> + /* Create the thread which takes care of all io to/from videoocore. */
> + instance->io_thread = kthread_create(&vc_sm_cma_vchi_videocore_io,
> + (void *)instance, "SMIO");
> + if (!instance->io_thread) {
> + pr_err("%s: failed to create SMIO thread", __func__);
> +
> + goto err_close_services;
> + }
> + instance->vpu_event = vpu_event;
> + set_user_nice(instance->io_thread, -10);
> + wake_up_process(instance->io_thread);
> +
> + pr_debug("%s: success - instance %p", __func__, instance);
> + return instance;
> +
> +err_close_services:
> + for (i = 0; i < instance->num_connections; i++) {
> + if (instance->service_handle[i])
> + vchiq_close_service(vchiq_instance, instance->service_handle[i]);
> + }
> + kfree(instance);
> +err_null:
> + pr_debug("%s: FAILED", __func__);
> + return NULL;
> +}
> +
> +int vc_sm_cma_vchi_stop(struct vchiq_instance *vchiq_instance, struct sm_instance **handle)
> +{
> + struct sm_instance *instance;
> + u32 i;
> +
> + if (!handle) {
> + pr_err("%s: invalid pointer to handle %p", __func__, handle);
> + goto lock;
> + }
> +
> + if (!*handle) {
> + pr_err("%s: invalid handle %p", __func__, *handle);
> + goto lock;
> + }
> +
> + instance = *handle;
> +
> + /* Close all VCHI service connections */
> + for (i = 0; i < instance->num_connections; i++) {
> + vchiq_use_service(vchiq_instance, instance->service_handle[i]);
> + vchiq_close_service(vchiq_instance, instance->service_handle[i]);
> + }
> +
> + kfree(instance);
> +
> + *handle = NULL;
> + return 0;
> +
> +lock:
> + return -EINVAL;
> +}
> +
> +static int vc_sm_cma_vchi_send_msg(struct sm_instance *handle,
> + enum vc_sm_msg_type msg_id, void *msg,
> + u32 msg_size, void *result, u32 result_size,
> + u32 *cur_trans_id, u8 wait_reply)
> +{
> + int status = 0;
> + struct sm_instance *instance = handle;
> + struct sm_cmd_rsp_blk *cmd_blk;
> +
> + if (!handle) {
> + pr_err("%s: invalid handle", __func__);
> + return -EINVAL;
> + }
> + if (!msg) {
> + pr_err("%s: invalid msg pointer", __func__);
> + return -EINVAL;
> + }
> +
> + cmd_blk =
> + vc_vchi_cmd_create(instance, msg_id, msg, msg_size, wait_reply);
> + if (!cmd_blk) {
> + pr_err("[%s]: failed to allocate global tracking resource",
> + __func__);
> + return -ENOMEM;
> + }
> +
> + if (cur_trans_id)
> + *cur_trans_id = cmd_blk->id;
> +
> + mutex_lock(&instance->lock);
> + list_add_tail(&cmd_blk->head, &instance->cmd_list);
> + mutex_unlock(&instance->lock);
> + complete(&instance->io_cmplt);
> +
> + if (!wait_reply)
> + /* We're done */
> + return 0;
> +
> + /* Wait for the response */
> + if (wait_for_completion_interruptible(&cmd_blk->cmplt)) {
> + mutex_lock(&instance->lock);
> + if (!cmd_blk->sent) {
> + list_del(&cmd_blk->head);
> + mutex_unlock(&instance->lock);
> + vc_vchi_cmd_delete(instance, cmd_blk);
> + return -ENXIO;
> + }
> +
> + list_move(&cmd_blk->head, &instance->dead_list);
> + mutex_unlock(&instance->lock);
> + complete(&instance->io_cmplt);
> + return -EINTR; /* We're done */
> + }
> +
> + if (result && result_size) {
> + memcpy(result, cmd_blk->msg, result_size);
> + } else {
> + struct vc_sm_result_t *res =
> + (struct vc_sm_result_t *)cmd_blk->msg;
> + status = (res->success == 0) ? 0 : -ENXIO;
> + }
> +
> + mutex_lock(&instance->lock);
> + list_del(&cmd_blk->head);
> + mutex_unlock(&instance->lock);
> + vc_vchi_cmd_delete(instance, cmd_blk);
> + return status;
> +}
> +
> +int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg,
> + u32 *cur_trans_id)
> +{
> + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_FREE,
> + msg, sizeof(*msg), 0, 0, cur_trans_id, 0);
> +}
> +
> +int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg,
> + struct vc_sm_import_result *result, u32 *cur_trans_id)
> +{
> + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_IMPORT,
> + msg, sizeof(*msg), result, sizeof(*result),
> + cur_trans_id, 1);
> +}
> +
> +int vc_sm_cma_vchi_client_version(struct sm_instance *handle,
> + struct vc_sm_version *msg,
> + struct vc_sm_result_t *result,
> + u32 *cur_trans_id)
> +{
> + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_CLIENT_VERSION,
> + //msg, sizeof(*msg), result, sizeof(*result),
> + //cur_trans_id, 1);
> + msg, sizeof(*msg), NULL, 0,
> + cur_trans_id, 0);
> +}
> +
> +int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle,
> + struct vc_sm_vc_mem_request_result *msg,
> + uint32_t *cur_trans_id)
> +{
> + return vc_sm_cma_vchi_send_msg(handle,
> + VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY,
> + msg, sizeof(*msg), 0, 0, cur_trans_id,
> + 0);
> +}
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> new file mode 100644
> index 000000000000..a4f40d4cef05
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> @@ -0,0 +1,63 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * VideoCore Shared Memory CMA allocator
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + * Copyright 2011-2012 Broadcom Corporation. All rights reserved.
> + *
> + * Based on vmcs_sm driver from Broadcom Corporation.
> + *
> + */
> +
> +#ifndef __VC_SM_CMA_VCHI_H__INCLUDED__
This is quite inconsistent with the existing coding style of
vc04_services and of the kernel as a whole. I'd go for VC_SM_CMA_VCHI_H,
or if preferred, __VC_SM_CMA_VCHI_H or __VC_SM_CMA_VCHI_H__.
> +#define __VC_SM_CMA_VCHI_H__INCLUDED__
> +
> +#include <linux/raspberrypi/vchiq.h>
> +
> +#include "vc_sm_defs.h"
> +
> +/*
> + * Forward declare.
> + */
Drop this.
> +struct sm_instance;
> +
> +typedef void (*vpu_event_cb)(struct sm_instance *instance,
> + struct vc_sm_result_t *reply, int reply_len);
> +
> +/*
> + * Initialize the shared memory service, opens up vchi connection to talk to it.
> + */
> +struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchi_instance,
> + unsigned int num_connections,
> + vpu_event_cb vpu_event);
> +
> +/*
> + * Terminates the shared memory service.
> + */
> +int vc_sm_cma_vchi_stop(struct vchiq_instance *vchi_instance, struct sm_instance **handle);
> +
> +/*
> + * Ask the shared memory service to free up some memory that was previously
> + * allocated by the vc_sm_cma_vchi_alloc function call.
> + */
> +int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg,
> + u32 *cur_trans_id);
> +
> +/*
> + * Import a contiguous block of memory and wrap it in a GPU MEM_HANDLE_T.
> + */
> +int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg,
> + struct vc_sm_import_result *result,
> + u32 *cur_trans_id);
> +
> +int vc_sm_cma_vchi_client_version(struct sm_instance *handle,
> + struct vc_sm_version *msg,
> + struct vc_sm_result_t *result,
> + u32 *cur_trans_id);
> +
> +int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle,
> + struct vc_sm_vc_mem_request_result *msg,
> + uint32_t *cur_trans_id);
> +
> +#endif /* __VC_SM_CMA_VCHI_H__INCLUDED__ */
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> new file mode 100644
> index 000000000000..ad4a3ec194d3
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> @@ -0,0 +1,187 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * VideoCore Shared Memory CMA allocator
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + *
> + * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation.
> + * All IPC messages are copied across to this file, even if the vc-sm-cma
> + * driver is not currently using them.
> + *
> + ****************************************************************************
> + */
> +
> +#ifndef __VC_SM_DEFS_H__INCLUDED__
> +#define __VC_SM_DEFS_H__INCLUDED__
> +
> +/* Maximum message length */
> +#define VC_SM_MAX_MSG_LEN (sizeof(union vc_sm_msg_union_t) + \
> + sizeof(struct vc_sm_msg_hdr_t))
> +#define VC_SM_MAX_RSP_LEN (sizeof(union vc_sm_msg_union_t))
> +
> +/* Resource name maximum size */
> +#define VC_SM_RESOURCE_NAME 32
> +
> +/*
> + * Version to be reported to the VPU
> + * VPU assumes 0 (aka 1) which does not require the released callback, nor
> + * expect the client to handle VC_MEM_REQUESTS.
> + * Version 2 requires the released callback, and must support VC_MEM_REQUESTS.
> + */
> +#define VC_SM_PROTOCOL_VERSION 2
> +
> +enum vc_sm_msg_type {
> + /* Message types supported for HOST->VC direction */
> +
> + /* Allocate shared memory block */
> + VC_SM_MSG_TYPE_ALLOC,
> + /* Lock allocated shared memory block */
> + VC_SM_MSG_TYPE_LOCK,
> + /* Unlock allocated shared memory block */
> + VC_SM_MSG_TYPE_UNLOCK,
> + /* Unlock allocated shared memory block, do not answer command */
> + VC_SM_MSG_TYPE_UNLOCK_NOANS,
> + /* Free shared memory block */
> + VC_SM_MSG_TYPE_FREE,
> + /* Resize a shared memory block */
> + VC_SM_MSG_TYPE_RESIZE,
> + /* Walk the allocated shared memory block(s) */
> + VC_SM_MSG_TYPE_WALK_ALLOC,
> +
> + /* A previously applied action will need to be reverted */
> + VC_SM_MSG_TYPE_ACTION_CLEAN,
> +
> + /*
> + * Import a physical address and wrap into a MEM_HANDLE_T.
> + * Release with VC_SM_MSG_TYPE_FREE.
> + */
> + VC_SM_MSG_TYPE_IMPORT,
> + /*
> + *Tells VC the protocol version supported by this client.
> + * 2 supports the async/cmd messages from the VPU for final release
> + * of memory, and for VC allocations.
> + */
> + VC_SM_MSG_TYPE_CLIENT_VERSION,
> + /* Response to VC request for memory */
> + VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY,
> +
> + /*
> + * Asynchronous/cmd messages supported for VC->HOST direction.
> + * Signalled by setting the top bit in vc_sm_result_t trans_id.
> + */
> +
> + /*
> + * VC has finished with an imported memory allocation.
> + * Release any Linux reference counts on the underlying block.
> + */
> + VC_SM_MSG_TYPE_RELEASED,
> + /* VC request for memory */
> + VC_SM_MSG_TYPE_VC_MEM_REQUEST,
> +
> + VC_SM_MSG_TYPE_MAX
> +};
> +
> +/* Type of memory to be allocated */
> +enum vc_sm_alloc_type_t {
> + VC_SM_ALLOC_CACHED,
> + VC_SM_ALLOC_NON_CACHED,
> +};
> +
> +/* Message header for all messages in HOST->VC direction */
> +struct vc_sm_msg_hdr_t {
> + u32 type;
> + u32 trans_id;
> + u8 body[0];
> +
> +};
> +
> +/* Request to free a previously allocated memory (HOST->VC) */
> +struct vc_sm_free_t {
> + /* Resource handle (returned from alloc) */
> + u32 res_handle;
> + /* Resource buffer (returned from alloc) */
> + u32 res_mem;
> +
> +};
> +
> +/* Generic result for a request (VC->HOST) */
> +struct vc_sm_result_t {
> + /* Transaction identifier */
> + u32 trans_id;
> +
> + s32 success;
> +
> +};
> +
> +/* Request to import memory (HOST->VC) */
> +struct vc_sm_import {
> + /* type of memory to allocate */
> + enum vc_sm_alloc_type_t type;
> + /* pointer to the VC (ie physical) address of the allocated memory */
> + u32 addr;
> + /* size of buffer */
> + u32 size;
> + /* opaque handle returned in RELEASED messages */
> + u32 kernel_id;
> + /* Allocator identifier */
> + u32 allocator;
> + /* resource name (for easier tracking on vc side) */
> + char name[VC_SM_RESOURCE_NAME];
> +};
> +
> +/* Result of a requested memory import (VC->HOST) */
> +struct vc_sm_import_result {
> + /* Transaction identifier */
> + u32 trans_id;
> +
> + /* Resource handle */
> + u32 res_handle;
> +};
> +
> +/* Notification that VC has finished with an allocation (VC->HOST) */
> +struct vc_sm_released {
> + /* cmd type / trans_id */
> + u32 cmd;
> +
> + /* pointer to the VC (ie physical) address of the allocated memory */
> + u32 addr;
> + /* size of buffer */
> + u32 size;
> + /* opaque handle returned in RELEASED messages */
> + u32 kernel_id;
> + u32 vc_handle;
> +};
> +
> +/*
> + * Client informing VC as to the protocol version it supports.
> + * >=2 requires the released callback, and supports VC asking for memory.
> + * Failure means that the firmware doesn't support this call, and therefore the
> + * client should either fail, or NOT rely on getting the released callback.
> + */
> +struct vc_sm_version {
> + u32 version;
> +};
> +
> +/* Response from the kernel to provide the VPU with some memory */
> +struct vc_sm_vc_mem_request_result {
> + /* Transaction identifier for the VPU */
> + u32 trans_id;
> + /* pointer to the physical address of the allocated memory */
> + u32 addr;
> + /* opaque handle returned in RELEASED messages */
> + u32 kernel_id;
> +};
> +
> +/* Union of ALL messages */
> +union vc_sm_msg_union_t {
> + struct vc_sm_free_t free;
> + struct vc_sm_result_t result;
> + struct vc_sm_import import;
> + struct vc_sm_import_result import_result;
> + struct vc_sm_version version;
> + struct vc_sm_released released;
> + struct vc_sm_vc_mem_request_result vc_request_result;
> +};
> +
> +#endif /* __VC_SM_DEFS_H__INCLUDED__ */
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> new file mode 100644
> index 000000000000..988fdd967922
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * VideoCore Shared Memory CMA allocator
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + *
> + * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation.
> + *
> + */
> +
> +#ifndef __VC_SM_KNL_H__INCLUDED__
> +#define __VC_SM_KNL_H__INCLUDED__
> +
> +#if !defined(__KERNEL__)
> +#error "This interface is for kernel use only..."
> +#endif
As this header isn't available to userspace, I think you can drop this.
> +
> +/* Free a previously allocated or imported shared memory handle and block. */
kerneldoc would be nice for such an inter-driver API.
> +int vc_sm_cma_free(void *handle);
> +
> +/* Get an internal resource handle mapped from the external one. */
> +int vc_sm_cma_int_handle(void *handle);
> +
> +/* Import a block of memory into the GPU space. */
> +int vc_sm_cma_import_dmabuf(struct dma_buf *dmabuf, void **handle);
I would order these function in the expected call order: import,
int_handle and free.
> +
> +#endif /* __VC_SM_KNL_H__INCLUDED__ */
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 01/14] staging: vc04_services: Add new vc-sm-cma driver
@ 2022-11-21 23:04 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:04 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang (and Dave),
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:09AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> Add Broadcom VideoCore Shared Memory support.
>
> This new driver allows contiguous memory blocks to be imported
> into the VideoCore VPU memory map, and manages the lifetime of
> those objects, only releasing the source dmabuf once the VPU has
> confirmed it has finished with it.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> drivers/staging/vc04_services/Kconfig | 2 +
> drivers/staging/vc04_services/Makefile | 1 +
> .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
> .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
> .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++++++++++++
> .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 ++
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++++++++
> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 ++
> .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++++
> .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
> 10 files changed, 1665 insertions(+)
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
>
> diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig
> index 31e58c9d1a11..6c0e77d64376 100644
> --- a/drivers/staging/vc04_services/Kconfig
> +++ b/drivers/staging/vc04_services/Kconfig
> @@ -46,5 +46,7 @@ source "drivers/staging/vc04_services/bcm2835-camera/Kconfig"
>
> source "drivers/staging/vc04_services/vchiq-mmal/Kconfig"
>
> +source "drivers/staging/vc04_services/vc-sm-cma/Kconfig"
> +
> endif
>
> diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile
> index 1fd191e2e2a5..01089f369fb4 100644
> --- a/drivers/staging/vc04_services/Makefile
> +++ b/drivers/staging/vc04_services/Makefile
> @@ -14,6 +14,7 @@ endif
> obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/
> obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/
> obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/
> +obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/
>
> ccflags-y += -I $(srctree)/$(src)/include
>
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/Kconfig b/drivers/staging/vc04_services/vc-sm-cma/Kconfig
> new file mode 100644
> index 000000000000..bbd296f5826b
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/Kconfig
> @@ -0,0 +1,10 @@
> +config BCM_VC_SM_CMA
> + tristate "VideoCore Shared Memory (CMA) driver"
> + depends on BCM2835_VCHIQ
> + select RBTREE
I don't see an RBTREE config option in mainline.
> + select DMA_SHARED_BUFFER
> + help
> + Say Y here to enable the shared memory interface that
> + supports sharing dmabufs with VideoCore.
> + This operates over the VCHIQ interface to a service
> + running on VideoCore.
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/Makefile b/drivers/staging/vc04_services/vc-sm-cma/Makefile
> new file mode 100644
> index 000000000000..c92a5775c62e
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/Makefile
> @@ -0,0 +1,12 @@
> +ccflags-y += \
> + -I$(srctree)/$(src)/../ \
> + -I$(srctree)/$(src)/../interface/vchiq_arm\
Missing space before \
> + -I$(srctree)/$(src)/../include
> +
> +ccflags-y += \
> + -D__VCCOREVER__=0
Is this needed ?
> +
> +vc-sm-cma-$(CONFIG_BCM_VC_SM_CMA) := \
> + vc_sm.o vc_sm_cma_vchi.o
> +
> +obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma.o
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> new file mode 100644
> index 000000000000..7fe81d259c7b
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> @@ -0,0 +1,801 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * VideoCore Shared Memory driver using CMA.
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + * Dave Stevenson <dave.stevenson@raspberrypi.com>
> + *
> + * Based on vmcs_sm driver from Broadcom Corporation for some API,
> + * and taking some code for buffer allocation and dmabuf handling from
> + * videobuf2.
> + *
> + * This driver handles taking a dmabuf, mapping it into the VPU address space,
> + * and returning a handle that the VPU can use.
> + */
> +
> +/* ---- Include Files ----------------------------------------------------- */
I'd drop this comment (same below), it feels a bit alien.
> +#include <linux/debugfs.h>
> +#include <linux/dma-buf.h>
> +#include <linux/errno.h>
> +#include <linux/miscdevice.h>
> +#include <linux/platform_device.h>
> +#include <linux/syscalls.h>
> +
> +#include "vchiq_connected.h"
> +#include "vc_sm_cma_vchi.h"
> +
> +#include "vc_sm.h"
> +#include "vc_sm_knl.h"
> +
> +MODULE_IMPORT_NS(DMA_BUF);
> +
> +/* ---- Private Constants and Types --------------------------------------- */
> +
> +#define VC_SM_RESOURCE_NAME_DEFAULT "sm-host-resource"
> +
> +#define VC_SM_DIR_ROOT_NAME "vcsm-cma"
> +#define VC_SM_STATE "state"
> +
> +/* Private file data associated with each opened device. */
> +struct vc_sm_privdata_t {
> +};
> +
> +typedef int (*VC_SM_SHOW) (struct seq_file *s, void *v);
> +struct sm_pde_t {
Should this structure (and the other ones below) have a vc_ prefix for
consistency ? Also, drop the _t suffix.
> + VC_SM_SHOW show; /* Debug fs function hookup. */
> + struct dentry *dir_entry; /* Debug fs directory entry. */
> + void *priv_data; /* Private data */
Never used.
> +};
> +
> +/* Global state information. */
> +struct sm_state_t {
> + struct platform_device *pdev;
> +
> + struct miscdevice misc_dev;
> +
> + struct sm_instance *sm_handle; /* Handle for videocore service. */
> +
> + spinlock_t kernelid_map_lock; /* Spinlock protecting kernelid_map */
> + struct idr kernelid_map;
> +
> + struct mutex map_lock; /* Global map lock. */
> + struct list_head buffer_list; /* List of buffer. */
> +
> + struct dentry *dir_root; /* Debug fs entries root. */
> + struct sm_pde_t dir_state; /* Debug fs entries state sub-tree. */
> +
> + bool require_released_callback; /* VPU will send a released msg when it
> + * has finished with a resource.
> + */
> + /* State for transactions */
> + int restart_sys; /* Tracks restart on interrupt. */
> + enum vc_sm_msg_type int_action; /* Interrupted action. */
Both of these are set but never used. Either there's dead code below, or
something is wrong.
> +
> + u32 int_trans_id; /* Interrupted transaction. */
> + struct vchiq_instance *vchiq_instance;
> +};
> +
> +struct vc_sm_dma_buf_attachment {
Never used.
> + struct device *dev;
> + struct sg_table sg_table;
> + struct list_head list;
> + enum dma_data_direction dma_dir;
> +};
> +
> +/* ---- Private Variables ----------------------------------------------- */
> +
> +static struct sm_state_t *sm_state;
> +static int sm_inited;
Can we avoid global variables please ?
> +
> +/* ---- Private Function Prototypes -------------------------------------- */
> +
> +/* ---- Private Functions ------------------------------------------------ */
If you want to split the driver in sections (which I personally like for
large files), I would group functions by purpose and add a section
header for each group.
> +
> +static int get_kernel_id(struct vc_sm_buffer *buffer)
> +{
> + int handle;
> +
> + spin_lock(&sm_state->kernelid_map_lock);
> + handle = idr_alloc(&sm_state->kernelid_map, buffer, 0, 0, GFP_KERNEL);
> + spin_unlock(&sm_state->kernelid_map_lock);
> +
> + return handle;
> +}
> +
> +static struct vc_sm_buffer *lookup_kernel_id(int handle)
> +{
> + return idr_find(&sm_state->kernelid_map, handle);
> +}
> +
> +static void free_kernel_id(int handle)
> +{
> + spin_lock(&sm_state->kernelid_map_lock);
> + idr_remove(&sm_state->kernelid_map, handle);
> + spin_unlock(&sm_state->kernelid_map_lock);
> +}
For instance those three functions could be in the IDR section. I would
probably inline them in their respective caller though, but if you wait
to keep them, you should give them better names. vc_sm_cma_idr_ sounds
like a good prefix, the functions could be vc_sm_cma_idr_alloc(),
vc_sm_cma_idr_find() and vc_sm_cma_idr_remove(). I would then move the
IDR init and cleanup code to two separate functions too.
> +
> +static int vc_sm_cma_seq_file_show(struct seq_file *s, void *v)
> +{
> + struct sm_pde_t *sm_pde;
> +
> + sm_pde = (struct sm_pde_t *)(s->private);
> +
> + if (sm_pde && sm_pde->show)
> + sm_pde->show(s, v);
The only show handler is vc_sm_cma_global_state_show(), you can simplify
all this by dropping this function and passing
vc_sm_cma_global_state_show to single_open(). The show field in sm_pde_t
can also be dropped. You can actually drop the sm_pde_t structure and
embed the dir_entry field directly in sm_state_t.
> +
> + return 0;
> +}
> +
> +static int vc_sm_cma_single_open(struct inode *inode, struct file *file)
> +{
> + return single_open(file, vc_sm_cma_seq_file_show, inode->i_private);
> +}
> +
> +static const struct file_operations vc_sm_cma_debug_fs_fops = {
> + .open = vc_sm_cma_single_open,
> + .read = seq_read,
> + .llseek = seq_lseek,
> + .release = single_release,
> +};
> +
> +static int vc_sm_cma_global_state_show(struct seq_file *s, void *v)
> +{
> + struct vc_sm_buffer *resource = NULL;
> + int resource_count = 0;
> +
> + if (!sm_state)
> + return 0;
> +
> + seq_printf(s, "\nVC-ServiceHandle %p\n", sm_state->sm_handle);
> +
> + /* Log all applicable mapping(s). */
> +
> + mutex_lock(&sm_state->map_lock);
> + seq_puts(s, "\nResources\n");
> + if (!list_empty(&sm_state->buffer_list)) {
> + list_for_each_entry(resource, &sm_state->buffer_list,
> + global_buffer_list) {
> + resource_count++;
> +
> + seq_printf(s, "\nResource %p\n",
> + resource);
> + seq_printf(s, " NAME %s\n",
> + resource->name);
> + seq_printf(s, " SIZE %zu\n",
> + resource->size);
> + seq_printf(s, " DMABUF %p\n",
> + resource->dma_buf);
> + seq_printf(s, " IMPORTED_DMABUF %p\n",
> + resource->imported_dma_buf);
> + seq_printf(s, " ATTACH %p\n",
> + resource->attach);
> + seq_printf(s, " SGT %p\n",
> + resource->sgt);
> + seq_printf(s, " DMA_ADDR %pad\n",
> + &resource->dma_addr);
> + seq_printf(s, " VC_HANDLE %08x\n",
> + resource->vc_handle);
> + seq_printf(s, " VC_MAPPING %d\n",
> + resource->vpu_state);
> + }
> + }
> + seq_printf(s, "\n\nTotal resource count: %d\n\n", resource_count);
> +
> + mutex_unlock(&sm_state->map_lock);
> +
> + return 0;
> +}
> +
> +/*
> + * Adds a buffer to the private data list which tracks all the allocated
> + * data.
> + */
> +static void vc_sm_add_resource(struct vc_sm_buffer *buffer)
> +{
> + mutex_lock(&sm_state->map_lock);
> + list_add(&buffer->global_buffer_list, &sm_state->buffer_list);
> + mutex_unlock(&sm_state->map_lock);
> +
> + pr_debug("[%s]: added buffer %p (name %s, size %zu)\n",
> + __func__, buffer, buffer->name, buffer->size);
Let's use dev_* instead of pr_* where possible.
> +}
> +
> +/*
> + * Cleans up imported dmabuf.
> + * Should be called with mutex held.
> + */
> +static void vc_sm_clean_up_dmabuf(struct vc_sm_buffer *buffer)
> +{
> + /* Handle cleaning up imported dmabufs */
> + if (buffer->sgt) {
> + dma_buf_unmap_attachment(buffer->attach,
> + buffer->sgt,
> + DMA_BIDIRECTIONAL);
> + buffer->sgt = NULL;
> + }
> + if (buffer->attach) {
> + dma_buf_detach(buffer->dma_buf, buffer->attach);
> + buffer->attach = NULL;
> + }
> +}
> +
> +/*
> + * Instructs VPU to decrement the refcount on a buffer.
> + */
> +static void vc_sm_vpu_free(struct vc_sm_buffer *buffer)
> +{
> + if (buffer->vc_handle && buffer->vpu_state == VPU_MAPPED) {
> + struct vc_sm_free_t free = { buffer->vc_handle, 0 };
> + int status = vc_sm_cma_vchi_free(sm_state->sm_handle, &free,
> + &sm_state->int_trans_id);
> + if (status != 0 && status != -EINTR) {
> + pr_err("[%s]: failed to free memory on videocore (status: %u, trans_id: %u)\n",
> + __func__, status, sm_state->int_trans_id);
> + }
> +
> + if (sm_state->require_released_callback) {
> + /* Need to wait for the VPU to confirm the free. */
> +
> + /* Retain a reference on this until the VPU has
> + * released it
> + */
> + buffer->vpu_state = VPU_UNMAPPING;
> + } else {
> + buffer->vpu_state = VPU_NOT_MAPPED;
> + buffer->vc_handle = 0;
> + }
> + }
> +}
> +
> +/*
> + * Release an allocation.
> + * All refcounting is done via the dma buf object.
> + *
> + * Must be called with the mutex held. The function will either release the
> + * mutex (if defering the release) or destroy it. The caller must therefore not
> + * reuse the buffer on return.
> + */
> +static void vc_sm_release_resource(struct vc_sm_buffer *buffer)
> +{
> + pr_debug("[%s]: buffer %p (name %s, size %zu)\n",
> + __func__, buffer, buffer->name, buffer->size);
> +
> + if (buffer->vc_handle) {
> + /* We've sent the unmap request but not had the response. */
> + pr_debug("[%s]: Waiting for VPU unmap response on %p\n",
> + __func__, buffer);
> + goto defer;
> + }
> + if (buffer->in_use) {
> + /* dmabuf still in use - we await the release */
> + pr_debug("[%s]: buffer %p is still in use\n", __func__, buffer);
> + goto defer;
> + }
> +
> + /* Release the allocation */
> + if (buffer->imported_dma_buf)
> + dma_buf_put(buffer->imported_dma_buf);
> + else
> + pr_err("%s: Imported dmabuf already been put for buf %p\n",
> + __func__, buffer);
> + buffer->imported_dma_buf = NULL;
> +
> + /* Free our buffer. Start by removing it from the list */
> + mutex_lock(&sm_state->map_lock);
> + list_del(&buffer->global_buffer_list);
> + mutex_unlock(&sm_state->map_lock);
> +
> + pr_debug("%s: Release our allocation - done\n", __func__);
> + mutex_unlock(&buffer->lock);
> +
> + mutex_destroy(&buffer->lock);
> +
> + kfree(buffer);
> + return;
> +
> +defer:
> + mutex_unlock(&buffer->lock);
> +}
> +
> +/* Dma_buf operations for chaining through to an imported dma_buf */
> +
> +static void vc_sm_dma_buf_release(struct dma_buf *dmabuf)
> +{
> + struct vc_sm_buffer *buffer;
> +
> + if (!dmabuf)
> + return;
> +
> + buffer = (struct vc_sm_buffer *)dmabuf->priv;
> +
> + mutex_lock(&buffer->lock);
> +
> + pr_debug("%s dmabuf %p, buffer %p\n", __func__, dmabuf, buffer);
> +
> + buffer->in_use = 0;
> +
> + /* Unmap on the VPU */
> + vc_sm_vpu_free(buffer);
> + pr_debug("%s vpu_free done\n", __func__);
> +
> + /* Unmap our dma_buf object (the vc_sm_buffer remains until released
> + * on the VPU).
> + */
> + vc_sm_clean_up_dmabuf(buffer);
> + pr_debug("%s clean_up dmabuf done\n", __func__);
> +
> + /* buffer->lock will be destroyed by vc_sm_release_resource if finished
> + * with, otherwise unlocked. Do NOT unlock here.
> + */
> + vc_sm_release_resource(buffer);
> + pr_debug("%s done\n", __func__);
> +}
> +
> +static
> +int vc_sm_import_dma_buf_attach(struct dma_buf *dmabuf,
> + struct dma_buf_attachment *attachment)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + return buf->imported_dma_buf->ops->attach(buf->imported_dma_buf,
> + attachment);
> +}
> +
> +static
> +void vc_sm_import_dma_buf_detatch(struct dma_buf *dmabuf,
> + struct dma_buf_attachment *attachment)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + buf->imported_dma_buf->ops->detach(buf->imported_dma_buf, attachment);
> +}
> +
> +static
> +struct sg_table *vc_sm_import_map_dma_buf(struct dma_buf_attachment *attachment,
> + enum dma_data_direction direction)
> +{
> + struct vc_sm_buffer *buf = attachment->dmabuf->priv;
> +
> + return buf->imported_dma_buf->ops->map_dma_buf(attachment,
> + direction);
> +}
> +
> +static
> +void vc_sm_import_unmap_dma_buf(struct dma_buf_attachment *attachment,
> + struct sg_table *table,
> + enum dma_data_direction direction)
> +{
> + struct vc_sm_buffer *buf = attachment->dmabuf->priv;
> +
> + buf->imported_dma_buf->ops->unmap_dma_buf(attachment, table, direction);
> +}
> +
> +static
> +int vc_sm_import_dmabuf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + pr_debug("%s: mmap dma_buf %p, buf %p, imported db %p\n", __func__,
> + dmabuf, buf, buf->imported_dma_buf);
> +
> + return buf->imported_dma_buf->ops->mmap(buf->imported_dma_buf, vma);
> +}
> +
> +static
> +int vc_sm_import_dma_buf_begin_cpu_access(struct dma_buf *dmabuf,
> + enum dma_data_direction direction)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + return buf->imported_dma_buf->ops->begin_cpu_access(buf->imported_dma_buf,
> + direction);
> +}
> +
> +static
> +int vc_sm_import_dma_buf_end_cpu_access(struct dma_buf *dmabuf,
> + enum dma_data_direction direction)
> +{
> + struct vc_sm_buffer *buf = dmabuf->priv;
> +
> + return buf->imported_dma_buf->ops->end_cpu_access(buf->imported_dma_buf,
> + direction);
> +}
> +
> +static const struct dma_buf_ops dma_buf_import_ops = {
> + .map_dma_buf = vc_sm_import_map_dma_buf,
> + .unmap_dma_buf = vc_sm_import_unmap_dma_buf,
> + .mmap = vc_sm_import_dmabuf_mmap,
> + .release = vc_sm_dma_buf_release,
> + .attach = vc_sm_import_dma_buf_attach,
> + .detach = vc_sm_import_dma_buf_detatch,
> + .begin_cpu_access = vc_sm_import_dma_buf_begin_cpu_access,
> + .end_cpu_access = vc_sm_import_dma_buf_end_cpu_access,
> +};
> +
> +/* Import a dma_buf to be shared with VC. */
> +int
> +vc_sm_cma_import_dmabuf_internal(struct sm_state_t *state,
> + struct dma_buf *dma_buf,
> + int fd,
> + struct dma_buf **imported_buf)
This can be static.
> +{
> + DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
> + struct vc_sm_buffer *buffer = NULL;
> + struct vc_sm_import import = { };
> + struct vc_sm_import_result result = { };
> + struct dma_buf_attachment *attach = NULL;
> + struct sg_table *sgt = NULL;
> + dma_addr_t dma_addr;
> + int ret = 0;
> + int status;
> +
> + /* Setup our allocation parameters */
> + pr_debug("%s: importing dma_buf %p/fd %d\n", __func__, dma_buf, fd);
> +
> + if (fd < 0)
> + get_dma_buf(dma_buf);
> + else
> + dma_buf = dma_buf_get(fd);
This function is always called with fd == -1. There seems to be quite a
bit of dead code in this driver, could you double-check everything and
trim it down ?
> +
> + if (!dma_buf)
> + return -EINVAL;
> +
> + attach = dma_buf_attach(dma_buf, &sm_state->pdev->dev);
> + if (IS_ERR(attach)) {
> + ret = PTR_ERR(attach);
> + goto error;
> + }
> +
> + sgt = dma_buf_map_attachment(attach, DMA_BIDIRECTIONAL);
> + if (IS_ERR(sgt)) {
> + ret = PTR_ERR(sgt);
> + goto error;
> + }
> +
> + /* Verify that the address block is contiguous */
> + if (sgt->nents != 1) {
> + ret = -ENOMEM;
> + goto error;
> + }
> +
> + /* Allocate local buffer to track this allocation. */
> + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL);
> + if (!buffer) {
> + ret = -ENOMEM;
> + goto error;
> + }
> +
> + import.type = VC_SM_ALLOC_NON_CACHED;
> + dma_addr = sg_dma_address(sgt->sgl);
> + import.addr = (u32)dma_addr;
> + if ((import.addr & 0xC0000000) != 0xC0000000) {
> + pr_err("%s: Expecting an uncached alias for dma_addr %pad\n",
> + __func__, &dma_addr);
> + import.addr |= 0xC0000000;
> + }
> + import.size = sg_dma_len(sgt->sgl);
> + import.allocator = current->tgid;
> + import.kernel_id = get_kernel_id(buffer);
> +
> + memcpy(import.name, VC_SM_RESOURCE_NAME_DEFAULT,
> + sizeof(VC_SM_RESOURCE_NAME_DEFAULT));
> +
> + pr_debug("[%s]: attempt to import \"%s\" data - type %u, addr %pad, size %u.\n",
> + __func__, import.name, import.type, &dma_addr, import.size);
> +
> + /* Allocate the videocore buffer. */
> + status = vc_sm_cma_vchi_import(sm_state->sm_handle, &import, &result,
> + &sm_state->int_trans_id);
> + if (status == -EINTR) {
> + pr_debug("[%s]: requesting import memory action restart (trans_id: %u)\n",
> + __func__, sm_state->int_trans_id);
> + ret = -ERESTARTSYS;
> + sm_state->restart_sys = -EINTR;
> + sm_state->int_action = VC_SM_MSG_TYPE_IMPORT;
> + goto error;
> + } else if (status || !result.res_handle) {
> + pr_debug("[%s]: failed to import memory on videocore (status: %u, trans_id: %u)\n",
> + __func__, status, sm_state->int_trans_id);
> + ret = -ENOMEM;
> + goto error;
> + }
> +
> + mutex_init(&buffer->lock);
> + INIT_LIST_HEAD(&buffer->attachments);
> + memcpy(buffer->name, import.name,
> + min(sizeof(buffer->name), sizeof(import.name) - 1));
> +
> + /* Keep track of the buffer we created. */
> + buffer->vc_handle = result.res_handle;
> + buffer->size = import.size;
> + buffer->vpu_state = VPU_MAPPED;
> +
> + buffer->imported_dma_buf = dma_buf;
> +
> + buffer->attach = attach;
> + buffer->sgt = sgt;
> + buffer->dma_addr = dma_addr;
> + buffer->in_use = 1;
> + buffer->kernel_id = import.kernel_id;
> +
> + /*
> + * We're done - we need to export a new dmabuf chaining through most
> + * functions, but enabling us to release our own internal references
> + * here.
> + */
Why do we need to export a new dmabuf ?
> + exp_info.ops = &dma_buf_import_ops;
> + exp_info.size = import.size;
> + exp_info.flags = O_RDWR;
> + exp_info.priv = buffer;
> +
> + buffer->dma_buf = dma_buf_export(&exp_info);
> + if (IS_ERR(buffer->dma_buf)) {
> + ret = PTR_ERR(buffer->dma_buf);
> + goto error;
> + }
> +
> + vc_sm_add_resource(buffer);
> +
> + *imported_buf = buffer->dma_buf;
> +
> + return 0;
> +
> +error:
> + if (result.res_handle) {
> + struct vc_sm_free_t free = { result.res_handle, 0 };
> +
> + vc_sm_cma_vchi_free(sm_state->sm_handle, &free,
> + &sm_state->int_trans_id);
> + }
> + free_kernel_id(import.kernel_id);
> + kfree(buffer);
> + if (sgt)
> + dma_buf_unmap_attachment(attach, sgt, DMA_BIDIRECTIONAL);
> + if (attach)
> + dma_buf_detach(dma_buf, attach);
> + dma_buf_put(dma_buf);
> + return ret;
> +}
> +
> +static void
> +vc_sm_vpu_event(struct sm_instance *instance, struct vc_sm_result_t *reply,
> + int reply_len)
> +{
> + switch (reply->trans_id & ~0x80000000) {
> + case VC_SM_MSG_TYPE_CLIENT_VERSION:
> + {
> + /* Acknowledge that the firmware supports the version command */
> + pr_debug("%s: firmware acked version msg. Require release cb\n",
> + __func__);
> + sm_state->require_released_callback = true;
> + }
> + break;
No need for braces.
> + case VC_SM_MSG_TYPE_RELEASED:
> + {
> + struct vc_sm_released *release = (struct vc_sm_released *)reply;
> + struct vc_sm_buffer *buffer =
> + lookup_kernel_id(release->kernel_id);
> + if (!buffer) {
> + pr_err("%s: VC released a buffer that is already released, kernel_id %d\n",
> + __func__, release->kernel_id);
> + break;
> + }
> + mutex_lock(&buffer->lock);
> +
> + pr_debug("%s: Released addr %08x, size %u, id %08x, mem_handle %08x\n",
> + __func__, release->addr, release->size,
> + release->kernel_id, release->vc_handle);
> +
> + buffer->vc_handle = 0;
> + buffer->vpu_state = VPU_NOT_MAPPED;
> + free_kernel_id(release->kernel_id);
> +
> + vc_sm_release_resource(buffer);
> + }
> + break;
> + default:
> + pr_err("%s: Unknown vpu cmd %x\n", __func__, reply->trans_id);
> + break;
> + }
> +}
> +
> +/* Driver load/unload functions */
> +/* Videocore connected. */
> +static void vc_sm_connected_init(void)
> +{
> + int ret;
> + struct vc_sm_version version;
> + struct vc_sm_result_t version_result;
> +
> + pr_info("[%s]: start\n", __func__);
> +
> + /*
> + * Initialize and create a VCHI connection for the shared memory service
> + * running on videocore.
> + */
> + ret = vchiq_initialise(&sm_state->vchiq_instance);
> + if (ret) {
> + pr_err("[%s]: failed to initialise VCHI instance (ret=%d)\n",
> + __func__, ret);
> +
> + return;
> + }
> +
> + ret = vchiq_connect(sm_state->vchiq_instance);
> + if (ret) {
> + pr_err("[%s]: failed to connect VCHI instance (ret=%d)\n",
> + __func__, ret);
> +
> + return;
> + }
> +
> + /* Initialize an instance of the shared memory service. */
> + sm_state->sm_handle = vc_sm_cma_vchi_init(sm_state->vchiq_instance, 1,
> + vc_sm_vpu_event);
> + if (!sm_state->sm_handle) {
> + pr_err("[%s]: failed to initialize shared memory service\n",
> + __func__);
> +
> + return;
> + }
> +
> + /* Create a debug fs directory entry (root). */
> + sm_state->dir_root = debugfs_create_dir(VC_SM_DIR_ROOT_NAME, NULL);
> +
> + sm_state->dir_state.show = &vc_sm_cma_global_state_show;
> + sm_state->dir_state.dir_entry =
> + debugfs_create_file(VC_SM_STATE, 0444, sm_state->dir_root,
> + &sm_state->dir_state,
> + &vc_sm_cma_debug_fs_fops);
Move this to a separate debugfs init function, grouped with the rest of
the debugfs code. debugfs is conditioned by CONFIG_DEBUG_FS, so you'll
need conditional compilation. You will also need to handle cleanup, you
never destroy the file and directory.
> +
> + INIT_LIST_HEAD(&sm_state->buffer_list);
> +
> + version.version = 2;
> + ret = vc_sm_cma_vchi_client_version(sm_state->sm_handle, &version,
> + &version_result,
> + &sm_state->int_trans_id);
> + if (ret) {
> + pr_err("[%s]: Failed to send version request %d\n", __func__,
> + ret);
> + }
> +
> + /* Done! */
> + sm_inited = 1;
> + pr_info("[%s]: installed successfully\n", __func__);
> +}
> +
> +/* Driver loading. */
> +static int bcm2835_vc_sm_cma_probe(struct platform_device *pdev)
> +{
> + pr_info("%s: Videocore shared memory driver\n", __func__);
Drop this, it only clutters the kernel log.
> +
> + sm_state = devm_kzalloc(&pdev->dev, sizeof(*sm_state), GFP_KERNEL);
> + if (!sm_state)
> + return -ENOMEM;
> + sm_state->pdev = pdev;
> + mutex_init(&sm_state->map_lock);
> +
> + spin_lock_init(&sm_state->kernelid_map_lock);
> + idr_init_base(&sm_state->kernelid_map, 1);
> +
> + pdev->dev.dma_parms = devm_kzalloc(&pdev->dev,
> + sizeof(*pdev->dev.dma_parms),
> + GFP_KERNEL);
> + /* dma_set_max_seg_size checks if dma_parms is NULL. */
> + dma_set_max_seg_size(&pdev->dev, 0x3FFFFFFF);
> +
> + vchiq_add_connected_callback(vc_sm_connected_init);
> + return 0;
> +}
> +
> +/* Driver unloading. */
> +static int bcm2835_vc_sm_cma_remove(struct platform_device *pdev)
> +{
> + pr_debug("[%s]: start\n", __func__);
You can drop this message and the one at the end of the function.
> + if (sm_inited) {
> + misc_deregister(&sm_state->misc_dev);
There's no misc_register() call, something seems wrong. Probably
leftover code ?
> +
> + /* Remove all proc entries. */
> + debugfs_remove_recursive(sm_state->dir_root);
> +
> + /* Stop the videocore shared memory service. */
> + vc_sm_cma_vchi_stop(sm_state->vchiq_instance, &sm_state->sm_handle);
> + }
> +
> + if (sm_state) {
Drop the condition, sm_state can never be NULL if probe() succeeded, and
if it hasn't succeeded, then remove() will not be called.
> + idr_destroy(&sm_state->kernelid_map);
> +
> + /* Free the memory for the state structure. */
> + mutex_destroy(&sm_state->map_lock);
> + }
> +
> + pr_debug("[%s]: end\n", __func__);
> + return 0;
> +}
> +
> +/* Kernel API calls */
> +/* Get an internal resource handle mapped from the external one. */
> +int vc_sm_cma_int_handle(void *handle)
> +{
> + struct dma_buf *dma_buf = (struct dma_buf *)handle;
> + struct vc_sm_buffer *buf;
> +
> + /* Validate we can work with this device. */
> + if (!sm_state || !handle) {
> + pr_err("[%s]: invalid input\n", __func__);
> + return 0;
> + }
> +
> + buf = (struct vc_sm_buffer *)dma_buf->priv;
> + return buf->vc_handle;
> +}
> +EXPORT_SYMBOL_GPL(vc_sm_cma_int_handle);
> +
> +/* Free a previously allocated shared memory handle and block. */
> +int vc_sm_cma_free(void *handle)
> +{
> + struct dma_buf *dma_buf = (struct dma_buf *)handle;
> +
> + /* Validate we can work with this device. */
> + if (!sm_state || !handle) {
> + pr_err("[%s]: invalid input\n", __func__);
> + return -EPERM;
That's a weird error code for an invalid input.
> + }
> +
> + pr_debug("%s: handle %p/dmabuf %p\n", __func__, handle, dma_buf);
> +
> + dma_buf_put(dma_buf);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(vc_sm_cma_free);
> +
> +/* Import a dmabuf to be shared with VC. */
> +int vc_sm_cma_import_dmabuf(struct dma_buf *src_dmabuf, void **handle)
> +{
> + struct dma_buf *new_dma_buf;
> + int ret;
> +
> + /* Validate we can work with this device. */
> + if (!sm_state || !src_dmabuf || !handle) {
> + pr_err("[%s]: invalid input\n", __func__);
> + return -EPERM;
Same here.
> + }
> +
> + ret = vc_sm_cma_import_dmabuf_internal(sm_state, src_dmabuf,
> + -1, &new_dma_buf);
> +
> + if (!ret) {
> + pr_debug("%s: imported to ptr %p\n", __func__, new_dma_buf);
> +
> + /* Assign valid handle at this time.*/
> + *handle = new_dma_buf;
> + } else {
> + /*
> + * succeeded in importing the dma_buf, but then
> + * failed to look it up again. How?
> + * Release the fd again.
> + */
> + pr_err("%s: imported vc_sm_cma_get_buffer failed %d\n",
> + __func__, ret);
> + }
> +
> + return ret;
> +}
> +EXPORT_SYMBOL_GPL(vc_sm_cma_import_dmabuf);
Those three functions are a bit ill-placed between the probe/remove
functions and the platform_driver. I'd move them just above
vc_sm_vpu_event(), and order them as mentioned below in the header file.
> +
> +static struct platform_driver bcm2835_vcsm_cma_driver = {
> + .probe = bcm2835_vc_sm_cma_probe,
> + .remove = bcm2835_vc_sm_cma_remove,
> + .driver = {
> + .name = "vcsm-cma",
> + .owner = THIS_MODULE,
> + },
Wrong indentation, should be
.driver = {
.name = "vcsm-cma",
.owner = THIS_MODULE,
},
> +};
> +
> +module_platform_driver(bcm2835_vcsm_cma_driver);
> +
> +MODULE_AUTHOR("Dave Stevenson");
> +MODULE_DESCRIPTION("VideoCore CMA Shared Memory Driver");
> +MODULE_LICENSE("GPL");
> +MODULE_ALIAS("platform:vcsm-cma");
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> new file mode 100644
> index 000000000000..61e110ec414d
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> @@ -0,0 +1,54 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * VideoCore Shared Memory driver using CMA.
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + *
> + */
> +
> +#ifndef VC_SM_H
> +#define VC_SM_H
> +
> +#define VC_SM_MAX_NAME_LEN 32
> +
> +enum vc_sm_vpu_mapping_state {
> + VPU_NOT_MAPPED,
> + VPU_MAPPED,
> + VPU_UNMAPPING
> +};
> +
> +struct vc_sm_buffer {
> + struct list_head global_buffer_list; /* Global list of buffers. */
> +
> + /* Index in the kernel_id idr so that we can find the
> + * mmal_msg_context again when servicing the VCHI reply.
> + */
> + int kernel_id;
> +
> + size_t size;
> +
> + /* Lock over all the following state for this buffer */
> + struct mutex lock;
> + struct list_head attachments;
> +
> + char name[VC_SM_MAX_NAME_LEN];
> +
> + int in_use:1; /* Kernel is still using this resource */
> +
> + enum vc_sm_vpu_mapping_state vpu_state;
> + u32 vc_handle; /* VideoCore handle for this buffer */
> +
> + /* DMABUF related fields */
> + struct dma_buf *dma_buf;
> + dma_addr_t dma_addr;
> + void *cookie;
> +
> + struct vc_sm_privdata_t *private;
> +
> + struct dma_buf *imported_dma_buf;
> + struct dma_buf_attachment *attach;
> + struct sg_table *sgt;
> +};
> +
> +#endif
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> new file mode 100644
> index 000000000000..c77ef0998a31
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> @@ -0,0 +1,507 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * VideoCore Shared Memory CMA allocator
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + * Copyright 2011-2012 Broadcom Corporation. All rights reserved.
> + *
> + * Based on vmcs_sm driver from Broadcom Corporation.
> + *
> + */
> +
> +/* ---- Include Files ----------------------------------------------------- */
> +#include <linux/kthread.h>
> +#include <linux/list.h>
> +#include <linux/semaphore.h>
> +#include <linux/slab.h>
> +#include <linux/types.h>
> +
> +#include "vc_sm_cma_vchi.h"
> +
> +#define VC_SM_VER 1
> +#define VC_SM_MIN_VER 0
> +
> +/* ---- Private Constants and Types -------------------------------------- */
> +
> +/* Command blocks come from a pool */
> +#define SM_MAX_NUM_CMD_RSP_BLKS 32
> +
> +/* The number of supported connections */
> +#define SM_MAX_NUM_CONNECTIONS 3
> +
> +struct sm_cmd_rsp_blk {
> + struct list_head head; /* To create lists */
> + /* To be signaled when the response is there */
> + struct completion cmplt;
> +
> + u32 id;
> + u16 length;
> +
> + u8 msg[VC_SM_MAX_MSG_LEN];
> +
> + uint32_t wait:1;
> + uint32_t sent:1;
> + uint32_t alloc:1;
> +
> +};
> +
> +struct sm_instance {
> + u32 num_connections;
> + unsigned int service_handle[SM_MAX_NUM_CONNECTIONS];
> + struct task_struct *io_thread;
> + struct completion io_cmplt;
> +
> + vpu_event_cb vpu_event;
> +
> + /* Mutex over the following lists */
> + struct mutex lock;
> + u32 trans_id;
> + struct list_head cmd_list;
> + struct list_head rsp_list;
> + struct list_head dead_list;
> +
> + struct sm_cmd_rsp_blk free_blk[SM_MAX_NUM_CMD_RSP_BLKS];
> +
> + /* Mutex over the free_list */
> + struct mutex free_lock;
> + struct list_head free_list;
> +
> + struct semaphore free_sema;
> + struct vchiq_instance *vchiq_instance;
> +};
> +
> +/* ---- Private Variables ------------------------------------------------ */
> +
> +/* ---- Private Function Prototypes -------------------------------------- */
> +
> +/* ---- Private Functions ------------------------------------------------ */
> +static int
> +bcm2835_vchi_msg_queue(struct vchiq_instance *vchiq_instance, unsigned int handle,
> + void *data,
> + unsigned int size)
> +{
> + return vchiq_queue_kernel_message(vchiq_instance, handle, data, size);
> +}
> +
> +static struct
> +sm_cmd_rsp_blk *vc_vchi_cmd_create(struct sm_instance *instance,
> + enum vc_sm_msg_type id, void *msg,
> + u32 size, int wait)
> +{
> + struct sm_cmd_rsp_blk *blk;
> + struct vc_sm_msg_hdr_t *hdr;
> +
> + if (down_interruptible(&instance->free_sema)) {
> + blk = kmalloc(sizeof(*blk), GFP_KERNEL);
> + if (!blk)
> + return NULL;
> +
> + blk->alloc = 1;
> + init_completion(&blk->cmplt);
> + } else {
> + mutex_lock(&instance->free_lock);
> + blk =
> + list_first_entry(&instance->free_list,
> + struct sm_cmd_rsp_blk, head);
> + list_del(&blk->head);
> + mutex_unlock(&instance->free_lock);
> + }
> +
> + blk->sent = 0;
> + blk->wait = wait;
> + blk->length = sizeof(*hdr) + size;
> +
> + hdr = (struct vc_sm_msg_hdr_t *)blk->msg;
> + hdr->type = id;
> + mutex_lock(&instance->lock);
> + instance->trans_id++;
> + /*
> + * Retain the top bit for identifying asynchronous events, or VPU cmds.
> + */
> + instance->trans_id &= ~0x80000000;
> + hdr->trans_id = instance->trans_id;
> + blk->id = instance->trans_id;
> + mutex_unlock(&instance->lock);
> +
> + if (size)
> + memcpy(hdr->body, msg, size);
> +
> + return blk;
> +}
> +
> +static void
> +vc_vchi_cmd_delete(struct sm_instance *instance, struct sm_cmd_rsp_blk *blk)
> +{
> + if (blk->alloc) {
> + kfree(blk);
> + return;
> + }
> +
> + mutex_lock(&instance->free_lock);
> + list_add(&blk->head, &instance->free_list);
> + mutex_unlock(&instance->free_lock);
> + up(&instance->free_sema);
> +}
> +
> +static void vc_sm_cma_vchi_rx_ack(struct sm_instance *instance,
> + struct sm_cmd_rsp_blk *cmd,
> + struct vc_sm_result_t *reply,
> + u32 reply_len)
> +{
> + mutex_lock(&instance->lock);
> + list_for_each_entry(cmd,
> + &instance->rsp_list,
> + head) {
> + if (cmd->id == reply->trans_id)
> + break;
> + }
> + mutex_unlock(&instance->lock);
> +
> + if (&cmd->head == &instance->rsp_list) {
> + //pr_debug("%s: received response %u, throw away...",
> + pr_err("%s: received response %u, throw away...",
> + __func__,
> + reply->trans_id);
> + } else if (reply_len > sizeof(cmd->msg)) {
> + pr_err("%s: reply too big (%u) %u, throw away...",
> + __func__, reply_len,
> + reply->trans_id);
> + } else {
> + memcpy(cmd->msg, reply,
> + reply_len);
> + complete(&cmd->cmplt);
> + }
> +}
> +
> +static int vc_sm_cma_vchi_videocore_io(void *arg)
> +{
> + struct sm_instance *instance = arg;
> + struct sm_cmd_rsp_blk *cmd = NULL, *cmd_tmp;
> + struct vc_sm_result_t *reply;
> + struct vchiq_header *header;
> + s32 status;
> + int svc_use = 1;
> +
> + while (1) {
> + if (svc_use)
> + vchiq_release_service(instance->vchiq_instance,
> + instance->service_handle[0]);
> + svc_use = 0;
> +
> + if (wait_for_completion_interruptible(&instance->io_cmplt))
> + continue;
Isn't it a bit overkill to use a thread, wouldn't a workqueue do ?
> + vchiq_use_service(instance->vchiq_instance, instance->service_handle[0]);
> + svc_use = 1;
> +
> + do {
> + /*
> + * Get new command and move it to response list
> + */
> + mutex_lock(&instance->lock);
> + if (list_empty(&instance->cmd_list)) {
> + /* no more commands to process */
> + mutex_unlock(&instance->lock);
> + break;
> + }
> + cmd = list_first_entry(&instance->cmd_list,
> + struct sm_cmd_rsp_blk, head);
> + list_move(&cmd->head, &instance->rsp_list);
> + cmd->sent = 1;
> + mutex_unlock(&instance->lock);
> + /* Send the command */
> + status =
> + bcm2835_vchi_msg_queue(instance->vchiq_instance,
> + instance->service_handle[0],
> + cmd->msg, cmd->length);
> + if (status) {
> + pr_err("%s: failed to queue message (%d)",
> + __func__, status);
> + }
> +
> + /* If no reply is needed then we're done */
> + if (!cmd->wait) {
> + mutex_lock(&instance->lock);
> + list_del(&cmd->head);
> + mutex_unlock(&instance->lock);
> + vc_vchi_cmd_delete(instance, cmd);
> + continue;
> + }
> +
> + if (status) {
> + complete(&cmd->cmplt);
> + continue;
> + }
> +
> + } while (1);
> +
> + while ((header = vchiq_msg_hold(instance->vchiq_instance,
> + instance->service_handle[0]))) {
> + reply = (struct vc_sm_result_t *)header->data;
> + if (reply->trans_id & 0x80000000) {
> + /* Async event or cmd from the VPU */
> + if (instance->vpu_event)
> + instance->vpu_event(instance, reply,
> + header->size);
> + } else {
> + vc_sm_cma_vchi_rx_ack(instance, cmd, reply,
> + header->size);
> + }
> +
> + vchiq_release_message(instance->vchiq_instance,
> + instance->service_handle[0],
> + header);
> + }
> +
> + /* Go through the dead list and free them */
> + mutex_lock(&instance->lock);
> + list_for_each_entry_safe(cmd, cmd_tmp, &instance->dead_list,
> + head) {
> + list_del(&cmd->head);
> + vc_vchi_cmd_delete(instance, cmd);
> + }
> + mutex_unlock(&instance->lock);
> + }
> +
> + return 0;
> +}
> +
> +static enum vchiq_status vc_sm_cma_vchi_callback(struct vchiq_instance *vchiq_instance,
> + enum vchiq_reason reason,
> + struct vchiq_header *header,
> + unsigned int handle, void *userdata)
> +{
> + struct sm_instance *instance = vchiq_get_service_userdata(vchiq_instance, handle);
> +
> + switch (reason) {
> + case VCHIQ_MESSAGE_AVAILABLE:
> + vchiq_msg_queue_push(vchiq_instance, handle, header);
> + complete(&instance->io_cmplt);
> + break;
> +
> + case VCHIQ_SERVICE_CLOSED:
> + pr_info("%s: service CLOSED!!", __func__);
> + default:
> + break;
> + }
> +
> + return VCHIQ_SUCCESS;
> +}
> +
> +struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchiq_instance,
> + unsigned int num_connections,
> + vpu_event_cb vpu_event)
> +{
> + u32 i;
> + struct sm_instance *instance;
> + int status;
> +
> + pr_debug("%s: start", __func__);
> +
> + if (num_connections > SM_MAX_NUM_CONNECTIONS) {
> + pr_err("%s: unsupported number of connections %u (max=%u)",
> + __func__, num_connections, SM_MAX_NUM_CONNECTIONS);
> +
> + goto err_null;
> + }
> + /* Allocate memory for this instance */
> + instance = kzalloc(sizeof(*instance), GFP_KERNEL);
> +
> + /* Misc initialisations */
> + mutex_init(&instance->lock);
> + init_completion(&instance->io_cmplt);
> + INIT_LIST_HEAD(&instance->cmd_list);
> + INIT_LIST_HEAD(&instance->rsp_list);
> + INIT_LIST_HEAD(&instance->dead_list);
> + INIT_LIST_HEAD(&instance->free_list);
> + sema_init(&instance->free_sema, SM_MAX_NUM_CMD_RSP_BLKS);
> + mutex_init(&instance->free_lock);
> + for (i = 0; i < SM_MAX_NUM_CMD_RSP_BLKS; i++) {
> + init_completion(&instance->free_blk[i].cmplt);
> + list_add(&instance->free_blk[i].head, &instance->free_list);
> + }
> +
> + instance->vchiq_instance = vchiq_instance;
> +
> + /* Open the VCHI service connections */
> + instance->num_connections = num_connections;
> + for (i = 0; i < num_connections; i++) {
> + struct vchiq_service_params_kernel params = {
> + .version = VC_SM_VER,
> + .version_min = VC_SM_MIN_VER,
> + .fourcc = VCHIQ_MAKE_FOURCC('S', 'M', 'E', 'M'),
> + .callback = vc_sm_cma_vchi_callback,
> + .userdata = instance,
> + };
> +
> + status = vchiq_open_service(vchiq_instance, ¶ms,
> + &instance->service_handle[i]);
> + if (status) {
> + pr_err("%s: failed to open VCHI service (%d)",
> + __func__, status);
> +
> + goto err_close_services;
> + }
> + }
> + /* Create the thread which takes care of all io to/from videoocore. */
> + instance->io_thread = kthread_create(&vc_sm_cma_vchi_videocore_io,
> + (void *)instance, "SMIO");
> + if (!instance->io_thread) {
> + pr_err("%s: failed to create SMIO thread", __func__);
> +
> + goto err_close_services;
> + }
> + instance->vpu_event = vpu_event;
> + set_user_nice(instance->io_thread, -10);
> + wake_up_process(instance->io_thread);
> +
> + pr_debug("%s: success - instance %p", __func__, instance);
> + return instance;
> +
> +err_close_services:
> + for (i = 0; i < instance->num_connections; i++) {
> + if (instance->service_handle[i])
> + vchiq_close_service(vchiq_instance, instance->service_handle[i]);
> + }
> + kfree(instance);
> +err_null:
> + pr_debug("%s: FAILED", __func__);
> + return NULL;
> +}
> +
> +int vc_sm_cma_vchi_stop(struct vchiq_instance *vchiq_instance, struct sm_instance **handle)
> +{
> + struct sm_instance *instance;
> + u32 i;
> +
> + if (!handle) {
> + pr_err("%s: invalid pointer to handle %p", __func__, handle);
> + goto lock;
> + }
> +
> + if (!*handle) {
> + pr_err("%s: invalid handle %p", __func__, *handle);
> + goto lock;
> + }
> +
> + instance = *handle;
> +
> + /* Close all VCHI service connections */
> + for (i = 0; i < instance->num_connections; i++) {
> + vchiq_use_service(vchiq_instance, instance->service_handle[i]);
> + vchiq_close_service(vchiq_instance, instance->service_handle[i]);
> + }
> +
> + kfree(instance);
> +
> + *handle = NULL;
> + return 0;
> +
> +lock:
> + return -EINVAL;
> +}
> +
> +static int vc_sm_cma_vchi_send_msg(struct sm_instance *handle,
> + enum vc_sm_msg_type msg_id, void *msg,
> + u32 msg_size, void *result, u32 result_size,
> + u32 *cur_trans_id, u8 wait_reply)
> +{
> + int status = 0;
> + struct sm_instance *instance = handle;
> + struct sm_cmd_rsp_blk *cmd_blk;
> +
> + if (!handle) {
> + pr_err("%s: invalid handle", __func__);
> + return -EINVAL;
> + }
> + if (!msg) {
> + pr_err("%s: invalid msg pointer", __func__);
> + return -EINVAL;
> + }
> +
> + cmd_blk =
> + vc_vchi_cmd_create(instance, msg_id, msg, msg_size, wait_reply);
> + if (!cmd_blk) {
> + pr_err("[%s]: failed to allocate global tracking resource",
> + __func__);
> + return -ENOMEM;
> + }
> +
> + if (cur_trans_id)
> + *cur_trans_id = cmd_blk->id;
> +
> + mutex_lock(&instance->lock);
> + list_add_tail(&cmd_blk->head, &instance->cmd_list);
> + mutex_unlock(&instance->lock);
> + complete(&instance->io_cmplt);
> +
> + if (!wait_reply)
> + /* We're done */
> + return 0;
> +
> + /* Wait for the response */
> + if (wait_for_completion_interruptible(&cmd_blk->cmplt)) {
> + mutex_lock(&instance->lock);
> + if (!cmd_blk->sent) {
> + list_del(&cmd_blk->head);
> + mutex_unlock(&instance->lock);
> + vc_vchi_cmd_delete(instance, cmd_blk);
> + return -ENXIO;
> + }
> +
> + list_move(&cmd_blk->head, &instance->dead_list);
> + mutex_unlock(&instance->lock);
> + complete(&instance->io_cmplt);
> + return -EINTR; /* We're done */
> + }
> +
> + if (result && result_size) {
> + memcpy(result, cmd_blk->msg, result_size);
> + } else {
> + struct vc_sm_result_t *res =
> + (struct vc_sm_result_t *)cmd_blk->msg;
> + status = (res->success == 0) ? 0 : -ENXIO;
> + }
> +
> + mutex_lock(&instance->lock);
> + list_del(&cmd_blk->head);
> + mutex_unlock(&instance->lock);
> + vc_vchi_cmd_delete(instance, cmd_blk);
> + return status;
> +}
> +
> +int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg,
> + u32 *cur_trans_id)
> +{
> + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_FREE,
> + msg, sizeof(*msg), 0, 0, cur_trans_id, 0);
> +}
> +
> +int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg,
> + struct vc_sm_import_result *result, u32 *cur_trans_id)
> +{
> + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_IMPORT,
> + msg, sizeof(*msg), result, sizeof(*result),
> + cur_trans_id, 1);
> +}
> +
> +int vc_sm_cma_vchi_client_version(struct sm_instance *handle,
> + struct vc_sm_version *msg,
> + struct vc_sm_result_t *result,
> + u32 *cur_trans_id)
> +{
> + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_CLIENT_VERSION,
> + //msg, sizeof(*msg), result, sizeof(*result),
> + //cur_trans_id, 1);
> + msg, sizeof(*msg), NULL, 0,
> + cur_trans_id, 0);
> +}
> +
> +int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle,
> + struct vc_sm_vc_mem_request_result *msg,
> + uint32_t *cur_trans_id)
> +{
> + return vc_sm_cma_vchi_send_msg(handle,
> + VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY,
> + msg, sizeof(*msg), 0, 0, cur_trans_id,
> + 0);
> +}
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> new file mode 100644
> index 000000000000..a4f40d4cef05
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> @@ -0,0 +1,63 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * VideoCore Shared Memory CMA allocator
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + * Copyright 2011-2012 Broadcom Corporation. All rights reserved.
> + *
> + * Based on vmcs_sm driver from Broadcom Corporation.
> + *
> + */
> +
> +#ifndef __VC_SM_CMA_VCHI_H__INCLUDED__
This is quite inconsistent with the existing coding style of
vc04_services and of the kernel as a whole. I'd go for VC_SM_CMA_VCHI_H,
or if preferred, __VC_SM_CMA_VCHI_H or __VC_SM_CMA_VCHI_H__.
> +#define __VC_SM_CMA_VCHI_H__INCLUDED__
> +
> +#include <linux/raspberrypi/vchiq.h>
> +
> +#include "vc_sm_defs.h"
> +
> +/*
> + * Forward declare.
> + */
Drop this.
> +struct sm_instance;
> +
> +typedef void (*vpu_event_cb)(struct sm_instance *instance,
> + struct vc_sm_result_t *reply, int reply_len);
> +
> +/*
> + * Initialize the shared memory service, opens up vchi connection to talk to it.
> + */
> +struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchi_instance,
> + unsigned int num_connections,
> + vpu_event_cb vpu_event);
> +
> +/*
> + * Terminates the shared memory service.
> + */
> +int vc_sm_cma_vchi_stop(struct vchiq_instance *vchi_instance, struct sm_instance **handle);
> +
> +/*
> + * Ask the shared memory service to free up some memory that was previously
> + * allocated by the vc_sm_cma_vchi_alloc function call.
> + */
> +int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg,
> + u32 *cur_trans_id);
> +
> +/*
> + * Import a contiguous block of memory and wrap it in a GPU MEM_HANDLE_T.
> + */
> +int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg,
> + struct vc_sm_import_result *result,
> + u32 *cur_trans_id);
> +
> +int vc_sm_cma_vchi_client_version(struct sm_instance *handle,
> + struct vc_sm_version *msg,
> + struct vc_sm_result_t *result,
> + u32 *cur_trans_id);
> +
> +int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle,
> + struct vc_sm_vc_mem_request_result *msg,
> + uint32_t *cur_trans_id);
> +
> +#endif /* __VC_SM_CMA_VCHI_H__INCLUDED__ */
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> new file mode 100644
> index 000000000000..ad4a3ec194d3
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> @@ -0,0 +1,187 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * VideoCore Shared Memory CMA allocator
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + *
> + * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation.
> + * All IPC messages are copied across to this file, even if the vc-sm-cma
> + * driver is not currently using them.
> + *
> + ****************************************************************************
> + */
> +
> +#ifndef __VC_SM_DEFS_H__INCLUDED__
> +#define __VC_SM_DEFS_H__INCLUDED__
> +
> +/* Maximum message length */
> +#define VC_SM_MAX_MSG_LEN (sizeof(union vc_sm_msg_union_t) + \
> + sizeof(struct vc_sm_msg_hdr_t))
> +#define VC_SM_MAX_RSP_LEN (sizeof(union vc_sm_msg_union_t))
> +
> +/* Resource name maximum size */
> +#define VC_SM_RESOURCE_NAME 32
> +
> +/*
> + * Version to be reported to the VPU
> + * VPU assumes 0 (aka 1) which does not require the released callback, nor
> + * expect the client to handle VC_MEM_REQUESTS.
> + * Version 2 requires the released callback, and must support VC_MEM_REQUESTS.
> + */
> +#define VC_SM_PROTOCOL_VERSION 2
> +
> +enum vc_sm_msg_type {
> + /* Message types supported for HOST->VC direction */
> +
> + /* Allocate shared memory block */
> + VC_SM_MSG_TYPE_ALLOC,
> + /* Lock allocated shared memory block */
> + VC_SM_MSG_TYPE_LOCK,
> + /* Unlock allocated shared memory block */
> + VC_SM_MSG_TYPE_UNLOCK,
> + /* Unlock allocated shared memory block, do not answer command */
> + VC_SM_MSG_TYPE_UNLOCK_NOANS,
> + /* Free shared memory block */
> + VC_SM_MSG_TYPE_FREE,
> + /* Resize a shared memory block */
> + VC_SM_MSG_TYPE_RESIZE,
> + /* Walk the allocated shared memory block(s) */
> + VC_SM_MSG_TYPE_WALK_ALLOC,
> +
> + /* A previously applied action will need to be reverted */
> + VC_SM_MSG_TYPE_ACTION_CLEAN,
> +
> + /*
> + * Import a physical address and wrap into a MEM_HANDLE_T.
> + * Release with VC_SM_MSG_TYPE_FREE.
> + */
> + VC_SM_MSG_TYPE_IMPORT,
> + /*
> + *Tells VC the protocol version supported by this client.
> + * 2 supports the async/cmd messages from the VPU for final release
> + * of memory, and for VC allocations.
> + */
> + VC_SM_MSG_TYPE_CLIENT_VERSION,
> + /* Response to VC request for memory */
> + VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY,
> +
> + /*
> + * Asynchronous/cmd messages supported for VC->HOST direction.
> + * Signalled by setting the top bit in vc_sm_result_t trans_id.
> + */
> +
> + /*
> + * VC has finished with an imported memory allocation.
> + * Release any Linux reference counts on the underlying block.
> + */
> + VC_SM_MSG_TYPE_RELEASED,
> + /* VC request for memory */
> + VC_SM_MSG_TYPE_VC_MEM_REQUEST,
> +
> + VC_SM_MSG_TYPE_MAX
> +};
> +
> +/* Type of memory to be allocated */
> +enum vc_sm_alloc_type_t {
> + VC_SM_ALLOC_CACHED,
> + VC_SM_ALLOC_NON_CACHED,
> +};
> +
> +/* Message header for all messages in HOST->VC direction */
> +struct vc_sm_msg_hdr_t {
> + u32 type;
> + u32 trans_id;
> + u8 body[0];
> +
> +};
> +
> +/* Request to free a previously allocated memory (HOST->VC) */
> +struct vc_sm_free_t {
> + /* Resource handle (returned from alloc) */
> + u32 res_handle;
> + /* Resource buffer (returned from alloc) */
> + u32 res_mem;
> +
> +};
> +
> +/* Generic result for a request (VC->HOST) */
> +struct vc_sm_result_t {
> + /* Transaction identifier */
> + u32 trans_id;
> +
> + s32 success;
> +
> +};
> +
> +/* Request to import memory (HOST->VC) */
> +struct vc_sm_import {
> + /* type of memory to allocate */
> + enum vc_sm_alloc_type_t type;
> + /* pointer to the VC (ie physical) address of the allocated memory */
> + u32 addr;
> + /* size of buffer */
> + u32 size;
> + /* opaque handle returned in RELEASED messages */
> + u32 kernel_id;
> + /* Allocator identifier */
> + u32 allocator;
> + /* resource name (for easier tracking on vc side) */
> + char name[VC_SM_RESOURCE_NAME];
> +};
> +
> +/* Result of a requested memory import (VC->HOST) */
> +struct vc_sm_import_result {
> + /* Transaction identifier */
> + u32 trans_id;
> +
> + /* Resource handle */
> + u32 res_handle;
> +};
> +
> +/* Notification that VC has finished with an allocation (VC->HOST) */
> +struct vc_sm_released {
> + /* cmd type / trans_id */
> + u32 cmd;
> +
> + /* pointer to the VC (ie physical) address of the allocated memory */
> + u32 addr;
> + /* size of buffer */
> + u32 size;
> + /* opaque handle returned in RELEASED messages */
> + u32 kernel_id;
> + u32 vc_handle;
> +};
> +
> +/*
> + * Client informing VC as to the protocol version it supports.
> + * >=2 requires the released callback, and supports VC asking for memory.
> + * Failure means that the firmware doesn't support this call, and therefore the
> + * client should either fail, or NOT rely on getting the released callback.
> + */
> +struct vc_sm_version {
> + u32 version;
> +};
> +
> +/* Response from the kernel to provide the VPU with some memory */
> +struct vc_sm_vc_mem_request_result {
> + /* Transaction identifier for the VPU */
> + u32 trans_id;
> + /* pointer to the physical address of the allocated memory */
> + u32 addr;
> + /* opaque handle returned in RELEASED messages */
> + u32 kernel_id;
> +};
> +
> +/* Union of ALL messages */
> +union vc_sm_msg_union_t {
> + struct vc_sm_free_t free;
> + struct vc_sm_result_t result;
> + struct vc_sm_import import;
> + struct vc_sm_import_result import_result;
> + struct vc_sm_version version;
> + struct vc_sm_released released;
> + struct vc_sm_vc_mem_request_result vc_request_result;
> +};
> +
> +#endif /* __VC_SM_DEFS_H__INCLUDED__ */
> diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> new file mode 100644
> index 000000000000..988fdd967922
> --- /dev/null
> +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> @@ -0,0 +1,28 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +
> +/*
> + * VideoCore Shared Memory CMA allocator
> + *
> + * Copyright: 2018, Raspberry Pi (Trading) Ltd
> + *
> + * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation.
> + *
> + */
> +
> +#ifndef __VC_SM_KNL_H__INCLUDED__
> +#define __VC_SM_KNL_H__INCLUDED__
> +
> +#if !defined(__KERNEL__)
> +#error "This interface is for kernel use only..."
> +#endif
As this header isn't available to userspace, I think you can drop this.
> +
> +/* Free a previously allocated or imported shared memory handle and block. */
kerneldoc would be nice for such an inter-driver API.
> +int vc_sm_cma_free(void *handle);
> +
> +/* Get an internal resource handle mapped from the external one. */
> +int vc_sm_cma_int_handle(void *handle);
> +
> +/* Import a block of memory into the GPU space. */
> +int vc_sm_cma_import_dmabuf(struct dma_buf *dmabuf, void **handle);
I would order these function in the expected call order: import,
int_handle and free.
> +
> +#endif /* __VC_SM_KNL_H__INCLUDED__ */
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 02/14] staging: vchiq_arm: Register vcsm-cma as a platform driver
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:07 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:07 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Dave,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:10AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> Following the same pattern as bcm2835-camera and bcm2835-audio,
> register the vcsm-cma driver as a platform driver.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
> drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> index dc33490ba7fb..642fdbc0d654 100644
> --- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> +++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> @@ -67,6 +67,7 @@ struct vchiq_state g_state;
>
> static struct platform_device *bcm2835_camera;
> static struct platform_device *bcm2835_audio;
> +static struct platform_device *vcsm_cma;
>
> struct vchiq_drvdata {
> const unsigned int cache_line_size;
> @@ -1832,6 +1833,7 @@ static int vchiq_probe(struct platform_device *pdev)
> goto error_exit;
> }
>
> + vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
> bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
> bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
>
> @@ -1847,6 +1849,7 @@ static int vchiq_remove(struct platform_device *pdev)
> {
> platform_device_unregister(bcm2835_audio);
> platform_device_unregister(bcm2835_camera);
> + platform_device_unregister(vcsm_cma);
> vchiq_debugfs_deinit();
> vchiq_deregister_chrdev();
>
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 02/14] staging: vchiq_arm: Register vcsm-cma as a platform driver
@ 2022-11-21 23:07 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:07 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Dave,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:10AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> Following the same pattern as bcm2835-camera and bcm2835-audio,
> register the vcsm-cma driver as a platform driver.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
> drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> index dc33490ba7fb..642fdbc0d654 100644
> --- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> +++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> @@ -67,6 +67,7 @@ struct vchiq_state g_state;
>
> static struct platform_device *bcm2835_camera;
> static struct platform_device *bcm2835_audio;
> +static struct platform_device *vcsm_cma;
>
> struct vchiq_drvdata {
> const unsigned int cache_line_size;
> @@ -1832,6 +1833,7 @@ static int vchiq_probe(struct platform_device *pdev)
> goto error_exit;
> }
>
> + vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
> bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
> bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
>
> @@ -1847,6 +1849,7 @@ static int vchiq_remove(struct platform_device *pdev)
> {
> platform_device_unregister(bcm2835_audio);
> platform_device_unregister(bcm2835_camera);
> + platform_device_unregister(vcsm_cma);
> vchiq_debugfs_deinit();
> vchiq_deregister_chrdev();
>
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 14/14] staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:10 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:10 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Naush,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:22AM +0530, Umang Jain wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> bcmn2835_isp is a platform driver dependent on vchiq,
> therefore add the load/unload functions for it to vchiq.
The commit message mentiones a platform driver, but the code handles a
platform device.
The rest looks fine to me.
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> index 642fdbc0d654..2d070f80a170 100644
> --- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> +++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> @@ -67,6 +67,7 @@ struct vchiq_state g_state;
>
> static struct platform_device *bcm2835_camera;
> static struct platform_device *bcm2835_audio;
> +static struct platform_device *bcm2835_isp;
> static struct platform_device *vcsm_cma;
>
> struct vchiq_drvdata {
> @@ -1836,6 +1837,7 @@ static int vchiq_probe(struct platform_device *pdev)
> vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
> bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
> bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
> + bcm2835_isp = vchiq_register_child(pdev, "bcm2835-isp");
>
> return 0;
>
> @@ -1847,6 +1849,7 @@ static int vchiq_probe(struct platform_device *pdev)
>
> static int vchiq_remove(struct platform_device *pdev)
> {
> + platform_device_unregister(bcm2835_isp);
> platform_device_unregister(bcm2835_audio);
> platform_device_unregister(bcm2835_camera);
> platform_device_unregister(vcsm_cma);
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 14/14] staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
@ 2022-11-21 23:10 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:10 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Naush,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:22AM +0530, Umang Jain wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> bcmn2835_isp is a platform driver dependent on vchiq,
> therefore add the load/unload functions for it to vchiq.
The commit message mentiones a platform driver, but the code handles a
platform device.
The rest looks fine to me.
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> index 642fdbc0d654..2d070f80a170 100644
> --- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> +++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c
> @@ -67,6 +67,7 @@ struct vchiq_state g_state;
>
> static struct platform_device *bcm2835_camera;
> static struct platform_device *bcm2835_audio;
> +static struct platform_device *bcm2835_isp;
> static struct platform_device *vcsm_cma;
>
> struct vchiq_drvdata {
> @@ -1836,6 +1837,7 @@ static int vchiq_probe(struct platform_device *pdev)
> vcsm_cma = vchiq_register_child(pdev, "vcsm-cma");
> bcm2835_camera = vchiq_register_child(pdev, "bcm2835-camera");
> bcm2835_audio = vchiq_register_child(pdev, "bcm2835_audio");
> + bcm2835_isp = vchiq_register_child(pdev, "bcm2835-isp");
>
> return 0;
>
> @@ -1847,6 +1849,7 @@ static int vchiq_probe(struct platform_device *pdev)
>
> static int vchiq_remove(struct platform_device *pdev)
> {
> + platform_device_unregister(bcm2835_isp);
> platform_device_unregister(bcm2835_audio);
> platform_device_unregister(bcm2835_camera);
> platform_device_unregister(vcsm_cma);
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:18 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:18 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham,
Dave Stevenson
Hi Umang and Dave,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:11AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.org>
>
> videobuf2 only allowed exporting a dmabuf as a file descriptor,
> but there are instances where having the struct dma_buf is
> useful within the kernel.
>
> Split the current implementation into two, one step which
> exports a struct dma_buf, and the second which converts that
> into an fd.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> .../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------
> include/media/videobuf2-core.h | 15 ++++++++
> 2 files changed, 40 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index ab9697f3b5f1..32b26737cac4 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -2184,49 +2184,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off,
> return -EINVAL;
> }
>
> -int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> - unsigned int index, unsigned int plane, unsigned int flags)
> +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> + unsigned int index, unsigned int plane,
> + unsigned int flags)
This function is used in the ISP driver, in bcm2835_isp_buf_prepare(),
for MMAP buffers, and as far as I can tell, its only purpose is to
create a dma_buf instance to then be imported in
vchiq_mmal_submit_buffer() with a call to vc_sm_cma_import_dmabuf().
That sounds like a very complicated set of operations, and quite
inefficient :-(
> {
> struct vb2_buffer *vb = NULL;
> struct vb2_plane *vb_plane;
> - int ret;
> struct dma_buf *dbuf;
>
> if (q->memory != VB2_MEMORY_MMAP) {
> dprintk(q, 1, "queue is not currently set up for mmap\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (!q->mem_ops->get_dmabuf) {
> dprintk(q, 1, "queue does not support DMA buffer exporting\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (flags & ~(O_CLOEXEC | O_ACCMODE)) {
> dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (type != q->type) {
> dprintk(q, 1, "invalid buffer type\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (index >= q->num_buffers) {
> dprintk(q, 1, "buffer index out of range\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> vb = q->bufs[index];
>
> if (plane >= vb->num_planes) {
> dprintk(q, 1, "buffer plane out of range\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (vb2_fileio_is_active(q)) {
> dprintk(q, 1, "expbuf: file io in progress\n");
> - return -EBUSY;
> + return ERR_PTR(-EBUSY);
> }
>
> vb_plane = &vb->planes[plane];
> @@ -2238,9 +2238,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> if (IS_ERR_OR_NULL(dbuf)) {
> dprintk(q, 1, "failed to export buffer %d, plane %d\n",
> index, plane);
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> + return dbuf;
> +}
> +EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf);
> +
> +int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> + unsigned int index, unsigned int plane, unsigned int flags)
> +{
> + struct dma_buf *dbuf;
> + int ret;
> +
> + dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags);
> + if (IS_ERR(dbuf))
> + return PTR_ERR(dbuf);
> +
> ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE);
> if (ret < 0) {
> dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n",
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 3253bd2f6fee..33629ed2b64f 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -911,6 +911,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
> */
> int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
>
> +/**
> + * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure
> + * @q: videobuf2 queue
> + * @type: buffer type
> + * @index: id number of the buffer
> + * @plane: index of the plane to be exported, 0 for single plane queues
> + * @flags: flags for newly created file, currently only O_CLOEXEC is
> + * supported, refer to manual of open syscall for more details
> + *
> + * Return: Returns the dmabuf pointer
> + */
> +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> + unsigned int index, unsigned int plane,
> + unsigned int flags);
> +
> /**
> * vb2_core_expbuf() - Export a buffer as a file descriptor.
> * @q: pointer to &struct vb2_queue with videobuf2 queue.
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf
@ 2022-11-21 23:18 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:18 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham,
Dave Stevenson
Hi Umang and Dave,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:11AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.org>
>
> videobuf2 only allowed exporting a dmabuf as a file descriptor,
> but there are instances where having the struct dma_buf is
> useful within the kernel.
>
> Split the current implementation into two, one step which
> exports a struct dma_buf, and the second which converts that
> into an fd.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> .../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------
> include/media/videobuf2-core.h | 15 ++++++++
> 2 files changed, 40 insertions(+), 11 deletions(-)
>
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index ab9697f3b5f1..32b26737cac4 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -2184,49 +2184,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off,
> return -EINVAL;
> }
>
> -int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> - unsigned int index, unsigned int plane, unsigned int flags)
> +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> + unsigned int index, unsigned int plane,
> + unsigned int flags)
This function is used in the ISP driver, in bcm2835_isp_buf_prepare(),
for MMAP buffers, and as far as I can tell, its only purpose is to
create a dma_buf instance to then be imported in
vchiq_mmal_submit_buffer() with a call to vc_sm_cma_import_dmabuf().
That sounds like a very complicated set of operations, and quite
inefficient :-(
> {
> struct vb2_buffer *vb = NULL;
> struct vb2_plane *vb_plane;
> - int ret;
> struct dma_buf *dbuf;
>
> if (q->memory != VB2_MEMORY_MMAP) {
> dprintk(q, 1, "queue is not currently set up for mmap\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (!q->mem_ops->get_dmabuf) {
> dprintk(q, 1, "queue does not support DMA buffer exporting\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (flags & ~(O_CLOEXEC | O_ACCMODE)) {
> dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (type != q->type) {
> dprintk(q, 1, "invalid buffer type\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (index >= q->num_buffers) {
> dprintk(q, 1, "buffer index out of range\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> vb = q->bufs[index];
>
> if (plane >= vb->num_planes) {
> dprintk(q, 1, "buffer plane out of range\n");
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> if (vb2_fileio_is_active(q)) {
> dprintk(q, 1, "expbuf: file io in progress\n");
> - return -EBUSY;
> + return ERR_PTR(-EBUSY);
> }
>
> vb_plane = &vb->planes[plane];
> @@ -2238,9 +2238,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> if (IS_ERR_OR_NULL(dbuf)) {
> dprintk(q, 1, "failed to export buffer %d, plane %d\n",
> index, plane);
> - return -EINVAL;
> + return ERR_PTR(-EINVAL);
> }
>
> + return dbuf;
> +}
> +EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf);
> +
> +int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> + unsigned int index, unsigned int plane, unsigned int flags)
> +{
> + struct dma_buf *dbuf;
> + int ret;
> +
> + dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags);
> + if (IS_ERR(dbuf))
> + return PTR_ERR(dbuf);
> +
> ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE);
> if (ret < 0) {
> dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n",
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 3253bd2f6fee..33629ed2b64f 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -911,6 +911,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
> */
> int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
>
> +/**
> + * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure
> + * @q: videobuf2 queue
> + * @type: buffer type
> + * @index: id number of the buffer
> + * @plane: index of the plane to be exported, 0 for single plane queues
> + * @flags: flags for newly created file, currently only O_CLOEXEC is
> + * supported, refer to manual of open syscall for more details
> + *
> + * Return: Returns the dmabuf pointer
> + */
> +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> + unsigned int index, unsigned int plane,
> + unsigned int flags);
> +
> /**
> * vb2_core_expbuf() - Export a buffer as a file descriptor.
> * @q: pointer to &struct vb2_queue with videobuf2 queue.
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 06/14] staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:21 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:21 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Dave,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:14AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> The bcm2835-isp supports Bayer, so add in the encodings for them.
> Also, Add support for monochrome image formats in the various MIPI
> packings.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> .../vc04_services/vchiq-mmal/mmal-encodings.h | 62 +++++++++++++++++++
> 1 file changed, 62 insertions(+)
>
> diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
> index e15ae7b24f73..4711877a9711 100644
> --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
> +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
> @@ -69,6 +69,68 @@
> */
> #define MMAL_ENCODING_OPAQUE MMAL_FOURCC('O', 'P', 'Q', 'V')
>
> +/* Bayer formats
/*
* Bayer formats
> + * FourCC values copied from V4L2 where defined.
> + */
> +/* 8 bit per pixel Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR8 MMAL_FOURCC('B', 'A', '8', '1')
> +#define MMAL_ENCODING_BAYER_SGBRG8 MMAL_FOURCC('G', 'B', 'R', 'G')
> +#define MMAL_ENCODING_BAYER_SGRBG8 MMAL_FOURCC('G', 'R', 'B', 'G')
> +#define MMAL_ENCODING_BAYER_SRGGB8 MMAL_FOURCC('R', 'G', 'G', 'B')
> +
> +/* 10 bit per pixel packed Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR10P MMAL_FOURCC('p', 'B', 'A', 'A')
> +#define MMAL_ENCODING_BAYER_SGRBG10P MMAL_FOURCC('p', 'g', 'A', 'A')
> +#define MMAL_ENCODING_BAYER_SGBRG10P MMAL_FOURCC('p', 'G', 'A', 'A')
SGBRG should go before SGRBG, like for the raw 8 formats. Same below for
the 12P, 10 and 12 formats.
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> +#define MMAL_ENCODING_BAYER_SRGGB10P MMAL_FOURCC('p', 'R', 'A', 'A')
> +
> +/* 12 bit per pixel packed Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR12P MMAL_FOURCC('p', 'B', '1', '2')
> +#define MMAL_ENCODING_BAYER_SGRBG12P MMAL_FOURCC('p', 'g', '1', '2')
> +#define MMAL_ENCODING_BAYER_SGBRG12P MMAL_FOURCC('p', 'G', '1', '2')
> +#define MMAL_ENCODING_BAYER_SRGGB12P MMAL_FOURCC('p', 'R', '1', '2')
> +
> +/* 14 bit per pixel Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR14P MMAL_FOURCC('p', 'B', 'E', 'E')
> +#define MMAL_ENCODING_BAYER_SGBRG14P MMAL_FOURCC('p', 'G', 'E', 'E')
> +#define MMAL_ENCODING_BAYER_SGRBG14P MMAL_FOURCC('p', 'g', 'E', 'E')
> +#define MMAL_ENCODING_BAYER_SRGGB14P MMAL_FOURCC('p', 'R', 'E', 'E')
> +
> +/* 16 bit per pixel Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR16 MMAL_FOURCC('B', 'G', '1', '6')
> +#define MMAL_ENCODING_BAYER_SGBRG16 MMAL_FOURCC('G', 'B', '1', '6')
> +#define MMAL_ENCODING_BAYER_SGRBG16 MMAL_FOURCC('G', 'R', '1', '6')
> +#define MMAL_ENCODING_BAYER_SRGGB16 MMAL_FOURCC('R', 'G', '1', '6')
> +
> +/* 10 bit per pixel unpacked (16bit) Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR10 MMAL_FOURCC('B', 'G', '1', '0')
> +#define MMAL_ENCODING_BAYER_SGRBG10 MMAL_FOURCC('B', 'A', '1', '0')
> +#define MMAL_ENCODING_BAYER_SGBRG10 MMAL_FOURCC('G', 'B', '1', '0')
> +#define MMAL_ENCODING_BAYER_SRGGB10 MMAL_FOURCC('R', 'G', '1', '0')
> +
> +/* 12 bit per pixel unpacked (16bit) Bayer formats */
> +#define MMAL_ENCODING_BAYER_SBGGR12 MMAL_FOURCC('B', 'G', '1', '2')
> +#define MMAL_ENCODING_BAYER_SGRBG12 MMAL_FOURCC('B', 'A', '1', '2')
> +#define MMAL_ENCODING_BAYER_SGBRG12 MMAL_FOURCC('G', 'B', '1', '2')
> +#define MMAL_ENCODING_BAYER_SRGGB12 MMAL_FOURCC('R', 'G', '1', '2')
> +
> +/* 14 bit per pixel unpacked (16bit) Bayer formats */
> +#define MMAL_ENCODING_BAYER_SBGGR14 MMAL_FOURCC('B', 'G', '1', '4')
> +#define MMAL_ENCODING_BAYER_SGBRG14 MMAL_FOURCC('G', 'B', '1', '4')
> +#define MMAL_ENCODING_BAYER_SGRBG14 MMAL_FOURCC('G', 'R', '1', '4')
> +#define MMAL_ENCODING_BAYER_SRGGB14 MMAL_FOURCC('R', 'G', '1', '4')
> +
> +/* MIPI packed monochrome images */
> +#define MMAL_ENCODING_GREY MMAL_FOURCC('G', 'R', 'E', 'Y')
> +#define MMAL_ENCODING_Y10P MMAL_FOURCC('Y', '1', '0', 'P')
> +#define MMAL_ENCODING_Y12P MMAL_FOURCC('Y', '1', '2', 'P')
> +#define MMAL_ENCODING_Y14P MMAL_FOURCC('Y', '1', '4', 'P')
> +#define MMAL_ENCODING_Y16 MMAL_FOURCC('Y', '1', '6', ' ')
> +/* Unpacked monochrome formats (16bit per sample, but only N LSBs used) */
> +#define MMAL_ENCODING_Y10 MMAL_FOURCC('Y', '1', '0', ' ')
> +#define MMAL_ENCODING_Y12 MMAL_FOURCC('Y', '1', '2', ' ')
> +#define MMAL_ENCODING_Y14 MMAL_FOURCC('Y', '1', '4', ' ')
> +
> /** An EGL image handle
> */
> #define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I')
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 06/14] staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
@ 2022-11-21 23:21 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:21 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Dave,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:14AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> The bcm2835-isp supports Bayer, so add in the encodings for them.
> Also, Add support for monochrome image formats in the various MIPI
> packings.
>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> .../vc04_services/vchiq-mmal/mmal-encodings.h | 62 +++++++++++++++++++
> 1 file changed, 62 insertions(+)
>
> diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
> index e15ae7b24f73..4711877a9711 100644
> --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
> +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h
> @@ -69,6 +69,68 @@
> */
> #define MMAL_ENCODING_OPAQUE MMAL_FOURCC('O', 'P', 'Q', 'V')
>
> +/* Bayer formats
/*
* Bayer formats
> + * FourCC values copied from V4L2 where defined.
> + */
> +/* 8 bit per pixel Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR8 MMAL_FOURCC('B', 'A', '8', '1')
> +#define MMAL_ENCODING_BAYER_SGBRG8 MMAL_FOURCC('G', 'B', 'R', 'G')
> +#define MMAL_ENCODING_BAYER_SGRBG8 MMAL_FOURCC('G', 'R', 'B', 'G')
> +#define MMAL_ENCODING_BAYER_SRGGB8 MMAL_FOURCC('R', 'G', 'G', 'B')
> +
> +/* 10 bit per pixel packed Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR10P MMAL_FOURCC('p', 'B', 'A', 'A')
> +#define MMAL_ENCODING_BAYER_SGRBG10P MMAL_FOURCC('p', 'g', 'A', 'A')
> +#define MMAL_ENCODING_BAYER_SGBRG10P MMAL_FOURCC('p', 'G', 'A', 'A')
SGBRG should go before SGRBG, like for the raw 8 formats. Same below for
the 12P, 10 and 12 formats.
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> +#define MMAL_ENCODING_BAYER_SRGGB10P MMAL_FOURCC('p', 'R', 'A', 'A')
> +
> +/* 12 bit per pixel packed Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR12P MMAL_FOURCC('p', 'B', '1', '2')
> +#define MMAL_ENCODING_BAYER_SGRBG12P MMAL_FOURCC('p', 'g', '1', '2')
> +#define MMAL_ENCODING_BAYER_SGBRG12P MMAL_FOURCC('p', 'G', '1', '2')
> +#define MMAL_ENCODING_BAYER_SRGGB12P MMAL_FOURCC('p', 'R', '1', '2')
> +
> +/* 14 bit per pixel Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR14P MMAL_FOURCC('p', 'B', 'E', 'E')
> +#define MMAL_ENCODING_BAYER_SGBRG14P MMAL_FOURCC('p', 'G', 'E', 'E')
> +#define MMAL_ENCODING_BAYER_SGRBG14P MMAL_FOURCC('p', 'g', 'E', 'E')
> +#define MMAL_ENCODING_BAYER_SRGGB14P MMAL_FOURCC('p', 'R', 'E', 'E')
> +
> +/* 16 bit per pixel Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR16 MMAL_FOURCC('B', 'G', '1', '6')
> +#define MMAL_ENCODING_BAYER_SGBRG16 MMAL_FOURCC('G', 'B', '1', '6')
> +#define MMAL_ENCODING_BAYER_SGRBG16 MMAL_FOURCC('G', 'R', '1', '6')
> +#define MMAL_ENCODING_BAYER_SRGGB16 MMAL_FOURCC('R', 'G', '1', '6')
> +
> +/* 10 bit per pixel unpacked (16bit) Bayer formats. */
> +#define MMAL_ENCODING_BAYER_SBGGR10 MMAL_FOURCC('B', 'G', '1', '0')
> +#define MMAL_ENCODING_BAYER_SGRBG10 MMAL_FOURCC('B', 'A', '1', '0')
> +#define MMAL_ENCODING_BAYER_SGBRG10 MMAL_FOURCC('G', 'B', '1', '0')
> +#define MMAL_ENCODING_BAYER_SRGGB10 MMAL_FOURCC('R', 'G', '1', '0')
> +
> +/* 12 bit per pixel unpacked (16bit) Bayer formats */
> +#define MMAL_ENCODING_BAYER_SBGGR12 MMAL_FOURCC('B', 'G', '1', '2')
> +#define MMAL_ENCODING_BAYER_SGRBG12 MMAL_FOURCC('B', 'A', '1', '2')
> +#define MMAL_ENCODING_BAYER_SGBRG12 MMAL_FOURCC('G', 'B', '1', '2')
> +#define MMAL_ENCODING_BAYER_SRGGB12 MMAL_FOURCC('R', 'G', '1', '2')
> +
> +/* 14 bit per pixel unpacked (16bit) Bayer formats */
> +#define MMAL_ENCODING_BAYER_SBGGR14 MMAL_FOURCC('B', 'G', '1', '4')
> +#define MMAL_ENCODING_BAYER_SGBRG14 MMAL_FOURCC('G', 'B', '1', '4')
> +#define MMAL_ENCODING_BAYER_SGRBG14 MMAL_FOURCC('G', 'R', '1', '4')
> +#define MMAL_ENCODING_BAYER_SRGGB14 MMAL_FOURCC('R', 'G', '1', '4')
> +
> +/* MIPI packed monochrome images */
> +#define MMAL_ENCODING_GREY MMAL_FOURCC('G', 'R', 'E', 'Y')
> +#define MMAL_ENCODING_Y10P MMAL_FOURCC('Y', '1', '0', 'P')
> +#define MMAL_ENCODING_Y12P MMAL_FOURCC('Y', '1', '2', 'P')
> +#define MMAL_ENCODING_Y14P MMAL_FOURCC('Y', '1', '4', 'P')
> +#define MMAL_ENCODING_Y16 MMAL_FOURCC('Y', '1', '6', ' ')
> +/* Unpacked monochrome formats (16bit per sample, but only N LSBs used) */
> +#define MMAL_ENCODING_Y10 MMAL_FOURCC('Y', '1', '0', ' ')
> +#define MMAL_ENCODING_Y12 MMAL_FOURCC('Y', '1', '2', ' ')
> +#define MMAL_ENCODING_Y14 MMAL_FOURCC('Y', '1', '4', ' ')
> +
> /** An EGL image handle
> */
> #define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I')
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 07/14] media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:25 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:25 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Naush,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:15AM +0530, Umang Jain wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Add V4L2_META_FMT_BCM2835_ISP_STATS V4L2 format type.
>
> This new format will be used by the bcm2835-isp device to return
> out ISP statistics for 3A.
>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> .../userspace-api/media/v4l/meta-formats.rst | 1 +
> .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +++++++++++++++++++
> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> include/uapi/linux/videodev2.h | 1 +
> 4 files changed, 44 insertions(+)
> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
>
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index d1ac3f3a2d8f..0c52f3df1576 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
> .. toctree::
> :maxdepth: 1
>
> + pixfmt-meta-bcm2835-isp-stats
> pixfmt-meta-d4xx
> pixfmt-meta-intel-ipu3
> pixfmt-meta-sensor-data
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> new file mode 100644
> index 000000000000..f974774c8252
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> @@ -0,0 +1,41 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/media/uapi/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
You can do so now.
> +
> +.. _v4l2-meta-fmt-bcm2835-isp-stats:
> +
> +*****************************************
> +V4L2_META_FMT_BCM2835_ISP_STATS ('BSTA')
> +*****************************************
> +
> +BCM2835 ISP Statistics
> +
> +Description
> +===========
> +
> +The BCM2835 ISP hardware calculate image statistics for an input Bayer frame.
> +These statistics are obtained from the "bcm2835-isp0-capture3" device node
> +using the :c:type:`v4l2_meta_format` interface. They are formatted as described
> +by the :c:type:`bcm2835_isp_stats` structure below.
> +
> +.. code-block:: c
> +
> + #define DEFAULT_AWB_REGIONS_X 16
> + #define DEFAULT_AWB_REGIONS_Y 12
> +
> + #define NUM_HISTOGRAMS 2
> + #define NUM_HISTOGRAM_BINS 128
> + #define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
> + #define FLOATING_REGIONS 16
> + #define AGC_REGIONS 16
> + #define FOCUS_REGIONS 12
> +
> +.. kernel-doc:: include/uapi/linux/bcm2835-isp.h
> + :functions: bcm2835_isp_stats_hist bcm2835_isp_stats_region
> + bcm2835_isp_stats_focus bcm2835_isp_stats
This patch should go after 08/14 or it will break bisection.
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> +
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 2680bc33f911..361e47ad6480 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1449,6 +1449,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> case V4L2_PIX_FMT_NV12M_8L128: descr = "NV12M (8x128 Linear)"; break;
> case V4L2_PIX_FMT_NV12M_10BE_8L128: descr = "10-bit NV12M (8x128 Linear, BE)"; break;
> case V4L2_META_FMT_SENSOR_DATA: descr = "Sensor Ancillary Metadata"; break;
> + case V4L2_META_FMT_BCM2835_ISP_STATS: descr = "BCM2835 ISP Image Statistics"; break;
>
> default:
> /* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 026fbc7af034..9269b45b3b64 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -807,6 +807,7 @@ struct v4l2_pix_format {
> #define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
> #define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D') /* Vivid Metadata */
> #define V4L2_META_FMT_SENSOR_DATA v4l2_fourcc('S', 'E', 'N', 'S') /* Sensor Ancillary metadata */
> +#define V4L2_META_FMT_BCM2835_ISP_STATS v4l2_fourcc('B', 'S', 'T', 'A') /* BCM2835 ISP image statistics output */
>
> /* Vendor specific - used for RK_ISP1 camera sub-system */
> #define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 07/14] media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
@ 2022-11-21 23:25 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:25 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Naush,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:15AM +0530, Umang Jain wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Add V4L2_META_FMT_BCM2835_ISP_STATS V4L2 format type.
>
> This new format will be used by the bcm2835-isp device to return
> out ISP statistics for 3A.
>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> .../userspace-api/media/v4l/meta-formats.rst | 1 +
> .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +++++++++++++++++++
> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> include/uapi/linux/videodev2.h | 1 +
> 4 files changed, 44 insertions(+)
> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
>
> diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst
> index d1ac3f3a2d8f..0c52f3df1576 100644
> --- a/Documentation/userspace-api/media/v4l/meta-formats.rst
> +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst
> @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only.
> .. toctree::
> :maxdepth: 1
>
> + pixfmt-meta-bcm2835-isp-stats
> pixfmt-meta-d4xx
> pixfmt-meta-intel-ipu3
> pixfmt-meta-sensor-data
> diff --git a/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> new file mode 100644
> index 000000000000..f974774c8252
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> @@ -0,0 +1,41 @@
> +.. Permission is granted to copy, distribute and/or modify this
> +.. document under the terms of the GNU Free Documentation License,
> +.. Version 1.1 or any later version published by the Free Software
> +.. Foundation, with no Invariant Sections, no Front-Cover Texts
> +.. and no Back-Cover Texts. A copy of the license is included at
> +.. Documentation/media/uapi/fdl-appendix.rst.
> +..
> +.. TODO: replace it to GFDL-1.1-or-later WITH no-invariant-sections
You can do so now.
> +
> +.. _v4l2-meta-fmt-bcm2835-isp-stats:
> +
> +*****************************************
> +V4L2_META_FMT_BCM2835_ISP_STATS ('BSTA')
> +*****************************************
> +
> +BCM2835 ISP Statistics
> +
> +Description
> +===========
> +
> +The BCM2835 ISP hardware calculate image statistics for an input Bayer frame.
> +These statistics are obtained from the "bcm2835-isp0-capture3" device node
> +using the :c:type:`v4l2_meta_format` interface. They are formatted as described
> +by the :c:type:`bcm2835_isp_stats` structure below.
> +
> +.. code-block:: c
> +
> + #define DEFAULT_AWB_REGIONS_X 16
> + #define DEFAULT_AWB_REGIONS_Y 12
> +
> + #define NUM_HISTOGRAMS 2
> + #define NUM_HISTOGRAM_BINS 128
> + #define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
> + #define FLOATING_REGIONS 16
> + #define AGC_REGIONS 16
> + #define FOCUS_REGIONS 12
> +
> +.. kernel-doc:: include/uapi/linux/bcm2835-isp.h
> + :functions: bcm2835_isp_stats_hist bcm2835_isp_stats_region
> + bcm2835_isp_stats_focus bcm2835_isp_stats
This patch should go after 08/14 or it will break bisection.
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> +
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 2680bc33f911..361e47ad6480 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1449,6 +1449,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> case V4L2_PIX_FMT_NV12M_8L128: descr = "NV12M (8x128 Linear)"; break;
> case V4L2_PIX_FMT_NV12M_10BE_8L128: descr = "10-bit NV12M (8x128 Linear, BE)"; break;
> case V4L2_META_FMT_SENSOR_DATA: descr = "Sensor Ancillary Metadata"; break;
> + case V4L2_META_FMT_BCM2835_ISP_STATS: descr = "BCM2835 ISP Image Statistics"; break;
>
> default:
> /* Compressed formats */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 026fbc7af034..9269b45b3b64 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -807,6 +807,7 @@ struct v4l2_pix_format {
> #define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */
> #define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D') /* Vivid Metadata */
> #define V4L2_META_FMT_SENSOR_DATA v4l2_fourcc('S', 'E', 'N', 'S') /* Sensor Ancillary metadata */
> +#define V4L2_META_FMT_BCM2835_ISP_STATS v4l2_fourcc('B', 'S', 'T', 'A') /* BCM2835 ISP image statistics output */
>
> /* Vendor specific - used for RK_ISP1 camera sub-system */
> #define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 08/14] uapi: bcm2835-isp: Add bcm2835-isp uapi header file
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:28 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:28 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Dave,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:16AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> This file defines the userland interface to the bcm2835-isp driver
> that will follow in a separate commit.
>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> include/uapi/linux/bcm2835-isp.h | 320 +++++++++++++++++++++++++++++
> include/uapi/linux/v4l2-controls.h | 6 +
> 2 files changed, 326 insertions(+)
> create mode 100644 include/uapi/linux/bcm2835-isp.h
>
> diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
> new file mode 100644
> index 000000000000..cf8c0437f159
> --- /dev/null
> +++ b/include/uapi/linux/bcm2835-isp.h
> @@ -0,0 +1,320 @@
> +/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */
> +/*
> + * bcm2835-isp.h
> + *
> + * BCM2835 ISP driver - user space header file.
> + *
> + * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
> + *
> + * Author: Naushir Patuck (naush@raspberrypi.com)
> + *
> + */
> +
> +#ifndef __BCM2835_ISP_H_
> +#define __BCM2835_ISP_H_
> +
> +#include <linux/v4l2-controls.h>
> +
> +#define V4L2_CID_USER_BCM2835_ISP_CC_MATRIX \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0001)
> +#define V4L2_CID_USER_BCM2835_ISP_LENS_SHADING \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0002)
> +#define V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0003)
> +#define V4L2_CID_USER_BCM2835_ISP_GEQ \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0004)
> +#define V4L2_CID_USER_BCM2835_ISP_GAMMA \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0005)
> +#define V4L2_CID_USER_BCM2835_ISP_DENOISE \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0006)
> +#define V4L2_CID_USER_BCM2835_ISP_SHARPEN \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
> +#define V4L2_CID_USER_BCM2835_ISP_DPC \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
> +
> +/*
> + * All structs below are directly mapped onto the equivalent structs in
> + * drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
> + * for convenience.
> + */
> +
> +/**
> + * struct bcm2835_isp_rational - Rational value type.
> + *
> + * @num: Numerator.
> + * @den: Denominator.
> + */
> +struct bcm2835_isp_rational {
> + __s32 num;
> + __u32 den;
> +};
> +
> +/**
> + * struct bcm2835_isp_ccm - Colour correction matrix.
> + *
> + * @ccm: 3x3 correction matrix coefficients.
> + * @offsets: 1x3 correction offsets.
> + */
> +struct bcm2835_isp_ccm {
> + struct bcm2835_isp_rational ccm[3][3];
> + __s32 offsets[3];
> +};
> +
> +/**
> + * struct bcm2835_isp_custom_ccm - Custom CCM applied with the
> + * V4L2_CID_USER_BCM2835_ISP_CC_MATRIX ctrl.
> + *
> + * @enabled: Enable custom CCM.
> + * @ccm: Custom CCM coefficients and offsets.
> + */
> +struct bcm2835_isp_custom_ccm {
> + __u32 enabled;
> + struct bcm2835_isp_ccm ccm;
> +};
> +
> +/**
> + * enum bcm2835_isp_gain_format - format of the gains in the lens shading
> + * tables used with the
> + * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING ctrl.
> + *
> + * @GAIN_FORMAT_U0P8_1: Gains are u0.8 format, starting at 1.0
> + * @GAIN_FORMAT_U1P7_0: Gains are u1.7 format, starting at 0.0
> + * @GAIN_FORMAT_U1P7_1: Gains are u1.7 format, starting at 1.0
> + * @GAIN_FORMAT_U2P6_0: Gains are u2.6 format, starting at 0.0
> + * @GAIN_FORMAT_U2P6_1: Gains are u2.6 format, starting at 1.0
> + * @GAIN_FORMAT_U3P5_0: Gains are u3.5 format, starting at 0.0
> + * @GAIN_FORMAT_U3P5_1: Gains are u3.5 format, starting at 1.0
> + * @GAIN_FORMAT_U4P10: Gains are u4.10 format, starting at 0.0
> + */
> +enum bcm2835_isp_gain_format {
> + GAIN_FORMAT_U0P8_1 = 0,
> + GAIN_FORMAT_U1P7_0 = 1,
> + GAIN_FORMAT_U1P7_1 = 2,
> + GAIN_FORMAT_U2P6_0 = 3,
> + GAIN_FORMAT_U2P6_1 = 4,
> + GAIN_FORMAT_U3P5_0 = 5,
> + GAIN_FORMAT_U3P5_1 = 6,
> + GAIN_FORMAT_U4P10 = 7,
> +};
> +
> +/**
> + * struct bcm2835_isp_lens_shading - Lens shading tables supplied with the
> + * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING
> + * ctrl.
> + *
> + * @enabled: Enable lens shading.
> + * @grid_cell_size: Size of grid cells in samples (16, 32, 64, 128 or 256).
> + * @grid_width: Width of lens shading tables in grid cells.
> + * @grid_stride: Row to row distance (in grid cells) between grid cells
> + * in the same horizontal location.
> + * @grid_height: Height of lens shading tables in grid cells.
> + * @dmabuf: dmabuf file handle containing the table.
> + * @ref_transform: Reference transform - unsupported, please pass zero.
> + * @corner_sampled: Whether the gains are sampled at the corner points
> + * of the grid cells or in the cell centres.
> + * @gain_format: Format of the gains (see enum &bcm2835_isp_gain_format).
> + */
> +struct bcm2835_isp_lens_shading {
> + __u32 enabled;
> + __u32 grid_cell_size;
> + __u32 grid_width;
> + __u32 grid_stride;
> + __u32 grid_height;
> + __s32 dmabuf;
> + __u32 ref_transform;
> + __u32 corner_sampled;
> + __u32 gain_format;
> +};
> +
> +/**
> + * struct bcm2835_isp_black_level - Sensor black level set with the
> + * V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL ctrl.
> + *
> + * @enabled: Enable black level.
> + * @black_level_r: Black level for red channel.
> + * @black_level_g: Black level for green channels.
> + * @black_level_b: Black level for blue channel.
> + */
> +struct bcm2835_isp_black_level {
> + __u32 enabled;
> + __u16 black_level_r;
> + __u16 black_level_g;
> + __u16 black_level_b;
> + __u8 padding[2]; /* Unused */
> +};
> +
> +/**
> + * struct bcm2835_isp_geq - Green equalisation parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_GEQ ctrl.
> + *
> + * @enabled: Enable green equalisation.
> + * @offset: Fixed offset of the green equalisation threshold.
> + * @slope: Slope of the green equalisation threshold.
> + */
> +struct bcm2835_isp_geq {
> + __u32 enabled;
> + __u32 offset;
> + struct bcm2835_isp_rational slope;
> +};
> +
> +#define BCM2835_NUM_GAMMA_PTS 33
> +
> +/**
> + * struct bcm2835_isp_gamma - Gamma parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_GAMMA ctrl.
> + *
> + * @enabled: Enable gamma adjustment.
> + * @X: X values of the points defining the gamma curve.
> + * Values should be scaled to 16 bits.
> + * @Y: Y values of the points defining the gamma curve.
> + * Values should be scaled to 16 bits.
> + */
> +struct bcm2835_isp_gamma {
> + __u32 enabled;
> + __u16 x[BCM2835_NUM_GAMMA_PTS];
> + __u16 y[BCM2835_NUM_GAMMA_PTS];
> +};
> +
> +/**
> + * struct bcm2835_isp_denoise - Denoise parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
> + *
> + * @enabled: Enable denoise.
> + * @constant: Fixed offset of the noise threshold.
> + * @slope: Slope of the noise threshold.
> + * @strength: Denoise strength between 0.0 (off) and 1.0 (maximum).
> + */
> +struct bcm2835_isp_denoise {
> + __u32 enabled;
> + __u32 constant;
> + struct bcm2835_isp_rational slope;
> + struct bcm2835_isp_rational strength;
> +};
> +
> +/**
> + * struct bcm2835_isp_sharpen - Sharpen parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_SHARPEN ctrl.
> + *
> + * @enabled: Enable sharpening.
> + * @threshold: Threshold at which to start sharpening pixels.
> + * @strength: Strength with which pixel sharpening increases.
> + * @limit: Limit to the amount of sharpening applied.
> + */
> +struct bcm2835_isp_sharpen {
> + __u32 enabled;
> + struct bcm2835_isp_rational threshold;
> + struct bcm2835_isp_rational strength;
> + struct bcm2835_isp_rational limit;
> +};
> +
> +/**
> + * enum bcm2835_isp_dpc_mode - defective pixel correction (DPC) strength.
> + *
> + * @DPC_MODE_OFF: No DPC.
> + * @DPC_MODE_NORMAL: Normal DPC.
> + * @DPC_MODE_STRONG: Strong DPC.
> + */
> +enum bcm2835_isp_dpc_mode {
> + DPC_MODE_OFF = 0,
> + DPC_MODE_NORMAL = 1,
> + DPC_MODE_STRONG = 2,
> +};
> +
> +/**
> + * struct bcm2835_isp_dpc - Defective pixel correction (DPC) parameters set
> + * with the V4L2_CID_USER_BCM2835_ISP_DPC ctrl.
> + *
> + * @enabled: Enable DPC.
> + * @strength: DPC strength (see enum &bcm2835_isp_dpc_mode).
> + */
> +struct bcm2835_isp_dpc {
> + __u32 enabled;
> + __u32 strength;
> +};
> +
> +/*
> + * ISP statistics structures.
> + *
> + * The bcm2835_isp_stats structure is generated at the output of the
> + * statistics node. Note that this does not directly map onto the statistics
> + * output of the ISP HW. Instead, the MMAL firmware code maps the HW statistics
> + * to the bcm2835_isp_stats structure.
> + */
> +#define DEFAULT_AWB_REGIONS_X 16
> +#define DEFAULT_AWB_REGIONS_Y 12
> +
> +#define NUM_HISTOGRAMS 2
> +#define NUM_HISTOGRAM_BINS 128
> +#define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
> +#define FLOATING_REGIONS 16
> +#define AGC_REGIONS 16
> +#define FOCUS_REGIONS 12
> +
> +/**
> + * struct bcm2835_isp_stats_hist - Histogram statistics
> + *
> + * @r_hist: Red channel histogram.
> + * @g_hist: Combined green channel histogram.
> + * @b_hist: Blue channel histogram.
> + */
> +struct bcm2835_isp_stats_hist {
> + __u32 r_hist[NUM_HISTOGRAM_BINS];
> + __u32 g_hist[NUM_HISTOGRAM_BINS];
> + __u32 b_hist[NUM_HISTOGRAM_BINS];
> +};
> +
> +/**
> + * struct bcm2835_isp_stats_region - Region sums.
> + *
> + * @counted: The number of 2x2 bayer tiles accumulated.
> + * @notcounted: The number of 2x2 bayer tiles not accumulated.
> + * @r_sum: Total sum of counted pixels in the red channel for a region.
> + * @g_sum: Total sum of counted pixels in the green channel for a region.
> + * @b_sum: Total sum of counted pixels in the blue channel for a region.
> + */
> +struct bcm2835_isp_stats_region {
> + __u32 counted;
> + __u32 notcounted;
> + __u64 r_sum;
> + __u64 g_sum;
> + __u64 b_sum;
> +};
> +
> +/**
> + * struct bcm2835_isp_stats_focus - Focus statistics.
> + *
> + * @contrast_val: Focus measure - accumulated output of the focus filter.
> + * In the first dimension, index [0] counts pixels below a
> + * preset threshold, and index [1] counts pixels above the
> + * threshold. In the second dimension, index [0] uses the
> + * first predefined filter, and index [1] uses the second
> + * predefined filter.
> + * @contrast_val_num: The number of counted pixels in the above accumulation.
> + */
> +struct bcm2835_isp_stats_focus {
> + __u64 contrast_val[2][2];
> + __u32 contrast_val_num[2][2];
> +};
> +
> +/**
> + * struct bcm2835_isp_stats - ISP statistics.
> + *
> + * @version: Version of the bcm2835_isp_stats structure.
> + * @size: Size of the bcm2835_isp_stats structure.
> + * @hist: Histogram statistics for the entire image.
> + * @awb_stats: Statistics for the regions defined for AWB calculations.
> + * @floating_stats: Statistics for arbitrarily placed (floating) regions.
> + * @agc_stats: Statistics for the regions defined for AGC calculations.
> + * @focus_stats: Focus filter statistics for the focus regions.
> + */
> +struct bcm2835_isp_stats {
> + __u32 version;
> + __u32 size;
> + struct bcm2835_isp_stats_hist hist[NUM_HISTOGRAMS];
> + struct bcm2835_isp_stats_region awb_stats[AWB_REGIONS];
> + struct bcm2835_isp_stats_region floating_stats[FLOATING_REGIONS];
> + struct bcm2835_isp_stats_region agc_stats[AGC_REGIONS];
> + struct bcm2835_isp_stats_focus focus_stats[FOCUS_REGIONS];
> +};
> +
> +#endif /* __BCM2835_ISP_H_ */
> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> index b5e7d082b8ad..31606d42ec58 100644
> --- a/include/uapi/linux/v4l2-controls.h
> +++ b/include/uapi/linux/v4l2-controls.h
> @@ -217,6 +217,7 @@ enum v4l2_colorfx {
> * The base for Allegro driver controls.
> * We reserve 16 controls for this driver.
> */
> +
Unrelated change.
> #define V4L2_CID_USER_ALLEGRO_BASE (V4L2_CID_USER_BASE + 0x1170)
>
> /*
> @@ -231,6 +232,11 @@ enum v4l2_colorfx {
> */
> #define V4L2_CID_USER_DW100_BASE (V4L2_CID_USER_BASE + 0x1190)
>
> +/* The base for the bcm2835-isp driver controls.
/*
* The base for the bcm2835-isp driver controls.
> + * We reserve 16 controls for this driver.
> + */
> +#define V4L2_CID_USER_BCM2835_ISP_BASE (V4L2_CID_USER_BASE + 0x11a0)
This will be interesting to handle in libcamera, as it breaks the ABI
with the downstream driver.
> +
> /* MPEG-class control IDs */
> /* The MPEG controls are applicable to all codec controls
> * and the 'MPEG' part of the define is historical */
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 08/14] uapi: bcm2835-isp: Add bcm2835-isp uapi header file
@ 2022-11-21 23:28 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:28 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Dave,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:16AM +0530, Umang Jain wrote:
> From: Dave Stevenson <dave.stevenson@raspberrypi.com>
>
> This file defines the userland interface to the bcm2835-isp driver
> that will follow in a separate commit.
>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> include/uapi/linux/bcm2835-isp.h | 320 +++++++++++++++++++++++++++++
> include/uapi/linux/v4l2-controls.h | 6 +
> 2 files changed, 326 insertions(+)
> create mode 100644 include/uapi/linux/bcm2835-isp.h
>
> diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h
> new file mode 100644
> index 000000000000..cf8c0437f159
> --- /dev/null
> +++ b/include/uapi/linux/bcm2835-isp.h
> @@ -0,0 +1,320 @@
> +/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */
> +/*
> + * bcm2835-isp.h
> + *
> + * BCM2835 ISP driver - user space header file.
> + *
> + * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd.
> + *
> + * Author: Naushir Patuck (naush@raspberrypi.com)
> + *
> + */
> +
> +#ifndef __BCM2835_ISP_H_
> +#define __BCM2835_ISP_H_
> +
> +#include <linux/v4l2-controls.h>
> +
> +#define V4L2_CID_USER_BCM2835_ISP_CC_MATRIX \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0001)
> +#define V4L2_CID_USER_BCM2835_ISP_LENS_SHADING \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0002)
> +#define V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0003)
> +#define V4L2_CID_USER_BCM2835_ISP_GEQ \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0004)
> +#define V4L2_CID_USER_BCM2835_ISP_GAMMA \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0005)
> +#define V4L2_CID_USER_BCM2835_ISP_DENOISE \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0006)
> +#define V4L2_CID_USER_BCM2835_ISP_SHARPEN \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007)
> +#define V4L2_CID_USER_BCM2835_ISP_DPC \
> + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008)
> +
> +/*
> + * All structs below are directly mapped onto the equivalent structs in
> + * drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h
> + * for convenience.
> + */
> +
> +/**
> + * struct bcm2835_isp_rational - Rational value type.
> + *
> + * @num: Numerator.
> + * @den: Denominator.
> + */
> +struct bcm2835_isp_rational {
> + __s32 num;
> + __u32 den;
> +};
> +
> +/**
> + * struct bcm2835_isp_ccm - Colour correction matrix.
> + *
> + * @ccm: 3x3 correction matrix coefficients.
> + * @offsets: 1x3 correction offsets.
> + */
> +struct bcm2835_isp_ccm {
> + struct bcm2835_isp_rational ccm[3][3];
> + __s32 offsets[3];
> +};
> +
> +/**
> + * struct bcm2835_isp_custom_ccm - Custom CCM applied with the
> + * V4L2_CID_USER_BCM2835_ISP_CC_MATRIX ctrl.
> + *
> + * @enabled: Enable custom CCM.
> + * @ccm: Custom CCM coefficients and offsets.
> + */
> +struct bcm2835_isp_custom_ccm {
> + __u32 enabled;
> + struct bcm2835_isp_ccm ccm;
> +};
> +
> +/**
> + * enum bcm2835_isp_gain_format - format of the gains in the lens shading
> + * tables used with the
> + * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING ctrl.
> + *
> + * @GAIN_FORMAT_U0P8_1: Gains are u0.8 format, starting at 1.0
> + * @GAIN_FORMAT_U1P7_0: Gains are u1.7 format, starting at 0.0
> + * @GAIN_FORMAT_U1P7_1: Gains are u1.7 format, starting at 1.0
> + * @GAIN_FORMAT_U2P6_0: Gains are u2.6 format, starting at 0.0
> + * @GAIN_FORMAT_U2P6_1: Gains are u2.6 format, starting at 1.0
> + * @GAIN_FORMAT_U3P5_0: Gains are u3.5 format, starting at 0.0
> + * @GAIN_FORMAT_U3P5_1: Gains are u3.5 format, starting at 1.0
> + * @GAIN_FORMAT_U4P10: Gains are u4.10 format, starting at 0.0
> + */
> +enum bcm2835_isp_gain_format {
> + GAIN_FORMAT_U0P8_1 = 0,
> + GAIN_FORMAT_U1P7_0 = 1,
> + GAIN_FORMAT_U1P7_1 = 2,
> + GAIN_FORMAT_U2P6_0 = 3,
> + GAIN_FORMAT_U2P6_1 = 4,
> + GAIN_FORMAT_U3P5_0 = 5,
> + GAIN_FORMAT_U3P5_1 = 6,
> + GAIN_FORMAT_U4P10 = 7,
> +};
> +
> +/**
> + * struct bcm2835_isp_lens_shading - Lens shading tables supplied with the
> + * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING
> + * ctrl.
> + *
> + * @enabled: Enable lens shading.
> + * @grid_cell_size: Size of grid cells in samples (16, 32, 64, 128 or 256).
> + * @grid_width: Width of lens shading tables in grid cells.
> + * @grid_stride: Row to row distance (in grid cells) between grid cells
> + * in the same horizontal location.
> + * @grid_height: Height of lens shading tables in grid cells.
> + * @dmabuf: dmabuf file handle containing the table.
> + * @ref_transform: Reference transform - unsupported, please pass zero.
> + * @corner_sampled: Whether the gains are sampled at the corner points
> + * of the grid cells or in the cell centres.
> + * @gain_format: Format of the gains (see enum &bcm2835_isp_gain_format).
> + */
> +struct bcm2835_isp_lens_shading {
> + __u32 enabled;
> + __u32 grid_cell_size;
> + __u32 grid_width;
> + __u32 grid_stride;
> + __u32 grid_height;
> + __s32 dmabuf;
> + __u32 ref_transform;
> + __u32 corner_sampled;
> + __u32 gain_format;
> +};
> +
> +/**
> + * struct bcm2835_isp_black_level - Sensor black level set with the
> + * V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL ctrl.
> + *
> + * @enabled: Enable black level.
> + * @black_level_r: Black level for red channel.
> + * @black_level_g: Black level for green channels.
> + * @black_level_b: Black level for blue channel.
> + */
> +struct bcm2835_isp_black_level {
> + __u32 enabled;
> + __u16 black_level_r;
> + __u16 black_level_g;
> + __u16 black_level_b;
> + __u8 padding[2]; /* Unused */
> +};
> +
> +/**
> + * struct bcm2835_isp_geq - Green equalisation parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_GEQ ctrl.
> + *
> + * @enabled: Enable green equalisation.
> + * @offset: Fixed offset of the green equalisation threshold.
> + * @slope: Slope of the green equalisation threshold.
> + */
> +struct bcm2835_isp_geq {
> + __u32 enabled;
> + __u32 offset;
> + struct bcm2835_isp_rational slope;
> +};
> +
> +#define BCM2835_NUM_GAMMA_PTS 33
> +
> +/**
> + * struct bcm2835_isp_gamma - Gamma parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_GAMMA ctrl.
> + *
> + * @enabled: Enable gamma adjustment.
> + * @X: X values of the points defining the gamma curve.
> + * Values should be scaled to 16 bits.
> + * @Y: Y values of the points defining the gamma curve.
> + * Values should be scaled to 16 bits.
> + */
> +struct bcm2835_isp_gamma {
> + __u32 enabled;
> + __u16 x[BCM2835_NUM_GAMMA_PTS];
> + __u16 y[BCM2835_NUM_GAMMA_PTS];
> +};
> +
> +/**
> + * struct bcm2835_isp_denoise - Denoise parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl.
> + *
> + * @enabled: Enable denoise.
> + * @constant: Fixed offset of the noise threshold.
> + * @slope: Slope of the noise threshold.
> + * @strength: Denoise strength between 0.0 (off) and 1.0 (maximum).
> + */
> +struct bcm2835_isp_denoise {
> + __u32 enabled;
> + __u32 constant;
> + struct bcm2835_isp_rational slope;
> + struct bcm2835_isp_rational strength;
> +};
> +
> +/**
> + * struct bcm2835_isp_sharpen - Sharpen parameters set with the
> + * V4L2_CID_USER_BCM2835_ISP_SHARPEN ctrl.
> + *
> + * @enabled: Enable sharpening.
> + * @threshold: Threshold at which to start sharpening pixels.
> + * @strength: Strength with which pixel sharpening increases.
> + * @limit: Limit to the amount of sharpening applied.
> + */
> +struct bcm2835_isp_sharpen {
> + __u32 enabled;
> + struct bcm2835_isp_rational threshold;
> + struct bcm2835_isp_rational strength;
> + struct bcm2835_isp_rational limit;
> +};
> +
> +/**
> + * enum bcm2835_isp_dpc_mode - defective pixel correction (DPC) strength.
> + *
> + * @DPC_MODE_OFF: No DPC.
> + * @DPC_MODE_NORMAL: Normal DPC.
> + * @DPC_MODE_STRONG: Strong DPC.
> + */
> +enum bcm2835_isp_dpc_mode {
> + DPC_MODE_OFF = 0,
> + DPC_MODE_NORMAL = 1,
> + DPC_MODE_STRONG = 2,
> +};
> +
> +/**
> + * struct bcm2835_isp_dpc - Defective pixel correction (DPC) parameters set
> + * with the V4L2_CID_USER_BCM2835_ISP_DPC ctrl.
> + *
> + * @enabled: Enable DPC.
> + * @strength: DPC strength (see enum &bcm2835_isp_dpc_mode).
> + */
> +struct bcm2835_isp_dpc {
> + __u32 enabled;
> + __u32 strength;
> +};
> +
> +/*
> + * ISP statistics structures.
> + *
> + * The bcm2835_isp_stats structure is generated at the output of the
> + * statistics node. Note that this does not directly map onto the statistics
> + * output of the ISP HW. Instead, the MMAL firmware code maps the HW statistics
> + * to the bcm2835_isp_stats structure.
> + */
> +#define DEFAULT_AWB_REGIONS_X 16
> +#define DEFAULT_AWB_REGIONS_Y 12
> +
> +#define NUM_HISTOGRAMS 2
> +#define NUM_HISTOGRAM_BINS 128
> +#define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
> +#define FLOATING_REGIONS 16
> +#define AGC_REGIONS 16
> +#define FOCUS_REGIONS 12
> +
> +/**
> + * struct bcm2835_isp_stats_hist - Histogram statistics
> + *
> + * @r_hist: Red channel histogram.
> + * @g_hist: Combined green channel histogram.
> + * @b_hist: Blue channel histogram.
> + */
> +struct bcm2835_isp_stats_hist {
> + __u32 r_hist[NUM_HISTOGRAM_BINS];
> + __u32 g_hist[NUM_HISTOGRAM_BINS];
> + __u32 b_hist[NUM_HISTOGRAM_BINS];
> +};
> +
> +/**
> + * struct bcm2835_isp_stats_region - Region sums.
> + *
> + * @counted: The number of 2x2 bayer tiles accumulated.
> + * @notcounted: The number of 2x2 bayer tiles not accumulated.
> + * @r_sum: Total sum of counted pixels in the red channel for a region.
> + * @g_sum: Total sum of counted pixels in the green channel for a region.
> + * @b_sum: Total sum of counted pixels in the blue channel for a region.
> + */
> +struct bcm2835_isp_stats_region {
> + __u32 counted;
> + __u32 notcounted;
> + __u64 r_sum;
> + __u64 g_sum;
> + __u64 b_sum;
> +};
> +
> +/**
> + * struct bcm2835_isp_stats_focus - Focus statistics.
> + *
> + * @contrast_val: Focus measure - accumulated output of the focus filter.
> + * In the first dimension, index [0] counts pixels below a
> + * preset threshold, and index [1] counts pixels above the
> + * threshold. In the second dimension, index [0] uses the
> + * first predefined filter, and index [1] uses the second
> + * predefined filter.
> + * @contrast_val_num: The number of counted pixels in the above accumulation.
> + */
> +struct bcm2835_isp_stats_focus {
> + __u64 contrast_val[2][2];
> + __u32 contrast_val_num[2][2];
> +};
> +
> +/**
> + * struct bcm2835_isp_stats - ISP statistics.
> + *
> + * @version: Version of the bcm2835_isp_stats structure.
> + * @size: Size of the bcm2835_isp_stats structure.
> + * @hist: Histogram statistics for the entire image.
> + * @awb_stats: Statistics for the regions defined for AWB calculations.
> + * @floating_stats: Statistics for arbitrarily placed (floating) regions.
> + * @agc_stats: Statistics for the regions defined for AGC calculations.
> + * @focus_stats: Focus filter statistics for the focus regions.
> + */
> +struct bcm2835_isp_stats {
> + __u32 version;
> + __u32 size;
> + struct bcm2835_isp_stats_hist hist[NUM_HISTOGRAMS];
> + struct bcm2835_isp_stats_region awb_stats[AWB_REGIONS];
> + struct bcm2835_isp_stats_region floating_stats[FLOATING_REGIONS];
> + struct bcm2835_isp_stats_region agc_stats[AGC_REGIONS];
> + struct bcm2835_isp_stats_focus focus_stats[FOCUS_REGIONS];
> +};
> +
> +#endif /* __BCM2835_ISP_H_ */
> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> index b5e7d082b8ad..31606d42ec58 100644
> --- a/include/uapi/linux/v4l2-controls.h
> +++ b/include/uapi/linux/v4l2-controls.h
> @@ -217,6 +217,7 @@ enum v4l2_colorfx {
> * The base for Allegro driver controls.
> * We reserve 16 controls for this driver.
> */
> +
Unrelated change.
> #define V4L2_CID_USER_ALLEGRO_BASE (V4L2_CID_USER_BASE + 0x1170)
>
> /*
> @@ -231,6 +232,11 @@ enum v4l2_colorfx {
> */
> #define V4L2_CID_USER_DW100_BASE (V4L2_CID_USER_BASE + 0x1190)
>
> +/* The base for the bcm2835-isp driver controls.
/*
* The base for the bcm2835-isp driver controls.
> + * We reserve 16 controls for this driver.
> + */
> +#define V4L2_CID_USER_BCM2835_ISP_BASE (V4L2_CID_USER_BASE + 0x11a0)
This will be interesting to handle in libcamera, as it breaks the ABI
with the downstream driver.
> +
> /* MPEG-class control IDs */
> /* The MPEG controls are applicable to all codec controls
> * and the 'MPEG' part of the define is historical */
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:29 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:29 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Naush,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:20AM +0530, Umang Jain wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Add a second (identical) set of device nodes to allow concurrent use of
> the bcm2835-isp hardware by another user. This change effectively
> creates a second state structure (struct bcm2835_isp_dev) to maintain
> independent state for the second user. Node and media entity names are
> appened with the instance index appropriately.
>
> Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
> define.
>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
Let's squash this with 09/14.
> ---
> .../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
> 1 file changed, 61 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> index cb7cdba76682..0dbcb25595e7 100644
> --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> @@ -28,13 +28,19 @@
>
> MODULE_IMPORT_NS(DMA_BUF);
>
> +/*
> + * We want to instantiate 2 independent instances allowing 2 simultaneous users
> + * of the ISP hardware.
> + */
> +#define BCM2835_ISP_NUM_INSTANCES 2
> +
> static unsigned int debug;
> module_param(debug, uint, 0644);
> MODULE_PARM_DESC(debug, "activates debug info");
>
> -static unsigned int video_nr = 13;
> -module_param(video_nr, uint, 0644);
> -MODULE_PARM_DESC(video_nr, "base video device number");
> +static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
> +module_param_array(video_nr, uint, NULL, 0644);
> +MODULE_PARM_DESC(video_nr, "base video device numbers");
>
> #define BCM2835_ISP_NAME "bcm2835-isp"
> #define BCM2835_ISP_ENTITY_NAME_LEN 32
> @@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
> * or output nodes.
> */
> static int register_node(struct bcm2835_isp_dev *dev,
> + unsigned int instance,
> struct bcm2835_isp_node *node,
> int index)
> {
> @@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
> snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
> node->name, node->id);
>
> - ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
> + ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
> if (ret) {
> v4l2_err(&dev->v4l2_dev,
> "Failed to register video %s[%d] device node\n",
> @@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
> return ret;
> }
>
> -static int bcm2835_isp_remove(struct platform_device *pdev)
> +static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
> {
> - struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
> unsigned int i;
>
> media_controller_unregister(dev);
> @@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
> dev->component);
>
> vchiq_mmal_finalise(dev->mmal_instance);
> -
> - return 0;
> }
>
> -static int bcm2835_isp_probe(struct platform_device *pdev)
> +static int bcm2835_isp_probe_instance(struct platform_device *pdev,
> + struct bcm2835_isp_dev **dev_int,
> + unsigned int instance)
> {
> struct bcm2835_isp_dev *dev;
> unsigned int i;
> @@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> if (!dev)
> return -ENOMEM;
>
> + *dev_int = dev;
> dev->dev = &pdev->dev;
>
> ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
> @@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> if (ret) {
> v4l2_err(&dev->v4l2_dev,
> "%s: failed to create ril.isp component\n", __func__);
> - goto error;
> + return ret;
> }
>
> if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
> @@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> BCM2835_ISP_NUM_OUTPUTS,
> dev->component->outputs,
> BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
> - goto error;
> + return -EINVAL;
> }
>
> atomic_set(&dev->num_streaming, 0);
> @@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
> struct bcm2835_isp_node *node = &dev->node[i];
>
> - ret = register_node(dev, node, i);
> + ret = register_node(dev, instance, node, i);
> if (ret)
> - goto error;
> + return ret;
> }
>
> ret = media_controller_register(dev);
> if (ret)
> - goto error;
> + return ret;
> +
> + return 0;
> +}
> +
> +static int bcm2835_isp_remove(struct platform_device *pdev)
> +{
> + struct bcm2835_isp_dev **bcm2835_isp_instances;
> + unsigned int i;
> +
> + bcm2835_isp_instances = platform_get_drvdata(pdev);
> + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> + if (bcm2835_isp_instances[i])
> + bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
> + }
> +
> + return 0;
> +}
> +
> +static int bcm2835_isp_probe(struct platform_device *pdev)
> +{
> + struct bcm2835_isp_dev **bcm2835_isp_instances;
> + unsigned int i;
> + int ret;
> +
> + bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
> + sizeof(bcm2835_isp_instances) *
> + BCM2835_ISP_NUM_INSTANCES,
> + GFP_KERNEL);
> + if (!bcm2835_isp_instances)
> + return -ENOMEM;
> +
> + platform_set_drvdata(pdev, bcm2835_isp_instances);
> +
> + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> + ret = bcm2835_isp_probe_instance(pdev,
> + &bcm2835_isp_instances[i], i);
> + if (ret)
> + goto error;
> + }
>
> - platform_set_drvdata(pdev, dev);
> - v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> + dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> return 0;
>
> error:
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users
@ 2022-11-21 23:29 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:29 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Naush,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:20AM +0530, Umang Jain wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Add a second (identical) set of device nodes to allow concurrent use of
> the bcm2835-isp hardware by another user. This change effectively
> creates a second state structure (struct bcm2835_isp_dev) to maintain
> independent state for the second user. Node and media entity names are
> appened with the instance index appropriately.
>
> Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
> define.
>
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
Let's squash this with 09/14.
> ---
> .../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
> 1 file changed, 61 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> index cb7cdba76682..0dbcb25595e7 100644
> --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> @@ -28,13 +28,19 @@
>
> MODULE_IMPORT_NS(DMA_BUF);
>
> +/*
> + * We want to instantiate 2 independent instances allowing 2 simultaneous users
> + * of the ISP hardware.
> + */
> +#define BCM2835_ISP_NUM_INSTANCES 2
> +
> static unsigned int debug;
> module_param(debug, uint, 0644);
> MODULE_PARM_DESC(debug, "activates debug info");
>
> -static unsigned int video_nr = 13;
> -module_param(video_nr, uint, 0644);
> -MODULE_PARM_DESC(video_nr, "base video device number");
> +static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
> +module_param_array(video_nr, uint, NULL, 0644);
> +MODULE_PARM_DESC(video_nr, "base video device numbers");
>
> #define BCM2835_ISP_NAME "bcm2835-isp"
> #define BCM2835_ISP_ENTITY_NAME_LEN 32
> @@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
> * or output nodes.
> */
> static int register_node(struct bcm2835_isp_dev *dev,
> + unsigned int instance,
> struct bcm2835_isp_node *node,
> int index)
> {
> @@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
> snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
> node->name, node->id);
>
> - ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
> + ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
> if (ret) {
> v4l2_err(&dev->v4l2_dev,
> "Failed to register video %s[%d] device node\n",
> @@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
> return ret;
> }
>
> -static int bcm2835_isp_remove(struct platform_device *pdev)
> +static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
> {
> - struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
> unsigned int i;
>
> media_controller_unregister(dev);
> @@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
> dev->component);
>
> vchiq_mmal_finalise(dev->mmal_instance);
> -
> - return 0;
> }
>
> -static int bcm2835_isp_probe(struct platform_device *pdev)
> +static int bcm2835_isp_probe_instance(struct platform_device *pdev,
> + struct bcm2835_isp_dev **dev_int,
> + unsigned int instance)
> {
> struct bcm2835_isp_dev *dev;
> unsigned int i;
> @@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> if (!dev)
> return -ENOMEM;
>
> + *dev_int = dev;
> dev->dev = &pdev->dev;
>
> ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
> @@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> if (ret) {
> v4l2_err(&dev->v4l2_dev,
> "%s: failed to create ril.isp component\n", __func__);
> - goto error;
> + return ret;
> }
>
> if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
> @@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> BCM2835_ISP_NUM_OUTPUTS,
> dev->component->outputs,
> BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
> - goto error;
> + return -EINVAL;
> }
>
> atomic_set(&dev->num_streaming, 0);
> @@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
> struct bcm2835_isp_node *node = &dev->node[i];
>
> - ret = register_node(dev, node, i);
> + ret = register_node(dev, instance, node, i);
> if (ret)
> - goto error;
> + return ret;
> }
>
> ret = media_controller_register(dev);
> if (ret)
> - goto error;
> + return ret;
> +
> + return 0;
> +}
> +
> +static int bcm2835_isp_remove(struct platform_device *pdev)
> +{
> + struct bcm2835_isp_dev **bcm2835_isp_instances;
> + unsigned int i;
> +
> + bcm2835_isp_instances = platform_get_drvdata(pdev);
> + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> + if (bcm2835_isp_instances[i])
> + bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
> + }
> +
> + return 0;
> +}
> +
> +static int bcm2835_isp_probe(struct platform_device *pdev)
> +{
> + struct bcm2835_isp_dev **bcm2835_isp_instances;
> + unsigned int i;
> + int ret;
> +
> + bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
> + sizeof(bcm2835_isp_instances) *
> + BCM2835_ISP_NUM_INSTANCES,
> + GFP_KERNEL);
> + if (!bcm2835_isp_instances)
> + return -ENOMEM;
> +
> + platform_set_drvdata(pdev, bcm2835_isp_instances);
> +
> + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> + ret = bcm2835_isp_probe_instance(pdev,
> + &bcm2835_isp_instances[i], i);
> + if (ret)
> + goto error;
> + }
>
> - platform_set_drvdata(pdev, dev);
> - v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> + dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> return 0;
>
> error:
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users
2022-11-21 23:29 ` Laurent Pinchart
@ 2022-11-21 23:35 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:35 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
On Tue, Nov 22, 2022 at 01:29:48AM +0200, Laurent Pinchart wrote:
> Hi Umang and Naush,
>
> Thank you for the patch.
>
> On Tue, Nov 22, 2022 at 03:17:20AM +0530, Umang Jain wrote:
> > From: Naushir Patuck <naush@raspberrypi.com>
> >
> > Add a second (identical) set of device nodes to allow concurrent use of
> > the bcm2835-isp hardware by another user. This change effectively
> > creates a second state structure (struct bcm2835_isp_dev) to maintain
> > independent state for the second user. Node and media entity names are
> > appened with the instance index appropriately.
> >
> > Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
> > define.
> >
> > Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> > Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
>
> Let's squash this with 09/14.
Also, the documentation in 13/14 should be updated.
> > ---
> > .../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
> > 1 file changed, 61 insertions(+), 16 deletions(-)
> >
> > diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > index cb7cdba76682..0dbcb25595e7 100644
> > --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > @@ -28,13 +28,19 @@
> >
> > MODULE_IMPORT_NS(DMA_BUF);
> >
> > +/*
> > + * We want to instantiate 2 independent instances allowing 2 simultaneous users
> > + * of the ISP hardware.
> > + */
> > +#define BCM2835_ISP_NUM_INSTANCES 2
> > +
> > static unsigned int debug;
> > module_param(debug, uint, 0644);
> > MODULE_PARM_DESC(debug, "activates debug info");
> >
> > -static unsigned int video_nr = 13;
> > -module_param(video_nr, uint, 0644);
> > -MODULE_PARM_DESC(video_nr, "base video device number");
> > +static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
> > +module_param_array(video_nr, uint, NULL, 0644);
> > +MODULE_PARM_DESC(video_nr, "base video device numbers");
> >
> > #define BCM2835_ISP_NAME "bcm2835-isp"
> > #define BCM2835_ISP_ENTITY_NAME_LEN 32
> > @@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
> > * or output nodes.
> > */
> > static int register_node(struct bcm2835_isp_dev *dev,
> > + unsigned int instance,
> > struct bcm2835_isp_node *node,
> > int index)
> > {
> > @@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
> > snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
> > node->name, node->id);
> >
> > - ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
> > + ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
> > if (ret) {
> > v4l2_err(&dev->v4l2_dev,
> > "Failed to register video %s[%d] device node\n",
> > @@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
> > return ret;
> > }
> >
> > -static int bcm2835_isp_remove(struct platform_device *pdev)
> > +static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
> > {
> > - struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
> > unsigned int i;
> >
> > media_controller_unregister(dev);
> > @@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
> > dev->component);
> >
> > vchiq_mmal_finalise(dev->mmal_instance);
> > -
> > - return 0;
> > }
> >
> > -static int bcm2835_isp_probe(struct platform_device *pdev)
> > +static int bcm2835_isp_probe_instance(struct platform_device *pdev,
> > + struct bcm2835_isp_dev **dev_int,
> > + unsigned int instance)
> > {
> > struct bcm2835_isp_dev *dev;
> > unsigned int i;
> > @@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > if (!dev)
> > return -ENOMEM;
> >
> > + *dev_int = dev;
> > dev->dev = &pdev->dev;
> >
> > ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
> > @@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > if (ret) {
> > v4l2_err(&dev->v4l2_dev,
> > "%s: failed to create ril.isp component\n", __func__);
> > - goto error;
> > + return ret;
> > }
> >
> > if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
> > @@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > BCM2835_ISP_NUM_OUTPUTS,
> > dev->component->outputs,
> > BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
> > - goto error;
> > + return -EINVAL;
> > }
> >
> > atomic_set(&dev->num_streaming, 0);
> > @@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
> > struct bcm2835_isp_node *node = &dev->node[i];
> >
> > - ret = register_node(dev, node, i);
> > + ret = register_node(dev, instance, node, i);
> > if (ret)
> > - goto error;
> > + return ret;
> > }
> >
> > ret = media_controller_register(dev);
> > if (ret)
> > - goto error;
> > + return ret;
> > +
> > + return 0;
> > +}
> > +
> > +static int bcm2835_isp_remove(struct platform_device *pdev)
> > +{
> > + struct bcm2835_isp_dev **bcm2835_isp_instances;
> > + unsigned int i;
> > +
> > + bcm2835_isp_instances = platform_get_drvdata(pdev);
> > + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> > + if (bcm2835_isp_instances[i])
> > + bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int bcm2835_isp_probe(struct platform_device *pdev)
> > +{
> > + struct bcm2835_isp_dev **bcm2835_isp_instances;
> > + unsigned int i;
> > + int ret;
> > +
> > + bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
> > + sizeof(bcm2835_isp_instances) *
> > + BCM2835_ISP_NUM_INSTANCES,
> > + GFP_KERNEL);
> > + if (!bcm2835_isp_instances)
> > + return -ENOMEM;
> > +
> > + platform_set_drvdata(pdev, bcm2835_isp_instances);
> > +
> > + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> > + ret = bcm2835_isp_probe_instance(pdev,
> > + &bcm2835_isp_instances[i], i);
> > + if (ret)
> > + goto error;
> > + }
> >
> > - platform_set_drvdata(pdev, dev);
> > - v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> > + dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> > return 0;
> >
> > error:
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users
@ 2022-11-21 23:35 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:35 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
On Tue, Nov 22, 2022 at 01:29:48AM +0200, Laurent Pinchart wrote:
> Hi Umang and Naush,
>
> Thank you for the patch.
>
> On Tue, Nov 22, 2022 at 03:17:20AM +0530, Umang Jain wrote:
> > From: Naushir Patuck <naush@raspberrypi.com>
> >
> > Add a second (identical) set of device nodes to allow concurrent use of
> > the bcm2835-isp hardware by another user. This change effectively
> > creates a second state structure (struct bcm2835_isp_dev) to maintain
> > independent state for the second user. Node and media entity names are
> > appened with the instance index appropriately.
> >
> > Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES
> > define.
> >
> > Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> > Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
>
> Let's squash this with 09/14.
Also, the documentation in 13/14 should be updated.
> > ---
> > .../bcm2835-isp/bcm2835-v4l2-isp.c | 77 +++++++++++++++----
> > 1 file changed, 61 insertions(+), 16 deletions(-)
> >
> > diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > index cb7cdba76682..0dbcb25595e7 100644
> > --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > @@ -28,13 +28,19 @@
> >
> > MODULE_IMPORT_NS(DMA_BUF);
> >
> > +/*
> > + * We want to instantiate 2 independent instances allowing 2 simultaneous users
> > + * of the ISP hardware.
> > + */
> > +#define BCM2835_ISP_NUM_INSTANCES 2
> > +
> > static unsigned int debug;
> > module_param(debug, uint, 0644);
> > MODULE_PARM_DESC(debug, "activates debug info");
> >
> > -static unsigned int video_nr = 13;
> > -module_param(video_nr, uint, 0644);
> > -MODULE_PARM_DESC(video_nr, "base video device number");
> > +static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 };
> > +module_param_array(video_nr, uint, NULL, 0644);
> > +MODULE_PARM_DESC(video_nr, "base video device numbers");
> >
> > #define BCM2835_ISP_NAME "bcm2835-isp"
> > #define BCM2835_ISP_ENTITY_NAME_LEN 32
> > @@ -1286,6 +1292,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node)
> > * or output nodes.
> > */
> > static int register_node(struct bcm2835_isp_dev *dev,
> > + unsigned int instance,
> > struct bcm2835_isp_node *node,
> > int index)
> > {
> > @@ -1447,7 +1454,7 @@ static int register_node(struct bcm2835_isp_dev *dev,
> > snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME,
> > node->name, node->id);
> >
> > - ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index);
> > + ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]);
> > if (ret) {
> > v4l2_err(&dev->v4l2_dev,
> > "Failed to register video %s[%d] device node\n",
> > @@ -1668,9 +1675,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev)
> > return ret;
> > }
> >
> > -static int bcm2835_isp_remove(struct platform_device *pdev)
> > +static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev)
> > {
> > - struct bcm2835_isp_dev *dev = platform_get_drvdata(pdev);
> > unsigned int i;
> >
> > media_controller_unregister(dev);
> > @@ -1685,11 +1691,11 @@ static int bcm2835_isp_remove(struct platform_device *pdev)
> > dev->component);
> >
> > vchiq_mmal_finalise(dev->mmal_instance);
> > -
> > - return 0;
> > }
> >
> > -static int bcm2835_isp_probe(struct platform_device *pdev)
> > +static int bcm2835_isp_probe_instance(struct platform_device *pdev,
> > + struct bcm2835_isp_dev **dev_int,
> > + unsigned int instance)
> > {
> > struct bcm2835_isp_dev *dev;
> > unsigned int i;
> > @@ -1699,6 +1705,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > if (!dev)
> > return -ENOMEM;
> >
> > + *dev_int = dev;
> > dev->dev = &pdev->dev;
> >
> > ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
> > @@ -1716,7 +1723,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > if (ret) {
> > v4l2_err(&dev->v4l2_dev,
> > "%s: failed to create ril.isp component\n", __func__);
> > - goto error;
> > + return ret;
> > }
> >
> > if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS ||
> > @@ -1728,7 +1735,7 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > BCM2835_ISP_NUM_OUTPUTS,
> > dev->component->outputs,
> > BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA);
> > - goto error;
> > + return -EINVAL;
> > }
> >
> > atomic_set(&dev->num_streaming, 0);
> > @@ -1736,17 +1743,55 @@ static int bcm2835_isp_probe(struct platform_device *pdev)
> > for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) {
> > struct bcm2835_isp_node *node = &dev->node[i];
> >
> > - ret = register_node(dev, node, i);
> > + ret = register_node(dev, instance, node, i);
> > if (ret)
> > - goto error;
> > + return ret;
> > }
> >
> > ret = media_controller_register(dev);
> > if (ret)
> > - goto error;
> > + return ret;
> > +
> > + return 0;
> > +}
> > +
> > +static int bcm2835_isp_remove(struct platform_device *pdev)
> > +{
> > + struct bcm2835_isp_dev **bcm2835_isp_instances;
> > + unsigned int i;
> > +
> > + bcm2835_isp_instances = platform_get_drvdata(pdev);
> > + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> > + if (bcm2835_isp_instances[i])
> > + bcm2835_isp_remove_instance(bcm2835_isp_instances[i]);
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static int bcm2835_isp_probe(struct platform_device *pdev)
> > +{
> > + struct bcm2835_isp_dev **bcm2835_isp_instances;
> > + unsigned int i;
> > + int ret;
> > +
> > + bcm2835_isp_instances = devm_kzalloc(&pdev->dev,
> > + sizeof(bcm2835_isp_instances) *
> > + BCM2835_ISP_NUM_INSTANCES,
> > + GFP_KERNEL);
> > + if (!bcm2835_isp_instances)
> > + return -ENOMEM;
> > +
> > + platform_set_drvdata(pdev, bcm2835_isp_instances);
> > +
> > + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) {
> > + ret = bcm2835_isp_probe_instance(pdev,
> > + &bcm2835_isp_instances[i], i);
> > + if (ret)
> > + goto error;
> > + }
> >
> > - platform_set_drvdata(pdev, dev);
> > - v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> > + dev_info(&pdev->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME);
> > return 0;
> >
> > error:
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 13/14] docs: admin-guide: media: bcm2835-isp: Add documentation for bcm2835-isp
2022-11-21 21:47 ` Umang Jain
@ 2022-11-21 23:41 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:41 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Naush,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:21AM +0530, Umang Jain wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Document device nodes, frame buffer formats and initial set of
> configuraiton that can be set from userspace to configure the pipeline.
s/configuraiton/configuration/
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> .../admin-guide/media/bcm2835-isp.rst | 127 ++++++++++++++++++
> .../userspace-api/media/drivers/index.rst | 1 +
> 2 files changed, 128 insertions(+)
> create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
>
> diff --git a/Documentation/admin-guide/media/bcm2835-isp.rst b/Documentation/admin-guide/media/bcm2835-isp.rst
> new file mode 100644
> index 000000000000..e1c19f78435e
> --- /dev/null
> +++ b/Documentation/admin-guide/media/bcm2835-isp.rst
> @@ -0,0 +1,127 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +BCM2835 ISP Driver
> +==================
> +
> +Introduction
> +------------
> +
> +The BCM2835 Image Sensor Pipeline (ISP) is a fixed function hardware pipeline
> +for performing image processing operations. Images are fed to the input
> +of the ISP through memory frame buffers. These images may be in various YUV,
> +RGB, or Bayer formats.
Does the ISP really support RGB or YUV input images ? Is this supported
by the driver ?
> A typical use case would have Bayer images obtained from
> +an image sensor by the BCM2835 Unicam peripheral, written to a memory
> +frame buffer, and finally fed into the input of the ISP. Two concurrent output
> +images may be generated in YUV or RGB format at different resolutions.
> +Statistics output is also generated for Bayer input images.
> +
> +The bcm2835-isp driver exposes the following media pads as V4L2 device nodes:
This sounds weird. Pads are not video nodes, are they ? Or are those
MMAL pads ?
> +
> +.. tabularcolumns:: |l|l|l|l|
> +
> +.. cssclass: longtable
> +
> +.. flat-table::
> +
> + * - *Pad*
> + - *Direction*
> + - *Purpose*
> + - *Formats*
> +
> + * - "bcm2835-isp0-output0"
> + - sink
> + - Accepts Bayer, RGB or YUV format frame buffers as input to the ISP HW
> + pipeline.
> + - :ref:`RAW8 <V4L2-PIX-FMT-SRGGB8>`,
> + :ref:`RAW10P <V4L2-PIX-FMT-SRGGB10P>`,
> + :ref:`RAW12P <V4L2-PIX-FMT-SRGGB12P>`,
> + :ref:`RAW14P <V4L2-PIX-FMT-SRGGB14P>`,
> + :ref:`RAW16 <V4L2-PIX-FMT-SRGGB16>`,
> + :ref:`RGB24/BGR24 <V4L2-PIX-FMT-RGB24>`,
> + :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
> + :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
> + :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
> + :ref:`VYUY <V4L2-PIX-FMT-VYUY>`,
> + :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`
> +
> + * - "bcm2835-isp0-capture1"
> + - source
> + - High resolution YUV or RGB processed output from the ISP.
> + - :ref:`RGB565 <V4L2-PIX-FMT-RGB565>`,
> + :ref:`RGB24/BGR24 <V4L2-PIX-FMT-RGB24>`,
> + :ref:`ABGR32 <V4L2-PIX-FMT-ABGR32>`,
> + :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
> + :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
> + :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
> + :ref:`VYUY <V4L2-PIX-FMT-VYUY>`.
> + :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`,
> + :ref:`NV12/NV21 <V4L2-PIX-FMT-NV12>`,
> +
> + * - "bcm2835-isp0-capture2"
> + - source
> + - Low resolution YUV processed output from the ISP. The output of
> + this pad cannot have a resolution larger than the "bcm2835-isp0-capture1" pad in any dimension.
> + - :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
> + :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
> + :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
> + :ref:`VYUY <V4L2-PIX-FMT-VYUY>`.
> + :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`,
> + :ref:`NV12/NV21 <V4L2-PIX-FMT-NV12>`,
> +
> + * - "bcm2835-isp0-capture1"
Is that capture3 ?
Maybe the video nodes should be renamed to make their purpose clearer ?
> + - source
> + - Image statistics calculated from the input image provided on the
> + "bcm2835-isp0-output0" pad. Statistics are only available for Bayer
> + format input images.
> + - :ref:`v4l2-meta-fmt-bcm2835-isp-stats`.
> +
> +Pipeline Configuration
> +----------------------
> +
> +The ISP pipeline can be configure through user-space by calling
> +:ref:`VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` on the “bcm2835-isp0-output0”
> +node with the appropriate parameters as shown in the table below.
> +
> +.. tabularcolumns:: |p{2cm}|p{5.0cm}|
> +
> +.. cssclass: longtable
> +
> +.. flat-table::
> +
> + * - *id*
> + - *Parameter*
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_CC_MATRIX``
> + - struct :c:type:`bcm2835_isp_custom_ccm`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_LENS_SHADING``
> + - struct :c:type:`bcm2835_isp_lens_shading`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL``
> + - struct :c:type:`bcm2835_isp_black_level`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_GEQ``
> + - struct :c:type:`bcm2835_isp_geq`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_GAMMA``
> + - struct :c:type:`bcm2835_isp_gamma`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_DENOISE``
> + - struct :c:type:`bcm2835_isp_denoise`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_SHARPEN``
> + - struct :c:type:`bcm2835_isp_sharpen`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_DPC``
> + - struct :c:type:`bcm2835_isp_dpc`
> +
> +++++++++++++++++++++++++
> +Configuration Parameters
> +++++++++++++++++++++++++
> +
> +.. kernel-doc:: include/uapi/linux/bcm2835-isp.h
> + :functions: bcm2835_isp_rational bcm2835_isp_ccm bcm2835_isp_custom_ccm
> + bcm2835_isp_gain_format bcm2835_isp_lens_shading
> + bcm2835_isp_black_level bcm2835_isp_geq bcm2835_isp_gamma
> + bcm2835_isp_denoise bcm2835_isp_sharpen
> + bcm2835_isp_dpc_mode bcm2835_isp_dpc
> diff --git a/Documentation/userspace-api/media/drivers/index.rst b/Documentation/userspace-api/media/drivers/index.rst
> index 32f82aed47d9..34e0d7102ef0 100644
> --- a/Documentation/userspace-api/media/drivers/index.rst
> +++ b/Documentation/userspace-api/media/drivers/index.rst
> @@ -31,6 +31,7 @@ For more details see the file COPYING in the source distribution of Linux.
> :maxdepth: 5
> :numbered:
>
> + bcm2835-isp
> ccs
> cx2341x-uapi
> dw100
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 13/14] docs: admin-guide: media: bcm2835-isp: Add documentation for bcm2835-isp
@ 2022-11-21 23:41 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-21 23:41 UTC (permalink / raw)
To: Umang Jain
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Naush,
Thank you for the patch.
On Tue, Nov 22, 2022 at 03:17:21AM +0530, Umang Jain wrote:
> From: Naushir Patuck <naush@raspberrypi.com>
>
> Document device nodes, frame buffer formats and initial set of
> configuraiton that can be set from userspace to configure the pipeline.
s/configuraiton/configuration/
> Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
> Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> ---
> .../admin-guide/media/bcm2835-isp.rst | 127 ++++++++++++++++++
> .../userspace-api/media/drivers/index.rst | 1 +
> 2 files changed, 128 insertions(+)
> create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
>
> diff --git a/Documentation/admin-guide/media/bcm2835-isp.rst b/Documentation/admin-guide/media/bcm2835-isp.rst
> new file mode 100644
> index 000000000000..e1c19f78435e
> --- /dev/null
> +++ b/Documentation/admin-guide/media/bcm2835-isp.rst
> @@ -0,0 +1,127 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +BCM2835 ISP Driver
> +==================
> +
> +Introduction
> +------------
> +
> +The BCM2835 Image Sensor Pipeline (ISP) is a fixed function hardware pipeline
> +for performing image processing operations. Images are fed to the input
> +of the ISP through memory frame buffers. These images may be in various YUV,
> +RGB, or Bayer formats.
Does the ISP really support RGB or YUV input images ? Is this supported
by the driver ?
> A typical use case would have Bayer images obtained from
> +an image sensor by the BCM2835 Unicam peripheral, written to a memory
> +frame buffer, and finally fed into the input of the ISP. Two concurrent output
> +images may be generated in YUV or RGB format at different resolutions.
> +Statistics output is also generated for Bayer input images.
> +
> +The bcm2835-isp driver exposes the following media pads as V4L2 device nodes:
This sounds weird. Pads are not video nodes, are they ? Or are those
MMAL pads ?
> +
> +.. tabularcolumns:: |l|l|l|l|
> +
> +.. cssclass: longtable
> +
> +.. flat-table::
> +
> + * - *Pad*
> + - *Direction*
> + - *Purpose*
> + - *Formats*
> +
> + * - "bcm2835-isp0-output0"
> + - sink
> + - Accepts Bayer, RGB or YUV format frame buffers as input to the ISP HW
> + pipeline.
> + - :ref:`RAW8 <V4L2-PIX-FMT-SRGGB8>`,
> + :ref:`RAW10P <V4L2-PIX-FMT-SRGGB10P>`,
> + :ref:`RAW12P <V4L2-PIX-FMT-SRGGB12P>`,
> + :ref:`RAW14P <V4L2-PIX-FMT-SRGGB14P>`,
> + :ref:`RAW16 <V4L2-PIX-FMT-SRGGB16>`,
> + :ref:`RGB24/BGR24 <V4L2-PIX-FMT-RGB24>`,
> + :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
> + :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
> + :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
> + :ref:`VYUY <V4L2-PIX-FMT-VYUY>`,
> + :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`
> +
> + * - "bcm2835-isp0-capture1"
> + - source
> + - High resolution YUV or RGB processed output from the ISP.
> + - :ref:`RGB565 <V4L2-PIX-FMT-RGB565>`,
> + :ref:`RGB24/BGR24 <V4L2-PIX-FMT-RGB24>`,
> + :ref:`ABGR32 <V4L2-PIX-FMT-ABGR32>`,
> + :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
> + :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
> + :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
> + :ref:`VYUY <V4L2-PIX-FMT-VYUY>`.
> + :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`,
> + :ref:`NV12/NV21 <V4L2-PIX-FMT-NV12>`,
> +
> + * - "bcm2835-isp0-capture2"
> + - source
> + - Low resolution YUV processed output from the ISP. The output of
> + this pad cannot have a resolution larger than the "bcm2835-isp0-capture1" pad in any dimension.
> + - :ref:`YUYV <V4L2-PIX-FMT-YUYV>`,
> + :ref:`YVYU <V4L2-PIX-FMT-YVYU>`,
> + :ref:`UYVY <V4L2-PIX-FMT-UYVY>`,
> + :ref:`VYUY <V4L2-PIX-FMT-VYUY>`.
> + :ref:`YUV420/YVU420 <V4L2-PIX-FMT-YUV420>`,
> + :ref:`NV12/NV21 <V4L2-PIX-FMT-NV12>`,
> +
> + * - "bcm2835-isp0-capture1"
Is that capture3 ?
Maybe the video nodes should be renamed to make their purpose clearer ?
> + - source
> + - Image statistics calculated from the input image provided on the
> + "bcm2835-isp0-output0" pad. Statistics are only available for Bayer
> + format input images.
> + - :ref:`v4l2-meta-fmt-bcm2835-isp-stats`.
> +
> +Pipeline Configuration
> +----------------------
> +
> +The ISP pipeline can be configure through user-space by calling
> +:ref:`VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` on the “bcm2835-isp0-output0”
> +node with the appropriate parameters as shown in the table below.
> +
> +.. tabularcolumns:: |p{2cm}|p{5.0cm}|
> +
> +.. cssclass: longtable
> +
> +.. flat-table::
> +
> + * - *id*
> + - *Parameter*
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_CC_MATRIX``
> + - struct :c:type:`bcm2835_isp_custom_ccm`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_LENS_SHADING``
> + - struct :c:type:`bcm2835_isp_lens_shading`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL``
> + - struct :c:type:`bcm2835_isp_black_level`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_GEQ``
> + - struct :c:type:`bcm2835_isp_geq`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_GAMMA``
> + - struct :c:type:`bcm2835_isp_gamma`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_DENOISE``
> + - struct :c:type:`bcm2835_isp_denoise`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_SHARPEN``
> + - struct :c:type:`bcm2835_isp_sharpen`
> +
> + * - ``V4L2_CID_USER_BCM2835_ISP_DPC``
> + - struct :c:type:`bcm2835_isp_dpc`
> +
> +++++++++++++++++++++++++
> +Configuration Parameters
> +++++++++++++++++++++++++
> +
> +.. kernel-doc:: include/uapi/linux/bcm2835-isp.h
> + :functions: bcm2835_isp_rational bcm2835_isp_ccm bcm2835_isp_custom_ccm
> + bcm2835_isp_gain_format bcm2835_isp_lens_shading
> + bcm2835_isp_black_level bcm2835_isp_geq bcm2835_isp_gamma
> + bcm2835_isp_denoise bcm2835_isp_sharpen
> + bcm2835_isp_dpc_mode bcm2835_isp_dpc
> diff --git a/Documentation/userspace-api/media/drivers/index.rst b/Documentation/userspace-api/media/drivers/index.rst
> index 32f82aed47d9..34e0d7102ef0 100644
> --- a/Documentation/userspace-api/media/drivers/index.rst
> +++ b/Documentation/userspace-api/media/drivers/index.rst
> @@ -31,6 +31,7 @@ For more details see the file COPYING in the source distribution of Linux.
> :maxdepth: 5
> :numbered:
>
> + bcm2835-isp
> ccs
> cx2341x-uapi
> dw100
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf
2022-11-21 23:18 ` Laurent Pinchart
@ 2022-11-22 11:35 ` Dave Stevenson
-1 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-11-22 11:35 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham, Dave Stevenson
On Mon, 21 Nov 2022 at 23:18, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Umang and Dave,
>
> Thank you for the patch.
>
> On Tue, Nov 22, 2022 at 03:17:11AM +0530, Umang Jain wrote:
> > From: Dave Stevenson <dave.stevenson@raspberrypi.org>
> >
> > videobuf2 only allowed exporting a dmabuf as a file descriptor,
> > but there are instances where having the struct dma_buf is
> > useful within the kernel.
> >
> > Split the current implementation into two, one step which
> > exports a struct dma_buf, and the second which converts that
> > into an fd.
> >
> > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
> > Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> > ---
> > .../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------
> > include/media/videobuf2-core.h | 15 ++++++++
> > 2 files changed, 40 insertions(+), 11 deletions(-)
> >
> > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> > index ab9697f3b5f1..32b26737cac4 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > @@ -2184,49 +2184,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off,
> > return -EINVAL;
> > }
> >
> > -int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > - unsigned int index, unsigned int plane, unsigned int flags)
> > +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> > + unsigned int index, unsigned int plane,
> > + unsigned int flags)
>
> This function is used in the ISP driver, in bcm2835_isp_buf_prepare(),
> for MMAP buffers, and as far as I can tell, its only purpose is to
> create a dma_buf instance to then be imported in
> vchiq_mmal_submit_buffer() with a call to vc_sm_cma_import_dmabuf().
> That sounds like a very complicated set of operations, and quite
> inefficient :-(
Are you saying that dmabufs are not the preferred route for sharing
buffers between kernel subsystems? What are you suggesting instead?
If the VPU (firmware) has a handle to the buffer then we need to
manage the lifetime such that it is not freed until the VPU has
released it. That is handled for you with dmabufs, therefore why
reinvent the wheel?
Dave
> > {
> > struct vb2_buffer *vb = NULL;
> > struct vb2_plane *vb_plane;
> > - int ret;
> > struct dma_buf *dbuf;
> >
> > if (q->memory != VB2_MEMORY_MMAP) {
> > dprintk(q, 1, "queue is not currently set up for mmap\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (!q->mem_ops->get_dmabuf) {
> > dprintk(q, 1, "queue does not support DMA buffer exporting\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (flags & ~(O_CLOEXEC | O_ACCMODE)) {
> > dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (type != q->type) {
> > dprintk(q, 1, "invalid buffer type\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (index >= q->num_buffers) {
> > dprintk(q, 1, "buffer index out of range\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > vb = q->bufs[index];
> >
> > if (plane >= vb->num_planes) {
> > dprintk(q, 1, "buffer plane out of range\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (vb2_fileio_is_active(q)) {
> > dprintk(q, 1, "expbuf: file io in progress\n");
> > - return -EBUSY;
> > + return ERR_PTR(-EBUSY);
> > }
> >
> > vb_plane = &vb->planes[plane];
> > @@ -2238,9 +2238,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > if (IS_ERR_OR_NULL(dbuf)) {
> > dprintk(q, 1, "failed to export buffer %d, plane %d\n",
> > index, plane);
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > + return dbuf;
> > +}
> > +EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf);
> > +
> > +int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > + unsigned int index, unsigned int plane, unsigned int flags)
> > +{
> > + struct dma_buf *dbuf;
> > + int ret;
> > +
> > + dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags);
> > + if (IS_ERR(dbuf))
> > + return PTR_ERR(dbuf);
> > +
> > ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE);
> > if (ret < 0) {
> > dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n",
> > diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> > index 3253bd2f6fee..33629ed2b64f 100644
> > --- a/include/media/videobuf2-core.h
> > +++ b/include/media/videobuf2-core.h
> > @@ -911,6 +911,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
> > */
> > int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
> >
> > +/**
> > + * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure
> > + * @q: videobuf2 queue
> > + * @type: buffer type
> > + * @index: id number of the buffer
> > + * @plane: index of the plane to be exported, 0 for single plane queues
> > + * @flags: flags for newly created file, currently only O_CLOEXEC is
> > + * supported, refer to manual of open syscall for more details
> > + *
> > + * Return: Returns the dmabuf pointer
> > + */
> > +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> > + unsigned int index, unsigned int plane,
> > + unsigned int flags);
> > +
> > /**
> > * vb2_core_expbuf() - Export a buffer as a file descriptor.
> > * @q: pointer to &struct vb2_queue with videobuf2 queue.
>
> --
> Regards,
>
> Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf
@ 2022-11-22 11:35 ` Dave Stevenson
0 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-11-22 11:35 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham, Dave Stevenson
On Mon, 21 Nov 2022 at 23:18, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Umang and Dave,
>
> Thank you for the patch.
>
> On Tue, Nov 22, 2022 at 03:17:11AM +0530, Umang Jain wrote:
> > From: Dave Stevenson <dave.stevenson@raspberrypi.org>
> >
> > videobuf2 only allowed exporting a dmabuf as a file descriptor,
> > but there are instances where having the struct dma_buf is
> > useful within the kernel.
> >
> > Split the current implementation into two, one step which
> > exports a struct dma_buf, and the second which converts that
> > into an fd.
> >
> > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
> > Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> > ---
> > .../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------
> > include/media/videobuf2-core.h | 15 ++++++++
> > 2 files changed, 40 insertions(+), 11 deletions(-)
> >
> > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> > index ab9697f3b5f1..32b26737cac4 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > @@ -2184,49 +2184,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off,
> > return -EINVAL;
> > }
> >
> > -int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > - unsigned int index, unsigned int plane, unsigned int flags)
> > +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> > + unsigned int index, unsigned int plane,
> > + unsigned int flags)
>
> This function is used in the ISP driver, in bcm2835_isp_buf_prepare(),
> for MMAP buffers, and as far as I can tell, its only purpose is to
> create a dma_buf instance to then be imported in
> vchiq_mmal_submit_buffer() with a call to vc_sm_cma_import_dmabuf().
> That sounds like a very complicated set of operations, and quite
> inefficient :-(
Are you saying that dmabufs are not the preferred route for sharing
buffers between kernel subsystems? What are you suggesting instead?
If the VPU (firmware) has a handle to the buffer then we need to
manage the lifetime such that it is not freed until the VPU has
released it. That is handled for you with dmabufs, therefore why
reinvent the wheel?
Dave
> > {
> > struct vb2_buffer *vb = NULL;
> > struct vb2_plane *vb_plane;
> > - int ret;
> > struct dma_buf *dbuf;
> >
> > if (q->memory != VB2_MEMORY_MMAP) {
> > dprintk(q, 1, "queue is not currently set up for mmap\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (!q->mem_ops->get_dmabuf) {
> > dprintk(q, 1, "queue does not support DMA buffer exporting\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (flags & ~(O_CLOEXEC | O_ACCMODE)) {
> > dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (type != q->type) {
> > dprintk(q, 1, "invalid buffer type\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (index >= q->num_buffers) {
> > dprintk(q, 1, "buffer index out of range\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > vb = q->bufs[index];
> >
> > if (plane >= vb->num_planes) {
> > dprintk(q, 1, "buffer plane out of range\n");
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > if (vb2_fileio_is_active(q)) {
> > dprintk(q, 1, "expbuf: file io in progress\n");
> > - return -EBUSY;
> > + return ERR_PTR(-EBUSY);
> > }
> >
> > vb_plane = &vb->planes[plane];
> > @@ -2238,9 +2238,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > if (IS_ERR_OR_NULL(dbuf)) {
> > dprintk(q, 1, "failed to export buffer %d, plane %d\n",
> > index, plane);
> > - return -EINVAL;
> > + return ERR_PTR(-EINVAL);
> > }
> >
> > + return dbuf;
> > +}
> > +EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf);
> > +
> > +int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > + unsigned int index, unsigned int plane, unsigned int flags)
> > +{
> > + struct dma_buf *dbuf;
> > + int ret;
> > +
> > + dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags);
> > + if (IS_ERR(dbuf))
> > + return PTR_ERR(dbuf);
> > +
> > ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE);
> > if (ret < 0) {
> > dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n",
> > diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> > index 3253bd2f6fee..33629ed2b64f 100644
> > --- a/include/media/videobuf2-core.h
> > +++ b/include/media/videobuf2-core.h
> > @@ -911,6 +911,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
> > */
> > int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
> >
> > +/**
> > + * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure
> > + * @q: videobuf2 queue
> > + * @type: buffer type
> > + * @index: id number of the buffer
> > + * @plane: index of the plane to be exported, 0 for single plane queues
> > + * @flags: flags for newly created file, currently only O_CLOEXEC is
> > + * supported, refer to manual of open syscall for more details
> > + *
> > + * Return: Returns the dmabuf pointer
> > + */
> > +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> > + unsigned int index, unsigned int plane,
> > + unsigned int flags);
> > +
> > /**
> > * vb2_core_expbuf() - Export a buffer as a file descriptor.
> > * @q: pointer to &struct vb2_queue with videobuf2 queue.
>
> --
> Regards,
>
> Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-11-21 22:16 ` Laurent Pinchart
@ 2022-11-22 11:42 ` Dave Stevenson
-1 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-11-22 11:42 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Laurent
On Mon, 21 Nov 2022 at 22:16, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Umang,
>
> Nice to see this series on the list !
>
> On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
> > This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > SoCs (namely BCM2711). Unicam is still under active development to work
> > with multistream support to get into mainline. Hence only the ISP driver
> > will remain the primary area of this series.
> >
> > Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
> > with the VC4 VPU.
> >
> > Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
> > dma_buf instead of a file-descriptor.
> >
> > Patch (04-05)/14 adds a couple of improvements/support for
> > bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
> >
> > Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
> > and format defintions.
> >
> > Patch (10-11)/14 deals with the colorspace support.
> > Note: This is still WIP since the implementation of colorspace is still
> > getting ironed out (especially around setting of colorspace flags handling).
> >
> > Patch 12/14 allows multiple instances of the ISP.
> >
> > Patch 13/14 adds a admin-guide document on bcm2835-isp.
> >
> > Patch 14/14 wires all this up with the vchiq-mmal driver.
> >
> > Testing:
> > Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
> > end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
> > bindings from [1]). Once done, functional testing was conducted with
> > libcamera[3] and its utility tools.
> >
> > Also note: Reviews given on [2] for the relevant ISP driver patches have
> > been incorporated in this version.
> >
> > Known issues:
> > - Colorspace handling
>
> This will require further discussions, I'll try to comment on this topic
> in the review of the ISP driver patch.
>
> > - vc-sm-cma spamming kernel log with
> > - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
>
> Do you have any plan to address this ? Is the root cause known ?
You've picked up an old version of the downstream driver.
Pi0&1 share the VPU L2 cache with the ARM in the architecture, so they
use the 0x8 cache alias.
See https://github.com/raspberrypi/linux/commit/e22927f8ec9dc87772ac61d6aba00dc8046b4f49
Dave
> > [1]: https://github.com/raspberrypi/linux
> > [2]: https://lore.kernel.org/linux-media/20200504092611.9798-1-laurent.pinchart@ideasonboard.com/
> > [3]: https://libcamera.org/getting-started.html
> >
> > Dave Stevenson (7):
> > staging: vc04_services: Add new vc-sm-cma driver
> > staging: vchiq_arm: Register vcsm-cma as a platform driver
> > media: videobuf2: Allow exporting of a struct dmabuf
> > staging: mmal-vchiq: Add support for event callbacks
> > staging: mmal-vchiq: Use vc-sm-cma to support zero copy
> > staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
> > uapi: bcm2835-isp: Add bcm2835-isp uapi header file
> >
> > David Plowman (2):
> > vc04_services: bcm2835-isp: Allow formats with different colour spaces
> > vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
> > outputs
> >
> > Naushir Patuck (5):
> > media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
> > staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
> > component
> > staging: vc04_services: bcm2835_isp: Allow multiple users
> > docs: admin-guide: media: bcm2835-isp: Add documentation for
> > bcm2835-isp
> > staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
> >
> > .../admin-guide/media/bcm2835-isp.rst | 127 ++
> > .../userspace-api/media/drivers/index.rst | 1 +
> > .../userspace-api/media/v4l/meta-formats.rst | 1 +
> > .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
> > MAINTAINERS | 9 +
> > .../media/common/videobuf2/videobuf2-core.c | 36 +-
> > drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> > drivers/staging/vc04_services/Kconfig | 4 +
> > drivers/staging/vc04_services/Makefile | 2 +
> > .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
> > .../vc04_services/bcm2835-isp/Makefile | 8 +
> > .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
> > .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
> > .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
> > .../interface/vchiq_arm/vchiq_arm.c | 6 +
> > .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
> > .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
> > .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
> > .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
> > .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
> > .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
> > .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
> > .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
> > .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
> > .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
> > .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
> > .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
> > .../vchiq-mmal/mmal-parameters.h | 165 +-
> > .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
> > .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
> > include/media/videobuf2-core.h | 15 +
> > include/uapi/linux/bcm2835-isp.h | 347 ++++
> > include/uapi/linux/v4l2-controls.h | 6 +
> > include/uapi/linux/videodev2.h | 1 +
> > 34 files changed, 5225 insertions(+), 33 deletions(-)
> > create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
> > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> > create mode 100644 include/uapi/linux/bcm2835-isp.h
>
> --
> Regards,
>
> Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-22 11:42 ` Dave Stevenson
0 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-11-22 11:42 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang and Laurent
On Mon, 21 Nov 2022 at 22:16, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Umang,
>
> Nice to see this series on the list !
>
> On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
> > This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > SoCs (namely BCM2711). Unicam is still under active development to work
> > with multistream support to get into mainline. Hence only the ISP driver
> > will remain the primary area of this series.
> >
> > Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
> > with the VC4 VPU.
> >
> > Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
> > dma_buf instead of a file-descriptor.
> >
> > Patch (04-05)/14 adds a couple of improvements/support for
> > bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
> >
> > Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
> > and format defintions.
> >
> > Patch (10-11)/14 deals with the colorspace support.
> > Note: This is still WIP since the implementation of colorspace is still
> > getting ironed out (especially around setting of colorspace flags handling).
> >
> > Patch 12/14 allows multiple instances of the ISP.
> >
> > Patch 13/14 adds a admin-guide document on bcm2835-isp.
> >
> > Patch 14/14 wires all this up with the vchiq-mmal driver.
> >
> > Testing:
> > Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
> > end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
> > bindings from [1]). Once done, functional testing was conducted with
> > libcamera[3] and its utility tools.
> >
> > Also note: Reviews given on [2] for the relevant ISP driver patches have
> > been incorporated in this version.
> >
> > Known issues:
> > - Colorspace handling
>
> This will require further discussions, I'll try to comment on this topic
> in the review of the ISP driver patch.
>
> > - vc-sm-cma spamming kernel log with
> > - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
>
> Do you have any plan to address this ? Is the root cause known ?
You've picked up an old version of the downstream driver.
Pi0&1 share the VPU L2 cache with the ARM in the architecture, so they
use the 0x8 cache alias.
See https://github.com/raspberrypi/linux/commit/e22927f8ec9dc87772ac61d6aba00dc8046b4f49
Dave
> > [1]: https://github.com/raspberrypi/linux
> > [2]: https://lore.kernel.org/linux-media/20200504092611.9798-1-laurent.pinchart@ideasonboard.com/
> > [3]: https://libcamera.org/getting-started.html
> >
> > Dave Stevenson (7):
> > staging: vc04_services: Add new vc-sm-cma driver
> > staging: vchiq_arm: Register vcsm-cma as a platform driver
> > media: videobuf2: Allow exporting of a struct dmabuf
> > staging: mmal-vchiq: Add support for event callbacks
> > staging: mmal-vchiq: Use vc-sm-cma to support zero copy
> > staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
> > uapi: bcm2835-isp: Add bcm2835-isp uapi header file
> >
> > David Plowman (2):
> > vc04_services: bcm2835-isp: Allow formats with different colour spaces
> > vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
> > outputs
> >
> > Naushir Patuck (5):
> > media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
> > staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
> > component
> > staging: vc04_services: bcm2835_isp: Allow multiple users
> > docs: admin-guide: media: bcm2835-isp: Add documentation for
> > bcm2835-isp
> > staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
> >
> > .../admin-guide/media/bcm2835-isp.rst | 127 ++
> > .../userspace-api/media/drivers/index.rst | 1 +
> > .../userspace-api/media/v4l/meta-formats.rst | 1 +
> > .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
> > MAINTAINERS | 9 +
> > .../media/common/videobuf2/videobuf2-core.c | 36 +-
> > drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
> > drivers/staging/vc04_services/Kconfig | 4 +
> > drivers/staging/vc04_services/Makefile | 2 +
> > .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
> > .../vc04_services/bcm2835-isp/Makefile | 8 +
> > .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
> > .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
> > .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
> > .../interface/vchiq_arm/vchiq_arm.c | 6 +
> > .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
> > .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
> > .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
> > .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
> > .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
> > .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
> > .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
> > .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
> > .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
> > .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
> > .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
> > .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
> > .../vchiq-mmal/mmal-parameters.h | 165 +-
> > .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
> > .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
> > include/media/videobuf2-core.h | 15 +
> > include/uapi/linux/bcm2835-isp.h | 347 ++++
> > include/uapi/linux/v4l2-controls.h | 6 +
> > include/uapi/linux/videodev2.h | 1 +
> > 34 files changed, 5225 insertions(+), 33 deletions(-)
> > create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
> > create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
> > create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
> > create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
> > create mode 100644 include/uapi/linux/bcm2835-isp.h
>
> --
> Regards,
>
> Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-11-22 11:42 ` Dave Stevenson
@ 2022-11-22 12:34 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-22 12:34 UTC (permalink / raw)
To: Dave Stevenson, Laurent Pinchart
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
On 11/22/22 5:12 PM, Dave Stevenson wrote:
> Hi Umang and Laurent
>
> On Mon, 21 Nov 2022 at 22:16, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>> Hi Umang,
>>
>> Nice to see this series on the list !
>>
>> On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>> with multistream support to get into mainline. Hence only the ISP driver
>>> will remain the primary area of this series.
>>>
>>> Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
>>> with the VC4 VPU.
>>>
>>> Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
>>> dma_buf instead of a file-descriptor.
>>>
>>> Patch (04-05)/14 adds a couple of improvements/support for
>>> bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
>>>
>>> Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
>>> and format defintions.
>>>
>>> Patch (10-11)/14 deals with the colorspace support.
>>> Note: This is still WIP since the implementation of colorspace is still
>>> getting ironed out (especially around setting of colorspace flags handling).
>>>
>>> Patch 12/14 allows multiple instances of the ISP.
>>>
>>> Patch 13/14 adds a admin-guide document on bcm2835-isp.
>>>
>>> Patch 14/14 wires all this up with the vchiq-mmal driver.
>>>
>>> Testing:
>>> Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
>>> end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
>>> bindings from [1]). Once done, functional testing was conducted with
>>> libcamera[3] and its utility tools.
>>>
>>> Also note: Reviews given on [2] for the relevant ISP driver patches have
>>> been incorporated in this version.
>>>
>>> Known issues:
>>> - Colorspace handling
>> This will require further discussions, I'll try to comment on this topic
>> in the review of the ISP driver patch.
>>
>>> - vc-sm-cma spamming kernel log with
>>> - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
>> Do you have any plan to address this ? Is the root cause known ?
> You've picked up an old version of the downstream driver.
I didn't pick version of driver rather, I picked commits that were
leading to the rpi-5.15.y branch of the downstream kernel
> Pi0&1 share the VPU L2 cache with the ARM in the architecture, so they
> use the 0x8 cache alias.
> See https://github.com/raspberrypi/linux/commit/e22927f8ec9dc87772ac61d6aba00dc8046b4f49
And certainly I missed to notice / pick this commit out on the vc-sm-cma
driver.
Thanks for providing the link! I'll squash it for v2.
>
> Dave
>
>>> [1]: https://github.com/raspberrypi/linux
>>> [2]: https://lore.kernel.org/linux-media/20200504092611.9798-1-laurent.pinchart@ideasonboard.com/
>>> [3]: https://libcamera.org/getting-started.html
>>>
>>> Dave Stevenson (7):
>>> staging: vc04_services: Add new vc-sm-cma driver
>>> staging: vchiq_arm: Register vcsm-cma as a platform driver
>>> media: videobuf2: Allow exporting of a struct dmabuf
>>> staging: mmal-vchiq: Add support for event callbacks
>>> staging: mmal-vchiq: Use vc-sm-cma to support zero copy
>>> staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
>>> uapi: bcm2835-isp: Add bcm2835-isp uapi header file
>>>
>>> David Plowman (2):
>>> vc04_services: bcm2835-isp: Allow formats with different colour spaces
>>> vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
>>> outputs
>>>
>>> Naushir Patuck (5):
>>> media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
>>> staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
>>> component
>>> staging: vc04_services: bcm2835_isp: Allow multiple users
>>> docs: admin-guide: media: bcm2835-isp: Add documentation for
>>> bcm2835-isp
>>> staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
>>>
>>> .../admin-guide/media/bcm2835-isp.rst | 127 ++
>>> .../userspace-api/media/drivers/index.rst | 1 +
>>> .../userspace-api/media/v4l/meta-formats.rst | 1 +
>>> .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
>>> MAINTAINERS | 9 +
>>> .../media/common/videobuf2/videobuf2-core.c | 36 +-
>>> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
>>> drivers/staging/vc04_services/Kconfig | 4 +
>>> drivers/staging/vc04_services/Makefile | 2 +
>>> .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
>>> .../vc04_services/bcm2835-isp/Makefile | 8 +
>>> .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
>>> .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
>>> .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
>>> .../interface/vchiq_arm/vchiq_arm.c | 6 +
>>> .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
>>> .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
>>> .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
>>> .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
>>> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
>>> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
>>> .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
>>> .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
>>> .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
>>> .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
>>> .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
>>> .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
>>> .../vchiq-mmal/mmal-parameters.h | 165 +-
>>> .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
>>> .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
>>> include/media/videobuf2-core.h | 15 +
>>> include/uapi/linux/bcm2835-isp.h | 347 ++++
>>> include/uapi/linux/v4l2-controls.h | 6 +
>>> include/uapi/linux/videodev2.h | 1 +
>>> 34 files changed, 5225 insertions(+), 33 deletions(-)
>>> create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
>>> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
>>> create mode 100644 include/uapi/linux/bcm2835-isp.h
>> --
>> Regards,
>>
>> Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-22 12:34 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-22 12:34 UTC (permalink / raw)
To: Dave Stevenson, Laurent Pinchart
Cc: linux-media, kernel-list, linux-kernel, linux-rpi-kernel,
linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
On 11/22/22 5:12 PM, Dave Stevenson wrote:
> Hi Umang and Laurent
>
> On Mon, 21 Nov 2022 at 22:16, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>> Hi Umang,
>>
>> Nice to see this series on the list !
>>
>> On Tue, Nov 22, 2022 at 03:17:08AM +0530, Umang Jain wrote:
>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>> with multistream support to get into mainline. Hence only the ISP driver
>>> will remain the primary area of this series.
>>>
>>> Patch (01-02)/14 adds a new driver named vc-sm-cma to handle memory sharing
>>> with the VC4 VPU.
>>>
>>> Patch 03/14 adds a small extension to videobuf2 to allow exporting as a
>>> dma_buf instead of a file-descriptor.
>>>
>>> Patch (04-05)/14 adds a couple of improvements/support for
>>> bcm2835-isp(event callback and zero-copy) to vchiq-mmal.
>>>
>>> Patch (06-09)/14 adds the core bcm2835-isp driver along with headers
>>> and format defintions.
>>>
>>> Patch (10-11)/14 deals with the colorspace support.
>>> Note: This is still WIP since the implementation of colorspace is still
>>> getting ironed out (especially around setting of colorspace flags handling).
>>>
>>> Patch 12/14 allows multiple instances of the ISP.
>>>
>>> Patch 13/14 adds a admin-guide document on bcm2835-isp.
>>>
>>> Patch 14/14 wires all this up with the vchiq-mmal driver.
>>>
>>> Testing:
>>> Tested with RPi Model 4B running linux mainline v6.1-rc6. To test
>>> end-to-end, I choose to cherry-pick the Unicam patches and OV5647 DT
>>> bindings from [1]). Once done, functional testing was conducted with
>>> libcamera[3] and its utility tools.
>>>
>>> Also note: Reviews given on [2] for the relevant ISP driver patches have
>>> been incorporated in this version.
>>>
>>> Known issues:
>>> - Colorspace handling
>> This will require further discussions, I'll try to comment on this topic
>> in the review of the ISP driver patch.
>>
>>> - vc-sm-cma spamming kernel log with
>>> - pr_err("%s: Expecting an uncached alias for dma_addr %pad\n"
>> Do you have any plan to address this ? Is the root cause known ?
> You've picked up an old version of the downstream driver.
I didn't pick version of driver rather, I picked commits that were
leading to the rpi-5.15.y branch of the downstream kernel
> Pi0&1 share the VPU L2 cache with the ARM in the architecture, so they
> use the 0x8 cache alias.
> See https://github.com/raspberrypi/linux/commit/e22927f8ec9dc87772ac61d6aba00dc8046b4f49
And certainly I missed to notice / pick this commit out on the vc-sm-cma
driver.
Thanks for providing the link! I'll squash it for v2.
>
> Dave
>
>>> [1]: https://github.com/raspberrypi/linux
>>> [2]: https://lore.kernel.org/linux-media/20200504092611.9798-1-laurent.pinchart@ideasonboard.com/
>>> [3]: https://libcamera.org/getting-started.html
>>>
>>> Dave Stevenson (7):
>>> staging: vc04_services: Add new vc-sm-cma driver
>>> staging: vchiq_arm: Register vcsm-cma as a platform driver
>>> media: videobuf2: Allow exporting of a struct dmabuf
>>> staging: mmal-vchiq: Add support for event callbacks
>>> staging: mmal-vchiq: Use vc-sm-cma to support zero copy
>>> staging: mmal_vchiq: Add image formats to be used by bcm2835-isp
>>> uapi: bcm2835-isp: Add bcm2835-isp uapi header file
>>>
>>> David Plowman (2):
>>> vc04_services: bcm2835-isp: Allow formats with different colour spaces
>>> vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP
>>> outputs
>>>
>>> Naushir Patuck (5):
>>> media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type
>>> staging: vc04_services: bcm2835-isp: Add a more complex ISP processing
>>> component
>>> staging: vc04_services: bcm2835_isp: Allow multiple users
>>> docs: admin-guide: media: bcm2835-isp: Add documentation for
>>> bcm2835-isp
>>> staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq
>>>
>>> .../admin-guide/media/bcm2835-isp.rst | 127 ++
>>> .../userspace-api/media/drivers/index.rst | 1 +
>>> .../userspace-api/media/v4l/meta-formats.rst | 1 +
>>> .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 41 +
>>> MAINTAINERS | 9 +
>>> .../media/common/videobuf2/videobuf2-core.c | 36 +-
>>> drivers/media/v4l2-core/v4l2-ioctl.c | 1 +
>>> drivers/staging/vc04_services/Kconfig | 4 +
>>> drivers/staging/vc04_services/Makefile | 2 +
>>> .../staging/vc04_services/bcm2835-isp/Kconfig | 14 +
>>> .../vc04_services/bcm2835-isp/Makefile | 8 +
>>> .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 +
>>> .../bcm2835-isp/bcm2835-isp-fmts.h | 558 +++++
>>> .../bcm2835-isp/bcm2835-v4l2-isp.c | 1817 +++++++++++++++++
>>> .../interface/vchiq_arm/vchiq_arm.c | 6 +
>>> .../staging/vc04_services/vc-sm-cma/Kconfig | 10 +
>>> .../staging/vc04_services/vc-sm-cma/Makefile | 12 +
>>> .../staging/vc04_services/vc-sm-cma/vc_sm.c | 801 ++++++++
>>> .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 +
>>> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++
>>> .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 +
>>> .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++
>>> .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 +
>>> .../staging/vc04_services/vchiq-mmal/Kconfig | 1 +
>>> .../vc04_services/vchiq-mmal/mmal-common.h | 5 +
>>> .../vc04_services/vchiq-mmal/mmal-encodings.h | 66 +
>>> .../vc04_services/vchiq-mmal/mmal-msg.h | 35 +
>>> .../vchiq-mmal/mmal-parameters.h | 165 +-
>>> .../vc04_services/vchiq-mmal/mmal-vchiq.c | 253 ++-
>>> .../vc04_services/vchiq-mmal/mmal-vchiq.h | 5 +
>>> include/media/videobuf2-core.h | 15 +
>>> include/uapi/linux/bcm2835-isp.h | 347 ++++
>>> include/uapi/linux/v4l2-controls.h | 6 +
>>> include/uapi/linux/videodev2.h | 1 +
>>> 34 files changed, 5225 insertions(+), 33 deletions(-)
>>> create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst
>>> create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h
>>> create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h
>>> create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h
>>> create mode 100644 include/uapi/linux/bcm2835-isp.h
>> --
>> Regards,
>>
>> Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf
2022-11-22 11:35 ` Dave Stevenson
@ 2022-11-25 1:49 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-25 1:49 UTC (permalink / raw)
To: Dave Stevenson
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham, Dave Stevenson
Hi Dave,
On Tue, Nov 22, 2022 at 11:35:31AM +0000, Dave Stevenson wrote:
> On Mon, 21 Nov 2022 at 23:18, Laurent Pinchart wrote:
> > On Tue, Nov 22, 2022 at 03:17:11AM +0530, Umang Jain wrote:
> > > From: Dave Stevenson <dave.stevenson@raspberrypi.org>
> > >
> > > videobuf2 only allowed exporting a dmabuf as a file descriptor,
> > > but there are instances where having the struct dma_buf is
> > > useful within the kernel.
> > >
> > > Split the current implementation into two, one step which
> > > exports a struct dma_buf, and the second which converts that
> > > into an fd.
> > >
> > > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
> > > Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> > > ---
> > > .../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------
> > > include/media/videobuf2-core.h | 15 ++++++++
> > > 2 files changed, 40 insertions(+), 11 deletions(-)
> > >
> > > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> > > index ab9697f3b5f1..32b26737cac4 100644
> > > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > > @@ -2184,49 +2184,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off,
> > > return -EINVAL;
> > > }
> > >
> > > -int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > > - unsigned int index, unsigned int plane, unsigned int flags)
> > > +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> > > + unsigned int index, unsigned int plane,
> > > + unsigned int flags)
> >
> > This function is used in the ISP driver, in bcm2835_isp_buf_prepare(),
> > for MMAP buffers, and as far as I can tell, its only purpose is to
> > create a dma_buf instance to then be imported in
> > vchiq_mmal_submit_buffer() with a call to vc_sm_cma_import_dmabuf().
> > That sounds like a very complicated set of operations, and quite
> > inefficient :-(
>
> Are you saying that dmabufs are not the preferred route for sharing
> buffers between kernel subsystems? What are you suggesting instead?
>
> If the VPU (firmware) has a handle to the buffer then we need to
> manage the lifetime such that it is not freed until the VPU has
> released it. That is handled for you with dmabufs, therefore why
> reinvent the wheel?
When we go through userspace, dmabuf is certainly the way to go. Here,
we need to share buffer information between two drivers that are
specific to the platform, so we could avoid going through so many layers
by using a custom abstraction. However, that would require additional
development, and probably reinventing the wheel to some extent, so it's
probably not worth it.
I'd like to explore if we could avoid creating a second dmabuf in
vc_sm_cma_import_dmabuf_internal() to make all this a bit more
lightweight. Let's discuss it in replies to patch 01/14.
> > > {
> > > struct vb2_buffer *vb = NULL;
> > > struct vb2_plane *vb_plane;
> > > - int ret;
> > > struct dma_buf *dbuf;
> > >
> > > if (q->memory != VB2_MEMORY_MMAP) {
> > > dprintk(q, 1, "queue is not currently set up for mmap\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (!q->mem_ops->get_dmabuf) {
> > > dprintk(q, 1, "queue does not support DMA buffer exporting\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (flags & ~(O_CLOEXEC | O_ACCMODE)) {
> > > dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (type != q->type) {
> > > dprintk(q, 1, "invalid buffer type\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (index >= q->num_buffers) {
> > > dprintk(q, 1, "buffer index out of range\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > vb = q->bufs[index];
> > >
> > > if (plane >= vb->num_planes) {
> > > dprintk(q, 1, "buffer plane out of range\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (vb2_fileio_is_active(q)) {
> > > dprintk(q, 1, "expbuf: file io in progress\n");
> > > - return -EBUSY;
> > > + return ERR_PTR(-EBUSY);
> > > }
> > >
> > > vb_plane = &vb->planes[plane];
> > > @@ -2238,9 +2238,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > > if (IS_ERR_OR_NULL(dbuf)) {
> > > dprintk(q, 1, "failed to export buffer %d, plane %d\n",
> > > index, plane);
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > + return dbuf;
> > > +}
> > > +EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf);
> > > +
> > > +int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > > + unsigned int index, unsigned int plane, unsigned int flags)
> > > +{
> > > + struct dma_buf *dbuf;
> > > + int ret;
> > > +
> > > + dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags);
> > > + if (IS_ERR(dbuf))
> > > + return PTR_ERR(dbuf);
> > > +
> > > ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE);
> > > if (ret < 0) {
> > > dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n",
> > > diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> > > index 3253bd2f6fee..33629ed2b64f 100644
> > > --- a/include/media/videobuf2-core.h
> > > +++ b/include/media/videobuf2-core.h
> > > @@ -911,6 +911,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
> > > */
> > > int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
> > >
> > > +/**
> > > + * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure
> > > + * @q: videobuf2 queue
> > > + * @type: buffer type
> > > + * @index: id number of the buffer
> > > + * @plane: index of the plane to be exported, 0 for single plane queues
> > > + * @flags: flags for newly created file, currently only O_CLOEXEC is
> > > + * supported, refer to manual of open syscall for more details
> > > + *
> > > + * Return: Returns the dmabuf pointer
> > > + */
> > > +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> > > + unsigned int index, unsigned int plane,
> > > + unsigned int flags);
> > > +
> > > /**
> > > * vb2_core_expbuf() - Export a buffer as a file descriptor.
> > > * @q: pointer to &struct vb2_queue with videobuf2 queue.
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf
@ 2022-11-25 1:49 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-11-25 1:49 UTC (permalink / raw)
To: Dave Stevenson
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham, Dave Stevenson
Hi Dave,
On Tue, Nov 22, 2022 at 11:35:31AM +0000, Dave Stevenson wrote:
> On Mon, 21 Nov 2022 at 23:18, Laurent Pinchart wrote:
> > On Tue, Nov 22, 2022 at 03:17:11AM +0530, Umang Jain wrote:
> > > From: Dave Stevenson <dave.stevenson@raspberrypi.org>
> > >
> > > videobuf2 only allowed exporting a dmabuf as a file descriptor,
> > > but there are instances where having the struct dma_buf is
> > > useful within the kernel.
> > >
> > > Split the current implementation into two, one step which
> > > exports a struct dma_buf, and the second which converts that
> > > into an fd.
> > >
> > > Signed-off-by: Dave Stevenson <dave.stevenson@raspberrypi.org>
> > > Signed-off-by: Umang Jain <umang.jain@ideasonboard.com>
> > > ---
> > > .../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------
> > > include/media/videobuf2-core.h | 15 ++++++++
> > > 2 files changed, 40 insertions(+), 11 deletions(-)
> > >
> > > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> > > index ab9697f3b5f1..32b26737cac4 100644
> > > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > > @@ -2184,49 +2184,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off,
> > > return -EINVAL;
> > > }
> > >
> > > -int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > > - unsigned int index, unsigned int plane, unsigned int flags)
> > > +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> > > + unsigned int index, unsigned int plane,
> > > + unsigned int flags)
> >
> > This function is used in the ISP driver, in bcm2835_isp_buf_prepare(),
> > for MMAP buffers, and as far as I can tell, its only purpose is to
> > create a dma_buf instance to then be imported in
> > vchiq_mmal_submit_buffer() with a call to vc_sm_cma_import_dmabuf().
> > That sounds like a very complicated set of operations, and quite
> > inefficient :-(
>
> Are you saying that dmabufs are not the preferred route for sharing
> buffers between kernel subsystems? What are you suggesting instead?
>
> If the VPU (firmware) has a handle to the buffer then we need to
> manage the lifetime such that it is not freed until the VPU has
> released it. That is handled for you with dmabufs, therefore why
> reinvent the wheel?
When we go through userspace, dmabuf is certainly the way to go. Here,
we need to share buffer information between two drivers that are
specific to the platform, so we could avoid going through so many layers
by using a custom abstraction. However, that would require additional
development, and probably reinventing the wheel to some extent, so it's
probably not worth it.
I'd like to explore if we could avoid creating a second dmabuf in
vc_sm_cma_import_dmabuf_internal() to make all this a bit more
lightweight. Let's discuss it in replies to patch 01/14.
> > > {
> > > struct vb2_buffer *vb = NULL;
> > > struct vb2_plane *vb_plane;
> > > - int ret;
> > > struct dma_buf *dbuf;
> > >
> > > if (q->memory != VB2_MEMORY_MMAP) {
> > > dprintk(q, 1, "queue is not currently set up for mmap\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (!q->mem_ops->get_dmabuf) {
> > > dprintk(q, 1, "queue does not support DMA buffer exporting\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (flags & ~(O_CLOEXEC | O_ACCMODE)) {
> > > dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (type != q->type) {
> > > dprintk(q, 1, "invalid buffer type\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (index >= q->num_buffers) {
> > > dprintk(q, 1, "buffer index out of range\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > vb = q->bufs[index];
> > >
> > > if (plane >= vb->num_planes) {
> > > dprintk(q, 1, "buffer plane out of range\n");
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > if (vb2_fileio_is_active(q)) {
> > > dprintk(q, 1, "expbuf: file io in progress\n");
> > > - return -EBUSY;
> > > + return ERR_PTR(-EBUSY);
> > > }
> > >
> > > vb_plane = &vb->planes[plane];
> > > @@ -2238,9 +2238,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > > if (IS_ERR_OR_NULL(dbuf)) {
> > > dprintk(q, 1, "failed to export buffer %d, plane %d\n",
> > > index, plane);
> > > - return -EINVAL;
> > > + return ERR_PTR(-EINVAL);
> > > }
> > >
> > > + return dbuf;
> > > +}
> > > +EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf);
> > > +
> > > +int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type,
> > > + unsigned int index, unsigned int plane, unsigned int flags)
> > > +{
> > > + struct dma_buf *dbuf;
> > > + int ret;
> > > +
> > > + dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags);
> > > + if (IS_ERR(dbuf))
> > > + return PTR_ERR(dbuf);
> > > +
> > > ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE);
> > > if (ret < 0) {
> > > dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n",
> > > diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> > > index 3253bd2f6fee..33629ed2b64f 100644
> > > --- a/include/media/videobuf2-core.h
> > > +++ b/include/media/videobuf2-core.h
> > > @@ -911,6 +911,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type);
> > > */
> > > int vb2_core_streamoff(struct vb2_queue *q, unsigned int type);
> > >
> > > +/**
> > > + * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure
> > > + * @q: videobuf2 queue
> > > + * @type: buffer type
> > > + * @index: id number of the buffer
> > > + * @plane: index of the plane to be exported, 0 for single plane queues
> > > + * @flags: flags for newly created file, currently only O_CLOEXEC is
> > > + * supported, refer to manual of open syscall for more details
> > > + *
> > > + * Return: Returns the dmabuf pointer
> > > + */
> > > +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type,
> > > + unsigned int index, unsigned int plane,
> > > + unsigned int flags);
> > > +
> > > /**
> > > * vb2_core_expbuf() - Export a buffer as a file descriptor.
> > > * @q: pointer to &struct vb2_queue with videobuf2 queue.
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-11-21 21:47 ` Umang Jain
@ 2022-11-26 14:42 ` Stefan Wahren
-1 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-11-26 14:42 UTC (permalink / raw)
To: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Umang,
Am 21.11.22 um 22:47 schrieb Umang Jain:
> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> SoCs (namely BCM2711). Unicam is still under active development to work
> with multistream support to get into mainline. Hence only the ISP driver
> will remain the primary area of this series.
thanks for working on this. But honestly i would prefer that vchiq comes
out of staging before adding more features. As Greg said some time ago
staging is not a place to "dump code and run away". These new files are
in the same bad shape as the rest of vc04 before the clean-up here in
staging started.
I agree that VCSM is on the TODO list for vchiq, but this driver is not
necessary for making bcm2835-audio & bcm2835-camera leave staging. It
just binds more resources on a new feature.
Unfortuntately i hadn't much time to work on vchiq by myself.
Just my two cents
Stefan
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-26 14:42 ` Stefan Wahren
0 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-11-26 14:42 UTC (permalink / raw)
To: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Umang,
Am 21.11.22 um 22:47 schrieb Umang Jain:
> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> SoCs (namely BCM2711). Unicam is still under active development to work
> with multistream support to get into mainline. Hence only the ISP driver
> will remain the primary area of this series.
thanks for working on this. But honestly i would prefer that vchiq comes
out of staging before adding more features. As Greg said some time ago
staging is not a place to "dump code and run away". These new files are
in the same bad shape as the rest of vc04 before the clean-up here in
staging started.
I agree that VCSM is on the TODO list for vchiq, but this driver is not
necessary for making bcm2835-audio & bcm2835-camera leave staging. It
just binds more resources on a new feature.
Unfortuntately i hadn't much time to work on vchiq by myself.
Just my two cents
Stefan
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-11-26 14:42 ` Stefan Wahren
@ 2022-11-26 16:26 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-26 16:26 UTC (permalink / raw)
To: Stefan Wahren, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Stefan
On 11/26/22 8:12 PM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 21.11.22 um 22:47 schrieb Umang Jain:
>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>> compatible
>> SoCs (namely BCM2711). Unicam is still under active development to work
>> with multistream support to get into mainline. Hence only the ISP driver
>> will remain the primary area of this series.
>
> thanks for working on this. But honestly i would prefer that vchiq
> comes out of staging before adding more features. As Greg said some
> time ago staging is not a place to "dump code and run away". These new
> files are in the same bad shape as the rest of vc04 before the
> clean-up here in staging started.
Certainly, I am not here to do that - but I am still learning the ropes.
If the staging issue is becoming a blocker for bcm2835-isp going
upstream, I would be happy to help here! Though I must mention that I
still have limited visibility so my aim would be to chart out a plan of
things needed to be done to get vc04_services out of staging!
>
> I agree that VCSM is on the TODO list for vchiq, but this driver is
> not necessary for making bcm2835-audio & bcm2835-camera leave staging.
> It just binds more resources on a new feature.
I see two TODO files in vc04_services:
./bcm2835-camera/TODO
./interface/TODO
One of the bcm2835-camera TODO points to the vc-sm-cma driver itself. So
that's address in the series. The other remaining one - I will need to
take a deeper look before commenting on it.
The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
cursory reading of them suggests that these apply to *all* vc04_services
components? Am I right?
Are these are the specific bits of cleanup you are referring to in your
comment?
>
> Unfortuntately i hadn't much time to work on vchiq by myself.
>
> Just my two cents
> Stefan
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-26 16:26 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-26 16:26 UTC (permalink / raw)
To: Stefan Wahren, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Stefan
On 11/26/22 8:12 PM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 21.11.22 um 22:47 schrieb Umang Jain:
>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>> compatible
>> SoCs (namely BCM2711). Unicam is still under active development to work
>> with multistream support to get into mainline. Hence only the ISP driver
>> will remain the primary area of this series.
>
> thanks for working on this. But honestly i would prefer that vchiq
> comes out of staging before adding more features. As Greg said some
> time ago staging is not a place to "dump code and run away". These new
> files are in the same bad shape as the rest of vc04 before the
> clean-up here in staging started.
Certainly, I am not here to do that - but I am still learning the ropes.
If the staging issue is becoming a blocker for bcm2835-isp going
upstream, I would be happy to help here! Though I must mention that I
still have limited visibility so my aim would be to chart out a plan of
things needed to be done to get vc04_services out of staging!
>
> I agree that VCSM is on the TODO list for vchiq, but this driver is
> not necessary for making bcm2835-audio & bcm2835-camera leave staging.
> It just binds more resources on a new feature.
I see two TODO files in vc04_services:
./bcm2835-camera/TODO
./interface/TODO
One of the bcm2835-camera TODO points to the vc-sm-cma driver itself. So
that's address in the series. The other remaining one - I will need to
take a deeper look before commenting on it.
The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
cursory reading of them suggests that these apply to *all* vc04_services
components? Am I right?
Are these are the specific bits of cleanup you are referring to in your
comment?
>
> Unfortuntately i hadn't much time to work on vchiq by myself.
>
> Just my two cents
> Stefan
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-11-26 16:26 ` Umang Jain
@ 2022-11-26 22:56 ` Stefan Wahren
-1 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-11-26 22:56 UTC (permalink / raw)
To: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Umang,
Am 26.11.22 um 17:26 schrieb Umang Jain:
> Hi Stefan
>
> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>> Hi Umang,
>>
>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>> compatible
>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>> with multistream support to get into mainline. Hence only the ISP
>>> driver
>>> will remain the primary area of this series.
>>
>> thanks for working on this. But honestly i would prefer that vchiq
>> comes out of staging before adding more features. As Greg said some
>> time ago staging is not a place to "dump code and run away". These
>> new files are in the same bad shape as the rest of vc04 before the
>> clean-up here in staging started.
>
> Certainly, I am not here to do that - but I am still learning the ropes.
no problem.
>
> If the staging issue is becoming a blocker for bcm2835-isp going
> upstream, I would be happy to help here! Though I must mention that I
> still have limited visibility so my aim would be to chart out a plan
> of things needed to be done to get vc04_services out of staging!
The vchiq driver is in staging since 2016, so every step forwards is
good. Unfortunately all of the low hanging fruits has been gathered.
For me the most important, but not to tricky steps to get vchiq out of
staging would be:
* Cleanup logging mechanism
* Get rid of custom function return values
There was already an attempt for this [1]
* Get rid of all non essential global structures and create a proper per
device structure
>
>>
>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>> not necessary for making bcm2835-audio & bcm2835-camera leave
>> staging. It just binds more resources on a new feature.
>
> I see two TODO files in vc04_services:
> ./bcm2835-camera/TODO
> ./interface/TODO
>
> One of the bcm2835-camera TODO points to the vc-sm-cma driver itself.
> So that's address in the series. The other remaining one - I will need
> to take a deeper look before commenting on it.
>
> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
> cursory reading of them suggests that these apply to *all*
> vc04_services components? Am I right?
Actually these applies just for the interfaces directory. Some of them
could apply to the services, but this is no priority.
>
> Are these are the specific bits of cleanup you are referring to in
> your comment?
You mean about bcm2835-isp? There were too many changes to vchiq that i
don't remember them all. The first that come to my mind was those fancy
comment sections which is not kernel coding style. It has been removed.
[1] -
https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>
>
>>
>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>
>> Just my two cents
>> Stefan
>>
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-26 22:56 ` Stefan Wahren
0 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-11-26 22:56 UTC (permalink / raw)
To: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Umang,
Am 26.11.22 um 17:26 schrieb Umang Jain:
> Hi Stefan
>
> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>> Hi Umang,
>>
>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>> compatible
>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>> with multistream support to get into mainline. Hence only the ISP
>>> driver
>>> will remain the primary area of this series.
>>
>> thanks for working on this. But honestly i would prefer that vchiq
>> comes out of staging before adding more features. As Greg said some
>> time ago staging is not a place to "dump code and run away". These
>> new files are in the same bad shape as the rest of vc04 before the
>> clean-up here in staging started.
>
> Certainly, I am not here to do that - but I am still learning the ropes.
no problem.
>
> If the staging issue is becoming a blocker for bcm2835-isp going
> upstream, I would be happy to help here! Though I must mention that I
> still have limited visibility so my aim would be to chart out a plan
> of things needed to be done to get vc04_services out of staging!
The vchiq driver is in staging since 2016, so every step forwards is
good. Unfortunately all of the low hanging fruits has been gathered.
For me the most important, but not to tricky steps to get vchiq out of
staging would be:
* Cleanup logging mechanism
* Get rid of custom function return values
There was already an attempt for this [1]
* Get rid of all non essential global structures and create a proper per
device structure
>
>>
>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>> not necessary for making bcm2835-audio & bcm2835-camera leave
>> staging. It just binds more resources on a new feature.
>
> I see two TODO files in vc04_services:
> ./bcm2835-camera/TODO
> ./interface/TODO
>
> One of the bcm2835-camera TODO points to the vc-sm-cma driver itself.
> So that's address in the series. The other remaining one - I will need
> to take a deeper look before commenting on it.
>
> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
> cursory reading of them suggests that these apply to *all*
> vc04_services components? Am I right?
Actually these applies just for the interfaces directory. Some of them
could apply to the services, but this is no priority.
>
> Are these are the specific bits of cleanup you are referring to in
> your comment?
You mean about bcm2835-isp? There were too many changes to vchiq that i
don't remember them all. The first that come to my mind was those fancy
comment sections which is not kernel coding style. It has been removed.
[1] -
https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>
>
>>
>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>
>> Just my two cents
>> Stefan
>>
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-11-26 22:56 ` Stefan Wahren
@ 2022-11-30 10:58 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-30 10:58 UTC (permalink / raw)
To: Stefan Wahren, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Stefan,
On 11/27/22 6:56 AM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 26.11.22 um 17:26 schrieb Umang Jain:
>> Hi Stefan
>>
>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>> Hi Umang,
>>>
>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and
>>>> is a
>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>> compatible
>>>> SoCs (namely BCM2711). Unicam is still under active development to
>>>> work
>>>> with multistream support to get into mainline. Hence only the ISP
>>>> driver
>>>> will remain the primary area of this series.
>>>
>>> thanks for working on this. But honestly i would prefer that vchiq
>>> comes out of staging before adding more features. As Greg said some
>>> time ago staging is not a place to "dump code and run away". These
>>> new files are in the same bad shape as the rest of vc04 before the
>>> clean-up here in staging started.
>>
>> Certainly, I am not here to do that - but I am still learning the ropes.
> no problem.
>>
>> If the staging issue is becoming a blocker for bcm2835-isp going
>> upstream, I would be happy to help here! Though I must mention that I
>> still have limited visibility so my aim would be to chart out a plan
>> of things needed to be done to get vc04_services out of staging!
>
> The vchiq driver is in staging since 2016, so every step forwards is
> good. Unfortunately all of the low hanging fruits has been gathered.
>
> For me the most important, but not to tricky steps to get vchiq out of
> staging would be:
>
> * Cleanup logging mechanism
>
> * Get rid of custom function return values
>
> There was already an attempt for this [1]
>
> * Get rid of all non essential global structures and create a proper per
> device structure
>
>>
>>>
>>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>>> not necessary for making bcm2835-audio & bcm2835-camera leave
>>> staging. It just binds more resources on a new feature.
bcm2835-camera is the legacy camera stack which probably need to be
dropped from hereon...
>>
>> I see two TODO files in vc04_services:
>> ./bcm2835-camera/TODO
>> ./interface/TODO
>>
>> One of the bcm2835-camera TODO points to the vc-sm-cma driver itself.
>> So that's address in the series. The other remaining one - I will
>> need to take a deeper look before commenting on it.
>>
>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
>> cursory reading of them suggests that these apply to *all*
>> vc04_services components? Am I right?
> Actually these applies just for the interfaces directory. Some of them
> could apply to the services, but this is no priority.
By no priority, you mean this doesn't affect the criteria required to
ful-fill to get these out of staging?
>>
>> Are these are the specific bits of cleanup you are referring to in
>> your comment?
>
> You mean about bcm2835-isp? There were too many changes to vchiq that
> i don't remember them all. The first that come to my mind was those
> fancy comment sections which is not kernel coding style. It has been
> removed.
No, I don't mean the bcm2835-isp changes (those are upcoming /
out-of-tree still so...). I mean what are the specific bits / points
that needs to be addressed to get vc04_services out of the staging.
You have mentioned it above now, so I'll follow up on those. The many
vchiq changes you referred to above comment (that you don't remember)
are from [1] as well or some other series ?
>
> [1] -
> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>
>>
>>
>>>
>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>
>>> Just my two cents
>>> Stefan
>>>
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-11-30 10:58 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-11-30 10:58 UTC (permalink / raw)
To: Stefan Wahren, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Stefan,
On 11/27/22 6:56 AM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 26.11.22 um 17:26 schrieb Umang Jain:
>> Hi Stefan
>>
>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>> Hi Umang,
>>>
>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and
>>>> is a
>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>> compatible
>>>> SoCs (namely BCM2711). Unicam is still under active development to
>>>> work
>>>> with multistream support to get into mainline. Hence only the ISP
>>>> driver
>>>> will remain the primary area of this series.
>>>
>>> thanks for working on this. But honestly i would prefer that vchiq
>>> comes out of staging before adding more features. As Greg said some
>>> time ago staging is not a place to "dump code and run away". These
>>> new files are in the same bad shape as the rest of vc04 before the
>>> clean-up here in staging started.
>>
>> Certainly, I am not here to do that - but I am still learning the ropes.
> no problem.
>>
>> If the staging issue is becoming a blocker for bcm2835-isp going
>> upstream, I would be happy to help here! Though I must mention that I
>> still have limited visibility so my aim would be to chart out a plan
>> of things needed to be done to get vc04_services out of staging!
>
> The vchiq driver is in staging since 2016, so every step forwards is
> good. Unfortunately all of the low hanging fruits has been gathered.
>
> For me the most important, but not to tricky steps to get vchiq out of
> staging would be:
>
> * Cleanup logging mechanism
>
> * Get rid of custom function return values
>
> There was already an attempt for this [1]
>
> * Get rid of all non essential global structures and create a proper per
> device structure
>
>>
>>>
>>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>>> not necessary for making bcm2835-audio & bcm2835-camera leave
>>> staging. It just binds more resources on a new feature.
bcm2835-camera is the legacy camera stack which probably need to be
dropped from hereon...
>>
>> I see two TODO files in vc04_services:
>> ./bcm2835-camera/TODO
>> ./interface/TODO
>>
>> One of the bcm2835-camera TODO points to the vc-sm-cma driver itself.
>> So that's address in the series. The other remaining one - I will
>> need to take a deeper look before commenting on it.
>>
>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
>> cursory reading of them suggests that these apply to *all*
>> vc04_services components? Am I right?
> Actually these applies just for the interfaces directory. Some of them
> could apply to the services, but this is no priority.
By no priority, you mean this doesn't affect the criteria required to
ful-fill to get these out of staging?
>>
>> Are these are the specific bits of cleanup you are referring to in
>> your comment?
>
> You mean about bcm2835-isp? There were too many changes to vchiq that
> i don't remember them all. The first that come to my mind was those
> fancy comment sections which is not kernel coding style. It has been
> removed.
No, I don't mean the bcm2835-isp changes (those are upcoming /
out-of-tree still so...). I mean what are the specific bits / points
that needs to be addressed to get vc04_services out of the staging.
You have mentioned it above now, so I'll follow up on those. The many
vchiq changes you referred to above comment (that you don't remember)
are from [1] as well or some other series ?
>
> [1] -
> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>
>>
>>
>>>
>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>
>>> Just my two cents
>>> Stefan
>>>
>>
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-11-30 10:58 ` Umang Jain
@ 2022-12-01 22:45 ` Stefan Wahren
-1 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-01 22:45 UTC (permalink / raw)
To: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Umang,
Am 30.11.22 um 11:58 schrieb Umang Jain:
> Hi Stefan,
>
> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>> Hi Umang,
>>
>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>> Hi Stefan
>>>
>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>> Hi Umang,
>>>>
>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and
>>>>> is a
>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>>> compatible
>>>>> SoCs (namely BCM2711). Unicam is still under active development to
>>>>> work
>>>>> with multistream support to get into mainline. Hence only the ISP
>>>>> driver
>>>>> will remain the primary area of this series.
>>>>
>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>> comes out of staging before adding more features. As Greg said some
>>>> time ago staging is not a place to "dump code and run away". These
>>>> new files are in the same bad shape as the rest of vc04 before the
>>>> clean-up here in staging started.
>>>
>>> Certainly, I am not here to do that - but I am still learning the
>>> ropes.
>> no problem.
>>>
>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>> upstream, I would be happy to help here! Though I must mention that
>>> I still have limited visibility so my aim would be to chart out a
>>> plan of things needed to be done to get vc04_services out of staging!
>>
>> The vchiq driver is in staging since 2016, so every step forwards is
>> good. Unfortunately all of the low hanging fruits has been gathered.
>>
>> For me the most important, but not to tricky steps to get vchiq out
>> of staging would be:
>>
>> * Cleanup logging mechanism
>>
>> * Get rid of custom function return values
>>
>> There was already an attempt for this [1]
>>
>> * Get rid of all non essential global structures and create a proper per
>> device structure
>>
>>>
>>>>
>>>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>>>> not necessary for making bcm2835-audio & bcm2835-camera leave
>>>> staging. It just binds more resources on a new feature.
>
> bcm2835-camera is the legacy camera stack which probably need to be
> dropped from hereon...
I don't not know if there any users left, so i would be careful here.
Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this dumb
question but i'm not expert here.
>>>
>>> I see two TODO files in vc04_services:
>>> ./bcm2835-camera/TODO
>>> ./interface/TODO
>>>
>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>> itself. So that's address in the series. The other remaining one - I
>>> will need to take a deeper look before commenting on it.
>>>
>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
>>> cursory reading of them suggests that these apply to *all*
>>> vc04_services components? Am I right?
>> Actually these applies just for the interfaces directory. Some of
>> them could apply to the services, but this is no priority.
>
> By no priority, you mean this doesn't affect the criteria required to
> ful-fill to get these out of staging?
Correct
>>>
>>> Are these are the specific bits of cleanup you are referring to in
>>> your comment?
>>
>> You mean about bcm2835-isp? There were too many changes to vchiq that
>> i don't remember them all. The first that come to my mind was those
>> fancy comment sections which is not kernel coding style. It has been
>> removed.
>
> No, I don't mean the bcm2835-isp changes (those are upcoming /
> out-of-tree still so...). I mean what are the specific bits / points
> that needs to be addressed to get vc04_services out of the staging.
These were the points which i mentioned in my last email. They came from
interface/TODO.
>
> You have mentioned it above now, so I'll follow up on those.
That would be great :)
> The many vchiq changes you referred to above comment (that you don't
> remember) are from [1] as well or some other series ?
Sorry, for the confusing. The many changes i refer were the dozens of
clean up patches for vc04_interfaces in mainline staging since the last
years. [1] was just a single patch which has been accepted yet.
>
>>
>> [1] -
>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>>
>>>
>>>
>>>>
>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>
>>>> Just my two cents
>>>> Stefan
>>>>
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-01 22:45 ` Stefan Wahren
0 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-01 22:45 UTC (permalink / raw)
To: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Umang,
Am 30.11.22 um 11:58 schrieb Umang Jain:
> Hi Stefan,
>
> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>> Hi Umang,
>>
>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>> Hi Stefan
>>>
>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>> Hi Umang,
>>>>
>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and
>>>>> is a
>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>>> compatible
>>>>> SoCs (namely BCM2711). Unicam is still under active development to
>>>>> work
>>>>> with multistream support to get into mainline. Hence only the ISP
>>>>> driver
>>>>> will remain the primary area of this series.
>>>>
>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>> comes out of staging before adding more features. As Greg said some
>>>> time ago staging is not a place to "dump code and run away". These
>>>> new files are in the same bad shape as the rest of vc04 before the
>>>> clean-up here in staging started.
>>>
>>> Certainly, I am not here to do that - but I am still learning the
>>> ropes.
>> no problem.
>>>
>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>> upstream, I would be happy to help here! Though I must mention that
>>> I still have limited visibility so my aim would be to chart out a
>>> plan of things needed to be done to get vc04_services out of staging!
>>
>> The vchiq driver is in staging since 2016, so every step forwards is
>> good. Unfortunately all of the low hanging fruits has been gathered.
>>
>> For me the most important, but not to tricky steps to get vchiq out
>> of staging would be:
>>
>> * Cleanup logging mechanism
>>
>> * Get rid of custom function return values
>>
>> There was already an attempt for this [1]
>>
>> * Get rid of all non essential global structures and create a proper per
>> device structure
>>
>>>
>>>>
>>>> I agree that VCSM is on the TODO list for vchiq, but this driver is
>>>> not necessary for making bcm2835-audio & bcm2835-camera leave
>>>> staging. It just binds more resources on a new feature.
>
> bcm2835-camera is the legacy camera stack which probably need to be
> dropped from hereon...
I don't not know if there any users left, so i would be careful here.
Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this dumb
question but i'm not expert here.
>>>
>>> I see two TODO files in vc04_services:
>>> ./bcm2835-camera/TODO
>>> ./interface/TODO
>>>
>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>> itself. So that's address in the series. The other remaining one - I
>>> will need to take a deeper look before commenting on it.
>>>
>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing a
>>> cursory reading of them suggests that these apply to *all*
>>> vc04_services components? Am I right?
>> Actually these applies just for the interfaces directory. Some of
>> them could apply to the services, but this is no priority.
>
> By no priority, you mean this doesn't affect the criteria required to
> ful-fill to get these out of staging?
Correct
>>>
>>> Are these are the specific bits of cleanup you are referring to in
>>> your comment?
>>
>> You mean about bcm2835-isp? There were too many changes to vchiq that
>> i don't remember them all. The first that come to my mind was those
>> fancy comment sections which is not kernel coding style. It has been
>> removed.
>
> No, I don't mean the bcm2835-isp changes (those are upcoming /
> out-of-tree still so...). I mean what are the specific bits / points
> that needs to be addressed to get vc04_services out of the staging.
These were the points which i mentioned in my last email. They came from
interface/TODO.
>
> You have mentioned it above now, so I'll follow up on those.
That would be great :)
> The many vchiq changes you referred to above comment (that you don't
> remember) are from [1] as well or some other series ?
Sorry, for the confusing. The many changes i refer were the dozens of
clean up patches for vc04_interfaces in mainline staging since the last
years. [1] was just a single patch which has been accepted yet.
>
>>
>> [1] -
>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>>
>>>
>>>
>>>>
>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>
>>>> Just my two cents
>>>> Stefan
>>>>
>>>
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-01 22:45 ` Stefan Wahren
@ 2022-12-02 3:57 ` Umang Jain
-1 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-12-02 3:57 UTC (permalink / raw)
To: Stefan Wahren, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Stefan,
On 12/2/22 6:45 AM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 30.11.22 um 11:58 schrieb Umang Jain:
>> Hi Stefan,
>>
>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>> Hi Umang,
>>>
>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>> Hi Stefan
>>>>
>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>> Hi Umang,
>>>>>
>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1]
>>>>>> and is a
>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>>>> compatible
>>>>>> SoCs (namely BCM2711). Unicam is still under active development
>>>>>> to work
>>>>>> with multistream support to get into mainline. Hence only the ISP
>>>>>> driver
>>>>>> will remain the primary area of this series.
>>>>>
>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>> comes out of staging before adding more features. As Greg said
>>>>> some time ago staging is not a place to "dump code and run away".
>>>>> These new files are in the same bad shape as the rest of vc04
>>>>> before the clean-up here in staging started.
>>>>
>>>> Certainly, I am not here to do that - but I am still learning the
>>>> ropes.
>>> no problem.
>>>>
>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>> upstream, I would be happy to help here! Though I must mention that
>>>> I still have limited visibility so my aim would be to chart out a
>>>> plan of things needed to be done to get vc04_services out of staging!
>>>
>>> The vchiq driver is in staging since 2016, so every step forwards is
>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>
>>> For me the most important, but not to tricky steps to get vchiq out
>>> of staging would be:
>>>
>>> * Cleanup logging mechanism
>>>
>>> * Get rid of custom function return values
>>>
>>> There was already an attempt for this [1]
>>>
>>> * Get rid of all non essential global structures and create a proper
>>> per
>>> device structure
>>>
>>>>
>>>>>
>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>> staging. It just binds more resources on a new feature.
>>
>> bcm2835-camera is the legacy camera stack which probably need to be
>> dropped from hereon...
> I don't not know if there any users left, so i would be careful here.
> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> dumb question but i'm not expert here.
I am careful too here and probably need Input from RaspberryPi in order
to proceed to drop it. But from my perspective - bcm2835-camera is _not_
going out of staging - it'll either sit here (or probably dropped) as
statied from [1]
```
+ * There are two camera drivers in the kernel for BCM283x - this one
+ * and bcm2835-camera (currently in staging).
```
The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
(bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
mentioned in my cover the testing of bcm2835-isp happened on top of
unicam patches.
[1]:
https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
>>>>
>>>> I see two TODO files in vc04_services:
>>>> ./bcm2835-camera/TODO
>>>> ./interface/TODO
>>>>
>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>>> itself. So that's address in the series. The other remaining one -
>>>> I will need to take a deeper look before commenting on it.
>>>>
>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
>>>> a cursory reading of them suggests that these apply to *all*
>>>> vc04_services components? Am I right?
>>> Actually these applies just for the interfaces directory. Some of
>>> them could apply to the services, but this is no priority.
>>
>> By no priority, you mean this doesn't affect the criteria required to
>> ful-fill to get these out of staging?
> Correct
>>>>
>>>> Are these are the specific bits of cleanup you are referring to in
>>>> your comment?
>>>
>>> You mean about bcm2835-isp? There were too many changes to vchiq
>>> that i don't remember them all. The first that come to my mind was
>>> those fancy comment sections which is not kernel coding style. It
>>> has been removed.
>>
>> No, I don't mean the bcm2835-isp changes (those are upcoming /
>> out-of-tree still so...). I mean what are the specific bits / points
>> that needs to be addressed to get vc04_services out of the staging.
> These were the points which i mentioned in my last email. They came
> from interface/TODO.
>>
>> You have mentioned it above now, so I'll follow up on those.
> That would be great :)
>> The many vchiq changes you referred to above comment (that you don't
>> remember) are from [1] as well or some other series ?
> Sorry, for the confusing. The many changes i refer were the dozens of
> clean up patches for vc04_interfaces in mainline staging since the
> last years. [1] was just a single patch which has been accepted yet.
Ah I see. There are many others that I've to dig out then. Thanks for
clarifying!
>>
>>>
>>> [1] -
>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>>>
>>>>
>>>>
>>>>>
>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>>
>>>>> Just my two cents
>>>>> Stefan
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> linux-arm-kernel@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 3:57 ` Umang Jain
0 siblings, 0 replies; 95+ messages in thread
From: Umang Jain @ 2022-12-02 3:57 UTC (permalink / raw)
To: Stefan Wahren, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list
Cc: Dave Stevenson, Florian Fainelli, Naushir Patuck, David Plowman,
Kieran Bingham, Laurent Pinchart
Hi Stefan,
On 12/2/22 6:45 AM, Stefan Wahren wrote:
> Hi Umang,
>
> Am 30.11.22 um 11:58 schrieb Umang Jain:
>> Hi Stefan,
>>
>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>> Hi Umang,
>>>
>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>> Hi Stefan
>>>>
>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>> Hi Umang,
>>>>>
>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1]
>>>>>> and is a
>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and
>>>>>> compatible
>>>>>> SoCs (namely BCM2711). Unicam is still under active development
>>>>>> to work
>>>>>> with multistream support to get into mainline. Hence only the ISP
>>>>>> driver
>>>>>> will remain the primary area of this series.
>>>>>
>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>> comes out of staging before adding more features. As Greg said
>>>>> some time ago staging is not a place to "dump code and run away".
>>>>> These new files are in the same bad shape as the rest of vc04
>>>>> before the clean-up here in staging started.
>>>>
>>>> Certainly, I am not here to do that - but I am still learning the
>>>> ropes.
>>> no problem.
>>>>
>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>> upstream, I would be happy to help here! Though I must mention that
>>>> I still have limited visibility so my aim would be to chart out a
>>>> plan of things needed to be done to get vc04_services out of staging!
>>>
>>> The vchiq driver is in staging since 2016, so every step forwards is
>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>
>>> For me the most important, but not to tricky steps to get vchiq out
>>> of staging would be:
>>>
>>> * Cleanup logging mechanism
>>>
>>> * Get rid of custom function return values
>>>
>>> There was already an attempt for this [1]
>>>
>>> * Get rid of all non essential global structures and create a proper
>>> per
>>> device structure
>>>
>>>>
>>>>>
>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>> staging. It just binds more resources on a new feature.
>>
>> bcm2835-camera is the legacy camera stack which probably need to be
>> dropped from hereon...
> I don't not know if there any users left, so i would be careful here.
> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> dumb question but i'm not expert here.
I am careful too here and probably need Input from RaspberryPi in order
to proceed to drop it. But from my perspective - bcm2835-camera is _not_
going out of staging - it'll either sit here (or probably dropped) as
statied from [1]
```
+ * There are two camera drivers in the kernel for BCM283x - this one
+ * and bcm2835-camera (currently in staging).
```
The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
(bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
mentioned in my cover the testing of bcm2835-isp happened on top of
unicam patches.
[1]:
https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
>>>>
>>>> I see two TODO files in vc04_services:
>>>> ./bcm2835-camera/TODO
>>>> ./interface/TODO
>>>>
>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>>> itself. So that's address in the series. The other remaining one -
>>>> I will need to take a deeper look before commenting on it.
>>>>
>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
>>>> a cursory reading of them suggests that these apply to *all*
>>>> vc04_services components? Am I right?
>>> Actually these applies just for the interfaces directory. Some of
>>> them could apply to the services, but this is no priority.
>>
>> By no priority, you mean this doesn't affect the criteria required to
>> ful-fill to get these out of staging?
> Correct
>>>>
>>>> Are these are the specific bits of cleanup you are referring to in
>>>> your comment?
>>>
>>> You mean about bcm2835-isp? There were too many changes to vchiq
>>> that i don't remember them all. The first that come to my mind was
>>> those fancy comment sections which is not kernel coding style. It
>>> has been removed.
>>
>> No, I don't mean the bcm2835-isp changes (those are upcoming /
>> out-of-tree still so...). I mean what are the specific bits / points
>> that needs to be addressed to get vc04_services out of the staging.
> These were the points which i mentioned in my last email. They came
> from interface/TODO.
>>
>> You have mentioned it above now, so I'll follow up on those.
> That would be great :)
>> The many vchiq changes you referred to above comment (that you don't
>> remember) are from [1] as well or some other series ?
> Sorry, for the confusing. The many changes i refer were the dozens of
> clean up patches for vc04_interfaces in mainline staging since the
> last years. [1] was just a single patch which has been accepted yet.
Ah I see. There are many others that I've to dig out then. Thanks for
clarifying!
>>
>>>
>>> [1] -
>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>>>
>>>>
>>>>
>>>>>
>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>>
>>>>> Just my two cents
>>>>> Stefan
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> linux-arm-kernel@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 3:57 ` Umang Jain
@ 2022-12-02 9:17 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-12-02 9:17 UTC (permalink / raw)
To: Umang Jain
Cc: Stefan Wahren, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang,
On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > Am 30.11.22 um 11:58 schrieb Umang Jain:
> >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>> will remain the primary area of this series.
> >>>>>
> >>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>> comes out of staging before adding more features. As Greg said
> >>>>> some time ago staging is not a place to "dump code and run away".
> >>>>> These new files are in the same bad shape as the rest of vc04
> >>>>> before the clean-up here in staging started.
> >>>>
> >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>
> >>> no problem.
> >>>
> >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>> upstream, I would be happy to help here! Though I must mention that
> >>>> I still have limited visibility so my aim would be to chart out a
> >>>> plan of things needed to be done to get vc04_services out of staging!
> >>>
> >>> The vchiq driver is in staging since 2016, so every step forwards is
> >>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>
> >>> For me the most important, but not to tricky steps to get vchiq out
> >>> of staging would be:
> >>>
> >>> * Cleanup logging mechanism
> >>>
> >>> * Get rid of custom function return values
> >>>
> >>> There was already an attempt for this [1]
> >>>
> >>> * Get rid of all non essential global structures and create a proper per
> >>> device structure
> >>>
> >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>> staging. It just binds more resources on a new feature.
> >>
> >> bcm2835-camera is the legacy camera stack which probably need to be
> >> dropped from hereon...
> >
> > I don't not know if there any users left, so i would be careful here.
> > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > dumb question but i'm not expert here.
>
> I am careful too here and probably need Input from RaspberryPi in order
> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> going out of staging - it'll either sit here (or probably dropped) as
> statied from [1]
>
> ```
> + * There are two camera drivers in the kernel for BCM283x - this one
> + * and bcm2835-camera (currently in staging).
> ```
>
> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> mentioned in my cover the testing of bcm2835-isp happened on top of
> unicam patches.
To be accurate, the bcm2835-camera driver supports the VC4
firmware-based camera stack. In that setup, the camera sensors (OV5647
or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
firmware, which provides a high-level interface towards the kernel. This
architecture has been replaced by Linux-side control of the camera
sensors (through existing drivers in drivers/media/i2c/), Unicam
(through the driver from [1]) and ISP (through this driver). Moving
control to the Linux side requires complex processing in userspace,
handled by libcamera.
bcm2835-camera is thus replaced by multiple drivers combined with
libcamera, and that is the camera stack that is shipped by Raspberry Pi
these days. While this may affect some userspace use cases), we will not
work on destaging bcm2835-camera, and as far as I'm aware, nobody else
is planning to do so either. I don't mind much if the driver stays in
staging for some more time, but I'd rather drop it if possible.
> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
>
> >>>>
> >>>> I see two TODO files in vc04_services:
> >>>> ./bcm2835-camera/TODO
> >>>> ./interface/TODO
> >>>>
> >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>> itself. So that's address in the series. The other remaining one -
> >>>> I will need to take a deeper look before commenting on it.
> >>>>
> >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>> a cursory reading of them suggests that these apply to *all*
> >>>> vc04_services components? Am I right?
> >>>
> >>> Actually these applies just for the interfaces directory. Some of
> >>> them could apply to the services, but this is no priority.
> >>
> >> By no priority, you mean this doesn't affect the criteria required to
> >> ful-fill to get these out of staging?
> >
> > Correct
> >
> >>>> Are these are the specific bits of cleanup you are referring to in
> >>>> your comment?
> >>>
> >>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>> that i don't remember them all. The first that come to my mind was
> >>> those fancy comment sections which is not kernel coding style. It
> >>> has been removed.
> >>
> >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >> out-of-tree still so...). I mean what are the specific bits / points
> >> that needs to be addressed to get vc04_services out of the staging.
> >
> > These were the points which i mentioned in my last email. They came
> > from interface/TODO.
> >
> >> You have mentioned it above now, so I'll follow up on those.
> >
> > That would be great :)
> >
> >> The many vchiq changes you referred to above comment (that you don't
> >> remember) are from [1] as well or some other series ?
> >
> > Sorry, for the confusing. The many changes i refer were the dozens of
> > clean up patches for vc04_interfaces in mainline staging since the
> > last years. [1] was just a single patch which has been accepted yet.
>
> Ah I see. There are many others that I've to dig out then. Thanks for
> clarifying!
>
> >>> [1] -
> >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> >>>
> >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>
> >>>>> Just my two cents
> >>>>> Stefan
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 9:17 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-12-02 9:17 UTC (permalink / raw)
To: Umang Jain
Cc: Stefan Wahren, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Dave Stevenson,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham
Hi Umang,
On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > Am 30.11.22 um 11:58 schrieb Umang Jain:
> >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>> will remain the primary area of this series.
> >>>>>
> >>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>> comes out of staging before adding more features. As Greg said
> >>>>> some time ago staging is not a place to "dump code and run away".
> >>>>> These new files are in the same bad shape as the rest of vc04
> >>>>> before the clean-up here in staging started.
> >>>>
> >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>
> >>> no problem.
> >>>
> >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>> upstream, I would be happy to help here! Though I must mention that
> >>>> I still have limited visibility so my aim would be to chart out a
> >>>> plan of things needed to be done to get vc04_services out of staging!
> >>>
> >>> The vchiq driver is in staging since 2016, so every step forwards is
> >>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>
> >>> For me the most important, but not to tricky steps to get vchiq out
> >>> of staging would be:
> >>>
> >>> * Cleanup logging mechanism
> >>>
> >>> * Get rid of custom function return values
> >>>
> >>> There was already an attempt for this [1]
> >>>
> >>> * Get rid of all non essential global structures and create a proper per
> >>> device structure
> >>>
> >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>> staging. It just binds more resources on a new feature.
> >>
> >> bcm2835-camera is the legacy camera stack which probably need to be
> >> dropped from hereon...
> >
> > I don't not know if there any users left, so i would be careful here.
> > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > dumb question but i'm not expert here.
>
> I am careful too here and probably need Input from RaspberryPi in order
> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> going out of staging - it'll either sit here (or probably dropped) as
> statied from [1]
>
> ```
> + * There are two camera drivers in the kernel for BCM283x - this one
> + * and bcm2835-camera (currently in staging).
> ```
>
> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> mentioned in my cover the testing of bcm2835-isp happened on top of
> unicam patches.
To be accurate, the bcm2835-camera driver supports the VC4
firmware-based camera stack. In that setup, the camera sensors (OV5647
or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
firmware, which provides a high-level interface towards the kernel. This
architecture has been replaced by Linux-side control of the camera
sensors (through existing drivers in drivers/media/i2c/), Unicam
(through the driver from [1]) and ISP (through this driver). Moving
control to the Linux side requires complex processing in userspace,
handled by libcamera.
bcm2835-camera is thus replaced by multiple drivers combined with
libcamera, and that is the camera stack that is shipped by Raspberry Pi
these days. While this may affect some userspace use cases), we will not
work on destaging bcm2835-camera, and as far as I'm aware, nobody else
is planning to do so either. I don't mind much if the driver stays in
staging for some more time, but I'd rather drop it if possible.
> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
>
> >>>>
> >>>> I see two TODO files in vc04_services:
> >>>> ./bcm2835-camera/TODO
> >>>> ./interface/TODO
> >>>>
> >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>> itself. So that's address in the series. The other remaining one -
> >>>> I will need to take a deeper look before commenting on it.
> >>>>
> >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>> a cursory reading of them suggests that these apply to *all*
> >>>> vc04_services components? Am I right?
> >>>
> >>> Actually these applies just for the interfaces directory. Some of
> >>> them could apply to the services, but this is no priority.
> >>
> >> By no priority, you mean this doesn't affect the criteria required to
> >> ful-fill to get these out of staging?
> >
> > Correct
> >
> >>>> Are these are the specific bits of cleanup you are referring to in
> >>>> your comment?
> >>>
> >>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>> that i don't remember them all. The first that come to my mind was
> >>> those fancy comment sections which is not kernel coding style. It
> >>> has been removed.
> >>
> >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >> out-of-tree still so...). I mean what are the specific bits / points
> >> that needs to be addressed to get vc04_services out of the staging.
> >
> > These were the points which i mentioned in my last email. They came
> > from interface/TODO.
> >
> >> You have mentioned it above now, so I'll follow up on those.
> >
> > That would be great :)
> >
> >> The many vchiq changes you referred to above comment (that you don't
> >> remember) are from [1] as well or some other series ?
> >
> > Sorry, for the confusing. The many changes i refer were the dozens of
> > clean up patches for vc04_interfaces in mainline staging since the
> > last years. [1] was just a single patch which has been accepted yet.
>
> Ah I see. There are many others that I've to dig out then. Thanks for
> clarifying!
>
> >>> [1] -
> >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> >>>
> >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>
> >>>>> Just my two cents
> >>>>> Stefan
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 9:17 ` Laurent Pinchart
@ 2022-12-02 11:23 ` Dave Stevenson
-1 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-12-02 11:23 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Umang Jain, Stefan Wahren, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Laurent, Umang, and Stefan.
On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Umang,
>
> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > >>>>>> will remain the primary area of this series.
> > >>>>>
> > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > >>>>> comes out of staging before adding more features. As Greg said
> > >>>>> some time ago staging is not a place to "dump code and run away".
> > >>>>> These new files are in the same bad shape as the rest of vc04
> > >>>>> before the clean-up here in staging started.
> > >>>>
> > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > >>>
> > >>> no problem.
> > >>>
> > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > >>>> upstream, I would be happy to help here! Though I must mention that
> > >>>> I still have limited visibility so my aim would be to chart out a
> > >>>> plan of things needed to be done to get vc04_services out of staging!
> > >>>
> > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > >>>
> > >>> For me the most important, but not to tricky steps to get vchiq out
> > >>> of staging would be:
> > >>>
> > >>> * Cleanup logging mechanism
> > >>>
> > >>> * Get rid of custom function return values
> > >>>
> > >>> There was already an attempt for this [1]
> > >>>
> > >>> * Get rid of all non essential global structures and create a proper per
> > >>> device structure
> > >>>
> > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > >>>>> staging. It just binds more resources on a new feature.
> > >>
> > >> bcm2835-camera is the legacy camera stack which probably need to be
> > >> dropped from hereon...
> > >
> > > I don't not know if there any users left, so i would be careful here.
> > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > dumb question but i'm not expert here.
> >
> > I am careful too here and probably need Input from RaspberryPi in order
> > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > going out of staging - it'll either sit here (or probably dropped) as
> > statied from [1]
> >
> > ```
> > + * There are two camera drivers in the kernel for BCM283x - this one
> > + * and bcm2835-camera (currently in staging).
> > ```
> >
> > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > mentioned in my cover the testing of bcm2835-isp happened on top of
> > unicam patches.
>
> To be accurate, the bcm2835-camera driver supports the VC4
> firmware-based camera stack. In that setup, the camera sensors (OV5647
> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> firmware, which provides a high-level interface towards the kernel. This
> architecture has been replaced by Linux-side control of the camera
> sensors (through existing drivers in drivers/media/i2c/), Unicam
> (through the driver from [1]) and ISP (through this driver). Moving
> control to the Linux side requires complex processing in userspace,
> handled by libcamera.
>
> bcm2835-camera is thus replaced by multiple drivers combined with
> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> these days. While this may affect some userspace use cases), we will not
> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> is planning to do so either. I don't mind much if the driver stays in
> staging for some more time, but I'd rather drop it if possible.
It would be reasonable to drop it at the point that Libcamera can work
to a similar level with at least the following list of applications:
- FFmpeg
- Gstreamer
- Chromium
- Firefox
- Motion
And that still leaves a huge number of existing V4L2 apps out in the cold.
Do you wish to make any predictions as to when that would be
achievable? Or even when a v1.0 release of libcamera is going to
happen?
Dropping anything prior to those points would be rather premature in my book.
The TODOs on bcm2835-camera are:
1) Zero copy. That comes almost for free as bcm2835-isp already does
this, but it does rely on vcsm-cma.
The main reason I haven't pushed it is that it then requires
reasonable amounts of CMA heap for all the buffers, which until
recently haven't been present in the default configurations. With the
vc4 DRM driver now being default (at least for the vendor kernel) and
also requiring CMA, making the change makes more sense.
AFAIK there is no easy way to have one driver choosing between using
vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
wrong.
Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
so perhaps we don't get very far.
2) This isn't workable within the current V4L2 frameworks. The
multi-planar V4L2 pixel formats are currently allocated as independent
buffers for each plane, whereas the firmware needs a single buffer
with (currently) specific offsets for the chroma planes. The
V4L2/videobuf2 core changes required to implement that are going to be
significant, and have minimal gain.
The specific stride handling is already dealt with (set bytesperline
appropriately), it's the padding of the height to a multiple of 16
before the chroma planes on YUV420 and NV12 formats that require the
firmware to do a small amount of repacking. The performance hit is
actually minimal anyway.
If bcm2835-camera is the only thing holding back vc04_services, then I
can have a look at it.
Dave
> > [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> >
> > >>>>
> > >>>> I see two TODO files in vc04_services:
> > >>>> ./bcm2835-camera/TODO
> > >>>> ./interface/TODO
> > >>>>
> > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > >>>> itself. So that's address in the series. The other remaining one -
> > >>>> I will need to take a deeper look before commenting on it.
> > >>>>
> > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > >>>> a cursory reading of them suggests that these apply to *all*
> > >>>> vc04_services components? Am I right?
> > >>>
> > >>> Actually these applies just for the interfaces directory. Some of
> > >>> them could apply to the services, but this is no priority.
> > >>
> > >> By no priority, you mean this doesn't affect the criteria required to
> > >> ful-fill to get these out of staging?
> > >
> > > Correct
> > >
> > >>>> Are these are the specific bits of cleanup you are referring to in
> > >>>> your comment?
> > >>>
> > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > >>> that i don't remember them all. The first that come to my mind was
> > >>> those fancy comment sections which is not kernel coding style. It
> > >>> has been removed.
> > >>
> > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > >> out-of-tree still so...). I mean what are the specific bits / points
> > >> that needs to be addressed to get vc04_services out of the staging.
> > >
> > > These were the points which i mentioned in my last email. They came
> > > from interface/TODO.
> > >
> > >> You have mentioned it above now, so I'll follow up on those.
> > >
> > > That would be great :)
> > >
> > >> The many vchiq changes you referred to above comment (that you don't
> > >> remember) are from [1] as well or some other series ?
> > >
> > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > clean up patches for vc04_interfaces in mainline staging since the
> > > last years. [1] was just a single patch which has been accepted yet.
> >
> > Ah I see. There are many others that I've to dig out then. Thanks for
> > clarifying!
> >
> > >>> [1] -
> > >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> > >>>
> > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > >>>>>
> > >>>>> Just my two cents
> > >>>>> Stefan
>
> --
> Regards,
>
> Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 11:23 ` Dave Stevenson
0 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-12-02 11:23 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Umang Jain, Stefan Wahren, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Laurent, Umang, and Stefan.
On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Umang,
>
> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > >>>>>> will remain the primary area of this series.
> > >>>>>
> > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > >>>>> comes out of staging before adding more features. As Greg said
> > >>>>> some time ago staging is not a place to "dump code and run away".
> > >>>>> These new files are in the same bad shape as the rest of vc04
> > >>>>> before the clean-up here in staging started.
> > >>>>
> > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > >>>
> > >>> no problem.
> > >>>
> > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > >>>> upstream, I would be happy to help here! Though I must mention that
> > >>>> I still have limited visibility so my aim would be to chart out a
> > >>>> plan of things needed to be done to get vc04_services out of staging!
> > >>>
> > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > >>>
> > >>> For me the most important, but not to tricky steps to get vchiq out
> > >>> of staging would be:
> > >>>
> > >>> * Cleanup logging mechanism
> > >>>
> > >>> * Get rid of custom function return values
> > >>>
> > >>> There was already an attempt for this [1]
> > >>>
> > >>> * Get rid of all non essential global structures and create a proper per
> > >>> device structure
> > >>>
> > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > >>>>> staging. It just binds more resources on a new feature.
> > >>
> > >> bcm2835-camera is the legacy camera stack which probably need to be
> > >> dropped from hereon...
> > >
> > > I don't not know if there any users left, so i would be careful here.
> > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > dumb question but i'm not expert here.
> >
> > I am careful too here and probably need Input from RaspberryPi in order
> > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > going out of staging - it'll either sit here (or probably dropped) as
> > statied from [1]
> >
> > ```
> > + * There are two camera drivers in the kernel for BCM283x - this one
> > + * and bcm2835-camera (currently in staging).
> > ```
> >
> > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > mentioned in my cover the testing of bcm2835-isp happened on top of
> > unicam patches.
>
> To be accurate, the bcm2835-camera driver supports the VC4
> firmware-based camera stack. In that setup, the camera sensors (OV5647
> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> firmware, which provides a high-level interface towards the kernel. This
> architecture has been replaced by Linux-side control of the camera
> sensors (through existing drivers in drivers/media/i2c/), Unicam
> (through the driver from [1]) and ISP (through this driver). Moving
> control to the Linux side requires complex processing in userspace,
> handled by libcamera.
>
> bcm2835-camera is thus replaced by multiple drivers combined with
> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> these days. While this may affect some userspace use cases), we will not
> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> is planning to do so either. I don't mind much if the driver stays in
> staging for some more time, but I'd rather drop it if possible.
It would be reasonable to drop it at the point that Libcamera can work
to a similar level with at least the following list of applications:
- FFmpeg
- Gstreamer
- Chromium
- Firefox
- Motion
And that still leaves a huge number of existing V4L2 apps out in the cold.
Do you wish to make any predictions as to when that would be
achievable? Or even when a v1.0 release of libcamera is going to
happen?
Dropping anything prior to those points would be rather premature in my book.
The TODOs on bcm2835-camera are:
1) Zero copy. That comes almost for free as bcm2835-isp already does
this, but it does rely on vcsm-cma.
The main reason I haven't pushed it is that it then requires
reasonable amounts of CMA heap for all the buffers, which until
recently haven't been present in the default configurations. With the
vc4 DRM driver now being default (at least for the vendor kernel) and
also requiring CMA, making the change makes more sense.
AFAIK there is no easy way to have one driver choosing between using
vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
wrong.
Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
so perhaps we don't get very far.
2) This isn't workable within the current V4L2 frameworks. The
multi-planar V4L2 pixel formats are currently allocated as independent
buffers for each plane, whereas the firmware needs a single buffer
with (currently) specific offsets for the chroma planes. The
V4L2/videobuf2 core changes required to implement that are going to be
significant, and have minimal gain.
The specific stride handling is already dealt with (set bytesperline
appropriately), it's the padding of the height to a multiple of 16
before the chroma planes on YUV420 and NV12 formats that require the
firmware to do a small amount of repacking. The performance hit is
actually minimal anyway.
If bcm2835-camera is the only thing holding back vc04_services, then I
can have a look at it.
Dave
> > [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> >
> > >>>>
> > >>>> I see two TODO files in vc04_services:
> > >>>> ./bcm2835-camera/TODO
> > >>>> ./interface/TODO
> > >>>>
> > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > >>>> itself. So that's address in the series. The other remaining one -
> > >>>> I will need to take a deeper look before commenting on it.
> > >>>>
> > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > >>>> a cursory reading of them suggests that these apply to *all*
> > >>>> vc04_services components? Am I right?
> > >>>
> > >>> Actually these applies just for the interfaces directory. Some of
> > >>> them could apply to the services, but this is no priority.
> > >>
> > >> By no priority, you mean this doesn't affect the criteria required to
> > >> ful-fill to get these out of staging?
> > >
> > > Correct
> > >
> > >>>> Are these are the specific bits of cleanup you are referring to in
> > >>>> your comment?
> > >>>
> > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > >>> that i don't remember them all. The first that come to my mind was
> > >>> those fancy comment sections which is not kernel coding style. It
> > >>> has been removed.
> > >>
> > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > >> out-of-tree still so...). I mean what are the specific bits / points
> > >> that needs to be addressed to get vc04_services out of the staging.
> > >
> > > These were the points which i mentioned in my last email. They came
> > > from interface/TODO.
> > >
> > >> You have mentioned it above now, so I'll follow up on those.
> > >
> > > That would be great :)
> > >
> > >> The many vchiq changes you referred to above comment (that you don't
> > >> remember) are from [1] as well or some other series ?
> > >
> > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > clean up patches for vc04_interfaces in mainline staging since the
> > > last years. [1] was just a single patch which has been accepted yet.
> >
> > Ah I see. There are many others that I've to dig out then. Thanks for
> > clarifying!
> >
> > >>> [1] -
> > >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> > >>>
> > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > >>>>>
> > >>>>> Just my two cents
> > >>>>> Stefan
>
> --
> Regards,
>
> Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 11:23 ` Dave Stevenson
@ 2022-12-02 12:10 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-12-02 12:10 UTC (permalink / raw)
To: Dave Stevenson
Cc: Umang Jain, Stefan Wahren, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > >>>>>> will remain the primary area of this series.
> > > >>>>>
> > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > >>>>> comes out of staging before adding more features. As Greg said
> > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > >>>>> before the clean-up here in staging started.
> > > >>>>
> > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > >>>
> > > >>> no problem.
> > > >>>
> > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > >>>> I still have limited visibility so my aim would be to chart out a
> > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > >>>
> > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > >>>
> > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > >>> of staging would be:
> > > >>>
> > > >>> * Cleanup logging mechanism
> > > >>>
> > > >>> * Get rid of custom function return values
> > > >>>
> > > >>> There was already an attempt for this [1]
> > > >>>
> > > >>> * Get rid of all non essential global structures and create a proper per
> > > >>> device structure
> > > >>>
> > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > >>>>> staging. It just binds more resources on a new feature.
> > > >>
> > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > >> dropped from hereon...
> > > >
> > > > I don't not know if there any users left, so i would be careful here.
> > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > dumb question but i'm not expert here.
> > >
> > > I am careful too here and probably need Input from RaspberryPi in order
> > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > going out of staging - it'll either sit here (or probably dropped) as
> > > statied from [1]
> > >
> > > ```
> > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > + * and bcm2835-camera (currently in staging).
> > > ```
> > >
> > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > unicam patches.
> >
> > To be accurate, the bcm2835-camera driver supports the VC4
> > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > firmware, which provides a high-level interface towards the kernel. This
> > architecture has been replaced by Linux-side control of the camera
> > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > (through the driver from [1]) and ISP (through this driver). Moving
> > control to the Linux side requires complex processing in userspace,
> > handled by libcamera.
> >
> > bcm2835-camera is thus replaced by multiple drivers combined with
> > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > these days. While this may affect some userspace use cases), we will not
> > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > is planning to do so either. I don't mind much if the driver stays in
> > staging for some more time, but I'd rather drop it if possible.
>
> It would be reasonable to drop it at the point that Libcamera can work
> to a similar level with at least the following list of applications:
> - FFmpeg
> - Gstreamer
> - Chromium
> - Firefox
> - Motion
> And that still leaves a huge number of existing V4L2 apps out in the cold.
That's exactly the kind of input we were looking for, thanks.
GStreamer is already addressed.
Chromium and Firefox will go through PipeWire. There is a working
implementation in libwebrtc, Kieran may be able to comment on the
upstreaming state. It will take some time for distributions to switch,
and I can't predict the time line, but that seems to clearly be the
direction the Linux desktop is taking.
I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
sense to add native libcamera support to FFmpeg, even if PipeWire
support could also make sense. It could also make sense to expose in the
libcamera V4L2 compat layer the libv4l2 API functions, as that would
allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
an immediate solution to the problem.
Same thing for motion, except it has no libv4l2 support. The V4L2 compat
layer could still be used with LD_PRELOAD, but that's not a great
solution. Native libcamera support would make more sense (or possibly
even GStreamer support, I don't know if upstream would accept that).
In a side note, how do the above applications work today on Raspberry Pi
platforms that use a sensor not supported by the legacy camera stack ?
> Do you wish to make any predictions as to when that would be
> achievable? Or even when a v1.0 release of libcamera is going to
> happen?
Now that we have started tagging releases, we've also decided to publish
a roadmap with the development still needed to stabilize the API. We'll
likely start working on it this month.
> Dropping anything prior to those points would be rather premature in my book.
Something I forgot to mention is that there should be no issue at all
keeping bcm2835-camera fully supported in the Raspberry Pi downstream
kernel for a longer period of time. It's in upstream that I don't think
it should be destaged, as it's already considered legacy and should be
phased out. Do you know if there are users of that driver with a
mainline kernel ?
> The TODOs on bcm2835-camera are:
> 1) Zero copy. That comes almost for free as bcm2835-isp already does
> this, but it does rely on vcsm-cma.
> The main reason I haven't pushed it is that it then requires
> reasonable amounts of CMA heap for all the buffers, which until
> recently haven't been present in the default configurations. With the
> vc4 DRM driver now being default (at least for the vendor kernel) and
> also requiring CMA, making the change makes more sense.
> AFAIK there is no easy way to have one driver choosing between using
> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> wrong.
Some drivers use a module parameter for that, but that's not great.
> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> so perhaps we don't get very far.
>
> 2) This isn't workable within the current V4L2 frameworks. The
> multi-planar V4L2 pixel formats are currently allocated as independent
> buffers for each plane, whereas the firmware needs a single buffer
> with (currently) specific offsets for the chroma planes. The
> V4L2/videobuf2 core changes required to implement that are going to be
> significant, and have minimal gain.
> The specific stride handling is already dealt with (set bytesperline
> appropriately), it's the padding of the height to a multiple of 16
> before the chroma planes on YUV420 and NV12 formats that require the
> firmware to do a small amount of repacking. The performance hit is
> actually minimal anyway.
>
> If bcm2835-camera is the only thing holding back vc04_services, then I
> can have a look at it.
I'll let Umang comment on whether it's holding vc04_services back, but
my understanding it that we could destage vc04_services while keeping
bcm2835-camera in staging for the time being. If anyone disagrees with
that, please let me know.
> > > [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> > >
> > > >>>>
> > > >>>> I see two TODO files in vc04_services:
> > > >>>> ./bcm2835-camera/TODO
> > > >>>> ./interface/TODO
> > > >>>>
> > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > >>>> itself. So that's address in the series. The other remaining one -
> > > >>>> I will need to take a deeper look before commenting on it.
> > > >>>>
> > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > >>>> a cursory reading of them suggests that these apply to *all*
> > > >>>> vc04_services components? Am I right?
> > > >>>
> > > >>> Actually these applies just for the interfaces directory. Some of
> > > >>> them could apply to the services, but this is no priority.
> > > >>
> > > >> By no priority, you mean this doesn't affect the criteria required to
> > > >> ful-fill to get these out of staging?
> > > >
> > > > Correct
> > > >
> > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > >>>> your comment?
> > > >>>
> > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > >>> that i don't remember them all. The first that come to my mind was
> > > >>> those fancy comment sections which is not kernel coding style. It
> > > >>> has been removed.
> > > >>
> > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > >> that needs to be addressed to get vc04_services out of the staging.
> > > >
> > > > These were the points which i mentioned in my last email. They came
> > > > from interface/TODO.
> > > >
> > > >> You have mentioned it above now, so I'll follow up on those.
> > > >
> > > > That would be great :)
> > > >
> > > >> The many vchiq changes you referred to above comment (that you don't
> > > >> remember) are from [1] as well or some other series ?
> > > >
> > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > last years. [1] was just a single patch which has been accepted yet.
> > >
> > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > clarifying!
> > >
> > > >>> [1] -
> > > >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> > > >>>
> > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > >>>>>
> > > >>>>> Just my two cents
> > > >>>>> Stefan
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 12:10 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-12-02 12:10 UTC (permalink / raw)
To: Dave Stevenson
Cc: Umang Jain, Stefan Wahren, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > >>>>>> will remain the primary area of this series.
> > > >>>>>
> > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > >>>>> comes out of staging before adding more features. As Greg said
> > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > >>>>> before the clean-up here in staging started.
> > > >>>>
> > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > >>>
> > > >>> no problem.
> > > >>>
> > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > >>>> I still have limited visibility so my aim would be to chart out a
> > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > >>>
> > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > >>>
> > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > >>> of staging would be:
> > > >>>
> > > >>> * Cleanup logging mechanism
> > > >>>
> > > >>> * Get rid of custom function return values
> > > >>>
> > > >>> There was already an attempt for this [1]
> > > >>>
> > > >>> * Get rid of all non essential global structures and create a proper per
> > > >>> device structure
> > > >>>
> > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > >>>>> staging. It just binds more resources on a new feature.
> > > >>
> > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > >> dropped from hereon...
> > > >
> > > > I don't not know if there any users left, so i would be careful here.
> > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > dumb question but i'm not expert here.
> > >
> > > I am careful too here and probably need Input from RaspberryPi in order
> > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > going out of staging - it'll either sit here (or probably dropped) as
> > > statied from [1]
> > >
> > > ```
> > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > + * and bcm2835-camera (currently in staging).
> > > ```
> > >
> > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > unicam patches.
> >
> > To be accurate, the bcm2835-camera driver supports the VC4
> > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > firmware, which provides a high-level interface towards the kernel. This
> > architecture has been replaced by Linux-side control of the camera
> > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > (through the driver from [1]) and ISP (through this driver). Moving
> > control to the Linux side requires complex processing in userspace,
> > handled by libcamera.
> >
> > bcm2835-camera is thus replaced by multiple drivers combined with
> > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > these days. While this may affect some userspace use cases), we will not
> > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > is planning to do so either. I don't mind much if the driver stays in
> > staging for some more time, but I'd rather drop it if possible.
>
> It would be reasonable to drop it at the point that Libcamera can work
> to a similar level with at least the following list of applications:
> - FFmpeg
> - Gstreamer
> - Chromium
> - Firefox
> - Motion
> And that still leaves a huge number of existing V4L2 apps out in the cold.
That's exactly the kind of input we were looking for, thanks.
GStreamer is already addressed.
Chromium and Firefox will go through PipeWire. There is a working
implementation in libwebrtc, Kieran may be able to comment on the
upstreaming state. It will take some time for distributions to switch,
and I can't predict the time line, but that seems to clearly be the
direction the Linux desktop is taking.
I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
sense to add native libcamera support to FFmpeg, even if PipeWire
support could also make sense. It could also make sense to expose in the
libcamera V4L2 compat layer the libv4l2 API functions, as that would
allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
an immediate solution to the problem.
Same thing for motion, except it has no libv4l2 support. The V4L2 compat
layer could still be used with LD_PRELOAD, but that's not a great
solution. Native libcamera support would make more sense (or possibly
even GStreamer support, I don't know if upstream would accept that).
In a side note, how do the above applications work today on Raspberry Pi
platforms that use a sensor not supported by the legacy camera stack ?
> Do you wish to make any predictions as to when that would be
> achievable? Or even when a v1.0 release of libcamera is going to
> happen?
Now that we have started tagging releases, we've also decided to publish
a roadmap with the development still needed to stabilize the API. We'll
likely start working on it this month.
> Dropping anything prior to those points would be rather premature in my book.
Something I forgot to mention is that there should be no issue at all
keeping bcm2835-camera fully supported in the Raspberry Pi downstream
kernel for a longer period of time. It's in upstream that I don't think
it should be destaged, as it's already considered legacy and should be
phased out. Do you know if there are users of that driver with a
mainline kernel ?
> The TODOs on bcm2835-camera are:
> 1) Zero copy. That comes almost for free as bcm2835-isp already does
> this, but it does rely on vcsm-cma.
> The main reason I haven't pushed it is that it then requires
> reasonable amounts of CMA heap for all the buffers, which until
> recently haven't been present in the default configurations. With the
> vc4 DRM driver now being default (at least for the vendor kernel) and
> also requiring CMA, making the change makes more sense.
> AFAIK there is no easy way to have one driver choosing between using
> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> wrong.
Some drivers use a module parameter for that, but that's not great.
> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> so perhaps we don't get very far.
>
> 2) This isn't workable within the current V4L2 frameworks. The
> multi-planar V4L2 pixel formats are currently allocated as independent
> buffers for each plane, whereas the firmware needs a single buffer
> with (currently) specific offsets for the chroma planes. The
> V4L2/videobuf2 core changes required to implement that are going to be
> significant, and have minimal gain.
> The specific stride handling is already dealt with (set bytesperline
> appropriately), it's the padding of the height to a multiple of 16
> before the chroma planes on YUV420 and NV12 formats that require the
> firmware to do a small amount of repacking. The performance hit is
> actually minimal anyway.
>
> If bcm2835-camera is the only thing holding back vc04_services, then I
> can have a look at it.
I'll let Umang comment on whether it's holding vc04_services back, but
my understanding it that we could destage vc04_services while keeping
bcm2835-camera in staging for the time being. If anyone disagrees with
that, please let me know.
> > > [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> > >
> > > >>>>
> > > >>>> I see two TODO files in vc04_services:
> > > >>>> ./bcm2835-camera/TODO
> > > >>>> ./interface/TODO
> > > >>>>
> > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > >>>> itself. So that's address in the series. The other remaining one -
> > > >>>> I will need to take a deeper look before commenting on it.
> > > >>>>
> > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > >>>> a cursory reading of them suggests that these apply to *all*
> > > >>>> vc04_services components? Am I right?
> > > >>>
> > > >>> Actually these applies just for the interfaces directory. Some of
> > > >>> them could apply to the services, but this is no priority.
> > > >>
> > > >> By no priority, you mean this doesn't affect the criteria required to
> > > >> ful-fill to get these out of staging?
> > > >
> > > > Correct
> > > >
> > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > >>>> your comment?
> > > >>>
> > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > >>> that i don't remember them all. The first that come to my mind was
> > > >>> those fancy comment sections which is not kernel coding style. It
> > > >>> has been removed.
> > > >>
> > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > >> that needs to be addressed to get vc04_services out of the staging.
> > > >
> > > > These were the points which i mentioned in my last email. They came
> > > > from interface/TODO.
> > > >
> > > >> You have mentioned it above now, so I'll follow up on those.
> > > >
> > > > That would be great :)
> > > >
> > > >> The many vchiq changes you referred to above comment (that you don't
> > > >> remember) are from [1] as well or some other series ?
> > > >
> > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > last years. [1] was just a single patch which has been accepted yet.
> > >
> > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > clarifying!
> > >
> > > >>> [1] -
> > > >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> > > >>>
> > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > >>>>>
> > > >>>>> Just my two cents
> > > >>>>> Stefan
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 11:23 ` Dave Stevenson
(?)
(?)
@ 2022-12-02 12:25 ` Stefan Wahren
-1 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-02 12:25 UTC (permalink / raw)
To: linux-arm-kernel
Hi Dave,
Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> Hi Laurent, Umang, and Stefan.
>
> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>> Hi Umang,
>>
>> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
>>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
>>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
>>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
>>>>>>>>> will remain the primary area of this series.
>>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>>>>> comes out of staging before adding more features. As Greg said
>>>>>>>> some time ago staging is not a place to "dump code and run away".
>>>>>>>> These new files are in the same bad shape as the rest of vc04
>>>>>>>> before the clean-up here in staging started.
>>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
>>>>>> no problem.
>>>>>>
>>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>>>>> upstream, I would be happy to help here! Though I must mention that
>>>>>>> I still have limited visibility so my aim would be to chart out a
>>>>>>> plan of things needed to be done to get vc04_services out of staging!
>>>>>> The vchiq driver is in staging since 2016, so every step forwards is
>>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>>>>
>>>>>> For me the most important, but not to tricky steps to get vchiq out
>>>>>> of staging would be:
>>>>>>
>>>>>> * Cleanup logging mechanism
>>>>>>
>>>>>> * Get rid of custom function return values
>>>>>>
>>>>>> There was already an attempt for this [1]
>>>>>>
>>>>>> * Get rid of all non essential global structures and create a proper per
>>>>>> device structure
>>>>>>
>>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>>>>> staging. It just binds more resources on a new feature.
>>>>> bcm2835-camera is the legacy camera stack which probably need to be
>>>>> dropped from hereon...
>>>> I don't not know if there any users left, so i would be careful here.
>>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
>>>> dumb question but i'm not expert here.
>>> I am careful too here and probably need Input from RaspberryPi in order
>>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
>>> going out of staging - it'll either sit here (or probably dropped) as
>>> statied from [1]
>>>
>>> ```
>>> + * There are two camera drivers in the kernel for BCM283x - this one
>>> + * and bcm2835-camera (currently in staging).
>>> ```
>>>
>>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
>>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
>>> mentioned in my cover the testing of bcm2835-isp happened on top of
>>> unicam patches.
>> To be accurate, the bcm2835-camera driver supports the VC4
>> firmware-based camera stack. In that setup, the camera sensors (OV5647
>> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
>> firmware, which provides a high-level interface towards the kernel. This
>> architecture has been replaced by Linux-side control of the camera
>> sensors (through existing drivers in drivers/media/i2c/), Unicam
>> (through the driver from [1]) and ISP (through this driver). Moving
>> control to the Linux side requires complex processing in userspace,
>> handled by libcamera.
Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
handle the old camera (OV5647)?
>>
>> bcm2835-camera is thus replaced by multiple drivers combined with
>> libcamera, and that is the camera stack that is shipped by Raspberry Pi
>> these days. While this may affect some userspace use cases), we will not
>> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
>> is planning to do so either. I don't mind much if the driver stays in
>> staging for some more time, but I'd rather drop it if possible.
> It would be reasonable to drop it at the point that Libcamera can work
> to a similar level with at least the following list of applications:
> - FFmpeg
> - Gstreamer
> - Chromium
> - Firefox
> - Motion
> And that still leaves a huge number of existing V4L2 apps out in the cold.
>
> Do you wish to make any predictions as to when that would be
> achievable? Or even when a v1.0 release of libcamera is going to
> happen?
> Dropping anything prior to those points would be rather premature in my book.
>
>
> The TODOs on bcm2835-camera are:
> 1) Zero copy. That comes almost for free as bcm2835-isp already does
> this, but it does rely on vcsm-cma.
> The main reason I haven't pushed it is that it then requires
> reasonable amounts of CMA heap for all the buffers, which until
> recently haven't been present in the default configurations. With the
> vc4 DRM driver now being default (at least for the vendor kernel) and
> also requiring CMA, making the change makes more sense.
> AFAIK there is no easy way to have one driver choosing between using
> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> wrong.
> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> so perhaps we don't get very far.
>
> 2) This isn't workable within the current V4L2 frameworks. The
> multi-planar V4L2 pixel formats are currently allocated as independent
> buffers for each plane, whereas the firmware needs a single buffer
> with (currently) specific offsets for the chroma planes. The
> V4L2/videobuf2 core changes required to implement that are going to be
> significant, and have minimal gain.
> The specific stride handling is already dealt with (set bytesperline
> appropriately), it's the padding of the height to a multiple of 16
> before the chroma planes on YUV420 and NV12 formats that require the
> firmware to do a small amount of repacking. The performance hit is
> actually minimal anyway.
>
> If bcm2835-camera is the only thing holding back vc04_services, then I
> can have a look at it.
No, it's the vchiq interface which needs the work.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 12:10 ` Laurent Pinchart
@ 2022-12-02 12:35 ` Stefan Wahren
-1 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-02 12:35 UTC (permalink / raw)
To: Laurent Pinchart, Dave Stevenson
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham, Peter Robinson,
Ivan T. Ivanov
Hi Laurent,
Am 02.12.22 um 13:10 schrieb Laurent Pinchart:
> Hi Dave,
>
> On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
>> Dropping anything prior to those points would be rather premature in my book.
> Something I forgot to mention is that there should be no issue at all
> keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> kernel for a longer period of time. It's in upstream that I don't think
> it should be destaged, as it's already considered legacy and should be
> phased out. Do you know if there are users of that driver with a
> mainline kernel ?
>
In Fedora there seems to be no official support [1], but openSuSE seems
to mention it [2].
[1] -
https://docs.fedoraproject.org/en-US/quick-docs/raspberry-pi/#_does_the_add_on_camera_work
[2] - https://en.opensuse.org/HCL:Raspberry_Pi_Camera_Modules
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 12:35 ` Stefan Wahren
0 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-02 12:35 UTC (permalink / raw)
To: Laurent Pinchart, Dave Stevenson
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham, Peter Robinson,
Ivan T. Ivanov
Hi Laurent,
Am 02.12.22 um 13:10 schrieb Laurent Pinchart:
> Hi Dave,
>
> On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
>> Dropping anything prior to those points would be rather premature in my book.
> Something I forgot to mention is that there should be no issue at all
> keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> kernel for a longer period of time. It's in upstream that I don't think
> it should be destaged, as it's already considered legacy and should be
> phased out. Do you know if there are users of that driver with a
> mainline kernel ?
>
In Fedora there seems to be no official support [1], but openSuSE seems
to mention it [2].
[1] -
https://docs.fedoraproject.org/en-US/quick-docs/raspberry-pi/#_does_the_add_on_camera_work
[2] - https://en.opensuse.org/HCL:Raspberry_Pi_Camera_Modules
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 12:10 ` Laurent Pinchart
@ 2022-12-02 12:38 ` Dave Stevenson
-1 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-12-02 12:38 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Umang Jain, Stefan Wahren, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Laurent
On Fri, 2 Dec 2022 at 12:10, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Dave,
>
> On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > > >>>>>> will remain the primary area of this series.
> > > > >>>>>
> > > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > > >>>>> comes out of staging before adding more features. As Greg said
> > > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > > >>>>> before the clean-up here in staging started.
> > > > >>>>
> > > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > > >>>
> > > > >>> no problem.
> > > > >>>
> > > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > > >>>> I still have limited visibility so my aim would be to chart out a
> > > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > > >>>
> > > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > > >>>
> > > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > > >>> of staging would be:
> > > > >>>
> > > > >>> * Cleanup logging mechanism
> > > > >>>
> > > > >>> * Get rid of custom function return values
> > > > >>>
> > > > >>> There was already an attempt for this [1]
> > > > >>>
> > > > >>> * Get rid of all non essential global structures and create a proper per
> > > > >>> device structure
> > > > >>>
> > > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > > >>>>> staging. It just binds more resources on a new feature.
> > > > >>
> > > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > > >> dropped from hereon...
> > > > >
> > > > > I don't not know if there any users left, so i would be careful here.
> > > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > > dumb question but i'm not expert here.
> > > >
> > > > I am careful too here and probably need Input from RaspberryPi in order
> > > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > > going out of staging - it'll either sit here (or probably dropped) as
> > > > statied from [1]
> > > >
> > > > ```
> > > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > > + * and bcm2835-camera (currently in staging).
> > > > ```
> > > >
> > > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > > unicam patches.
> > >
> > > To be accurate, the bcm2835-camera driver supports the VC4
> > > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > > firmware, which provides a high-level interface towards the kernel. This
> > > architecture has been replaced by Linux-side control of the camera
> > > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > > (through the driver from [1]) and ISP (through this driver). Moving
> > > control to the Linux side requires complex processing in userspace,
> > > handled by libcamera.
> > >
> > > bcm2835-camera is thus replaced by multiple drivers combined with
> > > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > > these days. While this may affect some userspace use cases), we will not
> > > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > > is planning to do so either. I don't mind much if the driver stays in
> > > staging for some more time, but I'd rather drop it if possible.
> >
> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
>
> That's exactly the kind of input we were looking for, thanks.
Please note that those are only the 5 major ones that came immediately
to mind as having significant numbers of users. There are obviously
many more V4L2 apps around which currently "just work" if they use the
API correctly.
> GStreamer is already addressed.
>
> Chromium and Firefox will go through PipeWire. There is a working
> implementation in libwebrtc, Kieran may be able to comment on the
> upstreaming state. It will take some time for distributions to switch,
> and I can't predict the time line, but that seems to clearly be the
> direction the Linux desktop is taking.
>
> I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
> sense to add native libcamera support to FFmpeg, even if PipeWire
> support could also make sense. It could also make sense to expose in the
> libcamera V4L2 compat layer the libv4l2 API functions, as that would
> allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
> instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
> an immediate solution to the problem.
Native would be nice in my book, but isn't always as straightforward.
> Same thing for motion, except it has no libv4l2 support. The V4L2 compat
> layer could still be used with LD_PRELOAD, but that's not a great
> solution. Native libcamera support would make more sense (or possibly
> even GStreamer support, I don't know if upstream would accept that).
The main bit missing with the compat layer is controls. AIUI fixing
that isn't a priority.
Things like Motion being used for security purposes need to be able to
set properties like exposure modes in order to achieve usable images
under changing lighting conditions.
> In a side note, how do the above applications work today on Raspberry Pi
> platforms that use a sensor not supported by the legacy camera stack ?
They don't for raw sensors, but do work with USB webcams.
Raspberry Pi have sold camera modules since 2013 (OV5647), with v2
(IMX219) coming along in 2016, and HQ (IMX477) in 2020. Linux has only
had frameworks to sensibly support raw image sensors since libcamera
came on the scene around 2019.
> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
>
> Now that we have started tagging releases, we've also decided to publish
> a roadmap with the development still needed to stabilize the API. We'll
> likely start working on it this month.
>
> > Dropping anything prior to those points would be rather premature in my book.
>
> Something I forgot to mention is that there should be no issue at all
> keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> kernel for a longer period of time. It's in upstream that I don't think
> it should be destaged, as it's already considered legacy and should be
> phased out. Do you know if there are users of that driver with a
> mainline kernel ?
There are a number of distros that use mainline kernels rather than
our vendor kernel. I don't follow distros in detail, but I believe
both Gentoo and Opensuse fall into that category.
Any users of the camera module under those will be using the mainline
bcm2835-camera module. I have no stats for who uses those on a Pi.
> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
>
> Some drivers use a module parameter for that, but that's not great.
I had thought of a module parameter, but it becomes yet another
configuration thing to get incorrectly set.
Perhaps Kconfig based on the setting for CMA_SIZE_MBYTES, but that can
be overridden.
Dave
> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
> >
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> I'll let Umang comment on whether it's holding vc04_services back, but
> my understanding it that we could destage vc04_services while keeping
> bcm2835-camera in staging for the time being. If anyone disagrees with
> that, please let me know.
>
> > > > [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> > > >
> > > > >>>>
> > > > >>>> I see two TODO files in vc04_services:
> > > > >>>> ./bcm2835-camera/TODO
> > > > >>>> ./interface/TODO
> > > > >>>>
> > > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > > >>>> itself. So that's address in the series. The other remaining one -
> > > > >>>> I will need to take a deeper look before commenting on it.
> > > > >>>>
> > > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > > >>>> a cursory reading of them suggests that these apply to *all*
> > > > >>>> vc04_services components? Am I right?
> > > > >>>
> > > > >>> Actually these applies just for the interfaces directory. Some of
> > > > >>> them could apply to the services, but this is no priority.
> > > > >>
> > > > >> By no priority, you mean this doesn't affect the criteria required to
> > > > >> ful-fill to get these out of staging?
> > > > >
> > > > > Correct
> > > > >
> > > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > > >>>> your comment?
> > > > >>>
> > > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > > >>> that i don't remember them all. The first that come to my mind was
> > > > >>> those fancy comment sections which is not kernel coding style. It
> > > > >>> has been removed.
> > > > >>
> > > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > > >> that needs to be addressed to get vc04_services out of the staging.
> > > > >
> > > > > These were the points which i mentioned in my last email. They came
> > > > > from interface/TODO.
> > > > >
> > > > >> You have mentioned it above now, so I'll follow up on those.
> > > > >
> > > > > That would be great :)
> > > > >
> > > > >> The many vchiq changes you referred to above comment (that you don't
> > > > >> remember) are from [1] as well or some other series ?
> > > > >
> > > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > > last years. [1] was just a single patch which has been accepted yet.
> > > >
> > > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > > clarifying!
> > > >
> > > > >>> [1] -
> > > > >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> > > > >>>
> > > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > > >>>>>
> > > > >>>>> Just my two cents
> > > > >>>>> Stefan
>
> --
> Regards,
>
> Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 12:38 ` Dave Stevenson
0 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-12-02 12:38 UTC (permalink / raw)
To: Laurent Pinchart
Cc: Umang Jain, Stefan Wahren, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Laurent
On Fri, 2 Dec 2022 at 12:10, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
>
> Hi Dave,
>
> On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > > >>>>>> will remain the primary area of this series.
> > > > >>>>>
> > > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > > >>>>> comes out of staging before adding more features. As Greg said
> > > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > > >>>>> before the clean-up here in staging started.
> > > > >>>>
> > > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > > >>>
> > > > >>> no problem.
> > > > >>>
> > > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > > >>>> I still have limited visibility so my aim would be to chart out a
> > > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > > >>>
> > > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > > >>>
> > > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > > >>> of staging would be:
> > > > >>>
> > > > >>> * Cleanup logging mechanism
> > > > >>>
> > > > >>> * Get rid of custom function return values
> > > > >>>
> > > > >>> There was already an attempt for this [1]
> > > > >>>
> > > > >>> * Get rid of all non essential global structures and create a proper per
> > > > >>> device structure
> > > > >>>
> > > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > > >>>>> staging. It just binds more resources on a new feature.
> > > > >>
> > > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > > >> dropped from hereon...
> > > > >
> > > > > I don't not know if there any users left, so i would be careful here.
> > > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > > dumb question but i'm not expert here.
> > > >
> > > > I am careful too here and probably need Input from RaspberryPi in order
> > > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > > going out of staging - it'll either sit here (or probably dropped) as
> > > > statied from [1]
> > > >
> > > > ```
> > > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > > + * and bcm2835-camera (currently in staging).
> > > > ```
> > > >
> > > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > > unicam patches.
> > >
> > > To be accurate, the bcm2835-camera driver supports the VC4
> > > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > > firmware, which provides a high-level interface towards the kernel. This
> > > architecture has been replaced by Linux-side control of the camera
> > > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > > (through the driver from [1]) and ISP (through this driver). Moving
> > > control to the Linux side requires complex processing in userspace,
> > > handled by libcamera.
> > >
> > > bcm2835-camera is thus replaced by multiple drivers combined with
> > > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > > these days. While this may affect some userspace use cases), we will not
> > > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > > is planning to do so either. I don't mind much if the driver stays in
> > > staging for some more time, but I'd rather drop it if possible.
> >
> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
>
> That's exactly the kind of input we were looking for, thanks.
Please note that those are only the 5 major ones that came immediately
to mind as having significant numbers of users. There are obviously
many more V4L2 apps around which currently "just work" if they use the
API correctly.
> GStreamer is already addressed.
>
> Chromium and Firefox will go through PipeWire. There is a working
> implementation in libwebrtc, Kieran may be able to comment on the
> upstreaming state. It will take some time for distributions to switch,
> and I can't predict the time line, but that seems to clearly be the
> direction the Linux desktop is taking.
>
> I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
> sense to add native libcamera support to FFmpeg, even if PipeWire
> support could also make sense. It could also make sense to expose in the
> libcamera V4L2 compat layer the libv4l2 API functions, as that would
> allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
> instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
> an immediate solution to the problem.
Native would be nice in my book, but isn't always as straightforward.
> Same thing for motion, except it has no libv4l2 support. The V4L2 compat
> layer could still be used with LD_PRELOAD, but that's not a great
> solution. Native libcamera support would make more sense (or possibly
> even GStreamer support, I don't know if upstream would accept that).
The main bit missing with the compat layer is controls. AIUI fixing
that isn't a priority.
Things like Motion being used for security purposes need to be able to
set properties like exposure modes in order to achieve usable images
under changing lighting conditions.
> In a side note, how do the above applications work today on Raspberry Pi
> platforms that use a sensor not supported by the legacy camera stack ?
They don't for raw sensors, but do work with USB webcams.
Raspberry Pi have sold camera modules since 2013 (OV5647), with v2
(IMX219) coming along in 2016, and HQ (IMX477) in 2020. Linux has only
had frameworks to sensibly support raw image sensors since libcamera
came on the scene around 2019.
> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
>
> Now that we have started tagging releases, we've also decided to publish
> a roadmap with the development still needed to stabilize the API. We'll
> likely start working on it this month.
>
> > Dropping anything prior to those points would be rather premature in my book.
>
> Something I forgot to mention is that there should be no issue at all
> keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> kernel for a longer period of time. It's in upstream that I don't think
> it should be destaged, as it's already considered legacy and should be
> phased out. Do you know if there are users of that driver with a
> mainline kernel ?
There are a number of distros that use mainline kernels rather than
our vendor kernel. I don't follow distros in detail, but I believe
both Gentoo and Opensuse fall into that category.
Any users of the camera module under those will be using the mainline
bcm2835-camera module. I have no stats for who uses those on a Pi.
> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
>
> Some drivers use a module parameter for that, but that's not great.
I had thought of a module parameter, but it becomes yet another
configuration thing to get incorrectly set.
Perhaps Kconfig based on the setting for CMA_SIZE_MBYTES, but that can
be overridden.
Dave
> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
> >
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> I'll let Umang comment on whether it's holding vc04_services back, but
> my understanding it that we could destage vc04_services while keeping
> bcm2835-camera in staging for the time being. If anyone disagrees with
> that, please let me know.
>
> > > > [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> > > >
> > > > >>>>
> > > > >>>> I see two TODO files in vc04_services:
> > > > >>>> ./bcm2835-camera/TODO
> > > > >>>> ./interface/TODO
> > > > >>>>
> > > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > > >>>> itself. So that's address in the series. The other remaining one -
> > > > >>>> I will need to take a deeper look before commenting on it.
> > > > >>>>
> > > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > > >>>> a cursory reading of them suggests that these apply to *all*
> > > > >>>> vc04_services components? Am I right?
> > > > >>>
> > > > >>> Actually these applies just for the interfaces directory. Some of
> > > > >>> them could apply to the services, but this is no priority.
> > > > >>
> > > > >> By no priority, you mean this doesn't affect the criteria required to
> > > > >> ful-fill to get these out of staging?
> > > > >
> > > > > Correct
> > > > >
> > > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > > >>>> your comment?
> > > > >>>
> > > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > > >>> that i don't remember them all. The first that come to my mind was
> > > > >>> those fancy comment sections which is not kernel coding style. It
> > > > >>> has been removed.
> > > > >>
> > > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > > >> that needs to be addressed to get vc04_services out of the staging.
> > > > >
> > > > > These were the points which i mentioned in my last email. They came
> > > > > from interface/TODO.
> > > > >
> > > > >> You have mentioned it above now, so I'll follow up on those.
> > > > >
> > > > > That would be great :)
> > > > >
> > > > >> The many vchiq changes you referred to above comment (that you don't
> > > > >> remember) are from [1] as well or some other series ?
> > > > >
> > > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > > last years. [1] was just a single patch which has been accepted yet.
> > > >
> > > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > > clarifying!
> > > >
> > > > >>> [1] -
> > > > >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> > > > >>>
> > > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > > >>>>>
> > > > >>>>> Just my two cents
> > > > >>>>> Stefan
>
> --
> Regards,
>
> Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 11:23 ` Dave Stevenson
@ 2022-12-02 12:41 ` Stefan Wahren
-1 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-02 12:41 UTC (permalink / raw)
To: Dave Stevenson, Laurent Pinchart
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> Hi Laurent, Umang, and Stefan.
>
> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>> Hi Umang,
>>
>> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
>>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
>>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
>>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
>>>>>>>>> will remain the primary area of this series.
>>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>>>>> comes out of staging before adding more features. As Greg said
>>>>>>>> some time ago staging is not a place to "dump code and run away".
>>>>>>>> These new files are in the same bad shape as the rest of vc04
>>>>>>>> before the clean-up here in staging started.
>>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
>>>>>> no problem.
>>>>>>
>>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>>>>> upstream, I would be happy to help here! Though I must mention that
>>>>>>> I still have limited visibility so my aim would be to chart out a
>>>>>>> plan of things needed to be done to get vc04_services out of staging!
>>>>>> The vchiq driver is in staging since 2016, so every step forwards is
>>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>>>>
>>>>>> For me the most important, but not to tricky steps to get vchiq out
>>>>>> of staging would be:
>>>>>>
>>>>>> * Cleanup logging mechanism
>>>>>>
>>>>>> * Get rid of custom function return values
>>>>>>
>>>>>> There was already an attempt for this [1]
>>>>>>
>>>>>> * Get rid of all non essential global structures and create a proper per
>>>>>> device structure
>>>>>>
>>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>>>>> staging. It just binds more resources on a new feature.
>>>>> bcm2835-camera is the legacy camera stack which probably need to be
>>>>> dropped from hereon...
>>>> I don't not know if there any users left, so i would be careful here.
>>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
>>>> dumb question but i'm not expert here.
>>> I am careful too here and probably need Input from RaspberryPi in order
>>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
>>> going out of staging - it'll either sit here (or probably dropped) as
>>> statied from [1]
>>>
>>> ```
>>> + * There are two camera drivers in the kernel for BCM283x - this one
>>> + * and bcm2835-camera (currently in staging).
>>> ```
>>>
>>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
>>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
>>> mentioned in my cover the testing of bcm2835-isp happened on top of
>>> unicam patches.
>> To be accurate, the bcm2835-camera driver supports the VC4
>> firmware-based camera stack. In that setup, the camera sensors (OV5647
>> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
>> firmware, which provides a high-level interface towards the kernel. This
>> architecture has been replaced by Linux-side control of the camera
>> sensors (through existing drivers in drivers/media/i2c/), Unicam
>> (through the driver from [1]) and ISP (through this driver). Moving
>> control to the Linux side requires complex processing in userspace,
>> handled by libcamera.
>>
>> bcm2835-camera is thus replaced by multiple drivers combined with
>> libcamera, and that is the camera stack that is shipped by Raspberry Pi
>> these days. While this may affect some userspace use cases), we will not
>> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
>> is planning to do so either. I don't mind much if the driver stays in
>> staging for some more time, but I'd rather drop it if possible.
Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
handle the old camera (OV5647)?
> It would be reasonable to drop it at the point that Libcamera can work
> to a similar level with at least the following list of applications:
> - FFmpeg
> - Gstreamer
> - Chromium
> - Firefox
> - Motion
> And that still leaves a huge number of existing V4L2 apps out in the cold.
>
> Do you wish to make any predictions as to when that would be
> achievable? Or even when a v1.0 release of libcamera is going to
> happen?
> Dropping anything prior to those points would be rather premature in my book.
>
>
> The TODOs on bcm2835-camera are:
> 1) Zero copy. That comes almost for free as bcm2835-isp already does
> this, but it does rely on vcsm-cma.
> The main reason I haven't pushed it is that it then requires
> reasonable amounts of CMA heap for all the buffers, which until
> recently haven't been present in the default configurations. With the
> vc4 DRM driver now being default (at least for the vendor kernel) and
> also requiring CMA, making the change makes more sense.
> AFAIK there is no easy way to have one driver choosing between using
> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> wrong.
> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> so perhaps we don't get very far.
CMA configuration should actually happen in device tree or kernel
cmdline. The bcm2835_defconfig is limited to make it work even with the
original Pi.
>
> 2) This isn't workable within the current V4L2 frameworks. The
> multi-planar V4L2 pixel formats are currently allocated as independent
> buffers for each plane, whereas the firmware needs a single buffer
> with (currently) specific offsets for the chroma planes. The
> V4L2/videobuf2 core changes required to implement that are going to be
> significant, and have minimal gain.
> The specific stride handling is already dealt with (set bytesperline
> appropriately), it's the padding of the height to a multiple of 16
> before the chroma planes on YUV420 and NV12 formats that require the
> firmware to do a small amount of repacking. The performance hit is
> actually minimal anyway.
>
> If bcm2835-camera is the only thing holding back vc04_services, then I
> can have a look at it.
No, it's the vchiq interface which needs the work.
Thanks Stefan
>
> Dave
>
>>> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
>>>
>>>>>>> I see two TODO files in vc04_services:
>>>>>>> ./bcm2835-camera/TODO
>>>>>>> ./interface/TODO
>>>>>>>
>>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>>>>>> itself. So that's address in the series. The other remaining one -
>>>>>>> I will need to take a deeper look before commenting on it.
>>>>>>>
>>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
>>>>>>> a cursory reading of them suggests that these apply to *all*
>>>>>>> vc04_services components? Am I right?
>>>>>> Actually these applies just for the interfaces directory. Some of
>>>>>> them could apply to the services, but this is no priority.
>>>>> By no priority, you mean this doesn't affect the criteria required to
>>>>> ful-fill to get these out of staging?
>>>> Correct
>>>>
>>>>>>> Are these are the specific bits of cleanup you are referring to in
>>>>>>> your comment?
>>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
>>>>>> that i don't remember them all. The first that come to my mind was
>>>>>> those fancy comment sections which is not kernel coding style. It
>>>>>> has been removed.
>>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
>>>>> out-of-tree still so...). I mean what are the specific bits / points
>>>>> that needs to be addressed to get vc04_services out of the staging.
>>>> These were the points which i mentioned in my last email. They came
>>>> from interface/TODO.
>>>>
>>>>> You have mentioned it above now, so I'll follow up on those.
>>>> That would be great :)
>>>>
>>>>> The many vchiq changes you referred to above comment (that you don't
>>>>> remember) are from [1] as well or some other series ?
>>>> Sorry, for the confusing. The many changes i refer were the dozens of
>>>> clean up patches for vc04_interfaces in mainline staging since the
>>>> last years. [1] was just a single patch which has been accepted yet.
>>> Ah I see. There are many others that I've to dig out then. Thanks for
>>> clarifying!
>>>
>>>>>> [1] -
>>>>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>>>>>>
>>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>>>>>
>>>>>>>> Just my two cents
>>>>>>>> Stefan
>> --
>> Regards,
>>
>> Laurent Pinchart
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 12:41 ` Stefan Wahren
0 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-02 12:41 UTC (permalink / raw)
To: Dave Stevenson, Laurent Pinchart
Cc: Umang Jain, linux-media, kernel-list, linux-kernel,
linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> Hi Laurent, Umang, and Stefan.
>
> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
> <laurent.pinchart@ideasonboard.com> wrote:
>> Hi Umang,
>>
>> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
>>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
>>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
>>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
>>>>>>>>> will remain the primary area of this series.
>>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>>>>> comes out of staging before adding more features. As Greg said
>>>>>>>> some time ago staging is not a place to "dump code and run away".
>>>>>>>> These new files are in the same bad shape as the rest of vc04
>>>>>>>> before the clean-up here in staging started.
>>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
>>>>>> no problem.
>>>>>>
>>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>>>>> upstream, I would be happy to help here! Though I must mention that
>>>>>>> I still have limited visibility so my aim would be to chart out a
>>>>>>> plan of things needed to be done to get vc04_services out of staging!
>>>>>> The vchiq driver is in staging since 2016, so every step forwards is
>>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>>>>
>>>>>> For me the most important, but not to tricky steps to get vchiq out
>>>>>> of staging would be:
>>>>>>
>>>>>> * Cleanup logging mechanism
>>>>>>
>>>>>> * Get rid of custom function return values
>>>>>>
>>>>>> There was already an attempt for this [1]
>>>>>>
>>>>>> * Get rid of all non essential global structures and create a proper per
>>>>>> device structure
>>>>>>
>>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>>>>> staging. It just binds more resources on a new feature.
>>>>> bcm2835-camera is the legacy camera stack which probably need to be
>>>>> dropped from hereon...
>>>> I don't not know if there any users left, so i would be careful here.
>>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
>>>> dumb question but i'm not expert here.
>>> I am careful too here and probably need Input from RaspberryPi in order
>>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
>>> going out of staging - it'll either sit here (or probably dropped) as
>>> statied from [1]
>>>
>>> ```
>>> + * There are two camera drivers in the kernel for BCM283x - this one
>>> + * and bcm2835-camera (currently in staging).
>>> ```
>>>
>>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
>>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
>>> mentioned in my cover the testing of bcm2835-isp happened on top of
>>> unicam patches.
>> To be accurate, the bcm2835-camera driver supports the VC4
>> firmware-based camera stack. In that setup, the camera sensors (OV5647
>> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
>> firmware, which provides a high-level interface towards the kernel. This
>> architecture has been replaced by Linux-side control of the camera
>> sensors (through existing drivers in drivers/media/i2c/), Unicam
>> (through the driver from [1]) and ISP (through this driver). Moving
>> control to the Linux side requires complex processing in userspace,
>> handled by libcamera.
>>
>> bcm2835-camera is thus replaced by multiple drivers combined with
>> libcamera, and that is the camera stack that is shipped by Raspberry Pi
>> these days. While this may affect some userspace use cases), we will not
>> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
>> is planning to do so either. I don't mind much if the driver stays in
>> staging for some more time, but I'd rather drop it if possible.
Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
handle the old camera (OV5647)?
> It would be reasonable to drop it at the point that Libcamera can work
> to a similar level with at least the following list of applications:
> - FFmpeg
> - Gstreamer
> - Chromium
> - Firefox
> - Motion
> And that still leaves a huge number of existing V4L2 apps out in the cold.
>
> Do you wish to make any predictions as to when that would be
> achievable? Or even when a v1.0 release of libcamera is going to
> happen?
> Dropping anything prior to those points would be rather premature in my book.
>
>
> The TODOs on bcm2835-camera are:
> 1) Zero copy. That comes almost for free as bcm2835-isp already does
> this, but it does rely on vcsm-cma.
> The main reason I haven't pushed it is that it then requires
> reasonable amounts of CMA heap for all the buffers, which until
> recently haven't been present in the default configurations. With the
> vc4 DRM driver now being default (at least for the vendor kernel) and
> also requiring CMA, making the change makes more sense.
> AFAIK there is no easy way to have one driver choosing between using
> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> wrong.
> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> so perhaps we don't get very far.
CMA configuration should actually happen in device tree or kernel
cmdline. The bcm2835_defconfig is limited to make it work even with the
original Pi.
>
> 2) This isn't workable within the current V4L2 frameworks. The
> multi-planar V4L2 pixel formats are currently allocated as independent
> buffers for each plane, whereas the firmware needs a single buffer
> with (currently) specific offsets for the chroma planes. The
> V4L2/videobuf2 core changes required to implement that are going to be
> significant, and have minimal gain.
> The specific stride handling is already dealt with (set bytesperline
> appropriately), it's the padding of the height to a multiple of 16
> before the chroma planes on YUV420 and NV12 formats that require the
> firmware to do a small amount of repacking. The performance hit is
> actually minimal anyway.
>
> If bcm2835-camera is the only thing holding back vc04_services, then I
> can have a look at it.
No, it's the vchiq interface which needs the work.
Thanks Stefan
>
> Dave
>
>>> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
>>>
>>>>>>> I see two TODO files in vc04_services:
>>>>>>> ./bcm2835-camera/TODO
>>>>>>> ./interface/TODO
>>>>>>>
>>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>>>>>> itself. So that's address in the series. The other remaining one -
>>>>>>> I will need to take a deeper look before commenting on it.
>>>>>>>
>>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
>>>>>>> a cursory reading of them suggests that these apply to *all*
>>>>>>> vc04_services components? Am I right?
>>>>>> Actually these applies just for the interfaces directory. Some of
>>>>>> them could apply to the services, but this is no priority.
>>>>> By no priority, you mean this doesn't affect the criteria required to
>>>>> ful-fill to get these out of staging?
>>>> Correct
>>>>
>>>>>>> Are these are the specific bits of cleanup you are referring to in
>>>>>>> your comment?
>>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
>>>>>> that i don't remember them all. The first that come to my mind was
>>>>>> those fancy comment sections which is not kernel coding style. It
>>>>>> has been removed.
>>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
>>>>> out-of-tree still so...). I mean what are the specific bits / points
>>>>> that needs to be addressed to get vc04_services out of the staging.
>>>> These were the points which i mentioned in my last email. They came
>>>> from interface/TODO.
>>>>
>>>>> You have mentioned it above now, so I'll follow up on those.
>>>> That would be great :)
>>>>
>>>>> The many vchiq changes you referred to above comment (that you don't
>>>>> remember) are from [1] as well or some other series ?
>>>> Sorry, for the confusing. The many changes i refer were the dozens of
>>>> clean up patches for vc04_interfaces in mainline staging since the
>>>> last years. [1] was just a single patch which has been accepted yet.
>>> Ah I see. There are many others that I've to dig out then. Thanks for
>>> clarifying!
>>>
>>>>>> [1] -
>>>>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>>>>>>
>>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>>>>>
>>>>>>>> Just my two cents
>>>>>>>> Stefan
>> --
>> Regards,
>>
>> Laurent Pinchart
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 12:35 ` Stefan Wahren
@ 2022-12-02 13:25 ` Peter Robinson
-1 siblings, 0 replies; 95+ messages in thread
From: Peter Robinson @ 2022-12-02 13:25 UTC (permalink / raw)
To: Stefan Wahren
Cc: Laurent Pinchart, Dave Stevenson, Umang Jain, linux-media,
kernel-list, linux-kernel, linux-rpi-kernel, linux-arm-kernel,
linux-staging, Broadcom internal kernel review list,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham,
Ivan T. Ivanov
Hi Folks,
> > On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> >> Dropping anything prior to those points would be rather premature in my book.
> > Something I forgot to mention is that there should be no issue at all
> > keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> > kernel for a longer period of time. It's in upstream that I don't think
> > it should be destaged, as it's already considered legacy and should be
> > phased out. Do you know if there are users of that driver with a
> > mainline kernel ?
> >
> In Fedora there seems to be no official support [1], but openSuSE seems
> to mention it [2].
It's enabled in Fedora but I've never had a huge amount of luck with
it, I would be OK with the firmware driver to disappear when ever.
From our point of view I'd sooner we can just move to the unicam in
kernel one sooner rather than later even swapping one for the other in
staging to get the libcamera one in and improve it in kernel via the
staging process, I feel that would be better use of the very few
cycles I have for RPi bits.
Peter
> [1] -
> https://docs.fedoraproject.org/en-US/quick-docs/raspberry-pi/#_does_the_add_on_camera_work
>
> [2] - https://en.opensuse.org/HCL:Raspberry_Pi_Camera_Modules
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 13:25 ` Peter Robinson
0 siblings, 0 replies; 95+ messages in thread
From: Peter Robinson @ 2022-12-02 13:25 UTC (permalink / raw)
To: Stefan Wahren
Cc: Laurent Pinchart, Dave Stevenson, Umang Jain, linux-media,
kernel-list, linux-kernel, linux-rpi-kernel, linux-arm-kernel,
linux-staging, Broadcom internal kernel review list,
Florian Fainelli, Naushir Patuck, David Plowman, Kieran Bingham,
Ivan T. Ivanov
Hi Folks,
> > On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> >> Dropping anything prior to those points would be rather premature in my book.
> > Something I forgot to mention is that there should be no issue at all
> > keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> > kernel for a longer period of time. It's in upstream that I don't think
> > it should be destaged, as it's already considered legacy and should be
> > phased out. Do you know if there are users of that driver with a
> > mainline kernel ?
> >
> In Fedora there seems to be no official support [1], but openSuSE seems
> to mention it [2].
It's enabled in Fedora but I've never had a huge amount of luck with
it, I would be OK with the firmware driver to disappear when ever.
From our point of view I'd sooner we can just move to the unicam in
kernel one sooner rather than later even swapping one for the other in
staging to get the libcamera one in and improve it in kernel via the
staging process, I feel that would be better use of the very few
cycles I have for RPi bits.
Peter
> [1] -
> https://docs.fedoraproject.org/en-US/quick-docs/raspberry-pi/#_does_the_add_on_camera_work
>
> [2] - https://en.opensuse.org/HCL:Raspberry_Pi_Camera_Modules
>
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 12:38 ` Dave Stevenson
@ 2022-12-02 13:29 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-12-02 13:29 UTC (permalink / raw)
To: Dave Stevenson
Cc: Umang Jain, Stefan Wahren, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
On Fri, Dec 02, 2022 at 12:38:09PM +0000, Dave Stevenson wrote:
> On Fri, 2 Dec 2022 at 12:10, Laurent Pinchart wrote:
> > On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> > > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > > > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > > > >>>>>> will remain the primary area of this series.
> > > > > >>>>>
> > > > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > > > >>>>> comes out of staging before adding more features. As Greg said
> > > > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > > > >>>>> before the clean-up here in staging started.
> > > > > >>>>
> > > > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > > > >>>
> > > > > >>> no problem.
> > > > > >>>
> > > > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > > > >>>> I still have limited visibility so my aim would be to chart out a
> > > > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > > > >>>
> > > > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > > > >>>
> > > > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > > > >>> of staging would be:
> > > > > >>>
> > > > > >>> * Cleanup logging mechanism
> > > > > >>>
> > > > > >>> * Get rid of custom function return values
> > > > > >>>
> > > > > >>> There was already an attempt for this [1]
> > > > > >>>
> > > > > >>> * Get rid of all non essential global structures and create a proper per
> > > > > >>> device structure
> > > > > >>>
> > > > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > > > >>>>> staging. It just binds more resources on a new feature.
> > > > > >>
> > > > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > > > >> dropped from hereon...
> > > > > >
> > > > > > I don't not know if there any users left, so i would be careful here.
> > > > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > > > dumb question but i'm not expert here.
> > > > >
> > > > > I am careful too here and probably need Input from RaspberryPi in order
> > > > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > > > going out of staging - it'll either sit here (or probably dropped) as
> > > > > statied from [1]
> > > > >
> > > > > ```
> > > > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > > > + * and bcm2835-camera (currently in staging).
> > > > > ```
> > > > >
> > > > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > > > unicam patches.
> > > >
> > > > To be accurate, the bcm2835-camera driver supports the VC4
> > > > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > > > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > > > firmware, which provides a high-level interface towards the kernel. This
> > > > architecture has been replaced by Linux-side control of the camera
> > > > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > > > (through the driver from [1]) and ISP (through this driver). Moving
> > > > control to the Linux side requires complex processing in userspace,
> > > > handled by libcamera.
> > > >
> > > > bcm2835-camera is thus replaced by multiple drivers combined with
> > > > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > > > these days. While this may affect some userspace use cases), we will not
> > > > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > > > is planning to do so either. I don't mind much if the driver stays in
> > > > staging for some more time, but I'd rather drop it if possible.
> > >
> > > It would be reasonable to drop it at the point that Libcamera can work
> > > to a similar level with at least the following list of applications:
> > > - FFmpeg
> > > - Gstreamer
> > > - Chromium
> > > - Firefox
> > > - Motion
> > > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > That's exactly the kind of input we were looking for, thanks.
>
> Please note that those are only the 5 major ones that came immediately
> to mind as having significant numbers of users. There are obviously
> many more V4L2 apps around which currently "just work" if they use the
> API correctly.
Sure, I'll keep that in mind.
> > GStreamer is already addressed.
> >
> > Chromium and Firefox will go through PipeWire. There is a working
> > implementation in libwebrtc, Kieran may be able to comment on the
> > upstreaming state. It will take some time for distributions to switch,
> > and I can't predict the time line, but that seems to clearly be the
> > direction the Linux desktop is taking.
> >
> > I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
> > sense to add native libcamera support to FFmpeg, even if PipeWire
> > support could also make sense. It could also make sense to expose in the
> > libcamera V4L2 compat layer the libv4l2 API functions, as that would
> > allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
> > instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
> > an immediate solution to the problem.
>
> Native would be nice in my book, but isn't always as straightforward.
I agree, native would be good.
> > Same thing for motion, except it has no libv4l2 support. The V4L2 compat
> > layer could still be used with LD_PRELOAD, but that's not a great
> > solution. Native libcamera support would make more sense (or possibly
> > even GStreamer support, I don't know if upstream would accept that).
>
> The main bit missing with the compat layer is controls. AIUI fixing
> that isn't a priority.
Good point. That's something we should fix. Not a high priority indeed,
but still good to have (and it shouldn't take too long).
> Things like Motion being used for security purposes need to be able to
> set properties like exposure modes in order to achieve usable images
> under changing lighting conditions.
>
> > In a side note, how do the above applications work today on Raspberry Pi
> > platforms that use a sensor not supported by the legacy camera stack ?
>
> They don't for raw sensors, but do work with USB webcams.
> Raspberry Pi have sold camera modules since 2013 (OV5647), with v2
> (IMX219) coming along in 2016, and HQ (IMX477) in 2020. Linux has only
> had frameworks to sensibly support raw image sensors since libcamera
> came on the scene around 2019.
Do I understand correctly that the need to keep bcm2835-camera working
is for support of the v1 and v2 camera modules only ? Or are there other
use cases ?
> > > Do you wish to make any predictions as to when that would be
> > > achievable? Or even when a v1.0 release of libcamera is going to
> > > happen?
> >
> > Now that we have started tagging releases, we've also decided to publish
> > a roadmap with the development still needed to stabilize the API. We'll
> > likely start working on it this month.
> >
> > > Dropping anything prior to those points would be rather premature in my book.
> >
> > Something I forgot to mention is that there should be no issue at all
> > keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> > kernel for a longer period of time. It's in upstream that I don't think
> > it should be destaged, as it's already considered legacy and should be
> > phased out. Do you know if there are users of that driver with a
> > mainline kernel ?
>
> There are a number of distros that use mainline kernels rather than
> our vendor kernel. I don't follow distros in detail, but I believe
> both Gentoo and Opensuse fall into that category.
> Any users of the camera module under those will be using the mainline
> bcm2835-camera module. I have no stats for who uses those on a Pi.
OK.
> > > The TODOs on bcm2835-camera are:
> > > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > > this, but it does rely on vcsm-cma.
> > > The main reason I haven't pushed it is that it then requires
> > > reasonable amounts of CMA heap for all the buffers, which until
> > > recently haven't been present in the default configurations. With the
> > > vc4 DRM driver now being default (at least for the vendor kernel) and
> > > also requiring CMA, making the change makes more sense.
> > > AFAIK there is no easy way to have one driver choosing between using
> > > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > > wrong.
> >
> > Some drivers use a module parameter for that, but that's not great.
>
> I had thought of a module parameter, but it becomes yet another
> configuration thing to get incorrectly set.
> Perhaps Kconfig based on the setting for CMA_SIZE_MBYTES, but that can
> be overridden.
The trouble with vb2-vmalloc is that you'll end up copying data, which I
expect to cause various performance issues that may generate bug reports
difficult to address. Increasing the CMA size would be better.
> > > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > > so perhaps we don't get very far.
> > >
> > > 2) This isn't workable within the current V4L2 frameworks. The
> > > multi-planar V4L2 pixel formats are currently allocated as independent
> > > buffers for each plane, whereas the firmware needs a single buffer
> > > with (currently) specific offsets for the chroma planes. The
> > > V4L2/videobuf2 core changes required to implement that are going to be
> > > significant, and have minimal gain.
> > > The specific stride handling is already dealt with (set bytesperline
> > > appropriately), it's the padding of the height to a multiple of 16
> > > before the chroma planes on YUV420 and NV12 formats that require the
> > > firmware to do a small amount of repacking. The performance hit is
> > > actually minimal anyway.
> > >
> > > If bcm2835-camera is the only thing holding back vc04_services, then I
> > > can have a look at it.
> >
> > I'll let Umang comment on whether it's holding vc04_services back, but
> > my understanding it that we could destage vc04_services while keeping
> > bcm2835-camera in staging for the time being. If anyone disagrees with
> > that, please let me know.
> >
> > > > > [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> > > > >
> > > > > >>>>
> > > > > >>>> I see two TODO files in vc04_services:
> > > > > >>>> ./bcm2835-camera/TODO
> > > > > >>>> ./interface/TODO
> > > > > >>>>
> > > > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > > > >>>> itself. So that's address in the series. The other remaining one -
> > > > > >>>> I will need to take a deeper look before commenting on it.
> > > > > >>>>
> > > > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > > > >>>> a cursory reading of them suggests that these apply to *all*
> > > > > >>>> vc04_services components? Am I right?
> > > > > >>>
> > > > > >>> Actually these applies just for the interfaces directory. Some of
> > > > > >>> them could apply to the services, but this is no priority.
> > > > > >>
> > > > > >> By no priority, you mean this doesn't affect the criteria required to
> > > > > >> ful-fill to get these out of staging?
> > > > > >
> > > > > > Correct
> > > > > >
> > > > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > > > >>>> your comment?
> > > > > >>>
> > > > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > > > >>> that i don't remember them all. The first that come to my mind was
> > > > > >>> those fancy comment sections which is not kernel coding style. It
> > > > > >>> has been removed.
> > > > > >>
> > > > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > > > >> that needs to be addressed to get vc04_services out of the staging.
> > > > > >
> > > > > > These were the points which i mentioned in my last email. They came
> > > > > > from interface/TODO.
> > > > > >
> > > > > >> You have mentioned it above now, so I'll follow up on those.
> > > > > >
> > > > > > That would be great :)
> > > > > >
> > > > > >> The many vchiq changes you referred to above comment (that you don't
> > > > > >> remember) are from [1] as well or some other series ?
> > > > > >
> > > > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > > > last years. [1] was just a single patch which has been accepted yet.
> > > > >
> > > > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > > > clarifying!
> > > > >
> > > > > >>> [1] -
> > > > > >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> > > > > >>>
> > > > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > > > >>>>>
> > > > > >>>>> Just my two cents
> > > > > >>>>> Stefan
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 13:29 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-12-02 13:29 UTC (permalink / raw)
To: Dave Stevenson
Cc: Umang Jain, Stefan Wahren, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
On Fri, Dec 02, 2022 at 12:38:09PM +0000, Dave Stevenson wrote:
> On Fri, 2 Dec 2022 at 12:10, Laurent Pinchart wrote:
> > On Fri, Dec 02, 2022 at 11:23:29AM +0000, Dave Stevenson wrote:
> > > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> > > > On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> > > > > On 12/2/22 6:45 AM, Stefan Wahren wrote:
> > > > > > Am 30.11.22 um 11:58 schrieb Umang Jain:
> > > > > >> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> > > > > >>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> > > > > >>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> > > > > >>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> > > > > >>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> > > > > >>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> > > > > >>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> > > > > >>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> > > > > >>>>>> with multistream support to get into mainline. Hence only the ISP driver
> > > > > >>>>>> will remain the primary area of this series.
> > > > > >>>>>
> > > > > >>>>> thanks for working on this. But honestly i would prefer that vchiq
> > > > > >>>>> comes out of staging before adding more features. As Greg said
> > > > > >>>>> some time ago staging is not a place to "dump code and run away".
> > > > > >>>>> These new files are in the same bad shape as the rest of vc04
> > > > > >>>>> before the clean-up here in staging started.
> > > > > >>>>
> > > > > >>>> Certainly, I am not here to do that - but I am still learning the ropes.
> > > > > >>>
> > > > > >>> no problem.
> > > > > >>>
> > > > > >>>> If the staging issue is becoming a blocker for bcm2835-isp going
> > > > > >>>> upstream, I would be happy to help here! Though I must mention that
> > > > > >>>> I still have limited visibility so my aim would be to chart out a
> > > > > >>>> plan of things needed to be done to get vc04_services out of staging!
> > > > > >>>
> > > > > >>> The vchiq driver is in staging since 2016, so every step forwards is
> > > > > >>> good. Unfortunately all of the low hanging fruits has been gathered.
> > > > > >>>
> > > > > >>> For me the most important, but not to tricky steps to get vchiq out
> > > > > >>> of staging would be:
> > > > > >>>
> > > > > >>> * Cleanup logging mechanism
> > > > > >>>
> > > > > >>> * Get rid of custom function return values
> > > > > >>>
> > > > > >>> There was already an attempt for this [1]
> > > > > >>>
> > > > > >>> * Get rid of all non essential global structures and create a proper per
> > > > > >>> device structure
> > > > > >>>
> > > > > >>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> > > > > >>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> > > > > >>>>> staging. It just binds more resources on a new feature.
> > > > > >>
> > > > > >> bcm2835-camera is the legacy camera stack which probably need to be
> > > > > >> dropped from hereon...
> > > > > >
> > > > > > I don't not know if there any users left, so i would be careful here.
> > > > > > Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> > > > > > dumb question but i'm not expert here.
> > > > >
> > > > > I am careful too here and probably need Input from RaspberryPi in order
> > > > > to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> > > > > going out of staging - it'll either sit here (or probably dropped) as
> > > > > statied from [1]
> > > > >
> > > > > ```
> > > > > + * There are two camera drivers in the kernel for BCM283x - this one
> > > > > + * and bcm2835-camera (currently in staging).
> > > > > ```
> > > > >
> > > > > The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> > > > > (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> > > > > mentioned in my cover the testing of bcm2835-isp happened on top of
> > > > > unicam patches.
> > > >
> > > > To be accurate, the bcm2835-camera driver supports the VC4
> > > > firmware-based camera stack. In that setup, the camera sensors (OV5647
> > > > or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> > > > firmware, which provides a high-level interface towards the kernel. This
> > > > architecture has been replaced by Linux-side control of the camera
> > > > sensors (through existing drivers in drivers/media/i2c/), Unicam
> > > > (through the driver from [1]) and ISP (through this driver). Moving
> > > > control to the Linux side requires complex processing in userspace,
> > > > handled by libcamera.
> > > >
> > > > bcm2835-camera is thus replaced by multiple drivers combined with
> > > > libcamera, and that is the camera stack that is shipped by Raspberry Pi
> > > > these days. While this may affect some userspace use cases), we will not
> > > > work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> > > > is planning to do so either. I don't mind much if the driver stays in
> > > > staging for some more time, but I'd rather drop it if possible.
> > >
> > > It would be reasonable to drop it at the point that Libcamera can work
> > > to a similar level with at least the following list of applications:
> > > - FFmpeg
> > > - Gstreamer
> > > - Chromium
> > > - Firefox
> > > - Motion
> > > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > That's exactly the kind of input we were looking for, thanks.
>
> Please note that those are only the 5 major ones that came immediately
> to mind as having significant numbers of users. There are obviously
> many more V4L2 apps around which currently "just work" if they use the
> API correctly.
Sure, I'll keep that in mind.
> > GStreamer is already addressed.
> >
> > Chromium and Firefox will go through PipeWire. There is a working
> > implementation in libwebrtc, Kieran may be able to comment on the
> > upstreaming state. It will take some time for distributions to switch,
> > and I can't predict the time line, but that seems to clearly be the
> > direction the Linux desktop is taking.
> >
> > I haven't looked at FFmpeg yet (maybe someone has ?). It probably makes
> > sense to add native libcamera support to FFmpeg, even if PipeWire
> > support could also make sense. It could also make sense to expose in the
> > libcamera V4L2 compat layer the libv4l2 API functions, as that would
> > allow linking FFmpeg (when compiled with CONFIG_LIBV4L2) to libcamera
> > instead of libv4l2, but Debian doesn't set CONFIG_LIBV4L2, so this isn't
> > an immediate solution to the problem.
>
> Native would be nice in my book, but isn't always as straightforward.
I agree, native would be good.
> > Same thing for motion, except it has no libv4l2 support. The V4L2 compat
> > layer could still be used with LD_PRELOAD, but that's not a great
> > solution. Native libcamera support would make more sense (or possibly
> > even GStreamer support, I don't know if upstream would accept that).
>
> The main bit missing with the compat layer is controls. AIUI fixing
> that isn't a priority.
Good point. That's something we should fix. Not a high priority indeed,
but still good to have (and it shouldn't take too long).
> Things like Motion being used for security purposes need to be able to
> set properties like exposure modes in order to achieve usable images
> under changing lighting conditions.
>
> > In a side note, how do the above applications work today on Raspberry Pi
> > platforms that use a sensor not supported by the legacy camera stack ?
>
> They don't for raw sensors, but do work with USB webcams.
> Raspberry Pi have sold camera modules since 2013 (OV5647), with v2
> (IMX219) coming along in 2016, and HQ (IMX477) in 2020. Linux has only
> had frameworks to sensibly support raw image sensors since libcamera
> came on the scene around 2019.
Do I understand correctly that the need to keep bcm2835-camera working
is for support of the v1 and v2 camera modules only ? Or are there other
use cases ?
> > > Do you wish to make any predictions as to when that would be
> > > achievable? Or even when a v1.0 release of libcamera is going to
> > > happen?
> >
> > Now that we have started tagging releases, we've also decided to publish
> > a roadmap with the development still needed to stabilize the API. We'll
> > likely start working on it this month.
> >
> > > Dropping anything prior to those points would be rather premature in my book.
> >
> > Something I forgot to mention is that there should be no issue at all
> > keeping bcm2835-camera fully supported in the Raspberry Pi downstream
> > kernel for a longer period of time. It's in upstream that I don't think
> > it should be destaged, as it's already considered legacy and should be
> > phased out. Do you know if there are users of that driver with a
> > mainline kernel ?
>
> There are a number of distros that use mainline kernels rather than
> our vendor kernel. I don't follow distros in detail, but I believe
> both Gentoo and Opensuse fall into that category.
> Any users of the camera module under those will be using the mainline
> bcm2835-camera module. I have no stats for who uses those on a Pi.
OK.
> > > The TODOs on bcm2835-camera are:
> > > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > > this, but it does rely on vcsm-cma.
> > > The main reason I haven't pushed it is that it then requires
> > > reasonable amounts of CMA heap for all the buffers, which until
> > > recently haven't been present in the default configurations. With the
> > > vc4 DRM driver now being default (at least for the vendor kernel) and
> > > also requiring CMA, making the change makes more sense.
> > > AFAIK there is no easy way to have one driver choosing between using
> > > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > > wrong.
> >
> > Some drivers use a module parameter for that, but that's not great.
>
> I had thought of a module parameter, but it becomes yet another
> configuration thing to get incorrectly set.
> Perhaps Kconfig based on the setting for CMA_SIZE_MBYTES, but that can
> be overridden.
The trouble with vb2-vmalloc is that you'll end up copying data, which I
expect to cause various performance issues that may generate bug reports
difficult to address. Increasing the CMA size would be better.
> > > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > > so perhaps we don't get very far.
> > >
> > > 2) This isn't workable within the current V4L2 frameworks. The
> > > multi-planar V4L2 pixel formats are currently allocated as independent
> > > buffers for each plane, whereas the firmware needs a single buffer
> > > with (currently) specific offsets for the chroma planes. The
> > > V4L2/videobuf2 core changes required to implement that are going to be
> > > significant, and have minimal gain.
> > > The specific stride handling is already dealt with (set bytesperline
> > > appropriately), it's the padding of the height to a multiple of 16
> > > before the chroma planes on YUV420 and NV12 formats that require the
> > > firmware to do a small amount of repacking. The performance hit is
> > > actually minimal anyway.
> > >
> > > If bcm2835-camera is the only thing holding back vc04_services, then I
> > > can have a look at it.
> >
> > I'll let Umang comment on whether it's holding vc04_services back, but
> > my understanding it that we could destage vc04_services while keeping
> > bcm2835-camera in staging for the time being. If anyone disagrees with
> > that, please let me know.
> >
> > > > > [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> > > > >
> > > > > >>>>
> > > > > >>>> I see two TODO files in vc04_services:
> > > > > >>>> ./bcm2835-camera/TODO
> > > > > >>>> ./interface/TODO
> > > > > >>>>
> > > > > >>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> > > > > >>>> itself. So that's address in the series. The other remaining one -
> > > > > >>>> I will need to take a deeper look before commenting on it.
> > > > > >>>>
> > > > > >>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> > > > > >>>> a cursory reading of them suggests that these apply to *all*
> > > > > >>>> vc04_services components? Am I right?
> > > > > >>>
> > > > > >>> Actually these applies just for the interfaces directory. Some of
> > > > > >>> them could apply to the services, but this is no priority.
> > > > > >>
> > > > > >> By no priority, you mean this doesn't affect the criteria required to
> > > > > >> ful-fill to get these out of staging?
> > > > > >
> > > > > > Correct
> > > > > >
> > > > > >>>> Are these are the specific bits of cleanup you are referring to in
> > > > > >>>> your comment?
> > > > > >>>
> > > > > >>> You mean about bcm2835-isp? There were too many changes to vchiq
> > > > > >>> that i don't remember them all. The first that come to my mind was
> > > > > >>> those fancy comment sections which is not kernel coding style. It
> > > > > >>> has been removed.
> > > > > >>
> > > > > >> No, I don't mean the bcm2835-isp changes (those are upcoming /
> > > > > >> out-of-tree still so...). I mean what are the specific bits / points
> > > > > >> that needs to be addressed to get vc04_services out of the staging.
> > > > > >
> > > > > > These were the points which i mentioned in my last email. They came
> > > > > > from interface/TODO.
> > > > > >
> > > > > >> You have mentioned it above now, so I'll follow up on those.
> > > > > >
> > > > > > That would be great :)
> > > > > >
> > > > > >> The many vchiq changes you referred to above comment (that you don't
> > > > > >> remember) are from [1] as well or some other series ?
> > > > > >
> > > > > > Sorry, for the confusing. The many changes i refer were the dozens of
> > > > > > clean up patches for vc04_interfaces in mainline staging since the
> > > > > > last years. [1] was just a single patch which has been accepted yet.
> > > > >
> > > > > Ah I see. There are many others that I've to dig out then. Thanks for
> > > > > clarifying!
> > > > >
> > > > > >>> [1] -
> > > > > >>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> > > > > >>>
> > > > > >>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> > > > > >>>>>
> > > > > >>>>> Just my two cents
> > > > > >>>>> Stefan
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 12:41 ` Stefan Wahren
@ 2022-12-02 13:32 ` Laurent Pinchart
-1 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-12-02 13:32 UTC (permalink / raw)
To: Stefan Wahren
Cc: Dave Stevenson, Umang Jain, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Stefan,
On Fri, Dec 02, 2022 at 01:41:53PM +0100, Stefan Wahren wrote:
> Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> >> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> >>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> >>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
> >>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>>>>> will remain the primary area of this series.
> >>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>>>>> comes out of staging before adding more features. As Greg said
> >>>>>>>> some time ago staging is not a place to "dump code and run away".
> >>>>>>>> These new files are in the same bad shape as the rest of vc04
> >>>>>>>> before the clean-up here in staging started.
> >>>>>>>
> >>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>>>>
> >>>>>> no problem.
> >>>>>>
> >>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>>>>> upstream, I would be happy to help here! Though I must mention that
> >>>>>>> I still have limited visibility so my aim would be to chart out a
> >>>>>>> plan of things needed to be done to get vc04_services out of staging!
> >>>>>>
> >>>>>> The vchiq driver is in staging since 2016, so every step forwards is
> >>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>>>>
> >>>>>> For me the most important, but not to tricky steps to get vchiq out
> >>>>>> of staging would be:
> >>>>>>
> >>>>>> * Cleanup logging mechanism
> >>>>>>
> >>>>>> * Get rid of custom function return values
> >>>>>>
> >>>>>> There was already an attempt for this [1]
> >>>>>>
> >>>>>> * Get rid of all non essential global structures and create a proper per
> >>>>>> device structure
> >>>>>>
> >>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>>>>> staging. It just binds more resources on a new feature.
> >>>>>
> >>>>> bcm2835-camera is the legacy camera stack which probably need to be
> >>>>> dropped from hereon...
> >>>>
> >>>> I don't not know if there any users left, so i would be careful here.
> >>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> >>>> dumb question but i'm not expert here.
> >>>
> >>> I am careful too here and probably need Input from RaspberryPi in order
> >>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> >>> going out of staging - it'll either sit here (or probably dropped) as
> >>> statied from [1]
> >>>
> >>> ```
> >>> + * There are two camera drivers in the kernel for BCM283x - this one
> >>> + * and bcm2835-camera (currently in staging).
> >>> ```
> >>>
> >>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> >>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> >>> mentioned in my cover the testing of bcm2835-isp happened on top of
> >>> unicam patches.
> >>
> >> To be accurate, the bcm2835-camera driver supports the VC4
> >> firmware-based camera stack. In that setup, the camera sensors (OV5647
> >> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> >> firmware, which provides a high-level interface towards the kernel. This
> >> architecture has been replaced by Linux-side control of the camera
> >> sensors (through existing drivers in drivers/media/i2c/), Unicam
> >> (through the driver from [1]) and ISP (through this driver). Moving
> >> control to the Linux side requires complex processing in userspace,
> >> handled by libcamera.
> >>
> >> bcm2835-camera is thus replaced by multiple drivers combined with
> >> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> >> these days. While this may affect some userspace use cases), we will not
> >> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> >> is planning to do so either. I don't mind much if the driver stays in
> >> staging for some more time, but I'd rather drop it if possible.
>
> Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
> handle the old camera (OV5647)?
Yes, but only for applications using libcamera, directly or indirectly
(through GStreamer for instance, or the V4L2 LD_PRELOAD-ed compatibility
later), not natively through V4L2 only.
> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
> > Dropping anything prior to those points would be rather premature in my book.
> >
> >
> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
>
> CMA configuration should actually happen in device tree or kernel
> cmdline. The bcm2835_defconfig is limited to make it work even with the
> original Pi.
>
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> No, it's the vchiq interface which needs the work.
>
> >>> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> >>>
> >>>>>>> I see two TODO files in vc04_services:
> >>>>>>> ./bcm2835-camera/TODO
> >>>>>>> ./interface/TODO
> >>>>>>>
> >>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>>>>> itself. So that's address in the series. The other remaining one -
> >>>>>>> I will need to take a deeper look before commenting on it.
> >>>>>>>
> >>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>>>>> a cursory reading of them suggests that these apply to *all*
> >>>>>>> vc04_services components? Am I right?
> >>>>>>
> >>>>>> Actually these applies just for the interfaces directory. Some of
> >>>>>> them could apply to the services, but this is no priority.
> >>>>>
> >>>>> By no priority, you mean this doesn't affect the criteria required to
> >>>>> ful-fill to get these out of staging?
> >>>>
> >>>> Correct
> >>>>
> >>>>>>> Are these are the specific bits of cleanup you are referring to in
> >>>>>>> your comment?
> >>>>>>
> >>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>>>>> that i don't remember them all. The first that come to my mind was
> >>>>>> those fancy comment sections which is not kernel coding style. It
> >>>>>> has been removed.
> >>>>>
> >>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >>>>> out-of-tree still so...). I mean what are the specific bits / points
> >>>>> that needs to be addressed to get vc04_services out of the staging.
> >>>>
> >>>> These were the points which i mentioned in my last email. They came
> >>>> from interface/TODO.
> >>>>
> >>>>> You have mentioned it above now, so I'll follow up on those.
> >>>>
> >>>> That would be great :)
> >>>>
> >>>>> The many vchiq changes you referred to above comment (that you don't
> >>>>> remember) are from [1] as well or some other series ?
> >>>> Sorry, for the confusing. The many changes i refer were the dozens of
> >>>> clean up patches for vc04_interfaces in mainline staging since the
> >>>> last years. [1] was just a single patch which has been accepted yet.
> >>>
> >>> Ah I see. There are many others that I've to dig out then. Thanks for
> >>> clarifying!
> >>>
> >>>>>> [1] -
> >>>>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> >>>>>>
> >>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>>>>
> >>>>>>>> Just my two cents
> >>>>>>>> Stefan
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 13:32 ` Laurent Pinchart
0 siblings, 0 replies; 95+ messages in thread
From: Laurent Pinchart @ 2022-12-02 13:32 UTC (permalink / raw)
To: Stefan Wahren
Cc: Dave Stevenson, Umang Jain, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Stefan,
On Fri, Dec 02, 2022 at 01:41:53PM +0100, Stefan Wahren wrote:
> Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart wrote:
> >> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> >>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> >>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
> >>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>>>>> will remain the primary area of this series.
> >>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>>>>> comes out of staging before adding more features. As Greg said
> >>>>>>>> some time ago staging is not a place to "dump code and run away".
> >>>>>>>> These new files are in the same bad shape as the rest of vc04
> >>>>>>>> before the clean-up here in staging started.
> >>>>>>>
> >>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>>>>
> >>>>>> no problem.
> >>>>>>
> >>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>>>>> upstream, I would be happy to help here! Though I must mention that
> >>>>>>> I still have limited visibility so my aim would be to chart out a
> >>>>>>> plan of things needed to be done to get vc04_services out of staging!
> >>>>>>
> >>>>>> The vchiq driver is in staging since 2016, so every step forwards is
> >>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>>>>
> >>>>>> For me the most important, but not to tricky steps to get vchiq out
> >>>>>> of staging would be:
> >>>>>>
> >>>>>> * Cleanup logging mechanism
> >>>>>>
> >>>>>> * Get rid of custom function return values
> >>>>>>
> >>>>>> There was already an attempt for this [1]
> >>>>>>
> >>>>>> * Get rid of all non essential global structures and create a proper per
> >>>>>> device structure
> >>>>>>
> >>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>>>>> staging. It just binds more resources on a new feature.
> >>>>>
> >>>>> bcm2835-camera is the legacy camera stack which probably need to be
> >>>>> dropped from hereon...
> >>>>
> >>>> I don't not know if there any users left, so i would be careful here.
> >>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> >>>> dumb question but i'm not expert here.
> >>>
> >>> I am careful too here and probably need Input from RaspberryPi in order
> >>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> >>> going out of staging - it'll either sit here (or probably dropped) as
> >>> statied from [1]
> >>>
> >>> ```
> >>> + * There are two camera drivers in the kernel for BCM283x - this one
> >>> + * and bcm2835-camera (currently in staging).
> >>> ```
> >>>
> >>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> >>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> >>> mentioned in my cover the testing of bcm2835-isp happened on top of
> >>> unicam patches.
> >>
> >> To be accurate, the bcm2835-camera driver supports the VC4
> >> firmware-based camera stack. In that setup, the camera sensors (OV5647
> >> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> >> firmware, which provides a high-level interface towards the kernel. This
> >> architecture has been replaced by Linux-side control of the camera
> >> sensors (through existing drivers in drivers/media/i2c/), Unicam
> >> (through the driver from [1]) and ISP (through this driver). Moving
> >> control to the Linux side requires complex processing in userspace,
> >> handled by libcamera.
> >>
> >> bcm2835-camera is thus replaced by multiple drivers combined with
> >> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> >> these days. While this may affect some userspace use cases), we will not
> >> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> >> is planning to do so either. I don't mind much if the driver stays in
> >> staging for some more time, but I'd rather drop it if possible.
>
> Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
> handle the old camera (OV5647)?
Yes, but only for applications using libcamera, directly or indirectly
(through GStreamer for instance, or the V4L2 LD_PRELOAD-ed compatibility
later), not natively through V4L2 only.
> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
> > Dropping anything prior to those points would be rather premature in my book.
> >
> >
> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
>
> CMA configuration should actually happen in device tree or kernel
> cmdline. The bcm2835_defconfig is limited to make it work even with the
> original Pi.
>
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> No, it's the vchiq interface which needs the work.
>
> >>> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> >>>
> >>>>>>> I see two TODO files in vc04_services:
> >>>>>>> ./bcm2835-camera/TODO
> >>>>>>> ./interface/TODO
> >>>>>>>
> >>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>>>>> itself. So that's address in the series. The other remaining one -
> >>>>>>> I will need to take a deeper look before commenting on it.
> >>>>>>>
> >>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>>>>> a cursory reading of them suggests that these apply to *all*
> >>>>>>> vc04_services components? Am I right?
> >>>>>>
> >>>>>> Actually these applies just for the interfaces directory. Some of
> >>>>>> them could apply to the services, but this is no priority.
> >>>>>
> >>>>> By no priority, you mean this doesn't affect the criteria required to
> >>>>> ful-fill to get these out of staging?
> >>>>
> >>>> Correct
> >>>>
> >>>>>>> Are these are the specific bits of cleanup you are referring to in
> >>>>>>> your comment?
> >>>>>>
> >>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>>>>> that i don't remember them all. The first that come to my mind was
> >>>>>> those fancy comment sections which is not kernel coding style. It
> >>>>>> has been removed.
> >>>>>
> >>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >>>>> out-of-tree still so...). I mean what are the specific bits / points
> >>>>> that needs to be addressed to get vc04_services out of the staging.
> >>>>
> >>>> These were the points which i mentioned in my last email. They came
> >>>> from interface/TODO.
> >>>>
> >>>>> You have mentioned it above now, so I'll follow up on those.
> >>>>
> >>>> That would be great :)
> >>>>
> >>>>> The many vchiq changes you referred to above comment (that you don't
> >>>>> remember) are from [1] as well or some other series ?
> >>>> Sorry, for the confusing. The many changes i refer were the dozens of
> >>>> clean up patches for vc04_interfaces in mainline staging since the
> >>>> last years. [1] was just a single patch which has been accepted yet.
> >>>
> >>> Ah I see. There are many others that I've to dig out then. Thanks for
> >>> clarifying!
> >>>
> >>>>>> [1] -
> >>>>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> >>>>>>
> >>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>>>>
> >>>>>>>> Just my two cents
> >>>>>>>> Stefan
--
Regards,
Laurent Pinchart
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 12:41 ` Stefan Wahren
@ 2022-12-02 13:42 ` Dave Stevenson
-1 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-12-02 13:42 UTC (permalink / raw)
To: Stefan Wahren
Cc: Laurent Pinchart, Umang Jain, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Stefan
On Fri, 2 Dec 2022 at 12:41, Stefan Wahren <stefan.wahren@i2se.com> wrote:
>
> Hi Dave,
>
> Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> > Hi Laurent, Umang, and Stefan.
> >
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
> > <laurent.pinchart@ideasonboard.com> wrote:
> >> Hi Umang,
> >>
> >> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> >>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> >>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
> >>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>>>>> will remain the primary area of this series.
> >>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>>>>> comes out of staging before adding more features. As Greg said
> >>>>>>>> some time ago staging is not a place to "dump code and run away".
> >>>>>>>> These new files are in the same bad shape as the rest of vc04
> >>>>>>>> before the clean-up here in staging started.
> >>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>>>> no problem.
> >>>>>>
> >>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>>>>> upstream, I would be happy to help here! Though I must mention that
> >>>>>>> I still have limited visibility so my aim would be to chart out a
> >>>>>>> plan of things needed to be done to get vc04_services out of staging!
> >>>>>> The vchiq driver is in staging since 2016, so every step forwards is
> >>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>>>>
> >>>>>> For me the most important, but not to tricky steps to get vchiq out
> >>>>>> of staging would be:
> >>>>>>
> >>>>>> * Cleanup logging mechanism
> >>>>>>
> >>>>>> * Get rid of custom function return values
> >>>>>>
> >>>>>> There was already an attempt for this [1]
> >>>>>>
> >>>>>> * Get rid of all non essential global structures and create a proper per
> >>>>>> device structure
> >>>>>>
> >>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>>>>> staging. It just binds more resources on a new feature.
> >>>>> bcm2835-camera is the legacy camera stack which probably need to be
> >>>>> dropped from hereon...
> >>>> I don't not know if there any users left, so i would be careful here.
> >>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> >>>> dumb question but i'm not expert here.
> >>> I am careful too here and probably need Input from RaspberryPi in order
> >>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> >>> going out of staging - it'll either sit here (or probably dropped) as
> >>> statied from [1]
> >>>
> >>> ```
> >>> + * There are two camera drivers in the kernel for BCM283x - this one
> >>> + * and bcm2835-camera (currently in staging).
> >>> ```
> >>>
> >>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> >>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> >>> mentioned in my cover the testing of bcm2835-isp happened on top of
> >>> unicam patches.
> >> To be accurate, the bcm2835-camera driver supports the VC4
> >> firmware-based camera stack. In that setup, the camera sensors (OV5647
> >> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> >> firmware, which provides a high-level interface towards the kernel. This
> >> architecture has been replaced by Linux-side control of the camera
> >> sensors (through existing drivers in drivers/media/i2c/), Unicam
> >> (through the driver from [1]) and ISP (through this driver). Moving
> >> control to the Linux side requires complex processing in userspace,
> >> handled by libcamera.
> >>
> >> bcm2835-camera is thus replaced by multiple drivers combined with
> >> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> >> these days. While this may affect some userspace use cases), we will not
> >> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> >> is planning to do so either. I don't mind much if the driver stays in
> >> staging for some more time, but I'd rather drop it if possible.
> Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
> handle the old camera (OV5647)?
There are kernel drivers in mainline for ov5647 (v1) and imx219 (v2)
cameras, with imx477 (HQ camera) due to be done. Connected to
bcm2835-unicam that gives you raw images from these sensors.
However you have to use libcamera to get useful images with these
drivers - raw Bayer isn't easy to consume.
Libcamera has support for all three (and more) sensors in the RPi IPA
(image processing algorithms), and will make use of bcm2835-isp to
process the images appropriately.
> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
> > Dropping anything prior to those points would be rather premature in my book.
> >
> >
> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
> CMA configuration should actually happen in device tree or kernel
> cmdline. The bcm2835_defconfig is limited to make it work even with the
> original Pi.
It's the original 256MB Pis that are the issue as memory is so tight.
Losing a bigger chunk to CMA is a problem there.
I see that bcm283x.dtsi sets it to 64MB, which is probably sufficient
for most bcm2835-camera use cases, but leaves me worried that you'll
steal it from other things.
> >
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> No, it's the vchiq interface which needs the work.
OK, I'll liaise with Umang and may look to deal with a couple of them.
Things like documenting the memory barriers probably falls to Phil.
Dave
> Thanks Stefan
>
> >
> > Dave
> >
> >>> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> >>>
> >>>>>>> I see two TODO files in vc04_services:
> >>>>>>> ./bcm2835-camera/TODO
> >>>>>>> ./interface/TODO
> >>>>>>>
> >>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>>>>> itself. So that's address in the series. The other remaining one -
> >>>>>>> I will need to take a deeper look before commenting on it.
> >>>>>>>
> >>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>>>>> a cursory reading of them suggests that these apply to *all*
> >>>>>>> vc04_services components? Am I right?
> >>>>>> Actually these applies just for the interfaces directory. Some of
> >>>>>> them could apply to the services, but this is no priority.
> >>>>> By no priority, you mean this doesn't affect the criteria required to
> >>>>> ful-fill to get these out of staging?
> >>>> Correct
> >>>>
> >>>>>>> Are these are the specific bits of cleanup you are referring to in
> >>>>>>> your comment?
> >>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>>>>> that i don't remember them all. The first that come to my mind was
> >>>>>> those fancy comment sections which is not kernel coding style. It
> >>>>>> has been removed.
> >>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >>>>> out-of-tree still so...). I mean what are the specific bits / points
> >>>>> that needs to be addressed to get vc04_services out of the staging.
> >>>> These were the points which i mentioned in my last email. They came
> >>>> from interface/TODO.
> >>>>
> >>>>> You have mentioned it above now, so I'll follow up on those.
> >>>> That would be great :)
> >>>>
> >>>>> The many vchiq changes you referred to above comment (that you don't
> >>>>> remember) are from [1] as well or some other series ?
> >>>> Sorry, for the confusing. The many changes i refer were the dozens of
> >>>> clean up patches for vc04_interfaces in mainline staging since the
> >>>> last years. [1] was just a single patch which has been accepted yet.
> >>> Ah I see. There are many others that I've to dig out then. Thanks for
> >>> clarifying!
> >>>
> >>>>>> [1] -
> >>>>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> >>>>>>
> >>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>>>>
> >>>>>>>> Just my two cents
> >>>>>>>> Stefan
> >> --
> >> Regards,
> >>
> >> Laurent Pinchart
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-02 13:42 ` Dave Stevenson
0 siblings, 0 replies; 95+ messages in thread
From: Dave Stevenson @ 2022-12-02 13:42 UTC (permalink / raw)
To: Stefan Wahren
Cc: Laurent Pinchart, Umang Jain, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Stefan
On Fri, 2 Dec 2022 at 12:41, Stefan Wahren <stefan.wahren@i2se.com> wrote:
>
> Hi Dave,
>
> Am 02.12.22 um 12:23 schrieb Dave Stevenson:
> > Hi Laurent, Umang, and Stefan.
> >
> > On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
> > <laurent.pinchart@ideasonboard.com> wrote:
> >> Hi Umang,
> >>
> >> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
> >>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
> >>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
> >>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
> >>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
> >>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
> >>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
> >>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
> >>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
> >>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
> >>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
> >>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
> >>>>>>>>> will remain the primary area of this series.
> >>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
> >>>>>>>> comes out of staging before adding more features. As Greg said
> >>>>>>>> some time ago staging is not a place to "dump code and run away".
> >>>>>>>> These new files are in the same bad shape as the rest of vc04
> >>>>>>>> before the clean-up here in staging started.
> >>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
> >>>>>> no problem.
> >>>>>>
> >>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
> >>>>>>> upstream, I would be happy to help here! Though I must mention that
> >>>>>>> I still have limited visibility so my aim would be to chart out a
> >>>>>>> plan of things needed to be done to get vc04_services out of staging!
> >>>>>> The vchiq driver is in staging since 2016, so every step forwards is
> >>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
> >>>>>>
> >>>>>> For me the most important, but not to tricky steps to get vchiq out
> >>>>>> of staging would be:
> >>>>>>
> >>>>>> * Cleanup logging mechanism
> >>>>>>
> >>>>>> * Get rid of custom function return values
> >>>>>>
> >>>>>> There was already an attempt for this [1]
> >>>>>>
> >>>>>> * Get rid of all non essential global structures and create a proper per
> >>>>>> device structure
> >>>>>>
> >>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
> >>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
> >>>>>>>> staging. It just binds more resources on a new feature.
> >>>>> bcm2835-camera is the legacy camera stack which probably need to be
> >>>>> dropped from hereon...
> >>>> I don't not know if there any users left, so i would be careful here.
> >>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
> >>>> dumb question but i'm not expert here.
> >>> I am careful too here and probably need Input from RaspberryPi in order
> >>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
> >>> going out of staging - it'll either sit here (or probably dropped) as
> >>> statied from [1]
> >>>
> >>> ```
> >>> + * There are two camera drivers in the kernel for BCM283x - this one
> >>> + * and bcm2835-camera (currently in staging).
> >>> ```
> >>>
> >>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
> >>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
> >>> mentioned in my cover the testing of bcm2835-isp happened on top of
> >>> unicam patches.
> >> To be accurate, the bcm2835-camera driver supports the VC4
> >> firmware-based camera stack. In that setup, the camera sensors (OV5647
> >> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
> >> firmware, which provides a high-level interface towards the kernel. This
> >> architecture has been replaced by Linux-side control of the camera
> >> sensors (through existing drivers in drivers/media/i2c/), Unicam
> >> (through the driver from [1]) and ISP (through this driver). Moving
> >> control to the Linux side requires complex processing in userspace,
> >> handled by libcamera.
> >>
> >> bcm2835-camera is thus replaced by multiple drivers combined with
> >> libcamera, and that is the camera stack that is shipped by Raspberry Pi
> >> these days. While this may affect some userspace use cases), we will not
> >> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
> >> is planning to do so either. I don't mind much if the driver stays in
> >> staging for some more time, but I'd rather drop it if possible.
> Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
> handle the old camera (OV5647)?
There are kernel drivers in mainline for ov5647 (v1) and imx219 (v2)
cameras, with imx477 (HQ camera) due to be done. Connected to
bcm2835-unicam that gives you raw images from these sensors.
However you have to use libcamera to get useful images with these
drivers - raw Bayer isn't easy to consume.
Libcamera has support for all three (and more) sensors in the RPi IPA
(image processing algorithms), and will make use of bcm2835-isp to
process the images appropriately.
> > It would be reasonable to drop it at the point that Libcamera can work
> > to a similar level with at least the following list of applications:
> > - FFmpeg
> > - Gstreamer
> > - Chromium
> > - Firefox
> > - Motion
> > And that still leaves a huge number of existing V4L2 apps out in the cold.
> >
> > Do you wish to make any predictions as to when that would be
> > achievable? Or even when a v1.0 release of libcamera is going to
> > happen?
> > Dropping anything prior to those points would be rather premature in my book.
> >
> >
> > The TODOs on bcm2835-camera are:
> > 1) Zero copy. That comes almost for free as bcm2835-isp already does
> > this, but it does rely on vcsm-cma.
> > The main reason I haven't pushed it is that it then requires
> > reasonable amounts of CMA heap for all the buffers, which until
> > recently haven't been present in the default configurations. With the
> > vc4 DRM driver now being default (at least for the vendor kernel) and
> > also requiring CMA, making the change makes more sense.
> > AFAIK there is no easy way to have one driver choosing between using
> > vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
> > wrong.
> > Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
> > so perhaps we don't get very far.
> CMA configuration should actually happen in device tree or kernel
> cmdline. The bcm2835_defconfig is limited to make it work even with the
> original Pi.
It's the original 256MB Pis that are the issue as memory is so tight.
Losing a bigger chunk to CMA is a problem there.
I see that bcm283x.dtsi sets it to 64MB, which is probably sufficient
for most bcm2835-camera use cases, but leaves me worried that you'll
steal it from other things.
> >
> > 2) This isn't workable within the current V4L2 frameworks. The
> > multi-planar V4L2 pixel formats are currently allocated as independent
> > buffers for each plane, whereas the firmware needs a single buffer
> > with (currently) specific offsets for the chroma planes. The
> > V4L2/videobuf2 core changes required to implement that are going to be
> > significant, and have minimal gain.
> > The specific stride handling is already dealt with (set bytesperline
> > appropriately), it's the padding of the height to a multiple of 16
> > before the chroma planes on YUV420 and NV12 formats that require the
> > firmware to do a small amount of repacking. The performance hit is
> > actually minimal anyway.
> >
> > If bcm2835-camera is the only thing holding back vc04_services, then I
> > can have a look at it.
>
> No, it's the vchiq interface which needs the work.
OK, I'll liaise with Umang and may look to deal with a couple of them.
Things like documenting the memory barriers probably falls to Phil.
Dave
> Thanks Stefan
>
> >
> > Dave
> >
> >>> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
> >>>
> >>>>>>> I see two TODO files in vc04_services:
> >>>>>>> ./bcm2835-camera/TODO
> >>>>>>> ./interface/TODO
> >>>>>>>
> >>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
> >>>>>>> itself. So that's address in the series. The other remaining one -
> >>>>>>> I will need to take a deeper look before commenting on it.
> >>>>>>>
> >>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
> >>>>>>> a cursory reading of them suggests that these apply to *all*
> >>>>>>> vc04_services components? Am I right?
> >>>>>> Actually these applies just for the interfaces directory. Some of
> >>>>>> them could apply to the services, but this is no priority.
> >>>>> By no priority, you mean this doesn't affect the criteria required to
> >>>>> ful-fill to get these out of staging?
> >>>> Correct
> >>>>
> >>>>>>> Are these are the specific bits of cleanup you are referring to in
> >>>>>>> your comment?
> >>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
> >>>>>> that i don't remember them all. The first that come to my mind was
> >>>>>> those fancy comment sections which is not kernel coding style. It
> >>>>>> has been removed.
> >>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
> >>>>> out-of-tree still so...). I mean what are the specific bits / points
> >>>>> that needs to be addressed to get vc04_services out of the staging.
> >>>> These were the points which i mentioned in my last email. They came
> >>>> from interface/TODO.
> >>>>
> >>>>> You have mentioned it above now, so I'll follow up on those.
> >>>> That would be great :)
> >>>>
> >>>>> The many vchiq changes you referred to above comment (that you don't
> >>>>> remember) are from [1] as well or some other series ?
> >>>> Sorry, for the confusing. The many changes i refer were the dozens of
> >>>> clean up patches for vc04_interfaces in mainline staging since the
> >>>> last years. [1] was just a single patch which has been accepted yet.
> >>> Ah I see. There are many others that I've to dig out then. Thanks for
> >>> clarifying!
> >>>
> >>>>>> [1] -
> >>>>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
> >>>>>>
> >>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
> >>>>>>>>
> >>>>>>>> Just my two cents
> >>>>>>>> Stefan
> >> --
> >> Regards,
> >>
> >> Laurent Pinchart
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
2022-12-02 13:42 ` Dave Stevenson
@ 2022-12-03 13:41 ` Stefan Wahren
-1 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-03 13:41 UTC (permalink / raw)
To: Dave Stevenson
Cc: Laurent Pinchart, Umang Jain, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
Am 02.12.22 um 14:42 schrieb Dave Stevenson:
> Hi Stefan
>
> On Fri, 2 Dec 2022 at 12:41, Stefan Wahren <stefan.wahren@i2se.com> wrote:
>> Hi Dave,
>>
>> Am 02.12.22 um 12:23 schrieb Dave Stevenson:
>>> Hi Laurent, Umang, and Stefan.
>>>
>>> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
>>> <laurent.pinchart@ideasonboard.com> wrote:
>>>> Hi Umang,
>>>>
>>>> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
>>>>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
>>>>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
>>>>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>>>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>>>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>>>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>>>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
>>>>>>>>>>> will remain the primary area of this series.
>>>>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>>>>>>> comes out of staging before adding more features. As Greg said
>>>>>>>>>> some time ago staging is not a place to "dump code and run away".
>>>>>>>>>> These new files are in the same bad shape as the rest of vc04
>>>>>>>>>> before the clean-up here in staging started.
>>>>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
>>>>>>>> no problem.
>>>>>>>>
>>>>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>>>>>>> upstream, I would be happy to help here! Though I must mention that
>>>>>>>>> I still have limited visibility so my aim would be to chart out a
>>>>>>>>> plan of things needed to be done to get vc04_services out of staging!
>>>>>>>> The vchiq driver is in staging since 2016, so every step forwards is
>>>>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>>>>>>
>>>>>>>> For me the most important, but not to tricky steps to get vchiq out
>>>>>>>> of staging would be:
>>>>>>>>
>>>>>>>> * Cleanup logging mechanism
>>>>>>>>
>>>>>>>> * Get rid of custom function return values
>>>>>>>>
>>>>>>>> There was already an attempt for this [1]
>>>>>>>>
>>>>>>>> * Get rid of all non essential global structures and create a proper per
>>>>>>>> device structure
>>>>>>>>
>>>>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>>>>>>> staging. It just binds more resources on a new feature.
>>>>>>> bcm2835-camera is the legacy camera stack which probably need to be
>>>>>>> dropped from hereon...
>>>>>> I don't not know if there any users left, so i would be careful here.
>>>>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
>>>>>> dumb question but i'm not expert here.
>>>>> I am careful too here and probably need Input from RaspberryPi in order
>>>>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
>>>>> going out of staging - it'll either sit here (or probably dropped) as
>>>>> statied from [1]
>>>>>
>>>>> ```
>>>>> + * There are two camera drivers in the kernel for BCM283x - this one
>>>>> + * and bcm2835-camera (currently in staging).
>>>>> ```
>>>>>
>>>>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
>>>>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
>>>>> mentioned in my cover the testing of bcm2835-isp happened on top of
>>>>> unicam patches.
>>>> To be accurate, the bcm2835-camera driver supports the VC4
>>>> firmware-based camera stack. In that setup, the camera sensors (OV5647
>>>> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
>>>> firmware, which provides a high-level interface towards the kernel. This
>>>> architecture has been replaced by Linux-side control of the camera
>>>> sensors (through existing drivers in drivers/media/i2c/), Unicam
>>>> (through the driver from [1]) and ISP (through this driver). Moving
>>>> control to the Linux side requires complex processing in userspace,
>>>> handled by libcamera.
>>>>
>>>> bcm2835-camera is thus replaced by multiple drivers combined with
>>>> libcamera, and that is the camera stack that is shipped by Raspberry Pi
>>>> these days. While this may affect some userspace use cases), we will not
>>>> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
>>>> is planning to do so either. I don't mind much if the driver stays in
>>>> staging for some more time, but I'd rather drop it if possible.
>> Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
>> handle the old camera (OV5647)?
> There are kernel drivers in mainline for ov5647 (v1) and imx219 (v2)
> cameras, with imx477 (HQ camera) due to be done. Connected to
> bcm2835-unicam that gives you raw images from these sensors.
>
> However you have to use libcamera to get useful images with these
> drivers - raw Bayer isn't easy to consume.
> Libcamera has support for all three (and more) sensors in the RPi IPA
> (image processing algorithms), and will make use of bcm2835-isp to
> process the images appropriately.
>
>>> It would be reasonable to drop it at the point that Libcamera can work
>>> to a similar level with at least the following list of applications:
>>> - FFmpeg
>>> - Gstreamer
>>> - Chromium
>>> - Firefox
>>> - Motion
>>> And that still leaves a huge number of existing V4L2 apps out in the cold.
>>>
>>> Do you wish to make any predictions as to when that would be
>>> achievable? Or even when a v1.0 release of libcamera is going to
>>> happen?
>>> Dropping anything prior to those points would be rather premature in my book.
>>>
>>>
>>> The TODOs on bcm2835-camera are:
>>> 1) Zero copy. That comes almost for free as bcm2835-isp already does
>>> this, but it does rely on vcsm-cma.
>>> The main reason I haven't pushed it is that it then requires
>>> reasonable amounts of CMA heap for all the buffers, which until
>>> recently haven't been present in the default configurations. With the
>>> vc4 DRM driver now being default (at least for the vendor kernel) and
>>> also requiring CMA, making the change makes more sense.
>>> AFAIK there is no easy way to have one driver choosing between using
>>> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
>>> wrong.
>>> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
>>> so perhaps we don't get very far.
>> CMA configuration should actually happen in device tree or kernel
>> cmdline. The bcm2835_defconfig is limited to make it work even with the
>> original Pi.
> It's the original 256MB Pis that are the issue as memory is so tight.
> Losing a bigger chunk to CMA is a problem there.
>
> I see that bcm283x.dtsi sets it to 64MB, which is probably sufficient
> for most bcm2835-camera use cases, but leaves me worried that you'll
> steal it from other things.
>
>>> 2) This isn't workable within the current V4L2 frameworks. The
>>> multi-planar V4L2 pixel formats are currently allocated as independent
>>> buffers for each plane, whereas the firmware needs a single buffer
>>> with (currently) specific offsets for the chroma planes. The
>>> V4L2/videobuf2 core changes required to implement that are going to be
>>> significant, and have minimal gain.
>>> The specific stride handling is already dealt with (set bytesperline
>>> appropriately), it's the padding of the height to a multiple of 16
>>> before the chroma planes on YUV420 and NV12 formats that require the
>>> firmware to do a small amount of repacking. The performance hit is
>>> actually minimal anyway.
>>>
>>> If bcm2835-camera is the only thing holding back vc04_services, then I
>>> can have a look at it.
>> No, it's the vchiq interface which needs the work.
> OK, I'll liaise with Umang and may look to deal with a couple of them.
> Things like documenting the memory barriers probably falls to Phil.
actually this item is already done. I missed to update the TODO file and
send a patch soon.
Best regards
>
> Dave
>
>> Thanks Stefan
>>
>>> Dave
>>>
>>>>> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
>>>>>
>>>>>>>>> I see two TODO files in vc04_services:
>>>>>>>>> ./bcm2835-camera/TODO
>>>>>>>>> ./interface/TODO
>>>>>>>>>
>>>>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>>>>>>>> itself. So that's address in the series. The other remaining one -
>>>>>>>>> I will need to take a deeper look before commenting on it.
>>>>>>>>>
>>>>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
>>>>>>>>> a cursory reading of them suggests that these apply to *all*
>>>>>>>>> vc04_services components? Am I right?
>>>>>>>> Actually these applies just for the interfaces directory. Some of
>>>>>>>> them could apply to the services, but this is no priority.
>>>>>>> By no priority, you mean this doesn't affect the criteria required to
>>>>>>> ful-fill to get these out of staging?
>>>>>> Correct
>>>>>>
>>>>>>>>> Are these are the specific bits of cleanup you are referring to in
>>>>>>>>> your comment?
>>>>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
>>>>>>>> that i don't remember them all. The first that come to my mind was
>>>>>>>> those fancy comment sections which is not kernel coding style. It
>>>>>>>> has been removed.
>>>>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
>>>>>>> out-of-tree still so...). I mean what are the specific bits / points
>>>>>>> that needs to be addressed to get vc04_services out of the staging.
>>>>>> These were the points which i mentioned in my last email. They came
>>>>>> from interface/TODO.
>>>>>>
>>>>>>> You have mentioned it above now, so I'll follow up on those.
>>>>>> That would be great :)
>>>>>>
>>>>>>> The many vchiq changes you referred to above comment (that you don't
>>>>>>> remember) are from [1] as well or some other series ?
>>>>>> Sorry, for the confusing. The many changes i refer were the dozens of
>>>>>> clean up patches for vc04_interfaces in mainline staging since the
>>>>>> last years. [1] was just a single patch which has been accepted yet.
>>>>> Ah I see. There are many others that I've to dig out then. Thanks for
>>>>> clarifying!
>>>>>
>>>>>>>> [1] -
>>>>>>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>>>>>>>>
>>>>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>>>>>>>
>>>>>>>>>> Just my two cents
>>>>>>>>>> Stefan
>>>> --
>>>> Regards,
>>>>
>>>> Laurent Pinchart
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [PATCH 00/14] staging: vc04_services: bcm2835-isp support
@ 2022-12-03 13:41 ` Stefan Wahren
0 siblings, 0 replies; 95+ messages in thread
From: Stefan Wahren @ 2022-12-03 13:41 UTC (permalink / raw)
To: Dave Stevenson
Cc: Laurent Pinchart, Umang Jain, linux-media, kernel-list,
linux-kernel, linux-rpi-kernel, linux-arm-kernel, linux-staging,
Broadcom internal kernel review list, Florian Fainelli,
Naushir Patuck, David Plowman, Kieran Bingham
Hi Dave,
Am 02.12.22 um 14:42 schrieb Dave Stevenson:
> Hi Stefan
>
> On Fri, 2 Dec 2022 at 12:41, Stefan Wahren <stefan.wahren@i2se.com> wrote:
>> Hi Dave,
>>
>> Am 02.12.22 um 12:23 schrieb Dave Stevenson:
>>> Hi Laurent, Umang, and Stefan.
>>>
>>> On Fri, 2 Dec 2022 at 09:17, Laurent Pinchart
>>> <laurent.pinchart@ideasonboard.com> wrote:
>>>> Hi Umang,
>>>>
>>>> On Fri, Dec 02, 2022 at 11:57:18AM +0800, Umang Jain wrote:
>>>>> On 12/2/22 6:45 AM, Stefan Wahren wrote:
>>>>>> Am 30.11.22 um 11:58 schrieb Umang Jain:
>>>>>>> On 11/27/22 6:56 AM, Stefan Wahren wrote:
>>>>>>>> Am 26.11.22 um 17:26 schrieb Umang Jain:
>>>>>>>>> On 11/26/22 8:12 PM, Stefan Wahren wrote:
>>>>>>>>>> Am 21.11.22 um 22:47 schrieb Umang Jain:
>>>>>>>>>>> This series aims to upport bcm2835-isp from the RPi kernel [1] and is a
>>>>>>>>>>> independent subset of earlier series [2] posted to upport CSI-2/CCP2
>>>>>>>>>>> receiver IP core("Unicam) + the ISP driver found in BCM283x and compatible
>>>>>>>>>>> SoCs (namely BCM2711). Unicam is still under active development to work
>>>>>>>>>>> with multistream support to get into mainline. Hence only the ISP driver
>>>>>>>>>>> will remain the primary area of this series.
>>>>>>>>>> thanks for working on this. But honestly i would prefer that vchiq
>>>>>>>>>> comes out of staging before adding more features. As Greg said
>>>>>>>>>> some time ago staging is not a place to "dump code and run away".
>>>>>>>>>> These new files are in the same bad shape as the rest of vc04
>>>>>>>>>> before the clean-up here in staging started.
>>>>>>>>> Certainly, I am not here to do that - but I am still learning the ropes.
>>>>>>>> no problem.
>>>>>>>>
>>>>>>>>> If the staging issue is becoming a blocker for bcm2835-isp going
>>>>>>>>> upstream, I would be happy to help here! Though I must mention that
>>>>>>>>> I still have limited visibility so my aim would be to chart out a
>>>>>>>>> plan of things needed to be done to get vc04_services out of staging!
>>>>>>>> The vchiq driver is in staging since 2016, so every step forwards is
>>>>>>>> good. Unfortunately all of the low hanging fruits has been gathered.
>>>>>>>>
>>>>>>>> For me the most important, but not to tricky steps to get vchiq out
>>>>>>>> of staging would be:
>>>>>>>>
>>>>>>>> * Cleanup logging mechanism
>>>>>>>>
>>>>>>>> * Get rid of custom function return values
>>>>>>>>
>>>>>>>> There was already an attempt for this [1]
>>>>>>>>
>>>>>>>> * Get rid of all non essential global structures and create a proper per
>>>>>>>> device structure
>>>>>>>>
>>>>>>>>>> I agree that VCSM is on the TODO list for vchiq, but this driver
>>>>>>>>>> is not necessary for making bcm2835-audio & bcm2835-camera leave
>>>>>>>>>> staging. It just binds more resources on a new feature.
>>>>>>> bcm2835-camera is the legacy camera stack which probably need to be
>>>>>>> dropped from hereon...
>>>>>> I don't not know if there any users left, so i would be careful here.
>>>>>> Can bcm2835-isp completely replace bcm2835-camera? Sorry, for this
>>>>>> dumb question but i'm not expert here.
>>>>> I am careful too here and probably need Input from RaspberryPi in order
>>>>> to proceed to drop it. But from my perspective - bcm2835-camera is _not_
>>>>> going out of staging - it'll either sit here (or probably dropped) as
>>>>> statied from [1]
>>>>>
>>>>> ```
>>>>> + * There are two camera drivers in the kernel for BCM283x - this one
>>>>> + * and bcm2835-camera (currently in staging).
>>>>> ```
>>>>>
>>>>> The bcm2835-camera is meant to be replaced by unicam [1] , but the ISP
>>>>> (bcm2835-isp) is meant to be worked with unicam [1]. In fact, I have
>>>>> mentioned in my cover the testing of bcm2835-isp happened on top of
>>>>> unicam patches.
>>>> To be accurate, the bcm2835-camera driver supports the VC4
>>>> firmware-based camera stack. In that setup, the camera sensors (OV5647
>>>> or IMX219), CSI-2 receiver (Unicam) and ISP are all controlled by the
>>>> firmware, which provides a high-level interface towards the kernel. This
>>>> architecture has been replaced by Linux-side control of the camera
>>>> sensors (through existing drivers in drivers/media/i2c/), Unicam
>>>> (through the driver from [1]) and ISP (through this driver). Moving
>>>> control to the Linux side requires complex processing in userspace,
>>>> handled by libcamera.
>>>>
>>>> bcm2835-camera is thus replaced by multiple drivers combined with
>>>> libcamera, and that is the camera stack that is shipped by Raspberry Pi
>>>> these days. While this may affect some userspace use cases), we will not
>>>> work on destaging bcm2835-camera, and as far as I'm aware, nobody else
>>>> is planning to do so either. I don't mind much if the driver stays in
>>>> staging for some more time, but I'd rather drop it if possible.
>> Thanks for clarification. Okay, so Unicam + bcm2835-isp are able to
>> handle the old camera (OV5647)?
> There are kernel drivers in mainline for ov5647 (v1) and imx219 (v2)
> cameras, with imx477 (HQ camera) due to be done. Connected to
> bcm2835-unicam that gives you raw images from these sensors.
>
> However you have to use libcamera to get useful images with these
> drivers - raw Bayer isn't easy to consume.
> Libcamera has support for all three (and more) sensors in the RPi IPA
> (image processing algorithms), and will make use of bcm2835-isp to
> process the images appropriately.
>
>>> It would be reasonable to drop it at the point that Libcamera can work
>>> to a similar level with at least the following list of applications:
>>> - FFmpeg
>>> - Gstreamer
>>> - Chromium
>>> - Firefox
>>> - Motion
>>> And that still leaves a huge number of existing V4L2 apps out in the cold.
>>>
>>> Do you wish to make any predictions as to when that would be
>>> achievable? Or even when a v1.0 release of libcamera is going to
>>> happen?
>>> Dropping anything prior to those points would be rather premature in my book.
>>>
>>>
>>> The TODOs on bcm2835-camera are:
>>> 1) Zero copy. That comes almost for free as bcm2835-isp already does
>>> this, but it does rely on vcsm-cma.
>>> The main reason I haven't pushed it is that it then requires
>>> reasonable amounts of CMA heap for all the buffers, which until
>>> recently haven't been present in the default configurations. With the
>>> vc4 DRM driver now being default (at least for the vendor kernel) and
>>> also requiring CMA, making the change makes more sense.
>>> AFAIK there is no easy way to have one driver choosing between using
>>> vb2_vmalloc_memops and vb2_dma_contig_memops at runtime, but I may be
>>> wrong.
>>> Actually bcm2835_defconfig appears to only allocate a 32MB CMA heap,
>>> so perhaps we don't get very far.
>> CMA configuration should actually happen in device tree or kernel
>> cmdline. The bcm2835_defconfig is limited to make it work even with the
>> original Pi.
> It's the original 256MB Pis that are the issue as memory is so tight.
> Losing a bigger chunk to CMA is a problem there.
>
> I see that bcm283x.dtsi sets it to 64MB, which is probably sufficient
> for most bcm2835-camera use cases, but leaves me worried that you'll
> steal it from other things.
>
>>> 2) This isn't workable within the current V4L2 frameworks. The
>>> multi-planar V4L2 pixel formats are currently allocated as independent
>>> buffers for each plane, whereas the firmware needs a single buffer
>>> with (currently) specific offsets for the chroma planes. The
>>> V4L2/videobuf2 core changes required to implement that are going to be
>>> significant, and have minimal gain.
>>> The specific stride handling is already dealt with (set bytesperline
>>> appropriately), it's the padding of the height to a multiple of 16
>>> before the chroma planes on YUV420 and NV12 formats that require the
>>> firmware to do a small amount of repacking. The performance hit is
>>> actually minimal anyway.
>>>
>>> If bcm2835-camera is the only thing holding back vc04_services, then I
>>> can have a look at it.
>> No, it's the vchiq interface which needs the work.
> OK, I'll liaise with Umang and may look to deal with a couple of them.
> Things like documenting the memory barriers probably falls to Phil.
actually this item is already done. I missed to update the TODO file and
send a patch soon.
Best regards
>
> Dave
>
>> Thanks Stefan
>>
>>> Dave
>>>
>>>>> [1]: https://lore.kernel.org/linux-media/20220208155027.891055-5-jeanmichel.hautbois@ideasonboard.com/
>>>>>
>>>>>>>>> I see two TODO files in vc04_services:
>>>>>>>>> ./bcm2835-camera/TODO
>>>>>>>>> ./interface/TODO
>>>>>>>>>
>>>>>>>>> One of the bcm2835-camera TODO points to the vc-sm-cma driver
>>>>>>>>> itself. So that's address in the series. The other remaining one -
>>>>>>>>> I will need to take a deeper look before commenting on it.
>>>>>>>>>
>>>>>>>>> The main chunk of TODO are in vc04_services/interfaces/TODO. Doing
>>>>>>>>> a cursory reading of them suggests that these apply to *all*
>>>>>>>>> vc04_services components? Am I right?
>>>>>>>> Actually these applies just for the interfaces directory. Some of
>>>>>>>> them could apply to the services, but this is no priority.
>>>>>>> By no priority, you mean this doesn't affect the criteria required to
>>>>>>> ful-fill to get these out of staging?
>>>>>> Correct
>>>>>>
>>>>>>>>> Are these are the specific bits of cleanup you are referring to in
>>>>>>>>> your comment?
>>>>>>>> You mean about bcm2835-isp? There were too many changes to vchiq
>>>>>>>> that i don't remember them all. The first that come to my mind was
>>>>>>>> those fancy comment sections which is not kernel coding style. It
>>>>>>>> has been removed.
>>>>>>> No, I don't mean the bcm2835-isp changes (those are upcoming /
>>>>>>> out-of-tree still so...). I mean what are the specific bits / points
>>>>>>> that needs to be addressed to get vc04_services out of the staging.
>>>>>> These were the points which i mentioned in my last email. They came
>>>>>> from interface/TODO.
>>>>>>
>>>>>>> You have mentioned it above now, so I'll follow up on those.
>>>>>> That would be great :)
>>>>>>
>>>>>>> The many vchiq changes you referred to above comment (that you don't
>>>>>>> remember) are from [1] as well or some other series ?
>>>>>> Sorry, for the confusing. The many changes i refer were the dozens of
>>>>>> clean up patches for vc04_interfaces in mainline staging since the
>>>>>> last years. [1] was just a single patch which has been accepted yet.
>>>>> Ah I see. There are many others that I've to dig out then. Thanks for
>>>>> clarifying!
>>>>>
>>>>>>>> [1] -
>>>>>>>> https://lore.kernel.org/linux-staging/20220712181928.17547-1-jslebodn@redhat.com/
>>>>>>>>
>>>>>>>>>> Unfortuntately i hadn't much time to work on vchiq by myself.
>>>>>>>>>>
>>>>>>>>>> Just my two cents
>>>>>>>>>> Stefan
>>>> --
>>>> Regards,
>>>>
>>>> Laurent Pinchart
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 95+ messages in thread
end of thread, other threads:[~2022-12-03 13:42 UTC | newest]
Thread overview: 95+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-21 21:47 [PATCH 00/14] staging: vc04_services: bcm2835-isp support Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 21:47 ` [PATCH 01/14] staging: vc04_services: Add new vc-sm-cma driver Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:04 ` Laurent Pinchart
2022-11-21 23:04 ` Laurent Pinchart
2022-11-21 21:47 ` [PATCH 02/14] staging: vchiq_arm: Register vcsm-cma as a platform driver Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:07 ` Laurent Pinchart
2022-11-21 23:07 ` Laurent Pinchart
2022-11-21 21:47 ` [PATCH 03/14] media: videobuf2: Allow exporting of a struct dmabuf Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:18 ` Laurent Pinchart
2022-11-21 23:18 ` Laurent Pinchart
2022-11-22 11:35 ` Dave Stevenson
2022-11-22 11:35 ` Dave Stevenson
2022-11-25 1:49 ` Laurent Pinchart
2022-11-25 1:49 ` Laurent Pinchart
2022-11-21 21:47 ` [PATCH 04/14] staging: mmal-vchiq: Add support for event callbacks Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 21:47 ` [PATCH 05/14] staging: mmal-vchiq: Use vc-sm-cma to support zero copy Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 21:47 ` [PATCH 06/14] staging: mmal_vchiq: Add image formats to be used by bcm2835-isp Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:21 ` Laurent Pinchart
2022-11-21 23:21 ` Laurent Pinchart
2022-11-21 21:47 ` [PATCH 07/14] media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:25 ` Laurent Pinchart
2022-11-21 23:25 ` Laurent Pinchart
2022-11-21 21:47 ` [PATCH 08/14] uapi: bcm2835-isp: Add bcm2835-isp uapi header file Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:28 ` Laurent Pinchart
2022-11-21 23:28 ` Laurent Pinchart
2022-11-21 21:47 ` [PATCH 09/14] staging: vc04_services: bcm2835-isp: Add a more complex ISP processing component Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 21:47 ` [PATCH 10/14] WIP: vc04_services: bcm2835-isp: Allow formats with different colour spaces Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 21:47 ` [PATCH 11/14] WIP: vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP outputs Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 21:47 ` [PATCH 12/14] staging: vc04_services: bcm2835_isp: Allow multiple users Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:29 ` Laurent Pinchart
2022-11-21 23:29 ` Laurent Pinchart
2022-11-21 23:35 ` Laurent Pinchart
2022-11-21 23:35 ` Laurent Pinchart
2022-11-21 21:47 ` [PATCH 13/14] docs: admin-guide: media: bcm2835-isp: Add documentation for bcm2835-isp Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:41 ` Laurent Pinchart
2022-11-21 23:41 ` Laurent Pinchart
2022-11-21 21:47 ` [PATCH 14/14] staging: vc04_services: vchiq: Load bcm2835_isp driver from vchiq Umang Jain
2022-11-21 21:47 ` Umang Jain
2022-11-21 23:10 ` Laurent Pinchart
2022-11-21 23:10 ` Laurent Pinchart
2022-11-21 22:16 ` [PATCH 00/14] staging: vc04_services: bcm2835-isp support Laurent Pinchart
2022-11-21 22:16 ` Laurent Pinchart
2022-11-22 11:42 ` Dave Stevenson
2022-11-22 11:42 ` Dave Stevenson
2022-11-22 12:34 ` Umang Jain
2022-11-22 12:34 ` Umang Jain
2022-11-26 14:42 ` Stefan Wahren
2022-11-26 14:42 ` Stefan Wahren
2022-11-26 16:26 ` Umang Jain
2022-11-26 16:26 ` Umang Jain
2022-11-26 22:56 ` Stefan Wahren
2022-11-26 22:56 ` Stefan Wahren
2022-11-30 10:58 ` Umang Jain
2022-11-30 10:58 ` Umang Jain
2022-12-01 22:45 ` Stefan Wahren
2022-12-01 22:45 ` Stefan Wahren
2022-12-02 3:57 ` Umang Jain
2022-12-02 3:57 ` Umang Jain
2022-12-02 9:17 ` Laurent Pinchart
2022-12-02 9:17 ` Laurent Pinchart
2022-12-02 11:23 ` Dave Stevenson
2022-12-02 11:23 ` Dave Stevenson
2022-12-02 12:10 ` Laurent Pinchart
2022-12-02 12:10 ` Laurent Pinchart
2022-12-02 12:35 ` Stefan Wahren
2022-12-02 12:35 ` Stefan Wahren
2022-12-02 13:25 ` Peter Robinson
2022-12-02 13:25 ` Peter Robinson
2022-12-02 12:38 ` Dave Stevenson
2022-12-02 12:38 ` Dave Stevenson
2022-12-02 13:29 ` Laurent Pinchart
2022-12-02 13:29 ` Laurent Pinchart
2022-12-02 12:25 ` Stefan Wahren
2022-12-02 12:41 ` Stefan Wahren
2022-12-02 12:41 ` Stefan Wahren
2022-12-02 13:32 ` Laurent Pinchart
2022-12-02 13:32 ` Laurent Pinchart
2022-12-02 13:42 ` Dave Stevenson
2022-12-02 13:42 ` Dave Stevenson
2022-12-03 13:41 ` Stefan Wahren
2022-12-03 13:41 ` Stefan Wahren
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.