Potential integration of thermal cameras into v4l

Potential integration of thermal cameras into v4l

[Sophie Friedrich]

Hello people,

over the past months I've written a driver for the Guide MobIR Air¹
USB-C thermal camera. The driver uses v4l2loopback and runs in user
space with python.² The driver manages camera state, calibration and
conversion of the raw microbolometer to temperature values.

In my research I couldn't find any mentions of a thermal camera support
in the Linux kernel nor the media subsystem, so I assume I'm the first
to ask here and we would presumably need lay out some ground work.

The current camera driver outputs the temperature in the Y16 format as
Kelvin * 100 (i.e. 28056 (raw) -> 280.56K -> 7.41°C). This allows for
further processing down the line with i.e. OpenCV, but has its 
limitations. Especially as temps of only up to 382.2°C can be displayed
currently.

Does it seem feasible to integrate support for thermal cameras in the
Linux kernel and more specifically in the v4l2 subsystem?

- Sophie

[1]: 
https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html
[2]: https://github.com/tyalie/pyMobirAir-v4l2/

Re: Potential integration of thermal cameras into v4l

[Pekka Paalanen]

[-- Attachment #1: Type: text/plain, Size: 1922 bytes --]

On Sat, 7 Jan 2023 22:58:41 +0100
Sophie Friedrich <lkml@flowerpot.me> wrote:

> Hello people,
> 
> over the past months I've written a driver for the Guide MobIR Air¹
> USB-C thermal camera. The driver uses v4l2loopback and runs in user
> space with python.² The driver manages camera state, calibration and
> conversion of the raw microbolometer to temperature values.
> 
> In my research I couldn't find any mentions of a thermal camera support
> in the Linux kernel nor the media subsystem, so I assume I'm the first
> to ask here and we would presumably need lay out some ground work.
> 
> The current camera driver outputs the temperature in the Y16 format as
> Kelvin * 100 (i.e. 28056 (raw) -> 280.56K -> 7.41°C). This allows for
> further processing down the line with i.e. OpenCV, but has its 
> limitations. Especially as temps of only up to 382.2°C can be displayed
> currently.
> 
> Does it seem feasible to integrate support for thermal cameras in the
> Linux kernel and more specifically in the v4l2 subsystem?

Hi,

since no-one else replied to you yet, I thought to mention my 2c
(I don't really do camera stuff myself so far):

Perhaps the best place is to reach out to the libcamera community:
https://libcamera.org/

It sounds to me like you want to do some hardware-specific
processing in userspace, and it might not be great to try to come
up with a generic thermal camera API at the kernel UAPI level.
That's where libcamera fits in.

As for the pixel type, maybe use a floating-point type to avoid
range/precision problems? E.g. DRM_FORMAT_R32F for application
facing API. That format does not seem to exist yet, but it's
trivial to add into kernel's drm_fourcc.h and should be well
accepted IMO.


Thanks,
pq

> 
> [1]: 
> https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html
> [2]: https://github.com/tyalie/pyMobirAir-v4l2/

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Potential integration of thermal cameras into v4l

[Jacopo Mondi]

[-- Attachment #1: Type: text/plain, Size: 2457 bytes --]

Hi Sophie and Pekka

On Tue, Jan 10, 2023 at 10:46:26AM +0200, Pekka Paalanen wrote:
> On Sat, 7 Jan 2023 22:58:41 +0100
> Sophie Friedrich <lkml@flowerpot.me> wrote:
>
> > Hello people,
> >
> > over the past months I've written a driver for the Guide MobIR Air¹
> > USB-C thermal camera. The driver uses v4l2loopback and runs in user
> > space with python.² The driver manages camera state, calibration and
> > conversion of the raw microbolometer to temperature values.
> >
> > In my research I couldn't find any mentions of a thermal camera support
> > in the Linux kernel nor the media subsystem, so I assume I'm the first
> > to ask here and we would presumably need lay out some ground work.
> >
> > The current camera driver outputs the temperature in the Y16 format as
> > Kelvin * 100 (i.e. 28056 (raw) -> 280.56K -> 7.41°C). This allows for
> > further processing down the line with i.e. OpenCV, but has its
> > limitations. Especially as temps of only up to 382.2°C can be displayed
> > currently.
> >
> > Does it seem feasible to integrate support for thermal cameras in the
> > Linux kernel and more specifically in the v4l2 subsystem?
>
> Hi,
>
> since no-one else replied to you yet, I thought to mention my 2c
> (I don't really do camera stuff myself so far):
>
> Perhaps the best place is to reach out to the libcamera community:
> https://libcamera.org/
>

cc-ed the libcamera list

I agree it would be interesting to better understand what you mean by
driver here.

The camera seems to be a UVC camera, does it deliver frames with the
current UVC driver or do you need kernel patches to support it ?

I would also be interested why it needs to go through v4l2loopback..

> It sounds to me like you want to do some hardware-specific
> processing in userspace, and it might not be great to try to come
> up with a generic thermal camera API at the kernel UAPI level.
> That's where libcamera fits in.
>
> As for the pixel type, maybe use a floating-point type to avoid
> range/precision problems? E.g. DRM_FORMAT_R32F for application
> facing API. That format does not seem to exist yet, but it's
> trivial to add into kernel's drm_fourcc.h and should be well
> accepted IMO.
>
>
> Thanks,
> pq
>
> >
> > [1]:
> > https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html
> > [2]: https://github.com/tyalie/pyMobirAir-v4l2/

This link is broken :)




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Potential integration of thermal cameras into v4l

[Sophie Tyalie]

Hi Pekka and Jacopo,

I'll not quote the whole exchange now, please tell me if this is an
anti-pattern.

On 10/01/2023 12:45, Jacopo Mondi wrote:
> On Tue, Jan 10, 2023 at 10:46:26AM +0200, Pekka Paalanen wrote:
>> Hi,
>>
>> since no-one else replied to you yet, I thought to mention my 2c
>> (I don't really do camera stuff myself so far):
>>
>> Perhaps the best place is to reach out to the libcamera community:
>> https://libcamera.org/
>>
>
> cc-ed the libcamera list

Thank you. Maybe libcamera is the better place to work on this.

> I agree it would be interesting to better understand what you mean by
> driver here.
>
> The camera seems to be a UVC camera, does it deliver frames with the
> current UVC driver or do you need kernel patches to support it ?

I'm not fully familiar with the UVC driver, but I very much doubt that
the camera speaks anything resembling the UVC protocol.

Guide build an RPC interface over USB, which they use to probe the
camera and retrieve data that is needed later on (manufacturing
calibration files, temperature curves, available modis, …).
Afterwards they tell the camera to send the raw frames on the
same channel (by sending `StartX=1`).

> I would also be interested why it needs to go through v4l2loopback..

I've currently used v4l2loopback in order to get the processed frames
from Python into the v4l2 system, so that I can access it further on.
There are probably better solutions, but it was one that I know and is
easy to work with.

An actual driver developed for v4l2 or libcamera would make use of the
standard ways of course.


>> It sounds to me like you want to do some hardware-specific
>> processing in userspace, and it might not be great to try to come
>> up with a generic thermal camera API at the kernel UAPI level.
>> That's where libcamera fits in.

This is probably true. I'm not sure if a generic thermal camera API is
even feasible, as every manufacturer does their own thing fully outside
of standards.

Together with the Guide MobIR Air, we could also implement the Flir ONE
USB camera which also already has an user space driver¹. But these two
cameras work completely differently in many regards (even conversion of
raw to thermal values).


>> As for the pixel type, maybe use a floating-point type to avoid
>> range/precision problems? E.g. DRM_FORMAT_R32F for application
>> facing API. That format does not seem to exist yet, but it's
>> trivial to add into kernel's drm_fourcc.h and should be well
>> accepted IMO.
>>
>> Thanks,
>> pq

Yeah, I would have preferred a floating point type too, but
v4l2loopback doesn't support one (yet).


>>> [1]:
>>> https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html
>>> [2]: https://github.com/tyalie/pyMobirAir-v4l2/
>
> This link is broken :)

Yeah sorry. I forgot to set the Github repo to public. It should be
fixed now.

Thank you all,
- Sophie

[1]: https://github.com/fnoop/flirone-v4l2

Re: Potential integration of thermal cameras into v4l

[Laurent Pinchart]

Hello,

On Tue, Jan 10, 2023 at 12:45:18PM +0100, Jacopo Mondi via libcamera-devel wrote:
> On Tue, Jan 10, 2023 at 10:46:26AM +0200, Pekka Paalanen wrote:
> > On Sat, 7 Jan 2023 22:58:41 +0100 Sophie Friedrich wrote:
> >
> > > Hello people,
> > >
> > > over the past months I've written a driver for the Guide MobIR Air¹
> > > USB-C thermal camera. The driver uses v4l2loopback and runs in user
> > > space with python.² The driver manages camera state, calibration and
> > > conversion of the raw microbolometer to temperature values.
> > >
> > > In my research I couldn't find any mentions of a thermal camera support
> > > in the Linux kernel nor the media subsystem, so I assume I'm the first
> > > to ask here and we would presumably need lay out some ground work.
> > >
> > > The current camera driver outputs the temperature in the Y16 format as
> > > Kelvin * 100 (i.e. 28056 (raw) -> 280.56K -> 7.41°C). This allows for
> > > further processing down the line with i.e. OpenCV, but has its
> > > limitations. Especially as temps of only up to 382.2°C can be displayed
> > > currently.
> > >
> > > Does it seem feasible to integrate support for thermal cameras in the
> > > Linux kernel and more specifically in the v4l2 subsystem?
> >
> > Hi,
> >
> > since no-one else replied to you yet, I thought to mention my 2c
> > (I don't really do camera stuff myself so far):
> >
> > Perhaps the best place is to reach out to the libcamera community:
> > https://libcamera.org/
> 
> cc-ed the libcamera list
> 
> I agree it would be interesting to better understand what you mean by
> driver here.
> 
> The camera seems to be a UVC camera, does it deliver frames with the
> current UVC driver or do you need kernel patches to support it ?

From a quick glance at the github project linked below, it doesn't seem
to be a UVC camera.

Sophie, could you please share the `lsusb -v` output for the device (you
can limit it to a specific device with `-d VID:PID`), running as root
(e.g. with `sudo`) if possible ?

> I would also be interested why it needs to go through v4l2loopback..
> 
> > It sounds to me like you want to do some hardware-specific
> > processing in userspace, and it might not be great to try to come
> > up with a generic thermal camera API at the kernel UAPI level.
> > That's where libcamera fits in.
> >
> > As for the pixel type, maybe use a floating-point type to avoid
> > range/precision problems? E.g. DRM_FORMAT_R32F for application
> > facing API. That format does not seem to exist yet, but it's
> > trivial to add into kernel's drm_fourcc.h and should be well
> > accepted IMO.
> >
> > > [1]: https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html
> > > [2]: https://github.com/tyalie/pyMobirAir-v4l2/
> 
> This link is broken :)

Works for me.

-- 
Regards,

Laurent Pinchart

Re: Potential integration of thermal cameras into v4l

[Sophie Friedrich]

Hello,

Yeah the link was broken before, but I already fixed it. I'm not
sure if my reply actually reached the mailing list (it accidentally
mixed up my sending adress). At least from the libcamera I got a
bounce, as I'm not a member.

On 10/01/2023 14:13, Laurent Pinchart wrote:
> Sophie, could you please share the `lsusb -v` output for the device (you
> can limit it to a specific device with `-d VID:PID`), running as root
> (e.g. with `sudo`) if possible ?

Note here that the idVendor and iManufacturer do not match. The
camera uses the same USB VID:PID as the Netchip "Gadget Zero", because
of course it does…

$ sudo lsusb -v -d 0525:a4a0
Bus 001 Device 019: ID 0525:a4a0 Netchip Technology, Inc. Linux-USB 
"Gadget Zero"
Device Descriptor:
   bLength                18
   bDescriptorType         1
   bcdUSB               2.01
   bDeviceClass            0
   bDeviceSubClass         0
   bDeviceProtocol         0
   bMaxPacketSize0        64
   idVendor           0x0525 Netchip Technology, Inc.
   idProduct          0xa4a0 Linux-USB "Gadget Zero"
   bcdDevice            2.00
   iManufacturer           1 Wuhan Guide Infrared Co., Ltd.
   iProduct                2 MobIR Air
   iSerial                 3 ZX01A19
   bNumConfigurations      1
   Configuration Descriptor:
     bLength                 9
     bDescriptorType         2
     wTotalLength       0x0040
     bNumInterfaces          2
     bConfigurationValue     1
     iConfiguration          4 MFI Configure
     bmAttributes         0xc0
       Self Powered
     MaxPower              100mA
     Interface Descriptor:
       bLength                 9
       bDescriptorType         4
       bInterfaceNumber        0
       bAlternateSetting       0
       bNumEndpoints           2
       bInterfaceClass       255 Vendor Specific Class
       bInterfaceSubClass    240
       bInterfaceProtocol      0
       iInterface              5 iAP Interface
       Endpoint Descriptor:
         bLength                 7
         bDescriptorType         5
         bEndpointAddress     0x82  EP 2 IN
         bmAttributes            2
           Transfer Type            Bulk
           Synch Type               None
           Usage Type               Data
         wMaxPacketSize     0x0040  1x 64 bytes
         bInterval               0
       Endpoint Descriptor:
         bLength                 7
         bDescriptorType         5
         bEndpointAddress     0x02  EP 2 OUT
         bmAttributes            2
           Transfer Type            Bulk
           Synch Type               None
           Usage Type               Data
         wMaxPacketSize     0x0040  1x 64 bytes
         bInterval               0
     Interface Descriptor:
       bLength                 9
       bDescriptorType         4
       bInterfaceNumber        1
       bAlternateSetting       0
       bNumEndpoints           0
       bInterfaceClass       255 Vendor Specific Class
       bInterfaceSubClass    240
       bInterfaceProtocol      1
       iInterface              6 com.guidesensmart.mobir-air
     Interface Descriptor:
       bLength                 9
       bDescriptorType         4
       bInterfaceNumber        1
       bAlternateSetting       1
       bNumEndpoints           2
       bInterfaceClass       255 Vendor Specific Class
       bInterfaceSubClass    240
       bInterfaceProtocol      1
       iInterface              6 com.guidesensmart.mobir-air
       Endpoint Descriptor:
         bLength                 7
         bDescriptorType         5
         bEndpointAddress     0x81  EP 1 IN
         bmAttributes            2
           Transfer Type            Bulk
           Synch Type               None
           Usage Type               Data
         wMaxPacketSize     0x0040  1x 64 bytes
         bInterval               0
       Endpoint Descriptor:
         bLength                 7
         bDescriptorType         5
         bEndpointAddress     0x01  EP 1 OUT
         bmAttributes            2
           Transfer Type            Bulk
           Synch Type               None
           Usage Type               Data
         wMaxPacketSize     0x0040  1x 64 bytes
         bInterval               0
Binary Object Store Descriptor:
   bLength                 5
   bDescriptorType        15
   wTotalLength       0x000c
   bNumDeviceCaps          1
   USB 2.0 Extension Device Capability:
     bLength                 7
     bDescriptorType        16
     bDevCapabilityType      2
     bmAttributes   0x00000002
       HIRD Link Power Management (LPM) Supported
Device Status:     0x0001
   Self Powered

Re: Potential integration of thermal cameras into v4l

[Laurent Pinchart]

Hi Sophie,

On Tue, Jan 10, 2023 at 02:33:17PM +0100, Sophie Friedrich wrote:
> Hello,
> 
> Yeah the link was broken before, but I already fixed it. I'm not
> sure if my reply actually reached the mailing list (it accidentally
> mixed up my sending adress). At least from the libcamera I got a
> bounce, as I'm not a member.

I've accepted both e-mails manually, and added the two addresses to the
approved senders.

> On 10/01/2023 14:13, Laurent Pinchart wrote:
> > Sophie, could you please share the `lsusb -v` output for the device (you
> > can limit it to a specific device with `-d VID:PID`), running as root
> > (e.g. with `sudo`) if possible ?
> 
> Note here that the idVendor and iManufacturer do not match. The
> camera uses the same USB VID:PID as the Netchip "Gadget Zero", because
> of course it does…

Of course... Thank you for the descriptors, even if they make me want to
cry...

> $ sudo lsusb -v -d 0525:a4a0
> Bus 001 Device 019: ID 0525:a4a0 Netchip Technology, Inc. Linux-USB 
> "Gadget Zero"
> Device Descriptor:
>    bLength                18
>    bDescriptorType         1
>    bcdUSB               2.01
>    bDeviceClass            0
>    bDeviceSubClass         0
>    bDeviceProtocol         0
>    bMaxPacketSize0        64
>    idVendor           0x0525 Netchip Technology, Inc.
>    idProduct          0xa4a0 Linux-USB "Gadget Zero"
>    bcdDevice            2.00
>    iManufacturer           1 Wuhan Guide Infrared Co., Ltd.
>    iProduct                2 MobIR Air
>    iSerial                 3 ZX01A19
>    bNumConfigurations      1
>    Configuration Descriptor:
>      bLength                 9
>      bDescriptorType         2
>      wTotalLength       0x0040
>      bNumInterfaces          2
>      bConfigurationValue     1
>      iConfiguration          4 MFI Configure
>      bmAttributes         0xc0
>        Self Powered
>      MaxPower              100mA
>      Interface Descriptor:
>        bLength                 9
>        bDescriptorType         4
>        bInterfaceNumber        0
>        bAlternateSetting       0
>        bNumEndpoints           2
>        bInterfaceClass       255 Vendor Specific Class
>        bInterfaceSubClass    240
>        bInterfaceProtocol      0
>        iInterface              5 iAP Interface
>        Endpoint Descriptor:
>          bLength                 7
>          bDescriptorType         5
>          bEndpointAddress     0x82  EP 2 IN
>          bmAttributes            2
>            Transfer Type            Bulk
>            Synch Type               None
>            Usage Type               Data
>          wMaxPacketSize     0x0040  1x 64 bytes
>          bInterval               0
>        Endpoint Descriptor:
>          bLength                 7
>          bDescriptorType         5
>          bEndpointAddress     0x02  EP 2 OUT
>          bmAttributes            2
>            Transfer Type            Bulk
>            Synch Type               None
>            Usage Type               Data
>          wMaxPacketSize     0x0040  1x 64 bytes
>          bInterval               0
>      Interface Descriptor:
>        bLength                 9
>        bDescriptorType         4
>        bInterfaceNumber        1
>        bAlternateSetting       0
>        bNumEndpoints           0
>        bInterfaceClass       255 Vendor Specific Class
>        bInterfaceSubClass    240
>        bInterfaceProtocol      1
>        iInterface              6 com.guidesensmart.mobir-air
>      Interface Descriptor:
>        bLength                 9
>        bDescriptorType         4
>        bInterfaceNumber        1
>        bAlternateSetting       1
>        bNumEndpoints           2
>        bInterfaceClass       255 Vendor Specific Class
>        bInterfaceSubClass    240
>        bInterfaceProtocol      1
>        iInterface              6 com.guidesensmart.mobir-air
>        Endpoint Descriptor:
>          bLength                 7
>          bDescriptorType         5
>          bEndpointAddress     0x81  EP 1 IN
>          bmAttributes            2
>            Transfer Type            Bulk
>            Synch Type               None
>            Usage Type               Data
>          wMaxPacketSize     0x0040  1x 64 bytes
>          bInterval               0
>        Endpoint Descriptor:
>          bLength                 7
>          bDescriptorType         5
>          bEndpointAddress     0x01  EP 1 OUT
>          bmAttributes            2
>            Transfer Type            Bulk
>            Synch Type               None
>            Usage Type               Data
>          wMaxPacketSize     0x0040  1x 64 bytes
>          bInterval               0
> Binary Object Store Descriptor:
>    bLength                 5
>    bDescriptorType        15
>    wTotalLength       0x000c
>    bNumDeviceCaps          1
>    USB 2.0 Extension Device Capability:
>      bLength                 7
>      bDescriptorType        16
>      bDevCapabilityType      2
>      bmAttributes   0x00000002
>        HIRD Link Power Management (LPM) Supported
> Device Status:     0x0001
>    Self Powered
> 

-- 
Regards,

Laurent Pinchart