* Potential integration of thermal cameras into v4l @ 2023-01-07 21:58 Sophie Friedrich 2023-01-10 8:46 ` Pekka Paalanen 0 siblings, 1 reply; 7+ messages in thread From: Sophie Friedrich @ 2023-01-07 21:58 UTC (permalink / raw) To: linux-media Hello people, over the past months I've written a driver for the Guide MobIR Air¹ USB-C thermal camera. The driver uses v4l2loopback and runs in user space with python.² The driver manages camera state, calibration and conversion of the raw microbolometer to temperature values. In my research I couldn't find any mentions of a thermal camera support in the Linux kernel nor the media subsystem, so I assume I'm the first to ask here and we would presumably need lay out some ground work. The current camera driver outputs the temperature in the Y16 format as Kelvin * 100 (i.e. 28056 (raw) -> 280.56K -> 7.41°C). This allows for further processing down the line with i.e. OpenCV, but has its limitations. Especially as temps of only up to 382.2°C can be displayed currently. Does it seem feasible to integrate support for thermal cameras in the Linux kernel and more specifically in the v4l2 subsystem? - Sophie [1]: https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html [2]: https://github.com/tyalie/pyMobirAir-v4l2/ ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential integration of thermal cameras into v4l 2023-01-07 21:58 Potential integration of thermal cameras into v4l Sophie Friedrich @ 2023-01-10 8:46 ` Pekka Paalanen 2023-01-10 11:45 ` Jacopo Mondi 0 siblings, 1 reply; 7+ messages in thread From: Pekka Paalanen @ 2023-01-10 8:46 UTC (permalink / raw) To: Sophie Friedrich; +Cc: linux-media [-- Attachment #1: Type: text/plain, Size: 1922 bytes --] On Sat, 7 Jan 2023 22:58:41 +0100 Sophie Friedrich <lkml@flowerpot.me> wrote: > Hello people, > > over the past months I've written a driver for the Guide MobIR Air¹ > USB-C thermal camera. The driver uses v4l2loopback and runs in user > space with python.² The driver manages camera state, calibration and > conversion of the raw microbolometer to temperature values. > > In my research I couldn't find any mentions of a thermal camera support > in the Linux kernel nor the media subsystem, so I assume I'm the first > to ask here and we would presumably need lay out some ground work. > > The current camera driver outputs the temperature in the Y16 format as > Kelvin * 100 (i.e. 28056 (raw) -> 280.56K -> 7.41°C). This allows for > further processing down the line with i.e. OpenCV, but has its > limitations. Especially as temps of only up to 382.2°C can be displayed > currently. > > Does it seem feasible to integrate support for thermal cameras in the > Linux kernel and more specifically in the v4l2 subsystem? Hi, since no-one else replied to you yet, I thought to mention my 2c (I don't really do camera stuff myself so far): Perhaps the best place is to reach out to the libcamera community: https://libcamera.org/ It sounds to me like you want to do some hardware-specific processing in userspace, and it might not be great to try to come up with a generic thermal camera API at the kernel UAPI level. That's where libcamera fits in. As for the pixel type, maybe use a floating-point type to avoid range/precision problems? E.g. DRM_FORMAT_R32F for application facing API. That format does not seem to exist yet, but it's trivial to add into kernel's drm_fourcc.h and should be well accepted IMO. Thanks, pq > > [1]: > https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html > [2]: https://github.com/tyalie/pyMobirAir-v4l2/ [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential integration of thermal cameras into v4l 2023-01-10 8:46 ` Pekka Paalanen @ 2023-01-10 11:45 ` Jacopo Mondi 2023-01-10 12:14 ` Sophie Tyalie 2023-01-10 13:13 ` Laurent Pinchart 0 siblings, 2 replies; 7+ messages in thread From: Jacopo Mondi @ 2023-01-10 11:45 UTC (permalink / raw) To: Pekka Paalanen; +Cc: Sophie Friedrich, linux-media, libcamera-devel [-- Attachment #1: Type: text/plain, Size: 2457 bytes --] Hi Sophie and Pekka On Tue, Jan 10, 2023 at 10:46:26AM +0200, Pekka Paalanen wrote: > On Sat, 7 Jan 2023 22:58:41 +0100 > Sophie Friedrich <lkml@flowerpot.me> wrote: > > > Hello people, > > > > over the past months I've written a driver for the Guide MobIR Air¹ > > USB-C thermal camera. The driver uses v4l2loopback and runs in user > > space with python.² The driver manages camera state, calibration and > > conversion of the raw microbolometer to temperature values. > > > > In my research I couldn't find any mentions of a thermal camera support > > in the Linux kernel nor the media subsystem, so I assume I'm the first > > to ask here and we would presumably need lay out some ground work. > > > > The current camera driver outputs the temperature in the Y16 format as > > Kelvin * 100 (i.e. 28056 (raw) -> 280.56K -> 7.41°C). This allows for > > further processing down the line with i.e. OpenCV, but has its > > limitations. Especially as temps of only up to 382.2°C can be displayed > > currently. > > > > Does it seem feasible to integrate support for thermal cameras in the > > Linux kernel and more specifically in the v4l2 subsystem? > > Hi, > > since no-one else replied to you yet, I thought to mention my 2c > (I don't really do camera stuff myself so far): > > Perhaps the best place is to reach out to the libcamera community: > https://libcamera.org/ > cc-ed the libcamera list I agree it would be interesting to better understand what you mean by driver here. The camera seems to be a UVC camera, does it deliver frames with the current UVC driver or do you need kernel patches to support it ? I would also be interested why it needs to go through v4l2loopback.. > It sounds to me like you want to do some hardware-specific > processing in userspace, and it might not be great to try to come > up with a generic thermal camera API at the kernel UAPI level. > That's where libcamera fits in. > > As for the pixel type, maybe use a floating-point type to avoid > range/precision problems? E.g. DRM_FORMAT_R32F for application > facing API. That format does not seem to exist yet, but it's > trivial to add into kernel's drm_fourcc.h and should be well > accepted IMO. > > > Thanks, > pq > > > > > [1]: > > https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html > > [2]: https://github.com/tyalie/pyMobirAir-v4l2/ This link is broken :) [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential integration of thermal cameras into v4l 2023-01-10 11:45 ` Jacopo Mondi @ 2023-01-10 12:14 ` Sophie Tyalie 2023-01-10 13:13 ` Laurent Pinchart 1 sibling, 0 replies; 7+ messages in thread From: Sophie Tyalie @ 2023-01-10 12:14 UTC (permalink / raw) To: Jacopo Mondi, Pekka Paalanen Cc: Sophie Friedrich, linux-media, libcamera-devel Hi Pekka and Jacopo, I'll not quote the whole exchange now, please tell me if this is an anti-pattern. On 10/01/2023 12:45, Jacopo Mondi wrote: > On Tue, Jan 10, 2023 at 10:46:26AM +0200, Pekka Paalanen wrote: >> Hi, >> >> since no-one else replied to you yet, I thought to mention my 2c >> (I don't really do camera stuff myself so far): >> >> Perhaps the best place is to reach out to the libcamera community: >> https://libcamera.org/ >> > > cc-ed the libcamera list Thank you. Maybe libcamera is the better place to work on this. > I agree it would be interesting to better understand what you mean by > driver here. > > The camera seems to be a UVC camera, does it deliver frames with the > current UVC driver or do you need kernel patches to support it ? I'm not fully familiar with the UVC driver, but I very much doubt that the camera speaks anything resembling the UVC protocol. Guide build an RPC interface over USB, which they use to probe the camera and retrieve data that is needed later on (manufacturing calibration files, temperature curves, available modis, …). Afterwards they tell the camera to send the raw frames on the same channel (by sending `StartX=1`). > I would also be interested why it needs to go through v4l2loopback.. I've currently used v4l2loopback in order to get the processed frames from Python into the v4l2 system, so that I can access it further on. There are probably better solutions, but it was one that I know and is easy to work with. An actual driver developed for v4l2 or libcamera would make use of the standard ways of course. >> It sounds to me like you want to do some hardware-specific >> processing in userspace, and it might not be great to try to come >> up with a generic thermal camera API at the kernel UAPI level. >> That's where libcamera fits in. This is probably true. I'm not sure if a generic thermal camera API is even feasible, as every manufacturer does their own thing fully outside of standards. Together with the Guide MobIR Air, we could also implement the Flir ONE USB camera which also already has an user space driver¹. But these two cameras work completely differently in many regards (even conversion of raw to thermal values). >> As for the pixel type, maybe use a floating-point type to avoid >> range/precision problems? E.g. DRM_FORMAT_R32F for application >> facing API. That format does not seem to exist yet, but it's >> trivial to add into kernel's drm_fourcc.h and should be well >> accepted IMO. >> >> Thanks, >> pq Yeah, I would have preferred a floating point type too, but v4l2loopback doesn't support one (yet). >>> [1]: >>> https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html >>> [2]: https://github.com/tyalie/pyMobirAir-v4l2/ > > This link is broken :) Yeah sorry. I forgot to set the Github repo to public. It should be fixed now. Thank you all, - Sophie [1]: https://github.com/fnoop/flirone-v4l2 ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential integration of thermal cameras into v4l 2023-01-10 11:45 ` Jacopo Mondi 2023-01-10 12:14 ` Sophie Tyalie @ 2023-01-10 13:13 ` Laurent Pinchart 2023-01-10 13:33 ` Sophie Friedrich 1 sibling, 1 reply; 7+ messages in thread From: Laurent Pinchart @ 2023-01-10 13:13 UTC (permalink / raw) To: Jacopo Mondi Cc: Pekka Paalanen, linux-media, libcamera-devel, Sophie Friedrich Hello, On Tue, Jan 10, 2023 at 12:45:18PM +0100, Jacopo Mondi via libcamera-devel wrote: > On Tue, Jan 10, 2023 at 10:46:26AM +0200, Pekka Paalanen wrote: > > On Sat, 7 Jan 2023 22:58:41 +0100 Sophie Friedrich wrote: > > > > > Hello people, > > > > > > over the past months I've written a driver for the Guide MobIR Air¹ > > > USB-C thermal camera. The driver uses v4l2loopback and runs in user > > > space with python.² The driver manages camera state, calibration and > > > conversion of the raw microbolometer to temperature values. > > > > > > In my research I couldn't find any mentions of a thermal camera support > > > in the Linux kernel nor the media subsystem, so I assume I'm the first > > > to ask here and we would presumably need lay out some ground work. > > > > > > The current camera driver outputs the temperature in the Y16 format as > > > Kelvin * 100 (i.e. 28056 (raw) -> 280.56K -> 7.41°C). This allows for > > > further processing down the line with i.e. OpenCV, but has its > > > limitations. Especially as temps of only up to 382.2°C can be displayed > > > currently. > > > > > > Does it seem feasible to integrate support for thermal cameras in the > > > Linux kernel and more specifically in the v4l2 subsystem? > > > > Hi, > > > > since no-one else replied to you yet, I thought to mention my 2c > > (I don't really do camera stuff myself so far): > > > > Perhaps the best place is to reach out to the libcamera community: > > https://libcamera.org/ > > cc-ed the libcamera list > > I agree it would be interesting to better understand what you mean by > driver here. > > The camera seems to be a UVC camera, does it deliver frames with the > current UVC driver or do you need kernel patches to support it ? From a quick glance at the github project linked below, it doesn't seem to be a UVC camera. Sophie, could you please share the `lsusb -v` output for the device (you can limit it to a specific device with `-d VID:PID`), running as root (e.g. with `sudo`) if possible ? > I would also be interested why it needs to go through v4l2loopback.. > > > It sounds to me like you want to do some hardware-specific > > processing in userspace, and it might not be great to try to come > > up with a generic thermal camera API at the kernel UAPI level. > > That's where libcamera fits in. > > > > As for the pixel type, maybe use a floating-point type to avoid > > range/precision problems? E.g. DRM_FORMAT_R32F for application > > facing API. That format does not seem to exist yet, but it's > > trivial to add into kernel's drm_fourcc.h and should be well > > accepted IMO. > > > > > [1]: https://www.guideir.com/products/mobileaccessories/mobirair/data_300.html > > > [2]: https://github.com/tyalie/pyMobirAir-v4l2/ > > This link is broken :) Works for me. -- Regards, Laurent Pinchart ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential integration of thermal cameras into v4l 2023-01-10 13:13 ` Laurent Pinchart @ 2023-01-10 13:33 ` Sophie Friedrich 2023-01-10 13:50 ` Laurent Pinchart 0 siblings, 1 reply; 7+ messages in thread From: Sophie Friedrich @ 2023-01-10 13:33 UTC (permalink / raw) To: Laurent Pinchart, Jacopo Mondi Cc: Pekka Paalanen, linux-media, libcamera-devel Hello, Yeah the link was broken before, but I already fixed it. I'm not sure if my reply actually reached the mailing list (it accidentally mixed up my sending adress). At least from the libcamera I got a bounce, as I'm not a member. On 10/01/2023 14:13, Laurent Pinchart wrote: > Sophie, could you please share the `lsusb -v` output for the device (you > can limit it to a specific device with `-d VID:PID`), running as root > (e.g. with `sudo`) if possible ? Note here that the idVendor and iManufacturer do not match. The camera uses the same USB VID:PID as the Netchip "Gadget Zero", because of course it does… $ sudo lsusb -v -d 0525:a4a0 Bus 001 Device 019: ID 0525:a4a0 Netchip Technology, Inc. Linux-USB "Gadget Zero" Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 2.01 bDeviceClass 0 bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 64 idVendor 0x0525 Netchip Technology, Inc. idProduct 0xa4a0 Linux-USB "Gadget Zero" bcdDevice 2.00 iManufacturer 1 Wuhan Guide Infrared Co., Ltd. iProduct 2 MobIR Air iSerial 3 ZX01A19 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 0x0040 bNumInterfaces 2 bConfigurationValue 1 iConfiguration 4 MFI Configure bmAttributes 0xc0 Self Powered MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 2 bInterfaceClass 255 Vendor Specific Class bInterfaceSubClass 240 bInterfaceProtocol 0 iInterface 5 iAP Interface Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x82 EP 2 IN bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0040 1x 64 bytes bInterval 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x02 EP 2 OUT bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0040 1x 64 bytes bInterval 0 Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 0 bNumEndpoints 0 bInterfaceClass 255 Vendor Specific Class bInterfaceSubClass 240 bInterfaceProtocol 1 iInterface 6 com.guidesensmart.mobir-air Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 1 bAlternateSetting 1 bNumEndpoints 2 bInterfaceClass 255 Vendor Specific Class bInterfaceSubClass 240 bInterfaceProtocol 1 iInterface 6 com.guidesensmart.mobir-air Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0040 1x 64 bytes bInterval 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x01 EP 1 OUT bmAttributes 2 Transfer Type Bulk Synch Type None Usage Type Data wMaxPacketSize 0x0040 1x 64 bytes bInterval 0 Binary Object Store Descriptor: bLength 5 bDescriptorType 15 wTotalLength 0x000c bNumDeviceCaps 1 USB 2.0 Extension Device Capability: bLength 7 bDescriptorType 16 bDevCapabilityType 2 bmAttributes 0x00000002 HIRD Link Power Management (LPM) Supported Device Status: 0x0001 Self Powered ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Potential integration of thermal cameras into v4l 2023-01-10 13:33 ` Sophie Friedrich @ 2023-01-10 13:50 ` Laurent Pinchart 0 siblings, 0 replies; 7+ messages in thread From: Laurent Pinchart @ 2023-01-10 13:50 UTC (permalink / raw) To: Sophie Friedrich Cc: Jacopo Mondi, Pekka Paalanen, linux-media, libcamera-devel Hi Sophie, On Tue, Jan 10, 2023 at 02:33:17PM +0100, Sophie Friedrich wrote: > Hello, > > Yeah the link was broken before, but I already fixed it. I'm not > sure if my reply actually reached the mailing list (it accidentally > mixed up my sending adress). At least from the libcamera I got a > bounce, as I'm not a member. I've accepted both e-mails manually, and added the two addresses to the approved senders. > On 10/01/2023 14:13, Laurent Pinchart wrote: > > Sophie, could you please share the `lsusb -v` output for the device (you > > can limit it to a specific device with `-d VID:PID`), running as root > > (e.g. with `sudo`) if possible ? > > Note here that the idVendor and iManufacturer do not match. The > camera uses the same USB VID:PID as the Netchip "Gadget Zero", because > of course it does… Of course... Thank you for the descriptors, even if they make me want to cry... > $ sudo lsusb -v -d 0525:a4a0 > Bus 001 Device 019: ID 0525:a4a0 Netchip Technology, Inc. Linux-USB > "Gadget Zero" > Device Descriptor: > bLength 18 > bDescriptorType 1 > bcdUSB 2.01 > bDeviceClass 0 > bDeviceSubClass 0 > bDeviceProtocol 0 > bMaxPacketSize0 64 > idVendor 0x0525 Netchip Technology, Inc. > idProduct 0xa4a0 Linux-USB "Gadget Zero" > bcdDevice 2.00 > iManufacturer 1 Wuhan Guide Infrared Co., Ltd. > iProduct 2 MobIR Air > iSerial 3 ZX01A19 > bNumConfigurations 1 > Configuration Descriptor: > bLength 9 > bDescriptorType 2 > wTotalLength 0x0040 > bNumInterfaces 2 > bConfigurationValue 1 > iConfiguration 4 MFI Configure > bmAttributes 0xc0 > Self Powered > MaxPower 100mA > Interface Descriptor: > bLength 9 > bDescriptorType 4 > bInterfaceNumber 0 > bAlternateSetting 0 > bNumEndpoints 2 > bInterfaceClass 255 Vendor Specific Class > bInterfaceSubClass 240 > bInterfaceProtocol 0 > iInterface 5 iAP Interface > Endpoint Descriptor: > bLength 7 > bDescriptorType 5 > bEndpointAddress 0x82 EP 2 IN > bmAttributes 2 > Transfer Type Bulk > Synch Type None > Usage Type Data > wMaxPacketSize 0x0040 1x 64 bytes > bInterval 0 > Endpoint Descriptor: > bLength 7 > bDescriptorType 5 > bEndpointAddress 0x02 EP 2 OUT > bmAttributes 2 > Transfer Type Bulk > Synch Type None > Usage Type Data > wMaxPacketSize 0x0040 1x 64 bytes > bInterval 0 > Interface Descriptor: > bLength 9 > bDescriptorType 4 > bInterfaceNumber 1 > bAlternateSetting 0 > bNumEndpoints 0 > bInterfaceClass 255 Vendor Specific Class > bInterfaceSubClass 240 > bInterfaceProtocol 1 > iInterface 6 com.guidesensmart.mobir-air > Interface Descriptor: > bLength 9 > bDescriptorType 4 > bInterfaceNumber 1 > bAlternateSetting 1 > bNumEndpoints 2 > bInterfaceClass 255 Vendor Specific Class > bInterfaceSubClass 240 > bInterfaceProtocol 1 > iInterface 6 com.guidesensmart.mobir-air > Endpoint Descriptor: > bLength 7 > bDescriptorType 5 > bEndpointAddress 0x81 EP 1 IN > bmAttributes 2 > Transfer Type Bulk > Synch Type None > Usage Type Data > wMaxPacketSize 0x0040 1x 64 bytes > bInterval 0 > Endpoint Descriptor: > bLength 7 > bDescriptorType 5 > bEndpointAddress 0x01 EP 1 OUT > bmAttributes 2 > Transfer Type Bulk > Synch Type None > Usage Type Data > wMaxPacketSize 0x0040 1x 64 bytes > bInterval 0 > Binary Object Store Descriptor: > bLength 5 > bDescriptorType 15 > wTotalLength 0x000c > bNumDeviceCaps 1 > USB 2.0 Extension Device Capability: > bLength 7 > bDescriptorType 16 > bDevCapabilityType 2 > bmAttributes 0x00000002 > HIRD Link Power Management (LPM) Supported > Device Status: 0x0001 > Self Powered > -- Regards, Laurent Pinchart ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-01-10 13:50 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-01-07 21:58 Potential integration of thermal cameras into v4l Sophie Friedrich 2023-01-10 8:46 ` Pekka Paalanen 2023-01-10 11:45 ` Jacopo Mondi 2023-01-10 12:14 ` Sophie Tyalie 2023-01-10 13:13 ` Laurent Pinchart 2023-01-10 13:33 ` Sophie Friedrich 2023-01-10 13:50 ` Laurent Pinchart
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.