i.MX6 status for IPU/VPU/GPU

i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

Hi there !

We have a custom board, based on i.MX6 SoC.
We are currently using Freescale's release of Linux, but this is a
3.10.17 kernel, and several drivers are lacking (adv7611 for instance)
or badly written (all the MXC part).
As we want to have nice things :) we would like to use a mainline
kernel, or at least a tree which can be mainlined.

It seems (#v4l told me so) that some people (Steeve :) ?) are working
on a rewriting of the IPU and all DRM part for i.MX6.
What is the current status (compared to Freescale's release maybe) ?
And what can we expect in a near future ? Maybe, how can we help too ?

Thanks,
JM

Re: i.MX6 status for IPU/VPU/GPU

[Robert Schwebel]

Hi,

On Mon, Jul 28, 2014 at 06:24:45PM +0200, Jean-Michel Hautbois wrote:
> We have a custom board, based on i.MX6 SoC.
> We are currently using Freescale's release of Linux, but this is a
> 3.10.17 kernel, and several drivers are lacking (adv7611 for instance)
> or badly written (all the MXC part).
> As we want to have nice things :) we would like to use a mainline
> kernel, or at least a tree which can be mainlined.
> 
> It seems (#v4l told me so) that some people (Steeve :) ?) are working
> on a rewriting of the IPU and all DRM part for i.MX6.
> What is the current status (compared to Freescale's release maybe) ?
> And what can we expect in a near future? Maybe, how can we help too ?

Pengutronix is continuously working on mainlining more parts of the
i.MX6 video and graphics subsystem, including the components you have
mentioned. We are posting patches here when they are ready for mainline
review.

Regards,
Robert (for commercial help, please contact me by email)
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

On 07/28/2014 11:59 AM, Robert Schwebel wrote:
> Hi,
>
> On Mon, Jul 28, 2014 at 06:24:45PM +0200, Jean-Michel Hautbois wrote:
>> We have a custom board, based on i.MX6 SoC.
>> We are currently using Freescale's release of Linux, but this is a
>> 3.10.17 kernel, and several drivers are lacking (adv7611 for instance)
>> or badly written (all the MXC part).
>> As we want to have nice things :) we would like to use a mainline
>> kernel, or at least a tree which can be mainlined.
>>
>> It seems (#v4l told me so) that some people (Steeve :) ?) are working
>> on a rewriting of the IPU and all DRM part for i.MX6.
>> What is the current status (compared to Freescale's release maybe) ?
>> And what can we expect in a near future? Maybe, how can we help too ?

Hi Jean-Michel,

I did post a v4l2 video capture driver for i.MX6 to linux-media.
The main complaint from Philip at Pengutronix is that it does not
support the media device framework.

The customer I am currently working for has no real interest in the
media controller API, and the driver I posted has all the features they
require, so any work I do to add that support to the driver would have
to be in my spare time, and I don't have much. If our customer were to
request and fund media control support, that would be ideal, but as it is
I can only spend limited time on it. So if you are interested in helping
out in the media device effort I can send what I have so far.

I have not provided any patches to i.MX6 DRM/KMS drivers. We have
developed new features (overlay plane global/local alpha, hardware gamma
correction, color-keying, and others) for for that component but haven't
posted them yet.

Steve

> Pengutronix is continuously working on mainlining more parts of the
> i.MX6 video and graphics subsystem, including the components you have
> mentioned. We are posting patches here when they are ready for mainline
> review.
>
> Regards,
> Robert (for commercial help, please contact me by email)

Re: i.MX6 status for IPU/VPU/GPU

[Tim Harvey]

On Mon, Jul 28, 2014 at 2:15 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote:
> On 07/28/2014 11:59 AM, Robert Schwebel wrote:
>> Hi,
>>
>> On Mon, Jul 28, 2014 at 06:24:45PM +0200, Jean-Michel Hautbois wrote:
>>> We have a custom board, based on i.MX6 SoC.
>>> We are currently using Freescale's release of Linux, but this is a
>>> 3.10.17 kernel, and several drivers are lacking (adv7611 for instance)
>>> or badly written (all the MXC part).
>>> As we want to have nice things :) we would like to use a mainline
>>> kernel, or at least a tree which can be mainlined.
>>>
>>> It seems (#v4l told me so) that some people (Steeve :) ?) are working
>>> on a rewriting of the IPU and all DRM part for i.MX6.
>>> What is the current status (compared to Freescale's release maybe) ?
>>> And what can we expect in a near future? Maybe, how can we help too ?
>
> Hi Jean-Michel,
>
> I did post a v4l2 video capture driver for i.MX6 to linux-media.
> The main complaint from Philip at Pengutronix is that it does not
> support the media device framework.

Philipp,

It is unfortunate that the lack of the media device framework is
holding back acceptance of Steve's patches. Is this something that can
be added later? Does your patchset which you posted for reference
resolve this issue and perhaps is something that everyone could agree
on for a starting point?

Regards,

Tim

>
> The customer I am currently working for has no real interest in the
> media controller API, and the driver I posted has all the features they
> require, so any work I do to add that support to the driver would have
> to be in my spare time, and I don't have much. If our customer were to
> request and fund media control support, that would be ideal, but as it is
> I can only spend limited time on it. So if you are interested in helping
> out in the media device effort I can send what I have so far.
>
> I have not provided any patches to i.MX6 DRM/KMS drivers. We have
> developed new features (overlay plane global/local alpha, hardware gamma
> correction, color-keying, and others) for for that component but haven't
> posted them yet.
>
> Steve
>
>> Pengutronix is continuously working on mainlining more parts of the
>> i.MX6 video and graphics subsystem, including the components you have
>> mentioned. We are posting patches here when they are ready for mainline
>> review.
>>
>> Regards,
>> Robert (for commercial help, please contact me by email)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: i.MX6 status for IPU/VPU/GPU

[Philipp Zabel]

Hi Tim,

Am Sonntag, den 03.08.2014, 23:14 -0700 schrieb Tim Harvey:
> Philipp,
> 
> It is unfortunate that the lack of the media device framework is
> holding back acceptance of Steve's patches. Is this something that can
> be added later? Does your patchset which you posted for reference
> resolve this issue and perhaps is something that everyone could agree
> on for a starting point?

We should take this step by step. First I'd like to get Steve's ipu-v3
series in, those don't have any major issues and are a prerequisite for
the media patches anyway.

The capture patches had a few more issues than just missing media device
support. But this is indeed the biggest one, especially where it
involves a userspace interface that we don't want to have to support in
the future.
My RFC series wasn't without problems either. I'll work on the IPU this
week and then post another RFC.

regards
Philipp

Re: i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

2014-08-04 13:54 GMT+02:00 Philipp Zabel <p.zabel@pengutronix.de>:
> Hi Tim,
>
> Am Sonntag, den 03.08.2014, 23:14 -0700 schrieb Tim Harvey:
>> Philipp,
>>
>> It is unfortunate that the lack of the media device framework is
>> holding back acceptance of Steve's patches. Is this something that can
>> be added later? Does your patchset which you posted for reference
>> resolve this issue and perhaps is something that everyone could agree
>> on for a starting point?
>
> We should take this step by step. First I'd like to get Steve's ipu-v3
> series in, those don't have any major issues and are a prerequisite for
> the media patches anyway.
>
> The capture patches had a few more issues than just missing media device
> support. But this is indeed the biggest one, especially where it
> involves a userspace interface that we don't want to have to support in
> the future.
> My RFC series wasn't without problems either. I'll work on the IPU this
> week and then post another RFC.

Are the capture patches using v4l2_async_notifier_register() ?
I can help integrating/testing all these patches, if you want...

JM

Re: i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

Hi Phillip,

2014-08-04 13:54 GMT+02:00 Philipp Zabel <p.zabel@pengutronix.de>:
> We should take this step by step. First I'd like to get Steve's ipu-v3
> series in, those don't have any major issues and are a prerequisite for
> the media patches anyway.
>
> The capture patches had a few more issues than just missing media device
> support. But this is indeed the biggest one, especially where it
> involves a userspace interface that we don't want to have to support in
> the future.
> My RFC series wasn't without problems either. I'll work on the IPU this
> week and then post another RFC.

Any news about this ? I saw your patchset from june 12th.
What is the current status of this RFC and is there a way to help
integrating/testing it ? Do you have a public git repository I can
fetch and merge in order to test ?


Thanks,
JM

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

On 08/27/2014 12:13 AM, Jean-Michel Hautbois wrote:
> Hi Phillip,
>
> 2014-08-04 13:54 GMT+02:00 Philipp Zabel <p.zabel@pengutronix.de>:
>> We should take this step by step. First I'd like to get Steve's ipu-v3
>> series in, those don't have any major issues and are a prerequisite for
>> the media patches anyway.
>>
>> The capture patches had a few more issues than just missing media device
>> support. But this is indeed the biggest one, especially where it
>> involves a userspace interface that we don't want to have to support in
>> the future.
>> My RFC series wasn't without problems either. I'll work on the IPU this
>> week and then post another RFC.
> Any news about this ? I saw your patchset from june 12th.
> What is the current status of this RFC and is there a way to help
> integrating/testing it ? Do you have a public git repository I can
> fetch and merge in order to test ?
>
>

Hi Jean-Michel, Phillip,

I've done some work on Philipp's June 12 patchset, converting
the CSI driver to a CSI subdev entity, and fixing some issues here
and there. This June 12 patchset doesn't appear to be a fully working
driver, Phillip correct me if I am wrong. I can post this work as it
exists, it is incomplete but compiles.

I've also worked out what I think is a workable video pipeline graph for i.MX,
suitable for defining the entities, pads, and links. Unfortunately I haven't
been able to spend as much time as I'd like on it.

The complete driver I posted to the list does have some minor issues
mostly suggested by Hans Verkuil (switch to new selection API instead
of cropping API for example). It is a full featured driver but it does not
implement the media device framework, i.e. user does not have direct
control of the video pipeline, rather the driver chooses the pipeline based
on the traditional inputs from user (video format and controls).

If there is interest I can submit another version of the traditional driver
to resolve the issues. But media device is a major rework, so I don't
know whether it would make sense to start from the traditional driver
and then implement media device on top later, since media device
is almost a complete rewrite.

Steve

Re: i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:

> Hi Jean-Michel, Phillip,

Hi Steve,

> I've done some work on Philipp's June 12 patchset, converting
> the CSI driver to a CSI subdev entity, and fixing some issues here
> and there. This June 12 patchset doesn't appear to be a fully working
> driver, Phillip correct me if I am wrong. I can post this work as it
> exists, it is incomplete but compiles.

Dos it compile against a 3.17-rc3 kernel :) ?

> I've also worked out what I think is a workable video pipeline graph for i.MX,
> suitable for defining the entities, pads, and links. Unfortunately I haven't
> been able to spend as much time as I'd like on it.

This is very interesting, do you have written this somewhere ?

> The complete driver I posted to the list does have some minor issues
> mostly suggested by Hans Verkuil (switch to new selection API instead
> of cropping API for example). It is a full featured driver but it does not
> implement the media device framework, i.e. user does not have direct
> control of the video pipeline, rather the driver chooses the pipeline based
> on the traditional inputs from user (video format and controls).
>
> If there is interest I can submit another version of the traditional driver
> to resolve the issues. But media device is a major rework, so I don't
> know whether it would make sense to start from the traditional driver
> and then implement media device on top later, since media device
> is almost a complete rewrite.

I, at least, am interested by this driver, even in its "traditionnal"
form :). If you don't want to submit it directly because this is not
using media controller, this is ok, you can provide me a git repo in
order to get it, or send a patchset.

JM

Re: i.MX6 status for IPU/VPU/GPU

[Hans Verkuil]

On 09/09/14 09:49, Jean-Michel Hautbois wrote:
> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
> 
>> Hi Jean-Michel, Phillip,
> 
> Hi Steve,
> 
>> I've done some work on Philipp's June 12 patchset, converting
>> the CSI driver to a CSI subdev entity, and fixing some issues here
>> and there. This June 12 patchset doesn't appear to be a fully working
>> driver, Phillip correct me if I am wrong. I can post this work as it
>> exists, it is incomplete but compiles.
> 
> Dos it compile against a 3.17-rc3 kernel :) ?
> 
>> I've also worked out what I think is a workable video pipeline graph for i.MX,
>> suitable for defining the entities, pads, and links. Unfortunately I haven't
>> been able to spend as much time as I'd like on it.
> 
> This is very interesting, do you have written this somewhere ?
> 
>> The complete driver I posted to the list does have some minor issues
>> mostly suggested by Hans Verkuil (switch to new selection API instead
>> of cropping API for example). It is a full featured driver but it does not
>> implement the media device framework, i.e. user does not have direct
>> control of the video pipeline, rather the driver chooses the pipeline based
>> on the traditional inputs from user (video format and controls).
>>
>> If there is interest I can submit another version of the traditional driver
>> to resolve the issues. But media device is a major rework, so I don't
>> know whether it would make sense to start from the traditional driver
>> and then implement media device on top later, since media device
>> is almost a complete rewrite.
> 
> I, at least, am interested by this driver, even in its "traditionnal"
> form :). If you don't want to submit it directly because this is not
> using media controller, this is ok, you can provide me a git repo in
> order to get it, or send a patchset.

Is it possible to create a staging driver? Even if there are bits missing,
having the code in the kernel as a staging driver would help a lot.

Regards,

	Hans

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

On 09/09/2014 12:52 AM, Hans Verkuil wrote:
> On 09/09/14 09:49, Jean-Michel Hautbois wrote:
>> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>>
>>> Hi Jean-Michel, Phillip,
>> Hi Steve,
>>
>>> I've done some work on Philipp's June 12 patchset, converting
>>> the CSI driver to a CSI subdev entity, and fixing some issues here
>>> and there. This June 12 patchset doesn't appear to be a fully working
>>> driver, Phillip correct me if I am wrong. I can post this work as it
>>> exists, it is incomplete but compiles.
>> Dos it compile against a 3.17-rc3 kernel :) ?
>>
>>> I've also worked out what I think is a workable video pipeline graph for i.MX,
>>> suitable for defining the entities, pads, and links. Unfortunately I haven't
>>> been able to spend as much time as I'd like on it.
>> This is very interesting, do you have written this somewhere ?
>>
>>> The complete driver I posted to the list does have some minor issues
>>> mostly suggested by Hans Verkuil (switch to new selection API instead
>>> of cropping API for example). It is a full featured driver but it does not
>>> implement the media device framework, i.e. user does not have direct
>>> control of the video pipeline, rather the driver chooses the pipeline based
>>> on the traditional inputs from user (video format and controls).
>>>
>>> If there is interest I can submit another version of the traditional driver
>>> to resolve the issues. But media device is a major rework, so I don't
>>> know whether it would make sense to start from the traditional driver
>>> and then implement media device on top later, since media device
>>> is almost a complete rewrite.
>> I, at least, am interested by this driver, even in its "traditionnal"
>> form :). If you don't want to submit it directly because this is not
>> using media controller, this is ok, you can provide me a git repo in
>> order to get it, or send a patchset.
> Is it possible to create a staging driver? Even if there are bits missing,
> having the code in the kernel as a staging driver would help a lot.

Hi Hans, that's a good idea. I can post it as a staging driver.

The capture driver does require more support in the i.MX IPU driver which is
not yet merged or proposed. Phillip has forwarded most of them to drm-next,
but there are a few more required. I need to post those patches to drm-next
as a first step.

I can start working on converting the driver to staging and addressing
the earlier issues, but posting it to media-tree will need to wait until the IPU
patches are merged, unless I include the IPU patches along with the capture
driver patchset.

Steve

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

Hi Jean-Michel,


On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>
>> Hi Jean-Michel, Phillip,
> Hi Steve,
>
>> I've done some work on Philipp's June 12 patchset, converting
>> the CSI driver to a CSI subdev entity, and fixing some issues here
>> and there. This June 12 patchset doesn't appear to be a fully working
>> driver, Phillip correct me if I am wrong. I can post this work as it
>> exists, it is incomplete but compiles.
> Dos it compile against a 3.17-rc3 kernel :) ?

No, not anymore, the original posted driver was against 3.16 IIRC.

>
>> I've also worked out what I think is a workable video pipeline graph for i.MX,
>> suitable for defining the entities, pads, and links. Unfortunately I haven't
>> been able to spend as much time as I'd like on it.
> This is very interesting, do you have written this somewhere ?

Yes, I'll try to find some time to create a pdf image.
>
>> The complete driver I posted to the list does have some minor issues
>> mostly suggested by Hans Verkuil (switch to new selection API instead
>> of cropping API for example). It is a full featured driver but it does not
>> implement the media device framework, i.e. user does not have direct
>> control of the video pipeline, rather the driver chooses the pipeline based
>> on the traditional inputs from user (video format and controls).
>>
>> If there is interest I can submit another version of the traditional driver
>> to resolve the issues. But media device is a major rework, so I don't
>> know whether it would make sense to start from the traditional driver
>> and then implement media device on top later, since media device
>> is almost a complete rewrite.
> I, at least, am interested by this driver, even in its "traditionnal"
> form :). If you don't want to submit it directly because this is not
> using media controller, this is ok, you can provide me a git repo in
> order to get it, or send a patchset.

I think I'll follow Hans' proposal and submit it again to media-tree as
a staging driver.

Steve

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>
>> The complete driver I posted to the list does have some minor issues
>> mostly suggested by Hans Verkuil (switch to new selection API instead
>> of cropping API for example). It is a full featured driver but it does not
>> implement the media device framework, i.e. user does not have direct
>> control of the video pipeline, rather the driver chooses the pipeline based
>> on the traditional inputs from user (video format and controls).
>>
>> If there is interest I can submit another version of the traditional driver
>> to resolve the issues. But media device is a major rework, so I don't
>> know whether it would make sense to start from the traditional driver
>> and then implement media device on top later, since media device
>> is almost a complete rewrite.
> I, at least, am interested by this driver, even in its "traditionnal"
> form :). If you don't want to submit it directly because this is not
> using media controller, this is ok, you can provide me a git repo in
> order to get it, or send a patchset.

Hi Jean-Michel, I forgot to mention I will be working on the staging
capture driver in a copy of the media-tree on github at:

git@github.com:slongerbeam/mediatree.git

Steve

Re: i.MX6 status for IPU/VPU/GPU

[Philipp Zabel]

Hi Steve,

Am Dienstag, den 09.09.2014, 09:12 -0700 schrieb Steve Longerbeam:
> Hi Jean-Michel,
> 
> 
> On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
> > 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
> >
> >> Hi Jean-Michel, Phillip,
> > Hi Steve,
> >
> >> I've done some work on Philipp's June 12 patchset, converting
> >> the CSI driver to a CSI subdev entity, and fixing some issues here
> >> and there. This June 12 patchset doesn't appear to be a fully working
> >> driver, Phillip correct me if I am wrong. I can post this work as it
> >> exists, it is incomplete but compiles.
> > Dos it compile against a 3.17-rc3 kernel :) ?
> 
> No, not anymore, the original posted driver was against 3.16 IIRC.
> 
> >
> >> I've also worked out what I think is a workable video pipeline graph for i.MX,
> >> suitable for defining the entities, pads, and links. Unfortunately I haven't
> >> been able to spend as much time as I'd like on it.
> > This is very interesting, do you have written this somewhere ?
> 
> Yes, I'll try to find some time to create a pdf image.

I'd be very interested in this, too. I have in the meantime started to
implement everything that has a source or destination selector in the
Frame Synchronization Unit (FSU) as media entity. I wonder which of
these parts should reasonably be unified into a single entity:

	CSI0
	CSI1
	SMFC0
	SMFC1
	SMFC2
	SMFC3
	IC preprocessor (input to VF and ENC, if I understood correctly)
	IC viewfinder task (scaling, csc)
	IC encoding task
	IC post processing task
	IRT viewfinder task (rotation)
	IRT encoding task
	IRT post processing task
	VDIC (deinterlacing, combining)
	(and probably some entry for DP/DC/DMFC for the direct
	 viewfinder path)

I suppose the SMFC channels need to be separate because they can belong
to different pipelines (and each entity can only belong to one).
The three IC task entities could probably be combined with their
corresponding IRT task entity somehow, but that would be at the cost of
not being able to tell the kernel whether to rotate before or after
scaling, which might be useful when handling chroma subsampled formats.

I have put my current state up here:

git://git.pengutronix.de/git/pza/linux.git test/nitrogen6x-ipu-media

So far I've captured video through the SMFC on a Nitrogen6X board with
OV5652 parallel camera with this.

> >> The complete driver I posted to the list does have some minor issues
> >> mostly suggested by Hans Verkuil (switch to new selection API instead
> >> of cropping API for example). It is a full featured driver but it does not
> >> implement the media device framework, i.e. user does not have direct
> >> control of the video pipeline, rather the driver chooses the pipeline based
> >> on the traditional inputs from user (video format and controls).
> >>
> >> If there is interest I can submit another version of the traditional driver
> >> to resolve the issues. But media device is a major rework, so I don't
> >> know whether it would make sense to start from the traditional driver
> >> and then implement media device on top later, since media device
> >> is almost a complete rewrite.
> > I, at least, am interested by this driver, even in its "traditionnal"
> > form :). If you don't want to submit it directly because this is not
> > using media controller, this is ok, you can provide me a git repo in
> > order to get it, or send a patchset.
> 
> I think I'll follow Hans' proposal and submit it again to media-tree as
> a staging driver.

I'm not too fond of adding a staging driver that we know will have to be
replaced. Maybe we could work together to get a media entity based
version up to speed?

regards
Philipp

Re: i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

2014-09-09 18:28 GMT+02:00 Steve Longerbeam <slongerbeam@gmail.com>:
> On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
>> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>>
>>> The complete driver I posted to the list does have some minor issues
>>> mostly suggested by Hans Verkuil (switch to new selection API instead
>>> of cropping API for example). It is a full featured driver but it does not
>>> implement the media device framework, i.e. user does not have direct
>>> control of the video pipeline, rather the driver chooses the pipeline based
>>> on the traditional inputs from user (video format and controls).
>>>
>>> If there is interest I can submit another version of the traditional driver
>>> to resolve the issues. But media device is a major rework, so I don't
>>> know whether it would make sense to start from the traditional driver
>>> and then implement media device on top later, since media device
>>> is almost a complete rewrite.
>> I, at least, am interested by this driver, even in its "traditionnal"
>> form :). If you don't want to submit it directly because this is not
>> using media controller, this is ok, you can provide me a git repo in
>> order to get it, or send a patchset.
>
> Hi Jean-Michel, I forgot to mention I will be working on the staging
> capture driver in a copy of the media-tree on github at:
>
> git@github.com:slongerbeam/mediatree.git
>

I took your mx6-camera-staging branch and merger it, but compile fails :
drivers/staging/media/imx6/capture/mx6-vdic.c:815:2: error: implicit
declaration of function 'ipu_mbus_code_to_fourcc'

Maybe isn't it ready yet ? I always want to go faster than music... :)

Thanks,
JM

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

On 09/10/2014 09:08 AM, Jean-Michel Hautbois wrote:
> 2014-09-09 18:28 GMT+02:00 Steve Longerbeam <slongerbeam@gmail.com>:
>> On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
>>> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>>>
>>>> The complete driver I posted to the list does have some minor issues
>>>> mostly suggested by Hans Verkuil (switch to new selection API instead
>>>> of cropping API for example). It is a full featured driver but it does not
>>>> implement the media device framework, i.e. user does not have direct
>>>> control of the video pipeline, rather the driver chooses the pipeline based
>>>> on the traditional inputs from user (video format and controls).
>>>>
>>>> If there is interest I can submit another version of the traditional driver
>>>> to resolve the issues. But media device is a major rework, so I don't
>>>> know whether it would make sense to start from the traditional driver
>>>> and then implement media device on top later, since media device
>>>> is almost a complete rewrite.
>>> I, at least, am interested by this driver, even in its "traditionnal"
>>> form :). If you don't want to submit it directly because this is not
>>> using media controller, this is ok, you can provide me a git repo in
>>> order to get it, or send a patchset.
>> Hi Jean-Michel, I forgot to mention I will be working on the staging
>> capture driver in a copy of the media-tree on github at:
>>
>> git@github.com:slongerbeam/mediatree.git
>>
> I took your mx6-camera-staging branch and merger it, but compile fails :
> drivers/staging/media/imx6/capture/mx6-vdic.c:815:2: error: implicit
> declaration of function 'ipu_mbus_code_to_fourcc'
>
> Maybe isn't it ready yet ? I always want to go faster than music... :)

Hi JM, yes I'm still working on it, I'll let you know when I think it's
in a good-enough state.

Steve

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

On 09/10/2014 09:25 AM, Steve Longerbeam wrote:
> On 09/10/2014 09:08 AM, Jean-Michel Hautbois wrote:
>> 2014-09-09 18:28 GMT+02:00 Steve Longerbeam <slongerbeam@gmail.com>:
>>> On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
>>>> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>>>>
>>>>> The complete driver I posted to the list does have some minor issues
>>>>> mostly suggested by Hans Verkuil (switch to new selection API instead
>>>>> of cropping API for example). It is a full featured driver but it does not
>>>>> implement the media device framework, i.e. user does not have direct
>>>>> control of the video pipeline, rather the driver chooses the pipeline based
>>>>> on the traditional inputs from user (video format and controls).
>>>>>
>>>>> If there is interest I can submit another version of the traditional driver
>>>>> to resolve the issues. But media device is a major rework, so I don't
>>>>> know whether it would make sense to start from the traditional driver
>>>>> and then implement media device on top later, since media device
>>>>> is almost a complete rewrite.
>>>> I, at least, am interested by this driver, even in its "traditionnal"
>>>> form :). If you don't want to submit it directly because this is not
>>>> using media controller, this is ok, you can provide me a git repo in
>>>> order to get it, or send a patchset.
>>> Hi Jean-Michel, I forgot to mention I will be working on the staging
>>> capture driver in a copy of the media-tree on github at:
>>>
>>> git@github.com:slongerbeam/mediatree.git
>>>
>> I took your mx6-camera-staging branch and merger it, but compile fails :
>> drivers/staging/media/imx6/capture/mx6-vdic.c:815:2: error: implicit
>> declaration of function 'ipu_mbus_code_to_fourcc'
>>
>> Maybe isn't it ready yet ? I always want to go faster than music... :)
> Hi JM, yes I'm still working on it, I'll let you know when I think it's
> in a good-enough state.

Hi JM, staging capture driver now compiles at
git@github.com:slongerbeam/mediatree.git, mx6-camera-staging
branch.

It is lightly tested on sabrelite quad (with parallel ov5642), sabreauto quad
(with adv7180 and NTSC/PAL sources), and sabresd quad (with MIPI CSI-2
ov5640).

I have not yet made most of the suggested changes from my version 1 posting
a couple months ago, so for the most part it is in the same state. Still in the process
of making those suggested changes.

Steve

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

Hi Phillip,

On 09/09/2014 10:40 AM, Philipp Zabel wrote:
>
>>>> I've also worked out what I think is a workable video pipeline graph for i.MX,
>>>> suitable for defining the entities, pads, and links. Unfortunately I haven't
>>>> been able to spend as much time as I'd like on it.
>>> This is very interesting, do you have written this somewhere ?
>> Yes, I'll try to find some time to create a pdf image.
> I'd be very interested in this, too.

I should have something to show tomorrow.

>  I have in the meantime started to
> implement everything that has a source or destination selector in the
> Frame Synchronization Unit (FSU) as media entity. I wonder which of
> these parts should reasonably be unified into a single entity:
>
> 	CSI0
> 	CSI1

Yes, we need a CSI subdev/entity, and it can be instantiated twice for
the two CSI ports.

> 	SMFC0
> 	SMFC1
> 	SMFC2
> 	SMFC3

I don't really see the need for an SMFC entity. The SMFC control can
be integrated into the CSI subdev.

> 	IC preprocessor (input to VF and ENC, if I understood correctly)
> 	IC viewfinder task (scaling, csc)
> 	IC encoding task
> 	IC post processing task

I see either three different IC subdev entities (IC prpenc, IC prpvf,
IC pp), or a single IC entity with three sink pads for each IC task.

> 	IRT viewfinder task (rotation)
> 	IRT encoding task
> 	IRT post processing task

well, the IRT is really just a submodule enable bit, I see no need
for an IRT subdev, in fact IRT has already been folded into ipu-ic.c
as a simple submodule enable/disable. Rotation support can be
implemented as part of the IC entities.

> 	VDIC (deinterlacing, combining)

I am thinking VDIC support can be part of the IC prpvf entity (well,
combining is not really on my radar, I haven't given that much thought).

> 	(and probably some entry for DP/DC/DMFC for the direct
> 	 viewfinder path)

Ugh, I've been ignoring that path as well. Freescale's BSP releases
and sample code from their SDK's have no example code for the
direct-to-DP/DC/DMFC camera viewfinder path, so given the quality
of the imx TRM, this could be a challenge to implement. Have you
gotten this path to work?

>
> I suppose the SMFC channels need to be separate because they can belong
> to different pipelines (and each entity can only belong to one).

I see the chosen SMFC channel as an internal decision by the
CSI subdev.

> The three IC task entities could probably be combined with their
> corresponding IRT task entity somehow, but that would be at the cost of
> not being able to tell the kernel whether to rotate before or after
> scaling, which might be useful when handling chroma subsampled formats.

I'm fairly sure IC rotation must always occur _after_ scaling. I.e.
raw frames are first passed through IC prpenc/prpvf/pp for scaling/CSC,
then EOF completion of that task is hardware linked to IRT.

>
> I have put my current state up here:
>
> git://git.pengutronix.de/git/pza/linux.git test/nitrogen6x-ipu-media
>
> So far I've captured video through the SMFC on a Nitrogen6X board with
> OV5652 parallel camera with this.

Thanks Phillip, I'll take a look! Sounds like a good place to start.
I assume this is with the video mux entity and CSI driver? I.e. no
IC entity support yet for scaling, CSC, or rotation.

Steve

Re: i.MX6 status for IPU/VPU/GPU

[Philipp Zabel]

Hi Steve,

Am Mittwoch, den 10.09.2014, 18:17 -0700 schrieb Steve Longerbeam:
[...]
> On 09/09/2014 10:40 AM, Philipp Zabel wrote:
[...]
> >  I have in the meantime started to
> > implement everything that has a source or destination selector in the
> > Frame Synchronization Unit (FSU) as media entity. I wonder which of
> > these parts should reasonably be unified into a single entity:
[...]
> > 	SMFC0
> > 	SMFC1
> > 	SMFC2
> > 	SMFC3
> 
> I don't really see the need for an SMFC entity. The SMFC control can
> be integrated into the CSI subdev.

Granted, this is currently is a theoretical question, but could we
handle a single MIPI link that carries two or more virtual channels with
different MIPI IDs this way?

> > 	IC preprocessor (input to VF and ENC, if I understood correctly)
> > 	IC viewfinder task (scaling, csc)
> > 	IC encoding task
> > 	IC post processing task
> 
> I see either three different IC subdev entities (IC prpenc, IC prpvf,
> IC pp), or a single IC entity with three sink pads for each IC task.

The former could work, the latter won't allow to have pre and post
processing on separate pipelines.

> > 	IRT viewfinder task (rotation)
> > 	IRT encoding task
> > 	IRT post processing task
> 
> well, the IRT is really just a submodule enable bit, I see no need
> for an IRT subdev, in fact IRT has already been folded into ipu-ic.c
> as a simple submodule enable/disable. Rotation support can be
> implemented as part of the IC entities.

My current understanding is that the IRT is strictly a mem2mem device
using its own DMA channels, which can be channel-linked to the IC (and
other blocks) in various ways.

> > 	VDIC (deinterlacing, combining)
> 
> I am thinking VDIC support can be part of the IC prpvf entity (well,
> combining is not really on my radar, I haven't given that much thought).
> 
> > 	(and probably some entry for DP/DC/DMFC for the direct
> > 	 viewfinder path)
> 
> Ugh, I've been ignoring that path as well. Freescale's BSP releases
> and sample code from their SDK's have no example code for the
> direct-to-DP/DC/DMFC camera viewfinder path, so given the quality
> of the imx TRM, this could be a challenge to implement. Have you
> gotten this path to work?

Not yet, no.

> > I suppose the SMFC channels need to be separate because they can belong
> > to different pipelines (and each entity can only belong to one).
> 
> I see the chosen SMFC channel as an internal decision by the
> CSI subdev.

Can we handle multiple outputs from a single CSI this way?

> > The three IC task entities could probably be combined with their
> > corresponding IRT task entity somehow, but that would be at the cost of
> > not being able to tell the kernel whether to rotate before or after
> > scaling, which might be useful when handling chroma subsampled formats.
> 
> I'm fairly sure IC rotation must always occur _after_ scaling. I.e.
> raw frames are first passed through IC prpenc/prpvf/pp for scaling/CSC,
> then EOF completion of that task is hardware linked to IRT.

There could be good reasons to do the rotation on the input side, for
example when upscaling or when the output is 4:2:2 subsampled. At least
the FSU registers suggest that channel linking the rotator before the IC
is possible. This probably won't be useful for the capture path in most
cases, but it might be for rotated playback.

> > I have put my current state up here:
> >
> > git://git.pengutronix.de/git/pza/linux.git test/nitrogen6x-ipu-media
> >
> > So far I've captured video through the SMFC on a Nitrogen6X board with
> > OV5652 parallel camera with this.
> 
> Thanks Phillip, I'll take a look! Sounds like a good place to start.
> I assume this is with the video mux entity and CSI driver? I.e. no
> IC entity support yet for scaling, CSC, or rotation.

Yes, exactly.

regards
Philipp

Re: i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

2014-09-11 15:26 GMT+02:00 Philipp Zabel <p.zabel@pengutronix.de>:
> Hi Steve,
>
> Am Mittwoch, den 10.09.2014, 18:17 -0700 schrieb Steve Longerbeam:
> [...]
>> On 09/09/2014 10:40 AM, Philipp Zabel wrote:
> [...]
>> >  I have in the meantime started to
>> > implement everything that has a source or destination selector in the
>> > Frame Synchronization Unit (FSU) as media entity. I wonder which of
>> > these parts should reasonably be unified into a single entity:
> [...]
>> >     SMFC0
>> >     SMFC1
>> >     SMFC2
>> >     SMFC3
>>
>> I don't really see the need for an SMFC entity. The SMFC control can
>> be integrated into the CSI subdev.
>
> Granted, this is currently is a theoretical question, but could we
> handle a single MIPI link that carries two or more virtual channels with
> different MIPI IDs this way?
>
>> >     IC preprocessor (input to VF and ENC, if I understood correctly)
>> >     IC viewfinder task (scaling, csc)
>> >     IC encoding task
>> >     IC post processing task
>>
>> I see either three different IC subdev entities (IC prpenc, IC prpvf,
>> IC pp), or a single IC entity with three sink pads for each IC task.
>
> The former could work, the latter won't allow to have pre and post
> processing on separate pipelines.
>
>> >     IRT viewfinder task (rotation)
>> >     IRT encoding task
>> >     IRT post processing task
>>
>> well, the IRT is really just a submodule enable bit, I see no need
>> for an IRT subdev, in fact IRT has already been folded into ipu-ic.c
>> as a simple submodule enable/disable. Rotation support can be
>> implemented as part of the IC entities.
>
> My current understanding is that the IRT is strictly a mem2mem device
> using its own DMA channels, which can be channel-linked to the IC (and
> other blocks) in various ways.
>
>> >     VDIC (deinterlacing, combining)
>>
>> I am thinking VDIC support can be part of the IC prpvf entity (well,
>> combining is not really on my radar, I haven't given that much thought).
>>
>> >     (and probably some entry for DP/DC/DMFC for the direct
>> >      viewfinder path)
>>
>> Ugh, I've been ignoring that path as well. Freescale's BSP releases
>> and sample code from their SDK's have no example code for the
>> direct-to-DP/DC/DMFC camera viewfinder path, so given the quality
>> of the imx TRM, this could be a challenge to implement. Have you
>> gotten this path to work?
>
> Not yet, no.
>
>> > I suppose the SMFC channels need to be separate because they can belong
>> > to different pipelines (and each entity can only belong to one).
>>
>> I see the chosen SMFC channel as an internal decision by the
>> CSI subdev.
>
> Can we handle multiple outputs from a single CSI this way?
>
>> > The three IC task entities could probably be combined with their
>> > corresponding IRT task entity somehow, but that would be at the cost of
>> > not being able to tell the kernel whether to rotate before or after
>> > scaling, which might be useful when handling chroma subsampled formats.
>>
>> I'm fairly sure IC rotation must always occur _after_ scaling. I.e.
>> raw frames are first passed through IC prpenc/prpvf/pp for scaling/CSC,
>> then EOF completion of that task is hardware linked to IRT.
>
> There could be good reasons to do the rotation on the input side, for
> example when upscaling or when the output is 4:2:2 subsampled. At least
> the FSU registers suggest that channel linking the rotator before the IC
> is possible. This probably won't be useful for the capture path in most
> cases, but it might be for rotated playback.
>
>> > I have put my current state up here:
>> >
>> > git://git.pengutronix.de/git/pza/linux.git test/nitrogen6x-ipu-media
>> >
>> > So far I've captured video through the SMFC on a Nitrogen6X board with
>> > OV5652 parallel camera with this.
>>
>> Thanks Phillip, I'll take a look! Sounds like a good place to start.
>> I assume this is with the video mux entity and CSI driver? I.e. no
>> IC entity support yet for scaling, CSC, or rotation.
>
> Yes, exactly.

I have tried both of your branches (Steve and Philip) and they both
are interesting.
I easily added adv76xx devices to Steve's work, but there is no media
controller support, as you previously said.
I cannot get adv7611 working on Philip's branch. I tried to do the
same as your "add OV5642 parallel camera" commit, but I don't see a
link between csi and adv even though I asked for it in DT (I removed
not useful stuff in the following paste) :

&i2c2 {
    hdmiin1: adv7611@4c {
            port {
                    hdmi0_out: endpoint@1 {
                            reg = <1>;
                            remote_endpoint = <&csi0_from_adv7611>;
                    };
    };
};

&csi0 {
        csi0_from_adv7611: endpoint {
                remote_endpoint = <&hdmi0_out>;
        };
};

Is there something specific that needs to be done in order to get the
link on boot ?

Even if this is still WIP, this is great, it helps me a lot :) !
Thanks,
JM

Re: i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

Hi Steve,

2014-09-09 18:28 GMT+02:00 Steve Longerbeam <slongerbeam@gmail.com>:
> On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
>> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>>
>>> The complete driver I posted to the list does have some minor issues
>>> mostly suggested by Hans Verkuil (switch to new selection API instead
>>> of cropping API for example). It is a full featured driver but it does not
>>> implement the media device framework, i.e. user does not have direct
>>> control of the video pipeline, rather the driver chooses the pipeline based
>>> on the traditional inputs from user (video format and controls).

Here is my first step toward MC support from your work :
https://github.com/Vodalys/linux-2.6-imx/commit/8f0318f53c48a9638a1963b395bc79fbd7ba4c07

This is a WIP, so some parts of code are commented out awaiting a
nicer solution.
I also keep using your eplist array for the moment, and open will
obviously fail when trying to power sensor.
But what I wanted was a complete MC support with parsing links from DT
and I used Laurent's work intensively :).

>>> I've also worked out what I think is a workable video pipeline graph for i.MX,
>>> suitable for defining the entities, pads, and links. Unfortunately I haven't
>>> been able to spend as much time as I'd like on it.

Did you find some time to write the pdf you mentioned ?

Thanks for your work again,
JM

Re: i.MX6 status for IPU/VPU/GPU

[Carlos Sanmartín Bustos]

Sorry for the resend, I forget send in plain text.
Hi all,

I'm interested in this driver with MC support too. I join the
conversation and if I have time can try to develop some functionality.

Only one question:

2014-10-02 16:50 GMT+02:00 Jean-Michel Hautbois
<jean-michel.hautbois@vodalys.com>:
>
> Hi Steve,
>
> 2014-09-09 18:28 GMT+02:00 Steve Longerbeam <slongerbeam@gmail.com>:
> > On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
> >> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
> >>
> >>> The complete driver I posted to the list does have some minor issues
> >>> mostly suggested by Hans Verkuil (switch to new selection API instead
> >>> of cropping API for example). It is a full featured driver but it does not
> >>> implement the media device framework, i.e. user does not have direct
> >>> control of the video pipeline, rather the driver chooses the pipeline based
> >>> on the traditional inputs from user (video format and controls).
>
> Here is my first step toward MC support from your work :
> https://github.com/Vodalys/linux-2.6-imx/commit/8f0318f53c48a9638a1963b395bc79fbd7ba4c07
>
> This is a WIP, so some parts of code are commented out awaiting a
> nicer solution.
> I also keep using your eplist array for the moment, and open will
> obviously fail when trying to power sensor.
> But what I wanted was a complete MC support with parsing links from DT
> and I used Laurent's work intensively :).
>

You are forking the Freescale linux-2.6-imx repository if I understood
well. Why not fork the linux-media repository? It's closer to mainline
kernel I think it's better.

Regards,
Carlos

2014-10-02 16:50 GMT+02:00 Jean-Michel Hautbois
<jean-michel.hautbois@vodalys.com>:
> Hi Steve,
>
> 2014-09-09 18:28 GMT+02:00 Steve Longerbeam <slongerbeam@gmail.com>:
>> On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
>>> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>>>
>>>> The complete driver I posted to the list does have some minor issues
>>>> mostly suggested by Hans Verkuil (switch to new selection API instead
>>>> of cropping API for example). It is a full featured driver but it does not
>>>> implement the media device framework, i.e. user does not have direct
>>>> control of the video pipeline, rather the driver chooses the pipeline based
>>>> on the traditional inputs from user (video format and controls).
>
> Here is my first step toward MC support from your work :
> https://github.com/Vodalys/linux-2.6-imx/commit/8f0318f53c48a9638a1963b395bc79fbd7ba4c07
>
> This is a WIP, so some parts of code are commented out awaiting a
> nicer solution.
> I also keep using your eplist array for the moment, and open will
> obviously fail when trying to power sensor.
> But what I wanted was a complete MC support with parsing links from DT
> and I used Laurent's work intensively :).
>
>>>> I've also worked out what I think is a workable video pipeline graph for i.MX,
>>>> suitable for defining the entities, pads, and links. Unfortunately I haven't
>>>> been able to spend as much time as I'd like on it.
>
> Did you find some time to write the pdf you mentioned ?
>
> Thanks for your work again,
> JM
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

2014-10-03 12:16 GMT+02:00 Carlos Sanmartín Bustos <carsanbu@gmail.com>:
> Hi all,
>
> I'm interested in this driver with MC support too. I join the conversation
> and if I have time can try to develop some functionality.
>
> Only one question:
>
> 2014-10-02 16:50 GMT+02:00 Jean-Michel Hautbois
> <jean-michel.hautbois@vodalys.com>:
>>
>> Hi Steve,
>>
>> 2014-09-09 18:28 GMT+02:00 Steve Longerbeam <slongerbeam@gmail.com>:
>> > On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
>> >> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam
>> >> <steve_longerbeam@mentor.com>:
>> >>
>> >>> The complete driver I posted to the list does have some minor issues
>> >>> mostly suggested by Hans Verkuil (switch to new selection API instead
>> >>> of cropping API for example). It is a full featured driver but it does
>> >>> not
>> >>> implement the media device framework, i.e. user does not have direct
>> >>> control of the video pipeline, rather the driver chooses the pipeline
>> >>> based
>> >>> on the traditional inputs from user (video format and controls).
>>
>> Here is my first step toward MC support from your work :
>>
>> https://github.com/Vodalys/linux-2.6-imx/commit/8f0318f53c48a9638a1963b395bc79fbd7ba4c07
>>
>> This is a WIP, so some parts of code are commented out awaiting a
>> nicer solution.
>> I also keep using your eplist array for the moment, and open will
>> obviously fail when trying to power sensor.
>> But what I wanted was a complete MC support with parsing links from DT
>> and I used Laurent's work intensively :).
>>
>
> You are forking the Freescale linux-2.6-imx repository if I understood well.
> Why not fork the linux-media repository? It's closer to mainline kernel I
> think it's better.

Well, this is kind of a mess in this github :).
But the branch indicated is a clone of vanilla, not on linux-2.6-imx
repository. I should have done a new repository, sorry for the
confusion.

JM

Re: i.MX6 status for IPU/VPU/GPU

[Steve Longerbeam]

On 10/02/2014 07:50 AM, Jean-Michel Hautbois wrote:
> Hi Steve,
> 
> 2014-09-09 18:28 GMT+02:00 Steve Longerbeam <slongerbeam@gmail.com>:
>> On 09/09/2014 12:49 AM, Jean-Michel Hautbois wrote:
>>> 2014-08-27 16:23 GMT+02:00 Steve Longerbeam <steve_longerbeam@mentor.com>:
>>>
>>>> The complete driver I posted to the list does have some minor issues
>>>> mostly suggested by Hans Verkuil (switch to new selection API instead
>>>> of cropping API for example). It is a full featured driver but it does not
>>>> implement the media device framework, i.e. user does not have direct
>>>> control of the video pipeline, rather the driver chooses the pipeline based
>>>> on the traditional inputs from user (video format and controls).
> 
> Here is my first step toward MC support from your work :
> https://github.com/Vodalys/linux-2.6-imx/commit/8f0318f53c48a9638a1963b395bc79fbd7ba4c07
> 
> This is a WIP, so some parts of code are commented out awaiting a
> nicer solution.
> I also keep using your eplist array for the moment, and open will
> obviously fail when trying to power sensor.
> But what I wanted was a complete MC support with parsing links from DT
> and I used Laurent's work intensively :).

Hi Jean-Michel,

Ok thanks for the work, I will try to find time to study it.


> 
>>>> I've also worked out what I think is a workable video pipeline graph for i.MX,
>>>> suitable for defining the entities, pads, and links. Unfortunately I haven't
>>>> been able to spend as much time as I'd like on it.
> 
> Did you find some time to write the pdf you mentioned ?

Finally did. I will send it directly to you, as I'm sure it will get
stripped here.

Steve

Re: i.MX6 status for IPU/VPU/GPU

[Jean-Michel Hautbois]

Hi Philipp,

2014-09-15 16:13 GMT+02:00 Jean-Michel Hautbois
<jean-michel.hautbois@vodalys.com>:
> 2014-09-11 15:26 GMT+02:00 Philipp Zabel <p.zabel@pengutronix.de>:
>> Hi Steve,
>>
>> Am Mittwoch, den 10.09.2014, 18:17 -0700 schrieb Steve Longerbeam:
>> [...]
>>> On 09/09/2014 10:40 AM, Philipp Zabel wrote:
>> [...]
>>> >  I have in the meantime started to
>>> > implement everything that has a source or destination selector in the
>>> > Frame Synchronization Unit (FSU) as media entity. I wonder which of
>>> > these parts should reasonably be unified into a single entity:
>> [...]
>>> >     SMFC0
>>> >     SMFC1
>>> >     SMFC2
>>> >     SMFC3
>>>
>>> I don't really see the need for an SMFC entity. The SMFC control can
>>> be integrated into the CSI subdev.
>>
>> Granted, this is currently is a theoretical question, but could we
>> handle a single MIPI link that carries two or more virtual channels with
>> different MIPI IDs this way?
>>
>>> >     IC preprocessor (input to VF and ENC, if I understood correctly)
>>> >     IC viewfinder task (scaling, csc)
>>> >     IC encoding task
>>> >     IC post processing task
>>>
>>> I see either three different IC subdev entities (IC prpenc, IC prpvf,
>>> IC pp), or a single IC entity with three sink pads for each IC task.
>>
>> The former could work, the latter won't allow to have pre and post
>> processing on separate pipelines.
>>
>>> >     IRT viewfinder task (rotation)
>>> >     IRT encoding task
>>> >     IRT post processing task
>>>
>>> well, the IRT is really just a submodule enable bit, I see no need
>>> for an IRT subdev, in fact IRT has already been folded into ipu-ic.c
>>> as a simple submodule enable/disable. Rotation support can be
>>> implemented as part of the IC entities.
>>
>> My current understanding is that the IRT is strictly a mem2mem device
>> using its own DMA channels, which can be channel-linked to the IC (and
>> other blocks) in various ways.
>>
>>> >     VDIC (deinterlacing, combining)
>>>
>>> I am thinking VDIC support can be part of the IC prpvf entity (well,
>>> combining is not really on my radar, I haven't given that much thought).
>>>
>>> >     (and probably some entry for DP/DC/DMFC for the direct
>>> >      viewfinder path)
>>>
>>> Ugh, I've been ignoring that path as well. Freescale's BSP releases
>>> and sample code from their SDK's have no example code for the
>>> direct-to-DP/DC/DMFC camera viewfinder path, so given the quality
>>> of the imx TRM, this could be a challenge to implement. Have you
>>> gotten this path to work?
>>
>> Not yet, no.
>>
>>> > I suppose the SMFC channels need to be separate because they can belong
>>> > to different pipelines (and each entity can only belong to one).
>>>
>>> I see the chosen SMFC channel as an internal decision by the
>>> CSI subdev.
>>
>> Can we handle multiple outputs from a single CSI this way?
>>
>>> > The three IC task entities could probably be combined with their
>>> > corresponding IRT task entity somehow, but that would be at the cost of
>>> > not being able to tell the kernel whether to rotate before or after
>>> > scaling, which might be useful when handling chroma subsampled formats.
>>>
>>> I'm fairly sure IC rotation must always occur _after_ scaling. I.e.
>>> raw frames are first passed through IC prpenc/prpvf/pp for scaling/CSC,
>>> then EOF completion of that task is hardware linked to IRT.
>>
>> There could be good reasons to do the rotation on the input side, for
>> example when upscaling or when the output is 4:2:2 subsampled. At least
>> the FSU registers suggest that channel linking the rotator before the IC
>> is possible. This probably won't be useful for the capture path in most
>> cases, but it might be for rotated playback.
>>
>>> > I have put my current state up here:
>>> >
>>> > git://git.pengutronix.de/git/pza/linux.git test/nitrogen6x-ipu-media
>>> >
>>> > So far I've captured video through the SMFC on a Nitrogen6X board with
>>> > OV5652 parallel camera with this.
>>>
>>> Thanks Phillip, I'll take a look! Sounds like a good place to start.
>>> I assume this is with the video mux entity and CSI driver? I.e. no
>>> IC entity support yet for scaling, CSC, or rotation.
>>
>> Yes, exactly.
>
> I have tried both of your branches (Steve and Philip) and they both
> are interesting.
> I easily added adv76xx devices to Steve's work, but there is no media
> controller support, as you previously said.
> I cannot get adv7611 working on Philip's branch. I tried to do the
> same as your "add OV5642 parallel camera" commit, but I don't see a
> link between csi and adv even though I asked for it in DT (I removed
> not useful stuff in the following paste) :
>
> &i2c2 {
>     hdmiin1: adv7611@4c {
>             port {
>                     hdmi0_out: endpoint@1 {
>                             reg = <1>;
>                             remote_endpoint = <&csi0_from_adv7611>;
>                     };
>     };
> };
>
> &csi0 {
>         csi0_from_adv7611: endpoint {
>                 remote_endpoint = <&hdmi0_out>;
>         };
> };
>
> Is there something specific that needs to be done in order to get the
> link on boot ?

I get back to this question, as I can't get your
test/nitrogen6x-ipu-media branch using my adv7611 device.
And in fact, I tried to draw the topology from media-ctl, and dot
crashes in SIGSEGV when I do this, I don't know if this is linked to
the way the topology is done in the driver ?

Can you also give an example of how you captured the video through the
camera ? Have you used gstreamer ?
If so, did you use the CODA encoder you wrote too ?
I am very interested by this particular question.

Thanks again for your work,

Regards,
JM

Re: i.MX6 status for IPU/VPU/GPU

[Carlos Sanmartín Bustos]

Hi Jean-Michel,

Long time no news of this. There are any progress?

I am interested in this. Can any of you send me the pdf of pipeline? I
want to take a look.

Regards,

Carlos S.

2014-10-24 15:42 GMT+02:00 Jean-Michel Hautbois
<jean-michel.hautbois@vodalys.com>:
> Hi Philipp,
>
> 2014-09-15 16:13 GMT+02:00 Jean-Michel Hautbois
> <jean-michel.hautbois@vodalys.com>:
>> 2014-09-11 15:26 GMT+02:00 Philipp Zabel <p.zabel@pengutronix.de>:
>>> Hi Steve,
>>>
>>> Am Mittwoch, den 10.09.2014, 18:17 -0700 schrieb Steve Longerbeam:
>>> [...]
>>>> On 09/09/2014 10:40 AM, Philipp Zabel wrote:
>>> [...]
>>>> >  I have in the meantime started to
>>>> > implement everything that has a source or destination selector in the
>>>> > Frame Synchronization Unit (FSU) as media entity. I wonder which of
>>>> > these parts should reasonably be unified into a single entity:
>>> [...]
>>>> >     SMFC0
>>>> >     SMFC1
>>>> >     SMFC2
>>>> >     SMFC3
>>>>
>>>> I don't really see the need for an SMFC entity. The SMFC control can
>>>> be integrated into the CSI subdev.
>>>
>>> Granted, this is currently is a theoretical question, but could we
>>> handle a single MIPI link that carries two or more virtual channels with
>>> different MIPI IDs this way?
>>>
>>>> >     IC preprocessor (input to VF and ENC, if I understood correctly)
>>>> >     IC viewfinder task (scaling, csc)
>>>> >     IC encoding task
>>>> >     IC post processing task
>>>>
>>>> I see either three different IC subdev entities (IC prpenc, IC prpvf,
>>>> IC pp), or a single IC entity with three sink pads for each IC task.
>>>
>>> The former could work, the latter won't allow to have pre and post
>>> processing on separate pipelines.
>>>
>>>> >     IRT viewfinder task (rotation)
>>>> >     IRT encoding task
>>>> >     IRT post processing task
>>>>
>>>> well, the IRT is really just a submodule enable bit, I see no need
>>>> for an IRT subdev, in fact IRT has already been folded into ipu-ic.c
>>>> as a simple submodule enable/disable. Rotation support can be
>>>> implemented as part of the IC entities.
>>>
>>> My current understanding is that the IRT is strictly a mem2mem device
>>> using its own DMA channels, which can be channel-linked to the IC (and
>>> other blocks) in various ways.
>>>
>>>> >     VDIC (deinterlacing, combining)
>>>>
>>>> I am thinking VDIC support can be part of the IC prpvf entity (well,
>>>> combining is not really on my radar, I haven't given that much thought).
>>>>
>>>> >     (and probably some entry for DP/DC/DMFC for the direct
>>>> >      viewfinder path)
>>>>
>>>> Ugh, I've been ignoring that path as well. Freescale's BSP releases
>>>> and sample code from their SDK's have no example code for the
>>>> direct-to-DP/DC/DMFC camera viewfinder path, so given the quality
>>>> of the imx TRM, this could be a challenge to implement. Have you
>>>> gotten this path to work?
>>>
>>> Not yet, no.
>>>
>>>> > I suppose the SMFC channels need to be separate because they can belong
>>>> > to different pipelines (and each entity can only belong to one).
>>>>
>>>> I see the chosen SMFC channel as an internal decision by the
>>>> CSI subdev.
>>>
>>> Can we handle multiple outputs from a single CSI this way?
>>>
>>>> > The three IC task entities could probably be combined with their
>>>> > corresponding IRT task entity somehow, but that would be at the cost of
>>>> > not being able to tell the kernel whether to rotate before or after
>>>> > scaling, which might be useful when handling chroma subsampled formats.
>>>>
>>>> I'm fairly sure IC rotation must always occur _after_ scaling. I.e.
>>>> raw frames are first passed through IC prpenc/prpvf/pp for scaling/CSC,
>>>> then EOF completion of that task is hardware linked to IRT.
>>>
>>> There could be good reasons to do the rotation on the input side, for
>>> example when upscaling or when the output is 4:2:2 subsampled. At least
>>> the FSU registers suggest that channel linking the rotator before the IC
>>> is possible. This probably won't be useful for the capture path in most
>>> cases, but it might be for rotated playback.
>>>
>>>> > I have put my current state up here:
>>>> >
>>>> > git://git.pengutronix.de/git/pza/linux.git test/nitrogen6x-ipu-media
>>>> >
>>>> > So far I've captured video through the SMFC on a Nitrogen6X board with
>>>> > OV5652 parallel camera with this.
>>>>
>>>> Thanks Phillip, I'll take a look! Sounds like a good place to start.
>>>> I assume this is with the video mux entity and CSI driver? I.e. no
>>>> IC entity support yet for scaling, CSC, or rotation.
>>>
>>> Yes, exactly.
>>
>> I have tried both of your branches (Steve and Philip) and they both
>> are interesting.
>> I easily added adv76xx devices to Steve's work, but there is no media
>> controller support, as you previously said.
>> I cannot get adv7611 working on Philip's branch. I tried to do the
>> same as your "add OV5642 parallel camera" commit, but I don't see a
>> link between csi and adv even though I asked for it in DT (I removed
>> not useful stuff in the following paste) :
>>
>> &i2c2 {
>>     hdmiin1: adv7611@4c {
>>             port {
>>                     hdmi0_out: endpoint@1 {
>>                             reg = <1>;
>>                             remote_endpoint = <&csi0_from_adv7611>;
>>                     };
>>     };
>> };
>>
>> &csi0 {
>>         csi0_from_adv7611: endpoint {
>>                 remote_endpoint = <&hdmi0_out>;
>>         };
>> };
>>
>> Is there something specific that needs to be done in order to get the
>> link on boot ?
>
> I get back to this question, as I can't get your
> test/nitrogen6x-ipu-media branch using my adv7611 device.
> And in fact, I tried to draw the topology from media-ctl, and dot
> crashes in SIGSEGV when I do this, I don't know if this is linked to
> the way the topology is done in the driver ?
>
> Can you also give an example of how you captured the video through the
> camera ? Have you used gstreamer ?
> If so, did you use the CODA encoder you wrote too ?
> I am very interested by this particular question.
>
> Thanks again for your work,
>
> Regards,
> JM
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html