cpufreq implementation for OMAP under xen hypervisor.

cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

Hi to all.

I want to implement an cpufreq support for OMAP processors in xen. I
use the Linux kernel as Dom0.

I know that there are 2 implementations of cpufreq: Domain0 based
cpufreq and Hypervisor based cpufreq. But those implementations are
made only for x86 architecture, not for the ARM architecture.

Could anybody give me an advise how to do that?

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Tue, 12 Aug 2014, Oleksandr Dmytryshyn wrote:
> Hi to all.
> 
> I want to implement an cpufreq support for OMAP processors in xen. I
> use the Linux kernel as Dom0.
> 
> I know that there are 2 implementations of cpufreq: Domain0 based
> cpufreq and Hypervisor based cpufreq. But those implementations are
> made only for x86 architecture, not for the ARM architecture.
> 
> Could anybody give me an advise how to do that?

I think that the best way would be to introduce cpufreq support directly
in the Xen hypervisor.  I would:

1) get cpufreq (xen/drivers/cpufreq) to build and run on ARM, with a dummy driver
although the cpufreq infrastructure has been written with x86 in mind,
it should be generalizable

2) write a cpufreq driver for OMAP

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

Hi, Stefano.

Thank You for the quick reply.

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt


On Tue, Aug 12, 2014 at 3:15 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 12 Aug 2014, Oleksandr Dmytryshyn wrote:
>> Hi to all.
>>
>> I want to implement an cpufreq support for OMAP processors in xen. I
>> use the Linux kernel as Dom0.
>>
>> I know that there are 2 implementations of cpufreq: Domain0 based
>> cpufreq and Hypervisor based cpufreq. But those implementations are
>> made only for x86 architecture, not for the ARM architecture.
>>
>> Could anybody give me an advise how to do that?
>
> I think that the best way would be to introduce cpufreq support directly
> in the Xen hypervisor.  I would:
>
> 1) get cpufreq (xen/drivers/cpufreq) to build and run on ARM, with a dummy driver
> although the cpufreq infrastructure has been written with x86 in mind,
> it should be generalizable
>
> 2) write a cpufreq driver for OMAP

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly V. Ch]

[-- Attachment #1.1: Type: text/plain, Size: 1239 bytes --]

Hi Stefano,

What do you think about implementation cpufreq driver as stub driver domain?

Just now I'm not familar enought with latest XEN fetures so my question can
be a little stupid :)

With best regards,

Vitaly


On Tue, Aug 12, 2014 at 3:15 PM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:

> On Tue, 12 Aug 2014, Oleksandr Dmytryshyn wrote:
> > Hi to all.
> >
> > I want to implement an cpufreq support for OMAP processors in xen. I
> > use the Linux kernel as Dom0.
> >
> > I know that there are 2 implementations of cpufreq: Domain0 based
> > cpufreq and Hypervisor based cpufreq. But those implementations are
> > made only for x86 architecture, not for the ARM architecture.
> >
> > Could anybody give me an advise how to do that?
>
> I think that the best way would be to introduce cpufreq support directly
> in the Xen hypervisor.  I would:
>
> 1) get cpufreq (xen/drivers/cpufreq) to build and run on ARM, with a dummy
> driver
> although the cpufreq infrastructure has been written with x86 in mind,
> it should be generalizable
>
> 2) write a cpufreq driver for OMAP
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 1977 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

Hi to all.

I'm planning to do next work:

1. Move file xen/include/acpi/cpufreq/cpufreq.h to the
xen/include/drivers/cpufreq/cpufreq.h
2. Create a new file xen/arch/x86/acpi/cpufreq/cpufreq_common.c
3. Move some acpi-specific functions from
xen/drivers/cpufreq/cpufreq.c to the
xen/arch/x86/acpi/cpufreq/cpufreq_common.c:
cpufreq_limit_change(), print_PCT(), print_PSS(), print_PSD(),
print_PPC(), set_px_pminfo().
4. Create a new file xen/arch/arm/cpufreq/cpufreq_common.c
5. Functions cpufreq_add_cpu()/cpufreq_del_cpu() should be implemented
separately for the x86 and ARM architecture (in the correspond file
cpufreq_common.c).
6. Port cpufreq driver for the OMAP from the Linux kernel.

In case ARM the cpufreq driver will read the settings for the
operating-points from the device tree and the
XENPF_set_processor_pminfo platform hypercall will not be necessary
for ARM.

Is this the right way to implement the cpufreq for OMAP under xen hypervisor?

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt


On Tue, Aug 12, 2014 at 3:15 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 12 Aug 2014, Oleksandr Dmytryshyn wrote:
>> Hi to all.
>>
>> I want to implement an cpufreq support for OMAP processors in xen. I
>> use the Linux kernel as Dom0.
>>
>> I know that there are 2 implementations of cpufreq: Domain0 based
>> cpufreq and Hypervisor based cpufreq. But those implementations are
>> made only for x86 architecture, not for the ARM architecture.
>>
>> Could anybody give me an advise how to do that?
>
> I think that the best way would be to introduce cpufreq support directly
> in the Xen hypervisor.  I would:
>
> 1) get cpufreq (xen/drivers/cpufreq) to build and run on ARM, with a dummy driver
> although the cpufreq infrastructure has been written with x86 in mind,
> it should be generalizable
>
> 2) write a cpufreq driver for OMAP

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]

It is not stupid, it is actually possible to do it that way.

It would be similar to do cpufreq in dom0: anything you can run in Dom0,
you can also run separately in a little stubdom.

However I think that it would be best to do it in Xen.

On Tue, 19 Aug 2014, Vitaly V. Ch wrote:
> Hi Stefano,
> 
> What do you think about implementation cpufreq driver as stub driver domain?
> 
> Just now I'm not familar enought with latest XEN fetures so my question can be a little stupid :)
> 
> With best regards,
> 
> Vitaly
> 
> 
> On Tue, Aug 12, 2014 at 3:15 PM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
>       On Tue, 12 Aug 2014, Oleksandr Dmytryshyn wrote:
>       > Hi to all.
>       >
>       > I want to implement an cpufreq support for OMAP processors in xen. I
>       > use the Linux kernel as Dom0.
>       >
>       > I know that there are 2 implementations of cpufreq: Domain0 based
>       > cpufreq and Hypervisor based cpufreq. But those implementations are
>       > made only for x86 architecture, not for the ARM architecture.
>       >
>       > Could anybody give me an advise how to do that?
> 
> I think that the best way would be to introduce cpufreq support directly
> in the Xen hypervisor.  I would:
> 
> 1) get cpufreq (xen/drivers/cpufreq) to build and run on ARM, with a dummy driver
> although the cpufreq infrastructure has been written with x86 in mind,
> it should be generalizable
> 
> 2) write a cpufreq driver for OMAP
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 
> 
> 
> 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

CC'ing the x86 maintainers and the cpufreq original author.


On Thu, 21 Aug 2014, Oleksandr Dmytryshyn wrote:
> Hi to all.
> 
> I'm planning to do next work:
> 
> 1. Move file xen/include/acpi/cpufreq/cpufreq.h to the
> xen/include/drivers/cpufreq/cpufreq.h
> 2. Create a new file xen/arch/x86/acpi/cpufreq/cpufreq_common.c

you can call it cpufreq.c or cpufreq_ops.c


> 3. Move some acpi-specific functions from
> xen/drivers/cpufreq/cpufreq.c to the
> xen/arch/x86/acpi/cpufreq/cpufreq_common.c:
> cpufreq_limit_change(), print_PCT(), print_PSS(), print_PSD(),
> print_PPC(), set_px_pminfo().

Why cpufreq_limit_change?


> 4. Create a new file xen/arch/arm/cpufreq/cpufreq_common.c
> 5. Functions cpufreq_add_cpu()/cpufreq_del_cpu() should be implemented
> separately for the x86 and ARM architecture (in the correspond file
> cpufreq_common.c).

Why? The implementation doesn't look x86 specific.


> 6. Port cpufreq driver for the OMAP from the Linux kernel.
> 
> In case ARM the cpufreq driver will read the settings for the
> operating-points from the device tree and the
> XENPF_set_processor_pminfo platform hypercall will not be necessary
> for ARM.
> 
> Is this the right way to implement the cpufreq for OMAP under xen hypervisor?

Yes, it's more or less what I had in mind.


> Oleksandr Dmytryshyn | Product Engineering and Development
> GlobalLogic
> M +38.067.382.2525
> www.globallogic.com
> 
> http://www.globallogic.com/email_disclaimer.txt
> 
> 
> On Tue, Aug 12, 2014 at 3:15 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Tue, 12 Aug 2014, Oleksandr Dmytryshyn wrote:
> >> Hi to all.
> >>
> >> I want to implement an cpufreq support for OMAP processors in xen. I
> >> use the Linux kernel as Dom0.
> >>
> >> I know that there are 2 implementations of cpufreq: Domain0 based
> >> cpufreq and Hypervisor based cpufreq. But those implementations are
> >> made only for x86 architecture, not for the ARM architecture.
> >>
> >> Could anybody give me an advise how to do that?
> >
> > I think that the best way would be to introduce cpufreq support directly
> > in the Xen hypervisor.  I would:
> >
> > 1) get cpufreq (xen/drivers/cpufreq) to build and run on ARM, with a dummy driver
> > although the cpufreq infrastructure has been written with x86 in mind,
> > it should be generalizable
> >
> > 2) write a cpufreq driver for OMAP
> 

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

Hi, Stefano.

On Fri, Aug 22, 2014 at 2:31 AM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> CC'ing the x86 maintainers and the cpufreq original author.
>
>
> On Thu, 21 Aug 2014, Oleksandr Dmytryshyn wrote:
>> Hi to all.
>>
>> I'm planning to do next work:
>>
>> 1. Move file xen/include/acpi/cpufreq/cpufreq.h to the
>> xen/include/drivers/cpufreq/cpufreq.h
>> 2. Create a new file xen/arch/x86/acpi/cpufreq/cpufreq_common.c
>
> you can call it cpufreq.c or cpufreq_ops.c
I'll call it cpufreq_ops.c

>> 3. Move some acpi-specific functions from
>> xen/drivers/cpufreq/cpufreq.c to the
>> xen/arch/x86/acpi/cpufreq/cpufreq_common.c:
>> cpufreq_limit_change(), print_PCT(), print_PSS(), print_PSD(),
>> print_PPC(), set_px_pminfo().
>
> Why cpufreq_limit_change?
The function cpufreq_limit_change() is called only in the set_px_pminfo()
function and will not be used for the ARM architecture.

>> 4. Create a new file xen/arch/arm/cpufreq/cpufreq_common.c
>> 5. Functions cpufreq_add_cpu()/cpufreq_del_cpu() should be implemented
>> separately for the x86 and ARM architecture (in the correspond file
>> cpufreq_common.c).
>
> Why? The implementation doesn't look x86 specific.
The function cpufreq_add_cpu()/cpufreq_del_cpu() uses the acpi-specific
data structures. I don't know how make them common for both architectures
but I'll try to do this.

>> 6. Port cpufreq driver for the OMAP from the Linux kernel.
>>
>> In case ARM the cpufreq driver will read the settings for the
>> operating-points from the device tree and the
>> XENPF_set_processor_pminfo platform hypercall will not be necessary
>> for ARM.
>>
>> Is this the right way to implement the cpufreq for OMAP under xen hypervisor?
>
> Yes, it's more or less what I had in mind.

I have a question. I see that the original file cpufreq.c contains
'Copyright (C)' fields. Could You please tell me which copyrights I should add
to the new cpufreq_ops.c files (for x86 and ARM arhitectures).
In case if cpufreq_add_cpu()/cpufreq_del_cpu() functions will be common for both
architectures there will be only one cpufreq_ops.c file for x86 architecture.
But there is a way when we will have two files file cpufreq_ops.c (for
x86 and ARM).



-- 
>> Oleksandr Dmytryshyn | Product Engineering and Development
>> GlobalLogic
>> M +38.067.382.2525
>> www.globallogic.com
>>
>> http://www.globallogic.com/email_disclaimer.txt
>>
>>
>> On Tue, Aug 12, 2014 at 3:15 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Tue, 12 Aug 2014, Oleksandr Dmytryshyn wrote:
>> >> Hi to all.
>> >>
>> >> I want to implement an cpufreq support for OMAP processors in xen. I
>> >> use the Linux kernel as Dom0.
>> >>
>> >> I know that there are 2 implementations of cpufreq: Domain0 based
>> >> cpufreq and Hypervisor based cpufreq. But those implementations are
>> >> made only for x86 architecture, not for the ARM architecture.
>> >>
>> >> Could anybody give me an advise how to do that?
>> >
>> > I think that the best way would be to introduce cpufreq support directly
>> > in the Xen hypervisor.  I would:
>> >
>> > 1) get cpufreq (xen/drivers/cpufreq) to build and run on ARM, with a dummy driver
>> > although the cpufreq infrastructure has been written with x86 in mind,
>> > it should be generalizable
>> >
>> > 2) write a cpufreq driver for OMAP
>>
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly Chernooky]

[-- Attachment #1.1: Type: text/plain, Size: 2517 bytes --]

On Fri, Aug 22, 2014 at 2:05 AM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:

> It is not stupid, it is actually possible to do it that way.
>
> It would be similar to do cpufreq in dom0: anything you can run in Dom0,
> you can also run separately in a little stubdom.
>
> However I think that it would be best to do it in Xen.
>

Ok, what the reason is to prefer to do cpufreq in Xen?

With driver domain we can once provide stable binary ABI for cpufreq and do
not have any confusion with sources, patches and branches. If we will do
cpufreq in Xen I expect in the nearest future large amount of patches for
cpufreq for wide variety of ARM platforms.

With best regards,


>
> On Tue, 19 Aug 2014, Vitaly V. Ch wrote:
> > Hi Stefano,
> >
> > What do you think about implementation cpufreq driver as stub driver
> domain?
> >
> > Just now I'm not familar enought with latest XEN fetures so my question
> can be a little stupid :)
> >
> > With best regards,
> >
> > Vitaly
> >
> >
> > On Tue, Aug 12, 2014 at 3:15 PM, Stefano Stabellini <
> stefano.stabellini@eu.citrix.com> wrote:
> >       On Tue, 12 Aug 2014, Oleksandr Dmytryshyn wrote:
> >       > Hi to all.
> >       >
> >       > I want to implement an cpufreq support for OMAP processors in
> xen. I
> >       > use the Linux kernel as Dom0.
> >       >
> >       > I know that there are 2 implementations of cpufreq: Domain0 based
> >       > cpufreq and Hypervisor based cpufreq. But those implementations
> are
> >       > made only for x86 architecture, not for the ARM architecture.
> >       >
> >       > Could anybody give me an advise how to do that?
> >
> > I think that the best way would be to introduce cpufreq support directly
> > in the Xen hypervisor.  I would:
> >
> > 1) get cpufreq (xen/drivers/cpufreq) to build and run on ARM, with a
> dummy driver
> > although the cpufreq infrastructure has been written with x86 in mind,
> > it should be generalizable
> >
> > 2) write a cpufreq driver for OMAP
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> >
> >
> >
> >
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
>


-- 
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 4038 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Fri, 22 Aug 2014, Vitaly Chernooky wrote:
> On Fri, Aug 22, 2014 at 2:05 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
>       It is not stupid, it is actually possible to do it that way.
> 
>       It would be similar to do cpufreq in dom0: anything you can run in Dom0,
>       you can also run separately in a little stubdom.
> 
>       However I think that it would be best to do it in Xen.
> 
> 
> Ok, what the reason is to prefer to do cpufreq in Xen?
> 
> With driver domain we can once provide stable binary ABI for cpufreq and do not have any confusion with sources,
> patches and branches. If we will do cpufreq in Xen I expect in the nearest future large amount of patches for cpufreq
> for wide variety of ARM platforms.
 
The reason is that cpu virtualization is done by Xen. The guest
operating system doesn't actually know anything about physical cpus,
because it is running on virtual cpus.

So it is OK to let a guest operating system manage one or more physical
devices. On the other hand letting it manage one ore more physical cpus
is more of a layering violation.

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Fri, 22 Aug 2014, Oleksandr Dmytryshyn wrote:
> Hi, Stefano.
> 
> On Fri, Aug 22, 2014 at 2:31 AM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > CC'ing the x86 maintainers and the cpufreq original author.
> >
> >
> > On Thu, 21 Aug 2014, Oleksandr Dmytryshyn wrote:
> >> Hi to all.
> >>
> >> I'm planning to do next work:
> >>
> >> 1. Move file xen/include/acpi/cpufreq/cpufreq.h to the
> >> xen/include/drivers/cpufreq/cpufreq.h
> >> 2. Create a new file xen/arch/x86/acpi/cpufreq/cpufreq_common.c
> >
> > you can call it cpufreq.c or cpufreq_ops.c
> I'll call it cpufreq_ops.c
> 
> >> 3. Move some acpi-specific functions from
> >> xen/drivers/cpufreq/cpufreq.c to the
> >> xen/arch/x86/acpi/cpufreq/cpufreq_common.c:
> >> cpufreq_limit_change(), print_PCT(), print_PSS(), print_PSD(),
> >> print_PPC(), set_px_pminfo().
> >
> > Why cpufreq_limit_change?
> The function cpufreq_limit_change() is called only in the set_px_pminfo()
> function and will not be used for the ARM architecture.

I see.
One thing to keep in mind is that although P states are obviously an
Intel concept, we could abstract them away and map them into
arch-independent power-saving states. That way we could share functions
like set_px_pminfo between ARM and x86. But I would have to see the
patches to actually know how feasible that is.


> >> 4. Create a new file xen/arch/arm/cpufreq/cpufreq_common.c
> >> 5. Functions cpufreq_add_cpu()/cpufreq_del_cpu() should be implemented
> >> separately for the x86 and ARM architecture (in the correspond file
> >> cpufreq_common.c).
> >
> > Why? The implementation doesn't look x86 specific.
> The function cpufreq_add_cpu()/cpufreq_del_cpu() uses the acpi-specific
> data structures. I don't know how make them common for both architectures
> but I'll try to do this.
> 
> >> 6. Port cpufreq driver for the OMAP from the Linux kernel.
> >>
> >> In case ARM the cpufreq driver will read the settings for the
> >> operating-points from the device tree and the
> >> XENPF_set_processor_pminfo platform hypercall will not be necessary
> >> for ARM.
> >>
> >> Is this the right way to implement the cpufreq for OMAP under xen hypervisor?
> >
> > Yes, it's more or less what I had in mind.
> 
> I have a question. I see that the original file cpufreq.c contains
> 'Copyright (C)' fields. Could You please tell me which copyrights I should add
> to the new cpufreq_ops.c files (for x86 and ARM arhitectures).
> In case if cpufreq_add_cpu()/cpufreq_del_cpu() functions will be common for both
> architectures there will be only one cpufreq_ops.c file for x86 architecture.
> But there is a way when we will have two files file cpufreq_ops.c (for
> x86 and ARM).

I would keep the current Copyright fields for the x86 implementation of
cpufreq_ops.c. You can use your own Copyright line for the ARM
implementation.

Re: cpufreq implementation for OMAP under xen hypervisor.

[Ian Campbell]

On Fri, 2014-08-22 at 17:31 +0100, Stefano Stabellini wrote:
> One thing to keep in mind is that although P states are obviously an
> Intel concept

I'm not sure about P-states but C-states at least even if they were
originally an Intel thing seem to have become more of an ACPI thing
these days and I think they are being used by the ARM ACPI stuff too.

(or am I confused?)

Ian.

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Fri, Aug 22, 2014 at 7:31 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Fri, 22 Aug 2014, Oleksandr Dmytryshyn wrote:
>> Hi, Stefano.
>>
>> On Fri, Aug 22, 2014 at 2:31 AM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > CC'ing the x86 maintainers and the cpufreq original author.
>> >
>> >
>> > On Thu, 21 Aug 2014, Oleksandr Dmytryshyn wrote:
>> >> Hi to all.
>> >>
>> >> I'm planning to do next work:
>> >>
>> >> 1. Move file xen/include/acpi/cpufreq/cpufreq.h to the
>> >> xen/include/drivers/cpufreq/cpufreq.h
>> >> 2. Create a new file xen/arch/x86/acpi/cpufreq/cpufreq_common.c
>> >
>> > you can call it cpufreq.c or cpufreq_ops.c
>> I'll call it cpufreq_ops.c
>>
>> >> 3. Move some acpi-specific functions from
>> >> xen/drivers/cpufreq/cpufreq.c to the
>> >> xen/arch/x86/acpi/cpufreq/cpufreq_common.c:
>> >> cpufreq_limit_change(), print_PCT(), print_PSS(), print_PSD(),
>> >> print_PPC(), set_px_pminfo().
>> >
>> > Why cpufreq_limit_change?
>> The function cpufreq_limit_change() is called only in the set_px_pminfo()
>> function and will not be used for the ARM architecture.
>
> I see.
> One thing to keep in mind is that although P states are obviously an
> Intel concept, we could abstract them away and map them into
> arch-independent power-saving states. That way we could share functions
> like set_px_pminfo between ARM and x86. But I would have to see the
> patches to actually know how feasible that is.
Here is my RFC implementation:
http://lists.xen.org/archives/html/xen-devel/2014-08/msg02919.html

>> >> 4. Create a new file xen/arch/arm/cpufreq/cpufreq_common.c
>> >> 5. Functions cpufreq_add_cpu()/cpufreq_del_cpu() should be implemented
>> >> separately for the x86 and ARM architecture (in the correspond file
>> >> cpufreq_common.c).
>> >
>> > Why? The implementation doesn't look x86 specific.
>> The function cpufreq_add_cpu()/cpufreq_del_cpu() uses the acpi-specific
>> data structures. I don't know how make them common for both architectures
>> but I'll try to do this.
>>
>> >> 6. Port cpufreq driver for the OMAP from the Linux kernel.
>> >>
>> >> In case ARM the cpufreq driver will read the settings for the
>> >> operating-points from the device tree and the
>> >> XENPF_set_processor_pminfo platform hypercall will not be necessary
>> >> for ARM.
>> >>
>> >> Is this the right way to implement the cpufreq for OMAP under xen hypervisor?
>> >
>> > Yes, it's more or less what I had in mind.
>>
>> I have a question. I see that the original file cpufreq.c contains
>> 'Copyright (C)' fields. Could You please tell me which copyrights I should add
>> to the new cpufreq_ops.c files (for x86 and ARM arhitectures).
>> In case if cpufreq_add_cpu()/cpufreq_del_cpu() functions will be common for both
>> architectures there will be only one cpufreq_ops.c file for x86 architecture.
>> But there is a way when we will have two files file cpufreq_ops.c (for
>> x86 and ARM).
>
> I would keep the current Copyright fields for the x86 implementation of
> cpufreq_ops.c. You can use your own Copyright line for the ARM
> implementation.


-- 
Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Andrii Tseglytskyi]

Hi,

Stefano, Ian,

Could you please clarify the following point:

I agree that decision about frequency change should be taken by Xen
hypervisor. But what about hardware frequency changing?
In general when frequency changed to bigger value (for example from 1
GHz to 1.5 GHz) for ARM kernels sequence looks like the following:

1) cpufreq governor decides that frequency should be changed. This
decision is taken after analysing of CPU performance data taking in
account governor policy.
2) cpufreq governor asks cpufreq driver about new frequency.
3) cpufreq driver compares current and target frequencies and asks
cpufreq regulator about voltage change.
4) cpufreq regulator send i2c command to standalone microchip, which
is responsible for voltage changing.
5) cpufreq driver asks clock framework about new frequency for CPU clock
6) clock framework performs frequency sanity checks, taking in account
clock parents and clock divider settings, and call platform specific
"set_frequency" callback.
7) platform specific callback performs proper HW registers
configuration for newly selected frequency

Also there are some special cases - for example for OMAP5+ when
frequency is changed to 1.5 GHz+, two additional HW IPs should be
triggered (ABB and DCC, if someone is familiar with OMAP5+ )

So, for generic ARM kernel we have 3 entities to change frequency:

- cpufreq governor
- cpufreq driver
- cpufreq regulator

+ 2 additional IP for OMAP5+
- ABB
- DCC

Taking in account all above, it looks like it would be better to
implement only Xen cpufreq governor. Xen will take a decision about
new frequency, and kernel dom0 will perform other steps. Dom0 contains
all generic and platform specific frameworks, needed for frequency
changing.

What do you think ?

Regards,
Andrii


On Fri, Aug 29, 2014 at 4:25 PM, Oleksandr Dmytryshyn
<oleksandr.dmytryshyn@globallogic.com> wrote:
> On Fri, Aug 22, 2014 at 7:31 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
>> On Fri, 22 Aug 2014, Oleksandr Dmytryshyn wrote:
>>> Hi, Stefano.
>>>
>>> On Fri, Aug 22, 2014 at 2:31 AM, Stefano Stabellini
>>> <stefano.stabellini@eu.citrix.com> wrote:
>>> > CC'ing the x86 maintainers and the cpufreq original author.
>>> >
>>> >
>>> > On Thu, 21 Aug 2014, Oleksandr Dmytryshyn wrote:
>>> >> Hi to all.
>>> >>
>>> >> I'm planning to do next work:
>>> >>
>>> >> 1. Move file xen/include/acpi/cpufreq/cpufreq.h to the
>>> >> xen/include/drivers/cpufreq/cpufreq.h
>>> >> 2. Create a new file xen/arch/x86/acpi/cpufreq/cpufreq_common.c
>>> >
>>> > you can call it cpufreq.c or cpufreq_ops.c
>>> I'll call it cpufreq_ops.c
>>>
>>> >> 3. Move some acpi-specific functions from
>>> >> xen/drivers/cpufreq/cpufreq.c to the
>>> >> xen/arch/x86/acpi/cpufreq/cpufreq_common.c:
>>> >> cpufreq_limit_change(), print_PCT(), print_PSS(), print_PSD(),
>>> >> print_PPC(), set_px_pminfo().
>>> >
>>> > Why cpufreq_limit_change?
>>> The function cpufreq_limit_change() is called only in the set_px_pminfo()
>>> function and will not be used for the ARM architecture.
>>
>> I see.
>> One thing to keep in mind is that although P states are obviously an
>> Intel concept, we could abstract them away and map them into
>> arch-independent power-saving states. That way we could share functions
>> like set_px_pminfo between ARM and x86. But I would have to see the
>> patches to actually know how feasible that is.
> Here is my RFC implementation:
> http://lists.xen.org/archives/html/xen-devel/2014-08/msg02919.html
>
>>> >> 4. Create a new file xen/arch/arm/cpufreq/cpufreq_common.c
>>> >> 5. Functions cpufreq_add_cpu()/cpufreq_del_cpu() should be implemented
>>> >> separately for the x86 and ARM architecture (in the correspond file
>>> >> cpufreq_common.c).
>>> >
>>> > Why? The implementation doesn't look x86 specific.
>>> The function cpufreq_add_cpu()/cpufreq_del_cpu() uses the acpi-specific
>>> data structures. I don't know how make them common for both architectures
>>> but I'll try to do this.
>>>
>>> >> 6. Port cpufreq driver for the OMAP from the Linux kernel.
>>> >>
>>> >> In case ARM the cpufreq driver will read the settings for the
>>> >> operating-points from the device tree and the
>>> >> XENPF_set_processor_pminfo platform hypercall will not be necessary
>>> >> for ARM.
>>> >>
>>> >> Is this the right way to implement the cpufreq for OMAP under xen hypervisor?
>>> >
>>> > Yes, it's more or less what I had in mind.
>>>
>>> I have a question. I see that the original file cpufreq.c contains
>>> 'Copyright (C)' fields. Could You please tell me which copyrights I should add
>>> to the new cpufreq_ops.c files (for x86 and ARM arhitectures).
>>> In case if cpufreq_add_cpu()/cpufreq_del_cpu() functions will be common for both
>>> architectures there will be only one cpufreq_ops.c file for x86 architecture.
>>> But there is a way when we will have two files file cpufreq_ops.c (for
>>> x86 and ARM).
>>
>> I would keep the current Copyright fields for the x86 implementation of
>> cpufreq_ops.c. You can use your own Copyright line for the ARM
>> implementation.
>
>
> --
> Oleksandr Dmytryshyn | Product Engineering and Development
> GlobalLogic
> M +38.067.382.2525
> www.globallogic.com
>
> http://www.globallogic.com/email_disclaimer.txt
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel



-- 

Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> Hi,
> 
> Stefano, Ian,
> 
> Could you please clarify the following point:
> 
> I agree that decision about frequency change should be taken by Xen
> hypervisor. But what about hardware frequency changing?
> In general when frequency changed to bigger value (for example from 1
> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> 
> 1) cpufreq governor decides that frequency should be changed. This
> decision is taken after analysing of CPU performance data taking in
> account governor policy.
> 2) cpufreq governor asks cpufreq driver about new frequency.
> 3) cpufreq driver compares current and target frequencies and asks
> cpufreq regulator about voltage change.
> 4) cpufreq regulator send i2c command to standalone microchip, which
> is responsible for voltage changing.
> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> 6) clock framework performs frequency sanity checks, taking in account
> clock parents and clock divider settings, and call platform specific
> "set_frequency" callback.
> 7) platform specific callback performs proper HW registers
> configuration for newly selected frequency
> 
> Also there are some special cases - for example for OMAP5+ when
> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> 
> So, for generic ARM kernel we have 3 entities to change frequency:
> 
> - cpufreq governor
> - cpufreq driver
> - cpufreq regulator
> 
> + 2 additional IP for OMAP5+
> - ABB
> - DCC
> 
> Taking in account all above, it looks like it would be better to
> implement only Xen cpufreq governor. Xen will take a decision about
> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> all generic and platform specific frameworks, needed for frequency
> changing.
> 
> What do you think ?
 
Keep in mind that the architecture must be able to handle the case where
dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
physical cpus.
Could dom0 change the frequency of a physical core or a physical cpu is
not even running on? If that is not a problem, because cpus and
frequency changing are decoupled enough in Linux to allow it, then I am
OK with it. But I suspect they are not.



> Regards,
> Andrii
> 
> 
> On Fri, Aug 29, 2014 at 4:25 PM, Oleksandr Dmytryshyn
> <oleksandr.dmytryshyn@globallogic.com> wrote:
> > On Fri, Aug 22, 2014 at 7:31 PM, Stefano Stabellini
> > <stefano.stabellini@eu.citrix.com> wrote:
> >> On Fri, 22 Aug 2014, Oleksandr Dmytryshyn wrote:
> >>> Hi, Stefano.
> >>>
> >>> On Fri, Aug 22, 2014 at 2:31 AM, Stefano Stabellini
> >>> <stefano.stabellini@eu.citrix.com> wrote:
> >>> > CC'ing the x86 maintainers and the cpufreq original author.
> >>> >
> >>> >
> >>> > On Thu, 21 Aug 2014, Oleksandr Dmytryshyn wrote:
> >>> >> Hi to all.
> >>> >>
> >>> >> I'm planning to do next work:
> >>> >>
> >>> >> 1. Move file xen/include/acpi/cpufreq/cpufreq.h to the
> >>> >> xen/include/drivers/cpufreq/cpufreq.h
> >>> >> 2. Create a new file xen/arch/x86/acpi/cpufreq/cpufreq_common.c
> >>> >
> >>> > you can call it cpufreq.c or cpufreq_ops.c
> >>> I'll call it cpufreq_ops.c
> >>>
> >>> >> 3. Move some acpi-specific functions from
> >>> >> xen/drivers/cpufreq/cpufreq.c to the
> >>> >> xen/arch/x86/acpi/cpufreq/cpufreq_common.c:
> >>> >> cpufreq_limit_change(), print_PCT(), print_PSS(), print_PSD(),
> >>> >> print_PPC(), set_px_pminfo().
> >>> >
> >>> > Why cpufreq_limit_change?
> >>> The function cpufreq_limit_change() is called only in the set_px_pminfo()
> >>> function and will not be used for the ARM architecture.
> >>
> >> I see.
> >> One thing to keep in mind is that although P states are obviously an
> >> Intel concept, we could abstract them away and map them into
> >> arch-independent power-saving states. That way we could share functions
> >> like set_px_pminfo between ARM and x86. But I would have to see the
> >> patches to actually know how feasible that is.
> > Here is my RFC implementation:
> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02919.html
> >
> >>> >> 4. Create a new file xen/arch/arm/cpufreq/cpufreq_common.c
> >>> >> 5. Functions cpufreq_add_cpu()/cpufreq_del_cpu() should be implemented
> >>> >> separately for the x86 and ARM architecture (in the correspond file
> >>> >> cpufreq_common.c).
> >>> >
> >>> > Why? The implementation doesn't look x86 specific.
> >>> The function cpufreq_add_cpu()/cpufreq_del_cpu() uses the acpi-specific
> >>> data structures. I don't know how make them common for both architectures
> >>> but I'll try to do this.
> >>>
> >>> >> 6. Port cpufreq driver for the OMAP from the Linux kernel.
> >>> >>
> >>> >> In case ARM the cpufreq driver will read the settings for the
> >>> >> operating-points from the device tree and the
> >>> >> XENPF_set_processor_pminfo platform hypercall will not be necessary
> >>> >> for ARM.
> >>> >>
> >>> >> Is this the right way to implement the cpufreq for OMAP under xen hypervisor?
> >>> >
> >>> > Yes, it's more or less what I had in mind.
> >>>
> >>> I have a question. I see that the original file cpufreq.c contains
> >>> 'Copyright (C)' fields. Could You please tell me which copyrights I should add
> >>> to the new cpufreq_ops.c files (for x86 and ARM arhitectures).
> >>> In case if cpufreq_add_cpu()/cpufreq_del_cpu() functions will be common for both
> >>> architectures there will be only one cpufreq_ops.c file for x86 architecture.
> >>> But there is a way when we will have two files file cpufreq_ops.c (for
> >>> x86 and ARM).
> >>
> >> I would keep the current Copyright fields for the x86 implementation of
> >> cpufreq_ops.c. You can use your own Copyright line for the ARM
> >> implementation.
> >
> >
> > --
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> 
> 
> 
> -- 
> 
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
> 

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly Chernooky]

[-- Attachment #1.1: Type: text/plain, Size: 7345 bytes --]

Hi All,


On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:

> On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > Hi,
> >
> > Stefano, Ian,
> >
> > Could you please clarify the following point:
> >
> > I agree that decision about frequency change should be taken by Xen
> > hypervisor. But what about hardware frequency changing?
> > In general when frequency changed to bigger value (for example from 1
> > GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> >
> > 1) cpufreq governor decides that frequency should be changed. This
> > decision is taken after analysing of CPU performance data taking in
> > account governor policy.
> > 2) cpufreq governor asks cpufreq driver about new frequency.
> > 3) cpufreq driver compares current and target frequencies and asks
> > cpufreq regulator about voltage change.
> > 4) cpufreq regulator send i2c command to standalone microchip, which
> > is responsible for voltage changing.
> > 5) cpufreq driver asks clock framework about new frequency for CPU clock
> > 6) clock framework performs frequency sanity checks, taking in account
> > clock parents and clock divider settings, and call platform specific
> > "set_frequency" callback.
> > 7) platform specific callback performs proper HW registers
> > configuration for newly selected frequency
> >
> > Also there are some special cases - for example for OMAP5+ when
> > frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> >
> > So, for generic ARM kernel we have 3 entities to change frequency:
> >
> > - cpufreq governor
> > - cpufreq driver
> > - cpufreq regulator
> >
> > + 2 additional IP for OMAP5+
> > - ABB
> > - DCC
> >
> > Taking in account all above, it looks like it would be better to
> > implement only Xen cpufreq governor. Xen will take a decision about
> > new frequency, and kernel dom0 will perform other steps. Dom0 contains
> > all generic and platform specific frameworks, needed for frequency
> > changing.
> >
> > What do you think ?
>
> Keep in mind that the architecture must be able to handle the case where
> dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> physical cpus.
> Could dom0 change the frequency of a physical core or a physical cpu is
> not even running on? If that is not a problem, because cpus and
> frequency changing are decoupled enough in Linux to allow it, then I am
> OK with it. But I suspect they are not.
>

Andrii's idea is reasonable because dom0 kernel already has enough
knowledge about physical hardware. But from opposite side frameworks
mentioned by Andrii don't have knowledge about vcpu and other virtualized
HW. So i'm still doubt than dom0 linux kernel is appropriate place for
introducing this functionality.

With best regards,


>
> > Regards,
> > Andrii
> >
> >
> > On Fri, Aug 29, 2014 at 4:25 PM, Oleksandr Dmytryshyn
> > <oleksandr.dmytryshyn@globallogic.com> wrote:
> > > On Fri, Aug 22, 2014 at 7:31 PM, Stefano Stabellini
> > > <stefano.stabellini@eu.citrix.com> wrote:
> > >> On Fri, 22 Aug 2014, Oleksandr Dmytryshyn wrote:
> > >>> Hi, Stefano.
> > >>>
> > >>> On Fri, Aug 22, 2014 at 2:31 AM, Stefano Stabellini
> > >>> <stefano.stabellini@eu.citrix.com> wrote:
> > >>> > CC'ing the x86 maintainers and the cpufreq original author.
> > >>> >
> > >>> >
> > >>> > On Thu, 21 Aug 2014, Oleksandr Dmytryshyn wrote:
> > >>> >> Hi to all.
> > >>> >>
> > >>> >> I'm planning to do next work:
> > >>> >>
> > >>> >> 1. Move file xen/include/acpi/cpufreq/cpufreq.h to the
> > >>> >> xen/include/drivers/cpufreq/cpufreq.h
> > >>> >> 2. Create a new file xen/arch/x86/acpi/cpufreq/cpufreq_common.c
> > >>> >
> > >>> > you can call it cpufreq.c or cpufreq_ops.c
> > >>> I'll call it cpufreq_ops.c
> > >>>
> > >>> >> 3. Move some acpi-specific functions from
> > >>> >> xen/drivers/cpufreq/cpufreq.c to the
> > >>> >> xen/arch/x86/acpi/cpufreq/cpufreq_common.c:
> > >>> >> cpufreq_limit_change(), print_PCT(), print_PSS(), print_PSD(),
> > >>> >> print_PPC(), set_px_pminfo().
> > >>> >
> > >>> > Why cpufreq_limit_change?
> > >>> The function cpufreq_limit_change() is called only in the
> set_px_pminfo()
> > >>> function and will not be used for the ARM architecture.
> > >>
> > >> I see.
> > >> One thing to keep in mind is that although P states are obviously an
> > >> Intel concept, we could abstract them away and map them into
> > >> arch-independent power-saving states. That way we could share
> functions
> > >> like set_px_pminfo between ARM and x86. But I would have to see the
> > >> patches to actually know how feasible that is.
> > > Here is my RFC implementation:
> > > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02919.html
> > >
> > >>> >> 4. Create a new file xen/arch/arm/cpufreq/cpufreq_common.c
> > >>> >> 5. Functions cpufreq_add_cpu()/cpufreq_del_cpu() should be
> implemented
> > >>> >> separately for the x86 and ARM architecture (in the correspond
> file
> > >>> >> cpufreq_common.c).
> > >>> >
> > >>> > Why? The implementation doesn't look x86 specific.
> > >>> The function cpufreq_add_cpu()/cpufreq_del_cpu() uses the
> acpi-specific
> > >>> data structures. I don't know how make them common for both
> architectures
> > >>> but I'll try to do this.
> > >>>
> > >>> >> 6. Port cpufreq driver for the OMAP from the Linux kernel.
> > >>> >>
> > >>> >> In case ARM the cpufreq driver will read the settings for the
> > >>> >> operating-points from the device tree and the
> > >>> >> XENPF_set_processor_pminfo platform hypercall will not be
> necessary
> > >>> >> for ARM.
> > >>> >>
> > >>> >> Is this the right way to implement the cpufreq for OMAP under xen
> hypervisor?
> > >>> >
> > >>> > Yes, it's more or less what I had in mind.
> > >>>
> > >>> I have a question. I see that the original file cpufreq.c contains
> > >>> 'Copyright (C)' fields. Could You please tell me which copyrights I
> should add
> > >>> to the new cpufreq_ops.c files (for x86 and ARM arhitectures).
> > >>> In case if cpufreq_add_cpu()/cpufreq_del_cpu() functions will be
> common for both
> > >>> architectures there will be only one cpufreq_ops.c file for x86
> architecture.
> > >>> But there is a way when we will have two files file cpufreq_ops.c
> (for
> > >>> x86 and ARM).
> > >>
> > >> I would keep the current Copyright fields for the x86 implementation
> of
> > >> cpufreq_ops.c. You can use your own Copyright line for the ARM
> > >> implementation.
> > >
> > >
> > > --
> > > Oleksandr Dmytryshyn | Product Engineering and Development
> > > GlobalLogic
> > > M +38.067.382.2525
> > > www.globallogic.com
> > >
> > > http://www.globallogic.com/email_disclaimer.txt
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > http://lists.xen.org/xen-devel
> >
> >
> >
> > --
> >
> > Andrii Tseglytskyi | Embedded Dev
> > GlobalLogic
> > www.globallogic.com
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>



-- 
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 10819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Andrii Tseglytskyi]

On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
>> Hi,
>>
>> Stefano, Ian,
>>
>> Could you please clarify the following point:
>>
>> I agree that decision about frequency change should be taken by Xen
>> hypervisor. But what about hardware frequency changing?
>> In general when frequency changed to bigger value (for example from 1
>> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
>>
>> 1) cpufreq governor decides that frequency should be changed. This
>> decision is taken after analysing of CPU performance data taking in
>> account governor policy.
>> 2) cpufreq governor asks cpufreq driver about new frequency.
>> 3) cpufreq driver compares current and target frequencies and asks
>> cpufreq regulator about voltage change.
>> 4) cpufreq regulator send i2c command to standalone microchip, which
>> is responsible for voltage changing.
>> 5) cpufreq driver asks clock framework about new frequency for CPU clock
>> 6) clock framework performs frequency sanity checks, taking in account
>> clock parents and clock divider settings, and call platform specific
>> "set_frequency" callback.
>> 7) platform specific callback performs proper HW registers
>> configuration for newly selected frequency
>>
>> Also there are some special cases - for example for OMAP5+ when
>> frequency is changed to 1.5 GHz+, two additional HW IPs should be
>> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
>>
>> So, for generic ARM kernel we have 3 entities to change frequency:
>>
>> - cpufreq governor
>> - cpufreq driver
>> - cpufreq regulator
>>
>> + 2 additional IP for OMAP5+
>> - ABB
>> - DCC
>>
>> Taking in account all above, it looks like it would be better to
>> implement only Xen cpufreq governor. Xen will take a decision about
>> new frequency, and kernel dom0 will perform other steps. Dom0 contains
>> all generic and platform specific frameworks, needed for frequency
>> changing.
>>
>> What do you think ?
>
> Keep in mind that the architecture must be able to handle the case where
> dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> physical cpus.
> Could dom0 change the frequency of a physical core or a physical cpu is
> not even running on? If that is not a problem, because cpus and
> frequency changing are decoupled enough in Linux to allow it, then I am
> OK with it. But I suspect they are not.
>

Not sure that I got your point correctly - dom0 will change frequency
on physical CPU.
And in case of OMAP - this changing affects on both ARM physical cpus
- changing is coupled.
In case of other ARM platforms - changing may be not coupled (I've
heard that Snapdragon can change cpu freqs independently on each
physical cpu)

Regrads,
Andrii

-- 

Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> >> Hi,
> >>
> >> Stefano, Ian,
> >>
> >> Could you please clarify the following point:
> >>
> >> I agree that decision about frequency change should be taken by Xen
> >> hypervisor. But what about hardware frequency changing?
> >> In general when frequency changed to bigger value (for example from 1
> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> >>
> >> 1) cpufreq governor decides that frequency should be changed. This
> >> decision is taken after analysing of CPU performance data taking in
> >> account governor policy.
> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> >> 3) cpufreq driver compares current and target frequencies and asks
> >> cpufreq regulator about voltage change.
> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> >> is responsible for voltage changing.
> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> >> 6) clock framework performs frequency sanity checks, taking in account
> >> clock parents and clock divider settings, and call platform specific
> >> "set_frequency" callback.
> >> 7) platform specific callback performs proper HW registers
> >> configuration for newly selected frequency
> >>
> >> Also there are some special cases - for example for OMAP5+ when
> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> >>
> >> So, for generic ARM kernel we have 3 entities to change frequency:
> >>
> >> - cpufreq governor
> >> - cpufreq driver
> >> - cpufreq regulator
> >>
> >> + 2 additional IP for OMAP5+
> >> - ABB
> >> - DCC
> >>
> >> Taking in account all above, it looks like it would be better to
> >> implement only Xen cpufreq governor. Xen will take a decision about
> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> >> all generic and platform specific frameworks, needed for frequency
> >> changing.
> >>
> >> What do you think ?
> >
> > Keep in mind that the architecture must be able to handle the case where
> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > physical cpus.
> > Could dom0 change the frequency of a physical core or a physical cpu is
> > not even running on? If that is not a problem, because cpus and
> > frequency changing are decoupled enough in Linux to allow it, then I am
> > OK with it. But I suspect they are not.
> >
> 
> Not sure that I got your point correctly - dom0 will change frequency
> on physical CPU.
> And in case of OMAP - this changing affects on both ARM physical cpus
> - changing is coupled.
> In case of other ARM platforms - changing may be not coupled (I've
> heard that Snapdragon can change cpu freqs independently on each
> physical cpu)

Let me explain with a concrete example.

Let's suppose that the platform has 2 physical cpus, each cpu has 4
cores.  Let's also supposed that dom0 has only 2 vcpus, currently
running on core0 and core1 of cpu0.

In this case would dom0 be able to change the frequency of core3 of
cpu1, given that is not even running on it?
If it can be done without any hacks, then we can go ahead with this
approach.

Re: cpufreq implementation for OMAP under xen hypervisor.

[Andrii Tseglytskyi]

Hi Stefano,

Thank you for explanation.
I think this requires more and deeper investigation, but for sure dom0
must be able to do this.
Let us investigate this.

Thank you,

Regards,
Andrii

On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
>> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
>> >> Hi,
>> >>
>> >> Stefano, Ian,
>> >>
>> >> Could you please clarify the following point:
>> >>
>> >> I agree that decision about frequency change should be taken by Xen
>> >> hypervisor. But what about hardware frequency changing?
>> >> In general when frequency changed to bigger value (for example from 1
>> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
>> >>
>> >> 1) cpufreq governor decides that frequency should be changed. This
>> >> decision is taken after analysing of CPU performance data taking in
>> >> account governor policy.
>> >> 2) cpufreq governor asks cpufreq driver about new frequency.
>> >> 3) cpufreq driver compares current and target frequencies and asks
>> >> cpufreq regulator about voltage change.
>> >> 4) cpufreq regulator send i2c command to standalone microchip, which
>> >> is responsible for voltage changing.
>> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
>> >> 6) clock framework performs frequency sanity checks, taking in account
>> >> clock parents and clock divider settings, and call platform specific
>> >> "set_frequency" callback.
>> >> 7) platform specific callback performs proper HW registers
>> >> configuration for newly selected frequency
>> >>
>> >> Also there are some special cases - for example for OMAP5+ when
>> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
>> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
>> >>
>> >> So, for generic ARM kernel we have 3 entities to change frequency:
>> >>
>> >> - cpufreq governor
>> >> - cpufreq driver
>> >> - cpufreq regulator
>> >>
>> >> + 2 additional IP for OMAP5+
>> >> - ABB
>> >> - DCC
>> >>
>> >> Taking in account all above, it looks like it would be better to
>> >> implement only Xen cpufreq governor. Xen will take a decision about
>> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
>> >> all generic and platform specific frameworks, needed for frequency
>> >> changing.
>> >>
>> >> What do you think ?
>> >
>> > Keep in mind that the architecture must be able to handle the case where
>> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
>> > physical cpus.
>> > Could dom0 change the frequency of a physical core or a physical cpu is
>> > not even running on? If that is not a problem, because cpus and
>> > frequency changing are decoupled enough in Linux to allow it, then I am
>> > OK with it. But I suspect they are not.
>> >
>>
>> Not sure that I got your point correctly - dom0 will change frequency
>> on physical CPU.
>> And in case of OMAP - this changing affects on both ARM physical cpus
>> - changing is coupled.
>> In case of other ARM platforms - changing may be not coupled (I've
>> heard that Snapdragon can change cpu freqs independently on each
>> physical cpu)
>
> Let me explain with a concrete example.
>
> Let's suppose that the platform has 2 physical cpus, each cpu has 4
> cores.  Let's also supposed that dom0 has only 2 vcpus, currently
> running on core0 and core1 of cpu0.
>
> In this case would dom0 be able to change the frequency of core3 of
> cpu1, given that is not even running on it?
> If it can be done without any hacks, then we can go ahead with this
> approach.



-- 

Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

Hi to all.

I want to implement cpufreq driver in the next way:
1. Cpufreq governor will be implemented in the Xen
2. dom0 will only change cpu frequency and voltage of the physical cpus

But there are some nuances:
1. dom0 driver should read an information about operation points
(frequencies and voltages) and cpu supply source from the device tree for each
physical cpu. In the omap processor case this driver suspects that
those settings
located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
for each vcpu
for kernel dom0 in the device tree and those information is lost in the dom0.
2. What about this case if we will have some physical cpus with different
operation points (for example 2 cpus) and we give only one cpu for dom0?

How should I transfer all information from the original cpu@0..cpu@n nodes
about all physical cpus to the kernel dom0 driver? Maybe an additional
nodes should be created by the hypervisor in the device tree for dom0
and named as pcpu@0..pcpu@n?


Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt


On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
<andrii.tseglytskyi@globallogic.com> wrote:
>
> Hi Stefano,
>
> Thank you for explanation.
> I think this requires more and deeper investigation, but for sure dom0
> must be able to do this.
> Let us investigate this.
>
> Thank you,
>
> Regards,
> Andrii
>
> On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> >> <stefano.stabellini@eu.citrix.com> wrote:
> >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> >> >> Hi,
> >> >>
> >> >> Stefano, Ian,
> >> >>
> >> >> Could you please clarify the following point:
> >> >>
> >> >> I agree that decision about frequency change should be taken by Xen
> >> >> hypervisor. But what about hardware frequency changing?
> >> >> In general when frequency changed to bigger value (for example from 1
> >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> >> >>
> >> >> 1) cpufreq governor decides that frequency should be changed. This
> >> >> decision is taken after analysing of CPU performance data taking in
> >> >> account governor policy.
> >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> >> >> 3) cpufreq driver compares current and target frequencies and asks
> >> >> cpufreq regulator about voltage change.
> >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> >> >> is responsible for voltage changing.
> >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> >> >> 6) clock framework performs frequency sanity checks, taking in account
> >> >> clock parents and clock divider settings, and call platform specific
> >> >> "set_frequency" callback.
> >> >> 7) platform specific callback performs proper HW registers
> >> >> configuration for newly selected frequency
> >> >>
> >> >> Also there are some special cases - for example for OMAP5+ when
> >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> >> >>
> >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> >> >>
> >> >> - cpufreq governor
> >> >> - cpufreq driver
> >> >> - cpufreq regulator
> >> >>
> >> >> + 2 additional IP for OMAP5+
> >> >> - ABB
> >> >> - DCC
> >> >>
> >> >> Taking in account all above, it looks like it would be better to
> >> >> implement only Xen cpufreq governor. Xen will take a decision about
> >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> >> >> all generic and platform specific frameworks, needed for frequency
> >> >> changing.
> >> >>
> >> >> What do you think ?
> >> >
> >> > Keep in mind that the architecture must be able to handle the case where
> >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> >> > physical cpus.
> >> > Could dom0 change the frequency of a physical core or a physical cpu is
> >> > not even running on? If that is not a problem, because cpus and
> >> > frequency changing are decoupled enough in Linux to allow it, then I am
> >> > OK with it. But I suspect they are not.
> >> >
> >>
> >> Not sure that I got your point correctly - dom0 will change frequency
> >> on physical CPU.
> >> And in case of OMAP - this changing affects on both ARM physical cpus
> >> - changing is coupled.
> >> In case of other ARM platforms - changing may be not coupled (I've
> >> heard that Snapdragon can change cpu freqs independently on each
> >> physical cpu)
> >
> > Let me explain with a concrete example.
> >
> > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
> > running on core0 and core1 of cpu0.
> >
> > In this case would dom0 be able to change the frequency of core3 of
> > cpu1, given that is not even running on it?
> > If it can be done without any hacks, then we can go ahead with this
> > approach.
>
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> Hi to all.
> 
> I want to implement cpufreq driver in the next way:
> 1. Cpufreq governor will be implemented in the Xen
> 2. dom0 will only change cpu frequency and voltage of the physical cpus
> 
> But there are some nuances:
> 1. dom0 driver should read an information about operation points
> (frequencies and voltages) and cpu supply source from the device tree for each
> physical cpu. In the omap processor case this driver suspects that
> those settings
> located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
> for each vcpu
> for kernel dom0 in the device tree and those information is lost in the dom0.
> 2. What about this case if we will have some physical cpus with different
> operation points (for example 2 cpus) and we give only one cpu for dom0?
> 
> How should I transfer all information from the original cpu@0..cpu@n nodes
> about all physical cpus to the kernel dom0 driver? Maybe an additional
> nodes should be created by the hypervisor in the device tree for dom0
> and named as pcpu@0..pcpu@n?

If we do that, wouldn't we require changes to the core OMAP drivers or
cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
nodes)?  I don't expect they would be easy to upstream or maintain going
forward.

I am trying to think of an alternative, such as passing the real cpu
nodes to dom0 but then adding status = "disabled", but I am not sure
whether Linux checks the status for cpu nodes. In addition this scheme
wouldn't support the case where dom0 has more vcpus than pcpus on the
system. Granted it is not very common and might even be detrimental for
performances, but we should be able to support it.

Ian, what do you think about this?


 
> Oleksandr Dmytryshyn | Product Engineering and Development
> GlobalLogic
> M +38.067.382.2525
> www.globallogic.com
> 
> http://www.globallogic.com/email_disclaimer.txt
> 
> 
> On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> <andrii.tseglytskyi@globallogic.com> wrote:
> >
> > Hi Stefano,
> >
> > Thank you for explanation.
> > I think this requires more and deeper investigation, but for sure dom0
> > must be able to do this.
> > Let us investigate this.
> >
> > Thank you,
> >
> > Regards,
> > Andrii
> >
> > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > <stefano.stabellini@eu.citrix.com> wrote:
> > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > >> <stefano.stabellini@eu.citrix.com> wrote:
> > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > >> >> Hi,
> > >> >>
> > >> >> Stefano, Ian,
> > >> >>
> > >> >> Could you please clarify the following point:
> > >> >>
> > >> >> I agree that decision about frequency change should be taken by Xen
> > >> >> hypervisor. But what about hardware frequency changing?
> > >> >> In general when frequency changed to bigger value (for example from 1
> > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> > >> >>
> > >> >> 1) cpufreq governor decides that frequency should be changed. This
> > >> >> decision is taken after analysing of CPU performance data taking in
> > >> >> account governor policy.
> > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > >> >> 3) cpufreq driver compares current and target frequencies and asks
> > >> >> cpufreq regulator about voltage change.
> > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> > >> >> is responsible for voltage changing.
> > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> > >> >> 6) clock framework performs frequency sanity checks, taking in account
> > >> >> clock parents and clock divider settings, and call platform specific
> > >> >> "set_frequency" callback.
> > >> >> 7) platform specific callback performs proper HW registers
> > >> >> configuration for newly selected frequency
> > >> >>
> > >> >> Also there are some special cases - for example for OMAP5+ when
> > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > >> >>
> > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> > >> >>
> > >> >> - cpufreq governor
> > >> >> - cpufreq driver
> > >> >> - cpufreq regulator
> > >> >>
> > >> >> + 2 additional IP for OMAP5+
> > >> >> - ABB
> > >> >> - DCC
> > >> >>
> > >> >> Taking in account all above, it looks like it would be better to
> > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> > >> >> all generic and platform specific frameworks, needed for frequency
> > >> >> changing.
> > >> >>
> > >> >> What do you think ?
> > >> >
> > >> > Keep in mind that the architecture must be able to handle the case where
> > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > >> > physical cpus.
> > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> > >> > not even running on? If that is not a problem, because cpus and
> > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> > >> > OK with it. But I suspect they are not.
> > >> >
> > >>
> > >> Not sure that I got your point correctly - dom0 will change frequency
> > >> on physical CPU.
> > >> And in case of OMAP - this changing affects on both ARM physical cpus
> > >> - changing is coupled.
> > >> In case of other ARM platforms - changing may be not coupled (I've
> > >> heard that Snapdragon can change cpu freqs independently on each
> > >> physical cpu)
> > >
> > > Let me explain with a concrete example.
> > >
> > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
> > > running on core0 and core1 of cpu0.
> > >
> > > In this case would dom0 be able to change the frequency of core3 of
> > > cpu1, given that is not even running on it?
> > > If it can be done without any hacks, then we can go ahead with this
> > > approach.
> >
> >
> >
> > --
> >
> > Andrii Tseglytskyi | Embedded Dev
> > GlobalLogic
> > www.globallogic.com
> 

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
>> Hi to all.
>>
>> I want to implement cpufreq driver in the next way:
>> 1. Cpufreq governor will be implemented in the Xen
>> 2. dom0 will only change cpu frequency and voltage of the physical cpus
>>
>> But there are some nuances:
>> 1. dom0 driver should read an information about operation points
>> (frequencies and voltages) and cpu supply source from the device tree for each
>> physical cpu. In the omap processor case this driver suspects that
>> those settings
>> located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
>> for each vcpu
>> for kernel dom0 in the device tree and those information is lost in the dom0.
>> 2. What about this case if we will have some physical cpus with different
>> operation points (for example 2 cpus) and we give only one cpu for dom0?
>>
>> How should I transfer all information from the original cpu@0..cpu@n nodes
>> about all physical cpus to the kernel dom0 driver? Maybe an additional
>> nodes should be created by the hypervisor in the device tree for dom0
>> and named as pcpu@0..pcpu@n?
>
> If we do that, wouldn't we require changes to the core OMAP drivers or
> cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> nodes)?  I don't expect they would be easy to upstream or maintain going
> forward.
>
> I am trying to think of an alternative, such as passing the real cpu
> nodes to dom0 but then adding status = "disabled", but I am not sure
> whether Linux checks the status for cpu nodes. In addition this scheme
> wouldn't support the case where dom0 has more vcpus than pcpus on the
> system. Granted it is not very common and might even be detrimental for
> performances, but we should be able to support it.
>
> Ian, what do you think about this?
>
>
>
>> Oleksandr Dmytryshyn | Product Engineering and Development
>> GlobalLogic
>> M +38.067.382.2525
>> www.globallogic.com
>>
>> http://www.globallogic.com/email_disclaimer.txt
>>
>>
>> On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
>> <andrii.tseglytskyi@globallogic.com> wrote:
>> >
>> > Hi Stefano,
>> >
>> > Thank you for explanation.
>> > I think this requires more and deeper investigation, but for sure dom0
>> > must be able to do this.
>> > Let us investigate this.
>> >
>> > Thank you,
>> >
>> > Regards,
>> > Andrii
>> >
>> > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
>> > <stefano.stabellini@eu.citrix.com> wrote:
>> > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
>> > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
>> > >> <stefano.stabellini@eu.citrix.com> wrote:
>> > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
>> > >> >> Hi,
>> > >> >>
>> > >> >> Stefano, Ian,
>> > >> >>
>> > >> >> Could you please clarify the following point:
>> > >> >>
>> > >> >> I agree that decision about frequency change should be taken by Xen
>> > >> >> hypervisor. But what about hardware frequency changing?
>> > >> >> In general when frequency changed to bigger value (for example from 1
>> > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
>> > >> >>
>> > >> >> 1) cpufreq governor decides that frequency should be changed. This
>> > >> >> decision is taken after analysing of CPU performance data taking in
>> > >> >> account governor policy.
>> > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
>> > >> >> 3) cpufreq driver compares current and target frequencies and asks
>> > >> >> cpufreq regulator about voltage change.
>> > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
>> > >> >> is responsible for voltage changing.
>> > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
>> > >> >> 6) clock framework performs frequency sanity checks, taking in account
>> > >> >> clock parents and clock divider settings, and call platform specific
>> > >> >> "set_frequency" callback.
>> > >> >> 7) platform specific callback performs proper HW registers
>> > >> >> configuration for newly selected frequency
>> > >> >>
>> > >> >> Also there are some special cases - for example for OMAP5+ when
>> > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
>> > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
>> > >> >>
>> > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
>> > >> >>
>> > >> >> - cpufreq governor
>> > >> >> - cpufreq driver
>> > >> >> - cpufreq regulator
>> > >> >>
>> > >> >> + 2 additional IP for OMAP5+
>> > >> >> - ABB
>> > >> >> - DCC
>> > >> >>
>> > >> >> Taking in account all above, it looks like it would be better to
>> > >> >> implement only Xen cpufreq governor. Xen will take a decision about
>> > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
>> > >> >> all generic and platform specific frameworks, needed for frequency
>> > >> >> changing.
>> > >> >>
>> > >> >> What do you think ?
>> > >> >
>> > >> > Keep in mind that the architecture must be able to handle the case where
>> > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
>> > >> > physical cpus.
>> > >> > Could dom0 change the frequency of a physical core or a physical cpu is
>> > >> > not even running on? If that is not a problem, because cpus and
>> > >> > frequency changing are decoupled enough in Linux to allow it, then I am
>> > >> > OK with it. But I suspect they are not.
>> > >> >
>> > >>
>> > >> Not sure that I got your point correctly - dom0 will change frequency
>> > >> on physical CPU.
>> > >> And in case of OMAP - this changing affects on both ARM physical cpus
>> > >> - changing is coupled.
>> > >> In case of other ARM platforms - changing may be not coupled (I've
>> > >> heard that Snapdragon can change cpu freqs independently on each
>> > >> physical cpu)
>> > >
>> > > Let me explain with a concrete example.
>> > >
>> > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
>> > > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
>> > > running on core0 and core1 of cpu0.
>> > >
>> > > In this case would dom0 be able to change the frequency of core3 of
>> > > cpu1, given that is not even running on it?
>> > > If it can be done without any hacks, then we can go ahead with this
>> > > approach.
>> >
>> >
>> >
>> > --
>> >
>> > Andrii Tseglytskyi | Embedded Dev
>> > GlobalLogic
>> > www.globallogic.com
>>

Hi to all.

I've done next work to check how dom0 can change the frequency of the cpus.
1. I've written a small HACK in the hypervisor which copies all settings
from the physical cpu0 to the vcpu0.
Thus kernel can read information about OPP states and power supply.
2. I've turned on the cpufreq driver in kernel dom0.
3. I've turned on all needed regulators.
4. I've reverted a patch whitch disables CPUFREQ in kernel dom0 when
it is running under the xen.

Now kernel dom0 has 2 virtual cpus and board has 2 physical cpus.
CPUFREQ driver uses vcpus as source to calculate frequency and changes this
frequency on the physical cpus.

I've checked that in this case kernel dom0 can change the frequency of the cpus.

I want to disable CPUFREQ governor driver in dom0 and leave only cpu0-cpufreq
driver (driver for OMAP processors). CPUFREQ governor driver will be implemented
in the xen.

Here are some questions:

1. How implement an mechanism which allows the hypervisor to give
commands to the
CPUFREQ driver in the dom0 (i. e. 'set frequency')?

2. Could we use a limitation when the number of VCPUs is domain0 must
be equal to
the number of physical CPUs, and the domain0 VCPU must be pinned to
the respective
physical CPU? This limitation us used for the existing Domain0 based cpufreq.

3. If we will use this limitation can I simply copy all settings from physical
cpus to the appropriate vcpus in the device tree in xen? In this case
the existing
kernel code will not be modified and simple CPUFREQ driver will be written
which will receive commands from xen and redirect them to the cpu0-cpufreq
driver (or another registered cpufreq driver for any cpu).

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Ian Campbell]

On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> I am trying to think of an alternative, such as passing the real cpu
> nodes to dom0 but then adding status = "disabled", but I am not sure
> whether Linux checks the status for cpu nodes.

status = "disabled" is defined to have a specific (i.e. non-default)
meaning for cpu nodes, Julien mentioned this when I tried to add a
similar patch to Xen to ignore them. I think it basically means "present
but not running, you should start them!".

>  In addition this scheme
> wouldn't support the case where dom0 has more vcpus than pcpus on the
> system. Granted it is not very common and might even be detrimental for
> performances, but we should be able to support it.

It's a bit of an edge case, for sure. I guess it wouldn't be totally
unreasonable to say that if you use this sort of configuration you may
not get cpufreq support.

> Ian, what do you think about this?

All the options suck in one way or another AFAICT. I think we are going
to be looking for the least bad solution not necessarily a good one.

Fundamentally are we trying to avoid having to have a i2c subsystem etc
in the hypervisor to be be able to change the voltages before/after
changing the frequency?

We can't just say "that's part of the cpufreq driver" since different
boards using the same SoC might use different voltage regulators, over
i2c or some other bus etc, so we end up with a matrix.

It's arguable that we should be letting dom0 poke at that regulator
functionality anyway, at least not all of it. Taking that ability away
would necessarily imply more platform specific functionality in the
hypervisor.

How does this stuff work on x86?

Sorry, more questions than answers here.

Ian.

> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> > 
> > http://www.globallogic.com/email_disclaimer.txt
> > 
> > 
> > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > <andrii.tseglytskyi@globallogic.com> wrote:
> > >
> > > Hi Stefano,
> > >
> > > Thank you for explanation.
> > > I think this requires more and deeper investigation, but for sure dom0
> > > must be able to do this.
> > > Let us investigate this.
> > >
> > > Thank you,
> > >
> > > Regards,
> > > Andrii
> > >
> > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > >> >> Hi,
> > > >> >>
> > > >> >> Stefano, Ian,
> > > >> >>
> > > >> >> Could you please clarify the following point:
> > > >> >>
> > > >> >> I agree that decision about frequency change should be taken by Xen
> > > >> >> hypervisor. But what about hardware frequency changing?
> > > >> >> In general when frequency changed to bigger value (for example from 1
> > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> > > >> >>
> > > >> >> 1) cpufreq governor decides that frequency should be changed. This
> > > >> >> decision is taken after analysing of CPU performance data taking in
> > > >> >> account governor policy.
> > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > > >> >> 3) cpufreq driver compares current and target frequencies and asks
> > > >> >> cpufreq regulator about voltage change.
> > > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> > > >> >> is responsible for voltage changing.
> > > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> > > >> >> 6) clock framework performs frequency sanity checks, taking in account
> > > >> >> clock parents and clock divider settings, and call platform specific
> > > >> >> "set_frequency" callback.
> > > >> >> 7) platform specific callback performs proper HW registers
> > > >> >> configuration for newly selected frequency
> > > >> >>
> > > >> >> Also there are some special cases - for example for OMAP5+ when
> > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > > >> >>
> > > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> > > >> >>
> > > >> >> - cpufreq governor
> > > >> >> - cpufreq driver
> > > >> >> - cpufreq regulator
> > > >> >>
> > > >> >> + 2 additional IP for OMAP5+
> > > >> >> - ABB
> > > >> >> - DCC
> > > >> >>
> > > >> >> Taking in account all above, it looks like it would be better to
> > > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> > > >> >> all generic and platform specific frameworks, needed for frequency
> > > >> >> changing.
> > > >> >>
> > > >> >> What do you think ?
> > > >> >
> > > >> > Keep in mind that the architecture must be able to handle the case where
> > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > > >> > physical cpus.
> > > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> > > >> > not even running on? If that is not a problem, because cpus and
> > > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> > > >> > OK with it. But I suspect they are not.
> > > >> >
> > > >>
> > > >> Not sure that I got your point correctly - dom0 will change frequency
> > > >> on physical CPU.
> > > >> And in case of OMAP - this changing affects on both ARM physical cpus
> > > >> - changing is coupled.
> > > >> In case of other ARM platforms - changing may be not coupled (I've
> > > >> heard that Snapdragon can change cpu freqs independently on each
> > > >> physical cpu)
> > > >
> > > > Let me explain with a concrete example.
> > > >
> > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
> > > > running on core0 and core1 of cpu0.
> > > >
> > > > In this case would dom0 be able to change the frequency of core3 of
> > > > cpu1, given that is not even running on it?
> > > > If it can be done without any hacks, then we can go ahead with this
> > > > approach.
> > >
> > >
> > >
> > > --
> > >
> > > Andrii Tseglytskyi | Embedded Dev
> > > GlobalLogic
> > > www.globallogic.com
> > 

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly Chernooky]

[-- Attachment #1.1: Type: text/plain, Size: 7545 bytes --]

Hi All!


On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:

> On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> > Hi to all.
> >
> > I want to implement cpufreq driver in the next way:
> > 1. Cpufreq governor will be implemented in the Xen
> > 2. dom0 will only change cpu frequency and voltage of the physical cpus
> >
> > But there are some nuances:
> > 1. dom0 driver should read an information about operation points
> > (frequencies and voltages) and cpu supply source from the device tree
> for each
> > physical cpu. In the omap processor case this driver suspects that
> > those settings
> > located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
> > for each vcpu
> > for kernel dom0 in the device tree and those information is lost in the
> dom0.
> > 2. What about this case if we will have some physical cpus with different
> > operation points (for example 2 cpus) and we give only one cpu for dom0?
> >
> > How should I transfer all information from the original cpu@0..cpu@n
> nodes
> > about all physical cpus to the kernel dom0 driver? Maybe an additional
> > nodes should be created by the hypervisor in the device tree for dom0
> > and named as pcpu@0..pcpu@n?
>
> If we do that, wouldn't we require changes to the core OMAP drivers or
> cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> nodes)?  I don't expect they would be easy to upstream or maintain going
> forward.
>
> I am trying to think of an alternative, such as passing the real cpu
> nodes to dom0 but then adding status = "disabled", but I am not sure
> whether Linux checks the status for cpu nodes. In addition this scheme
> wouldn't support the case where dom0 has more vcpus than pcpus on the
> system. Granted it is not very common and might even be detrimental for
> performances, but we should be able to support it.
>

In case where dom0 has more vcpus than pcpus on the
system, the dom0 kernel *is the most bug-prone place* for pcpu cpufreq
governor. So I still believe that separate driver domain with direct 1:1
vcpu:pcpu mapping is the best place for cpufreq governor. But it also
reasonable to run cpufreq governor as userspace daemon in dom0.

Also what do you think about PM QoS support? On bare metal cpufreq is
tightly integrated with PM QoS and intensively cooperate in frequency
scaling.

With best regards,


> Ian, what do you think about this?
>
>
>
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
> > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > <andrii.tseglytskyi@globallogic.com> wrote:
> > >
> > > Hi Stefano,
> > >
> > > Thank you for explanation.
> > > I think this requires more and deeper investigation, but for sure dom0
> > > must be able to do this.
> > > Let us investigate this.
> > >
> > > Thank you,
> > >
> > > Regards,
> > > Andrii
> > >
> > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > >> >> Hi,
> > > >> >>
> > > >> >> Stefano, Ian,
> > > >> >>
> > > >> >> Could you please clarify the following point:
> > > >> >>
> > > >> >> I agree that decision about frequency change should be taken by
> Xen
> > > >> >> hypervisor. But what about hardware frequency changing?
> > > >> >> In general when frequency changed to bigger value (for example
> from 1
> > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the
> following:
> > > >> >>
> > > >> >> 1) cpufreq governor decides that frequency should be changed.
> This
> > > >> >> decision is taken after analysing of CPU performance data taking
> in
> > > >> >> account governor policy.
> > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > > >> >> 3) cpufreq driver compares current and target frequencies and
> asks
> > > >> >> cpufreq regulator about voltage change.
> > > >> >> 4) cpufreq regulator send i2c command to standalone microchip,
> which
> > > >> >> is responsible for voltage changing.
> > > >> >> 5) cpufreq driver asks clock framework about new frequency for
> CPU clock
> > > >> >> 6) clock framework performs frequency sanity checks, taking in
> account
> > > >> >> clock parents and clock divider settings, and call platform
> specific
> > > >> >> "set_frequency" callback.
> > > >> >> 7) platform specific callback performs proper HW registers
> > > >> >> configuration for newly selected frequency
> > > >> >>
> > > >> >> Also there are some special cases - for example for OMAP5+ when
> > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > > >> >>
> > > >> >> So, for generic ARM kernel we have 3 entities to change
> frequency:
> > > >> >>
> > > >> >> - cpufreq governor
> > > >> >> - cpufreq driver
> > > >> >> - cpufreq regulator
> > > >> >>
> > > >> >> + 2 additional IP for OMAP5+
> > > >> >> - ABB
> > > >> >> - DCC
> > > >> >>
> > > >> >> Taking in account all above, it looks like it would be better to
> > > >> >> implement only Xen cpufreq governor. Xen will take a decision
> about
> > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0
> contains
> > > >> >> all generic and platform specific frameworks, needed for
> frequency
> > > >> >> changing.
> > > >> >>
> > > >> >> What do you think ?
> > > >> >
> > > >> > Keep in mind that the architecture must be able to handle the
> case where
> > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > > >> > physical cpus.
> > > >> > Could dom0 change the frequency of a physical core or a physical
> cpu is
> > > >> > not even running on? If that is not a problem, because cpus and
> > > >> > frequency changing are decoupled enough in Linux to allow it,
> then I am
> > > >> > OK with it. But I suspect they are not.
> > > >> >
> > > >>
> > > >> Not sure that I got your point correctly - dom0 will change
> frequency
> > > >> on physical CPU.
> > > >> And in case of OMAP - this changing affects on both ARM physical
> cpus
> > > >> - changing is coupled.
> > > >> In case of other ARM platforms - changing may be not coupled (I've
> > > >> heard that Snapdragon can change cpu freqs independently on each
> > > >> physical cpu)
> > > >
> > > > Let me explain with a concrete example.
> > > >
> > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
> > > > running on core0 and core1 of cpu0.
> > > >
> > > > In this case would dom0 be able to change the frequency of core3 of
> > > > cpu1, given that is not even running on it?
> > > > If it can be done without any hacks, then we can go ahead with this
> > > > approach.
> > >
> > >
> > >
> > > --
> > >
> > > Andrii Tseglytskyi | Embedded Dev
> > > GlobalLogic
> > > www.globallogic.com
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>



-- 
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 11085 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Tue, 9 Sep 2014, Ian Campbell wrote:
> On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > I am trying to think of an alternative, such as passing the real cpu
> > nodes to dom0 but then adding status = "disabled", but I am not sure
> > whether Linux checks the status for cpu nodes.
> 
> status = "disabled" is defined to have a specific (i.e. non-default)
> meaning for cpu nodes, Julien mentioned this when I tried to add a
> similar patch to Xen to ignore them. I think it basically means "present
> but not running, you should start them!".
> 
> >  In addition this scheme
> > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > system. Granted it is not very common and might even be detrimental for
> > performances, but we should be able to support it.
> 
> It's a bit of an edge case, for sure. I guess it wouldn't be totally
> unreasonable to say that if you use this sort of configuration you may
> not get cpufreq support.
> 
> > Ian, what do you think about this?
> 
> All the options suck in one way or another AFAICT. I think we are going
> to be looking for the least bad solution not necessarily a good one.
> 
> Fundamentally are we trying to avoid having to have a i2c subsystem etc
> in the hypervisor to be be able to change the voltages before/after
> changing the frequency?
>
> We can't just say "that's part of the cpufreq driver" since different
> boards using the same SoC might use different voltage regulators, over
> i2c or some other bus etc, so we end up with a matrix.
> 
> It's arguable that we should be letting dom0 poke at that regulator
> functionality anyway, at least not all of it. Taking that ability away
> would necessarily imply more platform specific functionality in the
> hypervisor.

Right.
I am afraid that in order to avoid more code in Xen, we end up with an
unmaintainable interface and unupstreamable hacks in dom0.


> How does this stuff work on x86?

As far as I can tell, after an initial version based on Dom0 doing the
work, the functionality has been moved into the hypervisor.
Of course doing that is easier on x86 where differences across
platforms are more limited.


> Sorry, more questions than answers here.
> 
> Ian.
> 
> > > Oleksandr Dmytryshyn | Product Engineering and Development
> > > GlobalLogic
> > > M +38.067.382.2525
> > > www.globallogic.com
> > > 
> > > http://www.globallogic.com/email_disclaimer.txt
> > > 
> > > 
> > > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > > <andrii.tseglytskyi@globallogic.com> wrote:
> > > >
> > > > Hi Stefano,
> > > >
> > > > Thank you for explanation.
> > > > I think this requires more and deeper investigation, but for sure dom0
> > > > must be able to do this.
> > > > Let us investigate this.
> > > >
> > > > Thank you,
> > > >
> > > > Regards,
> > > > Andrii
> > > >
> > > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > > > >> >> Hi,
> > > > >> >>
> > > > >> >> Stefano, Ian,
> > > > >> >>
> > > > >> >> Could you please clarify the following point:
> > > > >> >>
> > > > >> >> I agree that decision about frequency change should be taken by Xen
> > > > >> >> hypervisor. But what about hardware frequency changing?
> > > > >> >> In general when frequency changed to bigger value (for example from 1
> > > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> > > > >> >>
> > > > >> >> 1) cpufreq governor decides that frequency should be changed. This
> > > > >> >> decision is taken after analysing of CPU performance data taking in
> > > > >> >> account governor policy.
> > > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> > > > >> >> 3) cpufreq driver compares current and target frequencies and asks
> > > > >> >> cpufreq regulator about voltage change.
> > > > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> > > > >> >> is responsible for voltage changing.
> > > > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> > > > >> >> 6) clock framework performs frequency sanity checks, taking in account
> > > > >> >> clock parents and clock divider settings, and call platform specific
> > > > >> >> "set_frequency" callback.
> > > > >> >> 7) platform specific callback performs proper HW registers
> > > > >> >> configuration for newly selected frequency
> > > > >> >>
> > > > >> >> Also there are some special cases - for example for OMAP5+ when
> > > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> > > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> > > > >> >>
> > > > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> > > > >> >>
> > > > >> >> - cpufreq governor
> > > > >> >> - cpufreq driver
> > > > >> >> - cpufreq regulator
> > > > >> >>
> > > > >> >> + 2 additional IP for OMAP5+
> > > > >> >> - ABB
> > > > >> >> - DCC
> > > > >> >>
> > > > >> >> Taking in account all above, it looks like it would be better to
> > > > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> > > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> > > > >> >> all generic and platform specific frameworks, needed for frequency
> > > > >> >> changing.
> > > > >> >>
> > > > >> >> What do you think ?
> > > > >> >
> > > > >> > Keep in mind that the architecture must be able to handle the case where
> > > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> > > > >> > physical cpus.
> > > > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> > > > >> > not even running on? If that is not a problem, because cpus and
> > > > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> > > > >> > OK with it. But I suspect they are not.
> > > > >> >
> > > > >>
> > > > >> Not sure that I got your point correctly - dom0 will change frequency
> > > > >> on physical CPU.
> > > > >> And in case of OMAP - this changing affects on both ARM physical cpus
> > > > >> - changing is coupled.
> > > > >> In case of other ARM platforms - changing may be not coupled (I've
> > > > >> heard that Snapdragon can change cpu freqs independently on each
> > > > >> physical cpu)
> > > > >
> > > > > Let me explain with a concrete example.
> > > > >
> > > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> > > > > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
> > > > > running on core0 and core1 of cpu0.
> > > > >
> > > > > In this case would dom0 be able to change the frequency of core3 of
> > > > > cpu1, given that is not even running on it?
> > > > > If it can be done without any hacks, then we can go ahead with this
> > > > > approach.
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Andrii Tseglytskyi | Embedded Dev
> > > > GlobalLogic
> > > > www.globallogic.com
> > > 
> 
> 

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Tue, 9 Sep 2014, Oleksandr Dmytryshyn wrote:
> On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> >> Hi to all.
> >>
> >> I want to implement cpufreq driver in the next way:
> >> 1. Cpufreq governor will be implemented in the Xen
> >> 2. dom0 will only change cpu frequency and voltage of the physical cpus
> >>
> >> But there are some nuances:
> >> 1. dom0 driver should read an information about operation points
> >> (frequencies and voltages) and cpu supply source from the device tree for each
> >> physical cpu. In the omap processor case this driver suspects that
> >> those settings
> >> located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
> >> for each vcpu
> >> for kernel dom0 in the device tree and those information is lost in the dom0.
> >> 2. What about this case if we will have some physical cpus with different
> >> operation points (for example 2 cpus) and we give only one cpu for dom0?
> >>
> >> How should I transfer all information from the original cpu@0..cpu@n nodes
> >> about all physical cpus to the kernel dom0 driver? Maybe an additional
> >> nodes should be created by the hypervisor in the device tree for dom0
> >> and named as pcpu@0..pcpu@n?
> >
> > If we do that, wouldn't we require changes to the core OMAP drivers or
> > cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> > nodes)?  I don't expect they would be easy to upstream or maintain going
> > forward.
> >
> > I am trying to think of an alternative, such as passing the real cpu
> > nodes to dom0 but then adding status = "disabled", but I am not sure
> > whether Linux checks the status for cpu nodes. In addition this scheme
> > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > system. Granted it is not very common and might even be detrimental for
> > performances, but we should be able to support it.
> >
> > Ian, what do you think about this?
> >
> >
> >
> >> Oleksandr Dmytryshyn | Product Engineering and Development
> >> GlobalLogic
> >> M +38.067.382.2525
> >> www.globallogic.com
> >>
> >> http://www.globallogic.com/email_disclaimer.txt
> >>
> >>
> >> On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> >> <andrii.tseglytskyi@globallogic.com> wrote:
> >> >
> >> > Hi Stefano,
> >> >
> >> > Thank you for explanation.
> >> > I think this requires more and deeper investigation, but for sure dom0
> >> > must be able to do this.
> >> > Let us investigate this.
> >> >
> >> > Thank you,
> >> >
> >> > Regards,
> >> > Andrii
> >> >
> >> > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> >> > <stefano.stabellini@eu.citrix.com> wrote:
> >> > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> >> > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> >> > >> <stefano.stabellini@eu.citrix.com> wrote:
> >> > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> >> > >> >> Hi,
> >> > >> >>
> >> > >> >> Stefano, Ian,
> >> > >> >>
> >> > >> >> Could you please clarify the following point:
> >> > >> >>
> >> > >> >> I agree that decision about frequency change should be taken by Xen
> >> > >> >> hypervisor. But what about hardware frequency changing?
> >> > >> >> In general when frequency changed to bigger value (for example from 1
> >> > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> >> > >> >>
> >> > >> >> 1) cpufreq governor decides that frequency should be changed. This
> >> > >> >> decision is taken after analysing of CPU performance data taking in
> >> > >> >> account governor policy.
> >> > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> >> > >> >> 3) cpufreq driver compares current and target frequencies and asks
> >> > >> >> cpufreq regulator about voltage change.
> >> > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> >> > >> >> is responsible for voltage changing.
> >> > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> >> > >> >> 6) clock framework performs frequency sanity checks, taking in account
> >> > >> >> clock parents and clock divider settings, and call platform specific
> >> > >> >> "set_frequency" callback.
> >> > >> >> 7) platform specific callback performs proper HW registers
> >> > >> >> configuration for newly selected frequency
> >> > >> >>
> >> > >> >> Also there are some special cases - for example for OMAP5+ when
> >> > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> >> > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> >> > >> >>
> >> > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> >> > >> >>
> >> > >> >> - cpufreq governor
> >> > >> >> - cpufreq driver
> >> > >> >> - cpufreq regulator
> >> > >> >>
> >> > >> >> + 2 additional IP for OMAP5+
> >> > >> >> - ABB
> >> > >> >> - DCC
> >> > >> >>
> >> > >> >> Taking in account all above, it looks like it would be better to
> >> > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> >> > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> >> > >> >> all generic and platform specific frameworks, needed for frequency
> >> > >> >> changing.
> >> > >> >>
> >> > >> >> What do you think ?
> >> > >> >
> >> > >> > Keep in mind that the architecture must be able to handle the case where
> >> > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> >> > >> > physical cpus.
> >> > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> >> > >> > not even running on? If that is not a problem, because cpus and
> >> > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> >> > >> > OK with it. But I suspect they are not.
> >> > >> >
> >> > >>
> >> > >> Not sure that I got your point correctly - dom0 will change frequency
> >> > >> on physical CPU.
> >> > >> And in case of OMAP - this changing affects on both ARM physical cpus
> >> > >> - changing is coupled.
> >> > >> In case of other ARM platforms - changing may be not coupled (I've
> >> > >> heard that Snapdragon can change cpu freqs independently on each
> >> > >> physical cpu)
> >> > >
> >> > > Let me explain with a concrete example.
> >> > >
> >> > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> >> > > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
> >> > > running on core0 and core1 of cpu0.
> >> > >
> >> > > In this case would dom0 be able to change the frequency of core3 of
> >> > > cpu1, given that is not even running on it?
> >> > > If it can be done without any hacks, then we can go ahead with this
> >> > > approach.
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Andrii Tseglytskyi | Embedded Dev
> >> > GlobalLogic
> >> > www.globallogic.com
> >>
> 
> Hi to all.
> 
> I've done next work to check how dom0 can change the frequency of the cpus.
> 1. I've written a small HACK in the hypervisor which copies all settings
> from the physical cpu0 to the vcpu0.
> Thus kernel can read information about OPP states and power supply.
> 2. I've turned on the cpufreq driver in kernel dom0.
> 3. I've turned on all needed regulators.
> 4. I've reverted a patch whitch disables CPUFREQ in kernel dom0 when
> it is running under the xen.
> 
> Now kernel dom0 has 2 virtual cpus and board has 2 physical cpus.
> CPUFREQ driver uses vcpus as source to calculate frequency and changes this
> frequency on the physical cpus.
> 
> I've checked that in this case kernel dom0 can change the frequency of the cpus.
> 
> I want to disable CPUFREQ governor driver in dom0 and leave only cpu0-cpufreq
> driver (driver for OMAP processors). CPUFREQ governor driver will be implemented
> in the xen.
> 
> Here are some questions:
> 
> 1. How implement an mechanism which allows the hypervisor to give
> commands to the
> CPUFREQ driver in the dom0 (i. e. 'set frequency')?

We could set up an event channel for that. Dom0 would receive an
evtchn_irq interrupt and check what needs to be done.


> 2. Could we use a limitation when the number of VCPUs is domain0 must
> be equal to
> the number of physical CPUs, and the domain0 VCPU must be pinned to
> the respective
> physical CPU? This limitation us used for the existing Domain0 based cpufreq.

I think that the limitation would drastically limit the usefulness of the
solution. I wouldn't want to have to resort to this.
Nowadays it is extremely common to have dom0 running with fewer vcpus
that the amount of pcpus in the system.

Unless there is a way to tell Linux to leave the vcpus idle all the time.


> 3. If we will use this limitation can I simply copy all settings from physical
> cpus to the appropriate vcpus in the device tree in xen? In this case
> the existing
> kernel code will not be modified and simple CPUFREQ driver will be written
> which will receive commands from xen and redirect them to the cpu0-cpufreq
> driver (or another registered cpufreq driver for any cpu).

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 9159 bytes --]

On Tue, 9 Sep 2014, Vitaly Chernooky wrote:
> Hi All!
> 
> On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
>       On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
>       > Hi to all.
>       >
>       > I want to implement cpufreq driver in the next way:
>       > 1. Cpufreq governor will be implemented in the Xen
>       > 2. dom0 will only change cpu frequency and voltage of the physical cpus
>       >
>       > But there are some nuances:
>       > 1. dom0 driver should read an information about operation points
>       > (frequencies and voltages) and cpu supply source from the device tree for each
>       > physical cpu. In the omap processor case this driver suspects that
>       > those settings
>       > located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
>       > for each vcpu
>       > for kernel dom0 in the device tree and those information is lost in the dom0.
>       > 2. What about this case if we will have some physical cpus with different
>       > operation points (for example 2 cpus) and we give only one cpu for dom0?
>       >
>       > How should I transfer all information from the original cpu@0..cpu@n nodes
>       > about all physical cpus to the kernel dom0 driver? Maybe an additional
>       > nodes should be created by the hypervisor in the device tree for dom0
>       > and named as pcpu@0..pcpu@n?
> 
>       If we do that, wouldn't we require changes to the core OMAP drivers or
>       cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
>       nodes)?  I don't expect they would be easy to upstream or maintain going
>       forward.
> 
>       I am trying to think of an alternative, such as passing the real cpu
>       nodes to dom0 but then adding status = "disabled", but I am not sure
>       whether Linux checks the status for cpu nodes. In addition this scheme
>       wouldn't support the case where dom0 has more vcpus than pcpus on the
>       system. Granted it is not very common and might even be detrimental for
>       performances, but we should be able to support it.
> 
> 
> In case where dom0 has more vcpus than pcpus on the
> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq governor. So I still believe that separate driver
> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq governor. But it also reasonable to run cpufreq
> governor as userspace daemon in dom0.
> 
> Also what do you think about PM QoS support? On bare metal cpufreq is tightly integrated with PM QoS and intensively
> cooperate in frequency scaling.

Device PM needs to be done in Dom0.
CPU an Platform level PM architecturally belongs to Xen, but I do
understand that to do that in Xen we would need to add lots of code to
the hypervisor. There is no silver bullet here.

A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
are you going to use for that? Linux? Wouldn't Linux be too big for a
cpufreq driver domain, especially in embedded deployments? I think it
would need at least 32MB to run.


> With best regards,
>  
>       Ian, what do you think about this?
> 
> 
> 
>       > Oleksandr Dmytryshyn | Product Engineering and Development
>       > GlobalLogic
>       > M +38.067.382.2525
>       > www.globallogic.com
>       >
>       > http://www.globallogic.com/email_disclaimer.txt
>       >
>       >
>       > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
>       > <andrii.tseglytskyi@globallogic.com> wrote:
>       > >
>       > > Hi Stefano,
>       > >
>       > > Thank you for explanation.
>       > > I think this requires more and deeper investigation, but for sure dom0
>       > > must be able to do this.
>       > > Let us investigate this.
>       > >
>       > > Thank you,
>       > >
>       > > Regards,
>       > > Andrii
>       > >
>       > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
>       > > <stefano.stabellini@eu.citrix.com> wrote:
>       > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
>       > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
>       > > >> <stefano.stabellini@eu.citrix.com> wrote:
>       > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
>       > > >> >> Hi,
>       > > >> >>
>       > > >> >> Stefano, Ian,
>       > > >> >>
>       > > >> >> Could you please clarify the following point:
>       > > >> >>
>       > > >> >> I agree that decision about frequency change should be taken by Xen
>       > > >> >> hypervisor. But what about hardware frequency changing?
>       > > >> >> In general when frequency changed to bigger value (for example from 1
>       > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
>       > > >> >>
>       > > >> >> 1) cpufreq governor decides that frequency should be changed. This
>       > > >> >> decision is taken after analysing of CPU performance data taking in
>       > > >> >> account governor policy.
>       > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
>       > > >> >> 3) cpufreq driver compares current and target frequencies and asks
>       > > >> >> cpufreq regulator about voltage change.
>       > > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
>       > > >> >> is responsible for voltage changing.
>       > > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
>       > > >> >> 6) clock framework performs frequency sanity checks, taking in account
>       > > >> >> clock parents and clock divider settings, and call platform specific
>       > > >> >> "set_frequency" callback.
>       > > >> >> 7) platform specific callback performs proper HW registers
>       > > >> >> configuration for newly selected frequency
>       > > >> >>
>       > > >> >> Also there are some special cases - for example for OMAP5+ when
>       > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
>       > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
>       > > >> >>
>       > > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
>       > > >> >>
>       > > >> >> - cpufreq governor
>       > > >> >> - cpufreq driver
>       > > >> >> - cpufreq regulator
>       > > >> >>
>       > > >> >> + 2 additional IP for OMAP5+
>       > > >> >> - ABB
>       > > >> >> - DCC
>       > > >> >>
>       > > >> >> Taking in account all above, it looks like it would be better to
>       > > >> >> implement only Xen cpufreq governor. Xen will take a decision about
>       > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
>       > > >> >> all generic and platform specific frameworks, needed for frequency
>       > > >> >> changing.
>       > > >> >>
>       > > >> >> What do you think ?
>       > > >> >
>       > > >> > Keep in mind that the architecture must be able to handle the case where
>       > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
>       > > >> > physical cpus.
>       > > >> > Could dom0 change the frequency of a physical core or a physical cpu is
>       > > >> > not even running on? If that is not a problem, because cpus and
>       > > >> > frequency changing are decoupled enough in Linux to allow it, then I am
>       > > >> > OK with it. But I suspect they are not.
>       > > >> >
>       > > >>
>       > > >> Not sure that I got your point correctly - dom0 will change frequency
>       > > >> on physical CPU.
>       > > >> And in case of OMAP - this changing affects on both ARM physical cpus
>       > > >> - changing is coupled.
>       > > >> In case of other ARM platforms - changing may be not coupled (I've
>       > > >> heard that Snapdragon can change cpu freqs independently on each
>       > > >> physical cpu)
>       > > >
>       > > > Let me explain with a concrete example.
>       > > >
>       > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
>       > > > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
>       > > > running on core0 and core1 of cpu0.
>       > > >
>       > > > In this case would dom0 be able to change the frequency of core3 of
>       > > > cpu1, given that is not even running on it?
>       > > > If it can be done without any hacks, then we can go ahead with this
>       > > > approach.
>       > >
>       > >
>       > >
>       > > --
>       > >
>       > > Andrii Tseglytskyi | Embedded Dev
>       > > GlobalLogic
>       > > www.globallogic.com
>       >
> 
>       _______________________________________________
>       Xen-devel mailing list
>       Xen-devel@lists.xen.org
>       http://lists.xen.org/xen-devel
> 
> 
> 
> 
> --
> Vitaly Chernooky | Senior Developer - Product Engineering and Development
> GlobalLogic
> P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> www.globallogic.com
> 
> http://www.globallogic.com/email_disclaimer.txt
> 
> 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Ian Campbell]

On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> On Tue, 9 Sep 2014, Ian Campbell wrote:
> > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > > I am trying to think of an alternative, such as passing the real cpu
> > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > > whether Linux checks the status for cpu nodes.
> > 
> > status = "disabled" is defined to have a specific (i.e. non-default)
> > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > similar patch to Xen to ignore them. I think it basically means "present
> > but not running, you should start them!".
> > 
> > >  In addition this scheme
> > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > > system. Granted it is not very common and might even be detrimental for
> > > performances, but we should be able to support it.
> > 
> > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > unreasonable to say that if you use this sort of configuration you may
> > not get cpufreq support.
> > 
> > > Ian, what do you think about this?
> > 
> > All the options suck in one way or another AFAICT. I think we are going
> > to be looking for the least bad solution not necessarily a good one.
> > 
> > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > in the hypervisor to be be able to change the voltages before/after
> > changing the frequency?
> >
> > We can't just say "that's part of the cpufreq driver" since different
> > boards using the same SoC might use different voltage regulators, over
> > i2c or some other bus etc, so we end up with a matrix.
> > 
> > It's arguable that we should be letting dom0 poke at that regulator
> > functionality anyway, at least not all of it. Taking that ability away
> > would necessarily imply more platform specific functionality in the
> > hypervisor.
> 
> Right.
> I am afraid that in order to avoid more code in Xen, we end up with an
> unmaintainable interface and unupstreamable hacks in dom0.

That's what I'm worried about to. Hence I'm wondering if we should just
do this in the hypervisor.

Although there are a myriad of them the parts used to do voltage control
tend to be fairly simple.

One concern I have is that i2c busses also tend to have other things on
them which dom0 might legitimately access (e.g. rtc), I'm not sure what
to suggest here.

> > How does this stuff work on x86?
> 
> As far as I can tell, after an initial version based on Dom0 doing the
> work, the functionality has been moved into the hypervisor.
> Of course doing that is easier on x86 where differences across
> platforms are more limited.

This chimes with my limited understanding. I think we still need a dom0
component to feed stuff down from ACPI to let Xen do its work, is that
right? Or do we go under the hood and ignore ACPI PM somehow?

Ian.

Re: cpufreq implementation for OMAP under xen hypervisor.

[Andrii Tseglytskyi]

Hi,

>> In case where dom0 has more vcpus than pcpus on the
>> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq governor. So I still believe that separate driver
>> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq governor. But it also reasonable to run cpufreq
>> governor as userspace daemon in dom0.
>>
>> Also what do you think about PM QoS support? On bare metal cpufreq is tightly integrated with PM QoS and intensively
>> cooperate in frequency scaling.
>
> Device PM needs to be done in Dom0.
> CPU an Platform level PM architecturally belongs to Xen, but I do
> understand that to do that in Xen we would need to add lots of code to
> the hypervisor. There is no silver bullet here.
>
> A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> are you going to use for that? Linux? Wouldn't Linux be too big for a
> cpufreq driver domain, especially in embedded deployments? I think it
> would need at least 32MB to run.
>

I would suggest not to do this. Driver domain will need to share the
same HW with dom0, which is impossible to implement. Good examples are
I2C, which is needed for Display and for Cpufreq, and it cannot be
shared between domains. (At least in current Linux kernel it won't be
easy to implement). dom0 is the best place to do cpufreq.

Regards,
Andrii


-- 

Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com

Re: cpufreq implementation for OMAP under xen hypervisor.

[Andrii Tseglytskyi]

Hi,

On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>
> On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > > > I am trying to think of an alternative, such as passing the real cpu
> > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > > > whether Linux checks the status for cpu nodes.
> > >
> > > status = "disabled" is defined to have a specific (i.e. non-default)
> > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > > similar patch to Xen to ignore them. I think it basically means "present
> > > but not running, you should start them!".
> > >
> > > >  In addition this scheme
> > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > > > system. Granted it is not very common and might even be detrimental for
> > > > performances, but we should be able to support it.
> > >
> > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > > unreasonable to say that if you use this sort of configuration you may
> > > not get cpufreq support.
> > >
> > > > Ian, what do you think about this?
> > >
> > > All the options suck in one way or another AFAICT. I think we are going
> > > to be looking for the least bad solution not necessarily a good one.
> > >
> > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > > in the hypervisor to be be able to change the voltages before/after
> > > changing the frequency?
> > >
> > > We can't just say "that's part of the cpufreq driver" since different
> > > boards using the same SoC might use different voltage regulators, over
> > > i2c or some other bus etc, so we end up with a matrix.
> > >
> > > It's arguable that we should be letting dom0 poke at that regulator
> > > functionality anyway, at least not all of it. Taking that ability away
> > > would necessarily imply more platform specific functionality in the
> > > hypervisor.
> >
> > Right.
> > I am afraid that in order to avoid more code in Xen, we end up with an
> > unmaintainable interface and unupstreamable hacks in dom0.
>
> That's what I'm worried about to. Hence I'm wondering if we should just
> do this in the hypervisor.
>
> Although there are a myriad of them the parts used to do voltage control
> tend to be fairly simple.
>
> One concern I have is that i2c busses also tend to have other things on
> them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> to suggest here.

I would try to avoid i2c transactions in Xen. I2C driver is quite
complicated in Linux kernel. It consists of several parts - common
core + platform specific. I'm pretty sure Xen should not handle this.
I think that establishing of event channel for frequency changing is a
good idea. It would be good to try to implement this. In process of
implementation we will see what is need to be resolved. The only
question here is how to pass physical cpu to dom0.

Regarding x86.
I'm not sure but maybe ACPI interface encapsulate voltage changing as well?

Regards,
Andrii


-- 

Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly Chernooky]

[-- Attachment #1.1: Type: text/plain, Size: 1896 bytes --]

Hi Andrii,


On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi <
andrii.tseglytskyi@globallogic.com> wrote:

> Hi,
>
> >> In case where dom0 has more vcpus than pcpus on the
> >> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq
> governor. So I still believe that separate driver
> >> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq
> governor. But it also reasonable to run cpufreq
> >> governor as userspace daemon in dom0.
> >>
> >> Also what do you think about PM QoS support? On bare metal cpufreq is
> tightly integrated with PM QoS and intensively
> >> cooperate in frequency scaling.
> >
> > Device PM needs to be done in Dom0.
> > CPU an Platform level PM architecturally belongs to Xen, but I do
> > understand that to do that in Xen we would need to add lots of code to
> > the hypervisor. There is no silver bullet here.
> >
> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> > are you going to use for that? Linux? Wouldn't Linux be too big for a
> > cpufreq driver domain, especially in embedded deployments? I think it
> > would need at least 32MB to run.
> >
>
> I would suggest not to do this. Driver domain will need to share the
> same HW with dom0, which is impossible to implement. Good examples are
> I2C, which is needed for Display and for Cpufreq, and it cannot be
> shared between domains. (At least in current Linux kernel it won't be
> easy to implement). dom0 is the best place to do cpufreq.
>

What do you think about cpufreq in dom0 userspace?

With best regards,


> Regards,
> Andrii
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
>



-- 
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 3049 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Andrii Tseglytskyi]

On Wed, Sep 10, 2014 at 1:24 PM, Vitaly Chernooky
<vitalii.chernookyi@globallogic.com> wrote:
> Hi Andrii,
>
>
> On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi
> <andrii.tseglytskyi@globallogic.com> wrote:
>>
>> Hi,
>>
>> >> In case where dom0 has more vcpus than pcpus on the
>> >> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq
>> >> governor. So I still believe that separate driver
>> >> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq
>> >> governor. But it also reasonable to run cpufreq
>> >> governor as userspace daemon in dom0.
>> >>
>> >> Also what do you think about PM QoS support? On bare metal cpufreq is
>> >> tightly integrated with PM QoS and intensively
>> >> cooperate in frequency scaling.
>> >
>> > Device PM needs to be done in Dom0.
>> > CPU an Platform level PM architecturally belongs to Xen, but I do
>> > understand that to do that in Xen we would need to add lots of code to
>> > the hypervisor. There is no silver bullet here.
>> >
>> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
>> > are you going to use for that? Linux? Wouldn't Linux be too big for a
>> > cpufreq driver domain, especially in embedded deployments? I think it
>> > would need at least 32MB to run.
>> >
>>
>> I would suggest not to do this. Driver domain will need to share the
>> same HW with dom0, which is impossible to implement. Good examples are
>> I2C, which is needed for Display and for Cpufreq, and it cannot be
>> shared between domains. (At least in current Linux kernel it won't be
>> easy to implement). dom0 is the best place to do cpufreq.
>
>
> What do you think about cpufreq in dom0 userspace?
>

What do you mean? Could you please explain with more details?

Regards,
Andrii



-- 

Andrii Tseglytskyi | Embedded Dev
GlobalLogic
www.globallogic.com

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly Chernooky]

[-- Attachment #1.1: Type: text/plain, Size: 1893 bytes --]

On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi <
andrii.tseglytskyi@globallogic.com> wrote:

> Hi,
>
> >> In case where dom0 has more vcpus than pcpus on the
> >> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq
> governor. So I still believe that separate driver
> >> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq
> governor. But it also reasonable to run cpufreq
> >> governor as userspace daemon in dom0.
> >>
> >> Also what do you think about PM QoS support? On bare metal cpufreq is
> tightly integrated with PM QoS and intensively
> >> cooperate in frequency scaling.
> >
> > Device PM needs to be done in Dom0.
> > CPU an Platform level PM architecturally belongs to Xen, but I do
> > understand that to do that in Xen we would need to add lots of code to
> > the hypervisor. There is no silver bullet here.
> >
> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> > are you going to use for that? Linux? Wouldn't Linux be too big for a
> > cpufreq driver domain, especially in embedded deployments? I think it
> > would need at least 32MB to run.
> >
>
> I would suggest not to do this. Driver domain will need to share the
> same HW with dom0, which is impossible to implement. Good examples are
> I2C, which is needed for Display and for Cpufreq, and it cannot be
> shared between domains. (At least in current Linux kernel it won't be
> easy to implement). dom0 is the best place to do cpufreq.
>

Also how do you feel about providing virtualized i2c from driver domain to
dom0?


> Regards,
> Andrii
>
>
> --
>
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
>



-- 
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 3025 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly Chernooky]

[-- Attachment #1.1: Type: text/plain, Size: 10026 bytes --]

I've intensively discussed my suggestions here and now it is transparent to
me that we should not try to use Cpufreq on ARM SoCs without direct 1:1
pcpu:vcpu mapping in dom0. So if someone want to break 1:1 mapping he
should forget Cpufreq.

With best regards,


On Wed, Sep 10, 2014 at 12:58 AM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:

> On Tue, 9 Sep 2014, Vitaly Chernooky wrote:
> > Hi All!
> >
> > On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <
> stefano.stabellini@eu.citrix.com> wrote:
> >       On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> >       > Hi to all.
> >       >
> >       > I want to implement cpufreq driver in the next way:
> >       > 1. Cpufreq governor will be implemented in the Xen
> >       > 2. dom0 will only change cpu frequency and voltage of the
> physical cpus
> >       >
> >       > But there are some nuances:
> >       > 1. dom0 driver should read an information about operation points
> >       > (frequencies and voltages) and cpu supply source from the device
> tree for each
> >       > physical cpu. In the omap processor case this driver suspects
> that
> >       > those settings
> >       > located in the /cpus/cpu@0/ node. But hypervisor creates an cpu
> node
> >       > for each vcpu
> >       > for kernel dom0 in the device tree and those information is lost
> in the dom0.
> >       > 2. What about this case if we will have some physical cpus with
> different
> >       > operation points (for example 2 cpus) and we give only one cpu
> for dom0?
> >       >
> >       > How should I transfer all information from the original cpu@0.
> .cpu@n nodes
> >       > about all physical cpus to the kernel dom0 driver? Maybe an
> additional
> >       > nodes should be created by the hypervisor in the device tree for
> dom0
> >       > and named as pcpu@0..pcpu@n?
> >
> >       If we do that, wouldn't we require changes to the core OMAP
> drivers or
> >       cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
> >       nodes)?  I don't expect they would be easy to upstream or maintain
> going
> >       forward.
> >
> >       I am trying to think of an alternative, such as passing the real
> cpu
> >       nodes to dom0 but then adding status = "disabled", but I am not
> sure
> >       whether Linux checks the status for cpu nodes. In addition this
> scheme
> >       wouldn't support the case where dom0 has more vcpus than pcpus on
> the
> >       system. Granted it is not very common and might even be
> detrimental for
> >       performances, but we should be able to support it.
> >
> >
> > In case where dom0 has more vcpus than pcpus on the
> > system, the dom0 kernel is the most bug-prone place for pcpu cpufreq
> governor. So I still believe that separate driver
> > domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq
> governor. But it also reasonable to run cpufreq
> > governor as userspace daemon in dom0.
> >
> > Also what do you think about PM QoS support? On bare metal cpufreq is
> tightly integrated with PM QoS and intensively
> > cooperate in frequency scaling.
>
> Device PM needs to be done in Dom0.
> CPU an Platform level PM architecturally belongs to Xen, but I do
> understand that to do that in Xen we would need to add lots of code to
> the hypervisor. There is no silver bullet here.
>
> A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> are you going to use for that? Linux? Wouldn't Linux be too big for a
> cpufreq driver domain, especially in embedded deployments? I think it
> would need at least 32MB to run.
>
>
> > With best regards,
> >
> >       Ian, what do you think about this?
> >
> >
> >
> >       > Oleksandr Dmytryshyn | Product Engineering and Development
> >       > GlobalLogic
> >       > M +38.067.382.2525
> >       > www.globallogic.com
> >       >
> >       > http://www.globallogic.com/email_disclaimer.txt
> >       >
> >       >
> >       > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> >       > <andrii.tseglytskyi@globallogic.com> wrote:
> >       > >
> >       > > Hi Stefano,
> >       > >
> >       > > Thank you for explanation.
> >       > > I think this requires more and deeper investigation, but for
> sure dom0
> >       > > must be able to do this.
> >       > > Let us investigate this.
> >       > >
> >       > > Thank you,
> >       > >
> >       > > Regards,
> >       > > Andrii
> >       > >
> >       > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> >       > > <stefano.stabellini@eu.citrix.com> wrote:
> >       > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> >       > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> >       > > >> <stefano.stabellini@eu.citrix.com> wrote:
> >       > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> >       > > >> >> Hi,
> >       > > >> >>
> >       > > >> >> Stefano, Ian,
> >       > > >> >>
> >       > > >> >> Could you please clarify the following point:
> >       > > >> >>
> >       > > >> >> I agree that decision about frequency change should be
> taken by Xen
> >       > > >> >> hypervisor. But what about hardware frequency changing?
> >       > > >> >> In general when frequency changed to bigger value (for
> example from 1
> >       > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the
> following:
> >       > > >> >>
> >       > > >> >> 1) cpufreq governor decides that frequency should be
> changed. This
> >       > > >> >> decision is taken after analysing of CPU performance
> data taking in
> >       > > >> >> account governor policy.
> >       > > >> >> 2) cpufreq governor asks cpufreq driver about new
> frequency.
> >       > > >> >> 3) cpufreq driver compares current and target
> frequencies and asks
> >       > > >> >> cpufreq regulator about voltage change.
> >       > > >> >> 4) cpufreq regulator send i2c command to standalone
> microchip, which
> >       > > >> >> is responsible for voltage changing.
> >       > > >> >> 5) cpufreq driver asks clock framework about new
> frequency for CPU clock
> >       > > >> >> 6) clock framework performs frequency sanity checks,
> taking in account
> >       > > >> >> clock parents and clock divider settings, and call
> platform specific
> >       > > >> >> "set_frequency" callback.
> >       > > >> >> 7) platform specific callback performs proper HW
> registers
> >       > > >> >> configuration for newly selected frequency
> >       > > >> >>
> >       > > >> >> Also there are some special cases - for example for
> OMAP5+ when
> >       > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs
> should be
> >       > > >> >> triggered (ABB and DCC, if someone is familiar with
> OMAP5+ )
> >       > > >> >>
> >       > > >> >> So, for generic ARM kernel we have 3 entities to change
> frequency:
> >       > > >> >>
> >       > > >> >> - cpufreq governor
> >       > > >> >> - cpufreq driver
> >       > > >> >> - cpufreq regulator
> >       > > >> >>
> >       > > >> >> + 2 additional IP for OMAP5+
> >       > > >> >> - ABB
> >       > > >> >> - DCC
> >       > > >> >>
> >       > > >> >> Taking in account all above, it looks like it would be
> better to
> >       > > >> >> implement only Xen cpufreq governor. Xen will take a
> decision about
> >       > > >> >> new frequency, and kernel dom0 will perform other steps.
> Dom0 contains
> >       > > >> >> all generic and platform specific frameworks, needed for
> frequency
> >       > > >> >> changing.
> >       > > >> >>
> >       > > >> >> What do you think ?
> >       > > >> >
> >       > > >> > Keep in mind that the architecture must be able to handle
> the case where
> >       > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with
> multiple
> >       > > >> > physical cpus.
> >       > > >> > Could dom0 change the frequency of a physical core or a
> physical cpu is
> >       > > >> > not even running on? If that is not a problem, because
> cpus and
> >       > > >> > frequency changing are decoupled enough in Linux to allow
> it, then I am
> >       > > >> > OK with it. But I suspect they are not.
> >       > > >> >
> >       > > >>
> >       > > >> Not sure that I got your point correctly - dom0 will change
> frequency
> >       > > >> on physical CPU.
> >       > > >> And in case of OMAP - this changing affects on both ARM
> physical cpus
> >       > > >> - changing is coupled.
> >       > > >> In case of other ARM platforms - changing may be not
> coupled (I've
> >       > > >> heard that Snapdragon can change cpu freqs independently on
> each
> >       > > >> physical cpu)
> >       > > >
> >       > > > Let me explain with a concrete example.
> >       > > >
> >       > > > Let's suppose that the platform has 2 physical cpus, each
> cpu has 4
> >       > > > cores.  Let's also supposed that dom0 has only 2 vcpus,
> currently
> >       > > > running on core0 and core1 of cpu0.
> >       > > >
> >       > > > In this case would dom0 be able to change the frequency of
> core3 of
> >       > > > cpu1, given that is not even running on it?
> >       > > > If it can be done without any hacks, then we can go ahead
> with this
> >       > > > approach.
> >       > >
> >       > >
> >       > >
> >       > > --
> >       > >
> >       > > Andrii Tseglytskyi | Embedded Dev
> >       > > GlobalLogic
> >       > > www.globallogic.com
> >       >
> >
> >       _______________________________________________
> >       Xen-devel mailing list
> >       Xen-devel@lists.xen.org
> >       http://lists.xen.org/xen-devel
> >
> >
> >
> >
> > --
> > Vitaly Chernooky | Senior Developer - Product Engineering and Development
> > GlobalLogic
> > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
>



-- 
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 14830 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> Hi,
> 
> On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >
> > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > > > > I am trying to think of an alternative, such as passing the real cpu
> > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > > > > whether Linux checks the status for cpu nodes.
> > > >
> > > > status = "disabled" is defined to have a specific (i.e. non-default)
> > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > > > similar patch to Xen to ignore them. I think it basically means "present
> > > > but not running, you should start them!".
> > > >
> > > > >  In addition this scheme
> > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > > > > system. Granted it is not very common and might even be detrimental for
> > > > > performances, but we should be able to support it.
> > > >
> > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > > > unreasonable to say that if you use this sort of configuration you may
> > > > not get cpufreq support.
> > > >
> > > > > Ian, what do you think about this?
> > > >
> > > > All the options suck in one way or another AFAICT. I think we are going
> > > > to be looking for the least bad solution not necessarily a good one.
> > > >
> > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > > > in the hypervisor to be be able to change the voltages before/after
> > > > changing the frequency?
> > > >
> > > > We can't just say "that's part of the cpufreq driver" since different
> > > > boards using the same SoC might use different voltage regulators, over
> > > > i2c or some other bus etc, so we end up with a matrix.
> > > >
> > > > It's arguable that we should be letting dom0 poke at that regulator
> > > > functionality anyway, at least not all of it. Taking that ability away
> > > > would necessarily imply more platform specific functionality in the
> > > > hypervisor.
> > >
> > > Right.
> > > I am afraid that in order to avoid more code in Xen, we end up with an
> > > unmaintainable interface and unupstreamable hacks in dom0.
> >
> > That's what I'm worried about to. Hence I'm wondering if we should just
> > do this in the hypervisor.
> >
> > Although there are a myriad of them the parts used to do voltage control
> > tend to be fairly simple.
> >
> > One concern I have is that i2c busses also tend to have other things on
> > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > to suggest here.
> 
> I would try to avoid i2c transactions in Xen. I2C driver is quite
> complicated in Linux kernel. It consists of several parts - common
> core + platform specific. I'm pretty sure Xen should not handle this.
> I think that establishing of event channel for frequency changing is a
> good idea. It would be good to try to implement this. In process of
> implementation we will see what is need to be resolved.

OK, that's reasonable.


> The only question here is how to pass physical cpu to dom0.

We can use a device tree based interface to pass the information to
dom0, but requiring a number of dom0 vcpus equal to the number of
physical cpus and in addition to that having to pin the vcpus each to a
different pcpu is quite a stringent limitation. However I don't know the
frequency changing interfaces in Linux well enough to know how hard
would be to lift it.


> Regarding x86.
> I'm not sure but maybe ACPI interface encapsulate voltage changing as well?

I think so (but I am not an expert on that).



> Regards,
> Andrii
> 
> 
> -- 
> 
> Andrii Tseglytskyi | Embedded Dev
> GlobalLogic
> www.globallogic.com
> 

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]

On Wed, 10 Sep 2014, Vitaly Chernooky wrote:
> Hi Andrii,
> 
> 
> On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi <andrii.tseglytskyi@globallogic.com> wrote:
>       Hi,
> 
>       >> In case where dom0 has more vcpus than pcpus on the
>       >> system, the dom0 kernel is the most bug-prone place for pcpu cpufreq governor. So I still believe that
>       separate driver
>       >> domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq governor. But it also reasonable to
>       run cpufreq
>       >> governor as userspace daemon in dom0.
>       >>
>       >> Also what do you think about PM QoS support? On bare metal cpufreq is tightly integrated with PM QoS and
>       intensively
>       >> cooperate in frequency scaling.
>       >
>       > Device PM needs to be done in Dom0.
>       > CPU an Platform level PM architecturally belongs to Xen, but I do
>       > understand that to do that in Xen we would need to add lots of code to
>       > the hypervisor. There is no silver bullet here.
>       >
>       > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
>       > are you going to use for that? Linux? Wouldn't Linux be too big for a
>       > cpufreq driver domain, especially in embedded deployments? I think it
>       > would need at least 32MB to run.
>       >
> 
>       I would suggest not to do this. Driver domain will need to share the
>       same HW with dom0, which is impossible to implement. Good examples are
>       I2C, which is needed for Display and for Cpufreq, and it cannot be
>       shared between domains. (At least in current Linux kernel it won't be
>       easy to implement). dom0 is the best place to do cpufreq.
> 
> 
> What do you think about cpufreq in dom0 userspace?

Would you write an i2c driver in userspace? How would you suggest to
do the frequency and voltage changes in dom0 userspace otherwise?


> With best regards,
>  
>       Regards,
>       Andrii
> 
> 
>       --
> 
>       Andrii Tseglytskyi | Embedded Dev
>       GlobalLogic
>       www.globallogic.com
> 
> 
> 
> 
> --
> Vitaly Chernooky | Senior Developer - Product Engineering and Development
> GlobalLogic
> P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> www.globallogic.com
> 
> http://www.globallogic.com/email_disclaimer.txt
> 
> 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 11387 bytes --]

The issue with that limitation is that it doesn't scale well on large
systems. You really wouldn't want dom0 to have 18 vcpus on a Xeon E5
because it would badly affect performances. Even on a 8 cores SoC it
would be best to assign less than 8 vcpus to dom0.

On Wed, 10 Sep 2014, Vitaly Chernooky wrote:
> I've intensively discussed my suggestions here and now it is transparent to me that we should not try to use Cpufreq on ARM
> SoCs without direct 1:1 pcpu:vcpu mapping in dom0. So if someone want to break 1:1 mapping he should forget Cpufreq.
> With best regards,
> 
> 
> On Wed, Sep 10, 2014 at 12:58 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
>       On Tue, 9 Sep 2014, Vitaly Chernooky wrote:
>       > Hi All!
>       >
>       > On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
>       >       On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
>       >       > Hi to all.
>       >       >
>       >       > I want to implement cpufreq driver in the next way:
>       >       > 1. Cpufreq governor will be implemented in the Xen
>       >       > 2. dom0 will only change cpu frequency and voltage of the physical cpus
>       >       >
>       >       > But there are some nuances:
>       >       > 1. dom0 driver should read an information about operation points
>       >       > (frequencies and voltages) and cpu supply source from the device tree for each
>       >       > physical cpu. In the omap processor case this driver suspects that
>       >       > those settings
>       >       > located in the /cpus/cpu@0/ node. But hypervisor creates an cpu node
>       >       > for each vcpu
>       >       > for kernel dom0 in the device tree and those information is lost in the dom0.
>       >       > 2. What about this case if we will have some physical cpus with different
>       >       > operation points (for example 2 cpus) and we give only one cpu for dom0?
>       >       >
>       >       > How should I transfer all information from the original cpu@0..cpu@n nodes
>       >       > about all physical cpus to the kernel dom0 driver? Maybe an additional
>       >       > nodes should be created by the hypervisor in the device tree for dom0
>       >       > and named as pcpu@0..pcpu@n?
>       >
>       >       If we do that, wouldn't we require changes to the core OMAP drivers or
>       >       cpu initialization code in Linux (to parse "pcpu" instead of "cpu"
>       >       nodes)?  I don't expect they would be easy to upstream or maintain going
>       >       forward.
>       >
>       >       I am trying to think of an alternative, such as passing the real cpu
>       >       nodes to dom0 but then adding status = "disabled", but I am not sure
>       >       whether Linux checks the status for cpu nodes. In addition this scheme
>       >       wouldn't support the case where dom0 has more vcpus than pcpus on the
>       >       system. Granted it is not very common and might even be detrimental for
>       >       performances, but we should be able to support it.
>       >
>       >
>       > In case where dom0 has more vcpus than pcpus on the
>       > system, the dom0 kernel is the most bug-prone place for pcpu cpufreq governor. So I still believe that
>       separate driver
>       > domain with direct 1:1 vcpu:pcpu mapping is the best place for cpufreq governor. But it also reasonable to run
>       cpufreq
>       > governor as userspace daemon in dom0.
>       >
>       > Also what do you think about PM QoS support? On bare metal cpufreq is tightly integrated with PM QoS and
>       intensively
>       > cooperate in frequency scaling.
> 
> Device PM needs to be done in Dom0.
> CPU an Platform level PM architecturally belongs to Xen, but I do
> understand that to do that in Xen we would need to add lots of code to
> the hypervisor. There is no silver bullet here.
> 
> A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> are you going to use for that? Linux? Wouldn't Linux be too big for a
> cpufreq driver domain, especially in embedded deployments? I think it
> would need at least 32MB to run.
> 
> 
> > With best regards,
> >  
> >       Ian, what do you think about this?
> >
> >
> >
> >       > Oleksandr Dmytryshyn | Product Engineering and Development
> >       > GlobalLogic
> >       > M +38.067.382.2525
> >       > www.globallogic.com
> >       >
> >       > http://www.globallogic.com/email_disclaimer.txt
> >       >
> >       >
> >       > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> >       > <andrii.tseglytskyi@globallogic.com> wrote:
> >       > >
> >       > > Hi Stefano,
> >       > >
> >       > > Thank you for explanation.
> >       > > I think this requires more and deeper investigation, but for sure dom0
> >       > > must be able to do this.
> >       > > Let us investigate this.
> >       > >
> >       > > Thank you,
> >       > >
> >       > > Regards,
> >       > > Andrii
> >       > >
> >       > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> >       > > <stefano.stabellini@eu.citrix.com> wrote:
> >       > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> >       > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> >       > > >> <stefano.stabellini@eu.citrix.com> wrote:
> >       > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> >       > > >> >> Hi,
> >       > > >> >>
> >       > > >> >> Stefano, Ian,
> >       > > >> >>
> >       > > >> >> Could you please clarify the following point:
> >       > > >> >>
> >       > > >> >> I agree that decision about frequency change should be taken by Xen
> >       > > >> >> hypervisor. But what about hardware frequency changing?
> >       > > >> >> In general when frequency changed to bigger value (for example from 1
> >       > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like the following:
> >       > > >> >>
> >       > > >> >> 1) cpufreq governor decides that frequency should be changed. This
> >       > > >> >> decision is taken after analysing of CPU performance data taking in
> >       > > >> >> account governor policy.
> >       > > >> >> 2) cpufreq governor asks cpufreq driver about new frequency.
> >       > > >> >> 3) cpufreq driver compares current and target frequencies and asks
> >       > > >> >> cpufreq regulator about voltage change.
> >       > > >> >> 4) cpufreq regulator send i2c command to standalone microchip, which
> >       > > >> >> is responsible for voltage changing.
> >       > > >> >> 5) cpufreq driver asks clock framework about new frequency for CPU clock
> >       > > >> >> 6) clock framework performs frequency sanity checks, taking in account
> >       > > >> >> clock parents and clock divider settings, and call platform specific
> >       > > >> >> "set_frequency" callback.
> >       > > >> >> 7) platform specific callback performs proper HW registers
> >       > > >> >> configuration for newly selected frequency
> >       > > >> >>
> >       > > >> >> Also there are some special cases - for example for OMAP5+ when
> >       > > >> >> frequency is changed to 1.5 GHz+, two additional HW IPs should be
> >       > > >> >> triggered (ABB and DCC, if someone is familiar with OMAP5+ )
> >       > > >> >>
> >       > > >> >> So, for generic ARM kernel we have 3 entities to change frequency:
> >       > > >> >>
> >       > > >> >> - cpufreq governor
> >       > > >> >> - cpufreq driver
> >       > > >> >> - cpufreq regulator
> >       > > >> >>
> >       > > >> >> + 2 additional IP for OMAP5+
> >       > > >> >> - ABB
> >       > > >> >> - DCC
> >       > > >> >>
> >       > > >> >> Taking in account all above, it looks like it would be better to
> >       > > >> >> implement only Xen cpufreq governor. Xen will take a decision about
> >       > > >> >> new frequency, and kernel dom0 will perform other steps. Dom0 contains
> >       > > >> >> all generic and platform specific frameworks, needed for frequency
> >       > > >> >> changing.
> >       > > >> >>
> >       > > >> >> What do you think ?
> >       > > >> >
> >       > > >> > Keep in mind that the architecture must be able to handle the case where
> >       > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system with multiple
> >       > > >> > physical cpus.
> >       > > >> > Could dom0 change the frequency of a physical core or a physical cpu is
> >       > > >> > not even running on? If that is not a problem, because cpus and
> >       > > >> > frequency changing are decoupled enough in Linux to allow it, then I am
> >       > > >> > OK with it. But I suspect they are not.
> >       > > >> >
> >       > > >>
> >       > > >> Not sure that I got your point correctly - dom0 will change frequency
> >       > > >> on physical CPU.
> >       > > >> And in case of OMAP - this changing affects on both ARM physical cpus
> >       > > >> - changing is coupled.
> >       > > >> In case of other ARM platforms - changing may be not coupled (I've
> >       > > >> heard that Snapdragon can change cpu freqs independently on each
> >       > > >> physical cpu)
> >       > > >
> >       > > > Let me explain with a concrete example.
> >       > > >
> >       > > > Let's suppose that the platform has 2 physical cpus, each cpu has 4
> >       > > > cores.  Let's also supposed that dom0 has only 2 vcpus, currently
> >       > > > running on core0 and core1 of cpu0.
> >       > > >
> >       > > > In this case would dom0 be able to change the frequency of core3 of
> >       > > > cpu1, given that is not even running on it?
> >       > > > If it can be done without any hacks, then we can go ahead with this
> >       > > > approach.
> >       > >
> >       > >
> >       > >
> >       > > --
> >       > >
> >       > > Andrii Tseglytskyi | Embedded Dev
> >       > > GlobalLogic
> >       > > www.globallogic.com
> >       >
> >
> >       _______________________________________________
> >       Xen-devel mailing list
> >       Xen-devel@lists.xen.org
> >       http://lists.xen.org/xen-devel
> >
> >
> >
> >
> > --
> > Vitaly Chernooky | Senior Developer - Product Engineering and Development
> > GlobalLogic
> > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
> 
> 
> 
> 
> --
> Vitaly Chernooky | Senior Developer - Product Engineering and Development
> GlobalLogic
> P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> www.globallogic.com
> 
> http://www.globallogic.com/email_disclaimer.txt
> 
> 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Konrad Rzeszutek Wilk]

On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> > Hi,
> > 
> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > >
> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > > > > > I am trying to think of an alternative, such as passing the real cpu
> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > > > > > whether Linux checks the status for cpu nodes.
> > > > >
> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > > > > similar patch to Xen to ignore them. I think it basically means "present
> > > > > but not running, you should start them!".
> > > > >
> > > > > >  In addition this scheme
> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > > > > > system. Granted it is not very common and might even be detrimental for
> > > > > > performances, but we should be able to support it.
> > > > >
> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > > > > unreasonable to say that if you use this sort of configuration you may
> > > > > not get cpufreq support.
> > > > >
> > > > > > Ian, what do you think about this?
> > > > >
> > > > > All the options suck in one way or another AFAICT. I think we are going
> > > > > to be looking for the least bad solution not necessarily a good one.
> > > > >
> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > > > > in the hypervisor to be be able to change the voltages before/after
> > > > > changing the frequency?
> > > > >
> > > > > We can't just say "that's part of the cpufreq driver" since different
> > > > > boards using the same SoC might use different voltage regulators, over
> > > > > i2c or some other bus etc, so we end up with a matrix.
> > > > >
> > > > > It's arguable that we should be letting dom0 poke at that regulator
> > > > > functionality anyway, at least not all of it. Taking that ability away
> > > > > would necessarily imply more platform specific functionality in the
> > > > > hypervisor.
> > > >
> > > > Right.
> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> > > > unmaintainable interface and unupstreamable hacks in dom0.
> > >
> > > That's what I'm worried about to. Hence I'm wondering if we should just
> > > do this in the hypervisor.
> > >
> > > Although there are a myriad of them the parts used to do voltage control
> > > tend to be fairly simple.
> > >
> > > One concern I have is that i2c busses also tend to have other things on
> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > > to suggest here.
> > 
> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> > complicated in Linux kernel. It consists of several parts - common
> > core + platform specific. I'm pretty sure Xen should not handle this.
> > I think that establishing of event channel for frequency changing is a
> > good idea. It would be good to try to implement this. In process of
> > implementation we will see what is need to be resolved.
> 
> OK, that's reasonable.
> 
> 
> > The only question here is how to pass physical cpu to dom0.
> 
> We can use a device tree based interface to pass the information to
> dom0, but requiring a number of dom0 vcpus equal to the number of
> physical cpus and in addition to that having to pin the vcpus each to a
> different pcpu is quite a stringent limitation. However I don't know the
> frequency changing interfaces in Linux well enough to know how hard
> would be to lift it.
> 
> 
> > Regarding x86.
> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> 
> I think so (but I am not an expert on that).

The usual states are P and C states. The P states is the closes to what you
are looking at:

struct acpi_processor_px {                                                      
        u64 core_frequency;     /* megahertz */                                 
        u64 power;      /* milliWatts */                                        
        u64 transition_latency; /* microseconds */                              
        u64 bus_master_latency; /* microseconds */                              
        u64 control;    /* control value */                                     
        u64 status;     /* success indicator */                                 
};                               

> 
> 
> 
> > Regards,
> > Andrii
> > 
> > 
> > -- 
> > 
> > Andrii Tseglytskyi | Embedded Dev
> > GlobalLogic
> > www.globallogic.com
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly Chernooky]

[-- Attachment #1.1: Type: text/plain, Size: 11903 bytes --]

Stefano,

I don't know how to control voltages and frequency on x86 HW. May be there
is a *reasonable *way to implement Cpufreq with different numbers of pcpu
and vcpu. But there is definitely no *reasonable* ways to do if on TI and
Marvell ARM SoCs without 1:1 mapping. We need really huge amount of work on
ARM SoCs in steps to break 1:1 mapping dependency. I will be
really surprised if there is an ARM SOC not affected be 1:1 dependency.

With best regards,

On Wed, Sep 10, 2014 at 9:41 PM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:

> The issue with that limitation is that it doesn't scale well on large
> systems. You really wouldn't want dom0 to have 18 vcpus on a Xeon E5
> because it would badly affect performances. Even on a 8 cores SoC it
> would be best to assign less than 8 vcpus to dom0.
>




> On Wed, 10 Sep 2014, Vitaly Chernooky wrote:
> > I've intensively discussed my suggestions here and now it is transparent
> to me that we should not try to use Cpufreq on ARM
> > SoCs without direct 1:1 pcpu:vcpu mapping in dom0. So if someone want to
> break 1:1 mapping he should forget Cpufreq.
> > With best regards,
> >
> >
> > On Wed, Sep 10, 2014 at 12:58 AM, Stefano Stabellini <
> stefano.stabellini@eu.citrix.com> wrote:
> >       On Tue, 9 Sep 2014, Vitaly Chernooky wrote:
> >       > Hi All!
> >       >
> >       > On Fri, Sep 5, 2014 at 12:56 AM, Stefano Stabellini <
> stefano.stabellini@eu.citrix.com> wrote:
> >       >       On Thu, 4 Sep 2014, Oleksandr Dmytryshyn wrote:
> >       >       > Hi to all.
> >       >       >
> >       >       > I want to implement cpufreq driver in the next way:
> >       >       > 1. Cpufreq governor will be implemented in the Xen
> >       >       > 2. dom0 will only change cpu frequency and voltage of
> the physical cpus
> >       >       >
> >       >       > But there are some nuances:
> >       >       > 1. dom0 driver should read an information about
> operation points
> >       >       > (frequencies and voltages) and cpu supply source from
> the device tree for each
> >       >       > physical cpu. In the omap processor case this driver
> suspects that
> >       >       > those settings
> >       >       > located in the /cpus/cpu@0/ node. But hypervisor
> creates an cpu node
> >       >       > for each vcpu
> >       >       > for kernel dom0 in the device tree and those information
> is lost in the dom0.
> >       >       > 2. What about this case if we will have some physical
> cpus with different
> >       >       > operation points (for example 2 cpus) and we give only
> one cpu for dom0?
> >       >       >
> >       >       > How should I transfer all information from the original
> cpu@0..cpu@n nodes
> >       >       > about all physical cpus to the kernel dom0 driver? Maybe
> an additional
> >       >       > nodes should be created by the hypervisor in the device
> tree for dom0
> >       >       > and named as pcpu@0..pcpu@n?
> >       >
> >       >       If we do that, wouldn't we require changes to the core
> OMAP drivers or
> >       >       cpu initialization code in Linux (to parse "pcpu" instead
> of "cpu"
> >       >       nodes)?  I don't expect they would be easy to upstream or
> maintain going
> >       >       forward.
> >       >
> >       >       I am trying to think of an alternative, such as passing
> the real cpu
> >       >       nodes to dom0 but then adding status = "disabled", but I
> am not sure
> >       >       whether Linux checks the status for cpu nodes. In addition
> this scheme
> >       >       wouldn't support the case where dom0 has more vcpus than
> pcpus on the
> >       >       system. Granted it is not very common and might even be
> detrimental for
> >       >       performances, but we should be able to support it.
> >       >
> >       >
> >       > In case where dom0 has more vcpus than pcpus on the
> >       > system, the dom0 kernel is the most bug-prone place for pcpu
> cpufreq governor. So I still believe that
> >       separate driver
> >       > domain with direct 1:1 vcpu:pcpu mapping is the best place for
> cpufreq governor. But it also reasonable to run
> >       cpufreq
> >       > governor as userspace daemon in dom0.
> >       >
> >       > Also what do you think about PM QoS support? On bare metal
> cpufreq is tightly integrated with PM QoS and
> >       intensively
> >       > cooperate in frequency scaling.
> >
> > Device PM needs to be done in Dom0.
> > CPU an Platform level PM architecturally belongs to Xen, but I do
> > understand that to do that in Xen we would need to add lots of code to
> > the hypervisor. There is no silver bullet here.
> >
> > A driver domain with 1:1 vcpu:pcpu mapping could work, but what kernel
> > are you going to use for that? Linux? Wouldn't Linux be too big for a
> > cpufreq driver domain, especially in embedded deployments? I think it
> > would need at least 32MB to run.
> >
> >
> > > With best regards,
> > >
> > >       Ian, what do you think about this?
> > >
> > >
> > >
> > >       > Oleksandr Dmytryshyn | Product Engineering and Development
> > >       > GlobalLogic
> > >       > M +38.067.382.2525
> > >       > www.globallogic.com
> > >       >
> > >       > http://www.globallogic.com/email_disclaimer.txt
> > >       >
> > >       >
> > >       > On Tue, Sep 2, 2014 at 9:46 PM, Andrii Tseglytskyi
> > >       > <andrii.tseglytskyi@globallogic.com> wrote:
> > >       > >
> > >       > > Hi Stefano,
> > >       > >
> > >       > > Thank you for explanation.
> > >       > > I think this requires more and deeper investigation, but for
> sure dom0
> > >       > > must be able to do this.
> > >       > > Let us investigate this.
> > >       > >
> > >       > > Thank you,
> > >       > >
> > >       > > Regards,
> > >       > > Andrii
> > >       > >
> > >       > > On Tue, Sep 2, 2014 at 9:39 PM, Stefano Stabellini
> > >       > > <stefano.stabellini@eu.citrix.com> wrote:
> > >       > > > On Tue, 2 Sep 2014, Andrii Tseglytskyi wrote:
> > >       > > >> On Tue, Sep 2, 2014 at 4:00 AM, Stefano Stabellini
> > >       > > >> <stefano.stabellini@eu.citrix.com> wrote:
> > >       > > >> > On Fri, 29 Aug 2014, Andrii Tseglytskyi wrote:
> > >       > > >> >> Hi,
> > >       > > >> >>
> > >       > > >> >> Stefano, Ian,
> > >       > > >> >>
> > >       > > >> >> Could you please clarify the following point:
> > >       > > >> >>
> > >       > > >> >> I agree that decision about frequency change should be
> taken by Xen
> > >       > > >> >> hypervisor. But what about hardware frequency changing?
> > >       > > >> >> In general when frequency changed to bigger value (for
> example from 1
> > >       > > >> >> GHz to 1.5 GHz) for ARM kernels sequence looks like
> the following:
> > >       > > >> >>
> > >       > > >> >> 1) cpufreq governor decides that frequency should be
> changed. This
> > >       > > >> >> decision is taken after analysing of CPU performance
> data taking in
> > >       > > >> >> account governor policy.
> > >       > > >> >> 2) cpufreq governor asks cpufreq driver about new
> frequency.
> > >       > > >> >> 3) cpufreq driver compares current and target
> frequencies and asks
> > >       > > >> >> cpufreq regulator about voltage change.
> > >       > > >> >> 4) cpufreq regulator send i2c command to standalone
> microchip, which
> > >       > > >> >> is responsible for voltage changing.
> > >       > > >> >> 5) cpufreq driver asks clock framework about new
> frequency for CPU clock
> > >       > > >> >> 6) clock framework performs frequency sanity checks,
> taking in account
> > >       > > >> >> clock parents and clock divider settings, and call
> platform specific
> > >       > > >> >> "set_frequency" callback.
> > >       > > >> >> 7) platform specific callback performs proper HW
> registers
> > >       > > >> >> configuration for newly selected frequency
> > >       > > >> >>
> > >       > > >> >> Also there are some special cases - for example for
> OMAP5+ when
> > >       > > >> >> frequency is changed to 1.5 GHz+, two additional HW
> IPs should be
> > >       > > >> >> triggered (ABB and DCC, if someone is familiar with
> OMAP5+ )
> > >       > > >> >>
> > >       > > >> >> So, for generic ARM kernel we have 3 entities to
> change frequency:
> > >       > > >> >>
> > >       > > >> >> - cpufreq governor
> > >       > > >> >> - cpufreq driver
> > >       > > >> >> - cpufreq regulator
> > >       > > >> >>
> > >       > > >> >> + 2 additional IP for OMAP5+
> > >       > > >> >> - ABB
> > >       > > >> >> - DCC
> > >       > > >> >>
> > >       > > >> >> Taking in account all above, it looks like it would be
> better to
> > >       > > >> >> implement only Xen cpufreq governor. Xen will take a
> decision about
> > >       > > >> >> new frequency, and kernel dom0 will perform other
> steps. Dom0 contains
> > >       > > >> >> all generic and platform specific frameworks, needed
> for frequency
> > >       > > >> >> changing.
> > >       > > >> >>
> > >       > > >> >> What do you think ?
> > >       > > >> >
> > >       > > >> > Keep in mind that the architecture must be able to
> handle the case where
> > >       > > >> > dom0 has only 1 or 2 vcpus on a 4 or 8 cores system
> with multiple
> > >       > > >> > physical cpus.
> > >       > > >> > Could dom0 change the frequency of a physical core or a
> physical cpu is
> > >       > > >> > not even running on? If that is not a problem, because
> cpus and
> > >       > > >> > frequency changing are decoupled enough in Linux to
> allow it, then I am
> > >       > > >> > OK with it. But I suspect they are not.
> > >       > > >> >
> > >       > > >>
> > >       > > >> Not sure that I got your point correctly - dom0 will
> change frequency
> > >       > > >> on physical CPU.
> > >       > > >> And in case of OMAP - this changing affects on both ARM
> physical cpus
> > >       > > >> - changing is coupled.
> > >       > > >> In case of other ARM platforms - changing may be not
> coupled (I've
> > >       > > >> heard that Snapdragon can change cpu freqs independently
> on each
> > >       > > >> physical cpu)
> > >       > > >
> > >       > > > Let me explain with a concrete example.
> > >       > > >
> > >       > > > Let's suppose that the platform has 2 physical cpus, each
> cpu has 4
> > >       > > > cores.  Let's also supposed that dom0 has only 2 vcpus,
> currently
> > >       > > > running on core0 and core1 of cpu0.
> > >       > > >
> > >       > > > In this case would dom0 be able to change the frequency of
> core3 of
> > >       > > > cpu1, given that is not even running on it?
> > >       > > > If it can be done without any hacks, then we can go ahead
> with this
> > >       > > > approach.
> > >       > >
> > >       > >
> > >       > >
> > >       > > --
> > >       > >
> > >       > > Andrii Tseglytskyi | Embedded Dev
> > >       > > GlobalLogic
> > >       > > www.globallogic.com
> > >       >
> > >
> > >       _______________________________________________
> > >       Xen-devel mailing list
> > >       Xen-devel@lists.xen.org
> > >       http://lists.xen.org/xen-devel
> > >
> > >
> > >
> > >
> > > --
> > > Vitaly Chernooky | Senior Developer - Product Engineering and
> Development
> > > GlobalLogic
> > > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > > www.globallogic.com
> > >
> > > http://www.globallogic.com/email_disclaimer.txt
> > >
> > >
> >
> >
> >
> >
> > --
> > Vitaly Chernooky | Senior Developer - Product Engineering and Development
> > GlobalLogic
> > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
>



-- 
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 18172 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Vitaly Chernooky]

[-- Attachment #1.1: Type: text/plain, Size: 2916 bytes --]

It't not a problem. TI BSPs usually contain examples. There is another
problems which prevent me developing this direction.

With best regards,


On Wed, Sep 10, 2014 at 9:37 PM, Stefano Stabellini <
stefano.stabellini@eu.citrix.com> wrote:

> On Wed, 10 Sep 2014, Vitaly Chernooky wrote:
> > Hi Andrii,
> >
> >
> > On Wed, Sep 10, 2014 at 1:15 PM, Andrii Tseglytskyi <
> andrii.tseglytskyi@globallogic.com> wrote:
> >       Hi,
> >
> >       >> In case where dom0 has more vcpus than pcpus on the
> >       >> system, the dom0 kernel is the most bug-prone place for pcpu
> cpufreq governor. So I still believe that
> >       separate driver
> >       >> domain with direct 1:1 vcpu:pcpu mapping is the best place for
> cpufreq governor. But it also reasonable to
> >       run cpufreq
> >       >> governor as userspace daemon in dom0.
> >       >>
> >       >> Also what do you think about PM QoS support? On bare metal
> cpufreq is tightly integrated with PM QoS and
> >       intensively
> >       >> cooperate in frequency scaling.
> >       >
> >       > Device PM needs to be done in Dom0.
> >       > CPU an Platform level PM architecturally belongs to Xen, but I do
> >       > understand that to do that in Xen we would need to add lots of
> code to
> >       > the hypervisor. There is no silver bullet here.
> >       >
> >       > A driver domain with 1:1 vcpu:pcpu mapping could work, but what
> kernel
> >       > are you going to use for that? Linux? Wouldn't Linux be too big
> for a
> >       > cpufreq driver domain, especially in embedded deployments? I
> think it
> >       > would need at least 32MB to run.
> >       >
> >
> >       I would suggest not to do this. Driver domain will need to share
> the
> >       same HW with dom0, which is impossible to implement. Good examples
> are
> >       I2C, which is needed for Display and for Cpufreq, and it cannot be
> >       shared between domains. (At least in current Linux kernel it won't
> be
> >       easy to implement). dom0 is the best place to do cpufreq.
> >
> >
> > What do you think about cpufreq in dom0 userspace?
>
> Would you write an i2c driver in userspace? How would you suggest to
> do the frequency and voltage changes in dom0 userspace otherwise?
>
>
> > With best regards,
> >
> >       Regards,
> >       Andrii
> >
> >
> >       --
> >
> >       Andrii Tseglytskyi | Embedded Dev
> >       GlobalLogic
> >       www.globallogic.com
> >
> >
> >
> >
> > --
> > Vitaly Chernooky | Senior Developer - Product Engineering and Development
> > GlobalLogic
> > P +380.44.4929695 ext.1136 M +380.98.7920568 S cvv_2k
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt
> >
> >
>



-- 
*Vitaly Chernooky | Senior Developer - Product Engineering and Development*
GlobalLogic
P *+380.44.4929695 ext.1136* M *+380.98.7920568* S cvv_2k
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

[-- Attachment #1.2: Type: text/html, Size: 4656 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: cpufreq implementation for OMAP under xen hypervisor.

[Ian Campbell]

On Wed, 2014-09-10 at 19:35 +0100, Stefano Stabellini wrote:
> > > One concern I have is that i2c busses also tend to have other things on
> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > > to suggest here.
> > 
> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> > complicated in Linux kernel. It consists of several parts - common
> > core + platform specific. I'm pretty sure Xen should not handle this.
> > I think that establishing of event channel for frequency changing is a
> > good idea. It would be good to try to implement this. In process of
> > implementation we will see what is need to be resolved.
> 
> OK, that's reasonable.

I don't think a single purpose in-Xen i2c would need quite so much
infrastructure as what Linux provides, so it could be a whole lot
simpler, I think, so I wouldn't necessarily rule it out without trying
it.

> > The only question here is how to pass physical cpu to dom0.
> 
> We can use a device tree based interface to pass the information to
> dom0, but requiring a number of dom0 vcpus equal to the number of
> physical cpus and in addition to that having to pin the vcpus each to a
> different pcpu is quite a stringent limitation. However I don't know the
> frequency changing interfaces in Linux well enough to know how hard
> would be to lift it.
> 
> 
> > Regarding x86.
> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> 
> I think so (but I am not an expert on that).

Likewise.

Ian.

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
>> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
>> > Hi,
>> >
>> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > >
>> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
>> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
>> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
>> > > > > > I am trying to think of an alternative, such as passing the real cpu
>> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
>> > > > > > whether Linux checks the status for cpu nodes.
>> > > > >
>> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
>> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
>> > > > > similar patch to Xen to ignore them. I think it basically means "present
>> > > > > but not running, you should start them!".
>> > > > >
>> > > > > >  In addition this scheme
>> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
>> > > > > > system. Granted it is not very common and might even be detrimental for
>> > > > > > performances, but we should be able to support it.
>> > > > >
>> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
>> > > > > unreasonable to say that if you use this sort of configuration you may
>> > > > > not get cpufreq support.
>> > > > >
>> > > > > > Ian, what do you think about this?
>> > > > >
>> > > > > All the options suck in one way or another AFAICT. I think we are going
>> > > > > to be looking for the least bad solution not necessarily a good one.
>> > > > >
>> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
>> > > > > in the hypervisor to be be able to change the voltages before/after
>> > > > > changing the frequency?
>> > > > >
>> > > > > We can't just say "that's part of the cpufreq driver" since different
>> > > > > boards using the same SoC might use different voltage regulators, over
>> > > > > i2c or some other bus etc, so we end up with a matrix.
>> > > > >
>> > > > > It's arguable that we should be letting dom0 poke at that regulator
>> > > > > functionality anyway, at least not all of it. Taking that ability away
>> > > > > would necessarily imply more platform specific functionality in the
>> > > > > hypervisor.
>> > > >
>> > > > Right.
>> > > > I am afraid that in order to avoid more code in Xen, we end up with an
>> > > > unmaintainable interface and unupstreamable hacks in dom0.
>> > >
>> > > That's what I'm worried about to. Hence I'm wondering if we should just
>> > > do this in the hypervisor.
>> > >
>> > > Although there are a myriad of them the parts used to do voltage control
>> > > tend to be fairly simple.
>> > >
>> > > One concern I have is that i2c busses also tend to have other things on
>> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
>> > > to suggest here.
>> >
>> > I would try to avoid i2c transactions in Xen. I2C driver is quite
>> > complicated in Linux kernel. It consists of several parts - common
>> > core + platform specific. I'm pretty sure Xen should not handle this.
>> > I think that establishing of event channel for frequency changing is a
>> > good idea. It would be good to try to implement this. In process of
>> > implementation we will see what is need to be resolved.
>>
>> OK, that's reasonable.
>>
>>
>> > The only question here is how to pass physical cpu to dom0.
>>
>> We can use a device tree based interface to pass the information to
>> dom0, but requiring a number of dom0 vcpus equal to the number of
>> physical cpus and in addition to that having to pin the vcpus each to a
>> different pcpu is quite a stringent limitation. However I don't know the
>> frequency changing interfaces in Linux well enough to know how hard
>> would be to lift it.
>>
>>
>> > Regarding x86.
>> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>>
>> I think so (but I am not an expert on that).
>
> The usual states are P and C states. The P states is the closes to what you
> are looking at:
>
> struct acpi_processor_px {
>         u64 core_frequency;     /* megahertz */
>         u64 power;      /* milliWatts */
>         u64 transition_latency; /* microseconds */
>         u64 bus_master_latency; /* microseconds */
>         u64 control;    /* control value */
>         u64 status;     /* success indicator */
> };
>
>>
>>
>>
>> > Regards,
>> > Andrii
>> >
>> >
>> > --
>> >
>> > Andrii Tseglytskyi | Embedded Dev
>> > GlobalLogic
>> > www.globallogic.com
>> >
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel


                    Cpufreq driver implementation.
                                 ____________
                                /            \
                                | xenpm tool |
                                \____________/
 Dom0 kernel user-space
---------------------------------------------------------------------------

                          ________________               _____
                         /                \             /     \  CPU
                         | DevTree Parser |          /->| ARM | driver
                         \________________/          |  \_____/
 Dom0 kernel                                         |     |
-----------------------------------------------------|-----|---------------
                                                     |     |
              _____________________________________  |     |
             |     __________        ___________   | |     |
             |    /          \      /           \  | |     |
             |    | ondemand |      | userspace |  | |     |
 Registered  |    \__________/      \___________/  | |     |
  cpufreq    |   _____________       ___________   | |     |
 governor    |  /             \     /           \  | |     |
             |  | performance |     | powersave |  | |     |
             |  \_____________/     \___________/  | |     |
             |_____________________________________| |     |
                               ^                     |     |
                               |                     |     |
                         ______|_______              |     |
                        /              \             |     |  Change
                        | cpufreq core |-------------/     | frequency
                        \______________/ set/get freq      |
                                         commands          |
 Xen                                                       |
-----------------------------------------------------------|--------------
 Hardware                                                __V__
                                                        |     |
                                                        | CPU |
                                                        |_____|


Description of the implementation:
Cpufreq core and registered cpufreq governors are located in xen. Dom0
has CPU driver
which can only change frequency of the physical CPUs. In addition this driver
can change CPUs regulator voltage. I'll reuse some ACPI-specific
variables for ARM.
Thus I can make minimum modification in the xen cpufreq driver and all utilities
(as xenpm) will be working without modification if the xen code. In first
implementation xenpm tool won't show information about C-states, but it can show
information about P-states and can change cpufreq parameters and
change governor.
DevTree parser is a part of the CPU driver in Dom0 and it will read information
from /cpus/cpu@0/private_data path instead of the original /cpus path.

Steps of the initialization:
1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
/cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
any number
of VCPUs in Dom0 and we give all information about all physical CPUs in
the private_data node.

2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
path and give the information about CPUs parameters to the hypervisor via
XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
Dom0 driver and can not be calculated  in the hypervisor).

3. Cpufreq core driver in the hypervisor will communicate via some interface
with Dom0 (event channel can be used to notify Dom0) and give some commands
to the CPU driver in Dom0. Those command are set/get frequency, etc.

Can I implement cpufreq driver in this way?

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Konrad Rzeszutek Wilk]

On Tue, Sep 16, 2014 at 04:49:46PM +0300, Oleksandr Dmytryshyn wrote:
> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> >> > Hi,
> >> >
> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > >
> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> >> > > > > > whether Linux checks the status for cpu nodes.
> >> > > > >
> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> >> > > > > but not running, you should start them!".
> >> > > > >
> >> > > > > >  In addition this scheme
> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> >> > > > > > system. Granted it is not very common and might even be detrimental for
> >> > > > > > performances, but we should be able to support it.
> >> > > > >
> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> >> > > > > unreasonable to say that if you use this sort of configuration you may
> >> > > > > not get cpufreq support.
> >> > > > >
> >> > > > > > Ian, what do you think about this?
> >> > > > >
> >> > > > > All the options suck in one way or another AFAICT. I think we are going
> >> > > > > to be looking for the least bad solution not necessarily a good one.
> >> > > > >
> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> >> > > > > in the hypervisor to be be able to change the voltages before/after
> >> > > > > changing the frequency?
> >> > > > >
> >> > > > > We can't just say "that's part of the cpufreq driver" since different
> >> > > > > boards using the same SoC might use different voltage regulators, over
> >> > > > > i2c or some other bus etc, so we end up with a matrix.
> >> > > > >
> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> >> > > > > functionality anyway, at least not all of it. Taking that ability away
> >> > > > > would necessarily imply more platform specific functionality in the
> >> > > > > hypervisor.
> >> > > >
> >> > > > Right.
> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> >> > >
> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> >> > > do this in the hypervisor.
> >> > >
> >> > > Although there are a myriad of them the parts used to do voltage control
> >> > > tend to be fairly simple.
> >> > >
> >> > > One concern I have is that i2c busses also tend to have other things on
> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> >> > > to suggest here.
> >> >
> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> >> > complicated in Linux kernel. It consists of several parts - common
> >> > core + platform specific. I'm pretty sure Xen should not handle this.
> >> > I think that establishing of event channel for frequency changing is a
> >> > good idea. It would be good to try to implement this. In process of
> >> > implementation we will see what is need to be resolved.
> >>
> >> OK, that's reasonable.
> >>
> >>
> >> > The only question here is how to pass physical cpu to dom0.
> >>
> >> We can use a device tree based interface to pass the information to
> >> dom0, but requiring a number of dom0 vcpus equal to the number of
> >> physical cpus and in addition to that having to pin the vcpus each to a
> >> different pcpu is quite a stringent limitation. However I don't know the
> >> frequency changing interfaces in Linux well enough to know how hard
> >> would be to lift it.
> >>
> >>
> >> > Regarding x86.
> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> >>
> >> I think so (but I am not an expert on that).
> >
> > The usual states are P and C states. The P states is the closes to what you
> > are looking at:
> >
> > struct acpi_processor_px {
> >         u64 core_frequency;     /* megahertz */
> >         u64 power;      /* milliWatts */
> >         u64 transition_latency; /* microseconds */
> >         u64 bus_master_latency; /* microseconds */
> >         u64 control;    /* control value */
> >         u64 status;     /* success indicator */
> > };
> >
> >>
> >>
> >>
> >> > Regards,
> >> > Andrii
> >> >
> >> >
> >> > --
> >> >
> >> > Andrii Tseglytskyi | Embedded Dev
> >> > GlobalLogic
> >> > www.globallogic.com
> >> >
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> http://lists.xen.org/xen-devel
> 
> 
>                     Cpufreq driver implementation.
>                                  ____________
>                                 /            \
>                                 | xenpm tool |
>                                 \____________/
>  Dom0 kernel user-space
> ---------------------------------------------------------------------------
> 
>                           ________________               _____
>                          /                \             /     \  CPU
>                          | DevTree Parser |          /->| ARM | driver
>                          \________________/          |  \_____/
>  Dom0 kernel                                         |     |
> -----------------------------------------------------|-----|---------------
>                                                      |     |
>               _____________________________________  |     |
>              |     __________        ___________   | |     |
>              |    /          \      /           \  | |     |
>              |    | ondemand |      | userspace |  | |     |
>  Registered  |    \__________/      \___________/  | |     |
>   cpufreq    |   _____________       ___________   | |     |
>  governor    |  /             \     /           \  | |     |
>              |  | performance |     | powersave |  | |     |
>              |  \_____________/     \___________/  | |     |
>              |_____________________________________| |     |
>                                ^                     |     |
>                                |                     |     |
>                          ______|_______              |     |
>                         /              \             |     |  Change
>                         | cpufreq core |-------------/     | frequency
>                         \______________/ set/get freq      |
>                                          commands          |
>  Xen                                                       |
> -----------------------------------------------------------|--------------
>  Hardware                                                __V__
>                                                         |     |
>                                                         | CPU |
>                                                         |_____|
> 
> 
> Description of the implementation:
> Cpufreq core and registered cpufreq governors are located in xen. Dom0
> has CPU driver
> which can only change frequency of the physical CPUs. In addition this driver
> can change CPUs regulator voltage. I'll reuse some ACPI-specific
> variables for ARM.
> Thus I can make minimum modification in the xen cpufreq driver and all utilities
> (as xenpm) will be working without modification if the xen code. In first
> implementation xenpm tool won't show information about C-states, but it can show
> information about P-states and can change cpufreq parameters and
> change governor.
> DevTree parser is a part of the CPU driver in Dom0 and it will read information
> from /cpus/cpu@0/private_data path instead of the original /cpus path.
> 
> Steps of the initialization:
> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> any number
> of VCPUs in Dom0 and we give all information about all physical CPUs in
> the private_data node.
> 
> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> path and give the information about CPUs parameters to the hypervisor via
> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> Dom0 driver and can not be calculated  in the hypervisor).

Which driver? I presume it would be similar to the xen-acpi-processor.c driver
in drivers/xen?

> 
> 3. Cpufreq core driver in the hypervisor will communicate via some interface
> with Dom0 (event channel can be used to notify Dom0) and give some commands
> to the CPU driver in Dom0. Those command are set/get frequency, etc.

Like the 'xenpm' which does that?
> 
> Can I implement cpufreq driver in this way?

I don't see why not. Thought I am curious to what is the 'driver' you
are referring too. I presume it is the one that reads the voltage values
from something (what is that "Something" ?)?

> 
> Oleksandr Dmytryshyn | Product Engineering and Development
> GlobalLogic
> M +38.067.382.2525
> www.globallogic.com
> 
> http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Wed, Sep 17, 2014 at 8:06 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>
> On Tue, Sep 16, 2014 at 04:49:46PM +0300, Oleksandr Dmytryshyn wrote:
> > On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> > > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> > >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> > >> > Hi,
> > >> >
> > >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > >> > >
> > >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> > >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > >> > > > > > whether Linux checks the status for cpu nodes.
> > >> > > > >
> > >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> > >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> > >> > > > > but not running, you should start them!".
> > >> > > > >
> > >> > > > > >  In addition this scheme
> > >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > >> > > > > > system. Granted it is not very common and might even be detrimental for
> > >> > > > > > performances, but we should be able to support it.
> > >> > > > >
> > >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > >> > > > > unreasonable to say that if you use this sort of configuration you may
> > >> > > > > not get cpufreq support.
> > >> > > > >
> > >> > > > > > Ian, what do you think about this?
> > >> > > > >
> > >> > > > > All the options suck in one way or another AFAICT. I think we are going
> > >> > > > > to be looking for the least bad solution not necessarily a good one.
> > >> > > > >
> > >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > >> > > > > in the hypervisor to be be able to change the voltages before/after
> > >> > > > > changing the frequency?
> > >> > > > >
> > >> > > > > We can't just say "that's part of the cpufreq driver" since different
> > >> > > > > boards using the same SoC might use different voltage regulators, over
> > >> > > > > i2c or some other bus etc, so we end up with a matrix.
> > >> > > > >
> > >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> > >> > > > > functionality anyway, at least not all of it. Taking that ability away
> > >> > > > > would necessarily imply more platform specific functionality in the
> > >> > > > > hypervisor.
> > >> > > >
> > >> > > > Right.
> > >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> > >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> > >> > >
> > >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> > >> > > do this in the hypervisor.
> > >> > >
> > >> > > Although there are a myriad of them the parts used to do voltage control
> > >> > > tend to be fairly simple.
> > >> > >
> > >> > > One concern I have is that i2c busses also tend to have other things on
> > >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > >> > > to suggest here.
> > >> >
> > >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> > >> > complicated in Linux kernel. It consists of several parts - common
> > >> > core + platform specific. I'm pretty sure Xen should not handle this.
> > >> > I think that establishing of event channel for frequency changing is a
> > >> > good idea. It would be good to try to implement this. In process of
> > >> > implementation we will see what is need to be resolved.
> > >>
> > >> OK, that's reasonable.
> > >>
> > >>
> > >> > The only question here is how to pass physical cpu to dom0.
> > >>
> > >> We can use a device tree based interface to pass the information to
> > >> dom0, but requiring a number of dom0 vcpus equal to the number of
> > >> physical cpus and in addition to that having to pin the vcpus each to a
> > >> different pcpu is quite a stringent limitation. However I don't know the
> > >> frequency changing interfaces in Linux well enough to know how hard
> > >> would be to lift it.
> > >>
> > >>
> > >> > Regarding x86.
> > >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> > >>
> > >> I think so (but I am not an expert on that).
> > >
> > > The usual states are P and C states. The P states is the closes to what you
> > > are looking at:
> > >
> > > struct acpi_processor_px {
> > >         u64 core_frequency;     /* megahertz */
> > >         u64 power;      /* milliWatts */
> > >         u64 transition_latency; /* microseconds */
> > >         u64 bus_master_latency; /* microseconds */
> > >         u64 control;    /* control value */
> > >         u64 status;     /* success indicator */
> > > };
> > >
> > >>
> > >>
> > >>
> > >> > Regards,
> > >> > Andrii
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Andrii Tseglytskyi | Embedded Dev
> > >> > GlobalLogic
> > >> > www.globallogic.com
> > >> >
> > >>
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@lists.xen.org
> > >> http://lists.xen.org/xen-devel
> >
> >
> >                     Cpufreq driver implementation.
> >                                  ____________
> >                                 /            \
> >                                 | xenpm tool |
> >                                 \____________/
> >  Dom0 kernel user-space
> > ---------------------------------------------------------------------------
> >
> >                           ________________               _____
> >                          /                \             /     \  CPU
> >                          | DevTree Parser |          /->| ARM | driver
> >                          \________________/          |  \_____/
> >  Dom0 kernel                                         |     |
> > -----------------------------------------------------|-----|---------------
> >                                                      |     |
> >               _____________________________________  |     |
> >              |     __________        ___________   | |     |
> >              |    /          \      /           \  | |     |
> >              |    | ondemand |      | userspace |  | |     |
> >  Registered  |    \__________/      \___________/  | |     |
> >   cpufreq    |   _____________       ___________   | |     |
> >  governor    |  /             \     /           \  | |     |
> >              |  | performance |     | powersave |  | |     |
> >              |  \_____________/     \___________/  | |     |
> >              |_____________________________________| |     |
> >                                ^                     |     |
> >                                |                     |     |
> >                          ______|_______              |     |
> >                         /              \             |     |  Change
> >                         | cpufreq core |-------------/     | frequency
> >                         \______________/ set/get freq      |
> >                                          commands          |
> >  Xen                                                       |
> > -----------------------------------------------------------|--------------
> >  Hardware                                                __V__
> >                                                         |     |
> >                                                         | CPU |
> >                                                         |_____|
> >
> >
> > Description of the implementation:
> > Cpufreq core and registered cpufreq governors are located in xen. Dom0
> > has CPU driver
> > which can only change frequency of the physical CPUs. In addition this driver
> > can change CPUs regulator voltage. I'll reuse some ACPI-specific
> > variables for ARM.
> > Thus I can make minimum modification in the xen cpufreq driver and all utilities
> > (as xenpm) will be working without modification if the xen code. In first
> > implementation xenpm tool won't show information about C-states, but it can show
> > information about P-states and can change cpufreq parameters and
> > change governor.
> > DevTree parser is a part of the CPU driver in Dom0 and it will read information
> > from /cpus/cpu@0/private_data path instead of the original /cpus path.
> >
> > Steps of the initialization:
> > 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> > /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> > any number
> > of VCPUs in Dom0 and we give all information about all physical CPUs in
> > the private_data node.
> >
> > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> > path and give the information about CPUs parameters to the hypervisor via
> > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> > Dom0 driver and can not be calculated  in the hypervisor).
>
> Which driver? I presume it would be similar to the xen-acpi-processor.c driver
> in drivers/xen?
I meant slightly modified cpufreq-cpu0 driver in Linux kernel - driver
which changes
frequency and voltage on the OMAP CPU. And this driver will use part of the
xen-acpi-processor.c driver to upload P-states for xen.
xen-acpi-processor.c driver only uploads the information about
P-states and C-states
to the xen and xen can change CPUs frequency and voltage via ACPI interface.
In this case xen works with ACPI interface directly (without Dom0 kernel).
In case the ARM architecture every processor has own non-standardized interface
to change frequency. And in addition we should change voltage on CPU when we
change frequency. Every ARM CPU has own external Power Management IC
which can change voltage. But Power Management IC has own non-standardized
interface and command sequence to change voltage. In my case voltage can
be changed via I2C commands. We can not use I2C in the same time in xen
and in Dom0 (in this case I2C should be virtualized). Thus if we want
that cpufreq
driver work in the same way for ARM as for x86 we should port to xen a lot of
the frameworks from Linux kernel (I2C, clock and regulator framework) and
we should virtualize each framework. But we can not do that.

> >
> > 3. Cpufreq core driver in the hypervisor will communicate via some interface
> > with Dom0 (event channel can be used to notify Dom0) and give some commands
> > to the CPU driver in Dom0. Those command are set/get frequency, etc.
>
> Like the 'xenpm' which does that?
Not exactly. 'xenpm' communicates via hypercalls with Xen. In this case
xen acts as slave. And all cpufreq drivers are inside xen. In my case
part of the
cpufreq driver will be implemented in the Linux kernel (Dom0) and xen should
act as master and give commands to the cpufreq CPU driver in Dom0
(this driver will only change frequency and voltage on physical CPUs).

> >
> > Can I implement cpufreq driver in this way?
>
> I don't see why not. Thought I am curious to what is the 'driver' you
> are referring too. I presume it is the one that reads the voltage values
> from something (what is that "Something" ?)?
I've written about this driver above. This driver (in Dom0) will read
information
about voltages and frequencies from Device Tree. This Device Tree is created
by xen for Dom0.

> >
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Konrad Rzeszutek Wilk]

> > > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> > > path and give the information about CPUs parameters to the hypervisor via
> > > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> > > Dom0 driver and can not be calculated  in the hypervisor).
> >
> > Which driver? I presume it would be similar to the xen-acpi-processor.c driver
> > in drivers/xen?
> I meant slightly modified cpufreq-cpu0 driver in Linux kernel - driver
> which changes
> frequency and voltage on the OMAP CPU. And this driver will use part of the

OK, is that something you would upstream as well? Which of the ARM cpufreq
drivers is this?
> xen-acpi-processor.c driver to upload P-states for xen.
> xen-acpi-processor.c driver only uploads the information about
> P-states and C-states
> to the xen and xen can change CPUs frequency and voltage via ACPI interface.
> In this case xen works with ACPI interface directly (without Dom0 kernel).
> In case the ARM architecture every processor has own non-standardized interface
> to change frequency. And in addition we should change voltage on CPU when we
> change frequency. Every ARM CPU has own external Power Management IC
> which can change voltage. But Power Management IC has own non-standardized
> interface and command sequence to change voltage. In my case voltage can
> be changed via I2C commands. We can not use I2C in the same time in xen
> and in Dom0 (in this case I2C should be virtualized). Thus if we want
> that cpufreq
> driver work in the same way for ARM as for x86 we should port to xen a lot of
> the frameworks from Linux kernel (I2C, clock and regulator framework) and
> we should virtualize each framework. But we can not do that.
> 
> > >
> > > 3. Cpufreq core driver in the hypervisor will communicate via some interface
> > > with Dom0 (event channel can be used to notify Dom0) and give some commands
> > > to the CPU driver in Dom0. Those command are set/get frequency, etc.
> >
> > Like the 'xenpm' which does that?
> Not exactly. 'xenpm' communicates via hypercalls with Xen. In this case
> xen acts as slave. And all cpufreq drivers are inside xen. In my case
> part of the
> cpufreq driver will be implemented in the Linux kernel (Dom0) and xen should
> act as master and give commands to the cpufreq CPU driver in Dom0
> (this driver will only change frequency and voltage on physical CPUs).

Oh, I must have misunderstood your picture. In it the cpufreq and the
lot seemed to be all inside Xen.

> I
> > >
> > > Can I implement cpufreq driver in this way?
> >
> > I don't see why not. Thought I am curious to what is the 'driver' you
> > are referring too. I presume it is the one that reads the voltage values
> > from something (what is that "Something" ?)?
> I've written about this driver above. This driver (in Dom0) will read
> information
> about voltages and frequencies from Device Tree. This Device Tree is created
> by xen for Dom0.

And this driver is based on some Linux driver right ? Which one?

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Thu, Sep 18, 2014 at 5:59 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>> > > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
>> > > path and give the information about CPUs parameters to the hypervisor via
>> > > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
>> > > Dom0 driver and can not be calculated  in the hypervisor).
>> >
>> > Which driver? I presume it would be similar to the xen-acpi-processor.c driver
>> > in drivers/xen?
>> I meant slightly modified cpufreq-cpu0 driver in Linux kernel - driver
>> which changes
>> frequency and voltage on the OMAP CPU. And this driver will use part of the
>
> OK, is that something you would upstream as well? Which of the ARM cpufreq
> drivers is this?
I'll try to upstream it. But at first I should write and check that
cpufreq mechanism
is fully working.
This driver is drivers/cpufreq/cpufreq-cpu0.c

>> xen-acpi-processor.c driver to upload P-states for xen.
>> xen-acpi-processor.c driver only uploads the information about
>> P-states and C-states
>> to the xen and xen can change CPUs frequency and voltage via ACPI interface.
>> In this case xen works with ACPI interface directly (without Dom0 kernel).
>> In case the ARM architecture every processor has own non-standardized interface
>> to change frequency. And in addition we should change voltage on CPU when we
>> change frequency. Every ARM CPU has own external Power Management IC
>> which can change voltage. But Power Management IC has own non-standardized
>> interface and command sequence to change voltage. In my case voltage can
>> be changed via I2C commands. We can not use I2C in the same time in xen
>> and in Dom0 (in this case I2C should be virtualized). Thus if we want
>> that cpufreq
>> driver work in the same way for ARM as for x86 we should port to xen a lot of
>> the frameworks from Linux kernel (I2C, clock and regulator framework) and
>> we should virtualize each framework. But we can not do that.
>>
>> > >
>> > > 3. Cpufreq core driver in the hypervisor will communicate via some interface
>> > > with Dom0 (event channel can be used to notify Dom0) and give some commands
>> > > to the CPU driver in Dom0. Those command are set/get frequency, etc.
>> >
>> > Like the 'xenpm' which does that?
>> Not exactly. 'xenpm' communicates via hypercalls with Xen. In this case
>> xen acts as slave. And all cpufreq drivers are inside xen. In my case
>> part of the
>> cpufreq driver will be implemented in the Linux kernel (Dom0) and xen should
>> act as master and give commands to the cpufreq CPU driver in Dom0
>> (this driver will only change frequency and voltage on physical CPUs).
>
> Oh, I must have misunderstood your picture. In it the cpufreq and the
> lot seemed to be all inside Xen.
Yes, but there is a 'CPU driver' on the picture and this driver is
drivers/cpufreq/cpufreq-cpu0.c
in the Linux kernel.

>
>> I
>> > >
>> > > Can I implement cpufreq driver in this way?
>> >
>> > I don't see why not. Thought I am curious to what is the 'driver' you
>> > are referring too. I presume it is the one that reads the voltage values
>> > from something (what is that "Something" ?)?
>> I've written about this driver above. This driver (in Dom0) will read
>> information
>> about voltages and frequencies from Device Tree. This Device Tree is created
>> by xen for Dom0.
>
> And this driver is based on some Linux driver right ? Which one?
This driver is drivers/cpufreq/cpufreq-cpu0.c and it reads all information
from the Device Tree. Also this driver can change voltage and frequency
on physical CPU (using 'clock' and 'regulator' frameworks).

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> >> > Hi,
> >> >
> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> > >
> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> >> > > > > > whether Linux checks the status for cpu nodes.
> >> > > > >
> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> >> > > > > but not running, you should start them!".
> >> > > > >
> >> > > > > >  In addition this scheme
> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> >> > > > > > system. Granted it is not very common and might even be detrimental for
> >> > > > > > performances, but we should be able to support it.
> >> > > > >
> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> >> > > > > unreasonable to say that if you use this sort of configuration you may
> >> > > > > not get cpufreq support.
> >> > > > >
> >> > > > > > Ian, what do you think about this?
> >> > > > >
> >> > > > > All the options suck in one way or another AFAICT. I think we are going
> >> > > > > to be looking for the least bad solution not necessarily a good one.
> >> > > > >
> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> >> > > > > in the hypervisor to be be able to change the voltages before/after
> >> > > > > changing the frequency?
> >> > > > >
> >> > > > > We can't just say "that's part of the cpufreq driver" since different
> >> > > > > boards using the same SoC might use different voltage regulators, over
> >> > > > > i2c or some other bus etc, so we end up with a matrix.
> >> > > > >
> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> >> > > > > functionality anyway, at least not all of it. Taking that ability away
> >> > > > > would necessarily imply more platform specific functionality in the
> >> > > > > hypervisor.
> >> > > >
> >> > > > Right.
> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> >> > >
> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> >> > > do this in the hypervisor.
> >> > >
> >> > > Although there are a myriad of them the parts used to do voltage control
> >> > > tend to be fairly simple.
> >> > >
> >> > > One concern I have is that i2c busses also tend to have other things on
> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> >> > > to suggest here.
> >> >
> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> >> > complicated in Linux kernel. It consists of several parts - common
> >> > core + platform specific. I'm pretty sure Xen should not handle this.
> >> > I think that establishing of event channel for frequency changing is a
> >> > good idea. It would be good to try to implement this. In process of
> >> > implementation we will see what is need to be resolved.
> >>
> >> OK, that's reasonable.
> >>
> >>
> >> > The only question here is how to pass physical cpu to dom0.
> >>
> >> We can use a device tree based interface to pass the information to
> >> dom0, but requiring a number of dom0 vcpus equal to the number of
> >> physical cpus and in addition to that having to pin the vcpus each to a
> >> different pcpu is quite a stringent limitation. However I don't know the
> >> frequency changing interfaces in Linux well enough to know how hard
> >> would be to lift it.
> >>
> >>
> >> > Regarding x86.
> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> >>
> >> I think so (but I am not an expert on that).
> >
> > The usual states are P and C states. The P states is the closes to what you
> > are looking at:
> >
> > struct acpi_processor_px {
> >         u64 core_frequency;     /* megahertz */
> >         u64 power;      /* milliWatts */
> >         u64 transition_latency; /* microseconds */
> >         u64 bus_master_latency; /* microseconds */
> >         u64 control;    /* control value */
> >         u64 status;     /* success indicator */
> > };
> >
> >>
> >>
> >>
> >> > Regards,
> >> > Andrii
> >> >
> >> >
> >> > --
> >> >
> >> > Andrii Tseglytskyi | Embedded Dev
> >> > GlobalLogic
> >> > www.globallogic.com
> >> >
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> http://lists.xen.org/xen-devel
> 
> 
>                     Cpufreq driver implementation.
>                                  ____________
>                                 /            \
>                                 | xenpm tool |
>                                 \____________/
>  Dom0 kernel user-space
> ---------------------------------------------------------------------------
> 
>                           ________________               _____
>                          /                \             /     \  CPU
>                          | DevTree Parser |          /->| ARM | driver
>                          \________________/          |  \_____/
>  Dom0 kernel                                         |     |
> -----------------------------------------------------|-----|---------------
>                                                      |     |
>               _____________________________________  |     |
>              |     __________        ___________   | |     |
>              |    /          \      /           \  | |     |
>              |    | ondemand |      | userspace |  | |     |
>  Registered  |    \__________/      \___________/  | |     |
>   cpufreq    |   _____________       ___________   | |     |
>  governor    |  /             \     /           \  | |     |
>              |  | performance |     | powersave |  | |     |
>              |  \_____________/     \___________/  | |     |
>              |_____________________________________| |     |
>                                ^                     |     |
>                                |                     |     |
>                          ______|_______              |     |
>                         /              \             |     |  Change
>                         | cpufreq core |-------------/     | frequency
>                         \______________/ set/get freq      |
>                                          commands          |
>  Xen                                                       |
> -----------------------------------------------------------|--------------
>  Hardware                                                __V__
>                                                         |     |
>                                                         | CPU |
>                                                         |_____|
> 
> 
> Description of the implementation:
> Cpufreq core and registered cpufreq governors are located in xen. Dom0
> has CPU driver
> which can only change frequency of the physical CPUs. In addition this driver
> can change CPUs regulator voltage. I'll reuse some ACPI-specific
> variables for ARM.
> Thus I can make minimum modification in the xen cpufreq driver and all utilities
> (as xenpm) will be working without modification if the xen code. In first
> implementation xenpm tool won't show information about C-states, but it can show
> information about P-states and can change cpufreq parameters and
> change governor.
> DevTree parser is a part of the CPU driver in Dom0 and it will read information
> from /cpus/cpu@0/private_data path instead of the original /cpus path.
> 
> Steps of the initialization:
> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> any number
> of VCPUs in Dom0 and we give all information about all physical CPUs in
> the private_data node.
> 
> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> path and give the information about CPUs parameters to the hypervisor via
> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> Dom0 driver and can not be calculated  in the hypervisor).
> 
> 3. Cpufreq core driver in the hypervisor will communicate via some interface
> with Dom0 (event channel can be used to notify Dom0) and give some commands
> to the CPU driver in Dom0. Those command are set/get frequency, etc.
> 
> Can I implement cpufreq driver in this way?

The architecture looks sane to me. As Konrad pointed out, the difficulty
here is to be able to upstream the changes to the Linux driver in 2),
that you later in the thread identified as
drivers/cpufreq/cpufreq-cpu0.c.

If the changes are not invasive and you manage to upstream them in
Linux, I am all for this solution.

Re: cpufreq implementation for OMAP under xen hypervisor.

[Konrad Rzeszutek Wilk]

On Wed, Sep 24, 2014 at 03:36:15PM +0100, Stefano Stabellini wrote:
> On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
> > On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> > > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> > >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> > >> > Hi,
> > >> >
> > >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > >> > >
> > >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> > >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> > >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> > >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> > >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> > >> > > > > > whether Linux checks the status for cpu nodes.
> > >> > > > >
> > >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> > >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> > >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> > >> > > > > but not running, you should start them!".
> > >> > > > >
> > >> > > > > >  In addition this scheme
> > >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> > >> > > > > > system. Granted it is not very common and might even be detrimental for
> > >> > > > > > performances, but we should be able to support it.
> > >> > > > >
> > >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> > >> > > > > unreasonable to say that if you use this sort of configuration you may
> > >> > > > > not get cpufreq support.
> > >> > > > >
> > >> > > > > > Ian, what do you think about this?
> > >> > > > >
> > >> > > > > All the options suck in one way or another AFAICT. I think we are going
> > >> > > > > to be looking for the least bad solution not necessarily a good one.
> > >> > > > >
> > >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> > >> > > > > in the hypervisor to be be able to change the voltages before/after
> > >> > > > > changing the frequency?
> > >> > > > >
> > >> > > > > We can't just say "that's part of the cpufreq driver" since different
> > >> > > > > boards using the same SoC might use different voltage regulators, over
> > >> > > > > i2c or some other bus etc, so we end up with a matrix.
> > >> > > > >
> > >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> > >> > > > > functionality anyway, at least not all of it. Taking that ability away
> > >> > > > > would necessarily imply more platform specific functionality in the
> > >> > > > > hypervisor.
> > >> > > >
> > >> > > > Right.
> > >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> > >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> > >> > >
> > >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> > >> > > do this in the hypervisor.
> > >> > >
> > >> > > Although there are a myriad of them the parts used to do voltage control
> > >> > > tend to be fairly simple.
> > >> > >
> > >> > > One concern I have is that i2c busses also tend to have other things on
> > >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> > >> > > to suggest here.
> > >> >
> > >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> > >> > complicated in Linux kernel. It consists of several parts - common
> > >> > core + platform specific. I'm pretty sure Xen should not handle this.
> > >> > I think that establishing of event channel for frequency changing is a
> > >> > good idea. It would be good to try to implement this. In process of
> > >> > implementation we will see what is need to be resolved.
> > >>
> > >> OK, that's reasonable.
> > >>
> > >>
> > >> > The only question here is how to pass physical cpu to dom0.
> > >>
> > >> We can use a device tree based interface to pass the information to
> > >> dom0, but requiring a number of dom0 vcpus equal to the number of
> > >> physical cpus and in addition to that having to pin the vcpus each to a
> > >> different pcpu is quite a stringent limitation. However I don't know the
> > >> frequency changing interfaces in Linux well enough to know how hard
> > >> would be to lift it.
> > >>
> > >>
> > >> > Regarding x86.
> > >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> > >>
> > >> I think so (but I am not an expert on that).
> > >
> > > The usual states are P and C states. The P states is the closes to what you
> > > are looking at:
> > >
> > > struct acpi_processor_px {
> > >         u64 core_frequency;     /* megahertz */
> > >         u64 power;      /* milliWatts */
> > >         u64 transition_latency; /* microseconds */
> > >         u64 bus_master_latency; /* microseconds */
> > >         u64 control;    /* control value */
> > >         u64 status;     /* success indicator */
> > > };
> > >
> > >>
> > >>
> > >>
> > >> > Regards,
> > >> > Andrii
> > >> >
> > >> >
> > >> > --
> > >> >
> > >> > Andrii Tseglytskyi | Embedded Dev
> > >> > GlobalLogic
> > >> > www.globallogic.com
> > >> >
> > >>
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@lists.xen.org
> > >> http://lists.xen.org/xen-devel
> > 
> > 
> >                     Cpufreq driver implementation.
> >                                  ____________
> >                                 /            \
> >                                 | xenpm tool |
> >                                 \____________/
> >  Dom0 kernel user-space
> > ---------------------------------------------------------------------------
> > 
> >                           ________________               _____
> >                          /                \             /     \  CPU
> >                          | DevTree Parser |          /->| ARM | driver
> >                          \________________/          |  \_____/
> >  Dom0 kernel                                         |     |
> > -----------------------------------------------------|-----|---------------
> >                                                      |     |
> >               _____________________________________  |     |
> >              |     __________        ___________   | |     |
> >              |    /          \      /           \  | |     |
> >              |    | ondemand |      | userspace |  | |     |
> >  Registered  |    \__________/      \___________/  | |     |
> >   cpufreq    |   _____________       ___________   | |     |
> >  governor    |  /             \     /           \  | |     |
> >              |  | performance |     | powersave |  | |     |
> >              |  \_____________/     \___________/  | |     |
> >              |_____________________________________| |     |
> >                                ^                     |     |
> >                                |                     |     |
> >                          ______|_______              |     |
> >                         /              \             |     |  Change
> >                         | cpufreq core |-------------/     | frequency
> >                         \______________/ set/get freq      |
> >                                          commands          |
> >  Xen                                                       |
> > -----------------------------------------------------------|--------------
> >  Hardware                                                __V__
> >                                                         |     |
> >                                                         | CPU |
> >                                                         |_____|
> > 
> > 
> > Description of the implementation:
> > Cpufreq core and registered cpufreq governors are located in xen. Dom0
> > has CPU driver
> > which can only change frequency of the physical CPUs. In addition this driver
> > can change CPUs regulator voltage. I'll reuse some ACPI-specific
> > variables for ARM.
> > Thus I can make minimum modification in the xen cpufreq driver and all utilities
> > (as xenpm) will be working without modification if the xen code. In first
> > implementation xenpm tool won't show information about C-states, but it can show
> > information about P-states and can change cpufreq parameters and
> > change governor.
> > DevTree parser is a part of the CPU driver in Dom0 and it will read information
> > from /cpus/cpu@0/private_data path instead of the original /cpus path.
> > 
> > Steps of the initialization:
> > 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> > /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> > any number
> > of VCPUs in Dom0 and we give all information about all physical CPUs in
> > the private_data node.
> > 
> > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> > path and give the information about CPUs parameters to the hypervisor via
> > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> > Dom0 driver and can not be calculated  in the hypervisor).
> > 
> > 3. Cpufreq core driver in the hypervisor will communicate via some interface
> > with Dom0 (event channel can be used to notify Dom0) and give some commands
> > to the CPU driver in Dom0. Those command are set/get frequency, etc.
> > 
> > Can I implement cpufreq driver in this way?
> 
> The architecture looks sane to me. As Konrad pointed out, the difficulty
> here is to be able to upstream the changes to the Linux driver in 2),
> that you later in the thread identified as
> drivers/cpufreq/cpufreq-cpu0.c.
> 
> If the changes are not invasive and you manage to upstream them in
> Linux, I am all for this solution.

Looking at the driver, you could make some of the drivers functionality
be a library (all of those 'voltage-tolerance', 'clock-latency', regulator,
etc).

And then disable the cpufreq API from working altogether (disable_cpufreq())
and have your own driver (drivers/xen/xen-cpufreq-cpu0.c) use the libraries
and upload the data to the hypervisor.
(or 

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Wed, Sep 24, 2014 at 5:36 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
>> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com> wrote:
>> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
>> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
>> >> > Hi,
>> >> >
>> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> >> > >
>> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
>> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
>> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
>> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
>> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
>> >> > > > > > whether Linux checks the status for cpu nodes.
>> >> > > > >
>> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
>> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
>> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
>> >> > > > > but not running, you should start them!".
>> >> > > > >
>> >> > > > > >  In addition this scheme
>> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
>> >> > > > > > system. Granted it is not very common and might even be detrimental for
>> >> > > > > > performances, but we should be able to support it.
>> >> > > > >
>> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
>> >> > > > > unreasonable to say that if you use this sort of configuration you may
>> >> > > > > not get cpufreq support.
>> >> > > > >
>> >> > > > > > Ian, what do you think about this?
>> >> > > > >
>> >> > > > > All the options suck in one way or another AFAICT. I think we are going
>> >> > > > > to be looking for the least bad solution not necessarily a good one.
>> >> > > > >
>> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
>> >> > > > > in the hypervisor to be be able to change the voltages before/after
>> >> > > > > changing the frequency?
>> >> > > > >
>> >> > > > > We can't just say "that's part of the cpufreq driver" since different
>> >> > > > > boards using the same SoC might use different voltage regulators, over
>> >> > > > > i2c or some other bus etc, so we end up with a matrix.
>> >> > > > >
>> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
>> >> > > > > functionality anyway, at least not all of it. Taking that ability away
>> >> > > > > would necessarily imply more platform specific functionality in the
>> >> > > > > hypervisor.
>> >> > > >
>> >> > > > Right.
>> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
>> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
>> >> > >
>> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
>> >> > > do this in the hypervisor.
>> >> > >
>> >> > > Although there are a myriad of them the parts used to do voltage control
>> >> > > tend to be fairly simple.
>> >> > >
>> >> > > One concern I have is that i2c busses also tend to have other things on
>> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
>> >> > > to suggest here.
>> >> >
>> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
>> >> > complicated in Linux kernel. It consists of several parts - common
>> >> > core + platform specific. I'm pretty sure Xen should not handle this.
>> >> > I think that establishing of event channel for frequency changing is a
>> >> > good idea. It would be good to try to implement this. In process of
>> >> > implementation we will see what is need to be resolved.
>> >>
>> >> OK, that's reasonable.
>> >>
>> >>
>> >> > The only question here is how to pass physical cpu to dom0.
>> >>
>> >> We can use a device tree based interface to pass the information to
>> >> dom0, but requiring a number of dom0 vcpus equal to the number of
>> >> physical cpus and in addition to that having to pin the vcpus each to a
>> >> different pcpu is quite a stringent limitation. However I don't know the
>> >> frequency changing interfaces in Linux well enough to know how hard
>> >> would be to lift it.
>> >>
>> >>
>> >> > Regarding x86.
>> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>> >>
>> >> I think so (but I am not an expert on that).
>> >
>> > The usual states are P and C states. The P states is the closes to what you
>> > are looking at:
>> >
>> > struct acpi_processor_px {
>> >         u64 core_frequency;     /* megahertz */
>> >         u64 power;      /* milliWatts */
>> >         u64 transition_latency; /* microseconds */
>> >         u64 bus_master_latency; /* microseconds */
>> >         u64 control;    /* control value */
>> >         u64 status;     /* success indicator */
>> > };
>> >
>> >>
>> >>
>> >>
>> >> > Regards,
>> >> > Andrii
>> >> >
>> >> >
>> >> > --
>> >> >
>> >> > Andrii Tseglytskyi | Embedded Dev
>> >> > GlobalLogic
>> >> > www.globallogic.com
>> >> >
>> >>
>> >> _______________________________________________
>> >> Xen-devel mailing list
>> >> Xen-devel@lists.xen.org
>> >> http://lists.xen.org/xen-devel
>>
>>
>>                     Cpufreq driver implementation.
>>                                  ____________
>>                                 /            \
>>                                 | xenpm tool |
>>                                 \____________/
>>  Dom0 kernel user-space
>> ---------------------------------------------------------------------------
>>
>>                           ________________               _____
>>                          /                \             /     \  CPU
>>                          | DevTree Parser |          /->| ARM | driver
>>                          \________________/          |  \_____/
>>  Dom0 kernel                                         |     |
>> -----------------------------------------------------|-----|---------------
>>                                                      |     |
>>               _____________________________________  |     |
>>              |     __________        ___________   | |     |
>>              |    /          \      /           \  | |     |
>>              |    | ondemand |      | userspace |  | |     |
>>  Registered  |    \__________/      \___________/  | |     |
>>   cpufreq    |   _____________       ___________   | |     |
>>  governor    |  /             \     /           \  | |     |
>>              |  | performance |     | powersave |  | |     |
>>              |  \_____________/     \___________/  | |     |
>>              |_____________________________________| |     |
>>                                ^                     |     |
>>                                |                     |     |
>>                          ______|_______              |     |
>>                         /              \             |     |  Change
>>                         | cpufreq core |-------------/     | frequency
>>                         \______________/ set/get freq      |
>>                                          commands          |
>>  Xen                                                       |
>> -----------------------------------------------------------|--------------
>>  Hardware                                                __V__
>>                                                         |     |
>>                                                         | CPU |
>>                                                         |_____|
>>
>>
>> Description of the implementation:
>> Cpufreq core and registered cpufreq governors are located in xen. Dom0
>> has CPU driver
>> which can only change frequency of the physical CPUs. In addition this driver
>> can change CPUs regulator voltage. I'll reuse some ACPI-specific
>> variables for ARM.
>> Thus I can make minimum modification in the xen cpufreq driver and all utilities
>> (as xenpm) will be working without modification if the xen code. In first
>> implementation xenpm tool won't show information about C-states, but it can show
>> information about P-states and can change cpufreq parameters and
>> change governor.
>> DevTree parser is a part of the CPU driver in Dom0 and it will read information
>> from /cpus/cpu@0/private_data path instead of the original /cpus path.
>>
>> Steps of the initialization:
>> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
>> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
>> any number
>> of VCPUs in Dom0 and we give all information about all physical CPUs in
>> the private_data node.
>>
>> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
>> path and give the information about CPUs parameters to the hypervisor via
>> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
>> Dom0 driver and can not be calculated  in the hypervisor).
>>
>> 3. Cpufreq core driver in the hypervisor will communicate via some interface
>> with Dom0 (event channel can be used to notify Dom0) and give some commands
>> to the CPU driver in Dom0. Those command are set/get frequency, etc.
>>
>> Can I implement cpufreq driver in this way?
>
> The architecture looks sane to me. As Konrad pointed out, the difficulty
> here is to be able to upstream the changes to the Linux driver in 2),
> that you later in the thread identified as
> drivers/cpufreq/cpufreq-cpu0.c.
I'll write driver drivers/xen/xen-cpufreq.c and it replace original
drivers/cpufreq/cpufreq.c
And in the original cpufreq-cpu0 driver I'll chande only one string -
path in the device tree
with the settings for the CPUs opp:
string
np = of_find_node_by_path("/cpus/cpu@0");
will changed to:
np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");

> If the changes are not invasive and you manage to upstream them in
> Linux, I am all for this solution.
In Linux kernel I should make few changes:
1. Enable CONFIG_CPU_FREQ_TABLE
with disabled CONFIG_CPU_FREQ
2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
with disabled CONFIG_CPU_FREQ

I mean make those configs dependent on
CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
instead of
CONFIG_CPU_FREQ

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Wed, Sep 24, 2014 at 5:46 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Sep 24, 2014 at 03:36:15PM +0100, Stefano Stabellini wrote:
>> On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
>> > On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
>> > <konrad.wilk@oracle.com> wrote:
>> > > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
>> > >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
>> > >> > Hi,
>> > >> >
>> > >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> > >> > >
>> > >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
>> > >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
>> > >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
>> > >> > > > > > I am trying to think of an alternative, such as passing the real cpu
>> > >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
>> > >> > > > > > whether Linux checks the status for cpu nodes.
>> > >> > > > >
>> > >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
>> > >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
>> > >> > > > > similar patch to Xen to ignore them. I think it basically means "present
>> > >> > > > > but not running, you should start them!".
>> > >> > > > >
>> > >> > > > > >  In addition this scheme
>> > >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
>> > >> > > > > > system. Granted it is not very common and might even be detrimental for
>> > >> > > > > > performances, but we should be able to support it.
>> > >> > > > >
>> > >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
>> > >> > > > > unreasonable to say that if you use this sort of configuration you may
>> > >> > > > > not get cpufreq support.
>> > >> > > > >
>> > >> > > > > > Ian, what do you think about this?
>> > >> > > > >
>> > >> > > > > All the options suck in one way or another AFAICT. I think we are going
>> > >> > > > > to be looking for the least bad solution not necessarily a good one.
>> > >> > > > >
>> > >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
>> > >> > > > > in the hypervisor to be be able to change the voltages before/after
>> > >> > > > > changing the frequency?
>> > >> > > > >
>> > >> > > > > We can't just say "that's part of the cpufreq driver" since different
>> > >> > > > > boards using the same SoC might use different voltage regulators, over
>> > >> > > > > i2c or some other bus etc, so we end up with a matrix.
>> > >> > > > >
>> > >> > > > > It's arguable that we should be letting dom0 poke at that regulator
>> > >> > > > > functionality anyway, at least not all of it. Taking that ability away
>> > >> > > > > would necessarily imply more platform specific functionality in the
>> > >> > > > > hypervisor.
>> > >> > > >
>> > >> > > > Right.
>> > >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
>> > >> > > > unmaintainable interface and unupstreamable hacks in dom0.
>> > >> > >
>> > >> > > That's what I'm worried about to. Hence I'm wondering if we should just
>> > >> > > do this in the hypervisor.
>> > >> > >
>> > >> > > Although there are a myriad of them the parts used to do voltage control
>> > >> > > tend to be fairly simple.
>> > >> > >
>> > >> > > One concern I have is that i2c busses also tend to have other things on
>> > >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
>> > >> > > to suggest here.
>> > >> >
>> > >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
>> > >> > complicated in Linux kernel. It consists of several parts - common
>> > >> > core + platform specific. I'm pretty sure Xen should not handle this.
>> > >> > I think that establishing of event channel for frequency changing is a
>> > >> > good idea. It would be good to try to implement this. In process of
>> > >> > implementation we will see what is need to be resolved.
>> > >>
>> > >> OK, that's reasonable.
>> > >>
>> > >>
>> > >> > The only question here is how to pass physical cpu to dom0.
>> > >>
>> > >> We can use a device tree based interface to pass the information to
>> > >> dom0, but requiring a number of dom0 vcpus equal to the number of
>> > >> physical cpus and in addition to that having to pin the vcpus each to a
>> > >> different pcpu is quite a stringent limitation. However I don't know the
>> > >> frequency changing interfaces in Linux well enough to know how hard
>> > >> would be to lift it.
>> > >>
>> > >>
>> > >> > Regarding x86.
>> > >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>> > >>
>> > >> I think so (but I am not an expert on that).
>> > >
>> > > The usual states are P and C states. The P states is the closes to what you
>> > > are looking at:
>> > >
>> > > struct acpi_processor_px {
>> > >         u64 core_frequency;     /* megahertz */
>> > >         u64 power;      /* milliWatts */
>> > >         u64 transition_latency; /* microseconds */
>> > >         u64 bus_master_latency; /* microseconds */
>> > >         u64 control;    /* control value */
>> > >         u64 status;     /* success indicator */
>> > > };
>> > >
>> > >>
>> > >>
>> > >>
>> > >> > Regards,
>> > >> > Andrii
>> > >> >
>> > >> >
>> > >> > --
>> > >> >
>> > >> > Andrii Tseglytskyi | Embedded Dev
>> > >> > GlobalLogic
>> > >> > www.globallogic.com
>> > >> >
>> > >>
>> > >> _______________________________________________
>> > >> Xen-devel mailing list
>> > >> Xen-devel@lists.xen.org
>> > >> http://lists.xen.org/xen-devel
>> >
>> >
>> >                     Cpufreq driver implementation.
>> >                                  ____________
>> >                                 /            \
>> >                                 | xenpm tool |
>> >                                 \____________/
>> >  Dom0 kernel user-space
>> > ---------------------------------------------------------------------------
>> >
>> >                           ________________               _____
>> >                          /                \             /     \  CPU
>> >                          | DevTree Parser |          /->| ARM | driver
>> >                          \________________/          |  \_____/
>> >  Dom0 kernel                                         |     |
>> > -----------------------------------------------------|-----|---------------
>> >                                                      |     |
>> >               _____________________________________  |     |
>> >              |     __________        ___________   | |     |
>> >              |    /          \      /           \  | |     |
>> >              |    | ondemand |      | userspace |  | |     |
>> >  Registered  |    \__________/      \___________/  | |     |
>> >   cpufreq    |   _____________       ___________   | |     |
>> >  governor    |  /             \     /           \  | |     |
>> >              |  | performance |     | powersave |  | |     |
>> >              |  \_____________/     \___________/  | |     |
>> >              |_____________________________________| |     |
>> >                                ^                     |     |
>> >                                |                     |     |
>> >                          ______|_______              |     |
>> >                         /              \             |     |  Change
>> >                         | cpufreq core |-------------/     | frequency
>> >                         \______________/ set/get freq      |
>> >                                          commands          |
>> >  Xen                                                       |
>> > -----------------------------------------------------------|--------------
>> >  Hardware                                                __V__
>> >                                                         |     |
>> >                                                         | CPU |
>> >                                                         |_____|
>> >
>> >
>> > Description of the implementation:
>> > Cpufreq core and registered cpufreq governors are located in xen. Dom0
>> > has CPU driver
>> > which can only change frequency of the physical CPUs. In addition this driver
>> > can change CPUs regulator voltage. I'll reuse some ACPI-specific
>> > variables for ARM.
>> > Thus I can make minimum modification in the xen cpufreq driver and all utilities
>> > (as xenpm) will be working without modification if the xen code. In first
>> > implementation xenpm tool won't show information about C-states, but it can show
>> > information about P-states and can change cpufreq parameters and
>> > change governor.
>> > DevTree parser is a part of the CPU driver in Dom0 and it will read information
>> > from /cpus/cpu@0/private_data path instead of the original /cpus path.
>> >
>> > Steps of the initialization:
>> > 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
>> > /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
>> > any number
>> > of VCPUs in Dom0 and we give all information about all physical CPUs in
>> > the private_data node.
>> >
>> > 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
>> > path and give the information about CPUs parameters to the hypervisor via
>> > XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
>> > Dom0 driver and can not be calculated  in the hypervisor).
>> >
>> > 3. Cpufreq core driver in the hypervisor will communicate via some interface
>> > with Dom0 (event channel can be used to notify Dom0) and give some commands
>> > to the CPU driver in Dom0. Those command are set/get frequency, etc.
>> >
>> > Can I implement cpufreq driver in this way?
>>
>> The architecture looks sane to me. As Konrad pointed out, the difficulty
>> here is to be able to upstream the changes to the Linux driver in 2),
>> that you later in the thread identified as
>> drivers/cpufreq/cpufreq-cpu0.c.
>>
>> If the changes are not invasive and you manage to upstream them in
>> Linux, I am all for this solution.
>
> Looking at the driver, you could make some of the drivers functionality
> be a library (all of those 'voltage-tolerance', 'clock-latency', regulator,
> etc).
>
> And then disable the cpufreq API from working altogether (disable_cpufreq())
> and have your own driver (drivers/xen/xen-cpufreq-cpu0.c) use the libraries
> and upload the data to the hypervisor.
In this case I should copy original cpufreq-cpu0.c file to the
drivers/xen/xen-cpufreq-cpu0.c, change only one string -
path in the device tree
with the settings for the CPUs opp:
string
np = of_find_node_by_path("/cpus/cpu@0");
will changed to:
np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");

and write additional code. But I want to avoid copying code.

> (or

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Stefano Stabellini]

On Thu, 25 Sep 2014, Oleksandr Dmytryshyn wrote:
> On Wed, Sep 24, 2014 at 5:36 PM, Stefano Stabellini
> <stefano.stabellini@eu.citrix.com> wrote:
> > On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
> >> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
> >> <konrad.wilk@oracle.com> wrote:
> >> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
> >> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
> >> >> > Hi,
> >> >> >
> >> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> >> >> > >
> >> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
> >> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
> >> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
> >> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
> >> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
> >> >> > > > > > whether Linux checks the status for cpu nodes.
> >> >> > > > >
> >> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
> >> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
> >> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
> >> >> > > > > but not running, you should start them!".
> >> >> > > > >
> >> >> > > > > >  In addition this scheme
> >> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
> >> >> > > > > > system. Granted it is not very common and might even be detrimental for
> >> >> > > > > > performances, but we should be able to support it.
> >> >> > > > >
> >> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
> >> >> > > > > unreasonable to say that if you use this sort of configuration you may
> >> >> > > > > not get cpufreq support.
> >> >> > > > >
> >> >> > > > > > Ian, what do you think about this?
> >> >> > > > >
> >> >> > > > > All the options suck in one way or another AFAICT. I think we are going
> >> >> > > > > to be looking for the least bad solution not necessarily a good one.
> >> >> > > > >
> >> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
> >> >> > > > > in the hypervisor to be be able to change the voltages before/after
> >> >> > > > > changing the frequency?
> >> >> > > > >
> >> >> > > > > We can't just say "that's part of the cpufreq driver" since different
> >> >> > > > > boards using the same SoC might use different voltage regulators, over
> >> >> > > > > i2c or some other bus etc, so we end up with a matrix.
> >> >> > > > >
> >> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
> >> >> > > > > functionality anyway, at least not all of it. Taking that ability away
> >> >> > > > > would necessarily imply more platform specific functionality in the
> >> >> > > > > hypervisor.
> >> >> > > >
> >> >> > > > Right.
> >> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
> >> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
> >> >> > >
> >> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
> >> >> > > do this in the hypervisor.
> >> >> > >
> >> >> > > Although there are a myriad of them the parts used to do voltage control
> >> >> > > tend to be fairly simple.
> >> >> > >
> >> >> > > One concern I have is that i2c busses also tend to have other things on
> >> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
> >> >> > > to suggest here.
> >> >> >
> >> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
> >> >> > complicated in Linux kernel. It consists of several parts - common
> >> >> > core + platform specific. I'm pretty sure Xen should not handle this.
> >> >> > I think that establishing of event channel for frequency changing is a
> >> >> > good idea. It would be good to try to implement this. In process of
> >> >> > implementation we will see what is need to be resolved.
> >> >>
> >> >> OK, that's reasonable.
> >> >>
> >> >>
> >> >> > The only question here is how to pass physical cpu to dom0.
> >> >>
> >> >> We can use a device tree based interface to pass the information to
> >> >> dom0, but requiring a number of dom0 vcpus equal to the number of
> >> >> physical cpus and in addition to that having to pin the vcpus each to a
> >> >> different pcpu is quite a stringent limitation. However I don't know the
> >> >> frequency changing interfaces in Linux well enough to know how hard
> >> >> would be to lift it.
> >> >>
> >> >>
> >> >> > Regarding x86.
> >> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
> >> >>
> >> >> I think so (but I am not an expert on that).
> >> >
> >> > The usual states are P and C states. The P states is the closes to what you
> >> > are looking at:
> >> >
> >> > struct acpi_processor_px {
> >> >         u64 core_frequency;     /* megahertz */
> >> >         u64 power;      /* milliWatts */
> >> >         u64 transition_latency; /* microseconds */
> >> >         u64 bus_master_latency; /* microseconds */
> >> >         u64 control;    /* control value */
> >> >         u64 status;     /* success indicator */
> >> > };
> >> >
> >> >>
> >> >>
> >> >>
> >> >> > Regards,
> >> >> > Andrii
> >> >> >
> >> >> >
> >> >> > --
> >> >> >
> >> >> > Andrii Tseglytskyi | Embedded Dev
> >> >> > GlobalLogic
> >> >> > www.globallogic.com
> >> >> >
> >> >>
> >> >> _______________________________________________
> >> >> Xen-devel mailing list
> >> >> Xen-devel@lists.xen.org
> >> >> http://lists.xen.org/xen-devel
> >>
> >>
> >>                     Cpufreq driver implementation.
> >>                                  ____________
> >>                                 /            \
> >>                                 | xenpm tool |
> >>                                 \____________/
> >>  Dom0 kernel user-space
> >> ---------------------------------------------------------------------------
> >>
> >>                           ________________               _____
> >>                          /                \             /     \  CPU
> >>                          | DevTree Parser |          /->| ARM | driver
> >>                          \________________/          |  \_____/
> >>  Dom0 kernel                                         |     |
> >> -----------------------------------------------------|-----|---------------
> >>                                                      |     |
> >>               _____________________________________  |     |
> >>              |     __________        ___________   | |     |
> >>              |    /          \      /           \  | |     |
> >>              |    | ondemand |      | userspace |  | |     |
> >>  Registered  |    \__________/      \___________/  | |     |
> >>   cpufreq    |   _____________       ___________   | |     |
> >>  governor    |  /             \     /           \  | |     |
> >>              |  | performance |     | powersave |  | |     |
> >>              |  \_____________/     \___________/  | |     |
> >>              |_____________________________________| |     |
> >>                                ^                     |     |
> >>                                |                     |     |
> >>                          ______|_______              |     |
> >>                         /              \             |     |  Change
> >>                         | cpufreq core |-------------/     | frequency
> >>                         \______________/ set/get freq      |
> >>                                          commands          |
> >>  Xen                                                       |
> >> -----------------------------------------------------------|--------------
> >>  Hardware                                                __V__
> >>                                                         |     |
> >>                                                         | CPU |
> >>                                                         |_____|
> >>
> >>
> >> Description of the implementation:
> >> Cpufreq core and registered cpufreq governors are located in xen. Dom0
> >> has CPU driver
> >> which can only change frequency of the physical CPUs. In addition this driver
> >> can change CPUs regulator voltage. I'll reuse some ACPI-specific
> >> variables for ARM.
> >> Thus I can make minimum modification in the xen cpufreq driver and all utilities
> >> (as xenpm) will be working without modification if the xen code. In first
> >> implementation xenpm tool won't show information about C-states, but it can show
> >> information about P-states and can change cpufreq parameters and
> >> change governor.
> >> DevTree parser is a part of the CPU driver in Dom0 and it will read information
> >> from /cpus/cpu@0/private_data path instead of the original /cpus path.
> >>
> >> Steps of the initialization:
> >> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
> >> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
> >> any number
> >> of VCPUs in Dom0 and we give all information about all physical CPUs in
> >> the private_data node.
> >>
> >> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
> >> path and give the information about CPUs parameters to the hypervisor via
> >> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
> >> Dom0 driver and can not be calculated  in the hypervisor).
> >>
> >> 3. Cpufreq core driver in the hypervisor will communicate via some interface
> >> with Dom0 (event channel can be used to notify Dom0) and give some commands
> >> to the CPU driver in Dom0. Those command are set/get frequency, etc.
> >>
> >> Can I implement cpufreq driver in this way?
> >
> > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > here is to be able to upstream the changes to the Linux driver in 2),
> > that you later in the thread identified as
> > drivers/cpufreq/cpufreq-cpu0.c.
> I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> drivers/cpufreq/cpufreq.c
> And in the original cpufreq-cpu0 driver I'll chande only one string -
> path in the device tree
> with the settings for the CPUs opp:
> string
> np = of_find_node_by_path("/cpus/cpu@0");
> will changed to:
> np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");

There is no way that Linux upstream is going to accept code copied
like that. However if you refactor the code from
drivers/cpufreq/cpufreq-cpu0.c like Konrad suggested, moving the
functions we need to a separate file, then you can call into these
function from both cpufreq-cpu0.c and xen-cpufreq.c.


> > If the changes are not invasive and you manage to upstream them in
> > Linux, I am all for this solution.
> In Linux kernel I should make few changes:
> 1. Enable CONFIG_CPU_FREQ_TABLE
> with disabled CONFIG_CPU_FREQ
> 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> with disabled CONFIG_CPU_FREQ
> 
> I mean make those configs dependent on
> CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> instead of
> CONFIG_CPU_FREQ

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Thu, Sep 25, 2014 at 1:14 PM, Stefano Stabellini
<stefano.stabellini@eu.citrix.com> wrote:
> On Thu, 25 Sep 2014, Oleksandr Dmytryshyn wrote:
>> On Wed, Sep 24, 2014 at 5:36 PM, Stefano Stabellini
>> <stefano.stabellini@eu.citrix.com> wrote:
>> > On Tue, 16 Sep 2014, Oleksandr Dmytryshyn wrote:
>> >> On Wed, Sep 10, 2014 at 10:31 PM, Konrad Rzeszutek Wilk
>> >> <konrad.wilk@oracle.com> wrote:
>> >> > On Wed, Sep 10, 2014 at 07:35:47PM +0100, Stefano Stabellini wrote:
>> >> >> On Wed, 10 Sep 2014, Andrii Tseglytskyi wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > On Wed, Sep 10, 2014 at 12:42 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> >> >> > >
>> >> >> > > On Tue, 2014-09-09 at 22:41 +0100, Stefano Stabellini wrote:
>> >> >> > > > On Tue, 9 Sep 2014, Ian Campbell wrote:
>> >> >> > > > > On Thu, 2014-09-04 at 22:56 +0100, Stefano Stabellini wrote:
>> >> >> > > > > > I am trying to think of an alternative, such as passing the real cpu
>> >> >> > > > > > nodes to dom0 but then adding status = "disabled", but I am not sure
>> >> >> > > > > > whether Linux checks the status for cpu nodes.
>> >> >> > > > >
>> >> >> > > > > status = "disabled" is defined to have a specific (i.e. non-default)
>> >> >> > > > > meaning for cpu nodes, Julien mentioned this when I tried to add a
>> >> >> > > > > similar patch to Xen to ignore them. I think it basically means "present
>> >> >> > > > > but not running, you should start them!".
>> >> >> > > > >
>> >> >> > > > > >  In addition this scheme
>> >> >> > > > > > wouldn't support the case where dom0 has more vcpus than pcpus on the
>> >> >> > > > > > system. Granted it is not very common and might even be detrimental for
>> >> >> > > > > > performances, but we should be able to support it.
>> >> >> > > > >
>> >> >> > > > > It's a bit of an edge case, for sure. I guess it wouldn't be totally
>> >> >> > > > > unreasonable to say that if you use this sort of configuration you may
>> >> >> > > > > not get cpufreq support.
>> >> >> > > > >
>> >> >> > > > > > Ian, what do you think about this?
>> >> >> > > > >
>> >> >> > > > > All the options suck in one way or another AFAICT. I think we are going
>> >> >> > > > > to be looking for the least bad solution not necessarily a good one.
>> >> >> > > > >
>> >> >> > > > > Fundamentally are we trying to avoid having to have a i2c subsystem etc
>> >> >> > > > > in the hypervisor to be be able to change the voltages before/after
>> >> >> > > > > changing the frequency?
>> >> >> > > > >
>> >> >> > > > > We can't just say "that's part of the cpufreq driver" since different
>> >> >> > > > > boards using the same SoC might use different voltage regulators, over
>> >> >> > > > > i2c or some other bus etc, so we end up with a matrix.
>> >> >> > > > >
>> >> >> > > > > It's arguable that we should be letting dom0 poke at that regulator
>> >> >> > > > > functionality anyway, at least not all of it. Taking that ability away
>> >> >> > > > > would necessarily imply more platform specific functionality in the
>> >> >> > > > > hypervisor.
>> >> >> > > >
>> >> >> > > > Right.
>> >> >> > > > I am afraid that in order to avoid more code in Xen, we end up with an
>> >> >> > > > unmaintainable interface and unupstreamable hacks in dom0.
>> >> >> > >
>> >> >> > > That's what I'm worried about to. Hence I'm wondering if we should just
>> >> >> > > do this in the hypervisor.
>> >> >> > >
>> >> >> > > Although there are a myriad of them the parts used to do voltage control
>> >> >> > > tend to be fairly simple.
>> >> >> > >
>> >> >> > > One concern I have is that i2c busses also tend to have other things on
>> >> >> > > them which dom0 might legitimately access (e.g. rtc), I'm not sure what
>> >> >> > > to suggest here.
>> >> >> >
>> >> >> > I would try to avoid i2c transactions in Xen. I2C driver is quite
>> >> >> > complicated in Linux kernel. It consists of several parts - common
>> >> >> > core + platform specific. I'm pretty sure Xen should not handle this.
>> >> >> > I think that establishing of event channel for frequency changing is a
>> >> >> > good idea. It would be good to try to implement this. In process of
>> >> >> > implementation we will see what is need to be resolved.
>> >> >>
>> >> >> OK, that's reasonable.
>> >> >>
>> >> >>
>> >> >> > The only question here is how to pass physical cpu to dom0.
>> >> >>
>> >> >> We can use a device tree based interface to pass the information to
>> >> >> dom0, but requiring a number of dom0 vcpus equal to the number of
>> >> >> physical cpus and in addition to that having to pin the vcpus each to a
>> >> >> different pcpu is quite a stringent limitation. However I don't know the
>> >> >> frequency changing interfaces in Linux well enough to know how hard
>> >> >> would be to lift it.
>> >> >>
>> >> >>
>> >> >> > Regarding x86.
>> >> >> > I'm not sure but maybe ACPI interface encapsulate voltage changing as well?
>> >> >>
>> >> >> I think so (but I am not an expert on that).
>> >> >
>> >> > The usual states are P and C states. The P states is the closes to what you
>> >> > are looking at:
>> >> >
>> >> > struct acpi_processor_px {
>> >> >         u64 core_frequency;     /* megahertz */
>> >> >         u64 power;      /* milliWatts */
>> >> >         u64 transition_latency; /* microseconds */
>> >> >         u64 bus_master_latency; /* microseconds */
>> >> >         u64 control;    /* control value */
>> >> >         u64 status;     /* success indicator */
>> >> > };
>> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> > Regards,
>> >> >> > Andrii
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> >
>> >> >> > Andrii Tseglytskyi | Embedded Dev
>> >> >> > GlobalLogic
>> >> >> > www.globallogic.com
>> >> >> >
>> >> >>
>> >> >> _______________________________________________
>> >> >> Xen-devel mailing list
>> >> >> Xen-devel@lists.xen.org
>> >> >> http://lists.xen.org/xen-devel
>> >>
>> >>
>> >>                     Cpufreq driver implementation.
>> >>                                  ____________
>> >>                                 /            \
>> >>                                 | xenpm tool |
>> >>                                 \____________/
>> >>  Dom0 kernel user-space
>> >> ---------------------------------------------------------------------------
>> >>
>> >>                           ________________               _____
>> >>                          /                \             /     \  CPU
>> >>                          | DevTree Parser |          /->| ARM | driver
>> >>                          \________________/          |  \_____/
>> >>  Dom0 kernel                                         |     |
>> >> -----------------------------------------------------|-----|---------------
>> >>                                                      |     |
>> >>               _____________________________________  |     |
>> >>              |     __________        ___________   | |     |
>> >>              |    /          \      /           \  | |     |
>> >>              |    | ondemand |      | userspace |  | |     |
>> >>  Registered  |    \__________/      \___________/  | |     |
>> >>   cpufreq    |   _____________       ___________   | |     |
>> >>  governor    |  /             \     /           \  | |     |
>> >>              |  | performance |     | powersave |  | |     |
>> >>              |  \_____________/     \___________/  | |     |
>> >>              |_____________________________________| |     |
>> >>                                ^                     |     |
>> >>                                |                     |     |
>> >>                          ______|_______              |     |
>> >>                         /              \             |     |  Change
>> >>                         | cpufreq core |-------------/     | frequency
>> >>                         \______________/ set/get freq      |
>> >>                                          commands          |
>> >>  Xen                                                       |
>> >> -----------------------------------------------------------|--------------
>> >>  Hardware                                                __V__
>> >>                                                         |     |
>> >>                                                         | CPU |
>> >>                                                         |_____|
>> >>
>> >>
>> >> Description of the implementation:
>> >> Cpufreq core and registered cpufreq governors are located in xen. Dom0
>> >> has CPU driver
>> >> which can only change frequency of the physical CPUs. In addition this driver
>> >> can change CPUs regulator voltage. I'll reuse some ACPI-specific
>> >> variables for ARM.
>> >> Thus I can make minimum modification in the xen cpufreq driver and all utilities
>> >> (as xenpm) will be working without modification if the xen code. In first
>> >> implementation xenpm tool won't show information about C-states, but it can show
>> >> information about P-states and can change cpufreq parameters and
>> >> change governor.
>> >> DevTree parser is a part of the CPU driver in Dom0 and it will read information
>> >> from /cpus/cpu@0/private_data path instead of the original /cpus path.
>> >>
>> >> Steps of the initialization:
>> >> 1. Xen copies all cpu@0..cpu@N nodes (from input device tree) with properties to
>> >> /cpus/cpu@0/private_data node (device tree for Dom0). Thus we can have
>> >> any number
>> >> of VCPUs in Dom0 and we give all information about all physical CPUs in
>> >> the private_data node.
>> >>
>> >> 2. Driver in Dom0 will parse /cpus/cpu@0/private_data path instead of the /cpus
>> >> path and give the information about CPUs parameters to the hypervisor via
>> >> XENPF_set_processor_pminfo hypercall. (Some parameters are calculated in the
>> >> Dom0 driver and can not be calculated  in the hypervisor).
>> >>
>> >> 3. Cpufreq core driver in the hypervisor will communicate via some interface
>> >> with Dom0 (event channel can be used to notify Dom0) and give some commands
>> >> to the CPU driver in Dom0. Those command are set/get frequency, etc.
>> >>
>> >> Can I implement cpufreq driver in this way?
>> >
>> > The architecture looks sane to me. As Konrad pointed out, the difficulty
>> > here is to be able to upstream the changes to the Linux driver in 2),
>> > that you later in the thread identified as
>> > drivers/cpufreq/cpufreq-cpu0.c.
>> I'll write driver drivers/xen/xen-cpufreq.c and it replace original
>> drivers/cpufreq/cpufreq.c
>> And in the original cpufreq-cpu0 driver I'll chande only one string -
>> path in the device tree
>> with the settings for the CPUs opp:
>> string
>> np = of_find_node_by_path("/cpus/cpu@0");
>> will changed to:
>> np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
>
> There is no way that Linux upstream is going to accept code copied
> like that. However if you refactor the code from
> drivers/cpufreq/cpufreq-cpu0.c like Konrad suggested, moving the
> functions we need to a separate file, then you can call into these
> function from both cpufreq-cpu0.c and xen-cpufreq.c.
I'll see what can I do. At first I'll write and check the cpufreq
driver and then
I'll try to upstream patches for Linux and Xen hypervisor.

>
>> > If the changes are not invasive and you manage to upstream them in
>> > Linux, I am all for this solution.
>> In Linux kernel I should make few changes:
>> 1. Enable CONFIG_CPU_FREQ_TABLE
>> with disabled CONFIG_CPU_FREQ
>> 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
>> with disabled CONFIG_CPU_FREQ
>>
>> I mean make those configs dependent on
>> CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
>> instead of
>> CONFIG_CPU_FREQ

Re: cpufreq implementation for OMAP under xen hypervisor.

[Konrad Rzeszutek Wilk]

> > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > here is to be able to upstream the changes to the Linux driver in 2),
> > that you later in the thread identified as
> > drivers/cpufreq/cpufreq-cpu0.c.
> I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> drivers/cpufreq/cpufreq.c
> And in the original cpufreq-cpu0 driver I'll chande only one string -
> path in the device tree
> with the settings for the CPUs opp:
> string
> np = of_find_node_by_path("/cpus/cpu@0");
> will changed to:
> np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> 
> > If the changes are not invasive and you manage to upstream them in
> > Linux, I am all for this solution.
> In Linux kernel I should make few changes:
> 1. Enable CONFIG_CPU_FREQ_TABLE
> with disabled CONFIG_CPU_FREQ
> 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> with disabled CONFIG_CPU_FREQ
> 
> I mean make those configs dependent on
> CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> instead of
> CONFIG_CPU_FREQ

Gosh no. Please make it work runtime.
> 
> Oleksandr Dmytryshyn | Product Engineering and Development
> GlobalLogic
> M +38.067.382.2525
> www.globallogic.com
> 
> http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Fri, Sep 26, 2014 at 9:13 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>
> > > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > > here is to be able to upstream the changes to the Linux driver in 2),
> > > that you later in the thread identified as
> > > drivers/cpufreq/cpufreq-cpu0.c.
> > I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> > drivers/cpufreq/cpufreq.c
> > And in the original cpufreq-cpu0 driver I'll chande only one string -
> > path in the device tree
> > with the settings for the CPUs opp:
> > string
> > np = of_find_node_by_path("/cpus/cpu@0");
> > will changed to:
> > np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> >
> > > If the changes are not invasive and you manage to upstream them in
> > > Linux, I am all for this solution.
> > In Linux kernel I should make few changes:
> > 1. Enable CONFIG_CPU_FREQ_TABLE
> > with disabled CONFIG_CPU_FREQ
> > 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> > with disabled CONFIG_CPU_FREQ
> >
> > I mean make those configs dependent on
> > CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> > instead of
> > CONFIG_CPU_FREQ
>
> Gosh no. Please make it work runtime.
Sorry, Konrad, could You please explain what did You mean.

> >
> > Oleksandr Dmytryshyn | Product Engineering and Development
> > GlobalLogic
> > M +38.067.382.2525
> > www.globallogic.com
> >
> > http://www.globallogic.com/email_disclaimer.txt

Oleksandr Dmytryshyn | Product Engineering and Development
GlobalLogic
M +38.067.382.2525
www.globallogic.com

http://www.globallogic.com/email_disclaimer.txt

Re: cpufreq implementation for OMAP under xen hypervisor.

[Konrad Rzeszutek Wilk]

On Mon, Sep 29, 2014 at 12:45:36PM +0300, Oleksandr Dmytryshyn wrote:
> On Fri, Sep 26, 2014 at 9:13 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> >
> > > > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > > > here is to be able to upstream the changes to the Linux driver in 2),
> > > > that you later in the thread identified as
> > > > drivers/cpufreq/cpufreq-cpu0.c.
> > > I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> > > drivers/cpufreq/cpufreq.c
> > > And in the original cpufreq-cpu0 driver I'll chande only one string -
> > > path in the device tree
> > > with the settings for the CPUs opp:
> > > string
> > > np = of_find_node_by_path("/cpus/cpu@0");
> > > will changed to:
> > > np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> > >
> > > > If the changes are not invasive and you manage to upstream them in
> > > > Linux, I am all for this solution.
> > > In Linux kernel I should make few changes:
> > > 1. Enable CONFIG_CPU_FREQ_TABLE
> > > with disabled CONFIG_CPU_FREQ
> > > 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> > > with disabled CONFIG_CPU_FREQ
> > >
> > > I mean make those configs dependent on
> > > CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> > > instead of
> > > CONFIG_CPU_FREQ
> >
> > Gosh no. Please make it work runtime.
> Sorry, Konrad, could You please explain what did You mean.

Don't make XEN options disable other options.

Distributions want one kernel that can satisfi a variety of platforms - not just specifc ones.

Re: cpufreq implementation for OMAP under xen hypervisor.

[Oleksandr Dmytryshyn]

On Mon, Sep 29, 2014 at 6:18 PM, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>
> On Mon, Sep 29, 2014 at 12:45:36PM +0300, Oleksandr Dmytryshyn wrote:
> > On Fri, Sep 26, 2014 at 9:13 PM, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com> wrote:
> > >
> > > > > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > > > > here is to be able to upstream the changes to the Linux driver in 2),
> > > > > that you later in the thread identified as
> > > > > drivers/cpufreq/cpufreq-cpu0.c.
> > > > I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> > > > drivers/cpufreq/cpufreq.c
> > > > And in the original cpufreq-cpu0 driver I'll chande only one string -
> > > > path in the device tree
> > > > with the settings for the CPUs opp:
> > > > string
> > > > np = of_find_node_by_path("/cpus/cpu@0");
> > > > will changed to:
> > > > np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> > > >
> > > > > If the changes are not invasive and you manage to upstream them in
> > > > > Linux, I am all for this solution.
> > > > In Linux kernel I should make few changes:
> > > > 1. Enable CONFIG_CPU_FREQ_TABLE
> > > > with disabled CONFIG_CPU_FREQ
> > > > 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> > > > with disabled CONFIG_CPU_FREQ
> > > >
> > > > I mean make those configs dependent on
> > > > CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> > > > instead of
> > > > CONFIG_CPU_FREQ
> > >
> > > Gosh no. Please make it work runtime.
> > Sorry, Konrad, could You please explain what did You mean.
>
> Don't make XEN options disable other options.
>
> Distributions want one kernel that can satisfi a variety of platforms - not just specifc ones.
XEN option will not disable other options. On the contrary, it will
extend them. In my case XEN option
will allow to select more options.

Please, see example
diff --git a/drivers/cpufreq/Kconfig b/drivers/cpufreq/Kconfig
index cbcb21e..4531e04 100644
--- a/drivers/cpufreq/Kconfig
+++ b/drivers/cpufreq/Kconfig
@@ -15,11 +15,15 @@ config CPU_FREQ

          If in doubt, say N.

-if CPU_FREQ
+if CPU_FREQ || XEN_DOM0

 config CPU_FREQ_TABLE
        tristate

@@ -184,6 +188,10 @@ config CPU_FREQ_GOV_CONSERVATIVE

          If in doubt, say N.

+if CPU_FREQ || XEN_DOM0
+
 config GENERIC_CPUFREQ_CPU0
        tristate "Generic CPU0 cpufreq driver"
        depends on HAVE_CLK && REGULATOR && PM_OPP && OF

Re: cpufreq implementation for OMAP under xen hypervisor.

[Konrad Rzeszutek Wilk]

On Tue, Sep 30, 2014 at 01:28:42PM +0300, Oleksandr Dmytryshyn wrote:
> On Mon, Sep 29, 2014 at 6:18 PM, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com> wrote:
> >
> > On Mon, Sep 29, 2014 at 12:45:36PM +0300, Oleksandr Dmytryshyn wrote:
> > > On Fri, Sep 26, 2014 at 9:13 PM, Konrad Rzeszutek Wilk
> > > <konrad.wilk@oracle.com> wrote:
> > > >
> > > > > > The architecture looks sane to me. As Konrad pointed out, the difficulty
> > > > > > here is to be able to upstream the changes to the Linux driver in 2),
> > > > > > that you later in the thread identified as
> > > > > > drivers/cpufreq/cpufreq-cpu0.c.
> > > > > I'll write driver drivers/xen/xen-cpufreq.c and it replace original
> > > > > drivers/cpufreq/cpufreq.c
> > > > > And in the original cpufreq-cpu0 driver I'll chande only one string -
> > > > > path in the device tree
> > > > > with the settings for the CPUs opp:
> > > > > string
> > > > > np = of_find_node_by_path("/cpus/cpu@0");
> > > > > will changed to:
> > > > > np = of_find_node_by_path("/cpus/cpu@0/private_data/cpu@0");
> > > > >
> > > > > > If the changes are not invasive and you manage to upstream them in
> > > > > > Linux, I am all for this solution.
> > > > > In Linux kernel I should make few changes:
> > > > > 1. Enable CONFIG_CPU_FREQ_TABLE
> > > > > with disabled CONFIG_CPU_FREQ
> > > > > 2. Enable CONFIG_GENERIC_CPUFREQ_CPU0
> > > > > with disabled CONFIG_CPU_FREQ
> > > > >
> > > > > I mean make those configs dependent on
> > > > > CONFIG_CPU_FREQ or CONFIX_XEN_DOM0
> > > > > instead of
> > > > > CONFIG_CPU_FREQ
> > > >
> > > > Gosh no. Please make it work runtime.
> > > Sorry, Konrad, could You please explain what did You mean.
> >
> > Don't make XEN options disable other options.
> >
> > Distributions want one kernel that can satisfi a variety of platforms - not just specifc ones.
> XEN option will not disable other options. On the contrary, it will
> extend them. In my case XEN option
> will allow to select more options.
> 
> Please, see example
> diff --git a/drivers/cpufreq/Kconfig b/drivers/cpufreq/Kconfig
> index cbcb21e..4531e04 100644
> --- a/drivers/cpufreq/Kconfig
> +++ b/drivers/cpufreq/Kconfig
> @@ -15,11 +15,15 @@ config CPU_FREQ
> 
>           If in doubt, say N.
> 
> -if CPU_FREQ
> +if CPU_FREQ || XEN_DOM0
> 
>  config CPU_FREQ_TABLE
>         tristate
> 
> @@ -184,6 +188,10 @@ config CPU_FREQ_GOV_CONSERVATIVE
> 
>           If in doubt, say N.
> 
> +if CPU_FREQ || XEN_DOM0
> +

Ah, then it is fine. Sorry for the misunderstanding.
>  config GENERIC_CPUFREQ_CPU0
>         tristate "Generic CPU0 cpufreq driver"
>         depends on HAVE_CLK && REGULATOR && PM_OPP && OF