[Xenomai] NVIDIA Drivers w/ Xenomai

[Xenomai] NVIDIA Drivers w/ Xenomai

[Eric Huang]

Hi,

The specific question I am trying to answer is whether it is possible (at
all) to install an NVIDIA driver alongside Xenomai

My system is the following.

uname -a
Linux robot-Z68X-UD3H-B3 3.2.21-xenomai-2.6.1 #1 SMP Thu Jul 26 11:46:23
EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

xeno-config --version
2.5.5.2

For more details, my application is a real-time controller which needs to
do image processing on a GPU. The image processing code executes at 500ms
on a CPU and 15ms on a GPU. The image frame rate is 30hz. The control loop
frequency is 500hz. The control loop is the only part which needs to be
hard RT.

If I can't install and run the NVIDIA driver at all on my system I will
look into a solution with a separate control and perception computer.

Thanks,
Eric

Re: [Xenomai] NVIDIA Drivers w/ Xenomai

[Lennart Sorensen]

On Mon, Dec 21, 2015 at 07:22:26PM -0500, Eric Huang wrote:
> The specific question I am trying to answer is whether it is possible (at
> all) to install an NVIDIA driver alongside Xenomai
> 
> My system is the following.
> 
> uname -a
> Linux robot-Z68X-UD3H-B3 3.2.21-xenomai-2.6.1 #1 SMP Thu Jul 26 11:46:23
> EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> 
> xeno-config --version
> 2.5.5.2

Wow that is really really old.

> For more details, my application is a real-time controller which needs to
> do image processing on a GPU. The image processing code executes at 500ms
> on a CPU and 15ms on a GPU. The image frame rate is 30hz. The control loop
> frequency is 500hz. The control loop is the only part which needs to be
> hard RT.
> 
> If I can't install and run the NVIDIA driver at all on my system I will
> look into a solution with a separate control and perception computer.

Since xenomai uses ipipe which changes the way IRQs are handled, the
fact you don't have source code to recompile the nvidia driver means you
can make the nvidia driver work correctly with ipipe, so if you have the
nvidia driver running you really have no idea if you will get real time
or not since the nvidia driver won't be following the rules.

At least that is how I have understood the situation.

-- 
Len Sorensen

Re: [Xenomai] NVIDIA Drivers w/ Xenomai

[Gilles Chanteperdrix]

On Mon, Dec 21, 2015 at 09:43:04PM -0500, Lennart Sorensen wrote:
> On Mon, Dec 21, 2015 at 07:22:26PM -0500, Eric Huang wrote:
> > The specific question I am trying to answer is whether it is possible (at
> > all) to install an NVIDIA driver alongside Xenomai
> > 
> > My system is the following.
> > 
> > uname -a
> > Linux robot-Z68X-UD3H-B3 3.2.21-xenomai-2.6.1 #1 SMP Thu Jul 26 11:46:23
> > EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> > 
> > xeno-config --version
> > 2.5.5.2
> 
> Wow that is really really old.
> 
> > For more details, my application is a real-time controller which needs to
> > do image processing on a GPU. The image processing code executes at 500ms
> > on a CPU and 15ms on a GPU. The image frame rate is 30hz. The control loop
> > frequency is 500hz. The control loop is the only part which needs to be
> > hard RT.
> > 
> > If I can't install and run the NVIDIA driver at all on my system I will
> > look into a solution with a separate control and perception computer.
> 
> Since xenomai uses ipipe which changes the way IRQs are handled, the
> fact you don't have source code to recompile the nvidia driver means you
> can make the nvidia driver work correctly with ipipe, so if you have the
> nvidia driver running you really have no idea if you will get real time
> or not since the nvidia driver won't be following the rules.
> 
> At least that is how I have understood the situation.

Last time I looked, NVIDIA drivers used some glue code to adapt to
the particular kernel for which they were compiled. This glue made
use of the standard macro local_irq_save/spin_lock_irqsave, etc...
So, compiling the NVIDIA driver with an I-pipe kernel worked, and so
did running Xenomai programs along side graphic applications. But
that was a long time ago. In any case, running the latency test
while your image processing program is running on the GPU should
allow you to find whether the driver is generating large latencies.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] NVIDIA Drivers w/ Xenomai

[Lennart Sorensen]

On Tue, Dec 22, 2015 at 08:30:37AM +0100, Gilles Chanteperdrix wrote:
> Last time I looked, NVIDIA drivers used some glue code to adapt to
> the particular kernel for which they were compiled. This glue made
> use of the standard macro local_irq_save/spin_lock_irqsave, etc...
> So, compiling the NVIDIA driver with an I-pipe kernel worked, and so
> did running Xenomai programs along side graphic applications. But
> that was a long time ago. In any case, running the latency test
> while your image processing program is running on the GPU should
> allow you to find whether the driver is generating large latencies.

OK, maybe it isn't as bad as I was led to believe in the past then.

I suppose one could hope there are no interrupt disable calls in the
nvidia binary blob directly.

-- 
Len Sorensen

Re: [Xenomai] NVIDIA Drivers w/ Xenomai

[Jeff Webb]

On 12/22/2015 09:54 AM, Lennart Sorensen wrote:
> On Tue, Dec 22, 2015 at 08:30:37AM +0100, Gilles Chanteperdrix wrote:
>> Last time I looked, NVIDIA drivers used some glue code to adapt to
>> the particular kernel for which they were compiled. This glue made
>> use of the standard macro local_irq_save/spin_lock_irqsave, etc...
>> So, compiling the NVIDIA driver with an I-pipe kernel worked, and so
>> did running Xenomai programs along side graphic applications. But
>> that was a long time ago. In any case, running the latency test
>> while your image processing program is running on the GPU should
>> allow you to find whether the driver is generating large latencies.
>
> OK, maybe it isn't as bad as I was led to believe in the past then.
>
> I suppose one could hope there are no interrupt disable calls in the
> nvidia binary blob directly.

I have experienced some high worst-case latencies when using the NVIDIA drivers along side Xenomai real-time code in the past.  The open source drivers seemed to be more well-behaved from a real-time point of view.  You may be able to get the closed-source drivers to work for your application, depending on your timing constraints and GPU feature usage.  As Gilles mentioned, you really should try running the latency test while putting the system (and GPU) under load.  As I remember, switching to another virtual console sometimes triggered some nasty behavior.

-Jeff

Re: [Xenomai] NVIDIA Drivers w/ Xenomai

[Gilles Chanteperdrix]

On Tue, Dec 22, 2015 at 11:33:57AM -0600, Jeff Webb wrote:
> On 12/22/2015 09:54 AM, Lennart Sorensen wrote:
> > On Tue, Dec 22, 2015 at 08:30:37AM +0100, Gilles Chanteperdrix wrote:
> >> Last time I looked, NVIDIA drivers used some glue code to adapt to
> >> the particular kernel for which they were compiled. This glue made
> >> use of the standard macro local_irq_save/spin_lock_irqsave, etc...
> >> So, compiling the NVIDIA driver with an I-pipe kernel worked, and so
> >> did running Xenomai programs along side graphic applications. But
> >> that was a long time ago. In any case, running the latency test
> >> while your image processing program is running on the GPU should
> >> allow you to find whether the driver is generating large latencies.
> >
> > OK, maybe it isn't as bad as I was led to believe in the past then.
> >
> > I suppose one could hope there are no interrupt disable calls in the
> > nvidia binary blob directly.
> 
> I have experienced some high worst-case latencies when using the
> NVIDIA drivers along side Xenomai real-time code in the past. The
> open source drivers seemed to be more well-behaved from a
> real-time point of view.

If you are talking about the "nouveau" drivers, unfortunately I am
afraid they will not satisfy the user needs: last time I tried
writing shaders for the nouveau drivers, the shaders were run on the
CPU, not the GPU.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] NVIDIA Drivers w/ Xenomai

[Gilles Chanteperdrix]

On Mon, Dec 21, 2015 at 07:22:26PM -0500, Eric Huang wrote:
> Hi,
> 
> The specific question I am trying to answer is whether it is possible (at
> all) to install an NVIDIA driver alongside Xenomai
> 
> My system is the following.
> 
> uname -a
> Linux robot-Z68X-UD3H-B3 3.2.21-xenomai-2.6.1 #1 SMP Thu Jul 26 11:46:23
> EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> 
> xeno-config --version
> 2.5.5.2

By the way, you have a mismatch between Xenomai user-space and
kernel-space, xenomai 2.5.5.2 user-space should not work with a
xenomai 2.6.1 kernel space. You need a xenomai 2.6.x user-space for
a xenomai 2.6.1 kernel space. And Xenomai on Linux 3.2 is broken.
So, I would say a general upgrade is in order.

-- 
					    Gilles.
https://click-hack.org