Re: [Hardeneddrivers-discuss] RE: [cgl_discussion] Some Initial Comments on DDH-Spec-0.5h.pdf

Re: [Hardeneddrivers-discuss] RE: [cgl_discussion] Some Initial Comments on DDH-Spec-0.5h.pdf

[Jeff Garzik]

Gustafson, Geoffrey R wrote:
> Most of what makes a 'good' driver is common for all purposes - things you
> mention like don't make the system hang, don't cause fatal exceptions. But
> there are some things that would be different between a desktop, embedded
> system, enterprise server, or carrier server. For instance, when there is a
> tradeoff between reliability and performance; when reliability is king, it
> might be wise to do an insane amount of parameter checking to offset the
> merest chance of an undetected bug crashing a system.

This is not a valid example.  We do not make tradeoffs between 
performance and reliability.  Reliability _always_ comes first.  If it 
did not, it's a bug.


> Regarding the question: why not just fix the "bad" drivers? Drivers that are
> actually bad are probably for obscure hardware that is not really of
> concern. The purpose is to take good drivers and make sure they meet the
> last few percent of the objective standard of 'good'.

What exactly does this mean?  Can you give a concrete example of taking 
an existing driver and updating it for that last few percent?

[I venture to guess that Intel doesn't know yet what is necessary, but 
that's just a guess.]


> Another good point was about enforcement. Just because something is hardened
> at one point, doesn't necessarily mean some of the rules won't get
> accidentally violated by patches later on. So it either requires periodic
> reevaluation or strong buy-in from the respective maintainers. At least part
> of the beauty of open source is it _can_ be evaluated by an objective third
> party, if someone chose to do that.

It depends on what needs to be "enforced."  If its compliance to 
existing APIs, that's pretty much a given.  Non-compliance would be a bug.


> You asked several times for objective data, and I agree that would be great.
> However, drivers _are_ in the unique position of being both privileged code
> and yet specific to certain hardware. Thus they are capable of more damage
> than user-space code, but (on average) can't have been tested in as many
> configurations as core kernel code. So at least without data, they are a
> very logical starting point.

That's still not a concrete example :)  We already knew you were talking 
about drivers.

Does Intel even have --one-- example of a driver that could be hardened?

It seems to me Intel is writing a spec for an abstract objective, IOW 
writing a solution when the problem hasn't even been defined yet. 
Please define the problem ("<this> API needs updating to harden 
drivers") not the solution ("add <this> API and drivers will be hardened").

Finally, and this bears repeating -- I am very encouraged by Intel's 
participation, and initiative in hardening Linux drivers.  I actively 
encourage this effort, and think that Intel's resources can 
[potentially] dramatically improve the overall quality of Linux kernel 
drivers.  Guys, you have an excellent opportunity here ;-)  Please 
listen to the feedback from kernel developers, we are the guys in the 
trenches doing the Real Life software engineering day in and day out.

	Jeff

Re: [Hardeneddrivers-discuss] RE: [cgl_discussion] Some Initial Comments on DDH-Spec-0.5h.pdf

[Eric W. Biederman]

On the intel side I'm ready to ask for hardened hardware.
Maybe it's just the high volume but it feels like to use any piece
of intel hardware I half to use arounds for hardware bugs.

A lot of the problems feel like someone was doing a rush, job to meet
some marketing window.

My current favorite is the ICH3 I have my choice.  I can either monitor
Xeon temporatures via the i2c/smbus interface or I can reliably reset
the machine.

The useful point here is no amount of abstract talking about
hardening the hardware is going to reveal issues like these.  Only
testing and debugging.  And keeping both the hardware and the software
simple is an important piece of preventing bugs.


Jeff Garzik <jgarzik@pobox.com> writes:

> Gustafson, Geoffrey R wrote:
> > Most of what makes a 'good' driver is common for all purposes - things you
> > mention like don't make the system hang, don't cause fatal exceptions. But
> > there are some things that would be different between a desktop, embedded
> > system, enterprise server, or carrier server. For instance, when there is a
> > tradeoff between reliability and performance; when reliability is king, it
> > might be wise to do an insane amount of parameter checking to offset the
> > merest chance of an undetected bug crashing a system.
> 
> This is not a valid example.  We do not make tradeoffs between performance and
> reliability.  Reliability _always_ comes first.  If it did not, it's
> a bug.

Agreed.  

 
> > Regarding the question: why not just fix the "bad" drivers? Drivers that are
> > actually bad are probably for obscure hardware that is not really of
> > concern. The purpose is to take good drivers and make sure they meet the
> > last few percent of the objective standard of 'good'.

O.k. Then why not just fix the buggy hardware?
 
> What exactly does this mean?  Can you give a concrete example of taking an
> existing driver and updating it for that last few percent?

Does fixing intel hardware bugs count?
> 
> > You asked several times for objective data, and I agree that would be great.
> > However, drivers _are_ in the unique position of being both privileged code
> > and yet specific to certain hardware. Thus they are capable of more damage
> > than user-space code, but (on average) can't have been tested in as many
> > configurations as core kernel code. So at least without data, they are a
> > very logical starting point.

Oh, and don't forget that the hardware specification that drivers are
written to, many times are not generally available greatly reducing 
the pool of capable people who have the opportunity to review the and
debug the drivers.  I would make it a requirement for a hardened
driver that both the code and the hardware documentation be publicly
available so the code can easily be reviewed by as many people as wish
to.

> Finally, and this bears repeating -- I am very encouraged by Intel's
> participation, and initiative in hardening Linux drivers.  I actively encourage
> this effort, and think that Intel's resources can [potentially] dramatically
> improve the overall quality of Linux kernel drivers.  Guys, you have an
> excellent opportunity here ;-)  Please listen to the feedback from kernel
> developers, we are the guys in the trenches doing the Real Life software
> engineering day in and day out.

I will agree with that sentiment.  

Eric

Re: [Hardeneddrivers-discuss] RE: [cgl_discussion] Some Initial Comments on DDH-Spec-0.5h.pdf

[Jeff Garzik]

Eric W. Biederman wrote:
> Oh, and don't forget that the hardware specification that drivers are
> written to, many times are not generally available greatly reducing 
> the pool of capable people who have the opportunity to review the and
> debug the drivers.  I would make it a requirement for a hardened
> driver that both the code and the hardware documentation be publicly
> available so the code can easily be reviewed by as many people as wish
> to.


This is a good point that bears highlighting.  Donald Becker's [and thus 
the kernel's] eepro100.c had certain bugs for years, simply because 
access to Intel E100 hardware docs was damn near impossible to obtain.

I don't see driver hardening being very feasible on such drivers, where 
the vendor refuses to allow kernel engineers access needed to get their 
hardware working and stable.  [why vendors want crappy Linux support, 
I'll never know]

	Jeff


P.S.  In all fairness, Intel is doing a really good job maintaining the 
e100 and e1000 drivers nowadays, and e100 docs should be public very 
soon.  [e1000 docs? who knows...]

Re: [Hardeneddrivers-discuss] RE: [cgl_discussion] Some Initial Comments on DDH-Spec-0.5h.pdf

[Andre Hedrick]

On Tue, 24 Sep 2002, Jeff Garzik wrote:

> Eric W. Biederman wrote:
> > Oh, and don't forget that the hardware specification that drivers are
> > written to, many times are not generally available greatly reducing 
> > the pool of capable people who have the opportunity to review the and
> > debug the drivers.  I would make it a requirement for a hardened
> > driver that both the code and the hardware documentation be publicly
> > available so the code can easily be reviewed by as many people as wish
> > to.
> 
> 
> This is a good point that bears highlighting.  Donald Becker's [and thus 
> the kernel's] eepro100.c had certain bugs for years, simply because 
> access to Intel E100 hardware docs was damn near impossible to obtain.

Jeff, 

You know that every hardware vendor will clam it works well under
MicroSoft, so why does it fail under Linux.  This is the classic one-liner
we all have gotten.  The reality is closed software is used to hide all
the flaws and failures of made by the ASIC people.  I would love to shove
the brain dead asic designer of the original PIIX4 AB/EB off a cliff on
fire for being absolutely "stupid".  Sorry this is as nice an clean as I
can say this and not dust off the flame thrower.

> I don't see driver hardening being very feasible on such drivers, where 
> the vendor refuses to allow kernel engineers access needed to get their 
> hardware working and stable.  [why vendors want crappy Linux support, 
> I'll never know]

Worse is getting a spec that says, "no work around".
When the reality is the OEM hardware vendor will not take ownership of 
their errors and disclose a good proper work-around.

Cheers,

Andre Hedrick
LAD Storage Consulting Group

Re: [Hardeneddrivers-discuss] RE: [cgl_discussion] Some Initial Comments on DDH-Spec-0.5h.pdf

[Eric W. Biederman]

Andre Hedrick <andre@linux-ide.org> writes:


> Jeff, 
> 
> You know that every hardware vendor will clam it works well under
> MicroSoft, so why does it fail under Linux.  This is the classic one-liner
> we all have gotten.   The reality is closed software is used to hide all
> the flaws and failures of made by the ASIC people.  I would love to shove
> the brain dead asic designer of the original PIIX4 AB/EB off a cliff on
> fire for being absolutely "stupid".  Sorry this is as nice an clean as I
> can say this and not dust off the flame thrower.

Right.  So we need to get to a situation where ASIC designers are not afraid
to admit they messed up.  I have seen this from all vendors.  More than
anything this is the reason we have a documentation shortage.  Just when
we really need more documentation, and more code review to make certain
that the drivers are solid in the face of hardware bugs the vendors
stop talking.

As for ``It works well under windows so why does it fail under
Linux?'' line of arguing that is just a first reflex from people that
aren't used to dealing with Linux.  Putting it in a business case and
saying noting that the vendors can ship X million more in volume, or
become part of Y prestigious system.  People stop knee jerking and
start helping.

Getting those channels open through the business side takes time.  And
the more independent software developers don't always have access to
those kinds of arguments.  So we really need a way to shame a vendor
on a driver by driver basis that makes it worse to hide their
documentation than to admit to their bugs.  

Being able to say we could not ``harden'' the vendors driver because
the vendor did not give us the real specification, and errata
information, might be enough to shame change that.  If not we can try
other methods.

> > I don't see driver hardening being very feasible on such drivers, where 
> > the vendor refuses to allow kernel engineers access needed to get their 
> > hardware working and stable.  [why vendors want crappy Linux support, 
> > I'll never know]
> 
> Worse is getting a spec that says, "no work around".
> When the reality is the OEM hardware vendor will not take ownership of 
> their errors and disclose a good proper work-around.

If the vendor has not figured out a work around this is understandable
if undesirable.  

As for what can be done about it to get good Linux drivers, it is best
to remember that businesses do not have clear and consistent
policies.  Instead businesses are susceptible to the attack of many
pokes, and enticements by people waving cash.  So by pure persistence
and repetition we should be able to get the word out.

We just need to come up with arguments that are just as persistent as
the ip lawyers who say you need secret ``ip''.   And some
embarrassement that is stronger than the embarrassement at the quality
of their work.


Eric

Re: [Hardeneddrivers-discuss] RE: [cgl_discussion] Some Initial Comments on DDH-Spec-0.5h.pdf

[Pavel Machek]

Hi!

> > Most of what makes a 'good' driver is common for all purposes - things you
> > mention like don't make the system hang, don't cause fatal exceptions. But
> > there are some things that would be different between a desktop, embedded
> > system, enterprise server, or carrier server. For instance, when there is a
> > tradeoff between reliability and performance; when reliability is king, it
> > might be wise to do an insane amount of parameter checking to offset the
> > merest chance of an undetected bug crashing a system.
> 
> This is not a valid example.  We do not make tradeoffs between 
> performance and reliability.  Reliability _always_ comes first.  If it 
> did not, it's a bug.

No. We do run all drivers in *one* addressspace. That's bad for reliability.

-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.