On Wed, Sep 21, 2016 at 11:23:41AM +0200, gregkh@linuxfoundation.org wrote:
> On Wed, Sep 21, 2016 at 02:58:22PM +0800, Alex Shi wrote:

> > I am not a fun of some scheduler solution. But focus on this can not
> > explain why many distributions are using 'old' stable kernel. Looking
> > into product world, could you find some real product are using
> > 'upstream' kernel?

> > 'upstream first' is good for feature development, but isn't good for
> > product.

> Not true, IBM and Intel have shown that it saves you time and money to
> do "upstream first", why do people claim that their reports of this is
> somehow false?  Other companies also agree, but just don't want to take
> the initial "hit" of time to do it correctly as it will affect the
> device today to save time and money for the device tomorrow.

One problem that bites people fairly hard trying to do upstream first is
that upstream turns too slowly to do things for the current product in
some markets, when I was working on phones it'd off the top of my head
typically be 3-4 months before anything ended up in a release (depending
on where we were in the release cycle) and another year or so before
that filtered back out into product (and this is mostly with very low
resistance subsystems where I'm the maintainer).  This doesn't play very
nicely when your total product development lifecycle is on the order of
six months, even when you scale down to individual reviews of patches
the latencies involved just don't fit that well.  Upstream is
simultaneously very fast moving (when you look at the overall change
volume flowing in) and very slow moving (when you look at some
individual changes).

When I talk to people about this I tend to talk about doing upstream
simultaneously rather than first, that is a lot more tractable.  Work on
things, try to push your latest work upstream as you write it and
incorporate review feedback from upstream into your product as you go
but don't gate things on upstream.  Try to view anything that's still in
review as a problem that needs to be fixed but acknowledge that there's
also a need to actually get product out the door.

> > Many product guys talked to me that the non-upstream porting didn't
> > cost much and not the reason to pin on some stable kernel.

> You must be talking to product people who only have to make one device,
> not a family of devices :)

What Alex is seeing reflects my experience talking to people as well.
It's not like anyone is saying this is free but it's a thing that people
have been doing for a while, figured out and have incorporated into
their planning - it's managable and reasonably well understood even if
not super productive.

> > All of them said that testing and stability was the most cost part.

> Sure, software is always free, it's that pesky testing and fixing all of
> the bugs found that costs money :)
> (hint, all of those backports and non-upstrem stuff is what is causing
> lots of those bugs...)

All code has problems, it's not like we've got a silver bullet here -
let's not pretend that we're in a position where upstream is shippable
on all systems or where we never have any performance regressions.

> > Not only the regular test case, benchmarks, but also the long time
> > using for some trick/corner case bugs in whole system.

> What do you mean by this?

I think Alex is referring to the detailed full system testing that
people do, I could be wrong though.

> > I doubt the 'keep rebasing on upstream' guys have been really worked on
> > product?

> I doubt those "let's not work upstream" have been in this business for
> as long as those of us who say "work upstrem first" have :)

I have worked in this environment at various levels all the way up to
being sat on site with major consumer electronics companies pushing
patches into market leading products (and onto the lists).  I have
talked and continue to talk to colleagues in this space who have varying
degrees of engagement with upstream.

> Fine, you can ignore us, but realize that it will cost you time and
> money to _not_ work upstream.  We are just trying to help you out...

People aren't just ignoring the idea of working upstream entirely.
People are doing work here, it's producing results (or at least a pile
of mainline code).  Some are doing more than others, some are more
successful than others but there is a lot happening here.  There's a lot
of things going on that I'm critical of but there's also a lot of
perfectly rational, practical decisions being made.  Getting the more
complex consumer electronics devices to the point where they work as
well upstream as servers do is not trivial and will take some time.

Simply repeating "upstream first" over and over again and telling people
that doing anything else is just silly isn't really helping move things
forward.  People have heard this but for a good chunk of the industry
there's a big gap between that simple statement and something that can
be practically acted on in any sort of direct fashion, it can easily
just come over as dismissive and hostile.  It's going to be much more
productive to acknowledge the realities people are dealing with and talk
about how people can improve their engagement with upstream, make the
situation better and close the gaps.