From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC891C282C0 for ; Fri, 25 Jan 2019 15:02:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 867F0218A2 for ; Fri, 25 Jan 2019 15:02:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lixom-net.20150623.gappssmtp.com header.i=@lixom-net.20150623.gappssmtp.com header.b="cvxmoNRy" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726951AbfAYPCT (ORCPT ); Fri, 25 Jan 2019 10:02:19 -0500 Received: from mail-it1-f195.google.com ([209.85.166.195]:51119 "EHLO mail-it1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726265AbfAYPCT (ORCPT ); Fri, 25 Jan 2019 10:02:19 -0500 Received: by mail-it1-f195.google.com with SMTP id z7so10948004iti.0 for ; Fri, 25 Jan 2019 07:02:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lixom-net.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=1SBZv4y3/ZPU3Ilvzh5pc/GSNraEXdCcqSCA+bPGcpE=; b=cvxmoNRyWAQoFeN9tZu77sjcx68FADuuc/X5KZ0Op1H1uC7l+vGY8RSmEJvMI7QDnv yDGagvmZaSbjHTFEOCk0msWGg7dIrOrNFWURzOEAb36J8NPTkDXpDfQYYOLr8am5QIk9 zi+WCqMfwhvQHeVtn7HrgQsAtOdZHVcjo3P8MdaFXRVeVtULfw3t8C0MPwFLuvh8jROX Ka5k2znsWeVZrGMY8X5nO3AzxPZ+E8m0rltNaq3ch47kPx6R5IvppuHOU0SSPLRVzwl0 0sf0/HzPy38S78mdelG0X7olTWhQ8DfotB9RZptWVL+dYrTjY17lrUfMNFnMmxxAqTfX flnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=1SBZv4y3/ZPU3Ilvzh5pc/GSNraEXdCcqSCA+bPGcpE=; b=WBHLv0pTu/Fi5mqxKBerWxgypviznuI1rL2SbYjalRjNwo2728gnhnzS+mxvUjtfy3 AEtu5/jhUFPbuFE5nCzNMwAG10plruQ4rD8GjG8HXeEAO3spRsp/7qzzdTdJJCHlpxYs 2EomX8LGDTrbesvk3THajPG+3SADlkD3u6I2KGMi2ydlnFHlPA8LdiYNJvnVhsRL6WC4 A/nvMlk4zn4mkGW+OA2ev+h735SVqmgX2HUHwIXlKARnGTgzYonJtNkFK4qoZjujljF/ jKxcvJeYE+GyA46BlH8Ev20W4l9odON86cY11lfRGlloiiGsUbwpS5kXW8S41DMwII3W an/w== X-Gm-Message-State: AJcUukchS8KhDBfWCJeF9mf8AhB58wuB3g0HqLQjtmWScJCuYiWKFMu8 E92l8zPX+/Ri2x4O5wMVmlp5QmRcw/A7WOZnbU+PiA== X-Google-Smtp-Source: AHgI3IZP3BUkGSMIMvJrUzNhUO5WyamPDhblv9hB4Fr2yyq7Nwim/1PrHS5Y54zKLW5+xm83rMRLCUuRnfP8pbK/xcY= X-Received: by 2002:a24:ce42:: with SMTP id v63mr4249589itg.136.1548428537586; Fri, 25 Jan 2019 07:02:17 -0800 (PST) MIME-Version: 1.0 References: <20190123000057.31477-1-oded.gabbay@gmail.com> <20190123232052.GD1257@redhat.com> <20190123234817.GE1257@redhat.com> In-Reply-To: From: Olof Johansson Date: Fri, 25 Jan 2019 07:02:06 -0800 Message-ID: Subject: Re: [PATCH 00/15] Habana Labs kernel driver To: Daniel Vetter Cc: Dave Airlie , Oded Gabbay , Jerome Glisse , Greg Kroah-Hartman , LKML , ogabbay@habana.ai, Arnd Bergmann , fbarrat@linux.ibm.com, Andrew Donnellan Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jan 24, 2019 at 11:43 PM Daniel Vetter wrote: > > On Fri, Jan 25, 2019 at 1:14 AM Olof Johansson wrote: > > > > On Thu, Jan 24, 2019 at 2:23 AM Dave Airlie wrote: > > > > > > > I know I won't be able to convince you but I want to say that I think > > > > your arguments for full userspace open source are not really > > > > technical. > > > > > > There is more to keeping a kernel going than technical argument unfortunately. > > > > > > I guess the question for Greg, Olof etc, is do we care about Linux the > > > kernel, or Linux the open source ecosystem, if the former, these sort > > > of accelerator shim drivers are fine, useless to anyone who doesn't > > > have all the magic hidden userspace, and impossible to support for > > > anyone else, if the latter, we should leave the cost of maintenance to > > > the company benefiting from it and leave maintaining it out of tree. > > > > As mentioned in my reply to Daniel, I think we've got a history of > > being pragmatic and finding reasonable trade-offs of what can be open > > and what can be closed. For example, if truly care about open source > > ecosystem, drivers that require closed firmware should also be > > refused. > > Firmware has traditionally been different since usually it's looked > down, doesn't do much wrt functionality (dumb fifo scheduling at best, > not really power management) and so could be reasonably shrugged off > as "it's part of hw". If you care about the open graphics ecosystem, > i.e. your ability to port the stack to new cpu architectures, new > window systems (e.g. android -> xorg, or xorg -> android, or something > entirely new like wayland), new, more efficient client interface > (vulkan is a very new fad), then having a closed firmware is not going > to be a problem. Closed compiler, closed runtime, closed anything else > otoh is a serious practical pain. > > Unfortunately hw vendors seem to have realized that we (overall > community of customers, distro, upstream) are not insisting on open > firmware, so they're moving a lot of "valuable sauce" (no really, it's > not) into the firmware. PM governors, cpu scheduling algorithms, that > kind of stuff. We're not pleased, and there's lots of people doing the > behind the scenes work to fix it. One practical problem is that even > if we've demonstrated that r/e'ing a uc is no bigger challenge than > anything, there's usually this pesky issue with signatures. So we > can't force the vendors like we can with the userspace side. Otherwise > nouveau would have completely open firmware even for latest chips > (like it has for olders). > > > > Simple question like If I plug your accelerator into Power or ARM64, > > > where do I get the port of your userspace to use it? > > > > Does demanding complete open userspace get us closer to that goal in > > reality? By refusing to work with people to enable their hardware, > > they will still ship their platforms out of tree, using DKMS and all > > the other ways of getting kernel modules installed to talk to the > > hardware. And we'd be no closer. > > > > In the end, they'd open up their userspace when there's business > > reasons to do so. It's well-known how to work around refusal from us > > to merge drivers by now, so it's not much leverage in that area. > > Correct. None of the hw vendors had a business reason to open source > anything unforunately. Yes, eventually customers started demanding > open source and treatening to buy the competition, but this only works > if you have multiple reasonably performant & conformant stacks for > different vendors. The only way to get these is to reverse engineer > them. That's the grass-roots version of it, and it is indeed a lot of work. What _has_ proven to have success is when companies that would drive real revenue for the vendors have requirements for them to open up, contribute, and participate. In the graphics world it hasn't gotten things all the way to the right spot, but I know first hand that for example Chrome OS's insistence on upstream participation has made significant differences in how several companies interact with our communities. It's not something that's easy to do when the target is consumer-oriented hardware (which graphics mostly is), but at least for now, these accelerators aren't targeting end users as much as corporate environments, where we do have allies. > Now reverse-engineering is a major pain in itself (despite all the > great tooling gpu folks developed over the past 10 years to convert it > from a black art to a repeatable engineering excercise), but if you > additionally prefer the vendors closed stack (which you do by allowing > to get them to get merged) the r/e'd stack has no chance. And there is > not other way to get your open source stack. I can't really go into > all the details of the past 15+ of open source gpus, but without the > pressure of other r/e'ed stacks and the pressure of having stacks for > competitiors (all made possible through aggressive code sharing) we > would have 0 open source gfx stacks. All the ones we have either got > started with r/e first (and eventually the vendor jumped on board) or > survived through r/e and customer efforts (because the vendor planned > to abandon it). Another part of this is that we accept userspace only > when it's the common upstream (if there is one), to prevent vendors > closing down their stacks gradually. > > So yeah I think by not clearly preferring open source over > stacks-with-blobs (how radically you do that is a bit a balance act in > the end, I think we've maxed out in drivers/gpu on what's practically > possible) you'll just make sure that there's never going to be a > serious open source stack. I can confidently say that I would myself clearly give preferential treatment to open stacks when they show up. The trick is how we get there -- do we get there quicker by refusing to work with the partially closed stacks? My viewpoint is that we don't, and that the best way to get there is to bring them in and start working with what we have instead of building separate camps that we later need to figure out how to move. > > > I'm not the final arbiter on this sort of thing, but I'm definitely > > > going to make sure that anyone who lands this code is explicit in > > > ignoring any experience we've had in this area and in the future will > > > gladly accept "I told you so" :-) > > > > There's only one final arbiter on any inclusion to code to the kernel, > > but we tend to sort out most disagreements without going all the way > > there. > > > > I still think engaging has a better chance of success than rejecting > > the contributions, especially with clear expectations w.r.t. continued > > engagement and no second implementations over time. In all honestly, > > either approach might fail miserably. > > This is maybe not clear, but we still work together with the blob > folks as much as possible, for demonstration: nvidia sponsored XDC > this year, and nvidia engineers have been regularly presenting there. > Collaboration happens around the driver interfaces, like loaders (in > userspace), buffer sharing, synchronization, negotiation of buffer > formats and all that stuff. Do as much enganging as possible, but if > you give preferrential treatment to the closed stacks over the open > ones (and by default the vendor _always_ gives you a closed stack, or > as closed as possible, there's just no business case for them to open > up without a customer demanding it and competition providing it too), > you will end up with a closed stack for a very long time, maybe > forever. > > Even if you insist on an open stack it's going to take years, since > the only way to get there is lots of r/e, and you need to have at > least 2 stacks or otherwise the customers can't walk away from the > negotiation table. So again from gfx experience: The only way to get > open stacks is solid competition by open stacks, and customers/distros > investing ridiculous amounts of money to r/e the chips and write these > open&cross vendor stacks. The business case for vendors to open source > their stacks is just not there. Not until they can't sell their chips > any other way anymore (nvidia will embrace open stacks as soon as > their margins evaporate, not a second earlier, like all the others > before them). Maybe at the next hallway track we need to go through a > few examples of what all happened and is still happening in the > background (here's maybe not a good idea). Again, the graphics world is different since the volume market has traditionally been consumers, and the split got very deep. What we want to avoid here is to get into the same situation by avoiding the large split. Look at it another way, these are roughly the options and possible outcomes: 1a. We don't merge these drivers, vendors say "okay then" and open up their whole stacks. We merge the drivers. Then we start working together on moving to a common base. 1b. We don't merge these drivers, the vendors all do their own thing and over the next 5 years, we reverse engineer and start to bring in second implementations of all their code. 2a. We merge these drivers, start close engagement with the vendors, collaborate and converge with their next-gen products (possibly move first-gen over). 2b. We merge these drivers, and vendors still go off on their own and do their own thing. We spend the next 5 years reverse engineering and move over to open, new drivers even though the first ones are in-tree. 1a/2a are successful outcomes. I put 1a at very very low probability. 2b at medium probability with the right allies in the corporate world. 1b/2b are partial failure modes with huge effort needed and a passionate volunteer base. Both 1a/2a and 1b/2b have similar amounts of work involved. In other words, I don't see how anyone benefits from eliminating (2) as an approach, especially since 1a is an unlikely development of events. And, guess what -- if we do get open stacks early on, and give heavy preferential treatment to these, we can push others in that direction sooner rather than later, before stacks diverge too far. I just don't see how _not_ engaging is going to help here. Even if we do engage, the worst possible outcome is still the same as not engaging but a good chance for something better. -Olof