Re: [MAINTAINER SUMMIT] User-space requirements for accelerator drivers

[Arnd Bergmann <arnd@arndb.de>]

On Tue, Sep 14, 2021 at 11:23 AM Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> On Tue, Sep 14, 2021 at 11:09 AM Arnd Bergmann <arnd@arndb.de> wrote:

> > I can see two reasons why one would want to support a type (a)
> > interface even with the more versatile devices:
> >
> > - It can be done in a generic way so that simply adding a kernel
> >   driver and loading some firmware into it makes existing user space
> >   software work out of the box.
> >
> > - It gives the manufacturer a way to get an upstream kernel driver
> >   without open sourcing their firmware (a.k.a. compiler and user
> >   space driver). Whether you consider this a good or bad thing is
> >   of course a matter of perspective.
>
> I think for some embedded use-case this makes sense, especially around
> media stuff.
>
> I don't think it's BLAS, because on the compute side you really want a
> compiler that sees through the entire thing and can optimize it. Afaik
> BLAS is for some quick prototype of matrix algorithms and most
> importantly, for the top500 list :-)

It's probably not the only thing you need, but I would assume something
like sgemm and its variants are one of the building blocks you'd need
in this kind of interface. Note that oneDNN also comes with a
simplified interface similar to gemm[1] as well as straight wrapper around
gemm itself.

There are definitely frameworks that are successfully built just on top
of NumPy and blas (with NumPy itself being built on top of blas).
I used to make fun of linpack as the supercomputer benchmark that
has no practical use, but in the end it does spend most of its time in
the SGEMM function that is the most optimized algorithm in the world
and that is also where you end up spending your cycles in many AI
applications. I found a link to this blog post[2] explaining why this is still
used everywhere, and this matches what I've seen elsewhere, but
unlike me, the author seems to know what they are talking about ;-)

To get back to my own question from earlier about which part of oneAPI
is actually being used, I see that pytorch (to pick a common framework)
can use either mkl (oneMKL, BLAS) or mkldnn (dnnl, oneDNN) as a backend,
next to cuda, cudnn, openmp and certainly a number of third-party
backends.

The mkl backend seems to mostly be a wrapper around cblas_*gemm(),
though I may be reading that wrong.
The oneDNN backend operates on a higher level, calling into a
subset of the oneDNN interfaces. The other frameworks I looked at
(mxnet, tensorflow) look similar, probably each using other subsets of
oneDNN.

> > > We have lots of fixed function on GPUs, video codecs are on most x86
> > > GPUs. It's how you program them that matters, most of them are behind
> > > queues similar to the 3D engine, so you program them the same way.
> >
> > So these would go through /dev/dri instead of /dev/media0? I can definitely
> > see a lot of codec drivers in the kernel that use a /dev/media interfaces,
> > and the tradeoffs between those two seem very similar to the tradeoffs
> > you get for machine learning accelerators.
>
> Yeah we have plenty of codes running on top of /dev/dri0, with all the
> magic in userspace.
>
> They are all very far away from anything that is a machine learning accelerator.

Sure, I only meant the relation between dri codecs and media codecs
is similar to the relation between the ways one can implement the AI
accelerator APIs.

> Yeah for those I think a more fixed uapi like drivers/media has a lot
> of makes sense. What I don't like is when vendors then use that excuse
> of "oh you only upload a fixed model at boot" to shovel in an acccel
> driver with full generic interface, but not all the userspace
> bits&pieces. There's unfortunately another accel driver in
> drivers/misc for qualcom soc, which really should be either a media
> driver (for the fixed function use-case) or a drm driver (for the
> fully programmable) use-case.

I would argue that for the fixed-function use case, the media subsystem
isn't a great fit either. It would probably work just as well (as would the
crypto subsystem), but having a distinct interface that does just
one thing makes more sense conceptually, if only to make it clear
where to look for such drivers and to have a consistent interface
documentation.

> I think for the fixed-function interface case you can also make a
> reasonable argument that just documenting that fixed interface and all
> the parameters is good enough. But as soon as the interface becomes a
> generic "submit workload" style thing because you want to make it work
> for an entire set of "firmware" compiled by your closed stack, that's
> out of the window.

Right, agreed. If we add a fixed-function interface, that should ideally
not allow any vendor specific extensions at all, just a set of well-defined
operations, and certainly not a bypass mode that gets used to
send compiled binaries.

       Arnd

[1] https://oneapi-src.github.io/oneDNN/dev_guide_matmul.html
[1] https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/