Re: [PATCHv12 1/3] rdmacg: Added rdma cgroup controller

From: Matan Barak <matanb@mellanox.com>
To: Jason Gunthorpe <jgunthorpe@obsidianresearch.com>,
	Christoph Hellwig <hch@lst.de>
Cc: Parav Pandit <pandit.parav@gmail.com>, Tejun Heo <tj@kernel.org>,
	<cgroups@vger.kernel.org>, <linux-doc@vger.kernel.org>,
	"Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
	<linux-rdma@vger.kernel.org>, Li Zefan <lizefan@huawei.com>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Doug Ledford <dledford@redhat.com>,
	Liran Liss <liranl@mellanox.com>,
	"Hefty, Sean" <sean.hefty@intel.com>,
	Haggai Eran <haggaie@mellanox.com>,
	Jonathan Corbet <corbet@lwn.net>, <james.l.morris@oracle.com>,
	<serge@hallyn.com>, Or Gerlitz <ogerlitz@mellanox.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	<linux-security-module@vger.kernel.org>
Subject: Re: [PATCHv12 1/3] rdmacg: Added rdma cgroup controller
Date: Sun, 11 Sep 2016 11:07:57 +0300	[thread overview]
Message-ID: <ddbb4d5a-c352-925b-c3db-4f33f3e8ac50@mellanox.com> (raw)
In-Reply-To: <20160910170151.GA5230@obsidianresearch.com>

On 10/09/2016 20:01, Jason Gunthorpe wrote:
> On Sat, Sep 10, 2016 at 06:14:42PM +0200, Christoph Hellwig wrote:
>> OFVWG meetings have absolutely zero relevance for Linux development.
>
> Well, to be fair there are a fair number of kernel developers on that
> particular call..
>
>> More "flexibility" for drivers just means giving up on designing a
>> coherent API and leaving it to drivers authors to add crap to their
>> own drivers.  That's a major step backwards.
>
> Sadly, it isn't a step backwards, it is status quo - at least as far
> as the uapi is concerned.
>
> Every single user space driver has its own private abi file, carefully
> hidden in their driver, and dutifully copied over to user space:
>
> providers/cxgb3/iwch-abi.h
> providers/cxgb4/cxgb4-abi.h
> providers/hfi1verbs/hfi-abi.h
> providers/i40iw/i40iw-abi.h
> providers/ipathverbs/ipath-abi.h
> providers/mlx4/mlx4-abi.h
> providers/mlx5/mlx5-abi.h
> providers/mthca/mthca-abi.h
> providers/nes/nes-abi.h
> providers/ocrdma/ocrdma_abi.h
> providers/rxe/rxe-abi.h
>
> Just to pick two random examples:
>
> struct mlx5_create_cq {
>         struct ibv_create_cq            ibv_cmd;
>         __u64                           buf_addr;
>         __u64                           db_addr;
>         __u32				cqe_size;
> };
>
> struct iwch_create_cq {
>         struct ibv_create_cq ibv_cmd;
>         uint64_t user_rptr_addr;
> };
>
> Love to hear ideas on a way forward that doesn't involve rewriting
> everything :(
>

Yeah, unfortunately, the RDMA ABI is more driver specific ABI than a 
common user-kernel ABI. I guess this will become even worse, as the RDMA 
subsystem is evolving to serve more drivers with different object types. 
For example, I would like to hear how hfi1 are going to define their 
user-kernel ABI (once they leave the custom ioctls).

>> They should not be using the code in drivers/infiniband.  usnic is such
>> an example of a driver that should never have been added in it's current
>> form.
>
> +1
>
> Jason
>

Matan