[ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Liran Liss]

Following the recent discussions on unifying RDMA user-space, it looks like it is time to talk about a process to make it happen.
Let's dedicate this week's OFVWG slot for this.

Agenda items:
- Repositories
- Code submission and mailing lists
- Release process
- Schedules
- Other?

Thanks,
--Liran

Concall details
===========
JOIN WEBEX MEETING
https://mellanox-tech.webex.com/join/Liran.Liss  |  702 830 424

JOIN BY PHONE
1-80-9453550 Israel toll free
1-855-299-5224 USA/Canada toll free
Access code: 702 830 424

Global call-in numbers:
https://mellanox-tech.webex.com/mellanox-tech/globalcallin.php?serviceType=MC&ED=457637257&tollFree=1

Toll-free calling restrictions:
http://www.webex.com/pdf/tollfree_restrictions.pdf

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Parav Pandit]

Hi Liran,

We are almost reaching conclusions with Tejun, Christoph, Matan, Leon,
Jason and me with respect to rdma cgroup patches.

There were open question towards HFI driver about how it plans to use
new uverbs API.
If time permits can we discuss this so that we can close on rdma
cgroup patchset?
I haven't been active in OFVWG meeting for uapi yet, but I will also
join this call.

Regards,
Parav Pandit


On Mon, Sep 12, 2016 at 8:34 PM, Liran Liss <liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> Following the recent discussions on unifying RDMA user-space, it looks like it is time to talk about a process to make it happen.
> Let's dedicate this week's OFVWG slot for this.
>
> Agenda items:
> - Repositories
> - Code submission and mailing lists
> - Release process
> - Schedules
> - Other?
>
> Thanks,
> --Liran
>
> Concall details
> ===========
> JOIN WEBEX MEETING
> https://mellanox-tech.webex.com/join/Liran.Liss  |  702 830 424
>
> JOIN BY PHONE
> 1-80-9453550 Israel toll free
> 1-855-299-5224 USA/Canada toll free
> Access code: 702 830 424
>
> Global call-in numbers:
> https://mellanox-tech.webex.com/mellanox-tech/globalcallin.php?serviceType=MC&ED=457637257&tollFree=1
>
> Toll-free calling restrictions:
> http://www.webex.com/pdf/tollfree_restrictions.pdf
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Liran Liss]

> From: Parav Pandit [mailto:pandit.parav@gmail.com]

> Hi Liran,
> 
> We are almost reaching conclusions with Tejun, Christoph, Matan, Leon, Jason
> and me with respect to rdma cgroup patches.
> 
> There were open question towards HFI driver about how it plans to use new
> uverbs API.
> If time permits can we discuss this so that we can close on rdma cgroup
> patchset?

Sure.

> I haven't been active in OFVWG meeting for uapi yet, but I will also join this call.
> 

Welcome aboard! :)

> Regards,
> Parav Pandit

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2065 bytes --]

On Mon, Sep 12, 2016 at 09:31:04PM +0530, Parav Pandit wrote:
> Hi Liran,
>
> We are almost reaching conclusions with Tejun, Christoph, Matan, Leon,
> Jason and me with respect to rdma cgroup patches.

Interesting view on the situation :)
Feel free to join and participate.

If time doesn't permit us, we will talk next week about
these patches in regards to ABI.

>
> There were open question towards HFI driver about how it plans to use
> new uverbs API.
> If time permits can we discuss this so that we can close on rdma
> cgroup patchset?
> I haven't been active in OFVWG meeting for uapi yet, but I will also
> join this call.
>
> Regards,
> Parav Pandit
>
>
> On Mon, Sep 12, 2016 at 8:34 PM, Liran Liss <liranl-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org> wrote:
> > Following the recent discussions on unifying RDMA user-space, it looks like it is time to talk about a process to make it happen.
> > Let's dedicate this week's OFVWG slot for this.
> >
> > Agenda items:
> > - Repositories
> > - Code submission and mailing lists
> > - Release process
> > - Schedules
> > - Other?
> >
> > Thanks,
> > --Liran
> >
> > Concall details
> > ===========
> > JOIN WEBEX MEETING
> > https://mellanox-tech.webex.com/join/Liran.Liss  |  702 830 424
> >
> > JOIN BY PHONE
> > 1-80-9453550 Israel toll free
> > 1-855-299-5224 USA/Canada toll free
> > Access code: 702 830 424
> >
> > Global call-in numbers:
> > https://mellanox-tech.webex.com/mellanox-tech/globalcallin.php?serviceType=MC&ED=457637257&tollFree=1
> >
> > Toll-free calling restrictions:
> > http://www.webex.com/pdf/tollfree_restrictions.pdf
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Christoph Hellwig]

On Mon, Sep 12, 2016 at 09:31:04PM +0530, Parav Pandit wrote:
> I haven't been active in OFVWG meeting for uapi yet, but I will also
> join this call.

Please keep your discussions to the mailing list.  The phone call is a
total nightmare for keeping the coordination in one place.  Getting
right of it would be really helpful for improving the RDMA workflow and
adjusting it to the process used in the rest of the kernel.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Parav Pandit]

Hi Christoph,

On Tue, Sep 13, 2016 at 12:18 PM, Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> wrote:
> On Mon, Sep 12, 2016 at 09:31:04PM +0530, Parav Pandit wrote:
>> I haven't been active in OFVWG meeting for uapi yet, but I will also
>> join this call.
>
> Please keep your discussions to the mailing list.  The phone call is a
> total nightmare for keeping the coordination in one place.  Getting
> right of it would be really helpful for improving the RDMA workflow and
> adjusting it to the process used in the rest of the kernel.

Sure. I just want to remind HFI folks to give their comments. There
has been no response from them to Matan's question.

I am agreement with your point that most RDMA objects will have common
structure at the start and thats sufficient for cgroup to function
correctly. There hasn't been any disagreement yet on it.
So I am believing that we are good for now. Or I am missing something?
I will continue to have mailing list as one point for review and
discussion/feedback.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Christoph Hellwig]

On Tue, Sep 13, 2016 at 12:31:22PM +0530, Parav Pandit wrote:
> Sure. I just want to remind HFI folks to give their comments. There
> has been no response from them to Matan's question.
> 
> I am agreement with your point that most RDMA objects will have common
> structure at the start and thats sufficient for cgroup to function
> correctly. There hasn't been any disagreement yet on it.
> So I am believing that we are good for now. Or I am missing something?
> I will continue to have mailing list as one point for review and
> discussion/feedback.

I think we're fine now.  The non-verbs HFSI interfaces aren't really a
fit for our inifiband/rdma (it should really be called ibverbs I guess)
interface.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Christoph Hellwig]

Hi Liran,

please stop adding subscribers-only mailinglist to linux-rdma CC's.
Thank you!
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Parav Pandit]

On Tue, Sep 13, 2016 at 12:34 PM, Christoph Hellwig <hch-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> wrote:
> On Tue, Sep 13, 2016 at 12:31:22PM +0530, Parav Pandit wrote:
>> Sure. I just want to remind HFI folks to give their comments. There
>> has been no response from them to Matan's question.
>>
>> I am agreement with your point that most RDMA objects will have common
>> structure at the start and thats sufficient for cgroup to function
>> correctly. There hasn't been any disagreement yet on it.
>> So I am believing that we are good for now. Or I am missing something?
>> I will continue to have mailing list as one point for review and
>> discussion/feedback.
>
> I think we're fine now.  The non-verbs HFSI interfaces aren't really a
> fit for our inifiband/rdma (it should really be called ibverbs I guess)
> interface.

ok. Current patch should be able to manage rdma devices of below drivers.
cxgb3/4, i40iw, ib_rxe, mlx4/5, ocrdma, qed.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Tue, Sep 13, 2016 at 12:04:18AM -0700, Christoph Hellwig wrote:
> On Tue, Sep 13, 2016 at 12:31:22PM +0530, Parav Pandit wrote:
> > Sure. I just want to remind HFI folks to give their comments. There
> > has been no response from them to Matan's question.
> >
> > I am agreement with your point that most RDMA objects will have common
> > structure at the start and thats sufficient for cgroup to function
> > correctly. There hasn't been any disagreement yet on it.
> > So I am believing that we are good for now. Or I am missing something?
> > I will continue to have mailing list as one point for review and
> > discussion/feedback.
>
> I think we're fine now.  The non-verbs HFSI interfaces aren't really a
> fit for our inifiband/rdma (it should really be called ibverbs I guess)
> interface.

Out of curiosity, can you point me to the discussions where you
expressed your objections while hfi1 was added to drivers/inifiniband?

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Christoph Hellwig]

On Tue, Sep 13, 2016 at 01:32:38PM +0300, Leon Romanovsky wrote:
> > I think we're fine now.  The non-verbs HFSI interfaces aren't really a
> > fit for our inifiband/rdma (it should really be called ibverbs I guess)
> > interface.
> 
> Out of curiosity, can you point me to the discussions where you
> expressed your objections while hfi1 was added to drivers/inifiniband?

No idea what you mean.  I'm fine with HFI living in drivers/infiniband,
although other places for it might not be wrong either.

I'm not fine with HFI exposing the non-verbs interfaces through
drivers/infiniband/core interface, which as far as I can tell it
currently does not even do, although some discussions here suggest
that a future interface should cater to those interface.

The real mistake was adding usnic, which has not business in
drivers/infiniband/ at all.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

On Wed, Sep 14, 2016 at 05:30:59AM -0700, Christoph Hellwig wrote:
> On Tue, Sep 13, 2016 at 01:32:38PM +0300, Leon Romanovsky wrote:
> > > I think we're fine now.  The non-verbs HFSI interfaces aren't really a
> > > fit for our inifiband/rdma (it should really be called ibverbs I guess)
> > > interface.
> >
> > Out of curiosity, can you point me to the discussions where you
> > expressed your objections while hfi1 was added to drivers/inifiniband?
>
> No idea what you mean.  I'm fine with HFI living in drivers/infiniband,
> although other places for it might not be wrong either.
>
> I'm not fine with HFI exposing the non-verbs interfaces through
> drivers/infiniband/core interface, which as far as I can tell it
> currently does not even do, although some discussions here suggest
> that a future interface should cater to those interface.

I see your point. Latest discussionis or more correct lack of interest
from Intel folks for proposed ABI suggest that it is best to limit this
ABI to verbs objects only. In such way, memcg will fit natively into
the stack.

>
> The real mistake was adding usnic, which has not business in
> drivers/infiniband/ at all.



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Jason Gunthorpe]

On Wed, Sep 14, 2016 at 05:30:59AM -0700, Christoph Hellwig wrote:

> I'm not fine with HFI exposing the non-verbs interfaces through
> drivers/infiniband/core interface, which as far as I can tell it
> currently does not even do, although some discussions here suggest
> that a future interface should cater to those interface.

The counter balance is that the Kernel Community has had a lot of bad
experiences with drivers growing their own user APIs and 'subsystem'
like functionality.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

On Wed, Sep 14, 2016 at 11:13:49AM -0600, Jason Gunthorpe wrote:
> On Wed, Sep 14, 2016 at 05:30:59AM -0700, Christoph Hellwig wrote:
>
> > I'm not fine with HFI exposing the non-verbs interfaces through
> > drivers/infiniband/core interface, which as far as I can tell it
> > currently does not even do, although some discussions here suggest
> > that a future interface should cater to those interface.
>
> The counter balance is that the Kernel Community has had a lot of bad
> experiences with drivers growing their own user APIs and 'subsystem'
> like functionality.

Do you see any desire of HFI authors to move to new general user API?

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Christoph Hellwig]

On Thu, Sep 15, 2016 at 08:00:05AM +0300, Leon Romanovsky wrote:
> Do you see any desire of HFI authors to move to new general user API?

A general API requires something general to be implemented first.
That's usually been the problem with driver specific APIs..

Once in a while people try to implement generic concepts drivers
specific, in which case it's easy to redirect them to a generic
subsystem.  It's been a while since I look at HFI, but IIRC only
the eeprom bits feel into such an existing categeory.  In theory
anything PSM specific could fit into a PSM subsystem, but without
anyone else even looking into that it's hard to define a subsystem
for it.  And there's just lots of weird stuff which we'd either
need to reject entirely (too late for HFI now) or live with a driver
specific API.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [Ofvwg] [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Weiny, Ira]

> 
> On Wed, Sep 14, 2016 at 11:13:49AM -0600, Jason Gunthorpe wrote:
> > On Wed, Sep 14, 2016 at 05:30:59AM -0700, Christoph Hellwig wrote:
> >
> > > I'm not fine with HFI exposing the non-verbs interfaces through
> > > drivers/infiniband/core interface, which as far as I can tell it
> > > currently does not even do, although some discussions here suggest
> > > that a future interface should cater to those interface.
> >
> > The counter balance is that the Kernel Community has had a lot of bad
> > experiences with drivers growing their own user APIs and 'subsystem'
> > like functionality.
> 
> Do you see any desire of HFI authors to move to new general user API?

Obviously we designed to have PSM be it's own interface just like qib was.

I see advantages and disadvantages for both us and the community with both directions.

The largest issue we have right now is that we are not sure about the performance of PSM through the proposed interface.  Lack of performance would, of course, be a show stopper.  Validating performance will take time and will probably not be effective until we get farther down the interface.

Frankly, I just have not had time to try to evaluate what it would take to get a command through the new interface.  I have spent some time assessing which commands (verbs and PSM) need to be fast path.  And I have done a quick review of the latest rfc from a few weeks ago from Matan.  I hoping to find more time soon.

For the current set of patches, I noticed an "escape clause" for vendor specific IOCTLs.

        if (cmd == RDMA_DIRECT_IOCTL) {
                /* TODO? */
                err = -ENOSYS;
                goto out;
        } else {

I think we should have this but to Christophs point if you have something like this why not just have a separate cdev?

Christoph, do you think the addition of the PSM interface to infiniband/core is a pollution of that ABI?  I made a similar argument that from a user side opening /dev/infiniband/uverbsX and then doing PSM on that FD was "unclean".   Jason however, brought up some good points on why we may want to do that.

Frankly I think we should focus on verbs and keep the interface flexible, which is what we are doing.

Ira

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ofvwg] [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Christoph Hellwig]

On Fri, Sep 16, 2016 at 07:19:31AM +0000, Weiny, Ira wrote:
> I think we should have this but to Christophs point if you have something like this why not just have a separate cdev?

Yes, please.  It's not relate to the verbs interface in any way, so
don't shoer horn it into the verbs interface - that's just going to
create problems for both the verbs interface and psm.

> Christoph, do you think the addition of the PSM interface to infiniband/core is a pollution of that ABI?  I made a similar argument that from a user side opening /dev/infiniband/uverbsX and then doing PSM on that FD was "unclean".   Jason however, brought up some good points on why we may want to do that.

It's not infiniband infiniband core code, so it should not go there.  If
you think there is a reasonable chance of more PSM drivers we can have a
drivers/psm for it, otherwise leave it in the driver.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [ANNOUNCE] Open Fabrics Verbs Working Group (OFVWG) meeting tomorrow - 9/13/2016 at 11:00PDT

[Liran Liss]

> From: Liran Liss
> Sent: Monday, September 12, 2016 6:05 PM

> Agenda items:
> - Repositories
> - Code submission and mailing lists
> - Release process
> - Schedules
> - Other?
> 

Meeting notes:
- There is a general agreement to consolidate rdma user-space components into a single repo.
- In addition to shared libraries and provides, it seems that tools like srp-tools should also be included. Ibdiagnet may also be considered, but OpenSM does not seem to be suitable.
- Source should include all providers and libraries and all components are built
- Security patches should be applied immediately on master branch. No need to release with specific tag. Distros may cherry-pick such patches.
- Code submission options for co-maintainers:
-- Ack patches on mailing list, and core maintainer team would apply directly
-- Apply patches on local clone, send pull request
-- Direct right access
* Selected option depends on patch volume and maintainer preference.
- Releases will be synchronized with kernel schedules
-- Latest tree will be released with no delays
- Future work will include:
-- Travis CI integration
-- Open build systems
-- Distro backporting as a community effort
- Project will be hosted on github.com/linux-rdma organization
-- Doug Ledford was accepted to be the main project maintainer
-- Core services will be supported by maintainer teams to ensure continuous progress and availability
--- Doug/Yishai will be responsible for libibverbs
--- Sean/TBD will be responsible for librdmacm
--- Jason/TBD will be responsible for the build system
- The intention is to conduct the first release with Linux-4.9
-- We need to target starting the process towards the beginning of the nearest merge window (2-3 weeks)
-- Doug to advertise when the project git it ready
-- Jason to synchronize initial repo and complete integration gaps
-- Doug/Leon/Jason to workout component repos
-- Afterwards, all patches should go into the project main/component git repos


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html