Re: [SPDK] SPDK Event notifications

Re: [SPDK] SPDK Event notifications

[Wodkowski, PawelX]

[-- Attachment #1: Type: text/plain, Size: 2926 bytes --]

See comments inline.

Paweł

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Pelplinski, Piotr
> Sent: Wednesday, October 10, 2018 4:59 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] SPDK Event notifications
> 
> Hi,
> I started work on SPDK event notifications.  I have few doubts that you might
> help me resolve.
> This functionality adds ability to register for specific events (i.e. a new bdev),
> and be notified asynchronously when those events occur.
> 
> My proposal is three RPC calls based on following workflow:
> 
> 1.      After SPDK application starts, client sends RPCs to SPDK about what
> kinds of events it wants to register for.
> 
> 2.      Then, client subscribes to specified events.
> 
> 3.      Then the client periodically asks for new events.

Why client need to ask for new event? Why event can't be send just send to all clients ASAP they available?
You will have to track what events was send to what clients.
> 
> RPC Calls:

Are you describing the JSON RPC objects here or rpc.py commands?

> 
> 1.      show_async_events <component>
> component (optional) - show events for specified component only
> 
> 2.      register_async_events <unique_id>, <events>
> unique_id - unique identifier for client

What is the unique_id for? Isn't the connection itself an "unique id" enough?

> events - list of comma separated events which client wants to subscribe to.


> (Additional question: Does event should have parameters too? e.g. subscribe
> only to event: deletion of bdev with specific name)
> Response is a list of types of events to which client was subscribed.
> 
> 3.      poll_async_events <unique_id>
> unique_id - unique identifier for client
> Response is a list of events which happen from last call, which include:
> 
> -        Component, # component type for which event is raised
> 
> -        Notification,  # type of event, e.g. add, delete or update
> 
> -        Component_uuid, (e.g lvol uuid)
> 
> -        Parent_uuid (e.g. lvolstore uuid)
> 
> My question is what are the types of events we could register to?
> I can think of few generic bdev events like:
> -added bdev
> -deleted bdev
> -updated bdev
> 
> What other types of event do you see?

Runtime errors? Statistics? Nvmf/iscsi/vhost initiators connecting and disconnecting.
There will be many more but the solution you are proposing should be generic
enough to easly add more event in the future. bdev subsystem good starting point.

> 
> You can also look at the updates on this topic on trello:
> https://trello.com/c/ZTIHxp3w/28-asynchronous-rpc-notifications
> 
> --
> Best Regards,
> Piotr Pelpliński
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK Event notifications

[Sablok, Kunal]

[-- Attachment #1: Type: text/plain, Size: 3623 bytes --]

+Amit

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Pelplinski, Piotr
Sent: Friday, October 12, 2018 6:46 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK Event notifications



-- 
Best Regards,
Piotr Pelpliński


> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Wednesday, October 10, 2018 7:17 PM
> To: spdk(a)lists.01.org
> Subject: Re: [SPDK] SPDK Event notifications
> 
> On Wed, 2018-10-10 at 15:38 +0000, Wodkowski, PawelX wrote:
> > See comments inline.
> >
> > Paweł
> >
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Pelplinski, Piotr
> > >
> > > Hi,
> > > I started work on SPDK event notifications.  I have few doubts that you
> > > might help me resolve.
> > > This functionality adds ability to register for specific events (i.e. a new
> > > bdev), and be notified asynchronously when those events occur.
> > >
> > > My proposal is three RPC calls based on following workflow:
> > >
> > > 1.      After SPDK application starts, client sends RPCs to SPDK about what
> > > kinds of events it wants to register for.
> > >
> > > 2.      Then, client subscribes to specified events.
> > >
> > > 3.      Then the client periodically asks for new events.
> >
> > Why client need to ask for new event? Why event can't be send just send to all
> > clients ASAP they available?
> > You will have to track what events was send to what clients.
> 
> I think the above statement about tracking which events to send to which client
> outlines the major problem here. Tracking which events each client needs is very
> difficult - for instance, how long do you keep events around? A client could
> check in once a month, for example. Further, how many simultaneous clients do
> we
> need to keep track of? How much memory is that going to take? I'm positive
> that
> we don't want to go down this path.
> 
> Clients already have the ability to simply poll the state of the system by
> periodically issuing RPCs. That's good enough for most cases and is entirely
> stateless (the REST paradigm - we even use JSON!). We could consider an event
> stream API as an optimization only, where clients send an RPC and the response
> contains a file descriptor (maybe a domain socket?) where a stream of JSON
> objects corresponding to events as they happen is published (but not events
> from
> before the connection was established). A client would then begin by sending
> the
> existing RPCs to gather the current state of the world, then subscribe to this
> event API for changes only.
> 

How about returning events in response to RPC Call?

Client discovers events by calling show_events, then connects to RPC server
and asynchronously receives events as long as it is connected.

This way we have true asynchronous API. It doesn’t require caching of events or 
allocating any special resources or memory. 

Disconnected clients (i.e. due to reboot) would not receive those events, but do they
really want to receive them? I think that client after reboot would like to read entire
SPDK state anyway.

I updated trello card as well: 
https://trello.com/c/ZTIHxp3w/28-spdk-event-notifications

> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK Event notifications

[Pelplinski, Piotr]

[-- Attachment #1: Type: text/plain, Size: 3222 bytes --]



-- 
Best Regards,
Piotr Pelpliński


> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Wednesday, October 10, 2018 7:17 PM
> To: spdk(a)lists.01.org
> Subject: Re: [SPDK] SPDK Event notifications
> 
> On Wed, 2018-10-10 at 15:38 +0000, Wodkowski, PawelX wrote:
> > See comments inline.
> >
> > Paweł
> >
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Pelplinski, Piotr
> > >
> > > Hi,
> > > I started work on SPDK event notifications.  I have few doubts that you
> > > might help me resolve.
> > > This functionality adds ability to register for specific events (i.e. a new
> > > bdev), and be notified asynchronously when those events occur.
> > >
> > > My proposal is three RPC calls based on following workflow:
> > >
> > > 1.      After SPDK application starts, client sends RPCs to SPDK about what
> > > kinds of events it wants to register for.
> > >
> > > 2.      Then, client subscribes to specified events.
> > >
> > > 3.      Then the client periodically asks for new events.
> >
> > Why client need to ask for new event? Why event can't be send just send to all
> > clients ASAP they available?
> > You will have to track what events was send to what clients.
> 
> I think the above statement about tracking which events to send to which client
> outlines the major problem here. Tracking which events each client needs is very
> difficult - for instance, how long do you keep events around? A client could
> check in once a month, for example. Further, how many simultaneous clients do
> we
> need to keep track of? How much memory is that going to take? I'm positive
> that
> we don't want to go down this path.
> 
> Clients already have the ability to simply poll the state of the system by
> periodically issuing RPCs. That's good enough for most cases and is entirely
> stateless (the REST paradigm - we even use JSON!). We could consider an event
> stream API as an optimization only, where clients send an RPC and the response
> contains a file descriptor (maybe a domain socket?) where a stream of JSON
> objects corresponding to events as they happen is published (but not events
> from
> before the connection was established). A client would then begin by sending
> the
> existing RPCs to gather the current state of the world, then subscribe to this
> event API for changes only.
> 

How about returning events in response to RPC Call?

Client discovers events by calling show_events, then connects to RPC server
and asynchronously receives events as long as it is connected.

This way we have true asynchronous API. It doesn’t require caching of events or 
allocating any special resources or memory. 

Disconnected clients (i.e. due to reboot) would not receive those events, but do they
really want to receive them? I think that client after reboot would like to read entire
SPDK state anyway.

I updated trello card as well: 
https://trello.com/c/ZTIHxp3w/28-spdk-event-notifications

> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK Event notifications

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1056 bytes --]

On Thu, 2018-10-11 at 22:13 +0200, Andrey Kuzmin wrote:
> Just curious, how does SPDK NVMe driver handle AENs then (if at all), as
> the problem statement above is pretty similar to what AEN aimed to address?

The driver just sends down an AEN command at start up which sits there until an
event happens. Users may register for notifications upon completion of that AEN
command. Anyone registered at the time that the AEN fires gets notified. When
the driver sees the AEN completion, it automatically clears the event by sending
the appropriate get log page command. That means that events will be thrown away
if no one is subscribed at the time they occur.

AEN is similarly flawed in that someone must clear the event, which implies that
all clients are coordinating with one another somehow. If one client clears the
event, future clients won't be notified. In general this isn't a problem for
NVMe because there is only one client (the device driver) and when that isn't
connected the device is not powered.


> Regards,
> Andrey

Re: [SPDK] SPDK Event notifications

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 4698 bytes --]

On Wed, Oct 10, 2018, 19:19 Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:

> On Wed, 2018-10-10 at 15:38 +0000, Wodkowski, PawelX wrote:
> > See comments inline.
> >
> > Paweł
> >
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> Pelplinski, Piotr
> > >
> > > Hi,
> > > I started work on SPDK event notifications.  I have few doubts that you
> > > might help me resolve.
> > > This functionality adds ability to register for specific events (i.e.
> a new
> > > bdev), and be notified asynchronously when those events occur.
> > >
> > > My proposal is three RPC calls based on following workflow:
> > >
> > > 1.      After SPDK application starts, client sends RPCs to SPDK about
> what
> > > kinds of events it wants to register for.
> > >
> > > 2.      Then, client subscribes to specified events.
> > >
> > > 3.      Then the client periodically asks for new events.
> >
> > Why client need to ask for new event? Why event can't be send just send
> to all
> > clients ASAP they available?
> > You will have to track what events was send to what clients.
>
> I think the above statement about tracking which events to send to which
> client
> outlines the major problem here. Tracking which events each client needs
> is very
> difficult - for instance, how long do you keep events around? A client
> could
> check in once a month, for example. Further, how many simultaneous clients
> do we
> need to keep track of? How much memory is that going to take? I'm positive
> that
> we don't want to go down this path.
>

Just curious, how does SPDK NVMe driver handle AENs then (if at all), as
the problem statement above is pretty similar to what AEN aimed to address?

Regards,
Andrey


> Clients already have the ability to simply poll the state of the system by
> periodically issuing RPCs. That's good enough for most cases and is
> entirely
> stateless (the REST paradigm - we even use JSON!). We could consider an
> event
> stream API as an optimization only, where clients send an RPC and the
> response
> contains a file descriptor (maybe a domain socket?) where a stream of JSON
> objects corresponding to events as they happen is published (but not
> events from
> before the connection was established). A client would then begin by
> sending the
> existing RPCs to gather the current state of the world, then subscribe to
> this
> event API for changes only.
>
> > >
> > > RPC Calls:
> >
> > Are you describing the JSON RPC objects here or rpc.py commands?
> >
> > >
> > > 1.      show_async_events <component>
> > > component (optional) - show events for specified component only
> > >
> > > 2.      register_async_events <unique_id>, <events>
> > > unique_id - unique identifier for client
> >
> > What is the unique_id for? Isn't the connection itself an "unique id"
> enough?
>
> Presumably the client could crash, get restarted, and want to grab the
> events it
> missed by using the unique id.
>
> >
> >
> > > events - list of comma separated events which client wants to
> subscribe to.
> >
> >
> > > (Additional question: Does event should have parameters too? e.g.
> subscribe
> > > only to event: deletion of bdev with specific name)
> > > Response is a list of types of events to which client was subscribed.
> > >
> > > 3.      poll_async_events <unique_id>
> > > unique_id - unique identifier for client
> > > Response is a list of events which happen from last call, which
> include:
> > >
> > > -        Component, # component type for which event is raised
> > >
> > > -        Notification,  # type of event, e.g. add, delete or update
> > >
> > > -        Component_uuid, (e.g lvol uuid)
> > >
> > > -        Parent_uuid (e.g. lvolstore uuid)
> > >
> > > My question is what are the types of events we could register to?
> > > I can think of few generic bdev events like:
> > > -added bdev
> > > -deleted bdev
> > > -updated bdev
> > >
> > > What other types of event do you see?
> >
> > Runtime errors? Statistics? Nvmf/iscsi/vhost initiators connecting and
> > disconnecting.
> > There will be many more but the solution you are proposing should be
> generic
> > enough to easly add more event in the future. bdev subsystem good
> starting
> > point.
> >
> > >
> > > You can also look at the updates on this topic on trello:
> > > https://trello.com/c/ZTIHxp3w/28-asynchronous-rpc-notifications
> > >
> > > --
> > > Best Regards,
> > > Piotr Pelpliński
> > >
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] SPDK Event notifications

[Pelplinski, Piotr]

[-- Attachment #1: Type: text/plain, Size: 4844 bytes --]

> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Wednesday, October 10, 2018 7:17 PM
> To: spdk(a)lists.01.org
> Subject: Re: [SPDK] SPDK Event notifications
> 
> On Wed, 2018-10-10 at 15:38 +0000, Wodkowski, PawelX wrote:
> > See comments inline.
> >
> > Paweł
> >
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Pelplinski, Piotr
> > >
> > > Hi,
> > > I started work on SPDK event notifications.  I have few doubts that you
> > > might help me resolve.
> > > This functionality adds ability to register for specific events (i.e. a new
> > > bdev), and be notified asynchronously when those events occur.
> > >
> > > My proposal is three RPC calls based on following workflow:
> > >
> > > 1.      After SPDK application starts, client sends RPCs to SPDK about what
> > > kinds of events it wants to register for.
> > >
> > > 2.      Then, client subscribes to specified events.
> > >
> > > 3.      Then the client periodically asks for new events.
> >
> > Why client need to ask for new event? Why event can't be send just send to all
> > clients ASAP they available?
> > You will have to track what events was send to what clients.
> 
> I think the above statement about tracking which events to send to which client
> outlines the major problem here. Tracking which events each client needs is very
> difficult - for instance, how long do you keep events around? A client could
> check in once a month, for example. Further, how many simultaneous clients do
> we
> need to keep track of? How much memory is that going to take? I'm positive
> that
> we don't want to go down this path.

How about timeouts? We could specify requirement that client has to ask for new
events in specified finite amount of time, like one minute.

> 
> Clients already have the ability to simply poll the state of the system by
> periodically issuing RPCs. That's good enough for most cases and is entirely
> stateless (the REST paradigm - we even use JSON!). We could consider an event
> stream API as an optimization only, where clients send an RPC and the response
> contains a file descriptor (maybe a domain socket?) where a stream of JSON
> objects corresponding to events as they happen is published (but not events
> from
> before the connection was established). A client would then begin by sending
> the
> existing RPCs to gather the current state of the world, then subscribe to this
> event API for changes only.
> 
This solution looks very similar to websockets. Do you think that adding such
optimization is a priority?

> > >
> > > RPC Calls:
> >
> > Are you describing the JSON RPC objects here or rpc.py commands?
> >
> > >
> > > 1.      show_async_events <component>
> > > component (optional) - show events for specified component only
> > >
> > > 2.      register_async_events <unique_id>, <events>
> > > unique_id - unique identifier for client
> >
> > What is the unique_id for? Isn't the connection itself an "unique id" enough?
> 
> Presumably the client could crash, get restarted, and want to grab the events it
> missed by using the unique id.
> 
> >
> >
> > > events - list of comma separated events which client wants to subscribe to.
> >
> >
> > > (Additional question: Does event should have parameters too? e.g. subscribe
> > > only to event: deletion of bdev with specific name)
> > > Response is a list of types of events to which client was subscribed.
> > >
> > > 3.      poll_async_events <unique_id>
> > > unique_id - unique identifier for client
> > > Response is a list of events which happen from last call, which include:
> > >
> > > -        Component, # component type for which event is raised
> > >
> > > -        Notification,  # type of event, e.g. add, delete or update
> > >
> > > -        Component_uuid, (e.g lvol uuid)
> > >
> > > -        Parent_uuid (e.g. lvolstore uuid)
> > >
> > > My question is what are the types of events we could register to?
> > > I can think of few generic bdev events like:
> > > -added bdev
> > > -deleted bdev
> > > -updated bdev
> > >
> > > What other types of event do you see?
> >
> > Runtime errors? Statistics? Nvmf/iscsi/vhost initiators connecting and
> > disconnecting.
> > There will be many more but the solution you are proposing should be generic
> > enough to easly add more event in the future. bdev subsystem good starting
> > point.
> >
> > >
> > > You can also look at the updates on this topic on trello:
> > > https://trello.com/c/ZTIHxp3w/28-asynchronous-rpc-notifications
> > >
> > > --
> > > Best Regards,
> > > Piotr Pelpliński
> > >
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK Event notifications

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4054 bytes --]

On Wed, 2018-10-10 at 15:38 +0000, Wodkowski, PawelX wrote:
> See comments inline.
> 
> Paweł
> 
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Pelplinski, Piotr
> > 
> > Hi,
> > I started work on SPDK event notifications.  I have few doubts that you
> > might help me resolve.
> > This functionality adds ability to register for specific events (i.e. a new
> > bdev), and be notified asynchronously when those events occur.
> > 
> > My proposal is three RPC calls based on following workflow:
> > 
> > 1.      After SPDK application starts, client sends RPCs to SPDK about what
> > kinds of events it wants to register for.
> > 
> > 2.      Then, client subscribes to specified events.
> > 
> > 3.      Then the client periodically asks for new events.
> 
> Why client need to ask for new event? Why event can't be send just send to all
> clients ASAP they available?
> You will have to track what events was send to what clients.

I think the above statement about tracking which events to send to which client
outlines the major problem here. Tracking which events each client needs is very
difficult - for instance, how long do you keep events around? A client could
check in once a month, for example. Further, how many simultaneous clients do we
need to keep track of? How much memory is that going to take? I'm positive that
we don't want to go down this path.

Clients already have the ability to simply poll the state of the system by
periodically issuing RPCs. That's good enough for most cases and is entirely
stateless (the REST paradigm - we even use JSON!). We could consider an event
stream API as an optimization only, where clients send an RPC and the response
contains a file descriptor (maybe a domain socket?) where a stream of JSON
objects corresponding to events as they happen is published (but not events from
before the connection was established). A client would then begin by sending the
existing RPCs to gather the current state of the world, then subscribe to this
event API for changes only.

> > 
> > RPC Calls:
> 
> Are you describing the JSON RPC objects here or rpc.py commands?
> 
> > 
> > 1.      show_async_events <component>
> > component (optional) - show events for specified component only
> > 
> > 2.      register_async_events <unique_id>, <events>
> > unique_id - unique identifier for client
> 
> What is the unique_id for? Isn't the connection itself an "unique id" enough?

Presumably the client could crash, get restarted, and want to grab the events it
missed by using the unique id.

> 
> 
> > events - list of comma separated events which client wants to subscribe to.
> 
> 
> > (Additional question: Does event should have parameters too? e.g. subscribe
> > only to event: deletion of bdev with specific name)
> > Response is a list of types of events to which client was subscribed.
> > 
> > 3.      poll_async_events <unique_id>
> > unique_id - unique identifier for client
> > Response is a list of events which happen from last call, which include:
> > 
> > -        Component, # component type for which event is raised
> > 
> > -        Notification,  # type of event, e.g. add, delete or update
> > 
> > -        Component_uuid, (e.g lvol uuid)
> > 
> > -        Parent_uuid (e.g. lvolstore uuid)
> > 
> > My question is what are the types of events we could register to?
> > I can think of few generic bdev events like:
> > -added bdev
> > -deleted bdev
> > -updated bdev
> > 
> > What other types of event do you see?
> 
> Runtime errors? Statistics? Nvmf/iscsi/vhost initiators connecting and
> disconnecting.
> There will be many more but the solution you are proposing should be generic
> enough to easly add more event in the future. bdev subsystem good starting
> point.
> 
> > 
> > You can also look at the updates on this topic on trello:
> > https://trello.com/c/ZTIHxp3w/28-asynchronous-rpc-notifications
> > 
> > --
> > Best Regards,
> > Piotr Pelpliński
> > 

[SPDK] SPDK Event notifications

[Pelplinski, Piotr]

[-- Attachment #1: Type: text/plain, Size: 1787 bytes --]

Hi,
I started work on SPDK event notifications.  I have few doubts that you might help me resolve.
This functionality adds ability to register for specific events (i.e. a new bdev), and be notified asynchronously when those events occur.

My proposal is three RPC calls based on following workflow:

1.      After SPDK application starts, client sends RPCs to SPDK about what kinds of events it wants to register for.

2.      Then, client subscribes to specified events.

3.      Then the client periodically asks for new events.

RPC Calls:

1.      show_async_events <component>
component (optional) - show events for specified component only

2.      register_async_events <unique_id>, <events>
unique_id - unique identifier for client
events - list of comma separated events which client wants to subscribe to.
(Additional question: Does event should have parameters too? e.g. subscribe only to event: deletion of bdev with specific name)
Response is a list of types of events to which client was subscribed.

3.      poll_async_events <unique_id>
unique_id - unique identifier for client
Response is a list of events which happen from last call, which include:

-        Component, # component type for which event is raised

-        Notification,  # type of event, e.g. add, delete or update

-        Component_uuid, (e.g lvol uuid)

-        Parent_uuid (e.g. lvolstore uuid)

My question is what are the types of events we could register to?
I can think of few generic bdev events like:
-added bdev
-deleted bdev
-updated bdev

What other types of event do you see?

You can also look at the updates on this topic on trello:
https://trello.com/c/ZTIHxp3w/28-asynchronous-rpc-notifications

--
Best Regards,
Piotr Pelpliński