Plans for meshd

Plans for meshd

[Michał Lowas-Rzechonek]

Hi,

Given recent patches to mesh, what are your mid to long-term plans to
enable co-existence of bluetoothd and meshd on a single system?

From what I gather (please, correct me if I'm wrong), current
implementation of mesh-io-generic assumes that meshd is the sole owner
of the BLE adapter, because the HCI socket is bound using as
HCI_CHANNEL_USER, so at the moment the daemons are unable to share the
same HCI device. Is my understanding correct?

regards
-- 
Michał Lowas-Rzechonek
Lambda Team Leader

Silvair
Jasnogórska 44
31-358 Krakow
POLAND

http://www.silvair.com

RE: Plans for meshd

[Gix, Brian]

Hi Michal,

> -----Original Message-----
> From: linux-bluetooth-owner@vger.kernel.org [mailto:linux-bluetooth-
> owner@vger.kernel.org] On Behalf Of Michal Lowas-Rzechonek
> Sent: Monday, December 17, 2018 1:10 PM
> To: linux-bluetooth@vger.kernel.org
> Subject: Plans for meshd
> 
> Hi,
> 
> Given recent patches to mesh, what are your mid to long-term plans to
> enable co-existence of bluetoothd and meshd on a single system?

You are essentially correct, that the current implementation assumes mesh ownership of the Controller, and there is no co-existence of "Classic Bluetooth" as implemented in the Bluetooth Daemon, on the same controller.  So currently, if you want to simultaneously run an ACL connection based application with Mesh, you would do so with more than one controller.  (This should already work)

I consider this to be a practical approach, because the overhead consumed by ACL connections negatively affects Advertising based communications, and linux in particular handles a practically unlimited number of controllers.  Support for devices with controllers that have active ACL connections requires other mesh devices to send more redundant packets to ensure delivery.

The primary use cases targeted by this implementation assumes the controller is being used for one or the other.  And we want to give embedded platforms the option of not installing meshd if they don't need Mesh, or not installing bluetoothd if they don't need Classic Bluetooth. 

That said we *do* plan on supporting GATT Proxy Server, but offering this support through the Bluetooth daemon (bluetoothd) on a separate controller.  Is this your original question?  Same system, but multiple controllers, and both daemons running.  Also forward looking, a system might have as many as 5 controllers:  1 for Classic Bluetooth, 1 for Outbound Mesh traffic, and 3 (one on each LE adv channel) for incoming Mesh traffic.

And I am not opposed to someone starting a project that will allow the bluetoothd and meshd share a single controller and taking responsibility the reduced performance of each, but that is not work we have planned.

> From what I gather (please, correct me if I'm wrong), current implementation
> of mesh-io-generic assumes that meshd is the sole owner of the BLE
> adapter, because the HCI socket is bound using as HCI_CHANNEL_USER, so at
> the moment the daemons are unable to share the same HCI device. Is my
> understanding correct?
> 
> regards
> --
> Michał Lowas-Rzechonek
> Lambda Team Leader
> 
> Silvair
> Jasnogórska 44
> 31-358 Krakow
> POLAND
> 
> http://www.silvair.com

Regards,
Brian

Re: Plans for meshd

[Michał Lowas-Rzechonek]

Hi Brian,

Thanks for clarifying.

On Mon, Dec 17, 2018 at 09:48:34PM +0000, Gix, Brian wrote:
> I consider this to be a practical approach, because the overhead
> consumed by ACL connections negatively affects Advertising based
> communications, and linux in particular handles a practically
> unlimited number of controllers.  Support for devices with controllers
> that have active ACL connections requires other mesh devices to send
> more redundant packets to ensure delivery.
Ok, fair enough. We are well aware of performance considerations.

What I'm thinking about is having both modes of communication available,
even if the embedded device has just one Controller, as is the case with
the majority of off-the-shelf platforms.

The reason we want/need both BLE/GATT and Mesh on the same node has a
lot to do with provisioning - at the moment we configure our networks
using a mobile app, meaning we need devices to support PB-GATT. This is
because Controllers present in mobile devices do not support Mesh
natively, and we can't assume there is a GATT Proxy nearby, especially
when configuring a new network.

> And I am not opposed to someone starting a project that will allow the
> bluetoothd and meshd share a single controller and taking
> responsibility the reduced performance of each, but that is not work
> we have planned.
Ack.

FYI, it seems we might be interested in making that happen.

regards
-- 
Michał Lowas-Rzechonek <michal.lowas-rzechonek@silvair.com>
Silvair http://silvair.com
Jasnogórska 44, 31-358 Krakow, POLAND

RE: Plans for meshd

[Gix, Brian]

Hi Michal,

> -----Original Message-----
> From: linux-bluetooth-owner@vger.kernel.org [mailto:linux-bluetooth-
> owner@vger.kernel.org] On Behalf Of Michal Lowas-Rzechonek
> Sent: Tuesday, December 18, 2018 3:09 AM
> To: linux-bluetooth@vger.kernel.org
> Subject: Re: Plans for meshd
> 
[...]
> What I'm thinking about is having both modes of communication available,
> even if the embedded device has just one Controller, as is the case with the
> majority of off-the-shelf platforms.
> 
> The reason we want/need both BLE/GATT and Mesh on the same node has a
> lot to do with provisioning - at the moment we configure our networks using
> a mobile app, meaning we need devices to support PB-GATT. This is because
> Controllers present in mobile devices do not support Mesh natively, and we
> can't assume there is a GATT Proxy nearby, especially when configuring a
> new network.
> 
> > And I am not opposed to someone starting a project that will allow the
> > bluetoothd and meshd share a single controller and taking
> > responsibility the reduced performance of each, but that is not work
> > we have planned.
> Ack.
> 
> FYI, it seems we might be interested in making that happen.

Maybe we should start with what you actually need.
* Do you need GATT support only for the Provisioning procedure?
* or also as a GATT proxy Server?

There is a possibility...   We had an internal project where we supported a GATT Proxy server without using the bluetooothd services at all.  For bluetoothd, the result is the same:  The controller was entirely under the control of the mesh executable (it was not a daemon, but a proof-of-concept command line tool).  But there are problems with that particular idea given the architecture of the meshd daemon:

* I do not think we should try to support Proxy Client *at all*. The Proxy Client assumes a single Node, which defeats the purpose of having a daemon at all.  

* Only a single node (or unprovisioned GATT server) can use GATT at a time.  The protocol does not allow for the multi-node architecture of the daemon.

*  If we do try to put a GATT Proxy Server on a single controller implementation, we may need a special node that loops back data received from the Mesh to the Gatt Client.

* We will probably need to limit this special GATT supporting node to a single node on the daemon.

* We may want to make this a build-time option, because ADV handling will probably be affected whether an existing GATT connection is established or not, if it is possible that a GATT connection could be requested.

* This affects the times when we can use Randomized, ever changing BD ADDRs, and requires us to interleave connectable advertisements and "be connectable", which is not technically allowed with mesh network ADV packets.




> 
> regards
> --
> Michał Lowas-Rzechonek <michal.lowas-rzechonek@silvair.com>
> Silvair http://silvair.com
> Jasnogórska 44, 31-358 Krakow, POLAND

Re: Plans for meshd

[Danielle Costantino]

Thanks for the work done on BLE Mesh but I have to agree with the
request to support GATT and BLE Mesh simultaneously on a single
controller. There are many applications that use GATT for higher
bandwidth point to point connections used in OTA update where the
overhead and functionality of mesh are not needed or wanted. Until
Mesh supports routed packets large SAR uploads flood the mesh network
and make OTA over mesh nearly impossible when the node count goes
above 4 relay nodes. Limiting the number of GATT nodes to 1 at a time
is not a problem for our application needs.

What I am unclear on is why it is not possible to support generic GATT
and PB-GATT + Proxy Client/Server on the same controller. If PB-ADV is
only used for provisioning then the time spent in ADV will not affect
steady state operations.

Thanks,
Danielle

On Tue, Dec 18, 2018 at 8:28 AM Gix, Brian <brian.gix@intel.com> wrote:
>
> Hi Michal,
>
> > -----Original Message-----
> > From: linux-bluetooth-owner@vger.kernel.org [mailto:linux-bluetooth-
> > owner@vger.kernel.org] On Behalf Of Michal Lowas-Rzechonek
> > Sent: Tuesday, December 18, 2018 3:09 AM
> > To: linux-bluetooth@vger.kernel.org
> > Subject: Re: Plans for meshd
> >
> [...]
> > What I'm thinking about is having both modes of communication available,
> > even if the embedded device has just one Controller, as is the case with the
> > majority of off-the-shelf platforms.
> >
> > The reason we want/need both BLE/GATT and Mesh on the same node has a
> > lot to do with provisioning - at the moment we configure our networks using
> > a mobile app, meaning we need devices to support PB-GATT. This is because
> > Controllers present in mobile devices do not support Mesh natively, and we
> > can't assume there is a GATT Proxy nearby, especially when configuring a
> > new network.
> >
> > > And I am not opposed to someone starting a project that will allow the
> > > bluetoothd and meshd share a single controller and taking
> > > responsibility the reduced performance of each, but that is not work
> > > we have planned.
> > Ack.
> >
> > FYI, it seems we might be interested in making that happen.
>
> Maybe we should start with what you actually need.
> * Do you need GATT support only for the Provisioning procedure?
> * or also as a GATT proxy Server?
>
> There is a possibility...   We had an internal project where we supported a GATT Proxy server without using the bluetooothd services at all.  For bluetoothd, the result is the same:  The controller was entirely under the control of the mesh executable (it was not a daemon, but a proof-of-concept command line tool).  But there are problems with that particular idea given the architecture of the meshd daemon:
>
> * I do not think we should try to support Proxy Client *at all*. The Proxy Client assumes a single Node, which defeats the purpose of having a daemon at all.
>
> * Only a single node (or unprovisioned GATT server) can use GATT at a time.  The protocol does not allow for the multi-node architecture of the daemon.
>
> *  If we do try to put a GATT Proxy Server on a single controller implementation, we may need a special node that loops back data received from the Mesh to the Gatt Client.
>
> * We will probably need to limit this special GATT supporting node to a single node on the daemon.
>
> * We may want to make this a build-time option, because ADV handling will probably be affected whether an existing GATT connection is established or not, if it is possible that a GATT connection could be requested.
>
> * This affects the times when we can use Randomized, ever changing BD ADDRs, and requires us to interleave connectable advertisements and "be connectable", which is not technically allowed with mesh network ADV packets.
>
>
>
>
> >
> > regards
> > --
> > Michał Lowas-Rzechonek <michal.lowas-rzechonek@silvair.com>
> > Silvair http://silvair.com
> > Jasnogórska 44, 31-358 Krakow, POLAND



-- 
- Danielle Costantino

RE: Plans for meshd

[Gix, Brian]

Hi Danielle,

> -----Original Message-----
> From: linux-bluetooth-owner@vger.kernel.org [mailto:linux-bluetooth-
> owner@vger.kernel.org] On Behalf Of Danielle Costantino
> Sent: Tuesday, December 18, 2018 9:08 AM
> To: Gix, Brian <brian.gix@intel.com>
> Cc: michal.lowas-rzechonek@silvair.com; linux-bluetooth@vger.kernel.org
> Subject: Re: Plans for meshd
> 
> Thanks for the work done on BLE Mesh but I have to agree with the request
> to support GATT and BLE Mesh simultaneously on a single controller. There
> are many applications that use GATT for higher bandwidth point to point
> connections used in OTA update where the overhead and functionality of
> mesh are not needed or wanted. Until Mesh supports routed packets large
> SAR uploads flood the mesh network and make OTA over mesh nearly
> impossible when the node count goes above 4 relay nodes. Limiting the
> number of GATT nodes to 1 at a time is not a problem for our application
> needs.
> 
> What I am unclear on is why it is not possible to support generic GATT and
> PB-GATT + Proxy Client/Server on the same controller. If PB-ADV is only used
> for provisioning then the time spent in ADV will not affect steady state
> operations.

I am not entirely sure I understand your question, so I will answer the question I *think* you are asking.

Mesh is an Advertising based system, that works best in a steady state when everyone is just
passively scanning for connectionless advertising packets, and sending them when required.

GATT runs over an ACL connection which reserves a certain amount of the available "time" bandwidth, even when the ACL link is idle.  This reduces the opportunities for the controller to hear advertising packets, and there are more missed  mesh (ADV) packets as a result.

Also (with the possible exception of a higher speed Firmware update path) the  GATT Proxy Service is intended as a legacy method for a GATT Only device to join a mesh, and talk to node through the mesh.  It is *not* intended to be the main bearer of the mesh (i.e. all the nodes are *not* interconnected by GATT...   only the nodes that do not support discrete advertising control...  like perhaps older smartphones).

The "Backbone" of any Mesh Network will be ADV bearer only.


> Thanks,
> Danielle
> 
> On Tue, Dec 18, 2018 at 8:28 AM Gix, Brian <brian.gix@intel.com> wrote:
> >
> > Hi Michal,
> >
> > > -----Original Message-----
> > > From: linux-bluetooth-owner@vger.kernel.org [mailto:linux-bluetooth-
> > > owner@vger.kernel.org] On Behalf Of Michal Lowas-Rzechonek
> > > Sent: Tuesday, December 18, 2018 3:09 AM
> > > To: linux-bluetooth@vger.kernel.org
> > > Subject: Re: Plans for meshd
> > >
> > [...]
> > > What I'm thinking about is having both modes of communication
> > > available, even if the embedded device has just one Controller, as
> > > is the case with the majority of off-the-shelf platforms.
> > >
> > > The reason we want/need both BLE/GATT and Mesh on the same node
> has
> > > a lot to do with provisioning - at the moment we configure our
> > > networks using a mobile app, meaning we need devices to support
> > > PB-GATT. This is because Controllers present in mobile devices do
> > > not support Mesh natively, and we can't assume there is a GATT Proxy
> > > nearby, especially when configuring a new network.
> > >
> > > > And I am not opposed to someone starting a project that will allow
> > > > the bluetoothd and meshd share a single controller and taking
> > > > responsibility the reduced performance of each, but that is not
> > > > work we have planned.
> > > Ack.
> > >
> > > FYI, it seems we might be interested in making that happen.
> >
> > Maybe we should start with what you actually need.
> > * Do you need GATT support only for the Provisioning procedure?
> > * or also as a GATT proxy Server?
> >
> > There is a possibility...   We had an internal project where we supported a
> GATT Proxy server without using the bluetooothd services at all.  For
> bluetoothd, the result is the same:  The controller was entirely under the
> control of the mesh executable (it was not a daemon, but a proof-of-concept
> command line tool).  But there are problems with that particular idea given
> the architecture of the meshd daemon:
> >
> > * I do not think we should try to support Proxy Client *at all*. The Proxy
> Client assumes a single Node, which defeats the purpose of having a daemon
> at all.
> >
> > * Only a single node (or unprovisioned GATT server) can use GATT at a
> time.  The protocol does not allow for the multi-node architecture of the
> daemon.
> >
> > *  If we do try to put a GATT Proxy Server on a single controller
> implementation, we may need a special node that loops back data received
> from the Mesh to the Gatt Client.
> >
> > * We will probably need to limit this special GATT supporting node to a
> single node on the daemon.
> >
> > * We may want to make this a build-time option, because ADV handling will
> probably be affected whether an existing GATT connection is established or
> not, if it is possible that a GATT connection could be requested.
> >
> > * This affects the times when we can use Randomized, ever changing BD
> ADDRs, and requires us to interleave connectable advertisements and "be
> connectable", which is not technically allowed with mesh network ADV
> packets.
> >
> >
> >
> >
> > >
> > > regards
> > > --
> > > Michał Lowas-Rzechonek <michal.lowas-rzechonek@silvair.com>
> > > Silvair http://silvair.com
> > > Jasnogórska 44, 31-358 Krakow, POLAND
> 
> 
> 
> --
> - Danielle Costantino

Re: Plans for meshd

[Michal Lowas-Rzechonek]

Hi Brian,

On 12/18, Gix, Brian wrote:
> Maybe we should start with what you actually need.
> * Do you need GATT support only for the Provisioning procedure?
Primarily, but not only this. As Danielle mentioned, there are cases
where we would like to "fall back" to GATT in order to exchange large
amounts of data with a selected node. This is going to happen only
occassionally, so it won't affect the "steady state" performance of a
mesh network. Spec even mentions that particular use case, via Node
Identity procedure.

> * or also as a GATT proxy Server?
No, at the moment we don't need GATT Proxy Server support in Linux,
althouth long-term I think it would be beneficial for bluez to have
this.

> * I do not think we should try to support Proxy Client *at all*.
Agreed.

> The Proxy Client assumes a single Node, which defeats the purpose of
> having a daemon at all.
>
> * Only a single node (or unprovisioned GATT server) can use GATT at a
> time.  The protocol does not allow for the multi-node architecture of
> the daemon.
I do not think this is true. The spec doesn't seem to put any
limitations on the Network PDUs you can exchange with the GATT Proxy
Server, so I think it is entirely possible to have multiple Mesh nodes
share a single GATT connection to a proxy.

Noone does that, though, because performance of such a setup would
probably be abysmal :) In the end, I don't consider this a "real" use
case.

> *  If we do try to put a GATT Proxy Server on a single controller
> implementation, we may need a special node that loops back data
> received from the Mesh to the Gatt Client.
Well, I was wondering if it would be feasible to implement another
flavour of mesh-io layer, that would talk to Bluetooth Daemon (possibly
via API extensions) instead of opening the hci user channel on its own.

For that to work, the main Bluetooth Daemon would need to expose D-Bus
signals to retrieve "raw" advertising indications, and a method to send
a single advertising packet.

Alternatively (and this is just me thinking out loud), maybe it would be
possible to dup() the HCI socket and pass it to the Mesh Daemon,
assuming it gets spawneby by fork()ing Bluetooth Daemon? Not sure if the
kernel interface allows such sheganigans.

> * We may want to make this a build-time option, because ADV handling
> will probably be affected whether an existing GATT connection is
> established or not, if it is possible that a GATT connection could be
> requested.
> 
> * This affects the times when we can use Randomized, ever changing BD
> ADDRs, and requires us to interleave connectable advertisements and
> "be connectable", which is not technically allowed with mesh network
> ADV packets.
In our experience, /just enabling/ Proxy Server on a node (thus making
it advertise with connectables) does not seem affect delivery rate of
mesh packets, at least not meaningfully (0.1%, maybe). The performance
drop is observable only after an actual connection is established, and
is in 10-20% range, if I recall correctly.

regards
-- 
Michał Lowas-Rzechonek <michal.lowas-rzechonek@silvair.com>
Silvair http://silvair.com
Jasnogórska 44, 31-358 Krakow, POLAND