[RFC] need for new scan api for bluetooth low energy.

[RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

The way MGMT_OP_START_DISCOVERY work right now is:
- it return information about each device only once by sending
MGMT_EV_DEVICE_FOUND
- it allows only for active scanning
- After DISCOV_LE_TIMEOUT or DISCOV_INTERLEAVED_TIMEOUT the discovery
is automatically stopped
- If you restart discovery, you will again get MGMT_EV_DEVICE_FOUND
for each found device.

This is not good for Bluetooth Low Energy because:
- there is a wide range of connectionless devices that sends data in
advertisement, and there is no way to get each advertisement using
Management api right now, - for LE devices that you aren't connected
to you get RSSI value only once every DISCOV_LE_TIMEOUT, that is once
every 10 seconds, which is too infrequently to be useful for many use
cases, - for many LE applications passive scanning is enough, and it
uses less energy

I want to propose changing existing or creating new API that will
allow to start passive or active low energy scanning, and receive
event for each advertisement packet including its content and RSSI
value.

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

> The way MGMT_OP_START_DISCOVERY work right now is:
> - it return information about each device only once by sending
> MGMT_EV_DEVICE_FOUND
> - it allows only for active scanning
> - After DISCOV_LE_TIMEOUT or DISCOV_INTERLEAVED_TIMEOUT the discovery
> is automatically stopped
> - If you restart discovery, you will again get MGMT_EV_DEVICE_FOUND
> for each found device.
> 
> This is not good for Bluetooth Low Energy because:
> - there is a wide range of connectionless devices that sends data in
> advertisement, and there is no way to get each advertisement using
> Management api right now, - for LE devices that you aren't connected
> to you get RSSI value only once every DISCOV_LE_TIMEOUT, that is once
> every 10 seconds, which is too infrequently to be useful for many use
> cases, - for many LE applications passive scanning is enough, and it
> uses less energy
> 
> I want to propose changing existing or creating new API that will
> allow to start passive or active low energy scanning, and receive
> event for each advertisement packet including its content and RSSI
> value.

have you looked at the the Add Device management command with the 0x00 Action. That will just send out Device Found events for devices in range and uses passive scanning.

In addition if you are paired with a device and actually want to connect to it, you can use Action 0x02 to allow for background connection. This is currently in use for LE HID devices.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Arman Uguray]

Hi Marcel,

> have you looked at the the Add Device management command with the 0x00 Action. That will just send out Device Found events for devices in range and uses passive scanning.
>
> In addition if you are paired with a device and actually want to connect to it, you can use Action 0x02 to allow for background connection. This is currently in use for LE HID devices.
>

This still only sends one Device Found event though right? In Jakub's
case, a mgmt event is desired to let the userspace know every time a
new advertisement packet is received, with the updated RSSI and
TxPower and so on.

I think this ties in with some of the previous conversations on the
list about adding a device scan API that is geared towards
applications (rather than OS UI). That is, not only a mgmt API that
enables these kinds of use cases but also (probably) a high-level API
exposed from bluetoothd that allows applications to scan for devices
filtered by service UUIDs, contents of the manufacturer data field,
and so on.

As for the Add Device command, does bluetoothd currently use this somehow?

Thanks,
Arman

Re: [RFC] need for new scan api for bluetooth low energy.

[Johan Hedberg]

Hi Arman,

On Tue, Sep 16, 2014, Arman Uguray wrote:
> > have you looked at the the Add Device management command with the
> > 0x00 Action. That will just send out Device Found events for devices
> > in range and uses passive scanning.
> >
> > In addition if you are paired with a device and actually want to
> > connect to it, you can use Action 0x02 to allow for background
> > connection. This is currently in use for LE HID devices.
> 
> This still only sends one Device Found event though right? In Jakub's
> case, a mgmt event is desired to let the userspace know every time a
> new advertisement packet is received, with the updated RSSI and
> TxPower and so on.

It should send as many Device Found events as there are advertising
reports from the controller. We're enabling duplicate filtering in the
hci_req_add_le_passive_scan() function in hci_core.c so maybe that's
your issue? Some HW (like Broadcom) tend to be more conservative in how
many events they send when filtering is enabled whereas others (like
CSR) will give you many more of them (at least one whenever RSSI
changes).

> I think this ties in with some of the previous conversations on the
> list about adding a device scan API that is geared towards
> applications (rather than OS UI). That is, not only a mgmt API that
> enables these kinds of use cases but also (probably) a high-level API
> exposed from bluetoothd that allows applications to scan for devices
> filtered by service UUIDs, contents of the manufacturer data field,
> and so on.
> 
> As for the Add Device command, does bluetoothd currently use this somehow?

Yes, this feature (action 0x02 for Add Device) takes over the entire LE
background scanning and auto-connection mechanism from 3.17 kernel
onwards since BlueZ 5.22 (which was mentioned in our release notes btw).
This is much better than the active scanning based auto-connections that
have until now been done using Start/Stop Discovery.

Johan

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

On Tue, Sep 16, 2014 at 10:14 PM, Johan Hedberg <johan.hedberg@gmail.com> wrote:
> Hi Arman,
>
> On Tue, Sep 16, 2014, Arman Uguray wrote:
>> > have you looked at the the Add Device management command with the
>> > 0x00 Action. That will just send out Device Found events for devices
>> > in range and uses passive scanning.
>> >
>> > In addition if you are paired with a device and actually want to
>> > connect to it, you can use Action 0x02 to allow for background
>> > connection. This is currently in use for LE HID devices.
>>
>> This still only sends one Device Found event though right? In Jakub's
>> case, a mgmt event is desired to let the userspace know every time a
>> new advertisement packet is received, with the updated RSSI and
>> TxPower and so on.
>
> It should send as many Device Found events as there are advertising
> reports from the controller. We're enabling duplicate filtering in the
> hci_req_add_le_passive_scan() function in hci_core.c so maybe that's
> your issue? Some HW (like Broadcom) tend to be more conservative in how
> many events they send when filtering is enabled whereas others (like
> CSR) will give you many more of them (at least one whenever RSSI
> changes).

Why is duplicate filtering being enabled ? Does it filter packets that
have different advertisement content ? Can I turn it off ?

>
>> I think this ties in with some of the previous conversations on the
>> list about adding a device scan API that is geared towards
>> applications (rather than OS UI). That is, not only a mgmt API that
>> enables these kinds of use cases but also (probably) a high-level API
>> exposed from bluetoothd that allows applications to scan for devices
>> filtered by service UUIDs, contents of the manufacturer data field,
>> and so on.
>>
>> As for the Add Device command, does bluetoothd currently use this somehow?
>
> Yes, this feature (action 0x02 for Add Device) takes over the entire LE
> background scanning and auto-connection mechanism from 3.17 kernel
> onwards since BlueZ 5.22 (which was mentioned in our release notes btw).
> This is much better than the active scanning based auto-connections that
> have until now been done using Start/Stop Discovery.

Thanks for pointing that!
The release note says that:
"The kernel will even use the controller-side whitelist during
scanning (if no Resolvable Private Addresses are present), saving even
more power."
If I understand well, that mean that my peripheral device need to be
whitelisted in order for me to get advertisements ?
My use case include scanning for non-paired, non-connected devices,
checking RSSI value and deciding about connection based on
advertisement content, so that won't work. Am I right ?


>
> Johan

Re: [RFC] need for new scan api for bluetooth low energy.

[Johan Hedberg]

Hi Jakub,

On Tue, Sep 16, 2014, Jakub Pawlowski wrote:
> On Tue, Sep 16, 2014 at 10:14 PM, Johan Hedberg <johan.hedberg@gmail.com> wrote:
> > Hi Arman,
> >
> > On Tue, Sep 16, 2014, Arman Uguray wrote:
> >> > have you looked at the the Add Device management command with the
> >> > 0x00 Action. That will just send out Device Found events for devices
> >> > in range and uses passive scanning.
> >> >
> >> > In addition if you are paired with a device and actually want to
> >> > connect to it, you can use Action 0x02 to allow for background
> >> > connection. This is currently in use for LE HID devices.
> >>
> >> This still only sends one Device Found event though right? In Jakub's
> >> case, a mgmt event is desired to let the userspace know every time a
> >> new advertisement packet is received, with the updated RSSI and
> >> TxPower and so on.
> >
> > It should send as many Device Found events as there are advertising
> > reports from the controller. We're enabling duplicate filtering in the
> > hci_req_add_le_passive_scan() function in hci_core.c so maybe that's
> > your issue? Some HW (like Broadcom) tend to be more conservative in how
> > many events they send when filtering is enabled whereas others (like
> > CSR) will give you many more of them (at least one whenever RSSI
> > changes).
> 
> Why is duplicate filtering being enabled ? Does it filter packets that
> have different advertisement content ? Can I turn it off ?

At least in the HCI section of the core spec this parameter is quite
vaguely defined. We've enabled it to save power by avoiding the
controller waking up the host too often. With auto-connections it makes
sense to have it enabled since we're really only interested in the first
event which acts as a trigger to connect. Same goes for device discovery
when you're interested in setting up a new device for use.

Right now the filtering is hard-coded but we could consider making it
conditional to whether there are any entries with action 0x00 ("scan but
don't connect") and disable the filtering in that case.

> >> I think this ties in with some of the previous conversations on the
> >> list about adding a device scan API that is geared towards
> >> applications (rather than OS UI). That is, not only a mgmt API that
> >> enables these kinds of use cases but also (probably) a high-level API
> >> exposed from bluetoothd that allows applications to scan for devices
> >> filtered by service UUIDs, contents of the manufacturer data field,
> >> and so on.
> >>
> >> As for the Add Device command, does bluetoothd currently use this somehow?
> >
> > Yes, this feature (action 0x02 for Add Device) takes over the entire LE
> > background scanning and auto-connection mechanism from 3.17 kernel
> > onwards since BlueZ 5.22 (which was mentioned in our release notes btw).
> > This is much better than the active scanning based auto-connections that
> > have until now been done using Start/Stop Discovery.
> 
> Thanks for pointing that!
> The release note says that:
> "The kernel will even use the controller-side whitelist during
> scanning (if no Resolvable Private Addresses are present), saving even
> more power."
> If I understand well, that mean that my peripheral device need to be
> whitelisted in order for me to get advertisements ?

The controller-side whitelist just means that the controller only wakes
us up and gives us advertisements for entries in the whitelist. When not
using the whitelist the kernel gets woken up and needs to do itself the
filtering out of unknown devices. The kernel uses the controller-side
whitelist whenever possible, but e.g. when we have resolvable private
addresses in the list we can't use it as we can't know the address the
remote device will use in advance.

> My use case include scanning for non-paired, non-connected devices,
> checking RSSI value and deciding about connection based on
> advertisement content, so that won't work. Am I right ?

If you know the bdaddr that you're interested in in advance we should be
able to tweak the Add Device command to do what you want (e.g. disable
duplicate filtering in the case of Action=0x00 entries). However, if
what you need is general scanning for any devices (also previously
unknown ones) then Add Device doesn't really fit in (since it takes a
known bdaddr). For that we'd either need to consider allowing BDADDR_ANY
as an accepted parameter or then we'd need to define a new mgmt command.

Johan

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

On Tue, Sep 16, 2014 at 11:52 PM, Johan Hedberg <johan.hedberg@gmail.com> wrote:
> Hi Jakub,
>
> On Tue, Sep 16, 2014, Jakub Pawlowski wrote:
>> On Tue, Sep 16, 2014 at 10:14 PM, Johan Hedberg <johan.hedberg@gmail.com> wrote:
>> > Hi Arman,
>> >
>> > On Tue, Sep 16, 2014, Arman Uguray wrote:
>> >> > have you looked at the the Add Device management command with the
>> >> > 0x00 Action. That will just send out Device Found events for devices
>> >> > in range and uses passive scanning.
>> >> >
>> >> > In addition if you are paired with a device and actually want to
>> >> > connect to it, you can use Action 0x02 to allow for background
>> >> > connection. This is currently in use for LE HID devices.
>> >>
>> >> This still only sends one Device Found event though right? In Jakub's
>> >> case, a mgmt event is desired to let the userspace know every time a
>> >> new advertisement packet is received, with the updated RSSI and
>> >> TxPower and so on.
>> >
>> > It should send as many Device Found events as there are advertising
>> > reports from the controller. We're enabling duplicate filtering in the
>> > hci_req_add_le_passive_scan() function in hci_core.c so maybe that's
>> > your issue? Some HW (like Broadcom) tend to be more conservative in how
>> > many events they send when filtering is enabled whereas others (like
>> > CSR) will give you many more of them (at least one whenever RSSI
>> > changes).
>>
>> Why is duplicate filtering being enabled ? Does it filter packets that
>> have different advertisement content ? Can I turn it off ?
>
> At least in the HCI section of the core spec this parameter is quite
> vaguely defined. We've enabled it to save power by avoiding the
> controller waking up the host too often. With auto-connections it makes
> sense to have it enabled since we're really only interested in the first
> event which acts as a trigger to connect. Same goes for device discovery
> when you're interested in setting up a new device for use.
>
> Right now the filtering is hard-coded but we could consider making it
> conditional to whether there are any entries with action 0x00 ("scan but
> don't connect") and disable the filtering in that case.
>
>> >> I think this ties in with some of the previous conversations on the
>> >> list about adding a device scan API that is geared towards
>> >> applications (rather than OS UI). That is, not only a mgmt API that
>> >> enables these kinds of use cases but also (probably) a high-level API
>> >> exposed from bluetoothd that allows applications to scan for devices
>> >> filtered by service UUIDs, contents of the manufacturer data field,
>> >> and so on.
>> >>
>> >> As for the Add Device command, does bluetoothd currently use this somehow?
>> >
>> > Yes, this feature (action 0x02 for Add Device) takes over the entire LE
>> > background scanning and auto-connection mechanism from 3.17 kernel
>> > onwards since BlueZ 5.22 (which was mentioned in our release notes btw).
>> > This is much better than the active scanning based auto-connections that
>> > have until now been done using Start/Stop Discovery.
>>
>> Thanks for pointing that!
>> The release note says that:
>> "The kernel will even use the controller-side whitelist during
>> scanning (if no Resolvable Private Addresses are present), saving even
>> more power."
>> If I understand well, that mean that my peripheral device need to be
>> whitelisted in order for me to get advertisements ?
>
> The controller-side whitelist just means that the controller only wakes
> us up and gives us advertisements for entries in the whitelist. When not
> using the whitelist the kernel gets woken up and needs to do itself the
> filtering out of unknown devices. The kernel uses the controller-side
> whitelist whenever possible, but e.g. when we have resolvable private
> addresses in the list we can't use it as we can't know the address the
> remote device will use in advance.
>
>> My use case include scanning for non-paired, non-connected devices,
>> checking RSSI value and deciding about connection based on
>> advertisement content, so that won't work. Am I right ?
>
> If you know the bdaddr that you're interested in in advance we should be
> able to tweak the Add Device command to do what you want (e.g. disable
> duplicate filtering in the case of Action=0x00 entries). However, if
> what you need is general scanning for any devices (also previously
> unknown ones) then Add Device doesn't really fit in (since it takes a
> known bdaddr). For that we'd either need to consider allowing BDADDR_ANY
> as an accepted parameter or then we'd need to define a new mgmt command.
>

Unfortunately my case include previously unknown devices. I decide
wether I should connect (without pairing) based on advertised service
and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
add some parameter that will force not using controller-side whitelist
for scan?

> Johan

Re: [RFC] need for new scan api for bluetooth low energy.

[Johan Hedberg]

Hi Jakub,

On Wed, Sep 17, 2014, Jakub Pawlowski wrote:
> >> My use case include scanning for non-paired, non-connected devices,
> >> checking RSSI value and deciding about connection based on
> >> advertisement content, so that won't work. Am I right ?
> >
> > If you know the bdaddr that you're interested in in advance we should be
> > able to tweak the Add Device command to do what you want (e.g. disable
> > duplicate filtering in the case of Action=0x00 entries). However, if
> > what you need is general scanning for any devices (also previously
> > unknown ones) then Add Device doesn't really fit in (since it takes a
> > known bdaddr). For that we'd either need to consider allowing BDADDR_ANY
> > as an accepted parameter or then we'd need to define a new mgmt command.
> >
> 
> Unfortunately my case include previously unknown devices. I decide
> wether I should connect (without pairing) based on advertised service
> and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
> add some parameter that will force not using controller-side whitelist
> for scan?

With the BDADDR_ANY solution that'd be the trigger not to use the
controller-side white list. Simply disabling the controller side white
list isn't enough since the kernel would still filter out unknown
addresses and never send Device Found mgmt events.

Johan

Re: [RFC] need for new scan api for bluetooth low energy.

[Johan Hedberg]

On Wed, Sep 17, 2014, Johan Hedberg wrote:
> On Wed, Sep 17, 2014, Jakub Pawlowski wrote:
> > >> My use case include scanning for non-paired, non-connected devices,
> > >> checking RSSI value and deciding about connection based on
> > >> advertisement content, so that won't work. Am I right ?
> > >
> > > If you know the bdaddr that you're interested in in advance we should be
> > > able to tweak the Add Device command to do what you want (e.g. disable
> > > duplicate filtering in the case of Action=0x00 entries). However, if
> > > what you need is general scanning for any devices (also previously
> > > unknown ones) then Add Device doesn't really fit in (since it takes a
> > > known bdaddr). For that we'd either need to consider allowing BDADDR_ANY
> > > as an accepted parameter or then we'd need to define a new mgmt command.
> > >
> > 
> > Unfortunately my case include previously unknown devices. I decide
> > wether I should connect (without pairing) based on advertised service
> > and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
> > add some parameter that will force not using controller-side whitelist
> > for scan?
> 
> With the BDADDR_ANY solution that'd be the trigger not to use the
> controller-side white list. Simply disabling the controller side white
> list isn't enough since the kernel would still filter out unknown
> addresses and never send Device Found mgmt events.

Another question is do you guys want active scanning or passive
scanning? Right now Add Device is only used for passive scanning whereas
Start Discovery does active scanning. If you need also the scan response
data from devices (where some may put their UUIDs) then you wouldn't get
that with passive scanning.

If you need active scanning for any device it might be worth considering
sticking to the Start Discovery command and consider changing the kernel
to not use duplicate filtering when user space requests to only scan for
LE devices. It'd first need to be confirmed however that this whole
duplicate filtering parameter is an issue at all (since it has felt more
like speculation so far).

Johan

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

>>>>>> have you looked at the the Add Device management command with the
>>>>>> 0x00 Action. That will just send out Device Found events for devices
>>>>>> in range and uses passive scanning.
>>>>>> 
>>>>>> In addition if you are paired with a device and actually want to
>>>>>> connect to it, you can use Action 0x02 to allow for background
>>>>>> connection. This is currently in use for LE HID devices.
>>>>> 
>>>>> This still only sends one Device Found event though right? In Jakub's
>>>>> case, a mgmt event is desired to let the userspace know every time a
>>>>> new advertisement packet is received, with the updated RSSI and
>>>>> TxPower and so on.
>>>> 
>>>> It should send as many Device Found events as there are advertising
>>>> reports from the controller. We're enabling duplicate filtering in the
>>>> hci_req_add_le_passive_scan() function in hci_core.c so maybe that's
>>>> your issue? Some HW (like Broadcom) tend to be more conservative in how
>>>> many events they send when filtering is enabled whereas others (like
>>>> CSR) will give you many more of them (at least one whenever RSSI
>>>> changes).
>>> 
>>> Why is duplicate filtering being enabled ? Does it filter packets that
>>> have different advertisement content ? Can I turn it off ?
>> 
>> At least in the HCI section of the core spec this parameter is quite
>> vaguely defined. We've enabled it to save power by avoiding the
>> controller waking up the host too often. With auto-connections it makes
>> sense to have it enabled since we're really only interested in the first
>> event which acts as a trigger to connect. Same goes for device discovery
>> when you're interested in setting up a new device for use.
>> 
>> Right now the filtering is hard-coded but we could consider making it
>> conditional to whether there are any entries with action 0x00 ("scan but
>> don't connect") and disable the filtering in that case.
>> 
>>>>> I think this ties in with some of the previous conversations on the
>>>>> list about adding a device scan API that is geared towards
>>>>> applications (rather than OS UI). That is, not only a mgmt API that
>>>>> enables these kinds of use cases but also (probably) a high-level API
>>>>> exposed from bluetoothd that allows applications to scan for devices
>>>>> filtered by service UUIDs, contents of the manufacturer data field,
>>>>> and so on.
>>>>> 
>>>>> As for the Add Device command, does bluetoothd currently use this somehow?
>>>> 
>>>> Yes, this feature (action 0x02 for Add Device) takes over the entire LE
>>>> background scanning and auto-connection mechanism from 3.17 kernel
>>>> onwards since BlueZ 5.22 (which was mentioned in our release notes btw).
>>>> This is much better than the active scanning based auto-connections that
>>>> have until now been done using Start/Stop Discovery.
>>> 
>>> Thanks for pointing that!
>>> The release note says that:
>>> "The kernel will even use the controller-side whitelist during
>>> scanning (if no Resolvable Private Addresses are present), saving even
>>> more power."
>>> If I understand well, that mean that my peripheral device need to be
>>> whitelisted in order for me to get advertisements ?
>> 
>> The controller-side whitelist just means that the controller only wakes
>> us up and gives us advertisements for entries in the whitelist. When not
>> using the whitelist the kernel gets woken up and needs to do itself the
>> filtering out of unknown devices. The kernel uses the controller-side
>> whitelist whenever possible, but e.g. when we have resolvable private
>> addresses in the list we can't use it as we can't know the address the
>> remote device will use in advance.
>> 
>>> My use case include scanning for non-paired, non-connected devices,
>>> checking RSSI value and deciding about connection based on
>>> advertisement content, so that won't work. Am I right ?
>> 
>> If you know the bdaddr that you're interested in in advance we should be
>> able to tweak the Add Device command to do what you want (e.g. disable
>> duplicate filtering in the case of Action=0x00 entries). However, if
>> what you need is general scanning for any devices (also previously
>> unknown ones) then Add Device doesn't really fit in (since it takes a
>> known bdaddr). For that we'd either need to consider allowing BDADDR_ANY
>> as an accepted parameter or then we'd need to define a new mgmt command.
>> 
> 
> Unfortunately my case include previously unknown devices. I decide
> wether I should connect (without pairing) based on advertised service
> and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
> add some parameter that will force not using controller-side whitelist
> for scan?

actually BDADDR_ANY does not really look like a good solution. What we really want is to be able to define a list of UUIDs and maybe an RSSI range or threshold for this.

The real problem is that controllers behave so different when you talk about advertising event report. So even if you have duplicate filtering on or off they have different behaviors. And that is where the real challenge comes into play here. What works on one controller just fine, might turn into a huge battery drain on the other.

Get a few different controllers, Marvell, Intel, CSR and Broadcom and play with hcitool lescan to see the different behavior they provide. You can also write your own small little HCI program using the HCI User Channel feature we added in 3.13 and use src/shared/hci.c as helper for handling HCI commands. You really need to understand what you are getting yourself into here.

My main concern is battery drain. Once we figured out on how to avoid that, we can easily design a new mgmt command that allows you to do exactly what you want to do.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

On Wed, Sep 17, 2014 at 7:03 AM, Marcel Holtmann <marcel@holtmann.org> wrote:
> Hi Jakub,
>
>>>>>>> have you looked at the the Add Device management command with the
>>>>>>> 0x00 Action. That will just send out Device Found events for devices
>>>>>>> in range and uses passive scanning.
>>>>>>>
>>>>>>> In addition if you are paired with a device and actually want to
>>>>>>> connect to it, you can use Action 0x02 to allow for background
>>>>>>> connection. This is currently in use for LE HID devices.
>>>>>>
>>>>>> This still only sends one Device Found event though right? In Jakub's
>>>>>> case, a mgmt event is desired to let the userspace know every time a
>>>>>> new advertisement packet is received, with the updated RSSI and
>>>>>> TxPower and so on.
>>>>>
>>>>> It should send as many Device Found events as there are advertising
>>>>> reports from the controller. We're enabling duplicate filtering in the
>>>>> hci_req_add_le_passive_scan() function in hci_core.c so maybe that's
>>>>> your issue? Some HW (like Broadcom) tend to be more conservative in how
>>>>> many events they send when filtering is enabled whereas others (like
>>>>> CSR) will give you many more of them (at least one whenever RSSI
>>>>> changes).
>>>>
>>>> Why is duplicate filtering being enabled ? Does it filter packets that
>>>> have different advertisement content ? Can I turn it off ?
>>>
>>> At least in the HCI section of the core spec this parameter is quite
>>> vaguely defined. We've enabled it to save power by avoiding the
>>> controller waking up the host too often. With auto-connections it makes
>>> sense to have it enabled since we're really only interested in the first
>>> event which acts as a trigger to connect. Same goes for device discovery
>>> when you're interested in setting up a new device for use.
>>>
>>> Right now the filtering is hard-coded but we could consider making it
>>> conditional to whether there are any entries with action 0x00 ("scan but
>>> don't connect") and disable the filtering in that case.
>>>
>>>>>> I think this ties in with some of the previous conversations on the
>>>>>> list about adding a device scan API that is geared towards
>>>>>> applications (rather than OS UI). That is, not only a mgmt API that
>>>>>> enables these kinds of use cases but also (probably) a high-level API
>>>>>> exposed from bluetoothd that allows applications to scan for devices
>>>>>> filtered by service UUIDs, contents of the manufacturer data field,
>>>>>> and so on.
>>>>>>
>>>>>> As for the Add Device command, does bluetoothd currently use this somehow?
>>>>>
>>>>> Yes, this feature (action 0x02 for Add Device) takes over the entire LE
>>>>> background scanning and auto-connection mechanism from 3.17 kernel
>>>>> onwards since BlueZ 5.22 (which was mentioned in our release notes btw).
>>>>> This is much better than the active scanning based auto-connections that
>>>>> have until now been done using Start/Stop Discovery.
>>>>
>>>> Thanks for pointing that!
>>>> The release note says that:
>>>> "The kernel will even use the controller-side whitelist during
>>>> scanning (if no Resolvable Private Addresses are present), saving even
>>>> more power."
>>>> If I understand well, that mean that my peripheral device need to be
>>>> whitelisted in order for me to get advertisements ?
>>>
>>> The controller-side whitelist just means that the controller only wakes
>>> us up and gives us advertisements for entries in the whitelist. When not
>>> using the whitelist the kernel gets woken up and needs to do itself the
>>> filtering out of unknown devices. The kernel uses the controller-side
>>> whitelist whenever possible, but e.g. when we have resolvable private
>>> addresses in the list we can't use it as we can't know the address the
>>> remote device will use in advance.
>>>
>>>> My use case include scanning for non-paired, non-connected devices,
>>>> checking RSSI value and deciding about connection based on
>>>> advertisement content, so that won't work. Am I right ?
>>>
>>> If you know the bdaddr that you're interested in in advance we should be
>>> able to tweak the Add Device command to do what you want (e.g. disable
>>> duplicate filtering in the case of Action=0x00 entries). However, if
>>> what you need is general scanning for any devices (also previously
>>> unknown ones) then Add Device doesn't really fit in (since it takes a
>>> known bdaddr). For that we'd either need to consider allowing BDADDR_ANY
>>> as an accepted parameter or then we'd need to define a new mgmt command.
>>>
>>
>> Unfortunately my case include previously unknown devices. I decide
>> wether I should connect (without pairing) based on advertised service
>> and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
>> add some parameter that will force not using controller-side whitelist
>> for scan?
>
> actually BDADDR_ANY does not really look like a good solution. What we really want is to be able to define a list of UUIDs and maybe an RSSI range or threshold for this.

I think there should be option in mgmt api to capture all
advertisement packets, and later on application level decide what
property to filter on. I understand that this might be not energy
efficient, and that's why there is whitelisting and controller level
filtering.
Specifying list of UUIDs or RSSI value as parameter will be nice, but
controllers doesn't support this kind of complex filtering right now
anyway.
If you decide to allow UUID and RSSI filtering, maybe in a while
someone will come up with need to filter by something in advertisement
? Maybe it would be good to have this 'look for advertisements' API
very general.

>
> The real problem is that controllers behave so different when you talk about advertising event report. So even if you have duplicate filtering on or off they have different behaviors. And that is where the real challenge comes into play here. What works on one controller just fine, might turn into a huge battery drain on the other.
>

Thanks for pointing that out, I've already heard that argument from
Scott before, and it'll require more investigation.

> Get a few different controllers, Marvell, Intel, CSR and Broadcom and play with hcitool lescan to see the different behavior they provide. You can also write your own small little HCI program using the HCI User Channel feature we added in 3.13 and use src/shared/hci.c as helper for handling HCI commands. You really need to understand what you are getting yourself into here.

I've already implemented my functionality first using BLED112 dongle,
and then on linux using HCI api and tested it on two controllers, but
from what I understood MGMT is the  API of future. Also I want to make
it work for all controllers, not only those three.

>
> My main concern is battery drain. Once we figured out on how to avoid that, we can easily design a new mgmt command that allows you to do exactly what you want to do.

>From what I understand as long as controllers don't support
sophisticated filtering, that would be energy conuming. I would also
preffer to find a energy-friendly way, but is it better not to have
this functionality rather than have it in non-energy consuming way ?

>
> Regards
>
> Marcel
>

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

>>>>>>>> have you looked at the the Add Device management command with the
>>>>>>>> 0x00 Action. That will just send out Device Found events for devices
>>>>>>>> in range and uses passive scanning.
>>>>>>>> 
>>>>>>>> In addition if you are paired with a device and actually want to
>>>>>>>> connect to it, you can use Action 0x02 to allow for background
>>>>>>>> connection. This is currently in use for LE HID devices.
>>>>>>> 
>>>>>>> This still only sends one Device Found event though right? In Jakub's
>>>>>>> case, a mgmt event is desired to let the userspace know every time a
>>>>>>> new advertisement packet is received, with the updated RSSI and
>>>>>>> TxPower and so on.
>>>>>> 
>>>>>> It should send as many Device Found events as there are advertising
>>>>>> reports from the controller. We're enabling duplicate filtering in the
>>>>>> hci_req_add_le_passive_scan() function in hci_core.c so maybe that's
>>>>>> your issue? Some HW (like Broadcom) tend to be more conservative in how
>>>>>> many events they send when filtering is enabled whereas others (like
>>>>>> CSR) will give you many more of them (at least one whenever RSSI
>>>>>> changes).
>>>>> 
>>>>> Why is duplicate filtering being enabled ? Does it filter packets that
>>>>> have different advertisement content ? Can I turn it off ?
>>>> 
>>>> At least in the HCI section of the core spec this parameter is quite
>>>> vaguely defined. We've enabled it to save power by avoiding the
>>>> controller waking up the host too often. With auto-connections it makes
>>>> sense to have it enabled since we're really only interested in the first
>>>> event which acts as a trigger to connect. Same goes for device discovery
>>>> when you're interested in setting up a new device for use.
>>>> 
>>>> Right now the filtering is hard-coded but we could consider making it
>>>> conditional to whether there are any entries with action 0x00 ("scan but
>>>> don't connect") and disable the filtering in that case.
>>>> 
>>>>>>> I think this ties in with some of the previous conversations on the
>>>>>>> list about adding a device scan API that is geared towards
>>>>>>> applications (rather than OS UI). That is, not only a mgmt API that
>>>>>>> enables these kinds of use cases but also (probably) a high-level API
>>>>>>> exposed from bluetoothd that allows applications to scan for devices
>>>>>>> filtered by service UUIDs, contents of the manufacturer data field,
>>>>>>> and so on.
>>>>>>> 
>>>>>>> As for the Add Device command, does bluetoothd currently use this somehow?
>>>>>> 
>>>>>> Yes, this feature (action 0x02 for Add Device) takes over the entire LE
>>>>>> background scanning and auto-connection mechanism from 3.17 kernel
>>>>>> onwards since BlueZ 5.22 (which was mentioned in our release notes btw).
>>>>>> This is much better than the active scanning based auto-connections that
>>>>>> have until now been done using Start/Stop Discovery.
>>>>> 
>>>>> Thanks for pointing that!
>>>>> The release note says that:
>>>>> "The kernel will even use the controller-side whitelist during
>>>>> scanning (if no Resolvable Private Addresses are present), saving even
>>>>> more power."
>>>>> If I understand well, that mean that my peripheral device need to be
>>>>> whitelisted in order for me to get advertisements ?
>>>> 
>>>> The controller-side whitelist just means that the controller only wakes
>>>> us up and gives us advertisements for entries in the whitelist. When not
>>>> using the whitelist the kernel gets woken up and needs to do itself the
>>>> filtering out of unknown devices. The kernel uses the controller-side
>>>> whitelist whenever possible, but e.g. when we have resolvable private
>>>> addresses in the list we can't use it as we can't know the address the
>>>> remote device will use in advance.
>>>> 
>>>>> My use case include scanning for non-paired, non-connected devices,
>>>>> checking RSSI value and deciding about connection based on
>>>>> advertisement content, so that won't work. Am I right ?
>>>> 
>>>> If you know the bdaddr that you're interested in in advance we should be
>>>> able to tweak the Add Device command to do what you want (e.g. disable
>>>> duplicate filtering in the case of Action=0x00 entries). However, if
>>>> what you need is general scanning for any devices (also previously
>>>> unknown ones) then Add Device doesn't really fit in (since it takes a
>>>> known bdaddr). For that we'd either need to consider allowing BDADDR_ANY
>>>> as an accepted parameter or then we'd need to define a new mgmt command.
>>>> 
>>> 
>>> Unfortunately my case include previously unknown devices. I decide
>>> wether I should connect (without pairing) based on advertised service
>>> and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
>>> add some parameter that will force not using controller-side whitelist
>>> for scan?
>> 
>> actually BDADDR_ANY does not really look like a good solution. What we really want is to be able to define a list of UUIDs and maybe an RSSI range or threshold for this.
> 
> I think there should be option in mgmt api to capture all
> advertisement packets, and later on application level decide what
> property to filter on. I understand that this might be not energy
> efficient, and that's why there is whitelisting and controller level
> filtering.
> Specifying list of UUIDs or RSSI value as parameter will be nice, but
> controllers doesn't support this kind of complex filtering right now
> anyway.
> If you decide to allow UUID and RSSI filtering, maybe in a while
> someone will come up with need to filter by something in advertisement
> ? Maybe it would be good to have this 'look for advertisements' API
> very general.

we can always extend the mgmt API with a new command for filtering something new. And yes, the Bluetooth SIG will define a new AD element type and then it is not supported. There are also many new Bluetooth specifications in progress and also planned (which I will not iterate here since they are Buetooth SIG confidential) which extend LE advertising in various form and shapes.

For me it is important that we can do certain things in the kernel so that userspace can keep sleeping. And we really can just add extra mgmt commands to allow for this. Remember that kernel will have to disable certain "background" behavior anymore. An active discovery always comes first. An active connection establishment from a socket comes first. Pairing a device comes first. So all these major events that are driven by an user action will be prioritized. We have to do that since otherwise you end up with bad user experience. But then again, draining the battery is also bad user experience. So we need to find a balance here.

>> 
>> The real problem is that controllers behave so different when you talk about advertising event report. So even if you have duplicate filtering on or off they have different behaviors. And that is where the real challenge comes into play here. What works on one controller just fine, might turn into a huge battery drain on the other.
>> 
> 
> Thanks for pointing that out, I've already heard that argument from
> Scott before, and it'll require more investigation.
> 
>> Get a few different controllers, Marvell, Intel, CSR and Broadcom and play with hcitool lescan to see the different behavior they provide. You can also write your own small little HCI program using the HCI User Channel feature we added in 3.13 and use src/shared/hci.c as helper for handling HCI commands. You really need to understand what you are getting yourself into here.
> 
> I've already implemented my functionality first using BLED112 dongle,
> and then on linux using HCI api and tested it on two controllers, but
> from what I understood MGMT is the  API of future. Also I want to make
> it work for all controllers, not only those three.

This is not for an official API. You can not use HCI User Channel and mgmt at the time. They are mutually exclusive. Actually when opening a HCI User Channel you will see that the controller index disappears from the mgmt interface.

What I am saying is that you need to get yourself some understanding on how certain controllers from certain manufactures behave if you for example turn on/off the duplicate filtering. And then also have a rather busy LE environment with a bunch of devices advertising. Get a few beacons that are constant advertising and get a feel on what is going on on HCI. You will quickly see that we have to be smarter than just keep scanning. And for kicks, try to use BR/EDR for say A2DP at the same time. See how that goes.

>> 
>> My main concern is battery drain. Once we figured out on how to avoid that, we can easily design a new mgmt command that allows you to do exactly what you want to do.
> 
> From what I understand as long as controllers don't support
> sophisticated filtering, that would be energy conuming. I would also
> preffer to find a energy-friendly way, but is it better not to have
> this functionality rather than have it in non-energy consuming way ?

Even if we consume extra energy on older Bluetooth chips or certain manufactures, we want to try to not waste it. If it drains your battery, people will just disable Bluetooth and then you have not won anything.

Maybe what this really needs is that we go into the Bluetooth SIG and propose controller based filtering with standard HCI commands.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

[-- Attachment #1: Type: text/plain, Size: 11105 bytes --]

On Thu, Sep 18, 2014 at 8:37 AM, Marcel Holtmann <marcel@holtmann.org>
wrote:

> Hi Jakub,
>
> >>>>>>>> have you looked at the the Add Device management command with the
> >>>>>>>> 0x00 Action. That will just send out Device Found events for
> devices
> >>>>>>>> in range and uses passive scanning.
> >>>>>>>>
> >>>>>>>> In addition if you are paired with a device and actually want to
> >>>>>>>> connect to it, you can use Action 0x02 to allow for background
> >>>>>>>> connection. This is currently in use for LE HID devices.
> >>>>>>>
> >>>>>>> This still only sends one Device Found event though right? In
> Jakub's
> >>>>>>> case, a mgmt event is desired to let the userspace know every time
> a
> >>>>>>> new advertisement packet is received, with the updated RSSI and
> >>>>>>> TxPower and so on.
> >>>>>>
> >>>>>> It should send as many Device Found events as there are advertising
> >>>>>> reports from the controller. We're enabling duplicate filtering in
> the
> >>>>>> hci_req_add_le_passive_scan() function in hci_core.c so maybe that's
> >>>>>> your issue? Some HW (like Broadcom) tend to be more conservative in
> how
> >>>>>> many events they send when filtering is enabled whereas others (like
> >>>>>> CSR) will give you many more of them (at least one whenever RSSI
> >>>>>> changes).
> >>>>>
> >>>>> Why is duplicate filtering being enabled ? Does it filter packets
> that
> >>>>> have different advertisement content ? Can I turn it off ?
> >>>>
> >>>> At least in the HCI section of the core spec this parameter is quite
> >>>> vaguely defined. We've enabled it to save power by avoiding the
> >>>> controller waking up the host too often. With auto-connections it
> makes
> >>>> sense to have it enabled since we're really only interested in the
> first
> >>>> event which acts as a trigger to connect. Same goes for device
> discovery
> >>>> when you're interested in setting up a new device for use.
> >>>>
> >>>> Right now the filtering is hard-coded but we could consider making it
> >>>> conditional to whether there are any entries with action 0x00 ("scan
> but
> >>>> don't connect") and disable the filtering in that case.
> >>>>
> >>>>>>> I think this ties in with some of the previous conversations on the
> >>>>>>> list about adding a device scan API that is geared towards
> >>>>>>> applications (rather than OS UI). That is, not only a mgmt API that
> >>>>>>> enables these kinds of use cases but also (probably) a high-level
> API
> >>>>>>> exposed from bluetoothd that allows applications to scan for
> devices
> >>>>>>> filtered by service UUIDs, contents of the manufacturer data field,
> >>>>>>> and so on.
> >>>>>>>
> >>>>>>> As for the Add Device command, does bluetoothd currently use this
> somehow?
> >>>>>>
> >>>>>> Yes, this feature (action 0x02 for Add Device) takes over the
> entire LE
> >>>>>> background scanning and auto-connection mechanism from 3.17 kernel
> >>>>>> onwards since BlueZ 5.22 (which was mentioned in our release notes
> btw).
> >>>>>> This is much better than the active scanning based auto-connections
> that
> >>>>>> have until now been done using Start/Stop Discovery.
> >>>>>
> >>>>> Thanks for pointing that!
> >>>>> The release note says that:
> >>>>> "The kernel will even use the controller-side whitelist during
> >>>>> scanning (if no Resolvable Private Addresses are present), saving
> even
> >>>>> more power."
> >>>>> If I understand well, that mean that my peripheral device need to be
> >>>>> whitelisted in order for me to get advertisements ?
> >>>>
> >>>> The controller-side whitelist just means that the controller only
> wakes
> >>>> us up and gives us advertisements for entries in the whitelist. When
> not
> >>>> using the whitelist the kernel gets woken up and needs to do itself
> the
> >>>> filtering out of unknown devices. The kernel uses the controller-side
> >>>> whitelist whenever possible, but e.g. when we have resolvable private
> >>>> addresses in the list we can't use it as we can't know the address the
> >>>> remote device will use in advance.
> >>>>
> >>>>> My use case include scanning for non-paired, non-connected devices,
> >>>>> checking RSSI value and deciding about connection based on
> >>>>> advertisement content, so that won't work. Am I right ?
> >>>>
> >>>> If you know the bdaddr that you're interested in in advance we should
> be
> >>>> able to tweak the Add Device command to do what you want (e.g. disable
> >>>> duplicate filtering in the case of Action=0x00 entries). However, if
> >>>> what you need is general scanning for any devices (also previously
> >>>> unknown ones) then Add Device doesn't really fit in (since it takes a
> >>>> known bdaddr). For that we'd either need to consider allowing
> BDADDR_ANY
> >>>> as an accepted parameter or then we'd need to define a new mgmt
> command.
> >>>>
> >>>
> >>> Unfortunately my case include previously unknown devices. I decide
> >>> wether I should connect (without pairing) based on advertised service
> >>> and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
> >>> add some parameter that will force not using controller-side whitelist
> >>> for scan?
> >>
> >> actually BDADDR_ANY does not really look like a good solution. What we
> really want is to be able to define a list of UUIDs and maybe an RSSI range
> or threshold for this.
> >
> > I think there should be option in mgmt api to capture all
> > advertisement packets, and later on application level decide what
> > property to filter on. I understand that this might be not energy
> > efficient, and that's why there is whitelisting and controller level
> > filtering.
> > Specifying list of UUIDs or RSSI value as parameter will be nice, but
> > controllers doesn't support this kind of complex filtering right now
> > anyway.
> > If you decide to allow UUID and RSSI filtering, maybe in a while
> > someone will come up with need to filter by something in advertisement
> > ? Maybe it would be good to have this 'look for advertisements' API
> > very general.
>
> we can always extend the mgmt API with a new command for filtering
> something new. And yes, the Bluetooth SIG will define a new AD element type
> and then it is not supported. There are also many new Bluetooth
> specifications in progress and also planned (which I will not iterate here
> since they are Buetooth SIG confidential) which extend LE advertising in
> various form and shapes.
>
> For me it is important that we can do certain things in the kernel so that
> userspace can keep sleeping. And we really can just add extra mgmt commands
> to allow for this. Remember that kernel will have to disable certain
> "background" behavior anymore. An active discovery always comes first. An
> active connection establishment from a socket comes first. Pairing a device
> comes first. So all these major events that are driven by an user action
> will be prioritized. We have to do that since otherwise you end up with bad
> user experience. But then again, draining the battery is also bad user
> experience. So we need to find a balance here.
>

I will be very happy with mgmt api method that will filter by serice uuid
and RSSI value, and that's enough for me. From my use case it looks like
the user is going to be in this scan state for short time - few seconds,
maximum minute.

If it would be possible to also add general le scan method that would
return all advertisements, marked as "only for development", and "not for
use in production", and "behaves differently for different controllers" and
maybe somehow disabled by default and you need to set some special flag to
enable it on your developer machine then it would be awsome for developers
that want to play with it, but it's not my requirement.


>
> >>
> >> The real problem is that controllers behave so different when you talk
> about advertising event report. So even if you have duplicate filtering on
> or off they have different behaviors. And that is where the real challenge
> comes into play here. What works on one controller just fine, might turn
> into a huge battery drain on the other.
> >>
> >
> > Thanks for pointing that out, I've already heard that argument from
> > Scott before, and it'll require more investigation.
> >
> >> Get a few different controllers, Marvell, Intel, CSR and Broadcom and
> play with hcitool lescan to see the different behavior they provide. You
> can also write your own small little HCI program using the HCI User Channel
> feature we added in 3.13 and use src/shared/hci.c as helper for handling
> HCI commands. You really need to understand what you are getting yourself
> into here.
> >
> > I've already implemented my functionality first using BLED112 dongle,
> > and then on linux using HCI api and tested it on two controllers, but
> > from what I understood MGMT is the  API of future. Also I want to make
> > it work for all controllers, not only those three.
>
> This is not for an official API. You can not use HCI User Channel and mgmt
> at the time. They are mutually exclusive. Actually when opening a HCI User
> Channel you will see that the controller index disappears from the mgmt
> interface.
>

I was forced to use HCI, because mgmt didn't exposed required functionality
(the scan I want to add). And I want my code to work in the future without
causing the controller to disapear from mgmt.


>
> What I am saying is that you need to get yourself some understanding on
> how certain controllers from certain manufactures behave if you for example
> turn on/off the duplicate filtering. And then also have a rather busy LE
> environment with a bunch of devices advertising. Get a few beacons that are
> constant advertising and get a feel on what is going on on HCI. You will
> quickly see that we have to be smarter than just keep scanning. And for
> kicks, try to use BR/EDR for say A2DP at the same time. See how that goes.
>


>
> >>
> >> My main concern is battery drain. Once we figured out on how to avoid
> that, we can easily design a new mgmt command that allows you to do exactly
> what you want to do.
> >
> > From what I understand as long as controllers don't support
> > sophisticated filtering, that would be energy conuming. I would also
> > preffer to find a energy-friendly way, but is it better not to have
> > this functionality rather than have it in non-energy consuming way ?
>
> Even if we consume extra energy on older Bluetooth chips or certain
> manufactures, we want to try to not waste it. If it drains your battery,
> people will just disable Bluetooth and then you have not won anything.
>
> Maybe what this really needs is that we go into the Bluetooth SIG and
> propose controller based filtering with standard HCI commands.
>

Controller based filtering would be awsome, but it would probably take 2-3
years to get first controller that do it, and I hope to be able to ship my
functionality in maybe 2-3 months.

>From current discussion it seems that it's achievable to implement it in
kernel right now, and eventually talk with manufacturers to make it
standard in future.


> Regards
>
> Marcel
>
>

[-- Attachment #2: Type: text/html, Size: 14374 bytes --]

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

I will be very happy with mgmt api method that will filter by serice
uuid and RSSI value, and that's enough for me. From my use case it
looks like the user is going to be in this scan state for short time -
few seconds, maximum minute.

If it would be possible to also add general le scan method that would
return all advertisements, marked as "only for development", and "not
for use in production", and "behaves differently for different
controllers" and maybe somehow disabled by default and you need to set
some special flag to enable it on your developer machine then it would
be awsome for developers that want to play with it, but it's not my
requirement.

>
>
> >>
> >> The real problem is that controllers behave so different when you talk=
 about advertising event report. So even if you have duplicate filtering on=
 or off they have different behaviors. And that is where the real challenge=
 comes into play here. What works on one controller just fine, might turn i=
nto a huge battery drain on the other.
> >>
> >
> > Thanks for pointing that out, I've already heard that argument from
> > Scott before, and it'll require more investigation.
> >
> >> Get a few different controllers, Marvell, Intel, CSR and Broadcom and =
play with hcitool lescan to see the different behavior they provide. You ca=
n also write your own small little HCI program using the HCI User Channel f=
eature we added in 3.13 and use src/shared/hci.c as helper for handling HCI=
 commands. You really need to understand what you are getting yourself into=
 here.
> >
> > I've already implemented my functionality first using BLED112 dongle,
> > and then on linux using HCI api and tested it on two controllers, but
> > from what I understood MGMT is the  API of future. Also I want to make
> > it work for all controllers, not only those three.
>
> This is not for an official API. You can not use HCI User Channel and mgm=
t at the time. They are mutually exclusive. Actually when opening a HCI Use=
r Channel you will see that the controller index disappears from the mgmt i=
nterface.


I was forced to use HCI, because mgmt didn't exposed required
functionality (the scan I want to add). And I want my code to work in
the future without causing the controller to disapear from mgmt.

>
>
> What I am saying is that you need to get yourself some understanding on h=
ow certain controllers from certain manufactures behave if you for example =
turn on/off the duplicate filtering. And then also have a rather busy LE en=
vironment with a bunch of devices advertising. Get a few beacons that are c=
onstant advertising and get a feel on what is going on on HCI. You will qui=
ckly see that we have to be smarter than just keep scanning. And for kicks,=
 try to use BR/EDR for say A2DP at the same time. See how that goes.


>
>
> >>
> >> My main concern is battery drain. Once we figured out on how to avoid =
that, we can easily design a new mgmt command that allows you to do exactly=
 what you want to do.
> >
> > From what I understand as long as controllers don't support
> > sophisticated filtering, that would be energy conuming. I would also
> > preffer to find a energy-friendly way, but is it better not to have
> > this functionality rather than have it in non-energy consuming way ?
>
> Even if we consume extra energy on older Bluetooth chips or certain manuf=
actures, we want to try to not waste it. If it drains your battery, people =
will just disable Bluetooth and then you have not won anything.
>
> Maybe what this really needs is that we go into the Bluetooth SIG and pro=
pose controller based filtering with standard HCI commands.


Controller based filtering would be awsome, but it would probably take
2-3 years to get first controller that do it, and I hope to be able to
ship my functionality in maybe 2-3 months.

>From current discussion it seems that it's achievable to implement it
in kernel right now, and eventually talk with manufacturers to make it
standard in future.

>
> Regards
>
> Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

so you really need to follow the mailing list guidelines and disable HTML emails ;)

> I will be very happy with mgmt api method that will filter by serice
> uuid and RSSI value, and that's enough for me. From my use case it
> looks like the user is going to be in this scan state for short time -
> few seconds, maximum minute.

that is then something different. That is more like "scan for this UUID within this RSSI range for x amount of seconds, if any results, report them, and otherwise go back to idle".

We can talk about this one-shot kind of API. I would need to think about it a little bit more. However one thing that you might want to consider is using pathloss instead of RSSI. Especially if the advertisement contains the TX power. That way you get a way better idea of how close or far the device is.

> If it would be possible to also add general le scan method that would
> return all advertisements, marked as "only for development", and "not
> for use in production", and "behaves differently for different
> controllers" and maybe somehow disabled by default and you need to set
> some special flag to enable it on your developer machine then it would
> be awsome for developers that want to play with it, but it's not my
> requirement.

The mgmt API will never contain any development methods. It is an official API and we are committed to stable API promise here. Any kind of testing API has to go via debugfs.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

On Fri, Sep 19, 2014 at 5:35 PM, Marcel Holtmann <marcel@holtmann.org> wrote:
> Hi Jakub,
>
>> >>>>>>>> have you looked at the the Add Device management command with the
>> >>>>>>>> 0x00 Action. That will just send out Device Found events for
>> >>>>>>>> devices
>> >>>>>>>> in range and uses passive scanning.
>> >>>>>>>>
>> >>>>>>>> In addition if you are paired with a device and actually want to
>> >>>>>>>> connect to it, you can use Action 0x02 to allow for background
>> >>>>>>>> connection. This is currently in use for LE HID devices.
>> >>>>>>>
>> >>>>>>> This still only sends one Device Found event though right? In
>> >>>>>>> Jakub's
>> >>>>>>> case, a mgmt event is desired to let the userspace know every time
>> >>>>>>> a
>> >>>>>>> new advertisement packet is received, with the updated RSSI and
>> >>>>>>> TxPower and so on.
>> >>>>>>
>> >>>>>> It should send as many Device Found events as there are advertising
>> >>>>>> reports from the controller. We're enabling duplicate filtering in
>> >>>>>> the
>> >>>>>> hci_req_add_le_passive_scan() function in hci_core.c so maybe
>> >>>>>> that's
>> >>>>>> your issue? Some HW (like Broadcom) tend to be more conservative in
>> >>>>>> how
>> >>>>>> many events they send when filtering is enabled whereas others
>> >>>>>> (like
>> >>>>>> CSR) will give you many more of them (at least one whenever RSSI
>> >>>>>> changes).
>> >>>>>
>> >>>>> Why is duplicate filtering being enabled ? Does it filter packets
>> >>>>> that
>> >>>>> have different advertisement content ? Can I turn it off ?
>> >>>>
>> >>>> At least in the HCI section of the core spec this parameter is quite
>> >>>> vaguely defined. We've enabled it to save power by avoiding the
>> >>>> controller waking up the host too often. With auto-connections it
>> >>>> makes
>> >>>> sense to have it enabled since we're really only interested in the
>> >>>> first
>> >>>> event which acts as a trigger to connect. Same goes for device
>> >>>> discovery
>> >>>> when you're interested in setting up a new device for use.
>> >>>>
>> >>>> Right now the filtering is hard-coded but we could consider making it
>> >>>> conditional to whether there are any entries with action 0x00 ("scan
>> >>>> but
>> >>>> don't connect") and disable the filtering in that case.
>> >>>>
>> >>>>>>> I think this ties in with some of the previous conversations on
>> >>>>>>> the
>> >>>>>>> list about adding a device scan API that is geared towards
>> >>>>>>> applications (rather than OS UI). That is, not only a mgmt API
>> >>>>>>> that
>> >>>>>>> enables these kinds of use cases but also (probably) a high-level
>> >>>>>>> API
>> >>>>>>> exposed from bluetoothd that allows applications to scan for
>> >>>>>>> devices
>> >>>>>>> filtered by service UUIDs, contents of the manufacturer data
>> >>>>>>> field,
>> >>>>>>> and so on.
>> >>>>>>>
>> >>>>>>> As for the Add Device command, does bluetoothd currently use this
>> >>>>>>> somehow?
>> >>>>>>
>> >>>>>> Yes, this feature (action 0x02 for Add Device) takes over the
>> >>>>>> entire LE
>> >>>>>> background scanning and auto-connection mechanism from 3.17 kernel
>> >>>>>> onwards since BlueZ 5.22 (which was mentioned in our release notes
>> >>>>>> btw).
>> >>>>>> This is much better than the active scanning based auto-connections
>> >>>>>> that
>> >>>>>> have until now been done using Start/Stop Discovery.
>> >>>>>
>> >>>>> Thanks for pointing that!
>> >>>>> The release note says that:
>> >>>>> "The kernel will even use the controller-side whitelist during
>> >>>>> scanning (if no Resolvable Private Addresses are present), saving
>> >>>>> even
>> >>>>> more power."
>> >>>>> If I understand well, that mean that my peripheral device need to be
>> >>>>> whitelisted in order for me to get advertisements ?
>> >>>>
>> >>>> The controller-side whitelist just means that the controller only
>> >>>> wakes
>> >>>> us up and gives us advertisements for entries in the whitelist. When
>> >>>> not
>> >>>> using the whitelist the kernel gets woken up and needs to do itself
>> >>>> the
>> >>>> filtering out of unknown devices. The kernel uses the controller-side
>> >>>> whitelist whenever possible, but e.g. when we have resolvable private
>> >>>> addresses in the list we can't use it as we can't know the address
>> >>>> the
>> >>>> remote device will use in advance.
>> >>>>
>> >>>>> My use case include scanning for non-paired, non-connected devices,
>> >>>>> checking RSSI value and deciding about connection based on
>> >>>>> advertisement content, so that won't work. Am I right ?
>> >>>>
>> >>>> If you know the bdaddr that you're interested in in advance we should
>> >>>> be
>> >>>> able to tweak the Add Device command to do what you want (e.g.
>> >>>> disable
>> >>>> duplicate filtering in the case of Action=0x00 entries). However, if
>> >>>> what you need is general scanning for any devices (also previously
>> >>>> unknown ones) then Add Device doesn't really fit in (since it takes a
>> >>>> known bdaddr). For that we'd either need to consider allowing
>> >>>> BDADDR_ANY
>> >>>> as an accepted parameter or then we'd need to define a new mgmt
>> >>>> command.
>> >>>>
>> >>>
>> >>> Unfortunately my case include previously unknown devices. I decide
>> >>> wether I should connect (without pairing) based on advertised service
>> >>> and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
>> >>> add some parameter that will force not using controller-side whitelist
>> >>> for scan?
>> >>
>> >> actually BDADDR_ANY does not really look like a good solution. What we
>> >> really want is to be able to define a list of UUIDs and maybe an RSSI range
>> >> or threshold for this.
>> >
>> > I think there should be option in mgmt api to capture all
>> > advertisement packets, and later on application level decide what
>> > property to filter on. I understand that this might be not energy
>> > efficient, and that's why there is whitelisting and controller level
>> > filtering.
>> > Specifying list of UUIDs or RSSI value as parameter will be nice, but
>> > controllers doesn't support this kind of complex filtering right now
>> > anyway.
>> > If you decide to allow UUID and RSSI filtering, maybe in a while
>> > someone will come up with need to filter by something in advertisement
>> > ? Maybe it would be good to have this 'look for advertisements' API
>> > very general.
>>
>> we can always extend the mgmt API with a new command for filtering
>> something new. And yes, the Bluetooth SIG will define a new AD element type
>> and then it is not supported. There are also many new Bluetooth
>> specifications in progress and also planned (which I will not iterate here
>> since they are Buetooth SIG confidential) which extend LE advertising in
>> various form and shapes.
>>
>> For me it is important that we can do certain things in the kernel so that
>> userspace can keep sleeping. And we really can just add extra mgmt commands
>> to allow for this. Remember that kernel will have to disable certain
>> "background" behavior anymore. An active discovery always comes first. An
>> active connection establishment from a socket comes first. Pairing a device
>> comes first. So all these major events that are driven by an user action
>> will be prioritized. We have to do that since otherwise you end up with bad
>> user experience. But then again, draining the battery is also bad user
>> experience. So we need to find a balance here.
>
>
> I will be very happy with mgmt api method that will filter by serice uuid
> and RSSI value, and that's enough for me. From my use case it looks like the
> user is going to be in this scan state for short time - few seconds, maximum
> minute.
>
>
> that is then something different. That is more like "scan for this UUID
> within this RSSI range for x amount of seconds, if any results, report them,
> and otherwise go back to idle".
>
> We can talk about this one-shot kind of API. I would need to think about it
> a little bit more. However one thing that you might want to consider is
> using pathloss instead of RSSI. Especially if the advertisement contains the
> TX power. That way you get a way better idea of how close or far the device
> is.
>

Yay! Thanks!
Yes, pathloss will be much better, I was using it before.

> If it would be possible to also add general le scan method that would return
> all advertisements, marked as "only for development", and "not for use in
> production", and "behaves differently for different controllers" and maybe
> somehow disabled by default and you need to set some special flag to enable
> it on your developer machine then it would be awsome for developers that
> want to play with it, but it's not my requirement.
>
>
> The mgmt API will never contain any development methods. It is an official
> API and we are committed to stable API promise here. Any kind of testing API
> has to go via debugfs.
>
> Regards
>
> Marcel
>

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

---
 doc/mgmt-api.txt | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
index 11e2490..1c78f1c 100644
--- a/doc/mgmt-api.txt
+++ b/doc/mgmt-api.txt
@@ -2135,6 +2135,34 @@ Set Public Address Command
  Invalid Index


+Start LE Filtered Device Scan
+=======================
+
+ Command Code: 0x003A
+ Controller Index: <controller id>
+ Command Parameters: UUID (16 Octets)
+             max_pathloss (1 octet)
+             is_active (1 octet)
+
+ Return Parameters:
+
+ This command starts LE scan, and filter received advertisements: if
AD contains both TX power level and Service Solicitation, and UUID is
contained in Service Solicitation and computed pathloss is smaller
than max_pathloss parameter, then a Advertisement Received event will
be sent.
+
+ Possible values for the is_active parameter:
+ 0 Start passive scan
+ 1 Start active scan
+
+ This command can only be used when the controller is powered.
+
+ This command generates a Command Complete event on success or failure.
+
+ Possible errors: Busy
+ Not Supported
+ Invalid Parameters
+ Not Powered
+ Invalid Index
+
+
 Command Complete Event
 ======================

@@ -2851,3 +2879,44 @@ New Configuration Options Event

  This event indicates that one or more of the options for the
  controller configuration has changed.
+
+
+Advertisement Received Event
+==============================
+
+ Event Code: 0x0020
+ Controller Index: <controller id>
+ Event Parameters: Address (6 Octets)
+ Address_Type (1 Octet)
+ RSSI (1 Octet)
+ Flags (4 Octets)
+ AD_Data_Length (2 Octets)
+ AD_Data (0-65535 Octets)
+
+
+ This event indicates that we received event during LE Filtered Device Scan
+
+ Possible values for the Address_Type parameter:
+ 0 Reserved (not in use)
+ 1 LE Public
+ 2 LE Random
+
+ The following bits are defined for the Flags parameter:
+ 0 Reserved (not in use)
+ 1 Legacy Pairing
+ 2 Not Connectable
+
+ AD_Data contains AD if we're in passive scan mode, or EIR if we're
in active scan mode.
+
+ The Legacy Pairing flag indicates that Legacy Pairing is likely
+ to occur when pairing with this device. An application could use
+ this information to optimize the pairing process by locally
+ pre-generating a PIN code and thereby eliminate the risk of
+ local input timeout when pairing. Note that there is a risk of
+ false-positives for this flag so user space should be able to
+ handle getting something else as a PIN Request when pairing.
+
+ The Not Connectable flag indicates that the device will not
+ accept any connections. This can be indicated by Low Energy
+ devices that are in broadcaster role.
+
-- 
2.1.0.rc2.206.gedb03e5


On Fri, Sep 19, 2014 at 9:29 PM, Jakub Pawlowski <jpawlowski@google.com> wrote:
> On Fri, Sep 19, 2014 at 5:35 PM, Marcel Holtmann <marcel@holtmann.org> wrote:
>> Hi Jakub,
>>
>>> >>>>>>>> have you looked at the the Add Device management command with the
>>> >>>>>>>> 0x00 Action. That will just send out Device Found events for
>>> >>>>>>>> devices
>>> >>>>>>>> in range and uses passive scanning.
>>> >>>>>>>>
>>> >>>>>>>> In addition if you are paired with a device and actually want to
>>> >>>>>>>> connect to it, you can use Action 0x02 to allow for background
>>> >>>>>>>> connection. This is currently in use for LE HID devices.
>>> >>>>>>>
>>> >>>>>>> This still only sends one Device Found event though right? In
>>> >>>>>>> Jakub's
>>> >>>>>>> case, a mgmt event is desired to let the userspace know every time
>>> >>>>>>> a
>>> >>>>>>> new advertisement packet is received, with the updated RSSI and
>>> >>>>>>> TxPower and so on.
>>> >>>>>>
>>> >>>>>> It should send as many Device Found events as there are advertising
>>> >>>>>> reports from the controller. We're enabling duplicate filtering in
>>> >>>>>> the
>>> >>>>>> hci_req_add_le_passive_scan() function in hci_core.c so maybe
>>> >>>>>> that's
>>> >>>>>> your issue? Some HW (like Broadcom) tend to be more conservative in
>>> >>>>>> how
>>> >>>>>> many events they send when filtering is enabled whereas others
>>> >>>>>> (like
>>> >>>>>> CSR) will give you many more of them (at least one whenever RSSI
>>> >>>>>> changes).
>>> >>>>>
>>> >>>>> Why is duplicate filtering being enabled ? Does it filter packets
>>> >>>>> that
>>> >>>>> have different advertisement content ? Can I turn it off ?
>>> >>>>
>>> >>>> At least in the HCI section of the core spec this parameter is quite
>>> >>>> vaguely defined. We've enabled it to save power by avoiding the
>>> >>>> controller waking up the host too often. With auto-connections it
>>> >>>> makes
>>> >>>> sense to have it enabled since we're really only interested in the
>>> >>>> first
>>> >>>> event which acts as a trigger to connect. Same goes for device
>>> >>>> discovery
>>> >>>> when you're interested in setting up a new device for use.
>>> >>>>
>>> >>>> Right now the filtering is hard-coded but we could consider making it
>>> >>>> conditional to whether there are any entries with action 0x00 ("scan
>>> >>>> but
>>> >>>> don't connect") and disable the filtering in that case.
>>> >>>>
>>> >>>>>>> I think this ties in with some of the previous conversations on
>>> >>>>>>> the
>>> >>>>>>> list about adding a device scan API that is geared towards
>>> >>>>>>> applications (rather than OS UI). That is, not only a mgmt API
>>> >>>>>>> that
>>> >>>>>>> enables these kinds of use cases but also (probably) a high-level
>>> >>>>>>> API
>>> >>>>>>> exposed from bluetoothd that allows applications to scan for
>>> >>>>>>> devices
>>> >>>>>>> filtered by service UUIDs, contents of the manufacturer data
>>> >>>>>>> field,
>>> >>>>>>> and so on.
>>> >>>>>>>
>>> >>>>>>> As for the Add Device command, does bluetoothd currently use this
>>> >>>>>>> somehow?
>>> >>>>>>
>>> >>>>>> Yes, this feature (action 0x02 for Add Device) takes over the
>>> >>>>>> entire LE
>>> >>>>>> background scanning and auto-connection mechanism from 3.17 kernel
>>> >>>>>> onwards since BlueZ 5.22 (which was mentioned in our release notes
>>> >>>>>> btw).
>>> >>>>>> This is much better than the active scanning based auto-connections
>>> >>>>>> that
>>> >>>>>> have until now been done using Start/Stop Discovery.
>>> >>>>>
>>> >>>>> Thanks for pointing that!
>>> >>>>> The release note says that:
>>> >>>>> "The kernel will even use the controller-side whitelist during
>>> >>>>> scanning (if no Resolvable Private Addresses are present), saving
>>> >>>>> even
>>> >>>>> more power."
>>> >>>>> If I understand well, that mean that my peripheral device need to be
>>> >>>>> whitelisted in order for me to get advertisements ?
>>> >>>>
>>> >>>> The controller-side whitelist just means that the controller only
>>> >>>> wakes
>>> >>>> us up and gives us advertisements for entries in the whitelist. When
>>> >>>> not
>>> >>>> using the whitelist the kernel gets woken up and needs to do itself
>>> >>>> the
>>> >>>> filtering out of unknown devices. The kernel uses the controller-side
>>> >>>> whitelist whenever possible, but e.g. when we have resolvable private
>>> >>>> addresses in the list we can't use it as we can't know the address
>>> >>>> the
>>> >>>> remote device will use in advance.
>>> >>>>
>>> >>>>> My use case include scanning for non-paired, non-connected devices,
>>> >>>>> checking RSSI value and deciding about connection based on
>>> >>>>> advertisement content, so that won't work. Am I right ?
>>> >>>>
>>> >>>> If you know the bdaddr that you're interested in in advance we should
>>> >>>> be
>>> >>>> able to tweak the Add Device command to do what you want (e.g.
>>> >>>> disable
>>> >>>> duplicate filtering in the case of Action=0x00 entries). However, if
>>> >>>> what you need is general scanning for any devices (also previously
>>> >>>> unknown ones) then Add Device doesn't really fit in (since it takes a
>>> >>>> known bdaddr). For that we'd either need to consider allowing
>>> >>>> BDADDR_ANY
>>> >>>> as an accepted parameter or then we'd need to define a new mgmt
>>> >>>> command.
>>> >>>>
>>> >>>
>>> >>> Unfortunately my case include previously unknown devices. I decide
>>> >>> wether I should connect (without pairing) based on advertised service
>>> >>> and RSSI power only. BDADDR_ANY seems good solution. Or maybe we can
>>> >>> add some parameter that will force not using controller-side whitelist
>>> >>> for scan?
>>> >>
>>> >> actually BDADDR_ANY does not really look like a good solution. What we
>>> >> really want is to be able to define a list of UUIDs and maybe an RSSI range
>>> >> or threshold for this.
>>> >
>>> > I think there should be option in mgmt api to capture all
>>> > advertisement packets, and later on application level decide what
>>> > property to filter on. I understand that this might be not energy
>>> > efficient, and that's why there is whitelisting and controller level
>>> > filtering.
>>> > Specifying list of UUIDs or RSSI value as parameter will be nice, but
>>> > controllers doesn't support this kind of complex filtering right now
>>> > anyway.
>>> > If you decide to allow UUID and RSSI filtering, maybe in a while
>>> > someone will come up with need to filter by something in advertisement
>>> > ? Maybe it would be good to have this 'look for advertisements' API
>>> > very general.
>>>
>>> we can always extend the mgmt API with a new command for filtering
>>> something new. And yes, the Bluetooth SIG will define a new AD element type
>>> and then it is not supported. There are also many new Bluetooth
>>> specifications in progress and also planned (which I will not iterate here
>>> since they are Buetooth SIG confidential) which extend LE advertising in
>>> various form and shapes.
>>>
>>> For me it is important that we can do certain things in the kernel so that
>>> userspace can keep sleeping. And we really can just add extra mgmt commands
>>> to allow for this. Remember that kernel will have to disable certain
>>> "background" behavior anymore. An active discovery always comes first. An
>>> active connection establishment from a socket comes first. Pairing a device
>>> comes first. So all these major events that are driven by an user action
>>> will be prioritized. We have to do that since otherwise you end up with bad
>>> user experience. But then again, draining the battery is also bad user
>>> experience. So we need to find a balance here.
>>
>>
>> I will be very happy with mgmt api method that will filter by serice uuid
>> and RSSI value, and that's enough for me. From my use case it looks like the
>> user is going to be in this scan state for short time - few seconds, maximum
>> minute.
>>
>>
>> that is then something different. That is more like "scan for this UUID
>> within this RSSI range for x amount of seconds, if any results, report them,
>> and otherwise go back to idle".
>>
>> We can talk about this one-shot kind of API. I would need to think about it
>> a little bit more. However one thing that you might want to consider is
>> using pathloss instead of RSSI. Especially if the advertisement contains the
>> TX power. That way you get a way better idea of how close or far the device
>> is.
>>
>
> Yay! Thanks!
> Yes, pathloss will be much better, I was using it before.
>
>> If it would be possible to also add general le scan method that would return
>> all advertisements, marked as "only for development", and "not for use in
>> production", and "behaves differently for different controllers" and maybe
>> somehow disabled by default and you need to set some special flag to enable
>> it on your developer machine then it would be awsome for developers that
>> want to play with it, but it's not my requirement.
>>
>>
>> The mgmt API will never contain any development methods. It is an official
>> API and we are committed to stable API promise here. Any kind of testing API
>> has to go via debugfs.
>>
>> Regards
>>
>> Marcel
>>

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

> ---
> doc/mgmt-api.txt | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 69 insertions(+)
> 
> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
> index 11e2490..1c78f1c 100644
> --- a/doc/mgmt-api.txt
> +++ b/doc/mgmt-api.txt
> @@ -2135,6 +2135,34 @@ Set Public Address Command
>  Invalid Index
> 
> 
> +Start LE Filtered Device Scan
> +=======================
> +
> + Command Code: 0x003A
> + Controller Index: <controller id>
> + Command Parameters: UUID (16 Octets)
> +             max_pathloss (1 octet)
> +             is_active (1 octet)
> +
> + Return Parameters:
> +
> + This command starts LE scan, and filter received advertisements: if
> AD contains both TX power level and Service Solicitation, and UUID is
> contained in Service Solicitation and computed pathloss is smaller
> than max_pathloss parameter, then a Advertisement Received event will
> be sent.
> +
> + Possible values for the is_active parameter:
> + 0 Start passive scan
> + 1 Start active scan

is really like a one-shot scan? Or is this something you would be running all the time in the background?

Fundamentally if it is one-shot thing then you always want to do active scan, but if it runs in the background then you only want to do passive scanning.

> +
> + This command can only be used when the controller is powered.
> +
> + This command generates a Command Complete event on success or failure.
> +
> + Possible errors: Busy
> + Not Supported
> + Invalid Parameters
> + Not Powered
> + Invalid Index
> +
> +
> Command Complete Event
> ======================
> 
> @@ -2851,3 +2879,44 @@ New Configuration Options Event
> 
>  This event indicates that one or more of the options for the
>  controller configuration has changed.
> +
> +
> +Advertisement Received Event
> +==============================
> +
> + Event Code: 0x0020
> + Controller Index: <controller id>
> + Event Parameters: Address (6 Octets)
> + Address_Type (1 Octet)
> + RSSI (1 Octet)
> + Flags (4 Octets)
> + AD_Data_Length (2 Octets)
> + AD_Data (0-65535 Octets)

Device Found event not good enough here? Why would you need a new event with exactly the same parameters?

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Arman Uguray]

Hi Marcel,


>> +Advertisement Received Event
>> +==============================
>> +
>> + Event Code: 0x0020
>> + Controller Index: <controller id>
>> + Event Parameters: Address (6 Octets)
>> + Address_Type (1 Octet)
>> + RSSI (1 Octet)
>> + Flags (4 Octets)
>> + AD_Data_Length (2 Octets)
>> + AD_Data (0-65535 Octets)
>
> Device Found event not good enough here? Why would you need a new event with exactly the same parameters?
>

I guess the idea with this one is that it won't have duplicate
filtering, right? And all AD_IND types will be propagated instead of
just connectable ones, which is how this one differs from Device
Found. And I'm guessing AD_Data here is the data in the advertisement
packet in general as opposed to only being limited to EIR data.
Though, correct me if I'm wrong.

Cheers,
Arman

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Arman,

>>> +Advertisement Received Event
>>> +==============================
>>> +
>>> + Event Code: 0x0020
>>> + Controller Index: <controller id>
>>> + Event Parameters: Address (6 Octets)
>>> + Address_Type (1 Octet)
>>> + RSSI (1 Octet)
>>> + Flags (4 Octets)
>>> + AD_Data_Length (2 Octets)
>>> + AD_Data (0-65535 Octets)
>> 
>> Device Found event not good enough here? Why would you need a new event with exactly the same parameters?
>> 
> 
> I guess the idea with this one is that it won't have duplicate
> filtering, right? And all AD_IND types will be propagated instead of
> just connectable ones, which is how this one differs from Device
> Found. And I'm guessing AD_Data here is the data in the advertisement
> packet in general as opposed to only being limited to EIR data.
> Though, correct me if I'm wrong.

we fixed the advertising type handling. Device Found will report ADV_IND, ADV_NONCONN_IND and ADV_SCAN_IND. The latter two have the Non-Connectable flag set.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

On Tue, Sep 30, 2014 at 10:01 AM, Marcel Holtmann <marcel@holtmann.org> wrote:
> Hi Jakub,
>
>> ---
>> doc/mgmt-api.txt | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 69 insertions(+)
>>
>> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
>> index 11e2490..1c78f1c 100644
>> --- a/doc/mgmt-api.txt
>> +++ b/doc/mgmt-api.txt
>> @@ -2135,6 +2135,34 @@ Set Public Address Command
>>  Invalid Index
>>
>>
>> +Start LE Filtered Device Scan
>> +=======================
>> +
>> + Command Code: 0x003A
>> + Controller Index: <controller id>
>> + Command Parameters: UUID (16 Octets)
>> +             max_pathloss (1 octet)
>> +             is_active (1 octet)
>> +
>> + Return Parameters:
>> +
>> + This command starts LE scan, and filter received advertisements: if
>> AD contains both TX power level and Service Solicitation, and UUID is
>> contained in Service Solicitation and computed pathloss is smaller
>> than max_pathloss parameter, then a Advertisement Received event will
>> be sent.
>> +
>> + Possible values for the is_active parameter:
>> + 0 Start passive scan
>> + 1 Start active scan
>
> is really like a one-shot scan? Or is this something you would be running all the time in the background?
>
> Fundamentally if it is one-shot thing then you always want to do active scan, but if it runs in the background then you only want to do passive scanning.

I'm working on two projects, one require scanning up to 2 minutes,
second requires long term scanning in background.
I'd preffer to have scan type as argument, but having only passive
scan is also good option for me.

>
>> +
>> + This command can only be used when the controller is powered.
>> +
>> + This command generates a Command Complete event on success or failure.
>> +
>> + Possible errors: Busy
>> + Not Supported
>> + Invalid Parameters
>> + Not Powered
>> + Invalid Index
>> +
>> +
>> Command Complete Event
>> ======================
>>
>> @@ -2851,3 +2879,44 @@ New Configuration Options Event
>>
>>  This event indicates that one or more of the options for the
>>  controller configuration has changed.
>> +
>> +
>> +Advertisement Received Event
>> +==============================
>> +
>> + Event Code: 0x0020
>> + Controller Index: <controller id>
>> + Event Parameters: Address (6 Octets)
>> + Address_Type (1 Octet)
>> + RSSI (1 Octet)
>> + Flags (4 Octets)
>> + AD_Data_Length (2 Octets)
>> + AD_Data (0-65535 Octets)
>
> Device Found event not good enough here? Why would you need a new event with exactly the same parameters?

Because Device Found event always contains EIR data, and mine would
sometimes contain only AD data.
Also this will be send for each advertisement received, not for first one only.

>
> Regards
>
> Marcel
>

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

>>> ---
>>> doc/mgmt-api.txt | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 69 insertions(+)
>>> 
>>> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
>>> index 11e2490..1c78f1c 100644
>>> --- a/doc/mgmt-api.txt
>>> +++ b/doc/mgmt-api.txt
>>> @@ -2135,6 +2135,34 @@ Set Public Address Command
>>> Invalid Index
>>> 
>>> 
>>> +Start LE Filtered Device Scan
>>> +=======================
>>> +
>>> + Command Code: 0x003A
>>> + Controller Index: <controller id>
>>> + Command Parameters: UUID (16 Octets)
>>> +             max_pathloss (1 octet)
>>> +             is_active (1 octet)
>>> +
>>> + Return Parameters:
>>> +
>>> + This command starts LE scan, and filter received advertisements: if
>>> AD contains both TX power level and Service Solicitation, and UUID is
>>> contained in Service Solicitation and computed pathloss is smaller
>>> than max_pathloss parameter, then a Advertisement Received event will
>>> be sent.
>>> +
>>> + Possible values for the is_active parameter:
>>> + 0 Start passive scan
>>> + 1 Start active scan
>> 
>> is really like a one-shot scan? Or is this something you would be running all the time in the background?
>> 
>> Fundamentally if it is one-shot thing then you always want to do active scan, but if it runs in the background then you only want to do passive scanning.
> 
> I'm working on two projects, one require scanning up to 2 minutes,
> second requires long term scanning in background.
> I'd preffer to have scan type as argument, but having only passive
> scan is also good option for me.

I prefer to have these as separate commands. So one is actually a specialized active scanning to find a specific set of devices. Similar to Start Discovery. The other is to add an additional filter for our background scanning.

Remember that this has to work in sync with the auto-connection logic that the kernel is running. If you are messing that up, then you might find the device you are looking for, but the HID mouse will not re-connect.

>>> +
>>> + This command can only be used when the controller is powered.
>>> +
>>> + This command generates a Command Complete event on success or failure.
>>> +
>>> + Possible errors: Busy
>>> + Not Supported
>>> + Invalid Parameters
>>> + Not Powered
>>> + Invalid Index
>>> +
>>> +
>>> Command Complete Event
>>> ======================
>>> 
>>> @@ -2851,3 +2879,44 @@ New Configuration Options Event
>>> 
>>> This event indicates that one or more of the options for the
>>> controller configuration has changed.
>>> +
>>> +
>>> +Advertisement Received Event
>>> +==============================
>>> +
>>> + Event Code: 0x0020
>>> + Controller Index: <controller id>
>>> + Event Parameters: Address (6 Octets)
>>> + Address_Type (1 Octet)
>>> + RSSI (1 Octet)
>>> + Flags (4 Octets)
>>> + AD_Data_Length (2 Octets)
>>> + AD_Data (0-65535 Octets)
>> 
>> Device Found event not good enough here? Why would you need a new event with exactly the same parameters?
> 
> Because Device Found event always contains EIR data, and mine would
> sometimes contain only AD data.
> Also this will be send for each advertisement received, not for first one only.

Do not get confused because Device Found says EIR_Data. It will return EIR_Data for BR/EDR devices and AD_Data for LE devices. We just never bothered to find a more common name.

We could easily send multiple Device Found events. Besides Start Discovery, we also have Add Device with Action 0x00 sending out Device Found events. So that would not be a problem. We just need to define the context properly.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

Ok, new updated version - we do only passive scan, and we send Device
found event:


diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
index 11e2490..3d03617 100644
--- a/doc/mgmt-api.txt
+++ b/doc/mgmt-api.txt
@@ -2135,6 +2135,29 @@ Set Public Address Command
                                Invalid Index


+Start Passive LE Filtered Device Scan
+=======================
+
+       Command Code:           0x003A
+       Controller Index:       <controller id>
+       Command Parameters:     UUID (16 Octets)
+                    max_pathloss (1 octet)
+
+       Return Parameters:
+
+       This command starts passive LE scan, and filter received
advertisements: if AD contains both TX power level and Service
Solicitation, and UUID is contained in Service Solicitation and
computed pathloss is smaller than max_pathloss parameter, then a
Device Found event will be sent.
+
+       This command can only be used when the controller is powered.
+
+       This command generates a Command Complete event on success or failure.
+
+       Possible errors:        Busy
+                               Not Supported
+                               Invalid Parameters
+                               Not Powered
+                               Invalid Index
+
+
 Command Complete Event
 ======================


On Tue, Sep 30, 2014 at 10:31 AM, Marcel Holtmann <marcel@holtmann.org> wrote:
> Hi Jakub,
>
>>>> ---
>>>> doc/mgmt-api.txt | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> 1 file changed, 69 insertions(+)
>>>>
>>>> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
>>>> index 11e2490..1c78f1c 100644
>>>> --- a/doc/mgmt-api.txt
>>>> +++ b/doc/mgmt-api.txt
>>>> @@ -2135,6 +2135,34 @@ Set Public Address Command
>>>> Invalid Index
>>>>
>>>>
>>>> +Start LE Filtered Device Scan
>>>> +=======================
>>>> +
>>>> + Command Code: 0x003A
>>>> + Controller Index: <controller id>
>>>> + Command Parameters: UUID (16 Octets)
>>>> +             max_pathloss (1 octet)
>>>> +             is_active (1 octet)
>>>> +
>>>> + Return Parameters:
>>>> +
>>>> + This command starts LE scan, and filter received advertisements: if
>>>> AD contains both TX power level and Service Solicitation, and UUID is
>>>> contained in Service Solicitation and computed pathloss is smaller
>>>> than max_pathloss parameter, then a Advertisement Received event will
>>>> be sent.
>>>> +
>>>> + Possible values for the is_active parameter:
>>>> + 0 Start passive scan
>>>> + 1 Start active scan
>>>
>>> is really like a one-shot scan? Or is this something you would be running all the time in the background?
>>>
>>> Fundamentally if it is one-shot thing then you always want to do active scan, but if it runs in the background then you only want to do passive scanning.
>>
>> I'm working on two projects, one require scanning up to 2 minutes,
>> second requires long term scanning in background.
>> I'd preffer to have scan type as argument, but having only passive
>> scan is also good option for me.
>
> I prefer to have these as separate commands. So one is actually a specialized active scanning to find a specific set of devices. Similar to Start Discovery. The other is to add an additional filter for our background scanning.
>
> Remember that this has to work in sync with the auto-connection logic that the kernel is running. If you are messing that up, then you might find the device you are looking for, but the HID mouse will not re-connect.
>
>>>> +
>>>> + This command can only be used when the controller is powered.
>>>> +
>>>> + This command generates a Command Complete event on success or failure.
>>>> +
>>>> + Possible errors: Busy
>>>> + Not Supported
>>>> + Invalid Parameters
>>>> + Not Powered
>>>> + Invalid Index
>>>> +
>>>> +
>>>> Command Complete Event
>>>> ======================
>>>>
>>>> @@ -2851,3 +2879,44 @@ New Configuration Options Event
>>>>
>>>> This event indicates that one or more of the options for the
>>>> controller configuration has changed.
>>>> +
>>>> +
>>>> +Advertisement Received Event
>>>> +==============================
>>>> +
>>>> + Event Code: 0x0020
>>>> + Controller Index: <controller id>
>>>> + Event Parameters: Address (6 Octets)
>>>> + Address_Type (1 Octet)
>>>> + RSSI (1 Octet)
>>>> + Flags (4 Octets)
>>>> + AD_Data_Length (2 Octets)
>>>> + AD_Data (0-65535 Octets)
>>>
>>> Device Found event not good enough here? Why would you need a new event with exactly the same parameters?
>>
>> Because Device Found event always contains EIR data, and mine would
>> sometimes contain only AD data.
>> Also this will be send for each advertisement received, not for first one only.
>
> Do not get confused because Device Found says EIR_Data. It will return EIR_Data for BR/EDR devices and AD_Data for LE devices. We just never bothered to find a more common name.
>
> We could easily send multiple Device Found events. Besides Start Discovery, we also have Add Device with Action 0x00 sending out Device Found events. So that would not be a problem. We just need to define the context properly.
>
> Regards
>
> Marcel
>

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

> Ok, new updated version - we do only passive scan, and we send Device
> found event:
> 
> 
> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
> index 11e2490..3d03617 100644
> --- a/doc/mgmt-api.txt
> +++ b/doc/mgmt-api.txt
> @@ -2135,6 +2135,29 @@ Set Public Address Command
>                                Invalid Index
> 
> 
> +Start Passive LE Filtered Device Scan
> +=======================
> +
> +       Command Code:           0x003A
> +       Controller Index:       <controller id>
> +       Command Parameters:     UUID (16 Octets)
> +                    max_pathloss (1 octet)
> +
> +       Return Parameters:
> +
> +       This command starts passive LE scan, and filter received
> advertisements: if AD contains both TX power level and Service
> Solicitation, and UUID is contained in Service Solicitation and
> computed pathloss is smaller than max_pathloss parameter, then a
> Device Found event will be sent.

generally we tried to avoid to specific LE only commands. Our attempts with the kernel APIs where to make them as generic as possible. So I think something along the lines of adding and removing UUID filters should be something we should target at.

Add UUID Notification Filter
============================

	Command Parameters:	Address_Type (1 Octet)
				UUID (16 Octets)
				Max_Pathloss (1 Octet)
	Return Parameters:	Address_Type (1 Octet)


Remove UUID Notification Filter
===============================

	Command Parameters:	Address_Type (1 Octet)
				UUID (16 Octets)
	Return Parameters:	Address_Type (1 Octet)

The Address_Type would be defined the same way as for Start Discovery command.

Now we can discuss if on the mgmt level we want to use pathloss or rather a range of RSSI. I have seen that RSSI is preferred and that the translation from RSSI and advertising data with TX power level is done in the daemon. So for kernel API it might be better to expose RSSI range.

Besides a RSSI range, we might also want to allow giving a RSSI threshold that defines how much the RSSI to change between events to be reported again.

That said, the danger with a generic notification filter like this is that we can not implement it efficiently with current hardware without vendor commands. If you use duplicate filtering, then Broadcom and CSR chips behave fully different. Broadcom only filters on BD_ADDR and report devices once no matter what the RSSI, while CSR will report the same device again if the RSSI changes.

With that in mind, it might be safer to introduce a one-shot mgmt API that needs to be repeatedly called to keep scanning for new devices.

Start Service Discovery
=======================

	Command Parameters:	Address_Type (1 Octet)
				Min_RSSI (1 Octet)
				Max_RSSI (1 Octet)
				Num_UUID (1 Octet)
				UUID[i] (16 Octets)
	Return Parameters:	Address_Type (1 Octet)

Stop Service Discovery
======================

	Command Parameters:	Address_Type (1 Octet)
	Return Parameters:	Address_Type (1 Octet)

Without having dedicated controller support for filtering, having such a dedicated discovery for services with a specific UUID seems to be more appropriate and more safe. Since we could always add controller based filtering for background scanning once there is controller support.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

On Tue, Sep 30, 2014 at 11:54 AM, Marcel Holtmann <marcel@holtmann.org> wro=
te:
> Hi Jakub,
>
>> Ok, new updated version - we do only passive scan, and we send Device
>> found event:
>>
>>
>> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
>> index 11e2490..3d03617 100644
>> --- a/doc/mgmt-api.txt
>> +++ b/doc/mgmt-api.txt
>> @@ -2135,6 +2135,29 @@ Set Public Address Command
>>                                Invalid Index
>>
>>
>> +Start Passive LE Filtered Device Scan
>> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> +
>> +       Command Code:           0x003A
>> +       Controller Index:       <controller id>
>> +       Command Parameters:     UUID (16 Octets)
>> +                    max_pathloss (1 octet)
>> +
>> +       Return Parameters:
>> +
>> +       This command starts passive LE scan, and filter received
>> advertisements: if AD contains both TX power level and Service
>> Solicitation, and UUID is contained in Service Solicitation and
>> computed pathloss is smaller than max_pathloss parameter, then a
>> Device Found event will be sent.
>
> generally we tried to avoid to specific LE only commands. Our attempts wi=
th the kernel APIs where to make them as generic as possible. So I think so=
mething along the lines of adding and removing UUID filters should be somet=
hing we should target at.
>
> Add UUID Notification Filter
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
>
>         Command Parameters:     Address_Type (1 Octet)
>                                 UUID (16 Octets)
>                                 Max_Pathloss (1 Octet)
>         Return Parameters:      Address_Type (1 Octet)
>
>
> Remove UUID Notification Filter
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D
>
>         Command Parameters:     Address_Type (1 Octet)
>                                 UUID (16 Octets)
>         Return Parameters:      Address_Type (1 Octet)
>
> The Address_Type would be defined the same way as for Start Discovery com=
mand.
>
> Now we can discuss if on the mgmt level we want to use pathloss or rather=
 a range of RSSI. I have seen that RSSI is preferred and that the translati=
on from RSSI and advertising data with TX power level is done in the daemon=
. So for kernel API it might be better to expose RSSI range.
>
> Besides a RSSI range, we might also want to allow giving a RSSI threshold=
 that defines how much the RSSI to change between events to be reported aga=
in.

Some mobile devices that I'm working can change power level for
advertising, that makes raw RSSI filter useless. It must be pathloss,
that's why I require TX power in advertisement.

>
> That said, the danger with a generic notification filter like this is tha=
t we can not implement it efficiently with current hardware without vendor =
commands. If you use duplicate filtering, then Broadcom and CSR chips behav=
e fully different. Broadcom only filters on BD_ADDR and report devices once=
 no matter what the RSSI, while CSR will report the same device again if th=
e RSSI changes.
>
> With that in mind, it might be safer to introduce a one-shot mgmt API tha=
t needs to be repeatedly called to keep scanning for new devices.
>
> Start Service Discovery
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
>         Command Parameters:     Address_Type (1 Octet)
>                                 Min_RSSI (1 Octet)
>                                 Max_RSSI (1 Octet)
>                                 Num_UUID (1 Octet)
>                                 UUID[i] (16 Octets)
>         Return Parameters:      Address_Type (1 Octet)
>
> Stop Service Discovery
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
>         Command Parameters:     Address_Type (1 Octet)
>         Return Parameters:      Address_Type (1 Octet)
>
> Without having dedicated controller support for filtering, having such a =
dedicated discovery for services with a specific UUID seems to be more appr=
opriate and more safe. Since we could always add controller based filtering=
 for background scanning once there is controller support.

I preffer the "Add UUID Notification Filter" and "Remove UUID
Notification Filter". For controllers like Broadcom that reports only
one advertisement can we restart scan internally in kernel ? Why
propagate that event higher ?


>
> Regards
>
> Marcel
>

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

>>> Ok, new updated version - we do only passive scan, and we send Device
>>> found event:
>>> 
>>> 
>>> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
>>> index 11e2490..3d03617 100644
>>> --- a/doc/mgmt-api.txt
>>> +++ b/doc/mgmt-api.txt
>>> @@ -2135,6 +2135,29 @@ Set Public Address Command
>>>                               Invalid Index
>>> 
>>> 
>>> +Start Passive LE Filtered Device Scan
>>> +=======================
>>> +
>>> +       Command Code:           0x003A
>>> +       Controller Index:       <controller id>
>>> +       Command Parameters:     UUID (16 Octets)
>>> +                    max_pathloss (1 octet)
>>> +
>>> +       Return Parameters:
>>> +
>>> +       This command starts passive LE scan, and filter received
>>> advertisements: if AD contains both TX power level and Service
>>> Solicitation, and UUID is contained in Service Solicitation and
>>> computed pathloss is smaller than max_pathloss parameter, then a
>>> Device Found event will be sent.
>> 
>> generally we tried to avoid to specific LE only commands. Our attempts with the kernel APIs where to make them as generic as possible. So I think something along the lines of adding and removing UUID filters should be something we should target at.
>> 
>> Add UUID Notification Filter
>> ============================
>> 
>>        Command Parameters:     Address_Type (1 Octet)
>>                                UUID (16 Octets)
>>                                Max_Pathloss (1 Octet)
>>        Return Parameters:      Address_Type (1 Octet)
>> 
>> 
>> Remove UUID Notification Filter
>> ===============================
>> 
>>        Command Parameters:     Address_Type (1 Octet)
>>                                UUID (16 Octets)
>>        Return Parameters:      Address_Type (1 Octet)
>> 
>> The Address_Type would be defined the same way as for Start Discovery command.
>> 
>> Now we can discuss if on the mgmt level we want to use pathloss or rather a range of RSSI. I have seen that RSSI is preferred and that the translation from RSSI and advertising data with TX power level is done in the daemon. So for kernel API it might be better to expose RSSI range.
>> 
>> Besides a RSSI range, we might also want to allow giving a RSSI threshold that defines how much the RSSI to change between events to be reported again.
> 
> Some mobile devices that I'm working can change power level for
> advertising, that makes raw RSSI filter useless. It must be pathloss,
> that's why I require TX power in advertisement.

you will still get it, but just have to do the math in the daemon. The only difference is that the daemon gets woken up for RSSI ranges that in the end you might filter out.

>> 
>> That said, the danger with a generic notification filter like this is that we can not implement it efficiently with current hardware without vendor commands. If you use duplicate filtering, then Broadcom and CSR chips behave fully different. Broadcom only filters on BD_ADDR and report devices once no matter what the RSSI, while CSR will report the same device again if the RSSI changes.
>> 
>> With that in mind, it might be safer to introduce a one-shot mgmt API that needs to be repeatedly called to keep scanning for new devices.
>> 
>> Start Service Discovery
>> =======================
>> 
>>        Command Parameters:     Address_Type (1 Octet)
>>                                Min_RSSI (1 Octet)
>>                                Max_RSSI (1 Octet)
>>                                Num_UUID (1 Octet)
>>                                UUID[i] (16 Octets)
>>        Return Parameters:      Address_Type (1 Octet)
>> 
>> Stop Service Discovery
>> ======================
>> 
>>        Command Parameters:     Address_Type (1 Octet)
>>        Return Parameters:      Address_Type (1 Octet)
>> 
>> Without having dedicated controller support for filtering, having such a dedicated discovery for services with a specific UUID seems to be more appropriate and more safe. Since we could always add controller based filtering for background scanning once there is controller support.
> 
> I preffer the "Add UUID Notification Filter" and "Remove UUID
> Notification Filter". For controllers like Broadcom that reports only
> one advertisement can we restart scan internally in kernel ? Why
> propagate that event higher ?

Personally I would prefer just adding another filter as well. However it might be better to start with something like a one-shot handling and see where that is taking us. It is a little bit less intrusive and you could also use active scanning. At the same time you could also easily make it work for BR/EDR. I am thinking here along the lines of something wanting to pair a mouse. You just give the HID UUID and start interleaved discovery for that service on both transports.

For the Broadcom vs CSR differences, yes, we can work out certain quirks on how to improve this situation. However we might need to work these out before adding any new API. We are using controller whitelist whenever we can to avoid having to deal with this difference. But yes, stopping and re-starting has been considered as a possible solution to work with Broadcom controllers. Just never implemented.

I would propose you start looking at how you can make different controllers work smoothly and start by adding a debugfs based API. That way we can toy with this upstream without having to commit to a new API just yet. If we are satisfied, then we turn the debugfs API into a mgmt API for general consumption. Start sending kernel patches that allows some sort UUID filtering in our background scanning handling.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

>>>> Ok, new updated version - we do only passive scan, and we send Device
>>>> found event:
>>>> 
>>>> 
>>>> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
>>>> index 11e2490..3d03617 100644
>>>> --- a/doc/mgmt-api.txt
>>>> +++ b/doc/mgmt-api.txt
>>>> @@ -2135,6 +2135,29 @@ Set Public Address Command
>>>>                              Invalid Index
>>>> 
>>>> 
>>>> +Start Passive LE Filtered Device Scan
>>>> +=======================
>>>> +
>>>> +       Command Code:           0x003A
>>>> +       Controller Index:       <controller id>
>>>> +       Command Parameters:     UUID (16 Octets)
>>>> +                    max_pathloss (1 octet)
>>>> +
>>>> +       Return Parameters:
>>>> +
>>>> +       This command starts passive LE scan, and filter received
>>>> advertisements: if AD contains both TX power level and Service
>>>> Solicitation, and UUID is contained in Service Solicitation and
>>>> computed pathloss is smaller than max_pathloss parameter, then a
>>>> Device Found event will be sent.
>>> 
>>> generally we tried to avoid to specific LE only commands. Our attempts with the kernel APIs where to make them as generic as possible. So I think something along the lines of adding and removing UUID filters should be something we should target at.
>>> 
>>> Add UUID Notification Filter
>>> ============================
>>> 
>>>       Command Parameters:     Address_Type (1 Octet)
>>>                               UUID (16 Octets)
>>>                               Max_Pathloss (1 Octet)
>>>       Return Parameters:      Address_Type (1 Octet)
>>> 
>>> 
>>> Remove UUID Notification Filter
>>> ===============================
>>> 
>>>       Command Parameters:     Address_Type (1 Octet)
>>>                               UUID (16 Octets)
>>>       Return Parameters:      Address_Type (1 Octet)
>>> 
>>> The Address_Type would be defined the same way as for Start Discovery command.
>>> 
>>> Now we can discuss if on the mgmt level we want to use pathloss or rather a range of RSSI. I have seen that RSSI is preferred and that the translation from RSSI and advertising data with TX power level is done in the daemon. So for kernel API it might be better to expose RSSI range.
>>> 
>>> Besides a RSSI range, we might also want to allow giving a RSSI threshold that defines how much the RSSI to change between events to be reported again.
>> 
>> Some mobile devices that I'm working can change power level for
>> advertising, that makes raw RSSI filter useless. It must be pathloss,
>> that's why I require TX power in advertisement.
> 
> you will still get it, but just have to do the math in the daemon. The only difference is that the daemon gets woken up for RSSI ranges that in the end you might filter out.
> 
>>> 
>>> That said, the danger with a generic notification filter like this is that we can not implement it efficiently with current hardware without vendor commands. If you use duplicate filtering, then Broadcom and CSR chips behave fully different. Broadcom only filters on BD_ADDR and report devices once no matter what the RSSI, while CSR will report the same device again if the RSSI changes.
>>> 
>>> With that in mind, it might be safer to introduce a one-shot mgmt API that needs to be repeatedly called to keep scanning for new devices.
>>> 
>>> Start Service Discovery
>>> =======================
>>> 
>>>       Command Parameters:     Address_Type (1 Octet)
>>>                               Min_RSSI (1 Octet)
>>>                               Max_RSSI (1 Octet)
>>>                               Num_UUID (1 Octet)
>>>                               UUID[i] (16 Octets)
>>>       Return Parameters:      Address_Type (1 Octet)
>>> 
>>> Stop Service Discovery
>>> ======================
>>> 
>>>       Command Parameters:     Address_Type (1 Octet)
>>>       Return Parameters:      Address_Type (1 Octet)
>>> 
>>> Without having dedicated controller support for filtering, having such a dedicated discovery for services with a specific UUID seems to be more appropriate and more safe. Since we could always add controller based filtering for background scanning once there is controller support.
>> 
>> I preffer the "Add UUID Notification Filter" and "Remove UUID
>> Notification Filter". For controllers like Broadcom that reports only
>> one advertisement can we restart scan internally in kernel ? Why
>> propagate that event higher ?
> 
> Personally I would prefer just adding another filter as well. However it might be better to start with something like a one-shot handling and see where that is taking us. It is a little bit less intrusive and you could also use active scanning. At the same time you could also easily make it work for BR/EDR. I am thinking here along the lines of something wanting to pair a mouse. You just give the HID UUID and start interleaved discovery for that service on both transports.

I send a proposal for doing an one-shot service discovery. So you can propose a RSSI Threshold and a list of possible UUIDs to look for.

If RSSI Threshold is 127 or the Num_UUID is 0, then it will behave the same way as Start Discovery. I think this is how we should get started to tackle this problem.

Regards

Marcel

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

Thanks for info!

I'll look into that.

On Fri, Oct 10, 2014 at 3:52 AM, Marcel Holtmann <marcel@holtmann.org> wrot=
e:
> Hi Jakub,
>
>>>>> Ok, new updated version - we do only passive scan, and we send Device
>>>>> found event:
>>>>>
>>>>>
>>>>> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
>>>>> index 11e2490..3d03617 100644
>>>>> --- a/doc/mgmt-api.txt
>>>>> +++ b/doc/mgmt-api.txt
>>>>> @@ -2135,6 +2135,29 @@ Set Public Address Command
>>>>>                              Invalid Index
>>>>>
>>>>>
>>>>> +Start Passive LE Filtered Device Scan
>>>>> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
>>>>> +
>>>>> +       Command Code:           0x003A
>>>>> +       Controller Index:       <controller id>
>>>>> +       Command Parameters:     UUID (16 Octets)
>>>>> +                    max_pathloss (1 octet)
>>>>> +
>>>>> +       Return Parameters:
>>>>> +
>>>>> +       This command starts passive LE scan, and filter received
>>>>> advertisements: if AD contains both TX power level and Service
>>>>> Solicitation, and UUID is contained in Service Solicitation and
>>>>> computed pathloss is smaller than max_pathloss parameter, then a
>>>>> Device Found event will be sent.
>>>>
>>>> generally we tried to avoid to specific LE only commands. Our attempts=
 with the kernel APIs where to make them as generic as possible. So I think=
 something along the lines of adding and removing UUID filters should be so=
mething we should target at.
>>>>
>>>> Add UUID Notification Filter
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
>>>>
>>>>       Command Parameters:     Address_Type (1 Octet)
>>>>                               UUID (16 Octets)
>>>>                               Max_Pathloss (1 Octet)
>>>>       Return Parameters:      Address_Type (1 Octet)
>>>>
>>>>
>>>> Remove UUID Notification Filter
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
>>>>
>>>>       Command Parameters:     Address_Type (1 Octet)
>>>>                               UUID (16 Octets)
>>>>       Return Parameters:      Address_Type (1 Octet)
>>>>
>>>> The Address_Type would be defined the same way as for Start Discovery =
command.
>>>>
>>>> Now we can discuss if on the mgmt level we want to use pathloss or rat=
her a range of RSSI. I have seen that RSSI is preferred and that the transl=
ation from RSSI and advertising data with TX power level is done in the dae=
mon. So for kernel API it might be better to expose RSSI range.
>>>>
>>>> Besides a RSSI range, we might also want to allow giving a RSSI thresh=
old that defines how much the RSSI to change between events to be reported =
again.
>>>
>>> Some mobile devices that I'm working can change power level for
>>> advertising, that makes raw RSSI filter useless. It must be pathloss,
>>> that's why I require TX power in advertisement.
>>
>> you will still get it, but just have to do the math in the daemon. The o=
nly difference is that the daemon gets woken up for RSSI ranges that in the=
 end you might filter out.
>>
>>>>
>>>> That said, the danger with a generic notification filter like this is =
that we can not implement it efficiently with current hardware without vend=
or commands. If you use duplicate filtering, then Broadcom and CSR chips be=
have fully different. Broadcom only filters on BD_ADDR and report devices o=
nce no matter what the RSSI, while CSR will report the same device again if=
 the RSSI changes.
>>>>
>>>> With that in mind, it might be safer to introduce a one-shot mgmt API =
that needs to be repeatedly called to keep scanning for new devices.
>>>>
>>>> Start Service Discovery
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>
>>>>       Command Parameters:     Address_Type (1 Octet)
>>>>                               Min_RSSI (1 Octet)
>>>>                               Max_RSSI (1 Octet)
>>>>                               Num_UUID (1 Octet)
>>>>                               UUID[i] (16 Octets)
>>>>       Return Parameters:      Address_Type (1 Octet)
>>>>
>>>> Stop Service Discovery
>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>
>>>>       Command Parameters:     Address_Type (1 Octet)
>>>>       Return Parameters:      Address_Type (1 Octet)
>>>>
>>>> Without having dedicated controller support for filtering, having such=
 a dedicated discovery for services with a specific UUID seems to be more a=
ppropriate and more safe. Since we could always add controller based filter=
ing for background scanning once there is controller support.
>>>
>>> I preffer the "Add UUID Notification Filter" and "Remove UUID
>>> Notification Filter". For controllers like Broadcom that reports only
>>> one advertisement can we restart scan internally in kernel ? Why
>>> propagate that event higher ?
>>
>> Personally I would prefer just adding another filter as well. However it=
 might be better to start with something like a one-shot handling and see w=
here that is taking us. It is a little bit less intrusive and you could als=
o use active scanning. At the same time you could also easily make it work =
for BR/EDR. I am thinking here along the lines of something wanting to pair=
 a mouse. You just give the HID UUID and start interleaved discovery for th=
at service on both transports.
>
> I send a proposal for doing an one-shot service discovery. So you can pro=
pose a RSSI Threshold and a list of possible UUIDs to look for.
>
> If RSSI Threshold is 127 or the Num_UUID is 0, then it will behave the sa=
me way as Start Discovery. I think this is how we should get started to tac=
kle this problem.
>
> Regards
>
> Marcel
>

Re: [RFC] need for new scan api for bluetooth low energy.

[Jakub Pawlowski]

Hi Marcel.

I started implementing service scanning and I have a questions:

We initially agreed on version that uses le deduplication that needs
to be restarted after each device found.
When I started looking into that it's going to be much more restarting:
- need to restart after each packet from device that advertise service
we're interested that's not in RSSI range. That's because controller
that I use doesn't send advertisement again when RSSI changes.
- need to restart every 0.5 or 1 second in case when some device added
service I'm interested at to it's advertisement


Can we conseider implementation using le scan without deduplication,
and do packet de-duplication in kernel in hci_le_adv_report_evt ?
I've started implementing it this way and it works really nice. The
issue here might be that it might consume more energy.



On Fri, Oct 10, 2014 at 8:49 AM, Jakub Pawlowski <jpawlowski@google.com> wr=
ote:
> Thanks for info!
>
> I'll look into that.
>
> On Fri, Oct 10, 2014 at 3:52 AM, Marcel Holtmann <marcel@holtmann.org> wr=
ote:
>> Hi Jakub,
>>
>>>>>> Ok, new updated version - we do only passive scan, and we send Devic=
e
>>>>>> found event:
>>>>>>
>>>>>>
>>>>>> diff --git a/doc/mgmt-api.txt b/doc/mgmt-api.txt
>>>>>> index 11e2490..3d03617 100644
>>>>>> --- a/doc/mgmt-api.txt
>>>>>> +++ b/doc/mgmt-api.txt
>>>>>> @@ -2135,6 +2135,29 @@ Set Public Address Command
>>>>>>                              Invalid Index
>>>>>>
>>>>>>
>>>>>> +Start Passive LE Filtered Device Scan
>>>>>> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
>>>>>> +
>>>>>> +       Command Code:           0x003A
>>>>>> +       Controller Index:       <controller id>
>>>>>> +       Command Parameters:     UUID (16 Octets)
>>>>>> +                    max_pathloss (1 octet)
>>>>>> +
>>>>>> +       Return Parameters:
>>>>>> +
>>>>>> +       This command starts passive LE scan, and filter received
>>>>>> advertisements: if AD contains both TX power level and Service
>>>>>> Solicitation, and UUID is contained in Service Solicitation and
>>>>>> computed pathloss is smaller than max_pathloss parameter, then a
>>>>>> Device Found event will be sent.
>>>>>
>>>>> generally we tried to avoid to specific LE only commands. Our attempt=
s with the kernel APIs where to make them as generic as possible. So I thin=
k something along the lines of adding and removing UUID filters should be s=
omething we should target at.
>>>>>
>>>>> Add UUID Notification Filter
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D
>>>>>
>>>>>       Command Parameters:     Address_Type (1 Octet)
>>>>>                               UUID (16 Octets)
>>>>>                               Max_Pathloss (1 Octet)
>>>>>       Return Parameters:      Address_Type (1 Octet)
>>>>>
>>>>>
>>>>> Remove UUID Notification Filter
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>
>>>>>       Command Parameters:     Address_Type (1 Octet)
>>>>>                               UUID (16 Octets)
>>>>>       Return Parameters:      Address_Type (1 Octet)
>>>>>
>>>>> The Address_Type would be defined the same way as for Start Discovery=
 command.
>>>>>
>>>>> Now we can discuss if on the mgmt level we want to use pathloss or ra=
ther a range of RSSI. I have seen that RSSI is preferred and that the trans=
lation from RSSI and advertising data with TX power level is done in the da=
emon. So for kernel API it might be better to expose RSSI range.
>>>>>
>>>>> Besides a RSSI range, we might also want to allow giving a RSSI thres=
hold that defines how much the RSSI to change between events to be reported=
 again.
>>>>
>>>> Some mobile devices that I'm working can change power level for
>>>> advertising, that makes raw RSSI filter useless. It must be pathloss,
>>>> that's why I require TX power in advertisement.
>>>
>>> you will still get it, but just have to do the math in the daemon. The =
only difference is that the daemon gets woken up for RSSI ranges that in th=
e end you might filter out.
>>>
>>>>>
>>>>> That said, the danger with a generic notification filter like this is=
 that we can not implement it efficiently with current hardware without ven=
dor commands. If you use duplicate filtering, then Broadcom and CSR chips b=
ehave fully different. Broadcom only filters on BD_ADDR and report devices =
once no matter what the RSSI, while CSR will report the same device again i=
f the RSSI changes.
>>>>>
>>>>> With that in mind, it might be safer to introduce a one-shot mgmt API=
 that needs to be repeatedly called to keep scanning for new devices.
>>>>>
>>>>> Start Service Discovery
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>
>>>>>       Command Parameters:     Address_Type (1 Octet)
>>>>>                               Min_RSSI (1 Octet)
>>>>>                               Max_RSSI (1 Octet)
>>>>>                               Num_UUID (1 Octet)
>>>>>                               UUID[i] (16 Octets)
>>>>>       Return Parameters:      Address_Type (1 Octet)
>>>>>
>>>>> Stop Service Discovery
>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>
>>>>>       Command Parameters:     Address_Type (1 Octet)
>>>>>       Return Parameters:      Address_Type (1 Octet)
>>>>>
>>>>> Without having dedicated controller support for filtering, having suc=
h a dedicated discovery for services with a specific UUID seems to be more =
appropriate and more safe. Since we could always add controller based filte=
ring for background scanning once there is controller support.
>>>>
>>>> I preffer the "Add UUID Notification Filter" and "Remove UUID
>>>> Notification Filter". For controllers like Broadcom that reports only
>>>> one advertisement can we restart scan internally in kernel ? Why
>>>> propagate that event higher ?
>>>
>>> Personally I would prefer just adding another filter as well. However i=
t might be better to start with something like a one-shot handling and see =
where that is taking us. It is a little bit less intrusive and you could al=
so use active scanning. At the same time you could also easily make it work=
 for BR/EDR. I am thinking here along the lines of something wanting to pai=
r a mouse. You just give the HID UUID and start interleaved discovery for t=
hat service on both transports.
>>
>> I send a proposal for doing an one-shot service discovery. So you can pr=
opose a RSSI Threshold and a list of possible UUIDs to look for.
>>
>> If RSSI Threshold is 127 or the Num_UUID is 0, then it will behave the s=
ame way as Start Discovery. I think this is how we should get started to ta=
ckle this problem.
>>
>> Regards
>>
>> Marcel
>>

Re: [RFC] need for new scan api for bluetooth low energy.

[Marcel Holtmann]

Hi Jakub,

you really should not top-post on this mailing list. It breaks the flow for everybody else.

> I started implementing service scanning and I have a questions:
> 
> We initially agreed on version that uses le deduplication that needs
> to be restarted after each device found.
> When I started looking into that it's going to be much more restarting:
> - need to restart after each packet from device that advertise service
> we're interested that's not in RSSI range. That's because controller
> that I use doesn't send advertisement again when RSSI changes.
> - need to restart every 0.5 or 1 second in case when some device added
> service I'm interested at to it's advertisement

I think first we might need to have a HCI_QUIRK that we can set on hci_dev->quirks that tells us what kind of behavior the controller has. We can argue what is the expected default since the specification is pretty much unclear.

Since for all known devices, we are using the whitelist, I am happy to accept that Broadcom controllers just need a quirk to restart scanning. We have to find a nice sweat spot on when to do it. So this would need some testing. So just go and play with it.

My initial take would be that when you see a device with the matching filter, then you start a timeout. And after that timeout restart. Postpone the timeout if you find more devices that match your filter up to x seconds. That should get your there.

> Can we conseider implementation using le scan without deduplication,
> and do packet de-duplication in kernel in hci_le_adv_report_evt ?
> I've started implementing it this way and it works really nice. The
> issue here might be that it might consume more energy.

If you get a really bad controller, then this will go ballistic. Trust me, we had this and some controllers will report every single advertising packet up to the host. You can not process this quickly enough. It gets worse if you have a peripheral that advertises with a high duty cycle. Then you will consume extra power.

You need to test this in a real scenario where you actually have tons and tons peripherals around. It can get really bad. I mean it all depends on what you are actually want to do and how much precision you need. My first instinct is to just restart scanning to get new results.

Regards

Marcel