Adding support for DuplicateData into the kernel

Adding support for DuplicateData into the kernel

[Scott Shawcroft]

Hi!

I'm trying to create a bridge between BLE advertisements and the web
for sensor data logging. The BLE advertisements use manufacturer data
to transmit the sensor data. I'm running on a Raspberry Pi 3b.

I've been trying to use the Bluez dbus interface with DuplicateData,
trying both true and false, and I never seem to get duplicates back. I
dug into it via hcidump and noticed that the scan enable still has
filter duplicates set to true regardless. I can only manage to get it
set to False when using hcitool.

I believe I've tracked the issue down into the kernel here:
https://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next.git/tree/net/bluetooth/hci_request.c#n858
where enabling duplicate filtering is hardcoded. It seems that hcitool
side steps this by talking directly to the hci device.

Could someone confirm I'm on the right track? I'd be willing to plumb
the DuplicateData value through to the kernel if someone could help me
find the call path down.

Thanks!
~Scott

Re: Adding support for DuplicateData into the kernel

[Luiz Augusto von Dentz]

Hi Scott,

On Thu, Feb 20, 2020 at 8:34 PM Scott Shawcroft <scott@adafruit.com> wrote:
>
> Hi!
>
> I'm trying to create a bridge between BLE advertisements and the web
> for sensor data logging. The BLE advertisements use manufacturer data
> to transmit the sensor data. I'm running on a Raspberry Pi 3b.
>
> I've been trying to use the Bluez dbus interface with DuplicateData,
> trying both true and false, and I never seem to get duplicates back. I
> dug into it via hcidump and noticed that the scan enable still has
> filter duplicates set to true regardless. I can only manage to get it
> set to False when using hcitool.
>
> I believe I've tracked the issue down into the kernel here:
> https://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next.git/tree/net/bluetooth/hci_request.c#n858
> where enabling duplicate filtering is hardcoded. It seems that hcitool
> side steps this by talking directly to the hci device.

Yep, the kernel code don't actually allow setting the duplicate filter
unfortunately.

> Could someone confirm I'm on the right track? I'd be willing to plumb
> the DuplicateData value through to the kernel if someone could help me
> find the call path down.

In the past Marcel was against turning off the duplicate filtering but
since we do have a similar setting over D-Bus, though the later shall
not be confused since tracks duplicates by itself since over D-Bus we
are doing it to prevent duplicate signals, Im with the opinion that
DuplicateData could be used to disable duplicate filtering, default
should be still be enabled though.

-- 
Luiz Augusto von Dentz

Re: Adding support for DuplicateData into the kernel

[Scott Shawcroft]

On Mon, Feb 24, 2020 at 10:57 AM Luiz Augusto von Dentz
<luiz.dentz@gmail.com> wrote:
>
> Hi Scott,
>
> On Thu, Feb 20, 2020 at 8:34 PM Scott Shawcroft <scott@adafruit.com> wrote:
> >
> > Hi!
> >
> > I'm trying to create a bridge between BLE advertisements and the web
> > for sensor data logging. The BLE advertisements use manufacturer data
> > to transmit the sensor data. I'm running on a Raspberry Pi 3b.
> >
> > I've been trying to use the Bluez dbus interface with DuplicateData,
> > trying both true and false, and I never seem to get duplicates back. I
> > dug into it via hcidump and noticed that the scan enable still has
> > filter duplicates set to true regardless. I can only manage to get it
> > set to False when using hcitool.
> >
> > I believe I've tracked the issue down into the kernel here:
> > https://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next.git/tree/net/bluetooth/hci_request.c#n858
> > where enabling duplicate filtering is hardcoded. It seems that hcitool
> > side steps this by talking directly to the hci device.
>
> Yep, the kernel code don't actually allow setting the duplicate filter
> unfortunately.
>
> > Could someone confirm I'm on the right track? I'd be willing to plumb
> > the DuplicateData value through to the kernel if someone could help me
> > find the call path down.
>
> In the past Marcel was against turning off the duplicate filtering but
> since we do have a similar setting over D-Bus, though the later shall
> not be confused since tracks duplicates by itself since over D-Bus we
> are doing it to prevent duplicate signals, Im with the opinion that
> DuplicateData could be used to disable duplicate filtering, default
> should be still be enabled though.
>
> --
> Luiz Augusto von Dentz

Thanks for the reply!

I agree the default should be to filter duplicates. I'd love to see
the Dbus setting actually used in the kernel. Did I find the correct
place to change? I'm not sure how it is called from bluez though.
(Maybe through the mgmt API?)

In the meantime I'm using hcidump + hcitool just fine.

Thanks,
Scott

Re: Adding support for DuplicateData into the kernel

[Luiz Augusto von Dentz]

Hi Scott,

On Mon, Feb 24, 2020 at 1:51 PM Scott Shawcroft <scott@adafruit.com> wrote:
>
> On Mon, Feb 24, 2020 at 10:57 AM Luiz Augusto von Dentz
> <luiz.dentz@gmail.com> wrote:
> >
> > Hi Scott,
> >
> > On Thu, Feb 20, 2020 at 8:34 PM Scott Shawcroft <scott@adafruit.com> wrote:
> > >
> > > Hi!
> > >
> > > I'm trying to create a bridge between BLE advertisements and the web
> > > for sensor data logging. The BLE advertisements use manufacturer data
> > > to transmit the sensor data. I'm running on a Raspberry Pi 3b.
> > >
> > > I've been trying to use the Bluez dbus interface with DuplicateData,
> > > trying both true and false, and I never seem to get duplicates back. I
> > > dug into it via hcidump and noticed that the scan enable still has
> > > filter duplicates set to true regardless. I can only manage to get it
> > > set to False when using hcitool.
> > >
> > > I believe I've tracked the issue down into the kernel here:
> > > https://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next.git/tree/net/bluetooth/hci_request.c#n858
> > > where enabling duplicate filtering is hardcoded. It seems that hcitool
> > > side steps this by talking directly to the hci device.
> >
> > Yep, the kernel code don't actually allow setting the duplicate filter
> > unfortunately.
> >
> > > Could someone confirm I'm on the right track? I'd be willing to plumb
> > > the DuplicateData value through to the kernel if someone could help me
> > > find the call path down.
> >
> > In the past Marcel was against turning off the duplicate filtering but
> > since we do have a similar setting over D-Bus, though the later shall
> > not be confused since tracks duplicates by itself since over D-Bus we
> > are doing it to prevent duplicate signals, Im with the opinion that
> > DuplicateData could be used to disable duplicate filtering, default
> > should be still be enabled though.
> >
> > --
> > Luiz Augusto von Dentz
>
> Thanks for the reply!
>
> I agree the default should be to filter duplicates. I'd love to see
> the Dbus setting actually used in the kernel. Did I find the correct
> place to change? I'm not sure how it is called from bluez though.
> (Maybe through the mgmt API?)

Yes, we would need to add it to Start Service Discover or create a new
one, while at it we may add support for adding both the interval and
window to SetDiscoveryFilter, Id actually think we might be better off
adding a new command e.g. Start Filtered Discover which can then
accomodate all these details, but it probably won't be very simple to
do as we learn from the addition of the filters making it work with
multiple clients complicates it a little bit, anyway I don't think we
want this to be global but rather per client.


-- 
Luiz Augusto von Dentz

Re: Adding support for DuplicateData into the kernel

[Scott Shawcroft]

On Mon, Feb 24, 2020 at 2:33 PM Luiz Augusto von Dentz
<luiz.dentz@gmail.com> wrote:
>
> Hi Scott,
>
> On Mon, Feb 24, 2020 at 1:51 PM Scott Shawcroft <scott@adafruit.com> wrote:
> >
> > On Mon, Feb 24, 2020 at 10:57 AM Luiz Augusto von Dentz
> > <luiz.dentz@gmail.com> wrote:
> > >
> > > Hi Scott,
> > >
> > > On Thu, Feb 20, 2020 at 8:34 PM Scott Shawcroft <scott@adafruit.com> wrote:
> > > >
> > > > Hi!
> > > >
> > > > I'm trying to create a bridge between BLE advertisements and the web
> > > > for sensor data logging. The BLE advertisements use manufacturer data
> > > > to transmit the sensor data. I'm running on a Raspberry Pi 3b.
> > > >
> > > > I've been trying to use the Bluez dbus interface with DuplicateData,
> > > > trying both true and false, and I never seem to get duplicates back. I
> > > > dug into it via hcidump and noticed that the scan enable still has
> > > > filter duplicates set to true regardless. I can only manage to get it
> > > > set to False when using hcitool.
> > > >
> > > > I believe I've tracked the issue down into the kernel here:
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next.git/tree/net/bluetooth/hci_request.c#n858
> > > > where enabling duplicate filtering is hardcoded. It seems that hcitool
> > > > side steps this by talking directly to the hci device.
> > >
> > > Yep, the kernel code don't actually allow setting the duplicate filter
> > > unfortunately.
> > >
> > > > Could someone confirm I'm on the right track? I'd be willing to plumb
> > > > the DuplicateData value through to the kernel if someone could help me
> > > > find the call path down.
> > >
> > > In the past Marcel was against turning off the duplicate filtering but
> > > since we do have a similar setting over D-Bus, though the later shall
> > > not be confused since tracks duplicates by itself since over D-Bus we
> > > are doing it to prevent duplicate signals, Im with the opinion that
> > > DuplicateData could be used to disable duplicate filtering, default
> > > should be still be enabled though.
> > >
> > > --
> > > Luiz Augusto von Dentz
> >
> > Thanks for the reply!
> >
> > I agree the default should be to filter duplicates. I'd love to see
> > the Dbus setting actually used in the kernel. Did I find the correct
> > place to change? I'm not sure how it is called from bluez though.
> > (Maybe through the mgmt API?)
>
> Yes, we would need to add it to Start Service Discover or create a new
> one, while at it we may add support for adding both the interval and
> window to SetDiscoveryFilter, Id actually think we might be better off
> adding a new command e.g. Start Filtered Discover which can then
> accomodate all these details, but it probably won't be very simple to
> do as we learn from the addition of the filters making it work with
> multiple clients complicates it a little bit, anyway I don't think we
> want this to be global but rather per client.
>
>
> --
> Luiz Augusto von Dentz

I agree it makes sense to do it on a per-client basis. What I'm not
clear on is the call process from the DBUS api down into the kernel's
`hci_req_start_scan` function I found. This is my first time
considering kernel hacking so I'm not sure of the regular process. I
imagine we'll want to turn on duplicates if any dbus client wants them
and then filter them out for any clients that don't.

Thanks,
Scott