BlueZ Central to Peripheral latency issue

BlueZ Central to Peripheral latency issue

[Mathias Baert]

Hi,

We are using the BlueZ 5.48 version on a Raspberry PI with Raspbian Stretch 9.1.

The setup is this PI connected with a Nordic Semiconductor nRF52840
device, via an IPv6 over BLE connection. The connection is using a
connection interval of 7.5 ms. Via throughput measurements with iperf,
both ways (central to peripheral and peripheral to central), we are
able to achieve maximally ~ 100 kbps (using the 1 Mbps PHY).

However, when looking into individual packet exchanges, we notice a
significant delay (up to 1 sec) in the RTT when pinging from the
peripheral to the BlueZ central and back. We also see a huge
fluctuation in this value and it also depends on the intervals at
which the pings are fired (lower throughput gives a much higher
average individual latency). When firing ping packets at a 1 sec
interval, it is definitely visible.

When looking into this, we found that the latency between the
peripheral and the central is quite stable and low enough. But the
latency between the central and peripheral is fluctuating a lot and is
generally quite high. Is this something you have noticed before? We
think that it could be a scheduling issue on the kernel, where higher
throughput gives more priority to Bluetooth communication?

Thanks in advance.

Kind regards,

Mathias Baert

Re: BlueZ Central to Peripheral latency issue

[Luiz Augusto von Dentz]

Hi Mathias,

On Tue, Jul 2, 2019 at 3:33 PM Mathias Baert <mathiasrobaert@gmail.com> wrote:
>
> Hi,
>
> We are using the BlueZ 5.48 version on a Raspberry PI with Raspbian Stretch 9.1.
>
> The setup is this PI connected with a Nordic Semiconductor nRF52840
> device, via an IPv6 over BLE connection. The connection is using a
> connection interval of 7.5 ms. Via throughput measurements with iperf,
> both ways (central to peripheral and peripheral to central), we are
> able to achieve maximally ~ 100 kbps (using the 1 Mbps PHY).
>
> However, when looking into individual packet exchanges, we notice a
> significant delay (up to 1 sec) in the RTT when pinging from the
> peripheral to the BlueZ central and back. We also see a huge
> fluctuation in this value and it also depends on the intervals at
> which the pings are fired (lower throughput gives a much higher
> average individual latency). When firing ping packets at a 1 sec
> interval, it is definitely visible.
>
> When looking into this, we found that the latency between the
> peripheral and the central is quite stable and low enough. But the
> latency between the central and peripheral is fluctuating a lot and is
> generally quite high. Is this something you have noticed before? We
> think that it could be a scheduling issue on the kernel, where higher
> throughput gives more priority to Bluetooth communication?

So I assume this is using the experimental IPSP support with Zephyr as
peripheral? Usually, the Linux side is quite a bit more complex so it
is not surprising it can take more time but not 1 sec. so it got to be
something that is causing the extra lag, perhaps you can paste the
actual commands you are using to ping one another.

-- 
Luiz Augusto von Dentz

Re: BlueZ Central to Peripheral latency issue

[Mathias Baert]

Hi Barry,

I'm using the IPv6 over BLE implementation, which does not use the
GATT layer for actual data communication. It directly interacts with
the L2CAP channel, via LE Credit Based Flow Control Mode. So I'm using
neither of these methods.

Kind regards,

Mathias

On Tue, Jul 2, 2019 at 3:34 PM Barry Byford <31baz66@gmail.com> wrote:
>
> Hi Mathias,
>
> Are you using WriteValue:
> https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/doc/gatt-api.txt?h=5.48#n79
> or AcquireWrite
> https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/doc/gatt-api.txt?h=5.48#n94
> method?
>
> Regards,
> Barry
>
> On Tue, 2 Jul 2019 at 13:32, Mathias Baert <mathiasrobaert@gmail.com> wrote:
> >
> > Hi,
> >
> > We are using the BlueZ 5.48 version on a Raspberry PI with Raspbian Stretch 9.1.
> >
> > The setup is this PI connected with a Nordic Semiconductor nRF52840
> > device, via an IPv6 over BLE connection. The connection is using a
> > connection interval of 7.5 ms. Via throughput measurements with iperf,
> > both ways (central to peripheral and peripheral to central), we are
> > able to achieve maximally ~ 100 kbps (using the 1 Mbps PHY).
> >
> > However, when looking into individual packet exchanges, we notice a
> > significant delay (up to 1 sec) in the RTT when pinging from the
> > peripheral to the BlueZ central and back. We also see a huge
> > fluctuation in this value and it also depends on the intervals at
> > which the pings are fired (lower throughput gives a much higher
> > average individual latency). When firing ping packets at a 1 sec
> > interval, it is definitely visible.
> >
> > When looking into this, we found that the latency between the
> > peripheral and the central is quite stable and low enough. But the
> > latency between the central and peripheral is fluctuating a lot and is
> > generally quite high. Is this something you have noticed before? We
> > think that it could be a scheduling issue on the kernel, where higher
> > throughput gives more priority to Bluetooth communication?
> >
> > Thanks in advance.
> >
> > Kind regards,
> >
> > Mathias Baert

Re: BlueZ Central to Peripheral latency issue

[Mathias Baert]

[-- Attachment #1: Type: text/plain, Size: 6935 bytes --]

Hi Luiz,

Thank you for your reply. Apologies for my late reply back to you, I
was on holiday.

Actually, I'm using Nordic Semiconductor's implementation of IPv6 over
BLE for the 6LoWPAN peripheral node. More specifically, a custom
modified version of the UDP client example
(https://infocenter.nordicsemi.com/topic/com.nordic.infocenter.sdk5.v15.3.0/iot_sdk_app_ipv6_stack_udp_client.html?cp=5_1_4_2_4_10_0_0).
I started from this example, added serial SLIP based communication to
it in order to interact with the BLE application from outside the
board (send/receive payloads, update BLE connection status, etc.) and
allowed the BLE application to send and receive COAP packets by
preconfiguring the correct port and IP address to send to.

The Zephyr example can also suffice to reproduce our problem and so I
adapted the setup to the figure below, to be consistent with the
example (which preconfigures the IP address 2001:db8::1) but also with
what we are trying to achieve in our initial setup. In our initial
setup, we want to make the 6LoWPAN peripheral node part of an existing
IPv6 network, as shown by the figure in the attachment. Therefore, the
6LoWPAN border router acts as a neighbor proxy for the 6LoWPAN
peripheral node towards the existing network. Ultimately, any
application server within the existing network, is capable of sending
and receiving packets to/from the 6LoWPAN peripheral node.

Some configuration steps are required on the gateway/neighbor proxy to
setup the network shown in the figure:
          # Some preconfiguration
sysctl -w net.ipv6.conf.all.forwarding=1 # Enable forwarding
sysctl -w net.ipv6.conf.all.proxy_ndp=1 # Enable as neighbor proxy
modprobe bluetooth_6lowpan # Enable changing of 6lowpan config files
echo 1 > /sys/kernel/debug/bluetooth/6lowpan_enable # Enable 6lowpan
on this gateway

echo 6 > /sys/kernel/debug/bluetooth/hci0/conn_min_interval
echo 6 > /sys/kernel/debug/bluetooth/hci0/conn_max_interval

ip addr add 2001:db8::2/64 dev eth0

# Setting up the connection
echo "connect MAC 2" > /sys/kernel/debug/bluetooth/6lowpan_control #
change MAC with actual BLE MAC address

# Add route & neighbor proxy entry
ip -6 neigh add proxy 2001:db8::1 dev eth0  # To enable this gateway
to answer Neighbor Solicitation messages for the 6LoWPAN node
ip -6 neigh add proxy 2001:db8::3 dev bt0 # Normally this is
unnecessary as in our regular setup we use stateless autoconfiguration
to give the peripheral a public IPv6 address and do the exchange with
on-link flag to 0, ensuring that the peripheral
                                                            #  will
always send his packets to his default gateway (the 6LoWPAN gateway)
without performing any Neighbor Discovery. However, using the Zephyr
example with static address, this line is necessary to
                                                            # allow
Neighbor Discovery from the 6LoWPAN peripheral node.
ip route add 2001:db8::1 dev bt0

On the application server, the address should also be configured:
ip addr add 2001:db8::3/64 dev eth0

Unfortunately, I'm not able to configure this setup myself. When
following the steps described here
(https://docs.zephyrproject.org/latest/samples/bluetooth/ipsp/README.html)
to verify if the IPv6 over BLE connection was successfully
established, I'm not able to ping the device. The bt0 interface is
established and the BLE connection is there (checked via hcitool con)
but when pinging the all local-link nodes address on the bt0 interface
(ping6 -I bt0 ff02::1), only the bt0 interface link-local address is
reachable...

Should the setup have been successful, we would have tested the
latency of the connection in the same way as we tested it in our
initial setup: via nping (https://linux.die.net/man/1/nping). Using
the echo server (using port 4242) present on the Zephyr example, we
think this should also work, using this command:
nping -c 1 --delay 500ms -g 1884 -p 4242 2001:db8::1 --udp --data test

This command shows the RTT after completion. In our initial setup,
several runs give very fluctuating values... We could then see were
the fluctuation came from by running the application server &
peripheral on the same device, emulating the setup but getting the
advantage of a shared system time. By looking at the tcpdumps, we
could see that the peripheral to central side is quite stable but the
central to peripheral side is fluctuating. We expect that the same
fluctuation will occur in the simpler setup with Zephyr depicted in
the figure.

So my questions are:
Are you able to setup a working connection with the Zephyr example?
And which BlueZ version do you use for that? We used two different
5.xx versions but no luck. We also tried it with two different boards
(nRF52840 & nRF52832) but no luck either.

If you are able to replicate the setup shown in the figure, do you
also see the fluctuating RTT values when running nping? If yes, is
this something you have encountered before?

Thank you very much for your time.

Kind regards,

Mathias

On Tue, Jul 2, 2019 at 9:22 PM Luiz Augusto von Dentz
<luiz.dentz@gmail.com> wrote:
>
> Hi Mathias,
>
> On Tue, Jul 2, 2019 at 3:33 PM Mathias Baert <mathiasrobaert@gmail.com> wrote:
> >
> > Hi,
> >
> > We are using the BlueZ 5.48 version on a Raspberry PI with Raspbian Stretch 9.1.
> >
> > The setup is this PI connected with a Nordic Semiconductor nRF52840
> > device, via an IPv6 over BLE connection. The connection is using a
> > connection interval of 7.5 ms. Via throughput measurements with iperf,
> > both ways (central to peripheral and peripheral to central), we are
> > able to achieve maximally ~ 100 kbps (using the 1 Mbps PHY).
> >
> > However, when looking into individual packet exchanges, we notice a
> > significant delay (up to 1 sec) in the RTT when pinging from the
> > peripheral to the BlueZ central and back. We also see a huge
> > fluctuation in this value and it also depends on the intervals at
> > which the pings are fired (lower throughput gives a much higher
> > average individual latency). When firing ping packets at a 1 sec
> > interval, it is definitely visible.
> >
> > When looking into this, we found that the latency between the
> > peripheral and the central is quite stable and low enough. But the
> > latency between the central and peripheral is fluctuating a lot and is
> > generally quite high. Is this something you have noticed before? We
> > think that it could be a scheduling issue on the kernel, where higher
> > throughput gives more priority to Bluetooth communication?
>
> So I assume this is using the experimental IPSP support with Zephyr as
> peripheral? Usually, the Linux side is quite a bit more complex so it
> is not surprising it can take more time but not 1 sec. so it got to be
> something that is causing the extra lag, perhaps you can paste the
> actual commands you are using to ping one another.
>
> --
> Luiz Augusto von Dentz

[-- Attachment #2: Zephyr_setup.png --]
[-- Type: image/png, Size: 18266 bytes --]