[PATCH 0/3] RFC: prioritizing data over HCI

[PATCH 0/3] RFC: prioritizing data over HCI

[Luiz Augusto von Dentz]

From: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>

This set of patches uses the priority of the socket, set via SO_PRIORITY,
to prioritized data over HCI. Current priority range is from 0-7, where
0-6 priority can be set without any exatra capability which is according
to SO_PRIORITY documentation (socket(7)):

SO_PRIORITY

	Set  the  protocol-defined  priority  for  all packets to be sent
	on this socket.  Linux uses this value to order the networking
	queues: packets with a higher priority may be processed first
	depending on the selected device queueing discipline....
	Setting  a  priority outside  the  range  0  to  6 requires the
	CAP_NET_ADMIN capability.

This also enables in the future the use of guaranteed channels as it is
necessary to prioritized them over best effort (default) channels,
Bluetooth V4.0 spec (page 1497):

7.9 PRIORITIZING DATA OVER HCI

	In order for guaranteed channels to meet their guarantees, L2CAP
	should prioritize traffic over the HCI transport in devices that
	support HCI. Packets for Guaranteed channels should receive higher
	priority than packets for Best Effort channels.

Note: to be able to use guaranteed channels we need to negotiate QoS
parameters both in L2CAP and HCI, this is outside of the scope of this
patches since it probably requires new socket options to L2CAP sockets.

The main use case for this is A2DP, many people complained that when they
are using their HID devices (mouses) together with headset the mouse has
higher priority, which btw is not true since currently there is no
priority and connections receives the same quote.

With this changes it may reduce the problem since PulseAudio already sets
A2DP socket as low latency (priority 6), which means audio packets will be
processed before the HID packets, but this only affects tx and there still
exist the problem of some HID not letting us being master.

Ive been testing this using A2DP and HID connected simultaneusly without a
problem, though connecting/paging still makes audio skips, and also tried
OBEX transfers together to see if audio skips, indeed it does when HID is
active but it works better than having no priority (skips less frequent)
and in case of using priority 7 the audio doesn't skip anymore but it
eventually stall the OBEX transfer (this is maybe because OpenOBEX loop
while trying to write to socket and don't wait for POLLOUT).

Luiz Augusto von Dentz (3):
  Bluetooth: prioritizing data over HCI
  Bluetooth: set skbuffer priority based on L2CAP socket priority
  Bluetooth: make use sk_priority to priritize RFCOMM packets

 include/net/bluetooth/hci_core.h |    6 ++-
 include/net/bluetooth/l2cap.h    |    3 +-
 net/bluetooth/hci_conn.c         |    8 ++-
 net/bluetooth/hci_core.c         |  141 +++++++++++++++++++++++++++----------
 net/bluetooth/l2cap_core.c       |   47 +++++++++----
 net/bluetooth/l2cap_sock.c       |    2 +-
 net/bluetooth/rfcomm/core.c      |   51 +++++++++----
 net/bluetooth/rfcomm/sock.c      |    8 ++
 8 files changed, 194 insertions(+), 72 deletions(-)

-- 
1.7.6

[RFC 1/3] Bluetooth: prioritizing data over HCI

[Luiz Augusto von Dentz]

From: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>

This implement priority based scheduler using skbuffer priority set via
SO_PRIORITY socket option.

It reuses the quote calculation from old scheduler, but the difference is
that the number of connections used to calculate the quote is by priority,
so the quote may not be devided equally per connection if their priorities
are different.

To avoid starvation the priority is recalculated in a round robin fashion
so that the remaining queues are upgraded by 1 in priority until
HCI_PRIO_MAX - 1 (6).

HCI_PRIO_MAX (7) is considered special, it requires CAP_NET_ADMIN
capability and is used to send time critical command/respose for L2CAP
and RFCOMM to avoid timeouts, in addition to that it can be used to
provide more guaranteed channels but may starve other connections thus
need to be used with care.

Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
---
 include/net/bluetooth/hci_core.h |    6 ++-
 net/bluetooth/hci_conn.c         |    8 ++-
 net/bluetooth/hci_core.c         |  141 +++++++++++++++++++++++++++----------
 net/bluetooth/rfcomm/sock.c      |    8 ++
 4 files changed, 122 insertions(+), 41 deletions(-)

diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h
index 8f441b8..be226b0 100644
--- a/include/net/bluetooth/hci_core.h
+++ b/include/net/bluetooth/hci_core.h
@@ -32,6 +32,10 @@
 #define HCI_PROTO_L2CAP	0
 #define HCI_PROTO_SCO	1
 
+/* HCI priority queues */
+#define HCI_PRIO_QUEUES	8
+#define HCI_PRIO_MAX	HCI_PRIO_QUEUES - 1
+
 /* HCI Core structures */
 struct inquiry_data {
 	bdaddr_t	bdaddr;
@@ -274,7 +278,7 @@ struct hci_conn {
 
 	unsigned int	sent;
 
-	struct sk_buff_head data_q;
+	struct sk_buff_head data_q[HCI_PRIO_QUEUES];
 
 	struct timer_list disc_timer;
 	struct timer_list idle_timer;
diff --git a/net/bluetooth/hci_conn.c b/net/bluetooth/hci_conn.c
index ea7f031..4c09738 100644
--- a/net/bluetooth/hci_conn.c
+++ b/net/bluetooth/hci_conn.c
@@ -336,6 +336,7 @@ static void hci_conn_auto_accept(unsigned long arg)
 struct hci_conn *hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t *dst)
 {
 	struct hci_conn *conn;
+	int i;
 
 	BT_DBG("%s dst %s", hdev->name, batostr(dst));
 
@@ -372,7 +373,8 @@ struct hci_conn *hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t *dst)
 		break;
 	}
 
-	skb_queue_head_init(&conn->data_q);
+	for (i = 0; i < HCI_PRIO_QUEUES; i++)
+		skb_queue_head_init(&conn->data_q[i]);
 
 	setup_timer(&conn->disc_timer, hci_conn_timeout, (unsigned long)conn);
 	setup_timer(&conn->idle_timer, hci_conn_idle, (unsigned long)conn);
@@ -401,6 +403,7 @@ struct hci_conn *hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t *dst)
 int hci_conn_del(struct hci_conn *conn)
 {
 	struct hci_dev *hdev = conn->hdev;
+	int i;
 
 	BT_DBG("%s conn %p handle %d", hdev->name, conn, conn->handle);
 
@@ -438,7 +441,8 @@ int hci_conn_del(struct hci_conn *conn)
 
 	tasklet_enable(&hdev->tx_task);
 
-	skb_queue_purge(&conn->data_q);
+	for (i = 0; i < HCI_PRIO_QUEUES; i++)
+		skb_queue_purge(&conn->data_q[i]);
 
 	hci_conn_put_device(conn);
 
diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c
index ec0bc3f..7c34111 100644
--- a/net/bluetooth/hci_core.c
+++ b/net/bluetooth/hci_core.c
@@ -1975,10 +1975,21 @@ static void hci_add_acl_hdr(struct sk_buff *skb, __u16 handle, __u16 flags)
 	hdr->dlen   = cpu_to_le16(len);
 }
 
+static struct sk_buff_head *hci_prio_queue(struct hci_conn *conn, __u32 prio)
+{
+	BT_DBG("conn %p priority %u", conn, prio);
+
+	if (prio > HCI_PRIO_MAX)
+		prio = HCI_PRIO_MAX;
+
+	return &conn->data_q[prio];
+}
+
 void hci_send_acl(struct hci_conn *conn, struct sk_buff *skb, __u16 flags)
 {
 	struct hci_dev *hdev = conn->hdev;
 	struct sk_buff *list;
+	struct sk_buff_head *queue;
 
 	BT_DBG("%s conn %p flags 0x%x", hdev->name, conn, flags);
 
@@ -1986,12 +1997,14 @@ void hci_send_acl(struct hci_conn *conn, struct sk_buff *skb, __u16 flags)
 	bt_cb(skb)->pkt_type = HCI_ACLDATA_PKT;
 	hci_add_acl_hdr(skb, conn->handle, flags);
 
+	queue = hci_prio_queue(conn, skb->priority);
+
 	list = skb_shinfo(skb)->frag_list;
 	if (!list) {
 		/* Non fragmented */
 		BT_DBG("%s nonfrag skb %p len %d", hdev->name, skb, skb->len);
 
-		skb_queue_tail(&conn->data_q, skb);
+		skb_queue_tail(queue, skb);
 	} else {
 		/* Fragmented */
 		BT_DBG("%s frag %p len %d", hdev->name, skb, skb->len);
@@ -1999,9 +2012,9 @@ void hci_send_acl(struct hci_conn *conn, struct sk_buff *skb, __u16 flags)
 		skb_shinfo(skb)->frag_list = NULL;
 
 		/* Queue all fragments atomically */
-		spin_lock_bh(&conn->data_q.lock);
+		spin_lock_bh(&queue->lock);
 
-		__skb_queue_tail(&conn->data_q, skb);
+		__skb_queue_tail(queue, skb);
 
 		flags &= ~ACL_START;
 		flags |= ACL_CONT;
@@ -2014,10 +2027,10 @@ void hci_send_acl(struct hci_conn *conn, struct sk_buff *skb, __u16 flags)
 
 			BT_DBG("%s frag %p len %d", hdev->name, skb, skb->len);
 
-			__skb_queue_tail(&conn->data_q, skb);
+			__skb_queue_tail(queue, skb);
 		} while (list);
 
-		spin_unlock_bh(&conn->data_q.lock);
+		spin_unlock_bh(&queue->lock);
 	}
 
 	tasklet_schedule(&hdev->tx_task);
@@ -2029,6 +2042,7 @@ void hci_send_sco(struct hci_conn *conn, struct sk_buff *skb)
 {
 	struct hci_dev *hdev = conn->hdev;
 	struct hci_sco_hdr hdr;
+	struct sk_buff_head *queue;
 
 	BT_DBG("%s len %d", hdev->name, skb->len);
 
@@ -2042,7 +2056,8 @@ void hci_send_sco(struct hci_conn *conn, struct sk_buff *skb)
 	skb->dev = (void *) hdev;
 	bt_cb(skb)->pkt_type = HCI_SCODATA_PKT;
 
-	skb_queue_tail(&conn->data_q, skb);
+	queue = hci_prio_queue(conn, skb->priority);
+	skb_queue_tail(queue, skb);
 	tasklet_schedule(&hdev->tx_task);
 }
 EXPORT_SYMBOL(hci_send_sco);
@@ -2050,35 +2065,41 @@ EXPORT_SYMBOL(hci_send_sco);
 /* ---- HCI TX task (outgoing data) ---- */
 
 /* HCI Connection scheduler */
-static inline struct hci_conn *hci_low_sent(struct hci_dev *hdev, __u8 type, int *quote)
+static inline struct hci_conn *hci_prio_sent(struct hci_dev *hdev, __u8 type,
+					struct sk_buff_head **queue, int *quote)
 {
 	struct hci_conn_hash *h = &hdev->conn_hash;
 	struct hci_conn *conn = NULL;
-	int num = 0, min = ~0;
-	struct list_head *p;
+	int i;
 
 	/* We don't have to lock device here. Connections are always
 	 * added and removed with TX task disabled. */
-	list_for_each(p, &h->list) {
-		struct hci_conn *c;
-		c = list_entry(p, struct hci_conn, list);
+	for (i = HCI_PRIO_MAX; i >= 0; i--) {
+		int num = 0, min = ~0;
+		struct list_head *p;
+		int cnt, q;
 
-		if (c->type != type || skb_queue_empty(&c->data_q))
-			continue;
+		list_for_each(p, &h->list) {
+			struct hci_conn *c;
+			c = list_entry(p, struct hci_conn, list);
 
-		if (c->state != BT_CONNECTED && c->state != BT_CONFIG)
-			continue;
+			if (c->type != type || skb_queue_empty(&c->data_q[i]))
+				continue;
+
+			if (c->state != BT_CONNECTED && c->state != BT_CONFIG)
+				continue;
 
-		num++;
+			num++;
 
-		if (c->sent < min) {
-			min  = c->sent;
-			conn = c;
+			if (c->sent < min) {
+				min  = c->sent;
+				conn = c;
+				*queue = &c->data_q[i];
+			}
 		}
-	}
 
-	if (conn) {
-		int cnt, q;
+		if (!conn)
+			continue;
 
 		switch (conn->type) {
 		case ACL_LINK:
@@ -2098,10 +2119,13 @@ static inline struct hci_conn *hci_low_sent(struct hci_dev *hdev, __u8 type, int
 
 		q = cnt / num;
 		*quote = q ? q : 1;
-	} else
-		*quote = 0;
+		BT_DBG("conn %p index %d quote %d", conn, i, *quote);
+		return conn;
+	}
 
-	BT_DBG("conn %p quote %d", conn, *quote);
+	*quote = 0;
+	*queue = NULL;
+	BT_DBG("conn %p queue %p quote %d", conn, queue, *quote);
 	return conn;
 }
 
@@ -2128,6 +2152,7 @@ static inline void hci_sched_acl(struct hci_dev *hdev)
 {
 	struct hci_conn *conn;
 	struct sk_buff *skb;
+	struct sk_buff_head *queue = NULL;
 	int quote;
 
 	BT_DBG("%s", hdev->name);
@@ -2139,9 +2164,11 @@ static inline void hci_sched_acl(struct hci_dev *hdev)
 			hci_link_tx_to(hdev, ACL_LINK);
 	}
 
-	while (hdev->acl_cnt && (conn = hci_low_sent(hdev, ACL_LINK, &quote))) {
-		while (quote-- && (skb = skb_dequeue(&conn->data_q))) {
-			BT_DBG("skb %p len %d", skb, skb->len);
+	while (hdev->acl_cnt &&
+			(conn = hci_prio_sent(hdev, ACL_LINK, &queue, &quote))) {
+		while (quote-- && (skb = skb_dequeue(queue))) {
+			BT_DBG("skb %p len %d priority %u", skb, skb->len,
+							skb->priority);
 
 			hci_conn_enter_active_mode(conn, bt_cb(skb)->force_active);
 
@@ -2159,13 +2186,16 @@ static inline void hci_sched_sco(struct hci_dev *hdev)
 {
 	struct hci_conn *conn;
 	struct sk_buff *skb;
+	struct sk_buff_head *queue = NULL;
 	int quote;
 
 	BT_DBG("%s", hdev->name);
 
-	while (hdev->sco_cnt && (conn = hci_low_sent(hdev, SCO_LINK, &quote))) {
-		while (quote-- && (skb = skb_dequeue(&conn->data_q))) {
-			BT_DBG("skb %p len %d", skb, skb->len);
+	while (hdev->sco_cnt &&
+			(conn = hci_prio_sent(hdev, SCO_LINK, &queue, &quote))) {
+		while (quote-- && (skb = skb_dequeue(queue))) {
+			BT_DBG("conn %p skb %p len %d priority %u", conn, skb,
+						skb->len, skb->priority);
 			hci_send_frame(skb);
 
 			conn->sent++;
@@ -2179,13 +2209,16 @@ static inline void hci_sched_esco(struct hci_dev *hdev)
 {
 	struct hci_conn *conn;
 	struct sk_buff *skb;
+	struct sk_buff_head *queue = NULL;
 	int quote;
 
 	BT_DBG("%s", hdev->name);
 
-	while (hdev->sco_cnt && (conn = hci_low_sent(hdev, ESCO_LINK, &quote))) {
-		while (quote-- && (skb = skb_dequeue(&conn->data_q))) {
-			BT_DBG("skb %p len %d", skb, skb->len);
+	while (hdev->sco_cnt &&
+			(conn = hci_prio_sent(hdev, ESCO_LINK, &queue, &quote))) {
+		while (quote-- && (skb = skb_dequeue(queue))) {
+			BT_DBG("conn %p skb %p len %d priority %u", conn, skb,
+						skb->len, skb->priority);
 			hci_send_frame(skb);
 
 			conn->sent++;
@@ -2199,6 +2232,7 @@ static inline void hci_sched_le(struct hci_dev *hdev)
 {
 	struct hci_conn *conn;
 	struct sk_buff *skb;
+	struct sk_buff_head *queue = NULL;
 	int quote, cnt;
 
 	BT_DBG("%s", hdev->name);
@@ -2212,9 +2246,10 @@ static inline void hci_sched_le(struct hci_dev *hdev)
 	}
 
 	cnt = hdev->le_pkts ? hdev->le_cnt : hdev->acl_cnt;
-	while (cnt && (conn = hci_low_sent(hdev, LE_LINK, &quote))) {
-		while (quote-- && (skb = skb_dequeue(&conn->data_q))) {
-			BT_DBG("skb %p len %d", skb, skb->len);
+	while (cnt && (conn = hci_prio_sent(hdev, LE_LINK, &queue, &quote))) {
+		while (quote-- && (skb = skb_dequeue(queue))) {
+			BT_DBG("conn %p skb %p len %d priority %u", conn, skb,
+						skb->len, skb->priority);
 
 			hci_send_frame(skb);
 			hdev->le_last_tx = jiffies;
@@ -2229,6 +2264,34 @@ static inline void hci_sched_le(struct hci_dev *hdev)
 		hdev->acl_cnt = cnt;
 }
 
+static void hci_prio_recalculate(struct hci_dev *hdev)
+{
+	struct hci_conn_hash *h = &hdev->conn_hash;
+	int i;
+
+	BT_DBG("%s", hdev->name);
+
+	for (i = HCI_PRIO_MAX - 1; i > 0; i--) {
+		struct list_head *p;
+
+		list_for_each(p, &h->list) {
+			struct hci_conn *c;
+			c = list_entry(p, struct hci_conn, list);
+
+			if (c->sent || skb_queue_empty(&c->data_q[i - 1]))
+				continue;
+
+			if (c->state != BT_CONNECTED && c->state != BT_CONFIG)
+				continue;
+
+			BT_DBG("conn %p index %d promoted to %d", c, i - 1, i);
+
+			skb_queue_splice_tail_init(&c->data_q[i - 1],
+							&c->data_q[i]);
+		}
+	}
+}
+
 static void hci_tx_task(unsigned long arg)
 {
 	struct hci_dev *hdev = (struct hci_dev *) arg;
@@ -2253,6 +2316,8 @@ static void hci_tx_task(unsigned long arg)
 	while ((skb = skb_dequeue(&hdev->raw_q)))
 		hci_send_frame(skb);
 
+	hci_prio_recalculate(hdev);
+
 	read_unlock(&hci_task_lock);
 }
 
diff --git a/net/bluetooth/rfcomm/sock.c b/net/bluetooth/rfcomm/sock.c
index 8f01e6b..0228e1f 100644
--- a/net/bluetooth/rfcomm/sock.c
+++ b/net/bluetooth/rfcomm/sock.c
@@ -561,6 +561,7 @@ static int rfcomm_sock_sendmsg(struct kiocb *iocb, struct socket *sock,
 	struct sock *sk = sock->sk;
 	struct rfcomm_dlc *d = rfcomm_pi(sk)->dlc;
 	struct sk_buff *skb;
+	u32 priority;
 	int sent = 0;
 
 	if (test_bit(RFCOMM_DEFER_SETUP, &d->flags))
@@ -572,6 +573,11 @@ static int rfcomm_sock_sendmsg(struct kiocb *iocb, struct socket *sock,
 	if (sk->sk_shutdown & SEND_SHUTDOWN)
 		return -EPIPE;
 
+	if (msg->msg_flags & MSG_TRYHARD)
+		priority = HCI_PRIO_MAX;
+	else
+		priority = sk->sk_priority;
+
 	BT_DBG("sock %p, sk %p", sock, sk);
 
 	lock_sock(sk);
@@ -597,6 +603,8 @@ static int rfcomm_sock_sendmsg(struct kiocb *iocb, struct socket *sock,
 			break;
 		}
 
+		skb->priority = priority;
+
 		err = rfcomm_dlc_send(d, skb);
 		if (err < 0) {
 			kfree_skb(skb);
-- 
1.7.6

Re: [RFC 1/3] Bluetooth: prioritizing data over HCI

[Peter Hurley]

SGkgTHVpeiwNCg0KT24gV2VkLCAyMDExLTA4LTAzIGF0IDA5OjExIC0wNDAwLCBMdWl6IEF1Z3Vz
dG8gdm9uIERlbnR6IHdyb3RlOg0KPiBGcm9tOiBMdWl6IEF1Z3VzdG8gdm9uIERlbnR6IDxsdWl6
LnZvbi5kZW50ekBpbnRlbC5jb20+DQo+DQo+IFRoaXMgaW1wbGVtZW50IHByaW9yaXR5IGJhc2Vk
IHNjaGVkdWxlciB1c2luZyBza2J1ZmZlciBwcmlvcml0eSBzZXQgdmlhDQo+IFNPX1BSSU9SSVRZ
IHNvY2tldCBvcHRpb24uDQo+DQo+IEl0IHJldXNlcyB0aGUgcXVvdGUgY2FsY3VsYXRpb24gZnJv
bSBvbGQgc2NoZWR1bGVyLCBidXQgdGhlIGRpZmZlcmVuY2UgaXMNCj4gdGhhdCB0aGUgbnVtYmVy
IG9mIGNvbm5lY3Rpb25zIHVzZWQgdG8gY2FsY3VsYXRlIHRoZSBxdW90ZSBpcyBieSBwcmlvcml0
eSwNCj4gc28gdGhlIHF1b3RlIG1heSBub3QgYmUgZGV2aWRlZCBlcXVhbGx5IHBlciBjb25uZWN0
aW9uIGlmIHRoZWlyIHByaW9yaXRpZXMNCj4gYXJlIGRpZmZlcmVudC4NCj4NCj4gVG8gYXZvaWQg
c3RhcnZhdGlvbiB0aGUgcHJpb3JpdHkgaXMgcmVjYWxjdWxhdGVkIGluIGEgcm91bmQgcm9iaW4g
ZmFzaGlvbg0KPiBzbyB0aGF0IHRoZSByZW1haW5pbmcgcXVldWVzIGFyZSB1cGdyYWRlZCBieSAx
IGluIHByaW9yaXR5IHVudGlsDQo+IEhDSV9QUklPX01BWCAtIDEgKDYpLg0KPg0KPiBIQ0lfUFJJ
T19NQVggKDcpIGlzIGNvbnNpZGVyZWQgc3BlY2lhbCwgaXQgcmVxdWlyZXMgQ0FQX05FVF9BRE1J
Tg0KPiBjYXBhYmlsaXR5IGFuZCBpcyB1c2VkIHRvIHNlbmQgdGltZSBjcml0aWNhbCBjb21tYW5k
L3Jlc3Bvc2UgZm9yIEwyQ0FQDQo+IGFuZCBSRkNPTU0gdG8gYXZvaWQgdGltZW91dHMsIGluIGFk
ZGl0aW9uIHRvIHRoYXQgaXQgY2FuIGJlIHVzZWQgdG8NCj4gcHJvdmlkZSBtb3JlIGd1YXJhbnRl
ZWQgY2hhbm5lbHMgYnV0IG1heSBzdGFydmUgb3RoZXIgY29ubmVjdGlvbnMgdGh1cw0KPiBuZWVk
IHRvIGJlIHVzZWQgd2l0aCBjYXJlLg0KPg0KPiBTaWduZWQtb2ZmLWJ5OiBMdWl6IEF1Z3VzdG8g
dm9uIERlbnR6IDxsdWl6LnZvbi5kZW50ekBpbnRlbC5jb20+DQo+IC0tLQ0KPiAgaW5jbHVkZS9u
ZXQvYmx1ZXRvb3RoL2hjaV9jb3JlLmggfCAgICA2ICsrLQ0KPiAgbmV0L2JsdWV0b290aC9oY2lf
Y29ubi5jICAgICAgICAgfCAgICA4ICsrLQ0KPiAgbmV0L2JsdWV0b290aC9oY2lfY29yZS5jICAg
ICAgICAgfCAgMTQxICsrKysrKysrKysrKysrKysrKysrKysrKysrKy0tLS0tLS0tLS0NCj4gIG5l
dC9ibHVldG9vdGgvcmZjb21tL3NvY2suYyAgICAgIHwgICAgOCArKw0KPiAgNCBmaWxlcyBjaGFu
Z2VkLCAxMjIgaW5zZXJ0aW9ucygrKSwgNDEgZGVsZXRpb25zKC0pDQo+DQo+IGRpZmYgLS1naXQg
YS9pbmNsdWRlL25ldC9ibHVldG9vdGgvaGNpX2NvcmUuaCBiL2luY2x1ZGUvbmV0L2JsdWV0b290
aC9oY2lfY29yZS5oDQo+IGluZGV4IDhmNDQxYjguLmJlMjI2YjAgMTAwNjQ0DQo+IC0tLSBhL2lu
Y2x1ZGUvbmV0L2JsdWV0b290aC9oY2lfY29yZS5oDQo+ICsrKyBiL2luY2x1ZGUvbmV0L2JsdWV0
b290aC9oY2lfY29yZS5oDQo+IEBAIC0zMiw2ICszMiwxMCBAQA0KPiAgI2RlZmluZSBIQ0lfUFJP
VE9fTDJDQVAgICAgICAgIDANCj4gICNkZWZpbmUgSENJX1BST1RPX1NDTyAgMQ0KPg0KPiArLyog
SENJIHByaW9yaXR5IHF1ZXVlcyAqLw0KPiArI2RlZmluZSBIQ0lfUFJJT19RVUVVRVMgICAgICAg
IDgNCj4gKyNkZWZpbmUgSENJX1BSSU9fTUFYICAgSENJX1BSSU9fUVVFVUVTIC0gMQ0KPiArDQo+
ICAvKiBIQ0kgQ29yZSBzdHJ1Y3R1cmVzICovDQo+ICBzdHJ1Y3QgaW5xdWlyeV9kYXRhIHsNCj4g
ICAgICAgICBiZGFkZHJfdCAgICAgICAgYmRhZGRyOw0KPiBAQCAtMjc0LDcgKzI3OCw3IEBAIHN0
cnVjdCBoY2lfY29ubiB7DQo+DQo+ICAgICAgICAgdW5zaWduZWQgaW50ICAgIHNlbnQ7DQo+DQo+
IC0gICAgICAgc3RydWN0IHNrX2J1ZmZfaGVhZCBkYXRhX3E7DQo+ICsgICAgICAgc3RydWN0IHNr
X2J1ZmZfaGVhZCBkYXRhX3FbSENJX1BSSU9fUVVFVUVTXTsNCj4NCj4gICAgICAgICBzdHJ1Y3Qg
dGltZXJfbGlzdCBkaXNjX3RpbWVyOw0KPiAgICAgICAgIHN0cnVjdCB0aW1lcl9saXN0IGlkbGVf
dGltZXI7DQo+IGRpZmYgLS1naXQgYS9uZXQvYmx1ZXRvb3RoL2hjaV9jb25uLmMgYi9uZXQvYmx1
ZXRvb3RoL2hjaV9jb25uLmMNCj4gaW5kZXggZWE3ZjAzMS4uNGMwOTczOCAxMDA2NDQNCj4gLS0t
IGEvbmV0L2JsdWV0b290aC9oY2lfY29ubi5jDQo+ICsrKyBiL25ldC9ibHVldG9vdGgvaGNpX2Nv
bm4uYw0KPiBAQCAtMzM2LDYgKzMzNiw3IEBAIHN0YXRpYyB2b2lkIGhjaV9jb25uX2F1dG9fYWNj
ZXB0KHVuc2lnbmVkIGxvbmcgYXJnKQ0KPiAgc3RydWN0IGhjaV9jb25uICpoY2lfY29ubl9hZGQo
c3RydWN0IGhjaV9kZXYgKmhkZXYsIGludCB0eXBlLCBiZGFkZHJfdCAqZHN0KQ0KPiAgew0KPiAg
ICAgICAgIHN0cnVjdCBoY2lfY29ubiAqY29ubjsNCj4gKyAgICAgICBpbnQgaTsNCj4NCj4gICAg
ICAgICBCVF9EQkcoIiVzIGRzdCAlcyIsIGhkZXYtPm5hbWUsIGJhdG9zdHIoZHN0KSk7DQo+DQo+
IEBAIC0zNzIsNyArMzczLDggQEAgc3RydWN0IGhjaV9jb25uICpoY2lfY29ubl9hZGQoc3RydWN0
IGhjaV9kZXYgKmhkZXYsIGludCB0eXBlLCBiZGFkZHJfdCAqZHN0KQ0KPiAgICAgICAgICAgICAg
ICAgYnJlYWs7DQo+ICAgICAgICAgfQ0KPg0KPiAtICAgICAgIHNrYl9xdWV1ZV9oZWFkX2luaXQo
JmNvbm4tPmRhdGFfcSk7DQo+ICsgICAgICAgZm9yIChpID0gMDsgaSA8IEhDSV9QUklPX1FVRVVF
UzsgaSsrKQ0KPiArICAgICAgICAgICAgICAgc2tiX3F1ZXVlX2hlYWRfaW5pdCgmY29ubi0+ZGF0
YV9xW2ldKTsNCj4NCj4gICAgICAgICBzZXR1cF90aW1lcigmY29ubi0+ZGlzY190aW1lciwgaGNp
X2Nvbm5fdGltZW91dCwgKHVuc2lnbmVkIGxvbmcpY29ubik7DQo+ICAgICAgICAgc2V0dXBfdGlt
ZXIoJmNvbm4tPmlkbGVfdGltZXIsIGhjaV9jb25uX2lkbGUsICh1bnNpZ25lZCBsb25nKWNvbm4p
Ow0KPiBAQCAtNDAxLDYgKzQwMyw3IEBAIHN0cnVjdCBoY2lfY29ubiAqaGNpX2Nvbm5fYWRkKHN0
cnVjdCBoY2lfZGV2ICpoZGV2LCBpbnQgdHlwZSwgYmRhZGRyX3QgKmRzdCkNCj4gIGludCBoY2lf
Y29ubl9kZWwoc3RydWN0IGhjaV9jb25uICpjb25uKQ0KPiAgew0KPiAgICAgICAgIHN0cnVjdCBo
Y2lfZGV2ICpoZGV2ID0gY29ubi0+aGRldjsNCj4gKyAgICAgICBpbnQgaTsNCj4NCj4gICAgICAg
ICBCVF9EQkcoIiVzIGNvbm4gJXAgaGFuZGxlICVkIiwgaGRldi0+bmFtZSwgY29ubiwgY29ubi0+
aGFuZGxlKTsNCj4NCj4gQEAgLTQzOCw3ICs0NDEsOCBAQCBpbnQgaGNpX2Nvbm5fZGVsKHN0cnVj
dCBoY2lfY29ubiAqY29ubikNCj4NCj4gICAgICAgICB0YXNrbGV0X2VuYWJsZSgmaGRldi0+dHhf
dGFzayk7DQo+DQo+IC0gICAgICAgc2tiX3F1ZXVlX3B1cmdlKCZjb25uLT5kYXRhX3EpOw0KPiAr
ICAgICAgIGZvciAoaSA9IDA7IGkgPCBIQ0lfUFJJT19RVUVVRVM7IGkrKykNCj4gKyAgICAgICAg
ICAgICAgIHNrYl9xdWV1ZV9wdXJnZSgmY29ubi0+ZGF0YV9xW2ldKTsNCj4NCj4gICAgICAgICBo
Y2lfY29ubl9wdXRfZGV2aWNlKGNvbm4pOw0KPg0KPiBkaWZmIC0tZ2l0IGEvbmV0L2JsdWV0b290
aC9oY2lfY29yZS5jIGIvbmV0L2JsdWV0b290aC9oY2lfY29yZS5jDQo+IGluZGV4IGVjMGJjM2Yu
LjdjMzQxMTEgMTAwNjQ0DQo+IC0tLSBhL25ldC9ibHVldG9vdGgvaGNpX2NvcmUuYw0KPiArKysg
Yi9uZXQvYmx1ZXRvb3RoL2hjaV9jb3JlLmMNCj4gQEAgLTE5NzUsMTAgKzE5NzUsMjEgQEAgc3Rh
dGljIHZvaWQgaGNpX2FkZF9hY2xfaGRyKHN0cnVjdCBza19idWZmICpza2IsIF9fdTE2IGhhbmRs
ZSwgX191MTYgZmxhZ3MpDQo+ICAgICAgICAgaGRyLT5kbGVuICAgPSBjcHVfdG9fbGUxNihsZW4p
Ow0KPiAgfQ0KPg0KPiArc3RhdGljIHN0cnVjdCBza19idWZmX2hlYWQgKmhjaV9wcmlvX3F1ZXVl
KHN0cnVjdCBoY2lfY29ubiAqY29ubiwgX191MzIgcHJpbykNCj4gK3sNCj4gKyAgICAgICBCVF9E
QkcoImNvbm4gJXAgcHJpb3JpdHkgJXUiLCBjb25uLCBwcmlvKTsNCj4gKw0KPiArICAgICAgIGlm
IChwcmlvID4gSENJX1BSSU9fTUFYKQ0KPiArICAgICAgICAgICAgICAgcHJpbyA9IEhDSV9QUklP
X01BWDsNCj4gKw0KPiArICAgICAgIHJldHVybiAmY29ubi0+ZGF0YV9xW3ByaW9dOw0KPiArfQ0K
PiArDQo+ICB2b2lkIGhjaV9zZW5kX2FjbChzdHJ1Y3QgaGNpX2Nvbm4gKmNvbm4sIHN0cnVjdCBz
a19idWZmICpza2IsIF9fdTE2IGZsYWdzKQ0KPiAgew0KPiAgICAgICAgIHN0cnVjdCBoY2lfZGV2
ICpoZGV2ID0gY29ubi0+aGRldjsNCj4gICAgICAgICBzdHJ1Y3Qgc2tfYnVmZiAqbGlzdDsNCj4g
KyAgICAgICBzdHJ1Y3Qgc2tfYnVmZl9oZWFkICpxdWV1ZTsNCj4NCj4gICAgICAgICBCVF9EQkco
IiVzIGNvbm4gJXAgZmxhZ3MgMHgleCIsIGhkZXYtPm5hbWUsIGNvbm4sIGZsYWdzKTsNCj4NCj4g
QEAgLTE5ODYsMTIgKzE5OTcsMTQgQEAgdm9pZCBoY2lfc2VuZF9hY2woc3RydWN0IGhjaV9jb25u
ICpjb25uLCBzdHJ1Y3Qgc2tfYnVmZiAqc2tiLCBfX3UxNiBmbGFncykNCj4gICAgICAgICBidF9j
Yihza2IpLT5wa3RfdHlwZSA9IEhDSV9BQ0xEQVRBX1BLVDsNCj4gICAgICAgICBoY2lfYWRkX2Fj
bF9oZHIoc2tiLCBjb25uLT5oYW5kbGUsIGZsYWdzKTsNCj4NCj4gKyAgICAgICBxdWV1ZSA9IGhj
aV9wcmlvX3F1ZXVlKGNvbm4sIHNrYi0+cHJpb3JpdHkpOw0KPiArDQo+ICAgICAgICAgbGlzdCA9
IHNrYl9zaGluZm8oc2tiKS0+ZnJhZ19saXN0Ow0KPiAgICAgICAgIGlmICghbGlzdCkgew0KPiAg
ICAgICAgICAgICAgICAgLyogTm9uIGZyYWdtZW50ZWQgKi8NCj4gICAgICAgICAgICAgICAgIEJU
X0RCRygiJXMgbm9uZnJhZyBza2IgJXAgbGVuICVkIiwgaGRldi0+bmFtZSwgc2tiLCBza2ItPmxl
bik7DQo+DQo+IC0gICAgICAgICAgICAgICBza2JfcXVldWVfdGFpbCgmY29ubi0+ZGF0YV9xLCBz
a2IpOw0KPiArICAgICAgICAgICAgICAgc2tiX3F1ZXVlX3RhaWwocXVldWUsIHNrYik7DQo+ICAg
ICAgICAgfSBlbHNlIHsNCj4gICAgICAgICAgICAgICAgIC8qIEZyYWdtZW50ZWQgKi8NCj4gICAg
ICAgICAgICAgICAgIEJUX0RCRygiJXMgZnJhZyAlcCBsZW4gJWQiLCBoZGV2LT5uYW1lLCBza2Is
IHNrYi0+bGVuKTsNCj4gQEAgLTE5OTksOSArMjAxMiw5IEBAIHZvaWQgaGNpX3NlbmRfYWNsKHN0
cnVjdCBoY2lfY29ubiAqY29ubiwgc3RydWN0IHNrX2J1ZmYgKnNrYiwgX191MTYgZmxhZ3MpDQo+
ICAgICAgICAgICAgICAgICBza2Jfc2hpbmZvKHNrYiktPmZyYWdfbGlzdCA9IE5VTEw7DQo+DQo+
ICAgICAgICAgICAgICAgICAvKiBRdWV1ZSBhbGwgZnJhZ21lbnRzIGF0b21pY2FsbHkgKi8NCj4g
LSAgICAgICAgICAgICAgIHNwaW5fbG9ja19iaCgmY29ubi0+ZGF0YV9xLmxvY2spOw0KPiArICAg
ICAgICAgICAgICAgc3Bpbl9sb2NrX2JoKCZxdWV1ZS0+bG9jayk7DQo+DQo+IC0gICAgICAgICAg
ICAgICBfX3NrYl9xdWV1ZV90YWlsKCZjb25uLT5kYXRhX3EsIHNrYik7DQo+ICsgICAgICAgICAg
ICAgICBfX3NrYl9xdWV1ZV90YWlsKHF1ZXVlLCBza2IpOw0KPg0KPiAgICAgICAgICAgICAgICAg
ZmxhZ3MgJj0gfkFDTF9TVEFSVDsNCj4gICAgICAgICAgICAgICAgIGZsYWdzIHw9IEFDTF9DT05U
Ow0KPiBAQCAtMjAxNCwxMCArMjAyNywxMCBAQCB2b2lkIGhjaV9zZW5kX2FjbChzdHJ1Y3QgaGNp
X2Nvbm4gKmNvbm4sIHN0cnVjdCBza19idWZmICpza2IsIF9fdTE2IGZsYWdzKQ0KPg0KPiAgICAg
ICAgICAgICAgICAgICAgICAgICBCVF9EQkcoIiVzIGZyYWcgJXAgbGVuICVkIiwgaGRldi0+bmFt
ZSwgc2tiLCBza2ItPmxlbik7DQo+DQo+IC0gICAgICAgICAgICAgICAgICAgICAgIF9fc2tiX3F1
ZXVlX3RhaWwoJmNvbm4tPmRhdGFfcSwgc2tiKTsNCj4gKyAgICAgICAgICAgICAgICAgICAgICAg
X19za2JfcXVldWVfdGFpbChxdWV1ZSwgc2tiKTsNCj4gICAgICAgICAgICAgICAgIH0gd2hpbGUg
KGxpc3QpOw0KPg0KPiAtICAgICAgICAgICAgICAgc3Bpbl91bmxvY2tfYmgoJmNvbm4tPmRhdGFf
cS5sb2NrKTsNCj4gKyAgICAgICAgICAgICAgIHNwaW5fdW5sb2NrX2JoKCZxdWV1ZS0+bG9jayk7
DQo+ICAgICAgICAgfQ0KPg0KPiAgICAgICAgIHRhc2tsZXRfc2NoZWR1bGUoJmhkZXYtPnR4X3Rh
c2spOw0KPiBAQCAtMjAyOSw2ICsyMDQyLDcgQEAgdm9pZCBoY2lfc2VuZF9zY28oc3RydWN0IGhj
aV9jb25uICpjb25uLCBzdHJ1Y3Qgc2tfYnVmZiAqc2tiKQ0KPiAgew0KPiAgICAgICAgIHN0cnVj
dCBoY2lfZGV2ICpoZGV2ID0gY29ubi0+aGRldjsNCj4gICAgICAgICBzdHJ1Y3QgaGNpX3Njb19o
ZHIgaGRyOw0KPiArICAgICAgIHN0cnVjdCBza19idWZmX2hlYWQgKnF1ZXVlOw0KPg0KPiAgICAg
ICAgIEJUX0RCRygiJXMgbGVuICVkIiwgaGRldi0+bmFtZSwgc2tiLT5sZW4pOw0KPg0KPiBAQCAt
MjA0Miw3ICsyMDU2LDggQEAgdm9pZCBoY2lfc2VuZF9zY28oc3RydWN0IGhjaV9jb25uICpjb25u
LCBzdHJ1Y3Qgc2tfYnVmZiAqc2tiKQ0KPiAgICAgICAgIHNrYi0+ZGV2ID0gKHZvaWQgKikgaGRl
djsNCj4gICAgICAgICBidF9jYihza2IpLT5wa3RfdHlwZSA9IEhDSV9TQ09EQVRBX1BLVDsNCj4N
Cj4gLSAgICAgICBza2JfcXVldWVfdGFpbCgmY29ubi0+ZGF0YV9xLCBza2IpOw0KPiArICAgICAg
IHF1ZXVlID0gaGNpX3ByaW9fcXVldWUoY29ubiwgc2tiLT5wcmlvcml0eSk7DQo+ICsgICAgICAg
c2tiX3F1ZXVlX3RhaWwocXVldWUsIHNrYik7DQo+ICAgICAgICAgdGFza2xldF9zY2hlZHVsZSgm
aGRldi0+dHhfdGFzayk7DQo+ICB9DQo+ICBFWFBPUlRfU1lNQk9MKGhjaV9zZW5kX3Njbyk7DQo+
IEBAIC0yMDUwLDM1ICsyMDY1LDQxIEBAIEVYUE9SVF9TWU1CT0woaGNpX3NlbmRfc2NvKTsNCj4g
IC8qIC0tLS0gSENJIFRYIHRhc2sgKG91dGdvaW5nIGRhdGEpIC0tLS0gKi8NCj4NCj4gIC8qIEhD
SSBDb25uZWN0aW9uIHNjaGVkdWxlciAqLw0KPiAtc3RhdGljIGlubGluZSBzdHJ1Y3QgaGNpX2Nv
bm4gKmhjaV9sb3dfc2VudChzdHJ1Y3QgaGNpX2RldiAqaGRldiwgX191OCB0eXBlLCBpbnQgKnF1
b3RlKQ0KPiArc3RhdGljIGlubGluZSBzdHJ1Y3QgaGNpX2Nvbm4gKmhjaV9wcmlvX3NlbnQoc3Ry
dWN0IGhjaV9kZXYgKmhkZXYsIF9fdTggdHlwZSwNCj4gKyAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIHN0cnVjdCBza19idWZmX2hlYWQgKipxdWV1ZSwgaW50ICpxdW90ZSkN
Cj4gIHsNCj4gICAgICAgICBzdHJ1Y3QgaGNpX2Nvbm5faGFzaCAqaCA9ICZoZGV2LT5jb25uX2hh
c2g7DQo+ICAgICAgICAgc3RydWN0IGhjaV9jb25uICpjb25uID0gTlVMTDsNCj4gLSAgICAgICBp
bnQgbnVtID0gMCwgbWluID0gfjA7DQo+IC0gICAgICAgc3RydWN0IGxpc3RfaGVhZCAqcDsNCj4g
KyAgICAgICBpbnQgaTsNCj4NCj4gICAgICAgICAvKiBXZSBkb24ndCBoYXZlIHRvIGxvY2sgZGV2
aWNlIGhlcmUuIENvbm5lY3Rpb25zIGFyZSBhbHdheXMNCj4gICAgICAgICAgKiBhZGRlZCBhbmQg
cmVtb3ZlZCB3aXRoIFRYIHRhc2sgZGlzYWJsZWQuICovDQo+IC0gICAgICAgbGlzdF9mb3JfZWFj
aChwLCAmaC0+bGlzdCkgew0KPiAtICAgICAgICAgICAgICAgc3RydWN0IGhjaV9jb25uICpjOw0K
PiAtICAgICAgICAgICAgICAgYyA9IGxpc3RfZW50cnkocCwgc3RydWN0IGhjaV9jb25uLCBsaXN0
KTsNCj4gKyAgICAgICBmb3IgKGkgPSBIQ0lfUFJJT19NQVg7IGkgPj0gMDsgaS0tKSB7DQo+ICsg
ICAgICAgICAgICAgICBpbnQgbnVtID0gMCwgbWluID0gfjA7DQo+ICsgICAgICAgICAgICAgICBz
dHJ1Y3QgbGlzdF9oZWFkICpwOw0KPiArICAgICAgICAgICAgICAgaW50IGNudCwgcTsNCj4NCj4g
LSAgICAgICAgICAgICAgIGlmIChjLT50eXBlICE9IHR5cGUgfHwgc2tiX3F1ZXVlX2VtcHR5KCZj
LT5kYXRhX3EpKQ0KPiAtICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsNCj4gKyAgICAg
ICAgICAgICAgIGxpc3RfZm9yX2VhY2gocCwgJmgtPmxpc3QpIHsNCj4gKyAgICAgICAgICAgICAg
ICAgICAgICAgc3RydWN0IGhjaV9jb25uICpjOw0KPiArICAgICAgICAgICAgICAgICAgICAgICBj
ID0gbGlzdF9lbnRyeShwLCBzdHJ1Y3QgaGNpX2Nvbm4sIGxpc3QpOw0KPg0KPiAtICAgICAgICAg
ICAgICAgaWYgKGMtPnN0YXRlICE9IEJUX0NPTk5FQ1RFRCAmJiBjLT5zdGF0ZSAhPSBCVF9DT05G
SUcpDQo+IC0gICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0KPiArICAgICAgICAgICAg
ICAgICAgICAgICBpZiAoYy0+dHlwZSAhPSB0eXBlIHx8IHNrYl9xdWV1ZV9lbXB0eSgmYy0+ZGF0
YV9xW2ldKSkNCj4gKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsNCg0K
SSBiZWxpZXZlIHRoYXQgdGhlIGV4aXN0aW5nIHR4IHNjaGVkdWxlciBpcyBhbHJlYWR5IGluZWZm
aWNpZW50IGFuZCB0aGF0DQp0aGlzIGFwcHJvYWNoIHNpZ25pZmljYW50bHkgY29tcG91bmRzIHRo
YXQgaW5lZmZpY2llbmN5Lg0KDQpDb25zaWRlciB0aGUgZm9sbG93aW5nIHNjZW5hcmlvcyB3aXRo
IDQgQUNMIGNvbm5lY3Rpb25zIChrZXlib2FyZCwNCm1vdXNlLCBwaG9uZSwgaGVhZHNldCk6DQoN
CkZvciAqb25lKiBzY2hlZHVsZWQgQUNMIHR4IHBhY2tldCwgdGhlIGNvbm5lY3Rpb24gbGlzdCB3
aWxsIGJlIHNlYXJjaGVkDQozNX40MiB0aW1lcyAtLSB0aGF0J3MgMTQwfjE2OCBjb25uZWN0aW9u
IGxpc3Qgbm9kZSB2aXNpdHMgdG8gZmluZCBvbmUNCnBhY2tldCEgSGVyZSdzIHRoZSBtYXRoOg0K
ICAgICAgICAoNyAtIHByaW9yaXR5IG9mIHR4IHBhY2tldCkgeCA0IGNvbm5lY3Rpb25zID0+IGZv
dW5kIHRoZSBwYWNrZXQNCiAgICAgICAgNyBwcmkgbGV2ZWxzIHggNCBjb25uZWN0aW9ucyA9PiBu
byBtb3JlIEFDTCB0eCBwYWNrZXRzDQogICAgICAgIDcgcHJpIGxldmVscyB4IDQgY29ubmVjdGlv
bnMgPT4gbm8gU0NPIHR4IHBhY2tldHMNCiAgICAgICAgNyBwcmkgbGV2ZWxzIHggNCBjb25uZWN0
aW9ucyA9PiBubyBFU0NPIHR4IHBhY2tldHMNCiAgICAgICAgNyBwcmkgbGV2ZWxzIHggNCBjb25u
ZWN0aW9ucyA9PiBubyBMRSB0eCBwYWNrZXRzDQogICAgICAgIDcgcHJpIGxldmVscyB4IDQgY29u
bmVjdGlvbnMgPT4gcmVjYWxjIHByaW9yaXR5DQoNCklmIHRoZSBjb25uZWN0aW9uIHR5cGUgZG9l
c24ndCBtYXRjaCwgaXQncyBub3QgZ29pbmcgdG8gbWF0Y2ggYXQgYW55DQpwcmlvcml0eSBsZXZl
bC4NCg0KPiArDQo+ICsgICAgICAgICAgICAgICAgICAgICAgIGlmIChjLT5zdGF0ZSAhPSBCVF9D
T05ORUNURUQgJiYgYy0+c3RhdGUgIT0gQlRfQ09ORklHKQ0KPiArICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIGNvbnRpbnVlOw0KPg0KPiAtICAgICAgICAgICAgICAgbnVtKys7DQo+ICsg
ICAgICAgICAgICAgICAgICAgICAgIG51bSsrOw0KPg0KPiAtICAgICAgICAgICAgICAgaWYgKGMt
PnNlbnQgPCBtaW4pIHsNCj4gLSAgICAgICAgICAgICAgICAgICAgICAgbWluICA9IGMtPnNlbnQ7
DQo+IC0gICAgICAgICAgICAgICAgICAgICAgIGNvbm4gPSBjOw0KPiArICAgICAgICAgICAgICAg
ICAgICAgICBpZiAoYy0+c2VudCA8IG1pbikgew0KPiArICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIG1pbiAgPSBjLT5zZW50Ow0KPiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
IGNvbm4gPSBjOw0KPiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICpxdWV1ZSA9ICZj
LT5kYXRhX3FbaV07DQo+ICsgICAgICAgICAgICAgICAgICAgICAgIH0NCg0KV2h5IHByZXNlcnZl
IHRoZSBmYWlybmVzcyBsb2dpYz8NCg0KPiAgICAgICAgICAgICAgICAgfQ0KPiAtICAgICAgIH0N
Cj4NCj4gLSAgICAgICBpZiAoY29ubikgew0KPiAtICAgICAgICAgICAgICAgaW50IGNudCwgcTsN
Cj4gKyAgICAgICAgICAgICAgIGlmICghY29ubikNCj4gKyAgICAgICAgICAgICAgICAgICAgICAg
Y29udGludWU7DQo+DQo+ICAgICAgICAgICAgICAgICBzd2l0Y2ggKGNvbm4tPnR5cGUpIHsNCj4g
ICAgICAgICAgICAgICAgIGNhc2UgQUNMX0xJTks6DQo+IEBAIC0yMDk4LDEwICsyMTE5LDEzIEBA
IHN0YXRpYyBpbmxpbmUgc3RydWN0IGhjaV9jb25uICpoY2lfbG93X3NlbnQoc3RydWN0IGhjaV9k
ZXYgKmhkZXYsIF9fdTggdHlwZSwgaW50DQo+DQo+ICAgICAgICAgICAgICAgICBxID0gY250IC8g
bnVtOw0KPiAgICAgICAgICAgICAgICAgKnF1b3RlID0gcSA/IHEgOiAxOw0KPiAtICAgICAgIH0g
ZWxzZQ0KPiAtICAgICAgICAgICAgICAgKnF1b3RlID0gMDsNCj4gKyAgICAgICAgICAgICAgIEJU
X0RCRygiY29ubiAlcCBpbmRleCAlZCBxdW90ZSAlZCIsIGNvbm4sIGksICpxdW90ZSk7DQo+ICsg
ICAgICAgICAgICAgICByZXR1cm4gY29ubjsNCj4gKyAgICAgICB9DQo+DQo+IC0gICAgICAgQlRf
REJHKCJjb25uICVwIHF1b3RlICVkIiwgY29ubiwgKnF1b3RlKTsNCj4gKyAgICAgICAqcXVvdGUg
PSAwOw0KPiArICAgICAgICpxdWV1ZSA9IE5VTEw7DQo+ICsgICAgICAgQlRfREJHKCJjb25uICVw
IHF1ZXVlICVwIHF1b3RlICVkIiwgY29ubiwgcXVldWUsICpxdW90ZSk7DQo+ICAgICAgICAgcmV0
dXJuIGNvbm47DQo+ICB9DQo+DQo+IEBAIC0yMTI4LDYgKzIxNTIsNyBAQCBzdGF0aWMgaW5saW5l
IHZvaWQgaGNpX3NjaGVkX2FjbChzdHJ1Y3QgaGNpX2RldiAqaGRldikNCj4gIHsNCj4gICAgICAg
ICBzdHJ1Y3QgaGNpX2Nvbm4gKmNvbm47DQo+ICAgICAgICAgc3RydWN0IHNrX2J1ZmYgKnNrYjsN
Cj4gKyAgICAgICBzdHJ1Y3Qgc2tfYnVmZl9oZWFkICpxdWV1ZSA9IE5VTEw7DQo+ICAgICAgICAg
aW50IHF1b3RlOw0KPg0KPiAgICAgICAgIEJUX0RCRygiJXMiLCBoZGV2LT5uYW1lKTsNCj4gQEAg
LTIxMzksOSArMjE2NCwxMSBAQCBzdGF0aWMgaW5saW5lIHZvaWQgaGNpX3NjaGVkX2FjbChzdHJ1
Y3QgaGNpX2RldiAqaGRldikNCj4gICAgICAgICAgICAgICAgICAgICAgICAgaGNpX2xpbmtfdHhf
dG8oaGRldiwgQUNMX0xJTkspOw0KPiAgICAgICAgIH0NCj4NCj4gLSAgICAgICB3aGlsZSAoaGRl
di0+YWNsX2NudCAmJiAoY29ubiA9IGhjaV9sb3dfc2VudChoZGV2LCBBQ0xfTElOSywgJnF1b3Rl
KSkpIHsNCj4gLSAgICAgICAgICAgICAgIHdoaWxlIChxdW90ZS0tICYmIChza2IgPSBza2JfZGVx
dWV1ZSgmY29ubi0+ZGF0YV9xKSkpIHsNCj4gLSAgICAgICAgICAgICAgICAgICAgICAgQlRfREJH
KCJza2IgJXAgbGVuICVkIiwgc2tiLCBza2ItPmxlbik7DQo+ICsgICAgICAgd2hpbGUgKGhkZXYt
PmFjbF9jbnQgJiYNCj4gKyAgICAgICAgICAgICAgICAgICAgICAgKGNvbm4gPSBoY2lfcHJpb19z
ZW50KGhkZXYsIEFDTF9MSU5LLCAmcXVldWUsICZxdW90ZSkpKSB7DQo+ICsgICAgICAgICAgICAg
ICB3aGlsZSAocXVvdGUtLSAmJiAoc2tiID0gc2tiX2RlcXVldWUocXVldWUpKSkgew0KPiArICAg
ICAgICAgICAgICAgICAgICAgICBCVF9EQkcoInNrYiAlcCBsZW4gJWQgcHJpb3JpdHkgJXUiLCBz
a2IsIHNrYi0+bGVuLA0KPiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgIHNrYi0+cHJpb3JpdHkpOw0KPg0KPiAgICAgICAgICAgICAgICAgICAg
ICAgICBoY2lfY29ubl9lbnRlcl9hY3RpdmVfbW9kZShjb25uLCBidF9jYihza2IpLT5mb3JjZV9h
Y3RpdmUpOw0KPg0KPiBAQCAtMjE1OSwxMyArMjE4NiwxNiBAQCBzdGF0aWMgaW5saW5lIHZvaWQg
aGNpX3NjaGVkX3NjbyhzdHJ1Y3QgaGNpX2RldiAqaGRldikNCj4gIHsNCj4gICAgICAgICBzdHJ1
Y3QgaGNpX2Nvbm4gKmNvbm47DQo+ICAgICAgICAgc3RydWN0IHNrX2J1ZmYgKnNrYjsNCj4gKyAg
ICAgICBzdHJ1Y3Qgc2tfYnVmZl9oZWFkICpxdWV1ZSA9IE5VTEw7DQo+ICAgICAgICAgaW50IHF1
b3RlOw0KPg0KPiAgICAgICAgIEJUX0RCRygiJXMiLCBoZGV2LT5uYW1lKTsNCj4NCj4gLSAgICAg
ICB3aGlsZSAoaGRldi0+c2NvX2NudCAmJiAoY29ubiA9IGhjaV9sb3dfc2VudChoZGV2LCBTQ09f
TElOSywgJnF1b3RlKSkpIHsNCj4gLSAgICAgICAgICAgICAgIHdoaWxlIChxdW90ZS0tICYmIChz
a2IgPSBza2JfZGVxdWV1ZSgmY29ubi0+ZGF0YV9xKSkpIHsNCj4gLSAgICAgICAgICAgICAgICAg
ICAgICAgQlRfREJHKCJza2IgJXAgbGVuICVkIiwgc2tiLCBza2ItPmxlbik7DQo+ICsgICAgICAg
d2hpbGUgKGhkZXYtPnNjb19jbnQgJiYNCj4gKyAgICAgICAgICAgICAgICAgICAgICAgKGNvbm4g
PSBoY2lfcHJpb19zZW50KGhkZXYsIFNDT19MSU5LLCAmcXVldWUsICZxdW90ZSkpKSB7DQo+ICsg
ICAgICAgICAgICAgICB3aGlsZSAocXVvdGUtLSAmJiAoc2tiID0gc2tiX2RlcXVldWUocXVldWUp
KSkgew0KPiArICAgICAgICAgICAgICAgICAgICAgICBCVF9EQkcoImNvbm4gJXAgc2tiICVwIGxl
biAlZCBwcmlvcml0eSAldSIsIGNvbm4sIHNrYiwNCj4gKyAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgc2tiLT5sZW4sIHNrYi0+cHJpb3JpdHkpOw0KPiAgICAg
ICAgICAgICAgICAgICAgICAgICBoY2lfc2VuZF9mcmFtZShza2IpOw0KPg0KPiAgICAgICAgICAg
ICAgICAgICAgICAgICBjb25uLT5zZW50Kys7DQo+IEBAIC0yMTc5LDEzICsyMjA5LDE2IEBAIHN0
YXRpYyBpbmxpbmUgdm9pZCBoY2lfc2NoZWRfZXNjbyhzdHJ1Y3QgaGNpX2RldiAqaGRldikNCj4g
IHsNCj4gICAgICAgICBzdHJ1Y3QgaGNpX2Nvbm4gKmNvbm47DQo+ICAgICAgICAgc3RydWN0IHNr
X2J1ZmYgKnNrYjsNCj4gKyAgICAgICBzdHJ1Y3Qgc2tfYnVmZl9oZWFkICpxdWV1ZSA9IE5VTEw7
DQo+ICAgICAgICAgaW50IHF1b3RlOw0KPg0KPiAgICAgICAgIEJUX0RCRygiJXMiLCBoZGV2LT5u
YW1lKTsNCj4NCj4gLSAgICAgICB3aGlsZSAoaGRldi0+c2NvX2NudCAmJiAoY29ubiA9IGhjaV9s
b3dfc2VudChoZGV2LCBFU0NPX0xJTkssICZxdW90ZSkpKSB7DQo+IC0gICAgICAgICAgICAgICB3
aGlsZSAocXVvdGUtLSAmJiAoc2tiID0gc2tiX2RlcXVldWUoJmNvbm4tPmRhdGFfcSkpKSB7DQo+
IC0gICAgICAgICAgICAgICAgICAgICAgIEJUX0RCRygic2tiICVwIGxlbiAlZCIsIHNrYiwgc2ti
LT5sZW4pOw0KPiArICAgICAgIHdoaWxlIChoZGV2LT5zY29fY250ICYmDQo+ICsgICAgICAgICAg
ICAgICAgICAgICAgIChjb25uID0gaGNpX3ByaW9fc2VudChoZGV2LCBFU0NPX0xJTkssICZxdWV1
ZSwgJnF1b3RlKSkpIHsNCj4gKyAgICAgICAgICAgICAgIHdoaWxlIChxdW90ZS0tICYmIChza2Ig
PSBza2JfZGVxdWV1ZShxdWV1ZSkpKSB7DQo+ICsgICAgICAgICAgICAgICAgICAgICAgIEJUX0RC
RygiY29ubiAlcCBza2IgJXAgbGVuICVkIHByaW9yaXR5ICV1IiwgY29ubiwgc2tiLA0KPiArICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBza2ItPmxlbiwgc2ti
LT5wcmlvcml0eSk7DQo+ICAgICAgICAgICAgICAgICAgICAgICAgIGhjaV9zZW5kX2ZyYW1lKHNr
Yik7DQo+DQo+ICAgICAgICAgICAgICAgICAgICAgICAgIGNvbm4tPnNlbnQrKzsNCj4gQEAgLTIx
OTksNiArMjIzMiw3IEBAIHN0YXRpYyBpbmxpbmUgdm9pZCBoY2lfc2NoZWRfbGUoc3RydWN0IGhj
aV9kZXYgKmhkZXYpDQo+ICB7DQo+ICAgICAgICAgc3RydWN0IGhjaV9jb25uICpjb25uOw0KPiAg
ICAgICAgIHN0cnVjdCBza19idWZmICpza2I7DQo+ICsgICAgICAgc3RydWN0IHNrX2J1ZmZfaGVh
ZCAqcXVldWUgPSBOVUxMOw0KPiAgICAgICAgIGludCBxdW90ZSwgY250Ow0KPg0KPiAgICAgICAg
IEJUX0RCRygiJXMiLCBoZGV2LT5uYW1lKTsNCj4gQEAgLTIyMTIsOSArMjI0NiwxMCBAQCBzdGF0
aWMgaW5saW5lIHZvaWQgaGNpX3NjaGVkX2xlKHN0cnVjdCBoY2lfZGV2ICpoZGV2KQ0KPiAgICAg
ICAgIH0NCj4NCj4gICAgICAgICBjbnQgPSBoZGV2LT5sZV9wa3RzID8gaGRldi0+bGVfY250IDog
aGRldi0+YWNsX2NudDsNCj4gLSAgICAgICB3aGlsZSAoY250ICYmIChjb25uID0gaGNpX2xvd19z
ZW50KGhkZXYsIExFX0xJTkssICZxdW90ZSkpKSB7DQo+IC0gICAgICAgICAgICAgICB3aGlsZSAo
cXVvdGUtLSAmJiAoc2tiID0gc2tiX2RlcXVldWUoJmNvbm4tPmRhdGFfcSkpKSB7DQo+IC0gICAg
ICAgICAgICAgICAgICAgICAgIEJUX0RCRygic2tiICVwIGxlbiAlZCIsIHNrYiwgc2tiLT5sZW4p
Ow0KPiArICAgICAgIHdoaWxlIChjbnQgJiYgKGNvbm4gPSBoY2lfcHJpb19zZW50KGhkZXYsIExF
X0xJTkssICZxdWV1ZSwgJnF1b3RlKSkpIHsNCj4gKyAgICAgICAgICAgICAgIHdoaWxlIChxdW90
ZS0tICYmIChza2IgPSBza2JfZGVxdWV1ZShxdWV1ZSkpKSB7DQo+ICsgICAgICAgICAgICAgICAg
ICAgICAgIEJUX0RCRygiY29ubiAlcCBza2IgJXAgbGVuICVkIHByaW9yaXR5ICV1IiwgY29ubiwg
c2tiLA0KPiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBz
a2ItPmxlbiwgc2tiLT5wcmlvcml0eSk7DQo+DQo+ICAgICAgICAgICAgICAgICAgICAgICAgIGhj
aV9zZW5kX2ZyYW1lKHNrYik7DQo+ICAgICAgICAgICAgICAgICAgICAgICAgIGhkZXYtPmxlX2xh
c3RfdHggPSBqaWZmaWVzOw0KPiBAQCAtMjIyOSw2ICsyMjY0LDM0IEBAIHN0YXRpYyBpbmxpbmUg
dm9pZCBoY2lfc2NoZWRfbGUoc3RydWN0IGhjaV9kZXYgKmhkZXYpDQo+ICAgICAgICAgICAgICAg
ICBoZGV2LT5hY2xfY250ID0gY250Ow0KPiAgfQ0KPg0KPiArc3RhdGljIHZvaWQgaGNpX3ByaW9f
cmVjYWxjdWxhdGUoc3RydWN0IGhjaV9kZXYgKmhkZXYpDQo+ICt7DQo+ICsgICAgICAgc3RydWN0
IGhjaV9jb25uX2hhc2ggKmggPSAmaGRldi0+Y29ubl9oYXNoOw0KPiArICAgICAgIGludCBpOw0K
PiArDQo+ICsgICAgICAgQlRfREJHKCIlcyIsIGhkZXYtPm5hbWUpOw0KPiArDQo+ICsgICAgICAg
Zm9yIChpID0gSENJX1BSSU9fTUFYIC0gMTsgaSA+IDA7IGktLSkgew0KPiArICAgICAgICAgICAg
ICAgc3RydWN0IGxpc3RfaGVhZCAqcDsNCj4gKw0KPiArICAgICAgICAgICAgICAgbGlzdF9mb3Jf
ZWFjaChwLCAmaC0+bGlzdCkgew0KPiArICAgICAgICAgICAgICAgICAgICAgICBzdHJ1Y3QgaGNp
X2Nvbm4gKmM7DQo+ICsgICAgICAgICAgICAgICAgICAgICAgIGMgPSBsaXN0X2VudHJ5KHAsIHN0
cnVjdCBoY2lfY29ubiwgbGlzdCk7DQo+ICsNCj4gKyAgICAgICAgICAgICAgICAgICAgICAgaWYg
KGMtPnNlbnQgfHwgc2tiX3F1ZXVlX2VtcHR5KCZjLT5kYXRhX3FbaSAtIDFdKSkNCj4gKyAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICBjb250aW51ZTsNCg0KVGhlIHRlc3QgZm9yIHVuYWNr
ZWQgcGFja2V0cyBkb2VzIG5vdCBhbGxvdyB0aGUgcHJpb3JpdHkgdG8gYWR2YW5jZSBpZg0KdGhl
IHVuYWNrZWQgcGFja2V0cyBhcmUgZnJvbSBhIHByZXZpb3VzbHkgZXhlY3V0ZWQgaGNpX3R4X3Rh
c2sgKGllLiwgYQ0KY29ubmVjdGlvbiB0cmFuc21pdHRlZCBwYWNrZXRzIG9uIGFuIGVhcmxpZXIg
c2NoZWR1bGVkIHR4X3Rhc2sgYnV0IHRob3NlDQpwYWNrZXRzIGhhdmVuJ3QgYmVlbiBhY2tlZCB5
ZXQpLg0KDQpUaGlzIHdvdWxkIGFsbG93IGEgaGlnaGVyIHByaW9yaXR5IGNvbm5lY3Rpb24gdG8g
YWxtb3N0IG1vbm9wb2xpemUNCnRyYW5zbWlzc2lvbiB3aGVuIHRoZSBjb250cm9sbGVyIGlzIG5l
YXIgbWF4IGxvYWQuIEEgaGlnaGVyIHByaW9yaXR5DQpsaW5rIHdpbGwgcmVjZWl2ZSBhIHR4IHF1
b3RhIG9mIGFsbCBhdmFpbGFibGUgdHggYnVmZmVycywgd2hlcmVhcyBhDQpsb3dlciBwcmlvcml0
eSBsaW5rIHdpbGwgbm90IGFkdmFuY2UgdW50aWwgaXQncyBwcmV2aW91cyB0eCBwYWNrZXRzIGFy
ZQ0KYWNrZWQuIFdvcnN0LWNhc2Ugc2NoZWR1bGluZyBjb3VsZCB0YWtlIGEgbG9uZyB0aW1lIHRv
IHNjaGVkdWxlIGEgbG93ZXINCnByaW9yaXR5IHBhY2tldC4NCg0KPiArDQo+ICsgICAgICAgICAg
ICAgICAgICAgICAgIGlmIChjLT5zdGF0ZSAhPSBCVF9DT05ORUNURUQgJiYgYy0+c3RhdGUgIT0g
QlRfQ09ORklHKQ0KPiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0K
PiArDQo+ICsgICAgICAgICAgICAgICAgICAgICAgIEJUX0RCRygiY29ubiAlcCBpbmRleCAlZCBw
cm9tb3RlZCB0byAlZCIsIGMsIGkgLSAxLCBpKTsNCj4gKw0KPiArICAgICAgICAgICAgICAgICAg
ICAgICBza2JfcXVldWVfc3BsaWNlX3RhaWxfaW5pdCgmYy0+ZGF0YV9xW2kgLSAxXSwNCj4gKyAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAmYy0+
ZGF0YV9xW2ldKTsNCj4gKyAgICAgICAgICAgICAgIH0NCj4gKyAgICAgICB9DQo+ICt9DQo+ICsN
Cj4gIHN0YXRpYyB2b2lkIGhjaV90eF90YXNrKHVuc2lnbmVkIGxvbmcgYXJnKQ0KPiAgew0KPiAg
ICAgICAgIHN0cnVjdCBoY2lfZGV2ICpoZGV2ID0gKHN0cnVjdCBoY2lfZGV2ICopIGFyZzsNCj4g
QEAgLTIyNTMsNiArMjMxNiw4IEBAIHN0YXRpYyB2b2lkIGhjaV90eF90YXNrKHVuc2lnbmVkIGxv
bmcgYXJnKQ0KPiAgICAgICAgIHdoaWxlICgoc2tiID0gc2tiX2RlcXVldWUoJmhkZXYtPnJhd19x
KSkpDQo+ICAgICAgICAgICAgICAgICBoY2lfc2VuZF9mcmFtZShza2IpOw0KPg0KPiArICAgICAg
IGhjaV9wcmlvX3JlY2FsY3VsYXRlKGhkZXYpOw0KDQpUaGUgcHJpb3JpdHkgcmVjYWxjdWxhdGlv
biBzaG91bGQgb25seSBiZSBwZXJmb3JtZWQgaWYgb25lIG9mIHRoZQ0KaGNpX3NjaGVkXyogZnVu
Y3Rpb25zIHdhcyB1bmFibGUgdG8gc2NoZWR1bGUgYWxsIG91dHN0YW5kaW5nIHBhY2tldHMNCihp
ZS4sIGEgdHggYnVmZmVyIGNvdW50IGRyb3BwZWQgdG8gemVybykuDQoNCkhvdyBkb2VzIHRoaXMg
c2NoZWR1bGVyICsgcHJpb3JpdHkgcmVjYWxjIGJlaGF2ZSBpZiAqbm8qIHBhY2tldHMgY291bGQN
CmJlIHNjaGVkdWxlZCBhdCBhbGw/IEZvciBleGFtcGxlLCBpZiBhY2xfY250ID09IDAgd2hlbiB0
aGUgdHhfdGFzayBpcw0Kc2NoZWR1bGVkPw0KDQo+ICsNCj4gICAgICAgICByZWFkX3VubG9jaygm
aGNpX3Rhc2tfbG9jayk7DQo+ICB9DQo+DQo+IGRpZmYgLS1naXQgYS9uZXQvYmx1ZXRvb3RoL3Jm
Y29tbS9zb2NrLmMgYi9uZXQvYmx1ZXRvb3RoL3JmY29tbS9zb2NrLmMNCj4gaW5kZXggOGYwMWU2
Yi4uMDIyOGUxZiAxMDA2NDQNCj4gLS0tIGEvbmV0L2JsdWV0b290aC9yZmNvbW0vc29jay5jDQo+
ICsrKyBiL25ldC9ibHVldG9vdGgvcmZjb21tL3NvY2suYw0KPiBAQCAtNTYxLDYgKzU2MSw3IEBA
IHN0YXRpYyBpbnQgcmZjb21tX3NvY2tfc2VuZG1zZyhzdHJ1Y3Qga2lvY2IgKmlvY2IsIHN0cnVj
dCBzb2NrZXQgKnNvY2ssDQo+ICAgICAgICAgc3RydWN0IHNvY2sgKnNrID0gc29jay0+c2s7DQo+
ICAgICAgICAgc3RydWN0IHJmY29tbV9kbGMgKmQgPSByZmNvbW1fcGkoc2spLT5kbGM7DQo+ICAg
ICAgICAgc3RydWN0IHNrX2J1ZmYgKnNrYjsNCj4gKyAgICAgICB1MzIgcHJpb3JpdHk7DQo+ICAg
ICAgICAgaW50IHNlbnQgPSAwOw0KPg0KPiAgICAgICAgIGlmICh0ZXN0X2JpdChSRkNPTU1fREVG
RVJfU0VUVVAsICZkLT5mbGFncykpDQo+IEBAIC01NzIsNiArNTczLDExIEBAIHN0YXRpYyBpbnQg
cmZjb21tX3NvY2tfc2VuZG1zZyhzdHJ1Y3Qga2lvY2IgKmlvY2IsIHN0cnVjdCBzb2NrZXQgKnNv
Y2ssDQo+ICAgICAgICAgaWYgKHNrLT5za19zaHV0ZG93biAmIFNFTkRfU0hVVERPV04pDQo+ICAg
ICAgICAgICAgICAgICByZXR1cm4gLUVQSVBFOw0KPg0KPiArICAgICAgIGlmIChtc2ctPm1zZ19m
bGFncyAmIE1TR19UUllIQVJEKQ0KPiArICAgICAgICAgICAgICAgcHJpb3JpdHkgPSBIQ0lfUFJJ
T19NQVg7DQo+ICsgICAgICAgZWxzZQ0KPiArICAgICAgICAgICAgICAgcHJpb3JpdHkgPSBzay0+
c2tfcHJpb3JpdHk7DQo+ICsNCj4gICAgICAgICBCVF9EQkcoInNvY2sgJXAsIHNrICVwIiwgc29j
aywgc2spOw0KPg0KPiAgICAgICAgIGxvY2tfc29jayhzayk7DQo+IEBAIC01OTcsNiArNjAzLDgg
QEAgc3RhdGljIGludCByZmNvbW1fc29ja19zZW5kbXNnKHN0cnVjdCBraW9jYiAqaW9jYiwgc3Ry
dWN0IHNvY2tldCAqc29jaywNCj4gICAgICAgICAgICAgICAgICAgICAgICAgYnJlYWs7DQo+ICAg
ICAgICAgICAgICAgICB9DQo+DQo+ICsgICAgICAgICAgICAgICBza2ItPnByaW9yaXR5ID0gcHJp
b3JpdHk7DQo+ICsNCj4gICAgICAgICAgICAgICAgIGVyciA9IHJmY29tbV9kbGNfc2VuZChkLCBz
a2IpOw0KPiAgICAgICAgICAgICAgICAgaWYgKGVyciA8IDApIHsNCj4gICAgICAgICAgICAgICAg
ICAgICAgICAga2ZyZWVfc2tiKHNrYik7DQo+IC0tDQo+IDEuNy42DQo+DQo+IC0tDQo+IFRvIHVu
c3Vic2NyaWJlIGZyb20gdGhpcyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZSBsaW51
eC1ibHVldG9vdGgiIGluDQo+IHRoZSBib2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdl
ci5rZXJuZWwub3JnDQo+IE1vcmUgbWFqb3Jkb21vIGluZm8gYXQgIGh0dHA6Ly92Z2VyLmtlcm5l
bC5vcmcvbWFqb3Jkb21vLWluZm8uaHRtbA0KDQo=

Re: [RFC 1/3] Bluetooth: prioritizing data over HCI

[Luiz Augusto von Dentz]

Hi Peter,

On Wed, Aug 3, 2011 at 7:25 PM, Peter Hurley <peter@hurleysoftware.com> wrote:
>
> I believe that the existing tx scheduler is already inefficient and that
> this approach significantly compounds that inefficiency.
>
> Consider the following scenarios with 4 ACL connections (keyboard,
> mouse, phone, headset):
>
> For *one* scheduled ACL tx packet, the connection list will be searched
> 35~42 times -- that's 140~168 connection list node visits to find one
> packet! Here's the math:
>        (7 - priority of tx packet) x 4 connections => found the packet
>        7 pri levels x 4 connections => no more ACL tx packets
>        7 pri levels x 4 connections => no SCO tx packets
>        7 pri levels x 4 connections => no ESCO tx packets
>        7 pri levels x 4 connections => no LE tx packets
>        7 pri levels x 4 connections => recalc priority
>
> If the connection type doesn't match, it's not going to match at any
> priority level.

Obviously there are a few things that we might change to not traverse
the list of connection over and over, remember this is an RFC, besides
it 8 priorities (0-7), but we can skip much earlier if connection
doesn't match as you said. Also I guess for SCO/ESCO/LE e doesn't make
much sense to have many queues/priorities, it is basically ACL only,
that simplify a lot already.

>> +
>> +                       if (c->state != BT_CONNECTED && c->state != BT_CONFIG)
>> +                               continue;
>>
>> -               num++;
>> +                       num++;
>>
>> -               if (c->sent < min) {
>> -                       min  = c->sent;
>> -                       conn = c;
>> +                       if (c->sent < min) {
>> +                               min  = c->sent;
>> +                               conn = c;
>> +                               *queue = &c->data_q[i];
>> +                       }
>
> Why preserve the fairness logic?

It does need to fair if there is 2 or more sockets with the same
priority, otherwise the first connection in the list might get all the
quote and even if we promote the starving queues it may still happen
to top most priority since there it cannot be promoted anymore.

>> +static void hci_prio_recalculate(struct hci_dev *hdev)
>> +{
>> +       struct hci_conn_hash *h = &hdev->conn_hash;
>> +       int i;
>> +
>> +       BT_DBG("%s", hdev->name);
>> +
>> +       for (i = HCI_PRIO_MAX - 1; i > 0; i--) {
>> +               struct list_head *p;
>> +
>> +               list_for_each(p, &h->list) {
>> +                       struct hci_conn *c;
>> +                       c = list_entry(p, struct hci_conn, list);
>> +
>> +                       if (c->sent || skb_queue_empty(&c->data_q[i - 1]))
>> +                               continue;
>
> The test for unacked packets does not allow the priority to advance if
> the unacked packets are from a previously executed hci_tx_task (ie., a
> connection transmitted packets on an earlier scheduled tx_task but those
> packets haven't been acked yet).

IMO it does, why should it be promoted to a higher priority if it
still is pending from the last time, this is specifically to throttle
lower priority in favor of higher, if throughput is important the
application should request be able to sets its priority according, and
it can do it at any time.

> This would allow a higher priority connection to almost monopolize
> transmission when the controller is near max load. A higher priority
> link will receive a tx quota of all available tx buffers, whereas a
> lower priority link will not advance until it's previous tx packets are
> acked. Worst-case scheduling could take a long time to schedule a lower
> priority packet.

You probably have never run this code did you? Only priority 7 can
really monopolize the connection, and that is on purpose, and yes
lower priority are throttled so higher priority can get lower latency,
what is wrong with that?

>>  static void hci_tx_task(unsigned long arg)
>>  {
>>         struct hci_dev *hdev = (struct hci_dev *) arg;
>> @@ -2253,6 +2316,8 @@ static void hci_tx_task(unsigned long arg)
>>         while ((skb = skb_dequeue(&hdev->raw_q)))
>>                 hci_send_frame(skb);
>>
>> +       hci_prio_recalculate(hdev);
>
> The priority recalculation should only be performed if one of the
> hci_sched_* functions was unable to schedule all outstanding packets
> (ie., a tx buffer count dropped to zero).

Yep, gonna fix that too.

> How does this scheduler + priority recalc behave if *no* packets could
> be scheduled at all? For example, if acl_cnt == 0 when the tx_task is
> scheduled?

Right now it promotes starving queues, all of them, but maybe it
shouldn't so that we simplify the complexity a little bit.


-- 
Luiz Augusto von Dentz

Re: [RFC 1/3] Bluetooth: prioritizing data over HCI

[Gustavo Padovan]

Hi Luiz,

* Luiz Augusto von Dentz <luiz.dentz@gmail.com> [2011-08-03 20:49:19 +0300]:

> Hi Peter,
> 
> On Wed, Aug 3, 2011 at 7:25 PM, Peter Hurley <peter@hurleysoftware.com> wrote:
> >
> > I believe that the existing tx scheduler is already inefficient and that
> > this approach significantly compounds that inefficiency.
> >
> > Consider the following scenarios with 4 ACL connections (keyboard,
> > mouse, phone, headset):
> >
> > For *one* scheduled ACL tx packet, the connection list will be searched
> > 35~42 times -- that's 140~168 connection list node visits to find one
> > packet! Here's the math:
> >        (7 - priority of tx packet) x 4 connections => found the packet
> >        7 pri levels x 4 connections => no more ACL tx packets
> >        7 pri levels x 4 connections => no SCO tx packets
> >        7 pri levels x 4 connections => no ESCO tx packets
> >        7 pri levels x 4 connections => no LE tx packets
> >        7 pri levels x 4 connections => recalc priority
> >
> > If the connection type doesn't match, it's not going to match at any
> > priority level.
> 
> Obviously there are a few things that we might change to not traverse
> the list of connection over and over, remember this is an RFC, besides
> it 8 priorities (0-7), but we can skip much earlier if connection
> doesn't match as you said. Also I guess for SCO/ESCO/LE e doesn't make
> much sense to have many queues/priorities, it is basically ACL only,
> that simplify a lot already.

Maybe 8 is too much, I think it could work with 4. We keep the high
priority queue 7 and more 3. A2DP and SCO could have the 6, HID the queue 5,
Obex transfers, PAN, AVRCP, etc goes queue 4. Queueing in 3, 2, 1 and 0 could
actually queue on 4.

	Gustavo

Re: [RFC 1/3] Bluetooth: prioritizing data over HCI

[Peter Hurley]

SGkgTHVpeiwNCg0KT24gV2VkLCAyMDExLTA4LTAzIGF0IDEzOjQ5IC0wNDAwLCBMdWl6IEF1Z3Vz
dG8gdm9uIERlbnR6IHdyb3RlOg0KPiBIaSBQZXRlciwNCj4gDQo+IE9uIFdlZCwgQXVnIDMsIDIw
MTEgYXQgNzoyNSBQTSwgUGV0ZXIgSHVybGV5IDxwZXRlckBodXJsZXlzb2Z0d2FyZS5jb20+IHdy
b3RlOg0KPiA+DQo+ID4gSSBiZWxpZXZlIHRoYXQgdGhlIGV4aXN0aW5nIHR4IHNjaGVkdWxlciBp
cyBhbHJlYWR5IGluZWZmaWNpZW50IGFuZCB0aGF0DQo+ID4gdGhpcyBhcHByb2FjaCBzaWduaWZp
Y2FudGx5IGNvbXBvdW5kcyB0aGF0IGluZWZmaWNpZW5jeS4NCj4gPg0KPiA+IENvbnNpZGVyIHRo
ZSBmb2xsb3dpbmcgc2NlbmFyaW9zIHdpdGggNCBBQ0wgY29ubmVjdGlvbnMgKGtleWJvYXJkLA0K
PiA+IG1vdXNlLCBwaG9uZSwgaGVhZHNldCk6DQo+ID4NCj4gPiBGb3IgKm9uZSogc2NoZWR1bGVk
IEFDTCB0eCBwYWNrZXQsIHRoZSBjb25uZWN0aW9uIGxpc3Qgd2lsbCBiZSBzZWFyY2hlZA0KPiA+
IDM1fjQyIHRpbWVzIC0tIHRoYXQncyAxNDB+MTY4IGNvbm5lY3Rpb24gbGlzdCBub2RlIHZpc2l0
cyB0byBmaW5kIG9uZQ0KPiA+IHBhY2tldCEgSGVyZSdzIHRoZSBtYXRoOg0KPiA+ICAgICAgICAo
NyAtIHByaW9yaXR5IG9mIHR4IHBhY2tldCkgeCA0IGNvbm5lY3Rpb25zID0+IGZvdW5kIHRoZSBw
YWNrZXQNCj4gPiAgICAgICAgNyBwcmkgbGV2ZWxzIHggNCBjb25uZWN0aW9ucyA9PiBubyBtb3Jl
IEFDTCB0eCBwYWNrZXRzDQo+ID4gICAgICAgIDcgcHJpIGxldmVscyB4IDQgY29ubmVjdGlvbnMg
PT4gbm8gU0NPIHR4IHBhY2tldHMNCj4gPiAgICAgICAgNyBwcmkgbGV2ZWxzIHggNCBjb25uZWN0
aW9ucyA9PiBubyBFU0NPIHR4IHBhY2tldHMNCj4gPiAgICAgICAgNyBwcmkgbGV2ZWxzIHggNCBj
b25uZWN0aW9ucyA9PiBubyBMRSB0eCBwYWNrZXRzDQo+ID4gICAgICAgIDcgcHJpIGxldmVscyB4
IDQgY29ubmVjdGlvbnMgPT4gcmVjYWxjIHByaW9yaXR5DQo+ID4NCj4gPiBJZiB0aGUgY29ubmVj
dGlvbiB0eXBlIGRvZXNuJ3QgbWF0Y2gsIGl0J3Mgbm90IGdvaW5nIHRvIG1hdGNoIGF0IGFueQ0K
PiA+IHByaW9yaXR5IGxldmVsLg0KPiANCj4gT2J2aW91c2x5IHRoZXJlIGFyZSBhIGZldyB0aGlu
Z3MgdGhhdCB3ZSBtaWdodCBjaGFuZ2UgdG8gbm90IHRyYXZlcnNlDQo+IHRoZSBsaXN0IG9mIGNv
bm5lY3Rpb24gb3ZlciBhbmQgb3ZlciwgcmVtZW1iZXIgdGhpcyBpcyBhbiBSRkMsIGJlc2lkZXMN
Cj4gaXQgOCBwcmlvcml0aWVzICgwLTcpLCBidXQgd2UgY2FuIHNraXAgbXVjaCBlYXJsaWVyIGlm
IGNvbm5lY3Rpb24NCj4gZG9lc24ndCBtYXRjaCBhcyB5b3Ugc2FpZC4NCg0KSSdtIHNvcnJ5IGlm
IG15IGNvbW1lbnRzIGFwcGVhcmVkIGhhcnNoIGFuZCBJIGFwb2xvZ2l6ZSBpZiBJIG9mZmVuZGVk
DQp5b3UuIEkgZ2V0IGl0J3MgYW4gUkZDIDopDQpQZXJzb25hbGx5LCBJIGxpa2UgdGhlIGlkZWEg
b2YgcHJpb3JpdGl6aW5nIHRoZSB0eCBzY2hlZHVsZXIsIGFuZCBJDQphcHBsYXVkIHRoZSBlZmZv
cnQuDQoNCk15IHBvaW50IGhlcmUgKGJlc2lkZXMgcG9pbnRpbmcgb3V0IHRoZSBtYXNzaXZlIGl0
ZXJhdGlvbikgd2FzIHRvDQpzdWdnZXN0IHRoYXQgcGVyaGFwcyBzb21lIHNjaGVkdWxlciBzdGF0
ZSBzaG91bGQgYmUgcHJlc2VydmVkIHN1Y2ggdGhhdA0KYWxyZWFkeS12aXNpdGVkIHByaW9yaXR5
IGxldmVscyBhcmUgbm90IHJldmlzaXRlZCBtb3JlIHRoYW4gb25jZSBwZXIgdHgNCnRhc2tsZXQg
c2NoZWR1bGUuDQoNCj4gIEFsc28gSSBndWVzcyBmb3IgU0NPL0VTQ08vTEUgZSBkb2Vzbid0IG1h
a2UNCj4gbXVjaCBzZW5zZSB0byBoYXZlIG1hbnkgcXVldWVzL3ByaW9yaXRpZXMsIGl0IGlzIGJh
c2ljYWxseSBBQ0wgb25seSwNCj4gdGhhdCBzaW1wbGlmeSBhIGxvdCBhbHJlYWR5Lg0KDQpMRSBz
aG91bGQgc3RpbGwgaGF2ZSBwcmlvcml0aWVzLiBJbiBmYWN0LCBpbiBhIHByaXZhdGUgdGVzdCB3
ZSd2ZSBiZWVuDQpydW5uaW5nIGhlcmUsIHdlJ3ZlIG1lcmdlZCB0aGUgc2NoZWR1bGVyIHNvIHRo
YXQgTEUgY29ubnMgdGhhdCBzaGFyZSBBQ0wNCmJ1ZmZlcnMgYXJlIHNjaGVkdWxlZCBhbG9uZ3Np
ZGUgb3RoZXIgQUNMIGNvbm5zIChiZWNhdXNlIHRoZXkgc2hhcmUgdGhlDQpzYW1lIHJlc291cmNl
IC0tIG5hbWVseSwgdGhlIGFjbF9jbnQuIFdlIGFsc28gbWVyZ2VkIFNDTy9FU0NPIHNjaGVkdWxp
bmcNCnRvZ2V0aGVyIGFzIHdlbGwpLg0KDQo+IA0KPiA+PiArDQo+ID4+ICsgICAgICAgICAgICAg
ICAgICAgICAgIGlmIChjLT5zdGF0ZSAhPSBCVF9DT05ORUNURUQgJiYgYy0+c3RhdGUgIT0gQlRf
Q09ORklHKQ0KPiA+PiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0K
PiA+Pg0KPiA+PiAtICAgICAgICAgICAgICAgbnVtKys7DQo+ID4+ICsgICAgICAgICAgICAgICAg
ICAgICAgIG51bSsrOw0KPiA+Pg0KPiA+PiAtICAgICAgICAgICAgICAgaWYgKGMtPnNlbnQgPCBt
aW4pIHsNCj4gPj4gLSAgICAgICAgICAgICAgICAgICAgICAgbWluICA9IGMtPnNlbnQ7DQo+ID4+
IC0gICAgICAgICAgICAgICAgICAgICAgIGNvbm4gPSBjOw0KPiA+PiArICAgICAgICAgICAgICAg
ICAgICAgICBpZiAoYy0+c2VudCA8IG1pbikgew0KPiA+PiArICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIG1pbiAgPSBjLT5zZW50Ow0KPiA+PiArICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgIGNvbm4gPSBjOw0KPiA+PiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICpx
dWV1ZSA9ICZjLT5kYXRhX3FbaV07DQo+ID4+ICsgICAgICAgICAgICAgICAgICAgICAgIH0NCj4g
Pg0KPiA+IFdoeSBwcmVzZXJ2ZSB0aGUgZmFpcm5lc3MgbG9naWM/DQo+IA0KPiBJdCBkb2VzIG5l
ZWQgdG8gZmFpciBpZiB0aGVyZSBpcyAyIG9yIG1vcmUgc29ja2V0cyB3aXRoIHRoZSBzYW1lDQo+
IHByaW9yaXR5LCBvdGhlcndpc2UgdGhlIGZpcnN0IGNvbm5lY3Rpb24gaW4gdGhlIGxpc3QgbWln
aHQgZ2V0IGFsbCB0aGUNCj4gcXVvdGUgYW5kIGV2ZW4gaWYgd2UgcHJvbW90ZSB0aGUgc3RhcnZp
bmcgcXVldWVzIGl0IG1heSBzdGlsbCBoYXBwZW4NCj4gdG8gdG9wIG1vc3QgcHJpb3JpdHkgc2lu
Y2UgdGhlcmUgaXQgY2Fubm90IGJlIHByb21vdGVkIGFueW1vcmUuDQoNClJpZ2h0LiBXaGF0IEkg
bWVhbnQgaGVyZSB3YXMgdGhpczogaWYgdGhlIHR4IHNjaGVkdWxlciBpcyByZS1tYWRlIGFzDQpw
cmlvcml0eS1iYXNlZCwgdGhlbiBpdCBzZWVtcyBsb2dpY2FsIHRvIGRpc2NhcmQgdGhlIGZhaXJu
ZXNzIGxvZ2ljIGFuZA0KZGVzaWduIHRoZSBwcmlvcml0eSBzY2hlbWUgc3VjaCB0aGF0IHN0YXJ2
YXRpb24gaXMgbm90IHBvc3NpYmxlLiBGb3INCmV4YW1wbGUsIHRoZSBwcmkgbGV2ZWxzIGNvdWxk
IGJlIGxpa2UgdGhpczoNCgkwIH4gNQlzb2NrZXQtcHJvZ3JhbW1hYmxlIHByaW9yaXRpZXMNCgk2
CXNwZWNpYWwgQ0FQX05FVF9BRE1JTiBwcmlvcml0eQ0KCTcJb25seSBwb3NzaWJsZSB2aWEgcHJv
bW90aW9uDQooTm90IHRoYXQgSSdtIHN1Z2dlc3RpbmcgdGhhdCdzIHRoZSBvbmx5IG9yIGV2ZW4g
YmVzdCB3YXkgdG8gbWFrZQ0Kc3RhcnZhdGlvbiBub3QgcG9zc2libGUpLg0KDQo+ID4+ICtzdGF0
aWMgdm9pZCBoY2lfcHJpb19yZWNhbGN1bGF0ZShzdHJ1Y3QgaGNpX2RldiAqaGRldikNCj4gPj4g
K3sNCj4gPj4gKyAgICAgICBzdHJ1Y3QgaGNpX2Nvbm5faGFzaCAqaCA9ICZoZGV2LT5jb25uX2hh
c2g7DQo+ID4+ICsgICAgICAgaW50IGk7DQo+ID4+ICsNCj4gPj4gKyAgICAgICBCVF9EQkcoIiVz
IiwgaGRldi0+bmFtZSk7DQo+ID4+ICsNCj4gPj4gKyAgICAgICBmb3IgKGkgPSBIQ0lfUFJJT19N
QVggLSAxOyBpID4gMDsgaS0tKSB7DQo+ID4+ICsgICAgICAgICAgICAgICBzdHJ1Y3QgbGlzdF9o
ZWFkICpwOw0KPiA+PiArDQo+ID4+ICsgICAgICAgICAgICAgICBsaXN0X2Zvcl9lYWNoKHAsICZo
LT5saXN0KSB7DQo+ID4+ICsgICAgICAgICAgICAgICAgICAgICAgIHN0cnVjdCBoY2lfY29ubiAq
YzsNCj4gPj4gKyAgICAgICAgICAgICAgICAgICAgICAgYyA9IGxpc3RfZW50cnkocCwgc3RydWN0
IGhjaV9jb25uLCBsaXN0KTsNCj4gPj4gKw0KPiA+PiArICAgICAgICAgICAgICAgICAgICAgICBp
ZiAoYy0+c2VudCB8fCBza2JfcXVldWVfZW1wdHkoJmMtPmRhdGFfcVtpIC0gMV0pKQ0KPiA+PiAr
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlOw0KPiA+DQo+ID4gVGhlIHRl
c3QgZm9yIHVuYWNrZWQgcGFja2V0cyBkb2VzIG5vdCBhbGxvdyB0aGUgcHJpb3JpdHkgdG8gYWR2
YW5jZSBpZg0KPiA+IHRoZSB1bmFja2VkIHBhY2tldHMgYXJlIGZyb20gYSBwcmV2aW91c2x5IGV4
ZWN1dGVkIGhjaV90eF90YXNrIChpZS4sIGENCj4gPiBjb25uZWN0aW9uIHRyYW5zbWl0dGVkIHBh
Y2tldHMgb24gYW4gZWFybGllciBzY2hlZHVsZWQgdHhfdGFzayBidXQgdGhvc2UNCj4gPiBwYWNr
ZXRzIGhhdmVuJ3QgYmVlbiBhY2tlZCB5ZXQpLg0KPiANCj4gSU1PIGl0IGRvZXMsIHdoeSBzaG91
bGQgaXQgYmUgcHJvbW90ZWQgdG8gYSBoaWdoZXIgcHJpb3JpdHkgaWYgaXQNCj4gc3RpbGwgaXMg
cGVuZGluZyBmcm9tIHRoZSBsYXN0IHRpbWUsIHRoaXMgaXMgc3BlY2lmaWNhbGx5IHRvIHRocm90
dGxlDQo+IGxvd2VyIHByaW9yaXR5IGluIGZhdm9yIG9mIGhpZ2hlciwgaWYgdGhyb3VnaHB1dCBp
cyBpbXBvcnRhbnQgdGhlDQo+IGFwcGxpY2F0aW9uIHNob3VsZCByZXF1ZXN0IGJlIGFibGUgdG8g
c2V0cyBpdHMgcHJpb3JpdHkgYWNjb3JkaW5nLCBhbmQNCj4gaXQgY2FuIGRvIGl0IGF0IGFueSB0
aW1lLg0KPiANCj4gPiBUaGlzIHdvdWxkIGFsbG93IGEgaGlnaGVyIHByaW9yaXR5IGNvbm5lY3Rp
b24gdG8gYWxtb3N0IG1vbm9wb2xpemUNCj4gPiB0cmFuc21pc3Npb24gd2hlbiB0aGUgY29udHJv
bGxlciBpcyBuZWFyIG1heCBsb2FkLiBBIGhpZ2hlciBwcmlvcml0eQ0KPiA+IGxpbmsgd2lsbCBy
ZWNlaXZlIGEgdHggcXVvdGEgb2YgYWxsIGF2YWlsYWJsZSB0eCBidWZmZXJzLCB3aGVyZWFzIGEN
Cj4gPiBsb3dlciBwcmlvcml0eSBsaW5rIHdpbGwgbm90IGFkdmFuY2UgdW50aWwgaXQncyBwcmV2
aW91cyB0eCBwYWNrZXRzIGFyZQ0KPiA+IGFja2VkLiBXb3JzdC1jYXNlIHNjaGVkdWxpbmcgY291
bGQgdGFrZSBhIGxvbmcgdGltZSB0byBzY2hlZHVsZSBhIGxvd2VyDQo+ID4gcHJpb3JpdHkgcGFj
a2V0Lg0KPiANCj4gWW91IHByb2JhYmx5IGhhdmUgbmV2ZXIgcnVuIHRoaXMgY29kZSBkaWQgeW91
PyBPbmx5IHByaW9yaXR5IDcgY2FuDQo+IHJlYWxseSBtb25vcG9saXplIHRoZSBjb25uZWN0aW9u
LCBhbmQgdGhhdCBpcyBvbiBwdXJwb3NlLCBhbmQgeWVzDQo+IGxvd2VyIHByaW9yaXR5IGFyZSB0
aHJvdHRsZWQgc28gaGlnaGVyIHByaW9yaXR5IGNhbiBnZXQgbG93ZXIgbGF0ZW5jeSwNCj4gd2hh
dCBpcyB3cm9uZyB3aXRoIHRoYXQ/DQoNCldpdGggbm8gb2ZmZW5zZSBtZWFudCwgc2ltcGx5IHJ1
bm5pbmcgdGhpcyBjb2RlIHdvdWxkIGJlIGluc3VmZmljaWVudCB0bw0KcXVhbGlmeSBpdCdzIGFw
cHJvcHJpYXRlbmVzcyBhcyBhIHJlcGxhY2VtZW50IHNjaGVkdWxlci4gQSBiYXJlLW1pbmltdW0N
CnRlc3QgbWF0cml4IHdvdWxkIGFzY2VydGFpbiBjb21wYXJhdGl2ZSB2YWx1ZXMgZm9yIG1pbmlt
dW0vbWVhbi9tYXhpbXVtDQpsYXRlbmN5IGFuZCB0aHJvdWdocHV0IGZvciBiZXN0LWNhc2Uvbm9t
aW5hbC93b3JzdC1jYXNlIHR4IGxvYWRzLg0KDQpNeSBtYWluIGNvbmNlcm4gaGVyZSBpcyB0aGF0
IGFwcC1sZXZlbCBzb2NrZXQgc2V0dGluZ3MgY2FuIGhhdmUgZHJhbWF0aWMNCmVmZmVjdHMgb24g
dHggc2NoZWR1bGluZyBiZWhhdmlvciwgZXNwZWNpYWxseSBhdCBuZWFyIG1heCB0eCBsb2Fkcy4N
Cg0KV2hhdCBJIG1lYW50IGFib3V0IG5lYXIgbW9ub3BvbGl6YXRpb24gaXMgdGhpczogIGEgcHJp
b3JpdHkgNiBsaW5rIHRoYXQNCnJldGlyZXMgMiBwYWNrZXRzIGV2ZXJ5IDU1IG1zIHdvdWxkIG1l
YW4gdGhhdCBwcmlvcml0eSAwIGxpbmtzIGNvdWxkDQpvbmx5IHhtaXQgZXZlcnkgMzMwIG1zICgx
LzMgc2VjLiksIGFzc3VtaW5nIHRoZSBjb250cm9sbGVyIHdhcyBhbHJlYWR5DQpzYXR1cmF0ZWQg
YnkgdGhlIHByaW9yaXR5IDYgbGluay4gRXZlbiBwcmlvcml0eSAzIGxpbmtzIGFyZSBvbmx5IGdv
aW5nDQp0byBiZSBhYmxlIHRvIHhtaXQgZXZlcnkgMTY1bXMgdW5kZXIgdGhlc2UgY29uZGl0aW9u
cy4NCg0KV2hhdCBpZiwgaW5zdGVhZCwgbGlua3MgdGhhdCBhcmUgYmVpbmcgcHJvbW90ZWQgYmVj
YXVzZSB0aGUgY29udHJvbGxlcg0KaXMgc2F0dXJhdGVkLCBhcmUgcHJvbW90ZWQgdG8gdGhlIHNh
bWUgKG9yIG1heWJlLCBoaWdoZXIpIHByaW9yaXR5IGxldmVsDQp0aGFuIHdhcyBsYXN0IHN1Y2Nl
c3NmdWw/DQoNCj4gPj4gIHN0YXRpYyB2b2lkIGhjaV90eF90YXNrKHVuc2lnbmVkIGxvbmcgYXJn
KQ0KPiA+PiAgew0KPiA+PiAgICAgICAgIHN0cnVjdCBoY2lfZGV2ICpoZGV2ID0gKHN0cnVjdCBo
Y2lfZGV2ICopIGFyZzsNCj4gPj4gQEAgLTIyNTMsNiArMjMxNiw4IEBAIHN0YXRpYyB2b2lkIGhj
aV90eF90YXNrKHVuc2lnbmVkIGxvbmcgYXJnKQ0KPiA+PiAgICAgICAgIHdoaWxlICgoc2tiID0g
c2tiX2RlcXVldWUoJmhkZXYtPnJhd19xKSkpDQo+ID4+ICAgICAgICAgICAgICAgICBoY2lfc2Vu
ZF9mcmFtZShza2IpOw0KPiA+Pg0KPiA+PiArICAgICAgIGhjaV9wcmlvX3JlY2FsY3VsYXRlKGhk
ZXYpOw0KPiA+DQo+ID4gVGhlIHByaW9yaXR5IHJlY2FsY3VsYXRpb24gc2hvdWxkIG9ubHkgYmUg
cGVyZm9ybWVkIGlmIG9uZSBvZiB0aGUNCj4gPiBoY2lfc2NoZWRfKiBmdW5jdGlvbnMgd2FzIHVu
YWJsZSB0byBzY2hlZHVsZSBhbGwgb3V0c3RhbmRpbmcgcGFja2V0cw0KPiA+IChpZS4sIGEgdHgg
YnVmZmVyIGNvdW50IGRyb3BwZWQgdG8gemVybykuDQo+IA0KPiBZZXAsIGdvbm5hIGZpeCB0aGF0
IHRvby4NCj4gDQo+ID4gSG93IGRvZXMgdGhpcyBzY2hlZHVsZXIgKyBwcmlvcml0eSByZWNhbGMg
YmVoYXZlIGlmICpubyogcGFja2V0cyBjb3VsZA0KPiA+IGJlIHNjaGVkdWxlZCBhdCBhbGw/IEZv
ciBleGFtcGxlLCBpZiBhY2xfY250ID09IDAgd2hlbiB0aGUgdHhfdGFzayBpcw0KPiA+IHNjaGVk
dWxlZD8NCj4gDQo+IFJpZ2h0IG5vdyBpdCBwcm9tb3RlcyBzdGFydmluZyBxdWV1ZXMsIGFsbCBv
ZiB0aGVtLCBidXQgbWF5YmUgaXQNCj4gc2hvdWxkbid0IHNvIHRoYXQgd2Ugc2ltcGxpZnkgdGhl
IGNvbXBsZXhpdHkgYSBsaXR0bGUgYml0Lg0KDQpNeSB0aGlua2luZyBoZXJlIGlzIHRoYXQgdGhl
IHR4IHRhc2tsZXQgc2hvdWxkbid0IGJlIHNjaGVkdWxlZCBhdCBhbGwgaWYNCm5vIHdvcmsgY2Fu
IGJlIHBlcmZvcm1lZC4gT2YgY291cnNlLCB0aGUgZXhpc3RpbmcgY29kZSBzdWZmZXJzIGZyb20g
dGhlDQpzYW1lIGRlZmVjdC4gUmVhbGx5LCB0aGUgb25seSByZWFzb24gdGhlIHR4IHRhc2tsZXQg
Km5lZWRzKiB0byBiZQ0Kc2NoZWR1bGVkIHdoZW4gdGhlIHJlbGV2YW50IGJ1ZmZlciBjb3VudGVy
IGlzIDAgaXMgdG8gY2F0Y2ggdHggdGltZW91dHMuDQoNClJlc3BlY3RmdWxseSwNClBldGVyIEh1
cmxleQ0K

Re: [RFC 1/3] Bluetooth: prioritizing data over HCI

[Luiz Augusto von Dentz]

Hi Peter,

On Wed, Aug 3, 2011 at 11:53 PM, Peter Hurley <peter@hurleysoftware.com> wrote:
> My point here (besides pointing out the massive iteration) was to
> suggest that perhaps some scheduler state should be preserved such that
> already-visited priority levels are not revisited more than once per tx
> tasklet schedule.

That is a valid point, Im gonna try to come up with something that address that.

>>  Also I guess for SCO/ESCO/LE e doesn't make
>> much sense to have many queues/priorities, it is basically ACL only,
>> that simplify a lot already.
>
> LE should still have priorities. In fact, in a private test we've been
> running here, we've merged the scheduler so that LE conns that share ACL
> buffers are scheduled alongside other ACL conns (because they share the
> same resource -- namely, the acl_cnt. We also merged SCO/ESCO scheduling
> together as well).

Well I guess it would make sense, specially for SCO/ESCO, now LE Im
not sure, it is not always the case the it uses acl_cnt, but in case
it does it should be prioritized according.

>>
>> >> +
>> >> +                       if (c->state != BT_CONNECTED && c->state != BT_CONFIG)
>> >> +                               continue;
>> >>
>> >> -               num++;
>> >> +                       num++;
>> >>
>> >> -               if (c->sent < min) {
>> >> -                       min  = c->sent;
>> >> -                       conn = c;
>> >> +                       if (c->sent < min) {
>> >> +                               min  = c->sent;
>> >> +                               conn = c;
>> >> +                               *queue = &c->data_q[i];
>> >> +                       }
>> >
>> > Why preserve the fairness logic?
>>
>> It does need to fair if there is 2 or more sockets with the same
>> priority, otherwise the first connection in the list might get all the
>> quote and even if we promote the starving queues it may still happen
>> to top most priority since there it cannot be promoted anymore.
>
> Right. What I meant here was this: if the tx scheduler is re-made as
> priority-based, then it seems logical to discard the fairness logic and
> design the priority scheme such that starvation is not possible. For
> example, the pri levels could be like this:
>        0 ~ 5   socket-programmable priorities
>        6       special CAP_NET_ADMIN priority
>        7       only possible via promotion
> (Not that I'm suggesting that's the only or even best way to make
> starvation not possible).

I would like to stick with current approach with bigger than 6 needing
CAP_NET_ADMIN which is aligned with with SO_PRIORITY current does,
otherwise it would immediately break the current application that uses
priority 6. Besides I don't think having one possible promotion index
is a good idea, it would completely ignore priorities in starvation
case, but there is exactly where it is most needed to throttle lower
priorities in favor of higher.

>> You probably have never run this code did you? Only priority 7 can
>> really monopolize the connection, and that is on purpose, and yes
>> lower priority are throttled so higher priority can get lower latency,
>> what is wrong with that?
>
> With no offense meant, simply running this code would be insufficient to
> qualify it's appropriateness as a replacement scheduler. A bare-minimum
> test matrix would ascertain comparative values for minimum/mean/maximum
> latency and throughput for best-case/nominal/worst-case tx loads.

But it would give a hint if your assumption are correct or not.

> My main concern here is that app-level socket settings can have dramatic
> effects on tx scheduling behavior, especially at near max tx loads.

An that is ultimately what is needed for guaranteed channels (priority
7), best effort is nearly unchanged because there is no guarantees of
latency or throughput.

> What I meant about near monopolization is this:  a priority 6 link that
> retires 2 packets every 55 ms would mean that priority 0 links could
> only xmit every 330 ms (1/3 sec.), assuming the controller was already
> saturated by the priority 6 link. Even priority 3 links are only going
> to be able to xmit every 165ms under these conditions.

So what? the application can and should be set its priority according
to not get throttled, it should only be fair with the same priority,
the other are left behind. Im gonna be repeating this over and over,
best effort channel as we have today have no guarantees (latency,
throughput), so it is perfectly ok to throttle them.

> What if, instead, links that are being promoted because the controller
> is saturated, are promoted to the same (or maybe, higher) priority level
> than was last successful?

Ive tried that, it breaks the prioritization when it is most needed
because it returns the schedulers to its fairness state and complete
ignores the priorities.

>> Right now it promotes starving queues, all of them, but maybe it
>> shouldn't so that we simplify the complexity a little bit.
>
> My thinking here is that the tx tasklet shouldn't be scheduled at all if
> no work can be performed. Of course, the existing code suffers from the
> same defect. Really, the only reason the tx tasklet *needs* to be
> scheduled when the relevant buffer counter is 0 is to catch tx timeouts.

Yep, that can indeed be fixed but then we need to find some other
place to detect tx timeouts.

-- 
Luiz Augusto von Dentz

[RFC 2/3] Bluetooth: set skbuffer priority based on L2CAP socket priority

[Luiz Augusto von Dentz]

From: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>

This uses SO_PRIORITY to set the skbuffer priority field

Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
---
 include/net/bluetooth/l2cap.h |    3 +-
 net/bluetooth/l2cap_core.c    |   47 +++++++++++++++++++++++++++++-----------
 net/bluetooth/l2cap_sock.c    |    2 +-
 3 files changed, 37 insertions(+), 15 deletions(-)

diff --git a/include/net/bluetooth/l2cap.h b/include/net/bluetooth/l2cap.h
index 4f34ad2..f018e5d 100644
--- a/include/net/bluetooth/l2cap.h
+++ b/include/net/bluetooth/l2cap.h
@@ -511,7 +511,8 @@ struct l2cap_chan *l2cap_chan_create(struct sock *sk);
 void l2cap_chan_close(struct l2cap_chan *chan, int reason);
 void l2cap_chan_destroy(struct l2cap_chan *chan);
 int l2cap_chan_connect(struct l2cap_chan *chan);
-int l2cap_chan_send(struct l2cap_chan *chan, struct msghdr *msg, size_t len);
+int l2cap_chan_send(struct l2cap_chan *chan, struct msghdr *msg, size_t len,
+								u32 priority);
 void l2cap_chan_busy(struct l2cap_chan *chan, int busy);
 
 #endif /* __L2CAP_H */
diff --git a/net/bluetooth/l2cap_core.c b/net/bluetooth/l2cap_core.c
index 3204ba8..2be08ba 100644
--- a/net/bluetooth/l2cap_core.c
+++ b/net/bluetooth/l2cap_core.c
@@ -550,6 +550,8 @@ static void l2cap_send_cmd(struct l2cap_conn *conn, u8 ident, u8 code, u16 len,
 	if (!skb)
 		return;
 
+	skb->priority = HCI_PRIO_MAX;
+
 	if (lmp_no_flush_capable(conn->hcon->hdev))
 		flags = ACL_START_NO_FLUSH;
 	else
@@ -1245,7 +1247,8 @@ void l2cap_do_send(struct l2cap_chan *chan, struct sk_buff *skb)
 	struct hci_conn *hcon = chan->conn->hcon;
 	u16 flags;
 
-	BT_DBG("chan %p, skb %p len %d", chan, skb, skb->len);
+	BT_DBG("chan %p, skb %p len %d priority %u", chan, skb, skb->len,
+							skb->priority);
 
 	if (!chan->flushable && lmp_no_flush_capable(hcon->hdev))
 		flags = ACL_START_NO_FLUSH;
@@ -1451,6 +1454,8 @@ static inline int l2cap_skbuff_fromiovec(struct sock *sk, struct msghdr *msg, in
 		if (memcpy_fromiovec(skb_put(*frag, count), msg->msg_iov, count))
 			return -EFAULT;
 
+		(*frag)->priority = skb->priority;
+
 		sent += count;
 		len  -= count;
 
@@ -1460,7 +1465,9 @@ static inline int l2cap_skbuff_fromiovec(struct sock *sk, struct msghdr *msg, in
 	return sent;
 }
 
-struct sk_buff *l2cap_create_connless_pdu(struct l2cap_chan *chan, struct msghdr *msg, size_t len)
+static struct sk_buff *l2cap_create_connless_pdu(struct l2cap_chan *chan,
+						struct msghdr *msg, size_t len,
+						u32 priority)
 {
 	struct sock *sk = chan->sk;
 	struct l2cap_conn *conn = chan->conn;
@@ -1468,7 +1475,7 @@ struct sk_buff *l2cap_create_connless_pdu(struct l2cap_chan *chan, struct msghdr
 	int err, count, hlen = L2CAP_HDR_SIZE + 2;
 	struct l2cap_hdr *lh;
 
-	BT_DBG("sk %p len %d", sk, (int)len);
+	BT_DBG("sk %p len %d priority %u", sk, (int)len, priority);
 
 	count = min_t(unsigned int, (conn->mtu - hlen), len);
 	skb = bt_skb_send_alloc(sk, count + hlen,
@@ -1476,6 +1483,8 @@ struct sk_buff *l2cap_create_connless_pdu(struct l2cap_chan *chan, struct msghdr
 	if (!skb)
 		return ERR_PTR(err);
 
+	skb->priority = priority;
+
 	/* Create L2CAP header */
 	lh = (struct l2cap_hdr *) skb_put(skb, L2CAP_HDR_SIZE);
 	lh->cid = cpu_to_le16(chan->dcid);
@@ -1490,7 +1499,9 @@ struct sk_buff *l2cap_create_connless_pdu(struct l2cap_chan *chan, struct msghdr
 	return skb;
 }
 
-struct sk_buff *l2cap_create_basic_pdu(struct l2cap_chan *chan, struct msghdr *msg, size_t len)
+static struct sk_buff *l2cap_create_basic_pdu(struct l2cap_chan *chan,
+						struct msghdr *msg, size_t len,
+						u32 priority)
 {
 	struct sock *sk = chan->sk;
 	struct l2cap_conn *conn = chan->conn;
@@ -1506,6 +1517,8 @@ struct sk_buff *l2cap_create_basic_pdu(struct l2cap_chan *chan, struct msghdr *m
 	if (!skb)
 		return ERR_PTR(err);
 
+	skb->priority = priority;
+
 	/* Create L2CAP header */
 	lh = (struct l2cap_hdr *) skb_put(skb, L2CAP_HDR_SIZE);
 	lh->cid = cpu_to_le16(chan->dcid);
@@ -1519,7 +1532,10 @@ struct sk_buff *l2cap_create_basic_pdu(struct l2cap_chan *chan, struct msghdr *m
 	return skb;
 }
 
-struct sk_buff *l2cap_create_iframe_pdu(struct l2cap_chan *chan, struct msghdr *msg, size_t len, u16 control, u16 sdulen)
+static struct sk_buff *l2cap_create_iframe_pdu(struct l2cap_chan *chan,
+						struct msghdr *msg, size_t len,
+						u32 priority, u16 control,
+						u16 sdulen)
 {
 	struct sock *sk = chan->sk;
 	struct l2cap_conn *conn = chan->conn;
@@ -1527,7 +1543,7 @@ struct sk_buff *l2cap_create_iframe_pdu(struct l2cap_chan *chan, struct msghdr *
 	int err, count, hlen = L2CAP_HDR_SIZE + 2;
 	struct l2cap_hdr *lh;
 
-	BT_DBG("sk %p len %d", sk, (int)len);
+	BT_DBG("sk %p len %d priority %u", sk, (int)len, priority);
 
 	if (!conn)
 		return ERR_PTR(-ENOTCONN);
@@ -1544,6 +1560,8 @@ struct sk_buff *l2cap_create_iframe_pdu(struct l2cap_chan *chan, struct msghdr *
 	if (!skb)
 		return ERR_PTR(err);
 
+	skb->priority = priority;
+
 	/* Create L2CAP header */
 	lh = (struct l2cap_hdr *) skb_put(skb, L2CAP_HDR_SIZE);
 	lh->cid = cpu_to_le16(chan->dcid);
@@ -1574,7 +1592,8 @@ int l2cap_sar_segment_sdu(struct l2cap_chan *chan, struct msghdr *msg, size_t le
 
 	skb_queue_head_init(&sar_queue);
 	control = L2CAP_SDU_START;
-	skb = l2cap_create_iframe_pdu(chan, msg, chan->remote_mps, control, len);
+	skb = l2cap_create_iframe_pdu(chan, msg, chan->remote_mps,
+						HCI_PRIO_MAX, control, len);
 	if (IS_ERR(skb))
 		return PTR_ERR(skb);
 
@@ -1593,7 +1612,8 @@ int l2cap_sar_segment_sdu(struct l2cap_chan *chan, struct msghdr *msg, size_t le
 			buflen = len;
 		}
 
-		skb = l2cap_create_iframe_pdu(chan, msg, buflen, control, 0);
+		skb = l2cap_create_iframe_pdu(chan, msg, buflen, HCI_PRIO_MAX,
+								control, 0);
 		if (IS_ERR(skb)) {
 			skb_queue_purge(&sar_queue);
 			return PTR_ERR(skb);
@@ -1610,7 +1630,8 @@ int l2cap_sar_segment_sdu(struct l2cap_chan *chan, struct msghdr *msg, size_t le
 	return size;
 }
 
-int l2cap_chan_send(struct l2cap_chan *chan, struct msghdr *msg, size_t len)
+int l2cap_chan_send(struct l2cap_chan *chan, struct msghdr *msg, size_t len,
+								u32 priority)
 {
 	struct sk_buff *skb;
 	u16 control;
@@ -1618,7 +1639,7 @@ int l2cap_chan_send(struct l2cap_chan *chan, struct msghdr *msg, size_t len)
 
 	/* Connectionless channel */
 	if (chan->chan_type == L2CAP_CHAN_CONN_LESS) {
-		skb = l2cap_create_connless_pdu(chan, msg, len);
+		skb = l2cap_create_connless_pdu(chan, msg, len, priority);
 		if (IS_ERR(skb))
 			return PTR_ERR(skb);
 
@@ -1633,7 +1654,7 @@ int l2cap_chan_send(struct l2cap_chan *chan, struct msghdr *msg, size_t len)
 			return -EMSGSIZE;
 
 		/* Create a basic PDU */
-		skb = l2cap_create_basic_pdu(chan, msg, len);
+		skb = l2cap_create_basic_pdu(chan, msg, len, priority);
 		if (IS_ERR(skb))
 			return PTR_ERR(skb);
 
@@ -1646,8 +1667,8 @@ int l2cap_chan_send(struct l2cap_chan *chan, struct msghdr *msg, size_t len)
 		/* Entire SDU fits into one PDU */
 		if (len <= chan->remote_mps) {
 			control = L2CAP_SDU_UNSEGMENTED;
-			skb = l2cap_create_iframe_pdu(chan, msg, len, control,
-									0);
+			skb = l2cap_create_iframe_pdu(chan, msg, len, priority,
+								control, 0);
 			if (IS_ERR(skb))
 				return PTR_ERR(skb);
 
diff --git a/net/bluetooth/l2cap_sock.c b/net/bluetooth/l2cap_sock.c
index 5c36b3e..abe302b 100644
--- a/net/bluetooth/l2cap_sock.c
+++ b/net/bluetooth/l2cap_sock.c
@@ -706,7 +706,7 @@ static int l2cap_sock_sendmsg(struct kiocb *iocb, struct socket *sock, struct ms
 		return -ENOTCONN;
 	}
 
-	err = l2cap_chan_send(chan, msg, len);
+	err = l2cap_chan_send(chan, msg, len, sk->sk_priority);
 
 	release_sock(sk);
 	return err;
-- 
1.7.6

[RFC 3/3] Bluetooth: make use sk_priority to priritize RFCOMM packets

[Luiz Augusto von Dentz]

From: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>

Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
---
 net/bluetooth/rfcomm/core.c |   51 +++++++++++++++++++++++++++++-------------
 1 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/net/bluetooth/rfcomm/core.c b/net/bluetooth/rfcomm/core.c
index 5759bb7..3ec98bd 100644
--- a/net/bluetooth/rfcomm/core.c
+++ b/net/bluetooth/rfcomm/core.c
@@ -66,7 +66,8 @@ static unsigned long rfcomm_event;
 
 static LIST_HEAD(session_list);
 
-static int rfcomm_send_frame(struct rfcomm_session *s, u8 *data, int len);
+static int rfcomm_send_frame(struct rfcomm_session *s, u8 *data, int len,
+							u32 priority);
 static int rfcomm_send_sabm(struct rfcomm_session *s, u8 dlci);
 static int rfcomm_send_disc(struct rfcomm_session *s, u8 dlci);
 static int rfcomm_queue_disc(struct rfcomm_dlc *d);
@@ -751,19 +752,34 @@ void rfcomm_session_getaddr(struct rfcomm_session *s, bdaddr_t *src, bdaddr_t *d
 }
 
 /* ---- RFCOMM frame sending ---- */
-static int rfcomm_send_frame(struct rfcomm_session *s, u8 *data, int len)
+static int rfcomm_send_frame(struct rfcomm_session *s, u8 *data, int len,
+							u32 priority)
 {
 	struct socket *sock = s->sock;
+	struct sock *sk = sock->sk;
 	struct kvec iv = { data, len };
 	struct msghdr msg;
 
-	BT_DBG("session %p len %d", s, len);
+	BT_DBG("session %p len %d priority %u", s, len, priority);
+
+	if (sk->sk_priority != priority) {
+		lock_sock(sk);
+		sk->sk_priority = priority;
+		release_sock(sk);
+	}
 
 	memset(&msg, 0, sizeof(msg));
 
 	return kernel_sendmsg(sock, &msg, &iv, 1, len);
 }
 
+static int rfcomm_send_cmd(struct rfcomm_session *s, struct rfcomm_cmd *cmd)
+{
+	BT_DBG("%p cmd %u", s, cmd->ctrl);
+
+	return rfcomm_send_frame(s, (void *) cmd, sizeof(*cmd), HCI_PRIO_MAX);
+}
+
 static int rfcomm_send_sabm(struct rfcomm_session *s, u8 dlci)
 {
 	struct rfcomm_cmd cmd;
@@ -775,7 +791,7 @@ static int rfcomm_send_sabm(struct rfcomm_session *s, u8 dlci)
 	cmd.len  = __len8(0);
 	cmd.fcs  = __fcs2((u8 *) &cmd);
 
-	return rfcomm_send_frame(s, (void *) &cmd, sizeof(cmd));
+	return rfcomm_send_cmd(s, &cmd);
 }
 
 static int rfcomm_send_ua(struct rfcomm_session *s, u8 dlci)
@@ -789,7 +805,7 @@ static int rfcomm_send_ua(struct rfcomm_session *s, u8 dlci)
 	cmd.len  = __len8(0);
 	cmd.fcs  = __fcs2((u8 *) &cmd);
 
-	return rfcomm_send_frame(s, (void *) &cmd, sizeof(cmd));
+	return rfcomm_send_cmd(s, &cmd);
 }
 
 static int rfcomm_send_disc(struct rfcomm_session *s, u8 dlci)
@@ -803,7 +819,7 @@ static int rfcomm_send_disc(struct rfcomm_session *s, u8 dlci)
 	cmd.len  = __len8(0);
 	cmd.fcs  = __fcs2((u8 *) &cmd);
 
-	return rfcomm_send_frame(s, (void *) &cmd, sizeof(cmd));
+	return rfcomm_send_cmd(s, &cmd);
 }
 
 static int rfcomm_queue_disc(struct rfcomm_dlc *d)
@@ -817,6 +833,8 @@ static int rfcomm_queue_disc(struct rfcomm_dlc *d)
 	if (!skb)
 		return -ENOMEM;
 
+	skb->priority = HCI_PRIO_MAX;
+
 	cmd = (void *) __skb_put(skb, sizeof(*cmd));
 	cmd->addr = d->addr;
 	cmd->ctrl = __ctrl(RFCOMM_DISC, 1);
@@ -839,7 +857,7 @@ static int rfcomm_send_dm(struct rfcomm_session *s, u8 dlci)
 	cmd.len  = __len8(0);
 	cmd.fcs  = __fcs2((u8 *) &cmd);
 
-	return rfcomm_send_frame(s, (void *) &cmd, sizeof(cmd));
+	return rfcomm_send_cmd(s, &cmd);
 }
 
 static int rfcomm_send_nsc(struct rfcomm_session *s, int cr, u8 type)
@@ -864,7 +882,7 @@ static int rfcomm_send_nsc(struct rfcomm_session *s, int cr, u8 type)
 
 	*ptr = __fcs(buf); ptr++;
 
-	return rfcomm_send_frame(s, buf, ptr - buf);
+	return rfcomm_send_frame(s, buf, ptr - buf, HCI_PRIO_MAX);
 }
 
 static int rfcomm_send_pn(struct rfcomm_session *s, int cr, struct rfcomm_dlc *d)
@@ -906,7 +924,7 @@ static int rfcomm_send_pn(struct rfcomm_session *s, int cr, struct rfcomm_dlc *d
 
 	*ptr = __fcs(buf); ptr++;
 
-	return rfcomm_send_frame(s, buf, ptr - buf);
+	return rfcomm_send_frame(s, buf, ptr - buf, HCI_PRIO_MAX);
 }
 
 int rfcomm_send_rpn(struct rfcomm_session *s, int cr, u8 dlci,
@@ -944,7 +962,7 @@ int rfcomm_send_rpn(struct rfcomm_session *s, int cr, u8 dlci,
 
 	*ptr = __fcs(buf); ptr++;
 
-	return rfcomm_send_frame(s, buf, ptr - buf);
+	return rfcomm_send_frame(s, buf, ptr - buf, HCI_PRIO_MAX);
 }
 
 static int rfcomm_send_rls(struct rfcomm_session *s, int cr, u8 dlci, u8 status)
@@ -971,7 +989,7 @@ static int rfcomm_send_rls(struct rfcomm_session *s, int cr, u8 dlci, u8 status)
 
 	*ptr = __fcs(buf); ptr++;
 
-	return rfcomm_send_frame(s, buf, ptr - buf);
+	return rfcomm_send_frame(s, buf, ptr - buf, HCI_PRIO_MAX);
 }
 
 static int rfcomm_send_msc(struct rfcomm_session *s, int cr, u8 dlci, u8 v24_sig)
@@ -998,7 +1016,7 @@ static int rfcomm_send_msc(struct rfcomm_session *s, int cr, u8 dlci, u8 v24_sig
 
 	*ptr = __fcs(buf); ptr++;
 
-	return rfcomm_send_frame(s, buf, ptr - buf);
+	return rfcomm_send_frame(s, buf, ptr - buf, HCI_PRIO_MAX);
 }
 
 static int rfcomm_send_fcoff(struct rfcomm_session *s, int cr)
@@ -1020,7 +1038,7 @@ static int rfcomm_send_fcoff(struct rfcomm_session *s, int cr)
 
 	*ptr = __fcs(buf); ptr++;
 
-	return rfcomm_send_frame(s, buf, ptr - buf);
+	return rfcomm_send_frame(s, buf, ptr - buf, HCI_PRIO_MAX);
 }
 
 static int rfcomm_send_fcon(struct rfcomm_session *s, int cr)
@@ -1042,7 +1060,7 @@ static int rfcomm_send_fcon(struct rfcomm_session *s, int cr)
 
 	*ptr = __fcs(buf); ptr++;
 
-	return rfcomm_send_frame(s, buf, ptr - buf);
+	return rfcomm_send_frame(s, buf, ptr - buf, HCI_PRIO_MAX);
 }
 
 static int rfcomm_send_test(struct rfcomm_session *s, int cr, u8 *pattern, int len)
@@ -1093,7 +1111,7 @@ static int rfcomm_send_credits(struct rfcomm_session *s, u8 addr, u8 credits)
 
 	*ptr = __fcs(buf); ptr++;
 
-	return rfcomm_send_frame(s, buf, ptr - buf);
+	return rfcomm_send_frame(s, buf, ptr - buf, HCI_PRIO_MAX);
 }
 
 static void rfcomm_make_uih(struct sk_buff *skb, u8 addr)
@@ -1771,7 +1789,8 @@ static inline int rfcomm_process_tx(struct rfcomm_dlc *d)
 		return skb_queue_len(&d->tx_queue);
 
 	while (d->tx_credits && (skb = skb_dequeue(&d->tx_queue))) {
-		err = rfcomm_send_frame(d->session, skb->data, skb->len);
+		err = rfcomm_send_frame(d->session, skb->data, skb->len,
+							skb->priority);
 		if (err < 0) {
 			skb_queue_head(&d->tx_queue, skb);
 			break;
-- 
1.7.6

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Peter Hurley]

SGkgTHVpeiwNCg0KT24gV2VkLCAyMDExLTA4LTAzIGF0IDA5OjExIC0wNDAwLCBMdWl6IEF1Z3Vz
dG8gdm9uIERlbnR6IHdyb3RlOg0KLi4uLg0KPiBUaGUgbWFpbiB1c2UgY2FzZSBmb3IgdGhpcyBp
cyBBMkRQLCBtYW55IHBlb3BsZSBjb21wbGFpbmVkIHRoYXQgd2hlbiB0aGV5DQo+IGFyZSB1c2lu
ZyB0aGVpciBISUQgZGV2aWNlcyAobW91c2VzKSB0b2dldGhlciB3aXRoIGhlYWRzZXQgdGhlIG1v
dXNlIGhhcw0KPiBoaWdoZXIgcHJpb3JpdHksIHdoaWNoIGJ0dyBpcyBub3QgdHJ1ZSBzaW5jZSBj
dXJyZW50bHkgdGhlcmUgaXMgbm8NCj4gcHJpb3JpdHkgYW5kIGNvbm5lY3Rpb25zIHJlY2VpdmVz
IHRoZSBzYW1lIHF1b3RlLg0KPiANCj4gV2l0aCB0aGlzIGNoYW5nZXMgaXQgbWF5IHJlZHVjZSB0
aGUgcHJvYmxlbSBzaW5jZSBQdWxzZUF1ZGlvIGFscmVhZHkgc2V0cw0KPiBBMkRQIHNvY2tldCBh
cyBsb3cgbGF0ZW5jeSAocHJpb3JpdHkgNiksIHdoaWNoIG1lYW5zIGF1ZGlvIHBhY2tldHMgd2ls
bCBiZQ0KPiBwcm9jZXNzZWQgYmVmb3JlIHRoZSBISUQgcGFja2V0cywgYnV0IHRoaXMgb25seSBh
ZmZlY3RzIHR4IGFuZCB0aGVyZSBzdGlsbA0KPiBleGlzdCB0aGUgcHJvYmxlbSBvZiBzb21lIEhJ
RCBub3QgbGV0dGluZyB1cyBiZWluZyBtYXN0ZXIuDQoNCldoYXQgcGFja2V0cyBhcmUgYmVpbmcg
dHJhbnNtaXR0ZWQgX3RvXyB0aGUgSElEIGRldmljZXMgZHVyaW5nIGF1ZGlvDQpzdHJlYW1pbmc/
ICBJbiBhbGwgbXkgY2FwdHVyZXMgb2YgQTJEUCBjcmFja2xlL3NraXBzLCB0aGVyZSdzIG5vdCBh
DQpzaW5nbGUgSElEIHBhY2tldCBiZWluZyB0cmFuc21pdHRlZCBvdXQuIE9mIGNvdXJzZSwgSSB3
YXMgY2FyZWZ1bGx5IG5vdA0KdG8gcHJlc3MgYW55IG9mIHRoZSBsZWQtdHJhbnNpdGlvbiBrZXlz
IGR1cmluZyB0aGUgY2FwdHVyZSA6KS4NCg0KSXQncyBteSBiZWxpZWYgdGhhdCB0aGUgaW5jcmVk
aWJseSBjb21wbGljYXRlZCB3YXkgdGhhdCBQdWxzZUF1ZGlvDQpoYW5kbGVzIHRoZSBzdHJlYW0g
aW5kaWNlcyBhbmQgY29tcHV0ZWQgbGF0ZW5jeSBpcyByZWFsbHkgdGhlIGlzc3VlIChhdA0KbGVh
c3Qgd2l0aCBBMkRQKS4NCg0KRm9yIGV4YW1wbGUsIGluIHNldmVyYWwgY29uZmlndXJhdGlvbiBJ
IGhhdmUsIFB1bHNlQXVkaW8gaXMgcmVuZGVyaW5nDQoqdG9vIGZhc3QqLiBUaGlzIGlzIGV2aWRl
bmNlZCBieSB0aGUgcmVtb3RlIGRldmljZSBzZW5kaW5nIFNUT1AgcGFja2V0cw0KKHdoaWNoIGNh
biBmbG9vZCB0aGUgc3lzbG9nIGFzIDAtbGVuZ3RoIEwyQ0FQIHBhY2tldHMpIGJlY2F1c2UgaXRz
IHJlY3YNCmJ1ZmZlcnMgYXJlIGZ1bGwhIChPZiBjb3Vyc2UsIHRoZXNlIERNMSBwYWNrZXRzIGFy
ZSBlYXRpbmcgdXAgYWlyIHRpbWUNCnRvbyAtLSB3aGljaCBpcyBzbG93aW5nIGxlZ2l0aW1hdGUg
cnggcGFja2V0cyBmcm9tIHRoZSBISUQgZGV2aWNlcyEpDQoNCllvdSBzaG91bGQgaGVhciB0aGUg
J3NraXBwaW5nJyB3aGVuIHRoZSBzbGVlcCB0aW1lciB3aGljaCBkZWxheXMgdGhlDQpuZXh0IGJs
b2NrIHJlbmRlciBpcyBJRkRFRidkIG91dCBvZiB0aGUNCm1vZHVsZS1ibHVldG9vdGgtZGV2aWNl
LmMvdGhyZWFkZm4oKS4gV2l0aCBvdGhlciBzdWJzeXN0ZW1zIGJ1dCBub3Qgd2l0aA0KQmx1ZXos
IFB1bHNlQXVkaW8gdXNlcyBhICdzbW9vdGhlcicgdG8gZmluZS10dW5lIHdoZW4gdG8gcmVuZGVy
IHRoZSBuZXh0DQpibG9jay4NCg0KTXkgcG9pbnQgaGVyZSBpcyBJIHRoaW5rIGl0J3MgcHJlbWF0
dXJlIHRvIGNvbmNsdWRlIHRoYXQgUHVsc2VBdWRpbw0KZG9lc24ndCBoYXZlIGEgcGFydCB0byBw
bGF5Lg0KDQo+IEl2ZSBiZWVuIHRlc3RpbmcgdGhpcyB1c2luZyBBMkRQIGFuZCBISUQgY29ubmVj
dGVkIHNpbXVsdGFuZXVzbHkgd2l0aG91dCBhDQo+IHByb2JsZW0sIHRob3VnaCBjb25uZWN0aW5n
L3BhZ2luZyBzdGlsbCBtYWtlcyBhdWRpbyBza2lwcywgYW5kIGFsc28gdHJpZWQNCj4gT0JFWCB0
cmFuc2ZlcnMgdG9nZXRoZXIgdG8gc2VlIGlmIGF1ZGlvIHNraXBzLCBpbmRlZWQgaXQgZG9lcyB3
aGVuIEhJRCBpcw0KPiBhY3RpdmUgYnV0IGl0IHdvcmtzIGJldHRlciB0aGFuIGhhdmluZyBubyBw
cmlvcml0eSAoc2tpcHMgbGVzcyBmcmVxdWVudCkNCj4gYW5kIGluIGNhc2Ugb2YgdXNpbmcgcHJp
b3JpdHkgNyB0aGUgYXVkaW8gZG9lc24ndCBza2lwIGFueW1vcmUgYnV0IGl0DQo+IGV2ZW50dWFs
bHkgc3RhbGwgdGhlIE9CRVggdHJhbnNmZXIgKHRoaXMgaXMgbWF5YmUgYmVjYXVzZSBPcGVuT0JF
WCBsb29wDQo+IHdoaWxlIHRyeWluZyB0byB3cml0ZSB0byBzb2NrZXQgYW5kIGRvbid0IHdhaXQg
Zm9yIFBPTExPVVQpLg0KDQpJbnRlcmVzdGluZy4uLg0KDQpSZWdhcmRzLA0KUGV0ZXIgSHVybGV5
DQoNCg==

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Luiz Augusto von Dentz]

Hi Peter,

On Thu, Aug 4, 2011 at 12:14 AM, Peter Hurley <peter@hurleysoftware.com> wrote:
> Hi Luiz,
>
> On Wed, 2011-08-03 at 09:11 -0400, Luiz Augusto von Dentz wrote:
> ....
>> The main use case for this is A2DP, many people complained that when they
>> are using their HID devices (mouses) together with headset the mouse has
>> higher priority, which btw is not true since currently there is no
>> priority and connections receives the same quote.
>>
>> With this changes it may reduce the problem since PulseAudio already sets
>> A2DP socket as low latency (priority 6), which means audio packets will be
>> processed before the HID packets, but this only affects tx and there still
>> exist the problem of some HID not letting us being master.
>
> What packets are being transmitted _to_ the HID devices during audio
> streaming?  In all my captures of A2DP crackle/skips, there's not a
> single HID packet being transmitted out. Of course, I was carefully not
> to press any of the led-transition keys during the capture :).

Well with HID you will probably be only receiving, but that doesn't
change anything does it? I mean you still got less bandwidth to use
and we probably want low latency packets to be prioritized in any
case, besides there is probably some use case where HID transmit too,
see net/bluetooth/hidp/core.c:666(hidp_process_transmit).

> It's my belief that the incredibly complicated way that PulseAudio
> handles the stream indices and computed latency is really the issue (at
> least with A2DP).

Really, so you have a fix for that? I would be happy to see your
proposal and discussion on PulseAudio mailing list.

> For example, in several configuration I have, PulseAudio is rendering
> *too fast*. This is evidenced by the remote device sending STOP packets
> (which can flood the syslog as 0-length L2CAP packets) because its recv
> buffers are full! (Of course, these DM1 packets are eating up air time
> too -- which is slowing legitimate rx packets from the HID devices!)

But why the kernel is not stopping PulseAudio? I mean PA does use
POLLOUT to check if the socket is writable, it then writes as much as
possible and sleep, but without using guaranteed channels with proper
QoS support I don't this changing anytime soon.

> You should hear the 'skipping' when the sleep timer which delays the
> next block render is IFDEF'd out of the
> module-bluetooth-device.c/threadfn(). With other subsystems but not with
> Bluez, PulseAudio uses a 'smoother' to fine-tune when to render the next
> block.

This can be fixed, the source already uses the smoother anyway, but Im
afraid this can cause bad effects like much frequent wakeups and even
auto of sync with some headsets.

> My point here is I think it's premature to conclude that PulseAudio
> doesn't have a part to play.

It probably does, but it doesn't change the fact the prioritization is
pretty much needed, actually it will probably make such problems more
visible than ever.

-- 
Luiz Augusto von Dentz

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Peter Hurley]

SGkgTHVpeiwNCg0KT24gVGh1LCAyMDExLTA4LTA0IGF0IDA0OjIwIC0wNDAwLCBMdWl6IEF1Z3Vz
dG8gdm9uIERlbnR6IHdyb3RlOg0KPiBIaSBQZXRlciwNCj4gDQo+IE9uIFRodSwgQXVnIDQsIDIw
MTEgYXQgMTI6MTQgQU0sIFBldGVyIEh1cmxleSA8cGV0ZXJAaHVybGV5c29mdHdhcmUuY29tPiB3
cm90ZToNCj4gPiBIaSBMdWl6LA0KPiA+DQo+ID4gLi4uLi4NCj4gPg0KPiA+IEl0J3MgbXkgYmVs
aWVmIHRoYXQgdGhlIGluY3JlZGlibHkgY29tcGxpY2F0ZWQgd2F5IHRoYXQgUHVsc2VBdWRpbw0K
PiA+IGhhbmRsZXMgdGhlIHN0cmVhbSBpbmRpY2VzIGFuZCBjb21wdXRlZCBsYXRlbmN5IGlzIHJl
YWxseSB0aGUgaXNzdWUgKGF0DQo+ID4gbGVhc3Qgd2l0aCBBMkRQKS4NCj4gDQo+IFJlYWxseSwg
c28geW91IGhhdmUgYSBmaXggZm9yIHRoYXQ/IEkgd291bGQgYmUgaGFwcHkgdG8gc2VlIHlvdXIN
Cj4gcHJvcG9zYWwgYW5kIGRpc2N1c3Npb24gb24gUHVsc2VBdWRpbyBtYWlsaW5nIGxpc3QuDQoN
CkknZCByYXRoZXIgc3RhcnQgb3ZlciA6KQ0KDQo+ID4gRm9yIGV4YW1wbGUsIGluIHNldmVyYWwg
Y29uZmlndXJhdGlvbiBJIGhhdmUsIFB1bHNlQXVkaW8gaXMgcmVuZGVyaW5nDQo+ID4gKnRvbyBm
YXN0Ki4gVGhpcyBpcyBldmlkZW5jZWQgYnkgdGhlIHJlbW90ZSBkZXZpY2Ugc2VuZGluZyBTVE9Q
IHBhY2tldHMNCj4gPiAod2hpY2ggY2FuIGZsb29kIHRoZSBzeXNsb2cgYXMgMC1sZW5ndGggTDJD
QVAgcGFja2V0cykgYmVjYXVzZSBpdHMgcmVjdg0KPiA+IGJ1ZmZlcnMgYXJlIGZ1bGwhIChPZiBj
b3Vyc2UsIHRoZXNlIERNMSBwYWNrZXRzIGFyZSBlYXRpbmcgdXAgYWlyIHRpbWUNCj4gPiB0b28g
LS0gd2hpY2ggaXMgc2xvd2luZyBsZWdpdGltYXRlIHJ4IHBhY2tldHMgZnJvbSB0aGUgSElEIGRl
dmljZXMhKQ0KPiANCj4gQnV0IHdoeSB0aGUga2VybmVsIGlzIG5vdCBzdG9wcGluZyBQdWxzZUF1
ZGlvPyBJIG1lYW4gUEEgZG9lcyB1c2UNCj4gUE9MTE9VVCB0byBjaGVjayBpZiB0aGUgc29ja2V0
IGlzIHdyaXRhYmxlLCBpdCB0aGVuIHdyaXRlcyBhcyBtdWNoIGFzDQo+IHBvc3NpYmxlIGFuZCBz
bGVlcCwgYnV0IHdpdGhvdXQgdXNpbmcgZ3VhcmFudGVlZCBjaGFubmVscyB3aXRoIHByb3Blcg0K
PiBRb1Mgc3VwcG9ydCBJIGRvbid0IHRoaXMgY2hhbmdpbmcgYW55dGltZSBzb29uLg0KDQpUaGUg
a2VybmVsIGNhbid0IHN0b3AgdGhlIHNvY2tldCBiZWNhdXNlIHRoZXJlJ3Mgbm90IGVub3VnaCBp
bmZvcm1hdGlvbg0KdG8ga25vdyB3aGVuIHRvIHJlc3RhcnQgaXQgKGFuZCBJIGRvdWJ0IG90aGVy
IGNvbnRyb2xsZXJzIGZvcndhcmQgU1RPUA0KcGFja2V0cyB0byB0aGUgaG9zdCBsaWtlIEJDTSku
DQoNClRoaXMgaXMgbXkgd2hvbGUgcG9pbnQgYWJvdXQgdGhlIHN0cmVhbSBpbmRpY2VzIGFuZCBj
b21wdXRlZCBsYXRlbmN5Og0KUHVsc2VBdWRpbydzIGRlc2lnbiBpcyBwcmVkaWNhdGVkIG9uICpr
bm93aW5nKiB0aGUgcmVsYXRpb25zaGlwIGJldHdlZW4NCmVsYXBzZWQgdGltZSBhbmQgc3RyZWFt
IGluZGV4LCBhbmQgdGh1cyBiZWluZyBhYmxlIHRvIGRlbGl2ZXIgZGF0YSB0bw0KdGhlIHNpbmsg
anVzdCBiZWZvcmUgaXQncyBuZWVkZWQgKGVzcGVjaWFsbHkgd2l0aCBwcmUtZW1wdGlibGUga2Vy
bmVsKS4NCg0KQnV0IGZvciBhIHJlbW90ZSBkZXZpY2UgdG8gYmUgZmxvdy1jb250cm9sbGluZyBp
bXBsaWVzIHRoYXQgUHVsc2VBdWRpbw0KaXMgd2F5IGFoZWFkIG9mIHdoYXQncyBhY3R1YWxseSBw
bGF5aW5nLiBGdWxsIHJlbW90ZSByZWN2IGJ1ZmZlcnMgKyBmdWxsDQpob3N0IHR4IGJ1ZmZlcnMg
KiA4NTMtYnl0ZSBwYWNrZXRzIG9mIFJUUCBkYXRhIGlzIHByZXR0eSBmYXIgYWhlYWQhDQoNCj4g
PiBZb3Ugc2hvdWxkIGhlYXIgdGhlICdza2lwcGluZycgd2hlbiB0aGUgc2xlZXAgdGltZXIgd2hp
Y2ggZGVsYXlzIHRoZQ0KPiA+IG5leHQgYmxvY2sgcmVuZGVyIGlzIElGREVGJ2Qgb3V0IG9mIHRo
ZQ0KPiA+IG1vZHVsZS1ibHVldG9vdGgtZGV2aWNlLmMvdGhyZWFkZm4oKS4gV2l0aCBvdGhlciBz
dWJzeXN0ZW1zIGJ1dCBub3Qgd2l0aA0KPiA+IEJsdWV6LCBQdWxzZUF1ZGlvIHVzZXMgYSAnc21v
b3RoZXInIHRvIGZpbmUtdHVuZSB3aGVuIHRvIHJlbmRlciB0aGUgbmV4dA0KPiA+IGJsb2NrLg0K
PiANCj4gVGhpcyBjYW4gYmUgZml4ZWQsIHRoZSBzb3VyY2UgYWxyZWFkeSB1c2VzIHRoZSBzbW9v
dGhlciBhbnl3YXksIGJ1dCBJbQ0KPiBhZnJhaWQgdGhpcyBjYW4gY2F1c2UgYmFkIGVmZmVjdHMg
bGlrZSBtdWNoIGZyZXF1ZW50IHdha2V1cHMgYW5kIGV2ZW4NCj4gYXV0byBvZiBzeW5jIHdpdGgg
c29tZSBoZWFkc2V0cy4NCg0KRXhhY3RseS4gVGhlIGVsZW1lbnQgKHNtb290aGVyKSBkZXNpZ25l
ZCBzcGVjaWZpY2FsbHkgdG8gZW5zdXJlIHRoZQ0KY29ycmVjdCBsYXRlbmN5IGNhbGN1bGF0aW9u
LCBkb2Vzbid0Lg0KDQpbQXNpZGU6IHRoZSBwcm9ibGVtcyBjYXVzZWQgYnkgcmVtb3ZpbmcgdGhl
IHNsZWVwIHRpbWVyIGdvIGRlZXBlciB0aGFuDQpqdXN0IGZsb29kaW5nIEJUIC0gaXQgYWxzbyBl
eHBvc2VzIHNvbWUgc291cmNlIGluZGV4IHByb2JsZW1zLiBKdXN0DQpoYXZlbid0IGhhZCB0aW1l
IHRvIGludmVzdGlnYXRlIHRoYXQgZnVydGhlciB5ZXRdDQoNCj4gPiBNeSBwb2ludCBoZXJlIGlz
IEkgdGhpbmsgaXQncyBwcmVtYXR1cmUgdG8gY29uY2x1ZGUgdGhhdCBQdWxzZUF1ZGlvDQo+ID4g
ZG9lc24ndCBoYXZlIGEgcGFydCB0byBwbGF5Lg0KPiANCj4gSXQgcHJvYmFibHkgZG9lcywgYnV0
IGl0IGRvZXNuJ3QgY2hhbmdlIHRoZSBmYWN0IHRoZSBwcmlvcml0aXphdGlvbiBpcw0KPiBwcmV0
dHkgbXVjaCBuZWVkZWQsIGFjdHVhbGx5IGl0IHdpbGwgcHJvYmFibHkgbWFrZSBzdWNoIHByb2Js
ZW1zIG1vcmUNCj4gdmlzaWJsZSB0aGFuIGV2ZXIuDQoNCkFncmVlZC4gQXMgSSB3cm90ZSBpbiB0
aGUgb3RoZXIgcG9zdCwgSSB0aGluayBwcmlvcml0eS1iYXNlZCB0eA0Kc2NoZWR1bGluZyBpcyBh
biBleGNlbGxlbnQgaWRlYS4NCg0KUmVnYXJkcywNClBldGVyDQoNCg==

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Mat Martineau]

On Wed, 3 Aug 2011, Luiz Augusto von Dentz wrote:

> From: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
>
> This set of patches uses the priority of the socket, set via SO_PRIORITY,
> to prioritized data over HCI. Current priority range is from 0-7, where
> 0-6 priority can be set without any exatra capability which is according
> to SO_PRIORITY documentation (socket(7)):
>
> SO_PRIORITY
>
> 	Set  the  protocol-defined  priority  for  all packets to be sent
> 	on this socket.  Linux uses this value to order the networking
> 	queues: packets with a higher priority may be processed first
> 	depending on the selected device queueing discipline....
> 	Setting  a  priority outside  the  range  0  to  6 requires the
> 	CAP_NET_ADMIN capability.
>
> This also enables in the future the use of guaranteed channels as it is
> necessary to prioritized them over best effort (default) channels,
> Bluetooth V4.0 spec (page 1497):
>
> 7.9 PRIORITIZING DATA OVER HCI
>
> 	In order for guaranteed channels to meet their guarantees, L2CAP
> 	should prioritize traffic over the HCI transport in devices that
> 	support HCI. Packets for Guaranteed channels should receive higher
> 	priority than packets for Best Effort channels.
>
> Note: to be able to use guaranteed channels we need to negotiate QoS
> parameters both in L2CAP and HCI, this is outside of the scope of this
> patches since it probably requires new socket options to L2CAP sockets.
>
> The main use case for this is A2DP, many people complained that when they
> are using their HID devices (mouses) together with headset the mouse has
> higher priority, which btw is not true since currently there is no
> priority and connections receives the same quote.
>
> With this changes it may reduce the problem since PulseAudio already sets
> A2DP socket as low latency (priority 6), which means audio packets will be
> processed before the HID packets, but this only affects tx and there still
> exist the problem of some HID not letting us being master.
>
> Ive been testing this using A2DP and HID connected simultaneusly without a
> problem, though connecting/paging still makes audio skips, and also tried
> OBEX transfers together to see if audio skips, indeed it does when HID is
> active but it works better than having no priority (skips less frequent)
> and in case of using priority 7 the audio doesn't skip anymore but it
> eventually stall the OBEX transfer (this is maybe because OpenOBEX loop
> while trying to write to socket and don't wait for POLLOUT).
>
> Luiz Augusto von Dentz (3):
>  Bluetooth: prioritizing data over HCI
>  Bluetooth: set skbuffer priority based on L2CAP socket priority
>  Bluetooth: make use sk_priority to priritize RFCOMM packets
>
> include/net/bluetooth/hci_core.h |    6 ++-
> include/net/bluetooth/l2cap.h    |    3 +-
> net/bluetooth/hci_conn.c         |    8 ++-
> net/bluetooth/hci_core.c         |  141 +++++++++++++++++++++++++++----------
> net/bluetooth/l2cap_core.c       |   47 +++++++++----
> net/bluetooth/l2cap_sock.c       |    2 +-
> net/bluetooth/rfcomm/core.c      |   51 +++++++++----
> net/bluetooth/rfcomm/sock.c      |    8 ++
> 8 files changed, 194 insertions(+), 72 deletions(-)
>
> -- 
> 1.7.6


Luiz -

Thanks for making the effort to improve the scheduler!  The AMP code 
I'm upstreaming has some more building blocks for QoS, so we are 
getting closer to QoS support.

One concern I have is that an application that changes sk_priority 
while it has data in the HCI tx queue could have its frames delivered 
out of order.  While we can't control when sk_priority is set, it is 
possible to make a choice when setting the priority of each skb.  Does 
it make sense to detect an sk_priority change and only apply the new 
value when there are no skbs for the channel in the HCI tx queue?


I had a recent discussion with Gustavo about HCI queuing issues with 
ERTM:

http://www.spinics.net/lists/linux-bluetooth/msg13774.html

My proposal is to move tx queuing up to L2CAP, and have the HCI tx 
task only handle scheduling.  Senders would tell HCI they have data to 
send, and HCI would call back to pull data.  I've been focused on 
L2CAP - it would be possible to make a similar queuing change to 
SCO/eSCO/LE, but not strictly necessary.

This is different from the problem you're solving, but as long as 
we're talking about big changes to HCI tx, I think we could make some 
changes that help out with prioritization and queuing across all 
channels (whether they're going to the same remote device or different 
remote devices).

It could be more efficient to schedule this way since HCI wouldn't 
have to iterate over all of the connections and queues - it would only 
have to review the prioritized list of pending sends. Might be a more 
fitting structure for QoS too.

Since HCI wouldn't have a bunch of queued-up skbs, it would also 
eliminate the sk_priority change problem I noted above.

Your thoughts?

--
Mat Martineau
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Peter Hurley]

SGkgTWF0LA0KDQpPbiBUaHUsIDIwMTEtMDgtMDQgYXQgMTM6MzcgLTA0MDAsIE1hdCBNYXJ0aW5l
YXUgd3JvdGU6DQoNCj4gSSBoYWQgYSByZWNlbnQgZGlzY3Vzc2lvbiB3aXRoIEd1c3Rhdm8gYWJv
dXQgSENJIHF1ZXVpbmcgaXNzdWVzIHdpdGggDQo+IEVSVE06DQo+IA0KPiBodHRwOi8vd3d3LnNw
aW5pY3MubmV0L2xpc3RzL2xpbnV4LWJsdWV0b290aC9tc2cxMzc3NC5odG1sDQo+IA0KPiBNeSBw
cm9wb3NhbCBpcyB0byBtb3ZlIHR4IHF1ZXVpbmcgdXAgdG8gTDJDQVAsIGFuZCBoYXZlIHRoZSBI
Q0kgdHggDQo+IHRhc2sgb25seSBoYW5kbGUgc2NoZWR1bGluZy4gIFNlbmRlcnMgd291bGQgdGVs
bCBIQ0kgdGhleSBoYXZlIGRhdGEgdG8gDQo+IHNlbmQsIGFuZCBIQ0kgd291bGQgY2FsbCBiYWNr
IHRvIHB1bGwgZGF0YS4gIEkndmUgYmVlbiBmb2N1c2VkIG9uIA0KPiBMMkNBUCAtIGl0IHdvdWxk
IGJlIHBvc3NpYmxlIHRvIG1ha2UgYSBzaW1pbGFyIHF1ZXVpbmcgY2hhbmdlIHRvIA0KPiBTQ08v
ZVNDTy9MRSwgYnV0IG5vdCBzdHJpY3RseSBuZWNlc3NhcnkuDQoNCldvdWxkIHlvdSBwbGVhc2Ug
Y2xhcmlmeSB0aGlzIGFwcHJvYWNoIChwZXJoYXBzIGluIGEgc2VwYXJhdGUgdGhyZWFkKT8NCg0K
Rm9yIGV4YW1wbGUsIGhvdyBkb2VzIGhhdmluZyB0eCBxdWV1ZXMgaW4gbDJjYXBfY2hhbiAoaW5z
dGVhZCBvZiB0aGUNCmhjaV9jb25uKSBzb2x2ZSB0aGUgbGF0ZW5jeSBwcm9ibGVtcyBpbiBFUlRN
IHdoZW4gcmVwbHlpbmcgdG8NClJFSi9TUkVKL3BvbGw/IFdvbid0IHRoZXJlIHBvdGVudGlhbGx5
IGJlIGp1c3QgYXMgbXVjaCBkYXRhIGFscmVhZHkNCnF1ZXVlZCB1cD8gSXMgdGhlIHBsYW4gdG8g
bW92ZSB0aGUgcmVwbHkgdG8gdGhlIGZyb250IG9mIHRoZSB0eCBxdWV1ZQ0KYmVjYXVzZSByZXFz
ZXEgd29uJ3QgbmVlZCB0byBiZSBhc3NpZ25lZCB1bnRpbCB0aGUgZnJhbWUgaXMgYWN0dWFsbHkN
CnB1bGxlZCBvZmYgdGhlIHF1ZXVlPw0KDQo+IFRoaXMgaXMgZGlmZmVyZW50IGZyb20gdGhlIHBy
b2JsZW0geW91J3JlIHNvbHZpbmcsIGJ1dCBhcyBsb25nIGFzIA0KPiB3ZSdyZSB0YWxraW5nIGFi
b3V0IGJpZyBjaGFuZ2VzIHRvIEhDSSB0eCwgSSB0aGluayB3ZSBjb3VsZCBtYWtlIHNvbWUgDQo+
IGNoYW5nZXMgdGhhdCBoZWxwIG91dCB3aXRoIHByaW9yaXRpemF0aW9uIGFuZCBxdWV1aW5nIGFj
cm9zcyBhbGwgDQo+IGNoYW5uZWxzICh3aGV0aGVyIHRoZXkncmUgZ29pbmcgdG8gdGhlIHNhbWUg
cmVtb3RlIGRldmljZSBvciBkaWZmZXJlbnQgDQo+IHJlbW90ZSBkZXZpY2VzKS4NCj4gDQo+IEl0
IGNvdWxkIGJlIG1vcmUgZWZmaWNpZW50IHRvIHNjaGVkdWxlIHRoaXMgd2F5IHNpbmNlIEhDSSB3
b3VsZG4ndCANCj4gaGF2ZSB0byBpdGVyYXRlIG92ZXIgYWxsIG9mIHRoZSBjb25uZWN0aW9ucyBh
bmQgcXVldWVzIC0gaXQgd291bGQgb25seSANCj4gaGF2ZSB0byByZXZpZXcgdGhlIHByaW9yaXRp
emVkIGxpc3Qgb2YgcGVuZGluZyBzZW5kcy4gTWlnaHQgYmUgYSBtb3JlIA0KPiBmaXR0aW5nIHN0
cnVjdHVyZSBmb3IgUW9TIHRvby4NCg0KV291bGRuJ3QgdGhpcyAncHJpb3JpdGl6ZWQgbGlzdCBv
ZiBwZW5kaW5nIHNlbmRzJyBuZWVkIHRvIGJlIGxvY2tlZCB0bw0KcHJldmVudCB1cGRhdGUgd2hp
bGUgcmVhZGluZz8gSVNUTSB0aGF0IGNvbnRlbnRpb24gbWlnaHQgYmUgZmFpcmx5IGhpZ2gNCmZv
ciBhIHNpbmdsZSwgc2hhcmVkIHJlc291cmNlIGxpa2UgdGhhdC4NCg0KDQpSZWdhcmRzLA0KUGV0
ZXIgSHVybGV5DQoNCg==

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Gustavo Padovan]

* Peter Hurley <peter@hurleysoftware.com> [2011-08-04 19:09:37 -0400]:

> Hi Mat,
> 
> On Thu, 2011-08-04 at 13:37 -0400, Mat Martineau wrote:
> 
> > I had a recent discussion with Gustavo about HCI queuing issues with 
> > ERTM:
> > 
> > http://www.spinics.net/lists/linux-bluetooth/msg13774.html
> > 
> > My proposal is to move tx queuing up to L2CAP, and have the HCI tx 
> > task only handle scheduling.  Senders would tell HCI they have data to 
> > send, and HCI would call back to pull data.  I've been focused on 
> > L2CAP - it would be possible to make a similar queuing change to 
> > SCO/eSCO/LE, but not strictly necessary.
> 
> Would you please clarify this approach (perhaps in a separate thread)?
> 
> For example, how does having tx queues in l2cap_chan (instead of the
> hci_conn) solve the latency problems in ERTM when replying to
> REJ/SREJ/poll? Won't there potentially be just as much data already
> queued up? Is the plan to move the reply to the front of the tx queue
> because reqseq won't need to be assigned until the frame is actually
> pulled off the queue?

Exactly. ERTM connections can get dropped if the too much data is buffered and
we need to send final bit for example.

> 
> > This is different from the problem you're solving, but as long as 
> > we're talking about big changes to HCI tx, I think we could make some 
> > changes that help out with prioritization and queuing across all 
> > channels (whether they're going to the same remote device or different 
> > remote devices).
> > 
> > It could be more efficient to schedule this way since HCI wouldn't 
> > have to iterate over all of the connections and queues - it would only 
> > have to review the prioritized list of pending sends. Might be a more 
> > fitting structure for QoS too.
> 
> Wouldn't this 'prioritized list of pending sends' need to be locked to
> prevent update while reading? ISTM that contention might be fairly high
> for a single, shared resource like that.
> 
> 
> Regards,
> Peter Hurley
> 

	Gustavo

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Mat Martineau]

On Fri, 5 Aug 2011, Gustavo Padovan wrote:

> * Peter Hurley <peter@hurleysoftware.com> [2011-08-04 19:09:37 -0400]:
>
>> Hi Mat,
>>
>> On Thu, 2011-08-04 at 13:37 -0400, Mat Martineau wrote:
>>
>>> I had a recent discussion with Gustavo about HCI queuing issues with
>>> ERTM:
>>>
>>> http://www.spinics.net/lists/linux-bluetooth/msg13774.html
>>>
>>> My proposal is to move tx queuing up to L2CAP, and have the HCI tx
>>> task only handle scheduling.  Senders would tell HCI they have data to
>>> send, and HCI would call back to pull data.  I've been focused on
>>> L2CAP - it would be possible to make a similar queuing change to
>>> SCO/eSCO/LE, but not strictly necessary.
>>
>> Would you please clarify this approach (perhaps in a separate thread)?
>>
>> For example, how does having tx queues in l2cap_chan (instead of the
>> hci_conn) solve the latency problems in ERTM when replying to
>> REJ/SREJ/poll? Won't there potentially be just as much data already
>> queued up? Is the plan to move the reply to the front of the tx queue
>> because reqseq won't need to be assigned until the frame is actually
>> pulled off the queue?
>
> Exactly. ERTM connections can get dropped if the too much data is buffered and
> we need to send final bit for example.

Right now, an outgoing ERTM frame goes through two queues: a 
channel-specific ERTM tx queue and the HCI ACL data_q.  The ERTM 
control field is not constructed until a frame is removed from the 
ERTM tx queue and pushed to the HCI data_q, so the s-frame latency 
problem comes in when the the HCI data_q gets deep.  S-frames are 
already pushed directly in to the HCI data_q, bypassing the data tx 
queue.

>From an ERTM perspective, the goal is to defer assignment of reqseq 
and f-bit values as late as possible, so the remote device gets the 
most recent information on data frames and polls that have been 
received.  The optimal thing to do (by this measurement, anyway) is to 
build the ERTM control field as data is sent to the baseband -- in 
other words, to eliminate the HCI data_q altogether.

(Yeah, without the data_q, ERTM would need additional queues for 
s-frames and retransmitted i-frames)

So, without a data_q, what makes sense?  If there are ACL buffers 
available and no pending L2CAP senders, it would be great to push data 
straight out to the baseband.  If we're blocked waiting for 
num_completed_packets, then receipt of num_completed_packets is the 
natural time to pull data from the tx queues that now happen to be up 
in the L2CAP layer.

There are certainly locking, task scheduling, data scheduling, QoS, 
and efficiency issues to consider.  This is just a general description 
for now, and I'm trying to see if there's enough interest (or few 
enough obvious gotchas) to put some serious effort in to moving 
forward.


>>> This is different from the problem you're solving, but as long as
>>> we're talking about big changes to HCI tx, I think we could make some
>>> changes that help out with prioritization and queuing across all
>>> channels (whether they're going to the same remote device or different
>>> remote devices).
>>>
>>> It could be more efficient to schedule this way since HCI wouldn't
>>> have to iterate over all of the connections and queues - it would only
>>> have to review the prioritized list of pending sends. Might be a more
>>> fitting structure for QoS too.
>>
>> Wouldn't this 'prioritized list of pending sends' need to be locked to
>> prevent update while reading? ISTM that contention might be fairly high
>> for a single, shared resource like that.

If there was a prioritized list that was manipulated on multiple 
threads, then a lock certainly would be required.  Maybe it would work 
to keep the information sent between threads pretty simple (just an "I 
have data" bit), then any complex lists could be manipulated in a more 
controlled HCI-only context.  The prioritization code could run only 
when processing num_completed_packets.


Thanks for the great feedback so far!  Seeing any major flaws at this 
high level?


Regards,

--
Mat Martineau
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Gustavo Padovan]

* Mat Martineau <mathewm@codeaurora.org> [2011-08-08 16:29:51 -0700]:

> 
> On Fri, 5 Aug 2011, Gustavo Padovan wrote:
> 
> >* Peter Hurley <peter@hurleysoftware.com> [2011-08-04 19:09:37 -0400]:
> >
> >>Hi Mat,
> >>
> >>On Thu, 2011-08-04 at 13:37 -0400, Mat Martineau wrote:
> >>
> >>>I had a recent discussion with Gustavo about HCI queuing issues with
> >>>ERTM:
> >>>
> >>>http://www.spinics.net/lists/linux-bluetooth/msg13774.html
> >>>
> >>>My proposal is to move tx queuing up to L2CAP, and have the HCI tx
> >>>task only handle scheduling.  Senders would tell HCI they have data to
> >>>send, and HCI would call back to pull data.  I've been focused on
> >>>L2CAP - it would be possible to make a similar queuing change to
> >>>SCO/eSCO/LE, but not strictly necessary.
> >>
> >>Would you please clarify this approach (perhaps in a separate thread)?
> >>
> >>For example, how does having tx queues in l2cap_chan (instead of the
> >>hci_conn) solve the latency problems in ERTM when replying to
> >>REJ/SREJ/poll? Won't there potentially be just as much data already
> >>queued up? Is the plan to move the reply to the front of the tx queue
> >>because reqseq won't need to be assigned until the frame is actually
> >>pulled off the queue?
> >
> >Exactly. ERTM connections can get dropped if the too much data is buffered and
> >we need to send final bit for example.
> 
> Right now, an outgoing ERTM frame goes through two queues: a
> channel-specific ERTM tx queue and the HCI ACL data_q.  The ERTM
> control field is not constructed until a frame is removed from the
> ERTM tx queue and pushed to the HCI data_q, so the s-frame latency
> problem comes in when the the HCI data_q gets deep.  S-frames are
> already pushed directly in to the HCI data_q, bypassing the data tx
> queue.
> 
> From an ERTM perspective, the goal is to defer assignment of reqseq
> and f-bit values as late as possible, so the remote device gets the
> most recent information on data frames and polls that have been
> received.  The optimal thing to do (by this measurement, anyway) is
> to build the ERTM control field as data is sent to the baseband --
> in other words, to eliminate the HCI data_q altogether.
> 
> (Yeah, without the data_q, ERTM would need additional queues for
> s-frames and retransmitted i-frames)
> 
> So, without a data_q, what makes sense?  If there are ACL buffers
> available and no pending L2CAP senders, it would be great to push
> data straight out to the baseband.  If we're blocked waiting for
> num_completed_packets, then receipt of num_completed_packets is the
> natural time to pull data from the tx queues that now happen to be
> up in the L2CAP layer.
> 
> There are certainly locking, task scheduling, data scheduling, QoS,
> and efficiency issues to consider.  This is just a general
> description for now, and I'm trying to see if there's enough
> interest (or few enough obvious gotchas) to put some serious effort
> in to moving forward.

Getting rid of conn->data_q makes sense. I started a patch to create the
struct hci_chan that Luiz proposed. It would be one HCI channel per L2CAP
connection. The buffer (acl_cnt) would be now divided by the number of
channels and not the number of connections. This is a first step to support
QoS and priority inside ERTM. QoS then would just need new scheduler rules.

+struct hci_chan {
+       struct list_head list;
+       struct hci_conn *conn;
+       struct sk_buff_head data_q;
+       unsigned int    sent;
+}

So in the next step for ERTM we move the queue to L2CAP and create a callback
to call from HCI at the moment of push data to the baseband. The function in
L2CAP would set the last control bits in the first packet of the queue and
sent it through.

Then the queue can be split in two by adding a pointer that will mark which
element divides the queue between prio and normal. New prio skbs would just be
queued after this element and before the rest.

I still need to think on locking here. (and also finish my patches that move
all the bluetooth to workqueue)

	Gustavo

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Mat Martineau]

On Tue, 9 Aug 2011, Gustavo Padovan wrote:

> * Mat Martineau <mathewm@codeaurora.org> [2011-08-08 16:29:51 -0700]:
>
>>
>> On Fri, 5 Aug 2011, Gustavo Padovan wrote:
>>
>>> * Peter Hurley <peter@hurleysoftware.com> [2011-08-04 19:09:37 -0400]:
>>>
>>>> Hi Mat,
>>>>
>>>> On Thu, 2011-08-04 at 13:37 -0400, Mat Martineau wrote:
>>>>
>>>>> I had a recent discussion with Gustavo about HCI queuing issues with
>>>>> ERTM:
>>>>>
>>>>> http://www.spinics.net/lists/linux-bluetooth/msg13774.html
>>>>>
>>>>> My proposal is to move tx queuing up to L2CAP, and have the HCI tx
>>>>> task only handle scheduling.  Senders would tell HCI they have data to
>>>>> send, and HCI would call back to pull data.  I've been focused on
>>>>> L2CAP - it would be possible to make a similar queuing change to
>>>>> SCO/eSCO/LE, but not strictly necessary.
>>>>
>>>> Would you please clarify this approach (perhaps in a separate thread)?
>>>>
>>>> For example, how does having tx queues in l2cap_chan (instead of the
>>>> hci_conn) solve the latency problems in ERTM when replying to
>>>> REJ/SREJ/poll? Won't there potentially be just as much data already
>>>> queued up? Is the plan to move the reply to the front of the tx queue
>>>> because reqseq won't need to be assigned until the frame is actually
>>>> pulled off the queue?
>>>
>>> Exactly. ERTM connections can get dropped if the too much data is buffered and
>>> we need to send final bit for example.
>>
>> Right now, an outgoing ERTM frame goes through two queues: a
>> channel-specific ERTM tx queue and the HCI ACL data_q.  The ERTM
>> control field is not constructed until a frame is removed from the
>> ERTM tx queue and pushed to the HCI data_q, so the s-frame latency
>> problem comes in when the the HCI data_q gets deep.  S-frames are
>> already pushed directly in to the HCI data_q, bypassing the data tx
>> queue.
>>
>> From an ERTM perspective, the goal is to defer assignment of reqseq
>> and f-bit values as late as possible, so the remote device gets the
>> most recent information on data frames and polls that have been
>> received.  The optimal thing to do (by this measurement, anyway) is
>> to build the ERTM control field as data is sent to the baseband --
>> in other words, to eliminate the HCI data_q altogether.
>>
>> (Yeah, without the data_q, ERTM would need additional queues for
>> s-frames and retransmitted i-frames)
>>
>> So, without a data_q, what makes sense?  If there are ACL buffers
>> available and no pending L2CAP senders, it would be great to push
>> data straight out to the baseband.  If we're blocked waiting for
>> num_completed_packets, then receipt of num_completed_packets is the
>> natural time to pull data from the tx queues that now happen to be
>> up in the L2CAP layer.
>>
>> There are certainly locking, task scheduling, data scheduling, QoS,
>> and efficiency issues to consider.  This is just a general
>> description for now, and I'm trying to see if there's enough
>> interest (or few enough obvious gotchas) to put some serious effort
>> in to moving forward.
>
> Getting rid of conn->data_q makes sense. I started a patch to create the
> struct hci_chan that Luiz proposed. It would be one HCI channel per L2CAP
> connection. The buffer (acl_cnt) would be now divided by the number of
> channels and not the number of connections. This is a first step to support
> QoS and priority inside ERTM. QoS then would just need new scheduler rules.
>
> +struct hci_chan {
> +       struct list_head list;
> +       struct hci_conn *conn;
> +       struct sk_buff_head data_q;
> +       unsigned int    sent;
> +}

At the BlueZ summit last year, the group settled on using an "HCI 
channel" as an abstraction for AMP logical links.  We have a hci_chan 
struct already that you could add to.  Before making changes to HCI 
data structures, could we first work on upstreaming the AMP HCI 
changes (which also include some QoS-related code)?


> So in the next step for ERTM we move the queue to L2CAP and create a callback
> to call from HCI at the moment of push data to the baseband. The function in
> L2CAP would set the last control bits in the first packet of the queue and
> sent it through.

This actually causes some problems for ERTM, since skbs are cloned 
before they are pushed to HCI.  skb data is not supposed to be 
modified after cloning.

If there's a callback to L2CAP anyway, why not have L2CAP provide the 
skb at that time instead of modifying data it provided earlier?


> Then the queue can be split in two by adding a pointer that will mark which
> element divides the queue between prio and normal. New prio skbs would just be
> queued after this element and before the rest.

I think it's simpler and less bug-prone to just have two queues. 
Either way, it's one more pointer.

However, I'm still not sure we want any queues in hci_chan.  It's not 
very complicated to have the queue in the L2CAP layer, and gives ERTM 
the control it needs.


> I still need to think on locking here. (and also finish my patches 
> that move all the bluetooth to workqueue)

Keep in mind that skb_queue() and skb_dequeue() have their own 
locking. Non-ERTM modes shouldn't need locking when HCI calls back for 
skbs.

For ERTM, we need to figure out a good way to protect ERTM state (like 
buffer_seq and next_tx_seq) without using the socket lock.

--
Mat Martineau
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Luiz Augusto von Dentz]

Hi Mat,

On Wed, Aug 10, 2011 at 8:38 PM, Mat Martineau <mathewm@codeaurora.org> wrote:
>> +struct hci_chan {
>> +       struct list_head list;
>> +       struct hci_conn *conn;
>> +       struct sk_buff_head data_q;
>> +       unsigned int    sent;
>> +}
>
> At the BlueZ summit last year, the group settled on using an "HCI channel"
> as an abstraction for AMP logical links.  We have a hci_chan struct already
> that you could add to.  Before making changes to HCI data structures, could
> we first work on upstreaming the AMP HCI changes (which also include some
> QoS-related code)?

If just you told us this before I would have not spend time
implementing it, so could you please sent this patches ASAP, otherwise
I will just send mine that just address the prioritization

Btw, the thread intent is perhaps becoming too broad,  we should focus
in small steps lets say first HCI Channel, then we continue from
there.

-- 
Luiz Augusto von Dentz

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Mat Martineau]

Hi Luiz -

On Wed, 10 Aug 2011, Luiz Augusto von Dentz wrote:

> Hi Mat,
>
> On Wed, Aug 10, 2011 at 8:38 PM, Mat Martineau <mathewm@codeaurora.org> wrote:
>>> +struct hci_chan {
>>> +       struct list_head list;
>>> +       struct hci_conn *conn;
>>> +       struct sk_buff_head data_q;
>>> +       unsigned int    sent;
>>> +}
>>
>> At the BlueZ summit last year, the group settled on using an "HCI channel"
>> as an abstraction for AMP logical links.  We have a hci_chan struct already
>> that you could add to.  Before making changes to HCI data structures, could
>> we first work on upstreaming the AMP HCI changes (which also include some
>> QoS-related code)?
>
> If just you told us this before I would have not spend time
> implementing it, so could you please sent this patches ASAP, otherwise
> I will just send mine that just address the prioritization

Your RFC patches for prioritized HCI queues are not in conflict with 
the AMP code.  It's just the proposed hci_chan structure that came up 
in this email thread.

My recent patches are the beginning of AMP upstreaming.  I was trying 
to get the generic L2CAP improvements in place first.  One of my 
coworkers is starting to work in parallel on upstreaming the HCI 
changes, but we need to create a suitable patch series for 
bluetooth-next.


> Btw, the thread intent is perhaps becoming too broad,  we should focus
> in small steps lets say first HCI Channel, then we continue from
> there.

I agree, it's hard to keep track of the various sub-threads at this 
point.  The HCI channel is a good place to start.


--
Mat Martineau
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Peter Hurley]

T24gV2VkLCAyMDExLTA4LTEwIGF0IDEzOjM4IC0wNDAwLCBNYXQgTWFydGluZWF1IHdyb3RlOg0K
PiANCj4gT24gVHVlLCA5IEF1ZyAyMDExLCBHdXN0YXZvIFBhZG92YW4gd3JvdGU6DQo+IA0KPiA+
ICogTWF0IE1hcnRpbmVhdSA8bWF0aGV3bUBjb2RlYXVyb3JhLm9yZz4gWzIwMTEtMDgtMDggMTY6
Mjk6NTEgLTA3MDBdOg0KPiA+DQo+ID4+DQo+ID4+IE9uIEZyaSwgNSBBdWcgMjAxMSwgR3VzdGF2
byBQYWRvdmFuIHdyb3RlOg0KPiA+Pg0KPiA+Pj4gKiBQZXRlciBIdXJsZXkgPHBldGVyQGh1cmxl
eXNvZnR3YXJlLmNvbT4gWzIwMTEtMDgtMDQgMTk6MDk6MzcgLTA0MDBdOg0KPiA+Pj4NCj4gPj4+
PiBIaSBNYXQsDQo+ID4+Pj4NCj4gPj4+PiBPbiBUaHUsIDIwMTEtMDgtMDQgYXQgMTM6MzcgLTA0
MDAsIE1hdCBNYXJ0aW5lYXUgd3JvdGU6DQo+ID4+Pj4NCj4gPj4+Pj4gSSBoYWQgYSByZWNlbnQg
ZGlzY3Vzc2lvbiB3aXRoIEd1c3Rhdm8gYWJvdXQgSENJIHF1ZXVpbmcgaXNzdWVzIHdpdGgNCj4g
Pj4+Pj4gRVJUTToNCj4gPj4+Pj4NCj4gPj4+Pj4gaHR0cDovL3d3dy5zcGluaWNzLm5ldC9saXN0
cy9saW51eC1ibHVldG9vdGgvbXNnMTM3NzQuaHRtbA0KPiA+Pj4+Pg0KPiA+Pj4+PiBNeSBwcm9w
b3NhbCBpcyB0byBtb3ZlIHR4IHF1ZXVpbmcgdXAgdG8gTDJDQVAsIGFuZCBoYXZlIHRoZSBIQ0kg
dHgNCj4gPj4+Pj4gdGFzayBvbmx5IGhhbmRsZSBzY2hlZHVsaW5nLiAgU2VuZGVycyB3b3VsZCB0
ZWxsIEhDSSB0aGV5IGhhdmUgZGF0YSB0bw0KPiA+Pj4+PiBzZW5kLCBhbmQgSENJIHdvdWxkIGNh
bGwgYmFjayB0byBwdWxsIGRhdGEuICBJJ3ZlIGJlZW4gZm9jdXNlZCBvbg0KPiA+Pj4+PiBMMkNB
UCAtIGl0IHdvdWxkIGJlIHBvc3NpYmxlIHRvIG1ha2UgYSBzaW1pbGFyIHF1ZXVpbmcgY2hhbmdl
IHRvDQo+ID4+Pj4+IFNDTy9lU0NPL0xFLCBidXQgbm90IHN0cmljdGx5IG5lY2Vzc2FyeS4NCj4g
Pj4+Pg0KPiA+Pj4+IFdvdWxkIHlvdSBwbGVhc2UgY2xhcmlmeSB0aGlzIGFwcHJvYWNoIChwZXJo
YXBzIGluIGEgc2VwYXJhdGUgdGhyZWFkKT8NCj4gPj4+Pg0KPiA+Pj4+IEZvciBleGFtcGxlLCBo
b3cgZG9lcyBoYXZpbmcgdHggcXVldWVzIGluIGwyY2FwX2NoYW4gKGluc3RlYWQgb2YgdGhlDQo+
ID4+Pj4gaGNpX2Nvbm4pIHNvbHZlIHRoZSBsYXRlbmN5IHByb2JsZW1zIGluIEVSVE0gd2hlbiBy
ZXBseWluZyB0bw0KPiA+Pj4+IFJFSi9TUkVKL3BvbGw/IFdvbid0IHRoZXJlIHBvdGVudGlhbGx5
IGJlIGp1c3QgYXMgbXVjaCBkYXRhIGFscmVhZHkNCj4gPj4+PiBxdWV1ZWQgdXA/IElzIHRoZSBw
bGFuIHRvIG1vdmUgdGhlIHJlcGx5IHRvIHRoZSBmcm9udCBvZiB0aGUgdHggcXVldWUNCj4gPj4+
PiBiZWNhdXNlIHJlcXNlcSB3b24ndCBuZWVkIHRvIGJlIGFzc2lnbmVkIHVudGlsIHRoZSBmcmFt
ZSBpcyBhY3R1YWxseQ0KPiA+Pj4+IHB1bGxlZCBvZmYgdGhlIHF1ZXVlPw0KPiA+Pj4NCj4gPj4+
IEV4YWN0bHkuIEVSVE0gY29ubmVjdGlvbnMgY2FuIGdldCBkcm9wcGVkIGlmIHRoZSB0b28gbXVj
aCBkYXRhIGlzIGJ1ZmZlcmVkIGFuZA0KPiA+Pj4gd2UgbmVlZCB0byBzZW5kIGZpbmFsIGJpdCBm
b3IgZXhhbXBsZS4NCg0KSGkgTWF0LA0KDQpUaGFua3MgZm9yIHRha2luZyB0aGUgdGltZSB0byBj
bGFyaWZ5IHdoYXQgeW91IGFyZSBwcm9wb3NpbmcuDQoNCj4gPj4gUmlnaHQgbm93LCBhbiBvdXRn
b2luZyBFUlRNIGZyYW1lIGdvZXMgdGhyb3VnaCB0d28gcXVldWVzOiBhDQo+ID4+IGNoYW5uZWwt
c3BlY2lmaWMgRVJUTSB0eCBxdWV1ZSBhbmQgdGhlIEhDSSBBQ0wgZGF0YV9xLiAgVGhlIEVSVE0N
Cj4gPj4gY29udHJvbCBmaWVsZCBpcyBub3QgY29uc3RydWN0ZWQgdW50aWwgYSBmcmFtZSBpcyBy
ZW1vdmVkIGZyb20gdGhlDQo+ID4+IEVSVE0gdHggcXVldWUgYW5kIHB1c2hlZCB0byB0aGUgSENJ
IGRhdGFfcSwgc28gdGhlIHMtZnJhbWUgbGF0ZW5jeQ0KPiA+PiBwcm9ibGVtIGNvbWVzIGluIHdo
ZW4gdGhlIHRoZSBIQ0kgZGF0YV9xIGdldHMgZGVlcC4gIFMtZnJhbWVzIGFyZQ0KPiA+PiBhbHJl
YWR5IHB1c2hlZCBkaXJlY3RseSBpbiB0byB0aGUgSENJIGRhdGFfcSwgYnlwYXNzaW5nIHRoZSBk
YXRhIHR4DQo+ID4+IHF1ZXVlLg0KPiA+Pg0KPiA+PiBGcm9tIGFuIEVSVE0gcGVyc3BlY3RpdmUs
IHRoZSBnb2FsIGlzIHRvIGRlZmVyIGFzc2lnbm1lbnQgb2YgcmVxc2VxDQo+ID4+IGFuZCBmLWJp
dCB2YWx1ZXMgYXMgbGF0ZSBhcyBwb3NzaWJsZSwgc28gdGhlIHJlbW90ZSBkZXZpY2UgZ2V0cyB0
aGUNCj4gPj4gbW9zdCByZWNlbnQgaW5mb3JtYXRpb24gb24gZGF0YSBmcmFtZXMgYW5kIHBvbGxz
IHRoYXQgaGF2ZSBiZWVuDQo+ID4+IHJlY2VpdmVkLg0KDQpUaGF0J3Mgd2hhdCBJIHRob3VnaHQg
LSBqdXN0IHdhbnRlZCB0byBtYWtlIHN1cmUgdGhhdCdzIHdoYXQgeW91IG1lYW50Lg0KDQo+ID4+
ICAgICAgICAgICAgVGhlIG9wdGltYWwgdGhpbmcgdG8gZG8gKGJ5IHRoaXMgbWVhc3VyZW1lbnQs
IGFueXdheSkgaXMNCj4gPj4gdG8gYnVpbGQgdGhlIEVSVE0gY29udHJvbCBmaWVsZCBhcyBkYXRh
IGlzIHNlbnQgdG8gdGhlIGJhc2ViYW5kIC0tDQo+ID4+IGluIG90aGVyIHdvcmRzLCB0byBlbGlt
aW5hdGUgdGhlIEhDSSBkYXRhX3EgYWx0b2dldGhlci4NCj4gPj4NCj4gPj4gKFllYWgsIHdpdGhv
dXQgdGhlIGRhdGFfcSwgRVJUTSB3b3VsZCBuZWVkIGFkZGl0aW9uYWwgcXVldWVzIGZvcg0KPiA+
PiBzLWZyYW1lcyBhbmQgcmV0cmFuc21pdHRlZCBpLWZyYW1lcykNCj4gPj4NCj4gPj4gU28sIHdp
dGhvdXQgYSBkYXRhX3EsIHdoYXQgbWFrZXMgc2Vuc2U/ICBJZiB0aGVyZSBhcmUgQUNMIGJ1ZmZl
cnMNCj4gPj4gYXZhaWxhYmxlIGFuZCBubyBwZW5kaW5nIEwyQ0FQIHNlbmRlcnMsIGl0IHdvdWxk
IGJlIGdyZWF0IHRvIHB1c2gNCj4gPj4gZGF0YSBzdHJhaWdodCBvdXQgdG8gdGhlIGJhc2ViYW5k
LiAgSWYgd2UncmUgYmxvY2tlZCB3YWl0aW5nIGZvcg0KPiA+PiBudW1fY29tcGxldGVkX3BhY2tl
dHMsIHRoZW4gcmVjZWlwdCBvZiBudW1fY29tcGxldGVkX3BhY2tldHMgaXMgdGhlDQo+ID4+IG5h
dHVyYWwgdGltZSB0byBwdWxsIGRhdGEgZnJvbSB0aGUgdHggcXVldWVzIHRoYXQgbm93IGhhcHBl
biB0byBiZQ0KPiA+PiB1cCBpbiB0aGUgTDJDQVAgbGF5ZXIuDQo+ID4+DQo+ID4+IFRoZXJlIGFy
ZSBjZXJ0YWlubHkgbG9ja2luZywgdGFzayBzY2hlZHVsaW5nLCBkYXRhIHNjaGVkdWxpbmcsIFFv
UywNCj4gPj4gYW5kIGVmZmljaWVuY3kgaXNzdWVzIHRvIGNvbnNpZGVyLiAgVGhpcyBpcyBqdXN0
IGEgZ2VuZXJhbA0KPiA+PiBkZXNjcmlwdGlvbiBmb3Igbm93LCBhbmQgSSdtIHRyeWluZyB0byBz
ZWUgaWYgdGhlcmUncyBlbm91Z2gNCj4gPj4gaW50ZXJlc3QgKG9yIGZldyBlbm91Z2ggb2J2aW91
cyBnb3RjaGFzKSB0byBwdXQgc29tZSBzZXJpb3VzIGVmZm9ydA0KPiA+PiBpbiB0byBtb3Zpbmcg
Zm9yd2FyZC4NCg0KV2VsbCwgSSB0aGluayB0aGUgaW50ZXJlc3QgaXMgdGhlcmUgYnV0ICJ0aGUg
ZGV2aWwgaXMgaW4gdGhlIGRldGFpbHMiLg0KSXQncyBqdXN0IG15IG9waW5pb24gYnV0IEkgdGhp
bmsgJ2hhc2hpbmcgb3V0JyB0aGUgZGF0YSBmbG93IHdpbGwgaGVscA0KZXhwb3NlIGNvbnRlbnRp
b25zICh3aGljaCBJIHRoaW5rIGlzIGdvaW5nIHRvIGJlIHRoZSBtYWluIHByb2JsZW0pLiBTZWUN
Cm15IGNvbW1lbnQgYmVsb3cgcmU6IHNrYl9xdWV1ZSAmIHNrYl9kZXF1ZXVlLg0KDQo+ID4gR2V0
dGluZyByaWQgb2YgY29ubi0+ZGF0YV9xIG1ha2VzIHNlbnNlLiBJIHN0YXJ0ZWQgYSBwYXRjaCB0
byBjcmVhdGUgdGhlDQo+ID4gc3RydWN0IGhjaV9jaGFuIHRoYXQgTHVpeiBwcm9wb3NlZC4gSXQg
d291bGQgYmUgb25lIEhDSSBjaGFubmVsIHBlciBMMkNBUA0KPiA+IGNvbm5lY3Rpb24uIFRoZSBi
dWZmZXIgKGFjbF9jbnQpIHdvdWxkIGJlIG5vdyBkaXZpZGVkIGJ5IHRoZSBudW1iZXIgb2YNCj4g
PiBjaGFubmVscyBhbmQgbm90IHRoZSBudW1iZXIgb2YgY29ubmVjdGlvbnMuIFRoaXMgaXMgYSBm
aXJzdCBzdGVwIHRvIHN1cHBvcnQNCj4gPiBRb1MgYW5kIHByaW9yaXR5IGluc2lkZSBFUlRNLiBR
b1MgdGhlbiB3b3VsZCBqdXN0IG5lZWQgbmV3IHNjaGVkdWxlciBydWxlcy4NCj4gPg0KPiA+ICtz
dHJ1Y3QgaGNpX2NoYW4gew0KPiA+ICsgICAgICAgc3RydWN0IGxpc3RfaGVhZCBsaXN0Ow0KPiA+
ICsgICAgICAgc3RydWN0IGhjaV9jb25uICpjb25uOw0KPiA+ICsgICAgICAgc3RydWN0IHNrX2J1
ZmZfaGVhZCBkYXRhX3E7DQo+ID4gKyAgICAgICB1bnNpZ25lZCBpbnQgICAgc2VudDsNCj4gPiAr
fQ0KPiANCj4gQXQgdGhlIEJsdWVaIHN1bW1pdCBsYXN0IHllYXIsIHRoZSBncm91cCBzZXR0bGVk
IG9uIHVzaW5nIGFuICJIQ0kgDQo+IGNoYW5uZWwiIGFzIGFuIGFic3RyYWN0aW9uIGZvciBBTVAg
bG9naWNhbCBsaW5rcy4gIFdlIGhhdmUgYSBoY2lfY2hhbiANCj4gc3RydWN0IGFscmVhZHkgdGhh
dCB5b3UgY291bGQgYWRkIHRvLiAgQmVmb3JlIG1ha2luZyBjaGFuZ2VzIHRvIEhDSSANCj4gZGF0
YSBzdHJ1Y3R1cmVzLCBjb3VsZCB3ZSBmaXJzdCB3b3JrIG9uIHVwc3RyZWFtaW5nIHRoZSBBTVAg
SENJIA0KPiBjaGFuZ2VzICh3aGljaCBhbHNvIGluY2x1ZGUgc29tZSBRb1MtcmVsYXRlZCBjb2Rl
KT8NCj4gDQo+IA0KPiA+IFNvIGluIHRoZSBuZXh0IHN0ZXAgZm9yIEVSVE0gd2UgbW92ZSB0aGUg
cXVldWUgdG8gTDJDQVAgYW5kIGNyZWF0ZSBhIGNhbGxiYWNrDQo+ID4gdG8gY2FsbCBmcm9tIEhD
SSBhdCB0aGUgbW9tZW50IG9mIHB1c2ggZGF0YSB0byB0aGUgYmFzZWJhbmQuIFRoZSBmdW5jdGlv
biBpbg0KPiA+IEwyQ0FQIHdvdWxkIHNldCB0aGUgbGFzdCBjb250cm9sIGJpdHMgaW4gdGhlIGZp
cnN0IHBhY2tldCBvZiB0aGUgcXVldWUgYW5kDQo+ID4gc2VudCBpdCB0aHJvdWdoLg0KPiANCj4g
VGhpcyBhY3R1YWxseSBjYXVzZXMgc29tZSBwcm9ibGVtcyBmb3IgRVJUTSwgc2luY2Ugc2ticyBh
cmUgY2xvbmVkIA0KPiBiZWZvcmUgdGhleSBhcmUgcHVzaGVkIHRvIEhDSS4gIHNrYiBkYXRhIGlz
IG5vdCBzdXBwb3NlZCB0byBiZSANCj4gbW9kaWZpZWQgYWZ0ZXIgY2xvbmluZy4NCg0KbDJjYXAg
aXMgYWxyZWFkeSBkb2luZyB0aGlzIChtb2RpZnlpbmcgYWZ0ZXIgY2xvbmluZykgLSBlZy4sIHdo
ZW4NCnJldHJhbnNtaXR0aW5nLiBUaGUgYWx0ZXJuYXRpdmUgYmVpbmcgdG8gbGVhdmUgdGhlIHNr
Yl9wdXNoL2hlYWRlciBhZGQNCnVudGlsIHNlbmRpbmcgdG8gdGhlIHRyYW5zcG9ydCBkcml2ZXI/
DQoNCj4gSWYgdGhlcmUncyBhIGNhbGxiYWNrIHRvIEwyQ0FQIGFueXdheSwgd2h5IG5vdCBoYXZl
IEwyQ0FQIHByb3ZpZGUgdGhlIA0KPiBza2IgYXQgdGhhdCB0aW1lIGluc3RlYWQgb2YgbW9kaWZ5
aW5nIGRhdGEgaXQgcHJvdmlkZWQgZWFybGllcj8NCj4gDQo+IA0KPiA+IFRoZW4gdGhlIHF1ZXVl
IGNhbiBiZSBzcGxpdCBpbiB0d28gYnkgYWRkaW5nIGEgcG9pbnRlciB0aGF0IHdpbGwgbWFyayB3
aGljaA0KPiA+IGVsZW1lbnQgZGl2aWRlcyB0aGUgcXVldWUgYmV0d2VlbiBwcmlvIGFuZCBub3Jt
YWwuIE5ldyBwcmlvIHNrYnMgd291bGQganVzdCBiZQ0KPiA+IHF1ZXVlZCBhZnRlciB0aGlzIGVs
ZW1lbnQgYW5kIGJlZm9yZSB0aGUgcmVzdC4NCj4gDQo+IEkgdGhpbmsgaXQncyBzaW1wbGVyIGFu
ZCBsZXNzIGJ1Zy1wcm9uZSB0byBqdXN0IGhhdmUgdHdvIHF1ZXVlcy4gDQo+IEVpdGhlciB3YXks
IGl0J3Mgb25lIG1vcmUgcG9pbnRlci4NCj4gDQo+IEhvd2V2ZXIsIEknbSBzdGlsbCBub3Qgc3Vy
ZSB3ZSB3YW50IGFueSBxdWV1ZXMgaW4gaGNpX2NoYW4uICBJdCdzIG5vdCANCj4gdmVyeSBjb21w
bGljYXRlZCB0byBoYXZlIHRoZSBxdWV1ZSBpbiB0aGUgTDJDQVAgbGF5ZXIsIGFuZCBnaXZlcyBF
UlRNIA0KPiB0aGUgY29udHJvbCBpdCBuZWVkcy4NCg0KVGhlIG9ubHkgb2JqZWN0aW9uIGhlcmUg
aXMgdGhhdCB0aGlzIGRlc2lnbiBhcHByb2FjaCBsZWFkcyB0byBhY2NyZXRpb24NCm92ZXIgdGlt
ZSwgYW5kIGluIHRoZSBtb3N0IGRlbGljYXRlIGFyZWEgcG9zc2libGUuDQoNCj4gPiBJIHN0aWxs
IG5lZWQgdG8gdGhpbmsgb24gbG9ja2luZyBoZXJlLiAoYW5kIGFsc28gZmluaXNoIG15IHBhdGNo
ZXMgDQo+ID4gdGhhdCBtb3ZlIGFsbCB0aGUgYmx1ZXRvb3RoIHRvIHdvcmtxdWV1ZSkNCj4gDQo+
IEtlZXAgaW4gbWluZCB0aGF0IHNrYl9xdWV1ZSgpIGFuZCBza2JfZGVxdWV1ZSgpIGhhdmUgdGhl
aXIgb3duIA0KPiBsb2NraW5nLiBOb24tRVJUTSBtb2RlcyBzaG91bGRuJ3QgbmVlZCBsb2NraW5n
IHdoZW4gSENJIGNhbGxzIGJhY2sgZm9yIA0KPiBza2JzLg0KDQpFeGFjdGx5IC0tIGJ1dCByaWdo
dCBub3cgdGhlIEVSVE0gdHhfcSBpcyBhIHByaXZhdGUgcXVldWUuIFdoZW4gdGhpcw0KYmVjb21l
cyBzaGFyZWQsIHdlIG5lZWQgdG8gaGF2ZSBhbHJlYWR5IHdvcmtlZCBvdXQgaG93IHRvIGF2b2lk
IHRoZQ0Kc2l0dWF0aW9uIHdoZXJlIHRoZSBzY2hlZHVsZXIgaXMgaGVsZCB1cCB3YWl0aW5nIGZv
ciBhbiBFUlRNIGNoYW5uZWwgdG8NCmRyb3BwZWQgaXRzIGFja2VkIGZyYW1lcywgZm9yIGV4YW1w
bGUuDQoNCj4gRm9yIEVSVE0sIHdlIG5lZWQgdG8gZmlndXJlIG91dCBhIGdvb2Qgd2F5IHRvIHBy
b3RlY3QgRVJUTSBzdGF0ZSAobGlrZSANCj4gYnVmZmVyX3NlcSBhbmQgbmV4dF90eF9zZXEpIHdp
dGhvdXQgdXNpbmcgdGhlIHNvY2tldCBsb2NrLg0KDQpBbmQgdGhpcy4NCg0KVGhhbmtzIGFnYWlu
IGFuZCByZWdhcmRzLA0KUGV0ZXIgSHVybGV5DQoNCg==

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Marcel Holtmann]

Hi Mat,

<snip>

> > So in the next step for ERTM we move the queue to L2CAP and create a callback
> > to call from HCI at the moment of push data to the baseband. The function in
> > L2CAP would set the last control bits in the first packet of the queue and
> > sent it through.
> 
> This actually causes some problems for ERTM, since skbs are cloned 
> before they are pushed to HCI.  skb data is not supposed to be 
> modified after cloning.
> 
> If there's a callback to L2CAP anyway, why not have L2CAP provide the 
> skb at that time instead of modifying data it provided earlier?
> 
> 
> > Then the queue can be split in two by adding a pointer that will mark which
> > element divides the queue between prio and normal. New prio skbs would just be
> > queued after this element and before the rest.
> 
> I think it's simpler and less bug-prone to just have two queues. 
> Either way, it's one more pointer.
> 
> However, I'm still not sure we want any queues in hci_chan.  It's not 
> very complicated to have the queue in the L2CAP layer, and gives ERTM 
> the control it needs.

I am fine if we just queue in L2CAP actually. Since nobody should send
ACL packets directly (besides the crazy people that wanna abuse BlueZ's
support for HCI_RAW for 3rd party stacks) anyway.

In a real fully integrated stack only L2CAP will send ACL data packets
and we can just rely on that fact if want to.

Regards

Marcel

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Luiz Augusto von Dentz]

Hi Mat,

On Thu, Aug 4, 2011 at 8:37 PM, Mat Martineau <mathewm@codeaurora.org> wrote:
> One concern I have is that an application that changes sk_priority while it
> has data in the HCI tx queue could have its frames delivered out of order.
>  While we can't control when sk_priority is set, it is possible to make a
> choice when setting the priority of each skb.  Does it make sense to detect
> an sk_priority change and only apply the new value when there are no skbs
> for the channel in the HCI tx queue?

Yep, if we need to maintain the order I guess this would make sense,
the problem is that this could be a bit expensive.

> I had a recent discussion with Gustavo about HCI queuing issues with ERTM:
>
> http://www.spinics.net/lists/linux-bluetooth/msg13774.html
>
> My proposal is to move tx queuing up to L2CAP, and have the HCI tx task only
> handle scheduling.  Senders would tell HCI they have data to send, and HCI
> would call back to pull data.  I've been focused on L2CAP - it would be
> possible to make a similar queuing change to SCO/eSCO/LE, but not strictly
> necessary.

> This is different from the problem you're solving, but as long as we're
> talking about big changes to HCI tx, I think we could make some changes that
> help out with prioritization and queuing across all channels (whether
> they're going to the same remote device or different remote devices).
>
> It could be more efficient to schedule this way since HCI wouldn't have to
> iterate over all of the connections and queues - it would only have to
> review the prioritized list of pending sends. Might be a more fitting
> structure for QoS too.
>
> Since HCI wouldn't have a bunch of queued-up skbs, it would also eliminate
> the sk_priority change problem I noted above.
>
> Your thoughts?

If the idea is to have a list of chan/sockets and each having its own
queue, I guess that would make sense and we could possible preserve
the packet order in that case, but we still have to maintain the
fairness when dealing with the same priority so I would still have to
iterate to each connection and then to each channel evaluating their
priority. This way we don't have to defined any arbitrary number of
queues which is probably less items to visit on tx task, but the
locking may be a lot more complicated.

-- 
Luiz Augusto von Dentz

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Gustavo Padovan]

* Luiz Augusto von Dentz <luiz.dentz@gmail.com> [2011-08-05 09:09:38 +0300]:

> Hi Mat,
> 
> On Thu, Aug 4, 2011 at 8:37 PM, Mat Martineau <mathewm@codeaurora.org> wrote:
> > One concern I have is that an application that changes sk_priority while it
> > has data in the HCI tx queue could have its frames delivered out of order.
> >  While we can't control when sk_priority is set, it is possible to make a
> > choice when setting the priority of each skb.  Does it make sense to detect
> > an sk_priority change and only apply the new value when there are no skbs
> > for the channel in the HCI tx queue?
> 
> Yep, if we need to maintain the order I guess this would make sense,
> the problem is that this could be a bit expensive.
> 
> > I had a recent discussion with Gustavo about HCI queuing issues with ERTM:
> >
> > http://www.spinics.net/lists/linux-bluetooth/msg13774.html
> >
> > My proposal is to move tx queuing up to L2CAP, and have the HCI tx task only
> > handle scheduling.  Senders would tell HCI they have data to send, and HCI
> > would call back to pull data.  I've been focused on L2CAP - it would be
> > possible to make a similar queuing change to SCO/eSCO/LE, but not strictly
> > necessary.
> 
> > This is different from the problem you're solving, but as long as we're
> > talking about big changes to HCI tx, I think we could make some changes that
> > help out with prioritization and queuing across all channels (whether
> > they're going to the same remote device or different remote devices).
> >
> > It could be more efficient to schedule this way since HCI wouldn't have to
> > iterate over all of the connections and queues - it would only have to
> > review the prioritized list of pending sends. Might be a more fitting
> > structure for QoS too.
> >
> > Since HCI wouldn't have a bunch of queued-up skbs, it would also eliminate
> > the sk_priority change problem I noted above.
> >
> > Your thoughts?
> 
> If the idea is to have a list of chan/sockets and each having its own
> queue, I guess that would make sense and we could possible preserve
> the packet order in that case, but we still have to maintain the
> fairness when dealing with the same priority so I would still have to
> iterate to each connection and then to each channel evaluating their
> priority. This way we don't have to defined any arbitrary number of
> queues which is probably less items to visit on tx task, but the
> locking may be a lot more complicated.

Only ERTM needs to have its own queue, Basic and Streaming mode doesn't need
to change, they can use the same queue they are using now.

	Gustavo

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Luiz Augusto von Dentz]

Hi Gustavo,

On Fri, Aug 5, 2011 at 10:14 PM, Gustavo Padovan <padovan@profusion.mobi> wrote:
>
> Only ERTM needs to have its own queue, Basic and Streaming mode doesn't need
> to change, they can use the same queue they are using now.

The guaranteed channels also seems to need separated queue, actually
even best effort channels could use QoS (see page 1442  :5.3 QUALITY
OF SERVICE (QOS) OPTION). Note this is a general option which is then
negotiated also on HCI level, it seems completely independent of ERTM.

The per channel queuing is almost a must have if we really want to be
able to implement any QoS, because then we can track exactly what
channels are sending, calculate if we can attend the QoS needs and so
on. But I don't think doing this on L2CAP level is a good idea,
because it create a circular dependency with HCI  and the pull mode
might create a locking nightmare, not to mention L2CAP is already
quite big.

One of the solution Ive been thinking is to have an HCI Channel (e.g.
struct hci_chan) abstraction, which would be responsible for queueing
and storing necessary information for scheduling. This could be
implemented directly in HCI level (hci_conn.c) so L2CAP can still push
data to HCI by using the HCI Channel (l2cap_chan->hci_chan). How about
that?

-- 
Luiz Augusto von Dentz

Re: [PATCH 0/3] RFC: prioritizing data over HCI

[Gustavo Padovan]

Hi Luiz,

* Luiz Augusto von Dentz <luiz.dentz@gmail.com> [2011-08-06 01:49:24 +0300]:

> Hi Gustavo,
> 
> On Fri, Aug 5, 2011 at 10:14 PM, Gustavo Padovan <padovan@profusion.mobi> wrote:
> >
> > Only ERTM needs to have its own queue, Basic and Streaming mode doesn't need
> > to change, they can use the same queue they are using now.
> 
> The guaranteed channels also seems to need separated queue, actually
> even best effort channels could use QoS (see page 1442  :5.3 QUALITY
> OF SERVICE (QOS) OPTION). Note this is a general option which is then
> negotiated also on HCI level, it seems completely independent of ERTM.

What I tried to say is that ERTM needs to prioritize some frames (SREJ, REJ,
poll and final bits, resends) to improve the throughput. This is independent of
the QoS of the channel. So we need to postpone the queueing into conn->data_q
until the last moment before send it to the hci_dev.

> 
> The per channel queuing is almost a must have if we really want to be
> able to implement any QoS, because then we can track exactly what
> channels are sending, calculate if we can attend the QoS needs and so
> on. But I don't think doing this on L2CAP level is a good idea,
> because it create a circular dependency with HCI  and the pull mode
> might create a locking nightmare, not to mention L2CAP is already
> quite big.
> 
> One of the solution Ive been thinking is to have an HCI Channel (e.g.
> struct hci_chan) abstraction, which would be responsible for queueing
> and storing necessary information for scheduling. This could be
> implemented directly in HCI level (hci_conn.c) so L2CAP can still push
> data to HCI by using the HCI Channel (l2cap_chan->hci_chan). How about
> that?

This doesn't work ERTM. I've been thinking on keep both the normal ERTM queue
and the prio qeue inside L2CAP and the create some callbacks, then HCI can ask
L2CAP channel to deliver the frames it have queued. ERTM would first clear it
prio queue and then the normal queue, and before send each of packets to HCI
ERTM have to set packet header properly.

You approach can work for the other L2CAP modes.

	Gustavo