huawei: modem channel gets closed after PPP disconnection

huawei: modem channel gets closed after PPP disconnection

[Christophe Ronco]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

Hi,

I use a huawei E3372 modem. It connects to GPRS using PPP. I am able to 
activate context and then deactivate it. After that, I am not able to 
reactivate the context.
During deactivation, the AT channel used by PPP gets closed and nobody 
reopens it. During reactivation, the first AT command (AT+CGDCONT) fails.
Channel shutting down occurs just after NO CARRIER is received on modem 
channel (channel is back to AT state) along with a bunch of wrong data. 
Here is the log:
Modem: < \r\nNO CARRIER\r\n~\377}#\200!}%}"} }$\210*~
Shutting down is detected in function received_data (gatio.c). Flags 
G_IO_HUP and G_IO_ERR are received.

I have done a patch in huawei plugin that correct the problem@least 
with my modem. The idea is for plugin to get notified when modem AT 
channel gets closed and when this happens to remove gprs-context, reopen 
modem AT channel and restart gprs-context driver. This is possible 
because all other drivers (sim, gprs, ...) use PCUI AT channel and not 
modem AT channel.
But I am not sure this is:
- the right place to do it (in plugin or in gprs-context driver). I 
choose plugin because there are some AT commands to send just after 
channel opening and this is a plugin job.
  - the right way to do it (removing the whole gprs-context driver might 
be too much).
Please find attached my current patch. It might not apply exactly on 
HEAD of master. At this stage, this is more to have comment on my fix 
than to really send a patch proposal.

If you need more info on this problem, I should be able to send traces 
with AT debug. It does not help a lot and I had to add traces in gatchat 
module to investigate.

Best Regards,

Christophe

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0004-huawei-reopen-modem-channel-if-it-disconnects.patch --]
[-- Type: text/x-patch, Size: 3619 bytes --]

>From f840da4307634c98d6f1a100f0012d211521ed2a Mon Sep 17 00:00:00 2001
From: Christophe Ronco <c.ronco@kerlink.fr>
Date: Wed, 15 Feb 2017 12:11:13 +0100
Subject: [PATCH] huawei: reopen modem channel if it disconnects

On E3372 after a GPRS disconnection, modem AT channel (used by PPP) gets
disconnected (G_IO_HUP and G_IO_ERR is detected in received_data in gatio.c).
After that all connection attempts fail because we are not able to send
any AT commands on modem channel (it is closed).

With this patch, when this behavior is detected, we close gprs context
driver, reopen AT channel and gprs context driver.
---
 plugins/huawei.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 47 insertions(+), 7 deletions(-)

diff --git a/plugins/huawei.c b/plugins/huawei.c
index 5dc9bc3b..4f9b3956 100644
--- a/plugins/huawei.c
+++ b/plugins/huawei.c
@@ -78,6 +78,8 @@ enum {
 struct huawei_data {
 	GAtChat *modem;
 	GAtChat *pcui;
+	struct ofono_gprs *gprs;
+	struct ofono_gprs_context *gc;
 	gboolean have_sim;
 	int sim_state;
 	guint sysinfo_poll_source;
@@ -584,6 +586,45 @@ static GAtChat *open_device(struct ofono_modem *modem,
 	return chat;
 }
 
+static void modem_disconnect(gpointer user_data)
+{
+	struct ofono_modem *modem = user_data;
+	struct huawei_data *data = ofono_modem_get_data(modem);
+
+	DBG("");
+
+	ofono_warn("modem channel disconnected");
+
+	/* clean and close modem device */
+	g_at_chat_cancel_all(data->modem);
+	g_at_chat_unregister_all(data->modem);
+	g_at_chat_unref(data->modem);
+	data->modem = NULL;
+
+	/* close gprs context driver */
+	ofono_gprs_context_remove(data->gc);
+
+	/* reopen modem channel */
+	data->modem = open_device(modem, "Modem", "Modem: ");
+
+	if (data->modem == NULL) {
+		DBG("Can't reopen device");
+		return;
+	}
+
+	/* configure modem channel */
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+	g_at_chat_set_slave(data->modem, data->pcui);
+	g_at_chat_send(data->modem, "ATE0 +CMEE=1", NULL, NULL, NULL, NULL);
+
+	/* reopen gprs context driver */
+	data->gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
+						"atmodem", data->modem);
+
+	if (data->gprs && data->gc)
+		ofono_gprs_add_context(data->gprs, data->gc);
+}
+
 static int huawei_enable(struct ofono_modem *modem)
 {
 	struct huawei_data *data = ofono_modem_get_data(modem);
@@ -594,6 +635,8 @@ static int huawei_enable(struct ofono_modem *modem)
 	if (data->modem == NULL)
 		return -EINVAL;
 
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+
 	data->pcui = open_device(modem, "Pcui", "PCUI: ");
 	if (data->pcui == NULL) {
 		g_at_chat_unref(data->modem);
@@ -820,9 +863,6 @@ static void huawei_post_sim(struct ofono_modem *modem)
 	}
 
 	if (data->have_gsm == TRUE) {
-		struct ofono_gprs *gprs;
-		struct ofono_gprs_context *gc;
-
 		ofono_phonebook_create(modem, 0, "atmodem", data->pcui);
 		ofono_radio_settings_create(modem, 0,
 						"huaweimodem", data->pcui);
@@ -830,13 +870,13 @@ static void huawei_post_sim(struct ofono_modem *modem)
 		ofono_sms_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
 
-		gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
+		data->gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
-		gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
+		data->gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->modem);
 
-		if (gprs && gc)
-			ofono_gprs_add_context(gprs, gc);
+		if (data->gprs && data->gc)
+			ofono_gprs_add_context(data->gprs, data->gc);
 	}
 }
 
-- 
2.11.0

Re: huawei: modem channel gets closed after PPP disconnection

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 2141 bytes --]

Hi Christophe,

On 02/16/2017 03:26 AM, Christophe Ronco wrote:
> Hi,
>
> I use a huawei E3372 modem. It connects to GPRS using PPP. I am able to
> activate context and then deactivate it. After that, I am not able to
> reactivate the context.
> During deactivation, the AT channel used by PPP gets closed and nobody
> reopens it. During reactivation, the first AT command (AT+CGDCONT) fails.
> Channel shutting down occurs just after NO CARRIER is received on modem
> channel (channel is back to AT state) along with a bunch of wrong data.
> Here is the log:
> Modem: < \r\nNO CARRIER\r\n~\377}#\200!}%}"} }$\210*~
> Shutting down is detected in function received_data (gatio.c). Flags
> G_IO_HUP and G_IO_ERR are received.

This is a common problem with these devices, a hold-over from olden 
serial days.  Basically the modem signals a hangup and in addition to 
sending a NO CARRIER, it also twiddles with some serial status bits. 
This causes the socket to receive a 'shutdown' event.  I think that on 
some devices the serial port is also physically removed for a short 
time, so even opening the port again is likely to fail at least some of 
the time.

We use AT&C0 to turn off this behavior, but it sounds like your firmware 
is ignoring this.

>
> I have done a patch in huawei plugin that correct the problem at least
> with my modem. The idea is for plugin to get notified when modem AT
> channel gets closed and when this happens to remove gprs-context, reopen
> modem AT channel and restart gprs-context driver. This is possible
> because all other drivers (sim, gprs, ...) use PCUI AT channel and not
> modem AT channel.

Doing it in the modem driver (e.g. plugins/huawei.c) is the right place. 
  The fix you have come up with looks okayish.  However, you might want 
to strace/ptrace ofonod to see what the kernel is actually sending you 
to trigger this.  Also, a udev event monitor might help to tell if the 
tty is being physically removed / reinserted.

We have had a similar fix in the past and it simply didn't work due to 
the race conditions outlined.

Regards,
-Denis

Re: huawei: modem channel gets closed after PPP disconnection

[Christophe Ronco]

[-- Attachment #1: Type: text/plain, Size: 2765 bytes --]

Hello Denis,

On 02/16/2017 06:51 PM, Denis Kenzior wrote:
>
> Doing it in the modem driver (e.g. plugins/huawei.c) is the right 
> place.  The fix you have come up with looks okayish.  However, you 
> might want to strace/ptrace ofonod to see what the kernel is actually 
> sending you to trigger this.  Also, a udev event monitor might help to 
> tell if the tty is being physically removed / reinserted.
>
I've done the connect/disconnect operation with strace and a udev event 
monitor. The PPP modem channel is /dev/ttyUSB1. Here is the result of 
the udev event monitor:
root(a)klk-lpbs-040070:~ # udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[15443.637223] add      /devices/virtual/net/ppp0 (net)
KERNEL[15443.637377] add      /devices/virtual/net/ppp0/queues/rx-0 (queues)
KERNEL[15443.637464] add      /devices/virtual/net/ppp0/queues/tx-0 (queues)
UDEV  [15443.735949] add      /devices/virtual/net/ppp0 (net)
UDEV  [15443.742095] add      /devices/virtual/net/ppp0/queues/rx-0 (queues)
UDEV  [15443.754166] add      /devices/virtual/net/ppp0/queues/tx-0 (queues)
KERNEL[15449.168724] remove   /devices/virtual/net/ppp0/queues/rx-0 (queues)
KERNEL[15449.172883] remove   /devices/virtual/net/ppp0/queues/tx-0 (queues)
KERNEL[15449.176000] remove   /devices/virtual/net/ppp0 (net)
UDEV  [15449.183343] remove   /devices/virtual/net/ppp0/queues/tx-0 (queues)
UDEV  [15449.184839] remove   /devices/virtual/net/ppp0/queues/rx-0 (queues)
UDEV  [15449.275684] remove   /devices/virtual/net/ppp0 (net)

So only network interface appears (connection) and vanishes (disconnection).
The strace result is too big to be sent on the list, you can see it here:

https://framabin.org/?7e392864cc149771#8gyA/NLLDVVpnwxUV0FpR0wH5U6rwlogzy7S8yXmSMk=

What I learnt from it:

line 5839: open("/dev/ttyUSB1", O_RDWR|O_NOCTTY|O_NONBLOCK) = 10 => open
line 9041: read(10, "\r\nNO CARRIER\r\n~\377}#\200!}%}\"} }$\210*~", 
8192) = 31 => last data read
line 9049: poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=6, 
events=POLLIN}, {fd=7, events=POLLIN}, {fd=8, events=POLLIN}, {fd=9, 
events=POLLIN}, {fd=10, events=POLLIN}, {fd=11, events=POLLIN}], 8, -1) 
= 1 ([{fd=10, revents=POLLIN|POLLERR|POLLHUP}]) => poll error

So nothing more than a POLLERR|POLLHUP event on modem channel.

After that, /dev/ttyUSB1 is reopened (line 9068, before close, that's 
strange). This is because I 've done the trace with my patch. It does 
not give more information on why error happens, but we do know that 
/dev/ttyUSB1 does not disappear.

If you are OK, I'll send patch next week.

Best regards,

Christophe

Re: huawei: modem channel gets closed after PPP disconnection

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 343 bytes --]

Hi Christophe,
>
> So only network interface appears (connection) and vanishes
> (disconnection).

Okay, good.

>
> So nothing more than a POLLERR|POLLHUP event on modem channel.

okay, but the socket still has to be reopened I guess due to the POLLHUP.

> If you are OK, I'll send patch next week.

Sure.

Regards,
-Denis

[PATCH] huawei: reopen modem channel if it disconnects

[Christophe Ronco]

[-- Attachment #1: Type: text/plain, Size: 3475 bytes --]

On E3372 after a GPRS disconnection, modem AT channel (used by PPP) gets
disconnected (G_IO_HUP and G_IO_ERR are detected in received_data
in gatio.c).
After that all connection attempts fail because we are not able to send
any AT commands on modem channel (it is closed).

With this patch, when this behavior is detected, we close gprs context
driver, reopen AT channel and gprs context driver.
---
 plugins/huawei.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 47 insertions(+), 7 deletions(-)

diff --git a/plugins/huawei.c b/plugins/huawei.c
index 59995b30..fcec7994 100644
--- a/plugins/huawei.c
+++ b/plugins/huawei.c
@@ -78,6 +78,8 @@ enum {
 struct huawei_data {
 	GAtChat *modem;
 	GAtChat *pcui;
+	struct ofono_gprs *gprs;
+	struct ofono_gprs_context *gc;
 	gboolean have_sim;
 	int sim_state;
 	guint sysinfo_poll_source;
@@ -584,6 +586,45 @@ static GAtChat *open_device(struct ofono_modem *modem,
 	return chat;
 }
 
+static void modem_disconnect(gpointer user_data)
+{
+	struct ofono_modem *modem = user_data;
+	struct huawei_data *data = ofono_modem_get_data(modem);
+
+	DBG("");
+
+	ofono_warn("modem channel disconnected");
+
+	/* clean and close modem device */
+	g_at_chat_cancel_all(data->modem);
+	g_at_chat_unregister_all(data->modem);
+	g_at_chat_unref(data->modem);
+	data->modem = NULL;
+
+	/* close gprs context driver */
+	ofono_gprs_context_remove(data->gc);
+
+	/* reopen modem channel */
+	data->modem = open_device(modem, "Modem", "Modem: ");
+
+	if (data->modem == NULL) {
+		DBG("Can't reopen device");
+		return;
+	}
+
+	/* configure modem channel */
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+	g_at_chat_set_slave(data->modem, data->pcui);
+	g_at_chat_send(data->modem, "ATE0 +CMEE=1", NULL, NULL, NULL, NULL);
+
+	/* reopen gprs context driver */
+	data->gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
+						"atmodem", data->modem);
+
+	if (data->gprs && data->gc)
+		ofono_gprs_add_context(data->gprs, data->gc);
+}
+
 static int huawei_enable(struct ofono_modem *modem)
 {
 	struct huawei_data *data = ofono_modem_get_data(modem);
@@ -594,6 +635,8 @@ static int huawei_enable(struct ofono_modem *modem)
 	if (data->modem == NULL)
 		return -EINVAL;
 
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+
 	data->pcui = open_device(modem, "Pcui", "PCUI: ");
 	if (data->pcui == NULL) {
 		g_at_chat_unref(data->modem);
@@ -820,9 +863,6 @@ static void huawei_post_sim(struct ofono_modem *modem)
 	}
 
 	if (data->have_gsm == TRUE) {
-		struct ofono_gprs *gprs;
-		struct ofono_gprs_context *gc;
-
 		ofono_phonebook_create(modem, 0, "atmodem", data->pcui);
 		ofono_radio_settings_create(modem, 0,
 						"huaweimodem", data->pcui);
@@ -830,13 +870,13 @@ static void huawei_post_sim(struct ofono_modem *modem)
 		ofono_sms_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
 
-		gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
+		data->gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
-		gc = ofono_gprs_context_create(modem, 0,
+		data->gc = ofono_gprs_context_create(modem, 0,
 						"atmodem", data->modem);
 
-		if (gprs && gc)
-			ofono_gprs_add_context(gprs, gc);
+		if (data->gprs && data->gc)
+			ofono_gprs_add_context(data->gprs, data->gc);
 	}
 }
 
-- 
2.11.0

[PATCH v2 0/1] Fix possible seg fault in previous patch

[Christophe Ronco]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

A segmentation fault is possible with my previous patch.
It occurs if modem is removed. In this case, modem_disconnect is called after
huawei_remove.
ofono_modem_get_data(modem) will return NULL in this case.

Christophe Ronco (1):
  huawei: reopen modem channel if it disconnects

 plugins/huawei.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 7 deletions(-)

-- 
2.11.0

[PATCH v2 1/1] huawei: reopen modem channel if it disconnects

[Christophe Ronco]

[-- Attachment #1: Type: text/plain, Size: 3548 bytes --]

On E3372 after a GPRS disconnection, modem AT channel (used by PPP) gets
disconnected (G_IO_HUP and G_IO_ERR are detected in received_data
in gatio.c).
After that all connection attempts fail because we are not able to send
any AT commands on modem channel (it is closed).

With this patch, when this behavior is detected, we close gprs context
driver, reopen AT channel and gprs context driver.
---
 plugins/huawei.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 50 insertions(+), 7 deletions(-)

diff --git a/plugins/huawei.c b/plugins/huawei.c
index 59995b30..f02315d5 100644
--- a/plugins/huawei.c
+++ b/plugins/huawei.c
@@ -78,6 +78,8 @@ enum {
 struct huawei_data {
 	GAtChat *modem;
 	GAtChat *pcui;
+	struct ofono_gprs *gprs;
+	struct ofono_gprs_context *gc;
 	gboolean have_sim;
 	int sim_state;
 	guint sysinfo_poll_source;
@@ -584,6 +586,48 @@ static GAtChat *open_device(struct ofono_modem *modem,
 	return chat;
 }
 
+static void modem_disconnect(gpointer user_data)
+{
+	struct ofono_modem *modem = user_data;
+	struct huawei_data *data = ofono_modem_get_data(modem);
+
+	if (data == NULL) {
+		DBG("Modem has already been removed");
+		return;
+	}
+
+	ofono_warn("Modem channel disconnected");
+
+	/* clean and close modem device */
+	g_at_chat_cancel_all(data->modem);
+	g_at_chat_unregister_all(data->modem);
+	g_at_chat_unref(data->modem);
+	data->modem = NULL;
+
+	/* close gprs context driver */
+	ofono_gprs_context_remove(data->gc);
+
+	/* reopen modem channel */
+	data->modem = open_device(modem, "Modem", "Modem: ");
+
+	if (data->modem == NULL) {
+		DBG("Can't reopen device");
+		return;
+	}
+
+	/* configure modem channel */
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+	g_at_chat_set_slave(data->modem, data->pcui);
+	g_at_chat_send(data->modem, "ATE0 +CMEE=1", NULL, NULL, NULL, NULL);
+
+	/* reopen gprs context driver */
+	data->gc = ofono_gprs_context_create(modem, OFONO_VENDOR_HUAWEI,
+						"atmodem", data->modem);
+
+	if (data->gprs && data->gc)
+		ofono_gprs_add_context(data->gprs, data->gc);
+}
+
 static int huawei_enable(struct ofono_modem *modem)
 {
 	struct huawei_data *data = ofono_modem_get_data(modem);
@@ -594,6 +638,8 @@ static int huawei_enable(struct ofono_modem *modem)
 	if (data->modem == NULL)
 		return -EINVAL;
 
+	g_at_chat_set_disconnect_function(data->modem, modem_disconnect, modem);
+
 	data->pcui = open_device(modem, "Pcui", "PCUI: ");
 	if (data->pcui == NULL) {
 		g_at_chat_unref(data->modem);
@@ -820,9 +866,6 @@ static void huawei_post_sim(struct ofono_modem *modem)
 	}
 
 	if (data->have_gsm == TRUE) {
-		struct ofono_gprs *gprs;
-		struct ofono_gprs_context *gc;
-
 		ofono_phonebook_create(modem, 0, "atmodem", data->pcui);
 		ofono_radio_settings_create(modem, 0,
 						"huaweimodem", data->pcui);
@@ -830,13 +873,13 @@ static void huawei_post_sim(struct ofono_modem *modem)
 		ofono_sms_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
 
-		gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
+		data->gprs = ofono_gprs_create(modem, OFONO_VENDOR_HUAWEI,
 						"atmodem", data->pcui);
-		gc = ofono_gprs_context_create(modem, 0,
+		data->gc = ofono_gprs_context_create(modem, 0,
 						"atmodem", data->modem);
 
-		if (gprs && gc)
-			ofono_gprs_add_context(gprs, gc);
+		if (data->gprs && data->gc)
+			ofono_gprs_add_context(data->gprs, data->gc);
 	}
 }
 
-- 
2.11.0

Re: [PATCH v2 1/1] huawei: reopen modem channel if it disconnects

[Denis Kenzior]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

Hi Christophe,

On 02/21/2017 04:37 AM, Christophe Ronco wrote:
> On E3372 after a GPRS disconnection, modem AT channel (used by PPP) gets
> disconnected (G_IO_HUP and G_IO_ERR are detected in received_data
> in gatio.c).
> After that all connection attempts fail because we are not able to send
> any AT commands on modem channel (it is closed).
>
> With this patch, when this behavior is detected, we close gprs context
> driver, reopen AT channel and gprs context driver.
> ---
>  plugins/huawei.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 50 insertions(+), 7 deletions(-)
>

Applied, thanks.

Regards,
-Denis