Re: Question on pppd / reset?

Re: Question on pppd / reset?

[James Carlson]

PierrotLafouine wrote:
> My question is about reset.  I would like to understand more the stack
> of the system, and why it failed to reconnect.  The GPRS modem is
> connected trought full RS-232 signals, and RS-232 chips is on an ISA
> bus.  How does ppp deal with this kind of hadware configuration ?  I
> read ppp deal with modem reset, but how ?  Any idea what I should look
> at to understand more the root cause of this failure ?

It's not very different from any other tty-using application on UNIX,
but I'll give details anyway.

Loss of carrier on the modem will normally cause the RS-232 DCD (Data
Carrier Detect) signal to be de-asserted.  When this happens, the serial
driver in the host system will generate a hardware interrupt that in
turn sends SIGHUP to the process group that has this port as a
controlling terminal.

For normal use of pppd, we fork into the background and open the serial
port, so that becomes the controlling terminal for the process.  If you
run with "debug" mode in pppd and have syslog's "daemon.debug" directed
to a file, you should see SIGHUP being handled by pppd when this happens.

Internally, this causes "got_sighup" to be set, which then results in
"kill_link" being set.  We then terminate LCP, which eventually causes
the_channel->disconnect() and the_channel->cleanup() to run.  That last
one does a close() on the tty file descriptor, severing our connection
to the serial port.

The last close on the tty file descriptor should cause the driver in the
kernel to de-assert DTR (Data Terminal Ready) and RTS (Request To Send)
signals that the modem sees.  With a normally configured modem, these
should result in the modem being reset back to a known state.

What can go wrong?  Well, many things.  It helps a lot to have a
stand-alone serial analyzer when debugging things like this (Klos
Technologies makes a decent one; there are many others available),
rather than stumbling in the dark.

Here's a partial list of the sorts of things that I've seen or heard of:

  - The "modem" itself may have crashed or gotten stuck.  Note that with
    GPRS, you're really dealing with something that's more like an ISDN
    TA in terms of complexity.  It usually does protocol translation as
    well as local processing of some of the messages you send.  It's not
    just a simple character-based device as traditional wired telephone
    modems once were.

  - Timing problems on the wire combined with unusual modem issues
    sometimes cause the link to become wedged.  A classic case of this
    happens with a "smart" modem-like device seeing RTS from the host
    drop during the disconnect sequence, interpreting it as an assertion
    of flow control, and then locking up while waiting for the host to
    become ready again.  (Which of course never happens.)

  - It's possible that the serial driver has some sort of defect.  This
    isn't altogether unheard of; it's surprisingly hard to write a
    decent driver.  Defects here could cause the driver to cease
    handling interrupts (and thus never "see" the loss of DCD), or get
    "stuck" due to a mishandled transmit or receive interrupt such that
    all I/O stops.

    For what I not-too-humbly consider to be a nifty instance of such a
    problem, see:

    http://blogs.sun.com/carlson/entry/close_hang_or_4028137

  - It's possible that some other software on the system has the serial
    port open, or that a hidden fork() call in some library that pppd
    has called has leaked the tty descriptor, causing it to be held
    open.  In that case, you'd end up with pppd being unable to re-open
    the port, because it's now stuck "busy," and the modem unable to
    reset itself because DTR never drops.

  - It's possible for plug-ins or scripts used with pppd to cause
    erroneous operation.

  - It's possible for the OS you're using to have bugs related to serial
    port use (apart from the driver bugs suggested above).

  - It's possible that there are still bugs somewhere in pppd when
    serial ports are closed and reopened.  I haven't seen any, but that
    certainly doesn't mean they're not there.

If I had to bet on it, though, I'd bet on defects in the GPRS device
itself.  I haven't heard of one yet that was a worthwhile use of
someone's time and effort.  At least not mine.  ;-}

-- 
James Carlson         42.703N 71.076W         <carlsonj@workingcode.com>

Re: Question on pppd / reset?

[PierrotLafouine]

James Carlson wrote:
> PierrotLafouine wrote:
>   
>> My question is about reset.  I would like to understand more the stack
>> of the system, and why it failed to reconnect.  The GPRS modem is
>> connected trought full RS-232 signals, and RS-232 chips is on an ISA
>> bus.  How does ppp deal with this kind of hadware configuration ?  I
>> read ppp deal with modem reset, but how ?  Any idea what I should look
>> at to understand more the root cause of this failure ?
>>     
>
> If I had to bet on it, though, I'd bet on defects in the GPRS device
> itself. 
This is also my bet, as software reboot didn't work, and a power down
was needed to restore communication.  But now the question becomes : how
RS-232 chips translates a DCD into ISA bus signals ?  I'll check into
chips datasheet.

Your detailed answer (thank you very much) raise me an other questions :
some embedded boards (ex: BeagleBoard) are provided with GPRS modem. 
But looking at schematics, I notice most of them only connect Rx and Tx
between modem and UART/RS-232.  How would pppd deal with a lost of
carrier in these case ?  Is there a way (ex: command) to detect this
trought data ?

Thanks

Pierre

Re: Question on pppd / reset?

[James Carlson]

PierrotLafouine wrote:
> James Carlson wrote:
>> PierrotLafouine wrote:
>>   
>>> My question is about reset.  I would like to understand more the stack
>>> of the system, and why it failed to reconnect.  The GPRS modem is
>>> connected trought full RS-232 signals, and RS-232 chips is on an ISA
>>> bus.  How does ppp deal with this kind of hadware configuration ?  I
>>> read ppp deal with modem reset, but how ?  Any idea what I should look
>>> at to understand more the root cause of this failure ?
>>>     
>> If I had to bet on it, though, I'd bet on defects in the GPRS device
>> itself. 
> This is also my bet, as software reboot didn't work, and a power down
> was needed to restore communication.  But now the question becomes : how
> RS-232 chips translates a DCD into ISA bus signals ?  I'll check into
> chips datasheet.

It depends on the chipset in use, but there's typically one interrupt
for the chip itself, and the driver has to read an interrupt status
register to determine which interrupt(s) need to be handled.  The driver
will have a low-level interrupt handler that vectors off based on the
contents of this register to handler routines for modem status (DCD,
CTS) change, transmit complete, and receive buffer full conditions.  The
handler routines then either set state flags and signal a higher-level
task to send the signal to the process, or perhaps send the signal
directly, depending on the internal design of the driver.  You'd need to
read the driver source to find out exactly what it's doing.

Most (though certainly not all) serial chips used in PC-like hardware
are derived from the 8250/16450/16550 series.  There are some
differences among these, but at least for understanding how the
interrupt mechanisms work, any of the available documents would likely
be sufficient.

I'm not at all sure why you'd want or need to know that, though.  I
doubt it's related to the problem you're facing.

> Your detailed answer (thank you very much) raise me an other questions :
> some embedded boards (ex: BeagleBoard) are provided with GPRS modem. 
> But looking at schematics, I notice most of them only connect Rx and Tx
> between modem and UART/RS-232.  How would pppd deal with a lost of
> carrier in these case ?  Is there a way (ex: command) to detect this
> trought data ?

In short, it can't.  You mentioned in your message that you have modem
controls hooked up, and you asked about how those work, so that's what I
answered.

On a device without standard modem controls, there are two issues:
handling connection start/end and handling flow control.  Both must be
handled "in-band" (using special characters in the data stream) when the
"out-of-band" (separate control signals) are missing.

Connection start/end using in-band signaling is pretty complex and error
prone.  The typical way this is done is by using the Hayes "TIES"
sequence ("+++") to get the modem's attention, and then using "AT"
commands to direct the modem's behavior.  And to detect a disconnected
line, where the modem went down and the peer didn't do the normal LCP
termination handshake, you will need to set up the "lcp-echo-interval"
and "lcp-echo-failure" parameters in pppd, so that the daemon can check
the status of the link periodically and shut it down if it seems to have
failed.

(Note that you can profitably use those parameters on links with
standard modem control signals as well.  They guard against a different
class of failures, and I certainly recommend that they be used when
possible.)

The in-band flow control signaling mechanism is usually XON/XOFF (hex 11
and 13, respectively).  This must be implemented in the serial driver,
and pppd's only responsibility is making sure that the accm (async
control character mask) is set to 0xA0000 so that those characters never
appear in the data itself.

Unlike the connection start/end issue, the flow-control one is fairly
simple and workable in theory, though I can certainly say that bugs in
this area are _extremely_ common.

If this is really what you need, you're pretty much on your own to
design a robust solution.  I wish you luck.  (The simple solution is
defenestration of the offending hardware.  It's all a question of how
much you value your time.)

-- 
James Carlson         42.703N 71.076W         <carlsonj@workingcode.com>

Question on pppd / reset?

[PierrotLafouine]

Hi
If I'm not on the good forum, or too specific with my questions, please
provide me the address of the good mailing list/forum.

We have an application using ppp with a GPRS modem.  The system is
actually in the field and working 24/7 for several months.

Carrier disconnected modem often  (IP renewal, maintenance etc.) and
modem alway reconnected correctly, except last time.  I went onsite for
debugging and try different commands to make modem reconnect
(ifdown/ifdown, poff/pon, shutdown -r etc...) All commands failed to
reconnect the modem.  I had to pull the plug to make it connect.

My question is about reset.  I would like to understand more the stack
of the system, and why it failed to reconnect.  The GPRS modem is
connected trought full RS-232 signals, and RS-232 chips is on an ISA
bus.  How does ppp deal with this kind of hadware configuration ?  I
read ppp deal with modem reset, but how ?  Any idea what I should look
at to understand more the root cause of this failure ?

Thanks,

Pierre