Implementing tight time constraints for read/write with USB serial (ftdi_sio)

Implementing tight time constraints for read/write with USB serial (ftdi_sio)

[Thomas Bracht Laumann Jespersen]

Hello all!

I'm working on a project where we want to integrate into a production line and
listen on connected devices, and the challenge is that our setup needs to be
fast enough.

Let me try to explain the setup.

There are some 80+ devices on the network. They are connected in a multidrop
talking PROFIBUS at a speed of 1.5MB.

I have an RS485<->USB serial cable that uses the ftdi_sio driver to listen on
the network. It is identified as a FT232RL.

Just listening and de-constructing the PROFIBUS communication blocks that are
coming through is not a problem. The problem is when I need to reply to messages
in the network.

At first I was seeing reads from the serial port taking either ~.002ms or ~16ms,
which I assume is due to buffering on the chip. I then learned about setserial
and tried adjusting the latency to 1ms. This gives me read times at around ~4ms
on average (but sometimes going above 10ms). I'm looking for ways to
disable/minimize the buffering on the FTDI chip, but I also suspect that the
ftdi_sio driver does some buffering.

The challenge is that my application needs to act within 100 tbits, ie in around
~.066ms. That is the window in which I need to begin transmitting a reply. Is
there a way to achieve this with the ftdi_sio driver or do I need dedicated
hardware for this?

I guess my question is: Is it possible to achieve sub-millisecond response times
in a "standard" Linux (ie a desktop variant)? Or is it necessary to look into
real-time Linux or other specific kernel configuration?

As an aside, I'll mention that I've looked into libftdi as an alternative (could
it be?), but since it specifically unloads the kernel driver I'm not sure I
should be asking about that here :-)

To be honest, I'm not sure if kernelnewbies is the right place to ask, if not,
I'll be happy to take my questions elsewhere :-)


All the best,
-- Thomas

Implementing tight time constraints for read/write with USB serial (ftdi_sio)

[valdis.kletnieks at vt.edu]

On Mon, 03 Sep 2018 21:12:35 +0200, Thomas Bracht Laumann Jespersen said:

> The challenge is that my application needs to act within 100 tbits, ie in around
> ~.066ms. That is the window in which I need to begin transmitting a reply. Is
> there a way to achieve this with the ftdi_sio driver or do I need dedicated
> hardware for this?

(Hopefully you've already done all of the below - it's mostly for the
others out there thinking about doing real-time work...)

Step 0 is quantifying what a failure means.  Do you just miss a data point for
a graph, or does something on the assembly line explode or equally bad, or
something in between?

Step 1 is firming up exactly what that 0.66ms is.  Does it start at the
beginning of receiving the packet you care about, or when the last byte has
arrived?  And are you counting until the first byte of the response leaves, or
the last byte? Or are you counting until it's actually received at the target?
(Note that some statistical modeling may be needed, depending how PROFIBUS
behaves under load - what happens if another station is transmitting a very
large packet at the time you need to send your packet, etc etc..)

Step 2 is doing some userspace benchmarking of your response code - can the
target hardware do whatever processing is needed in the time allowed (remember
to subtract off any hardware-imposed latency like your PROFIBUS or your
RS485-USB converter).  This can be easy to meet if you have a beefy CPU and
just need to check the values of 3 bytes, add 2 other bytes together, and send
the result.  On the other hand, if you're on a low-end ARM and need to do a
512x512 FFT - the Realtime Gods may not smile favorably on your endeavor.

If your project is still alive at this point, *now* you start asking stuff like
realtime scheduling for your kernel thread, how to reduce interrupt
latency, and so on....

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 486 bytes
Desc: not available
URL: <http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20180903/9a573f53/attachment.sig>

Implementing tight time constraints for read/write with USB serial (ftdi_sio)

[Thomas Bracht Laumann Jespersen]

Hi Valdis,

Thanks for your reply!

Failure in our context means that we cannot participate as an alternative master
in the PROFIBUS network.

I'll try to explain briefly what the intended setup is: A PROFIBUS network
consists of exactly one master (a class 1 master, or MC1), and a number of
slaves. All the slaves are silent until queried for information. The MC1 goes
round-robin asking each slave for periodic status updates, to keep tabs on how
things are going. PROFIBUS is _very_ deterministic: only station may be
communicating at a time and the master orchestrates the exchange of information
[0].

Our challenge is that we want to pull run-time data out of this network, and
re-programming MC1 to extract more data is not an option. So failure also means
we cannot monitor the system as well as we want.

Fortunately, PROFIBUS also specifies a class 2 master (MC2) that is intended for
maintenance purposes [1]. The MC2 design allows other stations to dynamically
enter and leave the network. The idea is to use the gap time to act as a master.

To orchestrate this, the masters pass a token around. Specifically, the MC1
periodically tries all possible network addresses to see if any other stations
want to accept the token. If so, then the station must respond with a specific
"accept token" message. Once the MC1 station sees the accept, it passes the
token to the new MC2 and the two masters then pass the token back and forth.

An MC2 station leaves the network by simply failing to return the token three
times.

We have a test setup where I have been able to successfully send the "accept
token" message, but by the time the MC1 station passes the token (and my program
sees it), it has already failed to pass it back.

The problem is that response transmission has to begin within 100 tbits of
response receipt (this is called the slot time). That is, from the time I see a
message I should respond to, I must begin transmitting a response within 100
tbits. In our test setup we run 115 kBaud, so our slot time works out to about
0.868ms. In the production setup, we need to go even faster, at 1.5Mbaud, so the
slot time dips to ~0.066ms.

[0] https://www.felser.ch/profibus-manual/stationen.html
[1] https://www.felser.ch/profibus-manual/prinzip_des_token-passing.html


-- Thomas
On Tue, 4 Sep 2018 at 03:23, <valdis.kletnieks@vt.edu> wrote:
>
> On Mon, 03 Sep 2018 21:12:35 +0200, Thomas Bracht Laumann Jespersen said:
>
> > The challenge is that my application needs to act within 100 tbits, ie in around
> > ~.066ms. That is the window in which I need to begin transmitting a reply. Is
> > there a way to achieve this with the ftdi_sio driver or do I need dedicated
> > hardware for this?
>
> (Hopefully you've already done all of the below - it's mostly for the
> others out there thinking about doing real-time work...)
>
> Step 0 is quantifying what a failure means.  Do you just miss a data point for
> a graph, or does something on the assembly line explode or equally bad, or
> something in between?
>
> Step 1 is firming up exactly what that 0.66ms is.  Does it start at the
> beginning of receiving the packet you care about, or when the last byte has
> arrived?  And are you counting until the first byte of the response leaves, or
> the last byte? Or are you counting until it's actually received at the target?
> (Note that some statistical modeling may be needed, depending how PROFIBUS
> behaves under load - what happens if another station is transmitting a very
> large packet at the time you need to send your packet, etc etc..)
>
> Step 2 is doing some userspace benchmarking of your response code - can the
> target hardware do whatever processing is needed in the time allowed (remember
> to subtract off any hardware-imposed latency like your PROFIBUS or your
> RS485-USB converter).  This can be easy to meet if you have a beefy CPU and
> just need to check the values of 3 bytes, add 2 other bytes together, and send
> the result.  On the other hand, if you're on a low-end ARM and need to do a
> 512x512 FFT - the Realtime Gods may not smile favorably on your endeavor.
>
> If your project is still alive at this point, *now* you start asking stuff like
> realtime scheduling for your kernel thread, how to reduce interrupt
> latency, and so on....
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Implementing tight time constraints for read/write with USB serial (ftdi_sio)

[Greg KH]

On Mon, Sep 03, 2018 at 09:12:35PM +0200, Thomas Bracht Laumann Jespersen wrote:
> Hello all!
> 
> I'm working on a project where we want to integrate into a production line and
> listen on connected devices, and the challenge is that our setup needs to be
> fast enough.

First off, "tight time constraints" and "usb-serial" should never be in
the same sentence, unless you use the words "never will happen" in there
as well :)

There are loads of issues with trying to use USB with anything you wish
to have low-latency or determinism with, and then throw a usb-serial
device into the mix and all bets are off.  Remember there is a little
microcontroller out there at the end of the usb-serial device running a
UART and some unknown firmware blob that can take its time and do
anything it wants at any point in time, which can, and will, cause you
all sorts of headaches as you are finding out.

If you _really_ need to do something like this, use a UART attached to a
PCI card, you will have much better control and be able to actually do
what you are wanting to do here.

Good luck!

greg k-h

Implementing tight time constraints for read/write with USB serial (ftdi_sio)

[Thomas Bracht Laumann Jespersen]

On Tue, 4 Sep 2018 at 07:31, Greg KH <greg@kroah.com> wrote:
>
> On Mon, Sep 03, 2018 at 09:12:35PM +0200, Thomas Bracht Laumann Jespersen wrote:
> > Hello all!
> >
> > I'm working on a project where we want to integrate into a production line and
> > listen on connected devices, and the challenge is that our setup needs to be
> > fast enough.
>
> First off, "tight time constraints" and "usb-serial" should never be in
> the same sentence, unless you use the words "never will happen" in there
> as well :)

I'm learning (beginning to accept) that :-) This is pretty new territory for me,
so I appreciate the wisdom.

>
> There are loads of issues with trying to use USB with anything you wish
> to have low-latency or determinism with, and then throw a usb-serial
> device into the mix and all bets are off.  Remember there is a little
> microcontroller out there at the end of the usb-serial device running a
> UART and some unknown firmware blob that can take its time and do
> anything it wants at any point in time, which can, and will, cause you
> all sorts of headaches as you are finding out.
>
> If you _really_ need to do something like this, use a UART attached to a
> PCI card, you will have much better control and be able to actually do
> what you are wanting to do here.

We need to find a solution somehow, so thanks for the suggestion. I'll keep
digging for solutions, but if you have any good resources I should look at, I'm
all ears.

>
> Good luck!

Thank you!

Implementing tight time constraints for read/write with USB serial (ftdi_sio)

[Greg KH]

On Wed, Sep 05, 2018 at 12:25:47PM +0200, Thomas Bracht Laumann Jespersen wrote:
> We need to find a solution somehow, so thanks for the suggestion. I'll keep
> digging for solutions, but if you have any good resources I should look at, I'm
> all ears.

Do not use USB for anything that you have to do that is deterministic.
Unless you can handle dropping data on the floor (like USB audio will
do).  Other than that, why not just use the bus-specific hardware
controller for this type of bus to monitor and read the data?  That
hardware is available, right?

Good luck!

greg k-h

Implementing tight time constraints for read/write with USB serial (ftdi_sio)

[Thomas Bracht Laumann Jespersen]

On Wed, 5 Sep 2018 at 12:49, Greg KH <greg@kroah.com> wrote:
> Do not use USB for anything that you have to do that is deterministic.
> Unless you can handle dropping data on the floor (like USB audio will
> do).  Other than that, why not just use the bus-specific hardware
> controller for this type of bus to monitor and read the data?  That
> hardware is available, right?

Got it. We _could_ probably handle dropping data, but our solution would then
have to constantly reconnect to the network. I think that'd just be a major
headache.

Yes, there is hardware for this, but we wanted to see if we'd be able to solve
this primarily in software. We already have one these: [0]. The unfortunate
thing is that all the available software is Windows-based :-) There are no Linux
drivers, but hey, maybe I'll finally get to write a kernel driver :D

[0] https://www.thorsis.com/en/industrial-automation/usb-interfaces/profibus/ispro-usbx12/

Thanks!

-- Thomas