Simplifying the 802.15.4 stack

Simplifying the 802.15.4 stack

[Phoebe Buckheister]

(resent because I forgot CCs at first)

Hi,

the 802.15.4/6LoWPAN stack on Linux is pretty usable as it is now, much
due to the recent 6lowpan fixes by Alex. We can drive different radio
chips on different frequencies, IPv6 works well and we interoperate
just fine with RFC compliant stacks like the one implemented in Contiki.

There are, however, also a number of sore points, which I want to
outline in this message and for which I want to propose possible
solutions. These sore points range anywhere from minor annoyances to
what I view as major integrity problems. I'll list what comes to my
mind here, in no particular order. If you have any idea how to fix
something, if you have something to add to the list or merely want to
tell me where and why I am mistaken, please do so.

1) header handling in the stack

We currently have three points in the stack that parse or create
802.15.4 headers, where two of those are already duplicated code - in
the same source file no less. Header handling for mac802154 is
cumbersome, and for upper layers entirely impossible without
duplicating more code at the moment. I have a patch that attempts to
rectify this by adding structs for the 802.15.4 header and its
components, but that was shot down by davem due to some endianness
issues. I am reworking the patch to fix all of this and will resend it
soon, but it doesn't end with that.

2) endianness in the stack

In fact, that patch was consistent with how the mac802154 stack
currently handles endianness - which is, pretty much, not at all. For
example we have __le16 fields that the PAN ID of a given netdevice,
which is filled with data in host byte order and only happens to work
out because it's written to the header with explicit mask-shift
operations. Short addresses for nodes are given in host byte order,
extended addresses are however always held in big endian 8-byte arrays.
We really never use network byte order on purpose, and inside of the
stack we should really change that. Ideally we'd also change it in the
userspace API if we ever got the chance to do that, because
AF_IEEE802154 is the *only* protocol i know that uses all byte orders
in it's sockaddr struct *except* network byte order.

3) the mac802154_priv slave list

This is one of the biggest problems I think the current stack has. For
HardMAC, every netdevice corresponds to one specific WPAN (made up of
PAN ID, channel, and page) and it is pretty much guaranteed that
multiple HardMAC netdevs will always function as intended, even if they
are backed by the same piece of hardware.

SoftMAC, on the other hand, takes a single PHY chip and adds an
*arbitrary* number of specific WPANs on top of that PHY. No real
hardware can support an arbitrary number of specific WPANs, and no chip
I have yet come across can even support more than one. As far as I've
been told and was able to reconstruct from documentation and standards
documents, this was implemented to allow for ZigBee frequency agility,
but that was never implemented. What we have, though, is a means for
unprivileged users that can run socket(), write() and ip(8) to
effectively DoS a node that uses this feature.

Here's why: if somebody configures the stack with one PHY and two
specific WPANs on that PHY, an packet sent through one of those
specific WPAN netdevs will activate that specific WPAN and deactivate
all others on that PHY. This only holds when the specific WPANs differ
in channel or page, since we do not reconfigure the PHY addresses or
address filters on on the switch, which will most certainly yield
wildly incorrect behaviour if anybody were to ever use multiple
specific WPANs per PHY.

I thus propose to remove this feature entirely and instead have a 1:1
correspondence between PHYs and netdevs. If a driver can support
multiple specific WPANs per chip, it can register multiple PHYs. If
anybody actually needs to reconfigure the device to a different
specific WPAN, we already have an interface for that: the netlink API.
The netlink API also has the benefit that it does not impose arbitrary
delays before the switch, as switching by sending a packet does when
reordering/shaping Qdiscs are configured on the WPAN.

If we were to remove the slave list, the entire netlink API specific to
PHYs and adding/removing slaves would have to go. Listing PHYs would no
longer be necessary, instead we'd need an API to query PHY/MAC
parameters by netdev name. What is currently SET_PHYPARAMS (and my
fault) would also have to be split into commands that set PHY and MAC
params for consistency.

I propose a flag day to remove this feature entirely, as it will break
userspace, but it will do so to provide much better service,
semantically correct netdevs and it will remove the DoS path for
unprivileged users. And if we do that, we might as well think about
whether or not we should also change endianness of network-facing
fields exposed to userspace for consistency with all other protocols
implemented in Linux.

4) our *drivers* are expected to *sleep*

This is just wrong. Our RX/TX path invokes the scheduler at least once
for every packet on the assumption that drivers might sleep, which may
entail arbitrary delays even after the Qdiscs of the system have run.
Especially when somebody wanted to use latency sensitive Qdiscs, this
is a disaster, but also for general system throughput once you have a
certain number of packets going through your node.

I think this should also be reworked completely. Network packet
reception/transmission is inherently asynchronous, and the current
stack goes out of its way to make it a synchronous matter.
Interestingly, once a driver has received a packet, most if not all of
the RX path might actually be softirq-safe as it is now, without the
workqueue deferral. We don't fundamentally need the RX workqueue, and
the TX workqueue could also be dropped if we gave a PHY driver a
function pointer to call once transmission is complete. That would also
simplify the stack quite a bit, not introduce unnecessary latencies,
and generally be an improvement on drivers that invoke the scheduler.


Any thoughts/ideas?

Re: [Linux-zigbee-devel] Simplifying the 802.15.4 stack

[Werner Almesberger]

Phoebe Buckheister wrote:
> the 802.15.4/6LoWPAN stack on Linux is pretty usable as it is now, much
> due to the recent 6lowpan fixes by Alex.

Indeed, thanks to the work both of you did !

Regarding the proposed massive changes, I think everyone who had
a closer look at the stack knows that (too) much of it is still a
smoldering mess, and that most of the remaining problems are
caused by design issues that also touch APIs.

> 1) header handling in the stack
> 2) endianness in the stack

I don't care much about these, but jumping between byte orders is
certainly confusing. As long as an API user can tell easily and
unambiguously what was or will be in the air, I won't complain.

> 3) the mac802154_priv slave list

Yes, weird and of dubious use. Ironically, support for the only
piece of code that could have a chance to actually use this
without devastation - the coordinator - was dropped when 802.15.4
support originally went mainline.

And even if someone was to resurrect the coordinator and made it
multi-PAN, there are probably easier ways to accomplish all that.
So I don't see a problem with this sort of change, as long as
there is some way to set up the matching rules of 802.15.4-2011
section 5.1.6.2 (page 42), specifically the broadcast PAN ID
match.

This API has of course a non-negligible installed base since the
administrative tools touch it and everyone who uses 802.15.4 in
any way, including through 6LoWPAN, has to use the administrative
tools.

But then that very same installed base barely noticed that a few
months ago not even a ping really worked, so I wouldn't be overly
concerned about continuity.

In practical terms, it will be impossible to perfectly synchronize
kernel API changes and user space tools or documentation. Thus,
the sooner the change is made, the better.

Furthermore, if you could think of a way to preserve the old API
for a while, possibly reduced to just supporting the only
real-life case of having exactly one slave, that would ease the
transition.

This leads us to the heart of evil, ...

> 4) our *drivers* are expected to *sleep*

Linux has a perfectly good network device API that avoids such
perversions and also has a sane concept of flow control. Making
802.15.4 use that API will of course break every existing
802.15.4 driver, but there are few enough of them in mainline and
probably not a lot in active use out of mainline.

Specifically, we have in mainline:

- at86rf230.c which you are using yourself, so I'd assume you'd
  adapt is when such a change is made,
- mrf24j40.c by Alan Ott. Not sure what he thinks of the idea,
- fake*.c which should be fairly straightforward to adapt,

Outside mainline I know of atben which reuses at86rf230, so there
is nothing to do there, and atusb, which I'll be happy to adapt
(and should finally beat into shape for mainline anyway.)

Maybe one could also come up with a smooth transition strategy for
the driver API change, but I'd be careful not to waste time going
through the motions if nobody really needs that.


So to gauge the impact of the changes you're proposing, we'd need
to know:

1) whether there are any major users of AF_IEEE802154 (if you're
   planning to change that API),

2) whether anyone depends on the 802.15.4 administrative interface
   beyond having compiled a variant of
   http://sourceforge.net/apps/trac/linux-zigbee/wiki/GettingStarted-0.2

3) and whether there are any other 802.15.4 device drivers in active
   use with recent kernels besides the ones I mentioned above.

I don't think the sleepy driver API change needs to happen at the
same time as the rest so it may make sense to do all this in at
least two phases.

This brings us to the next question: when do you plan to do all
this ? :)

- Werner

Re: [Linux-zigbee-devel] Simplifying the 802.15.4 stack

[Phoebe Buckheister]

On Fri, March 7, 2014 8:38 pm, Werner Almesberger wrote:
>> 1) header handling in the stack
>> 2) endianness in the stack
>
> I don't care much about these, but jumping between byte orders is
> certainly confusing. As long as an API user can tell easily and
> unambiguously what was or will be in the air, I won't complain.

I'm not so sure that API users can tell, especially since even within the
sockaddr struct we have differing byte orders for different fields. At
least *that* should be consistent, even if only in itself and not with
what the stack uses internally.

>> 3) the mac802154_priv slave list
>
> Yes, weird and of dubious use. Ironically, support for the only
> piece of code that could have a chance to actually use this
> without devastation - the coordinator - was dropped when 802.15.4
> support originally went mainline.
>
> And even if someone was to resurrect the coordinator and made it
> multi-PAN, there are probably easier ways to accomplish all that.

Precisely. The coordinator is also *much* better of setting parameters via
netlink that would have otherwise been set implicitly. Changing specific
WPANs via a packet imposes all the possible delays and reorderings a qdisc
can cause, and that's most certainly not what a coordinator wants, and
it's also completely impossible to change some parameters like that (see
below).

> So I don't see a problem with this sort of change, as long as
> there is some way to set up the matching rules of 802.15.4-2011
> section 5.1.6.2 (page 42), specifically the broadcast PAN ID
> match.

That would still be possible, of course. Hardware will hand either all
packets up the stack, which can then easily be filtered, or hardware
supports its own address filtering and hands over only matching packets.
Currently, we do a mixture of both, and if someone uses the slaves feature
to set up multiple interface with different PAN IDs, we do hardware
address filtering wrong. Implicit specific WPAN switching changes only
channel and page, and nothing else, making the slaves even more harmful.

> This API has of course a non-negligible installed base since the
> administrative tools touch it and everyone who uses 802.15.4 in
> any way, including through 6LoWPAN, has to use the administrative
> tools.
>
> But then that very same installed base barely noticed that a few
> months ago not even a ping really worked, so I wouldn't be overly
> concerned about continuity.

6lowpan was broken for basically everything until Alex fixed it. Perhaps
the raw/dgram sockets worked fine all the time (I haven't used them much
yet). But then you can hand a MTU-sized packet to a dgram socket, which
might - depending on the driver - either silently truncate the frame,
report success and not transmit it at all, or do even wilder things ...

> In practical terms, it will be impossible to perfectly synchronize
> kernel API changes and user space tools or documentation. Thus,
> the sooner the change is made, the better.

To my understanding, the current API to the stack isn't even exposed to
userspace. It lives entirely in include/net, and all users I know of
(which is exactly lowpan-tools and my local wpan manager daemon) copy
headers from the kernel to have at least *some* definitions to use. As
such, I would consider those users at the mercy of whoever or whatever
might change those headers.

> Furthermore, if you could think of a way to preserve the old API
> for a while, possibly reduced to just supporting the only
> real-life case of having exactly one slave, that would ease the
> transition.

We could easily keep LIST_PHY for a while, let ADD_IFACE work *exactly*
once and DEL_IFACE not at all. That's not exactly preserving the API, but
it is the closest thing to preserving the API I can currently think of.
Removing the slaves is such an intrusive change that all API compatibility
will break at first or second poke, just like 6lowpan has for years -
ADD_IFACE for example would effectively be only a rename, and for
compatibility with LIST_PHY, we'd have to keep the old name around. I'd
rather not do that if the interface handling is fundamentally incompatible
with the old way anyway.

>> 4) our *drivers* are expected to *sleep*
>
> Linux has a perfectly good network device API that avoids such
> perversions and also has a sane concept of flow control. Making
> 802.15.4 use that API will of course break every existing
> 802.15.4 driver, but there are few enough of them in mainline and
> probably not a lot in active use out of mainline.

That's the funny thing. mac802154 disables/enables all netdev queues on a
phy before/after every TX action.

> Specifically, we have in mainline:
>
> - at86rf230.c which you are using yourself, so I'd assume you'd
>   adapt is when such a change is made,

Yes, though the driver change is not actually one of priorities at the
moment since it does "just work". It is freakishly ugly, but it works for
everything I've yet thrown at it.

> Maybe one could also come up with a smooth transition strategy for
> the driver API change, but I'd be careful not to waste time going
> through the motions if nobody really needs that.

That's will most likely force us to keep the workqueues, or queue the
RX/TX actions to the system queues and suffer from even more latencies
(possibly). I suppose it's possible to add a start_xmit to struct
ieee802154_ops and entirely rip out the mac802154 workqueues, running
driver that have only xmit

> So to gauge the impact of the changes you're proposing, we'd need
> to know:
>
> 1) whether there are any major users of AF_IEEE802154 (if you're
>    planning to change that API),

I want to change that, yes, but it's not hugely important for
AF_IEEE802154. The address format can stay as weird as it is now without
actively harming anybody.

> 2) whether anyone depends on the 802.15.4 administrative interface
>    beyond having compiled a variant of
>    http://sourceforge.net/apps/trac/linux-zigbee/wiki/GettingStarted-0.2

I would almost assume that not, and if so, be inclined to not care a lot
about those users since they too will have taken kernel definitions that
weren't actually exposed to userspace at any point. Also, since that guide
mentions mainly 6lowpan for anything other than the primitive chat thingy,
I'm inclined to believe that virtually nobody has ever gotten even that
far.

> 3) and whether there are any other 802.15.4 device drivers in active
>    use with recent kernels besides the ones I mentioned above.

I'm tempted to say "someone else's problem", but that won't help anyone in
the long run. Let's hope that anybody using such drivers reads this thread
and will reply ...

> I don't think the sleepy driver API change needs to happen at the
> same time as the rest so it may make sense to do all this in at
> least two phases.

Yes. What's really pressing is the slave list disaster and the netlink
APIs that concern themselves with it.

> This brings us to the next question: when do you plan to do all
> this ? :)

During some office hours and whatever time outside of office hours I want
to  dedicate to that task ;) My current priorities actually lie on
implementing link-layer security and what netlink APIs that requires,
which is at least partially done by now - so I might actually have time to
this not solely in my free time in the nearer future.

Re: Simplifying the 802.15.4 stack

[Phoebe Buckheister]

This is a reply to Dmitrys mail, which went only to linux-zigbee-devel.
The full text of Dmitrys mail is included for interested netdev readers
that are not on linux-zigbee-devel.

On Fri, March 7, 2014 11:16 pm, Dmitry Eremin-Solenikov wrote:
> Hello,
>
> On Fri, Mar 7, 2014 at 4:16 PM, Phoebe Buckheister
> <phoebe.buckheister@itwm.fraunhofer.de> wrote:
>> Hi,
>>
>> the 802.15.4/6LoWPAN stack on Linux is pretty usable as it is now, much
>> due to the recent 6lowpan fixes by Alex. We can drive different radio
>> chips on different frequencies, IPv6 works well and we interoperate
>> just fine with RFC compliant stacks like the one implemented in Contiki.
>>
>> There are, however, also a number of sore points, which I want to
>> outline in this message and for which I want to propose possible
>> solutions. These sore points range anywhere from minor annoyances to
>> what I view as major integrity problems. I'll list what comes to my
>> mind here, in no particular order. If you have any idea how to fix
>> something, if you have something to add to the list or merely want to
>> tell me where and why I am mistaken, please do so.
>>
>> 1) header handling in the stack
>>
>> We currently have three points in the stack that parse or create
>> 802.15.4 headers, where two of those are already duplicated code - in
>> the same source file no less. Header handling for mac802154 is
>> cumbersome, and for upper layers entirely impossible without
>> duplicating more code at the moment. I have a patch that attempts to
>> rectify this by adding structs for the 802.15.4 header and its
>> components, but that was shot down by davem due to some endianness
>> issues. I am reworking the patch to fix all of this and will resend it
>> soon, but it doesn't end with that.
>
> Strange. I don't remember issues with headers code, but that was looong
> time ago. I was puzzled by header-related code in 6lowpan. For the headers
> there is a simple requirement: dev_hard_header() should work for both
> SoftMAC and HardMAC drivers. That means that before calling
> dev_hard_header() the frame does not contain header at all.

The header handling in mac802154 is actually fine, save for a little code
duplication, as long as no frame with link-layer security are received.
Once that happens, the security header is simply not parsed and the frame
is not dropped as required, but the security header and the actual frame
payload are handed to upper layers as though they were the payload of an
unsecured frame.

6lowpan has an entirely different problem. Currently, 6lowpan assumes that
the ieee802154_mac_cb struct in an skb->cb will be valid when lowpan
receives the skb, but that's just not so. Instead, lowpan would have to
parse the MAC header again to get the addresses it needs for further
processing. To do that without duplicating the mac802154 header parsing
code, we need to move header handling around and make it available to all
interested users, not only mac802154.

>
>> 2) endianness in the stack
>
> The stack was tested on PowerPC boxes, but that was some time ago
> (at least before 6lowpan work). If you need an access to PowerPC box
> to run tests, I can think about arranging that.
>
>> We really never use network byte order on purpose, and inside of the
>> stack we should really change that. Ideally we'd also change it in the
>> userspace API if we ever got the chance to do that, because
>> AF_IEEE802154 is the *only* protocol i know that uses all byte orders
>> in it's sockaddr struct *except* network byte order.
>
> This can be hard and davem would probably disallow it :(
>
> Sockaddr for 802.15.4 should have used native byteorder, if I'm not
> mistaken.
>
>> 3) the mac802154_priv slave list
>>
>> This is one of the biggest problems I think the current stack has. For
>> HardMAC, every netdevice corresponds to one specific WPAN (made up of
>> PAN ID, channel, and page) and it is pretty much guaranteed that
>> multiple HardMAC netdevs will always function as intended, even if they
>> are backed by the same piece of hardware.
>>
>> SoftMAC, on the other hand, takes a single PHY chip and adds an
>> *arbitrary* number of specific WPANs on top of that PHY. No real
>> hardware can support an arbitrary number of specific WPANs, and no chip
>> I have yet come across can even support more than one. As far as I've
>> been told and was able to reconstruct from documentation and standards
>> documents, this was implemented to allow for ZigBee frequency agility,
>> but that was never implemented. What we have, though, is a means for
>> unprivileged users that can run socket(), write() and ip(8) to
>> effectively DoS a node that uses this feature.
>
> This was done to support different kinds of devices accessing the radio.
> Compare to 802.11 stack, which has phy & device conception.
> It really helps to separate between the radio-related parts and functional
> (MAC) things.
>
> 1) 802.15.4. At the moment of writing there was no difference between
>     plain device and coordinator. (It was expected that the need to
>     differentiate between device and coordinator may arise in the future.
>     Compare to 802.11 AP and Mesh devices.)
>
> 2) Monitor. A device returning a frame with _all_headers to userspace.
>     W/o any processing. Again, compare to 802.11 stack - they added
>     monitor devices.

Differentiating between different roles of a device with different types
of interfaces is a sensible idea. I'm not actually opposed to that, quite
the contrary - I am mainly interested in a working and semantically
correct system.

As far as 802.15.4 is concerned, I do want to be able to support all kinds
of devices that make sense. It is not sensible though to add a monitor
device on a PHY that is already running a WPAN if that PHY does not
support concurrent monitors, so that should not be allowed for 802.15.4
just as it is not allowed for 802.11. For why I think it is not sensible
to allow multiple 802.15.4 slave devices on one PHY, see below.

>
> Left over or not implemented:
>
> 3) SMAC - Freescale's 'Simple MAC'. Broadcast frames, two-byte header
>     to identify SMAC, no addressing. Used mostly for their demonstrations
>     and for hobbyist simple devices. It is present in linux-wpan kernel
> and
>     I think it's worth adding it to mainline
>
> 4) MiWi. Microchip's 802.15.4 variant. While it shares lot's of commons
>     with main 802.15.4 standard, it has some simplifications in the
>     areas of address negotiation and association/removal. Not implemented,
>     but it might be worth looking into it - it is a 'baby 802.15.4',
> so implementing
>     it would also help main 802.15.4 stack.
>
> 5) MiWi P2P - Another fancy from Microchip. Don't remember the details.
>
> 6) _anything_ else. This band and these radio modems are used for lots
>     pf hobbyist devices. Moreover, the notion of 'WPAN device is a device
>     supporting transmission of frames <= 127 bytes with 2-byte CRC16
>     at the end of the frame' should allow implementing other fancy devices
>     through this stack. In Siemens we were contacted by a French company
>     that developed powerline devices. They are not 802.15.4 devices,
>     because standard defines no powerline phy (yet). Those guys were
>     interested in transmitting 802.15.4 frames on top of these devices.
>     Or a custom frames. Another example: 802.15.7 Visible Light
>     Communications look very similar to 802.15.4. Very but not equal.
>     So it is possible to support 'VLC' drivers on top of 'WPAN phy'.
>     The driver would share lot's of code with 802.15.4. And still
>     allow to support VLC peculiarities.

Those are interesting protocols to support with the current PHY
infrastructure, yes. MiWi might benefit greatly from sharing code with
mac802154, so there is reason to keep that code reasonably generic if
anyone ever wanted to implement WiMi, the same holds for VLC.

All of those protocols may very well share the PHY and parts of the code
with regular 802.15.4, but 802.15.4 and these protocols are all in some
way incompatible with each other. For what I can tell, SMAC uses the first
two bytes of a frame as data, where 802.15.4 expects a frame control word,
so those cannot work together on the same PHY without interfering with
each other. MiWi, as an extended subset of 802.15.4, probably also cannot
coexist with 802.15.4 or SMAC on the same PHY. Two protocols can only
reliably coexist if they were explicitly designed to do that, but I
couldn't confirm that what you listed was designed to.

So yes, it is sensible to allow different protocols on top of the PHY, but
not two different protocols at once, and also not two different instances
of the same protocol that interfere with each other - which, for the
transceivers that have drivers in the kernel now, are any two instances at
all. The transceivers we support are designed by their manufacturers to be
as cheap as possible while still allowing a full implementation of an
802.15.4 stack, which notable does not include a requirement for one
device to be accessible in different specific WPANs at once.

>
>
> Note. I did not say a word about ZigBee or 6lowpan. Those are next-level
> protocols. They are not to be implemented as wpan phy slaves. Instead
> they use low level protocols (802.15.4 in this case) and operate them
> through protocol-related interface (netlink + data), be it a userspace
> implementation or a kernel-space one.
>
>
> I think I have said enough to convince you (and other developers) that
> slaves
> are important part of the stack and phy <-> dev separation should be
> supported. Just stop living in 802.15.4-only world. This parts of universe
> centers around 802.15.4,  but is not limited to it.

If the stated goal of mac802154 is to support anything that uses a
802.15.4 PHY, or a similar PHY, then maybe the stack should renamed to
reflect that. Or it could be split into mac80215x and ieee802154, smac,
miwi ... A basic stack for just the PHY functions and one stack for each
protocol, where those stack could of course share code where appropriate.
Since nothing has made a move for mainline since the 802.15.4 PHY
infrastructure was added, I do believe that this complexity is not
warranted at the moment. That might change when those protocols wish to go
mainline, but right now the implementation in the kernel is an 802.15.4
stack, and the assumption that it is a pure 802.15.4 stack is also present
in the PHY drivers. In my opinion, we should stick to what we know -
802.15.4 - until the need to diversify arises.

As explained above, I am not opposed to the separation into PHY devices
and netdevs. The current mode of seperation is at odds with what can be
implemented in reality without breaking at least one assumption of the
netdev model though, so I think we should rethink at least the mode of
separation - the separation itself is a sensible thing.

>
> Regarding multiple 802.15.4 devices per single phy, switching channels,
> etc.
> At the time we did this, stack contained (and it still contains) areas
> requiring more attention. To make multidev configurations work safely
> one just needs to add several checks in the code and several error codes
> around.
>
> In fact (yes, I'm not joking) the ability to add several devices and to
> switch channels from each of the device was considered as a feature,
> not as a bug. If you did this, you certainly knew what you are doing.
> And why. So if the driver fails, it's your problem (note: fails, not
> crashes).
> On the other hand, it allows running several interesting experiments
> and to deploy interesting configurations.

I don't think it's quite as easy as that. Making multidev work safely in
all configurations requires a lot of compatibility checking between the
devices configured on the PHY, and indications from the PHY driver about
what the chip can safely support. For the drivers we have now and chips I
have seen so far, this indication would always be "one specific WPAN",
plus maybe a monitor of the network, reducing the possible slaves to one
specific WPAN instance and one monitor device.

I can also think of different ways to switch specific WPANs by only
sending a packet down a device that do not require different device
instances. One would be to export a netlink controller for a device, such
that one could open a netlink socket, bind it to the wpan device, and from
then on be able to change channel/page/PAN-ID/other parameters of only
that wpan device via this socket. Other options include (possibly
privileged) ancillary data messages for socket sendmsg() calls that would
have the same effect. With these options, you can still know what you are
doing and not have the driver or a netdev fail on you in a manner that
might be hard to debug. I cannot comment on the experiments and
interesting configurations you mentioned, but I feel that most or all of
these are specialized enough to not care whether the switching mechanism
is via a simple packet, via a netlink command, or via a packet with
ancillary data.

>
>> I thus propose to remove this feature entirely and instead have a 1:1
>> correspondence between PHYs and netdevs. If a driver can support
>> multiple specific WPANs per chip, it can register multiple PHYs. If
>> anybody actually needs to reconfigure the device to a different
>> specific WPAN, we already have an interface for that: the netlink API.
>> The netlink API also has the benefit that it does not impose arbitrary
>> delays before the switch, as switching by sending a packet does when
>> reordering/shaping Qdiscs are configured on the WPAN.
>
> It's your community-based choice now. However I'd suggest to leave
> that in place. See 802.11 stack - there are much more similarities
> that one would have expected.
>
> [skipped]
>
>> 4) our *drivers* are expected to *sleep*
>>
>> This is just wrong. Our RX/TX path invokes the scheduler at least once
>> for every packet on the assumption that drivers might sleep, which may
>> entail arbitrary delays even after the Qdiscs of the system have run.
>> Especially when somebody wanted to use latency sensitive Qdiscs, this
>> is a disaster, but also for general system throughput once you have a
>> certain number of packets going through your node.
>
> I'm not sure about RX path (I just don't remember details at this point).
> There are netif_rx and netif_rx_ni functions.
>
> For TX path things are really simple. ndo_start_xmit can not sleep.
> at86, serial - all require sleeping in the tx path. For at86:
> Change the state, wait for the change to finish, program the buffer,
> toggle the gpio. Lot's of waiting and sleeping there.
>
> It would be possible to move the schedule&company to the driver
> itself. I don't remember if we tried that or not. However we felt that
> it would be simpler to handle all scheduling inside mac80154 core.
>
> You can shift schedule_work & company to the final drivers,
> You won't get rid of them completely.

Lots of waiting is required, yes, but sleeping is not necessary for this
waiting to happen. For at86: a TX operation writes a frame to the device
frame buffer, changes state, checks that the state change has succeeded
and reports and error if not, and finally waits for the TRX_END interrupt
to signal completion of the Transmission and to change the device state
back to receive mode. Writing to the device can be done asynchronously
with the SPI infrastructure the kernel has, sequencing of these operations
operations up until the wait for completion can be one via the completion
callbacks the SPI core offers. When TRX_END has fired, or when an error
has happened, we can inform mac802154 via a callback of our own about the
outcome of the operation. If mac802154 has not indicated that it wants to
send another frame right away, we can then change the device state to RX
asynchronously while mac802154 processes the packet we just gave to it. No
sleeping is actually required for these operations to work, but it
admittedly does make programming a little easier, at the cost of increased
latencies and decreased throughput.

>
> --
> With best wishes
> Dmitry
>

Re: Simplifying the 802.15.4 stack

[Dmitry Eremin-Solenikov]

Helo,

On Sat, Mar 8, 2014 at 8:58 PM, Phoebe Buckheister
<phoebe.buckheister-mPn0NPGs4xGatNDF+KUbs4QuADTiUCJX@public.gmane.org> wrote:
> On Fri, March 7, 2014 11:16 pm, Dmitry Eremin-Solenikov wrote:
>> On Fri, Mar 7, 2014 at 4:16 PM, Phoebe Buckheister
>> <phoebe.buckheister-mPn0NPGs4xGatNDF+KUbs4QuADTiUCJX@public.gmane.org> wrote:

[skipped]

>>> 3) the mac802154_priv slave list
>>>
>>> This is one of the biggest problems I think the current stack has. For
>>> HardMAC, every netdevice corresponds to one specific WPAN (made up of
>>> PAN ID, channel, and page) and it is pretty much guaranteed that
>>> multiple HardMAC netdevs will always function as intended, even if they
>>> are backed by the same piece of hardware.
>>>
>>> SoftMAC, on the other hand, takes a single PHY chip and adds an
>>> *arbitrary* number of specific WPANs on top of that PHY. No real
>>> hardware can support an arbitrary number of specific WPANs, and no chip
>>> I have yet come across can even support more than one. As far as I've
>>> been told and was able to reconstruct from documentation and standards
>>> documents, this was implemented to allow for ZigBee frequency agility,
>>> but that was never implemented. What we have, though, is a means for
>>> unprivileged users that can run socket(), write() and ip(8) to
>>> effectively DoS a node that uses this feature.
>>
>> This was done to support different kinds of devices accessing the radio.
>> Compare to 802.11 stack, which has phy & device conception.
>> It really helps to separate between the radio-related parts and functional
>> (MAC) things.
>>
>> 1) 802.15.4. At the moment of writing there was no difference between
>>     plain device and coordinator. (It was expected that the need to
>>     differentiate between device and coordinator may arise in the future.
>>     Compare to 802.11 AP and Mesh devices.)
>>
>> 2) Monitor. A device returning a frame with _all_headers to userspace.
>>     W/o any processing. Again, compare to 802.11 stack - they added
>>     monitor devices.
>
> Differentiating between different roles of a device with different types
> of interfaces is a sensible idea. I'm not actually opposed to that, quite
> the contrary - I am mainly interested in a working and semantically
> correct system.
>
> As far as 802.15.4 is concerned, I do want to be able to support all kinds
> of devices that make sense. It is not sensible though to add a monitor
> device on a PHY that is already running a WPAN if that PHY does not
> support concurrent monitors, so that should not be allowed for 802.15.4
> just as it is not allowed for 802.11. For why I think it is not sensible
> to allow multiple 802.15.4 slave devices on one PHY, see below.

If you wish to disallow multiple 802.15.4 devices on a single phy, just do it.

However before removing the rest of the phy/device API, please, Please,
PLEASE, go and consult the 802.11 separation in Linux kernel. This largest
part of slave support code was modeled after 802.11 stack. This is not about
the roles of the device - it is about the multiple roles being
attached to the device.
For example monitor mode would be hard to support on plain 802.15.4 - one
can expect that mac802.15.4 will change the skb. Especially after
adding security/encryption. Monitor (wireshark/whatever) must receive
skbs before all the rest of the handlers. It is not directly possible with
other architectures.

>> Left over or not implemented:
>>
>> 3) SMAC - Freescale's 'Simple MAC'. Broadcast frames, two-byte header
>>     to identify SMAC, no addressing. Used mostly for their demonstrations
>>     and for hobbyist simple devices. It is present in linux-wpan kernel
>> and
>>     I think it's worth adding it to mainline

If I have time, I'll update the code and send it to review & merge. This would
be a good example of supporting other protocols on top of 802.15.4 PHY in
parallel to the  802.15.4 MAC.

[skipped regarding miwi]

> Those are interesting protocols to support with the current PHY
> infrastructure, yes. MiWi might benefit greatly from sharing code with
> mac802154, so there is reason to keep that code reasonably generic if
> anyone ever wanted to implement WiMi, the same holds for VLC.
>
> All of those protocols may very well share the PHY and parts of the code
> with regular 802.15.4, but 802.15.4 and these protocols are all in some
> way incompatible with each other. For what I can tell, SMAC uses the first
> two bytes of a frame as data, where 802.15.4 expects a frame control word,
> so those cannot work together on the same PHY without interfering with
> each other. MiWi, as an extended subset of 802.15.4, probably also cannot
> coexist with 802.15.4 or SMAC on the same PHY. Two protocols can only
> reliably coexist if they were explicitly designed to do that, but I
> couldn't confirm that what you listed was designed to.

This is up to the protocol and application to decide, whether it can parse
packets or it can ignore them. Compare this to having IPv4, IPv6, IPX
and SMB (NetBEUI or whatever that was called) packets on top of
a wire. It is up to the protocol to decide, whether it can parse the
packet and whether the packet looks good enough to be passed
to upper layers.

> So yes, it is sensible to allow different protocols on top of the PHY, but
> not two different protocols at once, and also not two different instances
> of the same protocol that interfere with each other - which, for the
> transceivers that have drivers in the kernel now, are any two instances at
> all. The transceivers we support are designed by their manufacturers to be
> as cheap as possible while still allowing a full implementation of an
> 802.15.4 stack, which notable does not include a requirement for one
> device to be accessible in different specific WPANs at once.

And this is to the driver to decide. I don't remember, which got to the kernel.
My original code allowed the upper layers and protocol to agree on what
is allowed and what is not. It is the driver, who knows what is currently
enabled (address filtering, auto-ack, promisc, etc). Mac802.15.4 should
not impose additional restrictions on top of that.


> If the stated goal of mac802154 is to support anything that uses a
> 802.15.4 PHY, or a similar PHY, then maybe the stack should renamed to
> reflect that. Or it could be split into mac80215x and ieee802154, smac,
> miwi ... A basic stack for just the PHY functions and one stack for each
> protocol, where those stack could of course share code where appropriate.
> Since nothing has made a move for mainline since the 802.15.4 PHY
> infrastructure was added, I do believe that this complexity is not
> warranted at the moment. That might change when those protocols wish to go
> mainline, but right now the implementation in the kernel is an 802.15.4
> stack, and the assumption that it is a pure 802.15.4 stack is also present
> in the PHY drivers. In my opinion, we should stick to what we know -
> 802.15.4 - until the need to diversify arises.

PHY does/should not depend on having only 802.15.4. Monitoring and SMAC
worked w/o any issues. In fact this was the original cause for reduced and full
MLME ops, not that RFD/FFD crap, that is currently written in the comment
in ieee802154_netdev.h file.

> As explained above, I am not opposed to the separation into PHY devices
> and netdevs. The current mode of seperation is at odds with what can be
> implemented in reality without breaking at least one assumption of the
> netdev model though, so I think we should rethink at least the mode of
> separation - the separation itself is a sensible thing.

I'm open to any real suggestions. But please check the 80211 kernel code
first.

[skipped]

>>> 4) our *drivers* are expected to *sleep*
>>>
>>> This is just wrong. Our RX/TX path invokes the scheduler at least once
>>> for every packet on the assumption that drivers might sleep, which may
>>> entail arbitrary delays even after the Qdiscs of the system have run.
>>> Especially when somebody wanted to use latency sensitive Qdiscs, this
>>> is a disaster, but also for general system throughput once you have a
>>> certain number of packets going through your node.
>>
>> I'm not sure about RX path (I just don't remember details at this point).
>> There are netif_rx and netif_rx_ni functions.
>>
>> For TX path things are really simple. ndo_start_xmit can not sleep.
>> at86, serial - all require sleeping in the tx path. For at86:
>> Change the state, wait for the change to finish, program the buffer,
>> toggle the gpio. Lot's of waiting and sleeping there.
>>
>> It would be possible to move the schedule&company to the driver
>> itself. I don't remember if we tried that or not. However we felt that
>> it would be simpler to handle all scheduling inside mac80154 core.
>>
>> You can shift schedule_work & company to the final drivers,
>> You won't get rid of them completely.
>
> Lots of waiting is required, yes, but sleeping is not necessary for this
> waiting to happen. For at86: a TX operation writes a frame to the device
> frame buffer, changes state, checks that the state change has succeeded
> and reports and error if not, and finally waits for the TRX_END interrupt
> to signal completion of the Transmission and to change the device state
> back to receive mode. Writing to the device can be done asynchronously
> with the SPI infrastructure the kernel has, sequencing of these operations
> operations up until the wait for completion can be one via the completion
> callbacks the SPI core offers. When TRX_END has fired, or when an error
> has happened, we can inform mac802154 via a callback of our own about the
> outcome of the operation. If mac802154 has not indicated that it wants to
> send another frame right away, we can then change the device state to RX
> asynchronously while mac802154 processes the packet we just gave to it. No
> sleeping is actually required for these operations to work, but it
> admittedly does make programming a little easier, at the cost of increased
> latencies and decreased throughput.

First, just to make sure that you understand it. 'a TX operation writes a frame
to the device frame buffer, changes state, checks that the state
change has succeeded'.
This part already includes sleeping a lot. One can change the drivers
code to do all the job on asynchronous callbacks or in the special
driver-local worker. But you still face the issue. Again, send the patches
on how to improve the situation.


-- 
With best wishes
Dmitry

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech

Re: Simplifying the 802.15.4 stack

[Phoebe Buckheister]

On Tue, 11 Mar 2014 16:30:12 +0400
Dmitry Eremin-Solenikov <dbaryshkov@gmail.com> wrote:

> Helo,
[…]
> > Differentiating between different roles of a device with different
> > types of interfaces is a sensible idea. I'm not actually opposed to
> > that, quite the contrary - I am mainly interested in a working and
> > semantically correct system.
> >
> > As far as 802.15.4 is concerned, I do want to be able to support
> > all kinds of devices that make sense. It is not sensible though to
> > add a monitor device on a PHY that is already running a WPAN if
> > that PHY does not support concurrent monitors, so that should not
> > be allowed for 802.15.4 just as it is not allowed for 802.11. For
> > why I think it is not sensible to allow multiple 802.15.4 slave
> > devices on one PHY, see below.
> 
> If you wish to disallow multiple 802.15.4 devices on a single phy,
> just do it.
> 
> However before removing the rest of the phy/device API, please,
> Please, PLEASE, go and consult the 802.11 separation in Linux kernel.
> This largest part of slave support code was modeled after 802.11
> stack. This is not about the roles of the device - it is about the
> multiple roles being attached to the device.
> For example monitor mode would be hard to support on plain 802.15.4 -
> one can expect that mac802.15.4 will change the skb. Especially after
> adding security/encryption. Monitor (wireshark/whatever) must receive
> skbs before all the rest of the handlers. It is not directly possible
> with other architectures.

Removing the rest of the phy/device API is not my intention. The
separation between PHY and device was something I never wanted to
remove, though I did want to remove the ability to add multiple
devices/slaves to the same PHY. You've convinced me that that's a
really bad idea, but you've also convinced me that another kind of
restriction would be well in order: allow an arbitrary number of
low-level monitor devices, if the PHY supports that functionality, and
exactly one protocol implementation. For 802.15.4 WPAN, I think that
implementation should restrict itself to one netdev and an additional
list of slave roles as necessary, e.g. coordinator functions that are
only accessible through netlink.

[…]
> > Those are interesting protocols to support with the current PHY
> > infrastructure, yes. MiWi might benefit greatly from sharing code
> > with mac802154, so there is reason to keep that code reasonably
> > generic if anyone ever wanted to implement WiMi, the same holds for
> > VLC.
> >
> > All of those protocols may very well share the PHY and parts of the
> > code with regular 802.15.4, but 802.15.4 and these protocols are
> > all in some way incompatible with each other. For what I can tell,
> > SMAC uses the first two bytes of a frame as data, where 802.15.4
> > expects a frame control word, so those cannot work together on the
> > same PHY without interfering with each other. MiWi, as an extended
> > subset of 802.15.4, probably also cannot coexist with 802.15.4 or
> > SMAC on the same PHY. Two protocols can only reliably coexist if
> > they were explicitly designed to do that, but I couldn't confirm
> > that what you listed was designed to.
> 
> This is up to the protocol and application to decide, whether it can
> parse packets or it can ignore them. Compare this to having IPv4,
> IPv6, IPX and SMB (NetBEUI or whatever that was called) packets on
> top of a wire. It is up to the protocol to decide, whether it can
> parse the packet and whether the packet looks good enough to be passed
> to upper layers.

Usually, link layers support some kind of upper layer protocol type
indicator. Ethernet has the ethertype field, for example, so which
protocol payload a packet contains is described by the packet itself. On
802.15.4 PHYs, we don't usually have that - SMAC is just a pipe into
the air, 802.15.4 WPAN supports no protocol indication, and MiWi or
others probably do not either. As such, which protocol should be used
to process is a packet is not a property of the packet, but a property
of the network the PHY is in - barring interferers on the same channel
that send other things. On 802.15.4 WPANs, those interferers can be
easily ignored by switching to a different PAN and ignoring broadcast
frames from other PANs, but other protocols might not make this as easy.

Say, you have an SMAC device and a 802.15.4 WPAN on the same PHY, and
two machines with this setup. One machine sends an SMAC frame that just
so happens to look like an 802.15.4 WPAN dissasociation request, and
the other machine's 802.15.4 WPAN implementation might happily process
it as such even though it was never meant to. We could of course tell
SMAC to "just don't do that", but then that's not really SMAC anymore
and also not what an SMAC user would expect.

Our situation here, I think, is much more akin to an RS485 network or
similar networks that provide only raw, unadulterated bytestream
transport. For SMAC certainly, 802.15.4 WPAN does impose some
restrictions on this with the PAN concept and explicit non-broadcast
frames to separate different entities, but the protocol contained in a
frame taken from the PHY is still very much dependent on the network
itself. It will continue to be hard to have multiple protocols on the
same channel even with the restriction to one protocol implementation
per PHY, but that's already hard to impossible now - all depending on
the other protocols present on the channel.

> > So yes, it is sensible to allow different protocols on top of the
> > PHY, but not two different protocols at once, and also not two
> > different instances of the same protocol that interfere with each
> > other - which, for the transceivers that have drivers in the kernel
> > now, are any two instances at all. The transceivers we support are
> > designed by their manufacturers to be as cheap as possible while
> > still allowing a full implementation of an 802.15.4 stack, which
> > notable does not include a requirement for one device to be
> > accessible in different specific WPANs at once.
> 
> And this is to the driver to decide. I don't remember, which got to
> the kernel. My original code allowed the upper layers and protocol to
> agree on what is allowed and what is not. It is the driver, who knows
> what is currently enabled (address filtering, auto-ack, promisc,
> etc). Mac802.15.4 should not impose additional restrictions on top of
> that.

Yes, but it also shouldn't allow two slaves on the same PHY with
differing views of reality. In the best case, one slave will not be
serviced as expected, in the worst case, none of the slaves will be
serviced at all.

> > If the stated goal of mac802154 is to support anything that uses a
> > 802.15.4 PHY, or a similar PHY, then maybe the stack should renamed
> > to reflect that. Or it could be split into mac80215x and
> > ieee802154, smac, miwi ... A basic stack for just the PHY functions
> > and one stack for each protocol, where those stack could of course
> > share code where appropriate. Since nothing has made a move for
> > mainline since the 802.15.4 PHY infrastructure was added, I do
> > believe that this complexity is not warranted at the moment. That
> > might change when those protocols wish to go mainline, but right
> > now the implementation in the kernel is an 802.15.4 stack, and the
> > assumption that it is a pure 802.15.4 stack is also present in the
> > PHY drivers. In my opinion, we should stick to what we know -
> > 802.15.4 - until the need to diversify arises.
> 
> PHY does/should not depend on having only 802.15.4. Monitoring and
> SMAC worked w/o any issues. In fact this was the original cause for
> reduced and full MLME ops, not that RFD/FFD crap, that is currently
> written in the comment in ieee802154_netdev.h file.

It does not, that's correct, but activating even one 802.15.4 WPAN on a
PHY that supports address filtering will effectively lock out
everything else on the same PHY, including SMAC. Letting the PHY
perform only the lowest common denominator of all that is requested by
slaves will also not work, because we cannot possibly implement even
the ACK mechanism of 802.15.4 WPAN in the current stack without PHY
support. That's not only because we use workqueues for RX, but also
because some commonly SPI drivers use workqueues for their internals.
On a 100kbps OQPSK network, we have 12 symbol periods before an ACK for
a frame has to be sent, or about 0.5 milliseconds. Including the SPI
transfers required, there's just no way to keep that deadline.

> > As explained above, I am not opposed to the separation into PHY
> > devices and netdevs. The current mode of seperation is at odds with
> > what can be implemented in reality without breaking at least one
> > assumption of the netdev model though, so I think we should rethink
> > at least the mode of separation - the separation itself is a
> > sensible thing.
> 
> I'm open to any real suggestions. But please check the 80211 kernel
> code first.

 * split the slave list into a list of monitors and a protocol impl
 * have each protocol that wants to run on the phy provide a protocol
   impl, akin to packet_type mechanism of the netdev stack
 * protocol impls are free to do whatever they wish. For 802.15.4 WPAN,
   I'd wish the protocol impl to allow only one netdev and some set of
   other roles, as outlined above. That'd be one active slave that acts
   as a network device, and a number of other slaves that don't - but
   still process and transmit packets.

[…]
> First, just to make sure that you understand it. 'a TX operation
> writes a frame to the device frame buffer, changes state, checks that
> the state change has succeeded'.
> This part already includes sleeping a lot. One can change the drivers
> code to do all the job on asynchronous callbacks or in the special
> driver-local worker. But you still face the issue. Again, send the
> patches on how to improve the situation.

I would differentiate between sleeping (telling the scheduler to
ignore the current thread until some condition holds) and waiting
(telling an actor to provide asynchronous indication of completion).
The underlying actor may of course sleep, as some SPI drivers do, but
that doesn't force any other user to sleep as well.

Once I have some patches to show what I mean, I'll submit them. There
are other things I have to take of first, though, so this might not
happen for another few weeks.

------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/13534_NeoTech
_______________________________________________
Linux-zigbee-devel mailing list
Linux-zigbee-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-zigbee-devel