gsmtap design/extensions?

gsmtap design/extensions?

[Johannes Berg]

Hi,

As I'm looking into adding a generic cell modem framework to the linux
kernel (to create session netdevs etc.), I started looking for a
metadata encapsulation, a la Radiotap (I'm a wifi guy :-) ).

So obviously, I found gsmtap, but for my use case it doesn't really
address most of the interesting data, and it got me wondering. So a few
questions, if I may:

1) Why the design with encapsulating it in UDP? Radiotap is just a raw
   header without IP etc. in front, and you use it with tcpdump,
   wireshark or similar tools on the local system. What's the value in
   having something "network transparent"?

2) The format of gsmtap doesn't seem very extensible, but I guess a new
   version could be made that has a TLV-based format or so. I'd have
   argued that a new version isn't even needed, but the length field is
   only 8 bits right now which seems too short.

(speaking of versions - the docs say "version, set to 0x01 currently"
but "#define GSMTAP_VERSION 0x02")

3) Does the packet data follow the gsmtap header? It's not really clear
   to me based on reading the wireshark code.


In particular, the data I'm thinking of is higher-level things, like the
session ID for a frame when it's going through the kernel, or perhaps a
flow label on RX, etc.

Also, vendor-specific data would be useful, e.g. to encapsulate the
device-specific headers like QMI, where such metadata is encapsulated in
a vendor- or device-specific way, which you'd want to see for debugging
certain things, but for other things the generic "session ID" type
information - encoded in a vendor-agnostic way - would be better to show
in wireshark.


Since it doesn't seem possible to use gsmtap in the current version,
would it make sense to define a new gsmtap that (say) has version 3 or
something, followed by an overall length and TLVs? I do note that this
wouldn't be compatible with the current wireshark code as it doesn't
check the version, just shows it...

Or would it make more sense to define a new ARPHDR_WWANTAP like
ARPHDR_IEEE80211_RADIOTAP and just use that instead of encapsulating in
IP/UDP, and then have a completely new (extensible) protocol inside of
that? I'm not really sure I see the point of UDP encapsulation anyway.

Thanks,
johannes

Re: gsmtap design/extensions?

[Vadim Yanitskiy]

Hello Johannes,

FYI, there already was a discussion about GSMTAPv3:

https://www.youtube.com/watch?v=vum9jzavZi0&list=PL07C78AF831FFE8F9&index=10

but unfortunately, nobody has invested time into this (yet?).

> 1) Why the design with encapsulating it in UDP?

This gives us a possibility to "demux" multiple GSMTAP streams on the
receiving side, e.g. if you are running multiple processes.

> 2) The format of gsmtap doesn't seem very extensible [...]

ACK. I definitely support the idea of using TLVs.

With best regards,
Vadim Yanitskiy.

Re: gsmtap design/extensions?

[Johannes Berg]

On Wed, 2019-04-10 at 13:35 +0700, Vadim Yanitskiy wrote:
> Hello Johannes,
> 
> FYI, there already was a discussion about GSMTAPv3:
> 
> https://www.youtube.com/watch?v=vum9jzavZi0&list=PL07C78AF831FFE8F9&index=10
> 
> but unfortunately, nobody has invested time into this (yet?).

2012! But, umm, I don't really have time for a whole video right now -
anyone have the slides? :-)

But yeah, the first slides look sensible :-)

> > 1) Why the design with encapsulating it in UDP?
> 
> This gives us a possibility to "demux" multiple GSMTAP streams on the
> receiving side, e.g. if you are running multiple processes.

Not sure I get this, but I also don't really care all that much. It's
just a pretty strange design if the kernel were to output this, I'm not
even sure how I'd do that properly. I don't want to be generating UDP
packets there...

Perhaps we can define something (GSMTAPv3) to not really care how it's
encapsulated, and for 'native' packet captures like what I want on Linux
when integrated with the driver, actually use an ARPHDR_GSMTAP, and
encapsulate in UDP when you create it in an application and want to send
it elsewhere, rather than just writing it to a pcap file?

johannes

Re: gsmtap design/extensions?

[Marcel Holtmann]

Hi Johannes,

>> FYI, there already was a discussion about GSMTAPv3:
>> 
>> https://www.youtube.com/watch?v=vum9jzavZi0&list=PL07C78AF831FFE8F9&index=10
>> 
>> but unfortunately, nobody has invested time into this (yet?).
> 
> 2012! But, umm, I don't really have time for a whole video right now -
> anyone have the slides? :-)
> 
> But yeah, the first slides look sensible :-)
> 
>>> 1) Why the design with encapsulating it in UDP?
>> 
>> This gives us a possibility to "demux" multiple GSMTAP streams on the
>> receiving side, e.g. if you are running multiple processes.
> 
> Not sure I get this, but I also don't really care all that much. It's
> just a pretty strange design if the kernel were to output this, I'm not
> even sure how I'd do that properly. I don't want to be generating UDP
> packets there...
> 
> Perhaps we can define something (GSMTAPv3) to not really care how it's
> encapsulated, and for 'native' packet captures like what I want on Linux
> when integrated with the driver, actually use an ARPHDR_GSMTAP, and
> encapsulate in UDP when you create it in an application and want to send
> it elsewhere, rather than just writing it to a pcap file?

before you go all out and define this, it would suggest to understand what meta-data for the connection contexts you actually need as well. The data path itself is just a pipe and has not all the information attached with it. That goes via the control path and that is normally in user space and carries the real important information to make useful analysis of how the data path / context is setup.

From what I am seeing right now is that unless you have a method to also feed the control path into your GSMTAPv3, then this is rather useless. The majority of the debugging is really done for the control path. For oFono that is OFONO_DEBUG=1 environment variable and while it works it is not the most elegant solution. I would love to feed that into a generic debugging / tap that you can read out later.

As a side note, for Bluetooth we created a path where the bluetoothd can feed back its control debugging data back into the Bluetooth monitor in the kernel to allow combined userspace, mgmt and HCI tracing. Some really nasty issues could only be triaged by having all the meta data with a common timestamp.

Regards

Marcel

Re: gsmtap design/extensions?

[Harald Welte]

Hi Johannes,

On Tue, Apr 09, 2019 at 03:50:45PM +0200, Johannes Berg wrote:
> As I'm looking into adding a generic cell modem framework to the linux
> kernel (to create session netdevs etc.), I started looking for a
> metadata encapsulation, a la Radiotap (I'm a wifi guy :-) ).

Is there any discussion about "session netdevs, etc." anywhere?  What exactly
do you have in mind for that "generic cell modem framework"?  I'm quite
curious to learn more about it and happy to provide feedback from the
perspective of my cellular protocols/specs/systems knowledge.

> So obviously, I found gsmtap, but for my use case it doesn't really
> address most of the interesting data, 

I have no knowledge of your use case, so I don't feel like I can
comment on that :/

> 1) Why the design with encapsulating it in UDP? Radiotap is just a raw
>    header without IP etc. in front, and you use it with tcpdump,
>    wireshark or similar tools on the local system. What's the value in
>    having something "network transparent"?

GSMTAP was designed as a format to encapsulate protocols normally not spoken over IP
(such as classic GSM radio protocols, e.g. the Layer 2 LAPDm over GSM Um)
inside an IP transport.

The existing implementations of such protocols all live outside of the
kernel but in userspace.  So you either have

a) a pure SDR software implementation of a GSM phy, whether on the MS/UE
   side or on the BTS/network side, or whether in sniffing mode, running
   as a userspace process.  This could e.g. be airprobe or gr-gsm

b) a OsmocomBB phone attached over serial port running the Osmocom Layer1
   firmware on the baseband processor, with some userspace program on a Linux
   PC implementing higher layers

c) GSM base station software such as osmo-bts, running in userspace

d) Using an old nokia phone attached via serial cable to a Linux PC running
   dct3-gsmtap.

So for any of those, if you want to provide real-time data streams, you have to
somehow transmit those over some kind of network technology.   One could have
done raw ethernet frames or raw IP frames, but using UDP seemed straight-forward
as one can create UDP socket from non-privileged processes.

We also have a virtual physical layer between OsmocomBB and OsmoBTS called
virt_phy where the GSMTAP over multicast IP is used to simulate/virtualize
the entire Layer1 / PHY / radio interface between phone and base station.

Once again, all related network elements are implemented in userspace,
and having an 

> 2) The format of gsmtap doesn't seem very extensible, but I guess a new
>    version could be made that has a TLV-based format or so. I'd have
>    argued that a new version isn't even needed, but the length field is
>    only 8 bits right now which seems too short.

Yes, it's a known problem.  The format was originally designed for GSM,
that is circuit switched common and dedicated channels on the GSM Um interface.
It was later extended for GPRS, TETRA, SIM card protocol traces and many others,
but all of that was an afterthought.

For sure any future version should be more extensible and e.g. use TLVs

> (speaking of versions - the docs say "version, set to 0x01 currently"
> but "#define GSMTAP_VERSION 0x02")

Sorry, it's the usual "developer changes code but not comment". patches
are welcome, we use gerrit.osmocom.org :)

> 3) Does the packet data follow the gsmtap header? It's not really clear
>    to me based on reading the wireshark code.

Sure, the packet data follows the GSMTAP header, and the type of data
is defined by the GSMTAP type / sub-type as per the specific use case.  As
you can see, there's 18 TYPE by now (each of which has at least one
program/implementation generating the data).

I think the best way to learn about GSMTAP is to use any of the many
programs that support it by now.  I think not only the examples above
use it, but meanwhile many others like the independently-developed OpenBTS
project that's unrelated to Osmocom, as are other projects implementing LTE.

> In particular, the data I'm thinking of is higher-level things, like the
> session ID for a frame when it's going through the kernel, or perhaps a
> flow label on RX, etc.

I'm not quite sure how that relates to GSM?  GSMTAP so far was intended
to encapsulate ETSI/3GPP messages inside UDP/IP, particularly messages of
protocols that don't traditionally are transported over IP baesd transports.

Having said, we're open to extending the scope - it just all needs to make
sense

> Also, vendor-specific data would be useful, e.g. to encapsulate the
> device-specific headers like QMI, where such metadata is encapsulated in
> a vendor- or device-specific way, which you'd want to see for debugging
> certain things, but for other things the generic "session ID" type
> information - encoded in a vendor-agnostic way - would be better to show
> in wireshark.

I'm really not following you here.  What's a "session ID"?

In terms of vendor-specific encapsulations: We do have a GSMTAP sub-type
for Qualcomm DIAG, as well as the osmo-qcdiag program to take DIAG from
Qualcomm chips + put it in GSMTAP and I do have plenty of experimental
wireshark dissectors for various parts of the DIAG protocl in a branch
on git.osmocom.org.  This just never went anywhere complete enough that
I'd consider merging it.

> Since it doesn't seem possible to use gsmtap in the current version,
> would it make sense to define a new gsmtap that (say) has version 3 or
> something, followed by an overall length and TLVs? 

By all means.  Vadim has once presented some ideas/plans about exactly
that at an Osmocom conference, but I don't think any related work
ever started.

> I do note that this
> wouldn't be compatible with the current wireshark code as it doesn't
> check the version, just shows it...

If that's the case that's sad, and I should have paid more attention to
it when originally writing it.  We should get a related fix into
wireshark ASAP then.  But then, the current dissector would just state
something like unsupported version instead of showing some garbage it
cannot parse.  Either way, you will of course need new sources and
sink side implemetations.

> Or would it make more sense to define a new ARPHDR_WWANTAP like
> ARPHDR_IEEE80211_RADIOTAP and just use that instead of encapsulating in
> IP/UDP, and then have a completely new (extensible) protocol inside of
> that? 

No userspace source would ever be able to generate such data and stream
it real-time into wireshark, would it?  Sure, I can write pcap file with
such ARPHDR_* values, but I could never do this in real-time.  For many
but not all use cases, that's really what it is: A vehicle to stream
real-time non-IP protocol traces into wireshark so it can visualize
the protocol traces.

Regards,
	Harald
-- 
- Harald Welte <laforge-TgoAw6mPHtdg9hUCZPvPmw@public.gmane.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: gsmtap design/extensions?

[Harald Welte]

Hi Johannes,

On Wed, Apr 10, 2019 at 09:23:13AM +0200, Johannes Berg wrote:
> > but unfortunately, nobody has invested time into this (yet?).
> 
> 2012! 

Well, Osmocom is a very small community, with probably somewhere less than
25 active developers over the last few years (less than 15 full-time),
with an *incredibly* large scope:  Implement virtually any protocol
layer of any protocol stack on any of the 3GPP interfaces and all their
related network elements for 2G/3G as well as even other technologies
like TETRA, GMR-1, ...

And all that in a field of technology that has less free software than
the Operating Systems world had in the mid-1990ies.  It really feels a
bit like the Linux community 20 years ago.

So resources are always *extremely* tight, and given those limited
resources, I'm actually very happy with the results by now, having
automatied CI, build verifications, unit tests, functional test suites,
end-to-end testing, and all the code we implemented on git.osmocom.org :)

While current GSMTAPv2 is ugly, it works rather solid for all known
existing use cases, so there was no urgency to introduce a new version
of it.

> Not sure I get this, but I also don't really care all that much. 

Well, with all respect, GSMTAP was created for a variety of use cases,
see my other lengthy mail.  It's fine if you don't care, but unless you
could explain your use cases with a few paragraphs, neither you nor us
are able to determine if there is common functionality and if it makes
sense to use GSMTAP or not :)

So far I have not seen any explanation about what kind of data you want
to encapsulate at all.

> just a pretty strange design if the kernel were to output this, I'm not
> even sure how I'd do that properly. I don't want to be generating UDP
> packets there...

There are well-established APIs for having sockets in the kernel and for
generating + receiving UDP packets from it.  NFS has been doing this for
decades, as do various kernel-side tunneling helpers including the GTP
kernel module.

I'm not saying it's the right approach for your problem, I'm just saying
kernel-side code can for sure use UDP sockets.

> Perhaps we can define something (GSMTAPv3) to not really care how it's
> encapsulated, and for 'native' packet captures like what I want on Linux
> when integrated with the driver, actually use an ARPHDR_GSMTAP, and
> encapsulate in UDP when you create it in an application and want to send
> it elsewhere, rather than just writing it to a pcap file?

Sure, that works.  But the real question is, to me:  Are there common
GSMTAP payload types that both the existing GSMTAP users carry, as well
as what you would want to carry?  If yes, then it makes sense to think
about a common encapsulation like GSMTAP.  If the payload types differ,
then it seems rather like there are two distinct use cases that
wouldn't benefit from standardizing on one format.

Regards,
	Harald
-- 
- Harald Welte <laforge-TgoAw6mPHtdg9hUCZPvPmw@public.gmane.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: gsmtap design/extensions?

[Johannes Berg]

Hi Marcel,

> before you go all out and define this, it would suggest to understand
> what meta-data for the connection contexts you actually need as well.
> The data path itself is just a pipe and has not all the information
> attached with it. That goes via the control path and that is normally
> in user space and carries the real important information to make
> useful analysis of how the data path / context is setup.

Yes, that's true, though the control path is actually going through one
of the data pipes as well.

> From what I am seeing right now is that unless you have a method to
> also feed the control path into your GSMTAPv3, then this is rather
> useless. 

So the control path *itself* would be there, I guess, but ...

> The majority of the debugging is really done for the control path. For
> oFono that is OFONO_DEBUG=1 environment variable and while it works it
> is not the most elegant solution. I would love to feed that into a
> generic debugging / tap that you can read out later.

there's definitely room for more information than _just_ the control
path "chat", also application state etc. would be useful, and logging
etc.

Typically on wifi we feed all of this together into kernel tracing (to
record with trace-cmd) rather than trying to encapsulate it some other
way.

> As a side note, for Bluetooth we created a path where the bluetoothd
> can feed back its control debugging data back into the Bluetooth
> monitor in the kernel to allow combined userspace, mgmt and HCI
> tracing. Some really nasty issues could only be triaged by having all
> the meta data with a common timestamp.

Right. This is something we'd typically use tracing for in wifi.

I don't really know what the right model for WWAN would be, I guess.


Right now - and I really should've said this before - really the only
problem I was thinking of was how we can mux multiple "chat" sessions
with a device into a single data stream.

Currently, this is all vendor-specific. If you have a Qualcomm modem,
you'd be able to see all the open sessions on the underlying netdev, and
the QMI header tells you what session a given 'packet' belongs to, and
if you follow along maybe you can figure out if this is a control or IP
'packet' (could be an AT command).

cdc_mbim uses VLAN tags instead to achieve this, and decapsulates the
VLAN tags to send them down to the hardware in a different way.

There are a few reasons why I think that this model of having a single
underlying netdev controlled by the modem driver, with additional
netdevs layered on top in software will not work right in the future. I
think drivers should and will need to migrate to creating "real" netdevs
for the sessions instead of pure software ones.

But if you do this, you lose the ability to listen to all the session
streams at the same time, you can only do it for each netdev. Adding
this ability back seems worthwhile, but then we probably shouldn't do it
in a vendor-specific way, but rather in a generic way.

So basically right now that's all I'm trying to solve. In WiFi we don't
have the problem of "sessions" because we just use the addresses in the
frames to disambiguate - on such 'monitor' netdevs we see the frames
including full 802.11 headers.

Now, here's maybe where I'm getting off the right path - in wifi we
mostly couple that with PHY information as well, and so we have PHY
information + full 802.11 headers + data for capturing what's going on.
I figured that theoretically at least that would be possible/useful for
the modem as well (obviously control packets have no PHY data), but
there doesn't seem to be any hardware that would actually expose data in
this way. Everyone I've spoken to says these things are only available
as modem trace data. What I haven't figured out though is if that's by
some other design trade-off, or just because no such infrastructure
is/was available.

It may well be that doing kernel-tracing instead of doing it via some
kind of monitor netdev is perfectly sufficient, but I have a feeling
that the relative simplicity of starting tcpdump/wireshark might be
preferable.

johannes

Re: gsmtap design/extensions?

[Johannes Berg]

On Thu, 2019-04-11 at 01:32 +0200, Harald Welte wrote:
> Hi Johannes,
> 
> On Tue, Apr 09, 2019 at 03:50:45PM +0200, Johannes Berg wrote:
> > As I'm looking into adding a generic cell modem framework to the linux
> > kernel (to create session netdevs etc.), I started looking for a
> > metadata encapsulation, a la Radiotap (I'm a wifi guy :-) ).
> 
> Is there any discussion about "session netdevs, etc." anywhere?  What exactly
> do you have in mind for that "generic cell modem framework"?  I'm quite
> curious to learn more about it and happy to provide feedback from the
> perspective of my cellular protocols/specs/systems knowledge.

Yeah. I was too much in my little bubble when I wrote this. I just wrote
a lengthy reply to Marcel that explains a little more, but basically to
summarize:

Kernel drivers need to transport multiple data streams, like VoLTE (IP)
data, IP data, and control data. Currently, they do this in various ways
(cdc_mbim uses VLAN tags, rmnet/qmi_wwan uses Qualcomm QMI headers),
when you capture data you have to understand what you're dealing with
and maybe extend wireshark to even understand these headers.

I'm looking for a general vendor-agnostic encapsulation that would solve
this part of the problem.

At the same time, I come from WiFi, where - while we don't have these
sessions as such but can have different virtual interfaces as well, e.g.
for connections to different APs - it's often very useful to also
understand the PHY parameters that were used for TX/seen on RX. Now, I
realize this isn't something that modems will do right now - they just
don't seem to tell you any lower-level information on RX of a packet,
but it seemed like it might be worthwhile to not *exclude* this
possibility either.

> > 1) Why the design with encapsulating it in UDP? Radiotap is just a raw
> >    header without IP etc. in front, and you use it with tcpdump,
> >    wireshark or similar tools on the local system. What's the value in
> >    having something "network transparent"?
> 
> GSMTAP was designed as a format to encapsulate protocols normally not spoken over IP
> (such as classic GSM radio protocols, e.g. the Layer 2 LAPDm over GSM Um)
> inside an IP transport.

Sure, but wifi is also not spoken over IP, yet we don't encapsulate our
802.11 frames into IP to show them in wireshark :-)

> The existing implementations of such protocols all live outside of the
> kernel but in userspace.  So you either have
> 
> a) a pure SDR software implementation of a GSM phy, whether on the MS/UE
>    side or on the BTS/network side, or whether in sniffing mode, running
>    as a userspace process.  This could e.g. be airprobe or gr-gsm
> 
> b) a OsmocomBB phone attached over serial port running the Osmocom Layer1
>    firmware on the baseband processor, with some userspace program on a Linux
>    PC implementing higher layers
> 
> c) GSM base station software such as osmo-bts, running in userspace
> 
> d) Using an old nokia phone attached via serial cable to a Linux PC running
>    dct3-gsmtap.

Sure.

> So for any of those, if you want to provide real-time data streams, you have to
> somehow transmit those over some kind of network technology.   One could have
> done raw ethernet frames or raw IP frames, but using UDP seemed straight-forward
> as one can create UDP socket from non-privileged processes.

This is the part I guess I don't understand. I haven't really tried, but
it seems you should be able to encapsulate arbitrary protocols into
802.3 OUI-based ethertype or so?

But I guess if your primary use case is actually to transport them over
the network, this makes more sense.

Our primary use case with wifi is much more local - capture what's going
on - but do the capturing on the local system and write it to a file
there.

> We also have a virtual physical layer between OsmocomBB and OsmoBTS called
> virt_phy where the GSMTAP over multicast IP is used to simulate/virtualize
> the entire Layer1 / PHY / radio interface between phone and base station.
> 
> Once again, all related network elements are implemented in userspace,
> and having an 

...?

> > (speaking of versions - the docs say "version, set to 0x01 currently"
> > but "#define GSMTAP_VERSION 0x02")
> 
> Sorry, it's the usual "developer changes code but not comment". patches
> are welcome, we use gerrit.osmocom.org :)

Ok :-)

> > 3) Does the packet data follow the gsmtap header? It's not really clear
> >    to me based on reading the wireshark code.
> 
> Sure, the packet data follows the GSMTAP header, and the type of data
> is defined by the GSMTAP type / sub-type as per the specific use case.  As
> you can see, there's 18 TYPE by now (each of which has at least one
> program/implementation generating the data).

Right.

> I think the best way to learn about GSMTAP is to use any of the many
> programs that support it by now.  I think not only the examples above
> use it, but meanwhile many others like the independently-developed OpenBTS
> project that's unrelated to Osmocom, as are other projects implementing LTE.

OK.

> > In particular, the data I'm thinking of is higher-level things, like the
> > session ID for a frame when it's going through the kernel, or perhaps a
> > flow label on RX, etc.
> 
> I'm not quite sure how that relates to GSM?  GSMTAP so far was intended
> to encapsulate ETSI/3GPP messages inside UDP/IP, particularly messages of
> protocols that don't traditionally are transported over IP baesd transports.

Yeah, I think here's where I got confused or just don't know enough
about GSM.

Basically, I was looking at it as if it was like WiFi in a sense - you
have an IP frame, you're going to transport it over some physical link,
so it gets PHY information in the sense of modulation etc.

That doesn't seem to be the case for GSM, I guess? Does the IP frame get
encapsulated in some kind of 3GPP message first, and then transported
over the physical link, and the physical link info isn't even there in
GSMTAP?

> Having said, we're open to extending the scope - it just all needs to make
> sense

See above for a bit better description of what I'm trying to solve.

> > I do note that this
> > wouldn't be compatible with the current wireshark code as it doesn't
> > check the version, just shows it...
> 
> If that's the case that's sad, and I should have paid more attention to
> it when originally writing it.  We should get a related fix into
> wireshark ASAP then.  But then, the current dissector would just state
> something like unsupported version instead of showing some garbage it
> cannot parse.  Either way, you will of course need new sources and
> sink side implemetations.

Sure. Just a note in passing.

> > Or would it make more sense to define a new ARPHDR_WWANTAP like
> > ARPHDR_IEEE80211_RADIOTAP and just use that instead of encapsulating in
> > IP/UDP, and then have a completely new (extensible) protocol inside of
> > that? 
> 
> No userspace source would ever be able to generate such data and stream
> it real-time into wireshark, would it?  Sure, I can write pcap file with
> such ARPHDR_* values, but I could never do this in real-time.  For many
> but not all use cases, that's really what it is: A vehicle to stream
> real-time non-IP protocol traces into wireshark so it can visualize
> the protocol traces.

I think you can pipe a stream into wireshark?

To me it feels like the wrong thing to actually make wireshark listen on
"lo" or "eth0" or something to get data from the cellular that's
(locally) generated by another application, but I guess that's only
about how you think about it - and if it's not generated locally then
that's an easy transport. I'm not sure it makes *sense* because then you
need permissions to capture on the wired network etc. where you don't
*really* need that for this stream, but it's convenient for sure.

johannes

Re: gsmtap design/extensions?

[Johannes Berg]

On Thu, 2019-04-11 at 01:45 +0200, Harald Welte wrote:
> Hi Johannes,
> 
> On Wed, Apr 10, 2019 at 09:23:13AM +0200, Johannes Berg wrote:
> > > but unfortunately, nobody has invested time into this (yet?).
> > 
> > 2012! 
> 
> Well, Osmocom is a very small community, with probably somewhere less than
> 25 active developers over the last few years (less than 15 full-time),
> with an *incredibly* large scope:  Implement virtually any protocol
> layer of any protocol stack on any of the 3GPP interfaces and all their
> related network elements for 2G/3G as well as even other technologies
> like TETRA, GMR-1, ...
> 
> And all that in a field of technology that has less free software than
> the Operating Systems world had in the mid-1990ies.  It really feels a
> bit like the Linux community 20 years ago.
> 
> So resources are always *extremely* tight, and given those limited
> resources, I'm actually very happy with the results by now, having
> automatied CI, build verifications, unit tests, functional test suites,
> end-to-end testing, and all the code we implemented on git.osmocom.org :)

:-)
Sure, I get it. Just a bit surprised I guess.

> While current GSMTAPv2 is ugly, it works rather solid for all known
> existing use cases, so there was no urgency to introduce a new version
> of it.

OK.

> > Not sure I get this, but I also don't really care all that much. 
> 
> Well, with all respect, GSMTAP was created for a variety of use cases,
> see my other lengthy mail.  It's fine if you don't care, but unless you
> could explain your use cases with a few paragraphs, neither you nor us
> are able to determine if there is common functionality and if it makes
> sense to use GSMTAP or not :)

Agree. Sorry about that. No disrespect was intended, but I'm still not
sure I understand the need for UDP encapsulation *as part of the
protocol*. I guess saying "GSMTAP can optionally be encapsulated in UDP
with the well-known port xyz" would be something else, and it'd make
more sense to me than saying it has to be.

> So far I have not seen any explanation about what kind of data you want
> to encapsulate at all.

Right, see my other mail(s) from today as well. Basically the IP frame
that we're actually sending, and then attaching to that the "session" or
"mux id" that it's being sent on. Sorry, I probably also don't know the
right term - those show up in the drivers now.

> > just a pretty strange design if the kernel were to output this, I'm not
> > even sure how I'd do that properly. I don't want to be generating UDP
> > packets there...
> 
> There are well-established APIs for having sockets in the kernel and for
> generating + receiving UDP packets from it.  NFS has been doing this for
> decades, as do various kernel-side tunneling helpers including the GTP
> kernel module.
> 
> I'm not saying it's the right approach for your problem, I'm just saying
> kernel-side code can for sure use UDP sockets.

Of course it *can*. But I don't think it makes *sense*. The key feature
here isn't communicating with somebody else (unlike NFS, GTP, GENEVE and
whatever other protocol they have). In fact, you shouldn't really care
about the communication part per se at all, I'd think.

> Sure, that works.  But the real question is, to me:  Are there common
> GSMTAP payload types that both the existing GSMTAP users carry, as well
> as what you would want to carry?  If yes, then it makes sense to think
> about a common encapsulation like GSMTAP.  If the payload types differ,
> then it seems rather like there are two distinct use cases that
> wouldn't benefit from standardizing on one format.

Agree, and I don't really know.

Maybe I should start differently. Do you have an example GSMTAP capture
file that I could look at in wireshark? Yes, I see you've pointed out
how I can get all the software running, but if you have a file already
that's almost certainly faster :-)

And then the question I'd want answer is this: If there's an IP frame
that I send to the modem from the application using a regular UDP or TCP
socket, what would the corresponding GSMTAP capture look like? Surely it
includes the IP frame in some way?!

If the answer to that question is yes, then I think there is some
overlap, because you can always imagine the modem receiving an IP frame
and telling you more about how it was encapsulated over the air, no?
Mind, most if not all modems probably don't actually do that today, but
I wonder how much of that is because of lack of infrastructure to do it,
vs. it just not being necessary - since I've been told that the modems
do in fact often output tracing that contains information about this.
Just not directly combined with the IP frame.

johannes

Re: gsmtap design/extensions?

[Guy Harris]

On Apr 12, 2019, at 5:24 AM, Johannes Berg <johannes-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> wrote:

> On Thu, 2019-04-11 at 01:32 +0200, Harald Welte wrote:
> 
>> GSMTAP was designed as a format to encapsulate protocols normally not spoken over IP
>> (such as classic GSM radio protocols, e.g. the Layer 2 LAPDm over GSM Um)
>> inside an IP transport.
> 
> Sure, but wifi is also not spoken over IP, yet we don't encapsulate our
> 802.11 frames into IP to show them in wireshark :-)

That's just because the rpcap protocol hasn't been revved yet to handle the new create/set options/activate mechanism in libpcap to allow monitor-mode capturing, so you only get to see fake Ethernet frames. :-)

I.e., there's a split there between "capture" and "getting the packets from a capture delivered to you over an IP network".

Perhaps there should be a GSMTAP link-layer header type, so you can have a GSMTAP pcap file or GSMTAP interface in a pcapng file, combined with a more general "remote capture" mechanism in libpcap so that you could capture on gsmtap://host:port and capture from a host using the GSMTAP-over-UDP encapsulation - or capture using rpcap.

>> No userspace source would ever be able to generate such data and stream
>> it real-time into wireshark, would it?  Sure, I can write pcap file with
>> such ARPHDR_* values, but I could never do this in real-time.  For many
>> but not all use cases, that's really what it is: A vehicle to stream
>> real-time non-IP protocol traces into wireshark so it can visualize
>> the protocol traces.
> 
> I think you can pipe a stream into wireshark?

1) You could pipe into libpcap or otherwise have a way for a libpcap module to connect to a user space source and get packets from it.

2) You could pipe a pcap file into tcpdump/Wireshark/etc..

3) You could have an extcap program:

	https://www.wireshark.org/docs/wsdg_html_chunked/ChCaptureExtcap.html

provide packets to Wireshark.

> To me it feels like the wrong thing to actually make wireshark listen on
> "lo" or "eth0" or something to get data from the cellular that's
> (locally) generated by another application, but I guess that's only
> about how you think about it - and if it's not generated locally then
> that's an easy transport. I'm not sure it makes *sense* because then you
> need permissions to capture on the wired network etc.

Depending on how your system is set up:

	$ ls -l /dev/bpf*
	crw-rw----  1 root  access_bpf   23,   0 Apr 10 22:57 /dev/bpf0
	crw-rw----  1 root  access_bpf   23,   1 Apr 10 22:56 /dev/bpf1

		...

and it could just be rw-rw-rw-.  Perhaps other systems make it harder to grant capture privileges.

> where you don't *really* need that for this stream

If there's a need for that, whatever provides the packets could impose that (by finding out the UID/GID of the other process if this is going over a UNIX-domain socket, at least on some UN*Xes, or by requiring a login if this is going over a network).

Re: gsmtap design/extensions?

[Marcel Holtmann]

Hi Johannes,

>> before you go all out and define this, it would suggest to understand
>> what meta-data for the connection contexts you actually need as well.
>> The data path itself is just a pipe and has not all the information
>> attached with it. That goes via the control path and that is normally
>> in user space and carries the real important information to make
>> useful analysis of how the data path / context is setup.
> 
> Yes, that's true, though the control path is actually going through one
> of the data pipes as well.

I think that viewpoint is too simplistic. And for sure we have no such system where the control path is done as IP packets for Ethernet packets.

>> From what I am seeing right now is that unless you have a method to
>> also feed the control path into your GSMTAPv3, then this is rather
>> useless. 
> 
> So the control path *itself* would be there, I guess, but ...
> 
>> The majority of the debugging is really done for the control path. For
>> oFono that is OFONO_DEBUG=1 environment variable and while it works it
>> is not the most elegant solution. I would love to feed that into a
>> generic debugging / tap that you can read out later.
> 
> there's definitely room for more information than _just_ the control
> path "chat", also application state etc. would be useful, and logging
> etc.
> 
> Typically on wifi we feed all of this together into kernel tracing (to
> record with trace-cmd) rather than trying to encapsulate it some other
> way.
> 
>> As a side note, for Bluetooth we created a path where the bluetoothd
>> can feed back its control debugging data back into the Bluetooth
>> monitor in the kernel to allow combined userspace, mgmt and HCI
>> tracing. Some really nasty issues could only be triaged by having all
>> the meta data with a common timestamp.
> 
> Right. This is something we'd typically use tracing for in wifi.

Same thing, but different way of doing it. Mind you that Bluetooth support is older than tracing.

> I don't really know what the right model for WWAN would be, I guess.
> 
> 
> Right now - and I really should've said this before - really the only
> problem I was thinking of was how we can mux multiple "chat" sessions
> with a device into a single data stream.
> 
> Currently, this is all vendor-specific. If you have a Qualcomm modem,
> you'd be able to see all the open sessions on the underlying netdev, and
> the QMI header tells you what session a given 'packet' belongs to, and
> if you follow along maybe you can figure out if this is a control or IP
> 'packet' (could be an AT command).
> 
> cdc_mbim uses VLAN tags instead to achieve this, and decapsulates the
> VLAN tags to send them down to the hardware in a different way.
> 
> There are a few reasons why I think that this model of having a single
> underlying netdev controlled by the modem driver, with additional
> netdevs layered on top in software will not work right in the future. I
> think drivers should and will need to migrate to creating "real" netdevs
> for the sessions instead of pure software ones.

I do not follow on why is that. Why would I care if wwan0 is self-sustained of just a VLAN device. Or for that matter any other kind of slave/child device. 3GPP and 3GPP2 are not Ethernet frame based. We only have raw IPv4 and IPv6 packets for the data path.

> But if you do this, you lose the ability to listen to all the session
> streams at the same time, you can only do it for each netdev. Adding
> this ability back seems worthwhile, but then we probably shouldn't do it
> in a vendor-specific way, but rather in a generic way.
> 
> So basically right now that's all I'm trying to solve. In WiFi we don't
> have the problem of "sessions" because we just use the addresses in the
> frames to disambiguate - on such 'monitor' netdevs we see the frames
> including full 802.11 headers.

And for 3GPP you have context identifiers that at least within the context of the control path make logical sense of the data streams and what they are assigned to. This goes back to my original point. You need to capture the control path to see what APN context is set up and how. Mind you that you also have the fun between primary context and secondary context (primarily for quality of service in VoLTE cases).

> Now, here's maybe where I'm getting off the right path - in wifi we
> mostly couple that with PHY information as well, and so we have PHY
> information + full 802.11 headers + data for capturing what's going on.
> I figured that theoretically at least that would be possible/useful for
> the modem as well (obviously control packets have no PHY data), but
> there doesn't seem to be any hardware that would actually expose data in
> this way. Everyone I've spoken to says these things are only available
> as modem trace data. What I haven't figured out though is if that's by
> some other design trade-off, or just because no such infrastructure
> is/was available.
> 
> It may well be that doing kernel-tracing instead of doing it via some
> kind of monitor netdev is perfectly sufficient, but I have a feeling
> that the relative simplicity of starting tcpdump/wireshark might be
> preferable.

As I said, as long as you do not get the QMI, AT command, MBIM etc. control path session recorded as well, you have nothing to really analyze.

Regards

Marcel

Re: gsmtap design/extensions?

[Guy Harris]

On Apr 12, 2019, at 10:15 AM, Johannes Berg <johannes-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> wrote:

> Agree. Sorry about that. No disrespect was intended, but I'm still not
> sure I understand the need for UDP encapsulation *as part of the
> protocol*. I guess saying "GSMTAP can optionally be encapsulated in UDP
> with the well-known port xyz" would be something else, and it'd make
> more sense to me than saying it has to be.

I see nothing about a struct gsmtap_hdr:

	http://osmocom.org/projects/baseband/wiki/GSMTAP

that

	1) requires that it plus the payload be encapsulated in a UDP datagram

or

	2) would prevent it from being at the beginning of a LINKTYPE_GSMTAP/DLT_GSMTAP packet in a pcap or pcapng file.

Re: gsmtap design/extensions?

[Guy Harris]

On Apr 12, 2019, at 12:54 PM, Guy Harris <guy-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:

> I see nothing about a struct gsmtap_hdr:
> 
> 	http://osmocom.org/projects/baseband/wiki/GSMTAP
> 
> that...
> 
> 	2) would prevent it from being at the beginning of a LINKTYPE_GSMTAP/DLT_GSMTAP packet in a pcap or pcapng file.

With a specification based on

	http://cgit.osmocom.org/libosmocore/plain/include/osmocom/core/gsmtap.h

Re: gsmtap design/extensions?

[Harald Welte]

Dear Guy and others,

On Fri, Apr 12, 2019 at 03:47:26PM -0700, Guy Harris wrote:
> On Apr 12, 2019, at 12:54 PM, Guy Harris <guy-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:
> 
> > I see nothing about a struct gsmtap_hdr:
> > 	http://osmocom.org/projects/baseband/wiki/GSMTAP
> > that...
> > 	2) would prevent it from being at the beginning of a LINKTYPE_GSMTAP/DLT_GSMTAP packet in a pcap or pcapng file.
> With a specification based on
> 	http://cgit.osmocom.org/libosmocore/plain/include/osmocom/core/gsmtap.h

I completely agree that there is no technical reason why one couldn't put GSMTAP
headers in current (v2) or future formats inside an ETHERTYPE / DLT and have
them natively appearing in a pcap file of the given DLT.  There would be no objections
from my side to do so.

One could then simply call the same dissector in wireshark from that DLT or from
the existing UDP port number based dispatch via the IANA-registered GSMTAP port
number.

However, among the existing users of GSMTAP in the last decade or so,
there would be no advantage, as the related sources of GSMTAP frames all
exist in userspace and are feeding data, particularly from potentially
multiple sources, which can very well run on different hosts.

But that of course doesn't prevent new users from using different
transport mechanisms of getting GSMTAP from e.g. the kernel into
userspace.

Regards,
	Harald
-- 
- Harald Welte <laforge-TgoAw6mPHtdg9hUCZPvPmw@public.gmane.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: gsmtap design/extensions?

[Harald Welte]

Hi Johannes,

On Fri, Apr 12, 2019 at 07:15:56PM +0200, Johannes Berg wrote:

> Agree. Sorry about that. No disrespect was intended, but I'm still not
> sure I understand the need for UDP encapsulation *as part of the
> protocol*. I guess saying "GSMTAP can optionally be encapsulated in UDP
> with the well-known port xyz" would be something else, and it'd make
> more sense to me than saying it has to be.

Sure, like with most protocols you can wrap them in anything you want.

Let me put it like this:
You don't have to run RTP inside UDP, you could equally put the RTP
frames in to SCTP or DCTP.   It's just not what the original users of
the protocol/spec had envisioned, but it can for sure be done, and has
no side-effect other than not being interoperable with existing
implementations.

> Right, see my other mail(s) from today as well. Basically the IP frame
> that we're actually sending, and then attaching to that the "session" or
> "mux id" that it's being sent on. Sorry, I probably also don't know the
> right term - those show up in the drivers now.

So it's basically the information whch PDP/PDN context your user IP packet
belongs to.  A single integer "tag". that resembles the identifier that
was used by the control plane when activating that context (which is
basically a tunnel).

> > Sure, that works.  But the real question is, to me:  Are there common
> > GSMTAP payload types that both the existing GSMTAP users carry, as well
> > as what you would want to carry?  If yes, then it makes sense to think
> > about a common encapsulation like GSMTAP.  If the payload types differ,
> > then it seems rather like there are two distinct use cases that
> > wouldn't benefit from standardizing on one format.
> 
> Agree, and I don't really know.

>From what I can tell after the burst of e-mails in this thread so far,
I don't think there's much commonality here.  A modem will "never" give
you access to the actual cellular protocol layers that we care about
in the Osmocom/srsLTE/YateBTS/OpenBTS/airprobe/gr-gsm/... world, for
which GSMTAP was designed (see more below).

I'm not saying it cannot be done. If you want to use GSMTAP, I'm happy
to help.  But at least so far, I don't see why it would make any sense.

> Maybe I should start differently. Do you have an example GSMTAP capture
> file that I could look at in wireshark? Yes, I see you've pointed out
> how I can get all the software running, but if you have a file already
> that's almost certainly faster :-)

There are a couple of files attached at
https://osmocom.org/projects/baseband/wiki/WiresharkIntegration

> And then the question I'd want answer is this: If there's an IP frame
> that I send to the modem from the application using a regular UDP or TCP
> socket, what would the corresponding GSMTAP capture look like? Surely it
> includes the IP frame in some way?!

GSMTAP was generated initially for GSM. GSM is a circuit-switched digital
wireless telephony system that has no inherent/native way of transporting
IP payload.  As such, the GSMTAP Um captures linked above will not contain
IP user data but will contain the signaling plane data of the Um air interface
protocol stack, consisting of the LAPDm potocol on Layer2 and the TS 04.08
RR (Radio Resource), MM (Mobility Management) and CC (Call Control) data.

Of course one can also use GSMTAP for GPRS, which is packet oriented.  In
this case, you have the following layering stack inside a GSMTAP Um frame:

TCP
IP
SNDCP (TS 04.65)
LLC (TS 04.64)
RLC/MAC
PHY

There is segmentation/reassembly potentially at least at the LLC and at
the RLC/MAC layer.  There can be encryption at the LLC layer so the IP
payload is already invisible at that point.  You cab also have both
header and payload compression at the SNDCP layer.

In the end, you can have start/middle/end segments of LLC frames inside
a single RLC/MAC block, belonging to either one or multiple LLC frames,
and then the LLC frames contain [segments of] IP packets.

GSMTAP can be used on various other interfaces of the cellular network,
which are *not* the radio interface between modem/phone and base
station (such as e.g. between the Phone and the SIM card), so they're not
of interest to your use case and I'll not cover them here.

GSMTAP can also be used to encapsulate later cellular technologies such
as UMTS aka WCDMA aka 3G, or LTE aka 4G.  In all those cases you always
have a different (sometimes deeper and sometimes not so deep) protocol
stack between the user IP payload and the actual radio interface (PHY).

You can think of it a bit if you are not thinking about a User IP packet
but you assume for the sake of an argument that you want to see "HTML".

Then underneath HTML you either have, depending on the technoolgy, HTTP
1.0, 1.1 or 2.0.  And under that you can have any number of additional
protocol stacking whether it's TCP based, with or without SSL or TLS,
with IPv3 or IPv6, with or without IPsec, inside a VLAN or not, ...  -
and most importantly, next to all of that you have important "control
information" like let's say DHCP or neighbor discovery.

So, in other words, the user IP plan is *very* far away from the PHY on
the radio interface, and the really interesting things are those that
are happening beneath or next to the user IP.

However, commercially available cellular modems will go to the utmost
extent to make sure you never have access to any of this, and they
invent various different ways to make the user (whether that's ofono,
the ModemManager, ...) as far away from that as possible.  The
interfaces between the cellular protocol stacks and the user (whether
QMI, MBIM, AT commands, ...) are so abstract that virtually any useful
resemblance to what happens on the actual radio interface is lost.

You can get a glimpse to some of what's happening with Qualcomm DIAG
protocol, but even that doesn't really [always] expose full protocol
traces to you, particularly not on the user plane.

> If the answer to that question is yes, then I think there is some
> overlap, because you can always imagine the modem receiving an IP frame
> and telling you more about how it was encapsulated over the air, no?

Not really.  The modem implements the entire protocol stacks for the
various cellular technologies and in the end provides you as the user
something like a tunnel endpoint.

> Mind, most if not all modems probably don't actually do that today, but
> I wonder how much of that is because of lack of infrastructure to do it,
> vs. it just not being necessary - since I've been told that the modems
> do in fact often output tracing that contains information about this.
> Just not directly combined with the IP frame.

The most widespread technology to obtain tracing/debugging information
is the Qualcomm DIAG protocol, which is a collection of dozens of
sub-systems each of whcih has up to hundreds of individual
flags/settings of categories that can be enabled and/or disabled.

We once did a bit of development in that area, see osmo-qcdiag to enable
this (http://git.osmocom.org/osmo-qcdiag/) as well as *extremely
minimal* wireshark dissectors at http://git.osmocom.org/wireshark/log/?h=laforge/qcdiag

However, this is a proprietary mechanism available only to Qualcomm stack
based devices, will only work on those devices where one has figured out
how to enable an access it.  And the information it provides is
completely asynchronous to the user IP plane.

There is a similar project for (now very old) Infineon XGold based
phones, see https://github.com/2b-as/xgoldmon.  It also generates
GSMTAP.  But AFAIR only for the control plane data and not the user
plane data.

To be honest, I don't even think there's a lot of context that one could
theoretically provide "attached" to the user IP packet, as the
interesting bits are happening at lower protocol layers, and due to
segmentation/reassembly/compression/...  etc. there is no 1:1 mapping
between the user IP packet and what's happening on the radio interface.

Regards,
	Harald
-- 
- Harald Welte <laforge-TgoAw6mPHtdg9hUCZPvPmw@public.gmane.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: gsmtap design/extensions?

[Harald Welte]

Hi Johannes,

I'll only adress some of the points that haven't been clarified in other
mails in this thread alreasdy:

On Fri, Apr 12, 2019 at 02:24:25PM +0200, Johannes Berg wrote:

> Kernel drivers need to transport multiple data streams, like VoLTE (IP)
> data, IP data, and control data. Currently, they do this in various ways
> (cdc_mbim uses VLAN tags, rmnet/qmi_wwan uses Qualcomm QMI headers),
> when you capture data you have to understand what you're dealing with
> and maybe extend wireshark to even understand these headers.
> 
> I'm looking for a general vendor-agnostic encapsulation that would solve
> this part of the problem.

understood.

> At the same time, I come from WiFi, where - while we don't have these
> sessions as such but can have different virtual interfaces as well, e.g.
> for connections to different APs - it's often very useful to also
> understand the PHY parameters that were used for TX/seen on RX. Now, I
> realize this isn't something that modems will do right now - they just
> don't seem to tell you any lower-level information on RX of a packet,
> but it seemed like it might be worthwhile to not *exclude* this
> possibility either.

Agreed.  However, the protocol stacks on WiFi and cellular technologies
are very different, to say the least.  There is no easy mapping of PHY
related parameters to a given IP packet, and the related quality metrics
of the radio channels don't work that way.

But yes, I agree, that whatever transport mechanism you wan to use between the modem
and the user/Linux side should allow for additional, extensible metadata beyond
the identifier of the PDP/PDN context.

> > GSMTAP was designed as a format to encapsulate protocols normally not spoken over IP
> > (such as classic GSM radio protocols, e.g. the Layer 2 LAPDm over GSM Um)
> > inside an IP transport.
> 
> Sure, but wifi is also not spoken over IP, yet we don't encapsulate our
> 802.11 frames into IP to show them in wireshark :-)

This is because your 802.11 implementations are in hardware, while the
open source GSM/UMTS/LTE implementations using GSMTAP are implemented in
userspace - or at the very least the higher-level parts of the protocol
stacks, or the tracing frameworks that are used to generate the GSMTAP
data from some hardware.

> This is the part I guess I don't understand. I haven't really tried, but
> it seems you should be able to encapsulate arbitrary protocols into
> 802.3 OUI-based ethertype or so?

But why?  I'm in an userspace program, and I want to send data to one or
more other userspace programs.  Why would I not simply open a UDP socket
to do so?  I would have to have CAP_NET_RAW and open a packet socket,
and then generate ethernet frames from that?

I admit that the use case with wireshark is a bit odd, but there are
other receivers out there.

Also, for debugging cellular network elements, it's very useful if you
can look at the RAN or core network interfaces (Abis, Gb, Gp, A, Iu, S1,
...) in the same protocol trace in which you also get traces from the
radio interface in GSMTAP.  Then you can see what happens at each and
every interface / network element in one time-line.

And as the Osmcoom programs also allow generating log output wrapped in
GSMTAP, you get not only protocol traces of all the interfaces, but even
textual log information (with machine-readable log level + sub-system)
information all in one capture/timeline.

> But I guess if your primary use case is actually to transport them over
> the network, this makes more sense.

The use cases differ, but having UDP encapsulation enables a lot of
flexibility and has been proven very useful so far.

> Our primary use case with wifi is much more local - capture what's going
> on - but do the capturing on the local system and write it to a file
> there.

Yes, you're looking only at a single interface (the radio interface
between one BSS and one STA).  You're not looking at five different
interfaces at five different levels of network hierarchy/topology in the "wifi
controller" and want to mix in a radio interface trace in the same
timeline.

> > We also have a virtual physical layer between OsmocomBB and OsmoBTS called
> > virt_phy where the GSMTAP over multicast IP is used to simulate/virtualize
> > the entire Layer1 / PHY / radio interface between phone and base station.
> > 
> > Once again, all related network elements are implemented in userspace,
> > and having an 
> 
> ...?

sorry. '... UDP based transport is ahat enables theis use case'

> Basically, I was looking at it as if it was like WiFi in a sense - you
> have an IP frame, you're going to transport it over some physical link,
> so it gets PHY information in the sense of modulation etc.

As stated elsewhere, there's an M:N mapping between user (IP) payload
and actual radio interface "MAC blocks", so I'm not aware of anyone
mapping radio interface performance to user plane IP.

> That doesn't seem to be the case for GSM, I guess? Does the IP frame get
> encapsulated in some kind of 3GPP message first, and then transported
> over the physical link, and the physical link info isn't even there in
> GSMTAP?

the "physical link info" is present in GSMTAP, but the granularity of
GSMTAP frames is not user-IP frames, but "MAC blocks".  So your user IP
frame might not be visible as it's still compressed, encrypted,
fragmented, etc.

> > > Or would it make more sense to define a new ARPHDR_WWANTAP like
> > > ARPHDR_IEEE80211_RADIOTAP and just use that instead of encapsulating in
> > > IP/UDP, and then have a completely new (extensible) protocol inside of
> > > that? 
> > 
> > No userspace source would ever be able to generate such data and stream
> > it real-time into wireshark, would it?  Sure, I can write pcap file with
> > such ARPHDR_* values, but I could never do this in real-time.  For many
> > but not all use cases, that's really what it is: A vehicle to stream
> > real-time non-IP protocol traces into wireshark so it can visualize
> > the protocol traces.
> 
> I think you can pipe a stream into wireshark?
> 
> To me it feels like the wrong thing to actually make wireshark listen on
> "lo" or "eth0" or something to get data from the cellular that's
> (locally) generated by another application, 

I admit it's a bit odd.  But has been very useful, particularly in more
distributed setups with creating a shared timeline of various GSMTAP
sources that may or may not run on the same machine.

-- 
- Harald Welte <laforge-TgoAw6mPHtdg9hUCZPvPmw@public.gmane.org>           http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
                                                  (ETSI EN 300 175-7 Ch. A6)

Re: gsmtap design/extensions?

[Guy Harris]

On Apr 12, 2019, at 11:41 PM, Harald Welte <laforge-TgoAw6mPHtdg9hUCZPvPmw@public.gmane.org> wrote:

> But that of course doesn't prevent new users from using different
> transport mechanisms of getting GSMTAP from e.g. the kernel into
> userspace.

Nor does it prevent libpcap (once I generalize the remote-capture mechanism) from supporting gsmtap://{host}[:port] (port optional) URLs that cause it to set up a UDP ({PF_INET or PF_INET6}, SOCK_DGRAM, IPPROTO_UDP) socket on which to receive UDP packets from the specified host (at the specified port or, if no port is specified, the default GSMTAP socket) and supplying packets that begin with a GSMTAP header, with a new linktype of LINKTYPE_GSMTAP/DLT_GSMTAP.

Re: gsmtap design/extensions?

[Guy Harris]

On Apr 13, 2019, at 12:12 AM, Harald Welte <laforge-TgoAw6mPHtdg9hUCZPvPmw@public.gmane.org> wrote:

> On Fri, Apr 12, 2019 at 07:15:56PM +0200, Johannes Berg wrote:
> 
>> Agree. Sorry about that. No disrespect was intended, but I'm still not
>> sure I understand the need for UDP encapsulation *as part of the
>> protocol*. I guess saying "GSMTAP can optionally be encapsulated in UDP
>> with the well-known port xyz" would be something else, and it'd make
>> more sense to me than saying it has to be.
> 
> Sure, like with most protocols you can wrap them in anything you want.
> 
> Let me put it like this:
> You don't have to run RTP inside UDP, you could equally put the RTP
> frames in to SCTP or DCTP.   It's just not what the original users of
> the protocol/spec had envisioned, but it can for sure be done, and has
> no side-effect other than not being interoperable with existing
> implementations.

Or you can just have LINKTYPE_RTP/DLT_RTP and supply them inside nothing.

However, unlike RTP, there is no reason *not* to do that for GSMTAP - it's not as if the IP or UDP headers in a packet from a host supplying GSMTAP-encapsulated packets provide any information necessary or even useful for dissecting the encapsulated packets.

Whether it's useful, or possible, to have any interfaces on a *host* with cellular modem connectivity supply the cellular-network traffic as packets with GSMTAP headers - which appears to be what Johannes is thinking of - is another matter (but even if the answer is no, there is, as per my other message, a use for a LINKTYPE_GSMTAP/DLT_GSMTAP header type).  That might not be possible, as cellular modems, as you note, tend to hide a lot of lower-layer details from the host.

Re: gsmtap design/extensions?

[Guy Harris]

On Apr 13, 2019, at 12:35 AM, Harald Welte <laforge-TgoAw6mPHtdg9hUCZPvPmw@public.gmane.org> wrote:

> the "physical link info" is present in GSMTAP, but the granularity of
> GSMTAP frames is not user-IP frames, but "MAC blocks".  So your user IP
> frame might not be visible as it's still compressed, encrypted,
> fragmented, etc.

The granularity of Ethernet frames is not user IP frames, but Ethernet datagrams, so you user IP frame might not be visible as it's fragmented (the fragments might still be IP datagrams, but they would have to be reassembled - which Wireshark, for example, does, for those people still doing NFS-over-UDP :-)).

"Encrypted" may not apply there, but it *does* apply for 802.11 frames on a protected network (which Wireshark can decrypt, if you have 1) the network password for WEP or WPA-Personal and 2) the EAPOL handshake for WPA-Personal).

Re: gsmtap design/extensions?

[Johannes Berg]

On Fri, 2019-04-12 at 21:49 +0200, Marcel Holtmann wrote:
> 
> > Yes, that's true, though the control path is actually going through one
> > of the data pipes as well.
> 
> I think that viewpoint is too simplistic. And for sure we have no such
> system where the control path is done as IP packets for Ethernet
> packets.

That's true, but as far as I can tell in a lot of cases the pipe is
actually still a virtual netdev (session, encoded in whatever way) using
some sort of serial line format on top?

But I agree that typically this will not be sufficient anyway.

> > Right. This is something we'd typically use tracing for in wifi.
> 
> Same thing, but different way of doing it. Mind you that Bluetooth
> support is older than tracing.

Sure. Looking forward though, perhaps tracing *would* indeed be the
simplest thing to do.

> > There are a few reasons why I think that this model of having a single
> > underlying netdev controlled by the modem driver, with additional
> > netdevs layered on top in software will not work right in the future. I
> > think drivers should and will need to migrate to creating "real" netdevs
> > for the sessions instead of pure software ones.
> 
> I do not follow on why is that. Why would I care if wwan0 is self-
> sustained of just a VLAN device. Or for that matter any other kind of
> slave/child device.

>From a driver perspective I think it makes a difference. For example, if
you just have a single netdev from the driver, it gets harder to do
multiple TX/RX queues properly.

> 3GPP and 3GPP2 are not Ethernet frame based. We only have raw IPv4 and
> IPv6 packets for the data path.

Sure. Though some drivers/devices fake them anyway.

> And for 3GPP you have context identifiers that at least within the
> context of the control path make logical sense of the data streams and
> what they are assigned to. This goes back to my original point. You
> need to capture the control path to see what APN context is set up and
> how. Mind you that you also have the fun between primary context and
> secondary context (primarily for quality of service in VoLTE cases).

Right.

> > It may well be that doing kernel-tracing instead of doing it via some
> > kind of monitor netdev is perfectly sufficient, but I have a feeling
> > that the relative simplicity of starting tcpdump/wireshark might be
> > preferable.
> 
> As I said, as long as you do not get the QMI, AT command, MBIM etc.
> control path session recorded as well, you have nothing to really
> analyze.

But you do - afaict there are no ways other than the netdevs to
communicate with the device, and people do things like "socat" to set up
PTYs and pretend to have serial channels there, on top of the netdevs?

johannes

Re: gsmtap design/extensions?

[Johannes Berg]

Hi Harald,

> Agreed.  However, the protocol stacks on WiFi and cellular technologies
> are very different, to say the least.  There is no easy mapping of PHY
> related parameters to a given IP packet, and the related quality metrics
> of the radio channels don't work that way.

Sure. I'm (vaguely) aware of that.

> But yes, I agree, that whatever transport mechanism you wan to use between the modem
> and the user/Linux side should allow for additional, extensible metadata beyond
> the identifier of the PDP/PDN context.

Really the question it goes down to is whether there's possible overlap
here with GSMTAP or not.

If yes - we can define a GSMTAPv3, with TLVs or so, and add a "PDP/PDN
context" or "session" field or something like that, and maybe a tag that
says "IP packet is here", or some kind of "content type" field. I expect
there would be some such "sessions" that actually just transport the
local AT command chat.

If no, I guess we'll just define something else for this use case? Or,
perhaps, even get rid of it entirely and just rely on trace-cmd
recording or so, though then I guess we'd really want to teach libpcap
and/or wireshark to understand this in some way.

Though it almost sounds like in GSMTAP you're never actually
transporting IP data, but other types of packets that in turn transport
the (fragmented/compressed/encrypted) IP data. That would kinda mean
there's very little potential overlap.

> > This is the part I guess I don't understand. I haven't really tried, but
> > it seems you should be able to encapsulate arbitrary protocols into
> > 802.3 OUI-based ethertype or so?
> 
> But why?  I'm in an userspace program, and I want to send data to one or
> more other userspace programs.  Why would I not simply open a UDP socket
> to do so?  I would have to have CAP_NET_RAW and open a packet socket,
> and then generate ethernet frames from that?
> 
> I admit that the use case with wireshark is a bit odd, but there are
> other receivers out there.

Yeah, ok. I was thinking kinda the other way around - if all you need is
network transparency for wireshark it shouldn't matter, but that's
really discussed over in the other subthread with Guy Harris better.

Anyway it doesn't matter, and I'm beginning to understand that a (maybe
even the primary) use case is getting multiple parts of your stack to
talk to each other (over UDP).

> Yes, you're looking only at a single interface (the radio interface
> between one BSS and one STA).  You're not looking at five different
> interfaces at five different levels of network hierarchy/topology in the "wifi
> controller" and want to mix in a radio interface trace in the same
> timeline.

Indeed. Well, actually, we often do, but use different mechanics for
that (trace-cmd record comes to mind, it records all our driver/fw chat
as well as possibly adding in logging from wpa_supplicant etc.)

> > Basically, I was looking at it as if it was like WiFi in a sense - you
> > have an IP frame, you're going to transport it over some physical link,
> > so it gets PHY information in the sense of modulation etc.
> 
> As stated elsewhere, there's an M:N mapping between user (IP) payload
> and actual radio interface "MAC blocks", so I'm not aware of anyone
> mapping radio interface performance to user plane IP.

Right, OK.

> the "physical link info" is present in GSMTAP, but the granularity of
> GSMTAP frames is not user-IP frames, but "MAC blocks".  So your user IP
> frame might not be visible as it's still compressed, encrypted,
> fragmented, etc.

Sure. As Guy Harris pointed out, this is also true for wifi though - we
don't have compression, but certainly encryption & fragmentation,
wireshark was taught to handle that and recreate the original MSDU (mac
service data unit).

johannes

Re: gsmtap design/extensions?

[Johannes Berg]

On Fri, 2019-04-12 at 12:48 -0700, Guy Harris wrote:

> I.e., there's a split there between "capture" and "getting the packets
> from a capture delivered to you over an IP network".

Right. That's what I was trying to get at, you said it much more
succinctly.

> Depending on how your system is set up:
> 
> 	$ ls -l /dev/bpf*
> 	crw-rw----  1 root  access_bpf   23,   0 Apr 10 22:57 /dev/bpf0
> 	crw-rw----  1 root  access_bpf   23,   1 Apr 10 22:56 /dev/bpf1
> 
> 		...
> 
> and it could just be rw-rw-rw-.  Perhaps other systems make it harder
> to grant capture privileges.

Yeah, generally it's not that *hard*. It still seems *wrong*, and e.g.
it would also mean if you do need to do it remotely you'd have to filter
out your own remote control (ssh) traffic lest you cause traffic
amplification, etc.

johannes

Re: gsmtap design/extensions?

[Bjørn Mork]

Johannes Berg <johannes-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> writes:

>> As I said, as long as you do not get the QMI, AT command, MBIM etc.
>> control path session recorded as well, you have nothing to really
>> analyze.
>
> But you do - afaict there are no ways other than the netdevs to
> communicate with the device,

I don't understand where you are going here? Neither QMI, AT command nor
MBIM management are transported over netdevs. AT is usually transported
using simple USB bulk streams, exported to userspace as tty devices by
some USB serial driver.  Both QMI and MBIM management are transported as
USB control messages, and exported to userspace as chardevs.  There are
no netdevs involved.

> and people do things like "socat" to set up
> PTYs and pretend to have serial channels there, on top of the netdevs?

I assume you are referring to my MBIM DSS examples here.  I don't know
if anyone is actually using that, so you should probably ignore it...
I'll happily admit it was a bad idea.  Should have just added the
necessary code to map DSS channels to some sort of character device in
the driver, like most users requested.

But there really isn't anything in the MBIM spec saying how DSS should
be used. DSS is a generic data stream.  Could easily be connected to a
single TCP session for example, in which case you'd probably want to
connect it to a TCP session in the other end too.  So I wouldn't want to
force DSS into character devices on the host end.  This doesn't rule out
a userspace controlled optional mapping though. We could probably still
add that, replacing the VLAN mapping with a chardev for a specific DSS
session if requested by userspace.  I guess this is something to
consider for a generic cellular framework - supporting non-ip data
streams between modem and host.

Adding to my previous excuses: The DSS implementation in the cdc_mbim
driver was added without having seen a single modem firmware with *any*
type of DSS support.  It was purely based on spec{ification,culation}.
The VLAN mapping, along with examples of how to use socat to further map
the streams from VLANs into suitable application specific forms, seemed
like a simple and flexible enough solution.


Bjørn

Re: gsmtap design/extensions?

[Johannes Berg]

On Mon, 2019-04-15 at 12:29 +0200, Bjørn Mork wrote:
> 
> I don't understand where you are going here? Neither QMI, AT command nor
> MBIM management are transported over netdevs. AT is usually transported
> using simple USB bulk streams, exported to userspace as tty devices by
> some USB serial driver.  Both QMI and MBIM management are transported as
> USB control messages, and exported to userspace as chardevs.  There are
> no netdevs involved.

OK. I've also been looking at a driver for Intel modems, and I think it
works that way.

> > and people do things like "socat" to set up
> > PTYs and pretend to have serial channels there, on top of the netdevs?
> 
> I assume you are referring to my MBIM DSS examples here.

I was more thinking of what I've been told the Intel modem does,
actually. But that in turn might very well be inspired by what you
documented there. I think the folks doing this were just trying to make
it work and nobody really understands the whole landscape.

> I don't know
> if anyone is actually using that, so you should probably ignore it...
> I'll happily admit it was a bad idea.  Should have just added the
> necessary code to map DSS channels to some sort of character device in
> the driver, like most users requested.

OK. So I guess a framework should consider that possibility, and let you
create new chardevs for a given (DSS) channel?

> But there really isn't anything in the MBIM spec saying how DSS should
> be used. DSS is a generic data stream.  Could easily be connected to a
> single TCP session for example, in which case you'd probably want to
> connect it to a TCP session in the other end too.  So I wouldn't want to
> force DSS into character devices on the host end.  

Fair point.

I suppose some of these may also be debug channels that give you modem
debug information, which is useful (if you can interpret it).

> This doesn't rule out
> a userspace controlled optional mapping though. We could probably still
> add that, replacing the VLAN mapping with a chardev for a specific DSS
> session if requested by userspace.  I guess this is something to
> consider for a generic cellular framework - supporting non-ip data
> streams between modem and host.

Right.

> Adding to my previous excuses: The DSS implementation in the cdc_mbim
> driver was added without having seen a single modem firmware with *any*
> type of DSS support.  It was purely based on spec{ification,culation}.
> The VLAN mapping, along with examples of how to use socat to further map
> the streams from VLANs into suitable application specific forms, seemed
> like a simple and flexible enough solution.

:-)

johannes