[RFA] TLV fields for radiotap

[RFA] TLV fields for radiotap

[Johannes Berg]

Since there were no comments, let's make this more formal. Any
objections now?


--
title: TLV fields in radiotap
categories: [suggested]
--
Bit Number:
: 28

Structure:
: list of TLVs detailed below

Required alignment
: 4

Unit(s)
: unspecified

If this bit is set higher bits may not be set (as they cannot be parsed,
given the variable length of this field) and the data following it is to
be understand as a type-length-value format, with this structure:

 * u16 type
 * u16 length
 * data ("length" bytes)
 * 0-3 padding bytes
   The whole thing is padded to a multiple of 4 bytes length, but that
   padding is not taken into account in the 'length' field.

The type is taken from the regular radiotap type (bit) allocation, but
the special values (29 and 31) are not valid, vendor namespaces (type
30) may be used as TLV fields.

If field alignment is necessary beyond the 4-byte alignment, type 28
(i.e. this extension) shall be used as padding with an appropriate
length (which would likely often be 0 for 8-byte alignment). This
ensures parser simplicity, it doesn't need to be aware of alignment
rules.


Generators should generate data in fields with type < 28 in regular non-
TLV fields, this ensures backward compatibility.

Generators shall use TLV fields for all fields with type >= 32.

Parsers shall be able to parse TLV fields with partial data present,
i.e. allow parsing optional values at the end of a field.

Re: [RFA] TLV fields for radiotap

[Johannes Berg]

On Mon, 2018-07-09 at 10:47 +0200, Johannes Berg wrote:
> Since there were no comments, let's make this more formal. Any
> objections now?

Does this look like a good idea/compromise?

It's hard to do an entirely new version, then at least in Linux we'd
probably have to be able to switch versions on the fly for some time,
which is awkward ... but OTOH this feels a bit hackish.

No comments at all? :)

johannes

Re: [RFA] TLV fields for radiotap

[Johannes Berg]

On Tue, 2018-09-04 at 14:14 +0200, Johannes Berg wrote:
> On Mon, 2018-07-09 at 10:47 +0200, Johannes Berg wrote:
> > Since there were no comments, let's make this more formal. Any
> > objections now?
> 
> Does this look like a good idea/compromise?
> 
> It's hard to do an entirely new version, then at least in Linux we'd
> probably have to be able to switch versions on the fly for some time,
> which is awkward ... but OTOH this feels a bit hackish.
> 
> No comments at all? :)

*crickets*

I guess I'll propose this more formally, since nobody really seems to
have any thoughts on the matter and parsing radiotap is notoriously
difficult :-)

FWIW, especially with vendor fields, there's a problem today: these
always come more or less last, so you have to understand all bits before
them. This is also true for radiotap, but due to sequential allocation,
if you care about a certain field today, your software is guaranteed to
be able to read it in the future too. If your software cares about a
vendor radiotap field, that's not true since other new standard fields
may be inserted before it...

Speak up now if you do think it's a bad idea ;-)

johannes

Re: [RFA] TLV fields for radiotap

[David Young]

On Tue, Nov 20, 2018 at 09:09:39PM +0100, Johannes Berg wrote:
> On Tue, 2018-09-04 at 14:14 +0200, Johannes Berg wrote:
> > On Mon, 2018-07-09 at 10:47 +0200, Johannes Berg wrote:
> > > Since there were no comments, let's make this more formal. Any
> > > objections now?
> > 
> > Does this look like a good idea/compromise?
> > 
> > It's hard to do an entirely new version, then at least in Linux we'd
> > probably have to be able to switch versions on the fly for some time,
> > which is awkward ... but OTOH this feels a bit hackish.
> > 
> > No comments at all? :)
> 
> *crickets*

Sorry, I keep meaning to but forgetting to respond to this.

I had a thought about this tonight: maybe the solution is to define
a new radiotap field, "hint," that contains a hint about the offset
to a field.  The field begins with a 16-bit (?) field number, f, to
be interpreted in the current namespace.  Following that is a 16-bit
(?) absolute offset from the end of the hint field to field f.  If you
introduce your vendor namespace when the last-assigned presence bit is
bit p, and later you reuse the vendor namespace with newly-assigned
presence bit p+k, then you can supply a hint for the vendor namespace
field so that old readers can still benefit from presence bits 0..p and
your vendor fields.

Anyway, I've only just thought-up the hint field, so I am not even sure
that it's well-defined.

Dave

-- 
David Young
dyoung-e+AXbWqSrlAAvxtiuMwx3w@public.gmane.org    Urbana, IL    (217) 721-9981

Re: [RFA] TLV fields for radiotap

[Johannes Berg]

Hi Dave,

> Sorry, I keep meaning to but forgetting to respond to this.

No worries, but thanks you took the time to.

> I had a thought about this tonight: maybe the solution is to define
> a new radiotap field, "hint," that contains a hint about the offset
> to a field.  The field begins with a 16-bit (?) field number, f, to
> be interpreted in the current namespace.  Following that is a 16-bit
> (?) absolute offset from the end of the hint field to field f.  If you
> introduce your vendor namespace when the last-assigned presence bit is
> bit p, and later you reuse the vendor namespace with newly-assigned
> presence bit p+k, then you can supply a hint for the vendor namespace
> field so that old readers can still benefit from presence bits 0..p and
> your vendor fields.
> 
> Anyway, I've only just thought-up the hint field, so I am not even sure
> that it's well-defined.

Hmm. Actually, yes, I think that would work. However, it's hard to guess
which fields the parser can do, so unless you have a really smart parser
you'd end up with

fields 0..n, hint, n+1, hint, n+2, hint, n+3, vendor_data

A smarter parser that understands 0..n+2 might be able to read

fields 0..n, hint, n+1, n+2, n+3, vendor_data

but it'd have to keep track of the hint, and if it doesn't care about
the contents of the hint ... it could still restart parsing at that
point.

Also, you'd have to reserve a high bit for the "hint" field in every
presence bitmap (like we do for the extension and namespace switching
bits), and to have multiple hint fields you'd have to extend your
presence bitmap all the time too.

Actually, you say "in the current namespace", so you'd have to switch
namespaces for the hint field to vendor, and then back to normal. If you
imagine doing this in the near future when we've run out of bits in the
first presence DWORD, then to have a hint field followed by e.g.
radiotap field 32 (next presence dword bit 0), you'd need

<presence dword: (fields 0..n) | extension | switch to vendor>
<presence dword: 0 | hint | extension | switch to radiotap>
<presence dword: 0 | extension>
<presence dword: field 32 i.e. BIT(0) | extension | switch to vendor>
<presence dword: vendor field>

There, the third presence dword is just a filler to reach the field32
again after the extension. This is also something we didn't really
consider when we designed the extension/namespace switching.

Personally, I feel this is too complex, and only really addresses the
one case I mentioned with vendor fields. I could, btw, also address it
today without the hint field by ordering the presence bitmaps
accordingly, i.e. putting the new unknown fields after the vendor
extension, by just extending the presence bitmap further and further.
But again, then I actually need to know which parts the tools can parse,
etc.

With TLVs, you could also imagine a tool that really only cares about a
single radiotap field, and is able to parse it out easily without
knowing about all the other fields.

I'll need to think about it more, but I tend to think the complexity of
the parser and generator are a bit too much. It's already hard enough
(too hard really) to parse radiotap, this only makes it even harder.

johannes

Re: [RFA] TLV fields for radiotap

[Johannes Berg]

On Tue, 2018-12-18 at 14:45 +0100, Johannes Berg wrote:

> > I had a thought about this tonight: maybe the solution is to define
> > a new radiotap field, "hint," that contains a hint about the offset
> > to a field.  The field begins with a 16-bit (?) field number, f, to
> > be interpreted in the current namespace.  Following that is a 16-bit
> > (?) absolute offset from the end of the hint field to field f.  If you
> > introduce your vendor namespace when the last-assigned presence bit is
> > bit p, and later you reuse the vendor namespace with newly-assigned
> > presence bit p+k, then you can supply a hint for the vendor namespace
> > field so that old readers can still benefit from presence bits 0..p and
> > your vendor fields.

I thought about this again ... and I decided I still don't think it's a
good idea.

> Hmm. Actually, yes, I think that would work. However, it's hard to guess
> which fields the parser can do, so unless you have a really smart parser
> you'd end up with
> 
> fields 0..n, hint, n+1, hint, n+2, hint, n+3, vendor_data

I think this one's the killer to me, in a sense.

If you think about it - yes, right now, you could say

   Implementations currently understand fields 0..HE-MU, and I want to
   have some new field (e.g. S1G) and will add a hint.

With this thinking, you get:

 0..HE-MU, vendor-data-hint, S1G, vendor-data

But like I described above, the next time you do this you probably don't
want to change it to be

 0..HE-MU, S1G, vendor-data-hint, EHT, vendor-data

but rather

 0..HE-MU, vendor-data-hint, S1G, vendor-data-hint, EHT, vendor-data

and that gets complex really fast, and also *bigger* than any sort of
TLV formatting would ever get.

I'll resend the adoption proposal now, I think it's the better solution.

johannes

[RFA] TLV fields for radiotap

[Johannes Berg]

--
title: TLV fields in radiotap
categories: [suggested]
--
Bit Number:
: 28

Structure:
: list of TLVs detailed below

Required alignment
: 4

Unit(s)
: unspecified

If this bit is set higher bits may not be set (as they cannot be
parsed, given the variable length of this field) and the data of this
field fills the remaining radiotap data (as indicated by the length
field in the header); the data is understood as a type-length-value item
list with the following item structure:

 * u16 type
 * u16 length
 * data ("length" bytes)
 * 0-3 padding bytes
   The whole item is padded to a multiple of 4 bytes length, but that
   padding is not taken into account in the 'length' field.

The type is taken from the regular radiotap type (bit) allocation, but
the special values 29 and 31 are not valid.

The vendor namespace type (30) may be used as a TLV field, but has the
following modified format as a TLV:
 - u16 type (30)
 - u16 length (length of data, as usual)
 - data containing
   - u8 OUI[3]
   - u8 subtype
   - u16 presence type (index of the bit previously used in the
                        presence bitmap, e.g. 0 for BIT(0))
   - vendor data
 - padding (if needed)
As a consequence, multiple TLV fields are needed for multiple vendor
data items.


If field alignment is necessary beyond the 4-byte alignment, type 28
(i.e. this field value) shall be used as padding with an appropriate
length (which would likely often be 0 for 8-byte alignment). This
ensures parser simplicity, it doesn't need to be aware of alignment
rules.


Generators should generate data in fields with type < 28 in regular non-
TLV fields, this ensures backward compatibility.

Generators shall use TLV fields for all fields with type >= 32.

Parsers shall be able to parse TLV fields with partial data present as
indicated by the length of the data, i.e. producers can omit bytes they
don't fill at the end of the data. Such bytes are assumed to be 0.