* [RFC] capture file timestamping @ 2015-08-25 10:07 Johannes Berg [not found] ` <1440497229.2192.28.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> 0 siblings, 1 reply; 7+ messages in thread From: Johannes Berg @ 2015-08-25 10:07 UTC (permalink / raw) To: radiotap-S783fYmB3Ccdnm+yROfE0A; +Cc: Simon Barber Hi all, [Simon brought up a related topic a few years ago] We have a field for TSFT/mactime, which is defined as follows: Value in microseconds of the MAC's 64-bit 802.11 Time Synchronization Function timer when the first bit of the MPDU arrived at the MAC. For received frames only. This is great, but not always feasible. Particularly when operating within multiple virtual interfaces and trying to record data from the hardware at the same time, the TSFT is either unobtainable or practically useless. Similarly, in actual monitor mode, there might not be a TSF timer - just a base hardware timer that is essentially free-running (though with the same frequency.) As Simon also found, the reference might not be the "first bit of the MPDU" (which I'd also say is actually somewhat vague with OFDM symbols), but something else entirely. There's also something that I'm not sure about - in 5/10 MHz channels, as I understand this is often achieved by running hardware timers at different frequencies - does that affect the timestamps hardware reports here? I'm not sure we'll be able to capture everything in a single value, but we could add a field that has a "type" that can be extended in the future. Say, for example, we add a new field like this: Structure: u64 timestamp, u32 flags Alignment: 8 We'd probably need flags for * TSF timer * first bit/symbol of MPDU (these two allow reproducing the existing TSFT field functionality, which might not be necessary, but each could still be useful) * 32-bit only [our hardware only has an internal 32-bit timestamp] * end-of-frame [maybe define this better?] We could also go into the various other places of the frame where a timestamp can be taken - our hardware does it when it detects energy on the air. The most common use for this would really be looking at monitor files, so for me personally simply having a more generic "timestamp" field that doesn't specify much of its semantics would be sufficient - however, I believe that to be very dangerous as somebody's semantics will eventually get hard-coded into wireshark and then it's fixed anyway ... Anyone want to offer any thoughts? johannes ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <1440497229.2192.28.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org>]
* Re: [RFC] capture file timestamping [not found] ` <1440497229.2192.28.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> @ 2015-08-25 16:28 ` David Young [not found] ` <20150825162841.GR6823-e+AXbWqSrlAAvxtiuMwx3w@public.gmane.org> 0 siblings, 1 reply; 7+ messages in thread From: David Young @ 2015-08-25 16:28 UTC (permalink / raw) To: Johannes Berg; +Cc: radiotap-S783fYmB3Ccdnm+yROfE0A, Simon Barber On Tue, Aug 25, 2015 at 12:07:09PM +0200, Johannes Berg wrote: > Hi all, > > [Simon brought up a related topic a few years ago] > > We have a field for TSFT/mactime, which is defined as follows: > > Value in microseconds of the MAC's 64-bit 802.11 Time > Synchronization Function timer when the first bit of the MPDU > arrived at the MAC. For received frames only. > > This is great, but not always feasible. Particularly when operating > within multiple virtual interfaces and trying to record data from the > hardware at the same time, the TSFT is either unobtainable or > practically useless. Can you explain why it is unobtainable or practically useless? > Similarly, in actual monitor mode, there might not be a TSF timer - > just a base hardware timer that is essentially free-running (though > with the same frequency.) I agree that it's not quite right to call it TSF time when there is not actually any synchronization of stations. > As Simon also found, the reference might not be the "first bit of the > MPDU" (which I'd also say is actually somewhat vague with OFDM > symbols), but something else entirely. Hmm. What's the reference for beacons and probe responses? > There's also something that I'm not sure about - in 5/10 MHz channels, > as I understand this is often achieved by running hardware timers at > different frequencies - does that affect the timestamps hardware > reports here? Is it fair to say that you are generally concerned that not all hardware can stamp received frames with something like a TSF time, but some hardware can stamp frames with a time that is the "moral equivalent" for certain purposes? What are the use cases for the timestamps? My use cases were these: I. Measuring nanosecond intervals between several consecutive frames A. Using the measurements to verify settings of device registers B. Deriving from several intervals the meters distance between two 802.11 stations II. Using the timestamps to place graphic representations of frames on a timeline I can certainly imagine someone using hardware timestamps to put frames back into the order they were received if the order was scrambled by, for example, a multiprocessing network stack. Dave -- David Young dyoung-e+AXbWqSrlAAvxtiuMwx3w@public.gmane.org Urbana, IL (217) 721-9981 ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <20150825162841.GR6823-e+AXbWqSrlAAvxtiuMwx3w@public.gmane.org>]
* Re: [RFC] capture file timestamping [not found] ` <20150825162841.GR6823-e+AXbWqSrlAAvxtiuMwx3w@public.gmane.org> @ 2015-08-25 16:46 ` Johannes Berg [not found] ` <1440521168.2192.50.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> 0 siblings, 1 reply; 7+ messages in thread From: Johannes Berg @ 2015-08-25 16:46 UTC (permalink / raw) To: David Young; +Cc: radiotap-S783fYmB3Ccdnm+yROfE0A, Simon Barber On Tue, 2015-08-25 at 11:28 -0500, David Young wrote: > > This is great, but not always feasible. Particularly when operating > > within multiple virtual interfaces and trying to record data from > > the > > hardware at the same time, the TSFT is either unobtainable or > > practically useless. > > Can you explain why it is unobtainable or practically useless? If you have two stations, for example, you need to have synchronisation with the two different APs, so you can have two vastly different TSF values. Maybe your device gives those to you, or maybe not. It may make sense for some analysis to look at that, but if you need to look at timing inside the device to be evident then this is useless. > > Similarly, in actual monitor mode, there might not be a TSF timer - > > just a base hardware timer that is essentially free-running (though > > with the same frequency.) > > I agree that it's not quite right to call it TSF time when there is > not actually any synchronization of stations. It's not really a major defect though. > > As Simon also found, the reference might not be the "first bit of the > > MPDU" (which I'd also say is actually somewhat vague with OFDM > > symbols), but something else entirely. > > Hmm. What's the reference for beacons and probe responses? First symbol containing the first bit of the timestamp field, I believe. Haven't double-checked now. > Is it fair to say that you are generally concerned that not all hardware > can stamp received frames with something like a TSF time, but some > hardware can stamp frames with a time that is the "moral equivalent" for > certain purposes? Yeah, you could put it that way. > What are the use cases for the timestamps? My use cases were these: > > I. Measuring nanosecond intervals between several consecutive frames I don't think I have ever seen nanoseconds, and we don't currently allow that resolution. However, it's a good point - any new field should probably take resolution into account. > A. Using the measurements to verify settings of device registers This could be done I guess. > B. Deriving from several intervals the meters distance > between two 802.11 stations Possible, but I'm not sure it's all that feasible. > II. Using the timestamps to place graphic representations of frames > on a timeline This is one. One I ran into recently is that I needed to check the relative timings of packets to verify the backoff function was working correctly, which is really just very similar to your I-A case. > I can certainly imagine someone using hardware timestamps to put frames > back into the order they were received if the order was scrambled by, > for example, a multiprocessing network stack. The timestamps added by the capture (e.g. tcpdump) are also notoriously unreliable since there are processing delays - even without reordering. I was trying to be a bit more generic, but in my case it mostly boils down to this: a) the timestamps recorded by (e.g.) tcpdump in the pcap packet headers are unreliable b) the timestamps there are at the end of the frame, beginning is much more useful c) the timer I'm getting is an internal hardware timer and I can't easily derive either a TSF-based value, nor easily put it at the first bit/symbol of the MPDU (since it's earlier) [*] johannes [*] it's actually also somewhat unreliable, due to the reference it's taken at, but that's something I can generally live with, it's in the order of a few microseconds, not the milliseconds I sometimes see with tcpdump ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <1440521168.2192.50.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org>]
* Re: [RFC] capture file timestamping [not found] ` <1440521168.2192.50.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> @ 2015-08-25 17:47 ` Guy Harris [not found] ` <4E4FE63E-B636-4808-BBFF-6D2CE4EAFB2F-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> 2015-08-26 22:48 ` Simon Barber 1 sibling, 1 reply; 7+ messages in thread From: Guy Harris @ 2015-08-25 17:47 UTC (permalink / raw) To: Johannes Berg; +Cc: David Young, radiotap-S783fYmB3Ccdnm+yROfE0A, Simon Barber On Aug 25, 2015, at 9:46 AM, Johannes Berg <johannes-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> wrote: > The timestamps added by the capture (e.g. tcpdump) are also notoriously > unreliable since there are processing delays - even without reordering. If you have tcpdump 4.6 or later, with libpcap 1.6 or later, on Linux, you should have the --time-stamp-type and --list-time-stamp-types options. If so, what does tcpdump --list-time-stamp-types print? If it prints "adapter" or "adapter_unsynced", you might want to try those with the --time-stamp-type option, as those time stamp types mean that the time stamp will come from the adapter rather than from Linux. ("unsynced" means that the time stamps aren't synchronized with the host's clock.) That won't address > b) the timestamps there are at the end of the frame, > beginning is much more useful unless that's how the adapter time-stamps the packet and won't address > c) the timer I'm getting is an internal hardware timer and I can't > easily derive either a TSF-based value, nor easily put it at the > first bit/symbol of the MPDU (since it's earlier) [*] unless it's based on the TSF (which it probably won't be, as it should, at least in principle, be Epoch time, although I think Linux may have, at some point, changed adapter time stamps in a fashion making them not useful as libpcap time stamps). (I.e., the "host time stamps are applied at times not well correlated with actual network events" problem is not best solved with radiotap, it's best solved by libpcap and the capture mechanism atop which it runs, but "I want a time stamp that comes from the TSF timers" is one best solved with radiotap.) ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <4E4FE63E-B636-4808-BBFF-6D2CE4EAFB2F-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org>]
* Re: [RFC] capture file timestamping [not found] ` <4E4FE63E-B636-4808-BBFF-6D2CE4EAFB2F-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> @ 2015-08-25 17:55 ` Johannes Berg 0 siblings, 0 replies; 7+ messages in thread From: Johannes Berg @ 2015-08-25 17:55 UTC (permalink / raw) To: Guy Harris; +Cc: David Young, radiotap-S783fYmB3Ccdnm+yROfE0A, Simon Barber On Tue, 2015-08-25 at 10:47 -0700, Guy Harris wrote: > > If you have tcpdump 4.6 or later, with libpcap 1.6 or later, on > Linux, you should have the --time-stamp-type and --list-time-stamp > -types options. > > If so, what does > > > tcpdump --list-time-stamp-types > > print? > > If it prints "adapter" or "adapter_unsynced", you might want to try > those with the --time-stamp-type option, as those time stamp types > mean that the time stamp will come from the adapter rather than from > Linux. ("unsynced" means that the time stamps aren't synchronized > with the host's clock.) It does have "adapter_unsynced", but wouldn't the adapter also have to support it? Ah, yes, if I do tcpdump -i wlan0 --list-time-stamp-types it no longer shows it. Although perhaps I could possibly convince the driver to report this timestamp somehow, rather than changing radiotap. > unless it's based on the TSF (which it probably won't be, as it > should, at least in principle, be Epoch time, although I think Linux > may have, at some point, changed adapter time stamps in a fashion > making them not useful as libpcap time stamps). That, however, I almost certainly cannot do - I don't see how I'd synchronize from the adapter time (a free-running 32-bit counter at microseconds resolution) to the host time. johannes ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC] capture file timestamping [not found] ` <1440521168.2192.50.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> 2015-08-25 17:47 ` Guy Harris @ 2015-08-26 22:48 ` Simon Barber [not found] ` <55DE4244.9030407-vp0mx6+5gkqFX2APIN6yfw@public.gmane.org> 1 sibling, 1 reply; 7+ messages in thread From: Simon Barber @ 2015-08-26 22:48 UTC (permalink / raw) To: Johannes Berg, David Young; +Cc: radiotap-S783fYmB3Ccdnm+yROfE0A On 8/25/2015 9:46 AM, Johannes Berg wrote: > >> II. Using the timestamps to place graphic representations of frames >> on a timeline > This is one. > > One I ran into recently is that I needed to check the relative timings > of packets to verify the backoff function was working correctly, which > is really just very similar to your I-A case. > > This is exactly what I was doing when I ran into the current TSFT field limitations a few years ago. GPL code is here: https://github.com/parc-wifi/wireshark I'm working on getting this updated for 11ac now, so revisiting some of the same issues. The HT and VHT headers in radiotap are incomplete, and would be much nicer if they could just include the signal field from the plcp (perhaps with a mask to indicate parts that are missing). This would be a better approach for 60GHz I think. I'd like to display the L-SIG txop protection on the timeline as well. Simon ^ permalink raw reply [flat|nested] 7+ messages in thread
[parent not found: <55DE4244.9030407-vp0mx6+5gkqFX2APIN6yfw@public.gmane.org>]
* Re: [RFC] capture file timestamping [not found] ` <55DE4244.9030407-vp0mx6+5gkqFX2APIN6yfw@public.gmane.org> @ 2015-09-17 16:51 ` Johannes Berg 0 siblings, 0 replies; 7+ messages in thread From: Johannes Berg @ 2015-09-17 16:51 UTC (permalink / raw) To: Simon Barber, David Young; +Cc: radiotap-S783fYmB3Ccdnm+yROfE0A On Wed, 2015-08-26 at 15:48 -0700, Simon Barber wrote: > > This is exactly what I was doing when I ran into the current TSFT > field limitations a few years ago. Right, I vaguely remembered that, hence the CC :) > GPL code is here: > > https://github.com/parc-wifi/wireshark > I'm working on getting this updated for 11ac now, so revisiting some > of the same issues. Ah, that's really nice! I really needed something like this the other day, will check it out. > The HT and VHT headers in radiotap are incomplete, and would be much > nicer if they could just include the signal field from the plcp > (perhaps with a mask to indicate parts that are missing). This would > be a better approach for 60GHz I think. Yeah, can't really object to that, even if we might not be able to provide all the data for all the hardware, and might have to re -assemble the field from the available parsed data. As far as our device is concerned we might perhaps be able to include this data in firmware's monitor mode. > I'd like to display the L-SIG txop protection on the timeline as > well. > Good idea. As far as the timestamp is concerned though, what would you suggest? Also, what timestamp would be best for a tool like yours? I'm assuming ours, which is the signal detection timestamp and inherently vague by perhaps a dozen microseconds won't really work that well. johannes ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-09-17 16:51 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-08-25 10:07 [RFC] capture file timestamping Johannes Berg [not found] ` <1440497229.2192.28.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> 2015-08-25 16:28 ` David Young [not found] ` <20150825162841.GR6823-e+AXbWqSrlAAvxtiuMwx3w@public.gmane.org> 2015-08-25 16:46 ` Johannes Berg [not found] ` <1440521168.2192.50.camel-cdvu00un1VgdHxzADdlk8Q@public.gmane.org> 2015-08-25 17:47 ` Guy Harris [not found] ` <4E4FE63E-B636-4808-BBFF-6D2CE4EAFB2F-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> 2015-08-25 17:55 ` Johannes Berg 2015-08-26 22:48 ` Simon Barber [not found] ` <55DE4244.9030407-vp0mx6+5gkqFX2APIN6yfw@public.gmane.org> 2015-09-17 16:51 ` Johannes Berg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).