All of lore.kernel.org
 help / color / mirror / Atom feed
* Problems obtaining software TX timestamps
@ 2010-09-08 12:11 Ingo Kofler
  2010-09-08 15:12 ` Eric Dumazet
  0 siblings, 1 reply; 11+ messages in thread
From: Ingo Kofler @ 2010-09-08 12:11 UTC (permalink / raw)
  To: netdev

Dear all,

I am trying to obtain software TX timestamps for a each outgoing packet 
of an UDP socket. For that purpose I set the SO_TIMESTAMPING socket 
option to SOF_TIMESTAMPING_TX_SOFTWARE | SOF_TIMESTAMPING_SOFTWARE. The 
setsockopt call succeeds and also a subsequent getsockopt returns the 
correct flags. However, if a send a single datagram over the socket and 
try to fetch the packet as well as its timestamp via the socket's 
MSG_ERRQUEUE I do not get any information. Rather, the select call waits 
infinitely on the error queue....

I hardly found any documentation on this timestamping mechanism except 
the text file in the kernel source code. All I've figured out is that if 
I want to have hardware TX timestamps, I'll have to make a further ioctl 
which requires root permissions to active the hardware timestamping. But 
this is not what I want... software timestamps are sufficient for me. 
Here's my code....


int main (int argc, char **argv) {

    int rv;
    int sock;

    // allocate a socket
    printf("allocating socket\n");
    sock = socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP);
    if (sock==-1) return 1;

    // connect socket to IP + port passed via command line
    printf("connecting socket\n");
    struct sockaddr_in toAddr;
    memset(&toAddr, 0, sizeof(struct sockaddr_in));
    toAddr.sin_family = PF_INET;
    toAddr.sin_addr.s_addr = inet_addr(argv[1]);
    toAddr.sin_port = htons(atoi(argv[2]));
    rv = connect(sock, (struct sockaddr*) &toAddr, sizeof(struct 
sockaddr_in));
    if (rv!=0) return 1;

    // try to enable software TX timestamps
    printf("enabling (?) timestamping\n");
    int flags =  SOF_TIMESTAMPING_TX_SOFTWARE | SOF_TIMESTAMPING_SOFTWARE;
    rv = setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &flags, 
sizeof(flags));
    if (rv!=0) return 2;

    // check if the setsockopt was successful
    socklen_t slen = sizeof(flags);
    rv = getsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &flags, &slen);
    if (rv!=0) return 3;
    printf("getsockopt returned %d\n", flags);
   
    // send something via the socket
    char buffer[200];
    int len = sizeof(buffer);
    rv = send(sock, buffer, len, 0);
    printf("send returned %d\n", rv);

    // prepare select call to wait for packet on MSG_ERRQUEUE
    fd_set errorfs;
    FD_ZERO(&errorfs);   
    FD_SET(sock, &errorfs);
    rv = select(sock + 1, 0, 0, &errorfs, NULL);                 <<--- 
This select blocks forever...
    printf("select returned %d\n", rv);
    ...

Any suggestions what's wrong with my code?  Btw. I've tested this with 
kernel 2.6.31 and 2.6.33...

Thanks and best regards,
Ingo

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-08 12:11 Problems obtaining software TX timestamps Ingo Kofler
@ 2010-09-08 15:12 ` Eric Dumazet
  2010-09-08 21:21   ` Ingo Kofler
  0 siblings, 1 reply; 11+ messages in thread
From: Eric Dumazet @ 2010-09-08 15:12 UTC (permalink / raw)
  To: Ingo Kofler; +Cc: netdev

Le mercredi 08 septembre 2010 à 14:11 +0200, Ingo Kofler a écrit :
> Dear all,
> 
> I am trying to obtain software TX timestamps for a each outgoing packet 
> of an UDP socket. For that purpose I set the SO_TIMESTAMPING socket 
> option to SOF_TIMESTAMPING_TX_SOFTWARE | SOF_TIMESTAMPING_SOFTWARE. The 
> setsockopt call succeeds and also a subsequent getsockopt returns the 
> correct flags. However, if a send a single datagram over the socket and 
> try to fetch the packet as well as its timestamp via the socket's 
> MSG_ERRQUEUE I do not get any information. Rather, the select call waits 
> infinitely on the error queue....
> 
> I hardly found any documentation on this timestamping mechanism except 
> the text file in the kernel source code. All I've figured out is that if 
> I want to have hardware TX timestamps, I'll have to make a further ioctl 
> which requires root permissions to active the hardware timestamping. But 
> this is not what I want... software timestamps are sufficient for me. 

To my knowledge, no driver calls skb_tx_timestamp()

You'll need to tweak your NIC driver to play with this TX timestamps



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-08 15:12 ` Eric Dumazet
@ 2010-09-08 21:21   ` Ingo Kofler
  2010-09-08 21:24     ` David Miller
  2010-09-09  6:35     ` Richard Cochran
  0 siblings, 2 replies; 11+ messages in thread
From: Ingo Kofler @ 2010-09-08 21:21 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: netdev


On 09/08/2010 05:12 PM, Eric Dumazet wrote:
> Le mercredi 08 septembre 2010 à 14:11 +0200, Ingo Kofler a écrit :
>    
>> Dear all,
>>
>> I am trying to obtain software TX timestamps for a each outgoing packet
>> of an UDP socket. For that purpose I set the SO_TIMESTAMPING socket
>> option to SOF_TIMESTAMPING_TX_SOFTWARE | SOF_TIMESTAMPING_SOFTWARE. The
>> setsockopt call succeeds and also a subsequent getsockopt returns the
>> correct flags. However, if a send a single datagram over the socket and
>> try to fetch the packet as well as its timestamp via the socket's
>> MSG_ERRQUEUE I do not get any information. Rather, the select call waits
>> infinitely on the error queue....
>>
>> I hardly found any documentation on this timestamping mechanism except
>> the text file in the kernel source code. All I've figured out is that if
>> I want to have hardware TX timestamps, I'll have to make a further ioctl
>> which requires root permissions to active the hardware timestamping. But
>> this is not what I want... software timestamps are sufficient for me.
>>      
> To my knowledge, no driver calls skb_tx_timestamp()
>
> You'll need to tweak your NIC driver to play with this TX timestamps
>    
Ok, for hardware timestamps it seems logically for me that the driver 
has to support this. But what about the software timestamps - do they 
also require support from the driver? Is there no generic implementation 
in the networking stack?
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>    


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-08 21:21   ` Ingo Kofler
@ 2010-09-08 21:24     ` David Miller
  2010-09-09  6:35     ` Richard Cochran
  1 sibling, 0 replies; 11+ messages in thread
From: David Miller @ 2010-09-08 21:24 UTC (permalink / raw)
  To: ikofler; +Cc: eric.dumazet, netdev

From: Ingo Kofler <ikofler@edu.uni-klu.ac.at>
Date: Wed, 08 Sep 2010 23:21:36 +0200

> Ok, for hardware timestamps it seems logically for me that the driver
> has to support this. But what about the software timestamps - do they
> also require support from the driver? Is there no generic
> implementation in the networking stack?

Each and every driver must add the hook because it must be inside of
the driver's transmit function due to packet lifetime issues etc.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-08 21:21   ` Ingo Kofler
  2010-09-08 21:24     ` David Miller
@ 2010-09-09  6:35     ` Richard Cochran
  2010-09-09  8:26       ` Ingo Kofler
  1 sibling, 1 reply; 11+ messages in thread
From: Richard Cochran @ 2010-09-09  6:35 UTC (permalink / raw)
  To: Ingo Kofler; +Cc: Eric Dumazet, netdev

On Wed, Sep 08, 2010 at 11:21:36PM +0200, Ingo Kofler wrote:
> Ok, for hardware timestamps it seems logically for me that the
> driver has to support this. But what about the software timestamps -
> do they also require support from the driver? Is there no generic
> implementation in the networking stack?

There was once a completely generic implementation, but the Tx path
had issues. The generic Tx SW code was reverted in

cd4d8fdad1f13205c769266dfa99015e226b6e07

Now, there is a way to support this again, but it requires adding one
line into each and every MAC driver. It is not very hard to do:

linux/skbuff.h:1992
/**
 * skb_tx_timestamp() - Driver hook for transmit timestamping
 *
 * Ethernet MAC Drivers should call this function in their hard_xmit()
 * function as soon as possible after giving the sk_buff to the MAC
 * hardware, but before freeing the sk_buff.
 *
 * @skb: A socket buffer.
 */
static inline void skb_tx_timestamp(struct sk_buff *skb);

I am already carrying patches for this privately. If you post a patch
for your MAC, then you can be the first one to support this feature! ;^)

Richard


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-09  6:35     ` Richard Cochran
@ 2010-09-09  8:26       ` Ingo Kofler
  2010-09-09 13:57         ` Richard Cochran
  0 siblings, 1 reply; 11+ messages in thread
From: Ingo Kofler @ 2010-09-09  8:26 UTC (permalink / raw)
  To: Richard Cochran; +Cc: Eric Dumazet, netdev

Richard Cochran wrote:
> On Wed, Sep 08, 2010 at 11:21:36PM +0200, Ingo Kofler wrote:
>   
>> Ok, for hardware timestamps it seems logically for me that the
>> driver has to support this. But what about the software timestamps -
>> do they also require support from the driver? Is there no generic
>> implementation in the networking stack?
>>     
>
> There was once a completely generic implementation, but the Tx path
> had issues. The generic Tx SW code was reverted in
>
> cd4d8fdad1f13205c769266dfa99015e226b6e07
>
> Now, there is a way to support this again, but it requires adding one
> line into each and every MAC driver. It is not very hard to do:
>
> linux/skbuff.h:1992
> /**
>  * skb_tx_timestamp() - Driver hook for transmit timestamping
>  *
>  * Ethernet MAC Drivers should call this function in their hard_xmit()
>  * function as soon as possible after giving the sk_buff to the MAC
>  * hardware, but before freeing the sk_buff.
>  *
>  * @skb: A socket buffer.
>  */
> static inline void skb_tx_timestamp(struct sk_buff *skb);
>
> I am already carrying patches for this privately. If you post a patch
> for your MAC, then you can be the first one to support this feature! ;^)
>   
Thank you very much for this clarification. Are there any drawbacks with 
this approach, e.g. performance issues? Honestly, I am just wondering 
why driver developers do not add this single line if it's that easy....

Ingo
> Richard
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-09  8:26       ` Ingo Kofler
@ 2010-09-09 13:57         ` Richard Cochran
  2010-09-09 14:02           ` Eric Dumazet
  0 siblings, 1 reply; 11+ messages in thread
From: Richard Cochran @ 2010-09-09 13:57 UTC (permalink / raw)
  To: Ingo Kofler; +Cc: Eric Dumazet, netdev

On Thu, Sep 09, 2010 at 10:26:06AM +0200, Ingo Kofler wrote:
> Thank you very much for this clarification. Are there any drawbacks
> with this approach, e.g. performance issues? Honestly, I am just
> wondering why driver developers do not add this single line if it's
> that easy....

Adding the line only tests one flag, so it is not a huge performance
hit. This fix is very recent, that is why it has not been used in MAC
drivers yet.

Richard


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-09 13:57         ` Richard Cochran
@ 2010-09-09 14:02           ` Eric Dumazet
  2010-09-09 14:31             ` Loke, Chetan
  2010-09-09 17:32             ` Richard Cochran
  0 siblings, 2 replies; 11+ messages in thread
From: Eric Dumazet @ 2010-09-09 14:02 UTC (permalink / raw)
  To: Richard Cochran; +Cc: Ingo Kofler, netdev

Le jeudi 09 septembre 2010 à 15:57 +0200, Richard Cochran a écrit :
> On Thu, Sep 09, 2010 at 10:26:06AM +0200, Ingo Kofler wrote:
> > Thank you very much for this clarification. Are there any drawbacks
> > with this approach, e.g. performance issues? Honestly, I am just
> > wondering why driver developers do not add this single line if it's
> > that easy....
> 
> Adding the line only tests one flag, so it is not a huge performance
> hit. This fix is very recent, that is why it has not been used in MAC
> drivers yet.

Should drivers call it at start_xmit() time, or at tx completion time ?

1) start_xmit() time : Earlier than real hardware xmit (with up to 1000
packets queued in TX ring, delay might be very large)

2) tx completion time : After real hardware xmit
   (on tg3, ethtool -c gives : tx-usecs: 72  , tx-frames: 53, so we can
have a delay of 72 us before NIC acknowledges the frame xmit)




^ permalink raw reply	[flat|nested] 11+ messages in thread

* RE: Problems obtaining software TX timestamps
  2010-09-09 14:02           ` Eric Dumazet
@ 2010-09-09 14:31             ` Loke, Chetan
  2010-09-09 17:32             ` Richard Cochran
  1 sibling, 0 replies; 11+ messages in thread
From: Loke, Chetan @ 2010-09-09 14:31 UTC (permalink / raw)
  To: Eric Dumazet, Richard Cochran; +Cc: Ingo Kofler, netdev

> From: netdev-owner@vger.kernel.org [mailto:netdev-
> owner@vger.kernel.org] On Behalf Of Eric Dumazet
> Sent: September 09, 2010 10:03 AM
> To: Richard Cochran
> Cc: Ingo Kofler; netdev@vger.kernel.org
> Subject: Re: Problems obtaining software TX timestamps
> 
> Le jeudi 09 septembre 2010 à 15:57 +0200, Richard Cochran a écrit :
> > On Thu, Sep 09, 2010 at 10:26:06AM +0200, Ingo Kofler wrote:
> > > Thank you very much for this clarification. Are there any drawbacks
> > > with this approach, e.g. performance issues? Honestly, I am just
> > > wondering why driver developers do not add this single line if it's
> > > that easy....
> >
> > Adding the line only tests one flag, so it is not a huge performance
> > hit. This fix is very recent, that is why it has not been used in MAC
> > drivers yet.
> 
> Should drivers call it at start_xmit() time, or at tx completion time ?
> 
> 1) start_xmit() time : Earlier than real hardware xmit (with up to 1000
> packets queued in TX ring, delay might be very large)
> 
> 2) tx completion time : After real hardware xmit
>    (on tg3, ethtool -c gives : tx-usecs: 72  , tx-frames: 53, so we can
> have a delay of 72 us before NIC acknowledges the frame xmit)
> 

I'm not familiar w/ the code but I would think right before the Tx-descriptors are posted. That is, before a pci-write, where the Tx-queue put
ptrs are updated. Because if the NIC is overloaded then there will be some delay before the pkts are Xmit'd. But if there is a N/W TAP then we would atleast roughly know where we could start troubleshooting.

Chetan Loke

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-09 14:02           ` Eric Dumazet
  2010-09-09 14:31             ` Loke, Chetan
@ 2010-09-09 17:32             ` Richard Cochran
  2010-09-09 18:58               ` Eric Dumazet
  1 sibling, 1 reply; 11+ messages in thread
From: Richard Cochran @ 2010-09-09 17:32 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Ingo Kofler, netdev

On Thu, Sep 09, 2010 at 04:02:38PM +0200, Eric Dumazet wrote:
> Should drivers call it at start_xmit() time, or at tx completion time ?

If I could tell you that, I could also describe the sound of one hand
clapping.

IMHO, it should be done as immediately after the buffer has been
handed off to the hardware. Not all MACs have a completion method, so
that would at least make the "rule" consistent.

In general, the goals of high volume and timestamp accuracy are in
conflict (unless you have HW timestamping). If users are relying on SW
timestamping, then they should be aware of the fact that high network
traffic will spoil their fun.

Having said that, I don't know enough about all the different kinds of
MACs out there to really answer the question definitively.

Richard

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Problems obtaining software TX timestamps
  2010-09-09 17:32             ` Richard Cochran
@ 2010-09-09 18:58               ` Eric Dumazet
  0 siblings, 0 replies; 11+ messages in thread
From: Eric Dumazet @ 2010-09-09 18:58 UTC (permalink / raw)
  To: Richard Cochran; +Cc: Ingo Kofler, netdev

Le jeudi 09 septembre 2010 à 19:32 +0200, Richard Cochran a écrit :
> On Thu, Sep 09, 2010 at 04:02:38PM +0200, Eric Dumazet wrote:
> > Should drivers call it at start_xmit() time, or at tx completion time ?
> 

> IMHO, it should be done as immediately after the buffer has been
> handed off to the hardware. Not all MACs have a completion method, so
> that would at least make the "rule" consistent.
> 

Well, all drivers must release skb *somewhere*, either in their
ndo_start_xmit() or TX completion ;)

> In general, the goals of high volume and timestamp accuracy are in
> conflict (unless you have HW timestamping). If users are relying on SW
> timestamping, then they should be aware of the fact that high network
> traffic will spoil their fun.
> 

- Full duplex mode wanted
- And small TX rings




^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2010-09-09 18:59 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-09-08 12:11 Problems obtaining software TX timestamps Ingo Kofler
2010-09-08 15:12 ` Eric Dumazet
2010-09-08 21:21   ` Ingo Kofler
2010-09-08 21:24     ` David Miller
2010-09-09  6:35     ` Richard Cochran
2010-09-09  8:26       ` Ingo Kofler
2010-09-09 13:57         ` Richard Cochran
2010-09-09 14:02           ` Eric Dumazet
2010-09-09 14:31             ` Loke, Chetan
2010-09-09 17:32             ` Richard Cochran
2010-09-09 18:58               ` Eric Dumazet

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.