Network RX per process per interface statistics

Network RX per process per interface statistics

[Kamil Zaripov]

Hi everyone,

I trying to make a BPF program that can collect per process per interface statistics of network data consumption. Right now most difficult part for me is RX traffic.

I have tried to find some point in the sk_buff's way up to network stack where I can extract info both about the network interface which captured package and the process that will consume this data but failed. So I have to listen events in several points and somehow merge collected data.

The last point I found at which sk_buff still contains information about network device that captured this sk_buff is netif_receive_skb tracepoint. The first point where I can found information about process is protocol's rcv handlers (like tcp_v4_do_rcv). But I have some questions, to finish my program:

1. It seems that sk_buff modifies during handling, so how can I "match" sk_buff with same data in netif_receive_skb and in tcp_v4_do_rcv?
2. Maybe there is some good point where I can attach listener and where I can extract both process and interface info for each package?

Regards
Zaripov Kamil.

Re: Network RX per process per interface statistics

[Stanislav Fomichev]

On 03/24, Kamil Zaripov wrote:
> Hi everyone,

> I trying to make a BPF program that can collect per process per interface  
> statistics of network data consumption. Right now most difficult part for  
> me is RX traffic.

> I have tried to find some point in the sk_buff's way up to network stack  
> where I can extract info both about the network interface which captured  
> package and the process that will consume this data but failed. So I have  
> to listen events in several points and somehow merge collected data.

> The last point I found at which sk_buff still contains information about  
> network device that captured this sk_buff is netif_receive_skb  
> tracepoint. The first point where I can found information about process  
> is protocol's rcv handlers (like tcp_v4_do_rcv). But I have some  
> questions, to finish my program:

> 1. It seems that sk_buff modifies during handling, so how can I "match"  
> sk_buff with same data in netif_receive_skb and in tcp_v4_do_rcv?

By "modifies" - do you mean the payload/headers? You can probably use
the skb pointer address as a unique identifier to connect across different
tracepoints?

> 2. Maybe there is some good point where I can attach listener and where I  
> can extract both process and interface info for each package?

Nothing pops to my mind. But I think that if you store skbaddr=dev from
netif_receive_skb, you should be able to look this up at a later point
where you know skb->process association?

> Regards
> Zaripov Kamil.

Re: Network RX per process per interface statistics

[Kamil Zaripov]

> By "modifies" - do you mean the payload/headers? You can probably use
> the skb pointer address as a unique identifier to connect across different
> tracepoints?

No, I mean when situations when same package through its way to the network stack change sk_buff pointer. For example, after skb_clone() call. I have made some test and found out (empirically) that pointer to the skb->head a much better tracking ID. However, I am not sure that there is no other corner cases when skb->head also can change.

> Nothing pops to my mind. But I think that if you store skbaddr=dev from
> netif_receive_skb, you should be able to look this up at a later point
> where you know skb->process association?

Yes, I have already implemented and make some test of this approach: I’m listening at netif_receive_skb tracepoint to create skb_head->netif map and then listening for __kfree_skb calls to create pid->skb_head map. However, this approach have some weaknesses:
- Part of packages of TCP protocol packages (ACK, for example) are handled by the kernel, so I account this packages as kernel activity. But almost every TCP ACK package have some  associated socket, which, in turn, have associated process.
- I am not sure that all package consumes call __kfree_skb at the end. Maybe there is some other miscounting in this place.

Maybe there is some other approaches to map packages to processes?

Re: Network RX per process per interface statistics

[Stanislav Fomichev]

On 03/27, Kamil Zaripov wrote:
> > By "modifies" - do you mean the payload/headers? You can probably use
> > the skb pointer address as a unique identifier to connect across  
> different
> > tracepoints?

> No, I mean when situations when same package through its way to the  
> network stack change sk_buff pointer. For example, after skb_clone()  
> call. I have made some test and found out (empirically) that pointer to  
> the skb->head a much better tracking ID. However, I am not sure that  
> there is no other corner cases when skb->head also can change.

Yeah, those are tricky :-(

> > Nothing pops to my mind. But I think that if you store skbaddr=dev from
> > netif_receive_skb, you should be able to look this up at a later point
> > where you know skb->process association?

> Yes, I have already implemented and make some test of this approach: I’m  
> listening at netif_receive_skb tracepoint to create skb_head->netif map  
> and then listening for __kfree_skb calls to create pid->skb_head map.  
> However, this approach have some weaknesses:
> - Part of packages of TCP protocol packages (ACK, for example) are  
> handled by the kernel, so I account this packages as kernel activity. But  
> almost every TCP ACK package have some  associated socket, which, in  
> turn, have associated process.
> - I am not sure that all package consumes call __kfree_skb at the end.  
> Maybe there is some other miscounting in this place.

> Maybe there is some other approaches to map packages to processes?

I'm not super familiar with those tracing hooks. Maybe you need to
handle consume_skb as well?

If I were to solve something like this, I'd probably look at cgroup/ingress
hooks. Those are guaranteed to run for every incoming packet into cgroup's
sockets. (at least removes that kfree_skb vs consume_skb issue).

But it doesn't solve your problem with the clones...