Re: Allocation of sk_buffs in the kernel

Re: Allocation of sk_buffs in the kernel

[Andi Kleen]

"Jens Hoffrichter" <HOFFRICH@de.ibm.com> writes:

> I'm currently writing a kernel patch where it is essential to get known
> when a sk_buff is allocated. Or better said I have to get known when a
> sk_buff is effectively a new packet in the kernel-

I don't want to guess why you need that...

> 
> I currently identified 3 functions in the kernel where sk_buffs are
> allocated: alloc_skb (of course), skb_linearize and pskb_expand_head. Or at
> least there new data is defined for the sk_buffs.
> 
> Now I monitor a TCP session, a FTP download better said, and on the
> interface arrives around 30000 packets for 50 MB of data. But in my kernel
> patch only 2000 packets are allocated, or at least I see only the
> allocation of 2000 packets.
> 
> Can anyone help me where I can find my missing packets? ;)) I need them
> badly! *GG*

There should be no skbuff allocation outside net/core/skbuff.c and all
normal[1] networking drivers also don't use private pools. Perhaps
you forgot to instrument a case there.

-Andi

[1] There may be a few unnormal ones that do; e.g. vendor driver
writers seem to frequently try to reuse skbuffs privately because they're
used to that from other OS. It is discouraged and somewhat tricky, but
possible.

Re: Allocation of sk_buffs in the kernel

[Arnaldo Carvalho de Melo]

Em Thu, Aug 23, 2001 at 05:14:56AM +0200, Andi Kleen escreveu:
> [1] There may be a few unnormal ones that do; e.g. vendor driver
> writers seem to frequently try to reuse skbuffs privately because they're
> used to that from other OS. It is discouraged and somewhat tricky, but
> possible.

The original 802.2 stack from procom, used frame_t all over the core stack,
with only the entry/exit points manipulating skb's. On entry from the core
networking stack and from upper protocols (NetBEUI in procom case) they
allocated a frame_t from a pool and initialited pointers to the mac and llc
headers and stored the skb type in another frame_t member, pointing to the
skb memory and used those pointers instead of skb->h and skb->nh.  That
way, I think, they could use the core for several OSes.

Maybe Jens should use something like WAITQUEUE_DEBUG if he want to know
where alloc_skb and friends were called, see include/linux/wait.h 8)

- Arnaldo

Re: Allocation of sk_buffs in the kernel

[Arnaldo Carvalho de Melo]

Em Thu, Aug 23, 2001 at 11:29:11AM +0200, Jens Hoffrichter escreveu:
> > > > Maybe Jens should use something like WAITQUEUE_DEBUG if he want to
> know
> > > > where alloc_skb and friends were called, see include/linux/wait.h 8)
> > > Do you mean I should use something LIKE the WAITQUEUE_DEBUG (eg.
> > > implementing something like that in skbuff.c) or I should use
> > > WAITQUEUE_DEBUG?
> > no, just use the same idea that is used to debug wait_queues
> OK, then I interpreted the code in wait.h right ;)
> 
> > > Are there any examples how to use the WAITQUEUE_DEBUG?
> 
> > oops, I mean the __waker thing, for debugging you could get the address
> of
> > the caller with current_text_addr() and store it in an extra sk_buff
> field
> > so that later on you could know who create the skb.
> But where should I fill this field in the sk_buff? I know that alloc_skb
> creates an sk_buff, so this would be of no use for me. Or do you mean to
> add something like that to the initialization of the sk_buff struct, like a
> "long allocator = current_text_addr()" in skbuff.h? Is something like this
> possible? I'm not sure about it....

look, alloc_skb is not inline, so you need to use
__builtin_return_address(0) to know who called alloc_skb, you have to
change struct sk_buff to have an extra member where you will store the
return of __builtin_return_address(0), and possibly another field to store
the jiffies, so that you'll have the time when the alloc_skb was called,
then, later on, you could just printk the time it took from the alloc_skb
to the place where you want to know how long it took to traverse the
section of the network stack you're interested in.
 
> > About the example of WAITQUEUE_DEBUG:
> >
> > after being awaken you could do this:
> 
> >                 dprintk("sleeper=%p, waker=%lx\n",
> >                          current_text_addr(), wait.__waker);
> 
> > in a inline function does the trick, but this is just an example of a
> > function that uses an extra debug field in a structure that is alocated
> > somewhere and you want to know who allocated it later on.
> 
> > Yes, you'll have to decode the address from syslog, gotcha?
> But the __waker member is filled by a macro from wait.h, if I had seen it
> right. Where would you issue such a dprintk call?
 
> Sorry about my missing knowledge about wait queues, but I don't get the
> point where to fill the member. That I could print it later on, thats
> clear, but how to fill it?

creating another field in the struct sk_buff, like I said above. The whole
point of the wait_queue was to show you the idea, the fact that you could
add another field to the struct you want to watch, see this excerpt from
include/linux/wait.h:

struct __wait_queue {
        unsigned int flags;
#define WQ_FLAG_EXCLUSIVE       0x01
        struct task_struct * task;
        struct list_head task_list;
#if WAITQUEUE_DEBUG
        long __magic;
        long __waker;
#endif
};

see? you just add a 'long allocator; long jiffies_creation;' to struct
sk_buff like in the struct above, for WAITQUEUE_DEBUG debugging the kernel
has __magic and __waker, and store at alloc_skb time there the return of
__builtin_return_address(0) and jiffies, the later on you use it for your
purposes.

- Arnaldo

Re: Allocation of sk_buffs in the kernel

[Jens Hoffrichter]

> > > Maybe Jens should use something like WAITQUEUE_DEBUG if he want to
know
> > > where alloc_skb and friends were called, see include/linux/wait.h 8)
> > Do you mean I should use something LIKE the WAITQUEUE_DEBUG (eg.
> > implementing something like that in skbuff.c) or I should use
> > WAITQUEUE_DEBUG?
> no, just use the same idea that is used to debug wait_queues
OK, then I interpreted the code in wait.h right ;)

> > Are there any examples how to use the WAITQUEUE_DEBUG?

> oops, I mean the __waker thing, for debugging you could get the address
of
> the caller with current_text_addr() and store it in an extra sk_buff
field
> so that later on you could know who create the skb.
But where should I fill this field in the sk_buff? I know that alloc_skb
creates an sk_buff, so this would be of no use for me. Or do you mean to
add something like that to the initialization of the sk_buff struct, like a
"long allocator = current_text_addr()" in skbuff.h? Is something like this
possible? I'm not sure about it....

> About the example of WAITQUEUE_DEBUG:
>
> after being awaken you could do this:

>                 dprintk("sleeper=%p, waker=%lx\n",
>                          current_text_addr(), wait.__waker);

> in a inline function does the trick, but this is just an example of a
> function that uses an extra debug field in a structure that is alocated
> somewhere and you want to know who allocated it later on.

> Yes, you'll have to decode the address from syslog, gotcha?
But the __waker member is filled by a macro from wait.h, if I had seen it
right. Where would you issue such a dprintk call?

Sorry about my missing knowledge about wait queues, but I don't get the
point where to fill the member. That I could print it later on, thats
clear, but how to fill it?

CU,
Jens

Re: Allocation of sk_buffs in the kernel

[Arnaldo Carvalho de Melo]

Em Thu, Aug 23, 2001 at 11:01:42AM +0200, Jens Hoffrichter escreveu:
> Hi,
> 
> > Maybe Jens should use something like WAITQUEUE_DEBUG if he want to know
> > where alloc_skb and friends were called, see include/linux/wait.h 8)
> Do you mean I should use something LIKE the WAITQUEUE_DEBUG (eg.
> implementing something like that in skbuff.c) or I should use
> WAITQUEUE_DEBUG?

no, just use the same idea that is used to debug wait_queues
 
> The code in wait.h mainly seems to consist of issuing BUG() calls, and
> thats not quite what I want to ;) But how is it to use? I don't know much
> about waitqueues in the Linux kernel, I mainly played with the network
> stack...

heh, thats my use for the waitqueue debug now 8)
 
> Are there any examples how to use the WAITQUEUE_DEBUG?

oops, I mean the __waker thing, for debugging you could get the address of
the caller with current_text_addr() and store it in an extra sk_buff field
so that later on you could know who create the skb.

About the example of WAITQUEUE_DEBUG:

after being awaken you could do this:

                dprintk("sleeper=%p, waker=%lx\n",
                         current_text_addr(), wait.__waker);

in a inline function does the trick, but this is just an example of a
function that uses an extra debug field in a structure that is alocated
somewhere and you want to know who allocated it later on.

Yes, you'll have to decode the address from syslog, gotcha?

- Arnaldo

Re: Allocation of sk_buffs in the kernel

[Jens Hoffrichter]

> > I'm currently writing a kernel patch where it is essential to get
known>
> > when a sk_buff is allocated. Or better said I have to get known when a
> > sk_buff is effectively a new packet in the kernel-
>I don't want to guess why you need that...
I'm currently writing a stack profiler as my diploma thesis. It should give
a statistic how long a packet takes from one part of the network stack to
the other. So I have to watch the sk_buffs travelling through the kernel
and I need badly to know when they are allocated :) And, on top, the
deadline is in one week, and there are really strange things going on ;)

> > I currently identified 3 functions in the kernel where sk_buffs are
> > allocated: alloc_skb (of course), skb_linearize and pskb_expand_head.
Or at
> > least there new data is defined for the sk_buffs.
> >
> > Now I monitor a TCP session, a FTP download better said, and on the
> > interface arrives around 30000 packets for 50 MB of data. But in my
kernel
> > patch only 2000 packets are allocated, or at least I see only the
> > allocation of 2000 packets.
> >
> > Can anyone help me where I can find my missing packets? ;)) I need them
> > badly! *GG*

> There should be no skbuff allocation outside net/core/skbuff.c and all
> normal[1] networking drivers also don't use private pools. Perhaps
> you forgot to instrument a case there.
I have done some bit more testing yesterday, and the problem occurs in
either sending and receiving way, on two different ethernet drivers
(eepro100 and

Re: Allocation of sk_buffs in the kernel

[Jens Hoffrichter]

Hi,

> Maybe Jens should use something like WAITQUEUE_DEBUG if he want to know
> where alloc_skb and friends were called, see include/linux/wait.h 8)
Do you mean I should use something LIKE the WAITQUEUE_DEBUG (eg.
implementing something like that in skbuff.c) or I should use
WAITQUEUE_DEBUG?

The code in wait.h mainly seems to consist of issuing BUG() calls, and
thats not quite what I want to ;) But how is it to use? I don't know much
about waitqueues in the Linux kernel, I mainly played with the network
stack...

Are there any examples how to use the WAITQUEUE_DEBUG?

CU,
Jens

Allocation of sk_buffs in the kernel

[Jens Hoffrichter]

Hello everyone,

I'm currently writing a kernel patch where it is essential to get known
when a sk_buff is allocated. Or better said I have to get known when a
sk_buff is effectively a new packet in the kernel-

I currently identified 3 functions in the kernel where sk_buffs are
allocated: alloc_skb (of course), skb_linearize and pskb_expand_head. Or at
least there new data is defined for the sk_buffs.

Now I monitor a TCP session, a FTP download better said, and on the
interface arrives around 30000 packets for 50 MB of data. But in my kernel
patch only 2000 packets are allocated, or at least I see only the
allocation of 2000 packets.

Can anyone help me where I can find my missing packets? ;)) I need them
badly! *GG*

It's not quite easy to look into the TCP code, because its quite big and a
bit complicated.....

Thanks in advance!

Regards,
Jens