xdpsock poll syscall CPU 100%

xdpsock poll syscall CPU 100%

[William Tu]

Hi,

I'm trying to save some CPU cycles when there is no packet arrives.
I enable the poll syscall option of xdpsock, by doing

$ ./xdpsock -r -p -S -i ens16
 sock0@ens160:0 rxdrop xdp-skb poll()
                pps         pkts        1.00
rx              0           0
tx              0           0

Since there is no packet coming, I though by calling poll()
system call, the xdpsock process will be blocked and CPU utilization
should be way under 100%. However, I'm still seeing 100%
CPU utilization. Am I understanding this correctly?

Thanks,
William

Re: xdpsock poll syscall CPU 100%

[Magnus Karlsson]

On Thu, Feb 20, 2020 at 11:50 PM William Tu <u9012063@gmail.com> wrote:
>
> Hi,
>
> I'm trying to save some CPU cycles when there is no packet arrives.
> I enable the poll syscall option of xdpsock, by doing
>
> $ ./xdpsock -r -p -S -i ens16
>  sock0@ens160:0 rxdrop xdp-skb poll()
>                 pps         pkts        1.00
> rx              0           0
> tx              0           0
>
> Since there is no packet coming, I though by calling poll()
> system call, the xdpsock process will be blocked and CPU utilization
> should be way under 100%. However, I'm still seeing 100%
> CPU utilization. Am I understanding this correctly?

Yes, something seems to be wrong here. Calling poll() should put you
in sleep for the timeout period, here 1000 ms. Let me take a look at
it. What version of the kernel are you running?

/Magnus

> Thanks,
> William

Re: xdpsock poll syscall CPU 100%

[Eelco Chaudron]

On 20 Feb 2020, at 23:49, William Tu wrote:

> Hi,
>
> I'm trying to save some CPU cycles when there is no packet arrives.
> I enable the poll syscall option of xdpsock, by doing
>
> $ ./xdpsock -r -p -S -i ens16
>  sock0@ens160:0 rxdrop xdp-skb poll()
>                 pps         pkts        1.00
> rx              0           0
> tx              0           0
>
> Since there is no packet coming, I though by calling poll()
> system call, the xdpsock process will be blocked and CPU utilization
> should be way under 100%. However, I'm still seeing 100%
> CPU utilization. Am I understanding this correctly?

Hi William, I can remember I saw this in the past two with this code. It 
had something to do with the way xdpsock waits for the buffers to be 
free’ ed by the kernel. What I can remember it had something to do 
with the veth interfaces also.

I do remember that I fixed it in the tutorial for AF_XDP: 
https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c

Maybe you can see if you have the same problem with this example.

//Eelco


> Thanks,
> William

Re: xdpsock poll syscall CPU 100%

[Magnus Karlsson]

On Fri, Feb 21, 2020 at 9:30 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 20 Feb 2020, at 23:49, William Tu wrote:
>
> > Hi,
> >
> > I'm trying to save some CPU cycles when there is no packet arrives.
> > I enable the poll syscall option of xdpsock, by doing
> >
> > $ ./xdpsock -r -p -S -i ens16
> >  sock0@ens160:0 rxdrop xdp-skb poll()
> >                 pps         pkts        1.00
> > rx              0           0
> > tx              0           0
> >
> > Since there is no packet coming, I though by calling poll()
> > system call, the xdpsock process will be blocked and CPU utilization
> > should be way under 100%. However, I'm still seeing 100%
> > CPU utilization. Am I understanding this correctly?
>
> Hi William, I can remember I saw this in the past two with this code. It
> had something to do with the way xdpsock waits for the buffers to be
> free’ ed by the kernel. What I can remember it had something to do
> with the veth interfaces also.
>
> I do remember that I fixed it in the tutorial for AF_XDP:
> https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c

Eelco,

Do you remember exactly what you had to fix in the xdpsock sample?
Your tutorial is quite a rewrite so it is hard for me to tell exactly
which of all the changes that fix this problem. The reason I ask is
that it would be nice to fix this in the sample too.

Thanks: Magnus

> Maybe you can see if you have the same problem with this example.
>
> //Eelco
>
>
> > Thanks,
> > William
>

Re: xdpsock poll syscall CPU 100%

[Eelco Chaudron]

On 21 Feb 2020, at 9:33, Magnus Karlsson wrote:

> On Fri, Feb 21, 2020 at 9:30 AM Eelco Chaudron <echaudro@redhat.com> 
> wrote:
>>
>>
>>
>> On 20 Feb 2020, at 23:49, William Tu wrote:
>>
>>> Hi,
>>>
>>> I'm trying to save some CPU cycles when there is no packet arrives.
>>> I enable the poll syscall option of xdpsock, by doing
>>>
>>> $ ./xdpsock -r -p -S -i ens16
>>>  sock0@ens160:0 rxdrop xdp-skb poll()
>>>                 pps         pkts        1.00
>>> rx              0           0
>>> tx              0           0
>>>
>>> Since there is no packet coming, I though by calling poll()
>>> system call, the xdpsock process will be blocked and CPU utilization
>>> should be way under 100%. However, I'm still seeing 100%
>>> CPU utilization. Am I understanding this correctly?
>>
>> Hi William, I can remember I saw this in the past two with this code. 
>> It
>> had something to do with the way xdpsock waits for the buffers to be
>> free’ ed by the kernel. What I can remember it had something to do
>> with the veth interfaces also.
>>
>> I do remember that I fixed it in the tutorial for AF_XDP:
>> https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c
>
> Eelco,
>
> Do you remember exactly what you had to fix in the xdpsock sample?
> Your tutorial is quite a rewrite so it is hard for me to tell exactly
> which of all the changes that fix this problem. The reason I ask is
> that it would be nice to fix this in the sample too.
>
> Thanks: Magnus

 From an earlier email conversation we had this is where it looped in my 
case:

>>>>> One other thing I noticed, which I need to research is that if I
>>>>> use
>>>>> rx_drop() function from /xdpsock_user.c it loops a lot in:
>>>>>
>>>>>   	while (ret != rcvd) {
>>>>>    		if (ret < 0) {
>>>>> 	  			exit(-1);
>>>>> 		}
>>>>>    		ret = xsk_ring_prod__reserve(&xsk->umem-
>>> fq, rcvd, &idx_fq);
>>>>>
>>>>>    	}
>>>>>
>>>>> As ret return 0, until (it looks like) I send more packets. So 
>>>>> even
>>>>> in the poll() mode, it uses 100% cpu after sending a single 
>>>>> packet.
>>>>> Note this is with the default Fedora Kernel, as I’m working on 
>>>>> this
>>>>> from my laptop. Does this sound familiar? If not I’ll dig into 
>>>>> it
>>>>> once I’m back.
>>>>
>>>> The xdpsock test is a busypolling test, to compare against DPDK
>>>> speeds.  For real use-cases, I think people will want to trade-off
>>>> latency vs. burning CPU.
>>>
>>> I understand the use case, but even with the xdpsock test program, 
>>> if
>>> I send a single packet it’s not received, or at least not when 
>>> it's
>>> sent. It takes 16 (or a multiple of it) before the get
>>> detected/processed. I think it’s because of the
>>> xsk_ring_prod__reserve(), but I’ll try to debug it more today and 
>>> to
>>> understand the APIs better.

Re: xdpsock poll syscall CPU 100%

[Magnus Karlsson]

On Fri, Feb 21, 2020 at 9:40 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 21 Feb 2020, at 9:33, Magnus Karlsson wrote:
>
> > On Fri, Feb 21, 2020 at 9:30 AM Eelco Chaudron <echaudro@redhat.com>
> > wrote:
> >>
> >>
> >>
> >> On 20 Feb 2020, at 23:49, William Tu wrote:
> >>
> >>> Hi,
> >>>
> >>> I'm trying to save some CPU cycles when there is no packet arrives.
> >>> I enable the poll syscall option of xdpsock, by doing
> >>>
> >>> $ ./xdpsock -r -p -S -i ens16
> >>>  sock0@ens160:0 rxdrop xdp-skb poll()
> >>>                 pps         pkts        1.00
> >>> rx              0           0
> >>> tx              0           0
> >>>
> >>> Since there is no packet coming, I though by calling poll()
> >>> system call, the xdpsock process will be blocked and CPU utilization
> >>> should be way under 100%. However, I'm still seeing 100%
> >>> CPU utilization. Am I understanding this correctly?
> >>
> >> Hi William, I can remember I saw this in the past two with this code.
> >> It
> >> had something to do with the way xdpsock waits for the buffers to be
> >> free’ ed by the kernel. What I can remember it had something to do
> >> with the veth interfaces also.
> >>
> >> I do remember that I fixed it in the tutorial for AF_XDP:
> >> https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c
> >
> > Eelco,
> >
> > Do you remember exactly what you had to fix in the xdpsock sample?
> > Your tutorial is quite a rewrite so it is hard for me to tell exactly
> > which of all the changes that fix this problem. The reason I ask is
> > that it would be nice to fix this in the sample too.
> >
> > Thanks: Magnus
>
>  From an earlier email conversation we had this is where it looped in my
> case:

Thanks Eelco. Yes, the xdpsock sample is too simplistic in this case.
I will put this on my backlog to fix so that we do not have this
problem in the future. I might take some inspiration from your code
:-). Hope you do not mind.

/Magnus

> >>>>> One other thing I noticed, which I need to research is that if I
> >>>>> use
> >>>>> rx_drop() function from /xdpsock_user.c it loops a lot in:
> >>>>>
> >>>>>           while (ret != rcvd) {
> >>>>>                   if (ret < 0) {
> >>>>>                           exit(-1);
> >>>>>           }
> >>>>>                   ret = xsk_ring_prod__reserve(&xsk->umem-
> >>> fq, rcvd, &idx_fq);
> >>>>>
> >>>>>           }
> >>>>>
> >>>>> As ret return 0, until (it looks like) I send more packets. So
> >>>>> even
> >>>>> in the poll() mode, it uses 100% cpu after sending a single
> >>>>> packet.
> >>>>> Note this is with the default Fedora Kernel, as I’m working on
> >>>>> this
> >>>>> from my laptop. Does this sound familiar? If not I’ll dig into
> >>>>> it
> >>>>> once I’m back.
> >>>>
> >>>> The xdpsock test is a busypolling test, to compare against DPDK
> >>>> speeds.  For real use-cases, I think people will want to trade-off
> >>>> latency vs. burning CPU.
> >>>
> >>> I understand the use case, but even with the xdpsock test program,
> >>> if
> >>> I send a single packet it’s not received, or at least not when
> >>> it's
> >>> sent. It takes 16 (or a multiple of it) before the get
> >>> detected/processed. I think it’s because of the
> >>> xsk_ring_prod__reserve(), but I’ll try to debug it more today and
> >>> to
> >>> understand the APIs better.
>

Re: xdpsock poll syscall CPU 100%

[Eelco Chaudron]

On 21 Feb 2020, at 9:46, Magnus Karlsson wrote:

> On Fri, Feb 21, 2020 at 9:40 AM Eelco Chaudron <echaudro@redhat.com> 
> wrote:
>>
>>
>>
>> On 21 Feb 2020, at 9:33, Magnus Karlsson wrote:
>>
>>> On Fri, Feb 21, 2020 at 9:30 AM Eelco Chaudron <echaudro@redhat.com>
>>> wrote:
>>>>
>>>>
>>>>
>>>> On 20 Feb 2020, at 23:49, William Tu wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to save some CPU cycles when there is no packet 
>>>>> arrives.
>>>>> I enable the poll syscall option of xdpsock, by doing
>>>>>
>>>>> $ ./xdpsock -r -p -S -i ens16
>>>>>  sock0@ens160:0 rxdrop xdp-skb poll()
>>>>>                 pps         pkts        1.00
>>>>> rx              0           0
>>>>> tx              0           0
>>>>>
>>>>> Since there is no packet coming, I though by calling poll()
>>>>> system call, the xdpsock process will be blocked and CPU 
>>>>> utilization
>>>>> should be way under 100%. However, I'm still seeing 100%
>>>>> CPU utilization. Am I understanding this correctly?
>>>>
>>>> Hi William, I can remember I saw this in the past two with this 
>>>> code.
>>>> It
>>>> had something to do with the way xdpsock waits for the buffers to 
>>>> be
>>>> free’ ed by the kernel. What I can remember it had something to 
>>>> do
>>>> with the veth interfaces also.
>>>>
>>>> I do remember that I fixed it in the tutorial for AF_XDP:
>>>> https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c
>>>
>>> Eelco,
>>>
>>> Do you remember exactly what you had to fix in the xdpsock sample?
>>> Your tutorial is quite a rewrite so it is hard for me to tell 
>>> exactly
>>> which of all the changes that fix this problem. The reason I ask is
>>> that it would be nice to fix this in the sample too.
>>>
>>> Thanks: Magnus
>>
>>  From an earlier email conversation we had this is where it looped in 
>> my
>> case:
>
> Thanks Eelco. Yes, the xdpsock sample is too simplistic in this case.
> I will put this on my backlog to fix so that we do not have this
> problem in the future. I might take some inspiration from your code
> :-). Hope you do not mind.

Its GPL so take whatever you need ;)

Re: xdpsock poll syscall CPU 100%

[William Tu]

On Fri, Feb 21, 2020 at 12:46 AM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
>
> On Fri, Feb 21, 2020 at 9:40 AM Eelco Chaudron <echaudro@redhat.com> wrote:
> >
> >
> >
> > On 21 Feb 2020, at 9:33, Magnus Karlsson wrote:
> >
> > > On Fri, Feb 21, 2020 at 9:30 AM Eelco Chaudron <echaudro@redhat.com>
> > > wrote:
> > >>
> > >>
> > >>
> > >> On 20 Feb 2020, at 23:49, William Tu wrote:
> > >>
> > >>> Hi,
> > >>>
> > >>> I'm trying to save some CPU cycles when there is no packet arrives.
> > >>> I enable the poll syscall option of xdpsock, by doing
> > >>>
> > >>> $ ./xdpsock -r -p -S -i ens16
> > >>>  sock0@ens160:0 rxdrop xdp-skb poll()
> > >>>                 pps         pkts        1.00
> > >>> rx              0           0
> > >>> tx              0           0
> > >>>
> > >>> Since there is no packet coming, I though by calling poll()
> > >>> system call, the xdpsock process will be blocked and CPU utilization
> > >>> should be way under 100%. However, I'm still seeing 100%
> > >>> CPU utilization. Am I understanding this correctly?
> > >>
> > >> Hi William, I can remember I saw this in the past two with this code.
> > >> It
> > >> had something to do with the way xdpsock waits for the buffers to be
> > >> free’ ed by the kernel. What I can remember it had something to do
> > >> with the veth interfaces also.
> > >>
> > >> I do remember that I fixed it in the tutorial for AF_XDP:
> > >> https://github.com/xdp-project/xdp-tutorial/blob/master/advanced03-AF_XDP/af_xdp_user.c
> > >
> > > Eelco,
> > >
> > > Do you remember exactly what you had to fix in the xdpsock sample?
> > > Your tutorial is quite a rewrite so it is hard for me to tell exactly
> > > which of all the changes that fix this problem. The reason I ask is
> > > that it would be nice to fix this in the sample too.
> > >
> > > Thanks: Magnus
> >
> >  From an earlier email conversation we had this is where it looped in my
> > case:
>
> Thanks Eelco. Yes, the xdpsock sample is too simplistic in this case.
> I will put this on my backlog to fix so that we do not have this
> problem in the future. I might take some inspiration from your code
> :-). Hope you do not mind.
>
> /Magnus
>
Hi Magnus and Eelco,

Thanks for your reply.
I run Eelco's af_xdp_user.c and indeed, with poll syscall, the CPU is
way under 100%. But I'm still figuring out the difference and where to
fix in xdpsock_user.c.

Regards,
William