AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Konrad Djimeli]

Hello,

I am working on developing a Snabb Switch app which uses Af_XDP sockets
and so far I have not found any AF_XDP sample example online, except for
the sample in the Linux Kernel source. I have tried modifying the
example, to create two sockets (sock1 and sock2) sharing a UMEM with
custom write_sock and read_sock functions. But when I do a write with
"sock1" I am unable to read the data with "sock2". Here is the code
https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c ,
and I am new to working with AF_XDP sockets and I wish I could get some
pointers to what may be wrong with the program. 

Thanks,
Konrad
www.djimeli.me

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Marius Gerling]

Hello Konrad,

some things to think of while using AF_XDP.

 - Received data is forwarded to an AF_XDP socket with the bps_redirect_map call in the kernel part. You are sending the data to the first AF_XDP socket all the time.
 - With AF_XDP one bypasses all the TCP/IP stack of the Linux kernel. This means that one needs to set the Ethernet, IP, UDP (or TCP) headers by themselves at the beginning of the packet. (I don’t exactly know if this is needed for the loopback adapter you are using)

You might want to have a look at other AF_XDP implementations. One is in my repo: https://github.com/gerl1ng/afxdp-packet-processor (folder server_afxdp). Some more information is given in the PDF in that repo.

Marius

> On 1. Nov 2018, at 17:52, Konrad Djimeli <kdjimeli@igalia.com> wrote:
> 
> Hello,
> 
> I am working on developing a Snabb Switch app which uses Af_XDP sockets
> and so far I have not found any AF_XDP sample example online, except for
> the sample in the Linux Kernel source. I have tried modifying the
> example, to create two sockets (sock1 and sock2) sharing a UMEM with
> custom write_sock and read_sock functions. But when I do a write with
> "sock1" I am unable to read the data with "sock2". Here is the code
> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c ,
> and I am new to working with AF_XDP sockets and I wish I could get some
> pointers to what may be wrong with the program. 
> 
> Thanks,
> Konrad
> www.djimeli.me

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Björn Töpel]

Den tors 1 nov. 2018 kl 18:14 skrev Konrad Djimeli <kdjimeli@igalia.com>:
>
> Hello,
>
> I am working on developing a Snabb Switch app which uses Af_XDP sockets
> and so far I have not found any AF_XDP sample example online, except for
> the sample in the Linux Kernel source. I have tried modifying the
> example, to create two sockets (sock1 and sock2) sharing a UMEM with
> custom write_sock and read_sock functions. But when I do a write with
> "sock1" I am unable to read the data with "sock2". Here is the code
> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c ,
> and I am new to working with AF_XDP sockets and I wish I could get some
> pointers to what may be wrong with the program.
>

Hi Konrad,

Let's recap what your application does:

1. Create two AF_XDP sockets bound to ifindex and a shared UMEM.
2. The pass NUM_DESCS descriptors to the UMEM fill ring
3. In write_sock BATCH_SIZE packets are put on the Tx ring.
4. In read_sock up to BATCH_SIZE packets are picked from the Rx ring.
5. The application exits.

In terms of correctness:

In 3. you pass the same UMEM indicies to the Tx ring (ids={0,
FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill
ring. This is not correct. Userspace to track what ids are owned by
the user and by the kernel. Now, both the fill ring and Tx ring
contain the same id.

I'm not sure what the expected output of your program should be? Have
you verified that packets are being sent? Is the inferface/queue 0
receiving packets?

I think it would help if you explained what you are trying to do
first!

What XDP program are you running?


Cheers,
Björn



*

> Thanks,
> Konrad
> www.djimeli.me

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Konrad Djimeli]

On 2018-11-01 19:27, Marius Gerling wrote:
> Hello Konrad,
> 
> some things to think of while using AF_XDP.
> 
>  - Received data is forwarded to an AF_XDP socket with the
> bps_redirect_map call in the kernel part. You are sending the data to
> the first AF_XDP socket all the time.
>  - With AF_XDP one bypasses all the TCP/IP stack of the Linux kernel.
> This means that one needs to set the Ethernet, IP, UDP (or TCP)
> headers by themselves at the beginning of the packet. (I don’t exactly
> know if this is needed for the loopback adapter you are using)
> 
> You might want to have a look at other AF_XDP implementations. One is
> in my repo: https://github.com/gerl1ng/afxdp-packet-processor (folder
> server_afxdp). Some more information is given in the PDF in that repo.
> 
> Marius

Hello Marius,

Thanks for taking the time to  provide feedback and pointers. I am
currently going through your thesis and it is definitely helpful in
enabling me better understand XDP and AF_XDP.

Thanks
Konrad

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Konrad Djimeli]

On 2018-11-01 19:49, Björn Töpel wrote:
> Den tors 1 nov. 2018 kl 18:14 skrev Konrad Djimeli <kdjimeli@igalia.com>:
>>
>> Hello,
>>
>> I am working on developing a Snabb Switch app which uses Af_XDP sockets
>> and so far I have not found any AF_XDP sample example online, except for
>> the sample in the Linux Kernel source. I have tried modifying the
>> example, to create two sockets (sock1 and sock2) sharing a UMEM with
>> custom write_sock and read_sock functions. But when I do a write with
>> "sock1" I am unable to read the data with "sock2". Here is the code
>> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c ,
>> and I am new to working with AF_XDP sockets and I wish I could get some
>> pointers to what may be wrong with the program.
>>
> 
> Hi Konrad,
> 
> Let's recap what your application does:
> 
> 1. Create two AF_XDP sockets bound to ifindex and a shared UMEM.
> 2. The pass NUM_DESCS descriptors to the UMEM fill ring
> 3. In write_sock BATCH_SIZE packets are put on the Tx ring.
> 4. In read_sock up to BATCH_SIZE packets are picked from the Rx ring.
> 5. The application exits.
> 
> In terms of correctness:
> 
> In 3. you pass the same UMEM indicies to the Tx ring (ids={0,
> FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill
> ring. This is not correct. Userspace to track what ids are owned by
> the user and by the kernel. Now, both the fill ring and Tx ring
> contain the same id.

Hello Björn,

Thanks for taking time to go through the program in detail and providing
feedback.

> 
> I'm not sure what the expected output of your program should be? Have
> you verified that packets are being sent? Is the inferface/queue 0
> receiving packets?

Since I am calling the hex_dump(...) function within the write_sock and
read_sock function, I expect to see a printout of the packet that is
written and read when I call the functions.

I think the packet is actually sent because when I call the read with
the same socket that I used for the write "write_sock(sock1, data,
len);read_sock(sock1);", I get the example output below.
"
Writing = 1485946880
length = 42
addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00
01 36 15 FD 2A EE A3 C0 A8 08 64  | ...........]..........6.�*����.d
addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __
__ __ __ __ __ __ __ __ __ __ __  | ......�:�d

Reading = 1485946880
length = 42
addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00
01 36 15 FD 2A EE A3 C0 A8 08 64  | ...........]..........6.�*����.d
addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __
__ __ __ __ __ __ __ __ __ __ __  | ......�:�d
"

But when I use one socket for the write and the other socket for the
read "write_sock(sock1, data, len);read_sock(sock2);", I don't get any
output for the read call as seen in the example output below.
"
Writing = 1417486336
length = 42
addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00
01 36 15 FD 2A EE A3 C0 A8 08 64  | ...........]..........6.�*����.d
addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __
__ __ __ __ __ __ __ __ __ __ __  | ......�:�d
"

> 
> I think it would help if you explained what you are trying to do
> first!

I am just trying to test and understand how to send packets using an
AF_XDP socket and to receive the packets using an another AF_XDP socket.

> 
> What XDP program are you running?
> 

The XDP program can be found
https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c


Thanks
Konrad


> 
> Cheers,
> Björn
>  
 

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Björn Töpel]

Den fre 2 nov. 2018 kl 17:08 skrev Konrad Djimeli <kdjimeli@igalia.com>:
>
> On 2018-11-01 19:49, Björn Töpel wrote:
> > Den tors 1 nov. 2018 kl 18:14 skrev Konrad Djimeli <kdjimeli@igalia.com>:
> >>
> >> Hello,
> >>
> >> I am working on developing a Snabb Switch app which uses Af_XDP sockets
> >> and so far I have not found any AF_XDP sample example online, except for
> >> the sample in the Linux Kernel source. I have tried modifying the
> >> example, to create two sockets (sock1 and sock2) sharing a UMEM with
> >> custom write_sock and read_sock functions. But when I do a write with
> >> "sock1" I am unable to read the data with "sock2". Here is the code
> >> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c ,
> >> and I am new to working with AF_XDP sockets and I wish I could get some
> >> pointers to what may be wrong with the program.
> >>
> >
> > Hi Konrad,
> >
> > Let's recap what your application does:
> >
> > 1. Create two AF_XDP sockets bound to ifindex and a shared UMEM.
> > 2. The pass NUM_DESCS descriptors to the UMEM fill ring
> > 3. In write_sock BATCH_SIZE packets are put on the Tx ring.
> > 4. In read_sock up to BATCH_SIZE packets are picked from the Rx ring.
> > 5. The application exits.
> >
> > In terms of correctness:
> >
> > In 3. you pass the same UMEM indicies to the Tx ring (ids={0,
> > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill
> > ring. This is not correct. Userspace to track what ids are owned by
> > the user and by the kernel. Now, both the fill ring and Tx ring
> > contain the same id.
>
> Hello Björn,
>
> Thanks for taking time to go through the program in detail and providing
> feedback.
>
> >
> > I'm not sure what the expected output of your program should be? Have
> > you verified that packets are being sent? Is the inferface/queue 0
> > receiving packets?
>
> Since I am calling the hex_dump(...) function within the write_sock and
> read_sock function, I expect to see a printout of the packet that is
> written and read when I call the functions.
>
> I think the packet is actually sent because when I call the read with
> the same socket that I used for the write "write_sock(sock1, data,
> len);read_sock(sock1);", I get the example output below.
>

How is you networking set up? Everything you send out, is ping-ponged
back to the same interface? Then yes. :-)

"
> Writing = 1485946880
> length = 42
> addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00
> 01 36 15 FD 2A EE A3 C0 A8 08 64  | ...........]..........6.�*����.d
> addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __
> __ __ __ __ __ __ __ __ __ __ __  | ......�:�d
>
> Reading = 1485946880
> length = 42
> addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00
> 01 36 15 FD 2A EE A3 C0 A8 08 64  | ...........]..........6.�*����.d
> addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __
> __ __ __ __ __ __ __ __ __ __ __  | ......�:�d
> "
>
> But when I use one socket for the write and the other socket for the
> read "write_sock(sock1, data, len);read_sock(sock2);", I don't get any
> output for the read call as seen in the example output below.
> "
> Writing = 1417486336
> length = 42
> addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00
> 01 36 15 FD 2A EE A3 C0 A8 08 64  | ...........]..........6.�*����.d
> addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __
> __ __ __ __ __ __ __ __ __ __ __  | ......�:�d
> "
>

Your XDP program only redirects packets to the first AF_XDP socket,
according to:
     https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c#L33
and onward, so this would be excepted.

> >
> > I think it would help if you explained what you are trying to do
> > first!
>
> I am just trying to test and understand how to send packets using an
> AF_XDP socket and to receive the packets using an another AF_XDP socket.
>

OK!

Cheers,
Björn

> >
> > What XDP program are you running?
> >
>
> The XDP program can be found
> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c
>
>
> Thanks
> Konrad
>
>
> >
> > Cheers,
> > Björn
> >
>

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Konrad Djimeli]

Hello Björn,

On 2018-11-01 19:49, Björn Töpel wrote:
> Den tors 1 nov. 2018 kl 18:14 skrev Konrad Djimeli <kdjimeli@igalia.com>:
>>
>> Hello,
>>
>> I am working on developing a Snabb Switch app which uses Af_XDP sockets
>> and so far I have not found any AF_XDP sample example online, except for
>> the sample in the Linux Kernel source. I have tried modifying the
>> example, to create two sockets (sock1 and sock2) sharing a UMEM with
>> custom write_sock and read_sock functions. But when I do a write with
>> "sock1" I am unable to read the data with "sock2". Here is the code
>> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c ,
>> and I am new to working with AF_XDP sockets and I wish I could get some
>> pointers to what may be wrong with the program.
>>
> 
> Hi Konrad,
> 
> Let's recap what your application does:
> 
> 1. Create two AF_XDP sockets bound to ifindex and a shared UMEM.
> 2. The pass NUM_DESCS descriptors to the UMEM fill ring
> 3. In write_sock BATCH_SIZE packets are put on the Tx ring.
> 4. In read_sock up to BATCH_SIZE packets are picked from the Rx ring.
> 5. The application exits.
> 
> In terms of correctness:
> 
> In 3. you pass the same UMEM indicies to the Tx ring (ids={0,
> FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill
> ring. This is not correct. Userspace to track what ids are owned by
> the user and by the kernel. Now, both the fill ring and Tx ring
> contain the same id.
Please I have been looking into this for some days now, but I have not
been able to understand what you where referring to and how to fix it.

I also updated the xdp program bpf_redirect_map index to write to the
second socket (sock2) but I can not observe the reading working when I
use any other networking interface except for the 'lo' interface. I
would like to inquire if it may have something to do with the code.
- https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c
- https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c

Thanks
Konrad

> 
> I'm not sure what the expected output of your program should be? Have
> you verified that packets are being sent? Is the inferface/queue 0
> receiving packets?
> 
> I think it would help if you explained what you are trying to do
> first!
> 
> What XDP program are you running?
> 
> 
> Cheers,
> Björn
> 
> 
> 
> *
> 
>> Thanks,
>> Konrad
>> www.djimeli.me

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Björn Töpel]

Den tors 8 nov. 2018 kl 16:15 skrev Konrad Djimeli <kdjimeli@igalia.com>:
>
[...]
> > In terms of correctness:
> >
> > In 3. you pass the same UMEM indicies to the Tx ring (ids={0,
> > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill
> > ring. This is not correct. Userspace to track what ids are owned by
> > the user and by the kernel. Now, both the fill ring and Tx ring
> > contain the same id.
> Please I have been looking into this for some days now, but I have not
> been able to understand what you where referring to and how to fix it.
>

Ok, let's talk about the basics, and hopefully it will clear up. The
indicies/ids into the UMEM are used to point out packet buffers.

The Rx/Tx/Completion/Fill rings are used to pass ownership between the
kernel and the user application. E.g. if fill id X with data and pass
that to the kernel for transmission, the application has passed
ownership of buffer X to the kernel. The application cannot (well,
shouldn't) touch the buffer pointed out by X until the kernel is done
with the buffer -- in other words, when the buffer is passed back to
the application via the completion ring. Analogous, when you pass id Y
to the fill ring, Y is owned by the kernel. The application cannot
(again, shouldn't) touch the data pointed out by Y until ownership is
passed back to the application via the Rx ring.

What you did in your old code (I haven't looked at your updates!) is
that you passed X, Y and Z to the fill ring. And then, prior obtaining
ownership of the buffers, you passed X, Y and Z to the Tx ring.

> I also updated the xdp program bpf_redirect_map index to write to the
> second socket (sock2) but I can not observe the reading working when I
> use any other networking interface except for the 'lo' interface. I
> would like to inquire if it may have something to do with the code.
> - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c
> - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c
>

If you rely on that all sent packets out to a certain interface should
be hairpinned back to same interface, you need to set that up in your
switching plane. Or continue to use the loopback interface as you
do(?).


Björn


> Thanks
> Konrad
>
> >
> > I'm not sure what the expected output of your program should be? Have
> > you verified that packets are being sent? Is the inferface/queue 0
> > receiving packets?
> >
> > I think it would help if you explained what you are trying to do
> > first!
> >
> > What XDP program are you running?
> >
> >
> > Cheers,
> > Björn
> >
> >
> >
> > *
> >
> >> Thanks,
> >> Konrad
> >> www.djimeli.me

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[William Tu]

On Sat, Nov 10, 2018 at 6:31 AM Björn Töpel <bjorn.topel@gmail.com> wrote:
>
> Den tors 8 nov. 2018 kl 16:15 skrev Konrad Djimeli <kdjimeli@igalia.com>:
> >
> [...]
> > > In terms of correctness:
> > >
> > > In 3. you pass the same UMEM indicies to the Tx ring (ids={0,
> > > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill
> > > ring. This is not correct. Userspace to track what ids are owned by
> > > the user and by the kernel. Now, both the fill ring and Tx ring
> > > contain the same id.
> > Please I have been looking into this for some days now, but I have not
> > been able to understand what you where referring to and how to fix it.
> >
>
> Ok, let's talk about the basics, and hopefully it will clear up. The
> indicies/ids into the UMEM are used to point out packet buffers.
>
> The Rx/Tx/Completion/Fill rings are used to pass ownership between the
> kernel and the user application. E.g. if fill id X with data and pass
> that to the kernel for transmission, the application has passed
> ownership of buffer X to the kernel. The application cannot (well,
> shouldn't) touch the buffer pointed out by X until the kernel is done
> with the buffer -- in other words, when the buffer is passed back to
> the application via the completion ring. Analogous, when you pass id Y
> to the fill ring, Y is owned by the kernel. The application cannot
> (again, shouldn't) touch the data pointed out by Y until ownership is
> passed back to the application via the Rx ring.
>

Hi Konrad,

I made a slide about using how the rx/tx/completion/fill ring works for OVS.
Hope this is helpful for you.
https://drive.google.com/open?id=1gM3lndS6WeS16QNk74L22UNx5D5wlS2o

Another thing for debugging is first to add a print in the XDP code,
so making sure the packet is there at the XDP layer.
https://github.com/williamtu/ovs-ebpf/blob/afxdp-array/bpf/xdp.h#L71

Then I usually prints the xsk stats
https://github.com/williamtu/ovs-ebpf/blob/afxdp-array/lib/netdev-linux.c#L656
So making sure I'm not mess up with these rings.

Regards,
William

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Konrad Djimeli]

On 2018-11-10 15:58, William Tu wrote:
> On Sat, Nov 10, 2018 at 6:31 AM Björn Töpel <bjorn.topel@gmail.com> wrote:
>>
>> Den tors 8 nov. 2018 kl 16:15 skrev Konrad Djimeli <kdjimeli@igalia.com>:
>> >
>> [...]
>> > > In terms of correctness:
>> > >
>> > > In 3. you pass the same UMEM indicies to the Tx ring (ids={0,
>> > > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill
>> > > ring. This is not correct. Userspace to track what ids are owned by
>> > > the user and by the kernel. Now, both the fill ring and Tx ring
>> > > contain the same id.
>> > Please I have been looking into this for some days now, but I have not
>> > been able to understand what you where referring to and how to fix it.
>> >
>>
>> Ok, let's talk about the basics, and hopefully it will clear up. The
>> indicies/ids into the UMEM are used to point out packet buffers.
>>
>> The Rx/Tx/Completion/Fill rings are used to pass ownership between the
>> kernel and the user application. E.g. if fill id X with data and pass
>> that to the kernel for transmission, the application has passed
>> ownership of buffer X to the kernel. The application cannot (well,
>> shouldn't) touch the buffer pointed out by X until the kernel is done
>> with the buffer -- in other words, when the buffer is passed back to
>> the application via the completion ring. Analogous, when you pass id Y
>> to the fill ring, Y is owned by the kernel. The application cannot
>> (again, shouldn't) touch the data pointed out by Y until ownership is
>> passed back to the application via the Rx ring.
>>
> 
> Hi Konrad,
> 
> I made a slide about using how the rx/tx/completion/fill ring works for OVS.
> Hope this is helpful for you.
> https://drive.google.com/open?id=1gM3lndS6WeS16QNk74L22UNx5D5wlS2o
> 
> Another thing for debugging is first to add a print in the XDP code,
> so making sure the packet is there at the XDP layer.
> https://github.com/williamtu/ovs-ebpf/blob/afxdp-array/bpf/xdp.h#L71
> 
> Then I usually prints the xsk stats
> https://github.com/williamtu/ovs-ebpf/blob/afxdp-array/lib/netdev-linux.c#L656
> So making sure I'm not mess up with these rings.
> 
> Regards,
> William

Hello William,

Thanks a lot. I have gone through the resources you provided, and they
have been 
very helpful, especially the document on how the rx/tx/completion/fill
ring works.

Thanks,
Konrad

Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1)

[Konrad Djimeli]

Hello Björn,

On 2018-11-10 15:31, Björn Töpel wrote:
> Den tors 8 nov. 2018 kl 16:15 skrev Konrad Djimeli <kdjimeli@igalia.com>:
>>
> [...]
>> > In terms of correctness:
>> >
>> > In 3. you pass the same UMEM indicies to the Tx ring (ids={0,
>> > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill
>> > ring. This is not correct. Userspace to track what ids are owned by
>> > the user and by the kernel. Now, both the fill ring and Tx ring
>> > contain the same id.
>> Please I have been looking into this for some days now, but I have not
>> been able to understand what you where referring to and how to fix it.
>>
> 
> Ok, let's talk about the basics, and hopefully it will clear up. The
> indicies/ids into the UMEM are used to point out packet buffers.
> 
> The Rx/Tx/Completion/Fill rings are used to pass ownership between the
> kernel and the user application. E.g. if fill id X with data and pass
> that to the kernel for transmission, the application has passed
> ownership of buffer X to the kernel. The application cannot (well,
> shouldn't) touch the buffer pointed out by X until the kernel is done
> with the buffer -- in other words, when the buffer is passed back to
> the application via the completion ring. Analogous, when you pass id Y
> to the fill ring, Y is owned by the kernel. The application cannot
> (again, shouldn't) touch the data pointed out by Y until ownership is
> passed back to the application via the Rx ring.
> 
> What you did in your old code (I haven't looked at your updates!) is
> that you passed X, Y and Z to the fill ring. And then, prior obtaining
> ownership of the buffers, you passed X, Y and Z to the Tx ring.
> 

Sorry for the delayed response, I took some time off from the work I was
doing. Thanks a lot for the detailed explanation and I think I have been

able to identify the issue you were pointing out.

What I have done to address the problem is, since the location passed to

the fill ring are  {0, 2*FRAME_SIZE, 3*FRAME_SIZE, … , 
(NUM_DESCS-1)*FRAME_SIZE}, I updated the ids passed to the tx ring to 
{NUM_DESCS*FRAME_SIZE,  (NUM_DESCS+1)*FRAME_SIZE,
(NUM_DESCS+2)*FRAME_SIZE, … , 
(NUM_FRAMES-1)*FRAME_SIZE} [0].

The link below points to the commit addressing the issue. I hope I have
not
still gotten things mixedup. 

[0]
https://github.com/djkonro/afxdp/commit/c199b496ac6949771fc02260165f4822ccd2f51a#diff-e1b95b7ab6213d7c19f2eacc1290f955

Thanks,
Konrad

>> I also updated the xdp program bpf_redirect_map index to write to the
>> second socket (sock2) but I can not observe the reading working when I
>> use any other networking interface except for the 'lo' interface. I
>> would like to inquire if it may have something to do with the code.
>> - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c
>> - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c
>>
> 
> If you rely on that all sent packets out to a certain interface should
> be hairpinned back to same interface, you need to set that up in your
> switching plane. Or continue to use the loopback interface as you
> do(?).
> 
> 
> Björn
> 
> 
>> Thanks
>> Konrad
>>
>> >
>> > I'm not sure what the expected output of your program should be? Have
>> > you verified that packets are being sent? Is the inferface/queue 0
>> > receiving packets?
>> >
>> > I think it would help if you explained what you are trying to do
>> > first!
>> >
>> > What XDP program are you running?
>> >
>> >
>> > Cheers,
>> > Björn
>> >
>> >
>> >
>> > *
>> >
>> >> Thanks,
>> >> Konrad
>> >> www.djimeli.me