* AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) @ 2018-11-01 16:52 Konrad Djimeli 2018-11-01 18:27 ` Marius Gerling 2018-11-01 18:49 ` Björn Töpel 0 siblings, 2 replies; 11+ messages in thread From: Konrad Djimeli @ 2018-11-01 16:52 UTC (permalink / raw) To: xdp-newbies Hello, I am working on developing a Snabb Switch app which uses Af_XDP sockets and so far I have not found any AF_XDP sample example online, except for the sample in the Linux Kernel source. I have tried modifying the example, to create two sockets (sock1 and sock2) sharing a UMEM with custom write_sock and read_sock functions. But when I do a write with "sock1" I am unable to read the data with "sock2". Here is the code https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c , and I am new to working with AF_XDP sockets and I wish I could get some pointers to what may be wrong with the program. Thanks, Konrad www.djimeli.me ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-01 16:52 AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) Konrad Djimeli @ 2018-11-01 18:27 ` Marius Gerling 2018-11-02 15:30 ` Konrad Djimeli 2018-11-01 18:49 ` Björn Töpel 1 sibling, 1 reply; 11+ messages in thread From: Marius Gerling @ 2018-11-01 18:27 UTC (permalink / raw) To: Konrad Djimeli; +Cc: xdp-newbies Hello Konrad, some things to think of while using AF_XDP. - Received data is forwarded to an AF_XDP socket with the bps_redirect_map call in the kernel part. You are sending the data to the first AF_XDP socket all the time. - With AF_XDP one bypasses all the TCP/IP stack of the Linux kernel. This means that one needs to set the Ethernet, IP, UDP (or TCP) headers by themselves at the beginning of the packet. (I don’t exactly know if this is needed for the loopback adapter you are using) You might want to have a look at other AF_XDP implementations. One is in my repo: https://github.com/gerl1ng/afxdp-packet-processor (folder server_afxdp). Some more information is given in the PDF in that repo. Marius > On 1. Nov 2018, at 17:52, Konrad Djimeli <kdjimeli@igalia.com> wrote: > > Hello, > > I am working on developing a Snabb Switch app which uses Af_XDP sockets > and so far I have not found any AF_XDP sample example online, except for > the sample in the Linux Kernel source. I have tried modifying the > example, to create two sockets (sock1 and sock2) sharing a UMEM with > custom write_sock and read_sock functions. But when I do a write with > "sock1" I am unable to read the data with "sock2". Here is the code > https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c , > and I am new to working with AF_XDP sockets and I wish I could get some > pointers to what may be wrong with the program. > > Thanks, > Konrad > www.djimeli.me ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-01 18:27 ` Marius Gerling @ 2018-11-02 15:30 ` Konrad Djimeli 0 siblings, 0 replies; 11+ messages in thread From: Konrad Djimeli @ 2018-11-02 15:30 UTC (permalink / raw) To: Marius Gerling; +Cc: xdp-newbies On 2018-11-01 19:27, Marius Gerling wrote: > Hello Konrad, > > some things to think of while using AF_XDP. > > - Received data is forwarded to an AF_XDP socket with the > bps_redirect_map call in the kernel part. You are sending the data to > the first AF_XDP socket all the time. > - With AF_XDP one bypasses all the TCP/IP stack of the Linux kernel. > This means that one needs to set the Ethernet, IP, UDP (or TCP) > headers by themselves at the beginning of the packet. (I don’t exactly > know if this is needed for the loopback adapter you are using) > > You might want to have a look at other AF_XDP implementations. One is > in my repo: https://github.com/gerl1ng/afxdp-packet-processor (folder > server_afxdp). Some more information is given in the PDF in that repo. > > Marius Hello Marius, Thanks for taking the time to provide feedback and pointers. I am currently going through your thesis and it is definitely helpful in enabling me better understand XDP and AF_XDP. Thanks Konrad ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-01 16:52 AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) Konrad Djimeli 2018-11-01 18:27 ` Marius Gerling @ 2018-11-01 18:49 ` Björn Töpel 2018-11-02 16:08 ` Konrad Djimeli 2018-11-08 15:15 ` Konrad Djimeli 1 sibling, 2 replies; 11+ messages in thread From: Björn Töpel @ 2018-11-01 18:49 UTC (permalink / raw) To: kdjimeli; +Cc: xdp-newbies Den tors 1 nov. 2018 kl 18:14 skrev Konrad Djimeli <kdjimeli@igalia.com>: > > Hello, > > I am working on developing a Snabb Switch app which uses Af_XDP sockets > and so far I have not found any AF_XDP sample example online, except for > the sample in the Linux Kernel source. I have tried modifying the > example, to create two sockets (sock1 and sock2) sharing a UMEM with > custom write_sock and read_sock functions. But when I do a write with > "sock1" I am unable to read the data with "sock2". Here is the code > https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c , > and I am new to working with AF_XDP sockets and I wish I could get some > pointers to what may be wrong with the program. > Hi Konrad, Let's recap what your application does: 1. Create two AF_XDP sockets bound to ifindex and a shared UMEM. 2. The pass NUM_DESCS descriptors to the UMEM fill ring 3. In write_sock BATCH_SIZE packets are put on the Tx ring. 4. In read_sock up to BATCH_SIZE packets are picked from the Rx ring. 5. The application exits. In terms of correctness: In 3. you pass the same UMEM indicies to the Tx ring (ids={0, FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill ring. This is not correct. Userspace to track what ids are owned by the user and by the kernel. Now, both the fill ring and Tx ring contain the same id. I'm not sure what the expected output of your program should be? Have you verified that packets are being sent? Is the inferface/queue 0 receiving packets? I think it would help if you explained what you are trying to do first! What XDP program are you running? Cheers, Björn * > Thanks, > Konrad > www.djimeli.me ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-01 18:49 ` Björn Töpel @ 2018-11-02 16:08 ` Konrad Djimeli 2018-11-03 8:38 ` Björn Töpel 2018-11-08 15:15 ` Konrad Djimeli 1 sibling, 1 reply; 11+ messages in thread From: Konrad Djimeli @ 2018-11-02 16:08 UTC (permalink / raw) To: Björn Töpel; +Cc: xdp-newbies, xdp-newbies-owner On 2018-11-01 19:49, Björn Töpel wrote: > Den tors 1 nov. 2018 kl 18:14 skrev Konrad Djimeli <kdjimeli@igalia.com>: >> >> Hello, >> >> I am working on developing a Snabb Switch app which uses Af_XDP sockets >> and so far I have not found any AF_XDP sample example online, except for >> the sample in the Linux Kernel source. I have tried modifying the >> example, to create two sockets (sock1 and sock2) sharing a UMEM with >> custom write_sock and read_sock functions. But when I do a write with >> "sock1" I am unable to read the data with "sock2". Here is the code >> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c , >> and I am new to working with AF_XDP sockets and I wish I could get some >> pointers to what may be wrong with the program. >> > > Hi Konrad, > > Let's recap what your application does: > > 1. Create two AF_XDP sockets bound to ifindex and a shared UMEM. > 2. The pass NUM_DESCS descriptors to the UMEM fill ring > 3. In write_sock BATCH_SIZE packets are put on the Tx ring. > 4. In read_sock up to BATCH_SIZE packets are picked from the Rx ring. > 5. The application exits. > > In terms of correctness: > > In 3. you pass the same UMEM indicies to the Tx ring (ids={0, > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill > ring. This is not correct. Userspace to track what ids are owned by > the user and by the kernel. Now, both the fill ring and Tx ring > contain the same id. Hello Björn, Thanks for taking time to go through the program in detail and providing feedback. > > I'm not sure what the expected output of your program should be? Have > you verified that packets are being sent? Is the inferface/queue 0 > receiving packets? Since I am calling the hex_dump(...) function within the write_sock and read_sock function, I expect to see a printout of the packet that is written and read when I call the functions. I think the packet is actually sent because when I call the read with the same socket that I used for the write "write_sock(sock1, data, len);read_sock(sock1);", I get the example output below. " Writing = 1485946880 length = 42 addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00 01 36 15 FD 2A EE A3 C0 A8 08 64 | ...........]..........6.�*����.d addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ | ......�:�d Reading = 1485946880 length = 42 addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00 01 36 15 FD 2A EE A3 C0 A8 08 64 | ...........]..........6.�*����.d addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ | ......�:�d " But when I use one socket for the write and the other socket for the read "write_sock(sock1, data, len);read_sock(sock2);", I don't get any output for the read call as seen in the example output below. " Writing = 1417486336 length = 42 addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00 01 36 15 FD 2A EE A3 C0 A8 08 64 | ...........]..........6.�*����.d addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ | ......�:�d " > > I think it would help if you explained what you are trying to do > first! I am just trying to test and understand how to send packets using an AF_XDP socket and to receive the packets using an another AF_XDP socket. > > What XDP program are you running? > The XDP program can be found https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c Thanks Konrad > > Cheers, > Björn > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-02 16:08 ` Konrad Djimeli @ 2018-11-03 8:38 ` Björn Töpel 0 siblings, 0 replies; 11+ messages in thread From: Björn Töpel @ 2018-11-03 8:38 UTC (permalink / raw) To: kdjimeli; +Cc: xdp-newbies, xdp-newbies-owner Den fre 2 nov. 2018 kl 17:08 skrev Konrad Djimeli <kdjimeli@igalia.com>: > > On 2018-11-01 19:49, Björn Töpel wrote: > > Den tors 1 nov. 2018 kl 18:14 skrev Konrad Djimeli <kdjimeli@igalia.com>: > >> > >> Hello, > >> > >> I am working on developing a Snabb Switch app which uses Af_XDP sockets > >> and so far I have not found any AF_XDP sample example online, except for > >> the sample in the Linux Kernel source. I have tried modifying the > >> example, to create two sockets (sock1 and sock2) sharing a UMEM with > >> custom write_sock and read_sock functions. But when I do a write with > >> "sock1" I am unable to read the data with "sock2". Here is the code > >> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c , > >> and I am new to working with AF_XDP sockets and I wish I could get some > >> pointers to what may be wrong with the program. > >> > > > > Hi Konrad, > > > > Let's recap what your application does: > > > > 1. Create two AF_XDP sockets bound to ifindex and a shared UMEM. > > 2. The pass NUM_DESCS descriptors to the UMEM fill ring > > 3. In write_sock BATCH_SIZE packets are put on the Tx ring. > > 4. In read_sock up to BATCH_SIZE packets are picked from the Rx ring. > > 5. The application exits. > > > > In terms of correctness: > > > > In 3. you pass the same UMEM indicies to the Tx ring (ids={0, > > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill > > ring. This is not correct. Userspace to track what ids are owned by > > the user and by the kernel. Now, both the fill ring and Tx ring > > contain the same id. > > Hello Björn, > > Thanks for taking time to go through the program in detail and providing > feedback. > > > > > I'm not sure what the expected output of your program should be? Have > > you verified that packets are being sent? Is the inferface/queue 0 > > receiving packets? > > Since I am calling the hex_dump(...) function within the write_sock and > read_sock function, I expect to see a printout of the packet that is > written and read when I call the functions. > > I think the packet is actually sent because when I call the read with > the same socket that I used for the write "write_sock(sock1, data, > len);read_sock(sock1);", I get the example output below. > How is you networking set up? Everything you send out, is ping-ponged back to the same interface? Then yes. :-) " > Writing = 1485946880 > length = 42 > addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00 > 01 36 15 FD 2A EE A3 C0 A8 08 64 | ...........]..........6.�*����.d > addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __ > __ __ __ __ __ __ __ __ __ __ __ | ......�:�d > > Reading = 1485946880 > length = 42 > addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00 > 01 36 15 FD 2A EE A3 C0 A8 08 64 | ...........]..........6.�*����.d > addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __ > __ __ __ __ __ __ __ __ __ __ __ | ......�:�d > " > > But when I use one socket for the write and the other socket for the > read "write_sock(sock1, data, len);read_sock(sock2);", I don't get any > output for the read call as seen in the example output below. > " > Writing = 1417486336 > length = 42 > addr=0 | 01 01 01 01 01 01 04 01 03 02 12 5D 08 06 00 01 08 00 06 04 00 > 01 36 15 FD 2A EE A3 C0 A8 08 64 | ...........]..........6.�*����.d > addr=0 | 00 00 00 00 00 00 D8 3A D4 64 __ __ __ __ __ __ __ __ __ __ __ > __ __ __ __ __ __ __ __ __ __ __ | ......�:�d > " > Your XDP program only redirects packets to the first AF_XDP socket, according to: https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c#L33 and onward, so this would be excepted. > > > > I think it would help if you explained what you are trying to do > > first! > > I am just trying to test and understand how to send packets using an > AF_XDP socket and to receive the packets using an another AF_XDP socket. > OK! Cheers, Björn > > > > What XDP program are you running? > > > > The XDP program can be found > https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c > > > Thanks > Konrad > > > > > > Cheers, > > Björn > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-01 18:49 ` Björn Töpel 2018-11-02 16:08 ` Konrad Djimeli @ 2018-11-08 15:15 ` Konrad Djimeli 2018-11-10 14:31 ` Björn Töpel 1 sibling, 1 reply; 11+ messages in thread From: Konrad Djimeli @ 2018-11-08 15:15 UTC (permalink / raw) To: Björn Töpel; +Cc: xdp-newbies Hello Björn, On 2018-11-01 19:49, Björn Töpel wrote: > Den tors 1 nov. 2018 kl 18:14 skrev Konrad Djimeli <kdjimeli@igalia.com>: >> >> Hello, >> >> I am working on developing a Snabb Switch app which uses Af_XDP sockets >> and so far I have not found any AF_XDP sample example online, except for >> the sample in the Linux Kernel source. I have tried modifying the >> example, to create two sockets (sock1 and sock2) sharing a UMEM with >> custom write_sock and read_sock functions. But when I do a write with >> "sock1" I am unable to read the data with "sock2". Here is the code >> https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c , >> and I am new to working with AF_XDP sockets and I wish I could get some >> pointers to what may be wrong with the program. >> > > Hi Konrad, > > Let's recap what your application does: > > 1. Create two AF_XDP sockets bound to ifindex and a shared UMEM. > 2. The pass NUM_DESCS descriptors to the UMEM fill ring > 3. In write_sock BATCH_SIZE packets are put on the Tx ring. > 4. In read_sock up to BATCH_SIZE packets are picked from the Rx ring. > 5. The application exits. > > In terms of correctness: > > In 3. you pass the same UMEM indicies to the Tx ring (ids={0, > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill > ring. This is not correct. Userspace to track what ids are owned by > the user and by the kernel. Now, both the fill ring and Tx ring > contain the same id. Please I have been looking into this for some days now, but I have not been able to understand what you where referring to and how to fix it. I also updated the xdp program bpf_redirect_map index to write to the second socket (sock2) but I can not observe the reading working when I use any other networking interface except for the 'lo' interface. I would like to inquire if it may have something to do with the code. - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c Thanks Konrad > > I'm not sure what the expected output of your program should be? Have > you verified that packets are being sent? Is the inferface/queue 0 > receiving packets? > > I think it would help if you explained what you are trying to do > first! > > What XDP program are you running? > > > Cheers, > Björn > > > > * > >> Thanks, >> Konrad >> www.djimeli.me ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-08 15:15 ` Konrad Djimeli @ 2018-11-10 14:31 ` Björn Töpel 2018-11-10 14:58 ` William Tu 2018-11-26 16:04 ` Konrad Djimeli 0 siblings, 2 replies; 11+ messages in thread From: Björn Töpel @ 2018-11-10 14:31 UTC (permalink / raw) To: kdjimeli; +Cc: xdp-newbies Den tors 8 nov. 2018 kl 16:15 skrev Konrad Djimeli <kdjimeli@igalia.com>: > [...] > > In terms of correctness: > > > > In 3. you pass the same UMEM indicies to the Tx ring (ids={0, > > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill > > ring. This is not correct. Userspace to track what ids are owned by > > the user and by the kernel. Now, both the fill ring and Tx ring > > contain the same id. > Please I have been looking into this for some days now, but I have not > been able to understand what you where referring to and how to fix it. > Ok, let's talk about the basics, and hopefully it will clear up. The indicies/ids into the UMEM are used to point out packet buffers. The Rx/Tx/Completion/Fill rings are used to pass ownership between the kernel and the user application. E.g. if fill id X with data and pass that to the kernel for transmission, the application has passed ownership of buffer X to the kernel. The application cannot (well, shouldn't) touch the buffer pointed out by X until the kernel is done with the buffer -- in other words, when the buffer is passed back to the application via the completion ring. Analogous, when you pass id Y to the fill ring, Y is owned by the kernel. The application cannot (again, shouldn't) touch the data pointed out by Y until ownership is passed back to the application via the Rx ring. What you did in your old code (I haven't looked at your updates!) is that you passed X, Y and Z to the fill ring. And then, prior obtaining ownership of the buffers, you passed X, Y and Z to the Tx ring. > I also updated the xdp program bpf_redirect_map index to write to the > second socket (sock2) but I can not observe the reading working when I > use any other networking interface except for the 'lo' interface. I > would like to inquire if it may have something to do with the code. > - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c > - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c > If you rely on that all sent packets out to a certain interface should be hairpinned back to same interface, you need to set that up in your switching plane. Or continue to use the loopback interface as you do(?). Björn > Thanks > Konrad > > > > > I'm not sure what the expected output of your program should be? Have > > you verified that packets are being sent? Is the inferface/queue 0 > > receiving packets? > > > > I think it would help if you explained what you are trying to do > > first! > > > > What XDP program are you running? > > > > > > Cheers, > > Björn > > > > > > > > * > > > >> Thanks, > >> Konrad > >> www.djimeli.me ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-10 14:31 ` Björn Töpel @ 2018-11-10 14:58 ` William Tu 2018-11-26 16:03 ` Konrad Djimeli 2018-11-26 16:04 ` Konrad Djimeli 1 sibling, 1 reply; 11+ messages in thread From: William Tu @ 2018-11-10 14:58 UTC (permalink / raw) To: Björn Töpel; +Cc: kdjimeli, Xdp On Sat, Nov 10, 2018 at 6:31 AM Björn Töpel <bjorn.topel@gmail.com> wrote: > > Den tors 8 nov. 2018 kl 16:15 skrev Konrad Djimeli <kdjimeli@igalia.com>: > > > [...] > > > In terms of correctness: > > > > > > In 3. you pass the same UMEM indicies to the Tx ring (ids={0, > > > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill > > > ring. This is not correct. Userspace to track what ids are owned by > > > the user and by the kernel. Now, both the fill ring and Tx ring > > > contain the same id. > > Please I have been looking into this for some days now, but I have not > > been able to understand what you where referring to and how to fix it. > > > > Ok, let's talk about the basics, and hopefully it will clear up. The > indicies/ids into the UMEM are used to point out packet buffers. > > The Rx/Tx/Completion/Fill rings are used to pass ownership between the > kernel and the user application. E.g. if fill id X with data and pass > that to the kernel for transmission, the application has passed > ownership of buffer X to the kernel. The application cannot (well, > shouldn't) touch the buffer pointed out by X until the kernel is done > with the buffer -- in other words, when the buffer is passed back to > the application via the completion ring. Analogous, when you pass id Y > to the fill ring, Y is owned by the kernel. The application cannot > (again, shouldn't) touch the data pointed out by Y until ownership is > passed back to the application via the Rx ring. > Hi Konrad, I made a slide about using how the rx/tx/completion/fill ring works for OVS. Hope this is helpful for you. https://drive.google.com/open?id=1gM3lndS6WeS16QNk74L22UNx5D5wlS2o Another thing for debugging is first to add a print in the XDP code, so making sure the packet is there at the XDP layer. https://github.com/williamtu/ovs-ebpf/blob/afxdp-array/bpf/xdp.h#L71 Then I usually prints the xsk stats https://github.com/williamtu/ovs-ebpf/blob/afxdp-array/lib/netdev-linux.c#L656 So making sure I'm not mess up with these rings. Regards, William ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-10 14:58 ` William Tu @ 2018-11-26 16:03 ` Konrad Djimeli 0 siblings, 0 replies; 11+ messages in thread From: Konrad Djimeli @ 2018-11-26 16:03 UTC (permalink / raw) To: William Tu; +Cc: Björn Töpel, Xdp On 2018-11-10 15:58, William Tu wrote: > On Sat, Nov 10, 2018 at 6:31 AM Björn Töpel <bjorn.topel@gmail.com> wrote: >> >> Den tors 8 nov. 2018 kl 16:15 skrev Konrad Djimeli <kdjimeli@igalia.com>: >> > >> [...] >> > > In terms of correctness: >> > > >> > > In 3. you pass the same UMEM indicies to the Tx ring (ids={0, >> > > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill >> > > ring. This is not correct. Userspace to track what ids are owned by >> > > the user and by the kernel. Now, both the fill ring and Tx ring >> > > contain the same id. >> > Please I have been looking into this for some days now, but I have not >> > been able to understand what you where referring to and how to fix it. >> > >> >> Ok, let's talk about the basics, and hopefully it will clear up. The >> indicies/ids into the UMEM are used to point out packet buffers. >> >> The Rx/Tx/Completion/Fill rings are used to pass ownership between the >> kernel and the user application. E.g. if fill id X with data and pass >> that to the kernel for transmission, the application has passed >> ownership of buffer X to the kernel. The application cannot (well, >> shouldn't) touch the buffer pointed out by X until the kernel is done >> with the buffer -- in other words, when the buffer is passed back to >> the application via the completion ring. Analogous, when you pass id Y >> to the fill ring, Y is owned by the kernel. The application cannot >> (again, shouldn't) touch the data pointed out by Y until ownership is >> passed back to the application via the Rx ring. >> > > Hi Konrad, > > I made a slide about using how the rx/tx/completion/fill ring works for OVS. > Hope this is helpful for you. > https://drive.google.com/open?id=1gM3lndS6WeS16QNk74L22UNx5D5wlS2o > > Another thing for debugging is first to add a print in the XDP code, > so making sure the packet is there at the XDP layer. > https://github.com/williamtu/ovs-ebpf/blob/afxdp-array/bpf/xdp.h#L71 > > Then I usually prints the xsk stats > https://github.com/williamtu/ovs-ebpf/blob/afxdp-array/lib/netdev-linux.c#L656 > So making sure I'm not mess up with these rings. > > Regards, > William Hello William, Thanks a lot. I have gone through the resources you provided, and they have been very helpful, especially the document on how the rx/tx/completion/fill ring works. Thanks, Konrad ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) 2018-11-10 14:31 ` Björn Töpel 2018-11-10 14:58 ` William Tu @ 2018-11-26 16:04 ` Konrad Djimeli 1 sibling, 0 replies; 11+ messages in thread From: Konrad Djimeli @ 2018-11-26 16:04 UTC (permalink / raw) To: Björn Töpel; +Cc: xdp-newbies Hello Björn, On 2018-11-10 15:31, Björn Töpel wrote: > Den tors 8 nov. 2018 kl 16:15 skrev Konrad Djimeli <kdjimeli@igalia.com>: >> > [...] >> > In terms of correctness: >> > >> > In 3. you pass the same UMEM indicies to the Tx ring (ids={0, >> > FRAME_SIZE, 2*FRAME_SIZE, ...}), as you have passed to the fill >> > ring. This is not correct. Userspace to track what ids are owned by >> > the user and by the kernel. Now, both the fill ring and Tx ring >> > contain the same id. >> Please I have been looking into this for some days now, but I have not >> been able to understand what you where referring to and how to fix it. >> > > Ok, let's talk about the basics, and hopefully it will clear up. The > indicies/ids into the UMEM are used to point out packet buffers. > > The Rx/Tx/Completion/Fill rings are used to pass ownership between the > kernel and the user application. E.g. if fill id X with data and pass > that to the kernel for transmission, the application has passed > ownership of buffer X to the kernel. The application cannot (well, > shouldn't) touch the buffer pointed out by X until the kernel is done > with the buffer -- in other words, when the buffer is passed back to > the application via the completion ring. Analogous, when you pass id Y > to the fill ring, Y is owned by the kernel. The application cannot > (again, shouldn't) touch the data pointed out by Y until ownership is > passed back to the application via the Rx ring. > > What you did in your old code (I haven't looked at your updates!) is > that you passed X, Y and Z to the fill ring. And then, prior obtaining > ownership of the buffers, you passed X, Y and Z to the Tx ring. > Sorry for the delayed response, I took some time off from the work I was doing. Thanks a lot for the detailed explanation and I think I have been able to identify the issue you were pointing out. What I have done to address the problem is, since the location passed to the fill ring are {0, 2*FRAME_SIZE, 3*FRAME_SIZE, … , (NUM_DESCS-1)*FRAME_SIZE}, I updated the ids passed to the tx ring to {NUM_DESCS*FRAME_SIZE, (NUM_DESCS+1)*FRAME_SIZE, (NUM_DESCS+2)*FRAME_SIZE, … , (NUM_FRAMES-1)*FRAME_SIZE} [0]. The link below points to the commit addressing the issue. I hope I have not still gotten things mixedup. [0] https://github.com/djkonro/afxdp/commit/c199b496ac6949771fc02260165f4822ccd2f51a#diff-e1b95b7ab6213d7c19f2eacc1290f955 Thanks, Konrad >> I also updated the xdp program bpf_redirect_map index to write to the >> second socket (sock2) but I can not observe the reading working when I >> use any other networking interface except for the 'lo' interface. I >> would like to inquire if it may have something to do with the code. >> - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_kern.c >> - https://github.com/djkonro/afxdp/blob/master/xdpsock/xdpsock_user.c >> > > If you rely on that all sent packets out to a certain interface should > be hairpinned back to same interface, you need to set that up in your > switching plane. Or continue to use the loopback interface as you > do(?). > > > Björn > > >> Thanks >> Konrad >> >> > >> > I'm not sure what the expected output of your program should be? Have >> > you verified that packets are being sent? Is the inferface/queue 0 >> > receiving packets? >> > >> > I think it would help if you explained what you are trying to do >> > first! >> > >> > What XDP program are you running? >> > >> > >> > Cheers, >> > Björn >> > >> > >> > >> > * >> > >> >> Thanks, >> >> Konrad >> >> www.djimeli.me ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-11-27 2:59 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-11-01 16:52 AF_XDP socket (sock2) unable to read, after writing with another AF_XDP socket (sock1) Konrad Djimeli 2018-11-01 18:27 ` Marius Gerling 2018-11-02 15:30 ` Konrad Djimeli 2018-11-01 18:49 ` Björn Töpel 2018-11-02 16:08 ` Konrad Djimeli 2018-11-03 8:38 ` Björn Töpel 2018-11-08 15:15 ` Konrad Djimeli 2018-11-10 14:31 ` Björn Töpel 2018-11-10 14:58 ` William Tu 2018-11-26 16:03 ` Konrad Djimeli 2018-11-26 16:04 ` Konrad Djimeli
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.