Re: io_uring process termination/killing is not working

[Josef <josef.grieb@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3029 bytes --]

it seems to be that read event doesn't work properly, but I'm not sure
if it is related to what Pavel mentioned
poll<link>accept works but not poll<link>read -> cqe still receives
poll event but no read event, however I received a read event after
the third request via telnet

I just tested https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=io_uring-5.9&id=d4e7cd36a90e38e0276d6ce0c20f5ccef17ec38c
and
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=io_uring-5.9&id=227c0c9673d86732995474d277f84e08ee763e46
(but it works on Linux 5.7)


On Sat, 15 Aug 2020 at 18:50, Pavel Begunkov <asml.silence@gmail.com> wrote:
>
> On 15/08/2020 18:12, Jens Axboe wrote:
> > On 8/15/20 12:45 AM, Pavel Begunkov wrote:
> >> On 13/08/2020 02:32, Jens Axboe wrote:
> >>> On 8/12/20 12:28 PM, Pavel Begunkov wrote:
> >>>> On 12/08/2020 21:22, Pavel Begunkov wrote:
> >>>>> On 12/08/2020 21:20, Pavel Begunkov wrote:
> >>>>>> On 12/08/2020 21:05, Jens Axboe wrote:
> >>>>>>> On 8/12/20 11:58 AM, Josef wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I have a weird issue on kernel 5.8.0/5.8.1, SIGINT even SIGKILL
> >>>>>>>> doesn't work to kill this process(always state D or D+), literally I
> >>>>>>>> have to terminate my VM because even the kernel can't kill the process
> >>>>>>>> and no issue on 5.7.12-201, however if IOSQE_IO_LINK is not set, it
> >>>>>>>> works
> >>>>>>>>
> >>>>>>>> I've attached a file to reproduce it
> >>>>>>>> or here
> >>>>>>>> https://gist.github.com/1Jo1/15cb3c63439d0c08e3589cfa98418b2c
> >>>>>>>
> >>>>>>> Thanks, I'll take a look at this. It's stuck in uninterruptible
> >>>>>>> state, which is why you can't kill it.
> >>>>>>
> >>>>>> It looks like one of the hangs I've been talking about a few days ago,
> >>>>>> an accept is inflight but can't be found by cancel_files() because it's
> >>>>>> in a link.
> >>>>>
> >>>>> BTW, I described it a month ago, there were more details.
> >>>>
> >>>> https://lore.kernel.org/io-uring/34eb5e5a-8d37-0cae-be6c-c6ac4d85b5d4@gmail.com
> >>>
> >>> Yeah I think you're right. How about something like the below? That'll
> >>> potentially cancel more than just the one we're looking for, but seems
> >>> kind of silly to only cancel from the file table holding request and to
> >>> the end.
> >>
> >> The bug is not poll/t-out related, IIRC my test reproduces it with
> >> read(pipe)->open(). See the previously sent link.
> >
> > Right, but in this context for poll, I just mean any request that has a
> > poll handler armed. Not necessarily only a pure poll. The patch should
> > fix your case, too.
>
> Ok. I was thinking about sleeping in io_read(), etc. from io-wq context.
> That should have the same effect.
>
> >
> >> As mentioned, I'm going to patch that up, if you won't beat me on that.
> >
> > Please test and send a fix if you find something! I'm going to ship what
> > I have this weekend, but we can always add a fix on top if we need
> > anything.
>
> Sure
>
> --
> Pavel Begunkov

--
Josef Grieb

[-- Attachment #2: io_uring_read_issue.c --]
[-- Type: application/octet-stream, Size: 2793 bytes --]

#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <sys/socket.h>
#include <unistd.h>
#include <poll.h>
#include "liburing.h"

#define BACKLOG 512

#define PORT 9300

struct io_uring ring;

char buf[100];

void add_poll(int fd) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_poll_add(sqe, fd, POLLIN);
    sqe->user_data = 1;
    sqe->flags |= IOSQE_IO_LINK;
}

void add_accept(int fd) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_accept(sqe, fd, 0, 0, SOCK_NONBLOCK | SOCK_CLOEXEC);
    sqe->user_data = 2;
    sqe->flags |= IOSQE_IO_LINK;
}

void add_read(int fd) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_read(sqe, fd, &buf, 100, 0);
    sqe->user_data = 3;
    sqe->flags |= IOSQE_IO_LINK;
}

int setup_io_uring() {
    int ret = io_uring_queue_init(16, &ring, 0);
    if (ret) {
        fprintf(stderr, "Unable to setup io_uring: %s\n", strerror(-ret));
        return 1;
    }
    return 0;
}

int main(int argc, char *argv[]) {

    struct sockaddr_in serv_addr;

    setup_io_uring();
    
    int sock_listen_fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
    const int val = 1;
    setsockopt(sock_listen_fd, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val));

    memset(&serv_addr, 0, sizeof(serv_addr));
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_port = htons(PORT);
    serv_addr.sin_addr.s_addr = INADDR_ANY;

    if (bind(sock_listen_fd, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0) {
         perror("Error binding socket\n");
         exit(1);
     }
    if (listen(sock_listen_fd, BACKLOG) < 0) {
         perror("Error listening on socket\n");
         exit(1);
    }

    setup_io_uring();

    add_poll(sock_listen_fd);
    add_accept(sock_listen_fd);
    io_uring_submit(&ring);

    while (1) {
        struct io_uring_cqe *cqe;
        io_uring_wait_cqe(&ring, &cqe);

        printf("Res: res: %d\n", cqe->res);
        
        if (cqe->user_data == 1) {
            printf("Poll Event\n");
        }
        
        if (cqe->user_data == 2 && cqe->res > 0) {
            printf("Accept Event\n");
                    
            add_poll(sock_listen_fd);
            add_accept(sock_listen_fd);

            //avoid link between add_accept and add_poll
            struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
            io_uring_prep_nop(sqe); 

            add_poll(cqe->res);
            add_read(cqe->res);
        }

        if (cqe->user_data == 3) {
            printf("Read Buf: %s \n", buf);
        }
        io_uring_submit(&ring);

        io_uring_cqe_seen(&ring, cqe);
    }

    io_uring_queue_exit(&ring);

    return 0;
}