linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Stefano Garzarella <sgarzare@redhat.com>
To: Jens Axboe <axboe@kernel.dk>
Cc: Stefan Hajnoczi <stefanha@redhat.com>,
	Jeff Moyer <jmoyer@redhat.com>,
	io-uring@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [RFC] io_uring: add restrictions to support untrusted applications and guests
Date: Mon, 15 Jun 2020 09:23:10 +0200	[thread overview]
Message-ID: <20200615072310.iymkgr4dqdwzafg3@steredhat> (raw)
In-Reply-To: <f96beb0a-415c-92fb-96f4-3902b613e9e4@kernel.dk>

On Sun, Jun 14, 2020 at 09:52:30AM -0600, Jens Axboe wrote:
> On 6/9/20 8:24 AM, Stefano Garzarella wrote:
> > Hi Jens,
> > Stefan and I have a proposal to share with io_uring community.
> > Before implementing it we would like to discuss it to receive feedbacks and
> > to see if it could be accepted:
> > 
> > Adding restrictions to io_uring
> > =====================================
> > The io_uring API provides submission and completion queues for performing
> > asynchronous I/O operations. The queues are located in memory that is
> > accessible to both the host userspace application and the kernel, making it
> > possible to monitor for activity through polling instead of system calls. This
> > design offers good performance and this makes exposing io_uring to guests an
> > attractive idea for improving I/O performance in virtualization.
> > 
> > PoC and preliminary benchmarks
> > ---------------------------
> > We realized a PoC, using QEMU and virtio-blk device, to share io_uring
> > CQ and SQ rings with the guest.
> > QEMU initializes io_uring, registers the device (NVMe) fd through
> > io_uring_register(2), and maps the rings in the guest memory.
> > The virtio-blk driver uses these rings to send requests instead of using
> > the standard virtqueues.
> > 
> > The PoC implements a pure polling solution where the application is polling
> > (IOPOLL enabled) in the guest and the sqpoll_kthread is polling in the host
> > (SQPOLL and IOPOLL enabled).
> > 
> > These are the encouraging results we obtained from this preliminary work;
> > we used fio (rw=randread bs=4k) to measure the kIOPS on a NVMe device:
> > 
> > - bare-metal
> >                                                        iodepth
> >   | fio ioengine                              |  1  |  8  |  16 |  32 |
> >   |-------------------------------------------|----:|----:|----:|----:|
> >   | io_uring (SQPOLL + IOPOLL)                | 119 | 550 | 581 | 585 |
> >   | io_uring (IOPOLL)                         | 122 | 502 | 519 | 538 |
> > 
> > - QEMU/KVM guest (aio=io_uring)
> >                                                        iodepth
> >   | virtio-blk            | fio ioengine      |  1  |  8  |  16 |  32 |
> >   |-----------------------|-------------------|----:|----:|----:|----:|
> >   | virtqueues            | io_uring (IOPOLL) |  27 | 144 | 209 | 266 |
> >   | virtqueues + iothread | io_uring (IOPOLL) |  73 | 264 | 306 | 312 |
> >   | io_uring passthrough  | io_uring (IOPOLL) | 104 | 532 | 577 | 585 |
> > 
> >   All guest experiments are using the QEMU io_uring backend with SQPOLL and
> >   IOPOLL enabled. The virtio-blk driver is modified to support blovk io_poll
> >   on both virtqueues and io_uring passthrough.
> > 
> > Before developing this proof-of-concept further we would like to discuss
> > io_uring changes required to restrict rings since this mechanism is a
> > prerequisite for real-world use cases where guests are untrusted.
> > 
> > Restrictions
> > ------------
> > This document proposes io_uring API changes that safely allow untrusted
> > applications or guests to use io_uring. io_uring's existing security model is
> > that of kernel system call handler code. It is designed to reject invalid
> > inputs from host userspace applications. Supporting guests as io_uring API
> > clients adds a new trust domain with access to even fewer resources than host
> > userspace applications.
> > 
> > Guests do not have direct access to host userspace application file descriptors
> > or memory. The host userspace application, a Virtual Machine Monitor (VMM) such
> > as QEMU, grants access to a subset of its file descriptors and memory. The
> > allowed file descriptors are typically the disk image files belonging to the
> > guest. The memory is typically the virtual machine's RAM that the VMM has
> > allocated on behalf of the guest.
> > 
> > The following extensions to the io_uring API allow the host application to
> > grant access to some of its file descriptors.
> > 
> > These extensions are designed to be applicable to other use cases besides
> > untrusted guests and are not virtualization-specific. For example, the
> > restrictions can be used to allow only a subset of sqe operations available to
> > an application similar to seccomp syscall whitelisting.
> > 
> > An address translation and memory restriction mechanism would also be
> > necessary, but we can discuss this later.
> > 
> > The IOURING_REGISTER_RESTRICTIONS opcode
> > ----------------------------------------
> > The new io_uring_register(2) IOURING_REGISTER_RESTRICTIONS opcode permanently
> > installs a feature whitelist on an io_ring_ctx. The io_ring_ctx can then be
> > passed to untrusted code with the knowledge that only operations present in the
> > whitelist can be executed.
> > 
> > The whitelist approach ensures that new features added to io_uring do not
> > accidentally become available when an existing application is launched on a
> > newer kernel version.
> > 
> > The IORING_REGISTER_RESTRICTIONS opcode takes an array of struct
> > io_uring_restriction elements that describe whitelisted features:
> > 
> >   #define IORING_REGISTER_RESTRICTIONS 11
> > 
> >   /* struct io_uring_restriction::opcode values */
> >   enum {
> >       /* Allow an io_uring_register(2) opcode */
> >       IORING_RESTRICTION_REGISTER_OP,
> > 
> >       /* Allow an sqe opcode */
> >       IORING_RESTRICTION_SQE_OP,
> > 
> >       /* Only allow fixed files */
> >       IORING_RESTRICTION_FIXED_FILES_ONLY,
> > 
> >       /* Only allow registered addresses and translate them */
> >       IORING_RESTRICTION_BUFFER_CHECK
> >   };
> > 
> >   struct io_uring_restriction {
> >       __u16 opcode;
> >       union {
> >           __u8 register_op; /* IORING_RESTRICTION_REGISTER_OP */
> >           __u8 sqe_op;      /* IORING_RESTRICTION_SQE_OP */
> >       };
> >       __u8 resv;
> >       __u32 resv2[3];
> >   };
> > 
> > This call can only be made once. Afterwards it is not possible to change
> > restrictions anymore. This prevents untrusted code from removing restrictions.
> > 
> > Limiting access to io_uring operations
> > --------------------------------------
> > The following example shows how to whitelist IORING_OP_READV, IORING_OP_WRITEV,
> > and IORING_OP_FSYNC:
> > 
> >   struct io_uring_restriction restrictions[] = {
> >       {
> >           .opcode = IORING_RESTRICTION_SQE_OP,
> >           .sqe_op = IORING_OP_READV,
> >       },
> >       {
> >           .opcode = IORING_RESTRICTION_SQE_OP,
> >           .sqe_op = IORING_OP_WRITEV,
> >       },
> >       {
> >           .opcode = IORING_RESTRICTION_SQE_OP,
> >           .sqe_op = IORING_OP_FSYNC,
> >       },
> >       ...
> >   };
> > 
> >   io_uring_register(ringfd, IORING_REGISTER_RESTRICTIONS,
> >                     restrictions, ARRAY_SIZE(restrictions));
> > 
> > Limiting access to file descriptors
> > -----------------------------------
> > The fixed files mechanism can be used to limit access to a set of file
> > descriptors:
> > 
> >   struct io_uring_restriction restrictions[] = {
> >       {
> >           .opcode = IORING_RESTRICTION_FIXED_FILES_ONLY,
> >       },
> >       ...
> >   };
> > 
> >   io_uring_register(ringfd, IORING_REGISTER_RESTRICTIONS,
> >                     restrictions, ARRAY_SIZE(restrictions));
> > 
> > Only requests with the sqe->flags IOSQE_FIXED_FILE bit set will be allowed.
> 
> I don't think this sounds unreasonable, but I'd really like to see a
> prototype hacked up before rendering any further opinions on it :-)

Yeah :-) I'll be back with a prototype of this changes ASAP.

Thanks for you feedback,
Stefano


  reply	other threads:[~2020-06-15  7:23 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-09 14:24 Stefano Garzarella
2020-06-14 15:52 ` Jens Axboe
2020-06-15  7:23   ` Stefano Garzarella [this message]
2020-06-15  9:04 ` Jann Horn
2020-06-15 13:33   ` Stefano Garzarella
2020-06-15 17:00     ` Jens Axboe
2020-06-16  9:12       ` Stefano Garzarella
2020-06-16 11:32         ` Jann Horn
2020-06-16 14:07           ` Stefano Garzarella
2020-06-16 15:26         ` Jens Axboe
2020-06-16 16:07           ` Stefano Garzarella
2020-06-15 22:01   ` Christian Brauner
2020-06-15 23:26     ` Jann Horn

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200615072310.iymkgr4dqdwzafg3@steredhat \
    --to=sgarzare@redhat.com \
    --cc=axboe@kernel.dk \
    --cc=io-uring@vger.kernel.org \
    --cc=jmoyer@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stefanha@redhat.com \
    --subject='Re: [RFC] io_uring: add restrictions to support untrusted applications and guests' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).