LKML Archive on lore.kernel.org
 help / color / Atom feed
From: "Paraschiv, Andra-Irina" <andraprs@amazon.com>
To: "Longpeng (Mike,
	Cloud Infrastructure Service Product Dept.)" 
	<longpeng2@huawei.com>, Paolo Bonzini <pbonzini@redhat.com>,
	<linux-kernel@vger.kernel.org>
Cc: Anthony Liguori <aliguori@amazon.com>,
	Benjamin Herrenschmidt <benh@amazon.com>,
	Colm MacCarthaigh <colmmacc@amazon.com>,
	Bjoern Doebel <doebel@amazon.de>,
	David Woodhouse <dwmw@amazon.co.uk>,
	Frank van der Linden <fllinden@amazon.com>,
	Alexander Graf <graf@amazon.de>,
	Martin Pohlack <mpohlack@amazon.de>, Matt Wilson <msw@amazon.com>,
	Balbir Singh <sblbir@amazon.com>,
	Stewart Smith <trawets@amazon.com>,
	Uwe Dannowski <uwed@amazon.de>, <kvm@vger.kernel.org>,
	<ne-devel-upstream@amazon.com>,
	"Gonglei (Arei)" <arei.gonglei@huawei.com>
Subject: Re: [PATCH v1 00/15] Add support for Nitro Enclaves
Date: Mon, 27 Apr 2020 21:39:32 +0300
Message-ID: <4be2bd36-633c-8ac0-b503-91e54007f944@amazon.com> (raw)
In-Reply-To: <77af0b1c-9884-5a75-02bd-1cc63c57971c@huawei.com>



On 26/04/2020 04:55, Longpeng (Mike, Cloud Infrastructure Service 
Product Dept.) wrote:
>
> On 2020/4/24 17:54, Paraschiv, Andra-Irina wrote:
>>
>> On 24/04/2020 11:19, Paraschiv, Andra-Irina wrote:
>>>
>>> On 24/04/2020 06:04, Longpeng (Mike, Cloud Infrastructure Service Product
>>> Dept.) wrote:
>>>> On 2020/4/23 21:19, Paraschiv, Andra-Irina wrote:
>>>>> On 22/04/2020 00:46, Paolo Bonzini wrote:
>>>>>> On 21/04/20 20:41, Andra Paraschiv wrote:
>>>>>>> An enclave communicates with the primary VM via a local communication
>>>>>>> channel,
>>>>>>> using virtio-vsock [2]. An enclave does not have a disk or a network device
>>>>>>> attached.
>>>>>> Is it possible to have a sample of this in the samples/ directory?
>>>>> I can add in v2 a sample file including the basic flow of how to use the ioctl
>>>>> interface to create / terminate an enclave.
>>>>>
>>>>> Then we can update / build on top it based on the ongoing discussions on the
>>>>> patch series and the received feedback.
>>>>>
>>>>>> I am interested especially in:
>>>>>>
>>>>>> - the initial CPU state: CPL0 vs. CPL3, initial program counter, etc.
>>>>>>
>>>>>> - the communication channel; does the enclave see the usual local APIC
>>>>>> and IOAPIC interfaces in order to get interrupts from virtio-vsock, and
>>>>>> where is the virtio-vsock device (virtio-mmio I suppose) placed in memory?
>>>>>>
>>>>>> - what the enclave is allowed to do: can it change privilege levels,
>>>>>> what happens if the enclave performs an access to nonexistent memory, etc.
>>>>>>
>>>>>> - whether there are special hypercall interfaces for the enclave
>>>>> An enclave is a VM, running on the same host as the primary VM, that launched
>>>>> the enclave. They are siblings.
>>>>>
>>>>> Here we need to think of two components:
>>>>>
>>>>> 1. An enclave abstraction process - a process running in the primary VM guest,
>>>>> that uses the provided ioctl interface of the Nitro Enclaves kernel driver to
>>>>> spawn an enclave VM (that's 2 below).
>>>>>
>>>>> How does all gets to an enclave VM running on the host?
>>>>>
>>>>> There is a Nitro Enclaves emulated PCI device exposed to the primary VM. The
>>>>> driver for this new PCI device is included in the current patch series.
>>>>>
>>>> Hi Paraschiv,
>>>>
>>>> The new PCI device is emulated in QEMU ? If so, is there any plan to send the
>>>> QEMU code ?
>>> Hi,
>>>
>>> Nope, not that I know of so far.
>> And just to be a bit more clear, the reply above takes into consideration that
>> it's not emulated in QEMU.
>>
> Thanks.
>
> Guys in this thread are much more interested in the design of enclave VM and the
> new device, but there's no any document about this device yet, so I think the
> emulate code is a good alternative. However, Alex said the device specific will
> be published later, so I'll wait for it.

True, that was mentioned wrt device spec. The device interface could 
also be updated based on the ongoing discussions on the patch series. 
Refs to the device spec should be included e.g. in the .h file of the 
PCI device, once it's available.

Thanks,
Andra

>
>> Thanks,
>> Andra
>>
>>>>> The ioctl logic is mapped to PCI device commands e.g. the NE_ENCLAVE_START
>>>>> ioctl
>>>>> maps to an enclave start PCI command or the KVM_SET_USER_MEMORY_REGION maps to
>>>>> an add memory PCI command. The PCI device commands are then translated into
>>>>> actions taken on the hypervisor side; that's the Nitro hypervisor running on
>>>>> the
>>>>> host where the primary VM is running.
>>>>>
>>>>> 2. The enclave itself - a VM running on the same host as the primary VM that
>>>>> spawned it.
>>>>>
>>>>> The enclave VM has no persistent storage or network interface attached, it uses
>>>>> its own memory and CPUs + its virtio-vsock emulated device for communication
>>>>> with the primary VM.
>>>>>
>>>>> The memory and CPUs are carved out of the primary VM, they are dedicated for
>>>>> the
>>>>> enclave. The Nitro hypervisor running on the host ensures memory and CPU
>>>>> isolation between the primary VM and the enclave VM.
>>>>>
>>>>>
>>>>> These two components need to reflect the same state e.g. when the enclave
>>>>> abstraction process (1) is terminated, the enclave VM (2) is terminated as
>>>>> well.
>>>>>
>>>>> With regard to the communication channel, the primary VM has its own emulated
>>>>> virtio-vsock PCI device. The enclave VM has its own emulated virtio-vsock
>>>>> device
>>>>> as well. This channel is used, for example, to fetch data in the enclave and
>>>>> then process it. An application that sets up the vsock socket and connects or
>>>>> listens, depending on the use case, is then developed to use this channel; this
>>>>> happens on both ends - primary VM and enclave VM.
>>>>>
>>>>> Let me know if further clarifications are needed.
>>>>>
>>>>>>> The proposed solution is following the KVM model and uses the KVM API to
>>>>>>> be able
>>>>>>> to create and set resources for enclaves. An additional ioctl command,
>>>>>>> besides
>>>>>>> the ones provided by KVM, is used to start an enclave and setup the
>>>>>>> addressing
>>>>>>> for the communication channel and an enclave unique id.
>>>>>> Reusing some KVM ioctls is definitely a good idea, but I wouldn't really
>>>>>> say it's the KVM API since the VCPU file descriptor is basically non
>>>>>> functional (without KVM_RUN and mmap it's not really the KVM API).
>>>>> It uses part of the KVM API or a set of KVM ioctls to model the way a VM is
>>>>> created / terminated. That's true, KVM_RUN and mmap-ing the vcpu fd are not
>>>>> included.
>>>>>
>>>>> Thanks for the feedback regarding the reuse of KVM ioctls.
>>>>>
>>>>> Andra
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar
>>>>> Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in
>>>>> Romania. Registration number J22/2621/2005.
>>
>>
>>
>> Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar
>> Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in
>> Romania. Registration number J22/2621/2005.
> ---
> Regards,
> Longpeng(Mike)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

  reply index

Thread overview: 77+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-21 18:41 Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 01/15] nitro_enclaves: Add ioctl interface definition Andra Paraschiv
2020-04-21 18:47   ` Randy Dunlap
2020-04-21 21:45     ` Paolo Bonzini
2020-04-22 15:49       ` Paraschiv, Andra-Irina
2020-04-21 18:41 ` [PATCH v1 02/15] nitro_enclaves: Define the PCI device interface Andra Paraschiv
2020-04-21 21:22   ` Paolo Bonzini
2020-04-23 13:37     ` Paraschiv, Andra-Irina
2020-04-24 15:10       ` Paraschiv, Andra-Irina
2020-04-21 18:41 ` [PATCH v1 03/15] nitro_enclaves: Define enclave info for internal bookkeeping Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 04/15] nitro_enclaves: Init PCI device driver Andra Paraschiv
2020-04-25 14:25   ` Liran Alon
2020-04-29 16:31     ` Paraschiv, Andra-Irina
2020-04-21 18:41 ` [PATCH v1 05/15] nitro_enclaves: Handle PCI device command requests Andra Paraschiv
2020-04-25 14:52   ` Liran Alon
2020-04-29 17:00     ` Paraschiv, Andra-Irina
2020-04-21 18:41 ` [PATCH v1 06/15] nitro_enclaves: Handle out-of-band PCI device events Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 07/15] nitro_enclaves: Init misc device providing the ioctl interface Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 08/15] nitro_enclaves: Add logic for enclave vm creation Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 09/15] nitro_enclaves: Add logic for enclave vcpu creation Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 10/15] nitro_enclaves: Add logic for enclave memory region set Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 11/15] nitro_enclaves: Add logic for enclave start Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 12/15] nitro_enclaves: Add logic for enclave termination Andra Paraschiv
2020-04-21 18:41 ` [PATCH v1 13/15] nitro_enclaves: Add Kconfig for the Nitro Enclaves driver Andra Paraschiv
2020-04-21 18:50   ` Randy Dunlap
2020-04-22 14:35     ` Paraschiv, Andra-Irina
2020-04-21 18:41 ` [PATCH v1 14/15] nitro_enclaves: Add Makefile " Andra Paraschiv
2020-04-23  8:12   ` kbuild test robot
2020-04-24 17:00     ` Paraschiv, Andra-Irina
2020-04-23  8:43   ` kbuild test robot
2020-04-24 15:27     ` Paraschiv, Andra-Irina
2020-04-21 18:41 ` [PATCH v1 15/15] MAINTAINERS: Add entry " Andra Paraschiv
2020-04-21 21:46 ` [PATCH v1 00/15] Add support for Nitro Enclaves Paolo Bonzini
2020-04-23 13:19   ` Paraschiv, Andra-Irina
2020-04-23 13:42     ` Paolo Bonzini
2020-04-23 17:42       ` Paraschiv, Andra-Irina
2020-04-23 17:51         ` Paolo Bonzini
2020-04-23 20:56           ` Alexander Graf
2020-04-23 21:18             ` Paolo Bonzini
2020-04-24 12:56               ` Alexander Graf
2020-04-24 16:27                 ` Paolo Bonzini
2020-04-24 19:11                   ` Alexander Graf
2020-04-25 16:05                     ` Paolo Bonzini
2020-04-27  9:15                       ` Paraschiv, Andra-Irina
2020-04-27  9:22                       ` Paraschiv, Andra-Irina
2020-04-27  9:46                         ` Paolo Bonzini
2020-04-27 10:00                           ` Paraschiv, Andra-Irina
2020-04-28 15:07                       ` Alexander Graf
2020-04-29 13:20                         ` Paolo Bonzini
2020-04-30 13:59                           ` Paraschiv, Andra-Irina
2020-04-30 10:34                         ` Paolo Bonzini
2020-04-30 11:21                           ` Alexander Graf
2020-04-30 11:38                             ` Paolo Bonzini
2020-04-30 11:47                               ` Alexander Graf
2020-04-30 11:58                                 ` Paolo Bonzini
2020-04-30 12:19                                   ` Alexander Graf
2020-05-07 17:44       ` Pavel Machek
2020-05-08  7:00         ` Paraschiv, Andra-Irina
2020-05-09 19:21           ` Pavel Machek
2020-05-10 11:02             ` Herrenschmidt, Benjamin
2020-05-11 10:49               ` Paraschiv, Andra-Irina
2020-05-11 13:49               ` Stefan Hajnoczi
2020-04-24  3:04     ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
2020-04-24  8:19       ` Paraschiv, Andra-Irina
2020-04-24  9:54         ` Paraschiv, Andra-Irina
2020-04-26  1:55           ` Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
2020-04-27 18:39             ` Paraschiv, Andra-Irina [this message]
2020-04-24  9:59     ` Tian, Kevin
2020-04-24 13:59       ` Paraschiv, Andra-Irina
2020-04-26  8:16         ` Tian, Kevin
2020-04-27 19:05           ` Paraschiv, Andra-Irina
     [not found]         ` <CAKXe6SLonLQLAOY9Q_2AzTeg4uJxiknsAWnJpTF0hMcXEG5Tew@mail.gmail.com>
2020-05-11 12:05           ` Paraschiv, Andra-Irina
2020-04-25 15:25     ` Liran Alon
2020-04-27  7:56       ` Paraschiv, Andra-Irina
2020-04-27 11:44         ` Liran Alon
2020-04-28 15:25           ` Alexander Graf
2020-04-28 16:01             ` Liran Alon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4be2bd36-633c-8ac0-b503-91e54007f944@amazon.com \
    --to=andraprs@amazon.com \
    --cc=aliguori@amazon.com \
    --cc=arei.gonglei@huawei.com \
    --cc=benh@amazon.com \
    --cc=colmmacc@amazon.com \
    --cc=doebel@amazon.de \
    --cc=dwmw@amazon.co.uk \
    --cc=fllinden@amazon.com \
    --cc=graf@amazon.de \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=longpeng2@huawei.com \
    --cc=mpohlack@amazon.de \
    --cc=msw@amazon.com \
    --cc=ne-devel-upstream@amazon.com \
    --cc=pbonzini@redhat.com \
    --cc=sblbir@amazon.com \
    --cc=trawets@amazon.com \
    --cc=uwed@amazon.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git