From: "Paraschiv, Andra-Irina" <andraprs@amazon.com>
To: Alexander Graf <graf@amazon.de>, <linux-kernel@vger.kernel.org>
Cc: Anthony Liguori <aliguori@amazon.com>,
Benjamin Herrenschmidt <benh@kernel.crashing.org>,
Colm MacCarthaigh <colmmacc@amazon.com>,
"Bjoern Doebel" <doebel@amazon.de>,
David Woodhouse <dwmw@amazon.co.uk>,
"Frank van der Linden" <fllinden@amazon.com>,
Greg KH <gregkh@linuxfoundation.org>,
Martin Pohlack <mpohlack@amazon.de>, Matt Wilson <msw@amazon.com>,
"Paolo Bonzini" <pbonzini@redhat.com>,
Balbir Singh <sblbir@amazon.com>,
"Stefano Garzarella" <sgarzare@redhat.com>,
Stefan Hajnoczi <stefanha@redhat.com>,
Stewart Smith <trawets@amazon.com>,
Uwe Dannowski <uwed@amazon.de>, <kvm@vger.kernel.org>,
<ne-devel-upstream@amazon.com>
Subject: Re: [PATCH v4 06/18] nitro_enclaves: Handle out-of-band PCI device events
Date: Sat, 4 Jul 2020 18:43:04 +0300 [thread overview]
Message-ID: <96667e27-3379-4817-843f-988af9db4366@amazon.com> (raw)
In-Reply-To: <82768612-0d23-55a9-dedf-58ade57b37af@amazon.de>
On 02/07/2020 18:24, Alexander Graf wrote:
>
>
> On 22.06.20 22:03, Andra Paraschiv wrote:
>> In addition to the replies sent by the Nitro Enclaves PCI device in
>> response to command requests, out-of-band enclave events can happen e.g.
>> an enclave crashes. In this case, the Nitro Enclaves driver needs to be
>> aware of the event and notify the corresponding user space process that
>> abstracts the enclave.
>>
>> Register an MSI-X interrupt vector to be used for this kind of
>> out-of-band events. The interrupt notifies that the state of an enclave
>> changed and the driver logic scans the state of each running enclave to
>> identify for which this notification is intended.
>>
>> Create an workqueue to handle the out-of-band events. Notify user space
>> enclave process that is using a polling mechanism on the enclave fd.
>>
>> Signed-off-by: Alexandru-Catalin Vasile <lexnv@amazon.com>
>> Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
>> ---
>> Changelog
>>
>> v3 -> v4
>>
>> * Use dev_err instead of custom NE log pattern.
>> * Return IRQ_NONE when interrupts are not handled.
>>
>> v2 -> v3
>>
>> * Remove the WARN_ON calls.
>> * Update static calls sanity checks.
>> * Remove "ratelimited" from the logs that are not in the ioctl call
>> paths.
>>
>> v1 -> v2
>>
>> * Add log pattern for NE.
>> * Update goto labels to match their purpose.
>> ---
>> drivers/virt/nitro_enclaves/ne_pci_dev.c | 122 +++++++++++++++++++++++
>> 1 file changed, 122 insertions(+)
>>
>> diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c
>> b/drivers/virt/nitro_enclaves/ne_pci_dev.c
>> index c24230cfe7c0..9a137862cade 100644
>> --- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
>> +++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
>> @@ -239,6 +239,93 @@ static irqreturn_t ne_reply_handler(int irq,
>> void *args)
>> return IRQ_HANDLED;
>> }
>> +/**
>> + * ne_event_work_handler - Work queue handler for notifying enclaves on
>> + * a state change received by the event interrupt handler.
>> + *
>> + * An out-of-band event is being issued by the Nitro Hypervisor when
>> at least
>> + * one enclave is changing state without client interaction.
>> + *
>> + * @work: item containing the Nitro Enclaves PCI device for which a
>> + * out-of-band event was issued.
>> + */
>> +static void ne_event_work_handler(struct work_struct *work)
>> +{
>> + struct ne_pci_dev_cmd_reply cmd_reply = {};
>> + struct ne_enclave *ne_enclave = NULL;
>> + struct ne_pci_dev *ne_pci_dev =
>> + container_of(work, struct ne_pci_dev, notify_work);
>> + int rc = -EINVAL;
>> + struct slot_info_req slot_info_req = {};
>> +
>> + if (!ne_pci_dev)
>> + return;
>
> How?
Removed this check and the one below. Thank you.
Andra
>
>> +
>> + mutex_lock(&ne_pci_dev->enclaves_list_mutex);
>> +
>> + /*
>> + * Iterate over all enclaves registered for the Nitro Enclaves
>> + * PCI device and determine for which enclave(s) the out-of-band
>> event
>> + * is corresponding to.
>> + */
>> + list_for_each_entry(ne_enclave, &ne_pci_dev->enclaves_list,
>> + enclave_list_entry) {
>> + mutex_lock(&ne_enclave->enclave_info_mutex);
>> +
>> + /*
>> + * Enclaves that were never started cannot receive out-of-band
>> + * events.
>> + */
>> + if (ne_enclave->state != NE_STATE_RUNNING)
>> + goto unlock;
>> +
>> + slot_info_req.slot_uid = ne_enclave->slot_uid;
>> +
>> + rc = ne_do_request(ne_enclave->pdev, SLOT_INFO, &slot_info_req,
>> + sizeof(slot_info_req), &cmd_reply,
>> + sizeof(cmd_reply));
>> + if (rc < 0)
>> + dev_err(&ne_enclave->pdev->dev,
>> + "Error in slot info [rc=%d]\n", rc);
>> +
>> + /* Notify enclave process that the enclave state changed. */
>> + if (ne_enclave->state != cmd_reply.state) {
>> + ne_enclave->state = cmd_reply.state;
>> +
>> + ne_enclave->has_event = true;
>> +
>> + wake_up_interruptible(&ne_enclave->eventq);
>> + }
>> +
>> +unlock:
>> + mutex_unlock(&ne_enclave->enclave_info_mutex);
>> + }
>> +
>> + mutex_unlock(&ne_pci_dev->enclaves_list_mutex);
>> +}
>> +
>> +/**
>> + * ne_event_handler - Interrupt handler for PCI device out-of-band
>> + * events. This interrupt does not supply any data in the MMIO region.
>> + * It notifies a change in the state of any of the launched enclaves.
>> + *
>> + * @irq: received interrupt for an out-of-band event.
>> + * @args: PCI device private data structure.
>> + *
>> + * @returns: IRQ_HANDLED on handled interrupt, IRQ_NONE otherwise.
>> + */
>> +static irqreturn_t ne_event_handler(int irq, void *args)
>> +{
>> + struct ne_pci_dev *ne_pci_dev = (struct ne_pci_dev *)args;
>> +
>> + if (!ne_pci_dev)
>> + return IRQ_NONE;
>
> How can this happen?
>
>
> Alex
>
>> +
>> + queue_work(ne_pci_dev->event_wq, &ne_pci_dev->notify_work);
>> +
>> + return IRQ_HANDLED;
>> +}
>> +
>> /**
>> * ne_setup_msix - Setup MSI-X vectors for the PCI device.
>> *
>> @@ -284,8 +371,37 @@ static int ne_setup_msix(struct pci_dev *pdev)
>> goto free_irq_vectors;
>> }
>> + ne_pci_dev->event_wq =
>> create_singlethread_workqueue("ne_pci_dev_wq");
>> + if (!ne_pci_dev->event_wq) {
>> + rc = -ENOMEM;
>> +
>> + dev_err(&pdev->dev, "Cannot get wq for dev events [rc=%d]\n",
>> + rc);
>> +
>> + goto free_reply_irq_vec;
>> + }
>> +
>> + INIT_WORK(&ne_pci_dev->notify_work, ne_event_work_handler);
>> +
>> + /*
>> + * This IRQ gets triggered every time any enclave's state
>> changes. Its
>> + * handler then scans for the changes and propagates them to the
>> user
>> + * space.
>> + */
>> + rc = request_irq(pci_irq_vector(pdev, NE_VEC_EVENT),
>> + ne_event_handler, 0, "enclave_evt", ne_pci_dev);
>> + if (rc < 0) {
>> + dev_err(&pdev->dev, "Error in request irq event [rc=%d]\n",
>> rc);
>> +
>> + goto destroy_wq;
>> + }
>> +
>> return 0;
>> +destroy_wq:
>> + destroy_workqueue(ne_pci_dev->event_wq);
>> +free_reply_irq_vec:
>> + free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev);
>> free_irq_vectors:
>> pci_free_irq_vectors(pdev);
>> @@ -304,6 +420,12 @@ static void ne_teardown_msix(struct pci_dev
>> *pdev)
>> if (!ne_pci_dev)
>> return;
>> + free_irq(pci_irq_vector(pdev, NE_VEC_EVENT), ne_pci_dev);
>> +
>> + flush_work(&ne_pci_dev->notify_work);
>> + flush_workqueue(ne_pci_dev->event_wq);
>> + destroy_workqueue(ne_pci_dev->event_wq);
>> +
>> free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev);
>> pci_free_irq_vectors(pdev);
>>
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
next prev parent reply other threads:[~2020-07-04 15:43 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-22 20:03 [PATCH v4 00/18] Add support for Nitro Enclaves Andra Paraschiv
2020-06-22 20:03 ` [PATCH v4 01/18] nitro_enclaves: Add ioctl interface definition Andra Paraschiv
2020-06-23 8:56 ` Stefan Hajnoczi
2020-06-24 14:02 ` Paraschiv, Andra-Irina
2020-06-25 13:29 ` Stefan Hajnoczi
2020-06-25 17:42 ` Paraschiv, Andra-Irina
2020-07-02 15:24 ` Alexander Graf
2020-07-04 8:09 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 02/18] nitro_enclaves: Define the PCI device interface Andra Paraschiv
2020-07-02 15:24 ` Alexander Graf
2020-07-04 8:20 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 03/18] nitro_enclaves: Define enclave info for internal bookkeeping Andra Paraschiv
2020-07-02 15:24 ` Alexander Graf
2020-07-04 8:23 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 04/18] nitro_enclaves: Init PCI device driver Andra Paraschiv
2020-07-02 15:09 ` Alexander Graf
2020-07-04 10:00 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 05/18] nitro_enclaves: Handle PCI device command requests Andra Paraschiv
2020-07-02 15:19 ` Alexander Graf
2020-07-04 15:05 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 06/18] nitro_enclaves: Handle out-of-band PCI device events Andra Paraschiv
2020-07-02 15:24 ` Alexander Graf
2020-07-04 15:43 ` Paraschiv, Andra-Irina [this message]
2020-06-22 20:03 ` [PATCH v4 07/18] nitro_enclaves: Init misc device providing the ioctl interface Andra Paraschiv
2020-06-29 16:20 ` Greg KH
2020-06-29 17:45 ` Paraschiv, Andra-Irina
2020-06-30 8:05 ` Greg KH
2020-06-30 9:08 ` Paraschiv, Andra-Irina
2020-07-06 7:13 ` Alexander Graf
2020-07-06 7:49 ` Paraschiv, Andra-Irina
2020-07-06 8:01 ` Alexander Graf
2020-07-06 13:09 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 08/18] nitro_enclaves: Add logic for enclave vm creation Andra Paraschiv
2020-07-06 7:53 ` Alexander Graf
2020-07-06 13:12 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 09/18] nitro_enclaves: Add logic for enclave vcpu creation Andra Paraschiv
2020-07-06 10:12 ` Alexander Graf
2020-07-08 12:46 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 10/18] nitro_enclaves: Add logic for enclave image load info Andra Paraschiv
2020-07-06 10:16 ` Alexander Graf
2020-07-06 13:35 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 11/18] nitro_enclaves: Add logic for enclave memory region set Andra Paraschiv
2020-07-06 10:46 ` Alexander Graf
2020-07-09 7:36 ` Paraschiv, Andra-Irina
2020-07-09 8:40 ` Alexander Graf
2020-07-09 9:41 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 12/18] nitro_enclaves: Add logic for enclave start Andra Paraschiv
2020-07-06 11:21 ` Alexander Graf
2020-07-07 18:27 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 13/18] nitro_enclaves: Add logic for enclave termination Andra Paraschiv
2020-07-06 11:26 ` Alexander Graf
2020-07-06 14:15 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 14/18] nitro_enclaves: Add Kconfig for the Nitro Enclaves driver Andra Paraschiv
2020-07-06 11:28 ` Alexander Graf
2020-07-06 13:50 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 15/18] nitro_enclaves: Add Makefile " Andra Paraschiv
2020-07-06 11:30 ` Alexander Graf
2020-07-06 14:00 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 16/18] nitro_enclaves: Add sample for ioctl interface usage Andra Paraschiv
2020-07-06 11:39 ` Alexander Graf
2020-07-07 19:03 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 17/18] nitro_enclaves: Add overview documentation Andra Paraschiv
2020-06-23 8:59 ` Stefan Hajnoczi
2020-06-24 14:39 ` Paraschiv, Andra-Irina
2020-06-25 13:10 ` Stefan Hajnoczi
2020-06-25 17:36 ` Paraschiv, Andra-Irina
2020-06-22 20:03 ` [PATCH v4 18/18] MAINTAINERS: Add entry for the Nitro Enclaves driver Andra Paraschiv
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=96667e27-3379-4817-843f-988af9db4366@amazon.com \
--to=andraprs@amazon.com \
--cc=aliguori@amazon.com \
--cc=benh@kernel.crashing.org \
--cc=colmmacc@amazon.com \
--cc=doebel@amazon.de \
--cc=dwmw@amazon.co.uk \
--cc=fllinden@amazon.com \
--cc=graf@amazon.de \
--cc=gregkh@linuxfoundation.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mpohlack@amazon.de \
--cc=msw@amazon.com \
--cc=ne-devel-upstream@amazon.com \
--cc=pbonzini@redhat.com \
--cc=sblbir@amazon.com \
--cc=sgarzare@redhat.com \
--cc=stefanha@redhat.com \
--cc=trawets@amazon.com \
--cc=uwed@amazon.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).