Re: Unable to load large enclave

From: Jethro Beekman <jethro@fortanix.com>
To: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>,
	Sean Christopherson <sean.j.christopherson@intel.com>
Cc: "linux-sgx@vger.kernel.org" <linux-sgx@vger.kernel.org>
Subject: Re: Unable to load large enclave
Date: Wed, 7 Oct 2020 18:13:49 +0200	[thread overview]
Message-ID: <f362318e-2d14-c82d-82a8-35c35842f84c@fortanix.com> (raw)
In-Reply-To: <20201007154938.GA19072@linux.intel.com>

[-- Attachment #1: Type: text/plain, Size: 2753 bytes --]

On 2020-10-07 17:49, Jarkko Sakkinen wrote:
> On Tue, Oct 06, 2020 at 06:13:28PM +0300, Jarkko Sakkinen wrote:
>> On Mon, Oct 05, 2020 at 03:56:52PM -0700, Sean Christopherson wrote:
>>> On Wed, Sep 30, 2020 at 02:45:54PM +0300, Jarkko Sakkinen wrote:
>>>> On Wed, Sep 30, 2020 at 09:12:06AM +0200, Jethro Beekman wrote:
>>>>> On 2020-09-30 03:16, Jarkko Sakkinen wrote:
>>>>>> On Tue, Sep 29, 2020 at 05:52:48PM +0200, Jethro Beekman wrote:
>>>>>>> Since the latest API changes, I'm unable to load a large enclave. The
>>>>>>> test program at
>>>>>>> https://github.com/fortanix/rust-sgx/blob/sgx-load-large-enclave-test/src/main.rs
>>>>>>> always fails with ENOMEM after loading 0xffd6 pages.
>>>>>>>
>>>>>>> I've tested this with v36, if there's reason to believe it has been
>>>>>>> fixed I'd be happy to try it out on a newer patch set.
>>>>>>
>>>>>> I recommend using v39-rc1 tag that I created for testing because API is
>>>>>> reverted back to be compatible with v36.
>>>>>
>>>>> Not sure what you're saying. I tested with v36. You're saying v39-rc1
>>>>> will be the same? Or did you fix the issue since v36?
>>>>
>>>> v37 and v38 has an API change that is reverted in v39:
>>>>
>>>> https://lore.kernel.org/linux-sgx/20200921195822.GA58176@linux.intel.com/
>>>>
>>>> I'm not sure of the root cause yet but you asked to try to out a newer
>>>> patch set and v39-rc1 is the best option.
>>>>
>>>> There was off-by-one error in enclave maximum size calculation fixed in
>>>> v37 (it was actually a bug in SDM inherited to the code) but that should
>>>> not result the situation you just described.
>>>
>>> My money is on the XArray changes, that's the most notable change in v36 and
>>> IIRC the only thing that touched EPC/memory management.
>>
>> Yeah, that's what we've been speculating for some days now. That's
>> somewhat deprecated email. It all started to enroll when I asked
>> Haitao to turn CONFIG_PROVE_LOCKING on, and we got the information
>> required to root cause the bug.
> 
> I run the failing test and filtered SGX mmap's and ioctl's with this
> eBPF script:
> 
> kretprobe:sgx_ioctl /retval != 0/
> {
>         printf("sgx_ioctl: %d\n", retval)
> }
> 
> kretprobe:sgx_mmap /retval != 0/
> {
>         printf("sgx_mmap: %d\n", retval)
> }
> 
> This results zero positives, i.e. empty output, when run with bpftrace.
> 
> I'd go instead after RLIMIT_AS [*].
> 
> With these conclusions, I'm done with this bug.
> 

How can it be RLIMIT_AS? With the current flow, you mmap the whole range before mmaping the individual pages over it?

Also, I can easily load a 1GB enclave with the old driver.

Also:

$ ulimit -v
unlimited

--
Jethro Beekman | Fortanix

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4490 bytes --]