Re: userfaultfd: usability issue due to lack of UFFD events ordering

[David Hildenbrand <david@redhat.com>]

On 31.01.22 15:28, Mike Rapoport wrote:
> On Mon, Jan 31, 2022 at 03:12:36PM +0100, David Hildenbrand wrote:
>> On 31.01.22 15:05, Mike Rapoport wrote:
>>> On Mon, Jan 31, 2022 at 11:48:27AM +0100, David Hildenbrand wrote:
>>>> On 31.01.22 11:42, Mike Rapoport wrote:
>>>>> Hi Nadav,
>>>>>
>>>>> On Sat, Jan 29, 2022 at 10:23:55PM -0800, Nadav Amit wrote:
>>>>>> Using userfautlfd and looking at the kernel code, I encountered a usability
>>>>>> issue that complicates userspace UFFD-monitor implementation. I obviosuly
>>>>>> might be wrong, so I would appreciate a (polite?) feedback. I do have a
>>>>>> userspace workaround, but I thought it is worthy to share and to hear your
>>>>>> opinion, as well as feedback from other UFFD users.
>>>>>>
>>>>>> The issue I encountered regards the ordering of UFFD events tbat might not
>>>>>> reflect the actual order in which events took place.
>>>>>>
>>>>>> In more detail, UFFD events (e.g., unmap, fork) are not ordered against
>>>>>> themselves [*]. The mm-lock is dropped before notifying the userspace
>>>>>> UFFD-monitor, and therefore there is no guarantee as to whether the order of
>>>>>> the events actually reflects the order in which the events took place.
>>>>>> This can prevent a UFFD-monitor from using the events to track which
>>>>>> ranges are mapped. Specifically, UFFD_EVENT_FORK message and a
>>>>>> UFFD_EVENT_UNMAP message (which reflects unmap in the parent process) can
>>>>>> be reordered, if the events are triggered by two different threads. In
>>>>>> this case the UFFD-monitor cannot figure from the events whether the
>>>>>> child process has the unmapped memory range still mapped (because fork
>>>>>> happened first) or not.
>>>>>
>>>>> Yeah, it seems that something like this is possible:
>>>>>
>>>>>
>>>>> fork()					munmap()
>>>>> 	mmap_write_unlock();
>>>>> 						mmap_write_lock_killable();
>>>>> 						do_things();
>>>>> 						mmap_{read,write}_unlock();
>>>>> 						userfaultfd_unmap_complete();
>>>>> 	dup_userfaultfd_complete();
>>>>>
>>>>
>>>> I was thinking about other possible races, e.g., MADV_DONTNEED/MADV_FREE
>>>> racing with UFFD_EVENT_PAGEFAULT -- where we only hold the mmap_lock in
>>>> read mode. But not sure if they apply.
>>>
>>> The userspace can live with these, at least for uffd missing page faults.
>>> If the monitor will try to resolve a page fault for a removed area, the
>>> errno from UFFDIO_COPY/ZERO can be used to detect such races.
>>
>> I was wondering if the monitor could get confused if he just resolved a
>> page fault via UFFDIO_COPY/ZERO and then receives a REMOVE event.
> 
> And why would it be confused?

My thinking was that the monitor might use REMOVE events to track which
pages are actually populated. If you receive REMOVE after
UFFDIO_COPY/ZERO the monitor would conclude that the page is not
populated, just like if we'd get the MADV_DONTNEED/MADV_REMOVE
immediately after placing a page.

Of course, it heavily depends on the target use case in the monitor or I
might just be wrong.

-- 
Thanks,

David / dhildenb