From: Marcelo Tosatti <email@example.com> To: Christoph Lameter <firstname.lastname@example.org> Cc: Alex Belits <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, Juri Lelli <firstname.lastname@example.org>, Daniel Bristot de Oliveira <email@example.com> Subject: Re: [RFC] tentative prctl task isolation interface Date: Mon, 1 Feb 2021 15:20:17 -0300 [thread overview] Message-ID: <20210201182017.GA29345@fuller.cnet> (raw) In-Reply-To: <alpine.DEB.firstname.lastname@example.org> On Mon, Feb 01, 2021 at 10:48:18AM +0000, Christoph Lameter wrote: > On Thu, 21 Jan 2021, Marcelo Tosatti wrote: > > > Anyway, trying to improve Christoph's definition: > > > > F_ISOL_QUIESCE -> flush any pending operations that might cause > > the CPU to be interrupted (ex: free's > > per-CPU queues, sync MM statistics > > counters, etc). > > > > F_ISOL_ISOLATE -> inform the kernel that userspace is > > entering isolated mode (see description > > below on "ISOLATION MODES"). > > > > F_ISOL_UNISOLATE -> inform the kernel that userspace is > > leaving isolated mode. > > > > F_ISOL_NOTIFY -> notification mode of isolation breakage > > modes. > > Looks good to me. > > > > Isolation modes: > > --------------- > > > > There are two main types of isolation modes: > > > > - SOFT mode: does not prevent activities which might generate interruptions > > (such as CPU hotplug). > > > > - HARD mode: prevents all blockable activities that might generate interruptions. > > Administrators can override this via /sys. > > > Yup. > > > > > Notifications: > > ------------- > > > > Notification mode of isolation breakage can be configured as follows: > > > > - None (default): No notification is performed by the kernel on isolation > > breakage. > > > > - Syslog: Isolation breakage is reported to syslog. > > > - Abort with core dump > > This is useful for debugging and for hard core bare metalers that never > want any interrupts. > > One particular issue are page faults. One would have to prefault the > binary executable functions in order to avoid "interruptions" through page > faults. Are these proper interrutions of the code? Certainly major faults > are but minor faults may be ok? Dunno. mlockall man page: Real-time processes that are using mlockall() to prevent delays on page faults should reserve enough locked stack pages before entering the time-critical section, so that no page fault can be caused by function calls. This can be achieved by calling a function that allocates a sufficiently large automatic variable (an array) and writes to the memory occupied by this array in order to touch these stack pages. This way, enough pages will be mapped for the stack and can be locked into RAM. The dummy writes ensure that not even copy-on-write page faults can occur in the critical section. > In practice what I have often seen in such apps is that there is a "warm" > up mode where all critical functions are executed, all important variables > are touched and dummy I/Os are performed in order to populate the caches > and prefault all the data.I guess one would run these without isolation > first and then switch on some sort of isolation mode after warm up. So far > I think most people relied on the timer interrupt etc etc to be turned off > after a few secs of just running throught a polling loop without any OS > activities. Yep. > > > I ended up implementing a manager/helper task that talks to tasks over a > > > socket (when they are not isolated) and over ring buffers in shared memory > > > (when they are isolated). While the current implementation is rather > > > limited, the intention is to delegate to it everything that isolated task > > > either can't do at all (like, writing logs) or that it would be cumbersome > > > to implement (like monitoring the state of task, determining presence of > > > deferred work after the task returned to userspace), etc. > > > > Interesting. Are you considering opensourcing such library? Seems like a > > generic problem. > > Well everyone swears on having the right implementation. The people I know > would not do any thing with a socket in such situations. They would only > use shared memory and direct access to I/O devices via SPDK and DPDK or > the RDMA subsystem. > > > > > > Blocking? The app should fail if any deferred actions are triggered as a > > > > result of syscalls. It would give a warning with _WARN > > > > > > There are many supposedly innocent things, nowhere at the scale of CPU > > > hotplug, that happen in a system and result in synchronization implemented > > > as an IPI to every online CPU. We should consider them to be an ordinary > > > occurrence, so there is a choice: > > > > > > 1. Ignore them completely and allow them in isolated mode. This will delay > > > userspace with no indication and no isolation breaking. > > > > > > 2. Allow them, and notify userspace afterwards (through vdso or through > > > userspace helper/manager over shared memory). This may be useful in those > > > rare situations when the consequences of delay can be mitigated afterwards. > > > > > > 3. Make them break isolation, with userspace being notified normally (ex: > > > with a signal in the current implementation). I guess, can be used if > > > somehow most of the causes will be eliminated. > > > > > > 4. Prevent them from reaching the target CPU and make sure that whatever > > > synchronization they are intended to cause, will happen when intended target > > > CPU will enter to kernel later. Since we may have to synchronize things like > > > code modification, some of this synchronization has to happen very early on > > > kernel entry. > > > Or move the actions to a different victim processor like done with rcu and > vmstat etc etc. > > > > > > > I am most interested in (4), so this is what was implemented in my version > > > of the patch (and currently I am trying to achieve completeness and, if > > > possible, elegance of the implementation). > > > > Agree. (3) will be necessary as intermediate step. The proposed > > improvement to Christoph's reply, in this thread, separates notification > > and syscall blockage. > > I guess the notification mode will take care of the way we handle these > interruptions.
next prev parent reply other threads:[~2021-02-01 18:21 UTC|newest] Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-11-17 16:28 [PATCH] mm: introduce sysctl file to flush per-cpu vmstat statistics Marcelo Tosatti 2020-11-17 18:03 ` Matthew Wilcox 2020-11-17 19:06 ` Christopher Lameter 2020-11-17 19:09 ` Matthew Wilcox 2020-11-20 18:04 ` Christopher Lameter 2020-11-17 20:23 ` Marcelo Tosatti 2020-11-20 18:02 ` Marcelo Tosatti 2020-11-20 18:20 ` Christopher Lameter 2020-11-23 18:02 ` Marcelo Tosatti 2020-11-24 17:12 ` Vlastimil Babka 2020-11-24 19:52 ` Marcelo Tosatti 2020-11-27 15:48 ` Marcelo Tosatti 2020-11-28 3:49 ` [EXT] " Alex Belits 2020-11-30 18:18 ` Marcelo Tosatti 2020-11-30 18:29 ` Marcelo Tosatti 2020-12-03 22:47 ` Alex Belits 2020-12-03 22:21 ` Alex Belits 2020-11-30 9:31 ` Christoph Lameter 2020-12-02 12:43 ` Marcelo Tosatti 2020-12-02 15:57 ` Thomas Gleixner 2020-12-02 17:43 ` Christoph Lameter 2020-12-03 3:17 ` Thomas Gleixner 2020-12-07 8:08 ` Christoph Lameter 2020-12-07 16:09 ` Thomas Gleixner 2020-12-07 19:01 ` Thomas Gleixner 2020-12-02 18:38 ` Marcelo Tosatti 2020-12-04 0:20 ` Frederic Weisbecker 2020-12-04 13:31 ` Marcelo Tosatti 2020-12-04 1:43 ` [EXT] " Alex Belits 2021-01-13 12:15 ` [RFC] tentative prctl task isolation interface Marcelo Tosatti 2021-01-14 9:22 ` Christoph Lameter 2021-01-14 19:34 ` Marcelo Tosatti 2021-01-15 13:24 ` Christoph Lameter 2021-01-15 18:35 ` Alex Belits 2021-01-21 15:51 ` Marcelo Tosatti 2021-01-21 16:20 ` Marcelo Tosatti 2021-01-22 13:05 ` Marcelo Tosatti 2021-02-01 10:48 ` Christoph Lameter 2021-02-01 12:47 ` Alex Belits 2021-02-01 18:20 ` Marcelo Tosatti [this message] 2021-01-18 15:18 ` Marcelo Tosatti 2020-11-24 5:02 ` [mm] e655d17ffa: BUG:using_smp_processor_id()in_preemptible kernel test robot
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210201182017.GA29345@fuller.cnet \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --subject='Re: [RFC] tentative prctl task isolation interface' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions on how to clone and mirror all data and code used for this inbox