From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ian Jackson Subject: Re: libxl and malloc failure (Re: Current LibXL Status) Date: Mon, 22 Feb 2016 16:48:50 +0000 Message-ID: <22219.15346.138200.277441@mariner.uk.xensource.com> References: <564CC43B.1000904@ainfosec.com> <1447924858.5647.15.camel@citrix.com> <564DAA8D.5060305@citrix.com> <22214.2619.86697.724156@mariner.uk.xensource.com> <1455879131.6225.89.camel@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <1455879131.6225.89.camel@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Ian Campbell Cc: Andrew Cooper , Martin Osterloh , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org Ian Campbell writes ("Re: libxl and malloc failure (Re: [Xen-devel] Current= LibXL Status)"): > On Thu, 2016-02-18 at 18:15 +0000, Ian Jackson wrote: > > Andrew Cooper writes ("Re: [Xen-devel] Current LibXL Status"): > > > Languages such as OCaml use -ENOMEM as a hint to run the garbage > > > collector some more.=A0=A0I expect Haskell is the same. > > = > > This cannot possibly be true (if what you mean is that they use > > ENOMEM[1] from malloc as such an indication).=A0=A0It would make it > > impossible to write a correct binding to a normal C library. > > = > > Typically C library which calls malloc will do so in the middle of its > > execution.=A0=A0Even if the library returns the resulting error up as > > a distinguishable error code, you can't just make the same library > > call again - it may have done half its work but not the other half. > = > FWIW there is some code in the Ocaml runtime which raises an Out_of_memory > exception in response to ENOMEM in C land (and from ocaml code in some > circumstances too). ISTM that either there is something else which is turning ocaml out of heap errors into ENOMEM, or that this code you describe is fundamentally misconceived. > In principal it ought to be possible to write an ocaml program which > correctly dealt with those (and similar ones from other lower level > bindings), for example the interactive command line interpreter catches > Out_of_memory (but not Unix_error) and calls the gc before iterating. (I > did see one reference in a book which suggested Out_of_memory wasn't > catcheable, but that appears to be out of date from what I can tell). But iterating in the cli would not recover the operation which had failed. In the general case, there is no recovery possible when a C language library function returns ENOMEM. Sometimes the internal state of the library will now be unsuitable. Certainly, a policy of always retrying the operation after a gc would be very wrong indeed. Ie if ocaml has the behaviour claimed by Andrew (ie, that the need for a gc can be detected during malloc and reported as ENOMEM), all ocaml programs would already suffer from occasional random failures of operations being performed by code written in C. Note that "need for a gc" is not an abnormal part of operation in a consing garbage collected language like ocaml. It is entirely routine. Ie these random failures would occur in normal, otherwise entirely successful, operation. They would occur even if the program's actual memory use was modest. > This of course relies on all the C bindings correctly turning errors into > exceptions and the semantics of your particular daemon allowing for > abandoning things half way through (which I suppose is a relative of some > kind to crash-only s/w). Even if the operations are properly abandonded, they have still been erroneously made to fail. Ian.