From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E98BDC433E0 for ; Thu, 28 Jan 2021 16:54:18 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3596264DED for ; Thu, 28 Jan 2021 16:54:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3596264DED Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=bwidawsk.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:55508 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l5AYj-0000fO-7r for qemu-devel@archiver.kernel.org; Thu, 28 Jan 2021 11:54:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:40478) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5AWc-0007iC-W2 for qemu-devel@nongnu.org; Thu, 28 Jan 2021 11:52:07 -0500 Received: from zangief.bwidawsk.net ([107.170.211.233]:49834 helo=mail.bwidawsk.net) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1l5AWb-0006yd-5c for qemu-devel@nongnu.org; Thu, 28 Jan 2021 11:52:06 -0500 Received: by mail.bwidawsk.net (Postfix, from userid 5001) id 42D0A123395; Thu, 28 Jan 2021 08:52:00 -0800 (PST) Received: from mail.bwidawsk.net (c-73-37-61-164.hsd1.or.comcast.net [73.37.61.164]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by mail.bwidawsk.net (Postfix) with ESMTPSA id 10470122C5B; Thu, 28 Jan 2021 08:51:53 -0800 (PST) Date: Thu, 28 Jan 2021 08:51:51 -0800 From: Ben Widawsky To: Jonathan Cameron Subject: Re: [RFC PATCH v2 24/32] hw/cxl/device: Add a memory device (8.2.8.5) Message-ID: <20210128165151.skmzjvferaipavt6@mail.bwidawsk.net> References: <20210105165323.783725-1-ben.widawsky@intel.com> <20210105165323.783725-25-ben.widawsky@intel.com> <20210127220312.6850abe2@redhat.com> <20210127212645.ztuxaopjnwtm7u2i@intel.com> <20210128102538.000066ff@Huawei.com> <20210128150318.ujl2uzyzbygqnlg2@mail.bwidawsk.net> <20210128151444.2rjkcbhrwflq6zga@mail.bwidawsk.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210128151444.2rjkcbhrwflq6zga@mail.bwidawsk.net> Received-SPF: none client-ip=107.170.211.233; envelope-from=ben@bwidawsk.net; helo=mail.bwidawsk.net X-Spam_score_int: -14 X-Spam_score: -1.5 X-Spam_bar: - X-Spam_report: (-1.5 / 5.0 requ) BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.399, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Huth , "Michael S. Tsirkin" , Vishal Verma , Chris Browy , Philippe =?utf-8?Q?Mathieu-Daud=C3=A9?= , qemu-devel@nongnu.org, Prashant V Agarwal , Igor Mammedov , Dan Williams Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On 21-01-28 07:14:44, Ben Widawsky wrote: > On 21-01-28 07:03:18, Ben Widawsky wrote: > > On 21-01-28 10:25:38, Jonathan Cameron wrote: > > > On Wed, 27 Jan 2021 13:26:45 -0800 > > > Ben Widawsky wrote: > > > > > > > On 21-01-27 22:03:12, Igor Mammedov wrote: > > > > > On Tue, 5 Jan 2021 08:53:15 -0800 > > > > > Ben Widawsky wrote: > > > > > > > > > > > A CXL memory device (AKA Type 3) is a CXL component that contains some > > > > > > combination of volatile and persistent memory. It also implements the > > > > > > previously defined mailbox interface as well as the memory device > > > > > > firmware interface. > > > > > > > > > > > > The following example will create a 256M device in a 512M window: > > > > > > > > > > > > -object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M" > > > > > > -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M" > > > > > > > > > > I'd expect whole backend used by frontend, so one would not need "size" property > > > > > on frontend (like we do with memory devices). > > > > > So question is why it partially uses memdev? > > > > > > > > Answered in a separate thread... > > > > > > One possible suggestion inline. > > > > > > > > > + > > > > > > +static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp) > > > > > > +{ > > > > > > + MemoryRegionSection mrs; > > > > > > + MemoryRegion *mr; > > > > > > + uint64_t offset = 0; > > > > > > + size_t remaining_size; > > > > > > + > > > > > > + if (!ct3d->hostmem) { > > > > > > + error_setg(errp, "memdev property must be set"); > > > > > > + return; > > > > > > + } > > > > > > + > > > > > > + /* FIXME: need to check mr is the host bridge's MR */ > > > > > > + mr = host_memory_backend_get_memory(ct3d->hostmem); > > > > > > + > > > > > > + /* Create our new subregion */ > > > > > > + ct3d->cxl_dstate.pmem = g_new(MemoryRegion, 1); > > > > > > + > > > > > > + /* Find the first free space in the window */ > > > > > > + WITH_RCU_READ_LOCK_GUARD() > > > > > > + { > > > > > > + mrs = memory_region_find(mr, offset, 1); > > > > > > + while (mrs.mr && mrs.mr != mr) { > > > > > > + offset += memory_region_size(mrs.mr); > > > > > > + mrs = memory_region_find(mr, offset, 1); > > > > > > + } > > > > > > + } > > > > > > + > > > > > > + remaining_size = memory_region_size(mr) - offset; > > > > > > + if (remaining_size < ct3d->size) { > > > > > > + g_free(ct3d->cxl_dstate.pmem); > > > > > > + error_setg(errp, > > > > > > + "Not enough free space (%zd) required for device (%" PRId64 ")", > > > > > > + remaining_size, ct3d->size); > > > > > > + } > > > > > > + > > > > > > + /* Register our subregion as non-volatile */ > > > > > > + memory_region_init_ram(ct3d->cxl_dstate.pmem, OBJECT(ct3d), > > > > > > + "cxl_type3-memory", ct3d->size, errp); > > > > > this allocates ct3d->size of anon RAM, was this an intention? > > > > > If yes, can you clarify why extra RAM is used instead of using what > > > > > backend provides? > > > > > > > > It sounds like I'm doing the wrong thing then. There should be one chunk of > > > > memory which is a subset of the full memory backend object. Could you please > > > > advise on what I should be doing instead? Is add_subregion() sufficient? > > > > > > Taking inspiration from nvdimm I'm carrying a patch that uses > > > memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(qct3d)q, > > > "cxl_type3-memory", mr, offset, ct3d->size); > > > > > > I 'think' that's doing the right thing, but haven't fully tested it yet > > > so may be completely wrong :) > > > > > > Then for the pmem addr, call memory_region_set_address() to put it > > > in a particular location. > > > > > > > Yes - this is what I'd like to do and what I initially tried, and I also believe > > it's right, but it doesn't work. > > > > range_invariant: Assertion `range->lob <= range->upb || range->lob == range->upb + 1' failed. > > > > I was digging into this yesterday, but opted to start a new thread on the > > matter. > > > > Hmm. I think I need to figure out the right add_subregion after this and it > might work. I'll keep digging, but if you have ideas, let me know. [snip] I managed to get a bit further. With the following, I start getting complaints about fragmented memory when adding devices later. memory_region_init_alias(ct3d->cxl_dstate.pmem, OBJECT(ct3d), "cxl_type3-memory", mr, mr->addr + offset, ct3d->size); memory_region_set_nonvolatile(ct3d->cxl_dstate.pmem, true); memory_region_add_subregion(mr, offset, ct3d->cxl_dstate.pmem); -device nvdimm,memdev=nvmem1,id=nv1,label-size=2M,node=5: could not find position in guest address space for memory device - memory fragmented due to alignments