From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from ozlabs.org (ozlabs.org [IPv6:2401:3900:2:1::2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 88FDE1A0074 for ; Thu, 4 Sep 2014 18:47:20 +1000 (EST) Received: from smtp-outbound-2.vmware.com (smtp-outbound-2.vmware.com [208.91.2.13]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 1183714017E for ; Thu, 4 Sep 2014 18:47:19 +1000 (EST) Message-ID: <540826F6.9060505@vmware.com> Date: Thu, 4 Sep 2014 10:46:46 +0200 From: Thomas Hellstrom MIME-Version: 1.0 To: Benjamin Herrenschmidt Subject: Re: TTM placement & caching issue/questions References: <1409789547.30640.136.camel@pasglop> <54081844.7000604@vmware.com> <1409817962.4246.51.camel@pasglop> In-Reply-To: <1409817962.4246.51.camel@pasglop> Content-Type: text/plain; charset="ISO-8859-1" Cc: Jerome Glisse , linuxppc-dev@ozlabs.org, Michel Danzer , dri-devel@lists.freedesktop.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 09/04/2014 10:06 AM, Benjamin Herrenschmidt wrote: > On Thu, 2014-09-04 at 09:44 +0200, Thomas Hellstrom wrote: > >>> This will, from what I can tell, try to use the same caching mode as the >>> original object: >>> >>> if ((cur_placement & caching) != 0) >>> result |= (cur_placement & caching); >>> >>> And cur_placement comes from bo->mem.placement which as far as I can >>> tell is based on the placement array which the drivers set up. >> This originates from the fact that when evicting GTT memory, on x86 it's >> unnecessary and undesirable to switch caching mode when going to system. > But that's what I don't quite understand. We have two different mappings > here. The VRAM and the memory object. We wouldn't be "switching"... we > are creating a temporary mapping for the memory object in order to do > the memcpy, but we seem to be doing it by using the caching attributes > of the VRAM object.... or am I missing something ? I don't see how that > makes sense so I suppose I'm missing something here :-) Well, the intention when TTM was written was that the driver writer should be smart enough that when he wanted a move from unached VRAM to system, he'd request cached system in the placement flags in the first place. If TTM somehow overrides such a request, that's a bug in TTM. If the move, for example, is a result of an eviction, then the driver evict_flags() function should ideally look at the current placement and decide about a suitable placement based on that: vram-to-system moves should generally request cacheable memory if the next access is expected by the CPU. Probably write-combined otherwise. If the move is the result of a TTM swapout, TTM will automatically select cachable system, and for most other moves, I think the driver writer is in full control. > >> Last time I tested, (and it seems like Michel is on the same track), >> writing with the CPU to write-combined memory was substantially faster >> than writing to cached memory, with the additional side-effect that CPU >> caches are left unpolluted. > That's very strange indeed. It's certainly an x86 specific artifact, > even if we were allowed by our hypervisor to map memory non-cachable > (the HW somewhat can), we tend to have a higher throughput by going > cachable, but that could be due to the way the PowerBus works (it's > basically very biased toward cachable transactions). > >> I dislike the approach of rewriting placements. In some cases I think it >> won't even work, because placements are declared 'static const' >> >> What I'd suggest is instead to intercept the driver response from >> init_mem_type() and filter out undesired caching modes from >> available_caching and default_caching, > This was my original intent but Jerome seems to have different ideas > (see his proposed patches). I'm happy to revive mine as well and post it > as an alternative after I've tested it a bit more (tomorrow). > >> perhaps also looking at whether >> the memory type is mappable or not. This should have the additional >> benefit of working everywhere, and if a caching mode is selected that's >> not available on the platform, you'll simply get an error. (I guess?) > You mean that if not mappable we don't bother filtering ? > > The rule is really for me pretty simple: > > - If it's system memory (PL_SYSTEM/PL_TT), it MUST be cachable > > - If it's PCIe memory space (VRAM, registers, ...) it MUST be > non-cachable. Yes, something along these lines. I guess checking for VRAM or TTM_MEMTYPE_FLAG_FIXED would perhaps do the trick /Thomas > > Cheers, > Ben. > >> /Thomas >> >> >>> Cheers, >>> Ben. >>> >>> >>> _______________________________________________ >>> dri-devel mailing list >>> dri-devel@lists.freedesktop.org >>> https://urldefense.proofpoint.com/v1/url?u=http://lists.freedesktop.org/mailman/listinfo/dri-devel&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=l5Ago9ekmVFZ3c4M6eauqrJWGwjf6fTb%2BP3CxbBFkVM%3D%0A&m=C9AHL1VngKBOxe2UrNP2eCZo6FLqdlr6Y90rpfE5rUs%3D%0A&s=73da0633bafc5d54bf116bc861d48d13c39cf8f41832adfb739709e98ec05553 > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Hellstrom Subject: Re: TTM placement & caching issue/questions Date: Thu, 4 Sep 2014 10:46:46 +0200 Message-ID: <540826F6.9060505@vmware.com> References: <1409789547.30640.136.camel@pasglop> <54081844.7000604@vmware.com> <1409817962.4246.51.camel@pasglop> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <1409817962.4246.51.camel@pasglop> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+glppe-linuxppc-embedded-2=m.gmane.org@lists.ozlabs.org Sender: "Linuxppc-dev" To: Benjamin Herrenschmidt Cc: Jerome Glisse , linuxppc-dev@ozlabs.org, Michel Danzer , dri-devel@lists.freedesktop.org List-Id: dri-devel@lists.freedesktop.org T24gMDkvMDQvMjAxNCAxMDowNiBBTSwgQmVuamFtaW4gSGVycmVuc2NobWlkdCB3cm90ZToKPiBP biBUaHUsIDIwMTQtMDktMDQgYXQgMDk6NDQgKzAyMDAsIFRob21hcyBIZWxsc3Ryb20gd3JvdGU6 Cj4KPj4+IFRoaXMgd2lsbCwgZnJvbSB3aGF0IEkgY2FuIHRlbGwsIHRyeSB0byB1c2UgdGhlIHNh bWUgY2FjaGluZyBtb2RlIGFzIHRoZQo+Pj4gb3JpZ2luYWwgb2JqZWN0Ogo+Pj4KPj4+IAlpZiAo KGN1cl9wbGFjZW1lbnQgJiBjYWNoaW5nKSAhPSAwKQo+Pj4gCQlyZXN1bHQgfD0gKGN1cl9wbGFj ZW1lbnQgJiBjYWNoaW5nKTsKPj4+Cj4+PiBBbmQgY3VyX3BsYWNlbWVudCBjb21lcyBmcm9tIGJv LT5tZW0ucGxhY2VtZW50IHdoaWNoIGFzIGZhciBhcyBJIGNhbgo+Pj4gdGVsbCBpcyBiYXNlZCBv biB0aGUgcGxhY2VtZW50IGFycmF5IHdoaWNoIHRoZSBkcml2ZXJzIHNldCB1cC4KPj4gVGhpcyBv cmlnaW5hdGVzIGZyb20gdGhlIGZhY3QgdGhhdCB3aGVuIGV2aWN0aW5nIEdUVCBtZW1vcnksIG9u IHg4NiBpdCdzCj4+IHVubmVjZXNzYXJ5IGFuZCB1bmRlc2lyYWJsZSB0byBzd2l0Y2ggY2FjaGlu ZyBtb2RlIHdoZW4gZ29pbmcgdG8gc3lzdGVtLgo+IEJ1dCB0aGF0J3Mgd2hhdCBJIGRvbid0IHF1 aXRlIHVuZGVyc3RhbmQuIFdlIGhhdmUgdHdvIGRpZmZlcmVudCBtYXBwaW5ncwo+IGhlcmUuIFRo ZSBWUkFNIGFuZCB0aGUgbWVtb3J5IG9iamVjdC4gV2Ugd291bGRuJ3QgYmUgInN3aXRjaGluZyIu Li4gd2UKPiBhcmUgY3JlYXRpbmcgYSB0ZW1wb3JhcnkgbWFwcGluZyBmb3IgdGhlIG1lbW9yeSBv YmplY3QgaW4gb3JkZXIgdG8gZG8KPiB0aGUgbWVtY3B5LCBidXQgd2Ugc2VlbSB0byBiZSBkb2lu ZyBpdCBieSB1c2luZyB0aGUgY2FjaGluZyBhdHRyaWJ1dGVzCj4gb2YgdGhlIFZSQU0gb2JqZWN0 Li4uLiBvciBhbSBJIG1pc3Npbmcgc29tZXRoaW5nID8gSSBkb24ndCBzZWUgaG93IHRoYXQKPiBt YWtlcyBzZW5zZSBzbyBJIHN1cHBvc2UgSSdtIG1pc3Npbmcgc29tZXRoaW5nIGhlcmUgOi0pCgpX ZWxsLCB0aGUgaW50ZW50aW9uIHdoZW4gVFRNIHdhcyB3cml0dGVuIHdhcyB0aGF0IHRoZSBkcml2 ZXIgd3JpdGVyCnNob3VsZCBiZSBzbWFydCBlbm91Z2ggdGhhdCB3aGVuIGhlIHdhbnRlZCBhIG1v dmUgZnJvbSB1bmFjaGVkIFZSQU0gdG8Kc3lzdGVtLCBoZSdkIHJlcXVlc3QgY2FjaGVkIHN5c3Rl bSBpbiB0aGUgcGxhY2VtZW50IGZsYWdzIGluIHRoZSBmaXJzdApwbGFjZS4gIElmIFRUTSBzb21l aG93IG92ZXJyaWRlcyBzdWNoIGEgcmVxdWVzdCwgdGhhdCdzIGEgYnVnIGluIFRUTS4KCklmIHRo ZSBtb3ZlLCBmb3IgZXhhbXBsZSwgaXMgYSByZXN1bHQgb2YgYW4gZXZpY3Rpb24sIHRoZW4gdGhl IGRyaXZlcgpldmljdF9mbGFncygpIGZ1bmN0aW9uIHNob3VsZCBpZGVhbGx5IGxvb2sgYXQgdGhl IGN1cnJlbnQgcGxhY2VtZW50IGFuZApkZWNpZGUgYWJvdXQgYSBzdWl0YWJsZSBwbGFjZW1lbnQg YmFzZWQgb24gdGhhdDogdnJhbS10by1zeXN0ZW0gbW92ZXMKc2hvdWxkIGdlbmVyYWxseSByZXF1 ZXN0IGNhY2hlYWJsZSBtZW1vcnkgaWYgdGhlIG5leHQgYWNjZXNzIGlzIGV4cGVjdGVkCmJ5IHRo ZSBDUFUuIFByb2JhYmx5IHdyaXRlLWNvbWJpbmVkIG90aGVyd2lzZS4KSWYgdGhlIG1vdmUgaXMg dGhlIHJlc3VsdCBvZiBhIFRUTSBzd2Fwb3V0LCBUVE0gd2lsbCBhdXRvbWF0aWNhbGx5CnNlbGVj dCBjYWNoYWJsZSBzeXN0ZW0sIGFuZCBmb3IgbW9zdCBvdGhlciBtb3ZlcywgSSB0aGluayB0aGUg ZHJpdmVyCndyaXRlciBpcyBpbiBmdWxsIGNvbnRyb2wuCgo+Cj4+IExhc3QgdGltZSBJIHRlc3Rl ZCwgKGFuZCBpdCBzZWVtcyBsaWtlIE1pY2hlbCBpcyBvbiB0aGUgc2FtZSB0cmFjayksCj4+IHdy aXRpbmcgd2l0aCB0aGUgQ1BVIHRvIHdyaXRlLWNvbWJpbmVkIG1lbW9yeSB3YXMgc3Vic3RhbnRp YWxseSBmYXN0ZXIKPj4gdGhhbiB3cml0aW5nIHRvIGNhY2hlZCBtZW1vcnksIHdpdGggdGhlIGFk ZGl0aW9uYWwgc2lkZS1lZmZlY3QgdGhhdCBDUFUKPj4gY2FjaGVzIGFyZSBsZWZ0IHVucG9sbHV0 ZWQuCj4gVGhhdCdzIHZlcnkgc3RyYW5nZSBpbmRlZWQuIEl0J3MgY2VydGFpbmx5IGFuIHg4NiBz cGVjaWZpYyBhcnRpZmFjdCwKPiBldmVuIGlmIHdlIHdlcmUgYWxsb3dlZCBieSBvdXIgaHlwZXJ2 aXNvciB0byBtYXAgbWVtb3J5IG5vbi1jYWNoYWJsZQo+ICh0aGUgSFcgc29tZXdoYXQgY2FuKSwg d2UgdGVuZCB0byBoYXZlIGEgaGlnaGVyIHRocm91Z2hwdXQgYnkgZ29pbmcKPiBjYWNoYWJsZSwg YnV0IHRoYXQgY291bGQgYmUgZHVlIHRvIHRoZSB3YXkgdGhlIFBvd2VyQnVzIHdvcmtzIChpdCdz Cj4gYmFzaWNhbGx5IHZlcnkgYmlhc2VkIHRvd2FyZCBjYWNoYWJsZSB0cmFuc2FjdGlvbnMpLgo+ Cj4+IEkgZGlzbGlrZSB0aGUgYXBwcm9hY2ggb2YgcmV3cml0aW5nIHBsYWNlbWVudHMuIEluIHNv bWUgY2FzZXMgSSB0aGluayBpdAo+PiB3b24ndCBldmVuIHdvcmssIGJlY2F1c2UgcGxhY2VtZW50 cyBhcmUgZGVjbGFyZWQgJ3N0YXRpYyBjb25zdCcKPj4KPj4gV2hhdCBJJ2Qgc3VnZ2VzdCBpcyBp bnN0ZWFkIHRvIGludGVyY2VwdCB0aGUgZHJpdmVyIHJlc3BvbnNlIGZyb20KPj4gaW5pdF9tZW1f dHlwZSgpIGFuZCBmaWx0ZXIgb3V0IHVuZGVzaXJlZCBjYWNoaW5nIG1vZGVzIGZyb20KPj4gYXZh aWxhYmxlX2NhY2hpbmcgYW5kIGRlZmF1bHRfY2FjaGluZywgCj4gVGhpcyB3YXMgbXkgb3JpZ2lu YWwgaW50ZW50IGJ1dCBKZXJvbWUgc2VlbXMgdG8gaGF2ZSBkaWZmZXJlbnQgaWRlYXMKPiAoc2Vl IGhpcyBwcm9wb3NlZCBwYXRjaGVzKS4gSSdtIGhhcHB5IHRvIHJldml2ZSBtaW5lIGFzIHdlbGwg YW5kIHBvc3QgaXQKPiBhcyBhbiBhbHRlcm5hdGl2ZSBhZnRlciBJJ3ZlIHRlc3RlZCBpdCBhIGJp dCBtb3JlICh0b21vcnJvdykuCj4KPj4gcGVyaGFwcyBhbHNvIGxvb2tpbmcgYXQgd2hldGhlcgo+ PiB0aGUgbWVtb3J5IHR5cGUgaXMgbWFwcGFibGUgb3Igbm90LiBUaGlzIHNob3VsZCBoYXZlIHRo ZSBhZGRpdGlvbmFsCj4+IGJlbmVmaXQgb2Ygd29ya2luZyBldmVyeXdoZXJlLCBhbmQgaWYgYSBj YWNoaW5nIG1vZGUgaXMgc2VsZWN0ZWQgdGhhdCdzCj4+IG5vdCBhdmFpbGFibGUgb24gdGhlIHBs YXRmb3JtLCB5b3UnbGwgc2ltcGx5IGdldCBhbiBlcnJvci4gKEkgZ3Vlc3M/KQo+IFlvdSBtZWFu IHRoYXQgaWYgbm90IG1hcHBhYmxlIHdlIGRvbid0IGJvdGhlciBmaWx0ZXJpbmcgPwo+Cj4gVGhl IHJ1bGUgaXMgcmVhbGx5IGZvciBtZSBwcmV0dHkgc2ltcGxlOgo+Cj4gICAgLSBJZiBpdCdzIHN5 c3RlbSBtZW1vcnkgKFBMX1NZU1RFTS9QTF9UVCksIGl0IE1VU1QgYmUgY2FjaGFibGUKPgo+ICAg IC0gSWYgaXQncyBQQ0llIG1lbW9yeSBzcGFjZSAoVlJBTSwgcmVnaXN0ZXJzLCAuLi4pIGl0IE1V U1QgYmUKPiBub24tY2FjaGFibGUuCgpZZXMsIHNvbWV0aGluZyBhbG9uZyB0aGVzZSBsaW5lcy4g SSBndWVzcyBjaGVja2luZyBmb3IgVlJBTSBvcgpUVE1fTUVNVFlQRV9GTEFHX0ZJWEVEIHdvdWxk IHBlcmhhcHMgZG8gdGhlIHRyaWNrCgovVGhvbWFzCgo+Cj4gQ2hlZXJzLAo+IEJlbi4KPgo+PiAv VGhvbWFzCj4+Cj4+Cj4+PiBDaGVlcnMsCj4+PiBCZW4uCj4+Pgo+Pj4KPj4+IF9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4+PiBkcmktZGV2ZWwgbWFpbGlu ZyBsaXN0Cj4+PiBkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCj4+PiBodHRwczovL3Vy bGRlZmVuc2UucHJvb2Zwb2ludC5jb20vdjEvdXJsP3U9aHR0cDovL2xpc3RzLmZyZWVkZXNrdG9w Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RyaS1kZXZlbCZrPW9JdlJnMSUyQmRHQWdPb00xQklsTExx dyUzRCUzRCUwQSZyPWw1QWdvOWVrbVZGWjNjNE02ZWF1cXJKV0d3amY2ZlRiJTJCUDNDeGJCRmtW TSUzRCUwQSZtPUM5QUhMMVZuZ0tCT3hlMlVyTlAyZUNabzZGTHFkbHI2WTkwcnBmRTVyVXMlM0Ql MEEmcz03M2RhMDYzM2JhZmM1ZDU0YmYxMTZiYzg2MWQ0OGQxM2MzOWNmOGY0MTgzMmFkZmI3Mzk3 MDllOThlYzA1NTUzCj4KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCkxpbnV4cHBjLWRldiBtYWlsaW5nIGxpc3QKTGludXhwcGMtZGV2QGxpc3RzLm96bGFi cy5vcmcKaHR0cHM6Ly9saXN0cy5vemxhYnMub3JnL2xpc3RpbmZvL2xpbnV4cHBjLWRldg==