[LSF/MM TOPIC] User Directed Tiered Memory Management

[LSF/MM TOPIC] User Directed Tiered Memory Management

[Adam Manzanares]

V2l0aCB0aGUgaW50cm9kdWN0aW9uIG9mIGJ5dGUgYWRkcmVzc2FibGUgc3RvcmFnZSBkZXZpY2Vz
IHRoYXQgaGF2ZSBsb3cgDQpsYXRlbmNpZXMsIGl0IGJlY29tZXMgZGlmZmljdWx0IHRvIGRlY2lk
ZSBob3cgdG8gZXhwb3NlIHRoZXNlIGRldmljZXMgdG8gDQp1c2VyIHNwYWNlIGFwcGxpY2F0aW9u
cy4gRG8gd2UgdHJlYXQgdGhlbSBhcyB0cmFkaXRpb25hbCBibG9jayBkZXZpY2VzIA0Kb3IgZXhw
b3NlIHRoZW0gYXMgYSBEQVggY2FwYWJsZSBkZXZpY2U/IEEgdHJhZGl0aW9uYWwgYmxvY2sgZGV2
aWNlIA0KYWxsb3dzIHVzIHRvIHVzZSB0aGUgcGFnZSBjYWNoZSB0byB0YWtlIGFkdmFudGFnZSBv
ZiBsb2NhbGl0eSBpbiBhY2Nlc3MgDQpwYXR0ZXJucywgYnV0IGNvbWVzIGF0IHRoZSBleHBlbnNl
IG9mIGV4dHJhIG1lbW9yeSBjb3BpZXMgdGhhdCBhcmUgDQpleHRyZW1lbHkgY29zdGx5IGZvciBy
YW5kb20gd29ya2xvYWRzLiBBIERBWCBjYXBhYmxlIGRldmljZSBzZWVtcyBncmVhdCANCmZvciB0
aGUgYWZvcmVtZW50aW9uZWQgcmFuZG9tIGFjY2VzcyB3b3JrbG9hZCwgYnV0IHN1ZmZlcnMgb25j
ZSB0aGVyZSBpcyANCnNvbWUgbG9jYWxpdHkgaW4gdGhlIGFjY2VzcyBwYXR0ZXJuLg0KDQpXaGVu
IERBWC1jYXBhYmxlIGRldmljZXMgYXJlIHVzZWQgYXMgc2xvd2VyL2NoZWFwZXIgdm9sYXRpbGUg
bWVtb3J5LCANCnRyZWF0aW5nIHRoZW0gYXMgYSBzbG93ZXIgTlVNQSBub2RlIHdpdGggYW4gYXNz
b2NpYXRlZCBOVU1BIG1pZ3JhdGlvbiANCnBvbGljeSB3b3VsZCBhbGxvdyBmb3IgdGFraW5nIGFk
dmFudGFnZSBvZiBhY2Nlc3MgcGF0dGVybiBsb2NhbGl0eS4gDQpIb3dldmVyIHRoaXMgYXBwcm9h
Y2ggc3VmZmVycyBmcm9tIGEgZmV3IGRyYXdiYWNrcy4gRmlyc3QsIHdoZW4gdGhvc2UgDQpkZXZp
Y2VzIGFyZSBhbHNvIHBlcnNpc3RlbnQsIHRoZSB0aWVyaW5nIGFwcHJvYWNoIHVzZWQgaW4gTlVN
QSBtaWdyYXRpb24gDQptYXkgbm90IGd1YXJhbnRlZSBwZXJzaXN0ZW5jZS4gU2Vjb25kbHksIGZv
ciBkZXZpY2VzIHdpdGggc2lnbmlmaWNhbnRseSANCmhpZ2hlciBsYXRlbmNpZXMgdGhhbiBEUkFN
LCB0aGUgY29zdCBvZiBtb3ZpbmcgY2xlYW4gcGFnZXMgbWF5IGJlIA0Kc2lnbmlmaWNhbnQuIEZp
bmFsbHksIHBhZ2VzIGhhbmRsZWQgdmlhIE5VTUEgbWlncmF0aW9uIGFyZSBhIGNvbW1vbiANCnJl
c291cmNlIHN1YmplY3QgdG8gdGhyYXNoaW5nIGluIGNhc2Ugb2YgbWVtb3J5IHByZXNzdXJlLg0K
DQpJIHdvdWxkIGxpa2UgdG8gZGlzY3VzcyBhbiBhbHRlcm5hdGl2ZSBhcHByb2FjaCB3aGVyZSBt
ZW1vcnkgaW50ZW5zaXZlIA0KYXBwbGljYXRpb25zIG1tYXAgdGhlc2Ugc3RvcmFnZSBkZXZpY2Vz
IGludG8gdGhlaXIgYWRkcmVzcyBzcGFjZS4gVGhlIA0KYXBwbGljYXRpb24gY2FuIHNwZWNpZnkg
aG93IG11Y2ggRFJBTSBjb3VsZCBiZSB1c2VkIGFzIGEgY2FjaGUgYW5kIGhhdmUgDQpzb21lIGlu
Zmx1ZW5jZSBvbiBwcmVmZXRjaGluZyBhbmQgZXZpY3Rpb24gcG9saWNpZXMuIFRoZSBnb2FsIG9m
IHN1Y2ggYW4gDQphcHByb2FjaCB3b3VsZCBiZSB0byBtaW5pbWl6ZSB0aGUgaW1wYWN0IG9mIHRo
ZSBzbGlnaHRseSBzbG93ZXIgbWVtb3J5IA0KY291bGQgcG90ZW50aWFsbHkgaGF2ZSBvbiBhIHN5
c3RlbSB3aGVuIGl0IGlzIHRyZWF0ZWQgYXMga2VybmVsIG1hbmFnZWQgDQpnbG9iYWwgcmVzb3Vy
Y2UsIGFzIHdlbGwgYXMgZW5hYmxlIHVzZSBvZiB0aG9zZSBkZXZpY2VzIGFzIHBlcnNpc3RlbnQg
DQptZW1vcnkuIEJUVyB3ZSBjcmltaW5hbGx5IDspIHVzZWQgdGhlIHZtX2luc2VydF9wYWdlIGZ1
bmN0aW9uIGluIGEgDQpwcm90b3R5cGUgYW5kIGhhdmUgZm91bmQgdGhhdCBpdCBpcyBmYXN0ZXIg
dG8gdXNlIHZzIHBhZ2UgY2FjaGUgYW5kIA0Kc3dhcHBpbmcgbWVjaGFuaXNtcyBsaW1pdGVkIHRv
IHVzZSBhIHNtYWxsIGFtb3VudCBvZiBEUkFNLg==

[LSF/MM TOPIC] User Directed Tiered Memory Management

[Adam Manzanares]

With the introduction of byte addressable storage devices that have low 
latencies, it becomes difficult to decide how to expose these devices to 
user space applications. Do we treat them as traditional block devices 
or expose them as a DAX capable device? A traditional block device 
allows us to use the page cache to take advantage of locality in access 
patterns, but comes at the expense of extra memory copies that are 
extremely costly for random workloads. A DAX capable device seems great 
for the aforementioned random access workload, but suffers once there is 
some locality in the access pattern.

When DAX-capable devices are used as slower/cheaper volatile memory, 
treating them as a slower NUMA node with an associated NUMA migration 
policy would allow for taking advantage of access pattern locality. 
However this approach suffers from a few drawbacks. First, when those 
devices are also persistent, the tiering approach used in NUMA migration 
may not guarantee persistence. Secondly, for devices with significantly 
higher latencies than DRAM, the cost of moving clean pages may be 
significant. Finally, pages handled via NUMA migration are a common 
resource subject to thrashing in case of memory pressure.

I would like to discuss an alternative approach where memory intensive 
applications mmap these storage devices into their address space. The 
application can specify how much DRAM could be used as a cache and have 
some influence on prefetching and eviction policies. The goal of such an 
approach would be to minimize the impact of the slightly slower memory 
could potentially have on a system when it is treated as kernel managed 
global resource, as well as enable use of those devices as persistent 
memory. BTW we criminally ;) used the vm_insert_page function in a 
prototype and have found that it is faster to use vs page cache and 
swapping mechanisms limited to use a small amount of DRAM.