lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Stephen Bates]

SGkgQWxsDQoNCkkgYW0gaG9waW5nIHNvbWVvbmUgY2FuIGhlbHAgc2hlZCBzb21lIGxpZ2h0IG9u
IGFuIGlzc3VlIEkgYW0gc2VlaW5nIHdpdGggbXkgYXR0ZW1wdCB0byBhZGQgcDJwbWVtIFsxXSB0
byB0aGUgcHBjNjRlbCBrZXJuZWwuIHAycG1lbSBpcyBhIChjdXJyZW50bHkpIG91dC1vZi10cmVl
IHBhdGNoc2V0IHRoYXQgYWxsb3dzIG9uZSB0byBhZGQgZGV2aWNlIG1lbW9yeSB3aXRoIHN0cnVj
dCBwYWdlIGJhY2tpbmcgaW50byB0aGUgTGludXgga2VybmVsLiBJdCBsZXZlcmFnZXMgTUVNT1JZ
X0hPVFBMVUcgYW5kIFpPTkVfREVWSUNFIHdoaWNoIHdlcmUgYWRkZWQgdG8gcHBjNjQgaW4gNC4x
NCBzbyBJIHRob3VnaHQgaXQgd291bGQgYmUgZnVuIHRvIHRyeSBpdCBvdXQuDQoNCldlIGNvbnN0
cnVjdGVkIGEgcGF0Y2hzZXQgYmFzZWQgb2ZmIDQuMTQtcmMxIFsxXSBhbmQgYnVpbGQgYSBrZXJu
ZWwgYmFzZWQgb2ZmIHRoZSBwc2VyaWVzIGRlZmNvbmZpZyBhbmQgcmFuIHRoaXMgb24gdXBzdHJl
YW0gcWVtdS1zeXN0ZW0tcHBjNjQuIFRoZSBleGFjdCBjb21tYW5kIHRvIHJ1biBRRU1VIHdhczoN
Cg0KcWVtdS1zeXN0ZW0tcHBjNjQgXA0KICAgIC1tYWNoaW5lIHBzZXJpZXMgXA0KICAgIC1jcHUg
cG93ZXI4IFwNCiAgICAtc21wIDEgLW0gMjA0OCBcDQogICAgLWtlcm5lbCB+L2tlcm5lbC9saW51
eC1wcGM2NGVsL3ZtbGludXggXA0KICAgIC1hcHBlbmQgIm52bWUudXNlX2NtYj0yNCBjb25zb2xl
PWh2YyByb290PS9kZXYvc2RhIHJvb3R3YWl0IHJ3IiBcDQogICAgLXNlcmlhbCBtb246c3RkaW8g
LWRyaXZlIGZpbGU9aW1hZ2UtcHBjNjRlbC5pbWcsaWY9c2NzaSxmb3JtYXQ9cmF3LGluZGV4PTAg
XA0KICAgIC1ub2dyYXBoaWMgXA0KICAgIC1kcml2ZSBmaWxlPS4uL2ltYWdlcy9udm1lLnFjb3cy
LGlmPW5vbmUsaWQ9bnZtZTEsc25hcHNob3Q9b24gXA0KICAgIC1kZXZpY2UgbnZtZSxkcml2ZT1u
dm1lMSxzZXJpYWw9bnZtZTEgXA0KICAgIC1kcml2ZSBmaWxlPS4uL2ltYWdlcy9udm1lMi5xY293
MixpZj1ub25lLGlkPW52bWUyLHNuYXBzaG90PW9uIFwNCiAgICAtZGV2aWNlIG52bWUsZHJpdmU9
bnZtZTIsc2VyaWFsPW52bWUyLGNtYl9zaXplX21iPTY0DQoNClRoaXMgcmVzdWx0ZWQgaW4gdGhl
IGZvbGxvd2luZyBleHRyYWN0IGZyb20gZG1lc2cgd2hlbiByZWdpc3RlcmluZyB0aGUgcDJwbWVt
IGFzc29jaWF0ZWQgd2l0aCBvbmUgb2YgdGhlIE5WTWUgU1NEcy4NCg0KWyAgICAzLjUwODQ5N10g
bnZtZSAwMDAwOjAwOjAzLjA6IGVuYWJsaW5nIGRldmljZSAoMDEwMCAtPiAwMTAyKQ0KWyAgICAz
LjUxMDc0M10gbnZtZSAwMDAwOjAwOjAzLjA6IFVzaW5nIDY0LWJpdCBkaXJlY3QgRE1BIGF0IG9m
ZnNldCA4MDAwMDAwMDAwMDAwMDANClsgICAgMy41MzU3MDZdIHAycG1lbSBwMnBtZW0wOiByZWdp
c3RlcmVkDQpbICAgIDMuNTM3NzgwXSBscGFyOiBBdHRlbXB0aW5nIHRvIHJlc2l6ZSBIUFQgdG8g
c2hpZnQgMjENClsgICAgMy41MzkyNTFdIFVuYWJsZSB0byByZXNpemUgaGFzaCBwYWdlIHRhYmxl
IHRvIHRhcmdldCBvcmRlciAyMTogLTENClsgICAgMy41NDEwNzldIFVuYWJsZSB0byBjcmVhdGUg
bWFwcGluZyBmb3IgaG90IGFkZGVkIG1lbW9yeSAweGMwMDAyMTAwMDAwMDAwMDAuLjB4YzAwMDIx
MDAwNDAwMDAwMDogLTINClsgICAgMy41NTA0MDddIHAycG1lbSBwMnBtZW0wOiB1bnJlZ2lzdGVy
ZWQNCg0KSSBhbSBub3QgdGhhdCBmYW1pbGlhciB3aXRoIHRoZSBwc2VyaWVzIGFyY2hpdGVjdHVy
ZSBzbyBJIHdhcyBob3BpbmcgZm9yIHNvbWUgZ3VpZGFuY2UgY29uY2VybmluZyB0aGUgbHBhciBl
cnJvciBtZXNzYWdlLiBJZiBhbnkgcHBjNjQgY29kZXJzIGZhbmN5IGhhdmluZyBhIGdvIGF0IHRo
aXMgdGhlIGtlcm5lbCBjb2RlIGlzIGF0IFsxXSBhbmQgSeKAmWQgYmUgaGFwcHkgdG8gcHJvdmlk
ZSB0aGUgLmNvbmZpZyB1c2VkIGlmIG5lZWRlZC4gSnVzdCBsZXQgbWUga25vdy4NCg0KQ2hlZXJz
DQogDQpTdGVwaGVuDQoNClsxXSBodHRwczovL2dpdGh1Yi5jb20vc2JhdGVzMTMwMjcyL2xpbnV4
LXAycG1lbQ0KDQoNCg==

Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Balbir Singh]

On Wed, Oct 18, 2017 at 7:38 AM, Stephen  Bates <sbates@raithlin.com> wrote=
:
> Hi All
>
> I am hoping someone can help shed some light on an issue I am seeing with=
 my attempt to add p2pmem [1] to the ppc64el kernel. p2pmem is a (currently=
) out-of-tree patchset that allows one to add device memory with struct pag=
e backing into the Linux kernel. It leverages MEMORY_HOTPLUG and ZONE_DEVIC=
E which were added to ppc64 in 4.14 so I thought it would be fun to try it =
out.
>
> We constructed a patchset based off 4.14-rc1 [1] and build a kernel based=
 off the pseries defconfig and ran this on upstream qemu-system-ppc64. The =
exact command to run QEMU was:
>
> qemu-system-ppc64 \
>     -machine pseries \
>     -cpu power8 \
>     -smp 1 -m 2048 \
>     -kernel ~/kernel/linux-ppc64el/vmlinux \
>     -append "nvme.use_cmb=3D24 console=3Dhvc root=3D/dev/sda rootwait rw"=
 \
>     -serial mon:stdio -drive file=3Dimage-ppc64el.img,if=3Dscsi,format=3D=
raw,index=3D0 \
>     -nographic \
>     -drive file=3D../images/nvme.qcow2,if=3Dnone,id=3Dnvme1,snapshot=3Don=
 \
>     -device nvme,drive=3Dnvme1,serial=3Dnvme1 \
>     -drive file=3D../images/nvme2.qcow2,if=3Dnone,id=3Dnvme2,snapshot=3Do=
n \
>     -device nvme,drive=3Dnvme2,serial=3Dnvme2,cmb_size_mb=3D64
>
> This resulted in the following extract from dmesg when registering the p2=
pmem associated with one of the NVMe SSDs.
>
> [    3.508497] nvme 0000:00:03.0: enabling device (0100 -> 0102)
> [    3.510743] nvme 0000:00:03.0: Using 64-bit direct DMA at offset 80000=
0000000000
> [    3.535706] p2pmem p2pmem0: registered
> [    3.537780] lpar: Attempting to resize HPT to shift 21
> [    3.539251] Unable to resize hash page table to target order 21: -1

I am guessing that the hotplug of ZONE_DEVICE memory was done
incorrectly as it lead to HPT resizing (the system thinking this is
normal memory). Ideally one would expect that the driver would online
ZONE_DEVICE memory and not go through the HOTPLUG path. Are you using
devm_memremap_pages() path to add these pages?

Balbir Singh.

Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Stephen Bates]

DQo+IEkgYW0gZ3Vlc3NpbmcgdGhhdCB0aGUgaG90cGx1ZyBvZiBaT05FX0RFVklDRSBtZW1vcnkg
d2FzIGRvbmUNCj4gaW5jb3JyZWN0bHkgYXMgaXQgbGVhZCB0byBIUFQgcmVzaXppbmcgKHRoZSBz
eXN0ZW0gdGhpbmtpbmcgdGhpcyBpcw0KPiBub3JtYWwgbWVtb3J5KS4gSWRlYWxseSBvbmUgd291
bGQgZXhwZWN0IHRoYXQgdGhlIGRyaXZlciB3b3VsZCBvbmxpbmUNCj4gWk9ORV9ERVZJQ0UgbWVt
b3J5IGFuZCBub3QgZ28gdGhyb3VnaCB0aGUgSE9UUExVRyBwYXRoLiBBcmUgeW91IHVzaW5nDQo+
IGRldm1fbWVtcmVtYXBfcGFnZXMoKSBwYXRoIHRvIGFkZCB0aGVzZSBwYWdlcz8NCj4NCg0KVGhh
bmtzIGZvciB0aGUgcmVzcG9uc2UgQmFsYmlyLiBZZXMgd2UgdXNlIGRldm1fbWVtcmVtYXBfcGFn
ZXMoKSB0byBhZGQgdGhlc2UgcGFnZXMgYW5kIGl0IGRvZXMgY2FsbCBhcmNoX2FkZF9tZW1vcnko
KS4gV2UgZG8gaGF2ZSBhbiBhbHRlcm5hdGUgc2V0IG9mIHBhdGNoZXMgd2hpY2ggc3RpbGwgY2Fs
bHMgZGV2bV9tZW1yZW1hcF9wYWdlcygpIGJ1dCBjYW4gdGFrZSBhIGZsYWcgdG8gaW5kaWNhdGUg
dGhlIG1lbW9yeSBiZWluZyBhZGRlZCBpcyBpbyBtZW1vcnkgYW5kIHVzZXMgaW9fcmVtYXAoKSBy
YXRoZXIgdGhhbiBhcmNoX2FkZF9tZW1vcnkoKSBmb3IgdGhhdCB0eXBlIG9mIG1lbW9yeSBbMV0u
IFdvdWxkIHRoYXQgYmUgYSBiZXR0ZXIgYXBwcm9hY2ggZm9yIHRoaXMgYXJjaD8gSSBjYW4gdHJ5
IGFuZCBhcHBseSB0aGlzIHBhdGNoIGJ1dCBfX2FkZF9wYWdlcygpIGhhcyBnb25lIHRocm91Z2gg
c29tZSBjaGFuZ2VzIHJlY2VudGx5IHNvIGl0IHdpbGwgdGFrZSBtZSBhIGZldyBkYXlzIHRvIGdl
dCB0byB0aGF0LiANCg0KU3RlcGhlbg0KDQpbMV0gaHR0cHM6Ly9naXRodWIuY29tL3NiYXRlczEz
MDI3Mi9saW51eC1wMnBtZW0vY29tbWl0L2FjNzM1ODcxZmNkMmM2M2JkMzNjODE0YWEzOTQxY2Ez
ZWY1M2I2MzYNCg0KDQo=

Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Balbir Singh]

On Sat, 21 Oct 2017 15:03:29 +0000
"Stephen  Bates" <sbates@raithlin.com> wrote:

> > I am guessing that the hotplug of ZONE_DEVICE memory was done
> > incorrectly as it lead to HPT resizing (the system thinking this is
> > normal memory). Ideally one would expect that the driver would online
> > ZONE_DEVICE memory and not go through the HOTPLUG path. Are you using
> > devm_memremap_pages() path to add these pages?
> >  
> 
> Thanks for the response Balbir. Yes we use devm_memremap_pages() to add these pages and it does call arch_add_memory(). We do have an alternate set of patches which still calls devm_memremap_pages() but can take a flag to indicate the memory being added is io memory and uses io_remap() rather than arch_add_memory() for that type of memory [1]. Would that be a better approach for this arch? I can try and apply this patch but __add_pages() has gone through some changes recently so it will take me a few days to get to that. 

I just double checked, for pmem you do need to come in via arch_add_memory(). I was confused
by what we do for HMM, which is call __add_pages(), but we do need a section mapping so
the interface is correct.

The following

[    3.537780] lpar: Attempting to resize HPT to shift 21
[    3.539251] Unable to resize hash page table to target order 21: -1
[    3.541079] Unable to create mapping for hot added memory 0xc000210000000000..0xc000210004000000: -2

Needs to be debugged further. For #1 above please check if your qemu supports
H_RESIZE_HPT_* hcalls? For create mapping failures, the rc is -ENOENT. Can
you help debug this further? We could do hcall tracing or enable debugging.

Balbir Singh.

Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Stephen Bates]

DQo+IFsgICAgMy41Mzc3ODBdIGxwYXI6IEF0dGVtcHRpbmcgdG8gcmVzaXplIEhQVCB0byBzaGlm
dCAyMQ0KPiBbICAgIDMuNTM5MjUxXSBVbmFibGUgdG8gcmVzaXplIGhhc2ggcGFnZSB0YWJsZSB0
byB0YXJnZXQgb3JkZXIgMjE6IC0xDQo+IFsgICAgMy41NDEwNzldIFVuYWJsZSB0byBjcmVhdGUg
bWFwcGluZyBmb3IgaG90IGFkZGVkIG1lbW9yeSAweGMwMDAyMTAwMDAwMDAwMDAuLjB4YzAwMDIx
MDAwNDAwMDAwMDogLTINCg0KPiBGb3IgIzEgYWJvdmUgcGxlYXNlIGNoZWNrIGlmIHlvdXIgcWVt
dSBzdXBwb3J0cyBIX1JFU0laRV9IUFRfKiBoY2FsbHM/IA0KDQpCYWxiaXIgZG8geW91IGhhdmUg
YW55IHN1Z2dlc3Rpb25zIGFzIHRvIGhvdyB0byB0ZXN0IGZvciB0aGlzIHN1cHBvcnQ/IE5vdGUg
SSBhbSBydW5uaW5nIHRoaXMgb24gbXkgeDg2XzY0IGhvc3Qgc28gdGhlcmUgaXMgbm8gdmlydHVh
bGl6YXRpb24gaGFyZHdhcmUgaW4gbXkgUUVNVS4gTXkgcWVtdSBpcyB2ZXJ5IHJlY2VudCAoUUVN
VSBlbXVsYXRvciB2ZXJzaW9uIDIuMTAuNTAgKHYyLjEwLjAtMTAyNi1nZDhmOTMyYy1kaXJ0eSkp
Lg0KDQo+IEZvciBjcmVhdGUgbWFwcGluZyBmYWlsdXJlcywgdGhlIHJjIGlzIC1FTk9FTlQuIENh
biB5b3UgaGVscCBkZWJ1ZyB0aGlzIGZ1cnRoZXI/IFdlIGNvdWxkIGRvIGhjYWxsIHRyYWNpbmcg
b3IgZW5hYmxlIGRlYnVnZ2luZy4NCg0KU3VyZSBJIGNhbiBoZWxwIGRlYnVnLiBNeSBvcmlnaW5h
bCBlbWFpbCBhbHNvIGhhZCBhbGwgeW91IG5lZWRlZCB0byByZWNyZWF0ZSB0aGlzIGlzc3VlIHNv
IHRoYXTigJlzIGFuIG9wdGlvbiB0b28/DQoNClN0ZXBoZW4NCg0KDQo=

Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Oliver]

On Tue, Oct 24, 2017 at 7:17 AM, Stephen  Bates <sbates@raithlin.com> wrote=
:
>
>> [    3.537780] lpar: Attempting to resize HPT to shift 21
>> [    3.539251] Unable to resize hash page table to target order 21: -1
>> [    3.541079] Unable to create mapping for hot added memory 0xc00021000=
0000000..0xc000210004000000: -2
>
>> For #1 above please check if your qemu supports H_RESIZE_HPT_* hcalls?
>
> Balbir do you have any suggestions as to how to test for this support? No=
te I am running this on my x86_64 host so there is no virtualization hardwa=
re in my QEMU. My qemu is very recent (QEMU emulator version 2.10.50 (v2.10=
.0-1026-gd8f932c-dirty)).

Honestly I'd just ignore the resize error. The hash table stores PTE
entries so it should be sized based on the amount of memory in the
system. If it's drastically under sized there'll be a performance hit,
but  everything should still work.

>> For create mapping failures, the rc is -ENOENT. Can you help debug this =
further? We could do hcall tracing or enable debugging.
>
> Sure I can help debug. My original email also had all you needed to recre=
ate this issue so that=E2=80=99s an option too?

I'm not too sure what's happening there. My hunch is that the
hypervisor (qemu in this case) is rejecting the attempt to map the PCI
device MMIO space as cachable memory. On bare metal systems this can
result in cache paradoxes which will kill the system so the hypervisor
has an incentive to prevent that situation.

>
> Stephen
>
>

Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Oliver]

On Thu, Oct 26, 2017 at 1:34 AM, Oliver <oohall@gmail.com> wrote:
> On Tue, Oct 24, 2017 at 7:17 AM, Stephen  Bates <sbates@raithlin.com> wro=
te:
>>
>>> [    3.537780] lpar: Attempting to resize HPT to shift 21
>>> [    3.539251] Unable to resize hash page table to target order 21: -1
>>> [    3.541079] Unable to create mapping for hot added memory 0xc0002100=
00000000..0xc000210004000000: -2
>>
>>> For #1 above please check if your qemu supports H_RESIZE_HPT_* hcalls?
>>
>> Balbir do you have any suggestions as to how to test for this support? N=
ote I am running this on my x86_64 host so there is no virtualization hardw=
are in my QEMU. My qemu is very recent (QEMU emulator version 2.10.50 (v2.1=
0.0-1026-gd8f932c-dirty)).
>
> Honestly I'd just ignore the resize error. The hash table stores PTE
> entries so it should be sized based on the amount of memory in the
> system. If it's drastically under sized there'll be a performance hit,
> but  everything should still work.
>
>>> For create mapping failures, the rc is -ENOENT. Can you help debug this=
 further? We could do hcall tracing or enable debugging.
>>
>> Sure I can help debug. My original email also had all you needed to recr=
eate this issue so that=E2=80=99s an option too?
>
> I'm not too sure what's happening there. My hunch is that the
> hypervisor (qemu in this case) is rejecting the attempt to map the PCI
> device MMIO space as cachable memory. On bare metal systems this can
> result in cache paradoxes which will kill the system so the hypervisor
> has an incentive to prevent that situation.

So I had a deeper look and found the hypervisor interface spec (PAPR)
says the hypervisor should reject attempts to map memory with
inappropriate attributes for the type of memory being mapped. The
pseries model in qemu interprets this by only allowing cacheable
mappings on memory ranges that it considers as RAM. While KVM will
allow any mappings provided they have the same cachable attribute as
the hypervisor's mapping. Either way trying to use
devm_memremap_pages() like this on pseries is fundementally broken.
The alternative approach you mentioned that uses ioremap() should work
fine though.

Also, Alexy (+cc) said he was interested in trying this on some real
hardware. Is there a test suite for p2pmem floating around that he can
use?

Thanks,
Oliver

Re: lpar issue for ZONE_DEVICE p2pmem in 4.14-rc

[Stephen Bates]

DQpPbGl2ZXINCg0KPlRoZSBhbHRlcm5hdGl2ZSBhcHByb2FjaCB5b3UgbWVudGlvbmVkIHRoYXQg
dXNlcyBpb3JlbWFwKCkgc2hvdWxkIHdvcmsNCj5maW5lIHRob3VnaC4NCg0KR3JlYXQsIHRoYW5r
cyBmb3IgdGhpcyB1c2VmdWwgaW5mb3JtYXRpb24gT2xpdmVyLiBJIGFtIHdvcmtpbmcgb24gcmVi
YXNpbmcgdGhlIGlvcmVtYXAoKSANCnBhdGNoIG9uIDQuMTQgYW5kIHdpbGwgbGV0IHlvdSBrbm93
IGhvdyBpdCBnb2VzLg0KDQo+IEFsc28sIEFsZXh5ICgrY2MpIHNhaWQgaGUgd2FzIGludGVyZXN0
ZWQgaW4gdHJ5aW5nIHRoaXMgb24gc29tZSByZWFsDQo+IGhhcmR3YXJlLiBJcyB0aGVyZSBhIHRl
c3Qgc3VpdGUgZm9yIHAycG1lbSBmbG9hdGluZyBhcm91bmQgdGhhdCBoZSBjYW4NCj4gdXNlPw0K
DQpFeGNlbGxlbnQhIEkgdHlwaWNhbGx5IHVzZSBwMnBtZW0tdGVzdCBbMV0uIFlvdSB3aWxsIG5l
ZWQgYXQgbGVhc3Qgb25lIE5WTWUgU1NEIGFuZCBvbmUgcDJwbWVtIGNhcGFibGUgZGV2aWNlIGFu
ZCB5b3UgKm1pZ2h0KiBuZWVkIHRvIHBsYWNlIHRob3NlIHR3byBkZXZpY2VzIGJlaGluZCBhIFBD
SWUgc3dpdGNoIGFuZCBjb25maWd1cmUgQUNTIFsyLTNdIGRlcGVuZGluZyBvbiBob3cgdGhlIENQ
VSBSQyB0cmVhdHMgcGVlciB0byBwZWVyIFBDSSBUTFBzLiBJIGFtIGhhcHB5IHRvIGhlbHAgQWxl
eCBzZXQgdGhpcyBhbGwgdXAgc28gcmVhY2ggb3V0IGVpdGhlciBwdWJsaWNhbGx5IG9yIHByaXZh
dGVseSBpZiB5b3UgbGlrZS4gSSBqdXN0IGFkZGVkIGEgKGhvcGVmdWxseSkgZGVjZW50IFJFQURN
RSB0byBoZWxwIGdldCBwZW9wbGUgc3RhcnRlZC4NCg0KU3RlcGhlbg0KDQpbMV0gaHR0cHM6Ly9n
aXRodWIuY29tL3NiYXRlczEzMDI3Mi9wMnBtZW0tdGVzdA0KWzJdIGh0dHBzOi8vd3d3LnN1cGVy
bWljcm8uY29tL3N1cHBvcnQvZmFxcy9mYXEuY2ZtP2ZhcT0yMDczMg0KWzNdIGh0dHBzOi8vbGtt
bC5vcmcvbGttbC8yMDE3LzEwLzI2LzY3OCANCg0K