* [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup @ 2018-06-14 0:11 Luis R. Rodriguez 2018-06-14 10:01 ` Damien Le Moal ` (2 more replies) 0 siblings, 3 replies; 22+ messages in thread From: Luis R. Rodriguez @ 2018-06-14 0:11 UTC (permalink / raw) To: damien.lemoal Cc: hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block, Luis R. Rodriguez Setting up a zoned disks in a generic form is not so trivial. There is also quite a bit of tribal knowledge with these devices which is not easy to find. The currently supplied demo script works but it is not generic enough to be practical for Linux distributions or even developers which often move from one kernel to another. This tries to put a bit of this tribal knowledge into an initial udev rule for development with the hopes Linux distributions can later deploy. Three rule are added. One rule is optional for now, it should be extended later to be more distribution-friendly and then I think this may be ready for consideration for integration on distributions. 1) scheduler setup 2) backlist f2fs devices 3) run dmsetup for the rest of devices Note that this udev rule will not work well if you want to use a disk with f2fs on part of the disk and another filesystem on another part of the disk. That setup will require manual love so these setups can use the same backlist on rule 2). Its not widely known for instance that as of v4.16 it is mandated to use either deadline or the mq-deadline scheduler for *all* SMR drivers. Its also been determined that the Linux kernel is not the place to set this up, so a udev rule *is required* as per latest discussions. This is the first rule we add. Furthermore if you are *not* using f2fs you always have to run dmsetup. dmsetups do not persist, so you currently *always* have to run a custom sort of script, which is not ideal for Linux distributions. We can invert this logic into a udev rule to enable users to blacklist disks they know they want to use f2fs for. This the second optional rule. This blacklisting can be generalized further in the future with an exception list file, for instance using INPUT{db} or the like. The third and final rule added then runs dmsetup for the rest of the disks using the disk serial number for the new device mapper name. Note that it is currently easy for users to make a mistake and run mkfs on the the original disk, not the /dev/mapper/ device for non f2fs arrangements. If that is done experience shows things can easily fall apart with alignment *eventually*. We have no generic way today to error out on this condition and proactively prevent this. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> --- README | 10 +++++- udev/99-zoned-disks.rules | 78 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+), 1 deletion(-) create mode 100644 udev/99-zoned-disks.rules diff --git a/README b/README index 65e96c34fd04..f49541eaabc8 100644 --- a/README +++ b/README @@ -168,7 +168,15 @@ Options: reclaiming random zones if the percentage of free random data zones falls below <perc>. -V. Example scripts +V. Udev zone disk deployment +============================ + +A udev rule is provided which enables you to set the IO scheduler, blacklist +driver to run dmsetup, and runs dmsetup for the rest of the zone drivers. +If you use this udev rule the below script is not needed. Be sure to mkfs only +on the resulting /dev/mapper/zone-$serial device you end up with. + +VI. Example scripts ================== [[ diff --git a/udev/99-zoned-disks.rules b/udev/99-zoned-disks.rules new file mode 100644 index 000000000000..e19b738dcc0e --- /dev/null +++ b/udev/99-zoned-disks.rules @@ -0,0 +1,78 @@ +# To use a zone disks first thing you need to: +# +# 1) Enable zone disk support in your kernel +# 2) Use the deadline or mq-deadline scheduler for it - mandated as of v4.16 +# 3) Blacklist devices dedicated for f2fs as of v4.10 +# 4) Run dmsetup other disks +# 5) Create the filesystem -- NOTE: use mkfs /dev/mapper/zone-serial if +# you enabled use dmsetup on the disk. +# 6) Consider using nofail mount option in case you run an supported kernel +# +# You can use this udev rules file for 2) 3) and 4). Further details below. +# +# 1) Enable zone disk support in your kernel +# +# o CONFIG_BLK_DEV_ZONED +# o CONFIG_DM_ZONED +# +# This will let the kernel actually see these devices, ie, via fdisk /dev/sda +# for instance. Run: +# +# dmzadm --format /dev/sda + +# 2) Set deadline or mq-deadline for all disks which are zoned +# +# Zoned disks can only work with the deadline or mq-deadline scheduler. This is +# mandated for all SMR drives since v4.16. It has been determined this must be +# done through a udev rule, and the kernel should not set this up for disks. +# This magic will have to live for *all* zoned disks. +# XXX: what about distributions that want mq-deadline ? Probably easy for now +# to assume deadline and later have a mapping file to enable +# mq-deadline for specific serial devices? +ACTION=="add|change", KERNEL=="sd*[!0-9]", ATTRS{queue/zoned}=="host-managed", \ + ATTR{queue/scheduler}="deadline" + +# 3) Blacklist f2fs devices as of v4.10 +# We don't have to run dmsetup on on disks where you want to use f2fs, so you +# can use this rule to skip dmsetup for it. First get the serial short number. +# +# udevadm info --name=/dev/sda | grep -i serial_shor +# XXX: To generalize this for distributions consider using INPUT{db} to or so +# and then use that to check if the serial number matches one on the database. +#ACTION=="add", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="XXA1ZFFF", GOTO="zone_disk_group_end" + +# 4) We need to run dmsetup if you want to use other filesystems +# +# dmsetup is not persistent, so it needs to be run on upon every boot. We use +# the device serial number for the /dev/mapper/ name. +ACTION=="add", KERNEL=="sd*[!0-9]", ATTRS{queue/zoned}=="host-managed", \ + RUN+="/sbin/dmsetup create zoned-$env{ID_SERIAL_SHORT} --table '0 %s{size} zoned $devnode'", $attr{size} + +# 4) Create a filesystem for the device +# +# Be 100% sure you use /dev/mapper/zone-$YOUR_DEVICE_SERIAL for the mkfs +# command as otherwise things can break. +# +# XXX: preventing the above proactively in the kernel would be ideal however +# this may be hard. +# +# Once you create the filesystem it will get a UUID. +# +# Find out what UUID is, you can do this for instance if your zoned disk is +# your second device-mapper device, ie dm-1 by: +# +# ls -l /dev/disk/by-uuid/dm-1 +# +# To figure out which dm-$number it is, use dmsetup info, the minor number +# is the $number. +# +# 5) Add an etry in /etc/fstab with nofail for example: +# +# UUID=99999999-aaaa-bbbb-c1234aaaabbb33456 /media/monster xfs nofail 0 0 +# +# nofail will ensure system boots fine even if you boot into a kernel which +# lacks support for the device and so it is not found. Since the UUID will +# always match the device we don't care if the device moves around the bus +# on the system. We just need to get the UUID once. + +LABEL="zone_disk_group_end" -- 2.16.3 ^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 0:11 [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup Luis R. Rodriguez @ 2018-06-14 10:01 ` Damien Le Moal 2018-06-14 12:38 ` Mike Snitzer 2018-06-14 16:19 ` Bart Van Assche 2 siblings, 0 replies; 22+ messages in thread From: Damien Le Moal @ 2018-06-14 10:01 UTC (permalink / raw) To: Luis R. Rodriguez Cc: hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block DQoNCk9uIDYvMTQvMTggMDk6MTEsIEx1aXMgUi4gUm9kcmlndWV6IHdyb3RlOg0KPiBTZXR0aW5n IHVwIGEgem9uZWQgZGlza3MgaW4gYSBnZW5lcmljIGZvcm0gaXMgbm90IHNvIHRyaXZpYWwuIFRo ZXJlDQo+IGlzIGFsc28gcXVpdGUgYSBiaXQgb2YgdHJpYmFsIGtub3dsZWRnZSB3aXRoIHRoZXNl IGRldmljZXMgd2hpY2ggaXMgbm90DQo+IGVhc3kgdG8gZmluZC4NCj4gDQo+IFRoZSBjdXJyZW50 bHkgc3VwcGxpZWQgZGVtbyBzY3JpcHQgd29ya3MgYnV0IGl0IGlzIG5vdCBnZW5lcmljIGVub3Vn aCB0byBiZQ0KPiBwcmFjdGljYWwgZm9yIExpbnV4IGRpc3RyaWJ1dGlvbnMgb3IgZXZlbiBkZXZl bG9wZXJzIHdoaWNoIG9mdGVuIG1vdmUNCj4gZnJvbSBvbmUga2VybmVsIHRvIGFub3RoZXIuDQo+ IA0KPiBUaGlzIHRyaWVzIHRvIHB1dCBhIGJpdCBvZiB0aGlzIHRyaWJhbCBrbm93bGVkZ2UgaW50 byBhbiBpbml0aWFsIHVkZXYNCj4gcnVsZSBmb3IgZGV2ZWxvcG1lbnQgd2l0aCB0aGUgaG9wZXMg TGludXggZGlzdHJpYnV0aW9ucyBjYW4gbGF0ZXINCj4gZGVwbG95LiBUaHJlZSBydWxlIGFyZSBh ZGRlZC4gT25lIHJ1bGUgaXMgb3B0aW9uYWwgZm9yIG5vdywgaXQgc2hvdWxkIGJlDQo+IGV4dGVu ZGVkIGxhdGVyIHRvIGJlIG1vcmUgZGlzdHJpYnV0aW9uLWZyaWVuZGx5IGFuZCB0aGVuIEkgdGhp bmsgdGhpcw0KPiBtYXkgYmUgcmVhZHkgZm9yIGNvbnNpZGVyYXRpb24gZm9yIGludGVncmF0aW9u IG9uIGRpc3RyaWJ1dGlvbnMuDQo+IA0KPiAxKSBzY2hlZHVsZXIgc2V0dXANCj4gMikgYmFja2xp c3QgZjJmcyBkZXZpY2VzDQo+IDMpIHJ1biBkbXNldHVwIGZvciB0aGUgcmVzdCBvZiBkZXZpY2Vz DQo+IA0KPiBOb3RlIHRoYXQgdGhpcyB1ZGV2IHJ1bGUgd2lsbCBub3Qgd29yayB3ZWxsIGlmIHlv dSB3YW50IHRvIHVzZSBhIGRpc2sNCj4gd2l0aCBmMmZzIG9uIHBhcnQgb2YgdGhlIGRpc2sgYW5k IGFub3RoZXIgZmlsZXN5c3RlbSBvbiBhbm90aGVyIHBhcnQgb2YNCj4gdGhlIGRpc2suIFRoYXQg c2V0dXAgd2lsbCByZXF1aXJlIG1hbnVhbCBsb3ZlIHNvIHRoZXNlIHNldHVwcyBjYW4gdXNlDQo+ IHRoZSBzYW1lIGJhY2tsaXN0IG9uIHJ1bGUgMikuDQo+IA0KPiBJdHMgbm90IHdpZGVseSBrbm93 biBmb3IgaW5zdGFuY2UgdGhhdCBhcyBvZiB2NC4xNiBpdCBpcyBtYW5kYXRlZCB0byB1c2UNCj4g ZWl0aGVyIGRlYWRsaW5lIG9yIHRoZSBtcS1kZWFkbGluZSBzY2hlZHVsZXIgZm9yICphbGwqIFNN UiBkcml2ZXJzLiBJdHMNCj4gYWxzbyBiZWVuIGRldGVybWluZWQgdGhhdCB0aGUgTGludXgga2Vy bmVsIGlzIG5vdCB0aGUgcGxhY2UgdG8gc2V0IHRoaXMgdXAsDQo+IHNvIGEgdWRldiBydWxlICpp cyByZXF1aXJlZCogYXMgcGVyIGxhdGVzdCBkaXNjdXNzaW9ucy4gVGhpcyBpcyB0aGUNCj4gZmly c3QgcnVsZSB3ZSBhZGQuDQo+IA0KPiBGdXJ0aGVybW9yZSBpZiB5b3UgYXJlICpub3QqIHVzaW5n IGYyZnMgeW91IGFsd2F5cyBoYXZlIHRvIHJ1biBkbXNldHVwLg0KPiBkbXNldHVwcyBkbyBub3Qg cGVyc2lzdCwgc28geW91IGN1cnJlbnRseSAqYWx3YXlzKiBoYXZlIHRvIHJ1biBhIGN1c3RvbQ0K PiBzb3J0IG9mIHNjcmlwdCwgd2hpY2ggaXMgbm90IGlkZWFsIGZvciBMaW51eCBkaXN0cmlidXRp b25zLiBXZSBjYW4gaW52ZXJ0DQo+IHRoaXMgbG9naWMgaW50byBhIHVkZXYgcnVsZSB0byBlbmFi bGUgdXNlcnMgdG8gYmxhY2tsaXN0IGRpc2tzIHRoZXkga25vdyB0aGV5DQo+IHdhbnQgdG8gdXNl IGYyZnMgZm9yLiBUaGlzIHRoZSBzZWNvbmQgb3B0aW9uYWwgcnVsZS4gVGhpcyBibGFja2xpc3Rp bmcNCj4gY2FuIGJlIGdlbmVyYWxpemVkIGZ1cnRoZXIgaW4gdGhlIGZ1dHVyZSB3aXRoIGFuIGV4 Y2VwdGlvbiBsaXN0IGZpbGUsIGZvcg0KPiBpbnN0YW5jZSB1c2luZyBJTlBVVHtkYn0gb3IgdGhl IGxpa2UuDQo+IA0KPiBUaGUgdGhpcmQgYW5kIGZpbmFsIHJ1bGUgYWRkZWQgdGhlbiBydW5zIGRt c2V0dXAgZm9yIHRoZSByZXN0IG9mIHRoZSBkaXNrcw0KPiB1c2luZyB0aGUgZGlzayBzZXJpYWwg bnVtYmVyIGZvciB0aGUgbmV3IGRldmljZSBtYXBwZXIgbmFtZS4NCj4gDQo+IE5vdGUgdGhhdCBp dCBpcyBjdXJyZW50bHkgZWFzeSBmb3IgdXNlcnMgdG8gbWFrZSBhIG1pc3Rha2UgYW5kIHJ1biBt a2ZzDQo+IG9uIHRoZSB0aGUgb3JpZ2luYWwgZGlzaywgbm90IHRoZSAvZGV2L21hcHBlci8gZGV2 aWNlIGZvciBub24gZjJmcw0KPiBhcnJhbmdlbWVudHMuIElmIHRoYXQgaXMgZG9uZSBleHBlcmll bmNlIHNob3dzIHRoaW5ncyBjYW4gZWFzaWx5IGZhbGwNCj4gYXBhcnQgd2l0aCBhbGlnbm1lbnQg KmV2ZW50dWFsbHkqLiBXZSBoYXZlIG5vIGdlbmVyaWMgd2F5IHRvZGF5IHRvDQo+IGVycm9yIG91 dCBvbiB0aGlzIGNvbmRpdGlvbiBhbmQgcHJvYWN0aXZlbHkgcHJldmVudCB0aGlzLg0KPiANCj4g U2lnbmVkLW9mZi1ieTogTHVpcyBSLiBSb2RyaWd1ZXogPG1jZ3JvZkBrZXJuZWwub3JnPg0KPiAt LS0NCj4gIFJFQURNRSAgICAgICAgICAgICAgICAgICAgfCAxMCArKysrKy0NCj4gIHVkZXYvOTkt em9uZWQtZGlza3MucnVsZXMgfCA3OCArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysr KysrKysrKysrKysrKw0KPiAgMiBmaWxlcyBjaGFuZ2VkLCA4NyBpbnNlcnRpb25zKCspLCAxIGRl bGV0aW9uKC0pDQo+ICBjcmVhdGUgbW9kZSAxMDA2NDQgdWRldi85OS16b25lZC1kaXNrcy5ydWxl cw0KPiANCj4gZGlmZiAtLWdpdCBhL1JFQURNRSBiL1JFQURNRQ0KPiBpbmRleCA2NWU5NmMzNGZk MDQuLmY0OTU0MWVhYWJjOCAxMDA2NDQNCj4gLS0tIGEvUkVBRE1FDQo+ICsrKyBiL1JFQURNRQ0K PiBAQCAtMTY4LDcgKzE2OCwxNSBAQCBPcHRpb25zOg0KPiAgICAgICAgICAgICAgICAgICAgICAg cmVjbGFpbWluZyByYW5kb20gem9uZXMgaWYgdGhlIHBlcmNlbnRhZ2Ugb2YNCj4gICAgICAgICAg ICAgICAgICAgICAgIGZyZWUgcmFuZG9tIGRhdGEgem9uZXMgZmFsbHMgYmVsb3cgPHBlcmM+Lg0K PiAgDQo+IC1WLiBFeGFtcGxlIHNjcmlwdHMNCj4gK1YuIFVkZXYgem9uZSBkaXNrIGRlcGxveW1l bnQNCj4gKz09PT09PT09PT09PT09PT09PT09PT09PT09PT0NCj4gKw0KPiArQSB1ZGV2IHJ1bGUg aXMgcHJvdmlkZWQgd2hpY2ggZW5hYmxlcyB5b3UgdG8gc2V0IHRoZSBJTyBzY2hlZHVsZXIsIGJs YWNrbGlzdA0KPiArZHJpdmVyIHRvIHJ1biBkbXNldHVwLCBhbmQgcnVucyBkbXNldHVwIGZvciB0 aGUgcmVzdCBvZiB0aGUgem9uZSBkcml2ZXJzLg0KPiArSWYgeW91IHVzZSB0aGlzIHVkZXYgcnVs ZSB0aGUgYmVsb3cgc2NyaXB0IGlzIG5vdCBuZWVkZWQuIEJlIHN1cmUgdG8gbWtmcyBvbmx5DQo+ ICtvbiB0aGUgcmVzdWx0aW5nIC9kZXYvbWFwcGVyL3pvbmUtJHNlcmlhbCBkZXZpY2UgeW91IGVu ZCB1cCB3aXRoLg0KPiArDQo+ICtWSS4gRXhhbXBsZSBzY3JpcHRzDQo+ICA9PT09PT09PT09PT09 PT09PT0NCj4gIA0KPiAgW1sNCj4gZGlmZiAtLWdpdCBhL3VkZXYvOTktem9uZWQtZGlza3MucnVs ZXMgYi91ZGV2Lzk5LXpvbmVkLWRpc2tzLnJ1bGVzDQo+IG5ldyBmaWxlIG1vZGUgMTAwNjQ0DQo+ IGluZGV4IDAwMDAwMDAwMDAwMC4uZTE5YjczOGRjYzBlDQo+IC0tLSAvZGV2L251bGwNCj4gKysr IGIvdWRldi85OS16b25lZC1kaXNrcy5ydWxlcw0KPiBAQCAtMCwwICsxLDc4IEBADQo+ICsjIFRv IHVzZSBhIHpvbmUgZGlza3MgZmlyc3QgdGhpbmcgeW91IG5lZWQgdG86DQo+ICsjDQo+ICsjIDEp IEVuYWJsZSB6b25lIGRpc2sgc3VwcG9ydCBpbiB5b3VyIGtlcm5lbA0KPiArIyAyKSBVc2UgdGhl IGRlYWRsaW5lIG9yIG1xLWRlYWRsaW5lIHNjaGVkdWxlciBmb3IgaXQgLSBtYW5kYXRlZCBhcyBv ZiB2NC4xNg0KPiArIyAzKSBCbGFja2xpc3QgZGV2aWNlcyBkZWRpY2F0ZWQgZm9yIGYyZnMgYXMg b2YgdjQuMTANCj4gKyMgNCkgUnVuIGRtc2V0dXAgb3RoZXIgZGlza3MNCj4gKyMgNSkgQ3JlYXRl IHRoZSBmaWxlc3lzdGVtIC0tIE5PVEU6IHVzZSBta2ZzIC9kZXYvbWFwcGVyL3pvbmUtc2VyaWFs IGlmDQo+ICsjICAgIHlvdSBlbmFibGVkIHVzZSBkbXNldHVwIG9uIHRoZSBkaXNrLg0KPiArIyA2 KSBDb25zaWRlciB1c2luZyBub2ZhaWwgbW91bnQgb3B0aW9uIGluIGNhc2UgeW91IHJ1biBhbiBz dXBwb3J0ZWQga2VybmVsDQo+ICsjDQo+ICsjIFlvdSBjYW4gdXNlIHRoaXMgdWRldiBydWxlcyBm aWxlIGZvciAyKSAzKSBhbmQgNCkuIEZ1cnRoZXIgZGV0YWlscyBiZWxvdy4NCj4gKyMNCj4gKyMg MSkgRW5hYmxlIHpvbmUgZGlzayBzdXBwb3J0IGluIHlvdXIga2VybmVsDQo+ICsjDQo+ICsjICAg IG8gQ09ORklHX0JMS19ERVZfWk9ORUQNCj4gKyMgICAgbyBDT05GSUdfRE1fWk9ORUQNCj4gKyMN Cj4gKyMgVGhpcyB3aWxsIGxldCB0aGUga2VybmVsIGFjdHVhbGx5IHNlZSB0aGVzZSBkZXZpY2Vz LCBpZSwgdmlhIGZkaXNrIC9kZXYvc2RhDQo+ICsjIGZvciBpbnN0YW5jZS4gUnVuOg0KPiArIw0K PiArIyAJZG16YWRtIC0tZm9ybWF0IC9kZXYvc2RhDQo+ICsNCj4gKyMgMikgU2V0IGRlYWRsaW5l IG9yIG1xLWRlYWRsaW5lIGZvciBhbGwgZGlza3Mgd2hpY2ggYXJlIHpvbmVkDQo+ICsjDQo+ICsj IFpvbmVkIGRpc2tzIGNhbiBvbmx5IHdvcmsgd2l0aCB0aGUgZGVhZGxpbmUgb3IgbXEtZGVhZGxp bmUgc2NoZWR1bGVyLiBUaGlzIGlzDQo+ICsjIG1hbmRhdGVkIGZvciBhbGwgU01SIGRyaXZlcyBz aW5jZSB2NC4xNi4gSXQgaGFzIGJlZW4gZGV0ZXJtaW5lZCB0aGlzIG11c3QgYmUNCj4gKyMgZG9u ZSB0aHJvdWdoIGEgdWRldiBydWxlLCBhbmQgdGhlIGtlcm5lbCBzaG91bGQgbm90IHNldCB0aGlz IHVwIGZvciBkaXNrcy4NCj4gKyMgVGhpcyBtYWdpYyB3aWxsIGhhdmUgdG8gbGl2ZSBmb3IgKmFs bCogem9uZWQgZGlza3MuDQo+ICsjIFhYWDogd2hhdCBhYm91dCBkaXN0cmlidXRpb25zIHRoYXQg d2FudCBtcS1kZWFkbGluZSA/IFByb2JhYmx5IGVhc3kgZm9yIG5vdw0KPiArIyAgICAgIHRvIGFz c3VtZSBkZWFkbGluZSBhbmQgbGF0ZXIgaGF2ZSBhIG1hcHBpbmcgZmlsZSB0byBlbmFibGUNCj4g KyMgICAgICBtcS1kZWFkbGluZSBmb3Igc3BlY2lmaWMgc2VyaWFsIGRldmljZXM/DQo+ICtBQ1RJ T049PSJhZGR8Y2hhbmdlIiwgS0VSTkVMPT0ic2QqWyEwLTldIiwgQVRUUlN7cXVldWUvem9uZWR9 PT0iaG9zdC1tYW5hZ2VkIiwgXA0KPiArCUFUVFJ7cXVldWUvc2NoZWR1bGVyfT0iZGVhZGxpbmUi DQo+ICsNCj4gKyMgMykgQmxhY2tsaXN0IGYyZnMgZGV2aWNlcyBhcyBvZiB2NC4xMA0KPiArIyBX ZSBkb24ndCBoYXZlIHRvIHJ1biBkbXNldHVwIG9uIG9uIGRpc2tzIHdoZXJlIHlvdSB3YW50IHRv IHVzZSBmMmZzLCBzbyB5b3UNCj4gKyMgY2FuIHVzZSB0aGlzIHJ1bGUgdG8gc2tpcCBkbXNldHVw IGZvciBpdC4gRmlyc3QgZ2V0IHRoZSBzZXJpYWwgc2hvcnQgbnVtYmVyLg0KPiArIw0KPiArIwl1 ZGV2YWRtIGluZm8gLS1uYW1lPS9kZXYvc2RhICB8IGdyZXAgLWkgc2VyaWFsX3Nob3INCj4gKyMg WFhYOiBUbyBnZW5lcmFsaXplIHRoaXMgZm9yIGRpc3RyaWJ1dGlvbnMgY29uc2lkZXIgdXNpbmcg SU5QVVR7ZGJ9IHRvIG9yIHNvDQo+ICsjIGFuZCB0aGVuIHVzZSB0aGF0IHRvIGNoZWNrIGlmIHRo ZSBzZXJpYWwgbnVtYmVyIG1hdGNoZXMgb25lIG9uIHRoZSBkYXRhYmFzZS4NCj4gKyNBQ1RJT049 PSJhZGQiLCBTVUJTWVNURU09PSJibG9jayIsIEVOVntJRF9TRVJJQUxfU0hPUlR9PT0iWFhBMVpG RkYiLCBHT1RPPSJ6b25lX2Rpc2tfZ3JvdXBfZW5kIg0KPiArDQo+ICsjIDQpIFdlIG5lZWQgdG8g cnVuIGRtc2V0dXAgaWYgeW91IHdhbnQgdG8gdXNlIG90aGVyIGZpbGVzeXN0ZW1zDQo+ICsjDQo+ ICsjIGRtc2V0dXAgaXMgbm90IHBlcnNpc3RlbnQsIHNvIGl0IG5lZWRzIHRvIGJlIHJ1biBvbiB1 cG9uIGV2ZXJ5IGJvb3QuICBXZSB1c2UNCj4gKyMgdGhlIGRldmljZSBzZXJpYWwgbnVtYmVyIGZv ciB0aGUgL2Rldi9tYXBwZXIvIG5hbWUuDQo+ICtBQ1RJT049PSJhZGQiLCBLRVJORUw9PSJzZCpb ITAtOV0iLCBBVFRSU3txdWV1ZS96b25lZH09PSJob3N0LW1hbmFnZWQiLCBcDQo+ICsJUlVOKz0i L3NiaW4vZG1zZXR1cCBjcmVhdGUgem9uZWQtJGVudntJRF9TRVJJQUxfU0hPUlR9IC0tdGFibGUg JzAgJXN7c2l6ZX0gem9uZWQgJGRldm5vZGUnIiwgJGF0dHJ7c2l6ZX0NCj4gKw0KPiArIyA0KSBD cmVhdGUgYSBmaWxlc3lzdGVtIGZvciB0aGUgZGV2aWNlDQo+ICsjDQo+ICsjIEJlIDEwMCUgc3Vy ZSB5b3UgdXNlIC9kZXYvbWFwcGVyL3pvbmUtJFlPVVJfREVWSUNFX1NFUklBTCBmb3IgdGhlIG1r ZnMNCj4gKyMgY29tbWFuZCBhcyBvdGhlcndpc2UgdGhpbmdzIGNhbiBicmVhay4NCj4gKyMNCj4g KyMgWFhYOiBwcmV2ZW50aW5nIHRoZSBhYm92ZSBwcm9hY3RpdmVseSBpbiB0aGUga2VybmVsIHdv dWxkIGJlIGlkZWFsIGhvd2V2ZXINCj4gKyMgdGhpcyBtYXkgYmUgaGFyZC4NCj4gKyMNCj4gKyMg T25jZSB5b3UgY3JlYXRlIHRoZSBmaWxlc3lzdGVtIGl0IHdpbGwgZ2V0IGEgVVVJRC4NCj4gKyMN Cj4gKyMgRmluZCBvdXQgd2hhdCBVVUlEIGlzLCB5b3UgY2FuIGRvIHRoaXMgZm9yIGluc3RhbmNl IGlmIHlvdXIgem9uZWQgZGlzayBpcw0KPiArIyB5b3VyIHNlY29uZCBkZXZpY2UtbWFwcGVyIGRl dmljZSwgaWUgZG0tMSBieToNCj4gKyMNCj4gKyMgCWxzIC1sIC9kZXYvZGlzay9ieS11dWlkL2Rt LTENCj4gKyMNCj4gKyMgVG8gZmlndXJlIG91dCB3aGljaCBkbS0kbnVtYmVyIGl0IGlzLCB1c2Ug ZG1zZXR1cCBpbmZvLCB0aGUgbWlub3IgbnVtYmVyDQo+ICsjIGlzIHRoZSAkbnVtYmVyLg0KPiAr Iw0KPiArIyA1KSBBZGQgYW4gZXRyeSBpbiAvZXRjL2ZzdGFiIHdpdGggbm9mYWlsIGZvciBleGFt cGxlOg0KPiArIw0KPiArIyBVVUlEPTk5OTk5OTk5LWFhYWEtYmJiYi1jMTIzNGFhYWFiYmIzMzQ1 NiAvbWVkaWEvbW9uc3RlciB4ZnMgbm9mYWlsIDAgMA0KPiArIw0KPiArIyBub2ZhaWwgd2lsbCBl bnN1cmUgc3lzdGVtIGJvb3RzIGZpbmUgZXZlbiBpZiB5b3UgYm9vdCBpbnRvIGEga2VybmVsIHdo aWNoDQo+ICsjIGxhY2tzIHN1cHBvcnQgZm9yIHRoZSBkZXZpY2UgYW5kIHNvIGl0IGlzIG5vdCBm b3VuZC4gU2luY2UgdGhlIFVVSUQgd2lsbA0KPiArIyBhbHdheXMgbWF0Y2ggdGhlIGRldmljZSB3 ZSBkb24ndCBjYXJlIGlmIHRoZSBkZXZpY2UgbW92ZXMgYXJvdW5kIHRoZSBidXMNCj4gKyMgb24g dGhlIHN5c3RlbS4gV2UganVzdCBuZWVkIHRvIGdldCB0aGUgVVVJRCBvbmNlLg0KPiArDQo+ICtM QUJFTD0iem9uZV9kaXNrX2dyb3VwX2VuZCINCg0KQXBwbGllZC4gVGhhbmtzIEx1aXMgIQ0KDQot LSANCkRhbWllbiBMZSBNb2FsLA0KV2VzdGVybiBEaWdpdGFs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup @ 2018-06-14 10:01 ` Damien Le Moal 0 siblings, 0 replies; 22+ messages in thread From: Damien Le Moal @ 2018-06-14 10:01 UTC (permalink / raw) To: Luis R. Rodriguez Cc: hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block On 6/14/18 09:11, Luis R. Rodriguez wrote: > Setting up a zoned disks in a generic form is not so trivial. There > is also quite a bit of tribal knowledge with these devices which is not > easy to find. > > The currently supplied demo script works but it is not generic enough to be > practical for Linux distributions or even developers which often move > from one kernel to another. > > This tries to put a bit of this tribal knowledge into an initial udev > rule for development with the hopes Linux distributions can later > deploy. Three rule are added. One rule is optional for now, it should be > extended later to be more distribution-friendly and then I think this > may be ready for consideration for integration on distributions. > > 1) scheduler setup > 2) backlist f2fs devices > 3) run dmsetup for the rest of devices > > Note that this udev rule will not work well if you want to use a disk > with f2fs on part of the disk and another filesystem on another part of > the disk. That setup will require manual love so these setups can use > the same backlist on rule 2). > > Its not widely known for instance that as of v4.16 it is mandated to use > either deadline or the mq-deadline scheduler for *all* SMR drivers. Its > also been determined that the Linux kernel is not the place to set this up, > so a udev rule *is required* as per latest discussions. This is the > first rule we add. > > Furthermore if you are *not* using f2fs you always have to run dmsetup. > dmsetups do not persist, so you currently *always* have to run a custom > sort of script, which is not ideal for Linux distributions. We can invert > this logic into a udev rule to enable users to blacklist disks they know they > want to use f2fs for. This the second optional rule. This blacklisting > can be generalized further in the future with an exception list file, for > instance using INPUT{db} or the like. > > The third and final rule added then runs dmsetup for the rest of the disks > using the disk serial number for the new device mapper name. > > Note that it is currently easy for users to make a mistake and run mkfs > on the the original disk, not the /dev/mapper/ device for non f2fs > arrangements. If that is done experience shows things can easily fall > apart with alignment *eventually*. We have no generic way today to > error out on this condition and proactively prevent this. > > Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> > --- > README | 10 +++++- > udev/99-zoned-disks.rules | 78 +++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 87 insertions(+), 1 deletion(-) > create mode 100644 udev/99-zoned-disks.rules > > diff --git a/README b/README > index 65e96c34fd04..f49541eaabc8 100644 > --- a/README > +++ b/README > @@ -168,7 +168,15 @@ Options: > reclaiming random zones if the percentage of > free random data zones falls below <perc>. > > -V. Example scripts > +V. Udev zone disk deployment > +============================ > + > +A udev rule is provided which enables you to set the IO scheduler, blacklist > +driver to run dmsetup, and runs dmsetup for the rest of the zone drivers. > +If you use this udev rule the below script is not needed. Be sure to mkfs only > +on the resulting /dev/mapper/zone-$serial device you end up with. > + > +VI. Example scripts > ================== > > [[ > diff --git a/udev/99-zoned-disks.rules b/udev/99-zoned-disks.rules > new file mode 100644 > index 000000000000..e19b738dcc0e > --- /dev/null > +++ b/udev/99-zoned-disks.rules > @@ -0,0 +1,78 @@ > +# To use a zone disks first thing you need to: > +# > +# 1) Enable zone disk support in your kernel > +# 2) Use the deadline or mq-deadline scheduler for it - mandated as of v4.16 > +# 3) Blacklist devices dedicated for f2fs as of v4.10 > +# 4) Run dmsetup other disks > +# 5) Create the filesystem -- NOTE: use mkfs /dev/mapper/zone-serial if > +# you enabled use dmsetup on the disk. > +# 6) Consider using nofail mount option in case you run an supported kernel > +# > +# You can use this udev rules file for 2) 3) and 4). Further details below. > +# > +# 1) Enable zone disk support in your kernel > +# > +# o CONFIG_BLK_DEV_ZONED > +# o CONFIG_DM_ZONED > +# > +# This will let the kernel actually see these devices, ie, via fdisk /dev/sda > +# for instance. Run: > +# > +# dmzadm --format /dev/sda > + > +# 2) Set deadline or mq-deadline for all disks which are zoned > +# > +# Zoned disks can only work with the deadline or mq-deadline scheduler. This is > +# mandated for all SMR drives since v4.16. It has been determined this must be > +# done through a udev rule, and the kernel should not set this up for disks. > +# This magic will have to live for *all* zoned disks. > +# XXX: what about distributions that want mq-deadline ? Probably easy for now > +# to assume deadline and later have a mapping file to enable > +# mq-deadline for specific serial devices? > +ACTION=="add|change", KERNEL=="sd*[!0-9]", ATTRS{queue/zoned}=="host-managed", \ > + ATTR{queue/scheduler}="deadline" > + > +# 3) Blacklist f2fs devices as of v4.10 > +# We don't have to run dmsetup on on disks where you want to use f2fs, so you > +# can use this rule to skip dmsetup for it. First get the serial short number. > +# > +# udevadm info --name=/dev/sda | grep -i serial_shor > +# XXX: To generalize this for distributions consider using INPUT{db} to or so > +# and then use that to check if the serial number matches one on the database. > +#ACTION=="add", SUBSYSTEM=="block", ENV{ID_SERIAL_SHORT}=="XXA1ZFFF", GOTO="zone_disk_group_end" > + > +# 4) We need to run dmsetup if you want to use other filesystems > +# > +# dmsetup is not persistent, so it needs to be run on upon every boot. We use > +# the device serial number for the /dev/mapper/ name. > +ACTION=="add", KERNEL=="sd*[!0-9]", ATTRS{queue/zoned}=="host-managed", \ > + RUN+="/sbin/dmsetup create zoned-$env{ID_SERIAL_SHORT} --table '0 %s{size} zoned $devnode'", $attr{size} > + > +# 4) Create a filesystem for the device > +# > +# Be 100% sure you use /dev/mapper/zone-$YOUR_DEVICE_SERIAL for the mkfs > +# command as otherwise things can break. > +# > +# XXX: preventing the above proactively in the kernel would be ideal however > +# this may be hard. > +# > +# Once you create the filesystem it will get a UUID. > +# > +# Find out what UUID is, you can do this for instance if your zoned disk is > +# your second device-mapper device, ie dm-1 by: > +# > +# ls -l /dev/disk/by-uuid/dm-1 > +# > +# To figure out which dm-$number it is, use dmsetup info, the minor number > +# is the $number. > +# > +# 5) Add an etry in /etc/fstab with nofail for example: > +# > +# UUID=99999999-aaaa-bbbb-c1234aaaabbb33456 /media/monster xfs nofail 0 0 > +# > +# nofail will ensure system boots fine even if you boot into a kernel which > +# lacks support for the device and so it is not found. Since the UUID will > +# always match the device we don't care if the device moves around the bus > +# on the system. We just need to get the UUID once. > + > +LABEL="zone_disk_group_end" Applied. Thanks Luis ! -- Damien Le Moal, Western Digital ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 10:01 ` Damien Le Moal @ 2018-06-14 13:39 ` Bart Van Assche -1 siblings, 0 replies; 22+ messages in thread From: Bart Van Assche @ 2018-06-14 13:39 UTC (permalink / raw) To: Damien Le Moal, mcgrof Cc: mwilck, linux-block, agk, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal, zren T24gVGh1LCAyMDE4LTA2LTE0IGF0IDEwOjAxICswMDAwLCBEYW1pZW4gTGUgTW9hbCB3cm90ZToN Cj4gQXBwbGllZC4gVGhhbmtzIEx1aXMgIQ0KDQpIZWxsbyBEYW1pZW4sDQoNCkNhbiB0aGlzIHN0 aWxsIGJlIHVuZG9uZT8gSSBhZ3JlZSB3aXRoIE1pa2UgdGhhdCBpdCdzIHdyb25nIHRvIGludm9r ZQ0KIi9zYmluL2Rtc2V0dXAgY3JlYXRlIC4uLiB6b25lZCAuLi4iIGZyb20gYSB1ZGV2IHJ1bGUu DQoNClRoYW5rcywNCg0KQmFydC4NCg0KDQoNCg== ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup @ 2018-06-14 13:39 ` Bart Van Assche 0 siblings, 0 replies; 22+ messages in thread From: Bart Van Assche @ 2018-06-14 13:39 UTC (permalink / raw) To: Damien Le Moal, mcgrof Cc: mwilck, linux-block, agk, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal, zren On Thu, 2018-06-14 at 10:01 +0000, Damien Le Moal wrote: > Applied. Thanks Luis ! Hello Damien, Can this still be undone? I agree with Mike that it's wrong to invoke "/sbin/dmsetup create ... zoned ..." from a udev rule. Thanks, Bart. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 13:39 ` Bart Van Assche (?) @ 2018-06-14 13:42 ` Christoph Hellwig 2018-06-15 11:07 ` Martin Wilck -1 siblings, 1 reply; 22+ messages in thread From: Christoph Hellwig @ 2018-06-14 13:42 UTC (permalink / raw) To: Bart Van Assche Cc: Damien Le Moal, mcgrof, mwilck, linux-block, agk, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal, zren On Thu, Jun 14, 2018 at 01:39:50PM +0000, Bart Van Assche wrote: > On Thu, 2018-06-14 at 10:01 +0000, Damien Le Moal wrote: > > Applied. Thanks Luis ! > > Hello Damien, > > Can this still be undone? I agree with Mike that it's wrong to invoke > "/sbin/dmsetup create ... zoned ..." from a udev rule. Yes. We'll really need to verfify the device has dm-zoned metadata first. Preferably including a uuid for stable device naming. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 13:42 ` Christoph Hellwig @ 2018-06-15 11:07 ` Martin Wilck 0 siblings, 0 replies; 22+ messages in thread From: Martin Wilck @ 2018-06-15 11:07 UTC (permalink / raw) To: Christoph Hellwig, Bart Van Assche Cc: Damien Le Moal, mcgrof, linux-block, agk, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal On Thu, 2018-06-14 at 06:42 -0700, Christoph Hellwig wrote: > On Thu, Jun 14, 2018 at 01:39:50PM +0000, Bart Van Assche wrote: > > On Thu, 2018-06-14 at 10:01 +0000, Damien Le Moal wrote: > > > Applied. Thanks Luis ! > > > > Hello Damien, > > > > Can this still be undone? I agree with Mike that it's wrong to > > invoke > > "/sbin/dmsetup create ... zoned ..." from a udev rule. > > Yes. We'll really need to verfify the device has dm-zoned metadata > first. Preferably including a uuid for stable device naming. libblkid would be the central hub for metadata discovery, so perhaps a patch should be made to make libblkid dm-zoned-aware. Anyway, as Damien explained, dmzoned bails out if it doesn't find matching meta data, so AFAICS, little harm is done by calling it for a SMR device in host-managed mode. I fail to get the point why this would be wrong in general - what's the difference to e.g. calling "mdadm -I"? Regards Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup @ 2018-06-15 11:07 ` Martin Wilck 0 siblings, 0 replies; 22+ messages in thread From: Martin Wilck @ 2018-06-15 11:07 UTC (permalink / raw) To: Christoph Hellwig, Bart Van Assche Cc: Damien Le Moal, mcgrof, linux-block, agk, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal On Thu, 2018-06-14 at 06:42 -0700, Christoph Hellwig wrote: > On Thu, Jun 14, 2018 at 01:39:50PM +0000, Bart Van Assche wrote: > > On Thu, 2018-06-14 at 10:01 +0000, Damien Le Moal wrote: > > > Applied. Thanks Luis ! > > > > Hello Damien, > > > > Can this still be undone? I agree with Mike that it's wrong to > > invoke > > "/sbin/dmsetup create ... zoned ..." from a udev rule. > > Yes. We'll really need to verfify the device has dm-zoned metadata > first. Preferably including a uuid for stable device naming. libblkid would be the central hub for metadata discovery, so perhaps a patch should be made to make libblkid dm-zoned-aware. Anyway, as Damien explained, dmzoned bails out if it doesn't find matching meta data, so AFAICS, little harm is done by calling it for a SMR device in host-managed mode. I fail to get the point why this would be wrong in general - what's the difference to e.g. calling "mdadm -I"? Regards Martin -- Dr. Martin Wilck <mwilck@suse.com>, Tel. +49 (0)911 74053 2107 SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 0:11 [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup Luis R. Rodriguez 2018-06-14 10:01 ` Damien Le Moal @ 2018-06-14 12:38 ` Mike Snitzer 2018-06-14 16:23 ` Bart Van Assche ` (2 more replies) 2018-06-14 16:19 ` Bart Van Assche 2 siblings, 3 replies; 22+ messages in thread From: Mike Snitzer @ 2018-06-14 12:38 UTC (permalink / raw) To: Luis R. Rodriguez Cc: damien.lemoal, hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block On Wed, Jun 13 2018 at 8:11pm -0400, Luis R. Rodriguez <mcgrof@kernel.org> wrote: > Setting up a zoned disks in a generic form is not so trivial. There > is also quite a bit of tribal knowledge with these devices which is not > easy to find. > > The currently supplied demo script works but it is not generic enough to be > practical for Linux distributions or even developers which often move > from one kernel to another. > > This tries to put a bit of this tribal knowledge into an initial udev > rule for development with the hopes Linux distributions can later > deploy. Three rule are added. One rule is optional for now, it should be > extended later to be more distribution-friendly and then I think this > may be ready for consideration for integration on distributions. > > 1) scheduler setup This is wrong.. if zoned devices are so dependent on deadline or mq-deadline then the kernel should allow them to be hardcoded. I know Jens removed the API to do so but the fact that drivers need to rely on hacks like this udev rule to get a functional device is proof we need to allow drivers to impose the scheduler used. > 2) backlist f2fs devices There should porbably be support in dm-zoned for detecting whether a zoned device was formatted with f2fs (assuming there is a known f2fs superblock)? > 3) run dmsetup for the rest of devices automagically running dmsetup directly from udev to create a dm-zoned target is very much wrong. It just gets in the way of proper support that should be add to appropriate tools that admins use to setup their zoned devices. For instance, persistent use of dm-zoned target should be made reliable with a volume manager.. In general this udev script is unwelcome and makes things way worse for the long-term success of zoned devices. I don't dispute there is an obvious void for how to properly setup zoned devices, but this script is _not_ what should fill that void. So a heartfelt: Nacked-by: Mike Snitzer <snitzer@redhat.com> ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 12:38 ` Mike Snitzer @ 2018-06-14 16:23 ` Bart Van Assche 2018-06-14 17:37 ` Luis R. Rodriguez 2018-06-15 9:00 ` Damien Le Moal 2 siblings, 0 replies; 22+ messages in thread From: Bart Van Assche @ 2018-06-14 16:23 UTC (permalink / raw) To: snitzer, mcgrof Cc: mwilck, linux-block, agk, Damien Le Moal, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal, zren T24gVGh1LCAyMDE4LTA2LTE0IGF0IDA4OjM4IC0wNDAwLCBNaWtlIFNuaXR6ZXIgd3JvdGU6DQo+ IE9uIFdlZCwgSnVuIDEzIDIwMTggYXQgIDg6MTFwbSAtMDQwMCwNCj4gTHVpcyBSLiBSb2RyaWd1 ZXogPG1jZ3JvZkBrZXJuZWwub3JnPiB3cm90ZToNCj4gPiAxKSBzY2hlZHVsZXIgc2V0dXANCj4g DQo+IFRoaXMgaXMgd3JvbmcuLiBpZiB6b25lZCBkZXZpY2VzIGFyZSBzbyBkZXBlbmRlbnQgb24g ZGVhZGxpbmUgb3INCj4gbXEtZGVhZGxpbmUgdGhlbiB0aGUga2VybmVsIHNob3VsZCBhbGxvdyB0 aGVtIHRvIGJlIGhhcmRjb2RlZC4gIEkga25vdw0KPiBKZW5zIHJlbW92ZWQgdGhlIEFQSSB0byBk byBzbyBidXQgdGhlIGZhY3QgdGhhdCBkcml2ZXJzIG5lZWQgdG8gcmVseSBvbg0KPiBoYWNrcyBs aWtlIHRoaXMgdWRldiBydWxlIHRvIGdldCBhIGZ1bmN0aW9uYWwgZGV2aWNlIGlzIHByb29mIHdl IG5lZWQgdG8NCj4gYWxsb3cgZHJpdmVycyB0byBpbXBvc2UgdGhlIHNjaGVkdWxlciB1c2VkLg0K DQpIZWxsbyBNaWtlLA0KDQpBcyB5b3Uga25vdyB0aGUgTGludXgga2VybmVsIGJsb2NrIGxheWVy IHN0YWNrIGNhbiByZW9yZGVyIHJlcXVlc3RzLiBIb3dldmVyLA0KZm9yIHpvbmVkIGJsb2NrIGRl dmljZXMgaXQgaXMgZXNzZW50aWFsIHRoYXQgdGhlIGJsb2NrIGRldmljZSByZWNlaXZlcyB3cml0 ZQ0KcmVxdWVzdHMgaW4gdGhlIHNhbWUgb3JkZXIgYXMgdGhlc2Ugd2VyZSBzdWJtaXR0ZWQgYnkg dGhlICh1c2VyIHNwYWNlKQ0KYXBwbGljYXRpb24gb3IgYnkgdGhlIChrZXJuZWwpIGZpbGVzeXN0 ZW0uIEFmdGVyIGEgbG9uZyBkZWJhdGUgdGhlIGNob2ljZSB3YXMNCm1hZGUgdG8gbWFrZSBJL08g c2NoZWR1bGVycyByZXNwb25zaWJsZSBmb3IgZ3VhcmFudGVlaW5nIHRvIHByZXNlcnZlIHRoZSB3 cml0ZQ0Kb3JkZXIuIFRvZGF5IG9ubHkgdGhlIGRlYWRsaW5lIHNjaGVkdWxlciBndWFyYW50ZWVz IHRoYXQgdGhlIHdyaXRlIG9yZGVyIGlzDQpwcmVzZXJ2ZWQuIEhlbmNlIHRoZSB1ZGV2IHJ1bGUg dGhhdCBzZXRzIHRoZSBkZWFkbGluZSBzY2hlZHVsZXIgZm9yIHpvbmVkDQpibG9jayBkZXZpY2Vz Lg0KDQpCYXJ0Lg0KDQoNCg0KDQo= ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup @ 2018-06-14 16:23 ` Bart Van Assche 0 siblings, 0 replies; 22+ messages in thread From: Bart Van Assche @ 2018-06-14 16:23 UTC (permalink / raw) To: snitzer, mcgrof Cc: mwilck, linux-block, agk, Damien Le Moal, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal, zren On Thu, 2018-06-14 at 08:38 -0400, Mike Snitzer wrote: > On Wed, Jun 13 2018 at 8:11pm -0400, > Luis R. Rodriguez <mcgrof@kernel.org> wrote: > > 1) scheduler setup > > This is wrong.. if zoned devices are so dependent on deadline or > mq-deadline then the kernel should allow them to be hardcoded. I know > Jens removed the API to do so but the fact that drivers need to rely on > hacks like this udev rule to get a functional device is proof we need to > allow drivers to impose the scheduler used. Hello Mike, As you know the Linux kernel block layer stack can reorder requests. However, for zoned block devices it is essential that the block device receives write requests in the same order as these were submitted by the (user space) application or by the (kernel) filesystem. After a long debate the choice was made to make I/O schedulers responsible for guaranteeing to preserve the write order. Today only the deadline scheduler guarantees that the write order is preserved. Hence the udev rule that sets the deadline scheduler for zoned block devices. Bart. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 12:38 ` Mike Snitzer 2018-06-14 16:23 ` Bart Van Assche @ 2018-06-14 17:37 ` Luis R. Rodriguez 2018-06-14 17:46 ` Luis R. Rodriguez 2018-06-14 17:58 ` Mike Snitzer 2018-06-15 9:00 ` Damien Le Moal 2 siblings, 2 replies; 22+ messages in thread From: Luis R. Rodriguez @ 2018-06-14 17:37 UTC (permalink / raw) To: Mike Snitzer Cc: Luis R. Rodriguez, damien.lemoal, hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block On Thu, Jun 14, 2018 at 08:38:06AM -0400, Mike Snitzer wrote: > On Wed, Jun 13 2018 at 8:11pm -0400, > Luis R. Rodriguez <mcgrof@kernel.org> wrote: > > > Setting up a zoned disks in a generic form is not so trivial. There > > is also quite a bit of tribal knowledge with these devices which is not > > easy to find. > > > > The currently supplied demo script works but it is not generic enough to be > > practical for Linux distributions or even developers which often move > > from one kernel to another. > > > > This tries to put a bit of this tribal knowledge into an initial udev > > rule for development with the hopes Linux distributions can later > > deploy. Three rule are added. One rule is optional for now, it should be > > extended later to be more distribution-friendly and then I think this > > may be ready for consideration for integration on distributions. > > > > 1) scheduler setup > > This is wrong.. if zoned devices are so dependent on deadline or > mq-deadline then the kernel should allow them to be hardcoded. I know > Jens removed the API to do so but the fact that drivers need to rely on > hacks like this udev rule to get a functional device is proof we need to > allow drivers to impose the scheduler used. This is the point to the patch as well, I actually tend to agree with you, and I had tried to draw up a patch to do just that, however its *not* possible today to do this and would require some consensus. So from what I can tell we *have* to live with this one or a form of it. Ie a file describing which disk serial gets deadline and which one gets mq-deadline. Jens? Anyway, let's assume this is done in the kernel, which one would use deadline, which one would use mq-deadline? > > 2) backlist f2fs devices > > There should porbably be support in dm-zoned for detecting whether a > zoned device was formatted with f2fs (assuming there is a known f2fs > superblock)? Not sure what you mean. Are you suggesting we always setup dm-zoned for all zoned disks and just make an excemption on dm-zone code to somehow use the disk directly if a filesystem supports zoned disks directly somehow? f2fs does not require dm-zoned. What would be required is a bit more complex given one could dedicate portions of the disk to f2fs and other portions to another filesystem, which would require dm-zoned. Also filesystems which *do not* support zoned disks should *not* be allowing direct setup. Today that's all filesystems other than f2fs, in the future that may change. Those are bullets we are allowing to trigger for users just waiting to shot themselves on the foot with. So who's going to work on all the above? The point of the udev script is to illustrate the pains to properly deploy zoned disks on distributions today and without a roadmap... this is what at least I need on my systems today to reasonably deploy these disks for my own development. Consensus is indeed needed for a broader picture. > > 3) run dmsetup for the rest of devices > > automagically running dmsetup directly from udev to create a dm-zoned > target is very much wrong. It just gets in the way of proper support > that should be add to appropriate tools that admins use to setup their > zoned devices. For instance, persistent use of dm-zoned target should > be made reliable with a volume manager.. Ah yes, but who's working on that? How long will it take? I agree it is odd to expect one to use dmsetup and then use a volume manager on top of it, if we can just add proper support onto the volume manager... then that's a reasonable way to go. But *we're not there* yet, and as-is today, what is described in the udev script is the best we can do for a generic setup. > In general this udev script is unwelcome and makes things way worse for > the long-term success of zoned devices. dm-zoned-tools does not acknowledge in any way a roadmap, and just provides a script, which IMHO is less generic and less distribution friendly. Having a udev rule in place to demonstrate the current state of affairs IMHO is more scalable demonstrates the issues better than the script. If we have an agreed upon long term strategy lets document that. But from what I gather we are not even in consensus with regards to the scheduler stuff. If we have consensus on the other stuff lets document that as dm-zoned-tools is the only place I think folks could find to reasonably deploy these things. > I don't dispute there is an obvious void for how to properly setup zoned > devices, but this script is _not_ what should fill that void. Good to know! Again, consider it as an alternative to the script. I'm happy to adapt the language and supply it only as an example script developers can use, but we can't leave users hanging as well. Let's at least come up with a plan which we seem to agree on and document that. Luis ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 17:37 ` Luis R. Rodriguez @ 2018-06-14 17:46 ` Luis R. Rodriguez 2018-06-14 17:58 ` Mike Snitzer 1 sibling, 0 replies; 22+ messages in thread From: Luis R. Rodriguez @ 2018-06-14 17:46 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Mike Snitzer, damien.lemoal, hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block On Thu, Jun 14, 2018 at 07:37:19PM +0200, Luis R. Rodriguez wrote: > > Ie a file describing which > disk serial gets deadline and which one gets mq-deadline. > Anyway, let's assume this is done in the kernel, which one would use deadline, > which one would use mq-deadline? Nevermind this, deadline will work, Bart already stated why. Luis ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 17:37 ` Luis R. Rodriguez 2018-06-14 17:46 ` Luis R. Rodriguez @ 2018-06-14 17:58 ` Mike Snitzer 2018-06-15 9:59 ` Damien Le Moal 1 sibling, 1 reply; 22+ messages in thread From: Mike Snitzer @ 2018-06-14 17:58 UTC (permalink / raw) To: Luis R. Rodriguez Cc: damien.lemoal, hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block On Thu, Jun 14 2018 at 1:37pm -0400, Luis R. Rodriguez <mcgrof@kernel.org> wrote: > On Thu, Jun 14, 2018 at 08:38:06AM -0400, Mike Snitzer wrote: > > On Wed, Jun 13 2018 at 8:11pm -0400, > > Luis R. Rodriguez <mcgrof@kernel.org> wrote: > > > > > Setting up a zoned disks in a generic form is not so trivial. There > > > is also quite a bit of tribal knowledge with these devices which is not > > > easy to find. > > > > > > The currently supplied demo script works but it is not generic enough to be > > > practical for Linux distributions or even developers which often move > > > from one kernel to another. > > > > > > This tries to put a bit of this tribal knowledge into an initial udev > > > rule for development with the hopes Linux distributions can later > > > deploy. Three rule are added. One rule is optional for now, it should be > > > extended later to be more distribution-friendly and then I think this > > > may be ready for consideration for integration on distributions. > > > > > > 1) scheduler setup > > > > This is wrong.. if zoned devices are so dependent on deadline or > > mq-deadline then the kernel should allow them to be hardcoded. I know > > Jens removed the API to do so but the fact that drivers need to rely on > > hacks like this udev rule to get a functional device is proof we need to > > allow drivers to impose the scheduler used. > > This is the point to the patch as well, I actually tend to agree with you, > and I had tried to draw up a patch to do just that, however its *not* possible > today to do this and would require some consensus. So from what I can tell > we *have* to live with this one or a form of it. Ie a file describing which > disk serial gets deadline and which one gets mq-deadline. > > Jens? > > Anyway, let's assume this is done in the kernel, which one would use deadline, > which one would use mq-deadline? The zoned storage driver needs to make that call based on what mode it is in. If it is using blk-mq then it selects mq-deadline, otherwise deadline. > > > 2) backlist f2fs devices > > > > There should porbably be support in dm-zoned for detecting whether a > > zoned device was formatted with f2fs (assuming there is a known f2fs > > superblock)? > > Not sure what you mean. Are you suggesting we always setup dm-zoned for > all zoned disks and just make an excemption on dm-zone code to somehow > use the disk directly if a filesystem supports zoned disks directly somehow? No, I'm saying that a udev rule wouldn't be needed if dm-zoned just errored out if asked to consume disks that already have an f2fs superblock. And existing filesystems should get conflicting superblock awareness "for free" if blkid or whatever is trained to be aware of f2fs's superblock. > f2fs does not require dm-zoned. What would be required is a bit more complex > given one could dedicate portions of the disk to f2fs and other portions to > another filesystem, which would require dm-zoned. > > Also filesystems which *do not* support zoned disks should *not* be allowing > direct setup. Today that's all filesystems other than f2fs, in the future > that may change. Those are bullets we are allowing to trigger for users > just waiting to shot themselves on the foot with. > > So who's going to work on all the above? It should take care of itself if existing tools are trained to be aware of new signatures. E.g. ext4 and xfs already are aware of one another so that you cannot reformat a device with the other unless force is given. Same kind of mutual exclussion needs to happen for zoned devices. So the zoned device tools, dm-zoned, f2fs, whatever.. they need to be updated to not step on each others toes. And other filesystems' tools need to be updated to be zoned device aware. > The point of the udev script is to illustrate the pains to properly deploy > zoned disks on distributions today and without a roadmap... this is what > at least I need on my systems today to reasonably deploy these disks for > my own development. > > Consensus is indeed needed for a broader picture. Yeap. > > > 3) run dmsetup for the rest of devices > > > > automagically running dmsetup directly from udev to create a dm-zoned > > target is very much wrong. It just gets in the way of proper support > > that should be add to appropriate tools that admins use to setup their > > zoned devices. For instance, persistent use of dm-zoned target should > > be made reliable with a volume manager.. > > Ah yes, but who's working on that? How long will it take? No idea, as is (from my vantage point) there is close to zero demand for zoned devices. It won't be a priority until enough customers are asking for it. > I agree it is odd to expect one to use dmsetup and then use a volume manager on > top of it, if we can just add proper support onto the volume manager... then > that's a reasonable way to go. > > But *we're not there* yet, and as-is today, what is described in the udev > script is the best we can do for a generic setup. Just because doing things right takes work doesn't mean it makes sense to elevate this udev script to be packaged in some upstream project like udev or whatever. But if SUSE or some other distro wants to ship it that is fine. > > In general this udev script is unwelcome and makes things way worse for > > the long-term success of zoned devices. > > dm-zoned-tools does not acknowledge in any way a roadmap, and just provides > a script, which IMHO is less generic and less distribution friendly. Having > a udev rule in place to demonstrate the current state of affairs IMHO is > more scalable demonstrates the issues better than the script. > > If we have an agreed upon long term strategy lets document that. But from > what I gather we are not even in consensus with regards to the scheduler > stuff. If we have consensus on the other stuff lets document that as > dm-zoned-tools is the only place I think folks could find to reasonably > deploy these things. I'm sure Damien and others will have something to say here. > > I don't dispute there is an obvious void for how to properly setup zoned > > devices, but this script is _not_ what should fill that void. > > Good to know! Again, consider it as an alternative to the script. > > I'm happy to adapt the language and supply it only as an example script > developers can use, but we can't leave users hanging as well. Let's at > least come up with a plan which we seem to agree on and document that. Best to try to get Damien and others more invested in zoned devices to help you take up your cause. I think it is worthwhile to develop a strategy. But it needs to be done in terms of the norms of the existing infrastructure we all make use of today. So first step is making existing tools zoned device aware (even if to reject such devices). Mike ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 17:58 ` Mike Snitzer @ 2018-06-15 9:59 ` Damien Le Moal 0 siblings, 0 replies; 22+ messages in thread From: Damien Le Moal @ 2018-06-15 9:59 UTC (permalink / raw) To: Mike Snitzer, Luis R. Rodriguez Cc: hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block TWlrZSwNCg0KT24gNi8xNS8xOCAwMjo1OCwgTWlrZSBTbml0emVyIHdyb3RlOg0KPiBPbiBUaHUs IEp1biAxNCAyMDE4IGF0ICAxOjM3cG0gLTA0MDAsDQo+IEx1aXMgUi4gUm9kcmlndWV6IDxtY2dy b2ZAa2VybmVsLm9yZz4gd3JvdGU6DQo+IA0KPj4gT24gVGh1LCBKdW4gMTQsIDIwMTggYXQgMDg6 Mzg6MDZBTSAtMDQwMCwgTWlrZSBTbml0emVyIHdyb3RlOg0KPj4+IE9uIFdlZCwgSnVuIDEzIDIw MTggYXQgIDg6MTFwbSAtMDQwMCwNCj4+PiBMdWlzIFIuIFJvZHJpZ3VleiA8bWNncm9mQGtlcm5l bC5vcmc+IHdyb3RlOg0KPj4+DQo+Pj4+IFNldHRpbmcgdXAgYSB6b25lZCBkaXNrcyBpbiBhIGdl bmVyaWMgZm9ybSBpcyBub3Qgc28gdHJpdmlhbC4gVGhlcmUNCj4+Pj4gaXMgYWxzbyBxdWl0ZSBh IGJpdCBvZiB0cmliYWwga25vd2xlZGdlIHdpdGggdGhlc2UgZGV2aWNlcyB3aGljaCBpcyBub3QN Cj4+Pj4gZWFzeSB0byBmaW5kLg0KPj4+Pg0KPj4+PiBUaGUgY3VycmVudGx5IHN1cHBsaWVkIGRl bW8gc2NyaXB0IHdvcmtzIGJ1dCBpdCBpcyBub3QgZ2VuZXJpYyBlbm91Z2ggdG8gYmUNCj4+Pj4g cHJhY3RpY2FsIGZvciBMaW51eCBkaXN0cmlidXRpb25zIG9yIGV2ZW4gZGV2ZWxvcGVycyB3aGlj aCBvZnRlbiBtb3ZlDQo+Pj4+IGZyb20gb25lIGtlcm5lbCB0byBhbm90aGVyLg0KPj4+Pg0KPj4+ PiBUaGlzIHRyaWVzIHRvIHB1dCBhIGJpdCBvZiB0aGlzIHRyaWJhbCBrbm93bGVkZ2UgaW50byBh biBpbml0aWFsIHVkZXYNCj4+Pj4gcnVsZSBmb3IgZGV2ZWxvcG1lbnQgd2l0aCB0aGUgaG9wZXMg TGludXggZGlzdHJpYnV0aW9ucyBjYW4gbGF0ZXINCj4+Pj4gZGVwbG95LiBUaHJlZSBydWxlIGFy ZSBhZGRlZC4gT25lIHJ1bGUgaXMgb3B0aW9uYWwgZm9yIG5vdywgaXQgc2hvdWxkIGJlDQo+Pj4+ IGV4dGVuZGVkIGxhdGVyIHRvIGJlIG1vcmUgZGlzdHJpYnV0aW9uLWZyaWVuZGx5IGFuZCB0aGVu IEkgdGhpbmsgdGhpcw0KPj4+PiBtYXkgYmUgcmVhZHkgZm9yIGNvbnNpZGVyYXRpb24gZm9yIGlu dGVncmF0aW9uIG9uIGRpc3RyaWJ1dGlvbnMuDQo+Pj4+DQo+Pj4+IDEpIHNjaGVkdWxlciBzZXR1 cA0KPj4+DQo+Pj4gVGhpcyBpcyB3cm9uZy4uIGlmIHpvbmVkIGRldmljZXMgYXJlIHNvIGRlcGVu ZGVudCBvbiBkZWFkbGluZSBvcg0KPj4+IG1xLWRlYWRsaW5lIHRoZW4gdGhlIGtlcm5lbCBzaG91 bGQgYWxsb3cgdGhlbSB0byBiZSBoYXJkY29kZWQuICBJIGtub3cNCj4+PiBKZW5zIHJlbW92ZWQg dGhlIEFQSSB0byBkbyBzbyBidXQgdGhlIGZhY3QgdGhhdCBkcml2ZXJzIG5lZWQgdG8gcmVseSBv bg0KPj4+IGhhY2tzIGxpa2UgdGhpcyB1ZGV2IHJ1bGUgdG8gZ2V0IGEgZnVuY3Rpb25hbCBkZXZp Y2UgaXMgcHJvb2Ygd2UgbmVlZCB0bw0KPj4+IGFsbG93IGRyaXZlcnMgdG8gaW1wb3NlIHRoZSBz Y2hlZHVsZXIgdXNlZC4NCj4+DQo+PiBUaGlzIGlzIHRoZSBwb2ludCB0byB0aGUgcGF0Y2ggYXMg d2VsbCwgSSBhY3R1YWxseSB0ZW5kIHRvIGFncmVlIHdpdGggeW91LA0KPj4gYW5kIEkgaGFkIHRy aWVkIHRvIGRyYXcgdXAgYSBwYXRjaCB0byBkbyBqdXN0IHRoYXQsIGhvd2V2ZXIgaXRzICpub3Qq IHBvc3NpYmxlDQo+PiB0b2RheSB0byBkbyB0aGlzIGFuZCB3b3VsZCByZXF1aXJlIHNvbWUgY29u c2Vuc3VzLiBTbyBmcm9tIHdoYXQgSSBjYW4gdGVsbA0KPj4gd2UgKmhhdmUqIHRvIGxpdmUgd2l0 aCB0aGlzIG9uZSBvciBhIGZvcm0gb2YgaXQuIEllIGEgZmlsZSBkZXNjcmliaW5nIHdoaWNoDQo+ PiBkaXNrIHNlcmlhbCBnZXRzIGRlYWRsaW5lIGFuZCB3aGljaCBvbmUgZ2V0cyBtcS1kZWFkbGlu ZS4NCj4+DQo+PiBKZW5zPw0KPj4NCj4+IEFueXdheSwgbGV0J3MgYXNzdW1lIHRoaXMgaXMgZG9u ZSBpbiB0aGUga2VybmVsLCB3aGljaCBvbmUgd291bGQgdXNlIGRlYWRsaW5lLA0KPj4gd2hpY2gg b25lIHdvdWxkIHVzZSBtcS1kZWFkbGluZT8NCj4gDQo+IFRoZSB6b25lZCBzdG9yYWdlIGRyaXZl ciBuZWVkcyB0byBtYWtlIHRoYXQgY2FsbCBiYXNlZCBvbiB3aGF0IG1vZGUgaXQNCj4gaXMgaW4u ICBJZiBpdCBpcyB1c2luZyBibGstbXEgdGhlbiBpdCBzZWxlY3RzIG1xLWRlYWRsaW5lLCBvdGhl cndpc2UNCj4gZGVhZGxpbmUuDQoNCkFzIEJhcnQgcG9pbnRlZCBvdXQsIGRlYWRsaW5lIGlzIGFu IGFsaWFzIG9mIG1xLWRlYWRsaW5lLiBTbyB1c2luZw0KImRlYWRsaW5lIiBhcyB0aGUgc2NoZWR1 bGVyIG5hbWUgd29ya3MgaW4gYm90aCBsZWdhY3kgYW5kIG1xIGNhc2VzLg0KDQo+Pj4+IDIpIGJh Y2tsaXN0IGYyZnMgZGV2aWNlcw0KPj4+DQo+Pj4gVGhlcmUgc2hvdWxkIHBvcmJhYmx5IGJlIHN1 cHBvcnQgaW4gZG0tem9uZWQgZm9yIGRldGVjdGluZyB3aGV0aGVyIGENCj4+PiB6b25lZCBkZXZp Y2Ugd2FzIGZvcm1hdHRlZCB3aXRoIGYyZnMgKGFzc3VtaW5nIHRoZXJlIGlzIGEga25vd24gZjJm cw0KPj4+IHN1cGVyYmxvY2spPw0KPj4NCj4+IE5vdCBzdXJlIHdoYXQgeW91IG1lYW4uIEFyZSB5 b3Ugc3VnZ2VzdGluZyB3ZSBhbHdheXMgc2V0dXAgZG0tem9uZWQgZm9yDQo+PiBhbGwgem9uZWQg ZGlza3MgYW5kIGp1c3QgbWFrZSBhbiBleGNlbXB0aW9uIG9uIGRtLXpvbmUgY29kZSB0byBzb21l aG93DQo+PiB1c2UgdGhlIGRpc2sgZGlyZWN0bHkgaWYgYSBmaWxlc3lzdGVtIHN1cHBvcnRzIHpv bmVkIGRpc2tzIGRpcmVjdGx5IHNvbWVob3c/DQo+IA0KPiBObywgSSdtIHNheWluZyB0aGF0IGEg dWRldiBydWxlIHdvdWxkbid0IGJlIG5lZWRlZCBpZiBkbS16b25lZCBqdXN0DQo+IGVycm9yZWQg b3V0IGlmIGFza2VkIHRvIGNvbnN1bWUgZGlza3MgdGhhdCBhbHJlYWR5IGhhdmUgYW4gZjJmcw0K PiBzdXBlcmJsb2NrLiAgQW5kIGV4aXN0aW5nIGZpbGVzeXN0ZW1zIHNob3VsZCBnZXQgY29uZmxp Y3Rpbmcgc3VwZXJibG9jaw0KPiBhd2FyZW5lc3MgImZvciBmcmVlIiBpZiBibGtpZCBvciB3aGF0 ZXZlciBpcyB0cmFpbmVkIHRvIGJlIGF3YXJlIG9mDQo+IGYyZnMncyBzdXBlcmJsb2NrLg0KDQpX ZWxsIHRoYXQgaXMgdGhlIGNhc2UgYWxyZWFkeTogb24gc3RhcnR1cCwgZG0tem9uZWQgd2lsbCBy ZWFkIGl0cyBvd24NCm1ldGFkYXRhIGZyb20gc2VjdG9yIDAsIHNhbWUgYXMgZjJmcyB3b3VsZCBk byB3aXRoIGl0cyBzdXBlci1ibG9jay4gSWYNCnRoZSBmb3JtYXQvbWFnaWMgZG9lcyBub3QgbWF0 Y2ggZXhwZWN0ZWQgdmFsdWVzLCBkbS16b25lZCB3aWxsIGJhaWwgb3V0DQphbmQgcmV0dXJuIGFu IGVycm9yLiBkbS16b25lZCBtZXRhZGF0YSBhbmQgZjJmcyBtZXRhZGF0YSByZXNpZGUgaW4gdGhl DQpzYW1lIHBsYWNlIGFuZCBvdmVyd3JpdGUgZWFjaCBvdGhlci4gVGhlcmUgaXMgbm8gd2F5IHRv IGdldCBvbmUgd29ya2luZw0Kb24gdG9wIG9mIHRoZSBvdGhlci4gSSBkbyBub3Qgc2VlIGFueSBw b3NzaWJpbGl0eSBvZiBhIHByb2JsZW0gb24gc3RhcnR1cC4NCg0KQnV0IGRlZmluaXRlbHksIHRo ZSB1c2VyIGxhbmQgZm9ybWF0IHRvb2xzIGNhbiBzdGVwIG9uIGVhY2ggb3RoZXIgdG9lcy4NClRo YXQgbmVlZHMgZml4aW5nLg0KDQo+PiBmMmZzIGRvZXMgbm90IHJlcXVpcmUgZG0tem9uZWQuIFdo YXQgd291bGQgYmUgcmVxdWlyZWQgaXMgYSBiaXQgbW9yZSBjb21wbGV4DQo+PiBnaXZlbiBvbmUg Y291bGQgZGVkaWNhdGUgcG9ydGlvbnMgb2YgdGhlIGRpc2sgdG8gZjJmcyBhbmQgb3RoZXIgcG9y dGlvbnMgdG8NCj4+IGFub3RoZXIgZmlsZXN5c3RlbSwgd2hpY2ggd291bGQgcmVxdWlyZSBkbS16 b25lZC4NCj4+DQo+PiBBbHNvIGZpbGVzeXN0ZW1zIHdoaWNoICpkbyBub3QqIHN1cHBvcnQgem9u ZWQgZGlza3Mgc2hvdWxkICpub3QqIGJlIGFsbG93aW5nDQo+PiBkaXJlY3Qgc2V0dXAuIFRvZGF5 IHRoYXQncyBhbGwgZmlsZXN5c3RlbXMgb3RoZXIgdGhhbiBmMmZzLCBpbiB0aGUgZnV0dXJlDQo+ PiB0aGF0IG1heSBjaGFuZ2UuIFRob3NlIGFyZSBidWxsZXRzIHdlIGFyZSBhbGxvd2luZyB0byB0 cmlnZ2VyIGZvciB1c2Vycw0KPj4ganVzdCB3YWl0aW5nIHRvIHNob3QgdGhlbXNlbHZlcyBvbiB0 aGUgZm9vdCB3aXRoLg0KPj4NCj4+IFNvIHdobydzIGdvaW5nIHRvIHdvcmsgb24gYWxsIHRoZSBh Ym92ZT8NCj4gDQo+IEl0IHNob3VsZCB0YWtlIGNhcmUgb2YgaXRzZWxmIGlmIGV4aXN0aW5nIHRv b2xzIGFyZSB0cmFpbmVkIHRvIGJlIGF3YXJlDQo+IG9mIG5ldyBzaWduYXR1cmVzLiAgRS5nLiBl eHQ0IGFuZCB4ZnMgYWxyZWFkeSBhcmUgYXdhcmUgb2Ygb25lIGFub3RoZXINCj4gc28gdGhhdCB5 b3UgY2Fubm90IHJlZm9ybWF0IGEgZGV2aWNlIHdpdGggdGhlIG90aGVyIHVubGVzcyBmb3JjZSBp cw0KPiBnaXZlbi4NCj4gDQo+IFNhbWUga2luZCBvZiBtdXR1YWwgZXhjbHVzc2lvbiBuZWVkcyB0 byBoYXBwZW4gZm9yIHpvbmVkIGRldmljZXMuDQoNClllcy4NCg0KPiBTbyB0aGUgem9uZWQgZGV2 aWNlIHRvb2xzLCBkbS16b25lZCwgZjJmcywgd2hhdGV2ZXIuLiB0aGV5IG5lZWQgdG8gYmUNCj4g dXBkYXRlZCB0byBub3Qgc3RlcCBvbiBlYWNoIG90aGVycyB0b2VzLiAgQW5kIG90aGVyIGZpbGVz eXN0ZW1zJyB0b29scw0KPiBuZWVkIHRvIGJlIHVwZGF0ZWQgdG8gYmUgem9uZWQgZGV2aWNlIGF3 YXJlLg0KDQpJIHdpbGwgdXBkYXRlIGRtLXpvbmVkIHRvb2xzIHRvIGNoZWNrIGZvciBrbm93biBG UyBzdXBlcmJsb2NrcywNCnNpbWlsYXJseSB0byB3aGF0IG1rZnMuZXh0NCBhbmQgbWtmcy54ZnMg ZG8uDQoNCj4+Pj4gMykgcnVuIGRtc2V0dXAgZm9yIHRoZSByZXN0IG9mIGRldmljZXMNCj4+Pg0K Pj4+IGF1dG9tYWdpY2FsbHkgcnVubmluZyBkbXNldHVwIGRpcmVjdGx5IGZyb20gdWRldiB0byBj cmVhdGUgYSBkbS16b25lZA0KPj4+IHRhcmdldCBpcyB2ZXJ5IG11Y2ggd3JvbmcuICBJdCBqdXN0 IGdldHMgaW4gdGhlIHdheSBvZiBwcm9wZXIgc3VwcG9ydA0KPj4+IHRoYXQgc2hvdWxkIGJlIGFk ZCB0byBhcHByb3ByaWF0ZSB0b29scyB0aGF0IGFkbWlucyB1c2UgdG8gc2V0dXAgdGhlaXINCj4+ PiB6b25lZCBkZXZpY2VzLiAgRm9yIGluc3RhbmNlLCBwZXJzaXN0ZW50IHVzZSBvZiBkbS16b25l ZCB0YXJnZXQgc2hvdWxkDQo+Pj4gYmUgbWFkZSByZWxpYWJsZSB3aXRoIGEgdm9sdW1lIG1hbmFn ZXIuLg0KPj4NCj4+IEFoIHllcywgYnV0IHdobydzIHdvcmtpbmcgb24gdGhhdD8gSG93IGxvbmcg d2lsbCBpdCB0YWtlPw0KPiANCj4gTm8gaWRlYSwgYXMgaXMgKGZyb20gbXkgdmFudGFnZSBwb2lu dCkgdGhlcmUgaXMgY2xvc2UgdG8gemVybyBkZW1hbmQgZm9yDQo+IHpvbmVkIGRldmljZXMuICBJ dCB3b24ndCBiZSBhIHByaW9yaXR5IHVudGlsIGVub3VnaCBjdXN0b21lcnMgYXJlIGFza2luZw0K PiBmb3IgaXQuDQoNCkZyb20gbXkgcG9pbnQgb2YgdmlldyAoZHJpdmUgdmVuZG9yKSwgdGhpbmdz IGFyZSBkaWZmZXJlbnQuIFdlIGRvIHNlZSBhbg0KaW5jcmVhc2luZyBpbnRlcmVzdCBmb3IgdGhl c2UgZHJpdmVzLiBIb3dldmVyLCBtb3N0IHVzZSBjYXNlcyBhcmUgc3RpbGwNCmxpbWl0ZWQgdG8g YXBwbGljYXRpb24gYmFzZWQgZGlyZWN0IGRpc2sgYWNjZXNzIHdpdGggbWluaW1hbCBpbnZvbHZl bWVudA0KZnJvbSB0aGUga2VybmVsIGFuZCBzbyBmZXcgInN1cHBvcnQiIHJlcXVlc3RzLiBNYW55 IHJlYXNvbnMgdG8gdGhpcywgYnV0DQpvbmUgaXMgdG8gc29tZSBleHRlbnQgdGhlIGN1cnJlbnQg bGFjayBvZiBleHRlbmRlZCBzdXBwb3J0IGJ5IHRoZQ0Ka2VybmVsLiBEZXNwaXRlIGFsbCB0aGUg cmVjZW50IHdvcmsgZG9uZSwgYXMgTHVpcyBleHBlcmllbmNlZCwgem9uZWQNCmRyaXZlcyBhcmUg c3RpbGwgZmFyIGhhcmRlciB0byBlYXNpbHkgc2V0dXAgdGhhbiByZWd1bGFyIGRpc2tzLiBDaGlj a2VuDQphbmQgZWdnIHNpdHVhdGlvbi4uLg0KDQo+PiBJIGFncmVlIGl0IGlzIG9kZCB0byBleHBl Y3Qgb25lIHRvIHVzZSBkbXNldHVwIGFuZCB0aGVuIHVzZSBhIHZvbHVtZSBtYW5hZ2VyIG9uDQo+ PiB0b3Agb2YgaXQsIGlmIHdlIGNhbiBqdXN0IGFkZCBwcm9wZXIgc3VwcG9ydCBvbnRvIHRoZSB2 b2x1bWUgbWFuYWdlci4uLiB0aGVuDQo+PiB0aGF0J3MgYSByZWFzb25hYmxlIHdheSB0byBnby4N Cj4+DQo+PiBCdXQgKndlJ3JlIG5vdCB0aGVyZSogeWV0LCBhbmQgYXMtaXMgdG9kYXksIHdoYXQg aXMgZGVzY3JpYmVkIGluIHRoZSB1ZGV2DQo+PiBzY3JpcHQgaXMgdGhlIGJlc3Qgd2UgY2FuIGRv IGZvciBhIGdlbmVyaWMgc2V0dXAuDQo+IA0KPiBKdXN0IGJlY2F1c2UgZG9pbmcgdGhpbmdzIHJp Z2h0IHRha2VzIHdvcmsgZG9lc24ndCBtZWFuIGl0IG1ha2VzIHNlbnNlDQo+IHRvIGVsZXZhdGUg dGhpcyB1ZGV2IHNjcmlwdCB0byBiZSBwYWNrYWdlZCBpbiBzb21lIHVwc3RyZWFtIHByb2plY3Qg bGlrZQ0KPiB1ZGV2IG9yIHdoYXRldmVyLg0KDQpBZ3JlZS4gV2lsbCBzdGFydCBsb29raW5nIGlu dG8gYmV0dGVyIHNvbHV0aW9ucyBub3cgdGhhdCBhdCBsZWFzdCBvbmUNCnVzZXIgKEx1aXMpIGNv bXBsYWluZWQuIFRoZSBjdXN0b21lciBpcyBraW5nLg0KDQo+Pj4gSW4gZ2VuZXJhbCB0aGlzIHVk ZXYgc2NyaXB0IGlzIHVud2VsY29tZSBhbmQgbWFrZXMgdGhpbmdzIHdheSB3b3JzZSBmb3INCj4+ PiB0aGUgbG9uZy10ZXJtIHN1Y2Nlc3Mgb2Ygem9uZWQgZGV2aWNlcy4NCj4+DQo+PiBkbS16b25l ZC10b29scyBkb2VzIG5vdCBhY2tub3dsZWRnZSBpbiBhbnkgd2F5IGEgcm9hZG1hcCwgYW5kIGp1 c3QgcHJvdmlkZXMNCj4+IGEgc2NyaXB0LCB3aGljaCBJTUhPIGlzIGxlc3MgZ2VuZXJpYyBhbmQg bGVzcyBkaXN0cmlidXRpb24gZnJpZW5kbHkuIEhhdmluZw0KPj4gYSB1ZGV2IHJ1bGUgaW4gcGxh Y2UgdG8gZGVtb25zdHJhdGUgdGhlIGN1cnJlbnQgc3RhdGUgb2YgYWZmYWlycyBJTUhPIGlzDQo+ PiBtb3JlIHNjYWxhYmxlIGRlbW9uc3RyYXRlcyB0aGUgaXNzdWVzIGJldHRlciB0aGFuIHRoZSBz Y3JpcHQuDQo+Pg0KPj4gSWYgd2UgaGF2ZSBhbiBhZ3JlZWQgdXBvbiBsb25nIHRlcm0gc3RyYXRl Z3kgbGV0cyBkb2N1bWVudCB0aGF0LiBCdXQgZnJvbQ0KPj4gd2hhdCBJIGdhdGhlciB3ZSBhcmUg bm90IGV2ZW4gaW4gY29uc2Vuc3VzIHdpdGggcmVnYXJkcyB0byB0aGUgc2NoZWR1bGVyDQo+PiBz dHVmZi4gSWYgd2UgaGF2ZSBjb25zZW5zdXMgb24gdGhlIG90aGVyIHN0dWZmIGxldHMgZG9jdW1l bnQgdGhhdCBhcw0KPj4gZG0tem9uZWQtdG9vbHMgaXMgdGhlIG9ubHkgcGxhY2UgSSB0aGluayBm b2xrcyBjb3VsZCBmaW5kIHRvIHJlYXNvbmFibHkNCj4+IGRlcGxveSB0aGVzZSB0aGluZ3MuDQo+ IA0KPiBJJ20gc3VyZSBEYW1pZW4gYW5kIG90aGVycyB3aWxsIGhhdmUgc29tZXRoaW5nIHRvIHNh eSBoZXJlLg0KDQpZZXMuIFRoZSBzY2hlZHVsZXIgc2V0dXAgcGFpbiBpcyByZWFsLiBKZW5zIG1h ZGUgaXQgY2xlYXIgdGhhdCBoZQ0KcHJlZmVycyBhIHVkZXYgcnVsZS4gSSBmdWxseSB1bmRlcnN0 YW5kIGhpcyBwb2ludCBvZiB2aWV3LCB5ZXQsIEkgdGhpbmsNCmFuIGF1dG9tYXRpYyBzd2l0Y2gg aW4gdGhlIGJsb2NrIGxheWVyIHdvdWxkIGJlIGZhciBlYXNpZXIgYW5kIGdlbmVyYXRlDQphIGxv dCBsZXNzIHByb2JsZW0gZm9yIHVzZXJzLCBhbmQgbGlrZWx5IGxlc3MgImJ1ZyByZXBvcnQiIHRv DQpkaXN0cmlidXRpb25zIHZlbmRvcnMgKGFuZCB0byBteXNlbGYgdG9vKS4NCg0KVGhhdCBzYWlk LCBJIGFsc28gbGlrZSB0byBzZWUgdGhlIGN1cnJlbnQgZGVwZW5kZW5jeSBvZiB6b25lZCBkZXZp Y2VzIG9uDQp0aGUgZGVhZGxpbmUgc2NoZWR1bGVyIGFzIHRlbXBvcmFyeSB1bnRpbCBhIGJldHRl ciBzb2x1dGlvbiBmb3IgZW5zdXJpbmcNCndyaXRlIG9yZGVyaW5nIGlzIGZvdW5kLiBBZnRlciBh bGwsIHJlcXVpcmluZyBkZWFkbGluZSBhcyB0aGUgZGlzaw0Kc2NoZWR1bGVyIGRvZXMgaW1wb3Nl IG90aGVyIGxpbWl0YXRpb25zIG9uIHRoZSB1c2VyLiBMYWNrIG9mIEkvTw0KcHJpb3JpdHkgc3Vw cG9ydCBhbmQgbm8gY2dyb3VwIGJhc2VkIGZhaXJuZXNzIGFyZSB0d28gZXhhbXBsZXMgb2Ygd2hh dA0Kb3RoZXIgc2NoZWR1bGVycyBwcm92aWRlIGJ1dCBpcyBsb3N0IHdpdGggZm9yY2luZyBkZWFk bGluZS4NCg0KVGhlIG9idmlvdXMgZml4IGlzIG9mIGNvdXJzZSB0byBtYWtlIGFsbCBkaXNrIHNj aGVkdWxlcnMgem9uZSBkZXZpY2UNCmF3YXJlLiBBIGxpdHRsZSBoZWF2eSBoYW5kZWQsIHByb2Jh Ymx5IGxvdHMgb2YgZHVwbGljYXRlZC9zaW1pbGFyIGNvZGUsDQphbmQgbWFueSBtb3JlIHRlc3Qg Y2FzZXMgdG8gY292ZXIuIFRoaXMgYXBwcm9hY2ggZG9lcyBub3Qgc2VlbQ0Kc3VzdGFpbmFibGUg dG8gbWUuDQoNCldlIGRpc2N1c3NlZCBvdGhlciBwb3NzaWJpbGl0aWVzIGF0IExTRi9NTSAoc3Bl Y2lhbGl6ZWQgd3JpdGUgcXVldWUgaW4NCm11bHRpLXF1ZXVlIHBhdGgpLiBPbmUgY291bGQgYWxz byB0aGluayBvZiBtb3JlIGludmFzaXZlIGNoYW5nZXMgdG8gdGhlDQpibG9jayBsYXllciAoZS5n LiBhZGRpbmcgYW4gb3B0aW9uYWwgImRpc3BhdGNoZXIiIGxheWVyIHRvIHRpZ2h0bHkNCmNvbnRy b2wgY29tbWFuZCBvcmRlcmluZyA/KS4gQW5kIHByb2JhYmx5IGEgbG90IG1vcmUgb3B0aW9ucywg QnV0IEkgYW0NCm5vdCB5ZXQgc3VyZSB3aGF0IGFuIGFwcHJvcHJpYXRlIHJlcGxhY2VtZW50IHRv IGRlYWRsaW5lIHdvdWxkIGJlLg0KDQpFdmVudHVhbGx5LCB0aGUgcmVtb3ZhbCBvZiB0aGUgbGVn YWN5IEkvTyBwYXRoIG1heSBhbHNvIGJlIHRoZSB0cmlnZ2VyDQp0byBpbnRyb2R1Y2Ugc29tZSBk ZWVwZXIgZGVzaWduIGNoYW5nZXMgdG8gYmxrLW1xIHRvIGFjY29tbW9kYXRlIG1vcmUNCmVhc2ls eSB6b25lZCBibG9jayBkZXZpY2VzIG9yIG90aGVyIG5vbi1zdGFuZGFyZCBibG9jayBkZXZpY2Vz IChvcGVuDQpjaGFubmVsIFNTRHMgZm9yIGluc3RhbmNlKS4NCg0KQXMgeW91IGNhbiBzZWUgZnJv bSB0aGUgYWJvdmUsIHdvcmtpbmcgd2l0aCB0aGVzZSBkcml2ZXMgYWxsIGRheSBsb25nDQpkb2Vz IG5vdCBtYWtlIGZvciBhIGNsZWFyIHN0cmF0ZWd5LiBJbnB1dHMgZnJvbSBvdGhlciBoZXJlIGFy ZSBtb3JlIHRoYW4NCndlbGNvbWUuIEkgd291bGQgYmUgaGFwcHkgdG8gd3JpdGUgdXAgYWxsIHRo ZSBpZGVhcyBJIGhhdmUgdG8gc3RhcnQgYQ0KZGlzY3Vzc2lvbiBzbyB0aGF0IHdlIGNhbiBjb21l IHRvIGEgY29uc2Vuc3VzIGFuZCBoYXZlIGEgcGxhbi4NCg0KPj4+IEkgZG9uJ3QgZGlzcHV0ZSB0 aGVyZSBpcyBhbiBvYnZpb3VzIHZvaWQgZm9yIGhvdyB0byBwcm9wZXJseSBzZXR1cCB6b25lZA0K Pj4+IGRldmljZXMsIGJ1dCB0aGlzIHNjcmlwdCBpcyBfbm90XyB3aGF0IHNob3VsZCBmaWxsIHRo YXQgdm9pZC4NCj4+DQo+PiBHb29kIHRvIGtub3chIEFnYWluLCBjb25zaWRlciBpdCBhcyBhbiBh bHRlcm5hdGl2ZSB0byB0aGUgc2NyaXB0Lg0KPj4NCj4+IEknbSBoYXBweSB0byBhZGFwdCB0aGUg bGFuZ3VhZ2UgYW5kIHN1cHBseSBpdCBvbmx5IGFzIGFuIGV4YW1wbGUgc2NyaXB0DQo+PiBkZXZl bG9wZXJzIGNhbiB1c2UsIGJ1dCB3ZSBjYW4ndCBsZWF2ZSB1c2VycyBoYW5naW5nIGFzIHdlbGwu IExldCdzIGF0DQo+PiBsZWFzdCBjb21lIHVwIHdpdGggYSBwbGFuIHdoaWNoIHdlIHNlZW0gdG8g YWdyZWUgb24gYW5kIGRvY3VtZW50IHRoYXQuDQo+IA0KPiBCZXN0IHRvIHRyeSB0byBnZXQgRGFt aWVuIGFuZCBvdGhlcnMgbW9yZSBpbnZlc3RlZCBpbiB6b25lZCBkZXZpY2VzIHRvDQo+IGhlbHAg eW91IHRha2UgdXAgeW91ciBjYXVzZS4gIEkgdGhpbmsgaXQgaXMgd29ydGh3aGlsZSB0byBkZXZl bG9wIGENCj4gc3RyYXRlZ3kuICBCdXQgaXQgbmVlZHMgdG8gYmUgZG9uZSBpbiB0ZXJtcyBvZiB0 aGUgbm9ybXMgb2YgdGhlIGV4aXN0aW5nDQo+IGluZnJhc3RydWN0dXJlIHdlIGFsbCBtYWtlIHVz ZSBvZiB0b2RheS4gIFNvIGZpcnN0IHN0ZXAgaXMgbWFraW5nDQo+IGV4aXN0aW5nIHRvb2xzIHpv bmVkIGRldmljZSBhd2FyZSAoZXZlbiBpZiB0byByZWplY3Qgc3VjaCBkZXZpY2VzKS4NCg0KUmVz dCBhc3N1cmVkIHRoYXQgSSBhbSBmdWxseSBpbnZlc3RlZCBpbiBpbXByb3ZpbmcgdGhlIGV4aXN0 aW5nDQppbmZyYXN0cnVjdHVyZSBmb3Igem9uZWQgYmxvY2sgZGV2aWNlcy4gQXMgbWVudGlvbmVk IGFib3ZlLCBhcHBsaWNhdGlvbnMNCmJhc2VkIHVzZSBvZiB6b25lZCBibG9jayBkZXZpY2VzIHN0 aWxsIHByZXZhaWxzIHRvZGF5LiBTbyBJIGRvIHRlbmQgdG8NCndvcmsgbW9yZSBvbiB0aGF0IHNp ZGUgb2YgdGhpbmdzIChsaWJ6YmMsIHRjbXUsIHN5c3V0aWxzIGZvciBpbnN0YW5jZSkNCnJhdGhl ciB0aGFuIG9uIGEgYmV0dGVyIGludGVncmF0aW9uIHdpdGggbW9yZSBhZHZhbmNlZCB0b29scyAo c3VjaCBhcw0KTFZNKSByZWx5aW5nIG9uIGtlcm5lbCBmZWF0dXJlcy4gSSBhbSBob3dldmVyIHNl ZWluZyByaXNpbmcgaW50ZXJlc3QgaW4NCmZpbGUgc3lzdGVtcyBhbmQgYWxzbyBpbiBkbS16b25l ZC4gU28gZGVmaW5pdGVseSBpdCBpcyB0aW1lIHRvIHN0ZXAgdXANCndvcmsgaW4gdGhhdCBhcmVh IHRvIGZ1cnRoZXIgc2ltcGxpZnkgdXNpbmcgdGhlc2UgZHJpdmVzLg0KDQpUaGFuayB5b3UgZm9y IHRoZSBmZWVkYmFjay4NCg0KQmVzdCByZWdhcmRzLg0KDQotLSANCkRhbWllbiBMZSBNb2FsLA0K V2VzdGVybiBEaWdpdGFs ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup @ 2018-06-15 9:59 ` Damien Le Moal 0 siblings, 0 replies; 22+ messages in thread From: Damien Le Moal @ 2018-06-15 9:59 UTC (permalink / raw) To: Mike Snitzer, Luis R. Rodriguez Cc: hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block Mike, On 6/15/18 02:58, Mike Snitzer wrote: > On Thu, Jun 14 2018 at 1:37pm -0400, > Luis R. Rodriguez <mcgrof@kernel.org> wrote: > >> On Thu, Jun 14, 2018 at 08:38:06AM -0400, Mike Snitzer wrote: >>> On Wed, Jun 13 2018 at 8:11pm -0400, >>> Luis R. Rodriguez <mcgrof@kernel.org> wrote: >>> >>>> Setting up a zoned disks in a generic form is not so trivial. There >>>> is also quite a bit of tribal knowledge with these devices which is not >>>> easy to find. >>>> >>>> The currently supplied demo script works but it is not generic enough to be >>>> practical for Linux distributions or even developers which often move >>>> from one kernel to another. >>>> >>>> This tries to put a bit of this tribal knowledge into an initial udev >>>> rule for development with the hopes Linux distributions can later >>>> deploy. Three rule are added. One rule is optional for now, it should be >>>> extended later to be more distribution-friendly and then I think this >>>> may be ready for consideration for integration on distributions. >>>> >>>> 1) scheduler setup >>> >>> This is wrong.. if zoned devices are so dependent on deadline or >>> mq-deadline then the kernel should allow them to be hardcoded. I know >>> Jens removed the API to do so but the fact that drivers need to rely on >>> hacks like this udev rule to get a functional device is proof we need to >>> allow drivers to impose the scheduler used. >> >> This is the point to the patch as well, I actually tend to agree with you, >> and I had tried to draw up a patch to do just that, however its *not* possible >> today to do this and would require some consensus. So from what I can tell >> we *have* to live with this one or a form of it. Ie a file describing which >> disk serial gets deadline and which one gets mq-deadline. >> >> Jens? >> >> Anyway, let's assume this is done in the kernel, which one would use deadline, >> which one would use mq-deadline? > > The zoned storage driver needs to make that call based on what mode it > is in. If it is using blk-mq then it selects mq-deadline, otherwise > deadline. As Bart pointed out, deadline is an alias of mq-deadline. So using "deadline" as the scheduler name works in both legacy and mq cases. >>>> 2) backlist f2fs devices >>> >>> There should porbably be support in dm-zoned for detecting whether a >>> zoned device was formatted with f2fs (assuming there is a known f2fs >>> superblock)? >> >> Not sure what you mean. Are you suggesting we always setup dm-zoned for >> all zoned disks and just make an excemption on dm-zone code to somehow >> use the disk directly if a filesystem supports zoned disks directly somehow? > > No, I'm saying that a udev rule wouldn't be needed if dm-zoned just > errored out if asked to consume disks that already have an f2fs > superblock. And existing filesystems should get conflicting superblock > awareness "for free" if blkid or whatever is trained to be aware of > f2fs's superblock. Well that is the case already: on startup, dm-zoned will read its own metadata from sector 0, same as f2fs would do with its super-block. If the format/magic does not match expected values, dm-zoned will bail out and return an error. dm-zoned metadata and f2fs metadata reside in the same place and overwrite each other. There is no way to get one working on top of the other. I do not see any possibility of a problem on startup. But definitely, the user land format tools can step on each other toes. That needs fixing. >> f2fs does not require dm-zoned. What would be required is a bit more complex >> given one could dedicate portions of the disk to f2fs and other portions to >> another filesystem, which would require dm-zoned. >> >> Also filesystems which *do not* support zoned disks should *not* be allowing >> direct setup. Today that's all filesystems other than f2fs, in the future >> that may change. Those are bullets we are allowing to trigger for users >> just waiting to shot themselves on the foot with. >> >> So who's going to work on all the above? > > It should take care of itself if existing tools are trained to be aware > of new signatures. E.g. ext4 and xfs already are aware of one another > so that you cannot reformat a device with the other unless force is > given. > > Same kind of mutual exclussion needs to happen for zoned devices. Yes. > So the zoned device tools, dm-zoned, f2fs, whatever.. they need to be > updated to not step on each others toes. And other filesystems' tools > need to be updated to be zoned device aware. I will update dm-zoned tools to check for known FS superblocks, similarly to what mkfs.ext4 and mkfs.xfs do. >>>> 3) run dmsetup for the rest of devices >>> >>> automagically running dmsetup directly from udev to create a dm-zoned >>> target is very much wrong. It just gets in the way of proper support >>> that should be add to appropriate tools that admins use to setup their >>> zoned devices. For instance, persistent use of dm-zoned target should >>> be made reliable with a volume manager.. >> >> Ah yes, but who's working on that? How long will it take? > > No idea, as is (from my vantage point) there is close to zero demand for > zoned devices. It won't be a priority until enough customers are asking > for it. From my point of view (drive vendor), things are different. We do see an increasing interest for these drives. However, most use cases are still limited to application based direct disk access with minimal involvement from the kernel and so few "support" requests. Many reasons to this, but one is to some extent the current lack of extended support by the kernel. Despite all the recent work done, as Luis experienced, zoned drives are still far harder to easily setup than regular disks. Chicken and egg situation... >> I agree it is odd to expect one to use dmsetup and then use a volume manager on >> top of it, if we can just add proper support onto the volume manager... then >> that's a reasonable way to go. >> >> But *we're not there* yet, and as-is today, what is described in the udev >> script is the best we can do for a generic setup. > > Just because doing things right takes work doesn't mean it makes sense > to elevate this udev script to be packaged in some upstream project like > udev or whatever. Agree. Will start looking into better solutions now that at least one user (Luis) complained. The customer is king. >>> In general this udev script is unwelcome and makes things way worse for >>> the long-term success of zoned devices. >> >> dm-zoned-tools does not acknowledge in any way a roadmap, and just provides >> a script, which IMHO is less generic and less distribution friendly. Having >> a udev rule in place to demonstrate the current state of affairs IMHO is >> more scalable demonstrates the issues better than the script. >> >> If we have an agreed upon long term strategy lets document that. But from >> what I gather we are not even in consensus with regards to the scheduler >> stuff. If we have consensus on the other stuff lets document that as >> dm-zoned-tools is the only place I think folks could find to reasonably >> deploy these things. > > I'm sure Damien and others will have something to say here. Yes. The scheduler setup pain is real. Jens made it clear that he prefers a udev rule. I fully understand his point of view, yet, I think an automatic switch in the block layer would be far easier and generate a lot less problem for users, and likely less "bug report" to distributions vendors (and to myself too). That said, I also like to see the current dependency of zoned devices on the deadline scheduler as temporary until a better solution for ensuring write ordering is found. After all, requiring deadline as the disk scheduler does impose other limitations on the user. Lack of I/O priority support and no cgroup based fairness are two examples of what other schedulers provide but is lost with forcing deadline. The obvious fix is of course to make all disk schedulers zone device aware. A little heavy handed, probably lots of duplicated/similar code, and many more test cases to cover. This approach does not seem sustainable to me. We discussed other possibilities at LSF/MM (specialized write queue in multi-queue path). One could also think of more invasive changes to the block layer (e.g. adding an optional "dispatcher" layer to tightly control command ordering ?). And probably a lot more options, But I am not yet sure what an appropriate replacement to deadline would be. Eventually, the removal of the legacy I/O path may also be the trigger to introduce some deeper design changes to blk-mq to accommodate more easily zoned block devices or other non-standard block devices (open channel SSDs for instance). As you can see from the above, working with these drives all day long does not make for a clear strategy. Inputs from other here are more than welcome. I would be happy to write up all the ideas I have to start a discussion so that we can come to a consensus and have a plan. >>> I don't dispute there is an obvious void for how to properly setup zoned >>> devices, but this script is _not_ what should fill that void. >> >> Good to know! Again, consider it as an alternative to the script. >> >> I'm happy to adapt the language and supply it only as an example script >> developers can use, but we can't leave users hanging as well. Let's at >> least come up with a plan which we seem to agree on and document that. > > Best to try to get Damien and others more invested in zoned devices to > help you take up your cause. I think it is worthwhile to develop a > strategy. But it needs to be done in terms of the norms of the existing > infrastructure we all make use of today. So first step is making > existing tools zoned device aware (even if to reject such devices). Rest assured that I am fully invested in improving the existing infrastructure for zoned block devices. As mentioned above, applications based use of zoned block devices still prevails today. So I do tend to work more on that side of things (libzbc, tcmu, sysutils for instance) rather than on a better integration with more advanced tools (such as LVM) relying on kernel features. I am however seeing rising interest in file systems and also in dm-zoned. So definitely it is time to step up work in that area to further simplify using these drives. Thank you for the feedback. Best regards. -- Damien Le Moal, Western Digital ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-15 9:59 ` Damien Le Moal (?) @ 2018-06-15 14:50 ` Mike Snitzer -1 siblings, 0 replies; 22+ messages in thread From: Mike Snitzer @ 2018-06-15 14:50 UTC (permalink / raw) To: Damien Le Moal Cc: Luis R. Rodriguez, hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block On Fri, Jun 15 2018 at 5:59am -0400, Damien Le Moal <Damien.LeMoal@wdc.com> wrote: > Mike, > > On 6/15/18 02:58, Mike Snitzer wrote: > > On Thu, Jun 14 2018 at 1:37pm -0400, > > Luis R. Rodriguez <mcgrof@kernel.org> wrote: > > > >> On Thu, Jun 14, 2018 at 08:38:06AM -0400, Mike Snitzer wrote: > >>> On Wed, Jun 13 2018 at 8:11pm -0400, > >>> Luis R. Rodriguez <mcgrof@kernel.org> wrote: > >>> > >>>> Setting up a zoned disks in a generic form is not so trivial. There > >>>> is also quite a bit of tribal knowledge with these devices which is not > >>>> easy to find. > >>>> > >>>> The currently supplied demo script works but it is not generic enough to be > >>>> practical for Linux distributions or even developers which often move > >>>> from one kernel to another. > >>>> > >>>> This tries to put a bit of this tribal knowledge into an initial udev > >>>> rule for development with the hopes Linux distributions can later > >>>> deploy. Three rule are added. One rule is optional for now, it should be > >>>> extended later to be more distribution-friendly and then I think this > >>>> may be ready for consideration for integration on distributions. > >>>> > >>>> 1) scheduler setup > >>> > >>> This is wrong.. if zoned devices are so dependent on deadline or > >>> mq-deadline then the kernel should allow them to be hardcoded. I know > >>> Jens removed the API to do so but the fact that drivers need to rely on > >>> hacks like this udev rule to get a functional device is proof we need to > >>> allow drivers to impose the scheduler used. > >> > >> This is the point to the patch as well, I actually tend to agree with you, > >> and I had tried to draw up a patch to do just that, however its *not* possible > >> today to do this and would require some consensus. So from what I can tell > >> we *have* to live with this one or a form of it. Ie a file describing which > >> disk serial gets deadline and which one gets mq-deadline. > >> > >> Jens? > >> > >> Anyway, let's assume this is done in the kernel, which one would use deadline, > >> which one would use mq-deadline? > > > > The zoned storage driver needs to make that call based on what mode it > > is in. If it is using blk-mq then it selects mq-deadline, otherwise > > deadline. > > As Bart pointed out, deadline is an alias of mq-deadline. So using > "deadline" as the scheduler name works in both legacy and mq cases. > > >>>> 2) backlist f2fs devices > >>> > >>> There should porbably be support in dm-zoned for detecting whether a > >>> zoned device was formatted with f2fs (assuming there is a known f2fs > >>> superblock)? > >> > >> Not sure what you mean. Are you suggesting we always setup dm-zoned for > >> all zoned disks and just make an excemption on dm-zone code to somehow > >> use the disk directly if a filesystem supports zoned disks directly somehow? > > > > No, I'm saying that a udev rule wouldn't be needed if dm-zoned just > > errored out if asked to consume disks that already have an f2fs > > superblock. And existing filesystems should get conflicting superblock > > awareness "for free" if blkid or whatever is trained to be aware of > > f2fs's superblock. > > Well that is the case already: on startup, dm-zoned will read its own > metadata from sector 0, same as f2fs would do with its super-block. If > the format/magic does not match expected values, dm-zoned will bail out > and return an error. dm-zoned metadata and f2fs metadata reside in the > same place and overwrite each other. There is no way to get one working > on top of the other. I do not see any possibility of a problem on startup. > > But definitely, the user land format tools can step on each other toes. > That needs fixing. Right, I was talking about in the .ctr path for initial device creation, not activation of a previously created dm-zoned device. But I agree it makes most sense to do this check in userspace. > >> f2fs does not require dm-zoned. What would be required is a bit more complex > >> given one could dedicate portions of the disk to f2fs and other portions to > >> another filesystem, which would require dm-zoned. > >> > >> Also filesystems which *do not* support zoned disks should *not* be allowing > >> direct setup. Today that's all filesystems other than f2fs, in the future > >> that may change. Those are bullets we are allowing to trigger for users > >> just waiting to shot themselves on the foot with. > >> > >> So who's going to work on all the above? > > > > It should take care of itself if existing tools are trained to be aware > > of new signatures. E.g. ext4 and xfs already are aware of one another > > so that you cannot reformat a device with the other unless force is > > given. > > > > Same kind of mutual exclussion needs to happen for zoned devices. > > Yes. > > > So the zoned device tools, dm-zoned, f2fs, whatever.. they need to be > > updated to not step on each others toes. And other filesystems' tools > > need to be updated to be zoned device aware. > > I will update dm-zoned tools to check for known FS superblocks, > similarly to what mkfs.ext4 and mkfs.xfs do. Thanks. > >>>> 3) run dmsetup for the rest of devices > >>> > >>> automagically running dmsetup directly from udev to create a dm-zoned > >>> target is very much wrong. It just gets in the way of proper support > >>> that should be add to appropriate tools that admins use to setup their > >>> zoned devices. For instance, persistent use of dm-zoned target should > >>> be made reliable with a volume manager.. > >> > >> Ah yes, but who's working on that? How long will it take? > > > > No idea, as is (from my vantage point) there is close to zero demand for > > zoned devices. It won't be a priority until enough customers are asking > > for it. > > From my point of view (drive vendor), things are different. We do see an > increasing interest for these drives. However, most use cases are still > limited to application based direct disk access with minimal involvement > from the kernel and so few "support" requests. Many reasons to this, but > one is to some extent the current lack of extended support by the > kernel. Despite all the recent work done, as Luis experienced, zoned > drives are still far harder to easily setup than regular disks. Chicken > and egg situation... > > >> I agree it is odd to expect one to use dmsetup and then use a volume manager on > >> top of it, if we can just add proper support onto the volume manager... then > >> that's a reasonable way to go. > >> > >> But *we're not there* yet, and as-is today, what is described in the udev > >> script is the best we can do for a generic setup. > > > > Just because doing things right takes work doesn't mean it makes sense > > to elevate this udev script to be packaged in some upstream project like > > udev or whatever. > > Agree. Will start looking into better solutions now that at least one > user (Luis) complained. The customer is king. > > >>> In general this udev script is unwelcome and makes things way worse for > >>> the long-term success of zoned devices. > >> > >> dm-zoned-tools does not acknowledge in any way a roadmap, and just provides > >> a script, which IMHO is less generic and less distribution friendly. Having > >> a udev rule in place to demonstrate the current state of affairs IMHO is > >> more scalable demonstrates the issues better than the script. > >> > >> If we have an agreed upon long term strategy lets document that. But from > >> what I gather we are not even in consensus with regards to the scheduler > >> stuff. If we have consensus on the other stuff lets document that as > >> dm-zoned-tools is the only place I think folks could find to reasonably > >> deploy these things. > > > > I'm sure Damien and others will have something to say here. > > Yes. The scheduler setup pain is real. Jens made it clear that he > prefers a udev rule. I fully understand his point of view, yet, I think > an automatic switch in the block layer would be far easier and generate > a lot less problem for users, and likely less "bug report" to > distributions vendors (and to myself too). Yeap, Jens would say that ;) Unfortnately using udev to get this critical configuration correct is a real leap of faith that will prove to be a whack-a-mole across distributions. > That said, I also like to see the current dependency of zoned devices on > the deadline scheduler as temporary until a better solution for ensuring > write ordering is found. After all, requiring deadline as the disk > scheduler does impose other limitations on the user. Lack of I/O > priority support and no cgroup based fairness are two examples of what > other schedulers provide but is lost with forcing deadline. > > The obvious fix is of course to make all disk schedulers zone device > aware. A little heavy handed, probably lots of duplicated/similar code, > and many more test cases to cover. This approach does not seem > sustainable to me. Right, it isn't sustainable. There isn't enough zoned device developer expertise to go around. > We discussed other possibilities at LSF/MM (specialized write queue in > multi-queue path). One could also think of more invasive changes to the > block layer (e.g. adding an optional "dispatcher" layer to tightly > control command ordering ?). And probably a lot more options, But I am > not yet sure what an appropriate replacement to deadline would be. > > Eventually, the removal of the legacy I/O path may also be the trigger > to introduce some deeper design changes to blk-mq to accommodate more > easily zoned block devices or other non-standard block devices (open > channel SSDs for instance). > > As you can see from the above, working with these drives all day long > does not make for a clear strategy. Inputs from other here are more than > welcome. I would be happy to write up all the ideas I have to start a > discussion so that we can come to a consensus and have a plan. Doesn't hurt to establish a future plan(s) but we need to deal with the reality of what we have. And all we have for this particular issue is "deadline". Setting anything else is a bug. Short of the block layer reinstating the ability for a driver to specify an elevator: should the zoned driver put a check in place that errors out if anything other than deadline is configured? That'd at least save users from a very cutthroat learning curve. > >>> I don't dispute there is an obvious void for how to properly setup zoned > >>> devices, but this script is _not_ what should fill that void. > >> > >> Good to know! Again, consider it as an alternative to the script. > >> > >> I'm happy to adapt the language and supply it only as an example script > >> developers can use, but we can't leave users hanging as well. Let's at > >> least come up with a plan which we seem to agree on and document that. > > > > Best to try to get Damien and others more invested in zoned devices to > > help you take up your cause. I think it is worthwhile to develop a > > strategy. But it needs to be done in terms of the norms of the existing > > infrastructure we all make use of today. So first step is making > > existing tools zoned device aware (even if to reject such devices). > > Rest assured that I am fully invested in improving the existing > infrastructure for zoned block devices. As mentioned above, applications > based use of zoned block devices still prevails today. So I do tend to > work more on that side of things (libzbc, tcmu, sysutils for instance) > rather than on a better integration with more advanced tools (such as > LVM) relying on kernel features. I am however seeing rising interest in > file systems and also in dm-zoned. So definitely it is time to step up > work in that area to further simplify using these drives. > > Thank you for the feedback. Thanks for your insight. Sounds like you're ontop of it. Mike ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 12:38 ` Mike Snitzer @ 2018-06-15 9:00 ` Damien Le Moal 2018-06-14 17:37 ` Luis R. Rodriguez 2018-06-15 9:00 ` Damien Le Moal 2 siblings, 0 replies; 22+ messages in thread From: Damien Le Moal @ 2018-06-15 9:00 UTC (permalink / raw) To: Mike Snitzer, Luis R. Rodriguez Cc: hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block TWlrZSwNCg0KT24gNi8xNC8xOCAyMTozOCwgTWlrZSBTbml0emVyIHdyb3RlOg0KPiBPbiBXZWQs IEp1biAxMyAyMDE4IGF0ICA4OjExcG0gLTA0MDAsDQo+IEx1aXMgUi4gUm9kcmlndWV6IDxtY2dy b2ZAa2VybmVsLm9yZz4gd3JvdGU6DQo+IA0KPj4gU2V0dGluZyB1cCBhIHpvbmVkIGRpc2tzIGlu IGEgZ2VuZXJpYyBmb3JtIGlzIG5vdCBzbyB0cml2aWFsLiBUaGVyZQ0KPj4gaXMgYWxzbyBxdWl0 ZSBhIGJpdCBvZiB0cmliYWwga25vd2xlZGdlIHdpdGggdGhlc2UgZGV2aWNlcyB3aGljaCBpcyBu b3QNCj4+IGVhc3kgdG8gZmluZC4NCj4+DQo+PiBUaGUgY3VycmVudGx5IHN1cHBsaWVkIGRlbW8g c2NyaXB0IHdvcmtzIGJ1dCBpdCBpcyBub3QgZ2VuZXJpYyBlbm91Z2ggdG8gYmUNCj4+IHByYWN0 aWNhbCBmb3IgTGludXggZGlzdHJpYnV0aW9ucyBvciBldmVuIGRldmVsb3BlcnMgd2hpY2ggb2Z0 ZW4gbW92ZQ0KPj4gZnJvbSBvbmUga2VybmVsIHRvIGFub3RoZXIuDQo+Pg0KPj4gVGhpcyB0cmll cyB0byBwdXQgYSBiaXQgb2YgdGhpcyB0cmliYWwga25vd2xlZGdlIGludG8gYW4gaW5pdGlhbCB1 ZGV2DQo+PiBydWxlIGZvciBkZXZlbG9wbWVudCB3aXRoIHRoZSBob3BlcyBMaW51eCBkaXN0cmli dXRpb25zIGNhbiBsYXRlcg0KPj4gZGVwbG95LiBUaHJlZSBydWxlIGFyZSBhZGRlZC4gT25lIHJ1 bGUgaXMgb3B0aW9uYWwgZm9yIG5vdywgaXQgc2hvdWxkIGJlDQo+PiBleHRlbmRlZCBsYXRlciB0 byBiZSBtb3JlIGRpc3RyaWJ1dGlvbi1mcmllbmRseSBhbmQgdGhlbiBJIHRoaW5rIHRoaXMNCj4+ IG1heSBiZSByZWFkeSBmb3IgY29uc2lkZXJhdGlvbiBmb3IgaW50ZWdyYXRpb24gb24gZGlzdHJp YnV0aW9ucy4NCj4+DQo+PiAxKSBzY2hlZHVsZXIgc2V0dXANCj4gDQo+IFRoaXMgaXMgd3Jvbmcu LiBpZiB6b25lZCBkZXZpY2VzIGFyZSBzbyBkZXBlbmRlbnQgb24gZGVhZGxpbmUgb3INCj4gbXEt ZGVhZGxpbmUgdGhlbiB0aGUga2VybmVsIHNob3VsZCBhbGxvdyB0aGVtIHRvIGJlIGhhcmRjb2Rl ZC4gIEkga25vdw0KPiBKZW5zIHJlbW92ZWQgdGhlIEFQSSB0byBkbyBzbyBidXQgdGhlIGZhY3Qg dGhhdCBkcml2ZXJzIG5lZWQgdG8gcmVseSBvbg0KPiBoYWNrcyBsaWtlIHRoaXMgdWRldiBydWxl IHRvIGdldCBhIGZ1bmN0aW9uYWwgZGV2aWNlIGlzIHByb29mIHdlIG5lZWQgdG8NCj4gYWxsb3cg ZHJpdmVycyB0byBpbXBvc2UgdGhlIHNjaGVkdWxlciB1c2VkLg0KDQpJIGFncmVlLiBTd2l0Y2hp bmcgc2NoZWR1bGVyIGluIHRoZSBrZXJuZWwgZHVyaW5nIGRldmljZSBwcm9iZS9icmluZy11cA0K d291bGQgYmUgbXkgcHJlZmVycmVkIGNob2ljZS4gQnV0IHVudGlsIHdlIGNvbWUgdG8gYSBjb25z ZW5zdXMgb24gdGhlDQpiZXN0IHdheSB0byBkbyB0aGlzLCBJIHRoaW5rIHRoYXQgdGhpcyB1ZGV2 IHJ1bGUgaXMgdXNlZnVsIHNpbmNlIHRoZQ0KInNjaGVkdWxlcj0iIGtlcm5lbCBwYXJhbWV0ZXIg aXMgcmF0aGVyIGhlYXZ5IGhhbmRlZCBhbmQgYXBwbGllcyB0byBhbGwNCnNpbmdsZSBxdWV1ZSBk ZXZpY2VzLiBOb3QgdG8gbWVudGlvbiB0aGF0IGFkZGluZyB0aGlzIHBhcmFtZXRlciB0byB0aGUN Cmtlcm5lbCBhcmd1bWVudHMgaXMgaW4gZXNzZW5jZSBzaW1pbGFyIHRvIHRoZSB1ZGV2IHJ1bGUg YWRkaXRpb246IGFjdGlvbg0KZnJvbSB0aGUgc3lzdGVtIHVzZXIgaXMgbmVjZXNzYXJ5IHRvIGFj aGlldmUgYSBjb3JyZWN0IGNvbmZpZ3VyYXRpb24NCmV2ZW4gdGhvdWdoIHRoYXQgY291bGQgZWFz aWx5IGJlIGRvbmUgYXV0b21hdGljYWxseSBmcm9tIHdpdGhpbiB0aGUNCmJsb2NrIGxheWVyLg0K DQpJbiB0aGUgbWVhbnRpbWUsIGRvY3VtZW50aW5nIHByb3Blcmx5IHRoYXQgdGhlIGRlYWRsaW5l IHNjaGVkdWxlciBoYXMgYQ0Kc3BlY2lhbCByZWxhdGlvbnNoaXAgd2l0aCB6b25lZCBibG9jayBk ZXZpY2VzIGlzIHN0aWxsIGEgbmljZSB0aGluZyB0bw0KZG8uIEJ1dCB5ZXMsIGRtLXpvbmVkLXRv b2xzIG1heSBub3QgYmUgdGhlIGJlc3QgcGxhY2UgdG8gZG8gdGhhdC4gQQ0Kc2ltcGxlIHRleHQg ZmlsZSB1bmRlciBEb2N1bWVudGF0aW9uL2Jsb2NrIG1heSBiZSBiZXR0ZXIuIEF0IHRoZSB2ZXJ5 DQpsZWFzdCwgRG9jdW1lbnRhdGlvbi9ibG9jay9kZWFkbGluZS1pb3NjaGVkLnR4dCBzaG91bGQg aGF2ZSBzb21lIG1lbnRpb24NCm9mIGl0cyBhbG1vc3QgbWFuZGF0b3J5IHVzZSB3aXRoIHpvbmVk IGJsb2NrIGRldmljZXMuIEkgd2lsbCB3b3JrIG9uIHRoYXQuDQoNCj4+IDIpIGJhY2tsaXN0IGYy ZnMgZGV2aWNlcw0KPiANCj4gVGhlcmUgc2hvdWxkIHBvcmJhYmx5IGJlIHN1cHBvcnQgaW4gZG0t em9uZWQgZm9yIGRldGVjdGluZyB3aGV0aGVyIGENCj4gem9uZWQgZGV2aWNlIHdhcyBmb3JtYXR0 ZWQgd2l0aCBmMmZzIChhc3N1bWluZyB0aGVyZSBpcyBhIGtub3duIGYyZnMNCj4gc3VwZXJibG9j ayk/DQoNClRoYXQgd291bGQgY2VydGFpbmx5IGJlIG5pY2UgdG8gaGF2ZSBhbmQgd291bGQgbm90 IGJlIHRvbyBoYXJkIHRvIGNvZGUNCmluIGRtemFkbS4gSSBjYW4gYWRkIHRoYXQgYW5kIGRvIHdo YXQgZm9yIGluc3RhbmNlIG1rZnMuZXh0NCBvciBta2ZzLnhmcw0KZG8sIHdoaWNoIGlzIGFzayBm b3IgY29uZmlybWF0aW9uIG9yIGJhaWwgb3V0IGlmIGFuIGV4aXN0aW5nIHZhbGlkIEZTDQpmb3Jt YXQgaXMgZGV0ZWN0ZWQgb24gdGhlIGRpc2suDQoNClRoYXQgc2FpZCwgc3VjaCB0ZXN0IGFyZSBm YXIgZnJvbSBjb21tb24uIG1kYWRtIGZvciBpbnN0YW5jZSB3aWxsDQpoYXBwaWx5IGZvcm1hdCBh bmQgc3RhcnQgYW4gYXJyYXkgdXNpbmcgdW5tb3VudGVkIGRpc2tzIHdpdGggdmFsaWQgZmlsZQ0K c3lzdGVtcyBvbiB0aGVtLg0KDQo+PiAzKSBydW4gZG1zZXR1cCBmb3IgdGhlIHJlc3Qgb2YgZGV2 aWNlcw0KPiANCj4gYXV0b21hZ2ljYWxseSBydW5uaW5nIGRtc2V0dXAgZGlyZWN0bHkgZnJvbSB1 ZGV2IHRvIGNyZWF0ZSBhIGRtLXpvbmVkDQo+IHRhcmdldCBpcyB2ZXJ5IG11Y2ggd3JvbmcuICBJ dCBqdXN0IGdldHMgaW4gdGhlIHdheSBvZiBwcm9wZXIgc3VwcG9ydA0KPiB0aGF0IHNob3VsZCBi ZSBhZGQgdG8gYXBwcm9wcmlhdGUgdG9vbHMgdGhhdCBhZG1pbnMgdXNlIHRvIHNldHVwIHRoZWly DQo+IHpvbmVkIGRldmljZXMuICBGb3IgaW5zdGFuY2UsIHBlcnNpc3RlbnQgdXNlIG9mIGRtLXpv bmVkIHRhcmdldCBzaG91bGQNCj4gYmUgbWFkZSByZWxpYWJsZSB3aXRoIGEgdm9sdW1lIG1hbmFn ZXIuLg0KPiANCj4gSW4gZ2VuZXJhbCB0aGlzIHVkZXYgc2NyaXB0IGlzIHVud2VsY29tZSBhbmQg bWFrZXMgdGhpbmdzIHdheSB3b3JzZSBmb3INCj4gdGhlIGxvbmctdGVybSBzdWNjZXNzIG9mIHpv bmVkIGRldmljZXMuDQo+IA0KPiBJIGRvbid0IGRpc3B1dGUgdGhlcmUgaXMgYW4gb2J2aW91cyB2 b2lkIGZvciBob3cgdG8gcHJvcGVybHkgc2V0dXAgem9uZWQNCj4gZGV2aWNlcywgYnV0IHRoaXMg c2NyaXB0IGlzIF9ub3RfIHdoYXQgc2hvdWxkIGZpbGwgdGhhdCB2b2lkLg0KDQpGYWlyIHBvaW50 cy4gSSBhZ3JlZSB0aGF0IGl0IGlzIGhhY2tpc2guIFRoZSBpbnRlbnQgd2FzIGFzIHlvdSBzYXkg dG8NCnRlbXBvcmFyaWx5IGZpbGwgYSB2b2lkLiBCdXQgZ3JhbnRlZCwgdGVtcG9yYXJ5IGhhY2tz IHRlbmQgdG8gc3RheSAodG9vKQ0KbG9uZyBhcm91bmQgYW5kIGNhbiBpbiB0aGUgZW5kIGdldCBp biB0aGUgd2F5IG9mIGNsZWFuIHNvbHV0aW9ucy4gSSB3aWxsDQpzdGFydCBsb29raW5nIGludG8g YSBwcm9wZXIgZml4IGZvciBkbS16b25lZCBzZXR1cCBwZXJzaXN0ZW5jZS4NCg0KPiBTbyBhIGhl YXJ0ZmVsdDoNCj4gDQo+IE5hY2tlZC1ieTogTWlrZSBTbml0emVyIDxzbml0emVyQHJlZGhhdC5j b20+DQoNClVuZGVyc3Rvb2QuIEkgd2lsbCByZXZlcnQgYW5kIG5vdCBkb2N1bWVudCB0aGlzIHVk ZXYgcnVsZSBpbg0KZG0tem9uZWQtdG9vbHMuIEkgd2FzIGEgbGl0dGxlIHRvbyBxdWljayBpbiBh cHBseWluZyB0aGlzIHBhdGNoIGFuZCBkaWQNCm5vdCB3YWl0IHRvIHNlZSBjb21tZW50cyBmaXJz dC4gTXkgYmFkLg0KDQpUaGFuayB5b3UgZm9yIHlvdXIgY29tbWVudHMuDQoNCkJlc3QgcmVnYXJk cy4NCg0KLS0gDQpEYW1pZW4gTGUgTW9hbCwNCldlc3Rlcm4gRGlnaXRhbA== ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup @ 2018-06-15 9:00 ` Damien Le Moal 0 siblings, 0 replies; 22+ messages in thread From: Damien Le Moal @ 2018-06-15 9:00 UTC (permalink / raw) To: Mike Snitzer, Luis R. Rodriguez Cc: hare, axboe, jaegeuk, yuchao0, ghe, mwilck, tchvatal, zren, agk, linux-fsdevel, linux-block Mike, On 6/14/18 21:38, Mike Snitzer wrote: > On Wed, Jun 13 2018 at 8:11pm -0400, > Luis R. Rodriguez <mcgrof@kernel.org> wrote: > >> Setting up a zoned disks in a generic form is not so trivial. There >> is also quite a bit of tribal knowledge with these devices which is not >> easy to find. >> >> The currently supplied demo script works but it is not generic enough to be >> practical for Linux distributions or even developers which often move >> from one kernel to another. >> >> This tries to put a bit of this tribal knowledge into an initial udev >> rule for development with the hopes Linux distributions can later >> deploy. Three rule are added. One rule is optional for now, it should be >> extended later to be more distribution-friendly and then I think this >> may be ready for consideration for integration on distributions. >> >> 1) scheduler setup > > This is wrong.. if zoned devices are so dependent on deadline or > mq-deadline then the kernel should allow them to be hardcoded. I know > Jens removed the API to do so but the fact that drivers need to rely on > hacks like this udev rule to get a functional device is proof we need to > allow drivers to impose the scheduler used. I agree. Switching scheduler in the kernel during device probe/bring-up would be my preferred choice. But until we come to a consensus on the best way to do this, I think that this udev rule is useful since the "scheduler=" kernel parameter is rather heavy handed and applies to all single queue devices. Not to mention that adding this parameter to the kernel arguments is in essence similar to the udev rule addition: action from the system user is necessary to achieve a correct configuration even though that could easily be done automatically from within the block layer. In the meantime, documenting properly that the deadline scheduler has a special relationship with zoned block devices is still a nice thing to do. But yes, dm-zoned-tools may not be the best place to do that. A simple text file under Documentation/block may be better. At the very least, Documentation/block/deadline-iosched.txt should have some mention of its almost mandatory use with zoned block devices. I will work on that. >> 2) backlist f2fs devices > > There should porbably be support in dm-zoned for detecting whether a > zoned device was formatted with f2fs (assuming there is a known f2fs > superblock)? That would certainly be nice to have and would not be too hard to code in dmzadm. I can add that and do what for instance mkfs.ext4 or mkfs.xfs do, which is ask for confirmation or bail out if an existing valid FS format is detected on the disk. That said, such test are far from common. mdadm for instance will happily format and start an array using unmounted disks with valid file systems on them. >> 3) run dmsetup for the rest of devices > > automagically running dmsetup directly from udev to create a dm-zoned > target is very much wrong. It just gets in the way of proper support > that should be add to appropriate tools that admins use to setup their > zoned devices. For instance, persistent use of dm-zoned target should > be made reliable with a volume manager.. > > In general this udev script is unwelcome and makes things way worse for > the long-term success of zoned devices. > > I don't dispute there is an obvious void for how to properly setup zoned > devices, but this script is _not_ what should fill that void. Fair points. I agree that it is hackish. The intent was as you say to temporarily fill a void. But granted, temporary hacks tend to stay (too) long around and can in the end get in the way of clean solutions. I will start looking into a proper fix for dm-zoned setup persistence. > So a heartfelt: > > Nacked-by: Mike Snitzer <snitzer@redhat.com> Understood. I will revert and not document this udev rule in dm-zoned-tools. I was a little too quick in applying this patch and did not wait to see comments first. My bad. Thank you for your comments. Best regards. -- Damien Le Moal, Western Digital ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 0:11 [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup Luis R. Rodriguez @ 2018-06-14 16:19 ` Bart Van Assche 2018-06-14 12:38 ` Mike Snitzer 2018-06-14 16:19 ` Bart Van Assche 2 siblings, 0 replies; 22+ messages in thread From: Bart Van Assche @ 2018-06-14 16:19 UTC (permalink / raw) To: Damien Le Moal, mcgrof Cc: mwilck, linux-block, agk, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal, zren T24gV2VkLCAyMDE4LTA2LTEzIGF0IDE3OjExIC0wNzAwLCBMdWlzIFIuIFJvZHJpZ3VleiB3cm90 ZToNCj4gVGhpcyB0cmllcyB0byBwdXQgYSBiaXQgb2YgdGhpcyB0cmliYWwga25vd2xlZGdlIGlu dG8gYW4gaW5pdGlhbCB1ZGV2DQo+IHJ1bGUgZm9yIGRldmVsb3BtZW50IHdpdGggdGhlIGhvcGVz IExpbnV4IGRpc3RyaWJ1dGlvbnMgY2FuIGxhdGVyDQo+IGRlcGxveS4gVGhyZWUgcnVsZSBhcmUg YWRkZWQuIE9uZSBydWxlIGlzIG9wdGlvbmFsIGZvciBub3csIGl0IHNob3VsZCBiZQ0KPiBleHRl bmRlZCBsYXRlciB0byBiZSBtb3JlIGRpc3RyaWJ1dGlvbi1mcmllbmRseSBhbmQgdGhlbiBJIHRo aW5rIHRoaXMNCj4gbWF5IGJlIHJlYWR5IGZvciBjb25zaWRlcmF0aW9uIGZvciBpbnRlZ3JhdGlv biBvbiBkaXN0cmlidXRpb25zLg0KPiANCj4gMSkgc2NoZWR1bGVyIHNldHVwDQo+IDIpIGJhY2ts aXN0IGYyZnMgZGV2aWNlcw0KPiAzKSBydW4gZG1zZXR1cCBmb3IgdGhlIHJlc3Qgb2YgZGV2aWNl cw0KDQpIZWxsbyBMdWlzLA0KDQpJIHRoaW5rIGl0IGlzIHdyb25nIHRvIHBhY2thZ2UgdGhlIHpv bmVkIGJsb2NrIGRldmljZSBzY2hlZHVsZXIgcnVsZSBpbiB0aGUNCmRtLXpvbmVkLXRvb2xzIHBh Y2thZ2UuIFRoYXQgdWRldiBydWxlIHNob3VsZCBiZSBhY3RpdmF0ZWQgd2hldGhlciBvciBub3Qg dGhlDQpkbS16b25lZC10b29scyBwYWNrYWdlIGhhcyBiZWVuIGluc3RhbGxlZC4gSGF2ZSB5b3Ug Y29uc2lkZXJlZCB0byBzdWJtaXQgdGhlDQp6b25lZCBibG9jayBkZXZpY2Ugc2NoZWR1bGVyIHJ1 bGUgdG8gdGhlIHN5c3RlbWQgcHJvamVjdCBzaW5jZSB0b2RheSB0aGF0DQpwcm9qZWN0IGluY2x1 ZGVzIGFsbCBiYXNlIHVkZXYgcnVsZXM/DQoNCj4gKyMgWm9uZWQgZGlza3MgY2FuIG9ubHkgd29y ayB3aXRoIHRoZSBkZWFkbGluZSBvciBtcS1kZWFkbGluZSBzY2hlZHVsZXIuIFRoaXMgaXMNCj4g KyMgbWFuZGF0ZWQgZm9yIGFsbCBTTVIgZHJpdmVzIHNpbmNlIHY0LjE2LiBJdCBoYXMgYmVlbiBk ZXRlcm1pbmVkIHRoaXMgbXVzdCBiZQ0KPiArIyBkb25lIHRocm91Z2ggYSB1ZGV2IHJ1bGUsIGFu ZCB0aGUga2VybmVsIHNob3VsZCBub3Qgc2V0IHRoaXMgdXAgZm9yIGRpc2tzLg0KPiArIyBUaGlz IG1hZ2ljIHdpbGwgaGF2ZSB0byBsaXZlIGZvciAqYWxsKiB6b25lZCBkaXNrcy4NCj4gKyMgWFhY OiB3aGF0IGFib3V0IGRpc3RyaWJ1dGlvbnMgdGhhdCB3YW50IG1xLWRlYWRsaW5lID8gUHJvYmFi bHkgZWFzeSBmb3Igbm93DQo+ICsjICAgICAgdG8gYXNzdW1lIGRlYWRsaW5lIGFuZCBsYXRlciBo YXZlIGEgbWFwcGluZyBmaWxlIHRvIGVuYWJsZQ0KPiArIyAgICAgIG1xLWRlYWRsaW5lIGZvciBz cGVjaWZpYyBzZXJpYWwgZGV2aWNlcz8NCj4gK0FDVElPTj09ImFkZHxjaGFuZ2UiLCBLRVJORUw9 PSJzZCpbITAtOV0iLCBBVFRSU3txdWV1ZS96b25lZH09PSJob3N0LW1hbmFnZWQiLCBcDQo+ICsJ QVRUUntxdWV1ZS9zY2hlZHVsZXJ9PSJkZWFkbGluZSINCg0KSSB0aGluayBpdCBpcyB3cm9uZyB0 byBsaW1pdCB0aGlzIHJ1bGUgdG8gU0NTSSBkaXNrcyBvbmx5LiBXb3JrIGlzIG9uZ29pbmcgdG8N CmFkZCB6b25lZCBibG9jayBkZXZpY2Ugc3VwcG9ydCB0byB0aGUgbnVsbF9ibGsgZHJpdmVyLiBU aGF0IGlzIGEgYmxvY2sgZHJpdmVyDQphbmQgbm90IGEgU0NTSSBkcml2ZXIuIEkgdGhpbmsgdGhl IGFib3ZlIHVkZXYgcnVsZSBzaG91bGQgYXBwbHkgdG8gdGhhdCBibG9jaw0KZHJpdmVyIHRvby4N Cg0KUmVnYXJkaW5nIGJsay1tcSwgZnJvbSB0aGUgbXEtZGVhZGxpbmUgc291cmNlIGNvZGU6DQoJ LmVsZXZhdG9yX2FsaWFzID0gImRlYWRsaW5lIiwNCg0KSW4gb3RoZXIgd29yZHMsIHRoZSBuYW1l ICJkZWFkbGluZSIgc2hvdWxkIHdvcmsgYm90aCBmb3IgbGVnYWN5IGFuZCBmb3IgYmxrLW1xDQpi bG9jayBkZXZpY2VzLg0KDQpUaGFua3MsDQoNCkJhcnQuDQoNCg0K ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup @ 2018-06-14 16:19 ` Bart Van Assche 0 siblings, 0 replies; 22+ messages in thread From: Bart Van Assche @ 2018-06-14 16:19 UTC (permalink / raw) To: Damien Le Moal, mcgrof Cc: mwilck, linux-block, agk, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal, zren On Wed, 2018-06-13 at 17:11 -0700, Luis R. Rodriguez wrote: > This tries to put a bit of this tribal knowledge into an initial udev > rule for development with the hopes Linux distributions can later > deploy. Three rule are added. One rule is optional for now, it should be > extended later to be more distribution-friendly and then I think this > may be ready for consideration for integration on distributions. > > 1) scheduler setup > 2) backlist f2fs devices > 3) run dmsetup for the rest of devices Hello Luis, I think it is wrong to package the zoned block device scheduler rule in the dm-zoned-tools package. That udev rule should be activated whether or not the dm-zoned-tools package has been installed. Have you considered to submit the zoned block device scheduler rule to the systemd project since today that project includes all base udev rules? > +# Zoned disks can only work with the deadline or mq-deadline scheduler. This is > +# mandated for all SMR drives since v4.16. It has been determined this must be > +# done through a udev rule, and the kernel should not set this up for disks. > +# This magic will have to live for *all* zoned disks. > +# XXX: what about distributions that want mq-deadline ? Probably easy for now > +# to assume deadline and later have a mapping file to enable > +# mq-deadline for specific serial devices? > +ACTION=="add|change", KERNEL=="sd*[!0-9]", ATTRS{queue/zoned}=="host-managed", \ > + ATTR{queue/scheduler}="deadline" I think it is wrong to limit this rule to SCSI disks only. Work is ongoing to add zoned block device support to the null_blk driver. That is a block driver and not a SCSI driver. I think the above udev rule should apply to that block driver too. Regarding blk-mq, from the mq-deadline source code: .elevator_alias = "deadline", In other words, the name "deadline" should work both for legacy and for blk-mq block devices. Thanks, Bart. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup 2018-06-14 16:19 ` Bart Van Assche (?) @ 2018-06-14 17:44 ` Luis R. Rodriguez -1 siblings, 0 replies; 22+ messages in thread From: Luis R. Rodriguez @ 2018-06-14 17:44 UTC (permalink / raw) To: Bart Van Assche Cc: Damien Le Moal, mcgrof, mwilck, linux-block, agk, hare, axboe, yuchao0, ghe, linux-fsdevel, jaegeuk, tchvatal, zren On Thu, Jun 14, 2018 at 04:19:23PM +0000, Bart Van Assche wrote: > On Wed, 2018-06-13 at 17:11 -0700, Luis R. Rodriguez wrote: > > This tries to put a bit of this tribal knowledge into an initial udev > > rule for development with the hopes Linux distributions can later > > deploy. Three rule are added. One rule is optional for now, it should be > > extended later to be more distribution-friendly and then I think this > > may be ready for consideration for integration on distributions. > > > > 1) scheduler setup > > 2) backlist f2fs devices > > 3) run dmsetup for the rest of devices > > Hello Luis, > > I think it is wrong to package the zoned block device scheduler rule in the > dm-zoned-tools package. That udev rule should be activated whether or not the > dm-zoned-tools package has been installed. Have you considered to submit the > zoned block device scheduler rule to the systemd project since today that > project includes all base udev rules? Nope, this is a udev rule intended for developers wishing to brush up on our state of affairs. All I wanted to do is to get my own drive to work at home in a reasonably reliable and generic form, which also allowed me to hop around kernels without a fatal issue. Clearly there is much to discuss still and a roadmap to illustrate. We should update the documentation to reflect this first, and based on that enable then distributions based on that discussion. We should have short term and long term plans. That discussion may have taken place already so forgive me for not knowing about it, I did try to read as much from the code and existing tools and just didn't find anything else to be clear on it. > > +# Zoned disks can only work with the deadline or mq-deadline scheduler. This is > > +# mandated for all SMR drives since v4.16. It has been determined this must be > > +# done through a udev rule, and the kernel should not set this up for disks. > > +# This magic will have to live for *all* zoned disks. > > +# XXX: what about distributions that want mq-deadline ? Probably easy for now > > +# to assume deadline and later have a mapping file to enable > > +# mq-deadline for specific serial devices? > > +ACTION=="add|change", KERNEL=="sd*[!0-9]", ATTRS{queue/zoned}=="host-managed", \ > > + ATTR{queue/scheduler}="deadline" > > I think it is wrong to limit this rule to SCSI disks only. Work is ongoing to > add zoned block device support to the null_blk driver. That is a block driver > and not a SCSI driver. I think the above udev rule should apply to that block > driver too. Sure patches welcomed :) > Regarding blk-mq, from the mq-deadline source code: > .elevator_alias = "deadline", > > In other words, the name "deadline" should work both for legacy and for blk-mq > block devices. Groovy that helps. Luis ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2018-06-15 14:50 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-06-14 0:11 [PATCH] dm-zoned-tools: add zoned disk udev rules for scheduler / dmsetup Luis R. Rodriguez 2018-06-14 10:01 ` Damien Le Moal 2018-06-14 10:01 ` Damien Le Moal 2018-06-14 13:39 ` Bart Van Assche 2018-06-14 13:39 ` Bart Van Assche 2018-06-14 13:42 ` Christoph Hellwig 2018-06-15 11:07 ` Martin Wilck 2018-06-15 11:07 ` Martin Wilck 2018-06-14 12:38 ` Mike Snitzer 2018-06-14 16:23 ` Bart Van Assche 2018-06-14 16:23 ` Bart Van Assche 2018-06-14 17:37 ` Luis R. Rodriguez 2018-06-14 17:46 ` Luis R. Rodriguez 2018-06-14 17:58 ` Mike Snitzer 2018-06-15 9:59 ` Damien Le Moal 2018-06-15 9:59 ` Damien Le Moal 2018-06-15 14:50 ` Mike Snitzer 2018-06-15 9:00 ` Damien Le Moal 2018-06-15 9:00 ` Damien Le Moal 2018-06-14 16:19 ` [PATCH] " Bart Van Assche 2018-06-14 16:19 ` Bart Van Assche 2018-06-14 17:44 ` Luis R. Rodriguez
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.