All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v13 0/9] nvmet: add target passthru commands support
@ 2020-05-14 17:22 ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe

This is v13 of the passthru patchset which is mostly a resend of v12
with Sagi's reviewed-by tags collected.

Below, I'll reiterrate some points I made previously that haven't been
responded to:

I don't think cloning the ctrl_id or the subsysnqn is a good idea.
I sent an email trying to explain why here[1] but there was no response.
In short, I think cloning the ctrl_id will break multipathing over
fabrics and copying the subsysnqn only has the effect of breaking
loopback; the user can always copy the underlying subsysnqn if it
makes sense for their overall system.

I maintain overriding the CMIC bit in the ctrl id is necessary to
allow multipath over fabrics even if the underlying device did
not support multipath.

I also think the black list for admin commands is appropriate, and I
added it based on Sagi's feedback[2]. There are plenty of commands that
may be dangerous like firmware update and format NVM commands, and NS
attach commands won't work out of the box because we don't copy the
ctrl_id. It seems like there's more commands to be careful of than ones
that are that are obviously acceptable. So, I think the prudent course
is blacklisting by default until someone has a usecase and can show
the command is safe seems and makes sense. For our present use cases,
the identify, log page and vendor specific commands are all that we
care about.

A git branch is available here and is based on v5.7-rc5:

https://github.com/sbates130272/linux-p2pmem nvmet_passthru_v13

[1] https://lore.kernel.org/linux-block/247eca47-c3bc-6452-fb19-f7aa27b05a60@deltatee.com/
[2] https://lore.kernel.org/linux-block/e4430207-7def-8776-0289-0d58689dc0cd@grimberg.me/

--

v13 Changes:
  1. Rebased onto v5.7-rc5
  2. Collected Sagi's Reviewed-by tags

v12 Changes:
  1. Rebased onto v5.7-rc1
  2. Collected Sagi's Reviewed-by tags
  3. Per Sagi's feedback implement an whitelist for set/get features and
     audit the features for whether they are suitable to be passed-through

v11 Changes:
  1. Rebased onto v5.6-rc2
  2. Collected Max's Reviewed-By tag

v10 Changes:
  1. Rebased onto v5.5-rc1
  2. Disable all exports in core nvme if CONFIG_NVME_TARGET_PASSTHRU is
     not set and put them near the end of the file with a big fat
     comment (per Christoph)
  3. Don't fake up the vs field: pass it through as is and bump
     it to 1.2.1 if it is below that (per Christoph)
  4. Rework how passthru requests are submitted into the core
     with proper nvme_passthru_start/end handling (per Christoph)
  5. Rework how commands are parsed with passthru hooks in
     parse_admin_cmd() and nvmet_parse_io_cmd() (per Christoph)
  6. Rework commands are handled so they are only done in a work
     item if absolutely necessary (per Christoph)
  7. The data_len hack was dropped as a patchset was introduced to
     remove data_len altogether (per Christoph)
  8. The passthru accounting changes are now in v5.5-rc1
  9. A large number of other minor cleanups that were pointed out by
     Christoph

v9 Changes:
  1. Rebased onto v5.4-rc2 (required adjusting nvme_identify_ns() usage)
  2. Collected Sagi's Reviewed-By Tags
  3. Squashed seperate Kconfig patch into passthru patch (Per Sagi)
  4. Set REQ_FUA for flush requests and remove special casing
     on RQF_IO_STAT (Per Sagi)

v8 Changes:
  1. Rebased onto v5.3-rc6
  2. Collected Max's Reviewed-By tags
  3. Converted admin command black-list to a white-list, but
     allow all vendor specific commands. With this, we feel
     it's safe to allow multiple connections from hosts.
     (As per Sagi's feedback)

v7 Changes:
  1. Rebased onto v5.3-rc2
  2. Rework nvme_ctrl_get_by_path() to use filp_open() instead of
     the cdev changes that were in v6. (Per Al)
  3. Override the cmic bit to allow multipath and allow
     multiple connections from the same hostnqn. (At the same
     time I cleaned up the method of rejecting multiple connections.)
     See Patch 8)
  4. Found a bug when used with the tcp transport (See Patch 10)

v6 Changes:
  1. Rebased onto v5.3-rc1
  2. Rework configfs interface to simply be a passthru directory
     within the existing subsystem. The directory is similar to
     and consistent with a namespace directory.
  3. Have the configfs take a path instead of a bare controller name
  4. Renamed the main passthru file to io-cmd-passthru.c for consistency
     with the file and block-dev methods.
  5. Cleaned up all the CONFIG_NVME_TARGET_PASSTHRU usage to remove
     all the inline #ifdefs
  6. Restructured nvmet_passthru_make_request() a bit for clearer code
  7. Moved nvme_find_get_ns() call into nvmet_passthru_execute_cmd()
     seeing calling it in nvmet_req_init() causes a lockdep warning
     due to nvme_find_get_ns() being able to sleep.
  8. Added a check in nvmet_passthru_execute_cmd() to ensure we don't
     violate queue_max_segments or queue_max_hw_sectors and overrode
     mdts to ensure hosts don't intentionally submit commands
     that will exceed these limits.
  9. Reworked the code which ensures there's only one subsystem per
     passthru controller to use an xarray instead of a list as this is
     simpler and more easily fixed some bugs triggered by disabling
     subsystems that weren't enabled.
 10. Removed the overide of the target cntlid with the passthru cntlid;
     this seemed like a really bad idea especially in the presence of
     mixed systems as you could end up with two ctrlrs with the same
     cntlid. For now, commands that depend on cntlid are black listed.
 11. Implement block accounting for passthru so the target can track
     usage using /proc/diskstats
 12. A number of other minor bug fixes and cleanups

v5 Changes (not sent to list, from Chaitanya):
  1. Added workqueue for admin commands.
  2. Added kconfig option for the pass-thru feature.
  3. Restructure the parsing code according to your suggestion,
     call nvmet_xxx_parse_cmd() from nvmet_passthru_parse_cmd().
  4. Use pass-thru instead of pt.
  5. Several cleanups and add comments at the appropriate locations.
  6. Minimize the code for checking pass-thru ns across all the subsystems.
  7. Removed the delays in the ns related admin commands since I was
     not able to reproduce the previous bug.

v4 Changes:
  1. Add request polling interface to the block layer.
  2. Use request polling interface in the NVMEoF target passthru code
     path.
  3. Add checks suggested by Sagi for creating one target ctrl per
     passthru ctrl.
  4. Don't enable the namespace if it belongs to the configured passthru
     ctrl.
  5. Adjust the code latest kernel.

v3 Changes:
  1. Split the addition of passthru command handlers and integration
     into two different patches since we add guards to create one target
     controller per passthru controller. This way it will be easier to
     review the code.
  2. Adjust the code for 4.18.

v2 Changes:
  1. Update the new nvme core controller find API naming and
     changed the string comparison of the ctrl.
  2. Get rid of the newly added #defines for target ctrl values.
  3. Use the newly added structure members in the same patch where
     they are used. Aggregate the passthru command handling support
     and integration with nvmet-core into one patch.
  4. Introduce global NVMe Target subsystem list for connected and
     not connected subsystems on the target side.
  5. Add check when configuring the target ns and target
     passthru ctrl to allow only one target controller to be created
     for one passthru subsystem.
  6. Use the passthru ctrl cntlid when creating the
     target controller.


Chaitanya Kulkarni (1):
  nvmet-passthru: Introduce NVMet passthru Kconfig option

Logan Gunthorpe (8):
  nvme-core: Clear any SGL flags in passthru commands
  nvme: Create helper function to obtain command effects
  nvme: Move nvme_passthru_[start|end]() calls to common helper
  nvme-core: Introduce nvme_ctrl_get_by_path()
  nvme: Export existing nvme core functions
  nvmet-passthru: Add passthru code to process commands
  nvmet-passthru: Add enable/disable helpers
  nvmet-configfs: Introduce passthru configfs interface

 drivers/nvme/host/core.c        | 229 +++++++------
 drivers/nvme/host/nvme.h        |  14 +
 drivers/nvme/target/Kconfig     |  12 +
 drivers/nvme/target/Makefile    |   1 +
 drivers/nvme/target/admin-cmd.c |   7 +-
 drivers/nvme/target/configfs.c  | 103 ++++++
 drivers/nvme/target/core.c      |  13 +-
 drivers/nvme/target/nvmet.h     |  52 +++
 drivers/nvme/target/passthru.c  | 547 ++++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   4 +
 10 files changed, 889 insertions(+), 93 deletions(-)
 create mode 100644 drivers/nvme/target/passthru.c


base-commit: 8f3d9f354286745c751374f5f1fcafee6b3f3136
prerequisite-patch-id: 9d7238f8634ac0218556c6ea3b290ca3f9a9bfeb
prerequisite-patch-id: 7c28feea71d45760f162c9268e373af31a5906a4
prerequisite-patch-id: 38002717cce1b2917ba085e4107c08e1967bdc73
prerequisite-patch-id: bdf06c1eaeffba1067e28524281bcf85672f0ec6
prerequisite-patch-id: 6b36d776fa495e0a41a82efa5150139434bf3774
prerequisite-patch-id: 4989b0e17467b5c109c97875485ada07a561a98d
prerequisite-patch-id: 31a7495031ecb0d5f8f6a1ea35690a225d59f09d
prerequisite-patch-id: df18b77c596089e4378ae39dea3cd56cba81413d
prerequisite-patch-id: 83fc9c4937e86e1374b822c77b83e79fcda70c6c
prerequisite-patch-id: fe31fd60462907af7739d828e1da7477dd0c57fa
prerequisite-patch-id: cd157017f57ef7982d0aa6f0bd23e46530ecaa42
prerequisite-patch-id: ce4515317fd337e8ebb0f607644a131a4ca37ebd
prerequisite-patch-id: ec33e99ce296fdc99c2df2f50fc5b338bf44cf1c
prerequisite-patch-id: 66625c300e60dc72782a20118490f2998bad1b01
prerequisite-patch-id: d996f3880062d0f14d8fdd29e605282fe2f27035
prerequisite-patch-id: aed2a18de4fd4d167ff1c8255515e83ef04681d6
prerequisite-patch-id: 26939059a94177d43ae80e1291c75fd30a9928c7
prerequisite-patch-id: 3300246b7a6b0f592eea2277f887fff13f625f88
prerequisite-patch-id: 309fb4ee0caaa5f1b05154fc8d7b2967c7e6ac6e
prerequisite-patch-id: 710eb598f7fc11ba8e3240b43d13a82e01e35d83
prerequisite-patch-id: 9726bfa4b8e2a21fa50a819e1ee732d6ab369c31
prerequisite-patch-id: 5cf881ddbc6748cc2ce12293aa9a32a5c9269ef9
prerequisite-patch-id: 837840a24eb1efd4981270152aabd27b5653fc02
prerequisite-patch-id: f2c07cdd0fb2677ac655708acbda9a016d6e67a7
prerequisite-patch-id: 334c0e74332d15e15ed31763c0788ce892e287ad
prerequisite-patch-id: 835912e0a297405cccf128d4d7b8a4703244ff61
prerequisite-patch-id: 92f709c508faaf58207230a0122bcbb584a7f8ef
prerequisite-patch-id: a8c120b5e08269cca4d08412507abd2c93a34e0b
prerequisite-patch-id: 68cdf468ceb9a2b665fcf660ff1d9dc90b308c32
prerequisite-patch-id: ec86520c4537b8691623c6275765be46482e1d1e
prerequisite-patch-id: 7f48f11dd3cf0d4f1e9c155813a23561f86d76ec
prerequisite-patch-id: f12b6879d894861d59f0a52c65725c3a663dd0d0
prerequisite-patch-id: 2f55f5384e913e89819f7595bf06a6fc71074f72
prerequisite-patch-id: 66abb21c2745c20251f07d5450e18ba59ac1e3d9
prerequisite-patch-id: dc1159f014fcf438b10f6292d2affd8c1ba7c545
prerequisite-patch-id: ce9f25ee8122b61b303cd1e4ee71792e820e15fa
prerequisite-patch-id: 2629a09c4df6904eee0fa3c624065fefcae9a9ba
prerequisite-patch-id: 25e91f3381c956f3bf512f0f76c26e9c4917f3bd
prerequisite-patch-id: 416fdbc13fc712df5ed66ae0dd2cacd5fc45dc6d
prerequisite-patch-id: 2c688cc50d25eb016f80ca3ce130aef81afe61aa
prerequisite-patch-id: d738f17e5e853308d424caf2a8a16364b6590424
prerequisite-patch-id: d33f8d976c3fe9905a89bcd16a4ed3d54f4f781a
prerequisite-patch-id: 03bac0df161728e66d61993a6a847f0af2006f05
prerequisite-patch-id: 1029d41e51b1f6290746af3f4f05043f604c0b1b
prerequisite-patch-id: c6572f5733639e8f8f9d7cc0e8a246f1a6105c92
prerequisite-patch-id: 0badf7d8494fabc30a2967cd1faaddc8d3f73a5a
prerequisite-patch-id: 021e29c27905d8cdb1b47edc4a22321082a425a6
prerequisite-patch-id: 68569f42dfa917c069fd274bdde8a79b4fab3f48
prerequisite-patch-id: 1e2a91dc91a16102e9a9a459652ec7a8b7b5e677
prerequisite-patch-id: b1d458dfe2a8df06a7f9f9fe4c054672846ff7b6
prerequisite-patch-id: cc8bbd68781d9eb361eeb7ada60bb2d611c7a420
prerequisite-patch-id: 9d2a1df98901f9623bd0747a939a4369a385cc4c
prerequisite-patch-id: f3f9fefacd4daaa2968eeba36a9b317960dd6fc7
prerequisite-patch-id: 88dce86e6313e500889e7e0d0e4b2b7a2512262a
prerequisite-patch-id: 977b9cffb2a731d1d910c2c3c5751fccf2635b4a
prerequisite-patch-id: 9fd4c8c492fe5af9532e0292ab8d11a3105e8dd0
prerequisite-patch-id: e409f0f15c2d61a82dafdd34fcdff3a6d1919904
prerequisite-patch-id: c7c5151a581b577edbe80c3f1fd08e4d268bdadc
prerequisite-patch-id: 8fa60cc8887c6b24c0d6256ac2324c95c385e417
prerequisite-patch-id: d06da3377580154e7053958b47b2f8e69fba47db
prerequisite-patch-id: e0a1cd7adceadc7375e9fa2c2d71bd6c40decd84
prerequisite-patch-id: e8fc76121981dde48008657c3e3b04e625e26db2
prerequisite-patch-id: 4733ddcad5dd0658c25dc931c6368043daa8cd64
prerequisite-patch-id: a47b35a139038eb8776a2c8151380dc9e5945c34
prerequisite-patch-id: d1beb3f116818b224d79ce84784eaeb9ad037c4a
prerequisite-patch-id: 43fd2c8a5f4fa7e03f230b69a8dddd303fe49ac1
prerequisite-patch-id: 0dbe9548cfd6f249e101472be3069dcaba789979
prerequisite-patch-id: f84912cf01bea3e3bb2b22176caa92c581ff4f9b
prerequisite-patch-id: 879647838a9abeedf35ecadfcc95863ea4206db9
prerequisite-patch-id: 7c2791d63f0861316be998af4fc57c4b84cdbc5d
prerequisite-patch-id: 1c1c1674fc358312b470434e0be4c9feec5e5b63
prerequisite-patch-id: b0c5e83a0988a4514534970ab0245f5c943e49af
prerequisite-patch-id: 056e3603e23a67d9b050c041f8acb442c209244d
prerequisite-patch-id: 1877c0cf9459be83e044caf047a7698d16e744b9
prerequisite-patch-id: 4c907971f128c39db73b42394212cf5f79ba2c24
prerequisite-patch-id: 5f0c6241b518718570a362f2cea383ec5a65c732
prerequisite-patch-id: 6aa3b9dc5fdc13f4826f7017fc38975ac79e2896
prerequisite-patch-id: 9869e3ce9f6c3266d06d990f4e5b9d76f049d250
prerequisite-patch-id: 79e8ff4985dceb95b7c57ff232b4f5ba048f5ba0
prerequisite-patch-id: 2761fe023c74185047c24e5b5383e9ab95e8f2c1
prerequisite-patch-id: cbf6d86a18b4eca7ed99ddc4bd0c529cca38acb1
prerequisite-patch-id: 5f204bdb5f8fb455163f75778398c57e4df4745a
prerequisite-patch-id: 31f165cf29f08c57fc54fa7147d6f86a5faeaaed
prerequisite-patch-id: e8ded52ff2088a6067d651478d632501e415856d
prerequisite-patch-id: 6a463168443d4abf5c96f4bc7774d8e35794f4a3
prerequisite-patch-id: 537f61371635c4532bfeeaa5bd915ea97da5d201
prerequisite-patch-id: f68b88e7ba451ba5fc11da0997feb8886e290b99
prerequisite-patch-id: ef8fd30cae84eefc36b4092e16824f4732d604fa
prerequisite-patch-id: 6f90f5d2445cb332de0dc88395043a1a3c30267b
prerequisite-patch-id: 392d8792b9b1ed524fecb8f24c4756d655540d2c
prerequisite-patch-id: b1db472d486e1aa09bad7a488e5b54569e5868da
prerequisite-patch-id: 0e8b6d45cfbedd6d038602b6586952b44bcad8b0
prerequisite-patch-id: 2e6cc25733593b20256fe81980251abb24a913d6
prerequisite-patch-id: 32ae2785673ec8cdcfa6e05ace0ca10174071841
prerequisite-patch-id: 712bf5fec4c2d1280a5d8459977fe4c169c6d7f6
prerequisite-patch-id: 3d71f247bd9d6dc16bb59b1fc06a9a1ca6ba92b9
prerequisite-patch-id: 114f08b1cf5430d217d0e2caf6200c30aff3b121
prerequisite-patch-id: 782cea64700bc05c39bceb11ad313fb0a81c7df2
prerequisite-patch-id: bdbb9de7b0bcbaec6148437f14ad4ab077cd209b
prerequisite-patch-id: 5a29e15ec7c4dd2e7ad3889337b52c4e998eab57
prerequisite-patch-id: 4ad82fb5a21bebc80fb27f9e336665b9e72a1061
prerequisite-patch-id: 85d6ac957b30dadb251348951704319f3b238ec7
prerequisite-patch-id: 8813d16ecb6403951418bc1421941b753061f22a
prerequisite-patch-id: fc9e9b07db5d69e5453a20bb6bc451d1cd41616a
prerequisite-patch-id: 206d473ac8430bfb8310d6da95a1dddc6f6bcce3
prerequisite-patch-id: ee9ab694471ba519d2aead7231cd0f389192b1db
prerequisite-patch-id: d8a1856cfd49ad3dc12a160a4c287ed65940b9df
prerequisite-patch-id: 61dd08aec04d81138d09046036bc7cc4e32af2e3
prerequisite-patch-id: 30805f6b9c1f0044d53e92a0650c0410cc0acb38
prerequisite-patch-id: 4bc010c9f309625577bbdfe0e192c4f0d12f39ff
prerequisite-patch-id: 6afd9eeac577cb44c7f3180dbb9910fc17b17fc2
prerequisite-patch-id: 06b6d2c90cff7212519653374b5cf9ff87d0a50b
prerequisite-patch-id: f049381da80b9fc091c0ea9ce2270fc32d556035
prerequisite-patch-id: 5884d691f5b9f68e0c8a4069b3f1924e1bf731ef
prerequisite-patch-id: 32ae4b7ad9d9d3c0b8918219aec4390a334f149b
prerequisite-patch-id: 8a8bfc45bf2ebaf313031874db313444ef57f9f3
prerequisite-patch-id: 9a7bc44a5058d76798ed2479541b4de821145e78
prerequisite-patch-id: 635392e4d03dda79c324e215b27e782780a84161
prerequisite-patch-id: 1a50429cbb407fcb837722125364a04c7e8a9945
prerequisite-patch-id: baea216dbd90de43fb7eb98e8903a37b3fcc7f3c
prerequisite-patch-id: 3123868ff554a4e3a870fccd87034c3989167c56
prerequisite-patch-id: 6cd01a1482e406a3ac18fff08c3dce5073537434
prerequisite-patch-id: a8efe323e16316a40e75c135fe167465ff12ae22
prerequisite-patch-id: 4c0e8230bdf02ac424d3c3faef550ec9df9ebd05
prerequisite-patch-id: 970565876b8175c9979e9f9b413dd2f299f18c05
prerequisite-patch-id: addd3ea0c109ee8140a0f48f96cf1fbca5561a75
prerequisite-patch-id: 658d60c0042afabdcfe2792f2573e434e218ea7e
prerequisite-patch-id: 29008abb12edd72e8d9a7966ea93ef201635e0d0
prerequisite-patch-id: ef3b5095f29f83a82f379e3c813f1a3f5dba2f86
prerequisite-patch-id: 2b63f688032d142d3e1bc441ce103c77bf176f2b
prerequisite-patch-id: 939a366acfe1c510c168177440d735a178894bd0
prerequisite-patch-id: d97a8ad2a2af2142e3515794e0b692915212649c
prerequisite-patch-id: 6f27aeacbe016cb3d5b6931fee414f0c9b3c90d5
prerequisite-patch-id: a087411e8bc18caee7843d9d109a64fd83603376
prerequisite-patch-id: 2b886a469cbcd6188916a8ad2f42fb5c54bccac4
prerequisite-patch-id: b2c199b3ac6785ea456dbf4b87674899c9610ec8
prerequisite-patch-id: ccc4b2a34e0ca2ec9d88174870261efc6f47cdda
prerequisite-patch-id: 12ada6abaf98de410de33fec8bbc6245dacc2442
prerequisite-patch-id: 855406ba7270f034f3a88e6b9016376eafb31041
prerequisite-patch-id: 54419979f93951d3c7b656283f040d644c1bc526
prerequisite-patch-id: 084004427450ef334207220746099472acfa642a
prerequisite-patch-id: be329de732b4031c5a2cd686d99e8433eb62dc74
prerequisite-patch-id: 5a4dfbd0b614a8c015f20ae4185923e4d34c27cc
prerequisite-patch-id: 0af122faee9cf63e42604c25edcb70a3bb5e4d74
prerequisite-patch-id: b8dccba5b802c9548a48b5dec5bddf2f18e8ec12
prerequisite-patch-id: bc8f86dce2851ccd966c4a05a60270b6b08f8852
prerequisite-patch-id: 6f0b647063e0c34df925a827fabd79c284113951
prerequisite-patch-id: eb0a8c5b6a642c2b2dff1761015a7080e085d2d9
prerequisite-patch-id: 5c3b4f0c510ee2dbcb3ce354d0088c5e04b16894
prerequisite-patch-id: ea9f9ada0ef3d01d0b696e20c4644277ecffa5e7
prerequisite-patch-id: b1b2a0e22f38dae4e74ae09d59faf4aefc216edd
prerequisite-patch-id: d226c2836fc0004ff0a9d24ec3f38c6f263a22a0
prerequisite-patch-id: bf9a2c114cf9238ee172a5e3f284fb6c76dd20cc
prerequisite-patch-id: bace6948de8e0e25b3aa03545282857d87bf5db0
prerequisite-patch-id: 945d775dc66ac35fe43dd4893140820e539dd50f
prerequisite-patch-id: e2a6577071b9c11f867c5ddcf90bc705506415f7
prerequisite-patch-id: a85313a73be71d4f907edd346c902fe726ddcf02
prerequisite-patch-id: c1c3de603a51a8e895fc1e50161b9e2421aa7236
prerequisite-patch-id: 6727a2bc212edaec68f11394a9d1796550366974
prerequisite-patch-id: cda3c68a8cf7b2b0546d6e7a3c9eefb1d85bd52c
prerequisite-patch-id: 6c78fe8b57c1975287c549358d2436fc5e5bf11b
prerequisite-patch-id: c01293028f87a7a23dde1ed081533cb9032b4d3b
prerequisite-patch-id: 464a70cd7259ce1bfe51dced1fafb20a8290f2e3
prerequisite-patch-id: df721edf71c727d3921bf4b654fc99c487efa10f
prerequisite-patch-id: f07686d98b4ea4067122038628fcb05d719e0fd5
prerequisite-patch-id: d4400f0ed4f379bf789b350d7ff02726c1085c12
prerequisite-patch-id: 0982d00b7fdd6e59c96e146eebe50b06d0dd3eee
prerequisite-patch-id: 92fd699e1befda0d97d67509f86332cc9677c0bd
prerequisite-patch-id: 0bff6894ba9031d19a656281c98499b672745d43
prerequisite-patch-id: fcc062426abc792ac9b8caf8bb0e3774c71ce5e9
prerequisite-patch-id: 0007e834bb3ff82d400f66b8772509d3c9aeccbd
prerequisite-patch-id: 985a74df829b8a7002570e4903b8c76da3100c31
prerequisite-patch-id: 8b138daf1e6f0bde8b9fd23f11f286070e91c13f
prerequisite-patch-id: 6740651be7d8682288042571f150db5fdd2ce268
prerequisite-patch-id: db5df3bee5358aea7ae63239631a3e5d8778ddd2
prerequisite-patch-id: 43de8485e4f20f6095055679c675c9e69a8193e4
prerequisite-patch-id: f1d28f7616df3a856903601a12063996182dc00b
prerequisite-patch-id: cec1be3ac07f4316e48cfb93e93917c06026d20c
prerequisite-patch-id: 0b7ea0c1c2b4b5170f87386dd1d6913dea2bb10a
prerequisite-patch-id: 0090e96f64e8c931b1c36aff1be406d01c2dc743
prerequisite-patch-id: a73be4abe2b1dcc6193804a901a2a42b28ee5fad
prerequisite-patch-id: 75c264eaafaea09d6781ceeae699f6ca1ac133a3
prerequisite-patch-id: 3be4a9164b8fe0419232d39c68a00bacd530c883
prerequisite-patch-id: c0a3a19ff5045f257072638db01850a24dd69a8b
prerequisite-patch-id: 8a34f9f93292c6d0e9754200e0312e45cb119e96
prerequisite-patch-id: 40e6c024a047d4423e6a79716450f5fa02b890dc
prerequisite-patch-id: 5d1740a08cc624b72e4e4a73e32074ab7574818c
prerequisite-patch-id: baa4ead75a09f8d06352a0ea46f8e067cb079dcb
prerequisite-patch-id: f92e9f263d928e803a73103077224f6db1d6b905
prerequisite-patch-id: b0840c8c5009a5ea0f9475221ae430d5570a12a4
prerequisite-patch-id: 28b72495731e14acfc1a79668091c2363dfee9c4
prerequisite-patch-id: 81f2998bd78dbbf2e43fbad2465d8dc99e9b6f76
prerequisite-patch-id: ff568c436b5b1891f2c8c415b70f49c5e1ba79cd
prerequisite-patch-id: be4ea8275555f33bd4845acb68e565dbaa8ddf88
prerequisite-patch-id: 9d817b891d1614d97c6c6f6a5828a66db9625e5d
prerequisite-patch-id: 9f51a8baa9a9bc31d83838b11553e5011baedad4
prerequisite-patch-id: 3ac8fc898c030b6def3ef919a6e0d802e9104c6c
prerequisite-patch-id: 9ce11df7962457349322bf09648dcfe041cb4c15
prerequisite-patch-id: 39679f338d2884fb03eb568e4c05cde37c4b7d62
prerequisite-patch-id: 5a45be453070dae7523274936737228893055962
prerequisite-patch-id: bb46fcc0cc619d485e8aa421263f21a463e9249b
prerequisite-patch-id: 6d7b0777387449f497c646015db8340daa872ec3
prerequisite-patch-id: 57aa5a0ff2be4534319c5dcdb51bae3119655450
prerequisite-patch-id: f5b189582bccd0df51f92c384283fe62a662f361
prerequisite-patch-id: 8a4fab832c2421a3bbad6439aeb9634a78b2e8fa
prerequisite-patch-id: 83040c79ff4e22c528d036db30c41821a5de4eb6
prerequisite-patch-id: 8fdb93adf1b9088c913caca1a5c02a36307880c8
prerequisite-patch-id: f9b180a4a27cfe8e6d0f90be70166cc8bdd14372
prerequisite-patch-id: 3ac039e20a8b5217cd5b53216f7994c570a7d225
prerequisite-patch-id: d52630de6424c8a901e31aa234fb5451b6752e12
prerequisite-patch-id: a1ac61d41cec474193d099963e5a6d5d7eebb02e
prerequisite-patch-id: 243748f84e4c33995898ddf49ad79819661ea754
prerequisite-patch-id: 182e2105b1de611f3aa9fa1a5a6210c61eaebd0d
prerequisite-patch-id: bc92fbcf84a3ee1c3b44e3bdcf24d002a331296b
prerequisite-patch-id: 5257be3ed7f3a576e07c207bd13073f0c8fd1adb
prerequisite-patch-id: c42b55b057761321fbaf9d5e743878b19cb82f78
prerequisite-patch-id: 5ca26f7abf1c760569d2937b3168553a6c0245ba
prerequisite-patch-id: 9c743148e4c66db989af5f42f014d5f6dc79c435
prerequisite-patch-id: 75c649ba4ecb350855cfd43d3694f9c21c6bd934
prerequisite-patch-id: 44cfc7827d69c9f11676d83d4982800f1c4be6e4
prerequisite-patch-id: 110cc5d3f117b16549808a88b677bc55824bc91b
prerequisite-patch-id: 6e92cdcb7107c9217f045b05c859b484a2b70d23
prerequisite-patch-id: b6ddb390ebc52f9e85230c9db9f96b25d5000bc4
prerequisite-patch-id: 5ab0930bfd01508fab76ebb122d5cca85038a99f
prerequisite-patch-id: 1bc1845289729c172edbad7f13fbff782c3c40db
prerequisite-patch-id: 9a03e494dd6a3c0ed7345c24de70cf1579adfb32
prerequisite-patch-id: 901ea696d1d241a665d4ab3e762543988a326b0a
prerequisite-patch-id: 6a3959d15678641360877db80a19b09f7bf469b4
prerequisite-patch-id: 2878038b634ac466889423870dc6b83d59d1534f
prerequisite-patch-id: 070532c394ed3d32887e472b53ca9aa862f36592
prerequisite-patch-id: 11ddba0503ad58efd4ebba64483c4042d026f544
prerequisite-patch-id: 535de08752aa9395c5d80571a3e9713e21a9f861
prerequisite-patch-id: e986af1b3b2777fd612993a13cf3ddf3be0ea729
prerequisite-patch-id: c7dc3290f8ee97db0f290157d99eee748041203d
prerequisite-patch-id: a7d694c65dfff647fbbe680ce167239e6fc14ecf
prerequisite-patch-id: eefb293afc1dc654f4685a15d3074f1426064027
prerequisite-patch-id: 46862f1cd0820ece58de808166a9eff6460aee97
prerequisite-patch-id: c863dbb552254eeb3275b6d3732be8fe337f9b54
prerequisite-patch-id: bf44e01996d18d4155874cf785688432472f87d3
prerequisite-patch-id: 2c1a4a4034b7d9364f19d7eeee1cdb62411d736b
prerequisite-patch-id: b85804f5f63ce74bd0467a3d63cfc89688168065
prerequisite-patch-id: a724b02de89d063bde2cbfb069d583f11cbdc773
prerequisite-patch-id: 787bc2edc5b0eae2afcc0e456dbcdc3cead1fbf6
prerequisite-patch-id: 8f9ab0cb5f302a59bab15a1615ae40e2e594eab7
prerequisite-patch-id: 2220435d5a540e56319c4dd535549c0a7ec9d958
prerequisite-patch-id: e640a43975071c711bf2c8ce12e9e5480a7a017f
prerequisite-patch-id: 0f0c7bab25e97324eda731d5f9ecb2fcc666846c
prerequisite-patch-id: 4e10c81a0151f1b8c817a33810e4d675ed0e0d0d
prerequisite-patch-id: 06bf448d52b081e95329f7808b834d2656144ec3
prerequisite-patch-id: 06fc39732343a54f854b7c58514757ff0a5dbe75
prerequisite-patch-id: 7901ee09409c6bdd2865fbd38815eb998976aad9
prerequisite-patch-id: 06043139f97ba25f667f331e8944bb8f43002e2c
prerequisite-patch-id: 0fa78c1ddc1f1c8323dcf889bfdc5efb205069ce
prerequisite-patch-id: cf4d25c9e9ce9b8f735aed9c8257799f96309a64
prerequisite-patch-id: 273481041d1c4cc4975dde954d022b913e6410f5
prerequisite-patch-id: a626031549bfb89a292fd3798da5b373ebaae10b
prerequisite-patch-id: 6e3fad3f875bf8f1828ea93c7108146a97deb8a7
prerequisite-patch-id: e7a76a1aa9027fc7df9e279fac3bbdcc59e08742
prerequisite-patch-id: 963ef2431922e5e3ae870eb17c541420cc5bfeda
prerequisite-patch-id: db67610ca7b90e2cc1e764b3e375d38f06c2d004
prerequisite-patch-id: d989c522c82b45b3a03c6fec7594d7297f09bcec
prerequisite-patch-id: 85be256c94bed1da6b9ebb20e77b2b055783e55e
prerequisite-patch-id: a1a7e6e70582380d2f87dae8631814981bf5c287
prerequisite-patch-id: b42fe63328c91167725456f72605f155e7d35b5f
prerequisite-patch-id: 9a1f84c4d960c704059475b96e0d472ae2785718
prerequisite-patch-id: 73482edbcdf7cad22130c3c5f8fd19f72f4016b4
prerequisite-patch-id: 897e2ecbce6e9e2e015e23311ca0d0b8e770e5a0
prerequisite-patch-id: 5d1c8026b6edbf9502b477a88a7a473235f6833b
prerequisite-patch-id: 85e38b1bf4ac22908d6eb70518f643bfcaab36cb
prerequisite-patch-id: a1b1f7b720e0b0d9c5cea11aad878d9d20c4e9fe
prerequisite-patch-id: 2555ed927ed6531dbc8dfeff646721ce2e274534
prerequisite-patch-id: 49dd55b3ef8c2a67f8230d052073e96951a51a3b
prerequisite-patch-id: 92950220b15d4ca7b3c52870c4efbc21d0a806ea
prerequisite-patch-id: 54394e929bb94e0c53de402c715a6ed6377f336b
prerequisite-patch-id: 1ab1958b4aa4f8eedec46784a81cb3f13904061f
prerequisite-patch-id: 837fa2e1fa98d9533dda7ad634724dfcf64556f2
prerequisite-patch-id: 173cdcebf307869a6b6add803ef8139fb7ba58e8
prerequisite-patch-id: 147368cc73f4edc0eac6bec7e25803e476d4f3d9
prerequisite-patch-id: ae33a2cfde38bc2f42fe9e7348eb3f3803c16d45
prerequisite-patch-id: 9b8f4d34da917ef50bd8ef8e1ceb38d089131f54
prerequisite-patch-id: 31306cca2510419f8b0a554e3f4572428fabcb29
prerequisite-patch-id: 6051a56e8865fccf0b528bd75f572dd2d472c3de
prerequisite-patch-id: d5278762f173ba12f888eced152c27b0ce99bc83
prerequisite-patch-id: d1077b3b140d680b4206359912d6bf7c7627fbb0
prerequisite-patch-id: 2ad5206adfa3ff0d81ced89996e7262815eddff0
prerequisite-patch-id: eb7ce19dbd2693a536bc6316ccbac4374a42c40c
prerequisite-patch-id: ef3f2f60463cbc4f33327e7c0b8035e6169c19db
prerequisite-patch-id: dca90149f4c845ddf5e60428e6d69fef4580b827
prerequisite-patch-id: 1013ddc3522d45ead3a0cbb44827206724cc7f5c
prerequisite-patch-id: 3255c6028be9b549db15e7052fa0e3340d8cb4a6
prerequisite-patch-id: 6a95fc82ba5f3cf78990f89dca875b16b32eff3a
prerequisite-patch-id: 1cdb72a06de1f9e5afea8e02952baa366f4052d7
prerequisite-patch-id: 57cb23e1d4acdf5cbbb7174a54581239d35577c1
prerequisite-patch-id: e6a7b4fa9bb1ab451e843847772acfc8a6735069
prerequisite-patch-id: 1a1f2098dbe5268f615e42b90d3d70801f16be73
prerequisite-patch-id: 0c99ae56389a6a5589c84976f7ee427577206974
prerequisite-patch-id: 3073ddba81dddbdaf4fd2709457e9798704c2906
prerequisite-patch-id: dccaecee770dbafc7b93e5ad0db58f530353944c
prerequisite-patch-id: 8f5973e99751fdf81f94e80be83011b0114a65c7
prerequisite-patch-id: f6436859386a9ab728fe4d0e42095340d8d6280f
prerequisite-patch-id: 4f11d53e44f77b0eb96eab0c639014cc40c35a66
prerequisite-patch-id: 0b4cb280efeff9754c5c758c1ef997cac29b72c3
prerequisite-patch-id: 1dd1c3e988260d53a796375c8ef9d1444633acc7
prerequisite-patch-id: 1f687c7042ec3b629a5fcebd1dc83ec15e32cf4e
prerequisite-patch-id: 5879af0cbda416e5aa6c53d88863091d8b1e4407
prerequisite-patch-id: 6298f168fb6ef34ced7ae01afc294814146f90d9
prerequisite-patch-id: 9f8f121af776c2dd177e4ed66b80af1a92635b51
prerequisite-patch-id: 74303a5e89a3024438e16d236b02db78e87c176f
prerequisite-patch-id: 26455bec4a0ebf9bd68b483c891352c5fbe15154
prerequisite-patch-id: 883d9dc3cc0f21aa7237230166d759ec04c86da2
prerequisite-patch-id: f55667a5e87943047aae3c8acb771c945fd60a61
prerequisite-patch-id: 155dd929189593a223c9b87971fda6cfd4ee7d02
prerequisite-patch-id: 2795c504af025d6cf2a7cfdc53f6d3cb2e61aa01
prerequisite-patch-id: 66d437a9cf929a484c38f8816c3e6a159c4afd4f
prerequisite-patch-id: cfc847e49f7c3e9f7595cca9f9c27dd48f58ac31
prerequisite-patch-id: 13b8296ae2593007b44c0ad47b09b040a71b4237
prerequisite-patch-id: b372743afbcfcd99b271c7dc178636192d5554b9
prerequisite-patch-id: c1b86d0e2d874da4acdb11bd35c366f4dfca592b
prerequisite-patch-id: f58b279e8c6e23ad50f8d7c0c135d8e2fbb7dbf9
prerequisite-patch-id: 8e8506e63fc5acb6036ec13fed46cd6f64647b25
prerequisite-patch-id: c97de7d9dc36d59a7b6e0774db9ebeb00ee49cfa
prerequisite-patch-id: 4d22a15fc13dd8c22b68f32411f619f4f5f06031
prerequisite-patch-id: 030d6f481201eb00423b1f3f31fc4ac177ad744b
prerequisite-patch-id: fded371737b462926e4e46799d58bb5f2e662e9c
prerequisite-patch-id: 2b49e6f3184990f4dd090451a0d187beb836a33d
prerequisite-patch-id: 2dc38c769fea0e61d7ea26f5678d8f76ffc8385a
prerequisite-patch-id: aa3059e01d277c5b815df0172d9678ada062bdc1
prerequisite-patch-id: 56a33ee453fedb1a48dbc091298673c836872469
prerequisite-patch-id: d1a830b3c74494e78f2dd1cfca80989e8271db30
prerequisite-patch-id: ab66638db584af150e1b981f30a972f3ae16a678
prerequisite-patch-id: e97e372655090e136b7a3be940472501ebcf8563
prerequisite-patch-id: 11b966644927e3ea8a172a176d253d821c223435
prerequisite-patch-id: b452b372eb1f3ba03b4e352af744207c47756488
prerequisite-patch-id: 2ae8ed5a5867855f4419b42f998298060fefb1f2
prerequisite-patch-id: f9d9b9a274da20370907046f9f146c34ab58facc
prerequisite-patch-id: 3845235703cf6271a301e2126f1f881274eeb979
prerequisite-patch-id: ea1c337039108219d988446047511df63de6a231
prerequisite-patch-id: ed8515321afbc02262e365ea1af572ca51686fff
prerequisite-patch-id: 3df02fb90d43b4ca1d30216b8e2372b8fcaa3438
prerequisite-patch-id: 892ded69018e167f0e0d3a129a3cd9688c969a01
prerequisite-patch-id: fd51e453172c51f232102ff0ea1013f9ea06f086
prerequisite-patch-id: 896f4c9c7b60c7c806c3f89f5ff6816761ed6182
prerequisite-patch-id: 07f9d198c2990fc862f4d8f173e51371575f86ee
prerequisite-patch-id: 30c0bbd00c0aed5a58aacd637e0e2534bf3cf16f
prerequisite-patch-id: 8e8237761c16158f97d5b503b8913a4fbad7c3e6
prerequisite-patch-id: c4144bc4a038f29bc6f68fb2275f0b5ba8f31b60
prerequisite-patch-id: 3587d60e29dc0ca0dfbccb49ab631857f43b1632
prerequisite-patch-id: 85d16e7393dde39c0a14693a8cdb333cfaf0d007
prerequisite-patch-id: 5518761e2fd3b65a3b6bc9faeeef62cc2e8014b6
prerequisite-patch-id: ef94decfa59004cabf20b8981eace30c527f5176
prerequisite-patch-id: 3dcfd91371c737c82e41c63145de1535c8a9dd18
prerequisite-patch-id: 051f528001d215dec9dd066d2e5c340608f2daf6
prerequisite-patch-id: b2900ab223469c927ac55d1da6184286e6038f28
prerequisite-patch-id: 1c5d9553d3829565e87ae5cd7e2d5dbac81ae466
prerequisite-patch-id: de53fd2f0b77d1e61f09340e4ec1c127a191390f
prerequisite-patch-id: 1b9f205c1cef0b2d45e1e08321e31cc239da4789
prerequisite-patch-id: 45b260156c63c5a5bf062713305d9ae282a1e3b0
prerequisite-patch-id: 00286c2f4f8a84177313582fd6fa0524ddb5a098
prerequisite-patch-id: f8ce0a97233d0d537618ab101d0e410cdada3fba
prerequisite-patch-id: 13163fc64a35163f48b07e8f5595376b98d2ced3
prerequisite-patch-id: 3c2a2a0dd7492880a28f3e5cd02d25d9a6f4301c
prerequisite-patch-id: eed5cba1d574dada94e69d1af227392b247810b7
prerequisite-patch-id: f2c72dc4e0206fd4f3122c77b34cc438b8ba5707
prerequisite-patch-id: 9193eab5af26fd4d7c6bfece5b7492716fa502f6
prerequisite-patch-id: 217404b4b4de0b050d1a531d870932c46ecf0028
prerequisite-patch-id: 79d61b95cc07a3b27004ffabd807d24f6ee348f6
prerequisite-patch-id: 1294b9c3ebe6add3621d4fb4e79c4139adba8ac2
prerequisite-patch-id: 47ed8ae19ffcb8ba03f306cd02da2be7398ba95f
prerequisite-patch-id: 03e983abf83abb15c14790ea24bc4081b9269f1b
prerequisite-patch-id: 0de47f21f09c75037fdd26f5ebafb455cfd563cc
prerequisite-patch-id: bf439c65be96d2fe08498345b21da6dd01596ab6
prerequisite-patch-id: 959ff5ba7f097435837bbcc09e541b695cd012e6
prerequisite-patch-id: fc1db2b70dbea6428fba43cd8ee2f00c89104f2b
prerequisite-patch-id: 34b3e2033296413e181840eb9ab477df62217e72
prerequisite-patch-id: 12bb71fb40b298359db885ad1bbdd35ac5fa6523
prerequisite-patch-id: deb822cb734d5727efeb4224b2e4da12177fc46a
prerequisite-patch-id: 7509b386a3f790c4accae1174c11fd47f9559fe2
prerequisite-patch-id: 2de66e3ccaa3f0195072840484241d08a21fb6d1
prerequisite-patch-id: 57deebe1d329286efa8968ff310347642bc3c95d
prerequisite-patch-id: 30debd7f6462ed2aaefedc2ff6dc70476d28b8d0
prerequisite-patch-id: 04259df5dfb493f5b53274e8cc90065e4df4e703
prerequisite-patch-id: 153f9fe32b251f0abb63d87d08b3e3b230eb5b53
prerequisite-patch-id: ae05bd1ba17c4e4de8cbfaca973dc1d91f82f435
prerequisite-patch-id: 95b5c0242332dffed6dc485783a561a302de6492
prerequisite-patch-id: 382fc534cac40dfbe2f5f5a13024eaa3fafbe51e
prerequisite-patch-id: 2920ccf356b76054e0f9821c041183f951235bad
prerequisite-patch-id: cbc3c5805b699f8f02fb88be4b16522dc7af840b
prerequisite-patch-id: bf95630f37b72c1067b0df4628ede4bbae6ef334
prerequisite-patch-id: f382bd06d87893383a9296dd8bab20954a9a574e
prerequisite-patch-id: e88ef31d0d636c9fe1c1ba109f919e2199bdc392
prerequisite-patch-id: a3c7a30ba3325ab1334388a3c4b0c23fddadf8de
prerequisite-patch-id: d5f41560767c1a8ac6e1d8d983671db28aaface2
prerequisite-patch-id: 46f63af6109f65e9eb2ee6d8bbccda49c6172b4f
prerequisite-patch-id: 1a4fb82ef1995e8360691b8ede81502ea6e66813
prerequisite-patch-id: 436e404a9983065961e8230866cebc867d93c948
prerequisite-patch-id: 7b5c64e29651073ac56978199a79bf8ed084cc8b
prerequisite-patch-id: b2746064711f860070344d11a7cb9ae28e4eba5d
prerequisite-patch-id: 83de7b33033f8b7d107d2f9b960a864bcba65b16
prerequisite-patch-id: 01dc2c52438d0dfc501258954d7b324c5d11ae9c
prerequisite-patch-id: 1aa197807862c5439e16463d38ab5e7dfaf25cba
prerequisite-patch-id: 06a40bba59c3acebaa2099565e6f04a76420d573
prerequisite-patch-id: c081c33df5d5642f3015cbf355974d917ae28870
prerequisite-patch-id: ec5e9e149017e942793c397c237c2ae672b3f42a
prerequisite-patch-id: 7edff554cedbe18b8f3bc02a67dac3c915d0210f
prerequisite-patch-id: d00a5bdae7253fdc936ff552a2b58f6f52b3f5a0
prerequisite-patch-id: dc6e332dc81755ef34aeadf8240737f9dccb104c
prerequisite-patch-id: af84a91939a41f95fe2346c16a68aaeb37a4f641
prerequisite-patch-id: db7928bb79bd4aeffeaa60a6f9ff8b065919c94c
prerequisite-patch-id: 0e518075ae3209faa4893bf3187a64ec417ab442
prerequisite-patch-id: 268f2009013420b9487607933ad4aede4a2001f8
prerequisite-patch-id: 8f1ab5221864517cb3fc80441ae71e22db4baf51
prerequisite-patch-id: 62513866538318d3bae35f79177de7d15628097b
prerequisite-patch-id: 0c4ebc06ede9d1f9d3b07100ec76d394ee15c488
prerequisite-patch-id: 056b1e8010d15d36eae737d7413d2ccc0875890e
prerequisite-patch-id: d3160056bb417272c951586cda10d50c6388ee60
prerequisite-patch-id: eed160cfb978683231c670e42a6b205a28679174
prerequisite-patch-id: 58f389a1a5acf34a46436333ca0d909bf022a591
prerequisite-patch-id: 84589052265eb9229ac34659873944ad382b2524
prerequisite-patch-id: 21e68053eebc87cf71485d679666d7d9273d8487
prerequisite-patch-id: e5e1defce72e6945528876f1c2cde92c6c736dae
prerequisite-patch-id: 0341c1c00e5e1d04e5fd1097823b4d4c6dae92f6
prerequisite-patch-id: 96a3b8d529c250712433d2106215cce44dd8f64e
prerequisite-patch-id: 60784817fa91030b9ede4e1737d66d632fd999e3
prerequisite-patch-id: 68862ca84d0a1a785fa974d3ccfc929c70a49f66
prerequisite-patch-id: 7574d97104a0e7f145ee7d34d12cc675cd3d8a2a
prerequisite-patch-id: 55a9abc4ea006c525711c628c09ac7ab37898cc0
prerequisite-patch-id: 4eb1a7075906b49b533a1934bd5a0828def60a97
prerequisite-patch-id: 34c2518b2b49f91a2667e494571a4f8bac48faa3
prerequisite-patch-id: 7debe7e05adc4902deb0a9202e3dd97f29348ce1
prerequisite-patch-id: c9c9666c9c47ca1e5e68ad3a8442c4ae0e78d900
prerequisite-patch-id: b2c5e3b2f4003af18a75446959cae35ea1648bc3
prerequisite-patch-id: 7acd3a08bc7554519acca5819b56bbbdae70ed3d
prerequisite-patch-id: 1db655bf52599eab96b173cfaed4316b230b2bbe
prerequisite-patch-id: cc8dc88d1dcdd62c0a2de7c50cb32ffb8d7f102d
prerequisite-patch-id: 04e2903a8467c09aed7e4f4825d944c67ce4fd17
prerequisite-patch-id: ac1259ca8d62a36d76200d6aac21fd3c6775853d
prerequisite-patch-id: 5c6d9f1d978fa24e829126dc0418c542d51511d0
prerequisite-patch-id: db0645b5b42c0f59515a017b1fe93b4e4cbb2ff7
prerequisite-patch-id: d2769eeb5122bd4cb5fc5ca73f24c21cc9d5d1d5
prerequisite-patch-id: 08ac0bd42d6ee51d6fa74f4a18d540722eb46282
prerequisite-patch-id: dd9a925574c5e158ba425459fca382649b15117e
prerequisite-patch-id: 9217798eec18086c7af4b5cac83c2728962b1b27
prerequisite-patch-id: 73d271d19390a5f1d55bd2c0564e3be8c55a8bfa
prerequisite-patch-id: 8ecab5b382e4617328a76decb063643c08971330
prerequisite-patch-id: 06eedb9537c8908e16c4b73570805f147c0aa7a6
prerequisite-patch-id: 40f0ef1f64d8c8d5c67224eea72e09b8760bd7c6
prerequisite-patch-id: 731760e194dbf5f1a0d12660d1e3491b3a8a41b6
prerequisite-patch-id: cf9f958495047a114a853ce1d160f1f26483e9ae
prerequisite-patch-id: b1ec08a3514e4a4a5c1448299d13076f29d8977d
prerequisite-patch-id: 39e1c80db9e91cdf964cac898098e91436e70a2a
prerequisite-patch-id: 50a76bc20d70be0726a598aae7fc61f6edc36aa0
prerequisite-patch-id: 420c9896e24ec4f30b2f1994fbd08bb356f35909
prerequisite-patch-id: 467135ae1d015370502956d43c8f45dba8adadbd
prerequisite-patch-id: 618de01ea418223993d28780f508f664ae3074b7
prerequisite-patch-id: f5bcf4751282cc66a644b099dc66896c3cb420d2
prerequisite-patch-id: abebb2bffb7ed958c2a84d8121f96de1c84fc820
prerequisite-patch-id: abc36b610493ed09260e7e41f785ab3f6c3c9dd4
prerequisite-patch-id: 5448626e34003a90318069582a565c125179454f
prerequisite-patch-id: ea32d7782ae53cba2722df1472cffdfa594f4aee
prerequisite-patch-id: ff11f21f8b15cb9e4e50e7f27d8e1aa3d60357c9
prerequisite-patch-id: 85f6280796a6f843ed24cd4eb041a8c6621e8eaa
prerequisite-patch-id: 5992b5156d45a690506c0f5a3181f24460309b7f
prerequisite-patch-id: 63a5eed0ba540536e20ce524e2eb646ee451a261
prerequisite-patch-id: f0d0c5e8ecadee7fd64b9238e67586c49f229654
prerequisite-patch-id: 9b91d99b31e3f08bf502496867174735e21db660
prerequisite-patch-id: 237a67d1c83747b8bd2b94eda22e01977d83d919
prerequisite-patch-id: 2420f543c866f1cff1f204e5b3b036e960c30c42
prerequisite-patch-id: 5197dd4a37c0f0c70441ddb6523cf889b749ba32
prerequisite-patch-id: 5b149cf3f0706dec6e6ca8ec1fcb9f368c992b71
prerequisite-patch-id: 798a116f0c76a23e1fadf16b9284ec47f8aa3667
prerequisite-patch-id: 23aa7434ea7f9cacc52d13262c1e0c3fb291192d
prerequisite-patch-id: ffa8d17f55c6364b7a6b5d2c40621481f58088f9
prerequisite-patch-id: 243f8b4e9af1afc35c653ab83e35bac0a107bba5
prerequisite-patch-id: 935c536a4da7629f229b1d727ca8055d437336ba
prerequisite-patch-id: 4eb16edb090d0cf06ebf441609f98be95415dfa6
prerequisite-patch-id: e7829ac271b14df2977a5f070d73641184604145
prerequisite-patch-id: 9f920e698eaff83155a1c58b4bf8bfd3b53a4060
prerequisite-patch-id: a093bfa64e9320f9c4e2823f4bce8ca7e899734c
prerequisite-patch-id: 925957c035444aefb17b52633dc04cc563e26756
prerequisite-patch-id: 91d263f28fbd8ef57fd5ac1193e39e76fce1c9bb
prerequisite-patch-id: 6111f02d7daf8ede020a05dd202789f2eec0e71c
prerequisite-patch-id: fd3f9c62e0e1dcfd03ef7305568e44d1fa5b0572
prerequisite-patch-id: f0afe484af0e7dca6184d7e6266c0bc438d7a82e
prerequisite-patch-id: 031320e7b5b43497bd3a3b035c60d3c2b4590c6c
prerequisite-patch-id: 2412c486cebfd6cb9821c4c0a7e14c9a94e424b6
prerequisite-patch-id: 9ee897efa36f7087265fe0b59fa942147831cf84
prerequisite-patch-id: 78abe009993d877dfc43074f74d7b2c0ee6354e8
prerequisite-patch-id: ee9617761e10146b3932d5fb68435d3958a7c666
prerequisite-patch-id: edd891afc35358394ae991ebb84f8a59d7c3eec2
prerequisite-patch-id: 501fb93103dd732dec4b85ced2a667af30cfe2a4
prerequisite-patch-id: f3b0709ba690b45d9ac6fdbb5037e74e7d2c68c3
prerequisite-patch-id: 3e57f788314aa368144ab1c3ff6ed88b0f95a234
prerequisite-patch-id: 2a1f8aea4e4b1a2d09696c497d00374b8a1391ff
prerequisite-patch-id: ad7e10a9b6a20d0911089096fa56f67f41ea2686
prerequisite-patch-id: 8055d5c76756a428e6c0c9bab808460655e20228
prerequisite-patch-id: fc15f7c10dae0b5a34a1253006f16efd92ae6e4b
prerequisite-patch-id: 634e0f4b8b2e57ddd988da5db73b0a6f468f6bf9
prerequisite-patch-id: 76259d6e33054f6c3cad1e82f34d87adb88892ad
prerequisite-patch-id: 57b0bf73bab8e4c85c5e1ca6e794b9db4089474d
prerequisite-patch-id: 7354887d0fc6f4854f017eb96059cf4d4df57b75
prerequisite-patch-id: f40c6bc0b0e328762ea985479c5a0ef88e87daf8
prerequisite-patch-id: 8d013e9e82ba080c124f59feb8b19c786065048a
prerequisite-patch-id: 972b485a48a93b716e9b9179a15c3c46959a3e09
prerequisite-patch-id: 134e8dc206cfad79cfa3c68ceaf016e97eb2cdb1
prerequisite-patch-id: c7c01e5516a7f55134a4c61ef0415868f8ee66b7
prerequisite-patch-id: d36996df837725609eb046e6d36b7688bfe5f524
prerequisite-patch-id: 06f69d553f14f5af7a8714b03ea700375af896f3
prerequisite-patch-id: 9746cf6e46aa837873d4f6f13a22ffbba7e1ada6
prerequisite-patch-id: 7a0b0c6ec3ab34729eb3d43222672305efdcc323
prerequisite-patch-id: e8d98ce8b389cb1d7ac0e56622e5172618fbea97
prerequisite-patch-id: 3ebb5259fde093ee75deed39cd12045529aaa2ea
prerequisite-patch-id: 0d24e580f73e2c959e816d8dd15a8575bb73196f
prerequisite-patch-id: d2e33c5dfe970bed2d470961f9ad17f5ec71dc50
prerequisite-patch-id: 8504773a58376b3f2cc3cfd8ffcf700707ae7cdc
prerequisite-patch-id: 46315d860bf298e486cd88ff913879bc0af4c2e8
prerequisite-patch-id: 52d24f6fbe20ac26521909c5ab46a30f9e2dfb85
prerequisite-patch-id: 8ac4d5eb605b6ab6da24be349c6062d1acf771a5
prerequisite-patch-id: da6c959b278dd8c00a0116021075d621cdb65730
prerequisite-patch-id: 10090220c0e670d7a426e66d6fea8975a6acf60a
prerequisite-patch-id: 75d054fd4a935c96d181092416852a482a76f8d4
prerequisite-patch-id: db6c9727a795e424251a20f920081725b74da910
prerequisite-patch-id: cf357539b3934e3e493ddeb8f88d5734d52ba93b
prerequisite-patch-id: decea47e13839b2844f5303bbea5fb5659e4d71e
prerequisite-patch-id: 00764a4d6763fef4ada6659305885566f30bee12
prerequisite-patch-id: 60f3a283a56e77e666e51717d1eae8b3969cf78f
prerequisite-patch-id: ef53640393fa6c2e2067d2d263ac0992c46952b9
prerequisite-patch-id: 8caae48bf2e9d7b39608dadfc71d4b3bd3685985
prerequisite-patch-id: fa64087abcbcfbad49eb3ea8ffcf246a4f5dfbac
prerequisite-patch-id: 2d48caf1e8c93ba6210a71411f2fd75c572319c8
prerequisite-patch-id: 370f9da949fce1c586c916f045bfc28985a30ece
prerequisite-patch-id: 94d81d90b5097891839d33afca31c44c44f491d2
prerequisite-patch-id: 0ec93739dc84df97269b394b7e80b6d25d1b6760
prerequisite-patch-id: c4df0ca3a46c7527f02c7c81ce7567dd16c073c0
prerequisite-patch-id: aac3312a808f7ee8c1565cd9048b099956ff5231
prerequisite-patch-id: 173c14d7cc1cff25701efbe850cda32328ee505c
prerequisite-patch-id: 6f52d02aa5e9f89171e388d2aee1662d336270b5
prerequisite-patch-id: db0bedb3d8b059b203acc9aed821c35dfe382931
prerequisite-patch-id: f177a0cc475866c3e8acaddec8d9ea0710e8156c
prerequisite-patch-id: c7cca8ad687385e771c7f33bc05af75730dee6ca
prerequisite-patch-id: 39c7b7573e68a29a7c2d354e51795ad4c1e10801
prerequisite-patch-id: f73abe9607eb118b33a7f746ea916a6623d36af2
prerequisite-patch-id: f644f2653e0ed40155d4b944c2cf419bd300d188
prerequisite-patch-id: 491f90e9299928d3de6e0e9b8110b3bc3d8e089e
prerequisite-patch-id: fa63106e268842265418962017886678d6a33d05
prerequisite-patch-id: 852a6092ad10e01ea086fecf04a4dc2c1da801b4
prerequisite-patch-id: 7f61308971d8fabef648a8dfa0a937827cdff320
prerequisite-patch-id: 0ed80efecd479fa9606a7876c9605251e55ff8a1
prerequisite-patch-id: ef39dfc1a291c75d9ce1fce67d50670d07c7cd35
prerequisite-patch-id: a88f9664e09fb1a98f7300413b149cf9516174cd
prerequisite-patch-id: b68300512f8e688dd232f7af8d34c9189b53fb2e
prerequisite-patch-id: 238f1f3d163560545afa92cb4388403c0e6da0f2
prerequisite-patch-id: f1d92edc95c568075c3dbc5f2b419d9369aff19d
prerequisite-patch-id: f2210af95d27d0d4853dc725947b8e03826caa71
prerequisite-patch-id: e4d0a951916e2b25aeb598b646bdbf21ffd3c0b1
prerequisite-patch-id: a501326f9ec912b8fdb0ac5d8214db926b4dddc7
prerequisite-patch-id: 4cf88a276a32e9c811ed6e0933226aa5fbe9ff5d
prerequisite-patch-id: ff6eac459f6484b702adbd3c59eec3819f273395
prerequisite-patch-id: 590b30b6169d26f7789acd4b3e651195e5701f48
prerequisite-patch-id: 5b7ea4c03396f85d36158d3744ab7880f668f4fd
prerequisite-patch-id: e777df0fd9221ab6e2b5d5e568c002bb5b89d6de
prerequisite-patch-id: 2a77353ff3cad9e6f6e01756ccbf21244730384d
prerequisite-patch-id: 11d87266a0459d621f82ee66e3ac60881d443253
prerequisite-patch-id: 6404e0427f884bb493c2092d5d56f8a6f02b46f9
prerequisite-patch-id: 65cbfcf60cac34bea68780378183af8f13167a20
prerequisite-patch-id: ed5d375c62428a83f6b365166013723a01b87dac
prerequisite-patch-id: 81272a343fadb2140a7a7c2bef71a0853cf64573
prerequisite-patch-id: bb345139aafe19379621af356c258936f1c53e6b
prerequisite-patch-id: 1c06218ec8acd0062f7bf113b28c26ce693c3a35
prerequisite-patch-id: e3039112cf2da303d2093338b0ed8b6c472872f3
prerequisite-patch-id: 9f1841dbe42b7b7a9bca740de596568800ce0e1a
prerequisite-patch-id: 11a354408272fad479cd8c8e8635791f97b39d3e
prerequisite-patch-id: 2500a2d7329dad50fbbfd5c3dcea639964b3212f
prerequisite-patch-id: 6ad9b3cbf1b789d45075544e1106957bce69fed8
prerequisite-patch-id: f87f4833f78124b492910152d9587b6e9c6379b0
prerequisite-patch-id: 1d02c892708378e7343295b03272f9aca7041d3c
prerequisite-patch-id: df6ac078502bc3a345c92b56e98723ab8584d6ad
prerequisite-patch-id: e2f512dad5693cdbdcb3620c7dcee52ec689b11b
prerequisite-patch-id: e986e9cbc46097e0cb4626447f72dc4a1911e706
prerequisite-patch-id: c63e30ce4bde77c98d646ad6d592e2aa07c908af
prerequisite-patch-id: f014e34d52a73b3f8bfff9885cf3f2224f21c78e
prerequisite-patch-id: b5d5c7cb0fba43592272dba61a685f1cbccdc8c4
prerequisite-patch-id: 46f8abe0625b1bad66c0f28dc9d2e430f2f3363b
prerequisite-patch-id: 862c541f28c077844b0bfa48f2144262880c6a44
prerequisite-patch-id: 06ba9d289c88a63acaffa69cfbce0c40e9dea7b3
prerequisite-patch-id: 619e0ce36eaca824d1d8ca69efc1d4c45c636da1
prerequisite-patch-id: 4498c9031066bbde6797ebc36d776ea8964dba46
prerequisite-patch-id: 236c1ae8e9ca3fa0a148df79830e54b296482490
prerequisite-patch-id: 2bae2221ff9ab485f1c854cf0245fae43b5b1296
prerequisite-patch-id: e57b449459037835ecd520efd79244527bd7266a
prerequisite-patch-id: 8002f736b165a3f375193eb900f8d41877689964
prerequisite-patch-id: 8c1eb038e2613037f8db074d2f10b8da2efafc10
prerequisite-patch-id: 5018c489fb074d98ea95f28205c8b629c65f7564
prerequisite-patch-id: 0c31bcef7463624e2bcb38b8d187494530318fb1
prerequisite-patch-id: 3ab0a75893b2369bee1e5362cde1c75f6ba40e1e
prerequisite-patch-id: 42686eb87f20e6af2989daa9e0bb8e6a020708aa
prerequisite-patch-id: cbda5e6e72f52840923d28a97808049647de0dd6
prerequisite-patch-id: e4e63a698cb6fd898dd04b0bb216c20f5d6edddc
prerequisite-patch-id: 7fde90890c129d5d5c0ce301b1b895a786d00776
prerequisite-patch-id: 5aa1a9da7ef27a889c6470e9e4b447d34cd10d6b
prerequisite-patch-id: 5a8557148e77850c7992bca431c45724133a7bb7
prerequisite-patch-id: f21a243220a60cb79d22e8607c25cae9fec04282
prerequisite-patch-id: e649d3b34c98ca12645b0afb89df100a70be476d
prerequisite-patch-id: a89858f1d8f20b6d3b85017c930dfd1a577c8e60
prerequisite-patch-id: 6392cc5691299cdb45c8702a586c857934b51e75
prerequisite-patch-id: 62edf3e96891b53f1b663f2a97a8976dee962859
prerequisite-patch-id: 9d624768473afe293949fc751d756307242674b2
prerequisite-patch-id: 895665db6ce81238879cdeec1c90fa022a757603
prerequisite-patch-id: ba53a3692d50ff8e3e2711f092c7d646b8f1bd01
prerequisite-patch-id: f8725e60b459416c5ae3ce0d489bee97fa9cafa0
prerequisite-patch-id: 72b8a7a49939e764547be136004d65d197f3716b
prerequisite-patch-id: a833dd28954fa611365935352dab78e00b9e38d6
prerequisite-patch-id: 40a5eb3a1c54364c5f89d13a79033645503732aa
prerequisite-patch-id: 5add3c304c174d32c38d16115c6789ca977cb606
prerequisite-patch-id: f924b0f3d24bcd9b5000384b1bad24de79bf258d
prerequisite-patch-id: 3c7f9faf78ec7ec9cdd33384eab00897b6bb7e2c
prerequisite-patch-id: 2937f128b9bed96cf11f8e2bbc91777aa6b19df1
prerequisite-patch-id: 5040444f959fa3b5e75fad1aa1047f9b8d5142f8
prerequisite-patch-id: 74229139793c7cc8ccf388c90e35db447c062a13
prerequisite-patch-id: 4302d984cced34df4c4f01d2ca3ac75f14d89d64
prerequisite-patch-id: df9356a541b1747637e60d7c51924a3877bc8e9f
prerequisite-patch-id: bf77498ea30c954428ab0d4a6c5e93bab2dfc1e0
prerequisite-patch-id: b993bf3c8f09c3ef9644432c04e21d80781308de
prerequisite-patch-id: e27c6e49800d58ca352dca03565a266a6a25fce7
prerequisite-patch-id: f64485c81239177b91e6a1d64ddbe6a5f25f74c5
prerequisite-patch-id: f97b4de43506b4458aefd006ccca772421774e15
prerequisite-patch-id: f820c87452bcfb7f4f088852a8f599b959319267
prerequisite-patch-id: 23652244b104bc5da5acb783aec4bb174b785d41
prerequisite-patch-id: 0ebac18bec761e6cbdb48429679ddfaf19f705d0
prerequisite-patch-id: 3af126ab459c72e96e76309c19461522ea57b729
prerequisite-patch-id: 7fd66b146f5173793a315c5880a0732903124caa
prerequisite-patch-id: fa2c74ffcfbdd2cf99e049016050c89957499ef4
prerequisite-patch-id: 6253ff2f8cf4c59136507618e716f2959084e0e4
prerequisite-patch-id: 59c1ac91d9de5b98d520bc9edcffb36529be9817
prerequisite-patch-id: 065cc0efcd2b443ad6b5f814dd48a38740fe2cd7
prerequisite-patch-id: a2b0e8fe8e9a719632bb74fab07cb9f63931f37a
prerequisite-patch-id: 420949305b73dfdab1036f34a0929a587f378c33
prerequisite-patch-id: ecbd2dba3acf52ed7415bd9ed7e431421927a9ad
prerequisite-patch-id: 6668ca2ee1b6dfbe1b00465f039c7b605454ec6c
prerequisite-patch-id: 67f4c0ee0982828e11abeca1f118a7e987331a09
prerequisite-patch-id: 7621292e80f4a238091598d1eb3ae805bb070709
prerequisite-patch-id: 9d59321909b18786205de00ae6b5d7f28aad8fa6
prerequisite-patch-id: 314905339d5680f2fc67bb3cd5cc7192cf0c6400
prerequisite-patch-id: 5ae117618942a87c71078d6cc5738925b41e0bbf
prerequisite-patch-id: 4328b9ce7536a66dbebea79da293fabebc637e4e
prerequisite-patch-id: df9843af4a1c84898c0144e6294ab53809a23256
prerequisite-patch-id: 076d3da2781c0af86367d941fb299674c2d6958e
prerequisite-patch-id: 48d612f00a9a5efcb400eac052bb55593be38c1e
prerequisite-patch-id: a05901a51b37f4ef5b67e93301f2a23dc28fb4c7
prerequisite-patch-id: 6885e13276e46429c9dd94b717f02a94a3b6722b
prerequisite-patch-id: 0e9b811346a65d0fde20f3994b506bf73b74f5f6
prerequisite-patch-id: 4685d4acf2649f9280544b7936942f56b0e418fa
prerequisite-patch-id: 0f8dc20559ad260032ed723b51eff18adb6a37f6
prerequisite-patch-id: 58e396c87fcab1c03d75a28c676cd34747f460dc
prerequisite-patch-id: a34701d0b24f4fe994149b5090c43af00b4f47c0
prerequisite-patch-id: a1dfff515bab78ed1bd83df9b6a81f7e1f08c5cb
prerequisite-patch-id: 07521a53dfdaa91fd5f9b58a88d92587c7d44c31
prerequisite-patch-id: d46cc28d588403d355f91364c5b6dcb8e72929d3
prerequisite-patch-id: 6a88ff9f03af40ec4d43aa7f30c759af1f2066c4
prerequisite-patch-id: efb1ccf001d8a79fe49dfe40c688253825cf0c76
prerequisite-patch-id: 1fc9352f9602e0dab21dcd22d906c7bb75c9dd94
prerequisite-patch-id: c872169e584e9c573a2fe3fb88b10ea8ce421651
prerequisite-patch-id: 0aed61d304e9d9185def3e1d7d1be3c501d5d946
prerequisite-patch-id: c45d8e6ce731a5921cefb1b0eee5e9ef902380e1
prerequisite-patch-id: 2ce067f6c4e6b633ece97e2965a6f803248e8185
prerequisite-patch-id: b36d24f2552369a83e20f794a720cae23d391680
prerequisite-patch-id: 6d0244cea3e6cfddf6a7c32a696b8161593e3d66
prerequisite-patch-id: 0c30fd348dc1f8684df9e75b55534f18356e560d
prerequisite-patch-id: f15c04d4e79a495cd13cb719e0c27258dd3e25a1
prerequisite-patch-id: 64058243d8494592413d9d74c217645f6c8b032d
prerequisite-patch-id: 0518ab4c3c8c06a7bed10961b2cf64236aabe592
prerequisite-patch-id: 6ab8ba48056f078c4d332a1aa7a9af1e0da6a6c1
prerequisite-patch-id: 249215ad12d97209a16c258ae71a984ef36ab44d
prerequisite-patch-id: 069d07da14295d9868363f2c04fd543be9a3e5cc
prerequisite-patch-id: 869d630d593ac138f44a2ca14729c17ea3f4768c
prerequisite-patch-id: 50d4f44bef2d32534a8c708602ba7af7d9db473b
prerequisite-patch-id: 1784aa999916c851385bfb8ae008a9cb7b4b1125
prerequisite-patch-id: 69bc697cf25199ffcc0e66328659108e36415e34
prerequisite-patch-id: 0b6c08494a74ea5cfb7c5c40900bfa25f2a53751
prerequisite-patch-id: 41a1c6d276f5feb614f804c434eedeaebfa4b556
prerequisite-patch-id: ea322e76901f7a4b12bd9abf9e352f007ec296df
prerequisite-patch-id: c0a07b2f6c9fdcd903b4f6b167a1a509813da2ca
prerequisite-patch-id: 4f6cfa91375555d4a29eebb6ec1f6319d1d14026
prerequisite-patch-id: e9355d54b14f858cb4bb73a599042bb1283220ca
prerequisite-patch-id: 0e1965f5288854f9d99f2c8101f100d5aba21ee8
prerequisite-patch-id: d56cf389574ee6871b57afa853c3e6da8dbacf88
prerequisite-patch-id: 9897e81043dd13d347291cde084455a870597b88
prerequisite-patch-id: e370cdc3769c018c1386a74fc9bf164bbab74163
prerequisite-patch-id: d82cc2e2e156178a983191b6c5b878312c3da4c7
prerequisite-patch-id: c561efd60cb44130b603f044fdc9d4e241d21e11
prerequisite-patch-id: 2eccd04d6bb5f56ef6213a9ee727a1903c5963c4
prerequisite-patch-id: 1ea3449079f25fab84bd34d37ce2420d93602773
prerequisite-patch-id: aa074c15b1fce933239736244c5f3f5d41ce3310
prerequisite-patch-id: ae6211e7a6d58314bcb6f33f0444e903dc50fbc0
prerequisite-patch-id: ea807e03fd96c94581ffb86335a91fc9bd28f7ef
prerequisite-patch-id: cca7c828d0353e8b74addbed14961818a6a33c6a
prerequisite-patch-id: 12b659eb2b5b800d1ba9f349f07314a4508b5dcd
prerequisite-patch-id: ed0fd31d10d65561bfe5c68b951872d3c4eb291a
prerequisite-patch-id: 6f161438f92f980a48807a10a6c9f01fc3fce030
prerequisite-patch-id: c6971c072b807bc7977149f29d4ae69d23384946
prerequisite-patch-id: 0f8c6528433ee38a6910b40e96645ac7d2deccc7
prerequisite-patch-id: 3f1cbf311a4cac95e0635e2cb8088871b445e67b
prerequisite-patch-id: edc4747d25b5e7e8ef3a65667ded12b21b964cf3
prerequisite-patch-id: c4924037c953f426a56d88a04364de7302bdd3cd
prerequisite-patch-id: 3b5f5c3fe7a2d105c811c44e56ad255629cbd06f
prerequisite-patch-id: f22434618a55e3601ada6f903495ef431988d5c2
prerequisite-patch-id: 22979ad789d5f17cf1f220c2af4882bbefe6ab88
prerequisite-patch-id: 91b5211ad3732853ee471b3f207109e09906896e
prerequisite-patch-id: 8930a9baa2cea0f9bd74c324b710db82740a292b
prerequisite-patch-id: 72679f01dc5d8ad384313f6f547c9bd339f34a3b
prerequisite-patch-id: d905648e0ac0f3ee7bd7d990c84af20d9b01cc54
prerequisite-patch-id: 12003ba8d888014179461f859d3ecc34170694ef
prerequisite-patch-id: 51381be7ab0e9397abb311e6e1e4310807430661
prerequisite-patch-id: 513883391f89bda93b4709b802d2bd9549a0378a
prerequisite-patch-id: 6d5dd44268da4ae486fd194fbd49ded3303e3400
prerequisite-patch-id: b453cdfa0d1e993bf2481793f5efbd5958222ee6
prerequisite-patch-id: a3facdd4fc9120a292f4552e16410e3626086125
prerequisite-patch-id: eb38334e38ddc18a116e1de03f7ed0e030cc086c
prerequisite-patch-id: 4153608d6e683dbab3a7c76a43897c38647221cc
prerequisite-patch-id: cdeea42a7560a1397481d1b60ba628418510becb
prerequisite-patch-id: b78def513610f43e0b936eec83d93832c4e3f5d7
prerequisite-patch-id: ba6b2e6248f8fd3a3a3b8e973322d619628d8bc9
prerequisite-patch-id: 28b72495731e14acfc1a79668091c2363dfee9c4
prerequisite-patch-id: e430437c022a80ab7ee03a1afe6a10f643cd081d
prerequisite-patch-id: 93338df550d4d58e137631efd91e3a5ac05d1413
prerequisite-patch-id: a9ce52e16abcf2fbe39e144803e3284e969b9aef
prerequisite-patch-id: 9983bed851fb3bc3cbbc2902f388a12cb705d68a
prerequisite-patch-id: 40ac8ba7e2152b1e072b8d7e238aa0e45086dca6
prerequisite-patch-id: 8232ad4ec2eac07735bfccf53568060706cc0687
prerequisite-patch-id: 390757a385b27f66e0621e62dd2e8ec809e19a07
prerequisite-patch-id: e56c6f6c7ee9cc7ef502ab41d321b0f4742f7725
prerequisite-patch-id: 29a5de6fcbca758b63b9bc8b20b9fb773fd6f14e
prerequisite-patch-id: 34c0c842388720170bd8ac7c8cc88f716029693d
prerequisite-patch-id: 4c281e0621401e30347845f0b8443560214551d1
prerequisite-patch-id: 9d6fda847ce41c6ddad88ff47810fd19d032def0
prerequisite-patch-id: 55f2b9f8d9bc939b67f93da00173912d13c05832
prerequisite-patch-id: 59d42c6807eb990f444ffe80fa8dbb89ed095afe
prerequisite-patch-id: 0eb4a5890e7e74c294fc122ac253d96af39595cb
prerequisite-patch-id: 5ad8737a808ea48011766009c20c620ab051c90a
prerequisite-patch-id: a1d21cb660f9e285c57f694184314bb6d7c05ee3
prerequisite-patch-id: added19da0dcdf969ec0d853206f0e7f8b13cf68
prerequisite-patch-id: bd6a55036880849989d00947440b383dce7a9e3f
prerequisite-patch-id: 5b7dbaa60885fd3ed62b9c4bc54c9cac8d5c103c
prerequisite-patch-id: 8544e215a18151b2473ded8ef473d06483f623cd
prerequisite-patch-id: 5a9aa728ae9039ee9e8706bbc17d56b47a903545
prerequisite-patch-id: 1d5fc0ba5efb5697ea9772be3e19f3bc49e6579f
prerequisite-patch-id: 6bebfc55f1dfd63486b13865b8057b7feff1136f
prerequisite-patch-id: 7a7ae42590f236a5627f75aee6c3d7d16fbe3017
prerequisite-patch-id: 7e44486d318b0bf2e32911a032f00010eb2ded96
prerequisite-patch-id: 613a019b2610333b44ea720afbb6d241078501f5
prerequisite-patch-id: a29e2aaa5bee7ba5d3419a88223ecd0581b7d257
prerequisite-patch-id: 6588c1556c4220cae4198356ce9da3621c189d83
prerequisite-patch-id: 4587761a51718e5d576a635f87d3e88d67b7a6c7
prerequisite-patch-id: cc98b436a69297553f5c44dbfd742ddc1841c951
prerequisite-patch-id: 9004e2fbeae0eb7c21e752e627862a457858225d
prerequisite-patch-id: f0f3e426cfd96675212ff23d781fe8f1c5d6b7ce
prerequisite-patch-id: d6e8b03c99ccdb3ce66e39daf4d4452fcb1857bf
prerequisite-patch-id: f7b3bcb14a7030cf897ea355c20cfa5d1f339981
prerequisite-patch-id: c1d469ab765058c856af3ef843df7cdad8aa37f6
prerequisite-patch-id: ef79e8c9bae6a7d8cbfa4c3ce172704bc70f2325
prerequisite-patch-id: 3a8fbd85557b28416257fc602517a54b6147d95c
prerequisite-patch-id: b2687a0c0505e7e6528ec63d6685dd52908287a0
prerequisite-patch-id: cb363cbf087eeaea7f4aeeb1ec9a40c2c9d3dcdb
prerequisite-patch-id: f49eae3f854a76dc44ef80d09b299677d070ddbc
prerequisite-patch-id: 5795a2f1a370a04f58e85899dbc03f709dd2db42
prerequisite-patch-id: fbe4770bde4205b7cba6a8c892cee0457b8a9ce9
prerequisite-patch-id: b03e1b774dea3401c6fa71387586522e7420a65f
prerequisite-patch-id: 96ae3f7a56df8dea9dd35c1581e128287626cd4b
prerequisite-patch-id: 11bb0675775d4878eb9125bae8f626861a05a46b
prerequisite-patch-id: 04b8744a307c0b0260d59dbe88efdd1866e67fbf
prerequisite-patch-id: 9164ab9f5082c84db5b37d103b2e8a6679466871
prerequisite-patch-id: e3254bad10bdc24153d9f0fef4e12cd9fba036e3
prerequisite-patch-id: 1b4add5ef451065306a0aab093bc34d146efadf3
prerequisite-patch-id: 6930f9446637e9e49f0f24c1724e27bfe3d5defe
prerequisite-patch-id: 7ebb50e64bf5b423c8b1804a122c8ec8d5635841
prerequisite-patch-id: 9f24c7ac6d857c479e4cf3708601f1642f6edcaf
prerequisite-patch-id: e9f30266e6cba6804a7a46a81741e824362d11da
prerequisite-patch-id: afabdd8e1c6531dc3daf8d95e88c98126e16d463
prerequisite-patch-id: bf3c542db96c708f9097f424778576c53f2c8bc5
prerequisite-patch-id: 2019f0b2d75ccb9028ff2a4b9a1dc6ce02f6e36a
prerequisite-patch-id: 4e6405301350df8432aca2aa8ab0ed9af95b5c5a
prerequisite-patch-id: 9fe1a92110390af410ec9cc92c92e2f6dd2bacd1
prerequisite-patch-id: 6e9475b171b1cc647b39c6bae9ba515fe65d5a53
prerequisite-patch-id: c78a4f60e8c1c0e9e171f3e413b5c404c0827ac6
prerequisite-patch-id: 0cb10bfe1b1d12204a837ec965d1a6a912b7b679
prerequisite-patch-id: 81a64e30b15f874c83105022fcc686c49a32dd68
prerequisite-patch-id: 17cdc806d249433b8bf2266a23c763108a9c9408
prerequisite-patch-id: 0338123ec39e4429a183c49662b885a0a377a81e
prerequisite-patch-id: c77c125548addc0f8d731b8579db9da4803038b3
prerequisite-patch-id: fd7f3fcaa1352841531c098d61e6388e1c37cd56
prerequisite-patch-id: ee9d4f8eafb610399883b0f7005befae2d1fef7b
prerequisite-patch-id: c629deef06b699e6aa7dc91506d8253fcfeb022f
prerequisite-patch-id: 8dcb8909349aae97006c3a8e0aa552828738e754
prerequisite-patch-id: 98a0e4cafb71020c17bb575304ff5f2778ead28e
prerequisite-patch-id: dca1378fa1331680d4a8afcdf4bc3da71862e0a3
prerequisite-patch-id: 5270c136830952778d737fed71e5ef9f347d9c86
prerequisite-patch-id: 457a0cb1dc3a98ff9e1b1b572174b217639d7e91
prerequisite-patch-id: 4b6a848f30bc8b6e114e548d6a28ec57a0e9ce25
prerequisite-patch-id: e3af50feecc9c582b5b15de614fa1250717b5418
prerequisite-patch-id: 1bb8a7e233cc9fe1526c75aac1d2deb39008f494
prerequisite-patch-id: edffe1af5d3580e80b7ef1adbcc3b0af09f4bbb2
prerequisite-patch-id: 424877e0e5c4d662f63de27b19fd289df2028703
prerequisite-patch-id: 7427149f89d5c9fe60f677b4b56ebeb58dfdc68c
prerequisite-patch-id: b239cf2a9e946cab0320152750fcad4bd0c9213d
prerequisite-patch-id: d96ed1c21f228e476edfdcc31edd0bc518e6ad34
prerequisite-patch-id: 2b9a485f0761d759b22b62ce1e353bc28e3b9365
prerequisite-patch-id: 8e27a58e1e2f0e0c9737d8863f5712bf30dc232d
prerequisite-patch-id: c15d16c83925599fabd2ea3e43ca805da931e5ff
prerequisite-patch-id: f5eee1acc98ba563d0714ea9fbb9b0ad8764d235
prerequisite-patch-id: 678fa907cfc03b529d6ad167bf18e04b7b0a620c
prerequisite-patch-id: 0380ffb225c6adbeeed29212c8ad2f14bb9bd710
prerequisite-patch-id: 328ba258498823a3883c710b306becb0570e5830
prerequisite-patch-id: 0ee8819f7884ffa26455cda45081009a0b71d0cc
prerequisite-patch-id: 8c3e4787fb9d7f05824a231ee28ac58c0426af12
prerequisite-patch-id: 573bff398455437aa765156ad574737f15f32fa9
prerequisite-patch-id: 5ca7206a28906eaa8b21dc40de63b212aac4d94c
prerequisite-patch-id: dc9a4037536619aa12560f7cb9eff6f9c0f59a32
prerequisite-patch-id: 517e2990b4f1d8717e312437091aa1d55ae663f1
prerequisite-patch-id: 9ad4685669e2376f51dfb2842308f63af5c2846a
prerequisite-patch-id: dc1cbfc31413072c9091bbe232c401d03d7e8a81
prerequisite-patch-id: f0455cd2fc8ca138113b20ae572a2c0498fda538
prerequisite-patch-id: a418a1fc10bdd047358db375062f6b048c4c42cf
prerequisite-patch-id: b475573079da71f099a36293331785103f03004f
prerequisite-patch-id: d1980ce81fe7bfe2b8ac1a6bf06d83e80a98ab3f
prerequisite-patch-id: 59c719110fa5312a359dade0585a9b0a29d7158f
prerequisite-patch-id: 08ce3e29aac67a57a071f1572abc28c3f9bcd602
prerequisite-patch-id: 18c30eeaf9e62adc80b264fb9fc8c63aca874f69
prerequisite-patch-id: 750d3f1dd4b536d33ab6bc2e4230ce942c3df617
prerequisite-patch-id: 5b6a0cb8d72c59fc27d201b2fb3d459e81fefe1e
prerequisite-patch-id: 3c734df7b03f28fbe169161975dd3e7767d3790c
prerequisite-patch-id: 729973084f1e52d6ab70bc7798df960c3996eeac
prerequisite-patch-id: 31a9a36ca8a31053b2d3f2c8ce6adf5d94aae248
prerequisite-patch-id: a06e2876f6a229d884e4e105c997bda1b8428736
prerequisite-patch-id: 22965edce2d442d7356c81eca2e78d26619f9815
prerequisite-patch-id: 46c3492713aa91723fa4d72079c5537ae131454c
prerequisite-patch-id: 8aa9cdf4f039e742249376906342d21f2884afb1
prerequisite-patch-id: cf4a7e7cdb101d5649281314e84839fb39acc829
prerequisite-patch-id: 5faf2da2a3ad4cae080cfab9cbfa62d619e1ddcb
prerequisite-patch-id: 104d0273a3d05db4b8cadaa72ea8c40340611772
prerequisite-patch-id: b5ddab3b99cda3849b210591249a469f283fe12f
prerequisite-patch-id: 5c748ac476e9942a5fa5cab5961c3f3f2956794b
prerequisite-patch-id: 9ef670102b262f7b036d00539a0bb3c8ac6ece31
prerequisite-patch-id: 3fc845f4fd71a90ffca5ca4c7c167e9f06cbb49f
prerequisite-patch-id: 6e7cb4f94d5454c62b42720571f30721a77c72ad
prerequisite-patch-id: ebe8199f262c9f343ad318f67a1cb1e72fcf1348
prerequisite-patch-id: 0783ccea25059eb76bf1f15c697e947ae77444e3
prerequisite-patch-id: be25b982af9e6a74882845331d1b1948f30db564
prerequisite-patch-id: 90419591af246acc6b39e3664a29b775cf416aac
prerequisite-patch-id: c7b40acceb488402ee07b4a5f2bf5683c489fc8b
prerequisite-patch-id: 1cc311a2fccfd4b6019b4fcd8e818ca8d3fce31b
prerequisite-patch-id: 2b95afc329986cbfe8fa1a5ab889a7276612e310
prerequisite-patch-id: 5388256cfb73a2f2819ad2ae26894a5432e96dcd
prerequisite-patch-id: c1119aa8d56b78567b8a81f0c0c5d88ba0783812
prerequisite-patch-id: 32123f9a83387e01007eb2f6a73e78d75f34fdc7
prerequisite-patch-id: c048a1031d433c6ece7cb607815950073cac3111
prerequisite-patch-id: 6f24642e2ed9802f9a1a20ee8ebb00bcaaff92d7
prerequisite-patch-id: 04cb32ac39f28776801b3614140899e1bee9da12
prerequisite-patch-id: 1ba102f59bb9ce95f1bd3c870007a146138e5ae0
prerequisite-patch-id: 4a0d377303e738e2792b541f5314f85cd721b302
prerequisite-patch-id: a131b53bf5c55241cf39d9b7c13cd361aaa86c6b
prerequisite-patch-id: 3f92a1755c26674abb028a0fec019a8621758c4f
prerequisite-patch-id: 1d78b99f61e196688ec4f7d8a259255c2776f6f1
prerequisite-patch-id: c8823f3654f4190a1daeed3c0cc08771e3db4bfe
prerequisite-patch-id: d1fc47f2a0371da0cb93668670a6633929212d48
prerequisite-patch-id: 946c5585ef4aeeb12e4b9dd3ceff4fba51441ec7
prerequisite-patch-id: 3c9f21d9c6bacb573e570d25878cfb63bcd3e8d8
prerequisite-patch-id: a35e7cc47d6cebe4076c97b2d811742687a6fcf9
prerequisite-patch-id: 7baf4054e830376d4423ae2561f8acaebef27e14
prerequisite-patch-id: 542ad08acf7068aba4eb7efd8d2799147d342b1b
prerequisite-patch-id: 56a3b8f53f45c7384468f8fe064eea16f42adaaa
prerequisite-patch-id: 2c03310bd28f2e9e4b2d4d195e3edfde07d29f20
prerequisite-patch-id: 8ce1f8eb8f9d9d6b0d714e12ebfd9e358b8713c0
prerequisite-patch-id: cba2d358e7b3c11d35a593353bbd47f514f291b9
prerequisite-patch-id: 699d043b709125e8748ce4e73c32c3c00ed9246e
prerequisite-patch-id: 7ab9d831a894c4664e51a9ee253683aac7332c63
prerequisite-patch-id: eb72944306c1e146a1e0c9a0b4adc1c8c3bd566a
prerequisite-patch-id: 202b53077139415170c62c8a01d4342ca282bd6e
prerequisite-patch-id: 6913284408db7a031017569d128d29f150cbd2f3
prerequisite-patch-id: d437bc089602533e4e3ae7eeeb876471ba569be2
prerequisite-patch-id: b1a9e3d69c7bd8a01874e31278d05fd78b8c0941
prerequisite-patch-id: f6c5152a812c5490b8635bab89a3c9d48338c611
prerequisite-patch-id: 055bb55bfee8c8cb680883f7e86c0dee10f2d627
prerequisite-patch-id: b321d93ce021c1c4ce60a9dfdeb4c6c82b8fad9f
prerequisite-patch-id: 009891a3dcec0f36072ef751d411b031e66811d6
prerequisite-patch-id: 00ef3bed65ff2568e5502808406931ef6b33e70f
prerequisite-patch-id: c7de32de87e43e37934315cd89c8c292ed646778
prerequisite-patch-id: 3b8e08eca0ddc9c4c13c6aa7c7004e447dfd31b0
prerequisite-patch-id: f0e8fd344755cadcd77b8d2282de12ed83470ace
prerequisite-patch-id: 83798c45cc2d8b308618d756cbfc21caa3dace19
prerequisite-patch-id: a6d252ffdd5807db1f9b6122924434be87d7c644
prerequisite-patch-id: d3f1954a96025484f6735780670d25b59e3a12d3
prerequisite-patch-id: bf2855e7a2460c09dfb564ac2693e10607db1206
prerequisite-patch-id: 3cfc1d4be865b6520502fd0ed122c4765ad18bb6
prerequisite-patch-id: e2e34b7b2b84761da9bc39ddc4a8249efbf27d03
prerequisite-patch-id: 6cf5204d6ecefc40e03da7cba7f570f5a651db65
prerequisite-patch-id: 35e5690ade82282e16789d59aa187e820b248fcc
prerequisite-patch-id: 7ea494f1785ba68eb17fb7869e2eaad44bf4553a
prerequisite-patch-id: 523d9d44fac19a3f2f6d7d956cc6c88d8898e1b1
prerequisite-patch-id: 61ecac7c083e9b42e6ea22bb7bc0ee399bd04020
prerequisite-patch-id: c7ba7a92eaeee95d5fb206911141358390bb4c53
prerequisite-patch-id: cca97410a687557f6ba340d5f49f920d6dd0ccc7
prerequisite-patch-id: 922dc370a6b2d11627453a19eee9eca011b98a53
prerequisite-patch-id: d057779b53a2223c15cbcf94d81584b789d37489
prerequisite-patch-id: 872746b55c91b37391f7e8da91fcebba9721186b
prerequisite-patch-id: 0fa7ecf66abbbb38ec7e824ef12efd9c7ee22158
prerequisite-patch-id: 1f8c2f5622829312336065625589dff292fa9946
prerequisite-patch-id: 534ec5d08599e044c2c24486f4b41b39ab8cee55
prerequisite-patch-id: dc3743a908e2a305ee29f3a66f21555c72db202b
prerequisite-patch-id: fde2aaafdd62b094c1449bb2d287eac823cbf7fc
prerequisite-patch-id: c16fec73b92e3c609b6c42fd14be739c7e4e0ffc
prerequisite-patch-id: 3ed25009afbf79f4d5da11b1fee5c00f711be6fe
prerequisite-patch-id: cf50531f90bce8d9eb5594f16b695664d888e297
prerequisite-patch-id: a606f3291074e77536b96c27c28a2ee8de183232
prerequisite-patch-id: 6c898e6f1c93c9d9522e42d3619c784b1ea953c5
prerequisite-patch-id: 0934f7be2e482dee2e64b7c60a7d92c89ea1441c
prerequisite-patch-id: 205972fe555c28b53717266eb23436a82ed8daa5
prerequisite-patch-id: 6f341f17114775ed537733ebe4ab9ffc56b6e2d8
prerequisite-patch-id: 49f2585ec938bb3a6c10d378ea466e8c77ccc8d9
prerequisite-patch-id: b13efd215dd11daac648a14a1e6dff8efcfd02cb
prerequisite-patch-id: 46ff9da4dfc9f1c26c7635cdaf658f824ea21875
prerequisite-patch-id: 36d979b75d20d82323b874c15815d90dfddf4dd3
prerequisite-patch-id: cc835d5a72237c93e77737500b9d590f338dcaa3
prerequisite-patch-id: fd9eb12da903d07bcde709d7ae66988ca286e791
prerequisite-patch-id: 0bd65fd485eda542f630f1648afdc63f524f6eb1
prerequisite-patch-id: cb5b8a407ef968b45a63e7654d67914175f60c80
prerequisite-patch-id: 83a653ba5b80514a02a80b0a8fc48c2e28a8e66a
prerequisite-patch-id: bc5e5d35bbfc50a5703ff4f4c69c4d318141043a
prerequisite-patch-id: d66d5c5f952ddd2e84048f630537e5e93fee25e4
prerequisite-patch-id: 2877fae3c5a2e6700270f92474978637b548f7cb
prerequisite-patch-id: dab9a022f4365373ee1e867833216834d4900c1b
prerequisite-patch-id: d6a36198bb48d683ebe8525905e21ec2d726a671
prerequisite-patch-id: 6e7d502fb123eb46be07c2dae43443932ad77cdb
prerequisite-patch-id: c1ed4c595e5e09a26d95848c7fe1af12384b708c
prerequisite-patch-id: 0158886f82cf74d329f376309f51ff562c279554
prerequisite-patch-id: 540d4937eb1d8f9b0fafb1c0c49c4b9b5b6321c3
prerequisite-patch-id: 1721dcdf464226c90de74bbeb5d63220c8f82e23
prerequisite-patch-id: 64f5f96fa911f336c5a2e9f9ced4be310269dd8e
prerequisite-patch-id: caebe3e7cb4bfad451020384ecc9bb6b967540ea
prerequisite-patch-id: ad2dc73f27692d7ec838f84f1ff7b58ed7021a3f
prerequisite-patch-id: 6065c0f5d90b58514018f52c1fe25ca4c137f76d
prerequisite-patch-id: c15f7b1903fb0c1ef5ed4728fe80847100791197
prerequisite-patch-id: 040838aeaa059db802a4694ee09a618329bd3654
prerequisite-patch-id: 934571b5204014de9c3036c1c5abba2eae44e4ac
prerequisite-patch-id: 49d7182e3456b53fcf05dbad357ce22bc41efe3d
prerequisite-patch-id: dca53eab7db3d0c1e2fc239e07f13a6dd4af2233
prerequisite-patch-id: 7401cff7a5e9128327ec6c218246877896dc5f7a
prerequisite-patch-id: e11207c9ec36a8fb9dee082935b0b79c6955e068
prerequisite-patch-id: f16097f20f07fa2f37299c9c71cc20a30893b878
prerequisite-patch-id: f0d64711a91ae87869e931e71f2de4e16c1fef9b
prerequisite-patch-id: f0066fd29835a17e07368b26b074884fdd93f5ec
prerequisite-patch-id: 50acf42191545d507ab0ef486613a9faa823a9c8
prerequisite-patch-id: 3875845246e1405b7e3d0ae7e1daa337b46021cf
prerequisite-patch-id: 9c70cf6cbaf32e5ffbd4cff74df3cf90a60ad410
prerequisite-patch-id: 466ccbb6c611af3d67085d3fd3223c1a4a57959a
prerequisite-patch-id: 832a4c1bb236399012262a73a3e05bdad8a84db8
prerequisite-patch-id: 7d1679bd2fd4f85670f1b741b3442fd45bdbdd64
prerequisite-patch-id: dc37b46f26285e8d1e5d9b2bfdf214b2f765457a
prerequisite-patch-id: 70837893fc301a30e30739906ccf46d82baf5532
prerequisite-patch-id: 98810026a834c633c6089c83de8301611e312b28
prerequisite-patch-id: 63fcca22c2b47ecffbf941c2e6eb1f01bff43358
prerequisite-patch-id: 42a87d570def0bedbf2db6bff22e217baaedaa22
prerequisite-patch-id: 45f706cde07c5e5385ef4d1c25be5fcdc3071842
prerequisite-patch-id: f1e616db47b35ecf30be1e4c9706414caf51d8da
prerequisite-patch-id: 62ee9033ce48753c96f4c25fa2a66d869fe7ff97
prerequisite-patch-id: 92d3bb1adfefd958c8c442820ecfcaa18ccc3b0d
prerequisite-patch-id: c64d50fdced59de7d7eea7e0183304efdc080345
prerequisite-patch-id: 0755adcc12b58c70fda09cf1a629e31c5bfd21bf
prerequisite-patch-id: bc56910fb1dddc33d7e03344599ed83ebaf0040b
prerequisite-patch-id: 1ce6fa46332bb071ca2ec5a4e861786ac1e01fa8
prerequisite-patch-id: df253ba1486958b72d8329941062472f415b6fc7
prerequisite-patch-id: 183ff8d378d8a3b0729e00587adb72d8b8f14f22
prerequisite-patch-id: 72918841b84c51e65695ac292c46fc69878a2352
prerequisite-patch-id: 161efc73b3406eed1e6a4b731924a455532d0e5d
prerequisite-patch-id: 5294bf5a16224957d4f86d030d98199661ddf720
prerequisite-patch-id: 192fa81a8669eb7f39b456f07391f8a507b875df
prerequisite-patch-id: 732a95a179644747809aa6ac8ca4d06fc55b49e9
prerequisite-patch-id: 041e56a325f25688ad84b814cecb9db647a3a6d2
prerequisite-patch-id: b83c25218139fdbac6009959f8e064dd98628797
prerequisite-patch-id: ee523f6d39ac5c774c4ad9f3b8a5784de1152939
prerequisite-patch-id: a49df6d3fc26f6440240ddbf234e34cebf1ef16f
prerequisite-patch-id: 9c62854be71f74639b4e90d5abf6ccade9eff884
prerequisite-patch-id: 595ae66ceb1c12b4e4dc3b9490db91bd5ac74a15
prerequisite-patch-id: 43ddb6d44359b30938671c3a36af1ee4a05b0180
prerequisite-patch-id: c995202d269f5e4bb40f247e02db86ec5671adb1
prerequisite-patch-id: 91964f622cedb5a8be3713a74cdd28221fe65e72
prerequisite-patch-id: 39e875f28e9bb90d3254f46fb45aab19fc4342e6
prerequisite-patch-id: 0e71edf714ad88a0564a3a2dc132a1487cef0cf1
prerequisite-patch-id: 7d72e30bf9427af73dc2e6ccbd4c91f3b58ccfe9
prerequisite-patch-id: 2a9c4a3509dd4c7b6f7f50a776b909b77e516c14
prerequisite-patch-id: bcf624d9bef6e71660d62e71912a8ad3a71035c8
prerequisite-patch-id: e2386f60f4d0ab857afe7be6d83a026f204e8ff6
prerequisite-patch-id: b7ee5317c224a8070a7df9b9bda67eb5d3aac302
prerequisite-patch-id: 6db5e2bdbec7cf70c94eeb48c3432c3e5f0c0d50
prerequisite-patch-id: b99cef5dc6fad1bb57bd16bb740fc022f6c6b1c6
prerequisite-patch-id: b9c7f37353ccdfff2dd645d42a90f95514a1db54
prerequisite-patch-id: f35d20b4a97140484bb22268e1ca567141ab57cc
prerequisite-patch-id: f73734305c8fc96bc3a828f224225b47a0d25cfe
prerequisite-patch-id: c0e1107fa5b6072c94ca4abdfe90947412655d3a
prerequisite-patch-id: 184b53a10799774bebb924f287549d6bc9c6d6cb
prerequisite-patch-id: 0d462cd85a93821fdd600042478c695db4d687d7
prerequisite-patch-id: 0a14c5d87d64b8b850b2d6d16f88f09d1b4cb4d2
prerequisite-patch-id: 398bc65be166a871d5dbf89b61a6b8f749a75be1
prerequisite-patch-id: 7f3359413f59912996287f18e6395d0cad0efe39
prerequisite-patch-id: ad859862d2d367259d6a7dcd8bd77729bed39c21
prerequisite-patch-id: b22e67e99b2815334fcf73743fdf055ed153e005
prerequisite-patch-id: c8b4a954e20f953dbff008089440a1b2fa8cbf0f
prerequisite-patch-id: 312861abf86d5799e6f85ce5cf34ac8b9c46f95c
prerequisite-patch-id: 31856aa8d6ac44dc1ee30e37665eac1120360cd8
prerequisite-patch-id: 0e8e39a8c74c3dd80092a223e5729160be71451c
prerequisite-patch-id: 59f17bdb15c833f3627e99dcbb8ff5df56a1d685
prerequisite-patch-id: af943909b3b77614d39f7a35210dfec536d0c712
prerequisite-patch-id: a3eaed315e73b4cb584ae176de28903510ada792
prerequisite-patch-id: 75f8c758560362583bb69f21b65b49994e1d66e3
prerequisite-patch-id: 12cf2dad201e25272180a7e53d4f8481bdb08301
prerequisite-patch-id: d5155fad3e1418089bb22688df12680d8022d31f
prerequisite-patch-id: 7630ea4a5cedc6adf939c669489806a353af655e
prerequisite-patch-id: 32b446bd6dbe3fa4472b457d6c029fb4376150a6
prerequisite-patch-id: 14d4434a1e26c24f61a20824f147b04822d9ac1b
prerequisite-patch-id: 17565924a57ad42db80ab683d3ccd494ecb8186d
prerequisite-patch-id: 068d0b6f63964d45aa2a0bf5b2cae3d1c354d29e
prerequisite-patch-id: 4c755e71ad134f77cf6fd2a5743c102f6e6c04a6
prerequisite-patch-id: d6178d19892c900f5388ce55e0aeda22eab00c79
prerequisite-patch-id: c37d52c888c203fccb08e988893b48212c705d50
prerequisite-patch-id: d0fa642e5eda3a6de2f3f72bf35bc7676beed963
prerequisite-patch-id: 74ec9251231e15e63d22ade8c32cf6e9c02e863d
prerequisite-patch-id: 06bab0f0d2ed6eeef2e7801e134e324d6d54c5e3
prerequisite-patch-id: efdaa762203d7ca329c4a64db2609de5d3f0d37c
prerequisite-patch-id: 477d4405e109859bf72c54ef4ff6a5e176889eaa
prerequisite-patch-id: b126790065c81d4adb34fa1e6c9281b4db5ea07d
prerequisite-patch-id: 863520b4c6f94ee824b39c60b7298485b72f6d57
prerequisite-patch-id: 6478e0391435b494760ec557aaa4f2e30ea4ef34
prerequisite-patch-id: 25b42c970d9cbe5192bb8e8618debbf70aaf14ac
prerequisite-patch-id: 398ea1dca59d4953e3cab9477bb9ec884fd3ebd7
prerequisite-patch-id: 5c0236ba7e9f4d9263944a36a624d2008339339d
prerequisite-patch-id: cfa97a9b90ee31f14f1c11482125ddf382294949
prerequisite-patch-id: 8843b7292a2f48fe835188e1772d76470570d1ed
prerequisite-patch-id: 6242ad59ad854dde8c8dc69c1f4bc76847476762
prerequisite-patch-id: 2d4096321dc39ddb09592f30b69180ebf995101f
prerequisite-patch-id: 61982bb691cfe36a79c1be56eeeb503b5109d772
prerequisite-patch-id: 91a3bce85090b362190bd6140616ef75ddc4f8bc
prerequisite-patch-id: e0f2073b8a857f2730db86f09427c02bd032692e
prerequisite-patch-id: 7811cd31eccb9cc610a744231515b67ba271d33d
prerequisite-patch-id: cfff10eede96265d69ff1490ac7a16ab1f58abd8
prerequisite-patch-id: 15a921ffef8186b36b1bc9892fdcc2c23c58ab22
prerequisite-patch-id: 1c633a8a03923ebd3cb443472acee446df150017
prerequisite-patch-id: 4d398224253cb465a8e76e73607ab52d31b983ce
prerequisite-patch-id: 75de8b33aa5883b6f3cd5e97ed75a831974427df
prerequisite-patch-id: 8459349674b56d3add36df2f7af9f6db0fb1a350
prerequisite-patch-id: 1d17792aabed07b98dd84f0c5174fac27e10d522
prerequisite-patch-id: 603e021da94bb3396d63d9323cf4ee75c3e3529d
prerequisite-patch-id: c5136970f28a0e750cf989ce528a8632352a2c02
prerequisite-patch-id: c62a2df67dbc93c51f686b5ca8bfaf748802873c
prerequisite-patch-id: 27d76b2ac5e202eeb0d77b963a0111c55be7c542
prerequisite-patch-id: a61cb2e02ae351f044d61406c87ae4e45c049a2d
prerequisite-patch-id: e7944b362ca3924f15d484e987894dc26b9fcd16
prerequisite-patch-id: 225f1ded6ac55011457ace85eb9a0c5fb59760b7
prerequisite-patch-id: 0a099aadf367898a05990b6dc90f43bdb45e4353
prerequisite-patch-id: 5338bafc69d1ce2ceba53ead881862f5e5ceed12
prerequisite-patch-id: 753aa06e997ac9f07780ebf5cf3cef2a0f6ad8d4
prerequisite-patch-id: 647a84e6f3752159a3f219ef2acc0f47c4f89cbf
prerequisite-patch-id: 5c992979148cc8dd7ab6f39c688ede135aec246c
prerequisite-patch-id: 9b2432198d768c221e4995719753db1a96b282de
prerequisite-patch-id: e2d531670d3c996799fcc65b9b953b9d39e829cf
prerequisite-patch-id: dba1677a5304e1811fba1cb0e76f84c3d0f61b5e
prerequisite-patch-id: ea256490ff4b4f2933d4e1697d85bd574d1eea7a
prerequisite-patch-id: 7288e2f339581863a60bb7621af0bf53c15765a3
prerequisite-patch-id: 88d8b0c75971231a52362d72374e390856c91087
prerequisite-patch-id: 99080f960e5e939b79dcbb3762d7607f684f3223
prerequisite-patch-id: 443050737f9d410dafe4e6d66ad991fb30686ef7
prerequisite-patch-id: 75b00053d6584115ee478df398d206dff2ec0748
prerequisite-patch-id: 8fe187eb73b0c3756d22b740e0900c23b0e49685
prerequisite-patch-id: e9531cef6517857463ba70080480d28037dfc666
prerequisite-patch-id: 8fba074401cf5adddb55037eb4ac3131ef613a25
prerequisite-patch-id: 9dcde58fbe662f5e49c1bc59eef841076cea23a4
prerequisite-patch-id: 9e181e95b8abbf52ebfc3240f3c5677ed52db27e
prerequisite-patch-id: e144eb92024e96ef1f50ed2018e3181695ab1ac1
prerequisite-patch-id: 4bc8a9a6e40c176b54596cd9d76d3ed3e0918fee
prerequisite-patch-id: e86d956e63d866b7f714f27e0fe5c2dba2cda40c
prerequisite-patch-id: b98eb808bf62a850a3c165701c9c56f4a0d7eb1d
prerequisite-patch-id: 7e3beae83c76440cfecf4a229a6cb55cd2202fb0
prerequisite-patch-id: af1b8889603b162427a608893c5f19b04777e529
prerequisite-patch-id: 483ee1290f16e2ffb171b1ecfc5396d8d0b58dbd
prerequisite-patch-id: 4867d2771e267b9ee5fd9cf09466df54ad542565
prerequisite-patch-id: 26091f82ea89fa279f34e125568a5338e862fff1
prerequisite-patch-id: eead951ffdd1738e35dd97731722420a7e6e0197
prerequisite-patch-id: 8abf36d897ca028d5a9d25e04bf419b96ef20281
prerequisite-patch-id: 6f472485f30be3641acf15fab5e75acb95437371
prerequisite-patch-id: 761c632759b166ba0ec7ee3c51de846161d3ac0e
prerequisite-patch-id: 9b484611118e5b275c4b616ba18c7066614215ba
prerequisite-patch-id: c1e17c4fb70ea58840d2ab252c9cb54f43c4dfb0
prerequisite-patch-id: bfb19c706efa1f55d43781d7ef9987f8dfae1dc6
prerequisite-patch-id: deccfc8454cd071120492b40312d97c64028999d
prerequisite-patch-id: 7bfce196190c902064423360f73a74f978cdc153
prerequisite-patch-id: 215dd06efbc0d5700fc7a98b025b840a8268aa6d
prerequisite-patch-id: e858f5fc00fbe3ccf668c687dee2061c51a78432
prerequisite-patch-id: 85f224eb320f7fb107135b554993bc831047fa04
prerequisite-patch-id: c36ddc96e32b5f79af15df34b57ee7d8b4e32f24
prerequisite-patch-id: d387c6483b73d881c0d26cf5c64dead09f1afb7b
prerequisite-patch-id: b9b6c237825d78e3a860e608c021f910a756eb24
prerequisite-patch-id: 72a8be1dd38fe430d95c4277a3802ca8680554ca
prerequisite-patch-id: d7c9064734d9f90bae3ce8def4cbdd0564680a1a
prerequisite-patch-id: 2b7bdca9047d3e0ee2855905d6546b51a6a540c0
prerequisite-patch-id: 42cb03da4a1e374409cadd3277267cc471067e38
prerequisite-patch-id: a439f60634d74c2179712ca0e314facfa42fc0cf
prerequisite-patch-id: 2d6703c8330e3c5499398b3436e1c514b0c170ac
prerequisite-patch-id: 1b48e9eaee821261f3c926ad6862035f9988c161
prerequisite-patch-id: fa95486596301696ac5b042dda727e4803da4276
prerequisite-patch-id: 84128e305e9eed9c6b20dc983a7c794d20761210
prerequisite-patch-id: 66636e2ef226604d83077f3c6137142dd0fbe342
prerequisite-patch-id: 77015d5b571df095df24591ddd9ae1699d0d85eb
prerequisite-patch-id: 314bc0473ac43066f891cbfbf017a3c02bd6df80
prerequisite-patch-id: ca5ea54d426b2dfc91cc7d518f67049c0be40591
prerequisite-patch-id: 8037522480fec73cf06957a947095cedd386f62d
prerequisite-patch-id: 95817a13cfcca32ceae2372f737cc513d864db41
prerequisite-patch-id: 6f126035b087494f9ad1c200596b5b1b7cf21ee7
prerequisite-patch-id: da17979088e3bffc620e25dfef9ba042cd07c6b6
prerequisite-patch-id: 04f6291f75498217363dc4236f2edb0e0da1422b
prerequisite-patch-id: de36d82364ea2d98b80176eeb485657fa4e393b2
prerequisite-patch-id: de4a1010e6812616d397cce1a38737ac6e713c24
prerequisite-patch-id: 38157db92df65354c00325ca44410402e4092ee8
prerequisite-patch-id: 5f82f5da18887d17d3f102d0c1b641ecb798797b
prerequisite-patch-id: eaa92e037894a2714e027817117d18a15ec05852
prerequisite-patch-id: 03cfcfac835dd7977db94968973b1bff5ff669c5
prerequisite-patch-id: d06c810a891665cfaa9db769e2ef19f0d05a1fb3
prerequisite-patch-id: 68c65728eae1dce24053d5b866b5e37d636a7c24
prerequisite-patch-id: eb38334e38ddc18a116e1de03f7ed0e030cc086c
prerequisite-patch-id: b69a94a6026d4d3d94f3a447e94c71809908db67
prerequisite-patch-id: 951841a2e001c2c9a8e4c0b4713360457e325a9c
prerequisite-patch-id: 8075c8994f284a8608b1f0859755a1b8ffca8153
prerequisite-patch-id: d9f81256d272490df1ad829782fd2365dd20de7b
prerequisite-patch-id: 35b25f335c9e3e686356c63afcbe308cb31c1062
prerequisite-patch-id: 33fe5074755b6a819be186fa0c4a2c1ac6dc000c
prerequisite-patch-id: 184807e93c7f62fb37b32cc2af4f3ff5bbf6ac59
prerequisite-patch-id: bb6e202dc83dad6f3559c7b92e28b6b9c1b6a6fa
prerequisite-patch-id: 12bafa214ed2e7c9379ed84a40d6676293356849
prerequisite-patch-id: 0a65972e3977284d46188086b7f8e27d43306c4e
prerequisite-patch-id: 8877c9107b104c12e01db8c9b0d493c137d316ef
prerequisite-patch-id: ad17de915437c124c7fda10b750d5807d9ed1c8e
prerequisite-patch-id: e2f896af2e390a571a697776cdb3003c2db76a89
prerequisite-patch-id: a29eedd61de04f0a2aede926844f380d1ff6f79f
prerequisite-patch-id: d6bc8b3a26d70f4d2c88dae10be2785b840de4ec
prerequisite-patch-id: 580e27c186ebb36974da4a6d2ed96c54757137db
prerequisite-patch-id: 590954e09fa5fcf70cad3ad5842d3c25dab32761
prerequisite-patch-id: a6759838cb7a51c42bb14f357bdfb1fad6b767da
prerequisite-patch-id: 495c9001d25be9c1498552246f8fe7330803508a
prerequisite-patch-id: 3315fe58739db20b8bd4ee86d152b64686be4ea7
prerequisite-patch-id: 05eb01aeedd57d40b0a9ac6ad9d5fa0d64e36907
prerequisite-patch-id: fb0bd246455ced3e7e67fb31c036e9e850ad1b91
prerequisite-patch-id: 0229f6f684eea73cd60687e0b23b29dece8d2159
prerequisite-patch-id: eec7ab24b8bcdbd80e861e16452826f8b6043357
prerequisite-patch-id: ea859f13505822f7dcec69c04daa6b4bc10c3aba
prerequisite-patch-id: 8b59ec1d3b731adc1cba1c1071cae62784b20299
prerequisite-patch-id: 989d19717641e8c9b8a957dbda9cd1c67c2ae6e6
prerequisite-patch-id: 1eb489cbc0ef586f024d0925d68a5f5542b6142d
prerequisite-patch-id: eaf4187e6efdb486b91a2dcd6365454e48c4092a
prerequisite-patch-id: c079d7a0f3142e0a45ef9a726e9bfb63d2eee6d9
prerequisite-patch-id: 2c0039415add384a3809d83bdd7f1304344986dc
prerequisite-patch-id: 8f93732dd38c8f72bd1a6c459c2673114a49ecd1
prerequisite-patch-id: 0e554a72020d16d4f0f8f713377264bb9a4ba864
prerequisite-patch-id: e03e93492c22ca39321a8bd8b0b592868dbb1f9c
prerequisite-patch-id: 283e5d9773252832554e8c369022bf5d85d2cb66
prerequisite-patch-id: 83ae034672b242f88cee40c28dac69e0b5490d57
prerequisite-patch-id: 47af0933bad38dbe03fdd532180965b45a7fd005
prerequisite-patch-id: e4fb69da72ba5fc42aed60540e89fe6851e8e597
prerequisite-patch-id: b82050b0cbe129b9a91b407925ddaecb7983e50b
prerequisite-patch-id: e437fd86238d3c042dca4b0ab89201af6cc37d55
prerequisite-patch-id: f7a7e00fd75e5fe460f9bcad05fa4ce9452bad69
prerequisite-patch-id: 9d748bcba3ba3aba0836cb8956883d9996682b2a
prerequisite-patch-id: 4946ed5bf3398227b7362d316507bcec69143603
prerequisite-patch-id: 8e01fca2e9ca4db9c9da44e277ba0d1bb49fb06e
prerequisite-patch-id: 26978603b24ef02d2b050498ab711fb701efd8f2
prerequisite-patch-id: 91a17ff46b6e39a05567307d8a2e088585d5c5d4
prerequisite-patch-id: 0064859b4251406e5cfc808446cdefda8ab1229a
prerequisite-patch-id: 949f478edcdaa54e659759c9d7ca28df6ec96473
prerequisite-patch-id: b070529ca5dd9077f345001c24a41997de9cc1d8
prerequisite-patch-id: c2aedaa4cddc19a277e8a58387178616dd50cfa9
prerequisite-patch-id: d8ebe315e1e2a28f7b5a59274f0c8cfad3b58709
prerequisite-patch-id: 68720ef6a048d57ff024536ce55e39000831f56f
prerequisite-patch-id: 5a86814f227a75633f57cbd85427d81657a95415
prerequisite-patch-id: 35a7b7202ba6a7286874ec899485a1cf9eab5dea
prerequisite-patch-id: 6d292e9ea15b4e5713012e026e07444cb03817e7
prerequisite-patch-id: 734a40ab3cd8d906657279a4b84d768258677b32
prerequisite-patch-id: ba4b3c87691c9fc2b2ba9cdd5ee8d5e9b3395877
prerequisite-patch-id: 664d4c437ad91d48f79c72abd01b092641fe5051
prerequisite-patch-id: 57aa07b5a8f28666e091f46345792fedd5cff4ae
prerequisite-patch-id: c0b906886a830f76bc2933800dd84a1682a6780c
prerequisite-patch-id: 1e81e3f65f9256bfb8d584b22f15eea391a93135
prerequisite-patch-id: bc254e2dca43f77eb7e0dd14e04e53b0fe556ec8
prerequisite-patch-id: ec9b10fc0a84314ec28b3ee04041f05f2c6fbbf1
prerequisite-patch-id: 561b6d96fc9a16cfc4883525407ccf9f52767821
prerequisite-patch-id: a53fe52087d856b95c2744956649fe9327a75282
prerequisite-patch-id: 42bc72ce46ea2071d47cfc6f9323da9fd7ea44f3
prerequisite-patch-id: ef180abda60223444de4ad0c8cbc7e4c12fcc653
prerequisite-patch-id: 6d93bca7e00544f896614734239201b114619579
prerequisite-patch-id: 8214033f92fe970731589135b539adb405a417c0
--
2.20.1

^ permalink raw reply	[flat|nested] 46+ messages in thread

* [PATCH v13 0/9] nvmet: add target passthru commands support
@ 2020-05-14 17:22 ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

This is v13 of the passthru patchset which is mostly a resend of v12
with Sagi's reviewed-by tags collected.

Below, I'll reiterrate some points I made previously that haven't been
responded to:

I don't think cloning the ctrl_id or the subsysnqn is a good idea.
I sent an email trying to explain why here[1] but there was no response.
In short, I think cloning the ctrl_id will break multipathing over
fabrics and copying the subsysnqn only has the effect of breaking
loopback; the user can always copy the underlying subsysnqn if it
makes sense for their overall system.

I maintain overriding the CMIC bit in the ctrl id is necessary to
allow multipath over fabrics even if the underlying device did
not support multipath.

I also think the black list for admin commands is appropriate, and I
added it based on Sagi's feedback[2]. There are plenty of commands that
may be dangerous like firmware update and format NVM commands, and NS
attach commands won't work out of the box because we don't copy the
ctrl_id. It seems like there's more commands to be careful of than ones
that are that are obviously acceptable. So, I think the prudent course
is blacklisting by default until someone has a usecase and can show
the command is safe seems and makes sense. For our present use cases,
the identify, log page and vendor specific commands are all that we
care about.

A git branch is available here and is based on v5.7-rc5:

https://github.com/sbates130272/linux-p2pmem nvmet_passthru_v13

[1] https://lore.kernel.org/linux-block/247eca47-c3bc-6452-fb19-f7aa27b05a60@deltatee.com/
[2] https://lore.kernel.org/linux-block/e4430207-7def-8776-0289-0d58689dc0cd@grimberg.me/

--

v13 Changes:
  1. Rebased onto v5.7-rc5
  2. Collected Sagi's Reviewed-by tags

v12 Changes:
  1. Rebased onto v5.7-rc1
  2. Collected Sagi's Reviewed-by tags
  3. Per Sagi's feedback implement an whitelist for set/get features and
     audit the features for whether they are suitable to be passed-through

v11 Changes:
  1. Rebased onto v5.6-rc2
  2. Collected Max's Reviewed-By tag

v10 Changes:
  1. Rebased onto v5.5-rc1
  2. Disable all exports in core nvme if CONFIG_NVME_TARGET_PASSTHRU is
     not set and put them near the end of the file with a big fat
     comment (per Christoph)
  3. Don't fake up the vs field: pass it through as is and bump
     it to 1.2.1 if it is below that (per Christoph)
  4. Rework how passthru requests are submitted into the core
     with proper nvme_passthru_start/end handling (per Christoph)
  5. Rework how commands are parsed with passthru hooks in
     parse_admin_cmd() and nvmet_parse_io_cmd() (per Christoph)
  6. Rework commands are handled so they are only done in a work
     item if absolutely necessary (per Christoph)
  7. The data_len hack was dropped as a patchset was introduced to
     remove data_len altogether (per Christoph)
  8. The passthru accounting changes are now in v5.5-rc1
  9. A large number of other minor cleanups that were pointed out by
     Christoph

v9 Changes:
  1. Rebased onto v5.4-rc2 (required adjusting nvme_identify_ns() usage)
  2. Collected Sagi's Reviewed-By Tags
  3. Squashed seperate Kconfig patch into passthru patch (Per Sagi)
  4. Set REQ_FUA for flush requests and remove special casing
     on RQF_IO_STAT (Per Sagi)

v8 Changes:
  1. Rebased onto v5.3-rc6
  2. Collected Max's Reviewed-By tags
  3. Converted admin command black-list to a white-list, but
     allow all vendor specific commands. With this, we feel
     it's safe to allow multiple connections from hosts.
     (As per Sagi's feedback)

v7 Changes:
  1. Rebased onto v5.3-rc2
  2. Rework nvme_ctrl_get_by_path() to use filp_open() instead of
     the cdev changes that were in v6. (Per Al)
  3. Override the cmic bit to allow multipath and allow
     multiple connections from the same hostnqn. (At the same
     time I cleaned up the method of rejecting multiple connections.)
     See Patch 8)
  4. Found a bug when used with the tcp transport (See Patch 10)

v6 Changes:
  1. Rebased onto v5.3-rc1
  2. Rework configfs interface to simply be a passthru directory
     within the existing subsystem. The directory is similar to
     and consistent with a namespace directory.
  3. Have the configfs take a path instead of a bare controller name
  4. Renamed the main passthru file to io-cmd-passthru.c for consistency
     with the file and block-dev methods.
  5. Cleaned up all the CONFIG_NVME_TARGET_PASSTHRU usage to remove
     all the inline #ifdefs
  6. Restructured nvmet_passthru_make_request() a bit for clearer code
  7. Moved nvme_find_get_ns() call into nvmet_passthru_execute_cmd()
     seeing calling it in nvmet_req_init() causes a lockdep warning
     due to nvme_find_get_ns() being able to sleep.
  8. Added a check in nvmet_passthru_execute_cmd() to ensure we don't
     violate queue_max_segments or queue_max_hw_sectors and overrode
     mdts to ensure hosts don't intentionally submit commands
     that will exceed these limits.
  9. Reworked the code which ensures there's only one subsystem per
     passthru controller to use an xarray instead of a list as this is
     simpler and more easily fixed some bugs triggered by disabling
     subsystems that weren't enabled.
 10. Removed the overide of the target cntlid with the passthru cntlid;
     this seemed like a really bad idea especially in the presence of
     mixed systems as you could end up with two ctrlrs with the same
     cntlid. For now, commands that depend on cntlid are black listed.
 11. Implement block accounting for passthru so the target can track
     usage using /proc/diskstats
 12. A number of other minor bug fixes and cleanups

v5 Changes (not sent to list, from Chaitanya):
  1. Added workqueue for admin commands.
  2. Added kconfig option for the pass-thru feature.
  3. Restructure the parsing code according to your suggestion,
     call nvmet_xxx_parse_cmd() from nvmet_passthru_parse_cmd().
  4. Use pass-thru instead of pt.
  5. Several cleanups and add comments at the appropriate locations.
  6. Minimize the code for checking pass-thru ns across all the subsystems.
  7. Removed the delays in the ns related admin commands since I was
     not able to reproduce the previous bug.

v4 Changes:
  1. Add request polling interface to the block layer.
  2. Use request polling interface in the NVMEoF target passthru code
     path.
  3. Add checks suggested by Sagi for creating one target ctrl per
     passthru ctrl.
  4. Don't enable the namespace if it belongs to the configured passthru
     ctrl.
  5. Adjust the code latest kernel.

v3 Changes:
  1. Split the addition of passthru command handlers and integration
     into two different patches since we add guards to create one target
     controller per passthru controller. This way it will be easier to
     review the code.
  2. Adjust the code for 4.18.

v2 Changes:
  1. Update the new nvme core controller find API naming and
     changed the string comparison of the ctrl.
  2. Get rid of the newly added #defines for target ctrl values.
  3. Use the newly added structure members in the same patch where
     they are used. Aggregate the passthru command handling support
     and integration with nvmet-core into one patch.
  4. Introduce global NVMe Target subsystem list for connected and
     not connected subsystems on the target side.
  5. Add check when configuring the target ns and target
     passthru ctrl to allow only one target controller to be created
     for one passthru subsystem.
  6. Use the passthru ctrl cntlid when creating the
     target controller.


Chaitanya Kulkarni (1):
  nvmet-passthru: Introduce NVMet passthru Kconfig option

Logan Gunthorpe (8):
  nvme-core: Clear any SGL flags in passthru commands
  nvme: Create helper function to obtain command effects
  nvme: Move nvme_passthru_[start|end]() calls to common helper
  nvme-core: Introduce nvme_ctrl_get_by_path()
  nvme: Export existing nvme core functions
  nvmet-passthru: Add passthru code to process commands
  nvmet-passthru: Add enable/disable helpers
  nvmet-configfs: Introduce passthru configfs interface

 drivers/nvme/host/core.c        | 229 +++++++------
 drivers/nvme/host/nvme.h        |  14 +
 drivers/nvme/target/Kconfig     |  12 +
 drivers/nvme/target/Makefile    |   1 +
 drivers/nvme/target/admin-cmd.c |   7 +-
 drivers/nvme/target/configfs.c  | 103 ++++++
 drivers/nvme/target/core.c      |  13 +-
 drivers/nvme/target/nvmet.h     |  52 +++
 drivers/nvme/target/passthru.c  | 547 ++++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   4 +
 10 files changed, 889 insertions(+), 93 deletions(-)
 create mode 100644 drivers/nvme/target/passthru.c


base-commit: 8f3d9f354286745c751374f5f1fcafee6b3f3136
prerequisite-patch-id: 9d7238f8634ac0218556c6ea3b290ca3f9a9bfeb
prerequisite-patch-id: 7c28feea71d45760f162c9268e373af31a5906a4
prerequisite-patch-id: 38002717cce1b2917ba085e4107c08e1967bdc73
prerequisite-patch-id: bdf06c1eaeffba1067e28524281bcf85672f0ec6
prerequisite-patch-id: 6b36d776fa495e0a41a82efa5150139434bf3774
prerequisite-patch-id: 4989b0e17467b5c109c97875485ada07a561a98d
prerequisite-patch-id: 31a7495031ecb0d5f8f6a1ea35690a225d59f09d
prerequisite-patch-id: df18b77c596089e4378ae39dea3cd56cba81413d
prerequisite-patch-id: 83fc9c4937e86e1374b822c77b83e79fcda70c6c
prerequisite-patch-id: fe31fd60462907af7739d828e1da7477dd0c57fa
prerequisite-patch-id: cd157017f57ef7982d0aa6f0bd23e46530ecaa42
prerequisite-patch-id: ce4515317fd337e8ebb0f607644a131a4ca37ebd
prerequisite-patch-id: ec33e99ce296fdc99c2df2f50fc5b338bf44cf1c
prerequisite-patch-id: 66625c300e60dc72782a20118490f2998bad1b01
prerequisite-patch-id: d996f3880062d0f14d8fdd29e605282fe2f27035
prerequisite-patch-id: aed2a18de4fd4d167ff1c8255515e83ef04681d6
prerequisite-patch-id: 26939059a94177d43ae80e1291c75fd30a9928c7
prerequisite-patch-id: 3300246b7a6b0f592eea2277f887fff13f625f88
prerequisite-patch-id: 309fb4ee0caaa5f1b05154fc8d7b2967c7e6ac6e
prerequisite-patch-id: 710eb598f7fc11ba8e3240b43d13a82e01e35d83
prerequisite-patch-id: 9726bfa4b8e2a21fa50a819e1ee732d6ab369c31
prerequisite-patch-id: 5cf881ddbc6748cc2ce12293aa9a32a5c9269ef9
prerequisite-patch-id: 837840a24eb1efd4981270152aabd27b5653fc02
prerequisite-patch-id: f2c07cdd0fb2677ac655708acbda9a016d6e67a7
prerequisite-patch-id: 334c0e74332d15e15ed31763c0788ce892e287ad
prerequisite-patch-id: 835912e0a297405cccf128d4d7b8a4703244ff61
prerequisite-patch-id: 92f709c508faaf58207230a0122bcbb584a7f8ef
prerequisite-patch-id: a8c120b5e08269cca4d08412507abd2c93a34e0b
prerequisite-patch-id: 68cdf468ceb9a2b665fcf660ff1d9dc90b308c32
prerequisite-patch-id: ec86520c4537b8691623c6275765be46482e1d1e
prerequisite-patch-id: 7f48f11dd3cf0d4f1e9c155813a23561f86d76ec
prerequisite-patch-id: f12b6879d894861d59f0a52c65725c3a663dd0d0
prerequisite-patch-id: 2f55f5384e913e89819f7595bf06a6fc71074f72
prerequisite-patch-id: 66abb21c2745c20251f07d5450e18ba59ac1e3d9
prerequisite-patch-id: dc1159f014fcf438b10f6292d2affd8c1ba7c545
prerequisite-patch-id: ce9f25ee8122b61b303cd1e4ee71792e820e15fa
prerequisite-patch-id: 2629a09c4df6904eee0fa3c624065fefcae9a9ba
prerequisite-patch-id: 25e91f3381c956f3bf512f0f76c26e9c4917f3bd
prerequisite-patch-id: 416fdbc13fc712df5ed66ae0dd2cacd5fc45dc6d
prerequisite-patch-id: 2c688cc50d25eb016f80ca3ce130aef81afe61aa
prerequisite-patch-id: d738f17e5e853308d424caf2a8a16364b6590424
prerequisite-patch-id: d33f8d976c3fe9905a89bcd16a4ed3d54f4f781a
prerequisite-patch-id: 03bac0df161728e66d61993a6a847f0af2006f05
prerequisite-patch-id: 1029d41e51b1f6290746af3f4f05043f604c0b1b
prerequisite-patch-id: c6572f5733639e8f8f9d7cc0e8a246f1a6105c92
prerequisite-patch-id: 0badf7d8494fabc30a2967cd1faaddc8d3f73a5a
prerequisite-patch-id: 021e29c27905d8cdb1b47edc4a22321082a425a6
prerequisite-patch-id: 68569f42dfa917c069fd274bdde8a79b4fab3f48
prerequisite-patch-id: 1e2a91dc91a16102e9a9a459652ec7a8b7b5e677
prerequisite-patch-id: b1d458dfe2a8df06a7f9f9fe4c054672846ff7b6
prerequisite-patch-id: cc8bbd68781d9eb361eeb7ada60bb2d611c7a420
prerequisite-patch-id: 9d2a1df98901f9623bd0747a939a4369a385cc4c
prerequisite-patch-id: f3f9fefacd4daaa2968eeba36a9b317960dd6fc7
prerequisite-patch-id: 88dce86e6313e500889e7e0d0e4b2b7a2512262a
prerequisite-patch-id: 977b9cffb2a731d1d910c2c3c5751fccf2635b4a
prerequisite-patch-id: 9fd4c8c492fe5af9532e0292ab8d11a3105e8dd0
prerequisite-patch-id: e409f0f15c2d61a82dafdd34fcdff3a6d1919904
prerequisite-patch-id: c7c5151a581b577edbe80c3f1fd08e4d268bdadc
prerequisite-patch-id: 8fa60cc8887c6b24c0d6256ac2324c95c385e417
prerequisite-patch-id: d06da3377580154e7053958b47b2f8e69fba47db
prerequisite-patch-id: e0a1cd7adceadc7375e9fa2c2d71bd6c40decd84
prerequisite-patch-id: e8fc76121981dde48008657c3e3b04e625e26db2
prerequisite-patch-id: 4733ddcad5dd0658c25dc931c6368043daa8cd64
prerequisite-patch-id: a47b35a139038eb8776a2c8151380dc9e5945c34
prerequisite-patch-id: d1beb3f116818b224d79ce84784eaeb9ad037c4a
prerequisite-patch-id: 43fd2c8a5f4fa7e03f230b69a8dddd303fe49ac1
prerequisite-patch-id: 0dbe9548cfd6f249e101472be3069dcaba789979
prerequisite-patch-id: f84912cf01bea3e3bb2b22176caa92c581ff4f9b
prerequisite-patch-id: 879647838a9abeedf35ecadfcc95863ea4206db9
prerequisite-patch-id: 7c2791d63f0861316be998af4fc57c4b84cdbc5d
prerequisite-patch-id: 1c1c1674fc358312b470434e0be4c9feec5e5b63
prerequisite-patch-id: b0c5e83a0988a4514534970ab0245f5c943e49af
prerequisite-patch-id: 056e3603e23a67d9b050c041f8acb442c209244d
prerequisite-patch-id: 1877c0cf9459be83e044caf047a7698d16e744b9
prerequisite-patch-id: 4c907971f128c39db73b42394212cf5f79ba2c24
prerequisite-patch-id: 5f0c6241b518718570a362f2cea383ec5a65c732
prerequisite-patch-id: 6aa3b9dc5fdc13f4826f7017fc38975ac79e2896
prerequisite-patch-id: 9869e3ce9f6c3266d06d990f4e5b9d76f049d250
prerequisite-patch-id: 79e8ff4985dceb95b7c57ff232b4f5ba048f5ba0
prerequisite-patch-id: 2761fe023c74185047c24e5b5383e9ab95e8f2c1
prerequisite-patch-id: cbf6d86a18b4eca7ed99ddc4bd0c529cca38acb1
prerequisite-patch-id: 5f204bdb5f8fb455163f75778398c57e4df4745a
prerequisite-patch-id: 31f165cf29f08c57fc54fa7147d6f86a5faeaaed
prerequisite-patch-id: e8ded52ff2088a6067d651478d632501e415856d
prerequisite-patch-id: 6a463168443d4abf5c96f4bc7774d8e35794f4a3
prerequisite-patch-id: 537f61371635c4532bfeeaa5bd915ea97da5d201
prerequisite-patch-id: f68b88e7ba451ba5fc11da0997feb8886e290b99
prerequisite-patch-id: ef8fd30cae84eefc36b4092e16824f4732d604fa
prerequisite-patch-id: 6f90f5d2445cb332de0dc88395043a1a3c30267b
prerequisite-patch-id: 392d8792b9b1ed524fecb8f24c4756d655540d2c
prerequisite-patch-id: b1db472d486e1aa09bad7a488e5b54569e5868da
prerequisite-patch-id: 0e8b6d45cfbedd6d038602b6586952b44bcad8b0
prerequisite-patch-id: 2e6cc25733593b20256fe81980251abb24a913d6
prerequisite-patch-id: 32ae2785673ec8cdcfa6e05ace0ca10174071841
prerequisite-patch-id: 712bf5fec4c2d1280a5d8459977fe4c169c6d7f6
prerequisite-patch-id: 3d71f247bd9d6dc16bb59b1fc06a9a1ca6ba92b9
prerequisite-patch-id: 114f08b1cf5430d217d0e2caf6200c30aff3b121
prerequisite-patch-id: 782cea64700bc05c39bceb11ad313fb0a81c7df2
prerequisite-patch-id: bdbb9de7b0bcbaec6148437f14ad4ab077cd209b
prerequisite-patch-id: 5a29e15ec7c4dd2e7ad3889337b52c4e998eab57
prerequisite-patch-id: 4ad82fb5a21bebc80fb27f9e336665b9e72a1061
prerequisite-patch-id: 85d6ac957b30dadb251348951704319f3b238ec7
prerequisite-patch-id: 8813d16ecb6403951418bc1421941b753061f22a
prerequisite-patch-id: fc9e9b07db5d69e5453a20bb6bc451d1cd41616a
prerequisite-patch-id: 206d473ac8430bfb8310d6da95a1dddc6f6bcce3
prerequisite-patch-id: ee9ab694471ba519d2aead7231cd0f389192b1db
prerequisite-patch-id: d8a1856cfd49ad3dc12a160a4c287ed65940b9df
prerequisite-patch-id: 61dd08aec04d81138d09046036bc7cc4e32af2e3
prerequisite-patch-id: 30805f6b9c1f0044d53e92a0650c0410cc0acb38
prerequisite-patch-id: 4bc010c9f309625577bbdfe0e192c4f0d12f39ff
prerequisite-patch-id: 6afd9eeac577cb44c7f3180dbb9910fc17b17fc2
prerequisite-patch-id: 06b6d2c90cff7212519653374b5cf9ff87d0a50b
prerequisite-patch-id: f049381da80b9fc091c0ea9ce2270fc32d556035
prerequisite-patch-id: 5884d691f5b9f68e0c8a4069b3f1924e1bf731ef
prerequisite-patch-id: 32ae4b7ad9d9d3c0b8918219aec4390a334f149b
prerequisite-patch-id: 8a8bfc45bf2ebaf313031874db313444ef57f9f3
prerequisite-patch-id: 9a7bc44a5058d76798ed2479541b4de821145e78
prerequisite-patch-id: 635392e4d03dda79c324e215b27e782780a84161
prerequisite-patch-id: 1a50429cbb407fcb837722125364a04c7e8a9945
prerequisite-patch-id: baea216dbd90de43fb7eb98e8903a37b3fcc7f3c
prerequisite-patch-id: 3123868ff554a4e3a870fccd87034c3989167c56
prerequisite-patch-id: 6cd01a1482e406a3ac18fff08c3dce5073537434
prerequisite-patch-id: a8efe323e16316a40e75c135fe167465ff12ae22
prerequisite-patch-id: 4c0e8230bdf02ac424d3c3faef550ec9df9ebd05
prerequisite-patch-id: 970565876b8175c9979e9f9b413dd2f299f18c05
prerequisite-patch-id: addd3ea0c109ee8140a0f48f96cf1fbca5561a75
prerequisite-patch-id: 658d60c0042afabdcfe2792f2573e434e218ea7e
prerequisite-patch-id: 29008abb12edd72e8d9a7966ea93ef201635e0d0
prerequisite-patch-id: ef3b5095f29f83a82f379e3c813f1a3f5dba2f86
prerequisite-patch-id: 2b63f688032d142d3e1bc441ce103c77bf176f2b
prerequisite-patch-id: 939a366acfe1c510c168177440d735a178894bd0
prerequisite-patch-id: d97a8ad2a2af2142e3515794e0b692915212649c
prerequisite-patch-id: 6f27aeacbe016cb3d5b6931fee414f0c9b3c90d5
prerequisite-patch-id: a087411e8bc18caee7843d9d109a64fd83603376
prerequisite-patch-id: 2b886a469cbcd6188916a8ad2f42fb5c54bccac4
prerequisite-patch-id: b2c199b3ac6785ea456dbf4b87674899c9610ec8
prerequisite-patch-id: ccc4b2a34e0ca2ec9d88174870261efc6f47cdda
prerequisite-patch-id: 12ada6abaf98de410de33fec8bbc6245dacc2442
prerequisite-patch-id: 855406ba7270f034f3a88e6b9016376eafb31041
prerequisite-patch-id: 54419979f93951d3c7b656283f040d644c1bc526
prerequisite-patch-id: 084004427450ef334207220746099472acfa642a
prerequisite-patch-id: be329de732b4031c5a2cd686d99e8433eb62dc74
prerequisite-patch-id: 5a4dfbd0b614a8c015f20ae4185923e4d34c27cc
prerequisite-patch-id: 0af122faee9cf63e42604c25edcb70a3bb5e4d74
prerequisite-patch-id: b8dccba5b802c9548a48b5dec5bddf2f18e8ec12
prerequisite-patch-id: bc8f86dce2851ccd966c4a05a60270b6b08f8852
prerequisite-patch-id: 6f0b647063e0c34df925a827fabd79c284113951
prerequisite-patch-id: eb0a8c5b6a642c2b2dff1761015a7080e085d2d9
prerequisite-patch-id: 5c3b4f0c510ee2dbcb3ce354d0088c5e04b16894
prerequisite-patch-id: ea9f9ada0ef3d01d0b696e20c4644277ecffa5e7
prerequisite-patch-id: b1b2a0e22f38dae4e74ae09d59faf4aefc216edd
prerequisite-patch-id: d226c2836fc0004ff0a9d24ec3f38c6f263a22a0
prerequisite-patch-id: bf9a2c114cf9238ee172a5e3f284fb6c76dd20cc
prerequisite-patch-id: bace6948de8e0e25b3aa03545282857d87bf5db0
prerequisite-patch-id: 945d775dc66ac35fe43dd4893140820e539dd50f
prerequisite-patch-id: e2a6577071b9c11f867c5ddcf90bc705506415f7
prerequisite-patch-id: a85313a73be71d4f907edd346c902fe726ddcf02
prerequisite-patch-id: c1c3de603a51a8e895fc1e50161b9e2421aa7236
prerequisite-patch-id: 6727a2bc212edaec68f11394a9d1796550366974
prerequisite-patch-id: cda3c68a8cf7b2b0546d6e7a3c9eefb1d85bd52c
prerequisite-patch-id: 6c78fe8b57c1975287c549358d2436fc5e5bf11b
prerequisite-patch-id: c01293028f87a7a23dde1ed081533cb9032b4d3b
prerequisite-patch-id: 464a70cd7259ce1bfe51dced1fafb20a8290f2e3
prerequisite-patch-id: df721edf71c727d3921bf4b654fc99c487efa10f
prerequisite-patch-id: f07686d98b4ea4067122038628fcb05d719e0fd5
prerequisite-patch-id: d4400f0ed4f379bf789b350d7ff02726c1085c12
prerequisite-patch-id: 0982d00b7fdd6e59c96e146eebe50b06d0dd3eee
prerequisite-patch-id: 92fd699e1befda0d97d67509f86332cc9677c0bd
prerequisite-patch-id: 0bff6894ba9031d19a656281c98499b672745d43
prerequisite-patch-id: fcc062426abc792ac9b8caf8bb0e3774c71ce5e9
prerequisite-patch-id: 0007e834bb3ff82d400f66b8772509d3c9aeccbd
prerequisite-patch-id: 985a74df829b8a7002570e4903b8c76da3100c31
prerequisite-patch-id: 8b138daf1e6f0bde8b9fd23f11f286070e91c13f
prerequisite-patch-id: 6740651be7d8682288042571f150db5fdd2ce268
prerequisite-patch-id: db5df3bee5358aea7ae63239631a3e5d8778ddd2
prerequisite-patch-id: 43de8485e4f20f6095055679c675c9e69a8193e4
prerequisite-patch-id: f1d28f7616df3a856903601a12063996182dc00b
prerequisite-patch-id: cec1be3ac07f4316e48cfb93e93917c06026d20c
prerequisite-patch-id: 0b7ea0c1c2b4b5170f87386dd1d6913dea2bb10a
prerequisite-patch-id: 0090e96f64e8c931b1c36aff1be406d01c2dc743
prerequisite-patch-id: a73be4abe2b1dcc6193804a901a2a42b28ee5fad
prerequisite-patch-id: 75c264eaafaea09d6781ceeae699f6ca1ac133a3
prerequisite-patch-id: 3be4a9164b8fe0419232d39c68a00bacd530c883
prerequisite-patch-id: c0a3a19ff5045f257072638db01850a24dd69a8b
prerequisite-patch-id: 8a34f9f93292c6d0e9754200e0312e45cb119e96
prerequisite-patch-id: 40e6c024a047d4423e6a79716450f5fa02b890dc
prerequisite-patch-id: 5d1740a08cc624b72e4e4a73e32074ab7574818c
prerequisite-patch-id: baa4ead75a09f8d06352a0ea46f8e067cb079dcb
prerequisite-patch-id: f92e9f263d928e803a73103077224f6db1d6b905
prerequisite-patch-id: b0840c8c5009a5ea0f9475221ae430d5570a12a4
prerequisite-patch-id: 28b72495731e14acfc1a79668091c2363dfee9c4
prerequisite-patch-id: 81f2998bd78dbbf2e43fbad2465d8dc99e9b6f76
prerequisite-patch-id: ff568c436b5b1891f2c8c415b70f49c5e1ba79cd
prerequisite-patch-id: be4ea8275555f33bd4845acb68e565dbaa8ddf88
prerequisite-patch-id: 9d817b891d1614d97c6c6f6a5828a66db9625e5d
prerequisite-patch-id: 9f51a8baa9a9bc31d83838b11553e5011baedad4
prerequisite-patch-id: 3ac8fc898c030b6def3ef919a6e0d802e9104c6c
prerequisite-patch-id: 9ce11df7962457349322bf09648dcfe041cb4c15
prerequisite-patch-id: 39679f338d2884fb03eb568e4c05cde37c4b7d62
prerequisite-patch-id: 5a45be453070dae7523274936737228893055962
prerequisite-patch-id: bb46fcc0cc619d485e8aa421263f21a463e9249b
prerequisite-patch-id: 6d7b0777387449f497c646015db8340daa872ec3
prerequisite-patch-id: 57aa5a0ff2be4534319c5dcdb51bae3119655450
prerequisite-patch-id: f5b189582bccd0df51f92c384283fe62a662f361
prerequisite-patch-id: 8a4fab832c2421a3bbad6439aeb9634a78b2e8fa
prerequisite-patch-id: 83040c79ff4e22c528d036db30c41821a5de4eb6
prerequisite-patch-id: 8fdb93adf1b9088c913caca1a5c02a36307880c8
prerequisite-patch-id: f9b180a4a27cfe8e6d0f90be70166cc8bdd14372
prerequisite-patch-id: 3ac039e20a8b5217cd5b53216f7994c570a7d225
prerequisite-patch-id: d52630de6424c8a901e31aa234fb5451b6752e12
prerequisite-patch-id: a1ac61d41cec474193d099963e5a6d5d7eebb02e
prerequisite-patch-id: 243748f84e4c33995898ddf49ad79819661ea754
prerequisite-patch-id: 182e2105b1de611f3aa9fa1a5a6210c61eaebd0d
prerequisite-patch-id: bc92fbcf84a3ee1c3b44e3bdcf24d002a331296b
prerequisite-patch-id: 5257be3ed7f3a576e07c207bd13073f0c8fd1adb
prerequisite-patch-id: c42b55b057761321fbaf9d5e743878b19cb82f78
prerequisite-patch-id: 5ca26f7abf1c760569d2937b3168553a6c0245ba
prerequisite-patch-id: 9c743148e4c66db989af5f42f014d5f6dc79c435
prerequisite-patch-id: 75c649ba4ecb350855cfd43d3694f9c21c6bd934
prerequisite-patch-id: 44cfc7827d69c9f11676d83d4982800f1c4be6e4
prerequisite-patch-id: 110cc5d3f117b16549808a88b677bc55824bc91b
prerequisite-patch-id: 6e92cdcb7107c9217f045b05c859b484a2b70d23
prerequisite-patch-id: b6ddb390ebc52f9e85230c9db9f96b25d5000bc4
prerequisite-patch-id: 5ab0930bfd01508fab76ebb122d5cca85038a99f
prerequisite-patch-id: 1bc1845289729c172edbad7f13fbff782c3c40db
prerequisite-patch-id: 9a03e494dd6a3c0ed7345c24de70cf1579adfb32
prerequisite-patch-id: 901ea696d1d241a665d4ab3e762543988a326b0a
prerequisite-patch-id: 6a3959d15678641360877db80a19b09f7bf469b4
prerequisite-patch-id: 2878038b634ac466889423870dc6b83d59d1534f
prerequisite-patch-id: 070532c394ed3d32887e472b53ca9aa862f36592
prerequisite-patch-id: 11ddba0503ad58efd4ebba64483c4042d026f544
prerequisite-patch-id: 535de08752aa9395c5d80571a3e9713e21a9f861
prerequisite-patch-id: e986af1b3b2777fd612993a13cf3ddf3be0ea729
prerequisite-patch-id: c7dc3290f8ee97db0f290157d99eee748041203d
prerequisite-patch-id: a7d694c65dfff647fbbe680ce167239e6fc14ecf
prerequisite-patch-id: eefb293afc1dc654f4685a15d3074f1426064027
prerequisite-patch-id: 46862f1cd0820ece58de808166a9eff6460aee97
prerequisite-patch-id: c863dbb552254eeb3275b6d3732be8fe337f9b54
prerequisite-patch-id: bf44e01996d18d4155874cf785688432472f87d3
prerequisite-patch-id: 2c1a4a4034b7d9364f19d7eeee1cdb62411d736b
prerequisite-patch-id: b85804f5f63ce74bd0467a3d63cfc89688168065
prerequisite-patch-id: a724b02de89d063bde2cbfb069d583f11cbdc773
prerequisite-patch-id: 787bc2edc5b0eae2afcc0e456dbcdc3cead1fbf6
prerequisite-patch-id: 8f9ab0cb5f302a59bab15a1615ae40e2e594eab7
prerequisite-patch-id: 2220435d5a540e56319c4dd535549c0a7ec9d958
prerequisite-patch-id: e640a43975071c711bf2c8ce12e9e5480a7a017f
prerequisite-patch-id: 0f0c7bab25e97324eda731d5f9ecb2fcc666846c
prerequisite-patch-id: 4e10c81a0151f1b8c817a33810e4d675ed0e0d0d
prerequisite-patch-id: 06bf448d52b081e95329f7808b834d2656144ec3
prerequisite-patch-id: 06fc39732343a54f854b7c58514757ff0a5dbe75
prerequisite-patch-id: 7901ee09409c6bdd2865fbd38815eb998976aad9
prerequisite-patch-id: 06043139f97ba25f667f331e8944bb8f43002e2c
prerequisite-patch-id: 0fa78c1ddc1f1c8323dcf889bfdc5efb205069ce
prerequisite-patch-id: cf4d25c9e9ce9b8f735aed9c8257799f96309a64
prerequisite-patch-id: 273481041d1c4cc4975dde954d022b913e6410f5
prerequisite-patch-id: a626031549bfb89a292fd3798da5b373ebaae10b
prerequisite-patch-id: 6e3fad3f875bf8f1828ea93c7108146a97deb8a7
prerequisite-patch-id: e7a76a1aa9027fc7df9e279fac3bbdcc59e08742
prerequisite-patch-id: 963ef2431922e5e3ae870eb17c541420cc5bfeda
prerequisite-patch-id: db67610ca7b90e2cc1e764b3e375d38f06c2d004
prerequisite-patch-id: d989c522c82b45b3a03c6fec7594d7297f09bcec
prerequisite-patch-id: 85be256c94bed1da6b9ebb20e77b2b055783e55e
prerequisite-patch-id: a1a7e6e70582380d2f87dae8631814981bf5c287
prerequisite-patch-id: b42fe63328c91167725456f72605f155e7d35b5f
prerequisite-patch-id: 9a1f84c4d960c704059475b96e0d472ae2785718
prerequisite-patch-id: 73482edbcdf7cad22130c3c5f8fd19f72f4016b4
prerequisite-patch-id: 897e2ecbce6e9e2e015e23311ca0d0b8e770e5a0
prerequisite-patch-id: 5d1c8026b6edbf9502b477a88a7a473235f6833b
prerequisite-patch-id: 85e38b1bf4ac22908d6eb70518f643bfcaab36cb
prerequisite-patch-id: a1b1f7b720e0b0d9c5cea11aad878d9d20c4e9fe
prerequisite-patch-id: 2555ed927ed6531dbc8dfeff646721ce2e274534
prerequisite-patch-id: 49dd55b3ef8c2a67f8230d052073e96951a51a3b
prerequisite-patch-id: 92950220b15d4ca7b3c52870c4efbc21d0a806ea
prerequisite-patch-id: 54394e929bb94e0c53de402c715a6ed6377f336b
prerequisite-patch-id: 1ab1958b4aa4f8eedec46784a81cb3f13904061f
prerequisite-patch-id: 837fa2e1fa98d9533dda7ad634724dfcf64556f2
prerequisite-patch-id: 173cdcebf307869a6b6add803ef8139fb7ba58e8
prerequisite-patch-id: 147368cc73f4edc0eac6bec7e25803e476d4f3d9
prerequisite-patch-id: ae33a2cfde38bc2f42fe9e7348eb3f3803c16d45
prerequisite-patch-id: 9b8f4d34da917ef50bd8ef8e1ceb38d089131f54
prerequisite-patch-id: 31306cca2510419f8b0a554e3f4572428fabcb29
prerequisite-patch-id: 6051a56e8865fccf0b528bd75f572dd2d472c3de
prerequisite-patch-id: d5278762f173ba12f888eced152c27b0ce99bc83
prerequisite-patch-id: d1077b3b140d680b4206359912d6bf7c7627fbb0
prerequisite-patch-id: 2ad5206adfa3ff0d81ced89996e7262815eddff0
prerequisite-patch-id: eb7ce19dbd2693a536bc6316ccbac4374a42c40c
prerequisite-patch-id: ef3f2f60463cbc4f33327e7c0b8035e6169c19db
prerequisite-patch-id: dca90149f4c845ddf5e60428e6d69fef4580b827
prerequisite-patch-id: 1013ddc3522d45ead3a0cbb44827206724cc7f5c
prerequisite-patch-id: 3255c6028be9b549db15e7052fa0e3340d8cb4a6
prerequisite-patch-id: 6a95fc82ba5f3cf78990f89dca875b16b32eff3a
prerequisite-patch-id: 1cdb72a06de1f9e5afea8e02952baa366f4052d7
prerequisite-patch-id: 57cb23e1d4acdf5cbbb7174a54581239d35577c1
prerequisite-patch-id: e6a7b4fa9bb1ab451e843847772acfc8a6735069
prerequisite-patch-id: 1a1f2098dbe5268f615e42b90d3d70801f16be73
prerequisite-patch-id: 0c99ae56389a6a5589c84976f7ee427577206974
prerequisite-patch-id: 3073ddba81dddbdaf4fd2709457e9798704c2906
prerequisite-patch-id: dccaecee770dbafc7b93e5ad0db58f530353944c
prerequisite-patch-id: 8f5973e99751fdf81f94e80be83011b0114a65c7
prerequisite-patch-id: f6436859386a9ab728fe4d0e42095340d8d6280f
prerequisite-patch-id: 4f11d53e44f77b0eb96eab0c639014cc40c35a66
prerequisite-patch-id: 0b4cb280efeff9754c5c758c1ef997cac29b72c3
prerequisite-patch-id: 1dd1c3e988260d53a796375c8ef9d1444633acc7
prerequisite-patch-id: 1f687c7042ec3b629a5fcebd1dc83ec15e32cf4e
prerequisite-patch-id: 5879af0cbda416e5aa6c53d88863091d8b1e4407
prerequisite-patch-id: 6298f168fb6ef34ced7ae01afc294814146f90d9
prerequisite-patch-id: 9f8f121af776c2dd177e4ed66b80af1a92635b51
prerequisite-patch-id: 74303a5e89a3024438e16d236b02db78e87c176f
prerequisite-patch-id: 26455bec4a0ebf9bd68b483c891352c5fbe15154
prerequisite-patch-id: 883d9dc3cc0f21aa7237230166d759ec04c86da2
prerequisite-patch-id: f55667a5e87943047aae3c8acb771c945fd60a61
prerequisite-patch-id: 155dd929189593a223c9b87971fda6cfd4ee7d02
prerequisite-patch-id: 2795c504af025d6cf2a7cfdc53f6d3cb2e61aa01
prerequisite-patch-id: 66d437a9cf929a484c38f8816c3e6a159c4afd4f
prerequisite-patch-id: cfc847e49f7c3e9f7595cca9f9c27dd48f58ac31
prerequisite-patch-id: 13b8296ae2593007b44c0ad47b09b040a71b4237
prerequisite-patch-id: b372743afbcfcd99b271c7dc178636192d5554b9
prerequisite-patch-id: c1b86d0e2d874da4acdb11bd35c366f4dfca592b
prerequisite-patch-id: f58b279e8c6e23ad50f8d7c0c135d8e2fbb7dbf9
prerequisite-patch-id: 8e8506e63fc5acb6036ec13fed46cd6f64647b25
prerequisite-patch-id: c97de7d9dc36d59a7b6e0774db9ebeb00ee49cfa
prerequisite-patch-id: 4d22a15fc13dd8c22b68f32411f619f4f5f06031
prerequisite-patch-id: 030d6f481201eb00423b1f3f31fc4ac177ad744b
prerequisite-patch-id: fded371737b462926e4e46799d58bb5f2e662e9c
prerequisite-patch-id: 2b49e6f3184990f4dd090451a0d187beb836a33d
prerequisite-patch-id: 2dc38c769fea0e61d7ea26f5678d8f76ffc8385a
prerequisite-patch-id: aa3059e01d277c5b815df0172d9678ada062bdc1
prerequisite-patch-id: 56a33ee453fedb1a48dbc091298673c836872469
prerequisite-patch-id: d1a830b3c74494e78f2dd1cfca80989e8271db30
prerequisite-patch-id: ab66638db584af150e1b981f30a972f3ae16a678
prerequisite-patch-id: e97e372655090e136b7a3be940472501ebcf8563
prerequisite-patch-id: 11b966644927e3ea8a172a176d253d821c223435
prerequisite-patch-id: b452b372eb1f3ba03b4e352af744207c47756488
prerequisite-patch-id: 2ae8ed5a5867855f4419b42f998298060fefb1f2
prerequisite-patch-id: f9d9b9a274da20370907046f9f146c34ab58facc
prerequisite-patch-id: 3845235703cf6271a301e2126f1f881274eeb979
prerequisite-patch-id: ea1c337039108219d988446047511df63de6a231
prerequisite-patch-id: ed8515321afbc02262e365ea1af572ca51686fff
prerequisite-patch-id: 3df02fb90d43b4ca1d30216b8e2372b8fcaa3438
prerequisite-patch-id: 892ded69018e167f0e0d3a129a3cd9688c969a01
prerequisite-patch-id: fd51e453172c51f232102ff0ea1013f9ea06f086
prerequisite-patch-id: 896f4c9c7b60c7c806c3f89f5ff6816761ed6182
prerequisite-patch-id: 07f9d198c2990fc862f4d8f173e51371575f86ee
prerequisite-patch-id: 30c0bbd00c0aed5a58aacd637e0e2534bf3cf16f
prerequisite-patch-id: 8e8237761c16158f97d5b503b8913a4fbad7c3e6
prerequisite-patch-id: c4144bc4a038f29bc6f68fb2275f0b5ba8f31b60
prerequisite-patch-id: 3587d60e29dc0ca0dfbccb49ab631857f43b1632
prerequisite-patch-id: 85d16e7393dde39c0a14693a8cdb333cfaf0d007
prerequisite-patch-id: 5518761e2fd3b65a3b6bc9faeeef62cc2e8014b6
prerequisite-patch-id: ef94decfa59004cabf20b8981eace30c527f5176
prerequisite-patch-id: 3dcfd91371c737c82e41c63145de1535c8a9dd18
prerequisite-patch-id: 051f528001d215dec9dd066d2e5c340608f2daf6
prerequisite-patch-id: b2900ab223469c927ac55d1da6184286e6038f28
prerequisite-patch-id: 1c5d9553d3829565e87ae5cd7e2d5dbac81ae466
prerequisite-patch-id: de53fd2f0b77d1e61f09340e4ec1c127a191390f
prerequisite-patch-id: 1b9f205c1cef0b2d45e1e08321e31cc239da4789
prerequisite-patch-id: 45b260156c63c5a5bf062713305d9ae282a1e3b0
prerequisite-patch-id: 00286c2f4f8a84177313582fd6fa0524ddb5a098
prerequisite-patch-id: f8ce0a97233d0d537618ab101d0e410cdada3fba
prerequisite-patch-id: 13163fc64a35163f48b07e8f5595376b98d2ced3
prerequisite-patch-id: 3c2a2a0dd7492880a28f3e5cd02d25d9a6f4301c
prerequisite-patch-id: eed5cba1d574dada94e69d1af227392b247810b7
prerequisite-patch-id: f2c72dc4e0206fd4f3122c77b34cc438b8ba5707
prerequisite-patch-id: 9193eab5af26fd4d7c6bfece5b7492716fa502f6
prerequisite-patch-id: 217404b4b4de0b050d1a531d870932c46ecf0028
prerequisite-patch-id: 79d61b95cc07a3b27004ffabd807d24f6ee348f6
prerequisite-patch-id: 1294b9c3ebe6add3621d4fb4e79c4139adba8ac2
prerequisite-patch-id: 47ed8ae19ffcb8ba03f306cd02da2be7398ba95f
prerequisite-patch-id: 03e983abf83abb15c14790ea24bc4081b9269f1b
prerequisite-patch-id: 0de47f21f09c75037fdd26f5ebafb455cfd563cc
prerequisite-patch-id: bf439c65be96d2fe08498345b21da6dd01596ab6
prerequisite-patch-id: 959ff5ba7f097435837bbcc09e541b695cd012e6
prerequisite-patch-id: fc1db2b70dbea6428fba43cd8ee2f00c89104f2b
prerequisite-patch-id: 34b3e2033296413e181840eb9ab477df62217e72
prerequisite-patch-id: 12bb71fb40b298359db885ad1bbdd35ac5fa6523
prerequisite-patch-id: deb822cb734d5727efeb4224b2e4da12177fc46a
prerequisite-patch-id: 7509b386a3f790c4accae1174c11fd47f9559fe2
prerequisite-patch-id: 2de66e3ccaa3f0195072840484241d08a21fb6d1
prerequisite-patch-id: 57deebe1d329286efa8968ff310347642bc3c95d
prerequisite-patch-id: 30debd7f6462ed2aaefedc2ff6dc70476d28b8d0
prerequisite-patch-id: 04259df5dfb493f5b53274e8cc90065e4df4e703
prerequisite-patch-id: 153f9fe32b251f0abb63d87d08b3e3b230eb5b53
prerequisite-patch-id: ae05bd1ba17c4e4de8cbfaca973dc1d91f82f435
prerequisite-patch-id: 95b5c0242332dffed6dc485783a561a302de6492
prerequisite-patch-id: 382fc534cac40dfbe2f5f5a13024eaa3fafbe51e
prerequisite-patch-id: 2920ccf356b76054e0f9821c041183f951235bad
prerequisite-patch-id: cbc3c5805b699f8f02fb88be4b16522dc7af840b
prerequisite-patch-id: bf95630f37b72c1067b0df4628ede4bbae6ef334
prerequisite-patch-id: f382bd06d87893383a9296dd8bab20954a9a574e
prerequisite-patch-id: e88ef31d0d636c9fe1c1ba109f919e2199bdc392
prerequisite-patch-id: a3c7a30ba3325ab1334388a3c4b0c23fddadf8de
prerequisite-patch-id: d5f41560767c1a8ac6e1d8d983671db28aaface2
prerequisite-patch-id: 46f63af6109f65e9eb2ee6d8bbccda49c6172b4f
prerequisite-patch-id: 1a4fb82ef1995e8360691b8ede81502ea6e66813
prerequisite-patch-id: 436e404a9983065961e8230866cebc867d93c948
prerequisite-patch-id: 7b5c64e29651073ac56978199a79bf8ed084cc8b
prerequisite-patch-id: b2746064711f860070344d11a7cb9ae28e4eba5d
prerequisite-patch-id: 83de7b33033f8b7d107d2f9b960a864bcba65b16
prerequisite-patch-id: 01dc2c52438d0dfc501258954d7b324c5d11ae9c
prerequisite-patch-id: 1aa197807862c5439e16463d38ab5e7dfaf25cba
prerequisite-patch-id: 06a40bba59c3acebaa2099565e6f04a76420d573
prerequisite-patch-id: c081c33df5d5642f3015cbf355974d917ae28870
prerequisite-patch-id: ec5e9e149017e942793c397c237c2ae672b3f42a
prerequisite-patch-id: 7edff554cedbe18b8f3bc02a67dac3c915d0210f
prerequisite-patch-id: d00a5bdae7253fdc936ff552a2b58f6f52b3f5a0
prerequisite-patch-id: dc6e332dc81755ef34aeadf8240737f9dccb104c
prerequisite-patch-id: af84a91939a41f95fe2346c16a68aaeb37a4f641
prerequisite-patch-id: db7928bb79bd4aeffeaa60a6f9ff8b065919c94c
prerequisite-patch-id: 0e518075ae3209faa4893bf3187a64ec417ab442
prerequisite-patch-id: 268f2009013420b9487607933ad4aede4a2001f8
prerequisite-patch-id: 8f1ab5221864517cb3fc80441ae71e22db4baf51
prerequisite-patch-id: 62513866538318d3bae35f79177de7d15628097b
prerequisite-patch-id: 0c4ebc06ede9d1f9d3b07100ec76d394ee15c488
prerequisite-patch-id: 056b1e8010d15d36eae737d7413d2ccc0875890e
prerequisite-patch-id: d3160056bb417272c951586cda10d50c6388ee60
prerequisite-patch-id: eed160cfb978683231c670e42a6b205a28679174
prerequisite-patch-id: 58f389a1a5acf34a46436333ca0d909bf022a591
prerequisite-patch-id: 84589052265eb9229ac34659873944ad382b2524
prerequisite-patch-id: 21e68053eebc87cf71485d679666d7d9273d8487
prerequisite-patch-id: e5e1defce72e6945528876f1c2cde92c6c736dae
prerequisite-patch-id: 0341c1c00e5e1d04e5fd1097823b4d4c6dae92f6
prerequisite-patch-id: 96a3b8d529c250712433d2106215cce44dd8f64e
prerequisite-patch-id: 60784817fa91030b9ede4e1737d66d632fd999e3
prerequisite-patch-id: 68862ca84d0a1a785fa974d3ccfc929c70a49f66
prerequisite-patch-id: 7574d97104a0e7f145ee7d34d12cc675cd3d8a2a
prerequisite-patch-id: 55a9abc4ea006c525711c628c09ac7ab37898cc0
prerequisite-patch-id: 4eb1a7075906b49b533a1934bd5a0828def60a97
prerequisite-patch-id: 34c2518b2b49f91a2667e494571a4f8bac48faa3
prerequisite-patch-id: 7debe7e05adc4902deb0a9202e3dd97f29348ce1
prerequisite-patch-id: c9c9666c9c47ca1e5e68ad3a8442c4ae0e78d900
prerequisite-patch-id: b2c5e3b2f4003af18a75446959cae35ea1648bc3
prerequisite-patch-id: 7acd3a08bc7554519acca5819b56bbbdae70ed3d
prerequisite-patch-id: 1db655bf52599eab96b173cfaed4316b230b2bbe
prerequisite-patch-id: cc8dc88d1dcdd62c0a2de7c50cb32ffb8d7f102d
prerequisite-patch-id: 04e2903a8467c09aed7e4f4825d944c67ce4fd17
prerequisite-patch-id: ac1259ca8d62a36d76200d6aac21fd3c6775853d
prerequisite-patch-id: 5c6d9f1d978fa24e829126dc0418c542d51511d0
prerequisite-patch-id: db0645b5b42c0f59515a017b1fe93b4e4cbb2ff7
prerequisite-patch-id: d2769eeb5122bd4cb5fc5ca73f24c21cc9d5d1d5
prerequisite-patch-id: 08ac0bd42d6ee51d6fa74f4a18d540722eb46282
prerequisite-patch-id: dd9a925574c5e158ba425459fca382649b15117e
prerequisite-patch-id: 9217798eec18086c7af4b5cac83c2728962b1b27
prerequisite-patch-id: 73d271d19390a5f1d55bd2c0564e3be8c55a8bfa
prerequisite-patch-id: 8ecab5b382e4617328a76decb063643c08971330
prerequisite-patch-id: 06eedb9537c8908e16c4b73570805f147c0aa7a6
prerequisite-patch-id: 40f0ef1f64d8c8d5c67224eea72e09b8760bd7c6
prerequisite-patch-id: 731760e194dbf5f1a0d12660d1e3491b3a8a41b6
prerequisite-patch-id: cf9f958495047a114a853ce1d160f1f26483e9ae
prerequisite-patch-id: b1ec08a3514e4a4a5c1448299d13076f29d8977d
prerequisite-patch-id: 39e1c80db9e91cdf964cac898098e91436e70a2a
prerequisite-patch-id: 50a76bc20d70be0726a598aae7fc61f6edc36aa0
prerequisite-patch-id: 420c9896e24ec4f30b2f1994fbd08bb356f35909
prerequisite-patch-id: 467135ae1d015370502956d43c8f45dba8adadbd
prerequisite-patch-id: 618de01ea418223993d28780f508f664ae3074b7
prerequisite-patch-id: f5bcf4751282cc66a644b099dc66896c3cb420d2
prerequisite-patch-id: abebb2bffb7ed958c2a84d8121f96de1c84fc820
prerequisite-patch-id: abc36b610493ed09260e7e41f785ab3f6c3c9dd4
prerequisite-patch-id: 5448626e34003a90318069582a565c125179454f
prerequisite-patch-id: ea32d7782ae53cba2722df1472cffdfa594f4aee
prerequisite-patch-id: ff11f21f8b15cb9e4e50e7f27d8e1aa3d60357c9
prerequisite-patch-id: 85f6280796a6f843ed24cd4eb041a8c6621e8eaa
prerequisite-patch-id: 5992b5156d45a690506c0f5a3181f24460309b7f
prerequisite-patch-id: 63a5eed0ba540536e20ce524e2eb646ee451a261
prerequisite-patch-id: f0d0c5e8ecadee7fd64b9238e67586c49f229654
prerequisite-patch-id: 9b91d99b31e3f08bf502496867174735e21db660
prerequisite-patch-id: 237a67d1c83747b8bd2b94eda22e01977d83d919
prerequisite-patch-id: 2420f543c866f1cff1f204e5b3b036e960c30c42
prerequisite-patch-id: 5197dd4a37c0f0c70441ddb6523cf889b749ba32
prerequisite-patch-id: 5b149cf3f0706dec6e6ca8ec1fcb9f368c992b71
prerequisite-patch-id: 798a116f0c76a23e1fadf16b9284ec47f8aa3667
prerequisite-patch-id: 23aa7434ea7f9cacc52d13262c1e0c3fb291192d
prerequisite-patch-id: ffa8d17f55c6364b7a6b5d2c40621481f58088f9
prerequisite-patch-id: 243f8b4e9af1afc35c653ab83e35bac0a107bba5
prerequisite-patch-id: 935c536a4da7629f229b1d727ca8055d437336ba
prerequisite-patch-id: 4eb16edb090d0cf06ebf441609f98be95415dfa6
prerequisite-patch-id: e7829ac271b14df2977a5f070d73641184604145
prerequisite-patch-id: 9f920e698eaff83155a1c58b4bf8bfd3b53a4060
prerequisite-patch-id: a093bfa64e9320f9c4e2823f4bce8ca7e899734c
prerequisite-patch-id: 925957c035444aefb17b52633dc04cc563e26756
prerequisite-patch-id: 91d263f28fbd8ef57fd5ac1193e39e76fce1c9bb
prerequisite-patch-id: 6111f02d7daf8ede020a05dd202789f2eec0e71c
prerequisite-patch-id: fd3f9c62e0e1dcfd03ef7305568e44d1fa5b0572
prerequisite-patch-id: f0afe484af0e7dca6184d7e6266c0bc438d7a82e
prerequisite-patch-id: 031320e7b5b43497bd3a3b035c60d3c2b4590c6c
prerequisite-patch-id: 2412c486cebfd6cb9821c4c0a7e14c9a94e424b6
prerequisite-patch-id: 9ee897efa36f7087265fe0b59fa942147831cf84
prerequisite-patch-id: 78abe009993d877dfc43074f74d7b2c0ee6354e8
prerequisite-patch-id: ee9617761e10146b3932d5fb68435d3958a7c666
prerequisite-patch-id: edd891afc35358394ae991ebb84f8a59d7c3eec2
prerequisite-patch-id: 501fb93103dd732dec4b85ced2a667af30cfe2a4
prerequisite-patch-id: f3b0709ba690b45d9ac6fdbb5037e74e7d2c68c3
prerequisite-patch-id: 3e57f788314aa368144ab1c3ff6ed88b0f95a234
prerequisite-patch-id: 2a1f8aea4e4b1a2d09696c497d00374b8a1391ff
prerequisite-patch-id: ad7e10a9b6a20d0911089096fa56f67f41ea2686
prerequisite-patch-id: 8055d5c76756a428e6c0c9bab808460655e20228
prerequisite-patch-id: fc15f7c10dae0b5a34a1253006f16efd92ae6e4b
prerequisite-patch-id: 634e0f4b8b2e57ddd988da5db73b0a6f468f6bf9
prerequisite-patch-id: 76259d6e33054f6c3cad1e82f34d87adb88892ad
prerequisite-patch-id: 57b0bf73bab8e4c85c5e1ca6e794b9db4089474d
prerequisite-patch-id: 7354887d0fc6f4854f017eb96059cf4d4df57b75
prerequisite-patch-id: f40c6bc0b0e328762ea985479c5a0ef88e87daf8
prerequisite-patch-id: 8d013e9e82ba080c124f59feb8b19c786065048a
prerequisite-patch-id: 972b485a48a93b716e9b9179a15c3c46959a3e09
prerequisite-patch-id: 134e8dc206cfad79cfa3c68ceaf016e97eb2cdb1
prerequisite-patch-id: c7c01e5516a7f55134a4c61ef0415868f8ee66b7
prerequisite-patch-id: d36996df837725609eb046e6d36b7688bfe5f524
prerequisite-patch-id: 06f69d553f14f5af7a8714b03ea700375af896f3
prerequisite-patch-id: 9746cf6e46aa837873d4f6f13a22ffbba7e1ada6
prerequisite-patch-id: 7a0b0c6ec3ab34729eb3d43222672305efdcc323
prerequisite-patch-id: e8d98ce8b389cb1d7ac0e56622e5172618fbea97
prerequisite-patch-id: 3ebb5259fde093ee75deed39cd12045529aaa2ea
prerequisite-patch-id: 0d24e580f73e2c959e816d8dd15a8575bb73196f
prerequisite-patch-id: d2e33c5dfe970bed2d470961f9ad17f5ec71dc50
prerequisite-patch-id: 8504773a58376b3f2cc3cfd8ffcf700707ae7cdc
prerequisite-patch-id: 46315d860bf298e486cd88ff913879bc0af4c2e8
prerequisite-patch-id: 52d24f6fbe20ac26521909c5ab46a30f9e2dfb85
prerequisite-patch-id: 8ac4d5eb605b6ab6da24be349c6062d1acf771a5
prerequisite-patch-id: da6c959b278dd8c00a0116021075d621cdb65730
prerequisite-patch-id: 10090220c0e670d7a426e66d6fea8975a6acf60a
prerequisite-patch-id: 75d054fd4a935c96d181092416852a482a76f8d4
prerequisite-patch-id: db6c9727a795e424251a20f920081725b74da910
prerequisite-patch-id: cf357539b3934e3e493ddeb8f88d5734d52ba93b
prerequisite-patch-id: decea47e13839b2844f5303bbea5fb5659e4d71e
prerequisite-patch-id: 00764a4d6763fef4ada6659305885566f30bee12
prerequisite-patch-id: 60f3a283a56e77e666e51717d1eae8b3969cf78f
prerequisite-patch-id: ef53640393fa6c2e2067d2d263ac0992c46952b9
prerequisite-patch-id: 8caae48bf2e9d7b39608dadfc71d4b3bd3685985
prerequisite-patch-id: fa64087abcbcfbad49eb3ea8ffcf246a4f5dfbac
prerequisite-patch-id: 2d48caf1e8c93ba6210a71411f2fd75c572319c8
prerequisite-patch-id: 370f9da949fce1c586c916f045bfc28985a30ece
prerequisite-patch-id: 94d81d90b5097891839d33afca31c44c44f491d2
prerequisite-patch-id: 0ec93739dc84df97269b394b7e80b6d25d1b6760
prerequisite-patch-id: c4df0ca3a46c7527f02c7c81ce7567dd16c073c0
prerequisite-patch-id: aac3312a808f7ee8c1565cd9048b099956ff5231
prerequisite-patch-id: 173c14d7cc1cff25701efbe850cda32328ee505c
prerequisite-patch-id: 6f52d02aa5e9f89171e388d2aee1662d336270b5
prerequisite-patch-id: db0bedb3d8b059b203acc9aed821c35dfe382931
prerequisite-patch-id: f177a0cc475866c3e8acaddec8d9ea0710e8156c
prerequisite-patch-id: c7cca8ad687385e771c7f33bc05af75730dee6ca
prerequisite-patch-id: 39c7b7573e68a29a7c2d354e51795ad4c1e10801
prerequisite-patch-id: f73abe9607eb118b33a7f746ea916a6623d36af2
prerequisite-patch-id: f644f2653e0ed40155d4b944c2cf419bd300d188
prerequisite-patch-id: 491f90e9299928d3de6e0e9b8110b3bc3d8e089e
prerequisite-patch-id: fa63106e268842265418962017886678d6a33d05
prerequisite-patch-id: 852a6092ad10e01ea086fecf04a4dc2c1da801b4
prerequisite-patch-id: 7f61308971d8fabef648a8dfa0a937827cdff320
prerequisite-patch-id: 0ed80efecd479fa9606a7876c9605251e55ff8a1
prerequisite-patch-id: ef39dfc1a291c75d9ce1fce67d50670d07c7cd35
prerequisite-patch-id: a88f9664e09fb1a98f7300413b149cf9516174cd
prerequisite-patch-id: b68300512f8e688dd232f7af8d34c9189b53fb2e
prerequisite-patch-id: 238f1f3d163560545afa92cb4388403c0e6da0f2
prerequisite-patch-id: f1d92edc95c568075c3dbc5f2b419d9369aff19d
prerequisite-patch-id: f2210af95d27d0d4853dc725947b8e03826caa71
prerequisite-patch-id: e4d0a951916e2b25aeb598b646bdbf21ffd3c0b1
prerequisite-patch-id: a501326f9ec912b8fdb0ac5d8214db926b4dddc7
prerequisite-patch-id: 4cf88a276a32e9c811ed6e0933226aa5fbe9ff5d
prerequisite-patch-id: ff6eac459f6484b702adbd3c59eec3819f273395
prerequisite-patch-id: 590b30b6169d26f7789acd4b3e651195e5701f48
prerequisite-patch-id: 5b7ea4c03396f85d36158d3744ab7880f668f4fd
prerequisite-patch-id: e777df0fd9221ab6e2b5d5e568c002bb5b89d6de
prerequisite-patch-id: 2a77353ff3cad9e6f6e01756ccbf21244730384d
prerequisite-patch-id: 11d87266a0459d621f82ee66e3ac60881d443253
prerequisite-patch-id: 6404e0427f884bb493c2092d5d56f8a6f02b46f9
prerequisite-patch-id: 65cbfcf60cac34bea68780378183af8f13167a20
prerequisite-patch-id: ed5d375c62428a83f6b365166013723a01b87dac
prerequisite-patch-id: 81272a343fadb2140a7a7c2bef71a0853cf64573
prerequisite-patch-id: bb345139aafe19379621af356c258936f1c53e6b
prerequisite-patch-id: 1c06218ec8acd0062f7bf113b28c26ce693c3a35
prerequisite-patch-id: e3039112cf2da303d2093338b0ed8b6c472872f3
prerequisite-patch-id: 9f1841dbe42b7b7a9bca740de596568800ce0e1a
prerequisite-patch-id: 11a354408272fad479cd8c8e8635791f97b39d3e
prerequisite-patch-id: 2500a2d7329dad50fbbfd5c3dcea639964b3212f
prerequisite-patch-id: 6ad9b3cbf1b789d45075544e1106957bce69fed8
prerequisite-patch-id: f87f4833f78124b492910152d9587b6e9c6379b0
prerequisite-patch-id: 1d02c892708378e7343295b03272f9aca7041d3c
prerequisite-patch-id: df6ac078502bc3a345c92b56e98723ab8584d6ad
prerequisite-patch-id: e2f512dad5693cdbdcb3620c7dcee52ec689b11b
prerequisite-patch-id: e986e9cbc46097e0cb4626447f72dc4a1911e706
prerequisite-patch-id: c63e30ce4bde77c98d646ad6d592e2aa07c908af
prerequisite-patch-id: f014e34d52a73b3f8bfff9885cf3f2224f21c78e
prerequisite-patch-id: b5d5c7cb0fba43592272dba61a685f1cbccdc8c4
prerequisite-patch-id: 46f8abe0625b1bad66c0f28dc9d2e430f2f3363b
prerequisite-patch-id: 862c541f28c077844b0bfa48f2144262880c6a44
prerequisite-patch-id: 06ba9d289c88a63acaffa69cfbce0c40e9dea7b3
prerequisite-patch-id: 619e0ce36eaca824d1d8ca69efc1d4c45c636da1
prerequisite-patch-id: 4498c9031066bbde6797ebc36d776ea8964dba46
prerequisite-patch-id: 236c1ae8e9ca3fa0a148df79830e54b296482490
prerequisite-patch-id: 2bae2221ff9ab485f1c854cf0245fae43b5b1296
prerequisite-patch-id: e57b449459037835ecd520efd79244527bd7266a
prerequisite-patch-id: 8002f736b165a3f375193eb900f8d41877689964
prerequisite-patch-id: 8c1eb038e2613037f8db074d2f10b8da2efafc10
prerequisite-patch-id: 5018c489fb074d98ea95f28205c8b629c65f7564
prerequisite-patch-id: 0c31bcef7463624e2bcb38b8d187494530318fb1
prerequisite-patch-id: 3ab0a75893b2369bee1e5362cde1c75f6ba40e1e
prerequisite-patch-id: 42686eb87f20e6af2989daa9e0bb8e6a020708aa
prerequisite-patch-id: cbda5e6e72f52840923d28a97808049647de0dd6
prerequisite-patch-id: e4e63a698cb6fd898dd04b0bb216c20f5d6edddc
prerequisite-patch-id: 7fde90890c129d5d5c0ce301b1b895a786d00776
prerequisite-patch-id: 5aa1a9da7ef27a889c6470e9e4b447d34cd10d6b
prerequisite-patch-id: 5a8557148e77850c7992bca431c45724133a7bb7
prerequisite-patch-id: f21a243220a60cb79d22e8607c25cae9fec04282
prerequisite-patch-id: e649d3b34c98ca12645b0afb89df100a70be476d
prerequisite-patch-id: a89858f1d8f20b6d3b85017c930dfd1a577c8e60
prerequisite-patch-id: 6392cc5691299cdb45c8702a586c857934b51e75
prerequisite-patch-id: 62edf3e96891b53f1b663f2a97a8976dee962859
prerequisite-patch-id: 9d624768473afe293949fc751d756307242674b2
prerequisite-patch-id: 895665db6ce81238879cdeec1c90fa022a757603
prerequisite-patch-id: ba53a3692d50ff8e3e2711f092c7d646b8f1bd01
prerequisite-patch-id: f8725e60b459416c5ae3ce0d489bee97fa9cafa0
prerequisite-patch-id: 72b8a7a49939e764547be136004d65d197f3716b
prerequisite-patch-id: a833dd28954fa611365935352dab78e00b9e38d6
prerequisite-patch-id: 40a5eb3a1c54364c5f89d13a79033645503732aa
prerequisite-patch-id: 5add3c304c174d32c38d16115c6789ca977cb606
prerequisite-patch-id: f924b0f3d24bcd9b5000384b1bad24de79bf258d
prerequisite-patch-id: 3c7f9faf78ec7ec9cdd33384eab00897b6bb7e2c
prerequisite-patch-id: 2937f128b9bed96cf11f8e2bbc91777aa6b19df1
prerequisite-patch-id: 5040444f959fa3b5e75fad1aa1047f9b8d5142f8
prerequisite-patch-id: 74229139793c7cc8ccf388c90e35db447c062a13
prerequisite-patch-id: 4302d984cced34df4c4f01d2ca3ac75f14d89d64
prerequisite-patch-id: df9356a541b1747637e60d7c51924a3877bc8e9f
prerequisite-patch-id: bf77498ea30c954428ab0d4a6c5e93bab2dfc1e0
prerequisite-patch-id: b993bf3c8f09c3ef9644432c04e21d80781308de
prerequisite-patch-id: e27c6e49800d58ca352dca03565a266a6a25fce7
prerequisite-patch-id: f64485c81239177b91e6a1d64ddbe6a5f25f74c5
prerequisite-patch-id: f97b4de43506b4458aefd006ccca772421774e15
prerequisite-patch-id: f820c87452bcfb7f4f088852a8f599b959319267
prerequisite-patch-id: 23652244b104bc5da5acb783aec4bb174b785d41
prerequisite-patch-id: 0ebac18bec761e6cbdb48429679ddfaf19f705d0
prerequisite-patch-id: 3af126ab459c72e96e76309c19461522ea57b729
prerequisite-patch-id: 7fd66b146f5173793a315c5880a0732903124caa
prerequisite-patch-id: fa2c74ffcfbdd2cf99e049016050c89957499ef4
prerequisite-patch-id: 6253ff2f8cf4c59136507618e716f2959084e0e4
prerequisite-patch-id: 59c1ac91d9de5b98d520bc9edcffb36529be9817
prerequisite-patch-id: 065cc0efcd2b443ad6b5f814dd48a38740fe2cd7
prerequisite-patch-id: a2b0e8fe8e9a719632bb74fab07cb9f63931f37a
prerequisite-patch-id: 420949305b73dfdab1036f34a0929a587f378c33
prerequisite-patch-id: ecbd2dba3acf52ed7415bd9ed7e431421927a9ad
prerequisite-patch-id: 6668ca2ee1b6dfbe1b00465f039c7b605454ec6c
prerequisite-patch-id: 67f4c0ee0982828e11abeca1f118a7e987331a09
prerequisite-patch-id: 7621292e80f4a238091598d1eb3ae805bb070709
prerequisite-patch-id: 9d59321909b18786205de00ae6b5d7f28aad8fa6
prerequisite-patch-id: 314905339d5680f2fc67bb3cd5cc7192cf0c6400
prerequisite-patch-id: 5ae117618942a87c71078d6cc5738925b41e0bbf
prerequisite-patch-id: 4328b9ce7536a66dbebea79da293fabebc637e4e
prerequisite-patch-id: df9843af4a1c84898c0144e6294ab53809a23256
prerequisite-patch-id: 076d3da2781c0af86367d941fb299674c2d6958e
prerequisite-patch-id: 48d612f00a9a5efcb400eac052bb55593be38c1e
prerequisite-patch-id: a05901a51b37f4ef5b67e93301f2a23dc28fb4c7
prerequisite-patch-id: 6885e13276e46429c9dd94b717f02a94a3b6722b
prerequisite-patch-id: 0e9b811346a65d0fde20f3994b506bf73b74f5f6
prerequisite-patch-id: 4685d4acf2649f9280544b7936942f56b0e418fa
prerequisite-patch-id: 0f8dc20559ad260032ed723b51eff18adb6a37f6
prerequisite-patch-id: 58e396c87fcab1c03d75a28c676cd34747f460dc
prerequisite-patch-id: a34701d0b24f4fe994149b5090c43af00b4f47c0
prerequisite-patch-id: a1dfff515bab78ed1bd83df9b6a81f7e1f08c5cb
prerequisite-patch-id: 07521a53dfdaa91fd5f9b58a88d92587c7d44c31
prerequisite-patch-id: d46cc28d588403d355f91364c5b6dcb8e72929d3
prerequisite-patch-id: 6a88ff9f03af40ec4d43aa7f30c759af1f2066c4
prerequisite-patch-id: efb1ccf001d8a79fe49dfe40c688253825cf0c76
prerequisite-patch-id: 1fc9352f9602e0dab21dcd22d906c7bb75c9dd94
prerequisite-patch-id: c872169e584e9c573a2fe3fb88b10ea8ce421651
prerequisite-patch-id: 0aed61d304e9d9185def3e1d7d1be3c501d5d946
prerequisite-patch-id: c45d8e6ce731a5921cefb1b0eee5e9ef902380e1
prerequisite-patch-id: 2ce067f6c4e6b633ece97e2965a6f803248e8185
prerequisite-patch-id: b36d24f2552369a83e20f794a720cae23d391680
prerequisite-patch-id: 6d0244cea3e6cfddf6a7c32a696b8161593e3d66
prerequisite-patch-id: 0c30fd348dc1f8684df9e75b55534f18356e560d
prerequisite-patch-id: f15c04d4e79a495cd13cb719e0c27258dd3e25a1
prerequisite-patch-id: 64058243d8494592413d9d74c217645f6c8b032d
prerequisite-patch-id: 0518ab4c3c8c06a7bed10961b2cf64236aabe592
prerequisite-patch-id: 6ab8ba48056f078c4d332a1aa7a9af1e0da6a6c1
prerequisite-patch-id: 249215ad12d97209a16c258ae71a984ef36ab44d
prerequisite-patch-id: 069d07da14295d9868363f2c04fd543be9a3e5cc
prerequisite-patch-id: 869d630d593ac138f44a2ca14729c17ea3f4768c
prerequisite-patch-id: 50d4f44bef2d32534a8c708602ba7af7d9db473b
prerequisite-patch-id: 1784aa999916c851385bfb8ae008a9cb7b4b1125
prerequisite-patch-id: 69bc697cf25199ffcc0e66328659108e36415e34
prerequisite-patch-id: 0b6c08494a74ea5cfb7c5c40900bfa25f2a53751
prerequisite-patch-id: 41a1c6d276f5feb614f804c434eedeaebfa4b556
prerequisite-patch-id: ea322e76901f7a4b12bd9abf9e352f007ec296df
prerequisite-patch-id: c0a07b2f6c9fdcd903b4f6b167a1a509813da2ca
prerequisite-patch-id: 4f6cfa91375555d4a29eebb6ec1f6319d1d14026
prerequisite-patch-id: e9355d54b14f858cb4bb73a599042bb1283220ca
prerequisite-patch-id: 0e1965f5288854f9d99f2c8101f100d5aba21ee8
prerequisite-patch-id: d56cf389574ee6871b57afa853c3e6da8dbacf88
prerequisite-patch-id: 9897e81043dd13d347291cde084455a870597b88
prerequisite-patch-id: e370cdc3769c018c1386a74fc9bf164bbab74163
prerequisite-patch-id: d82cc2e2e156178a983191b6c5b878312c3da4c7
prerequisite-patch-id: c561efd60cb44130b603f044fdc9d4e241d21e11
prerequisite-patch-id: 2eccd04d6bb5f56ef6213a9ee727a1903c5963c4
prerequisite-patch-id: 1ea3449079f25fab84bd34d37ce2420d93602773
prerequisite-patch-id: aa074c15b1fce933239736244c5f3f5d41ce3310
prerequisite-patch-id: ae6211e7a6d58314bcb6f33f0444e903dc50fbc0
prerequisite-patch-id: ea807e03fd96c94581ffb86335a91fc9bd28f7ef
prerequisite-patch-id: cca7c828d0353e8b74addbed14961818a6a33c6a
prerequisite-patch-id: 12b659eb2b5b800d1ba9f349f07314a4508b5dcd
prerequisite-patch-id: ed0fd31d10d65561bfe5c68b951872d3c4eb291a
prerequisite-patch-id: 6f161438f92f980a48807a10a6c9f01fc3fce030
prerequisite-patch-id: c6971c072b807bc7977149f29d4ae69d23384946
prerequisite-patch-id: 0f8c6528433ee38a6910b40e96645ac7d2deccc7
prerequisite-patch-id: 3f1cbf311a4cac95e0635e2cb8088871b445e67b
prerequisite-patch-id: edc4747d25b5e7e8ef3a65667ded12b21b964cf3
prerequisite-patch-id: c4924037c953f426a56d88a04364de7302bdd3cd
prerequisite-patch-id: 3b5f5c3fe7a2d105c811c44e56ad255629cbd06f
prerequisite-patch-id: f22434618a55e3601ada6f903495ef431988d5c2
prerequisite-patch-id: 22979ad789d5f17cf1f220c2af4882bbefe6ab88
prerequisite-patch-id: 91b5211ad3732853ee471b3f207109e09906896e
prerequisite-patch-id: 8930a9baa2cea0f9bd74c324b710db82740a292b
prerequisite-patch-id: 72679f01dc5d8ad384313f6f547c9bd339f34a3b
prerequisite-patch-id: d905648e0ac0f3ee7bd7d990c84af20d9b01cc54
prerequisite-patch-id: 12003ba8d888014179461f859d3ecc34170694ef
prerequisite-patch-id: 51381be7ab0e9397abb311e6e1e4310807430661
prerequisite-patch-id: 513883391f89bda93b4709b802d2bd9549a0378a
prerequisite-patch-id: 6d5dd44268da4ae486fd194fbd49ded3303e3400
prerequisite-patch-id: b453cdfa0d1e993bf2481793f5efbd5958222ee6
prerequisite-patch-id: a3facdd4fc9120a292f4552e16410e3626086125
prerequisite-patch-id: eb38334e38ddc18a116e1de03f7ed0e030cc086c
prerequisite-patch-id: 4153608d6e683dbab3a7c76a43897c38647221cc
prerequisite-patch-id: cdeea42a7560a1397481d1b60ba628418510becb
prerequisite-patch-id: b78def513610f43e0b936eec83d93832c4e3f5d7
prerequisite-patch-id: ba6b2e6248f8fd3a3a3b8e973322d619628d8bc9
prerequisite-patch-id: 28b72495731e14acfc1a79668091c2363dfee9c4
prerequisite-patch-id: e430437c022a80ab7ee03a1afe6a10f643cd081d
prerequisite-patch-id: 93338df550d4d58e137631efd91e3a5ac05d1413
prerequisite-patch-id: a9ce52e16abcf2fbe39e144803e3284e969b9aef
prerequisite-patch-id: 9983bed851fb3bc3cbbc2902f388a12cb705d68a
prerequisite-patch-id: 40ac8ba7e2152b1e072b8d7e238aa0e45086dca6
prerequisite-patch-id: 8232ad4ec2eac07735bfccf53568060706cc0687
prerequisite-patch-id: 390757a385b27f66e0621e62dd2e8ec809e19a07
prerequisite-patch-id: e56c6f6c7ee9cc7ef502ab41d321b0f4742f7725
prerequisite-patch-id: 29a5de6fcbca758b63b9bc8b20b9fb773fd6f14e
prerequisite-patch-id: 34c0c842388720170bd8ac7c8cc88f716029693d
prerequisite-patch-id: 4c281e0621401e30347845f0b8443560214551d1
prerequisite-patch-id: 9d6fda847ce41c6ddad88ff47810fd19d032def0
prerequisite-patch-id: 55f2b9f8d9bc939b67f93da00173912d13c05832
prerequisite-patch-id: 59d42c6807eb990f444ffe80fa8dbb89ed095afe
prerequisite-patch-id: 0eb4a5890e7e74c294fc122ac253d96af39595cb
prerequisite-patch-id: 5ad8737a808ea48011766009c20c620ab051c90a
prerequisite-patch-id: a1d21cb660f9e285c57f694184314bb6d7c05ee3
prerequisite-patch-id: added19da0dcdf969ec0d853206f0e7f8b13cf68
prerequisite-patch-id: bd6a55036880849989d00947440b383dce7a9e3f
prerequisite-patch-id: 5b7dbaa60885fd3ed62b9c4bc54c9cac8d5c103c
prerequisite-patch-id: 8544e215a18151b2473ded8ef473d06483f623cd
prerequisite-patch-id: 5a9aa728ae9039ee9e8706bbc17d56b47a903545
prerequisite-patch-id: 1d5fc0ba5efb5697ea9772be3e19f3bc49e6579f
prerequisite-patch-id: 6bebfc55f1dfd63486b13865b8057b7feff1136f
prerequisite-patch-id: 7a7ae42590f236a5627f75aee6c3d7d16fbe3017
prerequisite-patch-id: 7e44486d318b0bf2e32911a032f00010eb2ded96
prerequisite-patch-id: 613a019b2610333b44ea720afbb6d241078501f5
prerequisite-patch-id: a29e2aaa5bee7ba5d3419a88223ecd0581b7d257
prerequisite-patch-id: 6588c1556c4220cae4198356ce9da3621c189d83
prerequisite-patch-id: 4587761a51718e5d576a635f87d3e88d67b7a6c7
prerequisite-patch-id: cc98b436a69297553f5c44dbfd742ddc1841c951
prerequisite-patch-id: 9004e2fbeae0eb7c21e752e627862a457858225d
prerequisite-patch-id: f0f3e426cfd96675212ff23d781fe8f1c5d6b7ce
prerequisite-patch-id: d6e8b03c99ccdb3ce66e39daf4d4452fcb1857bf
prerequisite-patch-id: f7b3bcb14a7030cf897ea355c20cfa5d1f339981
prerequisite-patch-id: c1d469ab765058c856af3ef843df7cdad8aa37f6
prerequisite-patch-id: ef79e8c9bae6a7d8cbfa4c3ce172704bc70f2325
prerequisite-patch-id: 3a8fbd85557b28416257fc602517a54b6147d95c
prerequisite-patch-id: b2687a0c0505e7e6528ec63d6685dd52908287a0
prerequisite-patch-id: cb363cbf087eeaea7f4aeeb1ec9a40c2c9d3dcdb
prerequisite-patch-id: f49eae3f854a76dc44ef80d09b299677d070ddbc
prerequisite-patch-id: 5795a2f1a370a04f58e85899dbc03f709dd2db42
prerequisite-patch-id: fbe4770bde4205b7cba6a8c892cee0457b8a9ce9
prerequisite-patch-id: b03e1b774dea3401c6fa71387586522e7420a65f
prerequisite-patch-id: 96ae3f7a56df8dea9dd35c1581e128287626cd4b
prerequisite-patch-id: 11bb0675775d4878eb9125bae8f626861a05a46b
prerequisite-patch-id: 04b8744a307c0b0260d59dbe88efdd1866e67fbf
prerequisite-patch-id: 9164ab9f5082c84db5b37d103b2e8a6679466871
prerequisite-patch-id: e3254bad10bdc24153d9f0fef4e12cd9fba036e3
prerequisite-patch-id: 1b4add5ef451065306a0aab093bc34d146efadf3
prerequisite-patch-id: 6930f9446637e9e49f0f24c1724e27bfe3d5defe
prerequisite-patch-id: 7ebb50e64bf5b423c8b1804a122c8ec8d5635841
prerequisite-patch-id: 9f24c7ac6d857c479e4cf3708601f1642f6edcaf
prerequisite-patch-id: e9f30266e6cba6804a7a46a81741e824362d11da
prerequisite-patch-id: afabdd8e1c6531dc3daf8d95e88c98126e16d463
prerequisite-patch-id: bf3c542db96c708f9097f424778576c53f2c8bc5
prerequisite-patch-id: 2019f0b2d75ccb9028ff2a4b9a1dc6ce02f6e36a
prerequisite-patch-id: 4e6405301350df8432aca2aa8ab0ed9af95b5c5a
prerequisite-patch-id: 9fe1a92110390af410ec9cc92c92e2f6dd2bacd1
prerequisite-patch-id: 6e9475b171b1cc647b39c6bae9ba515fe65d5a53
prerequisite-patch-id: c78a4f60e8c1c0e9e171f3e413b5c404c0827ac6
prerequisite-patch-id: 0cb10bfe1b1d12204a837ec965d1a6a912b7b679
prerequisite-patch-id: 81a64e30b15f874c83105022fcc686c49a32dd68
prerequisite-patch-id: 17cdc806d249433b8bf2266a23c763108a9c9408
prerequisite-patch-id: 0338123ec39e4429a183c49662b885a0a377a81e
prerequisite-patch-id: c77c125548addc0f8d731b8579db9da4803038b3
prerequisite-patch-id: fd7f3fcaa1352841531c098d61e6388e1c37cd56
prerequisite-patch-id: ee9d4f8eafb610399883b0f7005befae2d1fef7b
prerequisite-patch-id: c629deef06b699e6aa7dc91506d8253fcfeb022f
prerequisite-patch-id: 8dcb8909349aae97006c3a8e0aa552828738e754
prerequisite-patch-id: 98a0e4cafb71020c17bb575304ff5f2778ead28e
prerequisite-patch-id: dca1378fa1331680d4a8afcdf4bc3da71862e0a3
prerequisite-patch-id: 5270c136830952778d737fed71e5ef9f347d9c86
prerequisite-patch-id: 457a0cb1dc3a98ff9e1b1b572174b217639d7e91
prerequisite-patch-id: 4b6a848f30bc8b6e114e548d6a28ec57a0e9ce25
prerequisite-patch-id: e3af50feecc9c582b5b15de614fa1250717b5418
prerequisite-patch-id: 1bb8a7e233cc9fe1526c75aac1d2deb39008f494
prerequisite-patch-id: edffe1af5d3580e80b7ef1adbcc3b0af09f4bbb2
prerequisite-patch-id: 424877e0e5c4d662f63de27b19fd289df2028703
prerequisite-patch-id: 7427149f89d5c9fe60f677b4b56ebeb58dfdc68c
prerequisite-patch-id: b239cf2a9e946cab0320152750fcad4bd0c9213d
prerequisite-patch-id: d96ed1c21f228e476edfdcc31edd0bc518e6ad34
prerequisite-patch-id: 2b9a485f0761d759b22b62ce1e353bc28e3b9365
prerequisite-patch-id: 8e27a58e1e2f0e0c9737d8863f5712bf30dc232d
prerequisite-patch-id: c15d16c83925599fabd2ea3e43ca805da931e5ff
prerequisite-patch-id: f5eee1acc98ba563d0714ea9fbb9b0ad8764d235
prerequisite-patch-id: 678fa907cfc03b529d6ad167bf18e04b7b0a620c
prerequisite-patch-id: 0380ffb225c6adbeeed29212c8ad2f14bb9bd710
prerequisite-patch-id: 328ba258498823a3883c710b306becb0570e5830
prerequisite-patch-id: 0ee8819f7884ffa26455cda45081009a0b71d0cc
prerequisite-patch-id: 8c3e4787fb9d7f05824a231ee28ac58c0426af12
prerequisite-patch-id: 573bff398455437aa765156ad574737f15f32fa9
prerequisite-patch-id: 5ca7206a28906eaa8b21dc40de63b212aac4d94c
prerequisite-patch-id: dc9a4037536619aa12560f7cb9eff6f9c0f59a32
prerequisite-patch-id: 517e2990b4f1d8717e312437091aa1d55ae663f1
prerequisite-patch-id: 9ad4685669e2376f51dfb2842308f63af5c2846a
prerequisite-patch-id: dc1cbfc31413072c9091bbe232c401d03d7e8a81
prerequisite-patch-id: f0455cd2fc8ca138113b20ae572a2c0498fda538
prerequisite-patch-id: a418a1fc10bdd047358db375062f6b048c4c42cf
prerequisite-patch-id: b475573079da71f099a36293331785103f03004f
prerequisite-patch-id: d1980ce81fe7bfe2b8ac1a6bf06d83e80a98ab3f
prerequisite-patch-id: 59c719110fa5312a359dade0585a9b0a29d7158f
prerequisite-patch-id: 08ce3e29aac67a57a071f1572abc28c3f9bcd602
prerequisite-patch-id: 18c30eeaf9e62adc80b264fb9fc8c63aca874f69
prerequisite-patch-id: 750d3f1dd4b536d33ab6bc2e4230ce942c3df617
prerequisite-patch-id: 5b6a0cb8d72c59fc27d201b2fb3d459e81fefe1e
prerequisite-patch-id: 3c734df7b03f28fbe169161975dd3e7767d3790c
prerequisite-patch-id: 729973084f1e52d6ab70bc7798df960c3996eeac
prerequisite-patch-id: 31a9a36ca8a31053b2d3f2c8ce6adf5d94aae248
prerequisite-patch-id: a06e2876f6a229d884e4e105c997bda1b8428736
prerequisite-patch-id: 22965edce2d442d7356c81eca2e78d26619f9815
prerequisite-patch-id: 46c3492713aa91723fa4d72079c5537ae131454c
prerequisite-patch-id: 8aa9cdf4f039e742249376906342d21f2884afb1
prerequisite-patch-id: cf4a7e7cdb101d5649281314e84839fb39acc829
prerequisite-patch-id: 5faf2da2a3ad4cae080cfab9cbfa62d619e1ddcb
prerequisite-patch-id: 104d0273a3d05db4b8cadaa72ea8c40340611772
prerequisite-patch-id: b5ddab3b99cda3849b210591249a469f283fe12f
prerequisite-patch-id: 5c748ac476e9942a5fa5cab5961c3f3f2956794b
prerequisite-patch-id: 9ef670102b262f7b036d00539a0bb3c8ac6ece31
prerequisite-patch-id: 3fc845f4fd71a90ffca5ca4c7c167e9f06cbb49f
prerequisite-patch-id: 6e7cb4f94d5454c62b42720571f30721a77c72ad
prerequisite-patch-id: ebe8199f262c9f343ad318f67a1cb1e72fcf1348
prerequisite-patch-id: 0783ccea25059eb76bf1f15c697e947ae77444e3
prerequisite-patch-id: be25b982af9e6a74882845331d1b1948f30db564
prerequisite-patch-id: 90419591af246acc6b39e3664a29b775cf416aac
prerequisite-patch-id: c7b40acceb488402ee07b4a5f2bf5683c489fc8b
prerequisite-patch-id: 1cc311a2fccfd4b6019b4fcd8e818ca8d3fce31b
prerequisite-patch-id: 2b95afc329986cbfe8fa1a5ab889a7276612e310
prerequisite-patch-id: 5388256cfb73a2f2819ad2ae26894a5432e96dcd
prerequisite-patch-id: c1119aa8d56b78567b8a81f0c0c5d88ba0783812
prerequisite-patch-id: 32123f9a83387e01007eb2f6a73e78d75f34fdc7
prerequisite-patch-id: c048a1031d433c6ece7cb607815950073cac3111
prerequisite-patch-id: 6f24642e2ed9802f9a1a20ee8ebb00bcaaff92d7
prerequisite-patch-id: 04cb32ac39f28776801b3614140899e1bee9da12
prerequisite-patch-id: 1ba102f59bb9ce95f1bd3c870007a146138e5ae0
prerequisite-patch-id: 4a0d377303e738e2792b541f5314f85cd721b302
prerequisite-patch-id: a131b53bf5c55241cf39d9b7c13cd361aaa86c6b
prerequisite-patch-id: 3f92a1755c26674abb028a0fec019a8621758c4f
prerequisite-patch-id: 1d78b99f61e196688ec4f7d8a259255c2776f6f1
prerequisite-patch-id: c8823f3654f4190a1daeed3c0cc08771e3db4bfe
prerequisite-patch-id: d1fc47f2a0371da0cb93668670a6633929212d48
prerequisite-patch-id: 946c5585ef4aeeb12e4b9dd3ceff4fba51441ec7
prerequisite-patch-id: 3c9f21d9c6bacb573e570d25878cfb63bcd3e8d8
prerequisite-patch-id: a35e7cc47d6cebe4076c97b2d811742687a6fcf9
prerequisite-patch-id: 7baf4054e830376d4423ae2561f8acaebef27e14
prerequisite-patch-id: 542ad08acf7068aba4eb7efd8d2799147d342b1b
prerequisite-patch-id: 56a3b8f53f45c7384468f8fe064eea16f42adaaa
prerequisite-patch-id: 2c03310bd28f2e9e4b2d4d195e3edfde07d29f20
prerequisite-patch-id: 8ce1f8eb8f9d9d6b0d714e12ebfd9e358b8713c0
prerequisite-patch-id: cba2d358e7b3c11d35a593353bbd47f514f291b9
prerequisite-patch-id: 699d043b709125e8748ce4e73c32c3c00ed9246e
prerequisite-patch-id: 7ab9d831a894c4664e51a9ee253683aac7332c63
prerequisite-patch-id: eb72944306c1e146a1e0c9a0b4adc1c8c3bd566a
prerequisite-patch-id: 202b53077139415170c62c8a01d4342ca282bd6e
prerequisite-patch-id: 6913284408db7a031017569d128d29f150cbd2f3
prerequisite-patch-id: d437bc089602533e4e3ae7eeeb876471ba569be2
prerequisite-patch-id: b1a9e3d69c7bd8a01874e31278d05fd78b8c0941
prerequisite-patch-id: f6c5152a812c5490b8635bab89a3c9d48338c611
prerequisite-patch-id: 055bb55bfee8c8cb680883f7e86c0dee10f2d627
prerequisite-patch-id: b321d93ce021c1c4ce60a9dfdeb4c6c82b8fad9f
prerequisite-patch-id: 009891a3dcec0f36072ef751d411b031e66811d6
prerequisite-patch-id: 00ef3bed65ff2568e5502808406931ef6b33e70f
prerequisite-patch-id: c7de32de87e43e37934315cd89c8c292ed646778
prerequisite-patch-id: 3b8e08eca0ddc9c4c13c6aa7c7004e447dfd31b0
prerequisite-patch-id: f0e8fd344755cadcd77b8d2282de12ed83470ace
prerequisite-patch-id: 83798c45cc2d8b308618d756cbfc21caa3dace19
prerequisite-patch-id: a6d252ffdd5807db1f9b6122924434be87d7c644
prerequisite-patch-id: d3f1954a96025484f6735780670d25b59e3a12d3
prerequisite-patch-id: bf2855e7a2460c09dfb564ac2693e10607db1206
prerequisite-patch-id: 3cfc1d4be865b6520502fd0ed122c4765ad18bb6
prerequisite-patch-id: e2e34b7b2b84761da9bc39ddc4a8249efbf27d03
prerequisite-patch-id: 6cf5204d6ecefc40e03da7cba7f570f5a651db65
prerequisite-patch-id: 35e5690ade82282e16789d59aa187e820b248fcc
prerequisite-patch-id: 7ea494f1785ba68eb17fb7869e2eaad44bf4553a
prerequisite-patch-id: 523d9d44fac19a3f2f6d7d956cc6c88d8898e1b1
prerequisite-patch-id: 61ecac7c083e9b42e6ea22bb7bc0ee399bd04020
prerequisite-patch-id: c7ba7a92eaeee95d5fb206911141358390bb4c53
prerequisite-patch-id: cca97410a687557f6ba340d5f49f920d6dd0ccc7
prerequisite-patch-id: 922dc370a6b2d11627453a19eee9eca011b98a53
prerequisite-patch-id: d057779b53a2223c15cbcf94d81584b789d37489
prerequisite-patch-id: 872746b55c91b37391f7e8da91fcebba9721186b
prerequisite-patch-id: 0fa7ecf66abbbb38ec7e824ef12efd9c7ee22158
prerequisite-patch-id: 1f8c2f5622829312336065625589dff292fa9946
prerequisite-patch-id: 534ec5d08599e044c2c24486f4b41b39ab8cee55
prerequisite-patch-id: dc3743a908e2a305ee29f3a66f21555c72db202b
prerequisite-patch-id: fde2aaafdd62b094c1449bb2d287eac823cbf7fc
prerequisite-patch-id: c16fec73b92e3c609b6c42fd14be739c7e4e0ffc
prerequisite-patch-id: 3ed25009afbf79f4d5da11b1fee5c00f711be6fe
prerequisite-patch-id: cf50531f90bce8d9eb5594f16b695664d888e297
prerequisite-patch-id: a606f3291074e77536b96c27c28a2ee8de183232
prerequisite-patch-id: 6c898e6f1c93c9d9522e42d3619c784b1ea953c5
prerequisite-patch-id: 0934f7be2e482dee2e64b7c60a7d92c89ea1441c
prerequisite-patch-id: 205972fe555c28b53717266eb23436a82ed8daa5
prerequisite-patch-id: 6f341f17114775ed537733ebe4ab9ffc56b6e2d8
prerequisite-patch-id: 49f2585ec938bb3a6c10d378ea466e8c77ccc8d9
prerequisite-patch-id: b13efd215dd11daac648a14a1e6dff8efcfd02cb
prerequisite-patch-id: 46ff9da4dfc9f1c26c7635cdaf658f824ea21875
prerequisite-patch-id: 36d979b75d20d82323b874c15815d90dfddf4dd3
prerequisite-patch-id: cc835d5a72237c93e77737500b9d590f338dcaa3
prerequisite-patch-id: fd9eb12da903d07bcde709d7ae66988ca286e791
prerequisite-patch-id: 0bd65fd485eda542f630f1648afdc63f524f6eb1
prerequisite-patch-id: cb5b8a407ef968b45a63e7654d67914175f60c80
prerequisite-patch-id: 83a653ba5b80514a02a80b0a8fc48c2e28a8e66a
prerequisite-patch-id: bc5e5d35bbfc50a5703ff4f4c69c4d318141043a
prerequisite-patch-id: d66d5c5f952ddd2e84048f630537e5e93fee25e4
prerequisite-patch-id: 2877fae3c5a2e6700270f92474978637b548f7cb
prerequisite-patch-id: dab9a022f4365373ee1e867833216834d4900c1b
prerequisite-patch-id: d6a36198bb48d683ebe8525905e21ec2d726a671
prerequisite-patch-id: 6e7d502fb123eb46be07c2dae43443932ad77cdb
prerequisite-patch-id: c1ed4c595e5e09a26d95848c7fe1af12384b708c
prerequisite-patch-id: 0158886f82cf74d329f376309f51ff562c279554
prerequisite-patch-id: 540d4937eb1d8f9b0fafb1c0c49c4b9b5b6321c3
prerequisite-patch-id: 1721dcdf464226c90de74bbeb5d63220c8f82e23
prerequisite-patch-id: 64f5f96fa911f336c5a2e9f9ced4be310269dd8e
prerequisite-patch-id: caebe3e7cb4bfad451020384ecc9bb6b967540ea
prerequisite-patch-id: ad2dc73f27692d7ec838f84f1ff7b58ed7021a3f
prerequisite-patch-id: 6065c0f5d90b58514018f52c1fe25ca4c137f76d
prerequisite-patch-id: c15f7b1903fb0c1ef5ed4728fe80847100791197
prerequisite-patch-id: 040838aeaa059db802a4694ee09a618329bd3654
prerequisite-patch-id: 934571b5204014de9c3036c1c5abba2eae44e4ac
prerequisite-patch-id: 49d7182e3456b53fcf05dbad357ce22bc41efe3d
prerequisite-patch-id: dca53eab7db3d0c1e2fc239e07f13a6dd4af2233
prerequisite-patch-id: 7401cff7a5e9128327ec6c218246877896dc5f7a
prerequisite-patch-id: e11207c9ec36a8fb9dee082935b0b79c6955e068
prerequisite-patch-id: f16097f20f07fa2f37299c9c71cc20a30893b878
prerequisite-patch-id: f0d64711a91ae87869e931e71f2de4e16c1fef9b
prerequisite-patch-id: f0066fd29835a17e07368b26b074884fdd93f5ec
prerequisite-patch-id: 50acf42191545d507ab0ef486613a9faa823a9c8
prerequisite-patch-id: 3875845246e1405b7e3d0ae7e1daa337b46021cf
prerequisite-patch-id: 9c70cf6cbaf32e5ffbd4cff74df3cf90a60ad410
prerequisite-patch-id: 466ccbb6c611af3d67085d3fd3223c1a4a57959a
prerequisite-patch-id: 832a4c1bb236399012262a73a3e05bdad8a84db8
prerequisite-patch-id: 7d1679bd2fd4f85670f1b741b3442fd45bdbdd64
prerequisite-patch-id: dc37b46f26285e8d1e5d9b2bfdf214b2f765457a
prerequisite-patch-id: 70837893fc301a30e30739906ccf46d82baf5532
prerequisite-patch-id: 98810026a834c633c6089c83de8301611e312b28
prerequisite-patch-id: 63fcca22c2b47ecffbf941c2e6eb1f01bff43358
prerequisite-patch-id: 42a87d570def0bedbf2db6bff22e217baaedaa22
prerequisite-patch-id: 45f706cde07c5e5385ef4d1c25be5fcdc3071842
prerequisite-patch-id: f1e616db47b35ecf30be1e4c9706414caf51d8da
prerequisite-patch-id: 62ee9033ce48753c96f4c25fa2a66d869fe7ff97
prerequisite-patch-id: 92d3bb1adfefd958c8c442820ecfcaa18ccc3b0d
prerequisite-patch-id: c64d50fdced59de7d7eea7e0183304efdc080345
prerequisite-patch-id: 0755adcc12b58c70fda09cf1a629e31c5bfd21bf
prerequisite-patch-id: bc56910fb1dddc33d7e03344599ed83ebaf0040b
prerequisite-patch-id: 1ce6fa46332bb071ca2ec5a4e861786ac1e01fa8
prerequisite-patch-id: df253ba1486958b72d8329941062472f415b6fc7
prerequisite-patch-id: 183ff8d378d8a3b0729e00587adb72d8b8f14f22
prerequisite-patch-id: 72918841b84c51e65695ac292c46fc69878a2352
prerequisite-patch-id: 161efc73b3406eed1e6a4b731924a455532d0e5d
prerequisite-patch-id: 5294bf5a16224957d4f86d030d98199661ddf720
prerequisite-patch-id: 192fa81a8669eb7f39b456f07391f8a507b875df
prerequisite-patch-id: 732a95a179644747809aa6ac8ca4d06fc55b49e9
prerequisite-patch-id: 041e56a325f25688ad84b814cecb9db647a3a6d2
prerequisite-patch-id: b83c25218139fdbac6009959f8e064dd98628797
prerequisite-patch-id: ee523f6d39ac5c774c4ad9f3b8a5784de1152939
prerequisite-patch-id: a49df6d3fc26f6440240ddbf234e34cebf1ef16f
prerequisite-patch-id: 9c62854be71f74639b4e90d5abf6ccade9eff884
prerequisite-patch-id: 595ae66ceb1c12b4e4dc3b9490db91bd5ac74a15
prerequisite-patch-id: 43ddb6d44359b30938671c3a36af1ee4a05b0180
prerequisite-patch-id: c995202d269f5e4bb40f247e02db86ec5671adb1
prerequisite-patch-id: 91964f622cedb5a8be3713a74cdd28221fe65e72
prerequisite-patch-id: 39e875f28e9bb90d3254f46fb45aab19fc4342e6
prerequisite-patch-id: 0e71edf714ad88a0564a3a2dc132a1487cef0cf1
prerequisite-patch-id: 7d72e30bf9427af73dc2e6ccbd4c91f3b58ccfe9
prerequisite-patch-id: 2a9c4a3509dd4c7b6f7f50a776b909b77e516c14
prerequisite-patch-id: bcf624d9bef6e71660d62e71912a8ad3a71035c8
prerequisite-patch-id: e2386f60f4d0ab857afe7be6d83a026f204e8ff6
prerequisite-patch-id: b7ee5317c224a8070a7df9b9bda67eb5d3aac302
prerequisite-patch-id: 6db5e2bdbec7cf70c94eeb48c3432c3e5f0c0d50
prerequisite-patch-id: b99cef5dc6fad1bb57bd16bb740fc022f6c6b1c6
prerequisite-patch-id: b9c7f37353ccdfff2dd645d42a90f95514a1db54
prerequisite-patch-id: f35d20b4a97140484bb22268e1ca567141ab57cc
prerequisite-patch-id: f73734305c8fc96bc3a828f224225b47a0d25cfe
prerequisite-patch-id: c0e1107fa5b6072c94ca4abdfe90947412655d3a
prerequisite-patch-id: 184b53a10799774bebb924f287549d6bc9c6d6cb
prerequisite-patch-id: 0d462cd85a93821fdd600042478c695db4d687d7
prerequisite-patch-id: 0a14c5d87d64b8b850b2d6d16f88f09d1b4cb4d2
prerequisite-patch-id: 398bc65be166a871d5dbf89b61a6b8f749a75be1
prerequisite-patch-id: 7f3359413f59912996287f18e6395d0cad0efe39
prerequisite-patch-id: ad859862d2d367259d6a7dcd8bd77729bed39c21
prerequisite-patch-id: b22e67e99b2815334fcf73743fdf055ed153e005
prerequisite-patch-id: c8b4a954e20f953dbff008089440a1b2fa8cbf0f
prerequisite-patch-id: 312861abf86d5799e6f85ce5cf34ac8b9c46f95c
prerequisite-patch-id: 31856aa8d6ac44dc1ee30e37665eac1120360cd8
prerequisite-patch-id: 0e8e39a8c74c3dd80092a223e5729160be71451c
prerequisite-patch-id: 59f17bdb15c833f3627e99dcbb8ff5df56a1d685
prerequisite-patch-id: af943909b3b77614d39f7a35210dfec536d0c712
prerequisite-patch-id: a3eaed315e73b4cb584ae176de28903510ada792
prerequisite-patch-id: 75f8c758560362583bb69f21b65b49994e1d66e3
prerequisite-patch-id: 12cf2dad201e25272180a7e53d4f8481bdb08301
prerequisite-patch-id: d5155fad3e1418089bb22688df12680d8022d31f
prerequisite-patch-id: 7630ea4a5cedc6adf939c669489806a353af655e
prerequisite-patch-id: 32b446bd6dbe3fa4472b457d6c029fb4376150a6
prerequisite-patch-id: 14d4434a1e26c24f61a20824f147b04822d9ac1b
prerequisite-patch-id: 17565924a57ad42db80ab683d3ccd494ecb8186d
prerequisite-patch-id: 068d0b6f63964d45aa2a0bf5b2cae3d1c354d29e
prerequisite-patch-id: 4c755e71ad134f77cf6fd2a5743c102f6e6c04a6
prerequisite-patch-id: d6178d19892c900f5388ce55e0aeda22eab00c79
prerequisite-patch-id: c37d52c888c203fccb08e988893b48212c705d50
prerequisite-patch-id: d0fa642e5eda3a6de2f3f72bf35bc7676beed963
prerequisite-patch-id: 74ec9251231e15e63d22ade8c32cf6e9c02e863d
prerequisite-patch-id: 06bab0f0d2ed6eeef2e7801e134e324d6d54c5e3
prerequisite-patch-id: efdaa762203d7ca329c4a64db2609de5d3f0d37c
prerequisite-patch-id: 477d4405e109859bf72c54ef4ff6a5e176889eaa
prerequisite-patch-id: b126790065c81d4adb34fa1e6c9281b4db5ea07d
prerequisite-patch-id: 863520b4c6f94ee824b39c60b7298485b72f6d57
prerequisite-patch-id: 6478e0391435b494760ec557aaa4f2e30ea4ef34
prerequisite-patch-id: 25b42c970d9cbe5192bb8e8618debbf70aaf14ac
prerequisite-patch-id: 398ea1dca59d4953e3cab9477bb9ec884fd3ebd7
prerequisite-patch-id: 5c0236ba7e9f4d9263944a36a624d2008339339d
prerequisite-patch-id: cfa97a9b90ee31f14f1c11482125ddf382294949
prerequisite-patch-id: 8843b7292a2f48fe835188e1772d76470570d1ed
prerequisite-patch-id: 6242ad59ad854dde8c8dc69c1f4bc76847476762
prerequisite-patch-id: 2d4096321dc39ddb09592f30b69180ebf995101f
prerequisite-patch-id: 61982bb691cfe36a79c1be56eeeb503b5109d772
prerequisite-patch-id: 91a3bce85090b362190bd6140616ef75ddc4f8bc
prerequisite-patch-id: e0f2073b8a857f2730db86f09427c02bd032692e
prerequisite-patch-id: 7811cd31eccb9cc610a744231515b67ba271d33d
prerequisite-patch-id: cfff10eede96265d69ff1490ac7a16ab1f58abd8
prerequisite-patch-id: 15a921ffef8186b36b1bc9892fdcc2c23c58ab22
prerequisite-patch-id: 1c633a8a03923ebd3cb443472acee446df150017
prerequisite-patch-id: 4d398224253cb465a8e76e73607ab52d31b983ce
prerequisite-patch-id: 75de8b33aa5883b6f3cd5e97ed75a831974427df
prerequisite-patch-id: 8459349674b56d3add36df2f7af9f6db0fb1a350
prerequisite-patch-id: 1d17792aabed07b98dd84f0c5174fac27e10d522
prerequisite-patch-id: 603e021da94bb3396d63d9323cf4ee75c3e3529d
prerequisite-patch-id: c5136970f28a0e750cf989ce528a8632352a2c02
prerequisite-patch-id: c62a2df67dbc93c51f686b5ca8bfaf748802873c
prerequisite-patch-id: 27d76b2ac5e202eeb0d77b963a0111c55be7c542
prerequisite-patch-id: a61cb2e02ae351f044d61406c87ae4e45c049a2d
prerequisite-patch-id: e7944b362ca3924f15d484e987894dc26b9fcd16
prerequisite-patch-id: 225f1ded6ac55011457ace85eb9a0c5fb59760b7
prerequisite-patch-id: 0a099aadf367898a05990b6dc90f43bdb45e4353
prerequisite-patch-id: 5338bafc69d1ce2ceba53ead881862f5e5ceed12
prerequisite-patch-id: 753aa06e997ac9f07780ebf5cf3cef2a0f6ad8d4
prerequisite-patch-id: 647a84e6f3752159a3f219ef2acc0f47c4f89cbf
prerequisite-patch-id: 5c992979148cc8dd7ab6f39c688ede135aec246c
prerequisite-patch-id: 9b2432198d768c221e4995719753db1a96b282de
prerequisite-patch-id: e2d531670d3c996799fcc65b9b953b9d39e829cf
prerequisite-patch-id: dba1677a5304e1811fba1cb0e76f84c3d0f61b5e
prerequisite-patch-id: ea256490ff4b4f2933d4e1697d85bd574d1eea7a
prerequisite-patch-id: 7288e2f339581863a60bb7621af0bf53c15765a3
prerequisite-patch-id: 88d8b0c75971231a52362d72374e390856c91087
prerequisite-patch-id: 99080f960e5e939b79dcbb3762d7607f684f3223
prerequisite-patch-id: 443050737f9d410dafe4e6d66ad991fb30686ef7
prerequisite-patch-id: 75b00053d6584115ee478df398d206dff2ec0748
prerequisite-patch-id: 8fe187eb73b0c3756d22b740e0900c23b0e49685
prerequisite-patch-id: e9531cef6517857463ba70080480d28037dfc666
prerequisite-patch-id: 8fba074401cf5adddb55037eb4ac3131ef613a25
prerequisite-patch-id: 9dcde58fbe662f5e49c1bc59eef841076cea23a4
prerequisite-patch-id: 9e181e95b8abbf52ebfc3240f3c5677ed52db27e
prerequisite-patch-id: e144eb92024e96ef1f50ed2018e3181695ab1ac1
prerequisite-patch-id: 4bc8a9a6e40c176b54596cd9d76d3ed3e0918fee
prerequisite-patch-id: e86d956e63d866b7f714f27e0fe5c2dba2cda40c
prerequisite-patch-id: b98eb808bf62a850a3c165701c9c56f4a0d7eb1d
prerequisite-patch-id: 7e3beae83c76440cfecf4a229a6cb55cd2202fb0
prerequisite-patch-id: af1b8889603b162427a608893c5f19b04777e529
prerequisite-patch-id: 483ee1290f16e2ffb171b1ecfc5396d8d0b58dbd
prerequisite-patch-id: 4867d2771e267b9ee5fd9cf09466df54ad542565
prerequisite-patch-id: 26091f82ea89fa279f34e125568a5338e862fff1
prerequisite-patch-id: eead951ffdd1738e35dd97731722420a7e6e0197
prerequisite-patch-id: 8abf36d897ca028d5a9d25e04bf419b96ef20281
prerequisite-patch-id: 6f472485f30be3641acf15fab5e75acb95437371
prerequisite-patch-id: 761c632759b166ba0ec7ee3c51de846161d3ac0e
prerequisite-patch-id: 9b484611118e5b275c4b616ba18c7066614215ba
prerequisite-patch-id: c1e17c4fb70ea58840d2ab252c9cb54f43c4dfb0
prerequisite-patch-id: bfb19c706efa1f55d43781d7ef9987f8dfae1dc6
prerequisite-patch-id: deccfc8454cd071120492b40312d97c64028999d
prerequisite-patch-id: 7bfce196190c902064423360f73a74f978cdc153
prerequisite-patch-id: 215dd06efbc0d5700fc7a98b025b840a8268aa6d
prerequisite-patch-id: e858f5fc00fbe3ccf668c687dee2061c51a78432
prerequisite-patch-id: 85f224eb320f7fb107135b554993bc831047fa04
prerequisite-patch-id: c36ddc96e32b5f79af15df34b57ee7d8b4e32f24
prerequisite-patch-id: d387c6483b73d881c0d26cf5c64dead09f1afb7b
prerequisite-patch-id: b9b6c237825d78e3a860e608c021f910a756eb24
prerequisite-patch-id: 72a8be1dd38fe430d95c4277a3802ca8680554ca
prerequisite-patch-id: d7c9064734d9f90bae3ce8def4cbdd0564680a1a
prerequisite-patch-id: 2b7bdca9047d3e0ee2855905d6546b51a6a540c0
prerequisite-patch-id: 42cb03da4a1e374409cadd3277267cc471067e38
prerequisite-patch-id: a439f60634d74c2179712ca0e314facfa42fc0cf
prerequisite-patch-id: 2d6703c8330e3c5499398b3436e1c514b0c170ac
prerequisite-patch-id: 1b48e9eaee821261f3c926ad6862035f9988c161
prerequisite-patch-id: fa95486596301696ac5b042dda727e4803da4276
prerequisite-patch-id: 84128e305e9eed9c6b20dc983a7c794d20761210
prerequisite-patch-id: 66636e2ef226604d83077f3c6137142dd0fbe342
prerequisite-patch-id: 77015d5b571df095df24591ddd9ae1699d0d85eb
prerequisite-patch-id: 314bc0473ac43066f891cbfbf017a3c02bd6df80
prerequisite-patch-id: ca5ea54d426b2dfc91cc7d518f67049c0be40591
prerequisite-patch-id: 8037522480fec73cf06957a947095cedd386f62d
prerequisite-patch-id: 95817a13cfcca32ceae2372f737cc513d864db41
prerequisite-patch-id: 6f126035b087494f9ad1c200596b5b1b7cf21ee7
prerequisite-patch-id: da17979088e3bffc620e25dfef9ba042cd07c6b6
prerequisite-patch-id: 04f6291f75498217363dc4236f2edb0e0da1422b
prerequisite-patch-id: de36d82364ea2d98b80176eeb485657fa4e393b2
prerequisite-patch-id: de4a1010e6812616d397cce1a38737ac6e713c24
prerequisite-patch-id: 38157db92df65354c00325ca44410402e4092ee8
prerequisite-patch-id: 5f82f5da18887d17d3f102d0c1b641ecb798797b
prerequisite-patch-id: eaa92e037894a2714e027817117d18a15ec05852
prerequisite-patch-id: 03cfcfac835dd7977db94968973b1bff5ff669c5
prerequisite-patch-id: d06c810a891665cfaa9db769e2ef19f0d05a1fb3
prerequisite-patch-id: 68c65728eae1dce24053d5b866b5e37d636a7c24
prerequisite-patch-id: eb38334e38ddc18a116e1de03f7ed0e030cc086c
prerequisite-patch-id: b69a94a6026d4d3d94f3a447e94c71809908db67
prerequisite-patch-id: 951841a2e001c2c9a8e4c0b4713360457e325a9c
prerequisite-patch-id: 8075c8994f284a8608b1f0859755a1b8ffca8153
prerequisite-patch-id: d9f81256d272490df1ad829782fd2365dd20de7b
prerequisite-patch-id: 35b25f335c9e3e686356c63afcbe308cb31c1062
prerequisite-patch-id: 33fe5074755b6a819be186fa0c4a2c1ac6dc000c
prerequisite-patch-id: 184807e93c7f62fb37b32cc2af4f3ff5bbf6ac59
prerequisite-patch-id: bb6e202dc83dad6f3559c7b92e28b6b9c1b6a6fa
prerequisite-patch-id: 12bafa214ed2e7c9379ed84a40d6676293356849
prerequisite-patch-id: 0a65972e3977284d46188086b7f8e27d43306c4e
prerequisite-patch-id: 8877c9107b104c12e01db8c9b0d493c137d316ef
prerequisite-patch-id: ad17de915437c124c7fda10b750d5807d9ed1c8e
prerequisite-patch-id: e2f896af2e390a571a697776cdb3003c2db76a89
prerequisite-patch-id: a29eedd61de04f0a2aede926844f380d1ff6f79f
prerequisite-patch-id: d6bc8b3a26d70f4d2c88dae10be2785b840de4ec
prerequisite-patch-id: 580e27c186ebb36974da4a6d2ed96c54757137db
prerequisite-patch-id: 590954e09fa5fcf70cad3ad5842d3c25dab32761
prerequisite-patch-id: a6759838cb7a51c42bb14f357bdfb1fad6b767da
prerequisite-patch-id: 495c9001d25be9c1498552246f8fe7330803508a
prerequisite-patch-id: 3315fe58739db20b8bd4ee86d152b64686be4ea7
prerequisite-patch-id: 05eb01aeedd57d40b0a9ac6ad9d5fa0d64e36907
prerequisite-patch-id: fb0bd246455ced3e7e67fb31c036e9e850ad1b91
prerequisite-patch-id: 0229f6f684eea73cd60687e0b23b29dece8d2159
prerequisite-patch-id: eec7ab24b8bcdbd80e861e16452826f8b6043357
prerequisite-patch-id: ea859f13505822f7dcec69c04daa6b4bc10c3aba
prerequisite-patch-id: 8b59ec1d3b731adc1cba1c1071cae62784b20299
prerequisite-patch-id: 989d19717641e8c9b8a957dbda9cd1c67c2ae6e6
prerequisite-patch-id: 1eb489cbc0ef586f024d0925d68a5f5542b6142d
prerequisite-patch-id: eaf4187e6efdb486b91a2dcd6365454e48c4092a
prerequisite-patch-id: c079d7a0f3142e0a45ef9a726e9bfb63d2eee6d9
prerequisite-patch-id: 2c0039415add384a3809d83bdd7f1304344986dc
prerequisite-patch-id: 8f93732dd38c8f72bd1a6c459c2673114a49ecd1
prerequisite-patch-id: 0e554a72020d16d4f0f8f713377264bb9a4ba864
prerequisite-patch-id: e03e93492c22ca39321a8bd8b0b592868dbb1f9c
prerequisite-patch-id: 283e5d9773252832554e8c369022bf5d85d2cb66
prerequisite-patch-id: 83ae034672b242f88cee40c28dac69e0b5490d57
prerequisite-patch-id: 47af0933bad38dbe03fdd532180965b45a7fd005
prerequisite-patch-id: e4fb69da72ba5fc42aed60540e89fe6851e8e597
prerequisite-patch-id: b82050b0cbe129b9a91b407925ddaecb7983e50b
prerequisite-patch-id: e437fd86238d3c042dca4b0ab89201af6cc37d55
prerequisite-patch-id: f7a7e00fd75e5fe460f9bcad05fa4ce9452bad69
prerequisite-patch-id: 9d748bcba3ba3aba0836cb8956883d9996682b2a
prerequisite-patch-id: 4946ed5bf3398227b7362d316507bcec69143603
prerequisite-patch-id: 8e01fca2e9ca4db9c9da44e277ba0d1bb49fb06e
prerequisite-patch-id: 26978603b24ef02d2b050498ab711fb701efd8f2
prerequisite-patch-id: 91a17ff46b6e39a05567307d8a2e088585d5c5d4
prerequisite-patch-id: 0064859b4251406e5cfc808446cdefda8ab1229a
prerequisite-patch-id: 949f478edcdaa54e659759c9d7ca28df6ec96473
prerequisite-patch-id: b070529ca5dd9077f345001c24a41997de9cc1d8
prerequisite-patch-id: c2aedaa4cddc19a277e8a58387178616dd50cfa9
prerequisite-patch-id: d8ebe315e1e2a28f7b5a59274f0c8cfad3b58709
prerequisite-patch-id: 68720ef6a048d57ff024536ce55e39000831f56f
prerequisite-patch-id: 5a86814f227a75633f57cbd85427d81657a95415
prerequisite-patch-id: 35a7b7202ba6a7286874ec899485a1cf9eab5dea
prerequisite-patch-id: 6d292e9ea15b4e5713012e026e07444cb03817e7
prerequisite-patch-id: 734a40ab3cd8d906657279a4b84d768258677b32
prerequisite-patch-id: ba4b3c87691c9fc2b2ba9cdd5ee8d5e9b3395877
prerequisite-patch-id: 664d4c437ad91d48f79c72abd01b092641fe5051
prerequisite-patch-id: 57aa07b5a8f28666e091f46345792fedd5cff4ae
prerequisite-patch-id: c0b906886a830f76bc2933800dd84a1682a6780c
prerequisite-patch-id: 1e81e3f65f9256bfb8d584b22f15eea391a93135
prerequisite-patch-id: bc254e2dca43f77eb7e0dd14e04e53b0fe556ec8
prerequisite-patch-id: ec9b10fc0a84314ec28b3ee04041f05f2c6fbbf1
prerequisite-patch-id: 561b6d96fc9a16cfc4883525407ccf9f52767821
prerequisite-patch-id: a53fe52087d856b95c2744956649fe9327a75282
prerequisite-patch-id: 42bc72ce46ea2071d47cfc6f9323da9fd7ea44f3
prerequisite-patch-id: ef180abda60223444de4ad0c8cbc7e4c12fcc653
prerequisite-patch-id: 6d93bca7e00544f896614734239201b114619579
prerequisite-patch-id: 8214033f92fe970731589135b539adb405a417c0
--
2.20.1

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* [PATCH v13 1/9] nvme-core: Clear any SGL flags in passthru commands
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe

The host driver should decide whether to use SGLs or PRPs and they
currently assume the flags are cleared after the call to
nvme_setup_cmd(). However, passed-through commands may erroneously
set these bits; so clear them for all cases.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index f3c037f5a9ba..d22859543e4b 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -756,6 +756,8 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
 	case REQ_OP_DRV_IN:
 	case REQ_OP_DRV_OUT:
 		memcpy(cmd, nvme_req(req)->cmd, sizeof(*cmd));
+		/* passthru commands should let the driver set the SGL flags */
+		cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 		break;
 	case REQ_OP_FLUSH:
 		nvme_setup_flush(ns, cmd);
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 1/9] nvme-core: Clear any SGL flags in passthru commands
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

The host driver should decide whether to use SGLs or PRPs and they
currently assume the flags are cleared after the call to
nvme_setup_cmd(). However, passed-through commands may erroneously
set these bits; so clear them for all cases.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index f3c037f5a9ba..d22859543e4b 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -756,6 +756,8 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
 	case REQ_OP_DRV_IN:
 	case REQ_OP_DRV_OUT:
 		memcpy(cmd, nvme_req(req)->cmd, sizeof(*cmd));
+		/* passthru commands should let the driver set the SGL flags */
+		cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 		break;
 	case REQ_OP_FLUSH:
 		nvme_setup_flush(ns, cmd);
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 2/9] nvme: Create helper function to obtain command effects
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe

Separate the code to obtain command effects from the code
to start a passthru request and open code nvme_known_admin_effects()
in the new helper.

The new helper function will be necessary for nvmet passthru
code to determine if we need to change out of interrupt context
to handle the effects.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 39 ++++++++++++++++++++++-----------------
 1 file changed, 22 insertions(+), 17 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d22859543e4b..5062a83c3634 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1317,22 +1317,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
 }
 
-static u32 nvme_known_admin_effects(u8 opcode)
-{
-	switch (opcode) {
-	case nvme_admin_format_nvm:
-		return NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
-					NVME_CMD_EFFECTS_CSE_MASK;
-	case nvme_admin_sanitize_nvm:
-		return NVME_CMD_EFFECTS_CSE_MASK;
-	default:
-		break;
-	}
-	return 0;
-}
-
-static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-								u8 opcode)
+static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+				u8 opcode)
 {
 	u32 effects = 0;
 
@@ -1348,7 +1334,26 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 
 	if (ctrl->effects)
 		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
-	effects |= nvme_known_admin_effects(opcode);
+
+	switch (opcode) {
+	case nvme_admin_format_nvm:
+		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
+			NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	case nvme_admin_sanitize_nvm:
+		effects |= NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	default:
+		break;
+	}
+
+	return effects;
+}
+
+static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			       u8 opcode)
+{
+	u32 effects = nvme_command_effects(ctrl, ns, opcode);
 
 	/*
 	 * For simplicity, IO to all namespaces is quiesced even if the command
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 2/9] nvme: Create helper function to obtain command effects
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

Separate the code to obtain command effects from the code
to start a passthru request and open code nvme_known_admin_effects()
in the new helper.

The new helper function will be necessary for nvmet passthru
code to determine if we need to change out of interrupt context
to handle the effects.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 39 ++++++++++++++++++++++-----------------
 1 file changed, 22 insertions(+), 17 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d22859543e4b..5062a83c3634 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1317,22 +1317,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
 }
 
-static u32 nvme_known_admin_effects(u8 opcode)
-{
-	switch (opcode) {
-	case nvme_admin_format_nvm:
-		return NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
-					NVME_CMD_EFFECTS_CSE_MASK;
-	case nvme_admin_sanitize_nvm:
-		return NVME_CMD_EFFECTS_CSE_MASK;
-	default:
-		break;
-	}
-	return 0;
-}
-
-static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-								u8 opcode)
+static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+				u8 opcode)
 {
 	u32 effects = 0;
 
@@ -1348,7 +1334,26 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 
 	if (ctrl->effects)
 		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
-	effects |= nvme_known_admin_effects(opcode);
+
+	switch (opcode) {
+	case nvme_admin_format_nvm:
+		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
+			NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	case nvme_admin_sanitize_nvm:
+		effects |= NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	default:
+		break;
+	}
+
+	return effects;
+}
+
+static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			       u8 opcode)
+{
+	u32 effects = nvme_command_effects(ctrl, ns, opcode);
 
 	/*
 	 * For simplicity, IO to all namespaces is quiesced even if the command
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 3/9] nvme: Move nvme_passthru_[start|end]() calls to common helper
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe

Introduce a new nvme_execute_passthru_rq() helper which calls
nvme_passthru_[start|end]() around blk_execute_rq(). This ensures
all passthru calls (including nvme_submit_io()) will be wrapped
appropriately.

nvme_execute_passthru_rq() will also be useful for the nvmet passthru
code.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 193 ++++++++++++++++++++-------------------
 1 file changed, 100 insertions(+), 93 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 5062a83c3634..2ead7ad45a9d 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -890,6 +890,105 @@ static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf,
 	return ERR_PTR(ret);
 }
 
+static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+				u8 opcode)
+{
+	u32 effects = 0;
+
+	if (ns) {
+		if (ctrl->effects)
+			effects = le32_to_cpu(ctrl->effects->iocs[opcode]);
+		if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC))
+			dev_warn(ctrl->device,
+				 "IO command:%02x has unhandled effects:%08x\n",
+				 opcode, effects);
+		return 0;
+	}
+
+	if (ctrl->effects)
+		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
+
+	switch (opcode) {
+	case nvme_admin_format_nvm:
+		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
+			NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	case nvme_admin_sanitize_nvm:
+		effects |= NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	default:
+		break;
+	}
+
+	return effects & ~NVME_CMD_EFFECTS_CSUPP;
+}
+
+static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			       u8 opcode)
+{
+	u32 effects = nvme_command_effects(ctrl, ns, opcode);
+
+	/*
+	 * For simplicity, IO to all namespaces is quiesced even if the command
+	 * effects say only one namespace is affected.
+	 */
+	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
+		mutex_lock(&ctrl->scan_lock);
+		mutex_lock(&ctrl->subsys->lock);
+		nvme_mpath_start_freeze(ctrl->subsys);
+		nvme_mpath_wait_freeze(ctrl->subsys);
+		nvme_start_freeze(ctrl);
+		nvme_wait_freeze(ctrl);
+	}
+	return effects;
+}
+
+static void nvme_update_formats(struct nvme_ctrl *ctrl)
+{
+	struct nvme_ns *ns;
+
+	down_read(&ctrl->namespaces_rwsem);
+	list_for_each_entry(ns, &ctrl->namespaces, list)
+		if (ns->disk && nvme_revalidate_disk(ns->disk))
+			nvme_set_queue_dying(ns);
+	up_read(&ctrl->namespaces_rwsem);
+}
+
+static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
+{
+	/*
+	 * Revalidate LBA changes prior to unfreezing. This is necessary to
+	 * prevent memory corruption if a logical block size was changed by
+	 * this command.
+	 */
+	if (effects & NVME_CMD_EFFECTS_LBCC)
+		nvme_update_formats(ctrl);
+	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
+		nvme_unfreeze(ctrl);
+		nvme_mpath_unfreeze(ctrl->subsys);
+		mutex_unlock(&ctrl->subsys->lock);
+		nvme_remove_invalid_namespaces(ctrl, NVME_NSID_ALL);
+		mutex_unlock(&ctrl->scan_lock);
+	}
+	if (effects & NVME_CMD_EFFECTS_CCC)
+		nvme_init_identify(ctrl);
+	if (effects & (NVME_CMD_EFFECTS_NIC | NVME_CMD_EFFECTS_NCC))
+		nvme_queue_scan(ctrl);
+}
+
+static void nvme_execute_passthru_rq(struct request *rq)
+{
+	struct nvme_command *cmd = nvme_req(rq)->cmd;
+	struct nvme_ctrl *ctrl = nvme_req(rq)->ctrl;
+	struct nvme_ns *ns = rq->q->queuedata;
+	struct gendisk *disk = ns ? ns->disk : NULL;
+	u32 effects;
+
+	effects = nvme_passthru_start(ctrl, ns, cmd->common.opcode);
+	blk_execute_rq(rq->q, disk, rq, 0);
+	nvme_passthru_end(ctrl, effects);
+}
+
 static int nvme_submit_user_cmd(struct request_queue *q,
 		struct nvme_command *cmd, void __user *ubuffer,
 		unsigned bufflen, void __user *meta_buffer, unsigned meta_len,
@@ -928,7 +1027,7 @@ static int nvme_submit_user_cmd(struct request_queue *q,
 		}
 	}
 
-	blk_execute_rq(req->q, disk, req, 0);
+	nvme_execute_passthru_rq(req);
 	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
 		ret = -EINTR;
 	else
@@ -1317,99 +1416,12 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
 }
 
-static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-				u8 opcode)
-{
-	u32 effects = 0;
-
-	if (ns) {
-		if (ctrl->effects)
-			effects = le32_to_cpu(ctrl->effects->iocs[opcode]);
-		if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC))
-			dev_warn(ctrl->device,
-				 "IO command:%02x has unhandled effects:%08x\n",
-				 opcode, effects);
-		return 0;
-	}
-
-	if (ctrl->effects)
-		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
-
-	switch (opcode) {
-	case nvme_admin_format_nvm:
-		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
-			NVME_CMD_EFFECTS_CSE_MASK;
-		break;
-	case nvme_admin_sanitize_nvm:
-		effects |= NVME_CMD_EFFECTS_CSE_MASK;
-		break;
-	default:
-		break;
-	}
-
-	return effects;
-}
-
-static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-			       u8 opcode)
-{
-	u32 effects = nvme_command_effects(ctrl, ns, opcode);
-
-	/*
-	 * For simplicity, IO to all namespaces is quiesced even if the command
-	 * effects say only one namespace is affected.
-	 */
-	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
-		mutex_lock(&ctrl->scan_lock);
-		mutex_lock(&ctrl->subsys->lock);
-		nvme_mpath_start_freeze(ctrl->subsys);
-		nvme_mpath_wait_freeze(ctrl->subsys);
-		nvme_start_freeze(ctrl);
-		nvme_wait_freeze(ctrl);
-	}
-	return effects;
-}
-
-static void nvme_update_formats(struct nvme_ctrl *ctrl)
-{
-	struct nvme_ns *ns;
-
-	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
-		if (ns->disk && nvme_revalidate_disk(ns->disk))
-			nvme_set_queue_dying(ns);
-	up_read(&ctrl->namespaces_rwsem);
-}
-
-static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
-{
-	/*
-	 * Revalidate LBA changes prior to unfreezing. This is necessary to
-	 * prevent memory corruption if a logical block size was changed by
-	 * this command.
-	 */
-	if (effects & NVME_CMD_EFFECTS_LBCC)
-		nvme_update_formats(ctrl);
-	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
-		nvme_unfreeze(ctrl);
-		nvme_mpath_unfreeze(ctrl->subsys);
-		mutex_unlock(&ctrl->subsys->lock);
-		nvme_remove_invalid_namespaces(ctrl, NVME_NSID_ALL);
-		mutex_unlock(&ctrl->scan_lock);
-	}
-	if (effects & NVME_CMD_EFFECTS_CCC)
-		nvme_init_identify(ctrl);
-	if (effects & (NVME_CMD_EFFECTS_NIC | NVME_CMD_EFFECTS_NCC))
-		nvme_queue_scan(ctrl);
-}
-
 static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			struct nvme_passthru_cmd __user *ucmd)
 {
 	struct nvme_passthru_cmd cmd;
 	struct nvme_command c;
 	unsigned timeout = 0;
-	u32 effects;
 	u64 result;
 	int status;
 
@@ -1436,12 +1448,10 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.timeout_ms)
 		timeout = msecs_to_jiffies(cmd.timeout_ms);
 
-	effects = nvme_passthru_start(ctrl, ns, cmd.opcode);
 	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
 			nvme_to_user_ptr(cmd.addr), cmd.data_len,
 			nvme_to_user_ptr(cmd.metadata), cmd.metadata_len,
 			0, &result, timeout);
-	nvme_passthru_end(ctrl, effects);
 
 	if (status >= 0) {
 		if (put_user(result, &ucmd->result))
@@ -1457,7 +1467,6 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	struct nvme_passthru_cmd64 cmd;
 	struct nvme_command c;
 	unsigned timeout = 0;
-	u32 effects;
 	int status;
 
 	if (!capable(CAP_SYS_ADMIN))
@@ -1483,12 +1492,10 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.timeout_ms)
 		timeout = msecs_to_jiffies(cmd.timeout_ms);
 
-	effects = nvme_passthru_start(ctrl, ns, cmd.opcode);
 	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
 			nvme_to_user_ptr(cmd.addr), cmd.data_len,
 			nvme_to_user_ptr(cmd.metadata), cmd.metadata_len,
 			0, &cmd.result, timeout);
-	nvme_passthru_end(ctrl, effects);
 
 	if (status >= 0) {
 		if (put_user(cmd.result, &ucmd->result))
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 3/9] nvme: Move nvme_passthru_[start|end]() calls to common helper
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

Introduce a new nvme_execute_passthru_rq() helper which calls
nvme_passthru_[start|end]() around blk_execute_rq(). This ensures
all passthru calls (including nvme_submit_io()) will be wrapped
appropriately.

nvme_execute_passthru_rq() will also be useful for the nvmet passthru
code.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 193 ++++++++++++++++++++-------------------
 1 file changed, 100 insertions(+), 93 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 5062a83c3634..2ead7ad45a9d 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -890,6 +890,105 @@ static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf,
 	return ERR_PTR(ret);
 }
 
+static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+				u8 opcode)
+{
+	u32 effects = 0;
+
+	if (ns) {
+		if (ctrl->effects)
+			effects = le32_to_cpu(ctrl->effects->iocs[opcode]);
+		if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC))
+			dev_warn(ctrl->device,
+				 "IO command:%02x has unhandled effects:%08x\n",
+				 opcode, effects);
+		return 0;
+	}
+
+	if (ctrl->effects)
+		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
+
+	switch (opcode) {
+	case nvme_admin_format_nvm:
+		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
+			NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	case nvme_admin_sanitize_nvm:
+		effects |= NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	default:
+		break;
+	}
+
+	return effects & ~NVME_CMD_EFFECTS_CSUPP;
+}
+
+static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			       u8 opcode)
+{
+	u32 effects = nvme_command_effects(ctrl, ns, opcode);
+
+	/*
+	 * For simplicity, IO to all namespaces is quiesced even if the command
+	 * effects say only one namespace is affected.
+	 */
+	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
+		mutex_lock(&ctrl->scan_lock);
+		mutex_lock(&ctrl->subsys->lock);
+		nvme_mpath_start_freeze(ctrl->subsys);
+		nvme_mpath_wait_freeze(ctrl->subsys);
+		nvme_start_freeze(ctrl);
+		nvme_wait_freeze(ctrl);
+	}
+	return effects;
+}
+
+static void nvme_update_formats(struct nvme_ctrl *ctrl)
+{
+	struct nvme_ns *ns;
+
+	down_read(&ctrl->namespaces_rwsem);
+	list_for_each_entry(ns, &ctrl->namespaces, list)
+		if (ns->disk && nvme_revalidate_disk(ns->disk))
+			nvme_set_queue_dying(ns);
+	up_read(&ctrl->namespaces_rwsem);
+}
+
+static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
+{
+	/*
+	 * Revalidate LBA changes prior to unfreezing. This is necessary to
+	 * prevent memory corruption if a logical block size was changed by
+	 * this command.
+	 */
+	if (effects & NVME_CMD_EFFECTS_LBCC)
+		nvme_update_formats(ctrl);
+	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
+		nvme_unfreeze(ctrl);
+		nvme_mpath_unfreeze(ctrl->subsys);
+		mutex_unlock(&ctrl->subsys->lock);
+		nvme_remove_invalid_namespaces(ctrl, NVME_NSID_ALL);
+		mutex_unlock(&ctrl->scan_lock);
+	}
+	if (effects & NVME_CMD_EFFECTS_CCC)
+		nvme_init_identify(ctrl);
+	if (effects & (NVME_CMD_EFFECTS_NIC | NVME_CMD_EFFECTS_NCC))
+		nvme_queue_scan(ctrl);
+}
+
+static void nvme_execute_passthru_rq(struct request *rq)
+{
+	struct nvme_command *cmd = nvme_req(rq)->cmd;
+	struct nvme_ctrl *ctrl = nvme_req(rq)->ctrl;
+	struct nvme_ns *ns = rq->q->queuedata;
+	struct gendisk *disk = ns ? ns->disk : NULL;
+	u32 effects;
+
+	effects = nvme_passthru_start(ctrl, ns, cmd->common.opcode);
+	blk_execute_rq(rq->q, disk, rq, 0);
+	nvme_passthru_end(ctrl, effects);
+}
+
 static int nvme_submit_user_cmd(struct request_queue *q,
 		struct nvme_command *cmd, void __user *ubuffer,
 		unsigned bufflen, void __user *meta_buffer, unsigned meta_len,
@@ -928,7 +1027,7 @@ static int nvme_submit_user_cmd(struct request_queue *q,
 		}
 	}
 
-	blk_execute_rq(req->q, disk, req, 0);
+	nvme_execute_passthru_rq(req);
 	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
 		ret = -EINTR;
 	else
@@ -1317,99 +1416,12 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
 }
 
-static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-				u8 opcode)
-{
-	u32 effects = 0;
-
-	if (ns) {
-		if (ctrl->effects)
-			effects = le32_to_cpu(ctrl->effects->iocs[opcode]);
-		if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC))
-			dev_warn(ctrl->device,
-				 "IO command:%02x has unhandled effects:%08x\n",
-				 opcode, effects);
-		return 0;
-	}
-
-	if (ctrl->effects)
-		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
-
-	switch (opcode) {
-	case nvme_admin_format_nvm:
-		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
-			NVME_CMD_EFFECTS_CSE_MASK;
-		break;
-	case nvme_admin_sanitize_nvm:
-		effects |= NVME_CMD_EFFECTS_CSE_MASK;
-		break;
-	default:
-		break;
-	}
-
-	return effects;
-}
-
-static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-			       u8 opcode)
-{
-	u32 effects = nvme_command_effects(ctrl, ns, opcode);
-
-	/*
-	 * For simplicity, IO to all namespaces is quiesced even if the command
-	 * effects say only one namespace is affected.
-	 */
-	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
-		mutex_lock(&ctrl->scan_lock);
-		mutex_lock(&ctrl->subsys->lock);
-		nvme_mpath_start_freeze(ctrl->subsys);
-		nvme_mpath_wait_freeze(ctrl->subsys);
-		nvme_start_freeze(ctrl);
-		nvme_wait_freeze(ctrl);
-	}
-	return effects;
-}
-
-static void nvme_update_formats(struct nvme_ctrl *ctrl)
-{
-	struct nvme_ns *ns;
-
-	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
-		if (ns->disk && nvme_revalidate_disk(ns->disk))
-			nvme_set_queue_dying(ns);
-	up_read(&ctrl->namespaces_rwsem);
-}
-
-static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
-{
-	/*
-	 * Revalidate LBA changes prior to unfreezing. This is necessary to
-	 * prevent memory corruption if a logical block size was changed by
-	 * this command.
-	 */
-	if (effects & NVME_CMD_EFFECTS_LBCC)
-		nvme_update_formats(ctrl);
-	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
-		nvme_unfreeze(ctrl);
-		nvme_mpath_unfreeze(ctrl->subsys);
-		mutex_unlock(&ctrl->subsys->lock);
-		nvme_remove_invalid_namespaces(ctrl, NVME_NSID_ALL);
-		mutex_unlock(&ctrl->scan_lock);
-	}
-	if (effects & NVME_CMD_EFFECTS_CCC)
-		nvme_init_identify(ctrl);
-	if (effects & (NVME_CMD_EFFECTS_NIC | NVME_CMD_EFFECTS_NCC))
-		nvme_queue_scan(ctrl);
-}
-
 static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			struct nvme_passthru_cmd __user *ucmd)
 {
 	struct nvme_passthru_cmd cmd;
 	struct nvme_command c;
 	unsigned timeout = 0;
-	u32 effects;
 	u64 result;
 	int status;
 
@@ -1436,12 +1448,10 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.timeout_ms)
 		timeout = msecs_to_jiffies(cmd.timeout_ms);
 
-	effects = nvme_passthru_start(ctrl, ns, cmd.opcode);
 	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
 			nvme_to_user_ptr(cmd.addr), cmd.data_len,
 			nvme_to_user_ptr(cmd.metadata), cmd.metadata_len,
 			0, &result, timeout);
-	nvme_passthru_end(ctrl, effects);
 
 	if (status >= 0) {
 		if (put_user(result, &ucmd->result))
@@ -1457,7 +1467,6 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	struct nvme_passthru_cmd64 cmd;
 	struct nvme_command c;
 	unsigned timeout = 0;
-	u32 effects;
 	int status;
 
 	if (!capable(CAP_SYS_ADMIN))
@@ -1483,12 +1492,10 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.timeout_ms)
 		timeout = msecs_to_jiffies(cmd.timeout_ms);
 
-	effects = nvme_passthru_start(ctrl, ns, cmd.opcode);
 	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
 			nvme_to_user_ptr(cmd.addr), cmd.data_len,
 			nvme_to_user_ptr(cmd.metadata), cmd.metadata_len,
 			0, &cmd.result, timeout);
-	nvme_passthru_end(ctrl, effects);
 
 	if (status >= 0) {
 		if (put_user(cmd.result, &ucmd->result))
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 4/9] nvmet-passthru: Introduce NVMet passthru Kconfig option
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates,
	Chaitanya Kulkarni, Logan Gunthorpe

From: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>

This patch updates KConfig file for the NVMeOF target where we add new
option so that user can selectively enable/disable passthru code.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
[logang@deltatee.com: fixed some of the wording in the help message]
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/Kconfig | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/nvme/target/Kconfig b/drivers/nvme/target/Kconfig
index d7f48c0fb311..1ef775fa0048 100644
--- a/drivers/nvme/target/Kconfig
+++ b/drivers/nvme/target/Kconfig
@@ -15,6 +15,18 @@ config NVME_TARGET
 	  To configure the NVMe target you probably want to use the nvmetcli
 	  tool from http://git.infradead.org/users/hch/nvmetcli.git.
 
+config NVME_TARGET_PASSTHRU
+	bool "NVMe Target Passthrough support"
+	depends on NVME_CORE
+	depends on NVME_TARGET
+	help
+	  This enables target side NVMe passthru controller support for the
+	  NVMe Over Fabrics protocol. It allows for hosts to manage and
+	  directly access an actual NVMe controller residing on the target
+	  side, incuding executing Vendor Unique Commands.
+
+	  If unsure, say N.
+
 config NVME_TARGET_LOOP
 	tristate "NVMe loopback device support"
 	depends on NVME_TARGET
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 4/9] nvmet-passthru: Introduce NVMet passthru Kconfig option
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

From: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>

This patch updates KConfig file for the NVMeOF target where we add new
option so that user can selectively enable/disable passthru code.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
[logang@deltatee.com: fixed some of the wording in the help message]
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/Kconfig | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/nvme/target/Kconfig b/drivers/nvme/target/Kconfig
index d7f48c0fb311..1ef775fa0048 100644
--- a/drivers/nvme/target/Kconfig
+++ b/drivers/nvme/target/Kconfig
@@ -15,6 +15,18 @@ config NVME_TARGET
 	  To configure the NVMe target you probably want to use the nvmetcli
 	  tool from http://git.infradead.org/users/hch/nvmetcli.git.
 
+config NVME_TARGET_PASSTHRU
+	bool "NVMe Target Passthrough support"
+	depends on NVME_CORE
+	depends on NVME_TARGET
+	help
+	  This enables target side NVMe passthru controller support for the
+	  NVMe Over Fabrics protocol. It allows for hosts to manage and
+	  directly access an actual NVMe controller residing on the target
+	  side, incuding executing Vendor Unique Commands.
+
+	  If unsure, say N.
+
 config NVME_TARGET_LOOP
 	tristate "NVMe loopback device support"
 	depends on NVME_TARGET
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 5/9] nvme-core: Introduce nvme_ctrl_get_by_path()
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe

nvme_ctrl_get_by_path() is analagous to blkdev_get_by_path() except it
gets a struct nvme_ctrl from the path to its char dev (/dev/nvme0).
It makes use of filp_open() to open the file and uses the private
data to obtain a pointer to the struct nvme_ctrl. If the fops of the
file do not match, -EINVAL is returned.

The purpose of this function is to support NVMe-OF target passthru.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Max Gurtovoy <maxg@mellanox.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 31 +++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h |  9 +++++++++
 2 files changed, 40 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 2ead7ad45a9d..2604971362d8 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -4305,6 +4305,37 @@ void nvme_sync_queues(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_sync_queues);
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+/*
+ * The exports that follow within this ifdef are only for
+ * use by the nvmet-passthru and should not be used for
+ * other things.
+ */
+
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
+{
+	struct nvme_ctrl *ctrl;
+	struct file *f;
+
+	f = filp_open(path, O_RDWR, 0);
+	if (IS_ERR(f))
+		return ERR_CAST(f);
+
+	if (f->f_op != &nvme_dev_fops) {
+		ctrl = ERR_PTR(-EINVAL);
+		goto out_close;
+	}
+
+	ctrl = f->private_data;
+	nvme_get_ctrl(ctrl);
+
+out_close:
+	filp_close(f, NULL);
+	return ctrl;
+}
+EXPORT_SYMBOL_GPL(nvme_ctrl_get_by_path);
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 /*
  * Check we didn't inadvertently grow the command structure sizes:
  */
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 2e04a36296d9..9195dd97b61b 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -689,4 +689,13 @@ void nvme_hwmon_init(struct nvme_ctrl *ctrl);
 static inline void nvme_hwmon_init(struct nvme_ctrl *ctrl) { }
 #endif
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+/*
+ * The exports that follow within this ifdef are only for
+ * use by the nvmet-passthru and should not be used for
+ * other things.
+ */
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path);
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 #endif /* _NVME_H */
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 5/9] nvme-core: Introduce nvme_ctrl_get_by_path()
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

nvme_ctrl_get_by_path() is analagous to blkdev_get_by_path() except it
gets a struct nvme_ctrl from the path to its char dev (/dev/nvme0).
It makes use of filp_open() to open the file and uses the private
data to obtain a pointer to the struct nvme_ctrl. If the fops of the
file do not match, -EINVAL is returned.

The purpose of this function is to support NVMe-OF target passthru.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Max Gurtovoy <maxg@mellanox.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 31 +++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h |  9 +++++++++
 2 files changed, 40 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 2ead7ad45a9d..2604971362d8 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -4305,6 +4305,37 @@ void nvme_sync_queues(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_sync_queues);
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+/*
+ * The exports that follow within this ifdef are only for
+ * use by the nvmet-passthru and should not be used for
+ * other things.
+ */
+
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
+{
+	struct nvme_ctrl *ctrl;
+	struct file *f;
+
+	f = filp_open(path, O_RDWR, 0);
+	if (IS_ERR(f))
+		return ERR_CAST(f);
+
+	if (f->f_op != &nvme_dev_fops) {
+		ctrl = ERR_PTR(-EINVAL);
+		goto out_close;
+	}
+
+	ctrl = f->private_data;
+	nvme_get_ctrl(ctrl);
+
+out_close:
+	filp_close(f, NULL);
+	return ctrl;
+}
+EXPORT_SYMBOL_GPL(nvme_ctrl_get_by_path);
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 /*
  * Check we didn't inadvertently grow the command structure sizes:
  */
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 2e04a36296d9..9195dd97b61b 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -689,4 +689,13 @@ void nvme_hwmon_init(struct nvme_ctrl *ctrl);
 static inline void nvme_hwmon_init(struct nvme_ctrl *ctrl) { }
 #endif
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+/*
+ * The exports that follow within this ifdef are only for
+ * use by the nvmet-passthru and should not be used for
+ * other things.
+ */
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path);
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 #endif /* _NVME_H */
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 6/9] nvme: Export existing nvme core functions
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe,
	Chaitanya Kulkarni

Export nvme_put_ns(), nvme_command_effects(), nvme_execute_passthru_rq()
and nvme_find_get_ns() for use in the nvmet passthru code.

The exports are conditional on CONFIG_NVME_TARGET_PASSTHRU.

Based-on-a-patch-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Max Gurtovoy <maxg@mellanox.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 14 +++++++++-----
 drivers/nvme/host/nvme.h |  5 +++++
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 2604971362d8..c76f2e4b7c1c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -457,7 +457,7 @@ static void nvme_free_ns(struct kref *kref)
 	kfree(ns);
 }
 
-static void nvme_put_ns(struct nvme_ns *ns)
+void nvme_put_ns(struct nvme_ns *ns)
 {
 	kref_put(&ns->kref, nvme_free_ns);
 }
@@ -890,8 +890,8 @@ static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf,
 	return ERR_PTR(ret);
 }
 
-static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-				u8 opcode)
+u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			 u8 opcode)
 {
 	u32 effects = 0;
 
@@ -976,7 +976,7 @@ static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
 		nvme_queue_scan(ctrl);
 }
 
-static void nvme_execute_passthru_rq(struct request *rq)
+void nvme_execute_passthru_rq(struct request *rq)
 {
 	struct nvme_command *cmd = nvme_req(rq)->cmd;
 	struct nvme_ctrl *ctrl = nvme_req(rq)->ctrl;
@@ -3530,7 +3530,7 @@ static int ns_cmp(void *priv, struct list_head *a, struct list_head *b)
 	return nsa->head->ns_id - nsb->head->ns_id;
 }
 
-static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
+struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 {
 	struct nvme_ns *ns, *ret = NULL;
 
@@ -4311,6 +4311,10 @@ EXPORT_SYMBOL_GPL(nvme_sync_queues);
  * use by the nvmet-passthru and should not be used for
  * other things.
  */
+EXPORT_SYMBOL_GPL(nvme_put_ns);
+EXPORT_SYMBOL_GPL(nvme_command_effects);
+EXPORT_SYMBOL_GPL(nvme_execute_passthru_rq);
+EXPORT_SYMBOL_GPL(nvme_find_get_ns);
 
 struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
 {
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 9195dd97b61b..76b4aa8f35e1 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -695,6 +695,11 @@ static inline void nvme_hwmon_init(struct nvme_ctrl *ctrl) { }
  * use by the nvmet-passthru and should not be used for
  * other things.
  */
+void nvme_put_ns(struct nvme_ns *ns);
+u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			 u8 opcode);
+void nvme_execute_passthru_rq(struct request *rq);
+struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned int nsid);
 struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path);
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 6/9] nvme: Export existing nvme core functions
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

Export nvme_put_ns(), nvme_command_effects(), nvme_execute_passthru_rq()
and nvme_find_get_ns() for use in the nvmet passthru code.

The exports are conditional on CONFIG_NVME_TARGET_PASSTHRU.

Based-on-a-patch-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Max Gurtovoy <maxg@mellanox.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/host/core.c | 14 +++++++++-----
 drivers/nvme/host/nvme.h |  5 +++++
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 2604971362d8..c76f2e4b7c1c 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -457,7 +457,7 @@ static void nvme_free_ns(struct kref *kref)
 	kfree(ns);
 }
 
-static void nvme_put_ns(struct nvme_ns *ns)
+void nvme_put_ns(struct nvme_ns *ns)
 {
 	kref_put(&ns->kref, nvme_free_ns);
 }
@@ -890,8 +890,8 @@ static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf,
 	return ERR_PTR(ret);
 }
 
-static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-				u8 opcode)
+u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			 u8 opcode)
 {
 	u32 effects = 0;
 
@@ -976,7 +976,7 @@ static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
 		nvme_queue_scan(ctrl);
 }
 
-static void nvme_execute_passthru_rq(struct request *rq)
+void nvme_execute_passthru_rq(struct request *rq)
 {
 	struct nvme_command *cmd = nvme_req(rq)->cmd;
 	struct nvme_ctrl *ctrl = nvme_req(rq)->ctrl;
@@ -3530,7 +3530,7 @@ static int ns_cmp(void *priv, struct list_head *a, struct list_head *b)
 	return nsa->head->ns_id - nsb->head->ns_id;
 }
 
-static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
+struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 {
 	struct nvme_ns *ns, *ret = NULL;
 
@@ -4311,6 +4311,10 @@ EXPORT_SYMBOL_GPL(nvme_sync_queues);
  * use by the nvmet-passthru and should not be used for
  * other things.
  */
+EXPORT_SYMBOL_GPL(nvme_put_ns);
+EXPORT_SYMBOL_GPL(nvme_command_effects);
+EXPORT_SYMBOL_GPL(nvme_execute_passthru_rq);
+EXPORT_SYMBOL_GPL(nvme_find_get_ns);
 
 struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
 {
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 9195dd97b61b..76b4aa8f35e1 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -695,6 +695,11 @@ static inline void nvme_hwmon_init(struct nvme_ctrl *ctrl) { }
  * use by the nvmet-passthru and should not be used for
  * other things.
  */
+void nvme_put_ns(struct nvme_ns *ns);
+u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			 u8 opcode);
+void nvme_execute_passthru_rq(struct request *rq);
+struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned int nsid);
 struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path);
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 7/9] nvmet-passthru: Add passthru code to process commands
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe,
	Chaitanya Kulkarni

Add passthru command handling capability for the NVMeOF target and
export passthru APIs which are used to integrate passthru
code with nvmet-core.

The new file passthru.c handles passthru cmd parsing and execution.
In the passthru mode, we create a block layer request from the nvmet
request and map the data on to the block layer request.

Admin commands and features are on a white list as there are a number
of each that don't make too much sense with passthrough. We use a
white list so that new commands can be considered before being blindly
passed through. In both cases, vendor specific commands are always
allowed.

We also blacklist reservation IO commands as the underlying device
cannot differentiate between multiple hosts behind a fabric.

Based-on-a-patch-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/Makefile    |   1 +
 drivers/nvme/target/admin-cmd.c |   7 +-
 drivers/nvme/target/core.c      |   3 +
 drivers/nvme/target/nvmet.h     |  39 +++
 drivers/nvme/target/passthru.c  | 460 ++++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   4 +
 6 files changed, 512 insertions(+), 2 deletions(-)
 create mode 100644 drivers/nvme/target/passthru.c

diff --git a/drivers/nvme/target/Makefile b/drivers/nvme/target/Makefile
index 2b33836f3d3e..ebf91fc4c72e 100644
--- a/drivers/nvme/target/Makefile
+++ b/drivers/nvme/target/Makefile
@@ -11,6 +11,7 @@ obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
 			discovery.o io-cmd-file.o io-cmd-bdev.o
+nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvme-loop-y	+= loop.o
 nvmet-rdma-y	+= rdma.o
 nvmet-fc-y	+= fc.o
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 9d6f75cfa77c..a2dfd202eb32 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -734,7 +734,7 @@ u16 nvmet_set_feat_async_event(struct nvmet_req *req, u32 mask)
 	return 0;
 }
 
-static void nvmet_execute_set_features(struct nvmet_req *req)
+void nvmet_execute_set_features(struct nvmet_req *req)
 {
 	struct nvmet_subsys *subsys = req->sq->ctrl->subsys;
 	u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
@@ -809,7 +809,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req)
 	nvmet_set_result(req, READ_ONCE(req->sq->ctrl->aen_enabled));
 }
 
-static void nvmet_execute_get_features(struct nvmet_req *req)
+void nvmet_execute_get_features(struct nvmet_req *req)
 {
 	struct nvmet_subsys *subsys = req->sq->ctrl->subsys;
 	u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
@@ -925,6 +925,9 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
 	if (unlikely(ret))
 		return ret;
 
+	if (nvmet_req_passthru_ctrl(req))
+		return nvmet_parse_passthru_admin_cmd(req);
+
 	switch (cmd->common.opcode) {
 	case nvme_admin_get_log_page:
 		req->execute = nvmet_execute_get_log_page;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index b685f99d56a1..2547e0d8951c 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -841,6 +841,9 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 	if (unlikely(ret))
 		return ret;
 
+	if (nvmet_req_passthru_ctrl(req))
+		return nvmet_parse_passthru_io_cmd(req);
+
 	req->ns = nvmet_find_namespace(req->sq->ctrl, cmd->rw.nsid);
 	if (unlikely(!req->ns)) {
 		req->error_loc = offsetof(struct nvme_common_command, nsid);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 421dff3ea143..deb996c90804 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -237,6 +237,10 @@ struct nvmet_subsys {
 	struct config_group	allowed_hosts_group;
 
 	struct nvmet_subsys_model	__rcu *model;
+
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+	struct nvme_ctrl	*passthru_ctrl;
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
 static inline struct nvmet_subsys *to_subsys(struct config_item *item)
@@ -313,6 +317,11 @@ struct nvmet_req {
 			struct bio_vec          *bvec;
 			struct work_struct      work;
 		} f;
+		struct {
+			struct request		*rq;
+			struct work_struct      work;
+			u16 (*end_req)(struct nvmet_req *req);
+		} p;
 	};
 	int			sg_cnt;
 	/* data length as parsed from the SGL descriptor: */
@@ -390,6 +399,8 @@ void nvmet_req_complete(struct nvmet_req *req, u16 status);
 int nvmet_req_alloc_sgl(struct nvmet_req *req);
 void nvmet_req_free_sgl(struct nvmet_req *req);
 
+void nvmet_execute_set_features(struct nvmet_req *req);
+void nvmet_execute_get_features(struct nvmet_req *req);
 void nvmet_execute_keep_alive(struct nvmet_req *req);
 
 void nvmet_cq_setup(struct nvmet_ctrl *ctrl, struct nvmet_cq *cq, u16 qid,
@@ -511,6 +522,34 @@ static inline u32 nvmet_dsm_len(struct nvmet_req *req)
 		sizeof(struct nvme_dsm_range);
 }
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req);
+u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req);
+static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
+{
+	return subsys->passthru_ctrl;
+}
+#else /* CONFIG_NVME_TARGET_PASSTHRU */
+static inline u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
+{
+	return 0;
+}
+static inline u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req)
+{
+	return 0;
+}
+static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
+{
+	return NULL;
+}
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
+static inline struct nvme_ctrl *
+nvmet_req_passthru_ctrl(struct nvmet_req *req)
+{
+	return nvmet_passthru_ctrl(req->sq->ctrl->subsys);
+}
+
 u16 errno_to_nvme_status(struct nvmet_req *req, int errno);
 
 /* Convert a 32-bit number to a 16-bit 0's based number */
diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
new file mode 100644
index 000000000000..15b23521e59f
--- /dev/null
+++ b/drivers/nvme/target/passthru.c
@@ -0,0 +1,460 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe Over Fabrics Target Passthrough command implementation.
+ *
+ * Copyright (c) 2017-2018 Western Digital Corporation or its
+ * affiliates.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/module.h>
+
+#include "../host/nvme.h"
+#include "nvmet.h"
+
+static void nvmet_passthru_execute_cmd_work(struct work_struct *w)
+{
+	struct nvmet_req *req = container_of(w, struct nvmet_req, p.work);
+	struct request *rq = req->p.rq;
+	u16 status;
+
+	nvme_execute_passthru_rq(rq);
+
+	status = nvme_req(rq)->status;
+	if (status == NVME_SC_SUCCESS && req->p.end_req)
+		status = req->p.end_req(req);
+
+	req->cqe->result = nvme_req(rq)->result;
+	nvmet_req_complete(req, status);
+	blk_put_request(rq);
+}
+
+static void nvmet_passthru_req_done(struct request *rq,
+				    blk_status_t blk_status)
+{
+	struct nvmet_req *req = rq->end_io_data;
+
+	req->cqe->result = nvme_req(rq)->result;
+	nvmet_req_complete(req, nvme_req(rq)->status);
+	blk_put_request(rq);
+}
+
+static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
+{
+	int sg_cnt = req->sg_cnt;
+	struct scatterlist *sg;
+	int op_flags = 0;
+	struct bio *bio;
+	int i, ret;
+
+	if (req->cmd->common.opcode == nvme_cmd_flush)
+		op_flags = REQ_FUA;
+	else if (nvme_is_write(req->cmd))
+		op_flags = REQ_SYNC | REQ_IDLE;
+
+
+	bio = bio_alloc(GFP_KERNEL, min(sg_cnt, BIO_MAX_PAGES));
+	bio->bi_end_io = bio_put;
+
+	for_each_sg(req->sg, sg, req->sg_cnt, i) {
+		if (bio_add_page(bio, sg_page(sg), sg->length,
+				 sg->offset) != sg->length) {
+			ret = blk_rq_append_bio(rq, &bio);
+			if (unlikely(ret))
+				return ret;
+
+			bio->bi_opf = req_op(rq) | op_flags;
+			bio = bio_alloc(GFP_KERNEL,
+					min(sg_cnt, BIO_MAX_PAGES));
+		}
+		sg_cnt--;
+	}
+
+	ret = blk_rq_append_bio(rq, &bio);
+	if (unlikely(ret))
+		return ret;
+
+	return 0;
+}
+
+static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
+{
+	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
+	struct nvme_ns *ns = NULL;
+	struct request *rq = NULL;
+	struct request_queue *q;
+	u32 effects;
+	u16 status;
+	int ret;
+
+	if (likely(req->sq->qid != 0)) {
+		u32 nsid = le32_to_cpu(req->cmd->common.nsid);
+
+		ns = nvme_find_get_ns(ctrl, nsid);
+		if (unlikely(!ns)) {
+			pr_err("failed to get passthru ns nsid:%u\n", nsid);
+			status = NVME_SC_INVALID_NS | NVME_SC_DNR;
+			goto fail_out;
+		}
+	}
+
+	if (ns)
+		q = ns->queue;
+	else
+		q = ctrl->admin_q;
+
+	rq = nvme_alloc_request(q, req->cmd, BLK_MQ_REQ_NOWAIT, NVME_QID_ANY);
+	if (IS_ERR(rq)) {
+		rq = NULL;
+		status = NVME_SC_INTERNAL;
+		goto fail_out;
+	}
+
+	if (req->sg_cnt) {
+		ret = nvmet_passthru_map_sg(req, rq);
+		if (unlikely(ret)) {
+			status = NVME_SC_INTERNAL;
+			goto fail_out;
+		}
+	}
+
+	if (blk_rq_nr_phys_segments(rq) > queue_max_segments(rq->q)) {
+		status = NVME_SC_INVALID_FIELD;
+		goto fail_out;
+	}
+
+	if ((blk_rq_payload_bytes(rq) >> 9) > queue_max_hw_sectors(rq->q)) {
+		status = NVME_SC_INVALID_FIELD;
+		goto fail_out;
+	}
+
+	/*
+	 * If there are effects for the command we are about to execute, or
+	 * an end_req function we need to use nvme_execute_passthru_rq()
+	 * synchronously in a work item seeing the end_req function and
+	 * nvme_passthru_end() can't be called in the request done callback
+	 * which is typically in interrupt context.
+	 */
+	effects = nvme_command_effects(ctrl, ns, req->cmd->common.opcode);
+	if (req->p.end_req || effects) {
+		INIT_WORK(&req->p.work, nvmet_passthru_execute_cmd_work);
+		req->p.rq = rq;
+		schedule_work(&req->p.work);
+	} else {
+		rq->end_io_data = req;
+		blk_execute_rq_nowait(rq->q, ns ? ns->disk : NULL, rq, 0,
+				      nvmet_passthru_req_done);
+	}
+
+	if (ns)
+		nvme_put_ns(ns);
+
+	return;
+
+fail_out:
+	if (ns)
+		nvme_put_ns(ns);
+	nvmet_req_complete(req, status);
+	blk_put_request(rq);
+}
+
+static u16 nvmet_passthru_override_id_ctrl(struct nvmet_req *req)
+{
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	struct nvme_ctrl *pctrl = ctrl->subsys->passthru_ctrl;
+	u16 status = NVME_SC_SUCCESS;
+	struct nvme_id_ctrl *id;
+	u32 max_hw_sectors;
+	int page_shift;
+
+	id = kzalloc(sizeof(*id), GFP_KERNEL);
+	if (!id)
+		return NVME_SC_INTERNAL;
+
+	status = nvmet_copy_from_sgl(req, 0, id, sizeof(*id));
+	if (status)
+		goto out_free;
+
+	id->cntlid = cpu_to_le16(ctrl->cntlid);
+	id->ver = cpu_to_le32(ctrl->subsys->ver);
+
+	/*
+	 * The passthru NVMe driver may have a limit on the number of segments
+	 * which depends on the host's memory fragementation. To solve this,
+	 * ensure mdts is limitted to the pages equal to the number of
+	 * segments.
+	 */
+	max_hw_sectors = min_not_zero(pctrl->max_segments << (PAGE_SHIFT - 9),
+				      pctrl->max_hw_sectors);
+
+	page_shift = NVME_CAP_MPSMIN(ctrl->cap) + 12;
+
+	id->mdts = ilog2(max_hw_sectors) + 9 - page_shift;
+
+	id->acl = 3;
+	/*
+	 * We export aerl limit for the fabrics controller, update this when
+	 * passthru based aerl support is added.
+	 */
+	id->aerl = NVMET_ASYNC_EVENTS - 1;
+
+	/* emulate kas as most of the PCIe ctrl don't have a support for kas */
+	id->kas = cpu_to_le16(NVMET_KAS);
+
+	/* don't support host memory buffer */
+	id->hmpre = 0;
+	id->hmmin = 0;
+
+	id->sqes = min_t(__u8, ((0x6 << 4) | 0x6), id->sqes);
+	id->cqes = min_t(__u8, ((0x4 << 4) | 0x4), id->cqes);
+	id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
+
+	/* don't support fuse commands */
+	id->fuses = 0;
+
+	id->sgls = cpu_to_le32(1 << 0); /* we always support SGLs */
+	if (ctrl->ops->has_keyed_sgls)
+		id->sgls |= cpu_to_le32(1 << 2);
+	if (req->port->inline_data_size)
+		id->sgls |= cpu_to_le32(1 << 20);
+
+	/*
+	 * When passsthru controller is setup using nvme-loop transport it will
+	 * export the passthru ctrl subsysnqn (PCIe NVMe ctrl) and will fail in
+	 * the nvme/host/core.c in the nvme_init_subsystem()->nvme_active_ctrl()
+	 * code path with duplicate ctr subsynqn. In order to prevent that we
+	 * mask the passthru-ctrl subsysnqn with the target ctrl subsysnqn.
+	 */
+	memcpy(id->subnqn, ctrl->subsysnqn, sizeof(id->subnqn));
+
+	/* use fabric id-ctrl values */
+	id->ioccsz = cpu_to_le32((sizeof(struct nvme_command) +
+				req->port->inline_data_size) / 16);
+	id->iorcsz = cpu_to_le32(sizeof(struct nvme_completion) / 16);
+
+	id->msdbd = ctrl->ops->msdbd;
+
+	/* Support multipath connections with fabrics */
+	id->cmic |= 1 << 1;
+
+	/* Disable reservations, see nvmet_parse_passthru_io_cmd() */
+	id->oncs &= cpu_to_le16(~NVME_CTRL_ONCS_RESERVATIONS);
+
+	status = nvmet_copy_to_sgl(req, 0, id, sizeof(struct nvme_id_ctrl));
+
+out_free:
+	kfree(id);
+	return status;
+}
+
+static u16 nvmet_passthru_override_id_ns(struct nvmet_req *req)
+{
+	u16 status = NVME_SC_SUCCESS;
+	struct nvme_id_ns *id;
+	int i;
+
+	id = kzalloc(sizeof(*id), GFP_KERNEL);
+	if (!id)
+		return NVME_SC_INTERNAL;
+
+	status = nvmet_copy_from_sgl(req, 0, id, sizeof(struct nvme_id_ns));
+	if (status)
+		goto out_free;
+
+	for (i = 0; i < (id->nlbaf + 1); i++)
+		if (id->lbaf[i].ms)
+			memset(&id->lbaf[i], 0, sizeof(id->lbaf[i]));
+
+	id->flbas = id->flbas & ~(1 << 4);
+
+	/*
+	 * Presently the NVMEof target code does not support sending
+	 * metadata, so we must disable it here. This should be updated
+	 * once target starts supporting metadata.
+	 */
+	id->mc = 0;
+
+	status = nvmet_copy_to_sgl(req, 0, id, sizeof(*id));
+
+out_free:
+	kfree(id);
+	return status;
+}
+
+/*
+ * We need to emulate set host behaviour to ensure that any requested
+ * behaviour of the target's host matches the requested behaviour
+ * of the device's host and fail otherwise.
+ */
+static void nvmet_passthru_set_host_behaviour(struct nvmet_req *req)
+{
+	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
+	struct nvme_feat_host_behavior *host;
+	u16 status;
+	int ret;
+
+	host = kzalloc(sizeof(*host) * 2, GFP_KERNEL);
+	ret = nvme_get_features(ctrl, NVME_FEAT_HOST_BEHAVIOR, 0,
+				host, sizeof(*host), NULL);
+	if (ret) {
+		status = NVME_SC_INTERNAL;
+		goto free_and_exit;
+	}
+
+	status = nvmet_copy_from_sgl(req, 0, &host[1], sizeof(*host));
+	if (status)
+		goto free_and_exit;
+
+	if (memcmp(&host[0], &host[1], sizeof(host[0]))) {
+		pr_warn("target host has requested different behaviour from the local host\n");
+		status = NVME_SC_INTERNAL;
+	}
+
+free_and_exit:
+	kfree(host);
+	nvmet_req_complete(req, status);
+}
+
+static u16 nvmet_setup_passthru_command(struct nvmet_req *req)
+{
+	req->p.end_req = NULL;
+	req->execute = nvmet_passthru_execute_cmd;
+	return NVME_SC_SUCCESS;
+}
+
+u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req)
+{
+	switch (req->cmd->common.opcode) {
+	case nvme_cmd_resv_register:
+	case nvme_cmd_resv_report:
+	case nvme_cmd_resv_acquire:
+	case nvme_cmd_resv_release:
+		/*
+		 * Reservations cannot be supported properly because the
+		 * underlying device has no way of differentiating different
+		 * hosts that connect via fabrics. This could potentially be
+		 * emulated in the future if regular targets grow support for
+		 * this feature.
+		 */
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+	}
+
+	return nvmet_setup_passthru_command(req);
+}
+
+/*
+ * Only features that are emulated or specifically whitelisted are passed
+ * down to the controller. This function implements the whitelist for both
+ * get and set features.
+ */
+static u16 nvmet_passthru_get_set_features(struct nvmet_req *req)
+{
+	switch (le32_to_cpu(req->cmd->features.fid)) {
+	case NVME_FEAT_ARBITRATION:
+	case NVME_FEAT_POWER_MGMT:
+	case NVME_FEAT_LBA_RANGE:
+	case NVME_FEAT_TEMP_THRESH:
+	case NVME_FEAT_ERR_RECOVERY:
+	case NVME_FEAT_VOLATILE_WC:
+	case NVME_FEAT_WRITE_ATOMIC:
+	case NVME_FEAT_AUTO_PST:
+	case NVME_FEAT_TIMESTAMP:
+	case NVME_FEAT_HCTM:
+	case NVME_FEAT_NOPSC:
+	case NVME_FEAT_RRL:
+	case NVME_FEAT_PLM_CONFIG:
+	case NVME_FEAT_PLM_WINDOW:
+	case NVME_FEAT_HOST_BEHAVIOR:
+	case NVME_FEAT_SANITIZE:
+	case NVME_FEAT_VENDOR_START ... NVME_FEAT_VENDOR_END:
+		return nvmet_setup_passthru_command(req);
+
+	case NVME_FEAT_ASYNC_EVENT:
+		/* There is no support for forwarding ASYNC events */
+	case NVME_FEAT_IRQ_COALESCE:
+	case NVME_FEAT_IRQ_CONFIG:
+		/* The IRQ settings will not apply to the target controller */
+	case NVME_FEAT_HOST_MEM_BUF:
+		/*
+		 * Any HMB that's set will not be passed through and will
+		 * not work as expected
+		 */
+	case NVME_FEAT_SW_PROGRESS:
+		/*
+		 * The Pre-Boot Software Load Count doesn't make much
+		 * sense for a target to export
+		 */
+	case NVME_FEAT_RESV_MASK:
+	case NVME_FEAT_RESV_PERSIST:
+		/* No reservations, see nvmet_parse_passthru_io_cmd() */
+	default:
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+	}
+}
+
+u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
+{
+	/*
+	 * Passthru all vendor specific commands
+	 */
+	if (req->cmd->common.opcode >= nvme_admin_vendor_start)
+		return nvmet_setup_passthru_command(req);
+
+	switch (req->cmd->common.opcode) {
+	case nvme_admin_async_event:
+		req->execute = nvmet_execute_async_event;
+		return NVME_SC_SUCCESS;
+	case nvme_admin_keep_alive:
+		/*
+		 * Most PCIe ctrls don't support keep alive cmd, we route keep
+		 * alive to the non-passthru mode. In future please change this
+		 * code when PCIe ctrls with keep alive support available.
+		 */
+		req->execute = nvmet_execute_keep_alive;
+		return NVME_SC_SUCCESS;
+	case nvme_admin_set_features:
+		switch (le32_to_cpu(req->cmd->features.fid)) {
+		case NVME_FEAT_ASYNC_EVENT:
+		case NVME_FEAT_KATO:
+		case NVME_FEAT_NUM_QUEUES:
+		case NVME_FEAT_HOST_ID:
+			req->execute = nvmet_execute_set_features;
+			return NVME_SC_SUCCESS;
+		case NVME_FEAT_HOST_BEHAVIOR:
+			req->execute = nvmet_passthru_set_host_behaviour;
+			return NVME_SC_SUCCESS;
+		default:
+			return nvmet_passthru_get_set_features(req);
+		}
+		break;
+	case nvme_admin_get_features:
+		switch (le32_to_cpu(req->cmd->features.fid)) {
+		case NVME_FEAT_ASYNC_EVENT:
+		case NVME_FEAT_KATO:
+		case NVME_FEAT_NUM_QUEUES:
+		case NVME_FEAT_HOST_ID:
+			req->execute = nvmet_execute_get_features;
+			return NVME_SC_SUCCESS;
+		default:
+			return nvmet_passthru_get_set_features(req);
+		}
+		break;
+	case nvme_admin_identify:
+		switch (req->cmd->identify.cns) {
+		case NVME_ID_CNS_CTRL:
+			req->execute = nvmet_passthru_execute_cmd;
+			req->p.end_req = nvmet_passthru_override_id_ctrl;
+			return NVME_SC_SUCCESS;
+		case NVME_ID_CNS_NS:
+			req->execute = nvmet_passthru_execute_cmd;
+			req->p.end_req = nvmet_passthru_override_id_ns;
+			return NVME_SC_SUCCESS;
+		default:
+			return nvmet_setup_passthru_command(req);
+		}
+	case nvme_admin_get_log_page:
+		return nvmet_setup_passthru_command(req);
+	default:
+		/* Reject commands not in the whitelist above */
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+	}
+}
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 3d5189f46cb1..af807e0fb114 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -303,6 +303,7 @@ enum {
 	NVME_CTRL_ONCS_WRITE_UNCORRECTABLE	= 1 << 1,
 	NVME_CTRL_ONCS_DSM			= 1 << 2,
 	NVME_CTRL_ONCS_WRITE_ZEROES		= 1 << 3,
+	NVME_CTRL_ONCS_RESERVATIONS		= 1 << 5,
 	NVME_CTRL_ONCS_TIMESTAMP		= 1 << 6,
 	NVME_CTRL_VWC_PRESENT			= 1 << 0,
 	NVME_CTRL_OACS_SEC_SUPP                 = 1 << 0,
@@ -858,6 +859,7 @@ enum nvme_admin_opcode {
 	nvme_admin_security_recv	= 0x82,
 	nvme_admin_sanitize_nvm		= 0x84,
 	nvme_admin_get_lba_status	= 0x86,
+	nvme_admin_vendor_start		= 0xC0,
 };
 
 #define nvme_admin_opcode_name(opcode)	{ opcode, #opcode }
@@ -921,6 +923,8 @@ enum {
 	NVME_FEAT_RESV_MASK	= 0x82,
 	NVME_FEAT_RESV_PERSIST	= 0x83,
 	NVME_FEAT_WRITE_PROTECT	= 0x84,
+	NVME_FEAT_VENDOR_START	= 0xC0,
+	NVME_FEAT_VENDOR_END	= 0xFF,
 	NVME_LOG_ERROR		= 0x01,
 	NVME_LOG_SMART		= 0x02,
 	NVME_LOG_FW_SLOT	= 0x03,
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 7/9] nvmet-passthru: Add passthru code to process commands
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

Add passthru command handling capability for the NVMeOF target and
export passthru APIs which are used to integrate passthru
code with nvmet-core.

The new file passthru.c handles passthru cmd parsing and execution.
In the passthru mode, we create a block layer request from the nvmet
request and map the data on to the block layer request.

Admin commands and features are on a white list as there are a number
of each that don't make too much sense with passthrough. We use a
white list so that new commands can be considered before being blindly
passed through. In both cases, vendor specific commands are always
allowed.

We also blacklist reservation IO commands as the underlying device
cannot differentiate between multiple hosts behind a fabric.

Based-on-a-patch-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/Makefile    |   1 +
 drivers/nvme/target/admin-cmd.c |   7 +-
 drivers/nvme/target/core.c      |   3 +
 drivers/nvme/target/nvmet.h     |  39 +++
 drivers/nvme/target/passthru.c  | 460 ++++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   4 +
 6 files changed, 512 insertions(+), 2 deletions(-)
 create mode 100644 drivers/nvme/target/passthru.c

diff --git a/drivers/nvme/target/Makefile b/drivers/nvme/target/Makefile
index 2b33836f3d3e..ebf91fc4c72e 100644
--- a/drivers/nvme/target/Makefile
+++ b/drivers/nvme/target/Makefile
@@ -11,6 +11,7 @@ obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
 			discovery.o io-cmd-file.o io-cmd-bdev.o
+nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvme-loop-y	+= loop.o
 nvmet-rdma-y	+= rdma.o
 nvmet-fc-y	+= fc.o
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 9d6f75cfa77c..a2dfd202eb32 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -734,7 +734,7 @@ u16 nvmet_set_feat_async_event(struct nvmet_req *req, u32 mask)
 	return 0;
 }
 
-static void nvmet_execute_set_features(struct nvmet_req *req)
+void nvmet_execute_set_features(struct nvmet_req *req)
 {
 	struct nvmet_subsys *subsys = req->sq->ctrl->subsys;
 	u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
@@ -809,7 +809,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req)
 	nvmet_set_result(req, READ_ONCE(req->sq->ctrl->aen_enabled));
 }
 
-static void nvmet_execute_get_features(struct nvmet_req *req)
+void nvmet_execute_get_features(struct nvmet_req *req)
 {
 	struct nvmet_subsys *subsys = req->sq->ctrl->subsys;
 	u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
@@ -925,6 +925,9 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
 	if (unlikely(ret))
 		return ret;
 
+	if (nvmet_req_passthru_ctrl(req))
+		return nvmet_parse_passthru_admin_cmd(req);
+
 	switch (cmd->common.opcode) {
 	case nvme_admin_get_log_page:
 		req->execute = nvmet_execute_get_log_page;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index b685f99d56a1..2547e0d8951c 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -841,6 +841,9 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 	if (unlikely(ret))
 		return ret;
 
+	if (nvmet_req_passthru_ctrl(req))
+		return nvmet_parse_passthru_io_cmd(req);
+
 	req->ns = nvmet_find_namespace(req->sq->ctrl, cmd->rw.nsid);
 	if (unlikely(!req->ns)) {
 		req->error_loc = offsetof(struct nvme_common_command, nsid);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 421dff3ea143..deb996c90804 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -237,6 +237,10 @@ struct nvmet_subsys {
 	struct config_group	allowed_hosts_group;
 
 	struct nvmet_subsys_model	__rcu *model;
+
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+	struct nvme_ctrl	*passthru_ctrl;
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
 static inline struct nvmet_subsys *to_subsys(struct config_item *item)
@@ -313,6 +317,11 @@ struct nvmet_req {
 			struct bio_vec          *bvec;
 			struct work_struct      work;
 		} f;
+		struct {
+			struct request		*rq;
+			struct work_struct      work;
+			u16 (*end_req)(struct nvmet_req *req);
+		} p;
 	};
 	int			sg_cnt;
 	/* data length as parsed from the SGL descriptor: */
@@ -390,6 +399,8 @@ void nvmet_req_complete(struct nvmet_req *req, u16 status);
 int nvmet_req_alloc_sgl(struct nvmet_req *req);
 void nvmet_req_free_sgl(struct nvmet_req *req);
 
+void nvmet_execute_set_features(struct nvmet_req *req);
+void nvmet_execute_get_features(struct nvmet_req *req);
 void nvmet_execute_keep_alive(struct nvmet_req *req);
 
 void nvmet_cq_setup(struct nvmet_ctrl *ctrl, struct nvmet_cq *cq, u16 qid,
@@ -511,6 +522,34 @@ static inline u32 nvmet_dsm_len(struct nvmet_req *req)
 		sizeof(struct nvme_dsm_range);
 }
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req);
+u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req);
+static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
+{
+	return subsys->passthru_ctrl;
+}
+#else /* CONFIG_NVME_TARGET_PASSTHRU */
+static inline u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
+{
+	return 0;
+}
+static inline u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req)
+{
+	return 0;
+}
+static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
+{
+	return NULL;
+}
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
+static inline struct nvme_ctrl *
+nvmet_req_passthru_ctrl(struct nvmet_req *req)
+{
+	return nvmet_passthru_ctrl(req->sq->ctrl->subsys);
+}
+
 u16 errno_to_nvme_status(struct nvmet_req *req, int errno);
 
 /* Convert a 32-bit number to a 16-bit 0's based number */
diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
new file mode 100644
index 000000000000..15b23521e59f
--- /dev/null
+++ b/drivers/nvme/target/passthru.c
@@ -0,0 +1,460 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe Over Fabrics Target Passthrough command implementation.
+ *
+ * Copyright (c) 2017-2018 Western Digital Corporation or its
+ * affiliates.
+ */
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/module.h>
+
+#include "../host/nvme.h"
+#include "nvmet.h"
+
+static void nvmet_passthru_execute_cmd_work(struct work_struct *w)
+{
+	struct nvmet_req *req = container_of(w, struct nvmet_req, p.work);
+	struct request *rq = req->p.rq;
+	u16 status;
+
+	nvme_execute_passthru_rq(rq);
+
+	status = nvme_req(rq)->status;
+	if (status == NVME_SC_SUCCESS && req->p.end_req)
+		status = req->p.end_req(req);
+
+	req->cqe->result = nvme_req(rq)->result;
+	nvmet_req_complete(req, status);
+	blk_put_request(rq);
+}
+
+static void nvmet_passthru_req_done(struct request *rq,
+				    blk_status_t blk_status)
+{
+	struct nvmet_req *req = rq->end_io_data;
+
+	req->cqe->result = nvme_req(rq)->result;
+	nvmet_req_complete(req, nvme_req(rq)->status);
+	blk_put_request(rq);
+}
+
+static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
+{
+	int sg_cnt = req->sg_cnt;
+	struct scatterlist *sg;
+	int op_flags = 0;
+	struct bio *bio;
+	int i, ret;
+
+	if (req->cmd->common.opcode == nvme_cmd_flush)
+		op_flags = REQ_FUA;
+	else if (nvme_is_write(req->cmd))
+		op_flags = REQ_SYNC | REQ_IDLE;
+
+
+	bio = bio_alloc(GFP_KERNEL, min(sg_cnt, BIO_MAX_PAGES));
+	bio->bi_end_io = bio_put;
+
+	for_each_sg(req->sg, sg, req->sg_cnt, i) {
+		if (bio_add_page(bio, sg_page(sg), sg->length,
+				 sg->offset) != sg->length) {
+			ret = blk_rq_append_bio(rq, &bio);
+			if (unlikely(ret))
+				return ret;
+
+			bio->bi_opf = req_op(rq) | op_flags;
+			bio = bio_alloc(GFP_KERNEL,
+					min(sg_cnt, BIO_MAX_PAGES));
+		}
+		sg_cnt--;
+	}
+
+	ret = blk_rq_append_bio(rq, &bio);
+	if (unlikely(ret))
+		return ret;
+
+	return 0;
+}
+
+static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
+{
+	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
+	struct nvme_ns *ns = NULL;
+	struct request *rq = NULL;
+	struct request_queue *q;
+	u32 effects;
+	u16 status;
+	int ret;
+
+	if (likely(req->sq->qid != 0)) {
+		u32 nsid = le32_to_cpu(req->cmd->common.nsid);
+
+		ns = nvme_find_get_ns(ctrl, nsid);
+		if (unlikely(!ns)) {
+			pr_err("failed to get passthru ns nsid:%u\n", nsid);
+			status = NVME_SC_INVALID_NS | NVME_SC_DNR;
+			goto fail_out;
+		}
+	}
+
+	if (ns)
+		q = ns->queue;
+	else
+		q = ctrl->admin_q;
+
+	rq = nvme_alloc_request(q, req->cmd, BLK_MQ_REQ_NOWAIT, NVME_QID_ANY);
+	if (IS_ERR(rq)) {
+		rq = NULL;
+		status = NVME_SC_INTERNAL;
+		goto fail_out;
+	}
+
+	if (req->sg_cnt) {
+		ret = nvmet_passthru_map_sg(req, rq);
+		if (unlikely(ret)) {
+			status = NVME_SC_INTERNAL;
+			goto fail_out;
+		}
+	}
+
+	if (blk_rq_nr_phys_segments(rq) > queue_max_segments(rq->q)) {
+		status = NVME_SC_INVALID_FIELD;
+		goto fail_out;
+	}
+
+	if ((blk_rq_payload_bytes(rq) >> 9) > queue_max_hw_sectors(rq->q)) {
+		status = NVME_SC_INVALID_FIELD;
+		goto fail_out;
+	}
+
+	/*
+	 * If there are effects for the command we are about to execute, or
+	 * an end_req function we need to use nvme_execute_passthru_rq()
+	 * synchronously in a work item seeing the end_req function and
+	 * nvme_passthru_end() can't be called in the request done callback
+	 * which is typically in interrupt context.
+	 */
+	effects = nvme_command_effects(ctrl, ns, req->cmd->common.opcode);
+	if (req->p.end_req || effects) {
+		INIT_WORK(&req->p.work, nvmet_passthru_execute_cmd_work);
+		req->p.rq = rq;
+		schedule_work(&req->p.work);
+	} else {
+		rq->end_io_data = req;
+		blk_execute_rq_nowait(rq->q, ns ? ns->disk : NULL, rq, 0,
+				      nvmet_passthru_req_done);
+	}
+
+	if (ns)
+		nvme_put_ns(ns);
+
+	return;
+
+fail_out:
+	if (ns)
+		nvme_put_ns(ns);
+	nvmet_req_complete(req, status);
+	blk_put_request(rq);
+}
+
+static u16 nvmet_passthru_override_id_ctrl(struct nvmet_req *req)
+{
+	struct nvmet_ctrl *ctrl = req->sq->ctrl;
+	struct nvme_ctrl *pctrl = ctrl->subsys->passthru_ctrl;
+	u16 status = NVME_SC_SUCCESS;
+	struct nvme_id_ctrl *id;
+	u32 max_hw_sectors;
+	int page_shift;
+
+	id = kzalloc(sizeof(*id), GFP_KERNEL);
+	if (!id)
+		return NVME_SC_INTERNAL;
+
+	status = nvmet_copy_from_sgl(req, 0, id, sizeof(*id));
+	if (status)
+		goto out_free;
+
+	id->cntlid = cpu_to_le16(ctrl->cntlid);
+	id->ver = cpu_to_le32(ctrl->subsys->ver);
+
+	/*
+	 * The passthru NVMe driver may have a limit on the number of segments
+	 * which depends on the host's memory fragementation. To solve this,
+	 * ensure mdts is limitted to the pages equal to the number of
+	 * segments.
+	 */
+	max_hw_sectors = min_not_zero(pctrl->max_segments << (PAGE_SHIFT - 9),
+				      pctrl->max_hw_sectors);
+
+	page_shift = NVME_CAP_MPSMIN(ctrl->cap) + 12;
+
+	id->mdts = ilog2(max_hw_sectors) + 9 - page_shift;
+
+	id->acl = 3;
+	/*
+	 * We export aerl limit for the fabrics controller, update this when
+	 * passthru based aerl support is added.
+	 */
+	id->aerl = NVMET_ASYNC_EVENTS - 1;
+
+	/* emulate kas as most of the PCIe ctrl don't have a support for kas */
+	id->kas = cpu_to_le16(NVMET_KAS);
+
+	/* don't support host memory buffer */
+	id->hmpre = 0;
+	id->hmmin = 0;
+
+	id->sqes = min_t(__u8, ((0x6 << 4) | 0x6), id->sqes);
+	id->cqes = min_t(__u8, ((0x4 << 4) | 0x4), id->cqes);
+	id->maxcmd = cpu_to_le16(NVMET_MAX_CMD);
+
+	/* don't support fuse commands */
+	id->fuses = 0;
+
+	id->sgls = cpu_to_le32(1 << 0); /* we always support SGLs */
+	if (ctrl->ops->has_keyed_sgls)
+		id->sgls |= cpu_to_le32(1 << 2);
+	if (req->port->inline_data_size)
+		id->sgls |= cpu_to_le32(1 << 20);
+
+	/*
+	 * When passsthru controller is setup using nvme-loop transport it will
+	 * export the passthru ctrl subsysnqn (PCIe NVMe ctrl) and will fail in
+	 * the nvme/host/core.c in the nvme_init_subsystem()->nvme_active_ctrl()
+	 * code path with duplicate ctr subsynqn. In order to prevent that we
+	 * mask the passthru-ctrl subsysnqn with the target ctrl subsysnqn.
+	 */
+	memcpy(id->subnqn, ctrl->subsysnqn, sizeof(id->subnqn));
+
+	/* use fabric id-ctrl values */
+	id->ioccsz = cpu_to_le32((sizeof(struct nvme_command) +
+				req->port->inline_data_size) / 16);
+	id->iorcsz = cpu_to_le32(sizeof(struct nvme_completion) / 16);
+
+	id->msdbd = ctrl->ops->msdbd;
+
+	/* Support multipath connections with fabrics */
+	id->cmic |= 1 << 1;
+
+	/* Disable reservations, see nvmet_parse_passthru_io_cmd() */
+	id->oncs &= cpu_to_le16(~NVME_CTRL_ONCS_RESERVATIONS);
+
+	status = nvmet_copy_to_sgl(req, 0, id, sizeof(struct nvme_id_ctrl));
+
+out_free:
+	kfree(id);
+	return status;
+}
+
+static u16 nvmet_passthru_override_id_ns(struct nvmet_req *req)
+{
+	u16 status = NVME_SC_SUCCESS;
+	struct nvme_id_ns *id;
+	int i;
+
+	id = kzalloc(sizeof(*id), GFP_KERNEL);
+	if (!id)
+		return NVME_SC_INTERNAL;
+
+	status = nvmet_copy_from_sgl(req, 0, id, sizeof(struct nvme_id_ns));
+	if (status)
+		goto out_free;
+
+	for (i = 0; i < (id->nlbaf + 1); i++)
+		if (id->lbaf[i].ms)
+			memset(&id->lbaf[i], 0, sizeof(id->lbaf[i]));
+
+	id->flbas = id->flbas & ~(1 << 4);
+
+	/*
+	 * Presently the NVMEof target code does not support sending
+	 * metadata, so we must disable it here. This should be updated
+	 * once target starts supporting metadata.
+	 */
+	id->mc = 0;
+
+	status = nvmet_copy_to_sgl(req, 0, id, sizeof(*id));
+
+out_free:
+	kfree(id);
+	return status;
+}
+
+/*
+ * We need to emulate set host behaviour to ensure that any requested
+ * behaviour of the target's host matches the requested behaviour
+ * of the device's host and fail otherwise.
+ */
+static void nvmet_passthru_set_host_behaviour(struct nvmet_req *req)
+{
+	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
+	struct nvme_feat_host_behavior *host;
+	u16 status;
+	int ret;
+
+	host = kzalloc(sizeof(*host) * 2, GFP_KERNEL);
+	ret = nvme_get_features(ctrl, NVME_FEAT_HOST_BEHAVIOR, 0,
+				host, sizeof(*host), NULL);
+	if (ret) {
+		status = NVME_SC_INTERNAL;
+		goto free_and_exit;
+	}
+
+	status = nvmet_copy_from_sgl(req, 0, &host[1], sizeof(*host));
+	if (status)
+		goto free_and_exit;
+
+	if (memcmp(&host[0], &host[1], sizeof(host[0]))) {
+		pr_warn("target host has requested different behaviour from the local host\n");
+		status = NVME_SC_INTERNAL;
+	}
+
+free_and_exit:
+	kfree(host);
+	nvmet_req_complete(req, status);
+}
+
+static u16 nvmet_setup_passthru_command(struct nvmet_req *req)
+{
+	req->p.end_req = NULL;
+	req->execute = nvmet_passthru_execute_cmd;
+	return NVME_SC_SUCCESS;
+}
+
+u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req)
+{
+	switch (req->cmd->common.opcode) {
+	case nvme_cmd_resv_register:
+	case nvme_cmd_resv_report:
+	case nvme_cmd_resv_acquire:
+	case nvme_cmd_resv_release:
+		/*
+		 * Reservations cannot be supported properly because the
+		 * underlying device has no way of differentiating different
+		 * hosts that connect via fabrics. This could potentially be
+		 * emulated in the future if regular targets grow support for
+		 * this feature.
+		 */
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+	}
+
+	return nvmet_setup_passthru_command(req);
+}
+
+/*
+ * Only features that are emulated or specifically whitelisted are passed
+ * down to the controller. This function implements the whitelist for both
+ * get and set features.
+ */
+static u16 nvmet_passthru_get_set_features(struct nvmet_req *req)
+{
+	switch (le32_to_cpu(req->cmd->features.fid)) {
+	case NVME_FEAT_ARBITRATION:
+	case NVME_FEAT_POWER_MGMT:
+	case NVME_FEAT_LBA_RANGE:
+	case NVME_FEAT_TEMP_THRESH:
+	case NVME_FEAT_ERR_RECOVERY:
+	case NVME_FEAT_VOLATILE_WC:
+	case NVME_FEAT_WRITE_ATOMIC:
+	case NVME_FEAT_AUTO_PST:
+	case NVME_FEAT_TIMESTAMP:
+	case NVME_FEAT_HCTM:
+	case NVME_FEAT_NOPSC:
+	case NVME_FEAT_RRL:
+	case NVME_FEAT_PLM_CONFIG:
+	case NVME_FEAT_PLM_WINDOW:
+	case NVME_FEAT_HOST_BEHAVIOR:
+	case NVME_FEAT_SANITIZE:
+	case NVME_FEAT_VENDOR_START ... NVME_FEAT_VENDOR_END:
+		return nvmet_setup_passthru_command(req);
+
+	case NVME_FEAT_ASYNC_EVENT:
+		/* There is no support for forwarding ASYNC events */
+	case NVME_FEAT_IRQ_COALESCE:
+	case NVME_FEAT_IRQ_CONFIG:
+		/* The IRQ settings will not apply to the target controller */
+	case NVME_FEAT_HOST_MEM_BUF:
+		/*
+		 * Any HMB that's set will not be passed through and will
+		 * not work as expected
+		 */
+	case NVME_FEAT_SW_PROGRESS:
+		/*
+		 * The Pre-Boot Software Load Count doesn't make much
+		 * sense for a target to export
+		 */
+	case NVME_FEAT_RESV_MASK:
+	case NVME_FEAT_RESV_PERSIST:
+		/* No reservations, see nvmet_parse_passthru_io_cmd() */
+	default:
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+	}
+}
+
+u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
+{
+	/*
+	 * Passthru all vendor specific commands
+	 */
+	if (req->cmd->common.opcode >= nvme_admin_vendor_start)
+		return nvmet_setup_passthru_command(req);
+
+	switch (req->cmd->common.opcode) {
+	case nvme_admin_async_event:
+		req->execute = nvmet_execute_async_event;
+		return NVME_SC_SUCCESS;
+	case nvme_admin_keep_alive:
+		/*
+		 * Most PCIe ctrls don't support keep alive cmd, we route keep
+		 * alive to the non-passthru mode. In future please change this
+		 * code when PCIe ctrls with keep alive support available.
+		 */
+		req->execute = nvmet_execute_keep_alive;
+		return NVME_SC_SUCCESS;
+	case nvme_admin_set_features:
+		switch (le32_to_cpu(req->cmd->features.fid)) {
+		case NVME_FEAT_ASYNC_EVENT:
+		case NVME_FEAT_KATO:
+		case NVME_FEAT_NUM_QUEUES:
+		case NVME_FEAT_HOST_ID:
+			req->execute = nvmet_execute_set_features;
+			return NVME_SC_SUCCESS;
+		case NVME_FEAT_HOST_BEHAVIOR:
+			req->execute = nvmet_passthru_set_host_behaviour;
+			return NVME_SC_SUCCESS;
+		default:
+			return nvmet_passthru_get_set_features(req);
+		}
+		break;
+	case nvme_admin_get_features:
+		switch (le32_to_cpu(req->cmd->features.fid)) {
+		case NVME_FEAT_ASYNC_EVENT:
+		case NVME_FEAT_KATO:
+		case NVME_FEAT_NUM_QUEUES:
+		case NVME_FEAT_HOST_ID:
+			req->execute = nvmet_execute_get_features;
+			return NVME_SC_SUCCESS;
+		default:
+			return nvmet_passthru_get_set_features(req);
+		}
+		break;
+	case nvme_admin_identify:
+		switch (req->cmd->identify.cns) {
+		case NVME_ID_CNS_CTRL:
+			req->execute = nvmet_passthru_execute_cmd;
+			req->p.end_req = nvmet_passthru_override_id_ctrl;
+			return NVME_SC_SUCCESS;
+		case NVME_ID_CNS_NS:
+			req->execute = nvmet_passthru_execute_cmd;
+			req->p.end_req = nvmet_passthru_override_id_ns;
+			return NVME_SC_SUCCESS;
+		default:
+			return nvmet_setup_passthru_command(req);
+		}
+	case nvme_admin_get_log_page:
+		return nvmet_setup_passthru_command(req);
+	default:
+		/* Reject commands not in the whitelist above */
+		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
+	}
+}
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index 3d5189f46cb1..af807e0fb114 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -303,6 +303,7 @@ enum {
 	NVME_CTRL_ONCS_WRITE_UNCORRECTABLE	= 1 << 1,
 	NVME_CTRL_ONCS_DSM			= 1 << 2,
 	NVME_CTRL_ONCS_WRITE_ZEROES		= 1 << 3,
+	NVME_CTRL_ONCS_RESERVATIONS		= 1 << 5,
 	NVME_CTRL_ONCS_TIMESTAMP		= 1 << 6,
 	NVME_CTRL_VWC_PRESENT			= 1 << 0,
 	NVME_CTRL_OACS_SEC_SUPP                 = 1 << 0,
@@ -858,6 +859,7 @@ enum nvme_admin_opcode {
 	nvme_admin_security_recv	= 0x82,
 	nvme_admin_sanitize_nvm		= 0x84,
 	nvme_admin_get_lba_status	= 0x86,
+	nvme_admin_vendor_start		= 0xC0,
 };
 
 #define nvme_admin_opcode_name(opcode)	{ opcode, #opcode }
@@ -921,6 +923,8 @@ enum {
 	NVME_FEAT_RESV_MASK	= 0x82,
 	NVME_FEAT_RESV_PERSIST	= 0x83,
 	NVME_FEAT_WRITE_PROTECT	= 0x84,
+	NVME_FEAT_VENDOR_START	= 0xC0,
+	NVME_FEAT_VENDOR_END	= 0xFF,
 	NVME_LOG_ERROR		= 0x01,
 	NVME_LOG_SMART		= 0x02,
 	NVME_LOG_FW_SLOT	= 0x03,
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe,
	Chaitanya Kulkarni

This patch adds helper functions which are used in the NVMeOF configfs
when the user is configuring the passthru subsystem. Here we ensure
that only one subsys is assigned to each nvme_ctrl by using an xarray
on the cntlid.

The subsystem's version number is overridden by the passed through
controller's version. However, if that version is less than 1.2.1,
then we bump the advertised version to that and print a warning
in dmesg.

Based-on-a-patch-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/configfs.c |  4 ++
 drivers/nvme/target/core.c     | 10 +++-
 drivers/nvme/target/nvmet.h    | 12 +++++
 drivers/nvme/target/passthru.c | 87 ++++++++++++++++++++++++++++++++++
 4 files changed, 112 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 58cabd7b6fc5..e0ce6e5feb3a 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -826,6 +826,10 @@ static ssize_t nvmet_subsys_attr_version_store(struct config_item *item,
 	int major, minor, tertiary = 0;
 	int ret;
 
+	/* passthru subsystems use the underlying controller's version */
+	if (nvmet_passthru_ctrl(subsys))
+		return -EINVAL;
+
 	ret = sscanf(page, "%d.%d.%d\n", &major, &minor, &tertiary);
 	if (ret != 2 && ret != 3)
 		return -EINVAL;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 2547e0d8951c..a2cdd35ffb1d 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -522,6 +522,12 @@ int nvmet_ns_enable(struct nvmet_ns *ns)
 
 	mutex_lock(&subsys->lock);
 	ret = 0;
+
+	if (nvmet_passthru_ctrl(subsys)) {
+		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
+		goto out_unlock;
+	}
+
 	if (ns->enabled)
 		goto out_unlock;
 
@@ -1421,7 +1427,7 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 	if (!subsys)
 		return ERR_PTR(-ENOMEM);
 
-	subsys->ver = NVME_VS(1, 3, 0); /* NVMe 1.3.0 */
+	subsys->ver = NVMET_DEFAULT_VS;
 	/* generate a random serial number as our controllers are ephemeral: */
 	get_random_bytes(&subsys->serial, sizeof(subsys->serial));
 
@@ -1463,6 +1469,8 @@ static void nvmet_subsys_free(struct kref *ref)
 
 	WARN_ON_ONCE(!list_empty(&subsys->namespaces));
 
+	nvmet_passthru_subsys_free(subsys);
+
 	kfree(subsys->subsysnqn);
 	kfree_rcu(subsys->model, rcuhead);
 	kfree(subsys);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index deb996c90804..76c3a7cb9c89 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -20,6 +20,8 @@
 #include <linux/blkdev.h>
 #include <linux/radix-tree.h>
 
+#define NVMET_DEFAULT_VS		NVME_VS(1, 3, 0)
+
 #define NVMET_ASYNC_EVENTS		4
 #define NVMET_ERROR_LOG_SLOTS		128
 #define NVMET_NO_ERROR_LOC		((u16)-1)
@@ -240,6 +242,7 @@ struct nvmet_subsys {
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	struct nvme_ctrl	*passthru_ctrl;
+	char			*passthru_ctrl_path;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
@@ -523,6 +526,9 @@ static inline u32 nvmet_dsm_len(struct nvmet_req *req)
 }
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
+void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys);
+int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys);
+void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys);
 u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req);
 u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req);
 static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
@@ -530,6 +536,12 @@ static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
 	return subsys->passthru_ctrl;
 }
 #else /* CONFIG_NVME_TARGET_PASSTHRU */
+static inline void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
+{
+}
+static inline void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+}
 static inline u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
 {
 	return 0;
diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
index 15b23521e59f..a11c3ef34041 100644
--- a/drivers/nvme/target/passthru.c
+++ b/drivers/nvme/target/passthru.c
@@ -11,6 +11,11 @@
 #include "../host/nvme.h"
 #include "nvmet.h"
 
+/*
+ * xarray to maintain one passthru subsystem per nvme controller.
+ */
+static DEFINE_XARRAY(passthru_subsystems);
+
 static void nvmet_passthru_execute_cmd_work(struct work_struct *w)
 {
 	struct nvmet_req *req = container_of(w, struct nvmet_req, p.work);
@@ -458,3 +463,85 @@ u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
 		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
 	}
 }
+
+int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys)
+{
+	struct nvme_ctrl *ctrl;
+	int ret = -EINVAL;
+	void *old;
+
+	mutex_lock(&subsys->lock);
+	if (!subsys->passthru_ctrl_path)
+		goto out_unlock;
+	if (subsys->passthru_ctrl)
+		goto out_unlock;
+
+	if (subsys->nr_namespaces) {
+		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
+		goto out_unlock;
+	}
+
+	ctrl = nvme_ctrl_get_by_path(subsys->passthru_ctrl_path);
+	if (IS_ERR(ctrl)) {
+		ret = PTR_ERR(ctrl);
+		pr_err("failed to open nvme controller %s\n",
+		       subsys->passthru_ctrl_path);
+
+		goto out_unlock;
+	}
+
+	old = xa_cmpxchg(&passthru_subsystems, ctrl->cntlid, NULL,
+			 subsys, GFP_KERNEL);
+	if (xa_is_err(old)) {
+		ret = xa_err(old);
+		goto out_put_ctrl;
+	}
+
+	if (old)
+		goto out_put_ctrl;
+
+	subsys->passthru_ctrl = ctrl;
+	subsys->ver = ctrl->vs;
+
+	if (subsys->ver < NVME_VS(1, 2, 1)) {
+		pr_warn("nvme controller version is too old: %d.%d.%d, advertising 1.2.1\n",
+			(int)NVME_MAJOR(subsys->ver),
+			(int)NVME_MINOR(subsys->ver),
+			(int)NVME_TERTIARY(subsys->ver));
+		subsys->ver = NVME_VS(1, 2, 1);
+	}
+
+	mutex_unlock(&subsys->lock);
+	return 0;
+
+out_put_ctrl:
+	nvme_put_ctrl(ctrl);
+out_unlock:
+	mutex_unlock(&subsys->lock);
+	return ret;
+}
+
+static void __nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+	if (subsys->passthru_ctrl) {
+		xa_erase(&passthru_subsystems, subsys->passthru_ctrl->cntlid);
+		nvme_put_ctrl(subsys->passthru_ctrl);
+	}
+	subsys->passthru_ctrl = NULL;
+	subsys->ver = NVMET_DEFAULT_VS;
+}
+
+void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+	mutex_lock(&subsys->lock);
+	__nvmet_passthru_ctrl_disable(subsys);
+	mutex_unlock(&subsys->lock);
+}
+
+void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
+{
+	mutex_lock(&subsys->lock);
+	__nvmet_passthru_ctrl_disable(subsys);
+	mutex_unlock(&subsys->lock);
+	kfree(subsys->passthru_ctrl_path);
+}
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

This patch adds helper functions which are used in the NVMeOF configfs
when the user is configuring the passthru subsystem. Here we ensure
that only one subsys is assigned to each nvme_ctrl by using an xarray
on the cntlid.

The subsystem's version number is overridden by the passed through
controller's version. However, if that version is less than 1.2.1,
then we bump the advertised version to that and print a warning
in dmesg.

Based-on-a-patch-by: Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/configfs.c |  4 ++
 drivers/nvme/target/core.c     | 10 +++-
 drivers/nvme/target/nvmet.h    | 12 +++++
 drivers/nvme/target/passthru.c | 87 ++++++++++++++++++++++++++++++++++
 4 files changed, 112 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 58cabd7b6fc5..e0ce6e5feb3a 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -826,6 +826,10 @@ static ssize_t nvmet_subsys_attr_version_store(struct config_item *item,
 	int major, minor, tertiary = 0;
 	int ret;
 
+	/* passthru subsystems use the underlying controller's version */
+	if (nvmet_passthru_ctrl(subsys))
+		return -EINVAL;
+
 	ret = sscanf(page, "%d.%d.%d\n", &major, &minor, &tertiary);
 	if (ret != 2 && ret != 3)
 		return -EINVAL;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 2547e0d8951c..a2cdd35ffb1d 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -522,6 +522,12 @@ int nvmet_ns_enable(struct nvmet_ns *ns)
 
 	mutex_lock(&subsys->lock);
 	ret = 0;
+
+	if (nvmet_passthru_ctrl(subsys)) {
+		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
+		goto out_unlock;
+	}
+
 	if (ns->enabled)
 		goto out_unlock;
 
@@ -1421,7 +1427,7 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 	if (!subsys)
 		return ERR_PTR(-ENOMEM);
 
-	subsys->ver = NVME_VS(1, 3, 0); /* NVMe 1.3.0 */
+	subsys->ver = NVMET_DEFAULT_VS;
 	/* generate a random serial number as our controllers are ephemeral: */
 	get_random_bytes(&subsys->serial, sizeof(subsys->serial));
 
@@ -1463,6 +1469,8 @@ static void nvmet_subsys_free(struct kref *ref)
 
 	WARN_ON_ONCE(!list_empty(&subsys->namespaces));
 
+	nvmet_passthru_subsys_free(subsys);
+
 	kfree(subsys->subsysnqn);
 	kfree_rcu(subsys->model, rcuhead);
 	kfree(subsys);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index deb996c90804..76c3a7cb9c89 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -20,6 +20,8 @@
 #include <linux/blkdev.h>
 #include <linux/radix-tree.h>
 
+#define NVMET_DEFAULT_VS		NVME_VS(1, 3, 0)
+
 #define NVMET_ASYNC_EVENTS		4
 #define NVMET_ERROR_LOG_SLOTS		128
 #define NVMET_NO_ERROR_LOC		((u16)-1)
@@ -240,6 +242,7 @@ struct nvmet_subsys {
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	struct nvme_ctrl	*passthru_ctrl;
+	char			*passthru_ctrl_path;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
@@ -523,6 +526,9 @@ static inline u32 nvmet_dsm_len(struct nvmet_req *req)
 }
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
+void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys);
+int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys);
+void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys);
 u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req);
 u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req);
 static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
@@ -530,6 +536,12 @@ static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
 	return subsys->passthru_ctrl;
 }
 #else /* CONFIG_NVME_TARGET_PASSTHRU */
+static inline void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
+{
+}
+static inline void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+}
 static inline u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
 {
 	return 0;
diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
index 15b23521e59f..a11c3ef34041 100644
--- a/drivers/nvme/target/passthru.c
+++ b/drivers/nvme/target/passthru.c
@@ -11,6 +11,11 @@
 #include "../host/nvme.h"
 #include "nvmet.h"
 
+/*
+ * xarray to maintain one passthru subsystem per nvme controller.
+ */
+static DEFINE_XARRAY(passthru_subsystems);
+
 static void nvmet_passthru_execute_cmd_work(struct work_struct *w)
 {
 	struct nvmet_req *req = container_of(w, struct nvmet_req, p.work);
@@ -458,3 +463,85 @@ u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
 		return NVME_SC_INVALID_OPCODE | NVME_SC_DNR;
 	}
 }
+
+int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys)
+{
+	struct nvme_ctrl *ctrl;
+	int ret = -EINVAL;
+	void *old;
+
+	mutex_lock(&subsys->lock);
+	if (!subsys->passthru_ctrl_path)
+		goto out_unlock;
+	if (subsys->passthru_ctrl)
+		goto out_unlock;
+
+	if (subsys->nr_namespaces) {
+		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
+		goto out_unlock;
+	}
+
+	ctrl = nvme_ctrl_get_by_path(subsys->passthru_ctrl_path);
+	if (IS_ERR(ctrl)) {
+		ret = PTR_ERR(ctrl);
+		pr_err("failed to open nvme controller %s\n",
+		       subsys->passthru_ctrl_path);
+
+		goto out_unlock;
+	}
+
+	old = xa_cmpxchg(&passthru_subsystems, ctrl->cntlid, NULL,
+			 subsys, GFP_KERNEL);
+	if (xa_is_err(old)) {
+		ret = xa_err(old);
+		goto out_put_ctrl;
+	}
+
+	if (old)
+		goto out_put_ctrl;
+
+	subsys->passthru_ctrl = ctrl;
+	subsys->ver = ctrl->vs;
+
+	if (subsys->ver < NVME_VS(1, 2, 1)) {
+		pr_warn("nvme controller version is too old: %d.%d.%d, advertising 1.2.1\n",
+			(int)NVME_MAJOR(subsys->ver),
+			(int)NVME_MINOR(subsys->ver),
+			(int)NVME_TERTIARY(subsys->ver));
+		subsys->ver = NVME_VS(1, 2, 1);
+	}
+
+	mutex_unlock(&subsys->lock);
+	return 0;
+
+out_put_ctrl:
+	nvme_put_ctrl(ctrl);
+out_unlock:
+	mutex_unlock(&subsys->lock);
+	return ret;
+}
+
+static void __nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+	if (subsys->passthru_ctrl) {
+		xa_erase(&passthru_subsystems, subsys->passthru_ctrl->cntlid);
+		nvme_put_ctrl(subsys->passthru_ctrl);
+	}
+	subsys->passthru_ctrl = NULL;
+	subsys->ver = NVMET_DEFAULT_VS;
+}
+
+void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+	mutex_lock(&subsys->lock);
+	__nvmet_passthru_ctrl_disable(subsys);
+	mutex_unlock(&subsys->lock);
+}
+
+void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
+{
+	mutex_lock(&subsys->lock);
+	__nvmet_passthru_ctrl_disable(subsys);
+	mutex_unlock(&subsys->lock);
+	kfree(subsys->passthru_ctrl_path);
+}
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 9/9] nvmet-configfs: Introduce passthru configfs interface
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-05-14 17:22   ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Chaitanya Kulkarni, Max Gurtovoy, Stephen Bates, Logan Gunthorpe

When CONFIG_NVME_TARGET_PASSTHRU as 'passthru' directory will
be added to each subsystem. The directory is similar to a namespace
and has two attributes: device_path and enable. The user must set the
path to the nvme controller's char device and write '1' to enable the
subsystem to use passthru.

Any given subsystem is prevented from enabling both a regular namespace
and the passthru device. If one is enabled, enabling the other will
produce an error.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/configfs.c | 99 ++++++++++++++++++++++++++++++++++
 drivers/nvme/target/nvmet.h    |  1 +
 2 files changed, 100 insertions(+)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index e0ce6e5feb3a..59eb52a0d06c 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -613,6 +613,103 @@ static const struct config_item_type nvmet_namespaces_type = {
 	.ct_owner		= THIS_MODULE,
 };
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+
+static ssize_t nvmet_passthru_device_path_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return snprintf(page, PAGE_SIZE, "%s\n", subsys->passthru_ctrl_path);
+}
+
+static ssize_t nvmet_passthru_device_path_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	size_t len;
+	int ret;
+
+	mutex_lock(&subsys->lock);
+
+	ret = -EBUSY;
+	if (subsys->passthru_ctrl)
+		goto out_unlock;
+
+	ret = -EINVAL;
+	len = strcspn(page, "\n");
+	if (!len)
+		goto out_unlock;
+
+	kfree(subsys->passthru_ctrl_path);
+	ret = -ENOMEM;
+	subsys->passthru_ctrl_path = kstrndup(page, len, GFP_KERNEL);
+	if (!subsys->passthru_ctrl_path)
+		goto out_unlock;
+
+	mutex_unlock(&subsys->lock);
+
+	return count;
+out_unlock:
+	mutex_unlock(&subsys->lock);
+	return ret;
+}
+CONFIGFS_ATTR(nvmet_passthru_, device_path);
+
+static ssize_t nvmet_passthru_enable_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return sprintf(page, "%d\n", subsys->passthru_ctrl ? 1 : 0);
+}
+
+static ssize_t nvmet_passthru_enable_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	bool enable;
+	int ret = 0;
+
+	if (strtobool(page, &enable))
+		return -EINVAL;
+
+	if (enable)
+		ret = nvmet_passthru_ctrl_enable(subsys);
+	else
+		nvmet_passthru_ctrl_disable(subsys);
+
+	return ret ? ret : count;
+}
+CONFIGFS_ATTR(nvmet_passthru_, enable);
+
+static struct configfs_attribute *nvmet_passthru_attrs[] = {
+	&nvmet_passthru_attr_device_path,
+	&nvmet_passthru_attr_enable,
+	NULL,
+};
+
+static const struct config_item_type nvmet_passthru_type = {
+	.ct_attrs		= nvmet_passthru_attrs,
+	.ct_owner		= THIS_MODULE,
+};
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+	config_group_init_type_name(&subsys->passthru_group,
+				    "passthru", &nvmet_passthru_type);
+	configfs_add_default_group(&subsys->passthru_group,
+				   &subsys->group);
+}
+
+#else /* CONFIG_NVME_TARGET_PASSTHRU */
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+}
+
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 static int nvmet_port_subsys_allow_link(struct config_item *parent,
 		struct config_item *target)
 {
@@ -1047,6 +1144,8 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
 	configfs_add_default_group(&subsys->allowed_hosts_group,
 			&subsys->group);
 
+	nvmet_add_passthru_group(subsys);
+
 	return &subsys->group;
 }
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 76c3a7cb9c89..d40452e50212 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -243,6 +243,7 @@ struct nvmet_subsys {
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	struct nvme_ctrl	*passthru_ctrl;
 	char			*passthru_ctrl_path;
+	struct config_group	passthru_group;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* [PATCH v13 9/9] nvmet-configfs: Introduce passthru configfs interface
@ 2020-05-14 17:22   ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-05-14 17:22 UTC (permalink / raw)
  To: linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Chaitanya Kulkarni, Stephen Bates, Jens Axboe,
	Keith Busch, Max Gurtovoy, Logan Gunthorpe, Christoph Hellwig

When CONFIG_NVME_TARGET_PASSTHRU as 'passthru' directory will
be added to each subsystem. The directory is similar to a namespace
and has two attributes: device_path and enable. The user must set the
path to the nvme controller's char device and write '1' to enable the
subsystem to use passthru.

Any given subsystem is prevented from enabling both a regular namespace
and the passthru device. If one is enabled, enabling the other will
produce an error.

Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
---
 drivers/nvme/target/configfs.c | 99 ++++++++++++++++++++++++++++++++++
 drivers/nvme/target/nvmet.h    |  1 +
 2 files changed, 100 insertions(+)

diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index e0ce6e5feb3a..59eb52a0d06c 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -613,6 +613,103 @@ static const struct config_item_type nvmet_namespaces_type = {
 	.ct_owner		= THIS_MODULE,
 };
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+
+static ssize_t nvmet_passthru_device_path_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return snprintf(page, PAGE_SIZE, "%s\n", subsys->passthru_ctrl_path);
+}
+
+static ssize_t nvmet_passthru_device_path_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	size_t len;
+	int ret;
+
+	mutex_lock(&subsys->lock);
+
+	ret = -EBUSY;
+	if (subsys->passthru_ctrl)
+		goto out_unlock;
+
+	ret = -EINVAL;
+	len = strcspn(page, "\n");
+	if (!len)
+		goto out_unlock;
+
+	kfree(subsys->passthru_ctrl_path);
+	ret = -ENOMEM;
+	subsys->passthru_ctrl_path = kstrndup(page, len, GFP_KERNEL);
+	if (!subsys->passthru_ctrl_path)
+		goto out_unlock;
+
+	mutex_unlock(&subsys->lock);
+
+	return count;
+out_unlock:
+	mutex_unlock(&subsys->lock);
+	return ret;
+}
+CONFIGFS_ATTR(nvmet_passthru_, device_path);
+
+static ssize_t nvmet_passthru_enable_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return sprintf(page, "%d\n", subsys->passthru_ctrl ? 1 : 0);
+}
+
+static ssize_t nvmet_passthru_enable_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	bool enable;
+	int ret = 0;
+
+	if (strtobool(page, &enable))
+		return -EINVAL;
+
+	if (enable)
+		ret = nvmet_passthru_ctrl_enable(subsys);
+	else
+		nvmet_passthru_ctrl_disable(subsys);
+
+	return ret ? ret : count;
+}
+CONFIGFS_ATTR(nvmet_passthru_, enable);
+
+static struct configfs_attribute *nvmet_passthru_attrs[] = {
+	&nvmet_passthru_attr_device_path,
+	&nvmet_passthru_attr_enable,
+	NULL,
+};
+
+static const struct config_item_type nvmet_passthru_type = {
+	.ct_attrs		= nvmet_passthru_attrs,
+	.ct_owner		= THIS_MODULE,
+};
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+	config_group_init_type_name(&subsys->passthru_group,
+				    "passthru", &nvmet_passthru_type);
+	configfs_add_default_group(&subsys->passthru_group,
+				   &subsys->group);
+}
+
+#else /* CONFIG_NVME_TARGET_PASSTHRU */
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+}
+
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 static int nvmet_port_subsys_allow_link(struct config_item *parent,
 		struct config_item *target)
 {
@@ -1047,6 +1144,8 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
 	configfs_add_default_group(&subsys->allowed_hosts_group,
 			&subsys->group);
 
+	nvmet_add_passthru_group(subsys);
+
 	return &subsys->group;
 }
 
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 76c3a7cb9c89..d40452e50212 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -243,6 +243,7 @@ struct nvmet_subsys {
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	struct nvme_ctrl	*passthru_ctrl;
 	char			*passthru_ctrl_path;
+	struct config_group	passthru_group;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
-- 
2.20.1


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 0/9] nvmet: add target passthru commands support
  2020-05-14 17:22 ` Logan Gunthorpe
@ 2020-06-11 19:16   ` Chaitanya Kulkarni
  -1 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 19:16 UTC (permalink / raw)
  To: Logan Gunthorpe
  Cc: linux-kernel, linux-nvme, Christoph Hellwig, Sagi Grimberg,
	Keith Busch, Jens Axboe, Max Gurtovoy, Stephen Bates

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> This is v13 of the passthru patchset which is mostly a resend of v12
> with Sagi's reviewed-by tags collected.
> 
> Below, I'll reiterrate some points I made previously that haven't been
> responded to:
> 
> I don't think cloning the ctrl_id or the subsysnqn is a good idea.
> I sent an email trying to explain why here[1] but there was no response.
> In short, I think cloning the ctrl_id will break multipathing over
> fabrics and copying the subsysnqn only has the effect of breaking
> loopback; the user can always copy the underlying subsysnqn if it
> makes sense for their overall system.
> 
> I maintain overriding the CMIC bit in the ctrl id is necessary to
> allow multipath over fabrics even if the underlying device did
> not support multipath.
> 
> I also think the black list for admin commands is appropriate, and I
> added it based on Sagi's feedback[2]. There are plenty of commands that
> may be dangerous like firmware update and format NVM commands, and NS
> attach commands won't work out of the box because we don't copy the
> ctrl_id. It seems like there's more commands to be careful of than ones
> that are that are obviously acceptable. So, I think the prudent course
> is blacklisting by default until someone has a usecase and can show
> the command is safe seems and makes sense. For our present use cases,
> the identify, log page and vendor specific commands are all that we
> care about.
> 
> A git branch is available here and is based on v5.7-rc5:
> 
> https://github.com/sbates130272/linux-p2pmem  nvmet_passthru_v13
> 
> [1]https://lore.kernel.org/linux-block/247eca47-c3bc-6452-fb19-f7aa27b05a60@deltatee.com/
> [2]https://lore.kernel.org/linux-block/e4430207-7def-8776-0289-0d58689dc0cd@grimberg.me/
> 
> --
> 
> v13 Changes:
>    1. Rebased onto v5.7-rc5
>    2. Collected Sagi's Reviewed-by tags

Are you planning to send V14 based on nvme-5.9 branch ?

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 0/9] nvmet: add target passthru commands support
@ 2020-06-11 19:16   ` Chaitanya Kulkarni
  0 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 19:16 UTC (permalink / raw)
  To: Logan Gunthorpe
  Cc: Sagi Grimberg, linux-kernel, linux-nvme, Stephen Bates,
	Jens Axboe, Keith Busch, Max Gurtovoy, Christoph Hellwig

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> This is v13 of the passthru patchset which is mostly a resend of v12
> with Sagi's reviewed-by tags collected.
> 
> Below, I'll reiterrate some points I made previously that haven't been
> responded to:
> 
> I don't think cloning the ctrl_id or the subsysnqn is a good idea.
> I sent an email trying to explain why here[1] but there was no response.
> In short, I think cloning the ctrl_id will break multipathing over
> fabrics and copying the subsysnqn only has the effect of breaking
> loopback; the user can always copy the underlying subsysnqn if it
> makes sense for their overall system.
> 
> I maintain overriding the CMIC bit in the ctrl id is necessary to
> allow multipath over fabrics even if the underlying device did
> not support multipath.
> 
> I also think the black list for admin commands is appropriate, and I
> added it based on Sagi's feedback[2]. There are plenty of commands that
> may be dangerous like firmware update and format NVM commands, and NS
> attach commands won't work out of the box because we don't copy the
> ctrl_id. It seems like there's more commands to be careful of than ones
> that are that are obviously acceptable. So, I think the prudent course
> is blacklisting by default until someone has a usecase and can show
> the command is safe seems and makes sense. For our present use cases,
> the identify, log page and vendor specific commands are all that we
> care about.
> 
> A git branch is available here and is based on v5.7-rc5:
> 
> https://github.com/sbates130272/linux-p2pmem  nvmet_passthru_v13
> 
> [1]https://lore.kernel.org/linux-block/247eca47-c3bc-6452-fb19-f7aa27b05a60@deltatee.com/
> [2]https://lore.kernel.org/linux-block/e4430207-7def-8776-0289-0d58689dc0cd@grimberg.me/
> 
> --
> 
> v13 Changes:
>    1. Rebased onto v5.7-rc5
>    2. Collected Sagi's Reviewed-by tags

Are you planning to send V14 based on nvme-5.9 branch ?

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 0/9] nvmet: add target passthru commands support
  2020-06-11 19:16   ` Chaitanya Kulkarni
@ 2020-06-11 19:19     ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 19:19 UTC (permalink / raw)
  To: Chaitanya Kulkarni
  Cc: linux-kernel, linux-nvme, Christoph Hellwig, Sagi Grimberg,
	Keith Busch, Jens Axboe, Max Gurtovoy, Stephen Bates



On 2020-06-11 1:16 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> This is v13 of the passthru patchset which is mostly a resend of v12
>> with Sagi's reviewed-by tags collected.
>>
>> Below, I'll reiterrate some points I made previously that haven't been
>> responded to:
>>
>> I don't think cloning the ctrl_id or the subsysnqn is a good idea.
>> I sent an email trying to explain why here[1] but there was no response.
>> In short, I think cloning the ctrl_id will break multipathing over
>> fabrics and copying the subsysnqn only has the effect of breaking
>> loopback; the user can always copy the underlying subsysnqn if it
>> makes sense for their overall system.
>>
>> I maintain overriding the CMIC bit in the ctrl id is necessary to
>> allow multipath over fabrics even if the underlying device did
>> not support multipath.
>>
>> I also think the black list for admin commands is appropriate, and I
>> added it based on Sagi's feedback[2]. There are plenty of commands that
>> may be dangerous like firmware update and format NVM commands, and NS
>> attach commands won't work out of the box because we don't copy the
>> ctrl_id. It seems like there's more commands to be careful of than ones
>> that are that are obviously acceptable. So, I think the prudent course
>> is blacklisting by default until someone has a usecase and can show
>> the command is safe seems and makes sense. For our present use cases,
>> the identify, log page and vendor specific commands are all that we
>> care about.
>>
>> A git branch is available here and is based on v5.7-rc5:
>>
>> https://github.com/sbates130272/linux-p2pmem  nvmet_passthru_v13
>>
>> [1]https://lore.kernel.org/linux-block/247eca47-c3bc-6452-fb19-f7aa27b05a60@deltatee.com/
>> [2]https://lore.kernel.org/linux-block/e4430207-7def-8776-0289-0d58689dc0cd@grimberg.me/
>>
>> --
>>
>> v13 Changes:
>>    1. Rebased onto v5.7-rc5
>>    2. Collected Sagi's Reviewed-by tags
> 
> Are you planning to send V14 based on nvme-5.9 branch ?
> 

I don't have any plans to yet, but I certainly can. I usually wait until
after the merge window to resend. I can certainly look at rebasing onto
nvme-5.9 for next time.

Logan

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 0/9] nvmet: add target passthru commands support
@ 2020-06-11 19:19     ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 19:19 UTC (permalink / raw)
  To: Chaitanya Kulkarni
  Cc: Sagi Grimberg, linux-kernel, linux-nvme, Stephen Bates,
	Jens Axboe, Keith Busch, Max Gurtovoy, Christoph Hellwig



On 2020-06-11 1:16 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> This is v13 of the passthru patchset which is mostly a resend of v12
>> with Sagi's reviewed-by tags collected.
>>
>> Below, I'll reiterrate some points I made previously that haven't been
>> responded to:
>>
>> I don't think cloning the ctrl_id or the subsysnqn is a good idea.
>> I sent an email trying to explain why here[1] but there was no response.
>> In short, I think cloning the ctrl_id will break multipathing over
>> fabrics and copying the subsysnqn only has the effect of breaking
>> loopback; the user can always copy the underlying subsysnqn if it
>> makes sense for their overall system.
>>
>> I maintain overriding the CMIC bit in the ctrl id is necessary to
>> allow multipath over fabrics even if the underlying device did
>> not support multipath.
>>
>> I also think the black list for admin commands is appropriate, and I
>> added it based on Sagi's feedback[2]. There are plenty of commands that
>> may be dangerous like firmware update and format NVM commands, and NS
>> attach commands won't work out of the box because we don't copy the
>> ctrl_id. It seems like there's more commands to be careful of than ones
>> that are that are obviously acceptable. So, I think the prudent course
>> is blacklisting by default until someone has a usecase and can show
>> the command is safe seems and makes sense. For our present use cases,
>> the identify, log page and vendor specific commands are all that we
>> care about.
>>
>> A git branch is available here and is based on v5.7-rc5:
>>
>> https://github.com/sbates130272/linux-p2pmem  nvmet_passthru_v13
>>
>> [1]https://lore.kernel.org/linux-block/247eca47-c3bc-6452-fb19-f7aa27b05a60@deltatee.com/
>> [2]https://lore.kernel.org/linux-block/e4430207-7def-8776-0289-0d58689dc0cd@grimberg.me/
>>
>> --
>>
>> v13 Changes:
>>    1. Rebased onto v5.7-rc5
>>    2. Collected Sagi's Reviewed-by tags
> 
> Are you planning to send V14 based on nvme-5.9 branch ?
> 

I don't have any plans to yet, but I certainly can. I usually wait until
after the merge window to resend. I can certainly look at rebasing onto
nvme-5.9 for next time.

Logan

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 2/9] nvme: Create helper function to obtain command effects
  2020-05-14 17:22   ` Logan Gunthorpe
@ 2020-06-11 22:56     ` Chaitanya Kulkarni
  -1 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 22:56 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> Separate the code to obtain command effects from the code
> to start a passthru request and open code nvme_known_admin_effects()
> in the new helper.
> 
> The new helper function will be necessary for nvmet passthru
> code to determine if we need to change out of interrupt context
> to handle the effects.
> 
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
> Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
> ---
>   drivers/nvme/host/core.c | 39 ++++++++++++++++++++++-----------------
>   1 file changed, 22 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index d22859543e4b..5062a83c3634 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -1317,22 +1317,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
>   			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
>   }
>   
> -static u32 nvme_known_admin_effects(u8 opcode)
> -{
> -	switch (opcode) {
> -	case nvme_admin_format_nvm:
> -		return NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
> -					NVME_CMD_EFFECTS_CSE_MASK;
> -	case nvme_admin_sanitize_nvm:
> -		return NVME_CMD_EFFECTS_CSE_MASK;
> -	default:
> -		break;
> -	}
> -	return 0;
> -}
> -
> -static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> -								u8 opcode)
> +static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> +				u8 opcode)
>   {
>   	u32 effects = 0;
>   
> @@ -1348,7 +1334,26 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>   
>   	if (ctrl->effects)
>   		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
> -	effects |= nvme_known_admin_effects(opcode);
> +
> +	switch (opcode) {
> +	case nvme_admin_format_nvm:
> +		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
> +			NVME_CMD_EFFECTS_CSE_MASK;
> +		break;
> +	case nvme_admin_sanitize_nvm:
> +		effects |= NVME_CMD_EFFECTS_CSE_MASK;
> +		break;
> +	default:
> +		break;
> +	}
> +
> +	return effects;
> +}
> +
> +static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> +			       u8 opcode)
> +{
> +	u32 effects = nvme_command_effects(ctrl, ns, opcode);
>   
>   	/*
>   	 * For simplicity, IO to all namespaces is quiesced even if the command
> 

Seems like you have changed the existing function body from
returning from switch to returning at the end of the function along with
the name that is why diff is large, which also adds an extra variable 
named "effect".

How about following ? which keeps the diff small and removes the extra 
variable and keeps the existing code as it.

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 78fc38b8356f..f47013fdc5ee 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1326,7 +1326,8 @@ static int nvme_submit_io(struct nvme_ns *ns, 
struct nvme_user_io __user *uio)
                         metadata, meta_len, lower_32_bits(io.slba), 
NULL, 0);
  }

-static u32 nvme_known_admin_effects(u8 opcode)
+static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+                               u8 opcode)
  {
         switch (opcode) {
         case nvme_admin_format_nvm:
@@ -1343,7 +1344,7 @@ static u32 nvme_known_admin_effects(u8 opcode)
  static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
                                                                 u8 opcode)
  {
-       u32 effects = 0;
+       u32 effects = nvme_command_effects(ctrl, ns, opcode);

         if (ns) {
                 if (ctrl->effects)
@@ -1357,7 +1358,6 @@ static u32 nvme_passthru_start(struct nvme_ctrl 
*ctrl, struct nvme_ns *ns,

         if (ctrl->effects)
                 effects = le32_to_cpu(ctrl->effects->acs[opcode]);
-       effects |= nvme_known_admin_effects(opcode);

         /*
          * For simplicity, IO to all namespaces is quiesced even if the 
command
-- 
2.22.1

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 2/9] nvme: Create helper function to obtain command effects
@ 2020-06-11 22:56     ` Chaitanya Kulkarni
  0 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 22:56 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> Separate the code to obtain command effects from the code
> to start a passthru request and open code nvme_known_admin_effects()
> in the new helper.
> 
> The new helper function will be necessary for nvmet passthru
> code to determine if we need to change out of interrupt context
> to handle the effects.
> 
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
> Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
> ---
>   drivers/nvme/host/core.c | 39 ++++++++++++++++++++++-----------------
>   1 file changed, 22 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index d22859543e4b..5062a83c3634 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -1317,22 +1317,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
>   			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
>   }
>   
> -static u32 nvme_known_admin_effects(u8 opcode)
> -{
> -	switch (opcode) {
> -	case nvme_admin_format_nvm:
> -		return NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
> -					NVME_CMD_EFFECTS_CSE_MASK;
> -	case nvme_admin_sanitize_nvm:
> -		return NVME_CMD_EFFECTS_CSE_MASK;
> -	default:
> -		break;
> -	}
> -	return 0;
> -}
> -
> -static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> -								u8 opcode)
> +static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> +				u8 opcode)
>   {
>   	u32 effects = 0;
>   
> @@ -1348,7 +1334,26 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>   
>   	if (ctrl->effects)
>   		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
> -	effects |= nvme_known_admin_effects(opcode);
> +
> +	switch (opcode) {
> +	case nvme_admin_format_nvm:
> +		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
> +			NVME_CMD_EFFECTS_CSE_MASK;
> +		break;
> +	case nvme_admin_sanitize_nvm:
> +		effects |= NVME_CMD_EFFECTS_CSE_MASK;
> +		break;
> +	default:
> +		break;
> +	}
> +
> +	return effects;
> +}
> +
> +static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> +			       u8 opcode)
> +{
> +	u32 effects = nvme_command_effects(ctrl, ns, opcode);
>   
>   	/*
>   	 * For simplicity, IO to all namespaces is quiesced even if the command
> 

Seems like you have changed the existing function body from
returning from switch to returning at the end of the function along with
the name that is why diff is large, which also adds an extra variable 
named "effect".

How about following ? which keeps the diff small and removes the extra 
variable and keeps the existing code as it.

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 78fc38b8356f..f47013fdc5ee 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1326,7 +1326,8 @@ static int nvme_submit_io(struct nvme_ns *ns, 
struct nvme_user_io __user *uio)
                         metadata, meta_len, lower_32_bits(io.slba), 
NULL, 0);
  }

-static u32 nvme_known_admin_effects(u8 opcode)
+static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+                               u8 opcode)
  {
         switch (opcode) {
         case nvme_admin_format_nvm:
@@ -1343,7 +1344,7 @@ static u32 nvme_known_admin_effects(u8 opcode)
  static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
                                                                 u8 opcode)
  {
-       u32 effects = 0;
+       u32 effects = nvme_command_effects(ctrl, ns, opcode);

         if (ns) {
                 if (ctrl->effects)
@@ -1357,7 +1358,6 @@ static u32 nvme_passthru_start(struct nvme_ctrl 
*ctrl, struct nvme_ns *ns,

         if (ctrl->effects)
                 effects = le32_to_cpu(ctrl->effects->acs[opcode]);
-       effects |= nvme_known_admin_effects(opcode);

         /*
          * For simplicity, IO to all namespaces is quiesced even if the 
command
-- 
2.22.1

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 5/9] nvme-core: Introduce nvme_ctrl_get_by_path()
  2020-05-14 17:22   ` Logan Gunthorpe
@ 2020-06-11 23:02     ` Chaitanya Kulkarni
  -1 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:02 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> +#ifdef CONFIG_NVME_TARGET_PASSTHRU
> +/*
> + * The exports that follow within this ifdef are only for
> + * use by the nvmet-passthru and should not be used for
> + * other things.
> + */

The above comment is duplicate #ifdef is self explanatory and I didn't
find similar style in existing repo  (e.g. CONFIG_NVME_MULTIPATH,
CONFIG_BLK_SED_OPAL) so let's not introduce new style which will create
confusion to future code.

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 5/9] nvme-core: Introduce nvme_ctrl_get_by_path()
@ 2020-06-11 23:02     ` Chaitanya Kulkarni
  0 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:02 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> +#ifdef CONFIG_NVME_TARGET_PASSTHRU
> +/*
> + * The exports that follow within this ifdef are only for
> + * use by the nvmet-passthru and should not be used for
> + * other things.
> + */

The above comment is duplicate #ifdef is self explanatory and I didn't
find similar style in existing repo  (e.g. CONFIG_NVME_MULTIPATH,
CONFIG_BLK_SED_OPAL) so let's not introduce new style which will create
confusion to future code.

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
  2020-05-14 17:22   ` Logan Gunthorpe
@ 2020-06-11 23:05     ` Chaitanya Kulkarni
  -1 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:05 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> +	if (subsys->nr_namespaces) {
> +		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");

Let's try and keep the error message witin 80 char per line or split the
message into two pr_info() calls,how about this ?


pr_info("cannot enable passthru & regular namespaces\n")


^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
@ 2020-06-11 23:05     ` Chaitanya Kulkarni
  0 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:05 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> +	if (subsys->nr_namespaces) {
> +		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");

Let's try and keep the error message witin 80 char per line or split the
message into two pr_info() calls,how about this ?


pr_info("cannot enable passthru & regular namespaces\n")


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 2/9] nvme: Create helper function to obtain command effects
  2020-06-11 22:56     ` Chaitanya Kulkarni
@ 2020-06-11 23:09       ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:09 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates



On 2020-06-11 4:56 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> Separate the code to obtain command effects from the code
>> to start a passthru request and open code nvme_known_admin_effects()
>> in the new helper.
>>
>> The new helper function will be necessary for nvmet passthru
>> code to determine if we need to change out of interrupt context
>> to handle the effects.
>>
>> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
>> Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
>> ---
>>   drivers/nvme/host/core.c | 39 ++++++++++++++++++++++-----------------
>>   1 file changed, 22 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index d22859543e4b..5062a83c3634 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -1317,22 +1317,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
>>   			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
>>   }
>>   
>> -static u32 nvme_known_admin_effects(u8 opcode)
>> -{
>> -	switch (opcode) {
>> -	case nvme_admin_format_nvm:
>> -		return NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
>> -					NVME_CMD_EFFECTS_CSE_MASK;
>> -	case nvme_admin_sanitize_nvm:
>> -		return NVME_CMD_EFFECTS_CSE_MASK;
>> -	default:
>> -		break;
>> -	}
>> -	return 0;
>> -}
>> -
>> -static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>> -								u8 opcode)
>> +static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>> +				u8 opcode)
>>   {
>>   	u32 effects = 0;
>>   
>> @@ -1348,7 +1334,26 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>>   
>>   	if (ctrl->effects)
>>   		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
>> -	effects |= nvme_known_admin_effects(opcode);
>> +
>> +	switch (opcode) {
>> +	case nvme_admin_format_nvm:
>> +		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
>> +			NVME_CMD_EFFECTS_CSE_MASK;
>> +		break;
>> +	case nvme_admin_sanitize_nvm:
>> +		effects |= NVME_CMD_EFFECTS_CSE_MASK;
>> +		break;
>> +	default:
>> +		break;
>> +	}
>> +
>> +	return effects;
>> +}
>> +
>> +static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>> +			       u8 opcode)
>> +{
>> +	u32 effects = nvme_command_effects(ctrl, ns, opcode);
>>   
>>   	/*
>>   	 * For simplicity, IO to all namespaces is quiesced even if the command
>>
> 
> Seems like you have changed the existing function body from
> returning from switch to returning at the end of the function along with
> the name that is why diff is large, which also adds an extra variable 
> named "effect".
> 
> How about following ? which keeps the diff small and removes the extra 
> variable and keeps the existing code as it.

That doesn't really get what's needed for the new helper. We need a new
helper that gets the effects as nvme_passthru_start() sees them
(including ctrl->effects->acs[opcode]). Changing the name of
nvme_known_admin_effects() is not equivalent.

This patch would be possible to do without open coding
nvme_known_admin_effects() in the new helper, but doing so helps to
clean things up a bit as nvme_known_admin_effects() isn't used anywhere
else.

Logan


> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 78fc38b8356f..f47013fdc5ee 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -1326,7 +1326,8 @@ static int nvme_submit_io(struct nvme_ns *ns, 
> struct nvme_user_io __user *uio)
>                          metadata, meta_len, lower_32_bits(io.slba), 
> NULL, 0);
>   }
> 
> -static u32 nvme_known_admin_effects(u8 opcode)
> +static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> +                               u8 opcode)
>   {
>          switch (opcode) {
>          case nvme_admin_format_nvm:
> @@ -1343,7 +1344,7 @@ static u32 nvme_known_admin_effects(u8 opcode)
>   static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>                                                                  u8 opcode)
>   {
> -       u32 effects = 0;
> +       u32 effects = nvme_command_effects(ctrl, ns, opcode);
> 
>          if (ns) {
>                  if (ctrl->effects)
> @@ -1357,7 +1358,6 @@ static u32 nvme_passthru_start(struct nvme_ctrl 
> *ctrl, struct nvme_ns *ns,
> 
>          if (ctrl->effects)
>                  effects = le32_to_cpu(ctrl->effects->acs[opcode]);
> -       effects |= nvme_known_admin_effects(opcode);
> 
>          /*
>           * For simplicity, IO to all namespaces is quiesced even if the 
> command
> 

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 2/9] nvme: Create helper function to obtain command effects
@ 2020-06-11 23:09       ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:09 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig



On 2020-06-11 4:56 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> Separate the code to obtain command effects from the code
>> to start a passthru request and open code nvme_known_admin_effects()
>> in the new helper.
>>
>> The new helper function will be necessary for nvmet passthru
>> code to determine if we need to change out of interrupt context
>> to handle the effects.
>>
>> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
>> Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
>> ---
>>   drivers/nvme/host/core.c | 39 ++++++++++++++++++++++-----------------
>>   1 file changed, 22 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
>> index d22859543e4b..5062a83c3634 100644
>> --- a/drivers/nvme/host/core.c
>> +++ b/drivers/nvme/host/core.c
>> @@ -1317,22 +1317,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
>>   			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
>>   }
>>   
>> -static u32 nvme_known_admin_effects(u8 opcode)
>> -{
>> -	switch (opcode) {
>> -	case nvme_admin_format_nvm:
>> -		return NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
>> -					NVME_CMD_EFFECTS_CSE_MASK;
>> -	case nvme_admin_sanitize_nvm:
>> -		return NVME_CMD_EFFECTS_CSE_MASK;
>> -	default:
>> -		break;
>> -	}
>> -	return 0;
>> -}
>> -
>> -static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>> -								u8 opcode)
>> +static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>> +				u8 opcode)
>>   {
>>   	u32 effects = 0;
>>   
>> @@ -1348,7 +1334,26 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>>   
>>   	if (ctrl->effects)
>>   		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
>> -	effects |= nvme_known_admin_effects(opcode);
>> +
>> +	switch (opcode) {
>> +	case nvme_admin_format_nvm:
>> +		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
>> +			NVME_CMD_EFFECTS_CSE_MASK;
>> +		break;
>> +	case nvme_admin_sanitize_nvm:
>> +		effects |= NVME_CMD_EFFECTS_CSE_MASK;
>> +		break;
>> +	default:
>> +		break;
>> +	}
>> +
>> +	return effects;
>> +}
>> +
>> +static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>> +			       u8 opcode)
>> +{
>> +	u32 effects = nvme_command_effects(ctrl, ns, opcode);
>>   
>>   	/*
>>   	 * For simplicity, IO to all namespaces is quiesced even if the command
>>
> 
> Seems like you have changed the existing function body from
> returning from switch to returning at the end of the function along with
> the name that is why diff is large, which also adds an extra variable 
> named "effect".
> 
> How about following ? which keeps the diff small and removes the extra 
> variable and keeps the existing code as it.

That doesn't really get what's needed for the new helper. We need a new
helper that gets the effects as nvme_passthru_start() sees them
(including ctrl->effects->acs[opcode]). Changing the name of
nvme_known_admin_effects() is not equivalent.

This patch would be possible to do without open coding
nvme_known_admin_effects() in the new helper, but doing so helps to
clean things up a bit as nvme_known_admin_effects() isn't used anywhere
else.

Logan


> 
> diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
> index 78fc38b8356f..f47013fdc5ee 100644
> --- a/drivers/nvme/host/core.c
> +++ b/drivers/nvme/host/core.c
> @@ -1326,7 +1326,8 @@ static int nvme_submit_io(struct nvme_ns *ns, 
> struct nvme_user_io __user *uio)
>                          metadata, meta_len, lower_32_bits(io.slba), 
> NULL, 0);
>   }
> 
> -static u32 nvme_known_admin_effects(u8 opcode)
> +static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
> +                               u8 opcode)
>   {
>          switch (opcode) {
>          case nvme_admin_format_nvm:
> @@ -1343,7 +1344,7 @@ static u32 nvme_known_admin_effects(u8 opcode)
>   static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
>                                                                  u8 opcode)
>   {
> -       u32 effects = 0;
> +       u32 effects = nvme_command_effects(ctrl, ns, opcode);
> 
>          if (ns) {
>                  if (ctrl->effects)
> @@ -1357,7 +1358,6 @@ static u32 nvme_passthru_start(struct nvme_ctrl 
> *ctrl, struct nvme_ns *ns,
> 
>          if (ctrl->effects)
>                  effects = le32_to_cpu(ctrl->effects->acs[opcode]);
> -       effects |= nvme_known_admin_effects(opcode);
> 
>          /*
>           * For simplicity, IO to all namespaces is quiesced even if the 
> command
> 

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 5/9] nvme-core: Introduce nvme_ctrl_get_by_path()
  2020-06-11 23:02     ` Chaitanya Kulkarni
@ 2020-06-11 23:09       ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:09 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates



On 2020-06-11 5:02 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> +#ifdef CONFIG_NVME_TARGET_PASSTHRU
>> +/*
>> + * The exports that follow within this ifdef are only for
>> + * use by the nvmet-passthru and should not be used for
>> + * other things.
>> + */
> 
> The above comment is duplicate #ifdef is self explanatory and I didn't
> find similar style in existing repo  (e.g. CONFIG_NVME_MULTIPATH,
> CONFIG_BLK_SED_OPAL) so let's not introduce new style which will create
> confusion to future code.

Christoph specifically asked for this ifdef and comment.

Logan

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 5/9] nvme-core: Introduce nvme_ctrl_get_by_path()
@ 2020-06-11 23:09       ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:09 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig



On 2020-06-11 5:02 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> +#ifdef CONFIG_NVME_TARGET_PASSTHRU
>> +/*
>> + * The exports that follow within this ifdef are only for
>> + * use by the nvmet-passthru and should not be used for
>> + * other things.
>> + */
> 
> The above comment is duplicate #ifdef is self explanatory and I didn't
> find similar style in existing repo  (e.g. CONFIG_NVME_MULTIPATH,
> CONFIG_BLK_SED_OPAL) so let's not introduce new style which will create
> confusion to future code.

Christoph specifically asked for this ifdef and comment.

Logan

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
  2020-05-14 17:22   ` Logan Gunthorpe
@ 2020-06-11 23:13     ` Chaitanya Kulkarni
  -1 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:13 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> +	if (subsys->ver < NVME_VS(1, 2, 1)) {
> +		pr_warn("nvme controller version is too old: %d.%d.%d, advertising 1.2.1\n",
Is it more than 80 char ? can it be ?
                  pr_warn("nvme controller version is too old: ");
                  pr_warn("%d.%d.%d, advertising 1.2.1\n",
> +			(int)NVME_MAJOR(subsys->ver),
> +			(int)NVME_MINOR(subsys->ver),
> +			(int)NVME_TERTIARY(subsys->ver));
> +		subsys->ver = NVME_VS(1, 2, 1);
> +	}

NVMe blktests are running on QEMU based controller. This will generate 
warning every-time.

Also, I didn't understand int type cast, I wonder under what condition 
all these macros will return -ve values since ver of type u64 ?

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
@ 2020-06-11 23:13     ` Chaitanya Kulkarni
  0 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:13 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> +	if (subsys->ver < NVME_VS(1, 2, 1)) {
> +		pr_warn("nvme controller version is too old: %d.%d.%d, advertising 1.2.1\n",
Is it more than 80 char ? can it be ?
                  pr_warn("nvme controller version is too old: ");
                  pr_warn("%d.%d.%d, advertising 1.2.1\n",
> +			(int)NVME_MAJOR(subsys->ver),
> +			(int)NVME_MINOR(subsys->ver),
> +			(int)NVME_TERTIARY(subsys->ver));
> +		subsys->ver = NVME_VS(1, 2, 1);
> +	}

NVMe blktests are running on QEMU based controller. This will generate 
warning every-time.

Also, I didn't understand int type cast, I wonder under what condition 
all these macros will return -ve values since ver of type u64 ?

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
  2020-06-11 23:05     ` Chaitanya Kulkarni
@ 2020-06-11 23:14       ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:14 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates



On 2020-06-11 5:05 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> +	if (subsys->nr_namespaces) {
>> +		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
> 
> Let's try and keep the error message witin 80 char per line or split the
> message into two pr_info() calls,how about this ?
> 
> 
> pr_info("cannot enable passthru & regular namespaces\n")

Honestly, I think that is too brief. The error message is only 74 chars
and there's a long standing exception for long lines in the kernel for
printks. Even Linus has recently suggested that keeping to the 80 char
limit is not recommended when it harms readability (though I don't
generally agree with this 100%)[1].

Logan

[1]
https://lwn.net/ml/linux-kernel/CAHk-=wjR0H3+2ba0UUWwoYzYBH0GX9yTf5dj2MZyo0xvyzvJnA@mail.gmail.com/

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
@ 2020-06-11 23:14       ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:14 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig



On 2020-06-11 5:05 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> +	if (subsys->nr_namespaces) {
>> +		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
> 
> Let's try and keep the error message witin 80 char per line or split the
> message into two pr_info() calls,how about this ?
> 
> 
> pr_info("cannot enable passthru & regular namespaces\n")

Honestly, I think that is too brief. The error message is only 74 chars
and there's a long standing exception for long lines in the kernel for
printks. Even Linus has recently suggested that keeping to the 80 char
limit is not recommended when it harms readability (though I don't
generally agree with this 100%)[1].

Logan

[1]
https://lwn.net/ml/linux-kernel/CAHk-=wjR0H3+2ba0UUWwoYzYBH0GX9yTf5dj2MZyo0xvyzvJnA@mail.gmail.com/

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
  2020-06-11 23:13     ` Chaitanya Kulkarni
@ 2020-06-11 23:24       ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:24 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates



On 2020-06-11 5:13 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> +	if (subsys->ver < NVME_VS(1, 2, 1)) {
>> +		pr_warn("nvme controller version is too old: %d.%d.%d, advertising 1.2.1\n",
> Is it more than 80 char ? can it be ?
>                   pr_warn("nvme controller version is too old: ");
>                   pr_warn("%d.%d.%d, advertising 1.2.1\n",

Splitting printks is a bad idea because they won't necessarily end up in
dmesg right next to each other -- other prints can be interleaved.
Moreover, it breaks any tools that are dealing with kernel logs as
single records (ie through /dev/kmsg). Continuations (or printks without
a trailing "\n") would require using pr_cont() and generally that's
discouraged.

>> +			(int)NVME_MAJOR(subsys->ver),
>> +			(int)NVME_MINOR(subsys->ver),
>> +			(int)NVME_TERTIARY(subsys->ver));
>> +		subsys->ver = NVME_VS(1, 2, 1);
>> +	}
> 
> NVMe blktests are running on QEMU based controller. This will generate 
> warning every-time.

Yup.

> Also, I didn't understand int type cast, I wonder under what condition 
> all these macros will return -ve values since ver of type u64 ?

Yes, I'm not sure what I was thinking when I put in the cast. Probably
better just to prints "%llu". I'll fix this for the next revision.

Thanks,

Logan

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
@ 2020-06-11 23:24       ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:24 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig



On 2020-06-11 5:13 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> +	if (subsys->ver < NVME_VS(1, 2, 1)) {
>> +		pr_warn("nvme controller version is too old: %d.%d.%d, advertising 1.2.1\n",
> Is it more than 80 char ? can it be ?
>                   pr_warn("nvme controller version is too old: ");
>                   pr_warn("%d.%d.%d, advertising 1.2.1\n",

Splitting printks is a bad idea because they won't necessarily end up in
dmesg right next to each other -- other prints can be interleaved.
Moreover, it breaks any tools that are dealing with kernel logs as
single records (ie through /dev/kmsg). Continuations (or printks without
a trailing "\n") would require using pr_cont() and generally that's
discouraged.

>> +			(int)NVME_MAJOR(subsys->ver),
>> +			(int)NVME_MINOR(subsys->ver),
>> +			(int)NVME_TERTIARY(subsys->ver));
>> +		subsys->ver = NVME_VS(1, 2, 1);
>> +	}
> 
> NVMe blktests are running on QEMU based controller. This will generate 
> warning every-time.

Yup.

> Also, I didn't understand int type cast, I wonder under what condition 
> all these macros will return -ve values since ver of type u64 ?

Yes, I'm not sure what I was thinking when I put in the cast. Probably
better just to prints "%llu". I'll fix this for the next revision.

Thanks,

Logan

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
  2020-06-11 23:14       ` Logan Gunthorpe
@ 2020-06-11 23:32         ` Chaitanya Kulkarni
  -1 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:32 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates

On 6/11/20 4:15 PM, Logan Gunthorpe wrote:
> Honestly, I think that is too brief. The error message is only 74 chars
> and there's a long standing exception for long lines in the kernel for
> printks. Even Linus has recently suggested that keeping to the 80 char
> limit is not recommended when it harms readability (though I don't
> generally agree with this 100%)[1].

I'm aware of that and main cause is it creates unnecessary breaking 
which can lead to bugs, hard to read code and ugly wrapped code. I think
this case is different with printing message which is straight forward 
to read and not going to introduce any bugs in actual logic.

Based on your earlier reply about splitting printk I'll let Christoph
and Sagi decide on what to do about this and whether to enforce strict
80 lines or not for NVMe subsystem for printing messages and for future
code.

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers
@ 2020-06-11 23:32         ` Chaitanya Kulkarni
  0 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:32 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig

On 6/11/20 4:15 PM, Logan Gunthorpe wrote:
> Honestly, I think that is too brief. The error message is only 74 chars
> and there's a long standing exception for long lines in the kernel for
> printks. Even Linus has recently suggested that keeping to the 80 char
> limit is not recommended when it harms readability (though I don't
> generally agree with this 100%)[1].

I'm aware of that and main cause is it creates unnecessary breaking 
which can lead to bugs, hard to read code and ugly wrapped code. I think
this case is different with printing message which is straight forward 
to read and not going to introduce any bugs in actual logic.

Based on your earlier reply about splitting printk I'll let Christoph
and Sagi decide on what to do about this and whether to enforce strict
80 lines or not for NVMe subsystem for printing messages and for future
code.

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 7/9] nvmet-passthru: Add passthru code to process commands
  2020-05-14 17:22   ` Logan Gunthorpe
@ 2020-06-11 23:45     ` Chaitanya Kulkarni
  -1 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:45 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> +
> +static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
> +{
> +	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
> +	struct nvme_ns *ns = NULL;
> +	struct request *rq = NULL;
> +	struct request_queue *q;
> +	u32 effects;
> +	u16 status;
> +	int ret;
> +
> +	if (likely(req->sq->qid != 0)) {
> +		u32 nsid = le32_to_cpu(req->cmd->common.nsid);
> +
> +		ns = nvme_find_get_ns(ctrl, nsid);
> +		if (unlikely(!ns)) {
> +			pr_err("failed to get passthru ns nsid:%u\n", nsid);
> +			status = NVME_SC_INVALID_NS | NVME_SC_DNR;
> +			goto fail_out;
> +		}
> +	}
> +
> +	if (ns)
> +		q = ns->queue;
> +	else
> +		q = ctrl->admin_q;

Is it possible to avoid explicit if.. else in fast path given that the
condition exists and we can take an advantage of that, how about
following ?

diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
index 9e71fdfbdbb3..4f8e022d254c 100644
--- a/drivers/nvme/target/passthru.c
+++ b/drivers/nvme/target/passthru.c
@@ -84,9 +84,9 @@ static int nvmet_passthru_map_sg(struct nvmet_req 
*req, struct request *rq)
  static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
  {
         struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
+       struct request_queue *q = ctrl->admin_q;
         struct nvme_ns *ns = NULL;
         struct request *rq = NULL;
-       struct request_queue *q;
         u32 effects;
         u16 status;
         int ret;
@@ -100,12 +100,8 @@ static void nvmet_passthru_execute_cmd(struct 
nvmet_req *req)
                         status = NVME_SC_INVALID_NS | NVME_SC_DNR;
                         goto fail_out;
                 }
-       }
-
-       if (ns)
                 q = ns->queue;
-       else
-               q = ctrl->admin_q;
+       }

         rq = nvme_alloc_request(q, req->cmd, BLK_MQ_REQ_NOWAIT, 
NVME_QID_ANY);
         if (IS_ERR(rq)) {


^ permalink raw reply related	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 7/9] nvmet-passthru: Add passthru code to process commands
@ 2020-06-11 23:45     ` Chaitanya Kulkarni
  0 siblings, 0 replies; 46+ messages in thread
From: Chaitanya Kulkarni @ 2020-06-11 23:45 UTC (permalink / raw)
  To: Logan Gunthorpe, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig

On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
> +
> +static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
> +{
> +	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
> +	struct nvme_ns *ns = NULL;
> +	struct request *rq = NULL;
> +	struct request_queue *q;
> +	u32 effects;
> +	u16 status;
> +	int ret;
> +
> +	if (likely(req->sq->qid != 0)) {
> +		u32 nsid = le32_to_cpu(req->cmd->common.nsid);
> +
> +		ns = nvme_find_get_ns(ctrl, nsid);
> +		if (unlikely(!ns)) {
> +			pr_err("failed to get passthru ns nsid:%u\n", nsid);
> +			status = NVME_SC_INVALID_NS | NVME_SC_DNR;
> +			goto fail_out;
> +		}
> +	}
> +
> +	if (ns)
> +		q = ns->queue;
> +	else
> +		q = ctrl->admin_q;

Is it possible to avoid explicit if.. else in fast path given that the
condition exists and we can take an advantage of that, how about
following ?

diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
index 9e71fdfbdbb3..4f8e022d254c 100644
--- a/drivers/nvme/target/passthru.c
+++ b/drivers/nvme/target/passthru.c
@@ -84,9 +84,9 @@ static int nvmet_passthru_map_sg(struct nvmet_req 
*req, struct request *rq)
  static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
  {
         struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
+       struct request_queue *q = ctrl->admin_q;
         struct nvme_ns *ns = NULL;
         struct request *rq = NULL;
-       struct request_queue *q;
         u32 effects;
         u16 status;
         int ret;
@@ -100,12 +100,8 @@ static void nvmet_passthru_execute_cmd(struct 
nvmet_req *req)
                         status = NVME_SC_INVALID_NS | NVME_SC_DNR;
                         goto fail_out;
                 }
-       }
-
-       if (ns)
                 q = ns->queue;
-       else
-               q = ctrl->admin_q;
+       }

         rq = nvme_alloc_request(q, req->cmd, BLK_MQ_REQ_NOWAIT, 
NVME_QID_ANY);
         if (IS_ERR(rq)) {


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 7/9] nvmet-passthru: Add passthru code to process commands
  2020-06-11 23:45     ` Chaitanya Kulkarni
@ 2020-06-11 23:46       ` Logan Gunthorpe
  -1 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:46 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Christoph Hellwig, Sagi Grimberg, Keith Busch, Jens Axboe,
	Max Gurtovoy, Stephen Bates



On 2020-06-11 5:45 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> +
>> +static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
>> +{
>> +	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
>> +	struct nvme_ns *ns = NULL;
>> +	struct request *rq = NULL;
>> +	struct request_queue *q;
>> +	u32 effects;
>> +	u16 status;
>> +	int ret;
>> +
>> +	if (likely(req->sq->qid != 0)) {
>> +		u32 nsid = le32_to_cpu(req->cmd->common.nsid);
>> +
>> +		ns = nvme_find_get_ns(ctrl, nsid);
>> +		if (unlikely(!ns)) {
>> +			pr_err("failed to get passthru ns nsid:%u\n", nsid);
>> +			status = NVME_SC_INVALID_NS | NVME_SC_DNR;
>> +			goto fail_out;
>> +		}
>> +	}
>> +
>> +	if (ns)
>> +		q = ns->queue;
>> +	else
>> +		q = ctrl->admin_q;
> 
> Is it possible to avoid explicit if.. else in fast path given that the
> condition exists and we can take an advantage of that, how about
> following ?
> 
> diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
> index 9e71fdfbdbb3..4f8e022d254c 100644
> --- a/drivers/nvme/target/passthru.c
> +++ b/drivers/nvme/target/passthru.c
> @@ -84,9 +84,9 @@ static int nvmet_passthru_map_sg(struct nvmet_req 
> *req, struct request *rq)
>   static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
>   {
>          struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
> +       struct request_queue *q = ctrl->admin_q;
>          struct nvme_ns *ns = NULL;
>          struct request *rq = NULL;
> -       struct request_queue *q;
>          u32 effects;
>          u16 status;
>          int ret;
> @@ -100,12 +100,8 @@ static void nvmet_passthru_execute_cmd(struct 
> nvmet_req *req)
>                          status = NVME_SC_INVALID_NS | NVME_SC_DNR;
>                          goto fail_out;
>                  }
> -       }
> -
> -       if (ns)
>                  q = ns->queue;
> -       else
> -               q = ctrl->admin_q;
> +       }
> 
>          rq = nvme_alloc_request(q, req->cmd, BLK_MQ_REQ_NOWAIT, 
> NVME_QID_ANY);
>          if (IS_ERR(rq)) {
> 


This makes sense. Will fix for the next revision.

Thanks,

Logan

^ permalink raw reply	[flat|nested] 46+ messages in thread

* Re: [PATCH v13 7/9] nvmet-passthru: Add passthru code to process commands
@ 2020-06-11 23:46       ` Logan Gunthorpe
  0 siblings, 0 replies; 46+ messages in thread
From: Logan Gunthorpe @ 2020-06-11 23:46 UTC (permalink / raw)
  To: Chaitanya Kulkarni, linux-kernel, linux-nvme
  Cc: Sagi Grimberg, Stephen Bates, Jens Axboe, Keith Busch,
	Max Gurtovoy, Christoph Hellwig



On 2020-06-11 5:45 p.m., Chaitanya Kulkarni wrote:
> On 5/14/20 10:23 AM, Logan Gunthorpe wrote:
>> +
>> +static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
>> +{
>> +	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
>> +	struct nvme_ns *ns = NULL;
>> +	struct request *rq = NULL;
>> +	struct request_queue *q;
>> +	u32 effects;
>> +	u16 status;
>> +	int ret;
>> +
>> +	if (likely(req->sq->qid != 0)) {
>> +		u32 nsid = le32_to_cpu(req->cmd->common.nsid);
>> +
>> +		ns = nvme_find_get_ns(ctrl, nsid);
>> +		if (unlikely(!ns)) {
>> +			pr_err("failed to get passthru ns nsid:%u\n", nsid);
>> +			status = NVME_SC_INVALID_NS | NVME_SC_DNR;
>> +			goto fail_out;
>> +		}
>> +	}
>> +
>> +	if (ns)
>> +		q = ns->queue;
>> +	else
>> +		q = ctrl->admin_q;
> 
> Is it possible to avoid explicit if.. else in fast path given that the
> condition exists and we can take an advantage of that, how about
> following ?
> 
> diff --git a/drivers/nvme/target/passthru.c b/drivers/nvme/target/passthru.c
> index 9e71fdfbdbb3..4f8e022d254c 100644
> --- a/drivers/nvme/target/passthru.c
> +++ b/drivers/nvme/target/passthru.c
> @@ -84,9 +84,9 @@ static int nvmet_passthru_map_sg(struct nvmet_req 
> *req, struct request *rq)
>   static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
>   {
>          struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
> +       struct request_queue *q = ctrl->admin_q;
>          struct nvme_ns *ns = NULL;
>          struct request *rq = NULL;
> -       struct request_queue *q;
>          u32 effects;
>          u16 status;
>          int ret;
> @@ -100,12 +100,8 @@ static void nvmet_passthru_execute_cmd(struct 
> nvmet_req *req)
>                          status = NVME_SC_INVALID_NS | NVME_SC_DNR;
>                          goto fail_out;
>                  }
> -       }
> -
> -       if (ns)
>                  q = ns->queue;
> -       else
> -               q = ctrl->admin_q;
> +       }
> 
>          rq = nvme_alloc_request(q, req->cmd, BLK_MQ_REQ_NOWAIT, 
> NVME_QID_ANY);
>          if (IS_ERR(rq)) {
> 


This makes sense. Will fix for the next revision.

Thanks,

Logan

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 46+ messages in thread

end of thread, other threads:[~2020-06-11 23:47 UTC | newest]

Thread overview: 46+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-14 17:22 [PATCH v13 0/9] nvmet: add target passthru commands support Logan Gunthorpe
2020-05-14 17:22 ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 1/9] nvme-core: Clear any SGL flags in passthru commands Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 2/9] nvme: Create helper function to obtain command effects Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-06-11 22:56   ` Chaitanya Kulkarni
2020-06-11 22:56     ` Chaitanya Kulkarni
2020-06-11 23:09     ` Logan Gunthorpe
2020-06-11 23:09       ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 3/9] nvme: Move nvme_passthru_[start|end]() calls to common helper Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 4/9] nvmet-passthru: Introduce NVMet passthru Kconfig option Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 5/9] nvme-core: Introduce nvme_ctrl_get_by_path() Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-06-11 23:02   ` Chaitanya Kulkarni
2020-06-11 23:02     ` Chaitanya Kulkarni
2020-06-11 23:09     ` Logan Gunthorpe
2020-06-11 23:09       ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 6/9] nvme: Export existing nvme core functions Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 7/9] nvmet-passthru: Add passthru code to process commands Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-06-11 23:45   ` Chaitanya Kulkarni
2020-06-11 23:45     ` Chaitanya Kulkarni
2020-06-11 23:46     ` Logan Gunthorpe
2020-06-11 23:46       ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 8/9] nvmet-passthru: Add enable/disable helpers Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-06-11 23:05   ` Chaitanya Kulkarni
2020-06-11 23:05     ` Chaitanya Kulkarni
2020-06-11 23:14     ` Logan Gunthorpe
2020-06-11 23:14       ` Logan Gunthorpe
2020-06-11 23:32       ` Chaitanya Kulkarni
2020-06-11 23:32         ` Chaitanya Kulkarni
2020-06-11 23:13   ` Chaitanya Kulkarni
2020-06-11 23:13     ` Chaitanya Kulkarni
2020-06-11 23:24     ` Logan Gunthorpe
2020-06-11 23:24       ` Logan Gunthorpe
2020-05-14 17:22 ` [PATCH v13 9/9] nvmet-configfs: Introduce passthru configfs interface Logan Gunthorpe
2020-05-14 17:22   ` Logan Gunthorpe
2020-06-11 19:16 ` [PATCH v13 0/9] nvmet: add target passthru commands support Chaitanya Kulkarni
2020-06-11 19:16   ` Chaitanya Kulkarni
2020-06-11 19:19   ` Logan Gunthorpe
2020-06-11 19:19     ` Logan Gunthorpe

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.