From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B03B9C2D0A3 for ; Thu, 12 Nov 2020 14:08:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2704B22240 for ; Thu, 12 Nov 2020 14:08:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qoi+sFlb" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728089AbgKLOIK (ORCPT ); Thu, 12 Nov 2020 09:08:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727822AbgKLOII (ORCPT ); Thu, 12 Nov 2020 09:08:08 -0500 Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECC12C0613D1; Thu, 12 Nov 2020 06:08:06 -0800 (PST) Received: by mail-qk1-x743.google.com with SMTP id 11so5289212qkd.5; Thu, 12 Nov 2020 06:08:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Wq0TTtOfCmq5rw8H3/BW0bTDOXciTJIX0nn7uC7rhZE=; b=qoi+sFlbYvSA7V/w38GUcOZj555blWHiGPwdSpj6COfEV98uSSlrabdUmrpisgpgU2 o3bjcfgeoMPgX8e+uc7SCYW4ZavJSJzEDCL6aZNFxHf5P+MFlcyi1OML8TEurj7fNnx5 rReBKDTugOWS8YYaWSUjkGYta8G4s2ZwGhYooyzukHoCrPRGiBofwm4BP+ESHi26agOV 9Y5ExXWOgW7V40Xfnuw98yz4usPMfoVUk2WkVDR5JpTQsVqpQCb78tEhxV1xlk8FpR2X DRPoUMf7gKPJW3Z8PWxA5rB9Vm3fnF/rRZdE7USjWSJC6Zh5Pi1Abh6choNcBfXORtCw Ebog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Wq0TTtOfCmq5rw8H3/BW0bTDOXciTJIX0nn7uC7rhZE=; b=E1EZ4e3gukDr0OezWCEarwOYfoHK8XnxMpeGVmsmw+iSxam6JZY3VL8YQoi+WJLobi 8+F92T8rGwBVuj3R1O/5eZNLS4ElGQImTIFlrpWxC6JiDt9f4oeTW/KFoOsxLTAkYRQz XXSA284Gb3ujrc+sKrdC6VtplQNypfYTBV2JI659yvuK+vli2gFDLrQ52DtMB+GNsRrX UmkY9Nsh9DglzB65/7p4tf5fprpQTu0A7JP9y7q1wKAshZIOJqlY6w8U9Bmf7ISvb436 ZlHtx/Cj6doTAbp9a8GwJNQi1lD78G/Xze6/PLYXJ/x0eOh8XcPDIC9IiDxycwiwXQ3p YKMw== X-Gm-Message-State: AOAM5304RbPVAoyCZg9jOMaM0gEtLPPlip3sH4SEi8uDWP9r81dP7Mbi NUHIfFMzecsdZd86DuPPiw== X-Google-Smtp-Source: ABdhPJzL3B1l0WXfWIIu4w1iDcUICTeNQKVagY5WrNdCP9DefBDeEo2k+9b71PoEaeE92rVDL7u3UA== X-Received: by 2002:ae9:c211:: with SMTP id j17mr32011594qkg.458.1605190085900; Thu, 12 Nov 2020 06:08:05 -0800 (PST) Received: from localhost.localdomain ([68.175.153.174]) by smtp.gmail.com with ESMTPSA id g13sm4742881qth.27.2020.11.12.06.08.03 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Nov 2020 06:08:04 -0800 (PST) From: Rachit Agarwal To: Jens Axboe , Christoph Hellwig Cc: Rachit Agarwal , linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Keith Busch , Ming Lei , Jaehyun Hwang , Qizhe Cai , Midhul Vuppalapati , Rachit Agarwal , Sagi Grimberg , Rachit Agarwal Subject: [PATCH] iosched: Add i10 I/O Scheduler Date: Thu, 12 Nov 2020 09:07:52 -0500 Message-Id: <20201112140752.1554-1-rach4x0r@gmail.com> X-Mailer: git-send-email 2.15.2 (Apple Git-101.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Rachit Agarwal > Hi All, I/O batching is beneficial for optimizing IOPS and throughput for various applications. For instance, several kernel block drivers would benefit from batching, including mmc [1] and tcp-based storage drivers like nvme-tcp [2,3]. While we have support for batching dispatch [4], we need an I/O scheduler to efficiently enable batching. Such a scheduler is particularly interesting for disaggregated storage, where the access latency of remote disaggregated storage may be higher than local storage access; thus, batching can significantly help in amortizing the remote access latency while increasing the throughput. This patch introduces the i10 I/O scheduler, which performs batching per hctx in terms of #requests, #bytes, and timeouts (at microseconds granularity). i10 starts dispatching only when #requests or #bytes is larger than a default threshold or when a timer expires. After that, batching dispatch [3] would happen, allowing batching at device drivers along with "bd->last" and ".commit_rqs". The i10 I/O scheduler builds upon recent work on [6]. We have tested the i10 I/O scheduler with nvme-tcp optimizaitons [2,3] and batching dispatch [4], varying number of cores, varying read/write ratios, and varying request sizes, and with NVMe SSD and RAM block device. For NVMe SSDs, the i10 I/O scheduler achieves ~60% improvements in terms of IOPS per core over "noop" I/O scheduler. These results are available at [5], and many additional results are presented in [6]. While other schedulers may also batch I/O (e.g., mq-deadline), the optimization target in the i10 I/O scheduler is throughput maximization. Hence there is no latency target nor a need for a global tracking context, so a new scheduler is needed rather than to build this functionality to an existing scheduler. We currently use fixed default values as batching thresholds (e.g., 16 for #requests, 64KB for #bytes, and 50us for timeout). These default values are based on sensitivity tests in [6]. For our future work, we plan to support adaptive batching according to system load and to extend the scheduler to support isolation in multi-tenant deployments (to simultaneously achieve low tail latency for latency-sensitive applications and high throughput for throughput-bound applications). References [1] https://lore.kernel.org/linux-block/cover.1587888520.git.baolin.wang7@gmail.com/T/#mc48a8fb6069843827458f5fea722e1179d32af2a [2] https://git.infradead.org/nvme.git/commit/122e5b9f3d370ae11e1502d14ff5c7ea9b144a76 [3] https://git.infradead.org/nvme.git/commit/86f0348ace1510d7ac25124b096fb88a6ab45270 [4] https://lore.kernel.org/linux-block/20200630102501.2238972-1-ming.lei@redhat.com/ [5] https://github.com/i10-kernel/upstream-linux/blob/master/dss-evaluation.pdf [6] https://www.usenix.org/conference/nsdi20/presentation/hwang Signed-off-by: Jaehyun Hwang Signed-off-by: Qizhe Cai Signed-off-by: Midhul Vuppalapati Signed-off-by: Rachit Agarwal Signed-off-by: Sagi Grimberg --- Documentation/block/i10-iosched.rst | 69 +++++ block/Kconfig.iosched | 8 + block/Makefile | 1 + block/i10-iosched.c | 421 ++++++++++++++++++++++++++++ 4 files changed, 499 insertions(+) create mode 100644 Documentation/block/i10-iosched.rst create mode 100644 block/i10-iosched.c diff --git a/Documentation/block/i10-iosched.rst b/Documentation/block/i10-iosched.rst new file mode 100644 index 000000000000..3db7ca3ed4c1 --- /dev/null +++ b/Documentation/block/i10-iosched.rst @@ -0,0 +1,69 @@ +========================== +i10 I/O scheduler overview +========================== + +I/O batching is beneficial for optimizing IOPS and throughput for various +applications. For instance, several kernel block drivers would +benefit from batching, including mmc [1] and tcp-based storage drivers like +nvme-tcp [2,3]. While we have support for batching dispatch [4], we need an +I/O scheduler to efficiently enable batching. Such a scheduler is particularly +interesting for disaggregated storage, where the access latency of remote +disaggregated storage may be higher than local storage access; thus, batching +can significantly help in amortizing the remote access latency while increasing +the throughput. + +This patch introduces the i10 I/O scheduler, which performs batching per hctx in terms +of #requests, #bytes, and timeouts (at microseconds granularity). i10 starts +dispatching only when #requests or #bytes is larger than a default threshold or when +a timer expires. After that, batching dispatch [3] would happen, allowing batching +at device drivers along with "bd->last" and ".commit_rqs". + +The i10 I/O scheduler builds upon recent work on [6]. We have tested the i10 I/O +scheduler with nvme-tcp optimizaitons [2,3] and batching dispatch [4], varying number +of cores, varying read/write ratios, and varying request sizes, and with NVMe SSD and +RAM block device. For NVMe SSDs, the i10 I/O scheduler achieves ~60% improvements in +terms of IOPS per core over "noop" I/O scheduler. These results are available at [5], +and many additional results are presented in [6]. + +While other schedulers may also batch I/O (e.g., mq-deadline), the optimization target +in the i10 I/O scheduler is throughput maximization. Hence there is no latency target +nor a need for a global tracking context, so a new scheduler is needed rather than +to build this functionality to an existing scheduler. + +We currently use fixed default values as batching thresholds (e.g., 16 for #requests, +64KB for #bytes, and 50us for timeout). These default values are based on sensitivity +tests in [6]. For our future work, we plan to support adaptive batching according to +system load and to extend the scheduler to support isolation in multi-tenant deployments +(to simultaneously achieve low tail latency for latency-sensitive applications and high +throughput for throughput-bound applications). + +References +[1] https://lore.kernel.org/linux-block/cover.1587888520.git.baolin.wang7@gmail.com/T/#mc48a8fb6069843827458f5fea722e1179d32af2a +[2] https://git.infradead.org/nvme.git/commit/122e5b9f3d370ae11e1502d14ff5c7ea9b144a76 +[3] https://git.infradead.org/nvme.git/commit/86f0348ace1510d7ac25124b096fb88a6ab45270 +[4] https://lore.kernel.org/linux-block/20200630102501.2238972-1-ming.lei@redhat.com/ +[5] https://github.com/i10-kernel/upstream-linux/blob/master/dss-evaluation.pdf +[6] https://www.usenix.org/conference/nsdi20/presentation/hwang + +========================== +i10 I/O scheduler tunables +========================== + +The three tunables for the i10 scheduler are the number of requests for +reads/writes, the number of bytes for writes, and a timeout value. +i10 will use these values for batching requests. + +batch_nr +-------- +Number of requests for batching read/write requests +Default: 16 + +batch_bytes +----------- +Number of bytes for batching write requests +Default: 65536 (bytes) + +batch_timeout +------------- +Timeout value for batching (in microseconds) +Default: 50 (us) diff --git a/block/Kconfig.iosched b/block/Kconfig.iosched index 2f2158e05a91..5b3623b19487 100644 --- a/block/Kconfig.iosched +++ b/block/Kconfig.iosched @@ -44,6 +44,14 @@ config BFQ_CGROUP_DEBUG Enable some debugging help. Currently it exports additional stat files in a cgroup which can be useful for debugging. +config MQ_IOSCHED_I10 + tristate "i10 I/O scheduler" + default y + help + The i10 I/O Scheduler supports batching at BLK-MQ. + Any device driver that benefits from batching + (e.g., NVMe-over-TCP) can use this scheduler. + endmenu endif diff --git a/block/Makefile b/block/Makefile index 8d841f5f986f..27e0789589ea 100644 --- a/block/Makefile +++ b/block/Makefile @@ -21,6 +21,7 @@ obj-$(CONFIG_BLK_CGROUP_IOLATENCY) += blk-iolatency.o obj-$(CONFIG_BLK_CGROUP_IOCOST) += blk-iocost.o obj-$(CONFIG_MQ_IOSCHED_DEADLINE) += mq-deadline.o obj-$(CONFIG_MQ_IOSCHED_KYBER) += kyber-iosched.o +obj-$(CONFIG_MQ_IOSCHED_I10) += i10-iosched.o bfq-y := bfq-iosched.o bfq-wf2q.o bfq-cgroup.o obj-$(CONFIG_IOSCHED_BFQ) += bfq.o diff --git a/block/i10-iosched.c b/block/i10-iosched.c new file mode 100644 index 000000000000..b5451beaa66d --- /dev/null +++ b/block/i10-iosched.c @@ -0,0 +1,421 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * The i10 I/O Scheduler - supports batching at blk-mq. + * The main use case is disaggregated storage access + * using NVMe-over-Fabric (e.g., NVMe-over-TCP device driver). + * + * An early version of the idea is described and evaluated in + * "TCP ≈ RDMA: CPU-efficient Remote Storage Access with i10", + * USENIX NSDI 2020. + * + * Copyright (C) 2020 Cornell University + * Jaehyun Hwang + * Qizhe Cai + * Midhul Vuppalapati + * Rachit Agarwal + */ + +#include +#include +#include +#include +#include +#include + +#include "blk.h" +#include "blk-mq.h" +#include "blk-mq-debugfs.h" +#include "blk-mq-sched.h" +#include "blk-mq-tag.h" + +/* Default batch size in number of requests */ +#define I10_DEF_BATCH_NR 16 +/* Default batch size in bytes (for write requests) */ +#define I10_DEF_BATCH_BYTES 65536 +/* Default timeout value for batching (us units) */ +#define I10_DEF_BATCH_TIMEOUT 50 + +enum i10_state { + /* Batching state: + * Do not run dispatching until we have + * a certain amount of requests or a timer expires. + */ + I10_STATE_BATCH, + + /* Dispatching state: + * Run dispatching until all requests in the + * scheduler's hctx ihq are dispatched. + */ + I10_STATE_DISPATCH, +}; + +struct i10_queue_data { + struct request_queue *q; + + unsigned int def_batch_nr; + unsigned int def_batch_bytes; + unsigned int def_batch_timeout; +}; + +struct i10_hctx_queue { + spinlock_t lock; + struct list_head rq_list; + + struct blk_mq_hw_ctx *hctx; + + unsigned int batch_nr; + unsigned int batch_bytes; + unsigned int batch_timeout; + + unsigned int qlen_nr; + unsigned int qlen_bytes; + + struct hrtimer dispatch_timer; + enum i10_state state; +}; + +static struct i10_queue_data *i10_queue_data_alloc(struct request_queue *q) +{ + struct i10_queue_data *iqd; + + iqd = kzalloc_node(sizeof(*iqd), GFP_KERNEL, q->node); + if (!iqd) + return ERR_PTR(-ENOMEM); + + iqd->q = q; + iqd->def_batch_nr = I10_DEF_BATCH_NR; + iqd->def_batch_bytes = I10_DEF_BATCH_BYTES; + iqd->def_batch_timeout = I10_DEF_BATCH_TIMEOUT; + + return iqd; +} + +static int i10_init_sched(struct request_queue *q, struct elevator_type *e) +{ + struct i10_queue_data *iqd; + struct elevator_queue *eq; + + eq = elevator_alloc(q, e); + if (!eq) + return -ENOMEM; + + iqd = i10_queue_data_alloc(q); + if (IS_ERR(iqd)) { + kobject_put(&eq->kobj); + return PTR_ERR(iqd); + } + + blk_stat_enable_accounting(q); + + eq->elevator_data = iqd; + q->elevator = eq; + + return 0; +} + +static void i10_exit_sched(struct elevator_queue *e) +{ + struct i10_queue_data *iqd = e->elevator_data; + + kfree(iqd); +} + +enum hrtimer_restart i10_hctx_timeout_handler(struct hrtimer *timer) +{ + struct i10_hctx_queue *ihq = + container_of(timer, struct i10_hctx_queue, + dispatch_timer); + + ihq->state = I10_STATE_DISPATCH; + blk_mq_run_hw_queue(ihq->hctx, true); + + return HRTIMER_NORESTART; +} + +static void i10_hctx_queue_reset(struct i10_hctx_queue *ihq) +{ + ihq->qlen_nr = 0; + ihq->qlen_bytes = 0; + ihq->state = I10_STATE_BATCH; +} + +static int i10_init_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) +{ + struct i10_hctx_queue *ihq; + + ihq = kzalloc_node(sizeof(*ihq), GFP_KERNEL, hctx->numa_node); + if (!ihq) + return -ENOMEM; + + spin_lock_init(&ihq->lock); + INIT_LIST_HEAD(&ihq->rq_list); + + ihq->hctx = hctx; + ihq->batch_nr = 0; + ihq->batch_bytes = 0; + ihq->batch_timeout = 0; + + hrtimer_init(&ihq->dispatch_timer, + CLOCK_MONOTONIC, HRTIMER_MODE_REL); + ihq->dispatch_timer.function = &i10_hctx_timeout_handler; + + i10_hctx_queue_reset(ihq); + + hctx->sched_data = ihq; + + return 0; +} + +static void i10_exit_hctx(struct blk_mq_hw_ctx *hctx, unsigned int hctx_idx) +{ + struct i10_hctx_queue *ihq = hctx->sched_data; + + hrtimer_cancel(&ihq->dispatch_timer); + kfree(hctx->sched_data); +} + +static bool i10_hctx_bio_merge(struct blk_mq_hw_ctx *hctx, struct bio *bio, + unsigned int nr_segs) +{ + struct i10_hctx_queue *ihq = hctx->sched_data; + struct list_head *rq_list = &ihq->rq_list; + bool merged; + + spin_lock(&ihq->lock); + merged = blk_mq_bio_list_merge(hctx->queue, rq_list, bio, nr_segs); + spin_unlock(&ihq->lock); + + if (merged && bio_data_dir(bio) == WRITE) + ihq->qlen_bytes += bio->bi_iter.bi_size; + + return merged; +} + +/* + * The batch size can be adjusted dynamically on a per-hctx basis. + * Use per-hctx variables in that case. + */ +static inline unsigned int i10_hctx_batch_nr(struct blk_mq_hw_ctx *hctx) +{ + struct i10_queue_data *iqd = hctx->queue->elevator->elevator_data; + struct i10_hctx_queue *ihq = hctx->sched_data; + + return ihq->batch_nr ? + ihq->batch_nr : iqd->def_batch_nr; +} + +static inline unsigned int i10_hctx_batch_bytes(struct blk_mq_hw_ctx *hctx) +{ + struct i10_queue_data *iqd = hctx->queue->elevator->elevator_data; + struct i10_hctx_queue *ihq = hctx->sched_data; + + return ihq->batch_bytes ? + ihq->batch_bytes : iqd->def_batch_bytes; +} + +static inline unsigned int i10_hctx_batch_timeout(struct blk_mq_hw_ctx *hctx) +{ + struct i10_queue_data *iqd = hctx->queue->elevator->elevator_data; + struct i10_hctx_queue *ihq = hctx->sched_data; + + return ihq->batch_timeout ? + ihq->batch_timeout : iqd->def_batch_timeout; +} + +static void i10_hctx_insert_update(struct i10_hctx_queue *ihq, + struct request *rq) +{ + if (rq_data_dir(rq) == WRITE) + ihq->qlen_bytes += blk_rq_bytes(rq); + ihq->qlen_nr++; +} + +static void i10_hctx_insert_requests(struct blk_mq_hw_ctx *hctx, + struct list_head *rq_list, bool at_head) +{ + struct i10_hctx_queue *ihq = hctx->sched_data; + struct request *rq, *next; + + list_for_each_entry_safe(rq, next, rq_list, queuelist) { + struct list_head *head = &ihq->rq_list; + + spin_lock(&ihq->lock); + if (at_head) + list_move(&rq->queuelist, head); + else + list_move_tail(&rq->queuelist, head); + i10_hctx_insert_update(ihq, rq); + blk_mq_sched_request_inserted(rq); + spin_unlock(&ihq->lock); + } + + /* Start a new timer */ + if (ihq->state == I10_STATE_BATCH && + !hrtimer_active(&ihq->dispatch_timer)) + hrtimer_start(&ihq->dispatch_timer, + ns_to_ktime(i10_hctx_batch_timeout(hctx) + * NSEC_PER_USEC), + HRTIMER_MODE_REL); +} + +static struct request *i10_hctx_dispatch_request(struct blk_mq_hw_ctx *hctx) +{ + struct i10_hctx_queue *ihq = hctx->sched_data; + struct request *rq; + + spin_lock(&ihq->lock); + rq = list_first_entry_or_null(&ihq->rq_list, + struct request, queuelist); + if (rq) + list_del_init(&rq->queuelist); + else + i10_hctx_queue_reset(ihq); + spin_unlock(&ihq->lock); + + return rq; +} + +static inline bool i10_hctx_dispatch_now(struct blk_mq_hw_ctx *hctx) +{ + struct i10_hctx_queue *ihq = hctx->sched_data; + + return (ihq->qlen_nr >= i10_hctx_batch_nr(hctx)) || + (ihq->qlen_bytes >= i10_hctx_batch_bytes(hctx)); +} + +/* + * Return true if we are in the dispatching state. + */ +static bool i10_hctx_has_work(struct blk_mq_hw_ctx *hctx) +{ + struct i10_hctx_queue *ihq = hctx->sched_data; + + if (ihq->state == I10_STATE_BATCH) { + if (i10_hctx_dispatch_now(hctx)) { + ihq->state = I10_STATE_DISPATCH; + if (hrtimer_active(&ihq->dispatch_timer)) + hrtimer_cancel(&ihq->dispatch_timer); + } + } + + return (ihq->state == I10_STATE_DISPATCH); +} + +#define I10_DEF_BATCH_SHOW_STORE(name) \ +static ssize_t i10_def_batch_##name##_show(struct elevator_queue *e, \ + char *page) \ +{ \ + struct i10_queue_data *iqd = e->elevator_data; \ + \ + return sprintf(page, "%u\n", iqd->def_batch_##name); \ +} \ + \ +static ssize_t i10_def_batch_##name##_store(struct elevator_queue *e, \ + const char *page, size_t count) \ +{ \ + struct i10_queue_data *iqd = e->elevator_data; \ + unsigned long long value; \ + int ret; \ + \ + ret = kstrtoull(page, 10, &value); \ + if (ret) \ + return ret; \ + \ + iqd->def_batch_##name = value; \ + \ + return count; \ +} +I10_DEF_BATCH_SHOW_STORE(nr); +I10_DEF_BATCH_SHOW_STORE(bytes); +I10_DEF_BATCH_SHOW_STORE(timeout); +#undef I10_DEF_BATCH_SHOW_STORE + +#define I10_SCHED_ATTR(name) \ + __ATTR(batch_##name, 0644, i10_def_batch_##name##_show, i10_def_batch_##name##_store) +static struct elv_fs_entry i10_sched_attrs[] = { + I10_SCHED_ATTR(nr), + I10_SCHED_ATTR(bytes), + I10_SCHED_ATTR(timeout), + __ATTR_NULL +}; +#undef I10_SCHED_ATTR + +#ifdef CONFIG_BLK_DEBUG_FS +#define I10_DEBUGFS_SHOW(name) \ +static int i10_hctx_batch_##name##_show(void *data, struct seq_file *m) \ +{ \ + struct blk_mq_hw_ctx *hctx = data; \ + struct i10_hctx_queue *ihq = hctx->sched_data; \ + \ + seq_printf(m, "%u\n", ihq->batch_##name); \ + return 0; \ +} \ + \ +static int i10_hctx_qlen_##name##_show(void *data, struct seq_file *m) \ +{ \ + struct blk_mq_hw_ctx *hctx = data; \ + struct i10_hctx_queue *ihq = hctx->sched_data; \ + \ + seq_printf(m, "%u\n", ihq->qlen_##name); \ + return 0; \ +} +I10_DEBUGFS_SHOW(nr); +I10_DEBUGFS_SHOW(bytes); +#undef I10_DEBUGFS_SHOW + +static int i10_hctx_state_show(void *data, struct seq_file *m) +{ + struct blk_mq_hw_ctx *hctx = data; + struct i10_hctx_queue *ihq = hctx->sched_data; + + seq_printf(m, "%d\n", ihq->state); + return 0; +} + +#define I10_HCTX_QUEUE_ATTR(name) \ + {"batch_" #name, 0400, i10_hctx_batch_##name##_show}, \ + {"qlen_" #name, 0400, i10_hctx_qlen_##name##_show} +static const struct blk_mq_debugfs_attr i10_hctx_debugfs_attrs[] = { + I10_HCTX_QUEUE_ATTR(nr), + I10_HCTX_QUEUE_ATTR(bytes), + {"state", 0400, i10_hctx_state_show}, + {}, +}; +#undef I10_HCTX_QUEUE_ATTR +#endif + +static struct elevator_type i10_sched = { + .ops = { + .init_sched = i10_init_sched, + .exit_sched = i10_exit_sched, + .init_hctx = i10_init_hctx, + .exit_hctx = i10_exit_hctx, + .bio_merge = i10_hctx_bio_merge, + .insert_requests = i10_hctx_insert_requests, + .dispatch_request = i10_hctx_dispatch_request, + .has_work = i10_hctx_has_work, + }, +#ifdef CONFIG_BLK_DEBUG_FS + .hctx_debugfs_attrs = i10_hctx_debugfs_attrs, +#endif + .elevator_attrs = i10_sched_attrs, + .elevator_name = "i10", + .elevator_owner = THIS_MODULE, +}; + +static int __init i10_init(void) +{ + return elv_register(&i10_sched); +} + +static void __exit i10_exit(void) +{ + elv_unregister(&i10_sched); +} + +module_init(i10_init); +module_exit(i10_exit); + +MODULE_AUTHOR("Jaehyun Hwang, Qizhe Cai, Midhul Vuppalapati, Rachit Agarwal"); +MODULE_LICENSE("GPLv2"); +MODULE_DESCRIPTION("i10 I/O scheduler"); -- 2.22.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D151C2D0A3 for ; Thu, 12 Nov 2020 14:08:21 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BAE2C20A8B for ; Thu, 12 Nov 2020 14:08:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="mm6Ro7cJ"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qoi+sFlb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BAE2C20A8B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=qb6kshfGKlK/4DeEhKemK8+r+8EjsVZaUI9FSSe3zWE=; b=mm6Ro7cJ9edSkz4gxUStJ3FSEx dE8ZKzXcHuBlmr9pLSdiQ3xa5hPy6XVpHvuqzqKpY7ou5Bmb6qTf9qoJ7NfIDVDkysWwdxDc8HjZZ zcsAJHC48RiHMW99xLCXcPs9u+jksjEdt4DTPj9Rb0yredE1W2ZmBEDCmHyjgzZb7nAw+/u5Xb9rR b0i7F9kz7XXLGKjcGJHxxMQnaE8SzsbdwSF7mQpsIwm/oN5T8MhvF1p1PQUTdlTc2fsisjjvN416A x8Gj906OrjWthspFzG5Pkj4l9A/TdjHA3sbichrbanmgr3y1ZYFyMMpREa2+3izAWwSfzNTKVILG/ WpA76M0w==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdDGn-0007Oi-Sz; Thu, 12 Nov 2020 14:08:13 +0000 Received: from mail-qk1-x742.google.com ([2607:f8b0:4864:20::742]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kdDGj-0007Nb-VY for linux-nvme@lists.infradead.org; Thu, 12 Nov 2020 14:08:12 +0000 Received: by mail-qk1-x742.google.com with SMTP id h15so5245494qkl.13 for ; Thu, 12 Nov 2020 06:08:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Wq0TTtOfCmq5rw8H3/BW0bTDOXciTJIX0nn7uC7rhZE=; b=qoi+sFlbYvSA7V/w38GUcOZj555blWHiGPwdSpj6COfEV98uSSlrabdUmrpisgpgU2 o3bjcfgeoMPgX8e+uc7SCYW4ZavJSJzEDCL6aZNFxHf5P+MFlcyi1OML8TEurj7fNnx5 rReBKDTugOWS8YYaWSUjkGYta8G4s2ZwGhYooyzukHoCrPRGiBofwm4BP+ESHi26agOV 9Y5ExXWOgW7V40Xfnuw98yz4usPMfoVUk2WkVDR5JpTQsVqpQCb78tEhxV1xlk8FpR2X DRPoUMf7gKPJW3Z8PWxA5rB9Vm3fnF/rRZdE7USjWSJC6Zh5Pi1Abh6choNcBfXORtCw Ebog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=Wq0TTtOfCmq5rw8H3/BW0bTDOXciTJIX0nn7uC7rhZE=; b=pI6mcSAjFFMfZvOcTG4aDhhN88E11P7Ba09KaAdeC/NUDGylbDBaqnHCHchnJTQIG+ NJL0i5c7dERGETOon1H7s+CS6IeaZW4vtDGRsSwMxYNLk47D1nmFfQuvhbBzMB+uqjxo rIPbknFI7nDFrYxTfi+CfOxayhYtSiAGP8FkipLdu3F+oc24lXLBX+xZSOmVx0geAeqL GhxD+RSIfDkbwAHdRDLjc8QgK/0L+oXMr9HYBpuZkSpRCRVJE1a5WeEJ8dFkhEoSE6Kp ht9IIGigjhRdQ4oe+0YMhpfZX/HcFOOLO64RlFgVE4i81Ix01eoKeyK9EO7EvtLVukq1 KA6w== X-Gm-Message-State: AOAM532pptR7/p9j9ThZ3cCjGIcEMmnuwz/p2ugpeReuc7LVMI3ufG+u 3F/V9nAW4trMkNpCCPSa6w== X-Google-Smtp-Source: ABdhPJzL3B1l0WXfWIIu4w1iDcUICTeNQKVagY5WrNdCP9DefBDeEo2k+9b71PoEaeE92rVDL7u3UA== X-Received: by 2002:ae9:c211:: with SMTP id j17mr32011594qkg.458.1605190085900; Thu, 12 Nov 2020 06:08:05 -0800 (PST) Received: from localhost.localdomain ([68.175.153.174]) by smtp.gmail.com with ESMTPSA id g13sm4742881qth.27.2020.11.12.06.08.03 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Nov 2020 06:08:04 -0800 (PST) From: Rachit Agarwal To: Jens Axboe , Christoph Hellwig Subject: [PATCH] iosched: Add i10 I/O Scheduler Date: Thu, 12 Nov 2020 09:07:52 -0500 Message-Id: <20201112140752.1554-1-rach4x0r@gmail.com> X-Mailer: git-send-email 2.15.2 (Apple Git-101.1) MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201112_090810_065248_AF4317A7 X-CRM114-Status: GOOD ( 26.29 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Rachit Agarwal , Qizhe Cai , Rachit Agarwal , linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, Ming Lei , linux-block@vger.kernel.org, Midhul Vuppalapati , Jaehyun Hwang , Rachit Agarwal , Keith Busch , Sagi Grimberg Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org RnJvbTogUmFjaGl0IEFnYXJ3YWwgPHJhY2g0eDByQGdtYWlsLmNvbT4+CgpIaSBBbGwsCgpJL08g YmF0Y2hpbmcgaXMgYmVuZWZpY2lhbCBmb3Igb3B0aW1pemluZyBJT1BTIGFuZCB0aHJvdWdocHV0 IGZvciB2YXJpb3VzCmFwcGxpY2F0aW9ucy4gRm9yIGluc3RhbmNlLCBzZXZlcmFsIGtlcm5lbCBi bG9jayBkcml2ZXJzIHdvdWxkIGJlbmVmaXQgZnJvbSBiYXRjaGluZywKaW5jbHVkaW5nIG1tYyBb MV0gYW5kIHRjcC1iYXNlZCBzdG9yYWdlIGRyaXZlcnMgbGlrZSBudm1lLXRjcCBbMiwzXS4gV2hp bGUgd2UgaGF2ZQpzdXBwb3J0IGZvciBiYXRjaGluZyBkaXNwYXRjaCBbNF0sIHdlIG5lZWQgYW4g SS9PIHNjaGVkdWxlciB0byBlZmZpY2llbnRseSBlbmFibGUKYmF0Y2hpbmcuIFN1Y2ggYSBzY2hl ZHVsZXIgaXMgcGFydGljdWxhcmx5IGludGVyZXN0aW5nIGZvciBkaXNhZ2dyZWdhdGVkIHN0b3Jh Z2UsCndoZXJlIHRoZSBhY2Nlc3MgbGF0ZW5jeSBvZiByZW1vdGUgZGlzYWdncmVnYXRlZCBzdG9y YWdlIG1heSBiZSBoaWdoZXIgdGhhbiBsb2NhbApzdG9yYWdlIGFjY2VzczsgdGh1cywgYmF0Y2hp bmcgY2FuIHNpZ25pZmljYW50bHkgaGVscCBpbiBhbW9ydGl6aW5nIHRoZSByZW1vdGUgYWNjZXNz CmxhdGVuY3kgd2hpbGUgaW5jcmVhc2luZyB0aGUgdGhyb3VnaHB1dC4KClRoaXMgcGF0Y2ggaW50 cm9kdWNlcyB0aGUgaTEwIEkvTyBzY2hlZHVsZXIsIHdoaWNoIHBlcmZvcm1zIGJhdGNoaW5nIHBl ciBoY3R4IGluIHRlcm1zCm9mICNyZXF1ZXN0cywgI2J5dGVzLCBhbmQgdGltZW91dHMgKGF0IG1p Y3Jvc2Vjb25kcyBncmFudWxhcml0eSkuIGkxMCBzdGFydHMKZGlzcGF0Y2hpbmcgb25seSB3aGVu ICNyZXF1ZXN0cyBvciAjYnl0ZXMgaXMgbGFyZ2VyIHRoYW4gYSBkZWZhdWx0IHRocmVzaG9sZCBv ciB3aGVuCmEgdGltZXIgZXhwaXJlcy4gQWZ0ZXIgdGhhdCwgYmF0Y2hpbmcgZGlzcGF0Y2ggWzNd IHdvdWxkIGhhcHBlbiwgYWxsb3dpbmcgYmF0Y2hpbmcKYXQgZGV2aWNlIGRyaXZlcnMgYWxvbmcg d2l0aCAiYmQtPmxhc3QiIGFuZCAiLmNvbW1pdF9ycXMiLgoKVGhlIGkxMCBJL08gc2NoZWR1bGVy IGJ1aWxkcyB1cG9uIHJlY2VudCB3b3JrIG9uIFs2XS4gV2UgaGF2ZSB0ZXN0ZWQgdGhlIGkxMCBJ L08Kc2NoZWR1bGVyIHdpdGggbnZtZS10Y3Agb3B0aW1pemFpdG9ucyBbMiwzXSBhbmQgYmF0Y2hp bmcgZGlzcGF0Y2ggWzRdLCB2YXJ5aW5nIG51bWJlcgpvZiBjb3JlcywgdmFyeWluZyByZWFkL3dy aXRlIHJhdGlvcywgYW5kIHZhcnlpbmcgcmVxdWVzdCBzaXplcywgYW5kIHdpdGggTlZNZSBTU0Qg YW5kClJBTSBibG9jayBkZXZpY2UuIEZvciBOVk1lIFNTRHMsIHRoZSBpMTAgSS9PIHNjaGVkdWxl ciBhY2hpZXZlcyB+NjAlIGltcHJvdmVtZW50cyBpbgp0ZXJtcyBvZiBJT1BTIHBlciBjb3JlIG92 ZXIgIm5vb3AiIEkvTyBzY2hlZHVsZXIuIFRoZXNlIHJlc3VsdHMgYXJlIGF2YWlsYWJsZSBhdCBb NV0sCmFuZCBtYW55IGFkZGl0aW9uYWwgcmVzdWx0cyBhcmUgcHJlc2VudGVkIGluIFs2XS4KCldo aWxlIG90aGVyIHNjaGVkdWxlcnMgbWF5IGFsc28gYmF0Y2ggSS9PIChlLmcuLCBtcS1kZWFkbGlu ZSksIHRoZSBvcHRpbWl6YXRpb24gdGFyZ2V0CmluIHRoZSBpMTAgSS9PIHNjaGVkdWxlciBpcyB0 aHJvdWdocHV0IG1heGltaXphdGlvbi4gSGVuY2UgdGhlcmUgaXMgbm8gbGF0ZW5jeSB0YXJnZXQK bm9yIGEgbmVlZCBmb3IgYSBnbG9iYWwgdHJhY2tpbmcgY29udGV4dCwgc28gYSBuZXcgc2NoZWR1 bGVyIGlzIG5lZWRlZCByYXRoZXIgdGhhbgp0byBidWlsZCB0aGlzIGZ1bmN0aW9uYWxpdHkgdG8g YW4gZXhpc3Rpbmcgc2NoZWR1bGVyLgoKV2UgY3VycmVudGx5IHVzZSBmaXhlZCBkZWZhdWx0IHZh bHVlcyBhcyBiYXRjaGluZyB0aHJlc2hvbGRzIChlLmcuLCAxNiBmb3IgI3JlcXVlc3RzLAo2NEtC IGZvciAjYnl0ZXMsIGFuZCA1MHVzIGZvciB0aW1lb3V0KS4gVGhlc2UgZGVmYXVsdCB2YWx1ZXMg YXJlIGJhc2VkIG9uIHNlbnNpdGl2aXR5CnRlc3RzIGluIFs2XS4gRm9yIG91ciBmdXR1cmUgd29y aywgd2UgcGxhbiB0byBzdXBwb3J0IGFkYXB0aXZlIGJhdGNoaW5nIGFjY29yZGluZyB0bwpzeXN0 ZW0gbG9hZCBhbmQgdG8gZXh0ZW5kIHRoZSBzY2hlZHVsZXIgdG8gc3VwcG9ydCBpc29sYXRpb24g aW4gbXVsdGktdGVuYW50IGRlcGxveW1lbnRzCih0byBzaW11bHRhbmVvdXNseSBhY2hpZXZlIGxv dyB0YWlsIGxhdGVuY3kgZm9yIGxhdGVuY3ktc2Vuc2l0aXZlIGFwcGxpY2F0aW9ucyBhbmQgaGln aAp0aHJvdWdocHV0IGZvciB0aHJvdWdocHV0LWJvdW5kIGFwcGxpY2F0aW9ucykuCgpSZWZlcmVu Y2VzClsxXSBodHRwczovL2xvcmUua2VybmVsLm9yZy9saW51eC1ibG9jay9jb3Zlci4xNTg3ODg4 NTIwLmdpdC5iYW9saW4ud2FuZzdAZ21haWwuY29tL1QvI21jNDhhOGZiNjA2OTg0MzgyNzQ1OGY1 ZmVhNzIyZTExNzlkMzJhZjJhClsyXSBodHRwczovL2dpdC5pbmZyYWRlYWQub3JnL252bWUuZ2l0 L2NvbW1pdC8xMjJlNWI5ZjNkMzcwYWUxMWUxNTAyZDE0ZmY1YzdlYTliMTQ0YTc2ClszXSBodHRw czovL2dpdC5pbmZyYWRlYWQub3JnL252bWUuZ2l0L2NvbW1pdC84NmYwMzQ4YWNlMTUxMGQ3YWMy NTEyNGIwOTZmYjg4YTZhYjQ1MjcwCls0XSBodHRwczovL2xvcmUua2VybmVsLm9yZy9saW51eC1i bG9jay8yMDIwMDYzMDEwMjUwMS4yMjM4OTcyLTEtbWluZy5sZWlAcmVkaGF0LmNvbS8KWzVdIGh0 dHBzOi8vZ2l0aHViLmNvbS9pMTAta2VybmVsL3Vwc3RyZWFtLWxpbnV4L2Jsb2IvbWFzdGVyL2Rz cy1ldmFsdWF0aW9uLnBkZgpbNl0gaHR0cHM6Ly93d3cudXNlbml4Lm9yZy9jb25mZXJlbmNlL25z ZGkyMC9wcmVzZW50YXRpb24vaHdhbmcKClNpZ25lZC1vZmYtYnk6IEphZWh5dW4gSHdhbmcgPGph ZWh5dW4uaHdhbmdAY29ybmVsbC5lZHU+ClNpZ25lZC1vZmYtYnk6IFFpemhlIENhaSA8cWMyMjhA Y29ybmVsbC5lZHU+ClNpZ25lZC1vZmYtYnk6IE1pZGh1bCBWdXBwYWxhcGF0aSA8bXZ2MjVAY29y bmVsbC5lZHU+ClNpZ25lZC1vZmYtYnk6IFJhY2hpdCBBZ2Fyd2FsIDxyYWdhcndhbEBjb3JuZWxs LmVkdT4KU2lnbmVkLW9mZi1ieTogU2FnaSBHcmltYmVyZyA8c2FnaUBsaWdodGJpdHNsYWJzLmNv bT4KLS0tCiBEb2N1bWVudGF0aW9uL2Jsb2NrL2kxMC1pb3NjaGVkLnJzdCB8ICA2OSArKysrKwog YmxvY2svS2NvbmZpZy5pb3NjaGVkICAgICAgICAgICAgICAgfCAgIDggKwogYmxvY2svTWFrZWZp bGUgICAgICAgICAgICAgICAgICAgICAgfCAgIDEgKwogYmxvY2svaTEwLWlvc2NoZWQuYyAgICAg ICAgICAgICAgICAgfCA0MjEgKysrKysrKysrKysrKysrKysrKysrKysrKysrKwogNCBmaWxlcyBj aGFuZ2VkLCA0OTkgaW5zZXJ0aW9ucygrKQogY3JlYXRlIG1vZGUgMTAwNjQ0IERvY3VtZW50YXRp b24vYmxvY2svaTEwLWlvc2NoZWQucnN0CiBjcmVhdGUgbW9kZSAxMDA2NDQgYmxvY2svaTEwLWlv c2NoZWQuYwoKZGlmZiAtLWdpdCBhL0RvY3VtZW50YXRpb24vYmxvY2svaTEwLWlvc2NoZWQucnN0 IGIvRG9jdW1lbnRhdGlvbi9ibG9jay9pMTAtaW9zY2hlZC5yc3QKbmV3IGZpbGUgbW9kZSAxMDA2 NDQKaW5kZXggMDAwMDAwMDAwMDAwLi4zZGI3Y2EzZWQ0YzEKLS0tIC9kZXYvbnVsbAorKysgYi9E b2N1bWVudGF0aW9uL2Jsb2NrL2kxMC1pb3NjaGVkLnJzdApAQCAtMCwwICsxLDY5IEBACis9PT09 PT09PT09PT09PT09PT09PT09PT09PQoraTEwIEkvTyBzY2hlZHVsZXIgb3ZlcnZpZXcKKz09PT09 PT09PT09PT09PT09PT09PT09PT09CisKK0kvTyBiYXRjaGluZyBpcyBiZW5lZmljaWFsIGZvciBv cHRpbWl6aW5nIElPUFMgYW5kIHRocm91Z2hwdXQgZm9yIHZhcmlvdXMKK2FwcGxpY2F0aW9ucy4g Rm9yIGluc3RhbmNlLCBzZXZlcmFsIGtlcm5lbCBibG9jayBkcml2ZXJzIHdvdWxkCitiZW5lZml0 IGZyb20gYmF0Y2hpbmcsIGluY2x1ZGluZyBtbWMgWzFdIGFuZCB0Y3AtYmFzZWQgc3RvcmFnZSBk cml2ZXJzIGxpa2UKK252bWUtdGNwIFsyLDNdLiBXaGlsZSB3ZSBoYXZlIHN1cHBvcnQgZm9yIGJh dGNoaW5nIGRpc3BhdGNoIFs0XSwgd2UgbmVlZCBhbgorSS9PIHNjaGVkdWxlciB0byBlZmZpY2ll bnRseSBlbmFibGUgYmF0Y2hpbmcuIFN1Y2ggYSBzY2hlZHVsZXIgaXMgcGFydGljdWxhcmx5Citp bnRlcmVzdGluZyBmb3IgZGlzYWdncmVnYXRlZCBzdG9yYWdlLCB3aGVyZSB0aGUgYWNjZXNzIGxh dGVuY3kgb2YgcmVtb3RlCitkaXNhZ2dyZWdhdGVkIHN0b3JhZ2UgbWF5IGJlIGhpZ2hlciB0aGFu IGxvY2FsIHN0b3JhZ2UgYWNjZXNzOyB0aHVzLCBiYXRjaGluZworY2FuIHNpZ25pZmljYW50bHkg aGVscCBpbiBhbW9ydGl6aW5nIHRoZSByZW1vdGUgYWNjZXNzIGxhdGVuY3kgd2hpbGUgaW5jcmVh c2luZwordGhlIHRocm91Z2hwdXQuCisKK1RoaXMgcGF0Y2ggaW50cm9kdWNlcyB0aGUgaTEwIEkv TyBzY2hlZHVsZXIsIHdoaWNoIHBlcmZvcm1zIGJhdGNoaW5nIHBlciBoY3R4IGluIHRlcm1zCitv ZiAjcmVxdWVzdHMsICNieXRlcywgYW5kIHRpbWVvdXRzIChhdCBtaWNyb3NlY29uZHMgZ3JhbnVs YXJpdHkpLiBpMTAgc3RhcnRzCitkaXNwYXRjaGluZyBvbmx5IHdoZW4gI3JlcXVlc3RzIG9yICNi eXRlcyBpcyBsYXJnZXIgdGhhbiBhIGRlZmF1bHQgdGhyZXNob2xkIG9yIHdoZW4KK2EgdGltZXIg ZXhwaXJlcy4gQWZ0ZXIgdGhhdCwgYmF0Y2hpbmcgZGlzcGF0Y2ggWzNdIHdvdWxkIGhhcHBlbiwg YWxsb3dpbmcgYmF0Y2hpbmcKK2F0IGRldmljZSBkcml2ZXJzIGFsb25nIHdpdGggImJkLT5sYXN0 IiBhbmQgIi5jb21taXRfcnFzIi4KKworVGhlIGkxMCBJL08gc2NoZWR1bGVyIGJ1aWxkcyB1cG9u IHJlY2VudCB3b3JrIG9uIFs2XS4gV2UgaGF2ZSB0ZXN0ZWQgdGhlIGkxMCBJL08KK3NjaGVkdWxl ciB3aXRoIG52bWUtdGNwIG9wdGltaXphaXRvbnMgWzIsM10gYW5kIGJhdGNoaW5nIGRpc3BhdGNo IFs0XSwgdmFyeWluZyBudW1iZXIKK29mIGNvcmVzLCB2YXJ5aW5nIHJlYWQvd3JpdGUgcmF0aW9z LCBhbmQgdmFyeWluZyByZXF1ZXN0IHNpemVzLCBhbmQgd2l0aCBOVk1lIFNTRCBhbmQKK1JBTSBi bG9jayBkZXZpY2UuIEZvciBOVk1lIFNTRHMsIHRoZSBpMTAgSS9PIHNjaGVkdWxlciBhY2hpZXZl cyB+NjAlIGltcHJvdmVtZW50cyBpbgordGVybXMgb2YgSU9QUyBwZXIgY29yZSBvdmVyICJub29w IiBJL08gc2NoZWR1bGVyLiBUaGVzZSByZXN1bHRzIGFyZSBhdmFpbGFibGUgYXQgWzVdLAorYW5k IG1hbnkgYWRkaXRpb25hbCByZXN1bHRzIGFyZSBwcmVzZW50ZWQgaW4gWzZdLgorCitXaGlsZSBv dGhlciBzY2hlZHVsZXJzIG1heSBhbHNvIGJhdGNoIEkvTyAoZS5nLiwgbXEtZGVhZGxpbmUpLCB0 aGUgb3B0aW1pemF0aW9uIHRhcmdldAoraW4gdGhlIGkxMCBJL08gc2NoZWR1bGVyIGlzIHRocm91 Z2hwdXQgbWF4aW1pemF0aW9uLiBIZW5jZSB0aGVyZSBpcyBubyBsYXRlbmN5IHRhcmdldAorbm9y IGEgbmVlZCBmb3IgYSBnbG9iYWwgdHJhY2tpbmcgY29udGV4dCwgc28gYSBuZXcgc2NoZWR1bGVy IGlzIG5lZWRlZCByYXRoZXIgdGhhbgordG8gYnVpbGQgdGhpcyBmdW5jdGlvbmFsaXR5IHRvIGFu IGV4aXN0aW5nIHNjaGVkdWxlci4KKworV2UgY3VycmVudGx5IHVzZSBmaXhlZCBkZWZhdWx0IHZh bHVlcyBhcyBiYXRjaGluZyB0aHJlc2hvbGRzIChlLmcuLCAxNiBmb3IgI3JlcXVlc3RzLAorNjRL QiBmb3IgI2J5dGVzLCBhbmQgNTB1cyBmb3IgdGltZW91dCkuIFRoZXNlIGRlZmF1bHQgdmFsdWVz IGFyZSBiYXNlZCBvbiBzZW5zaXRpdml0eQordGVzdHMgaW4gWzZdLiBGb3Igb3VyIGZ1dHVyZSB3 b3JrLCB3ZSBwbGFuIHRvIHN1cHBvcnQgYWRhcHRpdmUgYmF0Y2hpbmcgYWNjb3JkaW5nIHRvCitz eXN0ZW0gbG9hZCBhbmQgdG8gZXh0ZW5kIHRoZSBzY2hlZHVsZXIgdG8gc3VwcG9ydCBpc29sYXRp b24gaW4gbXVsdGktdGVuYW50IGRlcGxveW1lbnRzCisodG8gc2ltdWx0YW5lb3VzbHkgYWNoaWV2 ZSBsb3cgdGFpbCBsYXRlbmN5IGZvciBsYXRlbmN5LXNlbnNpdGl2ZSBhcHBsaWNhdGlvbnMgYW5k IGhpZ2gKK3Rocm91Z2hwdXQgZm9yIHRocm91Z2hwdXQtYm91bmQgYXBwbGljYXRpb25zKS4KKwor UmVmZXJlbmNlcworWzFdIGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2xpbnV4LWJsb2NrL2NvdmVy LjE1ODc4ODg1MjAuZ2l0LmJhb2xpbi53YW5nN0BnbWFpbC5jb20vVC8jbWM0OGE4ZmI2MDY5ODQz ODI3NDU4ZjVmZWE3MjJlMTE3OWQzMmFmMmEKK1syXSBodHRwczovL2dpdC5pbmZyYWRlYWQub3Jn L252bWUuZ2l0L2NvbW1pdC8xMjJlNWI5ZjNkMzcwYWUxMWUxNTAyZDE0ZmY1YzdlYTliMTQ0YTc2 CitbM10gaHR0cHM6Ly9naXQuaW5mcmFkZWFkLm9yZy9udm1lLmdpdC9jb21taXQvODZmMDM0OGFj ZTE1MTBkN2FjMjUxMjRiMDk2ZmI4OGE2YWI0NTI3MAorWzRdIGh0dHBzOi8vbG9yZS5rZXJuZWwu b3JnL2xpbnV4LWJsb2NrLzIwMjAwNjMwMTAyNTAxLjIyMzg5NzItMS1taW5nLmxlaUByZWRoYXQu Y29tLworWzVdIGh0dHBzOi8vZ2l0aHViLmNvbS9pMTAta2VybmVsL3Vwc3RyZWFtLWxpbnV4L2Js b2IvbWFzdGVyL2Rzcy1ldmFsdWF0aW9uLnBkZgorWzZdIGh0dHBzOi8vd3d3LnVzZW5peC5vcmcv Y29uZmVyZW5jZS9uc2RpMjAvcHJlc2VudGF0aW9uL2h3YW5nCisKKz09PT09PT09PT09PT09PT09 PT09PT09PT09CitpMTAgSS9PIHNjaGVkdWxlciB0dW5hYmxlcworPT09PT09PT09PT09PT09PT09 PT09PT09PT0KKworVGhlIHRocmVlIHR1bmFibGVzIGZvciB0aGUgaTEwIHNjaGVkdWxlciBhcmUg dGhlIG51bWJlciBvZiByZXF1ZXN0cyBmb3IKK3JlYWRzL3dyaXRlcywgdGhlIG51bWJlciBvZiBi eXRlcyBmb3Igd3JpdGVzLCBhbmQgYSB0aW1lb3V0IHZhbHVlLgoraTEwIHdpbGwgdXNlIHRoZXNl IHZhbHVlcyBmb3IgYmF0Y2hpbmcgcmVxdWVzdHMuCisKK2JhdGNoX25yCistLS0tLS0tLQorTnVt YmVyIG9mIHJlcXVlc3RzIGZvciBiYXRjaGluZyByZWFkL3dyaXRlIHJlcXVlc3RzCitEZWZhdWx0 OiAxNgorCitiYXRjaF9ieXRlcworLS0tLS0tLS0tLS0KK051bWJlciBvZiBieXRlcyBmb3IgYmF0 Y2hpbmcgd3JpdGUgcmVxdWVzdHMKK0RlZmF1bHQ6IDY1NTM2IChieXRlcykKKworYmF0Y2hfdGlt ZW91dAorLS0tLS0tLS0tLS0tLQorVGltZW91dCB2YWx1ZSBmb3IgYmF0Y2hpbmcgKGluIG1pY3Jv c2Vjb25kcykKK0RlZmF1bHQ6IDUwICh1cykKZGlmZiAtLWdpdCBhL2Jsb2NrL0tjb25maWcuaW9z Y2hlZCBiL2Jsb2NrL0tjb25maWcuaW9zY2hlZAppbmRleCAyZjIxNThlMDVhOTEuLjViMzYyM2Ix OTQ4NyAxMDA2NDQKLS0tIGEvYmxvY2svS2NvbmZpZy5pb3NjaGVkCisrKyBiL2Jsb2NrL0tjb25m aWcuaW9zY2hlZApAQCAtNDQsNiArNDQsMTQgQEAgY29uZmlnIEJGUV9DR1JPVVBfREVCVUcKIAlF bmFibGUgc29tZSBkZWJ1Z2dpbmcgaGVscC4gQ3VycmVudGx5IGl0IGV4cG9ydHMgYWRkaXRpb25h bCBzdGF0CiAJZmlsZXMgaW4gYSBjZ3JvdXAgd2hpY2ggY2FuIGJlIHVzZWZ1bCBmb3IgZGVidWdn aW5nLgogCitjb25maWcgTVFfSU9TQ0hFRF9JMTAKKwl0cmlzdGF0ZSAiaTEwIEkvTyBzY2hlZHVs ZXIiCisJZGVmYXVsdCB5CisJaGVscAorCSAgVGhlIGkxMCBJL08gU2NoZWR1bGVyIHN1cHBvcnRz IGJhdGNoaW5nIGF0IEJMSy1NUS4KKwkgIEFueSBkZXZpY2UgZHJpdmVyIHRoYXQgYmVuZWZpdHMg ZnJvbSBiYXRjaGluZworCSAgKGUuZy4sIE5WTWUtb3Zlci1UQ1ApIGNhbiB1c2UgdGhpcyBzY2hl ZHVsZXIuCisKIGVuZG1lbnUKIAogZW5kaWYKZGlmZiAtLWdpdCBhL2Jsb2NrL01ha2VmaWxlIGIv YmxvY2svTWFrZWZpbGUKaW5kZXggOGQ4NDFmNWY5ODZmLi4yN2UwNzg5NTg5ZWEgMTAwNjQ0Ci0t LSBhL2Jsb2NrL01ha2VmaWxlCisrKyBiL2Jsb2NrL01ha2VmaWxlCkBAIC0yMSw2ICsyMSw3IEBA IG9iai0kKENPTkZJR19CTEtfQ0dST1VQX0lPTEFURU5DWSkJKz0gYmxrLWlvbGF0ZW5jeS5vCiBv YmotJChDT05GSUdfQkxLX0NHUk9VUF9JT0NPU1QpCSs9IGJsay1pb2Nvc3Qubwogb2JqLSQoQ09O RklHX01RX0lPU0NIRURfREVBRExJTkUpCSs9IG1xLWRlYWRsaW5lLm8KIG9iai0kKENPTkZJR19N UV9JT1NDSEVEX0tZQkVSKQkrPSBreWJlci1pb3NjaGVkLm8KK29iai0kKENPTkZJR19NUV9JT1ND SEVEX0kxMCkgICAgKz0gaTEwLWlvc2NoZWQubwogYmZxLXkJCQkJOj0gYmZxLWlvc2NoZWQubyBi ZnEtd2YycS5vIGJmcS1jZ3JvdXAubwogb2JqLSQoQ09ORklHX0lPU0NIRURfQkZRKQkrPSBiZnEu bwogCmRpZmYgLS1naXQgYS9ibG9jay9pMTAtaW9zY2hlZC5jIGIvYmxvY2svaTEwLWlvc2NoZWQu YwpuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAwMDAuLmI1NDUxYmVhYTY2ZAot LS0gL2Rldi9udWxsCisrKyBiL2Jsb2NrL2kxMC1pb3NjaGVkLmMKQEAgLTAsMCArMSw0MjEgQEAK Ky8vIFNQRFgtTGljZW5zZS1JZGVudGlmaWVyOiBHUEwtMi4wCisvKgorICogVGhlIGkxMCBJL08g U2NoZWR1bGVyIC0gc3VwcG9ydHMgYmF0Y2hpbmcgYXQgYmxrLW1xLgorICoJVGhlIG1haW4gdXNl IGNhc2UgaXMgZGlzYWdncmVnYXRlZCBzdG9yYWdlIGFjY2VzcworICoJdXNpbmcgTlZNZS1vdmVy LUZhYnJpYyAoZS5nLiwgTlZNZS1vdmVyLVRDUCBkZXZpY2UgZHJpdmVyKS4KKyAqCisgKiBBbiBl YXJseSB2ZXJzaW9uIG9mIHRoZSBpZGVhIGlzIGRlc2NyaWJlZCBhbmQgZXZhbHVhdGVkIGluCisg KiAiVENQIOKJiCBSRE1BOiBDUFUtZWZmaWNpZW50IFJlbW90ZSBTdG9yYWdlIEFjY2VzcyB3aXRo IGkxMCIsCisgKiBVU0VOSVggTlNESSAyMDIwLgorICoKKyAqIENvcHlyaWdodCAoQykgMjAyMCBD b3JuZWxsIFVuaXZlcnNpdHkKKyAqCUphZWh5dW4gSHdhbmcgPGphZWh5dW4uaHdhbmdAY29ybmVs bC5lZHU+CisgKglRaXpoZSBDYWkgPHFjMjI4QGNvcm5lbGwuZWR1PgorICoJTWlkaHVsIFZ1cHBh bGFwYXRpIDxtdnYyNUBjb3JuZWxsLmVkdSU+CisgKglSYWNoaXQgQWdhcndhbCA8cmFnYXJ3YWxA Y29ybmVsbC5lZHU+CisgKi8KKworI2luY2x1ZGUgPGxpbnV4L2tlcm5lbC5oPgorI2luY2x1ZGUg PGxpbnV4L2Jsa2Rldi5oPgorI2luY2x1ZGUgPGxpbnV4L2Jsay1tcS5oPgorI2luY2x1ZGUgPGxp bnV4L2VsZXZhdG9yLmg+CisjaW5jbHVkZSA8bGludXgvbW9kdWxlLmg+CisjaW5jbHVkZSA8bGlu dXgvc2JpdG1hcC5oPgorCisjaW5jbHVkZSAiYmxrLmgiCisjaW5jbHVkZSAiYmxrLW1xLmgiCisj aW5jbHVkZSAiYmxrLW1xLWRlYnVnZnMuaCIKKyNpbmNsdWRlICJibGstbXEtc2NoZWQuaCIKKyNp bmNsdWRlICJibGstbXEtdGFnLmgiCisKKy8qIERlZmF1bHQgYmF0Y2ggc2l6ZSBpbiBudW1iZXIg b2YgcmVxdWVzdHMgKi8KKyNkZWZpbmUgSTEwX0RFRl9CQVRDSF9OUgkxNgorLyogRGVmYXVsdCBi YXRjaCBzaXplIGluIGJ5dGVzIChmb3Igd3JpdGUgcmVxdWVzdHMpICovCisjZGVmaW5lIEkxMF9E RUZfQkFUQ0hfQllURVMJNjU1MzYKKy8qIERlZmF1bHQgdGltZW91dCB2YWx1ZSBmb3IgYmF0Y2hp bmcgKHVzIHVuaXRzKSAqLworI2RlZmluZSBJMTBfREVGX0JBVENIX1RJTUVPVVQJNTAKKworZW51 bSBpMTBfc3RhdGUgeworCS8qIEJhdGNoaW5nIHN0YXRlOgorCSAqIERvIG5vdCBydW4gZGlzcGF0 Y2hpbmcgdW50aWwgd2UgaGF2ZQorCSAqIGEgY2VydGFpbiBhbW91bnQgb2YgcmVxdWVzdHMgb3Ig YSB0aW1lciBleHBpcmVzLgorCSAqLworCUkxMF9TVEFURV9CQVRDSCwKKworCS8qIERpc3BhdGNo aW5nIHN0YXRlOgorCSAqIFJ1biBkaXNwYXRjaGluZyB1bnRpbCBhbGwgcmVxdWVzdHMgaW4gdGhl CisJICogc2NoZWR1bGVyJ3MgaGN0eCBpaHEgYXJlIGRpc3BhdGNoZWQuCisJICovCisJSTEwX1NU QVRFX0RJU1BBVENILAorfTsKKworc3RydWN0IGkxMF9xdWV1ZV9kYXRhIHsKKwlzdHJ1Y3QgcmVx dWVzdF9xdWV1ZSAqcTsKKworCXVuc2lnbmVkIGludAlkZWZfYmF0Y2hfbnI7CisJdW5zaWduZWQg aW50CWRlZl9iYXRjaF9ieXRlczsKKwl1bnNpZ25lZCBpbnQJZGVmX2JhdGNoX3RpbWVvdXQ7Cit9 OworCitzdHJ1Y3QgaTEwX2hjdHhfcXVldWUgeworCXNwaW5sb2NrX3QJCWxvY2s7CisJc3RydWN0 IGxpc3RfaGVhZAlycV9saXN0OworCisJc3RydWN0IGJsa19tcV9od19jdHgJKmhjdHg7CisKKwl1 bnNpZ25lZCBpbnQJYmF0Y2hfbnI7CisJdW5zaWduZWQgaW50CWJhdGNoX2J5dGVzOworCXVuc2ln bmVkIGludAliYXRjaF90aW1lb3V0OworCisJdW5zaWduZWQgaW50CXFsZW5fbnI7CisJdW5zaWdu ZWQgaW50CXFsZW5fYnl0ZXM7CisKKwlzdHJ1Y3QgaHJ0aW1lcglkaXNwYXRjaF90aW1lcjsKKwll bnVtIGkxMF9zdGF0ZQlzdGF0ZTsKK307CisKK3N0YXRpYyBzdHJ1Y3QgaTEwX3F1ZXVlX2RhdGEg KmkxMF9xdWV1ZV9kYXRhX2FsbG9jKHN0cnVjdCByZXF1ZXN0X3F1ZXVlICpxKQoreworCXN0cnVj dCBpMTBfcXVldWVfZGF0YSAqaXFkOworCisJaXFkID0ga3phbGxvY19ub2RlKHNpemVvZigqaXFk KSwgR0ZQX0tFUk5FTCwgcS0+bm9kZSk7CisJaWYgKCFpcWQpCisJCXJldHVybiBFUlJfUFRSKC1F Tk9NRU0pOworCisJaXFkLT5xID0gcTsKKwlpcWQtPmRlZl9iYXRjaF9uciA9IEkxMF9ERUZfQkFU Q0hfTlI7CisJaXFkLT5kZWZfYmF0Y2hfYnl0ZXMgPSBJMTBfREVGX0JBVENIX0JZVEVTOworCWlx ZC0+ZGVmX2JhdGNoX3RpbWVvdXQgPSBJMTBfREVGX0JBVENIX1RJTUVPVVQ7CisKKwlyZXR1cm4g aXFkOworfQorCitzdGF0aWMgaW50IGkxMF9pbml0X3NjaGVkKHN0cnVjdCByZXF1ZXN0X3F1ZXVl ICpxLCBzdHJ1Y3QgZWxldmF0b3JfdHlwZSAqZSkKK3sKKwlzdHJ1Y3QgaTEwX3F1ZXVlX2RhdGEg KmlxZDsKKwlzdHJ1Y3QgZWxldmF0b3JfcXVldWUgKmVxOworCisJZXEgPSBlbGV2YXRvcl9hbGxv YyhxLCBlKTsKKwlpZiAoIWVxKQorCQlyZXR1cm4gLUVOT01FTTsKKworCWlxZCA9IGkxMF9xdWV1 ZV9kYXRhX2FsbG9jKHEpOworCWlmIChJU19FUlIoaXFkKSkgeworCQlrb2JqZWN0X3B1dCgmZXEt PmtvYmopOworCQlyZXR1cm4gUFRSX0VSUihpcWQpOworCX0KKworCWJsa19zdGF0X2VuYWJsZV9h Y2NvdW50aW5nKHEpOworCisJZXEtPmVsZXZhdG9yX2RhdGEgPSBpcWQ7CisJcS0+ZWxldmF0b3Ig PSBlcTsKKworCXJldHVybiAwOworfQorCitzdGF0aWMgdm9pZCBpMTBfZXhpdF9zY2hlZChzdHJ1 Y3QgZWxldmF0b3JfcXVldWUgKmUpCit7CisJc3RydWN0IGkxMF9xdWV1ZV9kYXRhICppcWQgPSBl LT5lbGV2YXRvcl9kYXRhOworCisJa2ZyZWUoaXFkKTsKK30KKworZW51bSBocnRpbWVyX3Jlc3Rh cnQgaTEwX2hjdHhfdGltZW91dF9oYW5kbGVyKHN0cnVjdCBocnRpbWVyICp0aW1lcikKK3sKKwlz dHJ1Y3QgaTEwX2hjdHhfcXVldWUgKmlocSA9CisJCWNvbnRhaW5lcl9vZih0aW1lciwgc3RydWN0 IGkxMF9oY3R4X3F1ZXVlLAorCQkJZGlzcGF0Y2hfdGltZXIpOworCisJaWhxLT5zdGF0ZSA9IEkx MF9TVEFURV9ESVNQQVRDSDsKKwlibGtfbXFfcnVuX2h3X3F1ZXVlKGlocS0+aGN0eCwgdHJ1ZSk7 CisKKwlyZXR1cm4gSFJUSU1FUl9OT1JFU1RBUlQ7Cit9CisKK3N0YXRpYyB2b2lkIGkxMF9oY3R4 X3F1ZXVlX3Jlc2V0KHN0cnVjdCBpMTBfaGN0eF9xdWV1ZSAqaWhxKQoreworCWlocS0+cWxlbl9u ciA9IDA7CisJaWhxLT5xbGVuX2J5dGVzID0gMDsKKwlpaHEtPnN0YXRlID0gSTEwX1NUQVRFX0JB VENIOworfQorCitzdGF0aWMgaW50IGkxMF9pbml0X2hjdHgoc3RydWN0IGJsa19tcV9od19jdHgg KmhjdHgsIHVuc2lnbmVkIGludCBoY3R4X2lkeCkKK3sKKwlzdHJ1Y3QgaTEwX2hjdHhfcXVldWUg KmlocTsKKworCWlocSA9IGt6YWxsb2Nfbm9kZShzaXplb2YoKmlocSksIEdGUF9LRVJORUwsIGhj dHgtPm51bWFfbm9kZSk7CisJaWYgKCFpaHEpCisJCXJldHVybiAtRU5PTUVNOworCisJc3Bpbl9s b2NrX2luaXQoJmlocS0+bG9jayk7CisJSU5JVF9MSVNUX0hFQUQoJmlocS0+cnFfbGlzdCk7CisK KwlpaHEtPmhjdHggPSBoY3R4OworCWlocS0+YmF0Y2hfbnIgPSAwOworCWlocS0+YmF0Y2hfYnl0 ZXMgPSAwOworCWlocS0+YmF0Y2hfdGltZW91dCA9IDA7CisKKwlocnRpbWVyX2luaXQoJmlocS0+ ZGlzcGF0Y2hfdGltZXIsCisJCUNMT0NLX01PTk9UT05JQywgSFJUSU1FUl9NT0RFX1JFTCk7CisJ aWhxLT5kaXNwYXRjaF90aW1lci5mdW5jdGlvbiA9ICZpMTBfaGN0eF90aW1lb3V0X2hhbmRsZXI7 CisKKwlpMTBfaGN0eF9xdWV1ZV9yZXNldChpaHEpOworCisJaGN0eC0+c2NoZWRfZGF0YSA9IGlo cTsKKworCXJldHVybiAwOworfQorCitzdGF0aWMgdm9pZCBpMTBfZXhpdF9oY3R4KHN0cnVjdCBi bGtfbXFfaHdfY3R4ICpoY3R4LCB1bnNpZ25lZCBpbnQgaGN0eF9pZHgpCit7CisJc3RydWN0IGkx MF9oY3R4X3F1ZXVlICppaHEgPSBoY3R4LT5zY2hlZF9kYXRhOworCisJaHJ0aW1lcl9jYW5jZWwo JmlocS0+ZGlzcGF0Y2hfdGltZXIpOworCWtmcmVlKGhjdHgtPnNjaGVkX2RhdGEpOworfQorCitz dGF0aWMgYm9vbCBpMTBfaGN0eF9iaW9fbWVyZ2Uoc3RydWN0IGJsa19tcV9od19jdHggKmhjdHgs IHN0cnVjdCBiaW8gKmJpbywKKwkJdW5zaWduZWQgaW50IG5yX3NlZ3MpCit7CisJc3RydWN0IGkx MF9oY3R4X3F1ZXVlICppaHEgPSBoY3R4LT5zY2hlZF9kYXRhOworCXN0cnVjdCBsaXN0X2hlYWQg KnJxX2xpc3QgPSAmaWhxLT5ycV9saXN0OworCWJvb2wgbWVyZ2VkOworCisJc3Bpbl9sb2NrKCZp aHEtPmxvY2spOworCW1lcmdlZCA9IGJsa19tcV9iaW9fbGlzdF9tZXJnZShoY3R4LT5xdWV1ZSwg cnFfbGlzdCwgYmlvLCBucl9zZWdzKTsKKwlzcGluX3VubG9jaygmaWhxLT5sb2NrKTsKKworCWlm IChtZXJnZWQgJiYgYmlvX2RhdGFfZGlyKGJpbykgPT0gV1JJVEUpCisJCWlocS0+cWxlbl9ieXRl cyArPSBiaW8tPmJpX2l0ZXIuYmlfc2l6ZTsKKworCXJldHVybiBtZXJnZWQ7Cit9CisKKy8qCisg KiBUaGUgYmF0Y2ggc2l6ZSBjYW4gYmUgYWRqdXN0ZWQgZHluYW1pY2FsbHkgb24gYSBwZXItaGN0 eCBiYXNpcy4KKyAqIFVzZSBwZXItaGN0eCB2YXJpYWJsZXMgaW4gdGhhdCBjYXNlLgorICovCitz dGF0aWMgaW5saW5lIHVuc2lnbmVkIGludCBpMTBfaGN0eF9iYXRjaF9ucihzdHJ1Y3QgYmxrX21x X2h3X2N0eCAqaGN0eCkKK3sKKwlzdHJ1Y3QgaTEwX3F1ZXVlX2RhdGEgKmlxZCA9IGhjdHgtPnF1 ZXVlLT5lbGV2YXRvci0+ZWxldmF0b3JfZGF0YTsKKwlzdHJ1Y3QgaTEwX2hjdHhfcXVldWUgKmlo cSA9IGhjdHgtPnNjaGVkX2RhdGE7CisKKwlyZXR1cm4gaWhxLT5iYXRjaF9uciA/CisJCWlocS0+ YmF0Y2hfbnIgOiBpcWQtPmRlZl9iYXRjaF9ucjsKK30KKworc3RhdGljIGlubGluZSB1bnNpZ25l ZCBpbnQgaTEwX2hjdHhfYmF0Y2hfYnl0ZXMoc3RydWN0IGJsa19tcV9od19jdHggKmhjdHgpCit7 CisJc3RydWN0IGkxMF9xdWV1ZV9kYXRhICppcWQgPSBoY3R4LT5xdWV1ZS0+ZWxldmF0b3ItPmVs ZXZhdG9yX2RhdGE7CisJc3RydWN0IGkxMF9oY3R4X3F1ZXVlICppaHEgPSBoY3R4LT5zY2hlZF9k YXRhOworCisJcmV0dXJuIGlocS0+YmF0Y2hfYnl0ZXMgPworCQlpaHEtPmJhdGNoX2J5dGVzIDog aXFkLT5kZWZfYmF0Y2hfYnl0ZXM7Cit9CisKK3N0YXRpYyBpbmxpbmUgdW5zaWduZWQgaW50IGkx MF9oY3R4X2JhdGNoX3RpbWVvdXQoc3RydWN0IGJsa19tcV9od19jdHggKmhjdHgpCit7CisJc3Ry dWN0IGkxMF9xdWV1ZV9kYXRhICppcWQgPSBoY3R4LT5xdWV1ZS0+ZWxldmF0b3ItPmVsZXZhdG9y X2RhdGE7CisJc3RydWN0IGkxMF9oY3R4X3F1ZXVlICppaHEgPSBoY3R4LT5zY2hlZF9kYXRhOwor CisJcmV0dXJuIGlocS0+YmF0Y2hfdGltZW91dCA/CisJCWlocS0+YmF0Y2hfdGltZW91dCA6IGlx ZC0+ZGVmX2JhdGNoX3RpbWVvdXQ7Cit9CisKK3N0YXRpYyB2b2lkIGkxMF9oY3R4X2luc2VydF91 cGRhdGUoc3RydWN0IGkxMF9oY3R4X3F1ZXVlICppaHEsCisJCQkJc3RydWN0IHJlcXVlc3QgKnJx KQoreworCWlmIChycV9kYXRhX2RpcihycSkgPT0gV1JJVEUpCisJCWlocS0+cWxlbl9ieXRlcyAr PSBibGtfcnFfYnl0ZXMocnEpOworCWlocS0+cWxlbl9ucisrOworfQorCitzdGF0aWMgdm9pZCBp MTBfaGN0eF9pbnNlcnRfcmVxdWVzdHMoc3RydWN0IGJsa19tcV9od19jdHggKmhjdHgsCisJCQkJ c3RydWN0IGxpc3RfaGVhZCAqcnFfbGlzdCwgYm9vbCBhdF9oZWFkKQoreworCXN0cnVjdCBpMTBf aGN0eF9xdWV1ZSAqaWhxID0gaGN0eC0+c2NoZWRfZGF0YTsKKwlzdHJ1Y3QgcmVxdWVzdCAqcnEs ICpuZXh0OworCisJbGlzdF9mb3JfZWFjaF9lbnRyeV9zYWZlKHJxLCBuZXh0LCBycV9saXN0LCBx dWV1ZWxpc3QpIHsKKwkJc3RydWN0IGxpc3RfaGVhZCAqaGVhZCA9ICZpaHEtPnJxX2xpc3Q7CisK KwkJc3Bpbl9sb2NrKCZpaHEtPmxvY2spOworCQlpZiAoYXRfaGVhZCkKKwkJCWxpc3RfbW92ZSgm cnEtPnF1ZXVlbGlzdCwgaGVhZCk7CisJCWVsc2UKKwkJCWxpc3RfbW92ZV90YWlsKCZycS0+cXVl dWVsaXN0LCBoZWFkKTsKKwkJaTEwX2hjdHhfaW5zZXJ0X3VwZGF0ZShpaHEsIHJxKTsKKwkJYmxr X21xX3NjaGVkX3JlcXVlc3RfaW5zZXJ0ZWQocnEpOworCQlzcGluX3VubG9jaygmaWhxLT5sb2Nr KTsKKwl9CisKKwkvKiBTdGFydCBhIG5ldyB0aW1lciAqLworCWlmIChpaHEtPnN0YXRlID09IEkx MF9TVEFURV9CQVRDSCAmJgorCSAgICFocnRpbWVyX2FjdGl2ZSgmaWhxLT5kaXNwYXRjaF90aW1l cikpCisJCWhydGltZXJfc3RhcnQoJmlocS0+ZGlzcGF0Y2hfdGltZXIsCisJCQluc190b19rdGlt ZShpMTBfaGN0eF9iYXRjaF90aW1lb3V0KGhjdHgpCisJCQkJKiBOU0VDX1BFUl9VU0VDKSwKKwkJ CUhSVElNRVJfTU9ERV9SRUwpOworfQorCitzdGF0aWMgc3RydWN0IHJlcXVlc3QgKmkxMF9oY3R4 X2Rpc3BhdGNoX3JlcXVlc3Qoc3RydWN0IGJsa19tcV9od19jdHggKmhjdHgpCit7CisJc3RydWN0 IGkxMF9oY3R4X3F1ZXVlICppaHEgPSBoY3R4LT5zY2hlZF9kYXRhOworCXN0cnVjdCByZXF1ZXN0 ICpycTsKKworCXNwaW5fbG9jaygmaWhxLT5sb2NrKTsKKwlycSA9IGxpc3RfZmlyc3RfZW50cnlf b3JfbnVsbCgmaWhxLT5ycV9saXN0LAorCQkJCXN0cnVjdCByZXF1ZXN0LCBxdWV1ZWxpc3QpOwor CWlmIChycSkKKwkJbGlzdF9kZWxfaW5pdCgmcnEtPnF1ZXVlbGlzdCk7CisJZWxzZQorCQlpMTBf aGN0eF9xdWV1ZV9yZXNldChpaHEpOworCXNwaW5fdW5sb2NrKCZpaHEtPmxvY2spOworCisJcmV0 dXJuIHJxOworfQorCitzdGF0aWMgaW5saW5lIGJvb2wgaTEwX2hjdHhfZGlzcGF0Y2hfbm93KHN0 cnVjdCBibGtfbXFfaHdfY3R4ICpoY3R4KQoreworCXN0cnVjdCBpMTBfaGN0eF9xdWV1ZSAqaWhx ID0gaGN0eC0+c2NoZWRfZGF0YTsKKworCXJldHVybiAoaWhxLT5xbGVuX25yID49IGkxMF9oY3R4 X2JhdGNoX25yKGhjdHgpKSB8fAorCQkoaWhxLT5xbGVuX2J5dGVzID49IGkxMF9oY3R4X2JhdGNo X2J5dGVzKGhjdHgpKTsKK30KKworLyoKKyAqIFJldHVybiB0cnVlIGlmIHdlIGFyZSBpbiB0aGUg ZGlzcGF0Y2hpbmcgc3RhdGUuCisgKi8KK3N0YXRpYyBib29sIGkxMF9oY3R4X2hhc193b3JrKHN0 cnVjdCBibGtfbXFfaHdfY3R4ICpoY3R4KQoreworCXN0cnVjdCBpMTBfaGN0eF9xdWV1ZSAqaWhx ID0gaGN0eC0+c2NoZWRfZGF0YTsKKworCWlmIChpaHEtPnN0YXRlID09IEkxMF9TVEFURV9CQVRD SCkgeworCQlpZiAoaTEwX2hjdHhfZGlzcGF0Y2hfbm93KGhjdHgpKSB7CisJCQlpaHEtPnN0YXRl ID0gSTEwX1NUQVRFX0RJU1BBVENIOworCQkJaWYgKGhydGltZXJfYWN0aXZlKCZpaHEtPmRpc3Bh dGNoX3RpbWVyKSkKKwkJCQlocnRpbWVyX2NhbmNlbCgmaWhxLT5kaXNwYXRjaF90aW1lcik7CisJ CX0KKwl9CisKKwlyZXR1cm4gKGlocS0+c3RhdGUgPT0gSTEwX1NUQVRFX0RJU1BBVENIKTsKK30K KworI2RlZmluZSBJMTBfREVGX0JBVENIX1NIT1dfU1RPUkUobmFtZSkJCQkJCVwKK3N0YXRpYyBz c2l6ZV90IGkxMF9kZWZfYmF0Y2hfIyNuYW1lIyNfc2hvdyhzdHJ1Y3QgZWxldmF0b3JfcXVldWUg KmUsCVwKKwkJCQljaGFyICpwYWdlKQkJCQlcCit7CQkJCQkJCQkJXAorCXN0cnVjdCBpMTBfcXVl dWVfZGF0YSAqaXFkID0gZS0+ZWxldmF0b3JfZGF0YTsJCQlcCisJCQkJCQkJCQlcCisJcmV0dXJu IHNwcmludGYocGFnZSwgIiV1XG4iLCBpcWQtPmRlZl9iYXRjaF8jI25hbWUpOwkJXAorfQkJCQkJ CQkJCVwKKwkJCQkJCQkJCVwKK3N0YXRpYyBzc2l6ZV90IGkxMF9kZWZfYmF0Y2hfIyNuYW1lIyNf c3RvcmUoc3RydWN0IGVsZXZhdG9yX3F1ZXVlICplLAlcCisJCQljb25zdCBjaGFyICpwYWdlLCBz aXplX3QgY291bnQpCQkJXAorewkJCQkJCQkJCVwKKwlzdHJ1Y3QgaTEwX3F1ZXVlX2RhdGEgKmlx ZCA9IGUtPmVsZXZhdG9yX2RhdGE7CQkJXAorCXVuc2lnbmVkIGxvbmcgbG9uZyB2YWx1ZTsJCQkJ CVwKKwlpbnQgcmV0OwkJCQkJCQlcCisJCQkJCQkJCQlcCisJcmV0ID0ga3N0cnRvdWxsKHBhZ2Us IDEwLCAmdmFsdWUpOwkJCQlcCisJaWYgKHJldCkJCQkJCQkJXAorCQlyZXR1cm4gcmV0OwkJCQkJ CVwKKwkJCQkJCQkJCVwKKwlpcWQtPmRlZl9iYXRjaF8jI25hbWUgPSB2YWx1ZTsJCQkJCVwKKwkJ CQkJCQkJCVwKKwlyZXR1cm4gY291bnQ7CQkJCQkJCVwKK30KK0kxMF9ERUZfQkFUQ0hfU0hPV19T VE9SRShucik7CitJMTBfREVGX0JBVENIX1NIT1dfU1RPUkUoYnl0ZXMpOworSTEwX0RFRl9CQVRD SF9TSE9XX1NUT1JFKHRpbWVvdXQpOworI3VuZGVmIEkxMF9ERUZfQkFUQ0hfU0hPV19TVE9SRQor CisjZGVmaW5lIEkxMF9TQ0hFRF9BVFRSKG5hbWUpCVwKKwlfX0FUVFIoYmF0Y2hfIyNuYW1lLCAw NjQ0LCBpMTBfZGVmX2JhdGNoXyMjbmFtZSMjX3Nob3csIGkxMF9kZWZfYmF0Y2hfIyNuYW1lIyNf c3RvcmUpCitzdGF0aWMgc3RydWN0IGVsdl9mc19lbnRyeSBpMTBfc2NoZWRfYXR0cnNbXSA9IHsK KwlJMTBfU0NIRURfQVRUUihuciksCisJSTEwX1NDSEVEX0FUVFIoYnl0ZXMpLAorCUkxMF9TQ0hF RF9BVFRSKHRpbWVvdXQpLAorCV9fQVRUUl9OVUxMCit9OworI3VuZGVmIEkxMF9TQ0hFRF9BVFRS CisKKyNpZmRlZiBDT05GSUdfQkxLX0RFQlVHX0ZTCisjZGVmaW5lIEkxMF9ERUJVR0ZTX1NIT1co bmFtZSkJXAorc3RhdGljIGludCBpMTBfaGN0eF9iYXRjaF8jI25hbWUjI19zaG93KHZvaWQgKmRh dGEsIHN0cnVjdCBzZXFfZmlsZSAqbSkJXAorewkJCQkJCQkJCVwKKwlzdHJ1Y3QgYmxrX21xX2h3 X2N0eCAqaGN0eCA9IGRhdGE7CQkJCVwKKwlzdHJ1Y3QgaTEwX2hjdHhfcXVldWUgKmlocSA9IGhj dHgtPnNjaGVkX2RhdGE7CQkJXAorCQkJCQkJCQkJXAorCXNlcV9wcmludGYobSwgIiV1XG4iLCBp aHEtPmJhdGNoXyMjbmFtZSk7CQkJXAorCXJldHVybiAwOwkJCQkJCQlcCit9CQkJCQkJCQkJXAor CQkJCQkJCQkJXAorc3RhdGljIGludCBpMTBfaGN0eF9xbGVuXyMjbmFtZSMjX3Nob3codm9pZCAq ZGF0YSwgc3RydWN0IHNlcV9maWxlICptKQlcCit7CQkJCQkJCQkJXAorCXN0cnVjdCBibGtfbXFf aHdfY3R4ICpoY3R4ID0gZGF0YTsJCQkJXAorCXN0cnVjdCBpMTBfaGN0eF9xdWV1ZSAqaWhxID0g aGN0eC0+c2NoZWRfZGF0YTsJCQlcCisJCQkJCQkJCQlcCisJc2VxX3ByaW50ZihtLCAiJXVcbiIs IGlocS0+cWxlbl8jI25hbWUpOwkJCVwKKwlyZXR1cm4gMDsJCQkJCQkJXAorfQorSTEwX0RFQlVH RlNfU0hPVyhucik7CitJMTBfREVCVUdGU19TSE9XKGJ5dGVzKTsKKyN1bmRlZiBJMTBfREVCVUdG U19TSE9XCisKK3N0YXRpYyBpbnQgaTEwX2hjdHhfc3RhdGVfc2hvdyh2b2lkICpkYXRhLCBzdHJ1 Y3Qgc2VxX2ZpbGUgKm0pCit7CisJc3RydWN0IGJsa19tcV9od19jdHggKmhjdHggPSBkYXRhOwor CXN0cnVjdCBpMTBfaGN0eF9xdWV1ZSAqaWhxID0gaGN0eC0+c2NoZWRfZGF0YTsKKworCXNlcV9w cmludGYobSwgIiVkXG4iLCBpaHEtPnN0YXRlKTsKKwlyZXR1cm4gMDsKK30KKworI2RlZmluZSBJ MTBfSENUWF9RVUVVRV9BVFRSKG5hbWUpCQkJCQlcCisJeyJiYXRjaF8iICNuYW1lLCAwNDAwLCBp MTBfaGN0eF9iYXRjaF8jI25hbWUjI19zaG93fSwJCVwKKwl7InFsZW5fIiAjbmFtZSwgMDQwMCwg aTEwX2hjdHhfcWxlbl8jI25hbWUjI19zaG93fQorc3RhdGljIGNvbnN0IHN0cnVjdCBibGtfbXFf ZGVidWdmc19hdHRyIGkxMF9oY3R4X2RlYnVnZnNfYXR0cnNbXSA9IHsKKwlJMTBfSENUWF9RVUVV RV9BVFRSKG5yKSwKKwlJMTBfSENUWF9RVUVVRV9BVFRSKGJ5dGVzKSwKKwl7InN0YXRlIiwgMDQw MCwgaTEwX2hjdHhfc3RhdGVfc2hvd30sCisJe30sCit9OworI3VuZGVmIEkxMF9IQ1RYX1FVRVVF X0FUVFIKKyNlbmRpZgorCitzdGF0aWMgc3RydWN0IGVsZXZhdG9yX3R5cGUgaTEwX3NjaGVkID0g eworCS5vcHMgPSB7CisJCS5pbml0X3NjaGVkID0gaTEwX2luaXRfc2NoZWQsCisJCS5leGl0X3Nj aGVkID0gaTEwX2V4aXRfc2NoZWQsCisJCS5pbml0X2hjdHggPSBpMTBfaW5pdF9oY3R4LAorCQku ZXhpdF9oY3R4ID0gaTEwX2V4aXRfaGN0eCwKKwkJLmJpb19tZXJnZSA9IGkxMF9oY3R4X2Jpb19t ZXJnZSwKKwkJLmluc2VydF9yZXF1ZXN0cyA9IGkxMF9oY3R4X2luc2VydF9yZXF1ZXN0cywKKwkJ LmRpc3BhdGNoX3JlcXVlc3QgPSBpMTBfaGN0eF9kaXNwYXRjaF9yZXF1ZXN0LAorCQkuaGFzX3dv cmsgPSBpMTBfaGN0eF9oYXNfd29yaywKKwl9LAorI2lmZGVmIENPTkZJR19CTEtfREVCVUdfRlMK KwkuaGN0eF9kZWJ1Z2ZzX2F0dHJzID0gaTEwX2hjdHhfZGVidWdmc19hdHRycywKKyNlbmRpZgor CS5lbGV2YXRvcl9hdHRycyA9IGkxMF9zY2hlZF9hdHRycywKKwkuZWxldmF0b3JfbmFtZSA9ICJp MTAiLAorCS5lbGV2YXRvcl9vd25lciA9IFRISVNfTU9EVUxFLAorfTsKKworc3RhdGljIGludCBf X2luaXQgaTEwX2luaXQodm9pZCkKK3sKKwlyZXR1cm4gZWx2X3JlZ2lzdGVyKCZpMTBfc2NoZWQp OworfQorCitzdGF0aWMgdm9pZCBfX2V4aXQgaTEwX2V4aXQodm9pZCkKK3sKKwllbHZfdW5yZWdp c3RlcigmaTEwX3NjaGVkKTsKK30KKworbW9kdWxlX2luaXQoaTEwX2luaXQpOworbW9kdWxlX2V4 aXQoaTEwX2V4aXQpOworCitNT0RVTEVfQVVUSE9SKCJKYWVoeXVuIEh3YW5nLCBRaXpoZSBDYWks IE1pZGh1bCBWdXBwYWxhcGF0aSwgUmFjaGl0IEFnYXJ3YWwiKTsKK01PRFVMRV9MSUNFTlNFKCJH UEx2MiIpOworTU9EVUxFX0RFU0NSSVBUSU9OKCJpMTAgSS9PIHNjaGVkdWxlciIpOwotLSAKMi4y Mi4wCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KTGlu dXgtbnZtZSBtYWlsaW5nIGxpc3QKTGludXgtbnZtZUBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6 Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtbnZtZQo=