From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77F2CC433E1 for ; Wed, 10 Jun 2020 12:34:13 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 49929206F4 for ; Wed, 10 Jun 2020 12:34:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="HAxYu+GS"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=mediatek.com header.i=@mediatek.com header.b="HOdaBg95" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 49929206F4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=mediatek.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JyI8g885jtOOnTX2/DSCp+OOind0xwQwMTn6gNqteVs=; b=HAxYu+GSQySdDV +T37GP4PI6OpOvXVldc68dt6sPUFfQVjK2EDAGLBV7lSeF8ewcOv8T0ehpkbbOdFdZkONUqDt/LXX Aakxbd7WhxdAJJiWcNEKaulJeUNfHVsfipjpytM5+5/7kTFFRfvONl86bszPc1po/mkZCLB5KN1Y8 yNeAlbnCH3HnfSrDHgznt6dO5X244HfvPZsvCfSVML15tTchib24bqXzMYhryqyZkioDaSgRJx5HP OIOygmkaknMS/a2WWsO+H+XuwaGqSDHQcvNGgNVAZ5UnHzvcBXWUDekugdM5zY04chy5prqm4tiqW PJ/gGFFoqEbgmegwG3Sg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jizvi-0004D5-Hp; Wed, 10 Jun 2020 12:34:06 +0000 Received: from mailgw02.mediatek.com ([216.200.240.185]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jizvg-0004Bk-2e for linux-mediatek@lists.infradead.org; Wed, 10 Jun 2020 12:34:05 +0000 X-UUID: 6cee195427b142d6bcca5e112b44b981-20200610 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mediatek.com; s=dk; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:CC:To:From; bh=3UtQtFG20+VKtiEXcQZp+bogS+1pf5FFGXrTuHq/DJ8=; b=HOdaBg95cNZgZj8eNBJ7gfjho/jdcvtWuV0lEBAe6sNxlVn+gsF5dv8Y7L0nPHrFVOzv6CBnBW0URDJV6tztL91s0RYjcQTnyJ+ZvQ5/3p7VN4oXmiyD+V0H0MKd3JvAAbGEQ2md5RjIP2nr3MpvLfHfy32Zb4iDjpEVdjy7jts=; X-UUID: 6cee195427b142d6bcca5e112b44b981-20200610 Received: from mtkcas66.mediatek.inc [(172.29.193.44)] by mailgw02.mediatek.com (envelope-from ) (musrelay.mediatek.com ESMTP with TLS) with ESMTP id 734891992; Wed, 10 Jun 2020 04:33:52 -0800 Received: from MTKMBS01N1.mediatek.inc (172.21.101.68) by MTKMBS62N1.mediatek.inc (172.29.193.41) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 10 Jun 2020 05:23:55 -0700 Received: from mtkcas08.mediatek.inc (172.21.101.126) by mtkmbs01n1.mediatek.inc (172.21.101.68) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 10 Jun 2020 20:23:49 +0800 Received: from mtkswgap22.mediatek.inc (172.21.77.33) by mtkcas08.mediatek.inc (172.21.101.73) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 10 Jun 2020 20:23:49 +0800 From: Frankie Chang To: Todd Kjos , Greg Kroah-Hartman Subject: [PATCH v5 3/3] binder: add transaction latency tracer Date: Wed, 10 Jun 2020 20:23:47 +0800 Message-ID: <1591791827-23871-4-git-send-email-Frankie.Chang@mediatek.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1591791827-23871-1-git-send-email-Frankie.Chang@mediatek.com> References: <20200507085544.GB1097552@kroah.com> <1591791827-23871-1-git-send-email-Frankie.Chang@mediatek.com> MIME-Version: 1.0 X-MTK: N X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200610_053404_126239_B477BBC7 X-CRM114-Status: GOOD ( 18.83 ) X-BeenThere: linux-mediatek@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wsd_upstream , LKML , =?UTF-8?q?Arve=20Hj=C3=B8nnev=C3=A5g?= , Jian-Min Liu , linux-mediatek@lists.infradead.org, Joel Fernandes , "Frankie.Chang" , Martijn Coenen , Christian Brauner Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Linux-mediatek" Errors-To: linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org From: "Frankie.Chang" Record start/end timestamp for binder transaction. When transaction is completed or transaction is free, it would be checked if transaction latency over threshold (2 sec), if yes, printing related information for tracing. /* Implement details */ - Add latency tracer module to monitor transaction by attaching to new tracepoints introduced when transactions are allocated and freed. The trace_binder_txn_latency_free would not be enabled by default. Monitoring which transaction is too slow to cause some of exceptions is important. So we hook the tracepoint to call the monitor function. - Since some of modules would trigger timeout NE if their binder transaction don't finish in time, such as audio timeout (5 sec), even BT command timeout (2 sec), etc. Therefore, setting the timeout threshold as 2 seconds could be helpful to debug. - The reason why printing the related information to kernel information log but not trace buffer is that some abnormal transactions may be pending for a long time ago, they could not be recorded due to buffer limited. Signed-off-by: Frankie.Chang --- drivers/android/Kconfig | 8 +++ drivers/android/Makefile | 1 + drivers/android/binder.c | 2 + drivers/android/binder_internal.h | 13 ++++ drivers/android/binder_latency_tracer.c | 107 +++++++++++++++++++++++++++++++ drivers/android/binder_trace.h | 28 +++++++- 6 files changed, 156 insertions(+), 3 deletions(-) create mode 100644 drivers/android/binder_latency_tracer.c diff --git a/drivers/android/Kconfig b/drivers/android/Kconfig index 6fdf2ab..39d47be 100644 --- a/drivers/android/Kconfig +++ b/drivers/android/Kconfig @@ -54,6 +54,14 @@ config ANDROID_BINDER_IPC_SELFTEST exhaustively with combinations of various buffer sizes and alignments. +config BINDER_TRANSACTION_LATENCY_TRACKING + bool "Android Binder transaction tracking" + help + Used for track abnormal binder transaction which is over 2 seconds, + when the transaction is done or be free, this transaction would be + checked whether it executed overtime. + If yes, printing out the detailed info. + endif # if ANDROID endmenu diff --git a/drivers/android/Makefile b/drivers/android/Makefile index c9d3d0c9..c2ffdb6 100644 --- a/drivers/android/Makefile +++ b/drivers/android/Makefile @@ -4,3 +4,4 @@ ccflags-y += -I$(src) # needed for trace events obj-$(CONFIG_ANDROID_BINDERFS) += binderfs.o obj-$(CONFIG_ANDROID_BINDER_IPC) += binder.o binder_alloc.o obj-$(CONFIG_ANDROID_BINDER_IPC_SELFTEST) += binder_alloc_selftest.o +obj-$(CONFIG_BINDER_TRANSACTION_LATENCY_TRACKING) += binder_latency_tracer.o diff --git a/drivers/android/binder.c b/drivers/android/binder.c index 5ec9af8..846d69b 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -2657,6 +2657,7 @@ static void binder_transaction(struct binder_proc *proc, return_error_line = __LINE__; goto err_alloc_t_failed; } + trace_binder_txn_latency_alloc(t, e); INIT_LIST_HEAD(&t->fd_fixups); binder_stats_created(BINDER_STAT_TRANSACTION); spin_lock_init(&t->lock); @@ -5144,6 +5145,7 @@ static void print_binder_transaction_ilocked(struct seq_file *m, to_proc ? to_proc->pid : 0, t->to_thread ? t->to_thread->pid : 0, t->code, t->flags, t->priority, t->need_reply); + trace_binder_txn_latency_info(m, t); spin_unlock(&t->lock); if (proc != to_proc) { diff --git a/drivers/android/binder_internal.h b/drivers/android/binder_internal.h index ed61b3e..bebc185 100644 --- a/drivers/android/binder_internal.h +++ b/drivers/android/binder_internal.h @@ -12,6 +12,11 @@ #include #include +#ifdef CONFIG_BINDER_TRANSACTION_LATENCY_TRACKING +#include +#include +#endif + struct binder_context { struct binder_node *binder_context_mgr_node; struct mutex context_mgr_node_lock; @@ -131,6 +136,10 @@ struct binder_transaction_log_entry { uint32_t return_error; uint32_t return_error_param; char context_name[BINDERFS_MAX_NAME + 1]; +#ifdef CONFIG_BINDER_TRANSACTION_LATENCY_TRACKING + struct timespec timestamp; + struct timeval tv; +#endif }; struct binder_transaction_log { @@ -520,6 +529,10 @@ struct binder_transaction { * during thread teardown */ spinlock_t lock; +#ifdef CONFIG_BINDER_TRANSACTION_LATENCY_TRACKING + struct timespec timestamp; + struct timeval tv; +#endif }; /** diff --git a/drivers/android/binder_latency_tracer.c b/drivers/android/binder_latency_tracer.c new file mode 100644 index 0000000..1b89634 --- /dev/null +++ b/drivers/android/binder_latency_tracer.c @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2019 MediaTek Inc. + */ + +#include +#include +#include "binder_alloc.h" +#include "binder_internal.h" +#include "binder_trace.h" + +/* + * probe_binder_txn_latency_free - Output info of a delay transaction + * @t: pointer to the over-time transaction + */ +void probe_binder_txn_latency_free(void *ignore, struct binder_transaction *t) +{ + struct rtc_time tm; + struct timespec *startime; + struct timespec cur, sub_t; + + ktime_get_ts(&cur); + startime = &t->timestamp; + sub_t = timespec_sub(cur, *startime); + + /* if transaction time is over than 2 sec, + * show timeout warning log. + */ + if (sub_t.tv_sec < 2) + return; + + rtc_time_to_tm(t->tv.tv_sec, &tm); + + spin_lock(&t->lock); + pr_info_ratelimited("%d: from %d:%d to %d:%d", + t->debug_id, + t->from ? t->from->proc->pid : 0, + t->from ? t->from->pid : 0, + t->to_proc ? t->to_proc->pid : 0, + t->to_thread ? t->to_thread->pid : 0); + + pr_info_ratelimited(" total %u.%03ld s code %u start %lu.%03ld android %d-%02d-%02d %02d:%02d:%02d.%03lu\n", + (unsigned int)sub_t.tv_sec, + (sub_t.tv_nsec / NSEC_PER_MSEC), + t->code, + (unsigned long)startime->tv_sec, + (startime->tv_nsec / NSEC_PER_MSEC), + (tm.tm_year + 1900), (tm.tm_mon + 1), tm.tm_mday, + tm.tm_hour, tm.tm_min, tm.tm_sec, + (unsigned long)(t->tv.tv_usec / USEC_PER_MSEC)); + spin_unlock(&t->lock); +} + +static void probe_binder_txn_latency_alloc(void *ignore, + struct binder_transaction *t, + struct binder_transaction_log_entry *e) +{ + ktime_get_ts(&e->timestamp); + do_gettimeofday(&e->tv); + e->tv.tv_sec -= (sys_tz.tz_minuteswest * 60); + spin_lock(&t->lock); + memcpy(&t->timestamp, &e->timestamp, sizeof(struct timespec)); + memcpy(&t->tv, &e->tv, sizeof(struct timeval)); + spin_unlock(&t->lock); +} + +static void probe_binder_txn_latency_info(void *ignore, struct seq_file *m, + struct binder_transaction *t) +{ + struct rtc_time tm; + + rtc_time_to_tm(t->tv.tv_sec, &tm); + seq_printf(m, + " start %lu.%06lu android %d-%02d-%02d %02d:%02d:%02d.%03lu", + (unsigned long)t->timestamp.tv_sec, + (t->timestamp.tv_nsec / NSEC_PER_USEC), + (tm.tm_year + 1900), (tm.tm_mon + 1), tm.tm_mday, + tm.tm_hour, tm.tm_min, tm.tm_sec, + (unsigned long)(t->tv.tv_usec / USEC_PER_MSEC)); +} + +static int __init init_binder_latency_tracer(void) +{ + register_trace_binder_txn_latency_free( + probe_binder_txn_latency_free, NULL); + register_trace_binder_txn_latency_alloc( + probe_binder_txn_latency_alloc, NULL); + register_trace_binder_txn_latency_info( + probe_binder_txn_latency_info, NULL); + + return 0; +} + +static void exit_binder_latency_tracer(void) +{ + unregister_trace_binder_txn_latency_free( + probe_binder_txn_latency_free, NULL); + unregister_trace_binder_txn_latency_alloc( + probe_binder_txn_latency_alloc, NULL); + unregister_trace_binder_txn_latency_info( + probe_binder_txn_latency_info, NULL); +} + +module_init(init_binder_latency_tracer); +module_exit(exit_binder_latency_tracer); + +MODULE_LICENSE("GPL v2"); diff --git a/drivers/android/binder_trace.h b/drivers/android/binder_trace.h index 8ac87d1..ca41c7d 100644 --- a/drivers/android/binder_trace.h +++ b/drivers/android/binder_trace.h @@ -18,6 +18,7 @@ struct binder_ref_data; struct binder_thread; struct binder_transaction; +struct binder_transaction_log_entry; TRACE_EVENT(binder_ioctl, TP_PROTO(unsigned int cmd, unsigned long arg), @@ -95,6 +96,18 @@ __entry->thread_todo) ); +DECLARE_TRACE(binder_txn_latency_alloc, + TP_PROTO(struct binder_transaction *t, + struct binder_transaction_log_entry *e), + TP_ARGS(t, e) +); + +DECLARE_TRACE(binder_txn_latency_info, + TP_PROTO(struct seq_file *m, + struct binder_transaction *t), + TP_ARGS(m, t) +); + TRACE_EVENT(binder_txn_latency_free, TP_PROTO(struct binder_transaction *t), TP_ARGS(t), @@ -106,6 +119,8 @@ __field(int, to_thread) __field(unsigned int, code) __field(unsigned int, flags) + __field(unsigned long, start_sec) + __field(unsigned long, start_nsec) ), TP_fast_assign( __entry->debug_id = t->debug_id; @@ -115,11 +130,18 @@ __entry->to_thread = t->to_thread ? t->to_thread->pid : 0; __entry->code = t->code; __entry->flags = t->flags; - ), - TP_printk("transaction=%d from %d:%d to %d:%d flags=0x%x code=0x%x", +#ifdef CONFIG_BINDER_TRANSACTION_LATENCY_TRACKING + __entry->start_sec = t->timestamp.tv_sec; + __entry->start_nsec = t->timestamp.tv_nsec / NSEC_PER_MSEC; +#else + __entry->start_sec = 0; + __entry->start_nsec = 0; +#endif + ), + TP_printk("transaction=%d from %d:%d to %d:%d flags=0x%x code=0x%x start %lu.%03ld", __entry->debug_id, __entry->from_proc, __entry->from_thread, __entry->to_proc, __entry->to_thread, __entry->code, - __entry->flags) + __entry->flags, __entry->start_sec, __entry->start_nsec) ); TRACE_EVENT(binder_transaction, -- 1.7.9.5 _______________________________________________ Linux-mediatek mailing list Linux-mediatek@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-mediatek