From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.9 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 441BBC433EF for ; Fri, 3 Sep 2021 11:16:02 +0000 (UTC) Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by mail.kernel.org (Postfix) with ESMTP id 8B1BF6023B for ; Fri, 3 Sep 2021 11:16:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8B1BF6023B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ashroe.eu Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dpdk.org Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D09D740E64; Fri, 3 Sep 2021 13:16:00 +0200 (CEST) Received: from mail-108-mta6.mxroute.com (mail-108-mta6.mxroute.com [136.175.108.6]) by mails.dpdk.org (Postfix) with ESMTP id D213640E3C for ; Fri, 3 Sep 2021 13:15:57 +0200 (CEST) Received: from filter004.mxroute.com ([149.28.56.236] filter004.mxroute.com) (Authenticated sender: mN4UYu2MZsgR) by mail-108-mta6.mxroute.com (ZoneMTA) with ESMTPSA id 17bab5f4b6900074ba.001 for (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256); Fri, 03 Sep 2021 11:15:54 +0000 X-Zone-Loop: c4976bfa40ef9bdc4b72fb25062d79fcf73393e2ecb7 X-Originating-IP: [149.28.56.236] DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ashroe.eu; s=x; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=kL24GirSPWKHeRz6SsH0tiERedTIHPE6aFJC3w+b8dI=; b=rIByPHBsuMWPD7oSrhX9V4egKh TVvFEBdA4UQEBarH8eby7b0Uzb45wIEq+uuQpHNcWcO0xQJxl5TGALsqKCn37He/n1YjXZas3Vgk5 FpSJCiu8Z3HOTEjLYbMhkKk44LiYJ43+oOkpkUKYvticFGh4D0kf2N+0IKxvdb5bzKd+Dkgg8q/j8 iitBN6KZUVNhNnyBXMT6rUAteJmUAi4Xs9VbIGiK51vgWD2TwEXHTrF/6LLDTmlHRmyn92rHv5Mcn pokW/Nqa++Qjkl5CKiQTj3ZvIKjkmbUZOlhorZjhHVjZ1Hy6hZRC9B3FLiV2OnfQXf6cRWIXe6P+k w5oLBenQ==; To: Aaron Conole Cc: dev@dpdk.org, bruce.richardson@intel.com, stephen@networkplumber.org, ferruh.yigit@intel.com, thomas@monjalon.net, ktraynor@redhat.com References: <20210618163659.85933-1-mdr@ashroe.eu> <20210831145017.856776-1-mdr@ashroe.eu> <20210831145017.856776-3-mdr@ashroe.eu> From: "Kinsella, Ray" Message-ID: Date: Fri, 3 Sep 2021 12:15:50 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-AuthUser: mdr@ashroe.eu Subject: Re: [dpdk-dev] [PATCH v10 2/3] devtools: script to send notifications of expired symbols X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 01/09/2021 13:46, Aaron Conole wrote: > Ray Kinsella writes: > >> Use this script with the output of the DPDK symbol tool, to notify >> maintainers of expired symbols by email. You need to define the environment >> variable DPDK_GETMAINTAINER_PATH for this tool to work. >> >> Use terminal output to review the emails before sending. >> e.g. >> $ devtools/symbol-tool.py list-expired --format-output csv \ >> | DPDK_GETMAINTAINER_PATH=/get_maintainer.pl \ >> devtools/notify_expired_symbols.py --format-output terminal >> >> Then use email output to send the emails to the maintainers. >> e.g. >> $ devtools/symbol-tool.py list-expired --format-output csv \ >> | DPDK_GETMAINTAINER_PATH=/get_maintainer.pl \ >> devtools/notify_expired_symbols.py --format-output email \ >> --smtp-server --sender \ >> --password --cc >> >> Signed-off-by: Ray Kinsella >> --- >> devtools/notify-symbol-maintainers.py | 256 ++++++++++++++++++++++++++ >> 1 file changed, 256 insertions(+) >> create mode 100755 devtools/notify-symbol-maintainers.py >> >> diff --git a/devtools/notify-symbol-maintainers.py b/devtools/notify-symbol-maintainers.py >> new file mode 100755 >> index 0000000000..ee554687ff >> --- /dev/null >> +++ b/devtools/notify-symbol-maintainers.py >> @@ -0,0 +1,256 @@ >> +#!/usr/bin/env python3 >> +# SPDX-License-Identifier: BSD-3-Clause >> +# Copyright(c) 2021 Intel Corporation >> +# pylint: disable=invalid-name >> +'''Tool to notify maintainers of expired symbols''' >> +import smtplib >> +import ssl >> +import sys >> +import subprocess >> +import argparse >> +from argparse import RawTextHelpFormatter >> +import time >> +from email.message import EmailMessage >> + >> +DESCRIPTION = ''' >> +Use this script with the output of the DPDK symbol tool, to notify maintainers >> +and contributors of expired symbols by email. You need to define the environment >> +variable DPDK_GETMAINTAINER_PATH for this tool to work. >> + >> +Use terminal output to review the emails before sending. >> +e.g. >> +$ devtools/symbol-tool.py list-expired --format-output csv \\ >> +| DPDK_GETMAINTAINER_PATH=/get_maintainer.pl \\ >> +{s} --format-output terminal >> + >> +Then use email output to send the emails to the maintainers. >> +e.g. >> +$ devtools/symbol-tool.py list-expired --format-output csv \\ >> +| DPDK_GETMAINTAINER_PATH=/get_maintainer.pl \\ >> +{s} --format-output email \\ >> +--smtp-server --sender --password \\ >> +--cc >> +''' >> + >> +EMAIL_TEMPLATE = '''Hi there, >> + >> +Please note the symbols listed below have expired. In line with the DPDK ABI >> +policy, they should be scheduled for removal, in the next DPDK release. >> + >> +For more information, please see the DPDK ABI Policy, section 3.5.3. >> +https://doc.dpdk.org/guides/contributing/abi_policy.html >> + >> +Thanks, >> + >> +The DPDK Symbol Bot >> + >> +''' >> + >> +ABI_POLICY = 'doc/guides/contributing/abi_policy.rst' >> +MAINTAINERS = 'MAINTAINERS' >> +get_maintainer = ['devtools/get-maintainer.sh', \ >> + '--email', '-f'] > > Maybe it's best to make this something that can be overridden. There's > a series to change the .sh files to .py files. Perhaps an environment > variable or argument? ACK > >> +def _get_maintainers(libpath): >> + '''Get the maintainers for given library''' >> + try: >> + cmd = get_maintainer + [libpath] >> + result = subprocess.run(cmd, \ >> + stdout=subprocess.PIPE, \ >> + stderr=subprocess.PIPE, >> + check=True) >> + except subprocess.CalledProcessError: >> + return None > > You might consider handling > > except FileNotFoundError: > .... > > With a graceful exit and error message. In case the get_maintainers > path changes. ACK > >> + if result is None: >> + return None >> + >> + email = result.stdout.decode('utf-8') >> + if email == '': >> + return None >> + >> + email = list(filter(None,email.split('\n'))) >> + return email >> + >> +default_maintainers = _get_maintainers(ABI_POLICY) + \ >> + _get_maintainers(MAINTAINERS) >> + >> +def get_maintainers(libpath): >> + '''Get the maintainers for given library''' >> + maintainers=_get_maintainers(libpath) >> + >> + if maintainers is None: >> + maintainers = default_maintainers >> + >> + return maintainers >> + >> +def get_message(library, symbols, config): >> + '''Build email message from symbols, config and maintainers''' >> + contributors = {} >> + message = {} >> + maintainers = get_maintainers(library) >> + >> + if maintainers != default_maintainers: >> + message['CC'] = default_maintainers.copy() >> + >> + if 'CC' in config: >> + message.setdefault('CC',[]).append(config['CC']) >> + >> + message['Subject'] = 'Expired symbols in {}\n'.format(library) >> + >> + body = EMAIL_TEMPLATE >> + body += '{:<50}{:<25}{:<25}\n'.format('Symbol','Contributor','Email') >> + for sym in symbols: >> + body += ('{:<50}{:<25}{:<25}\n'.format(sym,\ >> + symbols[sym]['name'], >> + symbols[sym]['email'], >> + )) >> + email = symbols[sym]['email'] >> + contributors[email] = '' >> + >> + contributors = list(contributors.keys()) >> + >> + message['To'] = maintainers + contributors >> + message['Body'] = body >> + >> + return message >> + >> +class OutputEmail(): >> + '''Format the output for email''' >> + def __init__(self, config): >> + self.config = config >> + >> + self.terminal = OutputTerminal(config) >> + context = ssl.create_default_context() >> + >> + # Try to log in to server and send email >> + try: >> + self.server = smtplib.SMTP(config['smtp_server'], 587) >> + self.server.starttls(context=context) # Secure the connection >> + self.server.login(config['sender'], config['password']) >> + except Exception as exception: >> + print(exception) >> + raise exception >> + >> + def message(self,message): >> + '''send email''' >> + self.terminal.message(message) >> + >> + msg = EmailMessage() >> + msg.set_content(message.pop('Body')) >> + >> + for key in message.keys(): >> + msg[key] = message[key] >> + >> + msg['From'] = self.config['sender'] >> + msg['Reply-To'] = 'no-reply@dpdk.org' >> + >> + self.server.send_message(msg) >> + >> + time.sleep(1) > > Why this sleep is needed? Don't hammer the mail server :-) > >> + >> + def __del__(self): >> + self.server.quit() >> + >> +class OutputTerminal(): # pylint: disable=too-few-public-methods >> + '''Format the output for the terminal''' >> + def __init__(self, config): >> + self.config = config >> + >> + def message(self,message): >> + '''Print email to terminal''' >> + >> + terminal = 'To:' + ', '.join(message['To']) + '\n' >> + if 'sender' in self.config.keys(): >> + terminal += 'From:' + self.config['sender'] + '\n' >> + >> + terminal += 'Reply-To:' + 'no-reply@dpdk.org' + '\n' >> + >> + if 'CC' in message: >> + terminal += 'CC:' + ', '.join(message['CC']) + '\n' >> + >> + terminal += 'Subject:' + message['Subject'] + '\n' >> + terminal += 'Body:' + message['Body'] + '\n' >> + >> + print(terminal) >> + print('-' * 80) >> + >> +def parse_config(args): >> + '''put the command line args in the right places''' >> + config = {} >> + error_msg = None >> + >> + outputs = { >> + None : OutputTerminal, >> + 'terminal' : OutputTerminal, >> + 'email' : OutputEmail >> + } >> + >> + if args.format_output == 'email': >> + if args.smtp_server is None: >> + error_msg = 'SMTP server' >> + else: >> + config['smtp_server'] = args.smtp_server >> + >> + if args.sender is None: >> + error_msg = 'sender' >> + else: >> + config['sender'] = args.sender >> + >> + if args.password is None: >> + error_msg = 'password' >> + else: >> + config['password'] = args.password >> + >> + if args.cc is not None: >> + config['CC'] = args.cc >> + >> + if error_msg is not None: >> + print('Please specify a {} for email output'.format(error_msg)) >> + return None >> + >> + config['output'] = outputs[args.format_output] >> + return config >> + >> +def main(): >> + '''Main entry point''' >> + parser = argparse.ArgumentParser(description=DESCRIPTION.format(s=__file__), \ >> + formatter_class=RawTextHelpFormatter) >> + parser.add_argument('--format-output', choices=['terminal','email'], \ >> + default='terminal') >> + parser.add_argument('--smtp-server') >> + parser.add_argument('--password') >> + parser.add_argument('--sender') >> + parser.add_argument('--cc') >> + >> + args = parser.parse_args() >> + config = parse_config(args) >> + if config is None: >> + return >> + >> + symbols = {} >> + lastlib = library = '' >> + >> + output = config['output'](config) >> + >> + for line in sys.stdin: >> + line = line.rstrip('\n') >> + >> + if line.find('mapfile') >= 0: >> + continue >> + library, symbol, name, email = line.split(',') >> + >> + if library != lastlib: >> + message = get_message(lastlib, symbols, config) >> + output.message(message) >> + symbols = {} >> + >> + lastlib = library >> + symbols[symbol] = {'name' : name, 'email' : email} >> + >> + #print the last library >> + message = get_message(lastlib, symbols, config) >> + output.message(message) >> + >> +if __name__ == '__main__': >> + main() >