From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=3.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4961C433DF for ; Mon, 22 Jun 2020 19:39:54 +0000 (UTC) Received: from dpdk.org (dpdk.org [92.243.14.124]) by mail.kernel.org (Postfix) with ESMTP id 5851320767 for ; Mon, 22 Jun 2020 19:39:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OtSxZL0A" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5851320767 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8C50F1D8F2; Mon, 22 Jun 2020 21:39:51 +0200 (CEST) Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by dpdk.org (Postfix) with ESMTP id 36BD61D513 for ; Mon, 22 Jun 2020 21:39:49 +0200 (CEST) Received: by mail-lj1-f194.google.com with SMTP id s9so489036ljm.11 for ; Mon, 22 Jun 2020 12:39:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MZZJyCN8ABpCg0fIWM5ncTzehsfkUUFdq6IDnw2QUkQ=; b=OtSxZL0AWq7f9P1i/iMli4gI1Bu7qvuJGRYFXUxgiDqLm14CVnYIaofrljjdtD3MCj HI3xaoLwa+/EOntVqnmbyeqaL0DeFhANWETv6jK4oLr8mdXuQncQ8bM8nDphVribTEMx i+8pEZlGCPl/cZRnRGUEsGpgViEb4gI15Y9UYGigDUUZHTqTRhQQiZ/J2VV8HxwLJ8NG dHOP4SFeZNMlGZzjo/7XNWeUWI1jPavJIeFuGav4aiiH21+YUahyzmYtKCR8BhZ8iNOn BfMA140+SSLNLkhev3/yE1LjGNgrM0wZJymxAC+9ee8Hk39o5lmxWjWkL9hmESXs1th/ qv2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MZZJyCN8ABpCg0fIWM5ncTzehsfkUUFdq6IDnw2QUkQ=; b=rO5uJ7VGgwt1sCXCa0oXm6tEoaj0gzdJ7Kpq+AVZ6tR4KZ8QwplBROMVlMUbFhgH5a Py/eNxnTx/J+Saud4GWYX3yguER7Rb6mx+wparD95uAMtWKHMuZnbBeKgSCi+E3ovDyg nrtWzvH5fgeXZJPOm8ZcynZDbzKapPdFyqqC5/HVP1yDn8bjxttFfNp1QHqyK1BFIygR skbXOmUjiNJU2+H7J4tPcC+5NCMTS1Fyp4k97FdWbZHA2iHMCwgH+jN+VexyKn5ujXL3 Yxu1ozVhkA+LB6JtxuchTxWmQv0TL6rpTRTLTADUyzYWeb5dEhRcQB2TQOYwVeazlERd YNkQ== X-Gm-Message-State: AOAM533Ug6YemqtrLRjUaldD1fIyq/IJHx/V1JgwCQ/7X0Kr+OqzxtRF /J6L2E2+P0vMimElWtI5S7U= X-Google-Smtp-Source: ABdhPJwnDK3t31n+8MdnzkJ1IvBgn/grAc37hTwb4evLoKqIN8jVBbAx77pFIkVZvVcL7oRiQlu1tQ== X-Received: by 2002:a2e:a317:: with SMTP id l23mr9598343lje.175.1592854788476; Mon, 22 Jun 2020 12:39:48 -0700 (PDT) Received: from sovereign (broadband-37-110-65-23.ip.moscow.rt.ru. [37.110.65.23]) by smtp.gmail.com with ESMTPSA id t16sm2859298ljj.57.2020.06.22.12.39.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 Jun 2020 12:39:47 -0700 (PDT) Date: Mon, 22 Jun 2020 22:39:46 +0300 From: Dmitry Kozlyuk To: Neil Horman Cc: dev@dpdk.org, Bruce Richardson , Thomas Monjalon , robin.jarry@6wind.com Message-ID: <20200622223946.25977c13@sovereign> In-Reply-To: <20200622124117.GA216823@hmswarspite.think-freely.org> References: <20200622004503.29036-1-dmitry.kozliuk@gmail.com> <20200622124117.GA216823@hmswarspite.think-freely.org> X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [RFC PATCH 0/2] pmdinfogen: rewrite in Python X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, 22 Jun 2020 08:41:17 -0400, Neil Horman wrote: > On Mon, Jun 22, 2020 at 03:45:01AM +0300, Dmitry Kozlyuk wrote: [snip] > > 1. No standard ELF or COFF module for Python > > (amount of Python code without libelf on par with C code using it). > Thats not really true, pyelftools is pretty widely used, and packaged for > most(all) major distributions. Requiring this kicks the can for (2) above down > the road a bit, but I would prefer to see that used rather than a custom > implementation, just to reduce the maint. burden I must have underestimated pyelftools spread. I'll look into using it. The less new code the better, I agree here. Windows users can get it via PyPI. > > 2. struct rte_pci_id must be synchronized with header file > > (it's a few lines that never change). > > > I think you can just use ctypes and the native python C interface to just import > the rte_pci_id structure from the native dpdk header to avoid this, no? Not sure I follow. RFC script uses ctypes to describe struct rte_pci_id in Python. If you're suggesting to create a Python module in C just to include a header with a single small structure, I'd say it's not worth the trouble. > > 1. Support for >65K section headers seems present in current pmdinfogen. > > However, the data it reads is not used after. Is it really needed? > > > Its used. > The support you're referring to is primarily the ability to extract the true > number of section headers from the ELF file (in the event that its more than > 64k). Without that, on very large binaries, you may not be able to find the > symbol table in the ELF file. I was talking about these fields of struct elf_info: /* if Nth symbol table entry has .st_shndx = SHN_XINDEX, * take shndx from symtab_shndx_start[N] instead */ Elf32_Word *symtab_shndx_start; Elf32_Word *symtab_shndx_stop; They are not used after being filled in parse_elf() and their endianness fixed in the end despite the comment. > > 2. How much error-handling is required? This is a build-time tool, > > and Python gives nice stacktraces. However, segfaults are possible > > in Python version due to pointer handling. IMO, error checking > > must be just sufficient to prevent silent segfaults. > > > I'm not really sure what the question is here. You need to handle errors in the > parsing process, we can't just have the tool crashing during the build. How > thats handled is an implementation detail I would think. Consider a snippet from pmdinfogen.c: /* Check if file offset is correct */ if (hdr->e_shoff > info->size) { fprintf(stderr, "section header offset=%lu in file '%s' " "is bigger than filesize=%lu\n", (unsigned long)hdr->e_shoff, filename, info->size); return 0; } It is required in C, because otherwise pmdinfogen would crash without diagnostic. With Python, most errors like this result in a crash with a trace to the error line and a message from ctypes (or ELF parsing library). I'm arguing for leaving only the checks that prevent *silent* crashes (this can happen with ctypes) which are hard to diagnose. Motivation is to keep the code simple. > > 3. On Unix, pmdinfogen is called for each object file extracted with ar > > from an .a library by a shell script. On Windows, other tools > > have to be used, shell script will not work. On the other hand, COFF > > library format is quite simple. Would it be appropriate for > > pmdinfogen to handle it to avoid intermediate script? > > > I suppose you could, but extracting objects from archives seems outside the > scope of what pmdinfogen normally does. What is the problem with using a shell > script on windows? I thought WSL had support for executing bash syntax shell > scripts there? Why not key off an environment variable to make the relevant > scripts do the right thing on your environment? WSL2 is a Linux VM integrated in Windows, you can only cross-compile from there. Native builds can use Python or PowerShell, which is as capable as Unix shells, but incompatible with them. To call lib.exe (MSVC analog of ar) is probably simpler then parsing COFF, yes. So meson could select one of the following for a Windows target (objects are COFF): Host Toolchain Archive Script Extractor ------- --------- ------- --------- --------- Linux MinGW .a sh ar Windows MinGW .a PowerShell ar Windows Clang .lib PowerShell lib -- Dmitry Kozlyuk