From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43B8BC48BE0 for ; Fri, 11 Jun 2021 10:08:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1E218611CD for ; Fri, 11 Jun 2021 10:08:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231272AbhFKKKa (ORCPT ); Fri, 11 Jun 2021 06:10:30 -0400 Received: from mga07.intel.com ([134.134.136.100]:53626 "EHLO mga07.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229480AbhFKKK2 (ORCPT ); Fri, 11 Jun 2021 06:10:28 -0400 IronPort-SDR: H3dTZA6faqhc8Gde0/8oXJEYx2HZQipR5pGkrJ/Zht3YqE0PExP1uxyyHA6vGCL9NoDsLrg9m0 SnxnIEWCexxg== X-IronPort-AV: E=McAfee;i="6200,9189,10011"; a="269346684" X-IronPort-AV: E=Sophos;i="5.83,265,1616482800"; d="scan'208";a="269346684" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jun 2021 03:08:15 -0700 IronPort-SDR: MMv/k8ORRTiS9Mib26HqKMo2TnhmQUC40TC7Ph8aantSp3xu9n9O9OIFcltfPILziIJbncEBdd wve54RVKJJLA== X-IronPort-AV: E=Sophos;i="5.83,265,1616482800"; d="scan'208";a="402971775" Received: from smile.fi.intel.com (HELO smile) ([10.237.68.40]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Jun 2021 03:08:12 -0700 Received: from andy by smile with local (Exim 4.94) (envelope-from ) id 1lre5A-001Ttb-V4; Fri, 11 Jun 2021 13:08:08 +0300 Date: Fri, 11 Jun 2021 13:08:08 +0300 From: Andy Shevchenko To: "Leizhen (ThunderTown)" Cc: Andrew Morton , Nicolas Dichtel , Joe Perches , Jason Baron , Stefani Seibold , Jacob Keller , Thomas Graf , Herbert Xu , Jens Axboe , Petr Mladek , Sergey Senozhatsky , Rasmus Villemoes , linux-kernel Subject: Re: [PATCH 0/3] scripts/spelling.txt: add some spelling pairs and reorder Message-ID: References: <20210611071241.16728-1-thunder.leizhen@huawei.com> <908eb849-1925-8305-a1f3-aeb48f6f4c75@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <908eb849-1925-8305-a1f3-aeb48f6f4c75@huawei.com> Organization: Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 11, 2021 at 05:48:14PM +0800, Leizhen (ThunderTown) wrote: > On 2021/6/11 16:10, Andy Shevchenko wrote: > > On Fri, Jun 11, 2021 at 11:02 AM Andy Shevchenko > > wrote: > >> On Fri, Jun 11, 2021 at 10:19 AM Zhen Lei wrote: > >>> > >>> Add spelling_sanitizer.sh and use it to reorder, then add some spelling > >>> "mistake||correction" pairs. > >> > >> The sorting idea is good, but the order is not. > >> What you really need is to use language corpus [1] instead. So in such > >> case you will eliminate false positives (to some extent). > > > > Perhaps I need to elaborate what I meant. The (important) feature of > > the corpus is sorting by frequency of the word usage. That's what > > That's unlikely. Even if that were the case, no one could figure 'the frequency' out. What do you mean? It's a natural sorting for the language corpus. First you get the most used words down to less used. > > would be the best. Unfortunately I don't know if codespell uses linear > > search or hash based (i.o.w. does it convert the input file to the > > Python list() or set() object?). > > I think "spelling.txt" will be reprocessed by the tool. Of course, and that's what I meant in the parentheses, i.e. to check what data structure is used behind the scene. > For the same set of data, if the performance differs significantly or even does not work > due to the user's input order, the implementation of the tool itself must be problematic. > > So our ranking of "spelling.txt" is how to make it easy to see. > > I don't know if it's right. Pure logical reasoning. Depends on the above it might increase the ratio of false positives. Also it depends to the data base (spelling.txt). What you need here is to make sure that: 1) tool using arbitrary ordering already (by implying hash table, i.e. set() in Python); 2) the current ordering is arbitrary and doesn't imply any side effects. With this research conducted and explained, I'm fine with reordering. > >> [1]: https://en.wikipedia.org/wiki/Corpus_of_Contemporary_American_English -- With Best Regards, Andy Shevchenko