All of lore.kernel.org
 help / color / mirror / Atom feed
From: Andy Shevchenko <andy.shevchenko@gmail.com>
To: "Leizhen (ThunderTown)" <thunder.leizhen@huawei.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Nicolas Dichtel <nicolas.dichtel@6wind.com>,
	Joe Perches <joe@perches.com>, Jason Baron <jbaron@akamai.com>,
	Stefani Seibold <stefani@seibold.net>,
	Jacob Keller <jacob.e.keller@intel.com>,
	Thomas Graf <tgraf@suug.ch>,
	Herbert Xu <herbert@gondor.apana.org.au>,
	Jens Axboe <axboe@kernel.dk>, Petr Mladek <pmladek@suse.com>,
	Sergey Senozhatsky <senozhatsky@chromium.org>,
	Rasmus Villemoes <linux@rasmusvillemoes.dk>,
	linux-kernel <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 0/3] scripts/spelling.txt: add some spelling pairs and reorder
Date: Fri, 11 Jun 2021 13:08:08 +0300	[thread overview]
Message-ID: <YMM2CFHe2zxIkHcT@smile.fi.intel.com> (raw)
In-Reply-To: <908eb849-1925-8305-a1f3-aeb48f6f4c75@huawei.com>

On Fri, Jun 11, 2021 at 05:48:14PM +0800, Leizhen (ThunderTown) wrote:
> On 2021/6/11 16:10, Andy Shevchenko wrote:
> > On Fri, Jun 11, 2021 at 11:02 AM Andy Shevchenko
> > <andy.shevchenko@gmail.com> wrote:
> >> On Fri, Jun 11, 2021 at 10:19 AM Zhen Lei <thunder.leizhen@huawei.com> wrote:
> >>>
> >>> Add spelling_sanitizer.sh and use it to reorder, then add some spelling
> >>> "mistake||correction" pairs.
> >>
> >> The sorting idea is good, but the order is not.
> >> What you really need is to use language corpus [1] instead. So in such
> >> case you will eliminate false positives (to some extent).
> > 
> > Perhaps I need to elaborate what I meant. The (important) feature of
> > the corpus is sorting by frequency of the word usage. That's what
> 
> That's unlikely. Even if that were the case, no one could figure 'the frequency' out.

What do you mean? It's a natural sorting for the language corpus. First you get
the most used words down to less used.

> > would be the best. Unfortunately I don't know if codespell uses linear
> > search or hash based (i.o.w. does it convert the input file to the
> > Python list() or set() object?).
> 
> I think "spelling.txt" will be reprocessed by the tool.

Of course, and that's what I meant in the parentheses, i.e. to check what data
structure is used behind the scene.

> For the same set of data, if the performance differs significantly or even does not work
> due to the user's input order, the implementation of the tool itself must be problematic.
> 
> So our ranking of "spelling.txt" is how to make it easy to see.
> 
> I don't know if it's right. Pure logical reasoning.

Depends on the above it might increase the ratio of false positives.
Also it depends to the data base (spelling.txt).

What you need here is to make sure that:
 1) tool using arbitrary ordering already (by implying hash table, i.e. set()
    in Python);
 2) the current ordering is arbitrary and doesn't imply any side effects.

With this research conducted and explained, I'm fine with reordering.

> >> [1]: https://en.wikipedia.org/wiki/Corpus_of_Contemporary_American_English

-- 
With Best Regards,
Andy Shevchenko



  reply	other threads:[~2021-06-11 10:08 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-11  7:12 Zhen Lei
2021-06-11  7:12 ` [PATCH 1/3] scripts: add spelling_sanitizer.sh script Zhen Lei
2021-06-11  7:58   ` Andy Shevchenko
2021-06-11  9:30     ` Leizhen (ThunderTown)
2021-06-11  9:41       ` Andy Shevchenko
2021-06-11  9:58         ` Leizhen (ThunderTown)
2021-06-11 15:36   ` Joe Perches
2021-06-15  7:01     ` Leizhen (ThunderTown)
2021-06-16 11:58       ` Leizhen (ThunderTown)
2021-06-22  8:47         ` Leizhen (ThunderTown)
2021-06-11  7:12 ` [PATCH 2/3] scripts/spelling.txt: sort and remove duplicates Zhen Lei
2021-06-11  7:12 ` [PATCH 3/3] scripts/spelling.txt: add some spelling "mistake||correction" pairs Zhen Lei
2021-06-11  8:02 ` [PATCH 0/3] scripts/spelling.txt: add some spelling pairs and reorder Andy Shevchenko
2021-06-11  8:10   ` Andy Shevchenko
2021-06-11  9:48     ` Leizhen (ThunderTown)
2021-06-11 10:08       ` Andy Shevchenko [this message]
2021-06-11  8:12   ` Leizhen (ThunderTown)

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YMM2CFHe2zxIkHcT@smile.fi.intel.com \
    --to=andy.shevchenko@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=axboe@kernel.dk \
    --cc=herbert@gondor.apana.org.au \
    --cc=jacob.e.keller@intel.com \
    --cc=jbaron@akamai.com \
    --cc=joe@perches.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux@rasmusvillemoes.dk \
    --cc=nicolas.dichtel@6wind.com \
    --cc=pmladek@suse.com \
    --cc=senozhatsky@chromium.org \
    --cc=stefani@seibold.net \
    --cc=tgraf@suug.ch \
    --cc=thunder.leizhen@huawei.com \
    --subject='Re: [PATCH 0/3] scripts/spelling.txt: add some spelling pairs and reorder' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.