linux-ext4.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Theodore Y. Ts'o" <tytso@mit.edu>
To: Saranya Muruganandam <saranyamohan@google.com>
Cc: linux-ext4@vger.kernel.org, adilger.kernel@dilger.ca
Subject: Re: [RFC PATCH v3 00/61] Introduce parallel fsck to e2fsck pass1
Date: Mon, 23 Nov 2020 16:25:16 -0500	[thread overview]
Message-ID: <20201123212516.GC132317@mit.edu> (raw)
In-Reply-To: <20201118153947.3394530-1-saranyamohan@google.com>

On Wed, Nov 18, 2020 at 07:38:46AM -0800, Saranya Muruganandam wrote:
> I've tested the whole patch series using 'make test' of e2fsck itself, and i
> manually set default threads to 4 which still pass almost of test suite,
> failure cases are below:
> 
> f_h_badroot f_multithread f_multithread_logfile f_multithread_no f_multithread_ok
> 
> h_h_badroot failed because out of order checking output, and others are because
> of extra multiple threads log output.

I just tried the full series, and I'm only seeing one test failure.
Unfortunately, it's f_multithread, and it's double free crash:

...
Pass 5: Checking group summary information
Multiple threads triggered to read bitmaps
double free or corruption (!prev)
Signal (6) SIGABRT si_code=SI_TKILL
../e2fsck/e2fsck(+0x45fab)[0x556589911fab]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7fe52ec34140]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7fe52ea96c41]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7fe52ea80537]
/lib/x86_64-linux-gnu/libc.so.6(+0x7e6c8)[0x7fe52ead96c8]
/lib/x86_64-linux-gnu/libc.so.6(+0x859ba)[0x7fe52eae09ba]
/lib/x86_64-linux-gnu/libc.so.6(+0x86ffc)[0x7fe52eae1ffc]
../lib/libext2fs.so.2(ext2fs_free_mem+0x23)[0x7fe52ed1091a]
../lib/libext2fs.so.2(+0x47329)[0x7fe52ed1f329]
../lib/libext2fs.so.2(+0x475b4)[0x7fe52ed1f5b4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7)[0x7fe52ec28ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fe52eb58d4f]
Exit status is 8

Here's the contents of f_multithread_ok.1.log after running
"./test_script --valgrind f_multithread_ok" in the tests directory:

Pass 1: Checking inodes, blocks, and sizes
[Thread 0] Scan group range [0, 1)
[Thread 1] Scan group range [1, 2)
[Thread 2] Scan group range [2, 3)
[Thread 3] Scan group range [3, 4)
[Thread 2] Scanned group range [2, 3), inodes 8192
[Thread 1] Scanned group range [1, 2), inodes 8192
[Thread 3] Scanned group range [3, 4), inodes 8192
[Thread 0] Scanned group range [0, 1), inodes 8192
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Multiple threads triggered to read bitmaps
==182288== Thread 2:
==182288== Conditional jump or move depends on uninitialised value(s)
==182288==    at 0x488E31B: read_bitmaps_range_start (rw_bitmaps.c:437)
==182288==    by 0x488E5B3: read_bitmaps_thread (rw_bitmaps.c:532)
==182288==    by 0x4965EA6: start_thread (pthread_create.c:477)
==182288==    by 0x4A7CD4E: clone (clone.S:95)
==182288== 
==182288== Conditional jump or move depends on uninitialised value(s)
==182288==    at 0x4839961: free (vg_replace_malloc.c:538)
==182288==    by 0x487F919: ext2fs_free_mem (ext2fs.h:1891)
==182288==    by 0x488E328: read_bitmaps_range_start (rw_bitmaps.c:438)
==182288==    by 0x488E5B3: read_bitmaps_thread (rw_bitmaps.c:532)
==182288==    by 0x4965EA6: start_thread (pthread_create.c:477)
==182288==    by 0x4A7CD4E: clone (clone.S:95)
==182288== 
==182288== Invalid free() / delete / delete[] / realloc()
==182288==    at 0x48399AB: free (vg_replace_malloc.c:538)
==182288==    by 0x487F919: ext2fs_free_mem (ext2fs.h:1891)
==182288==    by 0x488E328: read_bitmaps_range_start (rw_bitmaps.c:438)
==182288==    by 0x488E5B3: read_bitmaps_thread (rw_bitmaps.c:532)
==182288==    by 0x4965EA6: start_thread (pthread_create.c:477)
==182288==    by 0x4A7CD4E: clone (clone.S:95)
==182288==  Address 0x4b46100 is 0 bytes inside a block of size 3,144 free'd
==182288==    at 0x48399AB: free (vg_replace_malloc.c:538)
==182288==    by 0x487F919: ext2fs_free_mem (ext2fs.h:1891)
==182288==    by 0x488E328: read_bitmaps_range_start (rw_bitmaps.c:438)
==182288==    by 0x488E5B3: read_bitmaps_thread (rw_bitmaps.c:532)
==182288==    by 0x4965EA6: start_thread (pthread_create.c:477)
==182288==    by 0x4A7CD4E: clone (clone.S:95)
==182288==  Block was alloc'd at
==182288==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==182288==    by 0x487F7C9: ext2fs_get_mem (ext2fs.h:1847)
==182288==    by 0x11EE47: e2fsck_allocate_context (e2fsck.c:27)
==182288==    by 0x11B228: PRS (unix.c:829)
==182288==    by 0x11CD55: main (unix.c:1465)
...

						- Ted

      parent reply	other threads:[~2020-11-23 21:25 UTC|newest]

Thread overview: 84+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-18 15:38 [RFC PATCH v3 00/61] Introduce parallel fsck to e2fsck pass1 Saranya Muruganandam
2020-11-18 15:38 ` [RFC PATCH v3 01/61] e2fsck: add -m option for multithread Saranya Muruganandam
2020-11-23 19:53   ` harshad shirwadkar
2020-11-23 21:28   ` Theodore Y. Ts'o
2020-11-18 15:38 ` [RFC PATCH v3 02/61] e2fsck: copy context when using multi-thread fsck Saranya Muruganandam
2020-11-23 19:55   ` harshad shirwadkar
2020-11-23 21:38   ` Theodore Y. Ts'o
2020-12-17 23:56   ` Darrick J. Wong
2020-12-18  1:13     ` Wang Shilong
2020-12-18  1:27       ` Darrick J. Wong
2020-11-18 15:38 ` [RFC PATCH v3 03/61] e2fsck: copy fs " Saranya Muruganandam
2020-11-23 22:12   ` Theodore Y. Ts'o
2020-11-18 15:38 ` [RFC PATCH v3 04/61] e2fsck: clear icache " Saranya Muruganandam
2020-11-23 22:27   ` Theodore Y. Ts'o
2020-11-18 15:38 ` [RFC PATCH v3 05/61] e2fsck: add assert when copying context Saranya Muruganandam
2020-11-18 15:38 ` [RFC PATCH v3 06/61] e2fsck: copy bitmaps " Saranya Muruganandam
2020-11-18 15:38 ` [RFC PATCH v3 07/61] e2fsck: copy badblocks when copying fs Saranya Muruganandam
2020-11-18 15:38 ` [RFC PATCH v3 08/61] e2fsck: open io-channel " Saranya Muruganandam
2020-11-23 22:38   ` Theodore Y. Ts'o
2020-11-24 14:17     ` Theodore Y. Ts'o
2020-11-18 15:38 ` [RFC PATCH v3 09/61] e2fsck: create logs for mult-threads Saranya Muruganandam
2020-11-23 23:05   ` Theodore Y. Ts'o
2020-11-18 15:38 ` [RFC PATCH v3 10/61] e2fsck: optionally configure one pfsck thread Saranya Muruganandam
2020-11-23 23:16   ` Theodore Y. Ts'o
2020-11-18 15:38 ` [RFC PATCH v3 11/61] e2fsck: add start/end group for thread Saranya Muruganandam
2020-11-18 15:38 ` [RFC PATCH v3 12/61] e2fsck: split groups to different threads Saranya Muruganandam
2020-11-18 15:38 ` [RFC PATCH v3 13/61] e2fsck: print thread log properly Saranya Muruganandam
2020-11-23 23:40   ` Theodore Y. Ts'o
2020-11-18 15:39 ` [RFC PATCH v3 14/61] e2fsck: merge bitmaps after thread completes Saranya Muruganandam
2020-11-24  2:00   ` Theodore Y. Ts'o
2020-11-18 15:39 ` [RFC PATCH v3 15/61] e2fsck: do not change global variables Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 16/61] e2fsck: optimize the inserting of dir_info_db Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 17/61] e2fsck: merge dir_info after thread finishes Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 18/61] e2fsck: rbtree bitmap for dir Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 19/61] e2fsck: merge badblocks after thread finishes Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 20/61] e2fsck: merge icounts " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 21/61] e2fsck: merge dblist " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 22/61] e2fsck: add debug codes for multiple threads Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 23/61] e2fsck: merge counts after threads finish Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 24/61] e2fsck: merge fs flags when " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 25/61] e2fsck: merge dx_dir_info after " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 26/61] e2fsck: merge dirs_to_hash when " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 27/61] e2fsck: merge context flags properly Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 28/61] e2fsck: merge quota context after threads finish Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 29/61] e2fsck: serialize fix operations Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 30/61] e2fsck: move some fixes out of parallel pthreads Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 31/61] e2fsck: split and merge invalid bitmaps Saranya Muruganandam
2020-12-18  0:05   ` Darrick J. Wong
2020-12-18  1:19     ` Wang Shilong
2020-11-18 15:39 ` [RFC PATCH v3 32/61] e2fsck: merge EA blocks properly Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 33/61] e2fsck: kickoff mutex lock for block found map Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 34/61] e2fsck: allow admin specify number of threads Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 35/61] e2fsck: adjust " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 36/61] e2fsck: fix readahead for pfsck of pass1 Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 37/61] e2fsck: merge options after threads finish Saranya Muruganandam
2020-12-17 23:30   ` Darrick J. Wong
2020-11-18 15:39 ` [RFC PATCH v3 38/61] e2fsck: reset lost_and_found " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 39/61] e2fsck: merge extent depth count " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 40/61] e2fsck: simplify e2fsck context merging codes Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 41/61] e2fsck: set E2F_FLAG_ALLOC_OK after threads Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 42/61] e2fsck: wait fix thread finish before checking Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 43/61] e2fsck: cleanup e2fsck_pass1_thread_join() Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 44/61] e2fsck: avoid too much memory allocation for pfsck Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 45/61] e2fsck: make default smallest RA size to 1M Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 46/61] ext2fs: parallel bitmap loading Saranya Muruganandam
2020-11-24  2:44   ` Theodore Y. Ts'o
2020-11-18 15:39 ` [RFC PATCH v3 47/61] e2fsck: update mmp block in one thread Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 48/61] e2fsck: reset @inodes_to_rebuild if restart Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 49/61] e2fsck: fix build for make rpm Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 50/61] e2fsck: move ext2fs_get_avg_group to rw_bitmaps.c Saranya Muruganandam
2020-11-24  2:12   ` Theodore Y. Ts'o
2020-11-18 15:39 ` [RFC PATCH v3 51/61] configure: enable pfsck by default Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 53/61] e2fsck: fix f_multithread_ok test Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 54/61] e2fsck: fix race in ext2fs_read_bitmaps() Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 55/61] e2fsck: fix readahead for pass1 without pfsck Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 56/61] e2fsck: fix memory leaks with pfsck enabled Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 57/61] ext2fs: fix to set tail flags " Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 58/61] e2fsck: misc cleanups for pfsck Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 59/61] e2fsck: update mmp block race Saranya Muruganandam
2020-11-18 15:39 ` [RFC PATCH v3 60/61] e2fsck: propagate number of threads Saranya Muruganandam
2020-11-24  3:56   ` Theodore Y. Ts'o
2020-11-18 15:39 ` [RFC PATCH v3 61/61] e2fsck: Annotating fields in e2fsck_struct Saranya Muruganandam
2020-11-19 15:58 ` [RFC PATCH v3 00/61] Introduce parallel fsck to e2fsck pass1 Theodore Y. Ts'o
2020-11-23 21:25 ` Theodore Y. Ts'o [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201123212516.GC132317@mit.edu \
    --to=tytso@mit.edu \
    --cc=adilger.kernel@dilger.ca \
    --cc=linux-ext4@vger.kernel.org \
    --cc=saranyamohan@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).