* [PATCH v7 1/4] xfs: Refactor xfs_isilocked()
2020-03-20 21:03 [PATCH v7 0/4] xfs: Remove wrappers for some semaphores Pavel Reichl
@ 2020-03-20 21:03 ` Pavel Reichl
2020-03-20 21:03 ` [PATCH v7 2/4] xfs: clean up whitespace in xfs_isilocked() calls Pavel Reichl
` (3 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Pavel Reichl @ 2020-03-20 21:03 UTC (permalink / raw)
To: linux-xfs
Refactor xfs_isilocked() to use newly introduced __xfs_rwsem_islocked().
__xfs_rwsem_islocked() is a helper function which encapsulates checking
state of rw_semaphores hold by inode.
Signed-off-by: Pavel Reichl <preichl@redhat.com>
Suggested-by: Dave Chinner <dchinner@redhat.com>
Suggested-by: Eric Sandeen <sandeen@redhat.com>
Suggested-by: Darrick J. Wong <darrick.wong@oracle.com>
---
Changes from V6:
* __xfs_rwsem_islocked() takes flags param
* changed logic of __xfs_rwsem_islocked() so that
holding a rwsem writte-locked implies read access as wel
fs/xfs/xfs_inode.c | 62 ++++++++++++++++++++++++++++++++++------------
fs/xfs/xfs_inode.h | 21 ++++++++++------
2 files changed, 60 insertions(+), 23 deletions(-)
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index c5077e6326c7..db356f815adc 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -345,32 +345,62 @@ xfs_ilock_demote(
}
#if defined(DEBUG) || defined(XFS_WARN)
-int
+static inline bool
+__xfs_rwsem_islocked(
+ struct rw_semaphore *rwsem,
+ int lock_flags)
+{
+ int arg;
+
+ if (!debug_locks)
+ return rwsem_is_locked(rwsem);
+
+ if (lock_flags & (1 << XFS_SHARED_LOCK_SHIFT)) {
+ /*
+ * The caller could be asking if we have (shared | excl)
+ * access to the lock. Ask lockdep if the rwsem is
+ * locked either for read or write access.
+ *
+ * The caller could also be asking if we have only
+ * shared access to the lock. Holding a rwsem
+ * write-locked implies read access as well, so the
+ * request to lockdep is the same for this case.
+ */
+ arg = -1;
+ } else {
+ /*
+ * The caller is asking if we have only exclusive access
+ * to the lock. Ask lockdep if the rwsem is locked for
+ * write access.
+ */
+ arg = 0;
+ }
+
+ return lockdep_is_held_type(rwsem, arg);
+}
+
+bool
xfs_isilocked(
- xfs_inode_t *ip,
+ struct xfs_inode *ip,
uint lock_flags)
{
- if (lock_flags & (XFS_ILOCK_EXCL|XFS_ILOCK_SHARED)) {
- if (!(lock_flags & XFS_ILOCK_SHARED))
- return !!ip->i_lock.mr_writer;
- return rwsem_is_locked(&ip->i_lock.mr_lock);
+ if (lock_flags & (XFS_ILOCK_EXCL | XFS_ILOCK_SHARED)) {
+ return __xfs_rwsem_islocked(&ip->i_lock,
+ (lock_flags >> XFS_ILOCK_FLAG_SHIFT));
}
- if (lock_flags & (XFS_MMAPLOCK_EXCL|XFS_MMAPLOCK_SHARED)) {
- if (!(lock_flags & XFS_MMAPLOCK_SHARED))
- return !!ip->i_mmaplock.mr_writer;
- return rwsem_is_locked(&ip->i_mmaplock.mr_lock);
+ if (lock_flags & (XFS_MMAPLOCK_EXCL | XFS_MMAPLOCK_SHARED)) {
+ return __xfs_rwsem_islocked(&ip->i_mmaplock,
+ (lock_flags >> XFS_MMAPLOCK_FLAG_SHIFT));
}
- if (lock_flags & (XFS_IOLOCK_EXCL|XFS_IOLOCK_SHARED)) {
- if (!(lock_flags & XFS_IOLOCK_SHARED))
- return !debug_locks ||
- lockdep_is_held_type(&VFS_I(ip)->i_rwsem, 0);
- return rwsem_is_locked(&VFS_I(ip)->i_rwsem);
+ if (lock_flags & (XFS_IOLOCK_EXCL | XFS_IOLOCK_SHARED)) {
+ return __xfs_rwsem_islocked(&VFS_I(ip)->i_rwsem,
+ (lock_flags >> XFS_IOLOCK_FLAG_SHIFT));
}
ASSERT(0);
- return 0;
+ return false;
}
#endif
diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
index 492e53992fa9..8df2c9e1a34e 100644
--- a/fs/xfs/xfs_inode.h
+++ b/fs/xfs/xfs_inode.h
@@ -272,12 +272,19 @@ static inline void xfs_ifunlock(struct xfs_inode *ip)
* Bit ranges: 1<<1 - 1<<16-1 -- iolock/ilock modes (bitfield)
* 1<<16 - 1<<32-1 -- lockdep annotation (integers)
*/
-#define XFS_IOLOCK_EXCL (1<<0)
-#define XFS_IOLOCK_SHARED (1<<1)
-#define XFS_ILOCK_EXCL (1<<2)
-#define XFS_ILOCK_SHARED (1<<3)
-#define XFS_MMAPLOCK_EXCL (1<<4)
-#define XFS_MMAPLOCK_SHARED (1<<5)
+
+#define XFS_IOLOCK_FLAG_SHIFT 0
+#define XFS_ILOCK_FLAG_SHIFT 2
+#define XFS_MMAPLOCK_FLAG_SHIFT 4
+
+#define XFS_SHARED_LOCK_SHIFT 1
+
+#define XFS_IOLOCK_EXCL (1 << (XFS_IOLOCK_FLAG_SHIFT))
+#define XFS_IOLOCK_SHARED (XFS_IOLOCK_EXCL << (XFS_SHARED_LOCK_SHIFT))
+#define XFS_ILOCK_EXCL (1 << (XFS_ILOCK_FLAG_SHIFT))
+#define XFS_ILOCK_SHARED (XFS_ILOCK_EXCL << (XFS_SHARED_LOCK_SHIFT))
+#define XFS_MMAPLOCK_EXCL (1 << (XFS_MMAPLOCK_FLAG_SHIFT))
+#define XFS_MMAPLOCK_SHARED (XFS_MMAPLOCK_EXCL << (XFS_SHARED_LOCK_SHIFT))
#define XFS_LOCK_MASK (XFS_IOLOCK_EXCL | XFS_IOLOCK_SHARED \
| XFS_ILOCK_EXCL | XFS_ILOCK_SHARED \
@@ -416,7 +423,7 @@ void xfs_ilock(xfs_inode_t *, uint);
int xfs_ilock_nowait(xfs_inode_t *, uint);
void xfs_iunlock(xfs_inode_t *, uint);
void xfs_ilock_demote(xfs_inode_t *, uint);
-int xfs_isilocked(xfs_inode_t *, uint);
+bool xfs_isilocked(xfs_inode_t *, uint);
uint xfs_ilock_data_map_shared(struct xfs_inode *);
uint xfs_ilock_attr_map_shared(struct xfs_inode *);
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 2/4] xfs: clean up whitespace in xfs_isilocked() calls
2020-03-20 21:03 [PATCH v7 0/4] xfs: Remove wrappers for some semaphores Pavel Reichl
2020-03-20 21:03 ` [PATCH v7 1/4] xfs: Refactor xfs_isilocked() Pavel Reichl
@ 2020-03-20 21:03 ` Pavel Reichl
2020-03-20 21:03 ` [PATCH v7 3/4] xfs: xfs_isilocked() can only check a single lock type Pavel Reichl
` (2 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: Pavel Reichl @ 2020-03-20 21:03 UTC (permalink / raw)
To: linux-xfs
Make whitespace follow the same pattern in all xfs_isilocked() calls.
Signed-off-by: Pavel Reichl <preichl@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
fs/xfs/libxfs/xfs_bmap.c | 2 +-
fs/xfs/xfs_file.c | 3 ++-
fs/xfs/xfs_inode.c | 6 +++---
fs/xfs/xfs_qm.c | 2 +-
4 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
index 9a6d7a84689a..bc2be29193aa 100644
--- a/fs/xfs/libxfs/xfs_bmap.c
+++ b/fs/xfs/libxfs/xfs_bmap.c
@@ -3906,7 +3906,7 @@ xfs_bmapi_read(
ASSERT(*nmap >= 1);
ASSERT(!(flags & ~(XFS_BMAPI_ATTRFORK|XFS_BMAPI_ENTIRE|
XFS_BMAPI_COWFORK)));
- ASSERT(xfs_isilocked(ip, XFS_ILOCK_SHARED|XFS_ILOCK_EXCL));
+ ASSERT(xfs_isilocked(ip, XFS_ILOCK_SHARED | XFS_ILOCK_EXCL));
if (XFS_IS_CORRUPT(mp, !xfs_ifork_has_extents(ip, whichfork)) ||
XFS_TEST_ERROR(false, mp, XFS_ERRTAG_BMAPIFORMAT)) {
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index b8a4a3f29b36..6b65572bdb21 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -770,7 +770,8 @@ xfs_break_layouts(
bool retry;
int error;
- ASSERT(xfs_isilocked(XFS_I(inode), XFS_IOLOCK_SHARED|XFS_IOLOCK_EXCL));
+ ASSERT(xfs_isilocked(XFS_I(inode),
+ XFS_IOLOCK_SHARED | XFS_IOLOCK_EXCL));
do {
retry = false;
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index db356f815adc..dec66059b045 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -2836,7 +2836,7 @@ static void
xfs_iunpin(
struct xfs_inode *ip)
{
- ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL|XFS_ILOCK_SHARED));
+ ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL | XFS_ILOCK_SHARED));
trace_xfs_inode_unpin_nowait(ip, _RET_IP_);
@@ -3700,7 +3700,7 @@ xfs_iflush(
XFS_STATS_INC(mp, xs_iflush_count);
- ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL|XFS_ILOCK_SHARED));
+ ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL | XFS_ILOCK_SHARED));
ASSERT(xfs_isiflocked(ip));
ASSERT(ip->i_d.di_format != XFS_DINODE_FMT_BTREE ||
ip->i_d.di_nextents > XFS_IFORK_MAXEXT(ip, XFS_DATA_FORK));
@@ -3832,7 +3832,7 @@ xfs_iflush_int(
struct xfs_dinode *dip;
struct xfs_mount *mp = ip->i_mount;
- ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL|XFS_ILOCK_SHARED));
+ ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL | XFS_ILOCK_SHARED));
ASSERT(xfs_isiflocked(ip));
ASSERT(ip->i_d.di_format != XFS_DINODE_FMT_BTREE ||
ip->i_d.di_nextents > XFS_IFORK_MAXEXT(ip, XFS_DATA_FORK));
diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
index 0b0909657bad..d2896925b5ca 100644
--- a/fs/xfs/xfs_qm.c
+++ b/fs/xfs/xfs_qm.c
@@ -1802,7 +1802,7 @@ xfs_qm_vop_chown_reserve(
int error;
- ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL|XFS_ILOCK_SHARED));
+ ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL | XFS_ILOCK_SHARED));
ASSERT(XFS_IS_QUOTA_RUNNING(mp));
delblks = ip->i_delayed_blks;
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 3/4] xfs: xfs_isilocked() can only check a single lock type
2020-03-20 21:03 [PATCH v7 0/4] xfs: Remove wrappers for some semaphores Pavel Reichl
2020-03-20 21:03 ` [PATCH v7 1/4] xfs: Refactor xfs_isilocked() Pavel Reichl
2020-03-20 21:03 ` [PATCH v7 2/4] xfs: clean up whitespace in xfs_isilocked() calls Pavel Reichl
@ 2020-03-20 21:03 ` Pavel Reichl
2020-03-20 21:03 ` [PATCH v7 4/4] xfs: replace mrlock_t with rw_semaphores Pavel Reichl
2020-03-23 3:28 ` [PATCH v7 0/4] xfs: Remove wrappers for some semaphores Darrick J. Wong
4 siblings, 0 replies; 10+ messages in thread
From: Pavel Reichl @ 2020-03-20 21:03 UTC (permalink / raw)
To: linux-xfs
In its current form, xfs_isilocked() is only able to test one lock type
at a time - ilock, iolock, or mmap lock, but combinations are not
properly handled. The intent here is to check that both XFS_IOLOCK_EXCL
and XFS_ILOCK_EXCL are held, so test them each separately.
The commit ecfea3f0c8c6 ("xfs: split xfs_bmap_shift_extents") ORed the
flags together which was an error, so this patch reverts that part of
the change and check the locks independently.
Fixes: ecfea3f0c8c6 ("xfs: split xfs_bmap_shift_extents")
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Suggested-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Pavel Reichl <preichl@redhat.com>
---
fs/xfs/libxfs/xfs_bmap.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
index bc2be29193aa..c9dc94f114ed 100644
--- a/fs/xfs/libxfs/xfs_bmap.c
+++ b/fs/xfs/libxfs/xfs_bmap.c
@@ -5829,7 +5829,8 @@ xfs_bmap_collapse_extents(
if (XFS_FORCED_SHUTDOWN(mp))
return -EIO;
- ASSERT(xfs_isilocked(ip, XFS_IOLOCK_EXCL | XFS_ILOCK_EXCL));
+ ASSERT(xfs_isilocked(ip, XFS_IOLOCK_EXCL));
+ ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL));
if (!(ifp->if_flags & XFS_IFEXTENTS)) {
error = xfs_iread_extents(tp, ip, whichfork);
@@ -5946,7 +5947,8 @@ xfs_bmap_insert_extents(
if (XFS_FORCED_SHUTDOWN(mp))
return -EIO;
- ASSERT(xfs_isilocked(ip, XFS_IOLOCK_EXCL | XFS_ILOCK_EXCL));
+ ASSERT(xfs_isilocked(ip, XFS_IOLOCK_EXCL));
+ ASSERT(xfs_isilocked(ip, XFS_ILOCK_EXCL));
if (!(ifp->if_flags & XFS_IFEXTENTS)) {
error = xfs_iread_extents(tp, ip, whichfork);
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 4/4] xfs: replace mrlock_t with rw_semaphores
2020-03-20 21:03 [PATCH v7 0/4] xfs: Remove wrappers for some semaphores Pavel Reichl
` (2 preceding siblings ...)
2020-03-20 21:03 ` [PATCH v7 3/4] xfs: xfs_isilocked() can only check a single lock type Pavel Reichl
@ 2020-03-20 21:03 ` Pavel Reichl
2020-03-23 3:28 ` [PATCH v7 0/4] xfs: Remove wrappers for some semaphores Darrick J. Wong
4 siblings, 0 replies; 10+ messages in thread
From: Pavel Reichl @ 2020-03-20 21:03 UTC (permalink / raw)
To: linux-xfs
Remove mrlock_t as it does not provide any extra value over
rw_semaphores. Make i_lock and i_mmaplock native rw_semaphores and
replace mr*() functions with native rwsem calls.
Signed-off-by: Pavel Reichl <preichl@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
---
fs/xfs/mrlock.h | 78 ----------------------------------------------
fs/xfs/xfs_inode.c | 36 +++++++++++----------
fs/xfs/xfs_inode.h | 4 +--
fs/xfs/xfs_iops.c | 4 +--
fs/xfs/xfs_linux.h | 2 +-
fs/xfs/xfs_super.c | 6 ++--
6 files changed, 27 insertions(+), 103 deletions(-)
delete mode 100644 fs/xfs/mrlock.h
diff --git a/fs/xfs/mrlock.h b/fs/xfs/mrlock.h
deleted file mode 100644
index 79155eec341b..000000000000
--- a/fs/xfs/mrlock.h
+++ /dev/null
@@ -1,78 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * Copyright (c) 2000-2006 Silicon Graphics, Inc.
- * All Rights Reserved.
- */
-#ifndef __XFS_SUPPORT_MRLOCK_H__
-#define __XFS_SUPPORT_MRLOCK_H__
-
-#include <linux/rwsem.h>
-
-typedef struct {
- struct rw_semaphore mr_lock;
-#if defined(DEBUG) || defined(XFS_WARN)
- int mr_writer;
-#endif
-} mrlock_t;
-
-#if defined(DEBUG) || defined(XFS_WARN)
-#define mrinit(mrp, name) \
- do { (mrp)->mr_writer = 0; init_rwsem(&(mrp)->mr_lock); } while (0)
-#else
-#define mrinit(mrp, name) \
- do { init_rwsem(&(mrp)->mr_lock); } while (0)
-#endif
-
-#define mrlock_init(mrp, t,n,s) mrinit(mrp, n)
-#define mrfree(mrp) do { } while (0)
-
-static inline void mraccess_nested(mrlock_t *mrp, int subclass)
-{
- down_read_nested(&mrp->mr_lock, subclass);
-}
-
-static inline void mrupdate_nested(mrlock_t *mrp, int subclass)
-{
- down_write_nested(&mrp->mr_lock, subclass);
-#if defined(DEBUG) || defined(XFS_WARN)
- mrp->mr_writer = 1;
-#endif
-}
-
-static inline int mrtryaccess(mrlock_t *mrp)
-{
- return down_read_trylock(&mrp->mr_lock);
-}
-
-static inline int mrtryupdate(mrlock_t *mrp)
-{
- if (!down_write_trylock(&mrp->mr_lock))
- return 0;
-#if defined(DEBUG) || defined(XFS_WARN)
- mrp->mr_writer = 1;
-#endif
- return 1;
-}
-
-static inline void mrunlock_excl(mrlock_t *mrp)
-{
-#if defined(DEBUG) || defined(XFS_WARN)
- mrp->mr_writer = 0;
-#endif
- up_write(&mrp->mr_lock);
-}
-
-static inline void mrunlock_shared(mrlock_t *mrp)
-{
- up_read(&mrp->mr_lock);
-}
-
-static inline void mrdemote(mrlock_t *mrp)
-{
-#if defined(DEBUG) || defined(XFS_WARN)
- mrp->mr_writer = 0;
-#endif
- downgrade_write(&mrp->mr_lock);
-}
-
-#endif /* __XFS_SUPPORT_MRLOCK_H__ */
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index dec66059b045..548f1b6e1a55 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -191,14 +191,15 @@ xfs_ilock(
}
if (lock_flags & XFS_MMAPLOCK_EXCL)
- mrupdate_nested(&ip->i_mmaplock, XFS_MMAPLOCK_DEP(lock_flags));
+ down_write_nested(&ip->i_mmaplock,
+ XFS_MMAPLOCK_DEP(lock_flags));
else if (lock_flags & XFS_MMAPLOCK_SHARED)
- mraccess_nested(&ip->i_mmaplock, XFS_MMAPLOCK_DEP(lock_flags));
+ down_read_nested(&ip->i_mmaplock, XFS_MMAPLOCK_DEP(lock_flags));
if (lock_flags & XFS_ILOCK_EXCL)
- mrupdate_nested(&ip->i_lock, XFS_ILOCK_DEP(lock_flags));
+ down_write_nested(&ip->i_lock, XFS_ILOCK_DEP(lock_flags));
else if (lock_flags & XFS_ILOCK_SHARED)
- mraccess_nested(&ip->i_lock, XFS_ILOCK_DEP(lock_flags));
+ down_read_nested(&ip->i_lock, XFS_ILOCK_DEP(lock_flags));
}
/*
@@ -242,27 +243,27 @@ xfs_ilock_nowait(
}
if (lock_flags & XFS_MMAPLOCK_EXCL) {
- if (!mrtryupdate(&ip->i_mmaplock))
+ if (!down_write_trylock(&ip->i_mmaplock))
goto out_undo_iolock;
} else if (lock_flags & XFS_MMAPLOCK_SHARED) {
- if (!mrtryaccess(&ip->i_mmaplock))
+ if (!down_read_trylock(&ip->i_mmaplock))
goto out_undo_iolock;
}
if (lock_flags & XFS_ILOCK_EXCL) {
- if (!mrtryupdate(&ip->i_lock))
+ if (!down_write_trylock(&ip->i_lock))
goto out_undo_mmaplock;
} else if (lock_flags & XFS_ILOCK_SHARED) {
- if (!mrtryaccess(&ip->i_lock))
+ if (!down_read_trylock(&ip->i_lock))
goto out_undo_mmaplock;
}
return 1;
out_undo_mmaplock:
if (lock_flags & XFS_MMAPLOCK_EXCL)
- mrunlock_excl(&ip->i_mmaplock);
+ up_write(&ip->i_mmaplock);
else if (lock_flags & XFS_MMAPLOCK_SHARED)
- mrunlock_shared(&ip->i_mmaplock);
+ up_read(&ip->i_mmaplock);
out_undo_iolock:
if (lock_flags & XFS_IOLOCK_EXCL)
up_write(&VFS_I(ip)->i_rwsem);
@@ -309,14 +310,14 @@ xfs_iunlock(
up_read(&VFS_I(ip)->i_rwsem);
if (lock_flags & XFS_MMAPLOCK_EXCL)
- mrunlock_excl(&ip->i_mmaplock);
+ up_write(&ip->i_mmaplock);
else if (lock_flags & XFS_MMAPLOCK_SHARED)
- mrunlock_shared(&ip->i_mmaplock);
+ up_read(&ip->i_mmaplock);
if (lock_flags & XFS_ILOCK_EXCL)
- mrunlock_excl(&ip->i_lock);
+ up_write(&ip->i_lock);
else if (lock_flags & XFS_ILOCK_SHARED)
- mrunlock_shared(&ip->i_lock);
+ up_read(&ip->i_lock);
trace_xfs_iunlock(ip, lock_flags, _RET_IP_);
}
@@ -335,9 +336,9 @@ xfs_ilock_demote(
~(XFS_IOLOCK_EXCL|XFS_MMAPLOCK_EXCL|XFS_ILOCK_EXCL)) == 0);
if (lock_flags & XFS_ILOCK_EXCL)
- mrdemote(&ip->i_lock);
+ downgrade_write(&ip->i_lock);
if (lock_flags & XFS_MMAPLOCK_EXCL)
- mrdemote(&ip->i_mmaplock);
+ downgrade_write(&ip->i_mmaplock);
if (lock_flags & XFS_IOLOCK_EXCL)
downgrade_write(&VFS_I(ip)->i_rwsem);
@@ -385,11 +386,14 @@ xfs_isilocked(
uint lock_flags)
{
if (lock_flags & (XFS_ILOCK_EXCL | XFS_ILOCK_SHARED)) {
+ ASSERT(!(lock_flags & ~(XFS_ILOCK_EXCL | XFS_ILOCK_SHARED)));
return __xfs_rwsem_islocked(&ip->i_lock,
(lock_flags >> XFS_ILOCK_FLAG_SHIFT));
}
if (lock_flags & (XFS_MMAPLOCK_EXCL | XFS_MMAPLOCK_SHARED)) {
+ ASSERT(!(lock_flags &
+ ~(XFS_MMAPLOCK_EXCL | XFS_MMAPLOCK_SHARED)));
return __xfs_rwsem_islocked(&ip->i_mmaplock,
(lock_flags >> XFS_MMAPLOCK_FLAG_SHIFT));
}
diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h
index 8df2c9e1a34e..b3664d818948 100644
--- a/fs/xfs/xfs_inode.h
+++ b/fs/xfs/xfs_inode.h
@@ -39,8 +39,8 @@ typedef struct xfs_inode {
/* Transaction and locking information. */
struct xfs_inode_log_item *i_itemp; /* logging information */
- mrlock_t i_lock; /* inode lock */
- mrlock_t i_mmaplock; /* inode mmap IO lock */
+ struct rw_semaphore i_lock; /* inode lock */
+ struct rw_semaphore i_mmaplock; /* inode mmap IO lock */
atomic_t i_pincount; /* inode pin count */
/*
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index 81f2f93caec0..7c3df574cf4c 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -1319,9 +1319,9 @@ xfs_setup_inode(
*/
lockdep_set_class(&inode->i_rwsem,
&inode->i_sb->s_type->i_mutex_dir_key);
- lockdep_set_class(&ip->i_lock.mr_lock, &xfs_dir_ilock_class);
+ lockdep_set_class(&ip->i_lock, &xfs_dir_ilock_class);
} else {
- lockdep_set_class(&ip->i_lock.mr_lock, &xfs_nondir_ilock_class);
+ lockdep_set_class(&ip->i_lock, &xfs_nondir_ilock_class);
}
/*
diff --git a/fs/xfs/xfs_linux.h b/fs/xfs/xfs_linux.h
index 8738bb03f253..8248744f1245 100644
--- a/fs/xfs/xfs_linux.h
+++ b/fs/xfs/xfs_linux.h
@@ -22,7 +22,6 @@ typedef __u32 xfs_nlink_t;
#include "xfs_types.h"
#include "kmem.h"
-#include "mrlock.h"
#include <linux/semaphore.h>
#include <linux/mm.h>
@@ -60,6 +59,7 @@ typedef __u32 xfs_nlink_t;
#include <linux/list_sort.h>
#include <linux/ratelimit.h>
#include <linux/rhashtable.h>
+#include <linux/rwsem.h>
#include <asm/page.h>
#include <asm/div64.h>
diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c
index 760901783944..1289ce1f4e9e 100644
--- a/fs/xfs/xfs_super.c
+++ b/fs/xfs/xfs_super.c
@@ -661,10 +661,8 @@ xfs_fs_inode_init_once(
atomic_set(&ip->i_pincount, 0);
spin_lock_init(&ip->i_flags_lock);
- mrlock_init(&ip->i_mmaplock, MRLOCK_ALLOW_EQUAL_PRI|MRLOCK_BARRIER,
- "xfsino", ip->i_ino);
- mrlock_init(&ip->i_lock, MRLOCK_ALLOW_EQUAL_PRI|MRLOCK_BARRIER,
- "xfsino", ip->i_ino);
+ init_rwsem(&ip->i_mmaplock);
+ init_rwsem(&ip->i_lock);
}
/*
--
2.25.1
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v7 0/4] xfs: Remove wrappers for some semaphores
2020-03-20 21:03 [PATCH v7 0/4] xfs: Remove wrappers for some semaphores Pavel Reichl
` (3 preceding siblings ...)
2020-03-20 21:03 ` [PATCH v7 4/4] xfs: replace mrlock_t with rw_semaphores Pavel Reichl
@ 2020-03-23 3:28 ` Darrick J. Wong
2020-03-23 9:22 ` Pavel Reichl
4 siblings, 1 reply; 10+ messages in thread
From: Darrick J. Wong @ 2020-03-23 3:28 UTC (permalink / raw)
To: Pavel Reichl; +Cc: linux-xfs
On Fri, Mar 20, 2020 at 10:03:13PM +0100, Pavel Reichl wrote:
> Remove some wrappers that we have in XFS around the read-write semaphore
> locks.
>
> The goal of this cleanup is to remove mrlock_t structure and its mr*()
> wrapper functions and replace it with native rw_semaphore type and its
> native calls.
Hmmm, there's something funny about this patchset that causes my fstests
vm to explode with isilocked assertions everywhere... I'll look more
tomorrow (it's still the weekend here) but figured I should tell you
sooner than later.
--D
> Pavel Reichl (4):
> xfs: Refactor xfs_isilocked()
> xfs: clean up whitespace in xfs_isilocked() calls
> xfs: xfs_isilocked() can only check a single lock type
> xfs: replace mrlock_t with rw_semaphores
>
> fs/xfs/libxfs/xfs_bmap.c | 8 +--
> fs/xfs/mrlock.h | 78 -----------------------------
> fs/xfs/xfs_file.c | 3 +-
> fs/xfs/xfs_inode.c | 104 ++++++++++++++++++++++++++-------------
> fs/xfs/xfs_inode.h | 25 ++++++----
> fs/xfs/xfs_iops.c | 4 +-
> fs/xfs/xfs_linux.h | 2 +-
> fs/xfs/xfs_qm.c | 2 +-
> fs/xfs/xfs_super.c | 6 +--
> 9 files changed, 98 insertions(+), 134 deletions(-)
> delete mode 100644 fs/xfs/mrlock.h
>
> --
> 2.25.1
>
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v7 0/4] xfs: Remove wrappers for some semaphores
2020-03-23 3:28 ` [PATCH v7 0/4] xfs: Remove wrappers for some semaphores Darrick J. Wong
@ 2020-03-23 9:22 ` Pavel Reichl
2020-03-23 16:33 ` Darrick J. Wong
0 siblings, 1 reply; 10+ messages in thread
From: Pavel Reichl @ 2020-03-23 9:22 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: linux-xfs
Oh, thanks for the heads up...I'll try to investigate.
On Mon, Mar 23, 2020 at 4:28 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> On Fri, Mar 20, 2020 at 10:03:13PM +0100, Pavel Reichl wrote:
> > Remove some wrappers that we have in XFS around the read-write semaphore
> > locks.
> >
> > The goal of this cleanup is to remove mrlock_t structure and its mr*()
> > wrapper functions and replace it with native rw_semaphore type and its
> > native calls.
>
> Hmmm, there's something funny about this patchset that causes my fstests
> vm to explode with isilocked assertions everywhere... I'll look more
> tomorrow (it's still the weekend here) but figured I should tell you
> sooner than later.
>
> --D
>
> > Pavel Reichl (4):
> > xfs: Refactor xfs_isilocked()
> > xfs: clean up whitespace in xfs_isilocked() calls
> > xfs: xfs_isilocked() can only check a single lock type
> > xfs: replace mrlock_t with rw_semaphores
> >
> > fs/xfs/libxfs/xfs_bmap.c | 8 +--
> > fs/xfs/mrlock.h | 78 -----------------------------
> > fs/xfs/xfs_file.c | 3 +-
> > fs/xfs/xfs_inode.c | 104 ++++++++++++++++++++++++++-------------
> > fs/xfs/xfs_inode.h | 25 ++++++----
> > fs/xfs/xfs_iops.c | 4 +-
> > fs/xfs/xfs_linux.h | 2 +-
> > fs/xfs/xfs_qm.c | 2 +-
> > fs/xfs/xfs_super.c | 6 +--
> > 9 files changed, 98 insertions(+), 134 deletions(-)
> > delete mode 100644 fs/xfs/mrlock.h
> >
> > --
> > 2.25.1
> >
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v7 0/4] xfs: Remove wrappers for some semaphores
2020-03-23 9:22 ` Pavel Reichl
@ 2020-03-23 16:33 ` Darrick J. Wong
2020-03-23 16:49 ` Christoph Hellwig
2020-03-23 21:59 ` Dave Chinner
0 siblings, 2 replies; 10+ messages in thread
From: Darrick J. Wong @ 2020-03-23 16:33 UTC (permalink / raw)
To: Pavel Reichl; +Cc: linux-xfs
On Mon, Mar 23, 2020 at 10:22:02AM +0100, Pavel Reichl wrote:
> Oh, thanks for the heads up...I'll try to investigate.
Ahah, I figured it out. It took me a while to pin down a solid reproducer,
but here's a stack trace that I see most often:
[ 812.102819] XFS: Assertion failed: xfs_isilocked(ip, XFS_ILOCK_EXCL), file: fs/xfs/libxfs/xfs_trans_inode.c, line: 91
[ 812.104017] ------------[ cut here ]------------
[ 812.104598] WARNING: CPU: 2 PID: 26250 at fs/xfs/xfs_message.c:112 assfail+0x30/0x50 [xfs]
[ 812.105505] Modules linked in: xfs libcrc32c ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip_set_hash_ip ip_set_hash_net xt_tcpudp xt_set ip_set_hash_mac ip_set nfnetlink ip6table_filter ip6_tables iptable_filter bfq sch_fq_codel ip_tables x_tables nfsv4 af_packet [last unloaded: xfs]
[ 812.108176] CPU: 2 PID: 26250 Comm: kworker/2:1 Tainted: G W 5.6.0-rc4-djw #rc4
[ 812.110742] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 812.111724] Workqueue: xfsalloc xfs_btree_split_worker [xfs]
[ 812.112404] RIP: 0010:assfail+0x30/0x50 [xfs]
[ 812.112905] Code: 41 89 c8 48 89 d1 48 89 f2 48 c7 c6 10 5d 37 a0 e8 c5 f8 ff ff 0f b6 1d fa 5b 11 00 80 fb 01 0f 87 aa 1b 06 00 83 e3 01 75 04 <0f> 0b 5b c3 0f 0b 48 c7 c7 80 a6 40 a0 e8 4d b8 0d e1 0f 1f 40 00
[ 812.114912] RSP: 0018:ffffc90000c7fc48 EFLAGS: 00010246
[ 812.115508] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 812.116286] RDX: 00000000ffffffc0 RSI: 000000000000000a RDI: ffffffffa0364b4d
[ 812.117087] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
[ 812.117875] R10: 000000000000000a R11: f000000000000000 R12: ffff888078695700
[ 812.118661] R13: 0000000000000000 R14: ffff8880787c4460 R15: 0000000000000000
[ 812.119445] FS: 0000000000000000(0000) GS:ffff88807e000000(0000) knlGS:0000000000000000
[ 812.120311] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 812.120969] CR2: 00007fe67192deb8 CR3: 00000000713a1002 CR4: 00000000001606a0
[ 812.121755] Call Trace:
[ 812.122097] xfs_trans_log_inode+0x128/0x140 [xfs]
[ 812.122685] xfs_bmbt_alloc_block+0x133/0x230 [xfs]
[ 812.123286] __xfs_btree_split+0x16d/0x980 [xfs]
[ 812.123876] xfs_btree_split_worker+0x61/0x90 [xfs]
[ 812.124446] process_one_work+0x250/0x5c0
[ 812.124910] ? worker_thread+0xcf/0x3a0
[ 812.125380] worker_thread+0x3d/0x3a0
[ 812.125807] ? process_one_work+0x5c0/0x5c0
[ 812.126297] kthread+0x121/0x140
[ 812.126684] ? kthread_park+0x80/0x80
[ 812.127111] ret_from_fork+0x3a/0x50
[ 812.127545] irq event stamp: 75298
[ 812.127947] hardirqs last enabled at (75297): [<ffffffff810d7f98>] console_unlock+0x428/0x580
[ 812.128895] hardirqs last disabled at (75298): [<ffffffff81001db0>] trace_hardirqs_off_thunk+0x1a/0x1c
[ 812.129942] softirqs last enabled at (74892): [<ffffffffa02c9f2b>] xfs_buf_find+0xa6b/0x1130 [xfs]
[ 812.130984] softirqs last disabled at (74890): [<ffffffffa02c98c0>] xfs_buf_find+0x400/0x1130 [xfs]
[ 812.131993] ---[ end trace 82fd2c9f1faba927 ]---
This one I could reproduce on my laptop by running generic/324 with a 1k
blocksize, but on my test vm fleet it was xfs/057 with 4k blocks. YMMV.
Anyway -- I augmented isilocked with some extra trace_printk to see what
was going on, and noticed this:
161011: <...>-28177 [000] 803.593121: xfs_ilock: dev 8:80 ino 0x46 flags ILOCK_EXCL caller xfs_alloc_file_space
161022: <...>-28177 [000] 803.593126: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
161038: <...>-28177 [000] 803.593132: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
161048: <...>-28177 [000] 803.593136: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
161064: <...>-28177 [000] 803.593172: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
161083: <...>-27081 [000] 803.593559: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 0
<assertion blows up>
162017: <...>-28177 [001] 803.641200: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
162036: <...>-28177 [001] 803.641215: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
162057: <...>-28177 [001] 803.641224: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
162073: <...>-28177 [001] 803.641243: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
162091: <...>-28177 [001] 803.641260: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
162106: <...>-28177 [001] 803.641288: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
162127: <...>-28177 [001] 803.641318: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
162140: <...>-28177 [001] 803.641397: bprint: xfs_isilocked: ino 0x46 lockf 0x4:0x1 debug_locks 1 arg 0 locked? 1
162158: <...>-28177 [001] 803.641416: xfs_iunlock: dev 8:80 ino 0x46 flags ILOCK_EXCL caller xfs_alloc_file_space
Process 28177 takes the ilock, and tries to do a bmapi_write which
involves a bmap btree split. The kernel libxfs delegates the split to a
workqueue (apparently to reduce stack usage?), which performs the bmbt
split and logs the inode core. The kworker is process 27081.
lockdep tracks the rwsem's lock state /and/ which process actually
holds the rwsem. This ownership doesn't transfer from 28177 to 27081,
so when the kworker asks lockdep if it holds ILOCK_EXCL, lockdep says
no, because 27081 doesn't own the lock, 28177 does. Kaboom.
The old mrlock_t had that 'int mr_writer' field which didn't care about
lock ownership and so isilocked would return true and so the assert was
happy.
So now comes the fun part -- the old isilocked code had a glaring hole
in which it would return true if *anyone* held the lock, even if the
owner is some other unrelated thread. That's probably good enough for
most of the fstests because we generally only run one thread at a time,
and developers will probably notice. :)
However, with your series applied, the isilocked function becomes much
more powerful when lockdep is active because now we can test that the
lock is held *by the caller*, which closes that hole.
Unfortunately, it also trips over this bmap split case, so if there's a
way to solve that problem we'd better find it quickly. Unfortunately, I
don't know of a way to gift a lock to another thread temporarily...
Thoughts?
--D
> On Mon, Mar 23, 2020 at 4:28 AM Darrick J. Wong <darrick.wong@oracle.com> wrote:
> >
> > On Fri, Mar 20, 2020 at 10:03:13PM +0100, Pavel Reichl wrote:
> > > Remove some wrappers that we have in XFS around the read-write semaphore
> > > locks.
> > >
> > > The goal of this cleanup is to remove mrlock_t structure and its mr*()
> > > wrapper functions and replace it with native rw_semaphore type and its
> > > native calls.
> >
> > Hmmm, there's something funny about this patchset that causes my fstests
> > vm to explode with isilocked assertions everywhere... I'll look more
> > tomorrow (it's still the weekend here) but figured I should tell you
> > sooner than later.
> >
> > --D
> >
> > > Pavel Reichl (4):
> > > xfs: Refactor xfs_isilocked()
> > > xfs: clean up whitespace in xfs_isilocked() calls
> > > xfs: xfs_isilocked() can only check a single lock type
> > > xfs: replace mrlock_t with rw_semaphores
> > >
> > > fs/xfs/libxfs/xfs_bmap.c | 8 +--
> > > fs/xfs/mrlock.h | 78 -----------------------------
> > > fs/xfs/xfs_file.c | 3 +-
> > > fs/xfs/xfs_inode.c | 104 ++++++++++++++++++++++++++-------------
> > > fs/xfs/xfs_inode.h | 25 ++++++----
> > > fs/xfs/xfs_iops.c | 4 +-
> > > fs/xfs/xfs_linux.h | 2 +-
> > > fs/xfs/xfs_qm.c | 2 +-
> > > fs/xfs/xfs_super.c | 6 +--
> > > 9 files changed, 98 insertions(+), 134 deletions(-)
> > > delete mode 100644 fs/xfs/mrlock.h
> > >
> > > --
> > > 2.25.1
> > >
> >
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v7 0/4] xfs: Remove wrappers for some semaphores
2020-03-23 16:33 ` Darrick J. Wong
@ 2020-03-23 16:49 ` Christoph Hellwig
2020-03-23 21:59 ` Dave Chinner
1 sibling, 0 replies; 10+ messages in thread
From: Christoph Hellwig @ 2020-03-23 16:49 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Pavel Reichl, linux-xfs
On Mon, Mar 23, 2020 at 09:33:42AM -0700, Darrick J. Wong wrote:
> lockdep tracks the rwsem's lock state /and/ which process actually
> holds the rwsem. This ownership doesn't transfer from 28177 to 27081,
> so when the kworker asks lockdep if it holds ILOCK_EXCL, lockdep says
> no, because 27081 doesn't own the lock, 28177 does. Kaboom.
> The old mrlock_t had that 'int mr_writer' field which didn't care about
> lock ownership and so isilocked would return true and so the assert was
> happy.
>
> So now comes the fun part -- the old isilocked code had a glaring hole
> in which it would return true if *anyone* held the lock, even if the
> owner is some other unrelated thread. That's probably good enough for
> most of the fstests because we generally only run one thread at a time,
> and developers will probably notice. :)
>
> However, with your series applied, the isilocked function becomes much
> more powerful when lockdep is active because now we can test that the
> lock is held *by the caller*, which closes that hole.
>
> Unfortunately, it also trips over this bmap split case, so if there's a
> way to solve that problem we'd better find it quickly. Unfortunately, I
> don't know of a way to gift a lock to another thread temporarily...
>
> Thoughts?
lock_release() in the donor, lock_acquire in the worker context, and
vice versa on return. We already do that dance indirectly with
xfs_setfilesize_trans_alloc and friends.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v7 0/4] xfs: Remove wrappers for some semaphores
2020-03-23 16:33 ` Darrick J. Wong
2020-03-23 16:49 ` Christoph Hellwig
@ 2020-03-23 21:59 ` Dave Chinner
1 sibling, 0 replies; 10+ messages in thread
From: Dave Chinner @ 2020-03-23 21:59 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Pavel Reichl, linux-xfs
On Mon, Mar 23, 2020 at 09:33:42AM -0700, Darrick J. Wong wrote:
> On Mon, Mar 23, 2020 at 10:22:02AM +0100, Pavel Reichl wrote:
> > Oh, thanks for the heads up...I'll try to investigate.
>
> Ahah, I figured it out. It took me a while to pin down a solid reproducer,
> but here's a stack trace that I see most often:
....
> lockdep tracks the rwsem's lock state /and/ which process actually
> holds the rwsem. This ownership doesn't transfer from 28177 to 27081,
> so when the kworker asks lockdep if it holds ILOCK_EXCL, lockdep says
> no, because 27081 doesn't own the lock, 28177 does. Kaboom.
Yeah, linux has this weird idea that semaphores can only be accessed
from a single process context, like a mutex. We've beaten our head
against this many times, and it's the only reason struct semaphore
still exists and everything isn't a mutex.
rwsems now do crazy stuff like track the owner for spinning
optimisations, despite the fact it's a semaphore and, by definition,
can be released by non-owner contexts....
> The old mrlock_t had that 'int mr_writer' field which didn't care about
> lock ownership and so isilocked would return true and so the assert was
> happy.
>
> So now comes the fun part -- the old isilocked code had a glaring hole
> in which it would return true if *anyone* held the lock, even if the
> owner is some other unrelated thread. That's probably good enough for
> most of the fstests because we generally only run one thread at a time,
> and developers will probably notice. :)
No, that's not a bug, that's how semaphores work - semaphores
protect the object, not the process context that took the lock.
i.e. the difference between a mutex and a semaphore is that the
mutex protects a code path from running concurrently with other
processes, while a semaphore protects an object from other accesses,
even when they don't have a process context associated with them
(e.g. while they are under IO).
> However, with your series applied, the isilocked function becomes much
> more powerful when lockdep is active because now we can test that the
> lock is held *by the caller*, which closes that hole.
Not really, it's just another "lockdep behaviour is wrong" issue we
have to work around. When we switch to a worker, we need to release
and acquire the lockdep context so that it thinks the current
process working on the object owns the lock.
> Unfortunately, it also trips over this bmap split case, so if there's a
> way to solve that problem we'd better find it quickly. Unfortunately, I
> don't know of a way to gift a lock to another thread temporarily...
rwsem_release() when the work is queued, rwsem_acquire() when the
work is run. And vice versa when the work is done.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 10+ messages in thread