* mkfs.xfs protofile and paths with spaces
@ 2023-01-23 21:13 Daan De Meyer
2023-01-25 1:53 ` Darrick J. Wong
0 siblings, 1 reply; 5+ messages in thread
From: Daan De Meyer @ 2023-01-23 21:13 UTC (permalink / raw)
To: linux-xfs
Hi,
We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
population of XFS filesystems. However, the man page does not specify
how to encode filenames with spaces in them. Spaces are used as the
token delimiter so I was wondering if there's some way to escape
filenames with spaces in them?
Cheers,
Daan De Meyer
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mkfs.xfs protofile and paths with spaces
2023-01-23 21:13 mkfs.xfs protofile and paths with spaces Daan De Meyer
@ 2023-01-25 1:53 ` Darrick J. Wong
2023-01-25 11:09 ` Daan De Meyer
0 siblings, 1 reply; 5+ messages in thread
From: Darrick J. Wong @ 2023-01-25 1:53 UTC (permalink / raw)
To: Daan De Meyer; +Cc: linux-xfs
On Mon, Jan 23, 2023 at 10:13:12PM +0100, Daan De Meyer wrote:
> Hi,
>
> We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
> population of XFS filesystems. However, the man page does not specify
> how to encode filenames with spaces in them. Spaces are used as the
> token delimiter so I was wondering if there's some way to escape
> filenames with spaces in them?
Spaces in filenames apparently weren't common when protofiles were
introduced in the Fourth Edition Unix in November 1973[1], so that
wasn't part of the specification for them:
"The prototype file contains tokens separated by spaces or new
lines."
The file format seems to have spread to other filesystems (minix, xenix,
afs, jfs, aix, etc.) without anybody adding support for spaces in
filenames.
One could make the argument that the protofile parsing code should
implicitly 's/\// /g' in the filename token since no Unix supports
slashes in directory entries, but that's not what people have been
doing for the past several decades.
At this point, 50 years later, it probably would make more sense to
clone the mke2fs -d functionality ("slurp up this directory tree") if
there's interest? Admittedly, at this point it's so old that we ought
to rev the entire format.
[1] https://dspinellis.github.io/unix-v4man/v4man.pdf (page 274)
or https://man.cat-v.org/unix-6th/8/mkfs
--D
> Cheers,
>
> Daan De Meyer
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mkfs.xfs protofile and paths with spaces
2023-01-25 1:53 ` Darrick J. Wong
@ 2023-01-25 11:09 ` Daan De Meyer
2023-02-02 0:56 ` Darrick J. Wong
0 siblings, 1 reply; 5+ messages in thread
From: Daan De Meyer @ 2023-01-25 11:09 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: linux-xfs
> ...
While a "-d" switch would be great to have, it'd also be great if we
could make the protofile format work with escaped spaces. That way we
can just add escaping for spaces in our tooling that calls mkfs.xfs
and we don't have to do ugly version checks on the mkfs binary version
to figure out which option to use.
On Wed, 25 Jan 2023 at 02:53, Darrick J. Wong <djwong@kernel.org> wrote:
>
> On Mon, Jan 23, 2023 at 10:13:12PM +0100, Daan De Meyer wrote:
> > Hi,
> >
> > We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
> > population of XFS filesystems. However, the man page does not specify
> > how to encode filenames with spaces in them. Spaces are used as the
> > token delimiter so I was wondering if there's some way to escape
> > filenames with spaces in them?
>
> Spaces in filenames apparently weren't common when protofiles were
> introduced in the Fourth Edition Unix in November 1973[1], so that
> wasn't part of the specification for them:
>
> "The prototype file contains tokens separated by spaces or new
> lines."
>
> The file format seems to have spread to other filesystems (minix, xenix,
> afs, jfs, aix, etc.) without anybody adding support for spaces in
> filenames.
>
> One could make the argument that the protofile parsing code should
> implicitly 's/\// /g' in the filename token since no Unix supports
> slashes in directory entries, but that's not what people have been
> doing for the past several decades.
>
> At this point, 50 years later, it probably would make more sense to
> clone the mke2fs -d functionality ("slurp up this directory tree") if
> there's interest? Admittedly, at this point it's so old that we ought
> to rev the entire format.
>
> [1] https://dspinellis.github.io/unix-v4man/v4man.pdf (page 274)
> or https://man.cat-v.org/unix-6th/8/mkfs
>
> --D
>
> > Cheers,
> >
> > Daan De Meyer
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mkfs.xfs protofile and paths with spaces
2023-01-25 11:09 ` Daan De Meyer
@ 2023-02-02 0:56 ` Darrick J. Wong
2023-02-02 8:46 ` Daan De Meyer
0 siblings, 1 reply; 5+ messages in thread
From: Darrick J. Wong @ 2023-02-02 0:56 UTC (permalink / raw)
To: Daan De Meyer; +Cc: linux-xfs
On Wed, Jan 25, 2023 at 12:09:28PM +0100, Daan De Meyer wrote:
> > ...
>
> While a "-d" switch would be great to have, it'd also be great if we
> could make the protofile format work with escaped spaces. That way we
> can just add escaping for spaces in our tooling that calls mkfs.xfs
> and we don't have to do ugly version checks on the mkfs binary version
> to figure out which option to use.
...which comes at a cost of making *us* figure out some gross hack to
retrofit that into the protofile format.
The easiest hack I can think of is to amend the protofile parser to
change any slash in the name to a space before creating the directory
entry. Slashes aren't allowed (and right now produce a corrupt
filesystem) so I guess that's the easy way out:
# cat /tmp/protofile
/
0 0
d--775 1000 1000
: Descending path /code/t/fstests
get/isk.sh ---775 1000 1000 /code/t/fstests/getdisk.sh
ext4.ftrace ---664 1000 1000 /code/t/fstests/ext4.ftrace
ocfs2.ftrace ---664 1000 1000 /code/t/fstests/ocfs2.ftrace
xfs.ftrace ---644 1000 1000 /code/t/fstests/xfs.ftrace
$
# mkfs.xfs -p /tmp/protofile /dev/sda -f
meta-data=/dev/sda isize=512 agcount=4, agsize=1298176
# xfs_db -c 'ls /' /dev/sda
/:
8 128 directory 0x0000002e 1 . (good)
10 128 directory 0x0000172e 2 .. (good)
12 131 regular 0xd8830694 10 get/isk.sh
(corrupt)
15 132 regular 0x3a30d3ae 11 ext4.ftrace
(good)
18 133 regular 0x3d313221 12 ocfs2.ftrace
(good)
21 134 regular 0x28becaee 10 xfs.ftrace
(good)
# xfs_repair -n /dev/sda
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
entry contains illegal character in shortform dir 128
would have junked entry "get/isk.sh" in directory inode 128
- agno = 1
- agno = 2
- agno = 3
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- check for inodes claiming duplicate blocks...
- agno = 0
entry contains illegal character in shortform dir 128
would have junked entry "get/isk.sh" in directory inode 128
- agno = 2
- agno = 1
- agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
- traversing filesystem ...
- traversal finished ...
- moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.
Would that (replace slash with space) help?
--D
> On Wed, 25 Jan 2023 at 02:53, Darrick J. Wong <djwong@kernel.org> wrote:
> >
> > On Mon, Jan 23, 2023 at 10:13:12PM +0100, Daan De Meyer wrote:
> > > Hi,
> > >
> > > We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
> > > population of XFS filesystems. However, the man page does not specify
> > > how to encode filenames with spaces in them. Spaces are used as the
> > > token delimiter so I was wondering if there's some way to escape
> > > filenames with spaces in them?
> >
> > Spaces in filenames apparently weren't common when protofiles were
> > introduced in the Fourth Edition Unix in November 1973[1], so that
> > wasn't part of the specification for them:
> >
> > "The prototype file contains tokens separated by spaces or new
> > lines."
> >
> > The file format seems to have spread to other filesystems (minix, xenix,
> > afs, jfs, aix, etc.) without anybody adding support for spaces in
> > filenames.
> >
> > One could make the argument that the protofile parsing code should
> > implicitly 's/\// /g' in the filename token since no Unix supports
> > slashes in directory entries, but that's not what people have been
> > doing for the past several decades.
> >
> > At this point, 50 years later, it probably would make more sense to
> > clone the mke2fs -d functionality ("slurp up this directory tree") if
> > there's interest? Admittedly, at this point it's so old that we ought
> > to rev the entire format.
> >
> > [1] https://dspinellis.github.io/unix-v4man/v4man.pdf (page 274)
> > or https://man.cat-v.org/unix-6th/8/mkfs
> >
> > --D
> >
> > > Cheers,
> > >
> > > Daan De Meyer
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mkfs.xfs protofile and paths with spaces
2023-02-02 0:56 ` Darrick J. Wong
@ 2023-02-02 8:46 ` Daan De Meyer
0 siblings, 0 replies; 5+ messages in thread
From: Daan De Meyer @ 2023-02-02 8:46 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: linux-xfs
> ...
> Would that (replace slash with space) help?
That sounds perfect. I can change our tooling to replace spaces with
slashes in the protofile. Theoretically you could also have newlines
and tabs and such in filenames but I don't think we need to support
that since it's pretty insane to do in the first place.
On Thu, 2 Feb 2023 at 01:56, Darrick J. Wong <djwong@kernel.org> wrote:
>
> On Wed, Jan 25, 2023 at 12:09:28PM +0100, Daan De Meyer wrote:
> > > ...
> >
> > While a "-d" switch would be great to have, it'd also be great if we
> > could make the protofile format work with escaped spaces. That way we
> > can just add escaping for spaces in our tooling that calls mkfs.xfs
> > and we don't have to do ugly version checks on the mkfs binary version
> > to figure out which option to use.
>
> ...which comes at a cost of making *us* figure out some gross hack to
> retrofit that into the protofile format.
>
> The easiest hack I can think of is to amend the protofile parser to
> change any slash in the name to a space before creating the directory
> entry. Slashes aren't allowed (and right now produce a corrupt
> filesystem) so I guess that's the easy way out:
>
> # cat /tmp/protofile
> /
> 0 0
> d--775 1000 1000
> : Descending path /code/t/fstests
> get/isk.sh ---775 1000 1000 /code/t/fstests/getdisk.sh
> ext4.ftrace ---664 1000 1000 /code/t/fstests/ext4.ftrace
> ocfs2.ftrace ---664 1000 1000 /code/t/fstests/ocfs2.ftrace
> xfs.ftrace ---644 1000 1000 /code/t/fstests/xfs.ftrace
> $
> # mkfs.xfs -p /tmp/protofile /dev/sda -f
> meta-data=/dev/sda isize=512 agcount=4, agsize=1298176
> # xfs_db -c 'ls /' /dev/sda
> /:
> 8 128 directory 0x0000002e 1 . (good)
> 10 128 directory 0x0000172e 2 .. (good)
> 12 131 regular 0xd8830694 10 get/isk.sh
> (corrupt)
> 15 132 regular 0x3a30d3ae 11 ext4.ftrace
> (good)
> 18 133 regular 0x3d313221 12 ocfs2.ftrace
> (good)
> 21 134 regular 0x28becaee 10 xfs.ftrace
> (good)
> # xfs_repair -n /dev/sda
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
> - zero log...
> - scan filesystem freespace and inode maps...
> - found root inode chunk
> Phase 3 - for each AG...
> - scan (but don't clear) agi unlinked lists...
> - process known inodes and perform inode discovery...
> - agno = 0
> entry contains illegal character in shortform dir 128
> would have junked entry "get/isk.sh" in directory inode 128
> - agno = 1
> - agno = 2
> - agno = 3
> - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
> - setting up duplicate extent list...
> - check for inodes claiming duplicate blocks...
> - agno = 0
> entry contains illegal character in shortform dir 128
> would have junked entry "get/isk.sh" in directory inode 128
> - agno = 2
> - agno = 1
> - agno = 3
> No modify flag set, skipping phase 5
> Phase 6 - check inode connectivity...
> - traversing filesystem ...
> - traversal finished ...
> - moving disconnected inodes to lost+found ...
> Phase 7 - verify link counts...
> No modify flag set, skipping filesystem flush and exiting.
>
> Would that (replace slash with space) help?
>
> --D
>
> > On Wed, 25 Jan 2023 at 02:53, Darrick J. Wong <djwong@kernel.org> wrote:
> > >
> > > On Mon, Jan 23, 2023 at 10:13:12PM +0100, Daan De Meyer wrote:
> > > > Hi,
> > > >
> > > > We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
> > > > population of XFS filesystems. However, the man page does not specify
> > > > how to encode filenames with spaces in them. Spaces are used as the
> > > > token delimiter so I was wondering if there's some way to escape
> > > > filenames with spaces in them?
> > >
> > > Spaces in filenames apparently weren't common when protofiles were
> > > introduced in the Fourth Edition Unix in November 1973[1], so that
> > > wasn't part of the specification for them:
> > >
> > > "The prototype file contains tokens separated by spaces or new
> > > lines."
> > >
> > > The file format seems to have spread to other filesystems (minix, xenix,
> > > afs, jfs, aix, etc.) without anybody adding support for spaces in
> > > filenames.
> > >
> > > One could make the argument that the protofile parsing code should
> > > implicitly 's/\// /g' in the filename token since no Unix supports
> > > slashes in directory entries, but that's not what people have been
> > > doing for the past several decades.
> > >
> > > At this point, 50 years later, it probably would make more sense to
> > > clone the mke2fs -d functionality ("slurp up this directory tree") if
> > > there's interest? Admittedly, at this point it's so old that we ought
> > > to rev the entire format.
> > >
> > > [1] https://dspinellis.github.io/unix-v4man/v4man.pdf (page 274)
> > > or https://man.cat-v.org/unix-6th/8/mkfs
> > >
> > > --D
> > >
> > > > Cheers,
> > > >
> > > > Daan De Meyer
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-02-02 8:47 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-23 21:13 mkfs.xfs protofile and paths with spaces Daan De Meyer
2023-01-25 1:53 ` Darrick J. Wong
2023-01-25 11:09 ` Daan De Meyer
2023-02-02 0:56 ` Darrick J. Wong
2023-02-02 8:46 ` Daan De Meyer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.