From: Bob McElrath <mcelrath+linux@draal.physics.wisc.edu>
To: Ingo Oeser <ingo.oeser@informatik.tu-chemnitz.de>
Cc: linux-kernel@vger.kernel.org
Subject: Re: Editing-in-place of a large file
Date: Sun, 2 Sep 2001 23:27:30 -0500 [thread overview]
Message-ID: <20010902232730.S23180@draal.physics.wisc.edu> (raw)
In-Reply-To: <20010902152137.L23180@draal.physics.wisc.edu> <20010902233008.Q9870@nightmaster.csn.tu-chemnitz.de> <20010902175938.D21576@work.bitmover.com> <20010903032439.A802@nightmaster.csn.tu-chemnitz.de>
In-Reply-To: <20010903032439.A802@nightmaster.csn.tu-chemnitz.de>; from ingo.oeser@informatik.tu-chemnitz.de on Mon, Sep 03, 2001 at 03:24:39AM +0200
[-- Attachment #1: Type: text/plain, Size: 3316 bytes --]
Ingo Oeser [ingo.oeser@informatik.tu-chemnitz.de] wrote:
> On Sun, Sep 02, 2001 at 05:59:38PM -0700, Larry McVoy wrote:
> > > What's needed is a generalisation of sparse files and truncate().
> > > They both handle similar problems.
> >
> > how about
> >
> > fzero(int fd, off_t off, size_t len)
> > fdelete(int fd, off_t off, size_t len)
>
> and
>
> finsert(int fd, off_t off, size_t len, void *buf, size_t buflen)
>
> > The main problem with this is if the off/len are not block aligned. If they
> > are, then this is just block twiddling, if they aren't, then this is a file
> > rewrite anyway.
*exactly* I don't know enough about ext2fs to know if this is possible
(i.e. a partially filled block in the middle of a file) so that's why I
asked.
> Another solution for the original problem is to rewrite the file
> in-place by coping from the end of the gap to the beginning of
> the gap until the gap is shifted to the end of the file and thus
> can be left to ftruncate().
For editing commercials, you'd still have to copy 90% of the data. In
the US, there's roughly 5 minutes of commercials for every 15 of the
show, so that would only save copying the first 15 minutes...
> This will at least not require more space on disk, but will take
> quite a while and risk data corruption for this file in case of
> abortion.
Yep. I should mention that the Linux/mjpeg tools
(http://mjpeg.sourceforge.net) already have an elegant way of "marking"
a portion of a video and skipping it when playing it, through the use of
"edit lists". (use xlav/glav to mark it, and then you can lavplay the
edit list, which just contains the start/end of skipped sections) They
also have a program to apply the edit list and create a new video
(lavtrans). But this requires copying the desired sections of video to
a new file, which requires 75% more disk space than the original file,
and takes a looong time.
The idea behind my first message should be obvious here...an almost
atomic operation modifying at most 2 blocks (and marking a bunch as
free) wouldn't require nearly as much disk-thrashing, and would be
nearly instantaneous from the user's perspective.
Disk fragmentation is unimportant when the contiguous chunks are 300MB
long.
> But fzero, fdelete and finsert might be worth considering, since
> some file systems, which pack tails could also pack these kind of
> partial used blocks and handle them properly.
Do the journaling filesystems use blocks in a similar manner to ext2fs?
Anyone know if any of them can handle partially filled blocks in the
middle of a file?
Are there any media-filesystems out there that have these kinds of
extensions? I'm not sure these extensions would be useful for anything
but editing media...
> We already handle partial pages, so why not handle them with
> offset/size pairs and enable this mechanisms? Multi media streams
> would love these kind of APIs ;-)
Yep yep yep. What do multimedia people use? Custom multi-thousand
dollar programs with their own filesystem layer? What about TiVo?
Didn't they contribute some fs-layer modifications a while back?
Cheers,
-- Bob
Bob McElrath (rsmcelrath@students.wisc.edu)
Univ. of Wisconsin at Madison, Department of Physics
[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]
next prev parent reply other threads:[~2001-09-03 4:28 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2001-09-02 20:21 Editing-in-place of a large file Bob McElrath
2001-09-02 21:28 ` COW fs (Re: Editing-in-place of a large file) VDA
2001-09-09 14:46 ` John Ripley
2001-09-09 16:30 ` John Ripley
2001-09-10 2:43 ` Daniel Phillips
2001-09-10 2:58 ` David Lang
2001-09-09 17:41 ` Xavier Bestel
2001-09-10 1:29 ` John Ripley
2001-09-10 6:45 ` Ragnar Kjørstad
2001-09-14 10:06 ` Pavel Machek
2001-09-10 11:11 ` Ihar Filipau
2001-09-10 16:10 ` Kari Hurtta
2001-09-14 10:03 ` Pavel Machek
2001-09-10 9:28 ` VDA
2001-09-10 9:35 ` John P. Looney
2001-09-02 21:30 ` Editing-in-place of a large file Ingo Oeser
2001-09-03 0:59 ` Larry McVoy
2001-09-03 1:24 ` Ingo Oeser
2001-09-03 1:31 ` Alan Cox
2001-09-03 1:50 ` Ingo Oeser
2001-09-03 10:48 ` Alan Cox
2001-09-03 14:31 ` Daniel Phillips
2001-09-03 14:46 ` Bob McElrath
2001-09-03 14:54 ` Alan Cox
2001-09-03 15:42 ` Doug McNaught
2001-09-03 15:11 ` Richard Guenther
2001-09-03 21:19 ` Ben Ford
2001-09-03 4:27 ` Bob McElrath [this message]
2001-09-03 1:30 ` Daniel Phillips
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20010902232730.S23180@draal.physics.wisc.edu \
--to=mcelrath+linux@draal.physics.wisc.edu \
--cc=ingo.oeser@informatik.tu-chemnitz.de \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).