Linux-EROFS Archive on lore.kernel.org
 help / color / Atom feed
* Support for uncompressed sparse files.
@ 2019-11-17 10:10 Pratik Shinde
  2019-11-17 17:30 ` Gao Xiang via Linux-erofs
  0 siblings, 1 reply; 4+ messages in thread
From: Pratik Shinde @ 2019-11-17 10:10 UTC (permalink / raw)
  To: Gao Xiang, Gao Xiang; +Cc: linux-erofs

[-- Attachment #1: Type: text/plain, Size: 385 bytes --]

Hello Gao,

I have started working on above functionality for erofs.
First thing we need to do is detect sparse files & determine location of
holes in it.

I was thinking of using lseek() with SEEK_HOLE & SEEK_DATA for detecting
holes.
Let me know what you think about the approach OR any other better approach
in your mind.

PS : support for SEEK_HOLE & SEEK_DATA came in 3.4 kernel.

[-- Attachment #2: Type: text/html, Size: 547 bytes --]

<div dir="ltr"><div>Hello Gao,</div><div><br></div><div>I have started working on above functionality for erofs.</div><div>First thing we need to do is detect sparse files &amp; determine location of holes in it.</div><div><br></div><div>I was thinking of using lseek() with SEEK_HOLE &amp; SEEK_DATA for detecting holes.</div><div>Let me know what you think about the approach OR any other better approach in your mind.<br></div><div><br></div><div>PS : support for SEEK_HOLE &amp; SEEK_DATA came in 3.4 kernel. <br></div><div> <br></div></div>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Support for uncompressed sparse files.
  2019-11-17 10:10 Support for uncompressed sparse files Pratik Shinde
@ 2019-11-17 17:30 ` Gao Xiang via Linux-erofs
  2019-11-24 18:30   ` Pratik Shinde
  0 siblings, 1 reply; 4+ messages in thread
From: Gao Xiang via Linux-erofs @ 2019-11-17 17:30 UTC (permalink / raw)
  To: Pratik Shinde; +Cc: linux-erofs

Hi Pratik,

On Sun, Nov 17, 2019 at 03:40:43PM +0530, Pratik Shinde wrote:
> Hello Gao,
> 
> I have started working on above functionality for erofs.
> First thing we need to do is detect sparse files & determine location of
> holes in it.
> 
> I was thinking of using lseek() with SEEK_HOLE & SEEK_DATA for detecting
> holes.
> Let me know what you think about the approach OR any other better approach
> in your mind.
> 
> PS : support for SEEK_HOLE & SEEK_DATA came in 3.4 kernel.

That is a good start to detect sparse files by SEEK_HOLE & SEEK_DATA.

And as the first step, we need to design the on-disk extent format
for uncompressed sparse files. Is there some preliminary proposed
ideas for this as well? :-) (I'm not sure whether Chao is busy in
other stuffs now, we'd get in line with sparse on-disk format.)

Thanks,
Gao Xiang


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Support for uncompressed sparse files.
  2019-11-17 17:30 ` Gao Xiang via Linux-erofs
@ 2019-11-24 18:30   ` Pratik Shinde
  2019-11-24 19:10     ` Gao Xiang via Linux-erofs
  0 siblings, 1 reply; 4+ messages in thread
From: Pratik Shinde @ 2019-11-24 18:30 UTC (permalink / raw)
  To: Gao Xiang; +Cc: linux-erofs

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

Hi Gao,

In the current design, for uncompressed files, we only maintain the
starting block address.because rest of the data blocks will follow it
(continuous allocation).
For sparse files we have to do following
1) We don't want to allocate space for holes (Holes will be multiple of
EROFS_BLKSZ ?)
2) For read() operation on holes, return null data  = '\0'.

I have few thoughts about it:
1) Without changing the current design much, we want to keep track of holes
in file.
    e.g maintaining some table OR bitmap(per inode), to check if given
offset falls inside hole or real data.
2) Accordingly changing the readpage() aop.

Let me know you thoughts on this.

--Pratik.

On Sun, Nov 17, 2019 at 11:01 PM Gao Xiang <hsiangkao@aol.com> wrote:

> Hi Pratik,
>
> On Sun, Nov 17, 2019 at 03:40:43PM +0530, Pratik Shinde wrote:
> > Hello Gao,
> >
> > I have started working on above functionality for erofs.
> > First thing we need to do is detect sparse files & determine location of
> > holes in it.
> >
> > I was thinking of using lseek() with SEEK_HOLE & SEEK_DATA for detecting
> > holes.
> > Let me know what you think about the approach OR any other better
> approach
> > in your mind.
> >
> > PS : support for SEEK_HOLE & SEEK_DATA came in 3.4 kernel.
>
> That is a good start to detect sparse files by SEEK_HOLE & SEEK_DATA.
>
> And as the first step, we need to design the on-disk extent format
> for uncompressed sparse files. Is there some preliminary proposed
> ideas for this as well? :-) (I'm not sure whether Chao is busy in
> other stuffs now, we'd get in line with sparse on-disk format.)
>
> Thanks,
> Gao Xiang
>
>

[-- Attachment #2: Type: text/html, Size: 2241 bytes --]

<div dir="ltr"><div>Hi Gao,</div><div><br></div><div>In the current design, for uncompressed files, we only maintain the starting block address.because rest of the data blocks will follow it (continuous allocation).</div><div>For sparse files we have to do following</div><div>1) We don&#39;t want to allocate space for holes (Holes will be multiple of EROFS_BLKSZ ?)</div><div>2) For read() operation on holes, return null data  = &#39;\0&#39;.</div><div><br></div><div>I have few thoughts about it:</div><div>1) Without changing the current design much, we want to keep track of holes in file.</div><div>    e.g maintaining some table OR bitmap(per inode), to check if given offset falls inside hole or real data.</div><div>2) Accordingly changing the readpage() aop.</div><div><br></div><div>Let me know you thoughts on this.</div><div><br></div><div>--Pratik.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 17, 2019 at 11:01 PM Gao Xiang &lt;<a href="mailto:hsiangkao@aol.com">hsiangkao@aol.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Pratik,<br>
<br>
On Sun, Nov 17, 2019 at 03:40:43PM +0530, Pratik Shinde wrote:<br>
&gt; Hello Gao,<br>
&gt; <br>
&gt; I have started working on above functionality for erofs.<br>
&gt; First thing we need to do is detect sparse files &amp; determine location of<br>
&gt; holes in it.<br>
&gt; <br>
&gt; I was thinking of using lseek() with SEEK_HOLE &amp; SEEK_DATA for detecting<br>
&gt; holes.<br>
&gt; Let me know what you think about the approach OR any other better approach<br>
&gt; in your mind.<br>
&gt; <br>
&gt; PS : support for SEEK_HOLE &amp; SEEK_DATA came in 3.4 kernel.<br>
<br>
That is a good start to detect sparse files by SEEK_HOLE &amp; SEEK_DATA.<br>
<br>
And as the first step, we need to design the on-disk extent format<br>
for uncompressed sparse files. Is there some preliminary proposed<br>
ideas for this as well? :-) (I&#39;m not sure whether Chao is busy in<br>
other stuffs now, we&#39;d get in line with sparse on-disk format.)<br>
<br>
Thanks,<br>
Gao Xiang<br>
<br>
</blockquote></div>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Support for uncompressed sparse files.
  2019-11-24 18:30   ` Pratik Shinde
@ 2019-11-24 19:10     ` Gao Xiang via Linux-erofs
  0 siblings, 0 replies; 4+ messages in thread
From: Gao Xiang via Linux-erofs @ 2019-11-24 19:10 UTC (permalink / raw)
  To: Pratik Shinde; +Cc: linux-erofs

Hi Pratik,

On Mon, Nov 25, 2019 at 12:00:28AM +0530, Pratik Shinde wrote:
> Hi Gao,
> 
> In the current design, for uncompressed files, we only maintain the
> starting block address.because rest of the data blocks will follow it
> (continuous allocation).
> For sparse files we have to do following
> 1) We don't want to allocate space for holes (Holes will be multiple of
> EROFS_BLKSZ ?)
> 2) For read() operation on holes, return null data  = '\0'.
> 
> I have few thoughts about it:
> 1) Without changing the current design much, we want to keep track of holes
> in file.
>     e.g maintaining some table OR bitmap(per inode), to check if given
> offset falls inside hole or real data.
> 2) Accordingly changing the readpage() aop.
> 
> Let me know you thoughts on this.

I think it's roughly correct. Assume that holes aren't greatly fragmented,
I think it's useful to introduce extent table format rather than
some bitmap for sparse inodes.

Maybe we can start off with RFC PATCH of sparse files for mkfs, and thus
we can have common sense with on-disk format as well.

Thanks,
Gao Xiang

> 
> --Pratik.
> 
> On Sun, Nov 17, 2019 at 11:01 PM Gao Xiang <hsiangkao@aol.com> wrote:
> 
> > Hi Pratik,
> >
> > On Sun, Nov 17, 2019 at 03:40:43PM +0530, Pratik Shinde wrote:
> > > Hello Gao,
> > >
> > > I have started working on above functionality for erofs.
> > > First thing we need to do is detect sparse files & determine location of
> > > holes in it.
> > >
> > > I was thinking of using lseek() with SEEK_HOLE & SEEK_DATA for detecting
> > > holes.
> > > Let me know what you think about the approach OR any other better
> > approach
> > > in your mind.
> > >
> > > PS : support for SEEK_HOLE & SEEK_DATA came in 3.4 kernel.
> >
> > That is a good start to detect sparse files by SEEK_HOLE & SEEK_DATA.
> >
> > And as the first step, we need to design the on-disk extent format
> > for uncompressed sparse files. Is there some preliminary proposed
> > ideas for this as well? :-) (I'm not sure whether Chao is busy in
> > other stuffs now, we'd get in line with sparse on-disk format.)
> >
> > Thanks,
> > Gao Xiang
> >
> >

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-17 10:10 Support for uncompressed sparse files Pratik Shinde
2019-11-17 17:30 ` Gao Xiang via Linux-erofs
2019-11-24 18:30   ` Pratik Shinde
2019-11-24 19:10     ` Gao Xiang via Linux-erofs

Linux-EROFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-erofs/0 linux-erofs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-erofs linux-erofs/ https://lore.kernel.org/linux-erofs \
		linux-erofs@lists.ozlabs.org linux-erofs@ozlabs.org
	public-inbox-index linux-erofs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.ozlabs.lists.linux-erofs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git