* Actual usage of files in ubifs
@ 2017-09-11 9:47 Ricard Wanderlof
2017-09-11 14:55 ` Richard Weinberger
0 siblings, 1 reply; 10+ messages in thread
From: Ricard Wanderlof @ 2017-09-11 9:47 UTC (permalink / raw)
To: Linux mtd; +Cc: Cristian Ionescu-Idbohrn, Jonas Holmberg
Is there a (simple) way to find out exactly how much space each specific
file uses in an ubifs fils system?
I looked into running mkfs.ubifs with some form of option to give me the
info, the closest I got was -g 3 which output a lot of sizes but it was
not obvious exactly what they meant.
I suppose it would be fully possible to add code to mkfs.ubifs to output
this information, as it surely must be there at some point, since it
selects the compression algorithm on a file per file basis, debending on
the amount of compression acheived for each algorithm.
/Ricard
--
Ricard Wolf Wanderlöf ricardw(at)axis.com
Axis Communications AB, Lund, Sweden www.axis.com
Phone +46 46 272 2016 Fax +46 46 13 61 30
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-11 9:47 Actual usage of files in ubifs Ricard Wanderlof
@ 2017-09-11 14:55 ` Richard Weinberger
2017-09-11 15:18 ` Ricard Wanderlof
0 siblings, 1 reply; 10+ messages in thread
From: Richard Weinberger @ 2017-09-11 14:55 UTC (permalink / raw)
To: Ricard Wanderlof; +Cc: Linux mtd, Jonas Holmberg, Cristian Ionescu-Idbohrn
On Mon, Sep 11, 2017 at 11:47 AM, Ricard Wanderlof
<ricard.wanderlof@axis.com> wrote:
>
> Is there a (simple) way to find out exactly how much space each specific
> file uses in an ubifs fils system?
>
> I looked into running mkfs.ubifs with some form of option to give me the
> info, the closest I got was -g 3 which output a lot of sizes but it was
> not obvious exactly what they meant.
>
> I suppose it would be fully possible to add code to mkfs.ubifs to output
> this information, as it surely must be there at some point, since it
> selects the compression algorithm on a file per file basis, debending on
> the amount of compression acheived for each algorithm.
Well, how exactly should it be?
I guess you want more than the compressed size on flash.
Do you also want to know how much bytes a file occupies in the index
tree and the inode itself?
This is non-trivial.
--
Thanks,
//richard
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-11 14:55 ` Richard Weinberger
@ 2017-09-11 15:18 ` Ricard Wanderlof
2017-09-11 15:44 ` Artem Bityutskiy
0 siblings, 1 reply; 10+ messages in thread
From: Ricard Wanderlof @ 2017-09-11 15:18 UTC (permalink / raw)
To: Richard Weinberger; +Cc: Linux mtd, Jonas Holmberg, Cristian Ionescu-Idbohrn
On Mon, 11 Sep 2017, Richard Weinberger wrote:
> On Mon, Sep 11, 2017 at 11:47 AM, Ricard Wanderlof
> <ricard.wanderlof@axis.com> wrote:
> >
> > Is there a (simple) way to find out exactly how much space each specific
> > file uses in an ubifs fils system?
> >
> > I looked into running mkfs.ubifs with some form of option to give me the
> > info, the closest I got was -g 3 which output a lot of sizes but it was
> > not obvious exactly what they meant.
> >
> > I suppose it would be fully possible to add code to mkfs.ubifs to output
> > this information, as it surely must be there at some point, since it
> > selects the compression algorithm on a file per file basis, debending on
> > the amount of compression acheived for each algorithm.
>
> Well, how exactly should it be?
> I guess you want more than the compressed size on flash.
> Do you also want to know how much bytes a file occupies in the index
> tree and the inode itself?
> This is non-trivial.
Good point. I was solely thinking along the lines of how much space the
actual file occupied, not considering metadata. That would be a good
starting point. I'm guessing that for moderate file sizes the metadata
would be relatively small compared to the file itself?
/Ricard
--
Ricard Wolf Wanderlöf ricardw(at)axis.com
Axis Communications AB, Lund, Sweden www.axis.com
Phone +46 46 272 2016 Fax +46 46 13 61 30
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-11 15:18 ` Ricard Wanderlof
@ 2017-09-11 15:44 ` Artem Bityutskiy
2017-09-11 19:57 ` Richard Weinberger
0 siblings, 1 reply; 10+ messages in thread
From: Artem Bityutskiy @ 2017-09-11 15:44 UTC (permalink / raw)
To: Ricard Wanderlof, Richard Weinberger
Cc: Jonas Holmberg, Linux mtd, Cristian Ionescu-Idbohrn
On Mon, 2017-09-11 at 17:18 +0200, Ricard Wanderlof wrote:
> On Mon, 11 Sep 2017, Richard Weinberger wrote:
>
> > On Mon, Sep 11, 2017 at 11:47 AM, Ricard Wanderlof
> > <ricard.wanderlof@axis.com> wrote:
> > >
> > > Is there a (simple) way to find out exactly how much space each
> > > specific
> > > file uses in an ubifs fils system?
> > >
> > > I looked into running mkfs.ubifs with some form of option to give
> > > me the
> > > info, the closest I got was -g 3 which output a lot of sizes but
> > > it was
> > > not obvious exactly what they meant.
> > >
> > > I suppose it would be fully possible to add code to mkfs.ubifs to
> > > output
> > > this information, as it surely must be there at some point, since
> > > it
> > > selects the compression algorithm on a file per file basis,
> > > debending on
> > > the amount of compression acheived for each algorithm.
> >
> > Well, how exactly should it be?
> > I guess you want more than the compressed size on flash.
> > Do you also want to know how much bytes a file occupies in the
> > index
> > tree and the inode itself?
> > This is non-trivial.
>
> Good point. I was solely thinking along the lines of how much space
> the
> actual file occupied, not considering metadata. That would be a good
> starting point. I'm guessing that for moderate file sizes the
> metadata
> would be relatively small compared to the file itself?
I would think a "slow" version of this would not be that hard to
implement - walk the index and sum up node sizes. Subtract header sizes
if you do not want metadata.
I am not sure what would be the API? Do other FSes implement something
like this?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-11 15:44 ` Artem Bityutskiy
@ 2017-09-11 19:57 ` Richard Weinberger
2017-09-12 9:15 ` Ricard Wanderlof
0 siblings, 1 reply; 10+ messages in thread
From: Richard Weinberger @ 2017-09-11 19:57 UTC (permalink / raw)
To: dedekind1, Jonas Holmberg
Cc: Ricard Wanderlof, Linux mtd, Cristian Ionescu-Idbohrn
Am Montag, 11. September 2017, 17:44:09 CEST schrieb Artem Bityutskiy:
> > Good point. I was solely thinking along the lines of how much space
> > the
> > actual file occupied, not considering metadata. That would be a good
> > starting point. I'm guessing that for moderate file sizes the
> > metadata
> > would be relatively small compared to the file itself?
>
> I would think a "slow" version of this would not be that hard to
> implement - walk the index and sum up node sizes. Subtract header sizes
> if you do not want metadata.
>
> I am not sure what would be the API? Do other FSes implement something
> like this?
I think a "show MTD usage by inode" should be implementable via debugfs.
Maybe, after a discussion on linux-fsdevel a per-file ioctl().
But first I'd like to know more about the use-case and where to draw the
border.
e.g. If a file as xattrs, do you also account them? UBIFS modules xattrs
via inodes. So, they have a rather huge space overhead.
Thanks,
//richard
--
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-11 19:57 ` Richard Weinberger
@ 2017-09-12 9:15 ` Ricard Wanderlof
2017-09-12 14:52 ` Richard Weinberger
0 siblings, 1 reply; 10+ messages in thread
From: Ricard Wanderlof @ 2017-09-12 9:15 UTC (permalink / raw)
To: Richard Weinberger
Cc: dedekind1, Jonas Holmberg, Linux mtd, Cristian Ionescu-Idbohrn
On Mon, 11 Sep 2017, Richard Weinberger wrote:
> Am Montag, 11. September 2017, 17:44:09 CEST schrieb Artem Bityutskiy:
> > > Good point. I was solely thinking along the lines of how much space
> > > the
> > > actual file occupied, not considering metadata. That would be a good
> > > starting point. I'm guessing that for moderate file sizes the
> > > metadata
> > > would be relatively small compared to the file itself?
> >
> > I would think a "slow" version of this would not be that hard to
> > implement - walk the index and sum up node sizes. Subtract header sizes
> > if you do not want metadata.
> >
> > I am not sure what would be the API? Do other FSes implement something
> > like this?
>
> I think a "show MTD usage by inode" should be implementable via debugfs.
> Maybe, after a discussion on linux-fsdevel a per-file ioctl().
>
> But first I'd like to know more about the use-case and where to draw the
> border. e.g. If a file as xattrs, do you also account them? UBIFS
> modules xattrs via inodes. So, they have a rather huge space overhead.
In our specific situation the use case is basically that given a specific
set files (in the root file system), how much flash is actually needed,
also considering that different types of files have different compression
ratios depending on the content. I.e. a large text file might not hurt as
much as a large binary since the former compresses better.
So primarily it is the total amount needed for the whole file system, but
it quickly comes down how much individual files consume. In our case it's
not necessary to do the operation on a running file system, but in the
more general case that might be more useful.
I would think that this is something that is known somewhere inside
mkfs.ubifs, or is the information too convoluted to be easily extractable?
/Ricard
--
Ricard Wolf Wanderlöf ricardw(at)axis.com
Axis Communications AB, Lund, Sweden www.axis.com
Phone +46 46 272 2016 Fax +46 46 13 61 30
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-12 9:15 ` Ricard Wanderlof
@ 2017-09-12 14:52 ` Richard Weinberger
2017-09-15 22:08 ` Ricard Wanderlof
0 siblings, 1 reply; 10+ messages in thread
From: Richard Weinberger @ 2017-09-12 14:52 UTC (permalink / raw)
To: Ricard Wanderlof
Cc: dedekind1, Jonas Holmberg, Linux mtd, Cristian Ionescu-Idbohrn
Ricard,
Am Dienstag, 12. September 2017, 11:15:25 CEST schrieb Ricard Wanderlof:
> On Mon, 11 Sep 2017, Richard Weinberger wrote:
> > Am Montag, 11. September 2017, 17:44:09 CEST schrieb Artem Bityutskiy:
> > > > Good point. I was solely thinking along the lines of how much space
> > > > the
> > > > actual file occupied, not considering metadata. That would be a good
> > > > starting point. I'm guessing that for moderate file sizes the
> > > > metadata
> > > > would be relatively small compared to the file itself?
> > >
> > > I would think a "slow" version of this would not be that hard to
> > > implement - walk the index and sum up node sizes. Subtract header sizes
> > > if you do not want metadata.
> > >
> > > I am not sure what would be the API? Do other FSes implement something
> > > like this?
> >
> > I think a "show MTD usage by inode" should be implementable via debugfs.
> > Maybe, after a discussion on linux-fsdevel a per-file ioctl().
> >
> > But first I'd like to know more about the use-case and where to draw the
> > border. e.g. If a file as xattrs, do you also account them? UBIFS
> > modules xattrs via inodes. So, they have a rather huge space overhead.
>
> In our specific situation the use case is basically that given a specific
> set files (in the root file system), how much flash is actually needed,
> also considering that different types of files have different compression
> ratios depending on the content. I.e. a large text file might not hurt as
> much as a large binary since the former compresses better.
>
> So primarily it is the total amount needed for the whole file system, but
> it quickly comes down how much individual files consume. In our case it's
> not necessary to do the operation on a running file system, but in the
> more general case that might be more useful.
>
> I would think that this is something that is known somewhere inside
> mkfs.ubifs, or is the information too convoluted to be easily extractable?
Adding this to mkfs.ubifs shouldn't be a big deal.
And I agree your use case makes sense.
Do you want to send me a patch? :)
Thanks,
//richard
--
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-12 14:52 ` Richard Weinberger
@ 2017-09-15 22:08 ` Ricard Wanderlof
2017-09-16 6:32 ` Richard Weinberger
0 siblings, 1 reply; 10+ messages in thread
From: Ricard Wanderlof @ 2017-09-15 22:08 UTC (permalink / raw)
To: Richard Weinberger
Cc: dedekind1, Jonas Holmberg, Linux mtd, Cristian Ionescu-Idbohrn
On Tue, 12 Sep 2017, Richard Weinberger wrote:
> > > But first I'd like to know more about the use-case and where to draw the
> > > border. e.g. If a file as xattrs, do you also account them? UBIFS
> > > modules xattrs via inodes. So, they have a rather huge space overhead.
> >
> > In our specific situation the use case is basically that given a specific
> > set files (in the root file system), how much flash is actually needed,
> > also considering that different types of files have different compression
> > ratios depending on the content. I.e. a large text file might not hurt as
> > much as a large binary since the former compresses better.
> >
> > So primarily it is the total amount needed for the whole file system, but
> > it quickly comes down how much individual files consume. In our case it's
> > not necessary to do the operation on a running file system, but in the
> > more general case that might be more useful.
> >
> > I would think that this is something that is known somewhere inside
> > mkfs.ubifs, or is the information too convoluted to be easily extractable?
>
> Adding this to mkfs.ubifs shouldn't be a big deal.
> And I agree your use case makes sense.
>
> Do you want to send me a patch? :)
Yes, that would be very much appreciated!
/Ricard
--
Ricard Wolf Wanderlöf ricardw(at)axis.com
Axis Communications AB, Lund, Sweden www.axis.com
Phone +46 46 272 2016 Fax +46 46 13 61 30
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-15 22:08 ` Ricard Wanderlof
@ 2017-09-16 6:32 ` Richard Weinberger
2017-09-18 6:37 ` Ricard Wanderlof
0 siblings, 1 reply; 10+ messages in thread
From: Richard Weinberger @ 2017-09-16 6:32 UTC (permalink / raw)
To: Ricard Wanderlof
Cc: dedekind1, Jonas Holmberg, Linux mtd, Cristian Ionescu-Idbohrn
Am Samstag, 16. September 2017, 00:08:53 CEST schrieb Ricard Wanderlof:
> > Do you want to send me a patch? :)
>
> Yes, that would be very much appreciated!
Uhh, sorry. Mad bad English tricked you.
I meant whether you want to create a patch and send it to me. :-)
Thanks,
//richard
--
sigma star gmbh - Eduard-Bodem-Gasse 6 - 6020 Innsbruck - Austria
ATU66964118 - FN 374287y
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Actual usage of files in ubifs
2017-09-16 6:32 ` Richard Weinberger
@ 2017-09-18 6:37 ` Ricard Wanderlof
0 siblings, 0 replies; 10+ messages in thread
From: Ricard Wanderlof @ 2017-09-18 6:37 UTC (permalink / raw)
To: Richard Weinberger
Cc: dedekind1, Jonas Holmberg, Linux mtd, Cristian Ionescu-Idbohrn
On Sat, 16 Sep 2017, Richard Weinberger wrote:
> Am Samstag, 16. September 2017, 00:08:53 CEST schrieb Ricard Wanderlof:
> > > Do you want to send me a patch? :)
> >
> > Yes, that would be very much appreciated!
>
> Uhh, sorry. Mad bad English tricked you.
> I meant whether you want to create a patch and send it to me. :-)
Sorry, my mistake, it's not your English at all, it's completely fine,
it's rather that I shouldn't be responding to mtd list posts on a Saturday
morning when I'm not really awake and not reading clearly. :-)
/Ricard
--
Ricard Wolf Wanderlöf ricardw(at)axis.com
Axis Communications AB, Lund, Sweden www.axis.com
Phone +46 46 272 2016 Fax +46 46 13 61 30
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2017-09-18 6:37 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-11 9:47 Actual usage of files in ubifs Ricard Wanderlof
2017-09-11 14:55 ` Richard Weinberger
2017-09-11 15:18 ` Ricard Wanderlof
2017-09-11 15:44 ` Artem Bityutskiy
2017-09-11 19:57 ` Richard Weinberger
2017-09-12 9:15 ` Ricard Wanderlof
2017-09-12 14:52 ` Richard Weinberger
2017-09-15 22:08 ` Ricard Wanderlof
2017-09-16 6:32 ` Richard Weinberger
2017-09-18 6:37 ` Ricard Wanderlof
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.