From: hujianyang <hujianyang@huawei.com> To: Andreas Dilger <adilger@dilger.ca>, <dedekind1@gmail.com> Cc: David Sterba <dsterba@suse.cz>, "markus.heininger@online.de" <markus.heininger@online.de>, <linux-mtd@lists.infradead.org>, "Linux Filesystem Development List" <linux-fsdevel@vger.kernel.org> Subject: Re: UBIFS: Is it possible to get the compressed size of a file? Date: Sat, 28 Feb 2015 15:31:14 +0800 [thread overview] Message-ID: <54F16EC2.3070703@huawei.com> (raw) In-Reply-To: <7B8DF06B-EBC8-4C85-AEEA-C2C6A52B8789@dilger.ca> Hi Artem and Andreas, I had worked on this feature for a few days. Now I meet some problems, I want to discuss them with you and wish you could give me some advises. On 2015/2/3 13:45, Andreas Dilger wrote: > On Feb 2, 2015, at 2:33 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote: >> >> Yes, no easy way, but I think implementing what you need is possible. I >> do not have plans and time to work on this, but I can help by giving >> advises and review. >> >> The question has 2 major parts. >> >> 1. The interface >> 2. The implementation >> >> For the former, one need to carefully investigate if there is something >> like this already implemented for other file-systems. I think btrfs may >> have it. If it is, then UBIFS should use similar interface, probably. >> >> And whatever is the interface choice, it should be discussed in the >> linux-fsdevel@vger.kernel.org mailing list, which I am CCing. > First, talking about the interface. > One option that was discussed for btrfs was to use the first fe_reserved > field for the FIEMAP ioctl struct fiemap_extent to fe_phys_length to > hold the compressed size of each extent in the file. > I don't think fiemap is a good interface for UBIFS and for compressed size reporting feature. The contents of files in UBIFS are sorted into *ubifs_data_node* which contains as much as UBIFS_BLOCK_SIZE(4096) bytes of data. A single file may contain lots of data nodes and these data nodes may locate on flash in order or in disorder because of out of place update. An fiemap ioctl from userspace need lots of memory to store discontinuous data mapping and copy these fiemap_extent may cost a lot of time. > http://lwn.net/Articles/607552/ > http://thread.gmane.org/gmane.comp.file-systems.btrfs/37312 > > I'm not sure what happened to that patch series - I was looking forward > to it landing, and it was in very good shape I think. > Since the *fe_phys_length* of fiemap_extent is not import to mainline, current fiemap can only report logical data length of an extent. Regret to say, it's no use for getting the compressed size of a file in UBIFS. Then, looking at the implement. >> >> a. 'struct ubifs_ino_node' has unused space, use it to add the >> compressed size field. >> b. maintain this field >> c. this field will only be correct for the part of the file which are on >> the media. The dirty data in the page cache has not yet been compressed, >> so we do not know its compressed size yet. >> e. when user asks for the compressed size, you have to sync the inode >> first, in order to make sure the compressed size is correct. >> >> And the implementation should be backward-compatible. That is, if new >> driver mounts the old media, you return something predicatable. I guess >> uncompressed size could be it. >> I'm worry about power cut recovery of compressed size if we introduce it into 'struct ubifs_ino_node'. We can't write a new metadata node after each changing of data node. Dirty data may not change the logical size of a file, but it must change the compressed size. How to keep the consistency between real compressed size(amount of each data nodes) and the record in metadata node? In logic size case, we could solve this problem by block number, because the size of each blocks are UBIFS_BLOCK_SIZE, each exist data node could tell the logic size of a file. Actually we use this method to fix the logic size of a file in recovery path. Since the physic size of each data node are not equal, we couldn't get the physic size of a file by a single data node in journal. And we couldn't record the total compressed size of a file in data node because it doesn't have enough reserve space, we couldn't use the same functionality for compressed size. Since metadata node(ubifs_ino_node) and data node(ubifs_data_node) are stored in different journals, I didn't find a easy way to keep consistency when a power cut happen. Seems a rebuilding whole scan can not be avoid in this case. So could we use a simply method? just a private ioctl which scan the tnc tree and report the compressed size of a UBIFS file? I found an old patch for Btrfs, but it is not import to mainline. https://patchwork.kernel.org/patch/117782/ Since the files in UBIFS are not too large, maybe we could test if the cost of time is acceptable for ordinary use case. Further, for both fiemap or this private ioctl method, current tnc tree lookup mechanism seems always copying the whole ubifs node, but only the header of a node is used in this case. Do we have a way to only read part of a node from tnc tree? Thanks, Hu
WARNING: multiple messages have this Message-ID (diff)
From: hujianyang <hujianyang@huawei.com> To: Andreas Dilger <adilger@dilger.ca>, <dedekind1@gmail.com> Cc: Linux Filesystem Development List <linux-fsdevel@vger.kernel.org>, linux-mtd@lists.infradead.org, David Sterba <dsterba@suse.cz>, "markus.heininger@online.de" <markus.heininger@online.de> Subject: Re: UBIFS: Is it possible to get the compressed size of a file? Date: Sat, 28 Feb 2015 15:31:14 +0800 [thread overview] Message-ID: <54F16EC2.3070703@huawei.com> (raw) In-Reply-To: <7B8DF06B-EBC8-4C85-AEEA-C2C6A52B8789@dilger.ca> Hi Artem and Andreas, I had worked on this feature for a few days. Now I meet some problems, I want to discuss them with you and wish you could give me some advises. On 2015/2/3 13:45, Andreas Dilger wrote: > On Feb 2, 2015, at 2:33 AM, Artem Bityutskiy <dedekind1@gmail.com> wrote: >> >> Yes, no easy way, but I think implementing what you need is possible. I >> do not have plans and time to work on this, but I can help by giving >> advises and review. >> >> The question has 2 major parts. >> >> 1. The interface >> 2. The implementation >> >> For the former, one need to carefully investigate if there is something >> like this already implemented for other file-systems. I think btrfs may >> have it. If it is, then UBIFS should use similar interface, probably. >> >> And whatever is the interface choice, it should be discussed in the >> linux-fsdevel@vger.kernel.org mailing list, which I am CCing. > First, talking about the interface. > One option that was discussed for btrfs was to use the first fe_reserved > field for the FIEMAP ioctl struct fiemap_extent to fe_phys_length to > hold the compressed size of each extent in the file. > I don't think fiemap is a good interface for UBIFS and for compressed size reporting feature. The contents of files in UBIFS are sorted into *ubifs_data_node* which contains as much as UBIFS_BLOCK_SIZE(4096) bytes of data. A single file may contain lots of data nodes and these data nodes may locate on flash in order or in disorder because of out of place update. An fiemap ioctl from userspace need lots of memory to store discontinuous data mapping and copy these fiemap_extent may cost a lot of time. > http://lwn.net/Articles/607552/ > http://thread.gmane.org/gmane.comp.file-systems.btrfs/37312 > > I'm not sure what happened to that patch series - I was looking forward > to it landing, and it was in very good shape I think. > Since the *fe_phys_length* of fiemap_extent is not import to mainline, current fiemap can only report logical data length of an extent. Regret to say, it's no use for getting the compressed size of a file in UBIFS. Then, looking at the implement. >> >> a. 'struct ubifs_ino_node' has unused space, use it to add the >> compressed size field. >> b. maintain this field >> c. this field will only be correct for the part of the file which are on >> the media. The dirty data in the page cache has not yet been compressed, >> so we do not know its compressed size yet. >> e. when user asks for the compressed size, you have to sync the inode >> first, in order to make sure the compressed size is correct. >> >> And the implementation should be backward-compatible. That is, if new >> driver mounts the old media, you return something predicatable. I guess >> uncompressed size could be it. >> I'm worry about power cut recovery of compressed size if we introduce it into 'struct ubifs_ino_node'. We can't write a new metadata node after each changing of data node. Dirty data may not change the logical size of a file, but it must change the compressed size. How to keep the consistency between real compressed size(amount of each data nodes) and the record in metadata node? In logic size case, we could solve this problem by block number, because the size of each blocks are UBIFS_BLOCK_SIZE, each exist data node could tell the logic size of a file. Actually we use this method to fix the logic size of a file in recovery path. Since the physic size of each data node are not equal, we couldn't get the physic size of a file by a single data node in journal. And we couldn't record the total compressed size of a file in data node because it doesn't have enough reserve space, we couldn't use the same functionality for compressed size. Since metadata node(ubifs_ino_node) and data node(ubifs_data_node) are stored in different journals, I didn't find a easy way to keep consistency when a power cut happen. Seems a rebuilding whole scan can not be avoid in this case. So could we use a simply method? just a private ioctl which scan the tnc tree and report the compressed size of a UBIFS file? I found an old patch for Btrfs, but it is not import to mainline. https://patchwork.kernel.org/patch/117782/ Since the files in UBIFS are not too large, maybe we could test if the cost of time is acceptable for ordinary use case. Further, for both fiemap or this private ioctl method, current tnc tree lookup mechanism seems always copying the whole ubifs node, but only the header of a node is used in this case. Do we have a way to only read part of a node from tnc tree? Thanks, Hu
next prev parent reply other threads:[~2015-02-28 7:32 UTC|newest] Thread overview: 106+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-12-12 8:10 UBIFS: Is it possible to get the compressed size of a file? markus.heininger 2014-12-17 6:22 ` hujianyang 2014-12-18 11:08 ` Aw: " markus.heininger 2015-02-02 9:33 ` Artem Bityutskiy 2015-02-02 9:33 ` Artem Bityutskiy 2015-02-03 5:45 ` Andreas Dilger 2015-02-03 5:45 ` Andreas Dilger 2015-02-03 9:07 ` Artem Bityutskiy 2015-02-03 9:07 ` Artem Bityutskiy 2015-02-28 7:31 ` hujianyang [this message] 2015-02-28 7:31 ` hujianyang 2015-03-26 14:19 ` David Sterba 2015-03-26 14:19 ` David Sterba -- strict thread matches above, loose matches on Subject: below -- 2015-03-10 12:03 OpenStack - Libvirt+Xen CI overview Bob Ball 2015-03-16 15:09 ` OpenStack - Libvirt+Xen CI overview : anyone interested in a walk/through or presentation on how it works Lars Kurth 2015-03-16 18:00 ` OpenStack - Libvirt+Xen CI overview Konrad Rzeszutek Wilk 2015-03-17 12:52 ` Dario Faggioli 2015-03-17 11:14 ` George Dunlap 2015-03-17 14:22 ` Jim Fehlig 2015-03-17 15:51 ` Alvin Starr 2015-03-18 13:12 ` OpenStack - Libvirt+Xen CI overview : anyone interested in a walk/through or presentation on how it works Lars Kurth 2015-03-18 14:37 ` Wei Liu 2015-03-24 18:17 ` Lars Kurth 2015-03-25 10:08 ` Ian Campbell 2015-03-25 10:09 ` Bob Ball 2015-03-25 12:26 ` Alvin Starr 2015-04-22 10:03 ` Lars Kurth 2015-06-04 17:12 ` Bob Ball 2015-06-05 11:47 ` Lars Kurth 2015-06-08 12:52 ` Bob Ball 2015-06-10 14:25 ` Bob Ball 2015-06-15 16:26 ` Bob Ball [not found] <10286402.11.1422626774586.JavaMail.kraghavendra@Raghavendra> 2015-01-30 14:08 ` Yocto-custom kernel build issue Raghavendra Kakarla 2015-01-30 16:34 ` Bruce Ashfield 2015-02-02 10:00 ` Raghavendra Kakarla 2015-02-03 12:40 ` yocto-custom-kernel issue Raghavendra Kakarla 2015-02-03 13:17 ` Sven Ebenfeld 2015-02-03 15:37 ` Bruce Ashfield 2015-02-05 8:56 ` MIPS32r2 little endian bsp Raghavendra Kakarla 2015-02-06 15:04 ` Mark Hatle 2015-02-10 8:17 ` Yocto-custom kernel build issue Raghavendra Kakarla 2015-02-10 11:00 ` Paul Eggleton [not found] ` <2974616.30.1423571125144.JavaMail.kraghavendra@Raghavendra> 2015-02-10 13:36 ` Paul Eggleton 2015-02-11 4:55 ` Raghavendra Kakarla 2015-02-11 9:30 ` Paul Eggleton 2015-02-11 10:00 ` Raghavendra Kakarla 2015-03-03 5:41 ` SDK generation issue Raghavendra Kakarla 2015-03-03 9:37 ` Paul Eggleton 2015-03-03 10:26 ` Raghavendra Kakarla 2015-03-03 10:31 ` Paul Eggleton 2015-03-03 11:55 ` Raghavendra Kakarla 2015-03-03 12:10 ` Paul Eggleton 2015-03-03 12:17 ` Raghavendra Kakarla 2015-03-03 13:16 ` Paul Eggleton 2015-03-10 4:21 ` Difference b/w yocto kernels and normal linux.org kernels Raghavendra Kakarla 2015-03-10 4:33 ` Khem Raj 2015-03-10 4:45 ` Nicholas Krause 2015-03-13 8:14 ` Raghavendra Kakarla 2015-03-13 8:53 ` Paul Eggleton 2015-03-13 9:05 ` Albert K 2015-03-13 9:12 ` Paul Eggleton 2015-04-03 6:04 ` Raghavendra Kakarla 2015-03-03 17:55 ` SDK generation issue Jim Rafert 2014-11-06 13:27 lots of connections in SYN_RECV state Puneet Agarwal 2014-11-06 14:30 ` Silvan Jegen 2014-11-06 15:15 ` Puneet Agarwal 2014-11-06 15:58 ` Silvan Jegen 2014-11-07 15:49 ` Dave Tian 2014-11-07 16:58 ` Valdis.Kletnieks at vt.edu 2014-11-07 23:48 ` Dave Tian 2014-11-07 17:41 ` Puneet Agarwal 2014-11-07 18:10 ` Valdis.Kletnieks at vt.edu 2014-11-08 2:05 ` Puneet Agarwal 2012-03-19 20:59 Linking two recipes simran singh 2012-03-19 21:22 ` Richard Purdie 2012-03-19 21:33 ` simran singh 2012-03-19 21:39 ` Christopher Larson 2012-03-19 21:57 ` simran singh 2012-03-19 21:59 ` Christopher Larson 2012-03-19 22:10 ` simran singh 2012-03-19 22:54 ` Richard Purdie 2012-03-20 19:25 ` simran singh 2012-04-07 3:44 ` Khem Raj 2005-01-11 9:28 [U-Boot-Users] PCI, Ethernet, IXDP425 Ara Avanesyan 2005-01-11 22:40 ` Wolfgang Denk 2005-01-12 6:18 ` [U-Boot-Users] " Ara Avanesyan 2005-01-12 6:41 ` Rodel Miguel 2005-01-12 7:41 ` Ara Avanesyan 2005-01-12 8:08 ` Ing.Gianfranco Morandi 2005-01-12 9:01 ` Wolfgang Denk 2005-01-12 9:32 ` Ara Avanesyan 2005-01-12 10:13 ` Anders Larsen 2005-01-12 14:16 ` Wolfgang Denk 2005-01-12 14:16 ` Wolfgang Denk 2005-01-12 15:07 ` Ara Avanesyan 2003-11-12 17:23 [U-Boot-Users] [PATCH] fix the ARM memory layout Anders Larsen 2003-11-12 19:52 ` Kyle Harris 2003-11-13 9:14 ` Anders Larsen 2003-11-13 14:19 ` Kyle Harris 2003-11-21 3:52 ` Kyle Harris 2003-11-21 7:47 ` Anders Larsen 2003-11-21 8:13 ` Wolfgang Denk 2003-12-07 0:17 ` Wolfgang Denk 2003-12-07 17:03 ` Anders Larsen 2004-01-09 11:14 ` Anders Larsen 2004-02-08 19:36 ` Wolfgang Denk
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=54F16EC2.3070703@huawei.com \ --to=hujianyang@huawei.com \ --cc=adilger@dilger.ca \ --cc=dedekind1@gmail.com \ --cc=dsterba@suse.cz \ --cc=linux-fsdevel@vger.kernel.org \ --cc=linux-mtd@lists.infradead.org \ --cc=markus.heininger@online.de \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.