* [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files
@ 2015-12-16 16:33 Joshua Lock
2015-12-17 12:21 ` Ed Bartosh
2015-12-17 16:09 ` [PATCH] oeqa/selftest/wic: add test case for sparse images Joshua Lock
0 siblings, 2 replies; 10+ messages in thread
From: Joshua Lock @ 2015-12-16 16:33 UTC (permalink / raw)
To: openembedded-core
The individual partitions created by wic are sparse but without
this change the assembled image is written as one (potentially
very) large file.
Preserve sparseness in the assembled image by passing the sparse
conversion symbol.
Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
---
scripts/lib/wic/utils/partitionedfs.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/wic/utils/partitionedfs.py b/scripts/lib/wic/utils/partitionedfs.py
index 5a103bb..3e2b420 100644
--- a/scripts/lib/wic/utils/partitionedfs.py
+++ b/scripts/lib/wic/utils/partitionedfs.py
@@ -340,7 +340,7 @@ class Image(object):
source = part['source_file']
if source:
# install source_file contents into a partition
- cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc" % \
+ cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc,sparse" % \
(source, image_file, self.sector_size,
part['start'], part['size'])
exec_cmd(cmd)
--
2.5.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files
2015-12-16 16:33 [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files Joshua Lock
@ 2015-12-17 12:21 ` Ed Bartosh
2015-12-17 13:19 ` Joshua Lock
2015-12-17 16:09 ` [PATCH] oeqa/selftest/wic: add test case for sparse images Joshua Lock
1 sibling, 1 reply; 10+ messages in thread
From: Ed Bartosh @ 2015-12-17 12:21 UTC (permalink / raw)
To: Joshua Lock; +Cc: openembedded-core
Hi Joshua,
Thank you for the patch! I like the change. It's a first step towards
supporting bmaptool, which is in my TODO list.
Would you be willing to write or modify wic test case to test this?
On Wed, Dec 16, 2015 at 04:33:13PM +0000, Joshua Lock wrote:
> The individual partitions created by wic are sparse but without
> this change the assembled image is written as one (potentially
> very) large file.
>
> Preserve sparseness in the assembled image by passing the sparse
> conversion symbol.
>
> Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
> ---
> scripts/lib/wic/utils/partitionedfs.py | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/lib/wic/utils/partitionedfs.py b/scripts/lib/wic/utils/partitionedfs.py
> index 5a103bb..3e2b420 100644
> --- a/scripts/lib/wic/utils/partitionedfs.py
> +++ b/scripts/lib/wic/utils/partitionedfs.py
> @@ -340,7 +340,7 @@ class Image(object):
> source = part['source_file']
> if source:
> # install source_file contents into a partition
> - cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc" % \
> + cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc,sparse" % \
> (source, image_file, self.sector_size,
> part['start'], part['size'])
> exec_cmd(cmd)
> --
> 2.5.0
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
--
Regards,
Ed
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files
2015-12-17 12:21 ` Ed Bartosh
@ 2015-12-17 13:19 ` Joshua Lock
2015-12-17 13:23 ` Burton, Ross
2015-12-17 13:53 ` Ed Bartosh
0 siblings, 2 replies; 10+ messages in thread
From: Joshua Lock @ 2015-12-17 13:19 UTC (permalink / raw)
To: ed.bartosh; +Cc: openembedded-core
Hi Ed,
On 17/12/15 12:21, Ed Bartosh wrote:
> Thank you for the patch! I like the change. It's a first step towards
> supporting bmaptool, which is in my TODO list.
No problem, I've been playing with bmaptool myself and that's what
prompted the change.
> Would you be willing to write or modify wic test case to test this?
I'd be willing to do that, is there any documentation on the development
of test cases?
I see meta/lib/oeqa/selftest/wic.py has a bunch of functions with
@testcase decorators but I'm not familiar with how this should all fit
together.
Regards,
Joshua
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files
2015-12-17 13:19 ` Joshua Lock
@ 2015-12-17 13:23 ` Burton, Ross
2015-12-17 13:53 ` Ed Bartosh
1 sibling, 0 replies; 10+ messages in thread
From: Burton, Ross @ 2015-12-17 13:23 UTC (permalink / raw)
To: Joshua Lock; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 487 bytes --]
On 17 December 2015 at 13:19, Joshua Lock <joshua.lock@collabora.co.uk>
wrote:
> I see meta/lib/oeqa/selftest/wic.py has a bunch of functions with
> @testcase decorators but I'm not familiar with how this should all fit
> together.
>
@testcase is just to map from test functions to testopia entries. This
doesn't (yet) have a testopia entry, so you can leave it.
Apart from that it's standard Python unittest, so find a test that's
somewhat similar and copy it.
Ross
[-- Attachment #2: Type: text/html, Size: 971 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files
2015-12-17 13:19 ` Joshua Lock
2015-12-17 13:23 ` Burton, Ross
@ 2015-12-17 13:53 ` Ed Bartosh
1 sibling, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2015-12-17 13:53 UTC (permalink / raw)
To: Joshua Lock; +Cc: openembedded-core
On Thu, Dec 17, 2015 at 01:19:47PM +0000, Joshua Lock wrote:
> Hi Ed,
>
> On 17/12/15 12:21, Ed Bartosh wrote:
> >Thank you for the patch! I like the change. It's a first step towards
> >supporting bmaptool, which is in my TODO list.
>
> No problem, I've been playing with bmaptool myself and that's what
> prompted the change.
>
> >Would you be willing to write or modify wic test case to test this?
>
> I'd be willing to do that, is there any documentation on the
> development of test cases?
You can find short info here:
https://wiki.yoctoproject.org/wiki/Oe-selftest
And here https://docs.python.org/2/library/unittest.html as oe-selftest
test cases are based on Python unittest.
> I see meta/lib/oeqa/selftest/wic.py has a bunch of functions with
> @testcase decorators but I'm not familiar with how this should all
> fit together.
You've spotted it right. All wic testcases are in that file.
You can run them this way: oe-selftest --run-tests wic
As Ross already pointed out you can omit @testcase decorator if you add
new test. QA team will add it later when they need it.
--
Regards,
Ed
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH] oeqa/selftest/wic: add test case for sparse images
2015-12-16 16:33 [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files Joshua Lock
2015-12-17 12:21 ` Ed Bartosh
@ 2015-12-17 16:09 ` Joshua Lock
2015-12-17 16:21 ` Burton, Ross
2015-12-17 19:43 ` [[PATCH v2] " Joshua Lock
1 sibling, 2 replies; 10+ messages in thread
From: Joshua Lock @ 2015-12-17 16:09 UTC (permalink / raw)
To: openembedded-core
Add a testcase to build a directdisk image and check that the
used disk size is less than the apparent size, as wic now
assembles images as sparse files.
Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
---
meta/lib/oeqa/selftest/wic.py | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 55f6f82..e85363c 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -264,3 +264,15 @@ class Wic(oeSelfTest):
self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
% image).status)
self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+ def test_sparseness(self):
+ """Test that assembled images are sparse; apparent size > disk usage"""
+ self.assertEqual(0, runCmd("wic create directdisk "
+ "--image-name core-image-minimal").status)
+ images = glob(self.resultdir + "directdisk-*.direct")
+ self.assertEqual(1, len(images))
+
+ path = images.pop()
+ used = int(runCmd("du %s | awk '{print $1}'" % path).output)
+ apparent = int(runCmd("du --apparent-size %s | awk '{print $1}'" % path).output)
+ self.assertLess(used, apparent)
--
2.5.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH] oeqa/selftest/wic: add test case for sparse images
2015-12-17 16:09 ` [PATCH] oeqa/selftest/wic: add test case for sparse images Joshua Lock
@ 2015-12-17 16:21 ` Burton, Ross
2015-12-17 16:52 ` Joshua Lock
2015-12-17 19:43 ` [[PATCH v2] " Joshua Lock
1 sibling, 1 reply; 10+ messages in thread
From: Burton, Ross @ 2015-12-17 16:21 UTC (permalink / raw)
To: Joshua Lock; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 304 bytes --]
On 17 December 2015 at 16:09, Joshua Lock <joshua.lock@collabora.co.uk>
wrote:
> + used = int(runCmd("du %s | awk '{print $1}'" % path).output)
> + apparent = int(runCmd("du --apparent-size %s | awk '{print $1}'"
> % path).output)
>
Surely os.stat() can do this instead?
Ross
[-- Attachment #2: Type: text/html, Size: 759 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH] oeqa/selftest/wic: add test case for sparse images
2015-12-17 16:21 ` Burton, Ross
@ 2015-12-17 16:52 ` Joshua Lock
0 siblings, 0 replies; 10+ messages in thread
From: Joshua Lock @ 2015-12-17 16:52 UTC (permalink / raw)
To: Burton, Ross; +Cc: OE-core
On 17/12/15 16:21, Burton, Ross wrote:
>
> On 17 December 2015 at 16:09, Joshua Lock <joshua.lock@collabora.co.uk
> <mailto:joshua.lock@collabora.co.uk>> wrote:
>
> + used = int(runCmd("du %s | awk '{print $1}'" % path).output)
> + apparent = int(runCmd("du --apparent-size %s | awk '{print
> $1}'" % path).output)
>
>
> Surely os.stat() can do this instead?
Of course it can — st_blocks is the "number of 512-byte blocks allocated
for file" we can multiply st_blocks by 512 to get the on-disk size
(whilst st_size is the apparent size).
https://docs.python.org/2/library/os.html#os.stat
I'll create a v2.
Joshua
^ permalink raw reply [flat|nested] 10+ messages in thread
* [[PATCH v2] oeqa/selftest/wic: add test case for sparse images
2015-12-17 16:09 ` [PATCH] oeqa/selftest/wic: add test case for sparse images Joshua Lock
2015-12-17 16:21 ` Burton, Ross
@ 2015-12-17 19:43 ` Joshua Lock
2015-12-30 9:53 ` Ed Bartosh
1 sibling, 1 reply; 10+ messages in thread
From: Joshua Lock @ 2015-12-17 19:43 UTC (permalink / raw)
To: openembedded-core
Add a testcase to build a directdisk image and check that the
used disk size is less than the apparent size, as wic now
assembles images as sparse files.
Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
---
meta/lib/oeqa/selftest/wic.py | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 55f6f82..9050d28 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -264,3 +264,16 @@ class Wic(oeSelfTest):
self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
% image).status)
self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+
+ def test_sparseness(self):
+ """Test that assembled images are sparse; apparent size > disk usage"""
+ self.assertEqual(0, runCmd("wic create directdisk "
+ "--image-name core-image-minimal").status)
+ images = glob(self.resultdir + "directdisk-*.direct")
+ self.assertEqual(1, len(images))
+
+ imagestat = os.stat(images.pop())
+ # st_blocks is the "number of 512-byte blocks allocated for file"
+ used = imagestat.st_blocks*512
+ apparent = imagestat.st_size
+ self.assertLess(used, apparent)
--
2.5.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [[PATCH v2] oeqa/selftest/wic: add test case for sparse images
2015-12-17 19:43 ` [[PATCH v2] " Joshua Lock
@ 2015-12-30 9:53 ` Ed Bartosh
0 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2015-12-30 9:53 UTC (permalink / raw)
To: Joshua Lock; +Cc: openembedded-core
Hi Joshua,
Thank you for the patch!
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
On Thu, Dec 17, 2015 at 07:43:41PM +0000, Joshua Lock wrote:
> Add a testcase to build a directdisk image and check that the
> used disk size is less than the apparent size, as wic now
> assembles images as sparse files.
>
> Signed-off-by: Joshua Lock <joshua.lock@collabora.co.uk>
> ---
> meta/lib/oeqa/selftest/wic.py | 13 +++++++++++++
> 1 file changed, 13 insertions(+)
>
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index 55f6f82..9050d28 100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -264,3 +264,16 @@ class Wic(oeSelfTest):
> self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \
> % image).status)
> self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
> +
> + def test_sparseness(self):
> + """Test that assembled images are sparse; apparent size > disk usage"""
> + self.assertEqual(0, runCmd("wic create directdisk "
> + "--image-name core-image-minimal").status)
> + images = glob(self.resultdir + "directdisk-*.direct")
> + self.assertEqual(1, len(images))
> +
> + imagestat = os.stat(images.pop())
> + # st_blocks is the "number of 512-byte blocks allocated for file"
> + used = imagestat.st_blocks*512
> + apparent = imagestat.st_size
> + self.assertLess(used, apparent)
--
Regards,
Ed
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-12-30 10:25 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-16 16:33 [PATCH] wic/utils/partitionedfs.py: assemble .wic images as sparse files Joshua Lock
2015-12-17 12:21 ` Ed Bartosh
2015-12-17 13:19 ` Joshua Lock
2015-12-17 13:23 ` Burton, Ross
2015-12-17 13:53 ` Ed Bartosh
2015-12-17 16:09 ` [PATCH] oeqa/selftest/wic: add test case for sparse images Joshua Lock
2015-12-17 16:21 ` Burton, Ross
2015-12-17 16:52 ` Joshua Lock
2015-12-17 19:43 ` [[PATCH v2] " Joshua Lock
2015-12-30 9:53 ` Ed Bartosh
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.