Query: UBIFS: How to detect empty volumes

Query: UBIFS: How to detect empty volumes

[Pintu Agarwal]

[-- Attachment #1.1: Type: text/plain, Size: 1552 bytes --]

Hi,

I have one basic query related to UBIFS volumes on a system with NAND
partition.
In short, how to detect a particular empty volume inside a system partition
while flashing the partition image?

Suppose I have one big system partition that consists of 4-5 ubi volumes
inside it with varying sizes.
Lets say:
-- System Partition (ubi image)
    - rootfs volume (ro, squashfs)
    - data volume (rw, ubifs)
    - firmware volume (ro, ubifs)
    - some-other volume (ro, squashfs)

Consider that all these could be flashed together as part of system.ubi
image in a single shot from the bootloader.
Now, suppose, one of the volume image (say firmware) is missing or remains
empty (as you know we can have empty volumes).

So, during system image flashing, we wanted to detect if one of the volume
(firmware) is empty.
Since this is an important volume, so we wanted to detect if this volume is
empty/missing we will abort flashing the system partition.
As there is no point in booting the system without this partition.

So, I am exploring options, how can this be detected ?
I mean is there any kind of magic number or header information which we can
read to detect a particular empty volume ?
Can we get any information from just "system.ubi" image to indicate about
the volume information ?
Also it could be possible that 2 or more volumes are empty, but we are only
concerned about one particular volume (firmware), so how to detect
particular volume ?

If anyone has any thoughts about this requirement, please share your
opinion.


Thanks,
Pintu

[-- Attachment #1.2: Type: text/html, Size: 3839 bytes --]

[-- Attachment #2: Type: text/plain, Size: 170 bytes --]

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Query: UBIFS: How to detect empty volumes

[Pintu Agarwal]

Hi,

I have one basic query related to UBIFS volumes on a system with NAND partition.
In short, how to detect a particular empty volume inside a system
partition while flashing the partition image?

Suppose I have one big system partition that consists of 4-5 ubi
volumes inside it with varying sizes.
Lets say:
-- System Partition (ubi image)
    - rootfs volume (ro, squashfs)
    - data volume (rw, ubifs)
    - firmware volume (ro, ubifs)
    - some-other volume (ro, squashfs)

Consider that all these could be flashed together as part of
system.ubi image in a single shot from the bootloader.
Now, suppose, one of the volume image (say firmware) is missing or
remains empty (as you know we can have empty volumes).

So, during system image flashing, we wanted to detect if one of the
volume (firmware) is empty.
Since this is an important volume, so we wanted to detect if this
volume is empty/missing we will abort flashing the system partition.
As there is no point in booting the system without this partition.

So, I am exploring options, how can this be detected ?
I mean is there any kind of magic number or header information which
we can read to detect a particular empty volume ?
Can we get any information from just "system.ubi" image to indicate
about the volume information ?
Also it could be possible that 2 or more volumes are empty, but we are
only concerned about one particular volume (firmware), so how to
detect particular volume ?

If anyone has any thoughts about this requirement, please share your opinion.


Thanks,
Pintu

Re: Query: UBIFS: How to detect empty volumes

[Pintu Agarwal]

Hi,

I have one basic query related to UBIFS volumes on a system with NAND partition.
In short, how to detect a particular empty volume inside a system
partition while flashing the partition image?

Suppose I have one big system partition that consists of 4-5 ubi
volumes inside it with varying sizes.
Lets say:
-- System Partition (ubi image)
    - rootfs volume (ro, squashfs)
    - data volume (rw, ubifs)
    - firmware volume (ro, ubifs)
    - some-other volume (ro, squashfs)

Consider that all these could be flashed together as part of
system.ubi image in a single shot from the bootloader.
Now, suppose, one of the volume image (say firmware) is missing or
remains empty (as you know we can have empty volumes).

So, during system image flashing, we wanted to detect if one of the
volume (firmware) is empty.
Since this is an important volume, so we wanted to detect if this
volume is empty/missing we will abort flashing the system partition.
As there is no point in booting the system without this partition.

So, I am exploring options, how can this be detected ?
I mean is there any kind of magic number or header information which
we can read to detect a particular empty volume ?
Can we get any information from just "system.ubi" image to indicate
about the volume information ?
Also it could be possible that 2 or more volumes are empty, but we are
only concerned about one particular volume (firmware), so how to
detect particular volume ?

If anyone has any thoughts about this requirement, please share your opinion.


Thanks,
Pintu

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Query: UBIFS: How to detect empty volumes

[Pintu Agarwal]

Hi,

I have one basic query related to UBIFS volumes on a system with NAND partition.
In short, how to detect a particular empty volume inside a system
partition while flashing the partition image?

Suppose I have one big system partition that consists of 4-5 ubi
volumes inside it with varying sizes.
Lets say:
-- System Partition (ubi image)
    - rootfs volume (ro, squashfs)
    - data volume (rw, ubifs)
    - firmware volume (ro, ubifs)
    - some-other volume (ro, squashfs)

Consider that all these could be flashed together as part of
system.ubi image in a single shot from the bootloader.
Now, suppose, one of the volume image (say firmware) is missing or
remains empty (as you know we can have empty volumes).

So, during system image flashing, we wanted to detect if one of the
volume (firmware) is empty.
Since this is an important volume, so we wanted to detect if this
volume is empty/missing we will abort flashing the system partition.
As there is no point in booting the system without this partition.

So, I am exploring options, how can this be detected ?
I mean is there any kind of magic number or header information which
we can read to detect a particular empty volume ?
Can we get any information from just "system.ubi" image to indicate
about the volume information ?
Also it could be possible that 2 or more volumes are empty, but we are
only concerned about one particular volume (firmware), so how to
detect particular volume ?

If anyone has any thoughts about this requirement, please share your opinion.


Thanks,
Pintu

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Query: UBIFS: How to detect empty volumes

[Richard Weinberger]

On Thu, Jun 24, 2021 at 6:09 PM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> I have one basic query related to UBIFS volumes on a system with NAND partition.

There is no such thing as a UBIFS volume. Do you mean UBI volumes?

> In short, how to detect a particular empty volume inside a system
> partition while flashing the partition image?

What do you mean by system partition? A MTD partition?

> Suppose I have one big system partition that consists of 4-5 ubi
> volumes inside it with varying sizes.
> Lets say:
> -- System Partition (ubi image)
>     - rootfs volume (ro, squashfs)
>     - data volume (rw, ubifs)
>     - firmware volume (ro, ubifs)
>     - some-other volume (ro, squashfs)

So by system partition you mean the MTD partition that hosts UBI itself?

> Consider that all these could be flashed together as part of
> system.ubi image in a single shot from the bootloader.
> Now, suppose, one of the volume image (say firmware) is missing or
> remains empty (as you know we can have empty volumes).
>
> So, during system image flashing, we wanted to detect if one of the
> volume (firmware) is empty.
> Since this is an important volume, so we wanted to detect if this
> volume is empty/missing we will abort flashing the system partition.
> As there is no point in booting the system without this partition.
>
> So, I am exploring options, how can this be detected ?

Read from the volume after flashing. If you get only 0xFF bytes it is empty.

> I mean is there any kind of magic number or header information which
> we can read to detect a particular empty volume ?
> Can we get any information from just "system.ubi" image to indicate
> about the volume information ?

You'll need to perform a proper UBI scan on all blocks.
If for the sad volume no LEBs have been found it is empty.

> Also it could be possible that 2 or more volumes are empty, but we are
> only concerned about one particular volume (firmware), so how to
> detect particular volume ?

I don't understand the use case. Is your image creation process so error prone
that you can't be sure whether critical parts got included or not?

-- 
Thanks,
//richard

Re: Query: UBIFS: How to detect empty volumes

[Richard Weinberger]

On Thu, Jun 24, 2021 at 6:09 PM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> I have one basic query related to UBIFS volumes on a system with NAND partition.

There is no such thing as a UBIFS volume. Do you mean UBI volumes?

> In short, how to detect a particular empty volume inside a system
> partition while flashing the partition image?

What do you mean by system partition? A MTD partition?

> Suppose I have one big system partition that consists of 4-5 ubi
> volumes inside it with varying sizes.
> Lets say:
> -- System Partition (ubi image)
>     - rootfs volume (ro, squashfs)
>     - data volume (rw, ubifs)
>     - firmware volume (ro, ubifs)
>     - some-other volume (ro, squashfs)

So by system partition you mean the MTD partition that hosts UBI itself?

> Consider that all these could be flashed together as part of
> system.ubi image in a single shot from the bootloader.
> Now, suppose, one of the volume image (say firmware) is missing or
> remains empty (as you know we can have empty volumes).
>
> So, during system image flashing, we wanted to detect if one of the
> volume (firmware) is empty.
> Since this is an important volume, so we wanted to detect if this
> volume is empty/missing we will abort flashing the system partition.
> As there is no point in booting the system without this partition.
>
> So, I am exploring options, how can this be detected ?

Read from the volume after flashing. If you get only 0xFF bytes it is empty.

> I mean is there any kind of magic number or header information which
> we can read to detect a particular empty volume ?
> Can we get any information from just "system.ubi" image to indicate
> about the volume information ?

You'll need to perform a proper UBI scan on all blocks.
If for the sad volume no LEBs have been found it is empty.

> Also it could be possible that 2 or more volumes are empty, but we are
> only concerned about one particular volume (firmware), so how to
> detect particular volume ?

I don't understand the use case. Is your image creation process so error prone
that you can't be sure whether critical parts got included or not?

-- 
Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Query: UBIFS: How to detect empty volumes

[Richard Weinberger]

On Thu, Jun 24, 2021 at 6:09 PM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> I have one basic query related to UBIFS volumes on a system with NAND partition.

There is no such thing as a UBIFS volume. Do you mean UBI volumes?

> In short, how to detect a particular empty volume inside a system
> partition while flashing the partition image?

What do you mean by system partition? A MTD partition?

> Suppose I have one big system partition that consists of 4-5 ubi
> volumes inside it with varying sizes.
> Lets say:
> -- System Partition (ubi image)
>     - rootfs volume (ro, squashfs)
>     - data volume (rw, ubifs)
>     - firmware volume (ro, ubifs)
>     - some-other volume (ro, squashfs)

So by system partition you mean the MTD partition that hosts UBI itself?

> Consider that all these could be flashed together as part of
> system.ubi image in a single shot from the bootloader.
> Now, suppose, one of the volume image (say firmware) is missing or
> remains empty (as you know we can have empty volumes).
>
> So, during system image flashing, we wanted to detect if one of the
> volume (firmware) is empty.
> Since this is an important volume, so we wanted to detect if this
> volume is empty/missing we will abort flashing the system partition.
> As there is no point in booting the system without this partition.
>
> So, I am exploring options, how can this be detected ?

Read from the volume after flashing. If you get only 0xFF bytes it is empty.

> I mean is there any kind of magic number or header information which
> we can read to detect a particular empty volume ?
> Can we get any information from just "system.ubi" image to indicate
> about the volume information ?

You'll need to perform a proper UBI scan on all blocks.
If for the sad volume no LEBs have been found it is empty.

> Also it could be possible that 2 or more volumes are empty, but we are
> only concerned about one particular volume (firmware), so how to
> detect particular volume ?

I don't understand the use case. Is your image creation process so error prone
that you can't be sure whether critical parts got included or not?

-- 
Thanks,
//richard

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Query: UBIFS: How to detect empty volumes

[Pintu Agarwal]

On Sun, 27 Jun 2021 at 14:12, Richard Weinberger
<richard.weinberger@gmail.com> wrote:
>
> On Thu, Jun 24, 2021 at 6:09 PM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> > I have one basic query related to UBIFS volumes on a system with NAND partition.
>
> There is no such thing as a UBIFS volume. Do you mean UBI volumes?
>
Yes I mean the ubi volumes which are created using the ubinize config file.

> > In short, how to detect a particular empty volume inside a system
> > partition while flashing the partition image?
>
> What do you mean by system partition? A MTD partition?
By this I mean the entire UBI partition that contains the magic header "UBI#"
>
> > Suppose I have one big system partition that consists of 4-5 ubi
> > volumes inside it with varying sizes.
> > Lets say:
> > -- System Partition (ubi image)
> >     - rootfs volume (ro, squashfs)
> >     - data volume (rw, ubifs)
> >     - firmware volume (ro, ubifs)
> >     - some-other volume (ro, squashfs)
>
> So by system partition you mean the MTD partition that hosts UBI itself?
>
Yes the entire UBI partition that contains UBI volumes

> > Consider that all these could be flashed together as part of
> > system.ubi image in a single shot from the bootloader.
> > Now, suppose, one of the volume image (say firmware) is missing or
> > remains empty (as you know we can have empty volumes).
> >
> > So, during system image flashing, we wanted to detect if one of the
> > volume (firmware) is empty.
> > Since this is an important volume, so we wanted to detect if this
> > volume is empty/missing we will abort flashing the system partition.
> > As there is no point in booting the system without this partition.
> >
> > So, I am exploring options, how can this be detected ?
>
> Read from the volume after flashing. If you get only 0xFF bytes it is empty.
>
I mean, without flashing the image, is it possible to determine/guess
from the image content/header
 that a volume in the image is empty.
If it is empty then do not allow to flash the image.

> > I mean is there any kind of magic number or header information which
> > we can read to detect a particular empty volume ?
> > Can we get any information from just "system.ubi" image to indicate
> > about the volume information ?
>
> You'll need to perform a proper UBI scan on all blocks.
> If for the sad volume no LEBs have been found it is empty.
>
hmm, this seems reasonable. We have something called
"ubi_scan_partition" that scans for bad blocks, during "ubi_open".
And we also have ubi_find_volume that checks for vtbl record in an
incoming image.
Let me check about the LEB option here.

> > Also it could be possible that 2 or more volumes are empty, but we are
> > only concerned about one particular volume (firmware), so how to
> > detect particular volume ?
>
> I don't understand the use case. Is your image creation process so error prone
> that you can't be sure whether critical parts got included or not?
>
Actually our UBI creation process contains multiple volumes, in which
one volume contains
firmware image, that comes from another subsystem.
So, it requires copying that firmware image manually before building
the etire yocto image.
But, it might be possible that some developer may miss/skip this
firmware copying part, thus
the resulting image may end up is having an empty firmware volume.
However, this firmware volume is an important volume, without which
the modem may not boot up.

Re: Query: UBIFS: How to detect empty volumes

[Pintu Agarwal]

On Sun, 27 Jun 2021 at 14:12, Richard Weinberger
<richard.weinberger@gmail.com> wrote:
>
> On Thu, Jun 24, 2021 at 6:09 PM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> > I have one basic query related to UBIFS volumes on a system with NAND partition.
>
> There is no such thing as a UBIFS volume. Do you mean UBI volumes?
>
Yes I mean the ubi volumes which are created using the ubinize config file.

> > In short, how to detect a particular empty volume inside a system
> > partition while flashing the partition image?
>
> What do you mean by system partition? A MTD partition?
By this I mean the entire UBI partition that contains the magic header "UBI#"
>
> > Suppose I have one big system partition that consists of 4-5 ubi
> > volumes inside it with varying sizes.
> > Lets say:
> > -- System Partition (ubi image)
> >     - rootfs volume (ro, squashfs)
> >     - data volume (rw, ubifs)
> >     - firmware volume (ro, ubifs)
> >     - some-other volume (ro, squashfs)
>
> So by system partition you mean the MTD partition that hosts UBI itself?
>
Yes the entire UBI partition that contains UBI volumes

> > Consider that all these could be flashed together as part of
> > system.ubi image in a single shot from the bootloader.
> > Now, suppose, one of the volume image (say firmware) is missing or
> > remains empty (as you know we can have empty volumes).
> >
> > So, during system image flashing, we wanted to detect if one of the
> > volume (firmware) is empty.
> > Since this is an important volume, so we wanted to detect if this
> > volume is empty/missing we will abort flashing the system partition.
> > As there is no point in booting the system without this partition.
> >
> > So, I am exploring options, how can this be detected ?
>
> Read from the volume after flashing. If you get only 0xFF bytes it is empty.
>
I mean, without flashing the image, is it possible to determine/guess
from the image content/header
 that a volume in the image is empty.
If it is empty then do not allow to flash the image.

> > I mean is there any kind of magic number or header information which
> > we can read to detect a particular empty volume ?
> > Can we get any information from just "system.ubi" image to indicate
> > about the volume information ?
>
> You'll need to perform a proper UBI scan on all blocks.
> If for the sad volume no LEBs have been found it is empty.
>
hmm, this seems reasonable. We have something called
"ubi_scan_partition" that scans for bad blocks, during "ubi_open".
And we also have ubi_find_volume that checks for vtbl record in an
incoming image.
Let me check about the LEB option here.

> > Also it could be possible that 2 or more volumes are empty, but we are
> > only concerned about one particular volume (firmware), so how to
> > detect particular volume ?
>
> I don't understand the use case. Is your image creation process so error prone
> that you can't be sure whether critical parts got included or not?
>
Actually our UBI creation process contains multiple volumes, in which
one volume contains
firmware image, that comes from another subsystem.
So, it requires copying that firmware image manually before building
the etire yocto image.
But, it might be possible that some developer may miss/skip this
firmware copying part, thus
the resulting image may end up is having an empty firmware volume.
However, this firmware volume is an important volume, without which
the modem may not boot up.

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Query: UBIFS: How to detect empty volumes

[Pintu Agarwal]

On Sun, 27 Jun 2021 at 14:12, Richard Weinberger
<richard.weinberger@gmail.com> wrote:
>
> On Thu, Jun 24, 2021 at 6:09 PM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> > I have one basic query related to UBIFS volumes on a system with NAND partition.
>
> There is no such thing as a UBIFS volume. Do you mean UBI volumes?
>
Yes I mean the ubi volumes which are created using the ubinize config file.

> > In short, how to detect a particular empty volume inside a system
> > partition while flashing the partition image?
>
> What do you mean by system partition? A MTD partition?
By this I mean the entire UBI partition that contains the magic header "UBI#"
>
> > Suppose I have one big system partition that consists of 4-5 ubi
> > volumes inside it with varying sizes.
> > Lets say:
> > -- System Partition (ubi image)
> >     - rootfs volume (ro, squashfs)
> >     - data volume (rw, ubifs)
> >     - firmware volume (ro, ubifs)
> >     - some-other volume (ro, squashfs)
>
> So by system partition you mean the MTD partition that hosts UBI itself?
>
Yes the entire UBI partition that contains UBI volumes

> > Consider that all these could be flashed together as part of
> > system.ubi image in a single shot from the bootloader.
> > Now, suppose, one of the volume image (say firmware) is missing or
> > remains empty (as you know we can have empty volumes).
> >
> > So, during system image flashing, we wanted to detect if one of the
> > volume (firmware) is empty.
> > Since this is an important volume, so we wanted to detect if this
> > volume is empty/missing we will abort flashing the system partition.
> > As there is no point in booting the system without this partition.
> >
> > So, I am exploring options, how can this be detected ?
>
> Read from the volume after flashing. If you get only 0xFF bytes it is empty.
>
I mean, without flashing the image, is it possible to determine/guess
from the image content/header
 that a volume in the image is empty.
If it is empty then do not allow to flash the image.

> > I mean is there any kind of magic number or header information which
> > we can read to detect a particular empty volume ?
> > Can we get any information from just "system.ubi" image to indicate
> > about the volume information ?
>
> You'll need to perform a proper UBI scan on all blocks.
> If for the sad volume no LEBs have been found it is empty.
>
hmm, this seems reasonable. We have something called
"ubi_scan_partition" that scans for bad blocks, during "ubi_open".
And we also have ubi_find_volume that checks for vtbl record in an
incoming image.
Let me check about the LEB option here.

> > Also it could be possible that 2 or more volumes are empty, but we are
> > only concerned about one particular volume (firmware), so how to
> > detect particular volume ?
>
> I don't understand the use case. Is your image creation process so error prone
> that you can't be sure whether critical parts got included or not?
>
Actually our UBI creation process contains multiple volumes, in which
one volume contains
firmware image, that comes from another subsystem.
So, it requires copying that firmware image manually before building
the etire yocto image.
But, it might be possible that some developer may miss/skip this
firmware copying part, thus
the resulting image may end up is having an empty firmware volume.
However, this firmware volume is an important volume, without which
the modem may not boot up.

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies

Re: Query: UBIFS: How to detect empty volumes

[Steve deRosier]

On Sun, Jun 27, 2021 at 6:52 AM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> But, it might be possible that some developer may miss/skip this
> firmware copying part, thus

This is your root problem.

If your build system requires any "magic incantations" such that it is
even possible that a developer can miss a step and result in this
situation, you *must* solve this problem.  I don't know what you're
doing, but honestly, this shouldn't be much more than adding a line or
two into a Makefile or build.sh script.  Fix this first.

I'm not saying it's a bad thing for your on-target flashing system to
sanity check the images, but that will never fix a broken process. Fix
the broken process.

- Steve

Re: Query: UBIFS: How to detect empty volumes

[Steve deRosier]

On Sun, Jun 27, 2021 at 6:52 AM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> But, it might be possible that some developer may miss/skip this
> firmware copying part, thus

This is your root problem.

If your build system requires any "magic incantations" such that it is
even possible that a developer can miss a step and result in this
situation, you *must* solve this problem.  I don't know what you're
doing, but honestly, this shouldn't be much more than adding a line or
two into a Makefile or build.sh script.  Fix this first.

I'm not saying it's a bad thing for your on-target flashing system to
sanity check the images, but that will never fix a broken process. Fix
the broken process.

- Steve

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Query: UBIFS: How to detect empty volumes

[Steve deRosier]

On Sun, Jun 27, 2021 at 6:52 AM Pintu Agarwal <pintu.ping@gmail.com> wrote:
> But, it might be possible that some developer may miss/skip this
> firmware copying part, thus

This is your root problem.

If your build system requires any "magic incantations" such that it is
even possible that a developer can miss a step and result in this
situation, you *must* solve this problem.  I don't know what you're
doing, but honestly, this shouldn't be much more than adding a line or
two into a Makefile or build.sh script.  Fix this first.

I'm not saying it's a bad thing for your on-target flashing system to
sanity check the images, but that will never fix a broken process. Fix
the broken process.

- Steve

_______________________________________________
Kernelnewbies mailing list
Kernelnewbies@kernelnewbies.org
https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies