linux-mtd.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* Re: lost busybox mysteriously
       [not found] <CAA=hcWRQmXa4_hh3KzKmqeaEkGsbdTEimsPjh=tZWribMkM0dA@mail.gmail.com>
@ 2020-01-26 23:35 ` JH
       [not found]   ` <emb3e01e23-7511-48c8-9229-85a767ab359c@andys-imac.leeshillfarm.local>
  0 siblings, 1 reply; 9+ messages in thread
From: JH @ 2020-01-26 23:35 UTC (permalink / raw)
  To: Yocto discussion list; +Cc: connman, linux-mtd

Hi,

I have further investigated the problem, here is more information:

That the same problem of missing busybox was not just occurred during
the device running in the middle of operation, it was also occurred
during booting image from NAND, I saw several times that the first and
second cycles of booting image from NAND were working well, then some
following booting process would be crashed by missing busybox, then
could not run whole shell commands. I have been pondering if it could
be caused by NAND issue or network virus / fishy? Appreciate any
clues.

Thank you.

Kind regards,

- jh

On 1/26/20, JH <jupiter.hce@gmail.com> wrote:
> Hi,
>
> I have been building and running Yocto image on iMX6 with NAND flash
> for a year, recently, I found that busybox was certainly missing
> mysteriously, I could not check ls and other shell commands. I could
> not figure out what caused that problem, it's like the flash is
> leaking. Appreciate tips how to debug it?
>
> Thank you.
>
> - jh
>

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [yocto] lost busybox mysteriously
       [not found]   ` <emb3e01e23-7511-48c8-9229-85a767ab359c@andys-imac.leeshillfarm.local>
@ 2020-01-27 11:13     ` JH
  2020-01-27 11:31       ` Jonatan Palsson
  2020-01-27 12:06       ` Quentin Schulz
  0 siblings, 2 replies; 9+ messages in thread
From: JH @ 2020-01-27 11:13 UTC (permalink / raw)
  To: Andy Pont; +Cc: connman, Yocto discussion list, linux-mtd

Hi Andy,

Thanks for the response.

On 1/27/20, Andy Pont <andy.pont@sdcsystems.com> wrote:
> JH wrote...
>
>>That the same problem of missing busybox was not just occurred during
>>the device running in the middle of operation, it was also occurred
>>during booting image from NAND, I saw several times that the first and
>>second cycles of booting image from NAND were working well, then some
>>following booting process would be crashed by missing busybox, then
>>could not run whole shell commands. I have been pondering if it could
>>be caused by NAND issue or network virus / fishy? Appreciate any
>>clues.
> The first step is for us to understand what “missing” means?  Have you
> got any mechanism (U-Boot, SD card boot, etc.) that will allow you to
> mount and look at the contents of the NAND file system?

Means that busybox was not there anymore, it mysteriously lost, all
shell commands would no longer available. It cannot to run mount or
any shell commands. There was two scenarios when that happened:

- In the middle of running, the device all of certain could not run
shell commands and failed mysteriously

- During the u-boot booting kernel process, there were full errors of
failing shell commands. Let me make it clear,  that booting error did
not occur in the first or second kernel booting after the new image
installation, it happened in the following kernel booting, but there
was nothing to delete busybox accidentally, busybox was just
mysteriously disappeared. Because I could not run ls, I did not know
if there are other things missing. If you ask how I could know the
busybox was missing, I ran the zImage-initramfs to boot the linux in
RAM, then mount the ubi0 to find  out busybox was gone.


> If you look at the /bin directory (ls -la /bin/busy*) what do you see?
> Have the files been deleted? Truncated? Zero length?

Could not run ls or any shell commands when the busybox was missing.

> What file system are you using on the NAND flash?  How are the devices
> being reset during the various boot cycles?  If it is a hardware reset
> then some file systems are less resilient to it than others but I would
> expect in that case more fundamental boot issues.

UBIFS, most device reset or boot cycles were calling halt or reboot,
but it sometime it could just use power cycle.

Thank you.

Kind regards,

- jh

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [yocto] lost busybox mysteriously
  2020-01-27 11:13     ` [yocto] " JH
@ 2020-01-27 11:31       ` Jonatan Palsson
  2020-01-28  5:58         ` JH
  2020-01-27 12:06       ` Quentin Schulz
  1 sibling, 1 reply; 9+ messages in thread
From: Jonatan Palsson @ 2020-01-27 11:31 UTC (permalink / raw)
  To: JH; +Cc: connman, Yocto discussion list, linux-mtd, Andy Pont

Hello JH,

On Mon, Jan 27, 2020 at 12:13 PM JH <jupiter.hce@gmail.com> wrote:
>
> > If you look at the /bin directory (ls -la /bin/busy*) what do you see?
> > Have the files been deleted? Truncated? Zero length?
>
> Could not run ls or any shell commands when the busybox was missing.

Do you have the "ubifsls" command available in your u-boot? This
should allow you to look around in the file system from u-boot. See
here: https://gitlab.denx.de/u-boot/u-boot/blob/master/doc/README.ubi#L177

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [yocto] lost busybox mysteriously
  2020-01-27 11:13     ` [yocto] " JH
  2020-01-27 11:31       ` Jonatan Palsson
@ 2020-01-27 12:06       ` Quentin Schulz
  2020-01-27 15:04         ` Laurent Gauthier
  2020-01-28  6:25         ` JH
  1 sibling, 2 replies; 9+ messages in thread
From: Quentin Schulz @ 2020-01-27 12:06 UTC (permalink / raw)
  To: JH; +Cc: connman, Yocto discussion list, linux-mtd, Andy Pont

Hi JH,

On Mon, Jan 27, 2020 at 10:13:37PM +1100, JH wrote:
> Hi Andy,
> 
> Thanks for the response.
> 
> On 1/27/20, Andy Pont <andy.pont@sdcsystems.com> wrote:
> > JH wrote...
> >
> >>That the same problem of missing busybox was not just occurred during
> >>the device running in the middle of operation, it was also occurred
> >>during booting image from NAND, I saw several times that the first and
> >>second cycles of booting image from NAND were working well, then some
> >>following booting process would be crashed by missing busybox, then
> >>could not run whole shell commands. I have been pondering if it could
> >>be caused by NAND issue or network virus / fishy? Appreciate any
> >>clues.
> > The first step is for us to understand what “missing” means?  Have you
> > got any mechanism (U-Boot, SD card boot, etc.) that will allow you to
> > mount and look at the contents of the NAND file system?
> 
> Means that busybox was not there anymore, it mysteriously lost, all
> shell commands would no longer available. It cannot to run mount or
> any shell commands. There was two scenarios when that happened:
> 
> - In the middle of running, the device all of certain could not run
> shell commands and failed mysteriously
> 
> - During the u-boot booting kernel process, there were full errors of
> failing shell commands. Let me make it clear,  that booting error did
> not occur in the first or second kernel booting after the new image
> installation, it happened in the following kernel booting, but there
> was nothing to delete busybox accidentally, busybox was just
> mysteriously disappeared. Because I could not run ls, I did not know
> if there are other things missing. If you ask how I could know the
> busybox was missing, I ran the zImage-initramfs to boot the linux in
> RAM, then mount the ubi0 to find  out busybox was gone.
> 
> 
> > If you look at the /bin directory (ls -la /bin/busy*) what do you see?
> > Have the files been deleted? Truncated? Zero length?
> 
> Could not run ls or any shell commands when the busybox was missing.
> 

/bin/ls -la /bin/busy* ?

Maybe something is messing with the PATH environment variable. Or
something is removing the symlinks from some binaries to busybox.

> > What file system are you using on the NAND flash?  How are the devices
> > being reset during the various boot cycles?  If it is a hardware reset
> > then some file systems are less resilient to it than others but I would
> > expect in that case more fundamental boot issues.
> 
> UBIFS, most device reset or boot cycles were calling halt or reboot,
> but it sometime it could just use power cycle.
> 

IIRC, UBIFS is safe from power cycles.

Quentin

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [yocto] lost busybox mysteriously
  2020-01-27 12:06       ` Quentin Schulz
@ 2020-01-27 15:04         ` Laurent Gauthier
  2020-01-28  5:55           ` JH
  2020-01-28  6:25         ` JH
  1 sibling, 1 reply; 9+ messages in thread
From: Laurent Gauthier @ 2020-01-27 15:04 UTC (permalink / raw)
  To: Quentin Schulz; +Cc: connman, Yocto discussion list, linux-mtd, Andy Pont, JH

Hi JH,

As you are not providing enough details in your request we have to
guess the nature of
the issue you are seeing.

I will just point out a few things that could help in your situation.

If this is a failure that happens almost every time after you flash a
new image it could be
due to NAND blocks being bad.

As you mentioned that you have been working with this system for a
year it could be that repeated
flashing of UBI images has ended up causing some NAND blocks to go bad.

When using UBI it is a good idea to use a flashing procedure that will
preserve UBI wear-levelling
information. An example of such a procedure can be found in section
4.4 of the following page:

http://variwiki.com/index.php?title=U-Boot_features#Flashing_UBIFS_to_NAND

These instructions (ubi part/ubi remove/ubi create/ubi write) need to
be adjusted for your specific
hardware layout, but this can help. In short you should update
individual UBIFS volumes in your UBI
partition rather then reflash the whole UBI partition.

Another option is to use ubiformat from the u-boot command line. For
this you would need to research
the subject by yourself.

Updating your UBIFS volumes this way you should be able to extend
greatly the life of your NAND if
you are flashing your development board multiple times a day.

I would like to also comment on a remark made by someone else about
UBIFS being safe across
power cycles:

While the assertion that "UBIFS is safe across power-cycles" is true
in theory in practice you probably
should avoid to rely on it too much.

My recommendations to improve resistance to power-cycles would be the following:

1. If possible mount your UBIFS root-filesystem as read-only, and will
avoid most issues. This means
   you should use tmpfs for temporary/volatile filesystems.
2. If your root-filesystem cannot be read-only then remount it as
read-only just before the final shutdown
   (using for example "mount -o remount,ro /" followed by a "sync") as
this will limit the possibility of a
   corruption of the UBIFS occurring on the next reboot.

I hope that you will find what your issue is.

Kind Regards, Laurent.

On Mon, Jan 27, 2020 at 1:53 PM Quentin Schulz
<quentin.schulz@streamunlimited.com> wrote:
>
> Hi JH,
>
> On Mon, Jan 27, 2020 at 10:13:37PM +1100, JH wrote:
> > Hi Andy,
> >
> > Thanks for the response.
> >
> > On 1/27/20, Andy Pont <andy.pont@sdcsystems.com> wrote:
> > > JH wrote...
> > >
> > >>That the same problem of missing busybox was not just occurred during
> > >>the device running in the middle of operation, it was also occurred
> > >>during booting image from NAND, I saw several times that the first and
> > >>second cycles of booting image from NAND were working well, then some
> > >>following booting process would be crashed by missing busybox, then
> > >>could not run whole shell commands. I have been pondering if it could
> > >>be caused by NAND issue or network virus / fishy? Appreciate any
> > >>clues.
> > > The first step is for us to understand what “missing” means?  Have you
> > > got any mechanism (U-Boot, SD card boot, etc.) that will allow you to
> > > mount and look at the contents of the NAND file system?
> >
> > Means that busybox was not there anymore, it mysteriously lost, all
> > shell commands would no longer available. It cannot to run mount or
> > any shell commands. There was two scenarios when that happened:
> >
> > - In the middle of running, the device all of certain could not run
> > shell commands and failed mysteriously
> >
> > - During the u-boot booting kernel process, there were full errors of
> > failing shell commands. Let me make it clear,  that booting error did
> > not occur in the first or second kernel booting after the new image
> > installation, it happened in the following kernel booting, but there
> > was nothing to delete busybox accidentally, busybox was just
> > mysteriously disappeared. Because I could not run ls, I did not know
> > if there are other things missing. If you ask how I could know the
> > busybox was missing, I ran the zImage-initramfs to boot the linux in
> > RAM, then mount the ubi0 to find  out busybox was gone.
> >
> >
> > > If you look at the /bin directory (ls -la /bin/busy*) what do you see?
> > > Have the files been deleted? Truncated? Zero length?
> >
> > Could not run ls or any shell commands when the busybox was missing.
> >
>
> /bin/ls -la /bin/busy* ?
>
> Maybe something is messing with the PATH environment variable. Or
> something is removing the symlinks from some binaries to busybox.
>
> > > What file system are you using on the NAND flash?  How are the devices
> > > being reset during the various boot cycles?  If it is a hardware reset
> > > then some file systems are less resilient to it than others but I would
> > > expect in that case more fundamental boot issues.
> >
> > UBIFS, most device reset or boot cycles were calling halt or reboot,
> > but it sometime it could just use power cycle.
> >
>
> IIRC, UBIFS is safe from power cycles.
>
> Quentin
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#48161): https://lists.yoctoproject.org/g/yocto/message/48161
> Mute This Topic: https://lists.yoctoproject.org/mt/70128245/3618354
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [laurent.gauthier@soccasys.com]
> -=-=-=-=-=-=-=-=-=-=-=-



-- 
Laurent Gauthier
Phone: +33 630 483 429
http://soccasys.com

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [yocto] lost busybox mysteriously
  2020-01-27 15:04         ` Laurent Gauthier
@ 2020-01-28  5:55           ` JH
  0 siblings, 0 replies; 9+ messages in thread
From: JH @ 2020-01-28  5:55 UTC (permalink / raw)
  To: Laurent Gauthier
  Cc: connman, Quentin Schulz, linux-mtd, Andy Pont, Yocto discussion list

Hi Laurent,

On 1/28/20, Laurent Gauthier <laurent.gauthier@soccasys.com> wrote:
> Hi JH,
>
> As you are not providing enough details in your request we have to
> guess the nature of
> the issue you are seeing.

Let me know what kind of details I haven't provided.

> I will just point out a few things that could help in your situation.
>
> If this is a failure that happens almost every time after you flash a
> new image it could be
> due to NAND blocks being bad.

There was one image, booted kernel fine for the first and second
cycle, then it got shell commands error due to lost busybox. I did see
NAND bad blocks list during booting, but that should be managed by
MTD, won't be a problem to case kernel crash or missing busybox,
right?


> As you mentioned that you have been working with this system for a
> year it could be that repeated
> flashing of UBI images has ended up causing some NAND blocks to go bad.

No, I mean I have been using Yocto to build image and to flash to the
NAND for a year, the devices I am currently using actually a new
revision hardware device, it was only used for the first time when the
new image was installed, the first and second booting was good, the
third got bad missing busybox and shell command, crashed kernel
booting.

> When using UBI it is a good idea to use a flashing procedure that will
> preserve UBI wear-levelling
> information. An example of such a procedure can be found in section
> 4.4 of the following page:
>
> http://variwiki.com/index.php?title=U-Boot_features#Flashing_UBIFS_to_NAND

Than you for the link, yes we have already been planing to work on a
new partition scheme to move MTD partitions to UBIFS volume
partitions. But I don't think the current problem is related to NAND
bad blocks.

> These instructions (ubi part/ubi remove/ubi create/ubi write) need to
> be adjusted for your specific
> hardware layout, but this can help. In short you should update
> individual UBIFS volumes in your UBI
> partition rather then reflash the whole UBI partition.
>
> Another option is to use ubiformat from the u-boot command line. For
> this you would need to research
> the subject by yourself.
>
> Updating your UBIFS volumes this way you should be able to extend
> greatly the life of your NAND if
> you are flashing your development board multiple times a day.
>
> I would like to also comment on a remark made by someone else about
> UBIFS being safe across
> power cycles:
>
> While the assertion that "UBIFS is safe across power-cycles" is true
> in theory in practice you probably
> should avoid to rely on it too much.

There is not practical we can avoid it, the device is an IoT device
installed and operated automatically, there is not practical we can
run command to turn it on and off.

> My recommendations to improve resistance to power-cycles would be the
> following:
>
> 1. If possible mount your UBIFS root-filesystem as read-only, and will
> avoid most issues. This means
>    you should use tmpfs for temporary/volatile filesystems.
> 2. If your root-filesystem cannot be read-only then remount it as
> read-only just before the final shutdown
>    (using for example "mount -o remount,ro /" followed by a "sync") as
> this will limit the possibility of a
>    corruption of the UBIFS occurring on the next reboot.
>
> I hope that you will find what your issue is.

> Kind Regards, Laurent.
>
> On Mon, Jan 27, 2020 at 1:53 PM Quentin Schulz
> <quentin.schulz@streamunlimited.com> wrote:
>>
>> Hi JH,
>>
>> On Mon, Jan 27, 2020 at 10:13:37PM +1100, JH wrote:
>> > Hi Andy,
>> >
>> > Thanks for the response.
>> >
>> > On 1/27/20, Andy Pont <andy.pont@sdcsystems.com> wrote:
>> > > JH wrote...
>> > >
>> > >>That the same problem of missing busybox was not just occurred during
>> > >>the device running in the middle of operation, it was also occurred
>> > >>during booting image from NAND, I saw several times that the first
>> > >> and
>> > >>second cycles of booting image from NAND were working well, then some
>> > >>following booting process would be crashed by missing busybox, then
>> > >>could not run whole shell commands. I have been pondering if it could
>> > >>be caused by NAND issue or network virus / fishy? Appreciate any
>> > >>clues.
>> > > The first step is for us to understand what “missing” means?  Have
>> > > you
>> > > got any mechanism (U-Boot, SD card boot, etc.) that will allow you to
>> > > mount and look at the contents of the NAND file system?
>> >
>> > Means that busybox was not there anymore, it mysteriously lost, all
>> > shell commands would no longer available. It cannot to run mount or
>> > any shell commands. There was two scenarios when that happened:
>> >
>> > - In the middle of running, the device all of certain could not run
>> > shell commands and failed mysteriously
>> >
>> > - During the u-boot booting kernel process, there were full errors of
>> > failing shell commands. Let me make it clear,  that booting error did
>> > not occur in the first or second kernel booting after the new image
>> > installation, it happened in the following kernel booting, but there
>> > was nothing to delete busybox accidentally, busybox was just
>> > mysteriously disappeared. Because I could not run ls, I did not know
>> > if there are other things missing. If you ask how I could know the
>> > busybox was missing, I ran the zImage-initramfs to boot the linux in
>> > RAM, then mount the ubi0 to find  out busybox was gone.
>> >
>> >
>> > > If you look at the /bin directory (ls -la /bin/busy*) what do you
>> > > see?
>> > > Have the files been deleted? Truncated? Zero length?
>> >
>> > Could not run ls or any shell commands when the busybox was missing.
>> >
>>
>> /bin/ls -la /bin/busy* ?
>>
>> Maybe something is messing with the PATH environment variable. Or
>> something is removing the symlinks from some binaries to busybox.
>>
>> > > What file system are you using on the NAND flash?  How are the
>> > > devices
>> > > being reset during the various boot cycles?  If it is a hardware
>> > > reset
>> > > then some file systems are less resilient to it than others but I
>> > > would
>> > > expect in that case more fundamental boot issues.
>> >
>> > UBIFS, most device reset or boot cycles were calling halt or reboot,
>> > but it sometime it could just use power cycle.
>> >
>>
>> IIRC, UBIFS is safe from power cycles.
>>
>> Quentin
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>>
>> View/Reply Online (#48161):
>> https://lists.yoctoproject.org/g/yocto/message/48161
>> Mute This Topic: https://lists.yoctoproject.org/mt/70128245/3618354
>> Group Owner: yocto+owner@lists.yoctoproject.org
>> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
>> [laurent.gauthier@soccasys.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>
>
>
> --
> Laurent Gauthier
> Phone: +33 630 483 429
> http://soccasys.com
>

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [yocto] lost busybox mysteriously
  2020-01-27 11:31       ` Jonatan Palsson
@ 2020-01-28  5:58         ` JH
  0 siblings, 0 replies; 9+ messages in thread
From: JH @ 2020-01-28  5:58 UTC (permalink / raw)
  To: Jonatan Palsson; +Cc: connman, Yocto discussion list, linux-mtd, Andy Pont

On 1/27/20, Jonatan Palsson <jonatan.p@gmail.com> wrote:
> Hello JH,
>
> On Mon, Jan 27, 2020 at 12:13 PM JH <jupiter.hce@gmail.com> wrote:
>>
>> > If you look at the /bin directory (ls -la /bin/busy*) what do you see?
>> > Have the files been deleted? Truncated? Zero length?
>>
>> Could not run ls or any shell commands when the busybox was missing.
>
> Do you have the "ubifsls" command available in your u-boot? This
> should allow you to look around in the file system from u-boot. See
> here: https://gitlab.denx.de/u-boot/u-boot/blob/master/doc/README.ubi#L177
>

Thanks Jonatan for the link.

Kind regards,

- jh

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [yocto] lost busybox mysteriously
  2020-01-27 12:06       ` Quentin Schulz
  2020-01-27 15:04         ` Laurent Gauthier
@ 2020-01-28  6:25         ` JH
       [not found]           ` <CAAx3WaA2uoCki+1AV9QFZybbszfS5wecWKf1PfDvgV5u9EVL=g@mail.gmail.com>
  1 sibling, 1 reply; 9+ messages in thread
From: JH @ 2020-01-28  6:25 UTC (permalink / raw)
  To: Quentin Schulz; +Cc: connman, Yocto discussion list, linux-mtd, Andy Pont

On 1/27/20, Quentin Schulz <quentin.schulz@streamunlimited.com> wrote:
> Hi JH,
>
> On Mon, Jan 27, 2020 at 10:13:37PM +1100, JH wrote:
>> Hi Andy,
>>
>> Thanks for the response.
>>
>> On 1/27/20, Andy Pont <andy.pont@sdcsystems.com> wrote:
>> > JH wrote...
>> >
>> >>That the same problem of missing busybox was not just occurred during
>> >>the device running in the middle of operation, it was also occurred
>> >>during booting image from NAND, I saw several times that the first and
>> >>second cycles of booting image from NAND were working well, then some
>> >>following booting process would be crashed by missing busybox, then
>> >>could not run whole shell commands. I have been pondering if it could
>> >>be caused by NAND issue or network virus / fishy? Appreciate any
>> >>clues.
>> > The first step is for us to understand what “missing” means?  Have you
>> > got any mechanism (U-Boot, SD card boot, etc.) that will allow you to
>> > mount and look at the contents of the NAND file system?
>>
>> Means that busybox was not there anymore, it mysteriously lost, all
>> shell commands would no longer available. It cannot to run mount or
>> any shell commands. There was two scenarios when that happened:
>>
>> - In the middle of running, the device all of certain could not run
>> shell commands and failed mysteriously
>>
>> - During the u-boot booting kernel process, there were full errors of
>> failing shell commands. Let me make it clear,  that booting error did
>> not occur in the first or second kernel booting after the new image
>> installation, it happened in the following kernel booting, but there
>> was nothing to delete busybox accidentally, busybox was just
>> mysteriously disappeared. Because I could not run ls, I did not know
>> if there are other things missing. If you ask how I could know the
>> busybox was missing, I ran the zImage-initramfs to boot the linux in
>> RAM, then mount the ubi0 to find  out busybox was gone.
>>
>>
>> > If you look at the /bin directory (ls -la /bin/busy*) what do you see?
>> > Have the files been deleted? Truncated? Zero length?
>>
>> Could not run ls or any shell commands when the busybox was missing.
>>
>
> /bin/ls -la /bin/busy* ?
>
> Maybe something is messing with the PATH environment variable. Or
> something is removing the symlinks from some binaries to busybox.

No, could not run /bin/ls as it was  linked to  /bin/busybox.nosuid,
the /bin/busybox.nosuid was damaged for some reason.

>> > What file system are you using on the NAND flash?  How are the devices
>> > being reset during the various boot cycles?  If it is a hardware reset
>> > then some file systems are less resilient to it than others but I would
>> > expect in that case more fundamental boot issues.
>>
>> UBIFS, most device reset or boot cycles were calling halt or reboot,
>> but it sometime it could just use power cycle.
>>
>
> IIRC, UBIFS is safe from power cycles.

Good to know. Thank you.

> Quentin
>

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [yocto] lost busybox mysteriously
       [not found]           ` <CAAx3WaA2uoCki+1AV9QFZybbszfS5wecWKf1PfDvgV5u9EVL=g@mail.gmail.com>
@ 2020-01-28 10:40             ` JH
  0 siblings, 0 replies; 9+ messages in thread
From: JH @ 2020-01-28 10:40 UTC (permalink / raw)
  To: Gabriele Zampieri
  Cc: connman, Quentin Schulz, linux-mtd, Andy Pont, Yocto discussion list

Hi Gabriele,

On 1/28/20, Gabriele Zampieri <gabbla.malist@gmail.com> wrote:
> HI JH,
>
> - Can you stop on uboot and try "nand bad"? This will list the bad blocks
> on your device.
=> nand bad

Device 0 bad blocks:


> - How do you flash your ubi partition(s)?

I boot zImage-initramfs kernel to Linux user space, then run:
ubidetach -p /dev/mtd5
ubiformat /dev/mtd5 -y
ubiattach -m 5
ubimkvol /dev/ubi0 -s 160MiB -N rootfs_data
mount -t ubifs ubi0:rootfs_data /mnt
cp -adr /yocto_rootfs/* /mnt

> - During boot, does ubifs layer complain about the partition is it trying
> to mount? It will in case of corrupted metadata and may result in a read
> only mount.

There is no partition involved during the boot, all partitions start
in user space after booting.

> - It's pretty weird that busybox has gone and the partition is intact.
> Didn't you messed with any script? Can you replicate on another board?

I don't think it was bad block problem, because it is a new version
hardware, which has some problem in power supply, could unstable or
insufficient power supply cause MTD / NAND crash?

Thank you Gabriele,

Kind regards,

- jh

>
> On Tue, Jan 28, 2020 at 7:25 AM JH <jupiter.hce@gmail.com> wrote:
>
>> On 1/27/20, Quentin Schulz <quentin.schulz@streamunlimited.com> wrote:
>> > Hi JH,
>> >
>> > On Mon, Jan 27, 2020 at 10:13:37PM +1100, JH wrote:
>> >> Hi Andy,
>> >>
>> >> Thanks for the response.
>> >>
>> >> On 1/27/20, Andy Pont <andy.pont@sdcsystems.com> wrote:
>> >> > JH wrote...
>> >> >
>> >> >>That the same problem of missing busybox was not just occurred
>> >> >> during
>> >> >>the device running in the middle of operation, it was also occurred
>> >> >>during booting image from NAND, I saw several times that the first
>> >> >> and
>> >> >>second cycles of booting image from NAND were working well, then
>> >> >> some
>> >> >>following booting process would be crashed by missing busybox, then
>> >> >>could not run whole shell commands. I have been pondering if it
>> >> >> could
>> >> >>be caused by NAND issue or network virus / fishy? Appreciate any
>> >> >>clues.
>> >> > The first step is for us to understand what “missing” means?  Have
>> >> > you
>> >> > got any mechanism (U-Boot, SD card boot, etc.) that will allow you
>> >> > to
>> >> > mount and look at the contents of the NAND file system?
>> >>
>> >> Means that busybox was not there anymore, it mysteriously lost, all
>> >> shell commands would no longer available. It cannot to run mount or
>> >> any shell commands. There was two scenarios when that happened:
>> >>
>> >> - In the middle of running, the device all of certain could not run
>> >> shell commands and failed mysteriously
>> >>
>> >> - During the u-boot booting kernel process, there were full errors of
>> >> failing shell commands. Let me make it clear,  that booting error did
>> >> not occur in the first or second kernel booting after the new image
>> >> installation, it happened in the following kernel booting, but there
>> >> was nothing to delete busybox accidentally, busybox was just
>> >> mysteriously disappeared. Because I could not run ls, I did not know
>> >> if there are other things missing. If you ask how I could know the
>> >> busybox was missing, I ran the zImage-initramfs to boot the linux in
>> >> RAM, then mount the ubi0 to find  out busybox was gone.
>> >>
>> >>
>> >> > If you look at the /bin directory (ls -la /bin/busy*) what do you
>> >> > see?
>> >> > Have the files been deleted? Truncated? Zero length?
>> >>
>> >> Could not run ls or any shell commands when the busybox was missing.
>> >>
>> >
>> > /bin/ls -la /bin/busy* ?
>> >
>> > Maybe something is messing with the PATH environment variable. Or
>> > something is removing the symlinks from some binaries to busybox.
>>
>> No, could not run /bin/ls as it was  linked to  /bin/busybox.nosuid,
>> the /bin/busybox.nosuid was damaged for some reason.
>>
>> >> > What file system are you using on the NAND flash?  How are the
>> >> > devices
>> >> > being reset during the various boot cycles?  If it is a hardware
>> >> > reset
>> >> > then some file systems are less resilient to it than others but I
>> would
>> >> > expect in that case more fundamental boot issues.
>> >>
>> >> UBIFS, most device reset or boot cycles were calling halt or reboot,
>> >> but it sometime it could just use power cycle.
>> >>
>> >
>> > IIRC, UBIFS is safe from power cycles.
>>
>> Good to know. Thank you.
>>
>> > Quentin
>> >
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>>
>> View/Reply Online (#48171):
>> https://lists.yoctoproject.org/g/yocto/message/48171
>> Mute This Topic: https://lists.yoctoproject.org/mt/70128245/3618237
>> Group Owner: yocto+owner@lists.yoctoproject.org
>> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub  [
>> gabbla.malist@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2020-01-28 10:40 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <CAA=hcWRQmXa4_hh3KzKmqeaEkGsbdTEimsPjh=tZWribMkM0dA@mail.gmail.com>
2020-01-26 23:35 ` lost busybox mysteriously JH
     [not found]   ` <emb3e01e23-7511-48c8-9229-85a767ab359c@andys-imac.leeshillfarm.local>
2020-01-27 11:13     ` [yocto] " JH
2020-01-27 11:31       ` Jonatan Palsson
2020-01-28  5:58         ` JH
2020-01-27 12:06       ` Quentin Schulz
2020-01-27 15:04         ` Laurent Gauthier
2020-01-28  5:55           ` JH
2020-01-28  6:25         ` JH
     [not found]           ` <CAAx3WaA2uoCki+1AV9QFZybbszfS5wecWKf1PfDvgV5u9EVL=g@mail.gmail.com>
2020-01-28 10:40             ` JH

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).