safety of save_env

safety of save_env

[Michael Zimmermann]

[-- Attachment #1: Type: text/plain, Size: 148 bytes --]

Hi,

how safe is the usage of save_env? I'm considering writing like 100MB to a
loop file using the technique used by this command.

Thanks
Michael

[-- Attachment #2: Type: text/html, Size: 224 bytes --]

Re: safety of save_env

[Andrei Borzenkov]

23.08.2016 21:51, Michael Zimmermann пишет:
> Hi,
> 
> how safe is the usage of save_env? I'm considering writing like 100MB to a
> loop file using the technique used by this command.
> 

Define "safe". Command is filesystem agnostic and expects fully
preallocated file; it writes on block level. It also builds full content
of environment file in memory before writing it out; so it may fail on
systems with small amount of memory.

It attempts to verify that file can be represented by block lists and it
does not overwrite blocks outside of allocated for the file.

Re: safety of save_env

[Michael Zimmermann]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

I mean safe in the sense that it would not destroy data outside of that
file (no matter if writing was successful or not) or even destroy/add
errors to  the filesystem.(ext*, f2fs, fat*, ...).

On Tue, Aug 23, 2016 at 9:59 PM, Andrei Borzenkov <arvidjaar@gmail.com>
wrote:

> 23.08.2016 21:51, Michael Zimmermann пишет:
> > Hi,
> >
> > how safe is the usage of save_env? I'm considering writing like 100MB to
> a
> > loop file using the technique used by this command.
> >
>
> Define "safe". Command is filesystem agnostic and expects fully
> preallocated file; it writes on block level. It also builds full content
> of environment file in memory before writing it out; so it may fail on
> systems with small amount of memory.
>
> It attempts to verify that file can be represented by block lists and it
> does not overwrite blocks outside of allocated for the file.
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 1575 bytes --]

Re: safety of save_env

[Andrei Borzenkov]

24.08.2016 07:15, Michael Zimmermann пишет:
> I mean safe in the sense that it would not destroy data outside of that
> file (no matter if writing was successful or not) or even destroy/add
> errors to  the filesystem.(ext*, f2fs, fat*, ...).
> 

It computes block list of file data and then overwrites these blocks. It
will abort if file is sparse. It actually even double checks block list
by reading directly from disk (bypassing filesystem driver) and
comparing results.

So it looks reasonably safe in that it should not write outside of space
already allocated for file data on disk. It obviously will not work for
filesystem that computes checksums over data like btrfs or zfs; these
filesystems currently ignore read hooks so always return empty block
list (leading to confusing error message).

For journalled filesystems there is non-zero probability that journal
replay will change content of those blocks.

Note that it depends on explicit fs driver support; looking e.g. at f2fs
patch inline data bypasses read hook. Probably not an issue for 100M
file though.

> On Tue, Aug 23, 2016 at 9:59 PM, Andrei Borzenkov <arvidjaar@gmail.com>
> wrote:
> 
>> 23.08.2016 21:51, Michael Zimmermann пишет:
>>> Hi,
>>>
>>> how safe is the usage of save_env? I'm considering writing like 100MB to
>> a
>>> loop file using the technique used by this command.
>>>
>>
>> Define "safe". Command is filesystem agnostic and expects fully
>> preallocated file; it writes on block level. It also builds full content
>> of environment file in memory before writing it out; so it may fail on
>> systems with small amount of memory.
>>
>> It attempts to verify that file can be represented by block lists and it
>> does not overwrite blocks outside of allocated for the file.
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
> 
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 

Re: safety of save_env

[Andrei Borzenkov]

24.08.2016 10:17, Andrei Borzenkov пишет:
> 24.08.2016 07:15, Michael Zimmermann пишет:
>> I mean safe in the sense that it would not destroy data outside of that
>> file (no matter if writing was successful or not) or even destroy/add
>> errors to  the filesystem.(ext*, f2fs, fat*, ...).
>>
> 
> It computes block list of file data and then overwrites these blocks. It
> will abort if file is sparse. It actually even double checks block list
> by reading directly from disk (bypassing filesystem driver) and
> comparing results.
> 
> So it looks reasonably safe in that it should not write outside of space
> already allocated for file data on disk. It obviously will not work for
> filesystem that computes checksums over data like btrfs or zfs; these
> filesystems currently ignore read hooks so always return empty block
> list (leading to confusing error message).
> 
> For journalled filesystems there is non-zero probability that journal
> replay will change content of those blocks.
> 
> Note that it depends on explicit fs driver support; looking e.g. at f2fs
> patch inline data bypasses read hook. Probably not an issue for 100M
> file though.
> 

Speaking about f2fs - does it use checksums for data or metadata? In
particular, if supporting block lists for inline data metadata checksums
may be an issue then.

Also I understand that it is designed for media which may not support
direct sector size write, or am I wrong?

Hmm ... inode checksums do mean problem for ext4 with inline data (not
yet in GRUB upstream).

>> On Tue, Aug 23, 2016 at 9:59 PM, Andrei Borzenkov <arvidjaar@gmail.com>
>> wrote:
>>
>>> 23.08.2016 21:51, Michael Zimmermann пишет:
>>>> Hi,
>>>>
>>>> how safe is the usage of save_env? I'm considering writing like 100MB to
>>> a
>>>> loop file using the technique used by this command.
>>>>
>>>
>>> Define "safe". Command is filesystem agnostic and expects fully
>>> preallocated file; it writes on block level. It also builds full content
>>> of environment file in memory before writing it out; so it may fail on
>>> systems with small amount of memory.
>>>
>>> It attempts to verify that file can be represented by block lists and it
>>> does not overwrite blocks outside of allocated for the file.
>>>
>>> _______________________________________________
>>> Grub-devel mailing list
>>> Grub-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>
>>
>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
> 


------------------------------------------------------------------------------
_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: safety of save_env

[Andrei Borzenkov]

24.08.2016 10:17, Andrei Borzenkov пишет:
> 24.08.2016 07:15, Michael Zimmermann пишет:
>> I mean safe in the sense that it would not destroy data outside of that
>> file (no matter if writing was successful or not) or even destroy/add
>> errors to  the filesystem.(ext*, f2fs, fat*, ...).
>>
> 
> It computes block list of file data and then overwrites these blocks. It
> will abort if file is sparse. It actually even double checks block list
> by reading directly from disk (bypassing filesystem driver) and
> comparing results.
> 
> So it looks reasonably safe in that it should not write outside of space
> already allocated for file data on disk. It obviously will not work for
> filesystem that computes checksums over data like btrfs or zfs; these
> filesystems currently ignore read hooks so always return empty block
> list (leading to confusing error message).
> 
> For journalled filesystems there is non-zero probability that journal
> replay will change content of those blocks.
> 
> Note that it depends on explicit fs driver support; looking e.g. at f2fs
> patch inline data bypasses read hook. Probably not an issue for 100M
> file though.
> 

Speaking about f2fs - does it use checksums for data or metadata? In
particular, if supporting block lists for inline data metadata checksums
may be an issue then.

Also I understand that it is designed for media which may not support
direct sector size write, or am I wrong?

Hmm ... inode checksums do mean problem for ext4 with inline data (not
yet in GRUB upstream).

>> On Tue, Aug 23, 2016 at 9:59 PM, Andrei Borzenkov <arvidjaar@gmail.com>
>> wrote:
>>
>>> 23.08.2016 21:51, Michael Zimmermann пишет:
>>>> Hi,
>>>>
>>>> how safe is the usage of save_env? I'm considering writing like 100MB to
>>> a
>>>> loop file using the technique used by this command.
>>>>
>>>
>>> Define "safe". Command is filesystem agnostic and expects fully
>>> preallocated file; it writes on block level. It also builds full content
>>> of environment file in memory before writing it out; so it may fail on
>>> systems with small amount of memory.
>>>
>>> It attempts to verify that file can be represented by block lists and it
>>> does not overwrite blocks outside of allocated for the file.
>>>
>>> _______________________________________________
>>> Grub-devel mailing list
>>> Grub-devel@gnu.org
>>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>>
>>
>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> https://lists.gnu.org/mailman/listinfo/grub-devel
>>
> 

Re: safety of save_env

[Michael Zimmermann]

[-- Attachment #1.1: Type: text/plain, Size: 3787 bytes --]

> Note that it depends on explicit fs driver support; looking e.g. at f2fs
> patch inline data bypasses read hook. Probably not an issue for 100M
> file though.
that probably has to be fixed then.

> Speaking about f2fs - does it use checksums for data or metadata? In
> particular, if supporting block lists for inline data metadata checksums
> may be an issue then.
@Jaegeuk Kim

> Also I understand that it is designed for media which may not support
> direct sector size write, or am I wrong?
@Jaegeuk Kim

> Hmm ... inode checksums do mean problem for ext4 with inline data (not
> yet in GRUB upstream).
Can all the problems you've listed be detected? If so we can just deny
writing if a specific fs feature is enabled.


On Wed, Aug 24, 2016 at 10:10 AM, Andrei Borzenkov <arvidjaar@gmail.com>
wrote:

> 24.08.2016 10:17, Andrei Borzenkov пишет:
> > 24.08.2016 07:15, Michael Zimmermann пишет:
> >> I mean safe in the sense that it would not destroy data outside of that
> >> file (no matter if writing was successful or not) or even destroy/add
> >> errors to  the filesystem.(ext*, f2fs, fat*, ...).
> >>
> >
> > It computes block list of file data and then overwrites these blocks. It
> > will abort if file is sparse. It actually even double checks block list
> > by reading directly from disk (bypassing filesystem driver) and
> > comparing results.
> >
> > So it looks reasonably safe in that it should not write outside of space
> > already allocated for file data on disk. It obviously will not work for
> > filesystem that computes checksums over data like btrfs or zfs; these
> > filesystems currently ignore read hooks so always return empty block
> > list (leading to confusing error message).
> >
> > For journalled filesystems there is non-zero probability that journal
> > replay will change content of those blocks.
> >
> > Note that it depends on explicit fs driver support; looking e.g. at f2fs
> > patch inline data bypasses read hook. Probably not an issue for 100M
> > file though.
> >
>
> Speaking about f2fs - does it use checksums for data or metadata? In
> particular, if supporting block lists for inline data metadata checksums
> may be an issue then.
>
> Also I understand that it is designed for media which may not support
> direct sector size write, or am I wrong?
>
> Hmm ... inode checksums do mean problem for ext4 with inline data (not
> yet in GRUB upstream).
>
> >> On Tue, Aug 23, 2016 at 9:59 PM, Andrei Borzenkov <arvidjaar@gmail.com>
> >> wrote:
> >>
> >>> 23.08.2016 21:51, Michael Zimmermann пишет:
> >>>> Hi,
> >>>>
> >>>> how safe is the usage of save_env? I'm considering writing like 100MB
> to
> >>> a
> >>>> loop file using the technique used by this command.
> >>>>
> >>>
> >>> Define "safe". Command is filesystem agnostic and expects fully
> >>> preallocated file; it writes on block level. It also builds full
> content
> >>> of environment file in memory before writing it out; so it may fail on
> >>> systems with small amount of memory.
> >>>
> >>> It attempts to verify that file can be represented by block lists and
> it
> >>> does not overwrite blocks outside of allocated for the file.
> >>>
> >>> _______________________________________________
> >>> Grub-devel mailing list
> >>> Grub-devel@gnu.org
> >>> https://lists.gnu.org/mailman/listinfo/grub-devel
> >>>
> >>
> >>
> >>
> >> _______________________________________________
> >> Grub-devel mailing list
> >> Grub-devel@gnu.org
> >> https://lists.gnu.org/mailman/listinfo/grub-devel
> >>
> >
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #1.2: Type: text/html, Size: 6216 bytes --]

[-- Attachment #2: Type: text/plain, Size: 141 bytes --]

_______________________________________________
Grub-devel mailing list
Grub-devel@gnu.org
https://lists.gnu.org/mailman/listinfo/grub-devel

Re: safety of save_env

[Michael Zimmermann]

[-- Attachment #1: Type: text/plain, Size: 3787 bytes --]

> Note that it depends on explicit fs driver support; looking e.g. at f2fs
> patch inline data bypasses read hook. Probably not an issue for 100M
> file though.
that probably has to be fixed then.

> Speaking about f2fs - does it use checksums for data or metadata? In
> particular, if supporting block lists for inline data metadata checksums
> may be an issue then.
@Jaegeuk Kim

> Also I understand that it is designed for media which may not support
> direct sector size write, or am I wrong?
@Jaegeuk Kim

> Hmm ... inode checksums do mean problem for ext4 with inline data (not
> yet in GRUB upstream).
Can all the problems you've listed be detected? If so we can just deny
writing if a specific fs feature is enabled.


On Wed, Aug 24, 2016 at 10:10 AM, Andrei Borzenkov <arvidjaar@gmail.com>
wrote:

> 24.08.2016 10:17, Andrei Borzenkov пишет:
> > 24.08.2016 07:15, Michael Zimmermann пишет:
> >> I mean safe in the sense that it would not destroy data outside of that
> >> file (no matter if writing was successful or not) or even destroy/add
> >> errors to  the filesystem.(ext*, f2fs, fat*, ...).
> >>
> >
> > It computes block list of file data and then overwrites these blocks. It
> > will abort if file is sparse. It actually even double checks block list
> > by reading directly from disk (bypassing filesystem driver) and
> > comparing results.
> >
> > So it looks reasonably safe in that it should not write outside of space
> > already allocated for file data on disk. It obviously will not work for
> > filesystem that computes checksums over data like btrfs or zfs; these
> > filesystems currently ignore read hooks so always return empty block
> > list (leading to confusing error message).
> >
> > For journalled filesystems there is non-zero probability that journal
> > replay will change content of those blocks.
> >
> > Note that it depends on explicit fs driver support; looking e.g. at f2fs
> > patch inline data bypasses read hook. Probably not an issue for 100M
> > file though.
> >
>
> Speaking about f2fs - does it use checksums for data or metadata? In
> particular, if supporting block lists for inline data metadata checksums
> may be an issue then.
>
> Also I understand that it is designed for media which may not support
> direct sector size write, or am I wrong?
>
> Hmm ... inode checksums do mean problem for ext4 with inline data (not
> yet in GRUB upstream).
>
> >> On Tue, Aug 23, 2016 at 9:59 PM, Andrei Borzenkov <arvidjaar@gmail.com>
> >> wrote:
> >>
> >>> 23.08.2016 21:51, Michael Zimmermann пишет:
> >>>> Hi,
> >>>>
> >>>> how safe is the usage of save_env? I'm considering writing like 100MB
> to
> >>> a
> >>>> loop file using the technique used by this command.
> >>>>
> >>>
> >>> Define "safe". Command is filesystem agnostic and expects fully
> >>> preallocated file; it writes on block level. It also builds full
> content
> >>> of environment file in memory before writing it out; so it may fail on
> >>> systems with small amount of memory.
> >>>
> >>> It attempts to verify that file can be represented by block lists and
> it
> >>> does not overwrite blocks outside of allocated for the file.
> >>>
> >>> _______________________________________________
> >>> Grub-devel mailing list
> >>> Grub-devel@gnu.org
> >>> https://lists.gnu.org/mailman/listinfo/grub-devel
> >>>
> >>
> >>
> >>
> >> _______________________________________________
> >> Grub-devel mailing list
> >> Grub-devel@gnu.org
> >> https://lists.gnu.org/mailman/listinfo/grub-devel
> >>
> >
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 6216 bytes --]

Re: safety of save_env

[Vladimir 'phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

Why do you need to write 100 MiB? Is GRUB the right place for it? Please
describe your end goal.

On Tue, 23 Aug 2016, 21:53 Michael Zimmermann <sigmaepsilon92@gmail.com>
wrote:

> Hi,
>
> how safe is the usage of save_env? I'm considering writing like 100MB to a
> loop file using the technique used by this command.
>
> Thanks
> Michael
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 1199 bytes --]