ext2 not NULLing deleted files?

ext2 not NULLing deleted files?

[Enver Haase]

Hi there,

I just recognized there's an "undelete" now for ext2 file systems [a KDE 
app]\b.

"The Other OS" in its professional version does of course clear the deleted 
blocks with 0's for security reasons; I would have bet a thousand bucks Linux 
would do so, too [seems I should have read the source code, good thing no-one 
wanted to take on the bet :) ].

So how to go about this? With that feature wanted, which fs should one choose 
under Linux? Is there a patch for ext2 for that feature? Am I the only one 
liking the idea?

Greetings,
Enver

Re: ext2 not NULLing deleted files?

[Robert Love]

On 17 Aug 2001 09:38:10 +0200, Enver Haase wrote:
> I just recognized there's an "undelete" now for ext2 file systems [a KDE 
> app].
> 
> "The Other OS" in its professional version does of course clear the deleted 
> blocks with 0's for security reasons; I would have bet a thousand bucks Linux 
> would do so, too [seems I should have read the source code, good thing no-one 
> wanted to take on the bet :) ].

By "The Other OS" I assume you mean NT.  NT does _not_ zero files on
delete, either with NTFS or anything else.  It merely unlinks them like
any other OS.  I can't think of anything that nullifies files, except
utilities meant solely to do that (often called "sweeping").

Do you have any idea how long it would take to zero files?  If you
removed even a moderately sized directory, it would take a _very long_
time.

-- 
Robert M. Love
rml at ufl.edu
rml at tech9.net

Re: ext2 not NULLing deleted files?

[Thomas Pornin]

In article <01081709381000.08800@haneman> you write:
> "The Other OS" in its professional version does of course clear the
> deleted blocks with 0's for security reasons

That's not much more than a smoke screen, actually. Wiping out data
from a disk requires more than merely writing 0's, if the bad guy is
assumed to physically have hold of the disk. See, for instance, the
paper from Peter Gutmann, "Secure Deletion of Data from Magnetic and
Solid-State Memory" (published in Usenix'96). Such games are much
dependant on the actual writing technology, and harddisk cache (which
cannot be deacticated on some modern IDE disks) gets in the way.
Besides, performance is terrible (if you need to rewrite seven times a
file with different patterns for each deletion, imagine what happens
when you delete a 100 MBytes file...).

The only truly secure way to do is encryption: if all data that gets to
the disk is encrypted, physical details about the disk are unimportant,
so security cannot be compromised by some smarter physicist with more
expensive tools.


	--Thomas Pornin

Re: ext2 not NULLing deleted files?

[Andreas Dilger]

On Aug 17, 2001  09:38 +0200, Enver Haase wrote:
> "The Other OS" in its professional version does of course clear the deleted 
> blocks with 0's for security reasons; I would have bet a thousand bucks Linux 
> would do so, too [seems I should have read the source code, good thing no-one 
> wanted to take on the bet :) ].
> 
> So how to go about this? With that feature wanted, which fs should one choose 
> under Linux? Is there a patch for ext2 for that feature? Am I the only one 
> liking the idea?

While there is an ext2 file attribute which sets "secure deletion" on a
per-file basis, it has never been implemented in the kernel.  Several
reasons for this:

1) Deleting a file really securely takes more than just a single write
   of zeros to the disk.
2) It would be a huge performance hit to overwrite a file the 15? or so
   times (some random, some patterned data) to really securely delete a
   file.
3) This is easily implemented in user-space, either by aliasing "rm" to
   a new function, or actually putting in your own "rm" binary which
   checks for the "S" attribute on ext2 files, and overwrites properly
   it if it a file only has a single link.  Then people can implement a
   level of security they are comfortable with for their particular needs.
4) Anything that really needs to be secure should not be stored in an
   insecure manner to begin with.  It should only be written to disk
   in encrypted form (see (a) and (b) above for why), and you also need
   something like tmpfs + encrypted swap so that you don't get unencrypted
   copies written to disk by mistake. Reasons for this are manyfold.
   With enough money and technology it is nearly impossible to really
   "delete" anything that was written to disk.  If it gets written on
   another part of the disk, you also have to scrub that (think /tmp or
   swap for editing documents).  If you make any backups of the disk,
   you need to scrub the tapes for every deletion (while keeping copies
   of all your other documents), very hard.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: ext2 not NULLing deleted files?

[Marc SCHAEFER]

In article <01081709381000.08800@haneman> you wrote:

> I just recognized there's an "undelete" now for ext2 file systems [a KDE 

not new.

> "The Other OS" in its professional version does of course clear the deleted 

(assuming NT)

No it doesn't, and there even has been cases in the past where its
journaling filesystem was, under some conditions, extending files
with old data blocks without deleting them (a bit like the OLE `let's
put anything which is in RAM in this MS-Word file'), allowing other
users to snoop on each other's data / deleted data [no references
sorry, from memory].

Special care, as far as I understand it, must be taken when allocating
fs data blocks. The following sequence must be followed:

   1. reserve them
   2. clear them
   3. mark them as allocated.

if 2 is too expensive, maybe it's sufficient to mark them as dirty
and zero them in memory. But what happens if the system crashes, with
the metadata to the disk (block allocated), but the data block not
yet filled/zeroed ?

Maybe some flags somewhere telling that those data blocks are allocated
but not yet committed ?

Re: ext2 not NULLing deleted files?

[Andreas Dilger]

On Aug 17, 2001  18:32 +0200, Marc SCHAEFER wrote:
> Special care, as far as I understand it, must be taken when allocating
> fs data blocks. The following sequence must be followed:
> 
>    1. reserve them
>    2. clear them
>    3. mark them as allocated.
> 
> if 2 is too expensive, maybe it's sufficient to mark them as dirty
> and zero them in memory. But what happens if the system crashes, with
> the metadata to the disk (block allocated), but the data block not
> yet filled/zeroed ?

Ext2 and ext3 both do this already (with caveats).  Since ext2 doesn't
impose write ordering constraints, there is not a hard guarantee that
the data block makes it to disk before the metadata is updated.  If
you run ext3 in data=ordered or data=journal mode, then you do have
such a guarantee. 

If you run in data=writeback mode, you basically have the same
situation as ext2 (data may be written before or after the metadata).
This is the same as the _current_ reiserfs code, but there are
apparently patches available which allow data=ordered mode also.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: ext2 not NULLing deleted files?

[Mark H. Wood]

On 17 Aug 2001, Robert Love wrote:
> On 17 Aug 2001 09:38:10 +0200, Enver Haase wrote:
> > I just recognized there's an "undelete" now for ext2 file systems [a KDE
> > app].
> >
> > "The Other OS" in its professional version does of course clear the deleted
> > blocks with 0's for security reasons; I would have bet a thousand bucks Linux
> > would do so, too [seems I should have read the source code, good thing no-one
> > wanted to take on the bet :) ].
>
> By "The Other OS" I assume you mean NT.  NT does _not_ zero files on
> delete, either with NTFS or anything else.  It merely unlinks them like
> any other OS.  I can't think of anything that nullifies files, except
> utilities meant solely to do that (often called "sweeping").

VMS.  "DELETE/ERASE FOO.BAR"  I don't recall whether it's done by the
filesystem code or by the DELETE command in userspace, but I'm guessing
it's built into FILES-11.  (Sorry, my Gray Wall is at home.)

> Do you have any idea how long it would take to zero files?  If you
> removed even a moderately sized directory, it would take a _very long_
> time.

That's why it's optional.

-- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
Make a good day.

Re: ext2 not NULLing deleted files?

[Mark H. Wood]

Regarding the need to do more than just zero unwanted data, I note that
there is a U.S. DOD MIL-SPEC (no, I do not know the number) which defines
a sequence of patterns to be used for erasing magnetic media.  VMS has a
hook on which one may hang one's own erasure pattern generator, and I
think DEC provided an unsupported implementation of the MIL-SPEC patterns
as an example of its use.  INITIALIZE /ERASE can use the patterns, but I
don't recall whether DELETE /ERASE does.  If you don't provide a
generator, I think erasure just uses zeros.

I recall hearing that highly-classified data must be destroyed by
physically shredding the medium.  Yes, throw your disk drive in the
shredder!  (Just imagine the class of machinery required to digest an RA81
HDA.)

Most of this goes way beyond the need to deter casual snooping.

-- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
Make a good day.

Re: ext2 not NULLing deleted files?

[Andreas Dilger]

On Aug 17, 2001  12:55 -0500, Mark H. Wood wrote:
> Regarding the need to do more than just zero unwanted data, I note that
> there is a U.S. DOD MIL-SPEC (no, I do not know the number) which defines
> a sequence of patterns to be used for erasing magnetic media.

In the Usenix paper quoted earlier in this thread (I believe) it was
stated that the MIL-SPEC document was actually bogus.  REAL secure
deletion requirements were much more strict (something like 15 passes of
various random and non-random patterns vs. 7 passes of alternating all 0
and all 1 data), but the US government made it think that the MIL-SPEC
requirements were enough, so that naive users would follow it, still
leaving enough trace data on the disk for the government to retrieve it.

Still, even a single pass of zero writes is enough to prevent 99.9%
of attackers from getting the data back.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: ext2 not NULLing deleted files?

[Andreas Bombe]

On Fri, Aug 17, 2001 at 02:02:41AM -0600, Andreas Dilger wrote:
> 3) This is easily implemented in user-space, either by aliasing "rm" to
>    a new function, or actually putting in your own "rm" binary which
>    checks for the "S" attribute on ext2 files, and overwrites properly
>    it if it a file only has a single link.  Then people can implement a
>    level of security they are comfortable with for their particular needs.

Bad, the file may still be open and in use.  So this rm would 1) destroy
work data and 2) leave the data in clear that are written after the
sweep.

-- 
Andreas E. Bombe <andreas.bombe@munich.netsurf.de>    DSA key 0x04880A44

Re: ext2 not NULLing deleted files?

[Jesse Pollard]

--
> 
> On Aug 17, 2001  12:55 -0500, Mark H. Wood wrote:
> > Regarding the need to do more than just zero unwanted data, I note that
> > there is a U.S. DOD MIL-SPEC (no, I do not know the number) which defines
> > a sequence of patterns to be used for erasing magnetic media.
> 
> In the Usenix paper quoted earlier in this thread (I believe) it was
> stated that the MIL-SPEC document was actually bogus.  REAL secure
> deletion requirements were much more strict (something like 15 passes of
> various random and non-random patterns vs. 7 passes of alternating all 0
> and all 1 data), but the US government made it think that the MIL-SPEC
> requirements were enough, so that naive users would follow it, still
> leaving enough trace data on the disk for the government to retrieve it.

Actually, it does exist as part of the rainbow series under object reuse.
I have a copy of the current renewed memo draft + addendum (this year) for
purging.

No change.

> Still, even a single pass of zero writes is enough to prevent 99.9%
> of attackers from getting the data back.

Absolutely - the only people that can still retrieve data are the data
recovery companies out there (even fire doesn't fully erase the data unless
above 2-3,000 degrees). Tunneling magnetic microscopes are amazing at
data retrieval. Polishing off the top layer even allows reading some data
recorded many times earlier, though the newer thin film surfaces make this
harder.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: ext2 not NULLing deleted files?

[Kent Borg]

Andi Kleen <freitag@alancoxonachip.com> writes:
>Just NULLing alone is quite useless anyways; just 0ed data can be
>easily recovered in a special laboratory by using old traces of
>magnetism on the surfaces.  If you care about real data deletion you
>should probably use an utility like wipe which does about 20-30
>passes with random data.

The services of such a laboratory are quite expensive, and invasive
(they need the disk).  An unerase utility is quite cheap (free) and
can be quietly run from the other side of the planet.

It seems to me there be room for something simple that raises the cost
of recovering deleted files to a price significantly above the current
sale price of "free".  Simple NULLing would do that and it could be
done cheaply by a low priority daemon that goes around sweeping up
deleted bits when nothing much else is happening.

Yes, there would still be a window when files will not have been
NULLed, and some machines are too busy to allow such a daemon to run
(are those machines also too busy to do encryption?), but it would be
much better than the case now where nearly all of us have tons of
deleted stuff just sitting there.  (Do you?)

In the physical world documents are sometimes shredded.  Yes, there is
a window between when a document is designated to be shredded and when
it can be shredded, and, yes, most shredders leave big enough pieces
to reassemble the original.  But shredders significantly lower one's
exposure and they significantly raise the cost of recovering that
data.  Just because they are not perfect doesn't mean they are "quite
useless".  The same is true of NULLing deleted files.

Also, I note that such a userland daemon is not a kernel issue.


-kb, the Kent who doesn't consider seatbelts "quite useless" just
because there are accidents for which they will not save his life.


P.S.  We still don't know what was in the 18-1/2 minute gap.  Maybe,
after all these years, we will soon find out, but most of the folks
involved are now dead.  (How many on this list were not yet born then?
Do they even know what I am talking about?  That's pretty good
security for daily use.)  Simple erasure is not perfect security, but
it is pretty damn good, and all it took was Rosemary Woods stretching
to reach that "record" button.

Re: ext2 not NULLing deleted files?

[Andi Kleen]

Enver Haase <ehaase@inf.fu-berlin.de> writes:

> Hi there,
> 
> I just recognized there's an "undelete" now for ext2 file systems [a KDE 
> app]\b.

There have been ext2 undeletes for many years now (and howtos how to do
it manually even longer), nothing new.

> 
> "The Other OS" in its professional version does of course clear the deleted 
> blocks with 0's for security reasons; I would have bet a thousand bucks Linux 
> would do so, too [seems I should have read the source code, good thing no-one 
> wanted to take on the bet :) ].
> 
> So how to go about this? With that feature wanted, which fs should one choose 
> under Linux? Is there a patch for ext2 for that feature? Am I the only one 
> liking the idea?

Old ext2 (before 2.0) supported this with a special attribute bit; but it was 
removed for good reasons.
Just NULLing alone is quite useless anyways; just 0ed data can be easily
recovered in a special laboratory by using old traces of magnetism on the
surfaces.
If you care about real data deletion you should probably use an utility
like wipe which does about 20-30 passes with random data. That is far too
complex to do in kernel space of course, but you can run it in user space
as needed. 0ing would just give you a false sense of security.

-Andi