Kiriansky et al. (2002) wrote an interesting paper on what they call “program sheperding”.  The basic idea is to control how the program counter changes and where it points to.  The PC should not point to data areas (this is somewhat similar in concept to non-executable stacks or memory pages).  The PC should enter library code through approved entry points only.  It would be capable in principle to enforce that the return target of a function should be the instruction located right after the call.

Their solution keeps track of “code origins”, which resembles a multi-level taint tracking.  The authors argue that this is better than execute flags on memory pages, because those could be “inadvertently or maliciously changed” (and they have three states instead of only two).  I thought those flags were managed by the kernel and could not be changed in user space?  If the kernel is compromised, then program sheperding will be compromised too.  The mechanism tracking code origins heavily uses write-protected memory pages, so the question that comes to mind is why couldn’t those also be “inadvertently or maliciously changed” if we have to worry about that for execute flags?  I must be missing something.

The potential versatility of this technology is impressive.  The authors test only one policy.  Policies have to be written, tested and approved;  it is not clear to me why that policy was chosen and the compromises it implies.

The crux of the whole system is code interpretation, which, despite the use of advanced optimizations, slows the execution.  It would be interesting to see how it would fare inside the framework of a virtual machine (e.g., VMWare).  Enterprises are already embracing VMWare and virtual machine solutions for its easier management of hardware, software, and disaster recovery.  With a price already paid for sandboxing, using this new sandboxing technology may not be so expensive after all.  Whereas it may not be as appealing as some solutions requiring hardware support, it may be easier to deploy.