GentleSecurity

[Andrew]

I was able to use the exploit against RPC DCOM to get a shell and then install a service on the machine. I know accessing the registry from a protected application(or something downloaded through it) should be impossible. Is this correct or can services be installed using valid CreateService() system calls?

GeSWall prevents installing services by registry modification and by system APIs (including CreateService) but only for isolated applications. In your case, "RPC DCOM" service is not isolated because it is hosted by svchost.exe, which is always trusted (means never isolated) by default settings (GeSWallConsole\Applications\system\svchost.exe). Though, you can change this, we do not recommend this because there are no rules for the process and machine may get to unbootable state. GeSWall for desktops is not supposed to isolated services. For that you should use GeSWall Server Edition, which targets services and other non-interactive applications. But even in this case we are not going to isolate certain system services, because it is quite tricky and could lead to many failures. That means GeSWall's security depends on the security of core system components: kernel, drivers and key services - TCB (Trusted Computer Base). If one of those components has a hole, then GeSWall, as well as other security products, is out of business, because at that point the whole system can be subverted.