Update, the Logger
One of the central parts of NDM is observing what an analyzed npm package actually does during execution. We want to collect information about its processes, file system activity, network connections and potentially sensitive data access.
Sounds relatively straightforward, right?
Well, it turned out to be quite a bit more complicated than expected. We have already tested several approaches, and our current solution is quite different from what we originally planned.
Our initial plan
Our first approach was to use Tetragon, a security observability and runtime enforcement tool based on eBPF.
Getting Tetragon running inside our Docker environment was not the problem. The container started successfully, and our policies were accepted without any configuration errors.
The real problems started when we began testing the actual event collection.
Our policies did not produce the events we expected. In particular, filtering events to only include processes belonging to our sandbox turned out to be problematic. The container selection mechanisms we initially tried did not work properly with our Docker-based setup, because teragon was build for Kubernetes.
This meant that even though Tetragon was running, we could not reliably collect the information we needed.
So, instead of spending even more time trying to fix an approach that was not working as intended, we decided to move on to our second option.
Plan "B"
Of course, we had a backup plan.
We removed the Tetragon container from our Docker Compose setup and started experimenting with Falco.
This time, things looked considerably better. We managed to configure container selection, and our custom rules started producing the events we were looking for.
We successfully tested several types of activity:
- Process activity: Detecting processes started inside the sandbox.
- File system activity: Observing file access, creation and modification.
- Network activity: Detecting outgoing network connections.
Falco also showed lower CPU usage in our tests compared to our previous Tetragon setup.
For the first time, we had a working proof of concept that could collect most of the information we needed.
However, there was still one major problem: our sandbox runtime.
gVisor
NDM does not execute potentially untrusted packages in ordinary Docker containers. Instead, we use gVisor as an additional isolation layer.
But what exactly is gVisor?
Unlike traditional containers, which primarily rely on Linux kernel isolation mechanisms such as namespaces and cgroups, gVisor introduces an additional user-space kernel called Sentry.
This means that system calls made by applications inside the sandbox are intercepted and handled by gVisor instead of being passed directly to the host kernel.
This is great from a security perspective, but it creates a problem for traditional system monitoring tools.
Falco normally observes system activity through events generated by the host kernel. With gVisor, many of the operations performed inside the sandbox are handled by Sentry and are therefore not visible to Falco in the same way as ordinary container activity.
As a result, Falco could not provide the complete view of the package's activity that we needed.
Our monitoring solution needs to observe what happens inside the sandbox, not just what happens on the host running it. Simply monitoring the host kernel is not sufficient for our setup.
We could have reconsidered our sandbox architecture, but that would have meant compromising on an important part of NDM's security model.
Instead, we started looking for a solution that works directly with gVisor.
A new hope
This is where SecCheck comes into play.
SecCheck is a monitoring mechanism built into gVisor itself. It allows gVisor to report security-relevant events from inside the sandbox through a dedicated event stream.
And that is exactly what we were looking for.
Instead of trying to observe sandbox activity from outside through the host kernel, we can use the monitoring capabilities that gVisor already provides.
This approach also means that we do not have to abandon our existing sandbox architecture or introduce another monitoring layer that cannot properly see what happens inside it.
- Native gVisor integration: Events originate directly from the component responsible for sandbox isolation.
- Better visibility: We can collect information about activity that traditional host-level monitoring might miss.
- Fits our architecture: The event stream can be received by a separate service and integrated into our existing Django backend.
We have already started implementing the communication between gVisor and our backend. The first tests confirmed that gVisor reaches our configured SecCheck socket.
There is still work to do before we can reliably collect, process and store all relevant events. In particular, we need to handle multiple concurrent sandboxes and make sure that events are correctly assigned to their respective analyses.
We are not quite there yet, but this time we seem to be heading in the right direction.
More details about the implementation will follow in our documentation once everything is working as intended.