Sounds relatively straightforward, right?

Well, it turned out to be quite a bit more complicated than expected. We have already tested several approaches, and our current solution is quite different from what we originally planned.

Our initial plan​

Our first approach was to use Tetragon, a security observability and runtime enforcement tool based on eBPF.

Getting Tetragon running inside our Docker environment was not the problem. The container started successfully, and our policies were accepted without any configuration errors.

The real problems started when we began testing the actual event collection.

Our policies did not produce the events we expected. In particular, filtering events to only include processes belonging to our sandbox turned out to be problematic. The container selection mechanisms we initially tried did not work properly with our Docker-based setup, because teragon was build for Kubernetes.

This meant that even though Tetragon was running, we could not reliably collect the information we needed.

So, instead of spending even more time trying to fix an approach that was not working as intended, we decided to move on to our second option.

Plan "B"​

Of course, we had a backup plan.

We removed the Tetragon container from our Docker Compose setup and started experimenting with Falco.

This time, things looked considerably better. We managed to configure container selection, and our custom rules started producing the events we were looking for.

We successfully tested several types of activity:

• Process activity: Detecting processes started inside the sandbox.
• File system activity: Observing file access, creation and modification.
• Network activity: Detecting outgoing network connections.

Falco also showed lower CPU usage in our tests compared to our previous Tetragon setup.

For the first time, we had a working proof of concept that could collect most of the information we needed.

However, there was still one major problem: our sandbox runtime.

gVisor​

NDM does not execute potentially untrusted packages in ordinary Docker containers. Instead, we use gVisor as an additional isolation layer.

But what exactly is gVisor?​

Unlike traditional containers, which primarily rely on Linux kernel isolation mechanisms such as namespaces and cgroups, gVisor introduces an additional user-space kernel called Sentry.

This means that system calls made by applications inside the sandbox are intercepted and handled by gVisor instead of being passed directly to the host kernel.

This is great from a security perspective, but it creates a problem for traditional system monitoring tools.

Falco normally observes system activity through events generated by the host kernel. With gVisor, many of the operations performed inside the sandbox are handled by Sentry and are therefore not visible to Falco in the same way as ordinary container activity.

As a result, Falco could not provide the complete view of the package's activity that we needed.

We could have reconsidered our sandbox architecture, but that would have meant compromising on an important part of NDM's security model.

Instead, we started looking for a solution that works directly with gVisor.

A new hope​

This is where SecCheck comes into play.

SecCheck is a monitoring mechanism built into gVisor itself. It allows gVisor to report security-relevant events from inside the sandbox through a dedicated event stream.

And that is exactly what we were looking for.

Instead of trying to observe sandbox activity from outside through the host kernel, we can use the monitoring capabilities that gVisor already provides.

This approach also means that we do not have to abandon our existing sandbox architecture or introduce another monitoring layer that cannot properly see what happens inside it.

We have already started implementing the communication between gVisor and our backend. The first tests confirmed that gVisor reaches our configured SecCheck socket.

There is still work to do before we can reliably collect, process and store all relevant events. In particular, we need to handle multiple concurrent sandboxes and make sure that events are correctly assigned to their respective analyses.

We are not quite there yet, but this time we seem to be heading in the right direction.

More details about the implementation will follow in our documentation once everything is working as intended.

Also worth mentioning:​

The server will be asynchronous. Currently, we are testing the limits of the server.

We also built a tool that stops building more Sandboxes as needed. For example, if two users start a request with the same package and version at the same time, only one sandbox will start. Only one sandbox will start for both users.

We also implemented a PostgreSQL database so that the sandbox doesn't run for every request. The response includes a value that indicates whether the results were cached or performed in a sandbox during your request.

This is the first post.

Read the documentation to understand what NDM is.