Update SecCheck
SecCheck is working!
We have been investing a lot of time in the logger for our sandboxes.
Now, we can happily announce that our current development is working out!
We are now in the testing phase. We want to see what is working and where fine-tuning is needed.
This is essential for NDM because we want to guarantee a good and working product for our users.
Update SecCheck
After our previous attempts with Tetragon and Falco, we decided to investigate gVisor's built-in SecCheck functionality.
And it finally paid off!
We now have a working setup that allows us to collect events from inside our gVisor sandboxes and forward them to an external receiver.
But what exactly have we achieved so far?
What is working now?
The biggest milestone was getting the communication between gVisor and our external receiver working.
We configured gVisor to use its remote SecCheck sink, which sends trace events to an external monitoring process. The communication uses a Unix domain socket, and the events are transmitted using Protocol Buffers.
Our receiver can accept incoming connections and process the events it receives. This means that we are no longer just trying to observe what happens inside the sandbox from the outside. We are now receiving events directly from gVisor.
We have successfully received and processed several types of events:
- OPEN: Opening files.
- READ: Reading from files or file descriptors.
- WRITE: Writing to files or file descriptors.
- EXECVE: Executing programs, for example
/bin/shand Node.js. - CONNECT: Network connection events.
We can also associate collected events with their sandbox using the available container information.
Another important detail is that the events are not just printed to the console. We can store them in JSONL format, giving us a structured event log that we can process later.
During our tested runs, the receiver also reported dropped_count: 0.
The receiver did not report any dropped events during these test runs. This is a good result, although it does not guarantee that events can never be lost under different conditions or higher workloads.
Testing phase
Now that the basic communication and event collection are working, we can finally focus on testing the actual logger.
Until now, a lot of our work was about getting the infrastructure running. We had to figure out how gVisor exposes its events, how to receive them and how to process the incoming data.
Now we can concentrate on the more interesting part: Does our logger actually capture the activities we want to detect?
We are currently looking at several things:
- Event coverage: Are all relevant file, process and network events being captured?
- Correct attribution: Can we reliably associate every event with the correct sandbox?
- Event metadata: Do we receive enough information to understand what happened?
- Reliability: Are events lost during execution, especially when multiple sandboxes run simultaneously?
There is also an important distinction between observing an operation and observing its result. For example, we need to make sure that our logger can correctly interpret both successful and unsuccessful system calls.
We do not want to assume that everything works just because some events appear in our logs.
A logger that produces events is not automatically a good security monitoring solution.
We need to make sure that the events are sufficiently complete, correctly assigned and useful for the analysis we want to perform later.
What comes next?
There are still a few important steps before we can consider the logger finished.
First, we want to complete our current tests and improve the event handling wherever necessary.
After that, we will focus on integrating the logger properly into the NDM backend.
Our goal is to have the receiver work alongside our existing Django backend and support multiple analyses running at the same time. Each sandbox must have its own correctly associated event stream.
The collected events will then become the foundation for the next stage of NDM: analyzing the behavior of npm packages.
Instead of simply collecting events, we eventually want to use them to identify suspicious behavior and provide useful information about what a package actually does during execution.
We also plan to test the logger with real npm packages rather than just controlled test operations.
Our goal is not just to build another logging tool.
We want to understand what an npm package does while it is running, collect that information independently of the package itself and use it to make NDM's runtime analysis more meaningful.
Finally, some progress!
After quite a few attempts and a lot of debugging, we finally have a working foundation for our runtime logger.
There is still plenty of work ahead, especially when it comes to testing, reliability and backend integration. But we have reached an important milestone.
SecCheck is working, and we can finally move on to the next stage of development!