On online judging, part 3: the macOS sandbox

In a previous installment, we talked about creating a Linux sandbox using the ptrace(2) system call. Today, we'll look at setting up a similar sandbox for macOS.

It's a historical fact that the Darwin operating system that macOS relies on was originally forked from FreeBSD. This line of reasoning would lead us to assume that Darwin has a ptrace(2) system call just like FreeBSD, and we can therefore use it to sandbox code on macOS.

This is partially true — there exists ptrace(2) in Darwin, but unfortunately, we can't use it to sandbox on macOS.

At this point, most people say something among the lines of:

…but it has to work, otherwise gdb wouldn't!

Well, not really.

Let's start from the top: ptrace(2), on macOS, allows you to attach and detach from a process. Practically every other function from it is unimplemented.

The Mach kernel does roundabout ways for accomplishing most of what ptrace(2) can — people have butted heads with this in the past, and for your entertainment you can read more about their struggles here, here, and here.

At the end of the day, though, this functionality may be enough for gdb, but it's not enough for us: we need to be able to break on syscalls, and the Mach API has no support for this (nor does gdb require it to).

And so, just like Barbara's fairies, our ideal of a reusable ptrace(2) sandbox for macOS is also imaginary. Apple's App Sandbox is a promising start for online judging, albeit in an entirely different way than our Linux sandbox. However, I don't own a Mac, so our adventures with sandboxing on Mac must end here.