A common issue we see with self-hosted runners is that they can leave behind files that were created or modified by the action. This is because the action runs in a container and the container is using a
root user to do its work.
The GitHub documentation says to run the the runner service as
root as well, to have the most compatibility with most runners. This is not a good idea, as it can lead to security issues, so a lot of people run the runner service as a non-root user. This can lead to permission issues when the action touches files in the workspace directory that get’s mounted.
One common example is the super-linter action. Depending on the checks that run, it can touch files on disc that then get owned by the
Another example is running the entire job inside of a container, with using the keyword
container on the job level:
jobs: build: runs-on: self-hosted container: ubuntu:22.04 steps: - uses: actions/checkout@v2 - run: echo "Hello World"
The checkout action will create a
.git directory in the workspace directory with the repo contents, which will be owned by the
root user as the
ubuntu-22.04 container runs as
root. The job itself will complete just fine, but the next time you run another job for this repository on the same runner, the checkout action will try to cleanup the $GITHUB_WORKSPACE directory to get a clean starting point, and will fail with a permission error since the job is not running as root, but as the non-root user the runner service is executing under.
There are multiple ways to tackle this issue:
rootand run the job as
rootas well. This is recommended by GitHub to prevent this from happening, that is how the service has been designed. Most people I talk with do not agree since the user has to much permissions and it can lead to security issues.
ephemeral, where the runner is alive for a single job execution, and then gets completely deleted and cleaned up so there is no reuse.
I’ll go over the last 4 options in this post.
You could hunt for the jobs that cause this issue, and ‘ask’ the user to add a cleanup action in the job itself. There is for example a cleanup action available in the marketplace that runs in a container that uses root, and that cleans up the workspace directory. This is not a good solution, as it requires a lot of manual work, and you will have to keep track of the jobs that cause this issue. The following options are probably manageable in a better way.
You can configure the runner with specific customization commands that get executed before and after the job, as well as other options:
By setting 2 environment variables before the job starts, you can run specific scripts before and after the job starts. The completed job can then be a docker run command that mounts the $GITHUB_WORKSPACE and cleans everything up while running as
root. This is the easiest option in my opinion:
The best advice is to always start runners as
ephemeral, where the runner is alive for a single job execution, and then gets completely deleted and cleaned up so there is no reuse. You will need to provide a mechanism to cleanup the Virtual Machine or Container setup where you execute the runner on. The best method I’ve found is to use Actions Runner Controller where the runner is executing inside of a container that gets deleted after the job is done. This will prevent any data to linger around on the runner as well, and gets you a clean execution environment every time.
There might be some valid reasons to still have a persistent runner, but I would recommend to use ephemeral runners as much as possible.