Running LLMs on air-gapped machines with Docker

In my research, I often have to run machine learning experiments on very sensitive data. This data lives in a protected environment, and one of the features of this environment is that it is sealed off from the internet. This makes running experiments a bit challenging.

We don't want to install a bunch of dependencies in the shared environment, since this might break other people's experiments. On the other hand, it's basically impossible to create a meaningful virtual environment without internet access, unless you have a mirror of PyPI running in your offline environment. Instead, this is an excellent use-case for Docker containers!

FROM nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04
# (or whatever base image fits your use case)

WORKDIR /app

RUN apt-get update
RUN apt-get install -y python3 python3-pip

COPY requirements.txt .

RUN pip3 install -r requirements.txt

ENTRYPOINT bash

The Dockerfile above is an example of how you could create a CUDA-enabled environment that has all dependencies needed to run your code. You can mount directories outside the docker container as volumes when starting the container. This gives you controlled access to files on your file system and is very useful for accessing models and code, and for reading and writing data. More on that towards the end of the post.

To build the Docker image, use the following commands:

cd /path/to/project
docker build . -t my-docker-image -f path/to/Dockerfile

This will create a docker image called my-docker-image based on the definition in the Dockerfile and the build command will have access to everything in the current directory (hence the .). Crucially, for our purposes, there needs to be a file called requirements.txt in the root directory of the build context (which is why I suggest you cd to the root of your Python project).

Note that Docker will not follow symlinks! You will only be able to reference files in the current directory (if you keep . after build) or whichever directory you pick as the build context.

Now that the image has been built, we can save it to a gzipped file using the following command:

docker save my-docker-image | gzip > /path/to/my-docker-image.tar.gz

Saving and compressing the file will often take a while. Once it's done, the image is now ready for transport! By this time I have typically made sure that the /path/to/my-docker-image.tar.gz is on a USB flash drive which I can use to physically move the image to the offline and air-gapped server.

Before we can actually run the image, however, we need to ensure that the server has the software required to run the docker container. You need to have the NVIDIA Container Toolkit installed on the machine^[1]. The link contains installation instructions and as far as I know, there is no way to use CUDA from the container without this toolkit.

With that sorted out, next step is to load it onto this machine using the following command:

docker load -i /usb/path/to/my-docker-image.tar.gz

A nice bonus is that Docker decompresses the file on the fly when loading it. Now, the only thing left is to actually start a container:

docker run --gpus all \
 -n my-named-container \
 -v /path/to/code:/app \
 -v /path/to/data:/data \
 -v /path/to/model:/model \
 -it my-docker-image

In this example, I've used the -v flags to specify that I want volumes in the docker container (reachable inside the container as /app, /data, and /model) that map to locations in my filesystem. Deciding what to mount and where to mount it is up to you.

The command docker run creates a new container every time you run it. In the example above, we have included the -n option, which allows us to give a name^[2] to the container. If you wanted to start this container without creating a new one, you can use docker start my-named-container.

Since we provided the -i flag, you will now see a bash terminal. From here, you can check that the volumes were mounted as expected, and then you can run your workload. Now go grab a nice cup of coffee while you wait for your experiments to run! After all, machine learning has allowed us to relive the good old days of the mainframe.

Tips & Tricks

Remember, docker run creates a container from an image, whereas docker start starts an existing container based on a container's name. Repeatedly running docker run will result in more and more containers being stored to disk. These are not automatically deleted! To remove our example container, you would run docker rm my-named-container.

For our purposes, there are two primary ways to disconnect from a container. The most straightforward way is to shut down the session. If, as in our example, our container starts in a bash shell, then you could close the session using CTRL+D or by typing exit. Doing so will stop the container. If you want to disconnect without stopping the container, you can press CTRL+P and then CTRL+Q.

You can also connect to a running container. To open a new terminal in our running container, you can run docker exec -it my-named-container bash. This can be useful if you want to debug the environment without disrupting whatever is running in the main terminal session. To reconnect to a session, you can run docker attach my-named-container.

Finally, there are some useful commands you can use to monitor the docker daemon and the containers being run. docker ps will show running containers (adding -a will show closed ones, too). You can also, instead of attaching to the container, look at its logs using docker logs my-named-container. Add -f to follow the logs and get new logs as they are produced.

2024-02-14: Updated the base image in the Dockerfile since the previous one had problems with public keys. Also changed the setup to start into bash and mount the model directory when starting the docker container.

2026-03-20: Major revision of the post, based on feedback from Martin Hansson, adding explanations, many more examples, and some useful commands.

Written on March 20, 2026