How to run a Docker in LXC container on ZFS on Proxmox

Cover Image for How to run a Docker in LXC container on ZFS on Proxmox

How to run a Docker in LXC container on ZFS on Proxmox

How-to written and screenshots taken on 2021 October 63 min read

Introduction

What started as a project to run "some crypto applications" grew into something much bigger, but also more useful. I learned a tremendous amount about virtual machines, lxc containers and Proxmox. And although I'm just scratching the surface and I don't actually understand it, I know how to do some things. I'll write them down here, partly for myself, but also for you in hopes it will save one of us a lot of time and frustration.

ZFS and docker

Currently it doesn't seem possible to get Docker to work on ZFS. Well you can with a lot of manual work, but my experience was really poor. I wanted to run docker inside an LXC container, which might sound weird, but alas, I wanted it. However, running docker in LXC with a ZFS mount was really slow. So I changed the setup a bit.

Create dedicated docker storage

  1. In Proxmox GUI go to "Datacenter" -> "Storage" and click "Add" and choose "Directory"

    Add directory storage

  2. Create the directory:

    • ID: docker-pve
    • Directory: /mnt/docker
    • Content: Container

    Create directory storage

Create an LXC container

  1. Go to "nfs-pool1" -> "CT Templates" and click "Templates". Download Ubuntu 20.

    Download container template

  2. Click "Create CT":

    • Hostname: docker-example
    • Check "Unprivileged container" and "Nesting"
    • Add a password

    Create a container

  3. Select a container template:

    • Storage: nfs-pool1
    • Template: ubuntu-20.04-standard_20.04-1_amd64.tar.gz

    Select a container template

  4. Select storage:

    • Storage: local-zfs-pve
    • Disk size (GiB): 8

    Select container storage

  5. Enter amount of cores:

    • Cores: 2

    Enter amount of cores for container

  6. Enter amount of memory:

    • Memory (MiB): 1024

    Enter amount of memory for container

  7. Configure network

    Configure container network connection

  8. Confirm next two screens.

  9. Click on the container and go to "Resources". Click "Add" -> "Mount point"

    • Storage: docker-pve
    • Disk size (GiB): 8
    • Path: /var/lib/docker
    • Uncheck "Backup"
    • Check "Skip replication"

    Add docker mount point

Install docker

  1. Start the container and open a shellsession.

    • You can open a shellsession directly to the IP or hostname of the container.
    • Or you can open a shellsession to the host and enter the container pct enter <id>
  2. Add supporting package

    apt update
    apt upgrade
    apt install apt-transport-https ca-certificates curl gnupg lsb-release
  3. Add Docker’s official GPG key:

    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
  4. Add Apt repository:

    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
  5. Install Docker:

    apt update
    apt install docker-ce docker-ce-cli containerd.io docker-compose
  6. Run Docker:

    docker run hello-world

    Docker running in LXC