During the start-up of my Fedora powered laptop I noticed in the boot messages list that some service has failed to start. Looking more closely I noticed that docker-storage-setup.service was failing, now, let’s go down the rabbit hole.
First thing i checked were logs, of course, and journalctl -u docker-storage-setup said:
Jun 16 21:44:48 scotty.tomica.local docker-storage-setup[1096]: ERROR: Docker has been previously configured for use with devicemapper graph driver. Not creating a new thin pool as existing docker metadata will fail to work with it. Manual cleanup is required before this will succeed.
Jun 16 21:44:48 scotty.tomica.local docker-storage-setup[1096]: INFO: Docker state can be reset by stopping docker and by removing /var/lib/docker directory. This will destroy existing docker images and containers and all the docker metadata.
Jun 16 21:44:48 scotty.tomica.local systemd[1]: Failed to start Docker Storage Setup.
Jun 16 21:44:48 scotty.tomica.local systemd[1]: docker-storage-setup.service: Unit entered failed state.
Now, that doesn’t seem right, so what actually happens here? Docker tries to set up it’s storage system but it looks like it fails in the middle of it. How so?
So let’s do what it suggests, bare in mind that this docker setup was on my testing machine and not nowhere near production so I just deleted whole /var/lib/docker. After I deleted that directory and started docker.service again I was faced with a little bit different error:
Jun 12 21:21:28 scotty.tomica.local systemd[1]: Starting Docker Storage Setup...
Jun 12 21:21:29 scotty.tomica.local docker-storage-setup[1078]: Rounding up size to full physical extent 232.00 MiB
Jun 12 21:21:29 scotty.tomica.local docker-storage-setup[1078]: Volume group "vg" has insufficient free space (0 extents): 58 required.
And there it was. While setting up the docker environment it wanted to create LVM thin pool and use it to store containers and container images, problem was that I fully allocated my disk to “vg” volume group and there was no space left on device to create another volume group to use for this purpose, so it failed.
If this were some production system for Docker I would probably configure base os and partitioning otherwise, but since I don’t run Docker in production, nor I was needing some excellent performance I decided for another solution.
You see, there is this default configuration file which docker-storage-setup.service looks when it starts up and it’s located in /usr/lib/docker-storage-setup/docker-storage-setup and it’s default config says to use devicemapper as storage driver:
STORAGE_DRIVER=devicemapper
Since I don’t have enough free (unallocated) disk space, overlay option is not an option with SELinux enabled, I’ll just settle with docker mounting those “stores” as loopback devices and override this option. To “trick” docker to mount “stores” as loopback devices I’ve just put:
STORAGE_DRIVER=""
in my /etc/sysconfig/docker-storage-setup, this will override default options specified in it’s default config and start up docker at boot without any errors.