Moving docker to SD card on Firewalla Purple
I was running Portainer and Home Assistant in the default location /var/lib/docker. On the Purple, there's only 2 gigs on that partition, so I was running out of room.
Here's how I set up an SD card and configured docker to run from it. I've only been running this for a few hours, but the only difference I've noticed is that it takes a bit longer to extract images when I download them.
Prepare SD card with an ext4 filesystem
Basically follow the instructions from Firewalla support
sudo parted -s /dev/mmcblk1 mklabel gpt
sudo parted -s /dev/mmcblk1 mkpart primary ext4 2 100%
sudo parted -s /dev/mmcblk1 align-check optimal 1
sudo mkfs.ext4 /dev/mmcblk1p1Mount the SD card and create directory for docker
sudo mount /dev/mmcblk1p1 /mnt -o defaults
sudo mkdir /mnt/dockerDefault mount options match those of the other block devices of FWP (rw,relatime,data=ordered)
Stop docker, copy docker info to new location
sudo systemctl stop docker
sudo rsync -aP /var/lib/docker/ /mnt/dockerCreate executable script to mount SD card and configure docker to use it on reboot
mkdir /home/pi/.firewalla/config/post_main.d
vi /home/pi/.firewalla/config/post_main.d/sdcard_docker.shUse :set paste to paste the following into vi
sudo mount /dev/mmcblk1p1 /mnt -o defaults
sudo bash -c 'cat << UPDATECONFIG > /etc/docker/daemon.json
{
"data-root": "/mnt/docker"
}
UPDATECONFIG'
sudo systemctl restart docker*see the updated script in the comment below*
Set the script to executable
chmod 774 /home/pi/.firewalla/config/post_main.d/sdcard_docker.shReboot or just run the script to try the new config
Test any existing containers/volumes to make sure they work as expected
Especially check that
sudo docker info
shows 'Docker Root Dir: /mnt/docker', and that it persists through reboots.
Optionally clean up old docker
Remove the config pointing to the new location, so cleanup operations happen on the original docker root location.
sudo rm /etc/docker/daemon.jsonRestart docker for the old root to be used.
sudo systemctl restart dockerClear out that docker (/var/lib/docker) per Firewalla support instructions
Finally, reboot or run the script to point docker back to the SD card.
Note this doesn't remove previous docker volumes
-
Hi Michael,
I don't understand what you mean. Update my docker-compose.yml files to make those apps use the SD card? I wasn't aware that was an option.
I'm hardly a docker expert; everything I know, I've learned in the past few weeks. When I searched for information about using something other than /var/lib/docker, I found info about changing the service itself, or using the /etc/docker/daemon.json file.
Also, I may not always use docker-compose, and would like those components to use the external location as well.
-
If you look at this example,
https://github.com/mbierman/homebridge-installer/blob/main/docker-compose.yaml
The part under `volumes`. The left side would be out mount point /mnt/docker or whatever leaving the right side of the statement alone. This permanently tells this is your config and assuming the SD card is mounted, docker will use that file system.
if you are using command line and not yaml, then adjust the “-v” parameter in the same way. https://docs.portainer.io/v/ce-2.11/start/install/server/docker/linux
-
I _do_ like the idea of explicitly specifying where the volumes are.
My question is whether containers (and images) would go on the SD card with this method. The issue I was facing is that my partition was running out of space because of image sizes, though moving the user data volumes off the internal partition to the card would help the issue to some extent.
-
What I suggested puts the all of the data for the docker image where you specify. So it does address disk space to an extent. Using:
sudo docker infoYou can find where containers and images are stored. The default on Ubuntu is: /var/lib/docker
I have been able to install two large docker images on Purple by just setting the volumes as I described above so I have not needed to do it, but you can relocate the docker root:
https://www.ibm.com/docs/en/z-logdata-analytics/5.1.0?topic=compose-relocating-docker-root-directory
another good reference: https://www.freecodecamp.org/news/where-are-docker-images-stored-docker-container-paths-explained/
Also, be sure to read up and use docker prune appropriately.
-
I just thought about it and I suppose if you do it right, you could link /mnt/docker to /var/lib/docker be sure to move anything you have in /var/lib/docker first. Something like:
- stop all docker images and docker itself.
- cp everything in /var/lib/docker to /mnt/docker
- remove /var/lib/docker
- then create a link from /mnt/docker to /var/lib/docker
If you do this, I would put the volumes of your containers in like, /mnt/docker_images or something different than /mnt/docker. I think you want them separated.
I have not tried this so do it at your own risk. :)
-
That IBM article is what I ended up doing. The HomeAssisstant image alone is 1.2 gigs. Portainer isn't nearly as large, but it obviously takes up space, too. Since I hadn't relocated the volumes I had made (including a Webthings volume), that 2 gig was getting mighty full.
Everything I read about moving the docker stuff out of /var/lib/docker recommended against doing it via a link, though that would have been easier for me to grasp. 😏
I'll definitely keep those docker prune commands handy. I played with them while trying to minimize /var/lib/docker usage, and running them from time to time will help keep things heathy.
Thanks for your help and feedback. The Firewalla community is pretty awesome.
-
Hi David,
I put this together based on your excellent work. Thought I'd share.#!/bin/bash
if [ -z "$(mount | grep mnt )" ] ; then
sudo mount /dev/mmcblk1p1 /mnt
sudo chown 1000:1000 /mnt
else
echo "/mnt is already mounted..."
fi
if [ -z $(grep foo /etc/docker/daemon.json) ] ; then
echo -e '{\n"data-root": "/mnt/docker"\n}' > /etc/docker/daemon.json
sudo systemctl restart docker
else
echo "modified docker location already set"
fi -
What little bash knowledge I had, it was rusty, so thank you for that updated script.
One issue is that the pi user can't write to /etc/docker, so I had to update that line with
sudo bash -c "echo -e '{\n\"data-root\": \"/mnt/docker\"\n}' > /etc/docker/daemon.json"Now, to figure out how to write a script to show ports on the host which were opened via a container (in host networking mode, in my case).
-
That's the first thing I tried, and got permission denied. I'm guessing the sudo doesn't pass through to the output redirection.
I made a few tweaks to the conditionals of the script. Using findmnt to determine if the filesystem is mounted, and bash testing to see if the file exists.
#!/bin/bash
if [ -z "$(findmnt -n /mnt)" ] ; then
sudo mount /dev/mmcblk1p1 /mnt
sudo chown 1000:1000 /mnt
else
echo "/mnt is already mounted..."
fiif [[ ! -f /etc/docker/daemon.json ]] ; then
sudo bash -c "echo -e '{\n\"data-root\": \"/mnt/docker\"\n}' > /etc/docker/daemon.json"
sudo systemctl restart docker
else
echo "modified docker location already set"
fi -
Was looking at Michaels post about changing the location of the volume. I changed up my pihole yaml to this and wondering if this looks accurate? i didn't change 2nd and 3rd line under volume and thinking i could leave as is?
pi@firewalla:~/.firewalla/run/docker/pi-hole (Firewalla) $ cat docker-compose.yaml
version: "3"# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
pihole:
container_name: pihole
image: pihole/pihole:latest
environment:
# set a secure password here or the default will be firewalla
WEBPASSWORD: 'firewalla'
# Volumes store your data between container upgrades
volumes:
- '/mnt/docker_images/data/pi-hole/etc-pihole/:/etc/pihole/'
- './etc-dnsmasq.d/:/etc/dnsmasq.d/'
- '/etc/localtime:/etc/localtime:ro'
restart: unless-stopped
cap_add:
- NET_ADMIN
networks:
default:
# static IP address for pi-hole
ipv4_address: 172.16.0.2
networks:
default:
driver: bridge
ipam:
config:
# your chosen docker network here
- subnet: 172.16.0.0/24
Then:- stop all docker images and docker itself.
- cp everything in /var/lib/docker to /mnt/docker
- remove /var/lib/docker
- then create a link from /mnt/docker to /var/lib/docker
For the link, is it as simple as 'docker run -d -p --name /mnt/docker --link /var/lib/docker' ?
-
Thanks, put this in and got the output below:
pi@firewalla:~ (Firewalla) $ sudo su
root@firewalla:/home/pi# systemctl enable docker-compose@pi-hole
Created symlink /etc/systemd/system/multi-user.target.wants/docker-compose@pi-hole.service → /etc/systemd/system/docker-compose@.service. -
I think i'm doing something wrong with new containers, created traefik and portainer after moving pihole over to SSD /mnt/docker (I should rename docker to docker_containers maybe? my goal is to have any new container i create have it's volumes associated to /mnt/docker_containers/<docker name>
docker root is under /mnt/docker
pihole data volume not under /mnt and under default path still
traefik docker-compose and other files under /mnt/traefikportainer docker-compose and other files under /mnt/portainer
pi@firewalla:/mnt/portainer/data (Firewalla) $ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 790M 91M 699M 12% /run
/dev/mmcblk0p3 3.4G 2.8G 492M 85% /media/root-ro
tmpfs-root 200M 27M 174M 14% /media/root-rw
overlayroot 200M 27M 174M 14% /
tmpfs 3.9G 4.0K 3.9G 1% /dev/shm
tmpfs 5.0M 8.0K 5.0M 1% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mmcblk0p2 477M 77M 371M 18% /boot
/dev/mmcblk0p9 253M 242K 252M 1% /boot/efi
/dev/mmcblk0p4 3.4G 15M 3.2G 1% /var/lib/docker
/dev/mmcblk0p8 3.9G 600M 3.1G 16% /data
/dev/mmcblk0p7 976M 477M 432M 53% /log
/dev/mmcblk0p5 2.0G 947M 887M 52% /media/home-ro
/dev/mmcblk0p6 2.0G 790M 1.1G 44% /media/home-rw
overlay 2.0G 790M 1.1G 44% /home
tmpfs 20M 792K 20M 4% /alog
/dev/sda1 469G 878M 444G 1% /mnt
tmpfs 30M 404K 30M 2% /bspool
tmpfs 790M 0 790M 0% /run/user/1000
root is was looks concerning to me which multiple overlays and limited space for each container. Am i accurate in my concern and doing this wrong? i'd expect not to see such a small available space per container. Maybe not understanding this layout.root@firewalla:/mnt/portainer/data# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 4011396 0 4011396 0% /dev
tmpfs 808308 93020 715288 12% /run
/dev/mmcblk0p3 3546848 2843428 503536 85% /media/root-ro
tmpfs-root 204800 27572 177228 14% /media/root-rw
overlayroot 204800 27572 177228 14% /
tmpfs 4041540 4 4041536 1% /dev/shm
tmpfs 5120 8 5112 1% /run/lock
tmpfs 4041540 0 4041540 0% /sys/fs/cgroup
/dev/mmcblk0p2 487634 78217 379721 18% /boot
/dev/mmcblk0p9 258095 242 257853 1% /boot/efi
/dev/mmcblk0p4 3546848 15180 3331784 1% /var/lib/docker
/dev/mmcblk0p8 4062912 613560 3223256 16% /data
/dev/mmcblk0p7 999320 488280 442228 53% /log
/dev/mmcblk0p5 1998672 969612 907820 52% /media/home-ro
/dev/mmcblk0p6 1998672 808716 1068716 44% /media/home-rw
overlay 1998672 808716 1068716 44% /home
tmpfs 20480 804 19676 4% /alog
/dev/sda1 491207920 898232 465287976 1% /mnt
overlay 491207920 898232 465287976 1% /mnt/docker/overlay2/7c6ea681f22daded 51a20cdb90e5443a7a8660fb9611178492295c5f0d399c88/merged
shm 65536 2852 62684 5% /mnt/docker/containers/f3f14d7128646a 13af15ed20882439c310dbd5b4fcdd9d1f05fd0c253983a3d2/mounts/shm
tmpfs 30720 304 30416 1% /bspool
tmpfs 808308 0 808308 0% /run/user/1000
overlay 491207920 898232 465287976 1% /mnt/docker/overlay2/7b8bbc4ed55cafc9 4386a7a051d09baac5b015bc3c3bc6a3fb3eb8cf2461322b/merged
shm 65536 0 65536 0% /mnt/docker/containers/3891817a3d9762 2925ce2533becda0fbc6e61dfe9c090b457f844a94283aef5e/mounts/shm
overlay 491207920 898232 465287976 1% /mnt/docker/overlay2/3ba60f35c929b345 b659fb13736223e930dc69245fea2d0a864f77b3522523d7/merged
shm 65536 0 65536 0% /mnt/docker/containers/9627a12ea0e5c9 95875e2053aeb338e456444495e3eb3c3062a3deb675072e5b/mounts/shm -
Trying to isolate why my /var/lib/docker path is showing half used out of the blue after i had previously moved everything over to /mnt using your guide. I think this is happening after a restart of the FWG (had some recent power outages).
'sudo docker info' shows correct = '/mnt/docker' as i expect after previously moving everything over before the unexpected power outage.
Decided to run 'df -h' today to double check and sure enough
(Firewalla) $ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 790M 92M 699M 12% /run
/dev/mmcblk0p3 3.4G 2.8G 492M 85% /media/root-ro
tmpfs-root 200M 26M 175M 13% /media/root-rw
overlayroot 200M 26M 175M 13% /
tmpfs 3.9G 4.0K 3.9G 1% /dev/shm
tmpfs 5.0M 8.0K 5.0M 1% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mmcblk0p2 477M 77M 371M 18% /boot
/dev/mmcblk0p9 253M 242K 252M 1% /boot/efi
/dev/mmcblk0p7 976M 359M 551M 40% /log
/dev/mmcblk0p4 3.4G 1.6G 1.7G 48% /var/lib/docker
/dev/mmcblk0p8 3.9G 705M 3.0G 19% /data
/dev/mmcblk0p5 2.0G 947M 887M 52% /media/home-ro
/dev/mmcblk0p6 2.0G 805M 1.1G 44% /media/home-rw
overlay 2.0G 805M 1.1G 44% /home
/dev/sda1 469G 3.1G 442G 1% /mnt
tmpfs 20M 756K 20M 4% /alog
tmpfs 30M 456K 30M 2% /bspool
tmpfs 790M 0 790M 0% /run/user/1000 -
Thanks Michael, i do remember doing this after i did the initial xfer of data and saw /var/lib/docker at 0%. It's my understanding moving the path to /mnt/docker stops space being consumed in /var/lib/docker if done correctly and pruned following the xfer?
Will do this again now and monitor. -
(Firewalla) $ cat daemon.json
{
"data-root": "/mnt/docker"
}pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $ cat sdcard_docker.sh
#!/bin/bash
if [ -z "$(findmnt -n /mnt)" ] ; then
sudo mount /dev/sda1 /mnt
sudo chown 1000:1000 /mnt
else
echo "/mnt is already mounted..."
fiif [[ ! -f /etc/docker/daemon.json ]] ; then
sudo bash -c "echo -e '{\n\"data-root\": \"/mnt/docker\"\n}' > /etc/docker/daemon.json"
sudo systemctl restart docker
else
echo "modified docker location already set"
fi
pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $ ls -lrth
total 12K
-rwxr-xr-x 1 root root 308 Nov 20 09:26 start_pi_hole.sh
-rwxrwxr-- 1 pi pi 384 Nov 21 21:09 sdcard_docker.sh
-rwxrwxr-x 1 pi pi 318 Nov 25 16:43 start_unifi.sh
pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $ -
pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $ sudo docker info
Client:
Debug Mode: falseServer:
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 6
Server Version: 19.03.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-70-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.709GiB
Name: firewalla
ID: NNWA:Z5XN:R3FX:X7YJ:OWWM:7WWH:YRHD:6J3N:RCNT:JXFX:72Y4:F7Z7
Docker Root Dir: /mnt/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: falseWARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $ -
Just so i understand correctly about how this old directory is filling up post doing a prune in the past, if i have pointed docker to /mnt/docker and create new containers in the future, this would not cause this old directory to start filling up right? Trying to think why it's filling up when i had previously performed a prune on /var/lib/docker following moving docker dir. I remember seeing /var/lib/docker going to 0% after the prune was done. Only difference i can see is that the FWG went through a reload from power outage.
Please sign in to leave a comment.
Comments
36 comments