Moving docker to SD card on Firewalla Purple

Comments

33 comments

  • Avatar
    David Koppenhofer

    Since you have a docker-compose.yaml file, I think you'd start it with something like sudo systemctl start docker-compose@pi-hole, not "docker run"

    2
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    I think if you enable the service, it'll recover upon reboot.
    systemctl enable docker-compose@pi-hole

    1
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    I can't remember exactly the output, but that looks reasonable for enabling the service.

    1
    Comment actions Permalink
  • Avatar
    Michael Bierman

    @Lynk,

    Make sure you have pruned your docker containers. If you don't, a lot of space is wasted and the available space can quickly be consumed. 

    1
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    It looks like your docker is not running on your /mnt mount point.

    What does your /etc/docker/daemon.json have in it?

    Is your /home/pi/.firewalla/config/post_main.d/sdcard_docker.sh script intact?

    1
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    So, your docker is running with Docker Root Dir: /mnt/docker. I would temporarily disable that and then run prune commands on the original docker root as Michael suggested.

    1
    Comment actions Permalink
  • Avatar
    Michael Bierman

    Why not update the yaml file? 

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    Hi Michael,

    I don't understand what you mean. Update my docker-compose.yml files to make those apps use the SD card? I wasn't aware that was an option.

    I'm hardly a docker expert; everything I know, I've learned in the past few weeks. When I searched for information about using something other than /var/lib/docker, I found info about changing the service itself, or using the /etc/docker/daemon.json file.

    Also, I may not always use docker-compose, and would like those components to use the external location as well.

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    If you look at this example,

    https://github.com/mbierman/homebridge-installer/blob/main/docker-compose.yaml

    The part under `volumes`. The left side would be out mount point /mnt/docker or whatever leaving the right side of the statement alone. This permanently tells this is your config and assuming the SD card is mounted, docker will use that file system.

     

    if you are using command line and not yaml, then adjust the “-v” parameter in the same way. https://docs.portainer.io/v/ce-2.11/start/install/server/docker/linux

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    I _do_ like the idea of explicitly specifying where the volumes are.

    My question is whether containers (and images) would go on the SD card with this method. The issue I was facing is that my partition was running out of space because of image sizes, though moving the user data volumes off the internal partition to the card would help the issue to some extent.

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    What I suggested puts the all of the data for the docker image where you specify. So it does address disk space to an extent. Using: 

    sudo docker info

    You can find where containers and images are stored. The default on Ubuntu is: /var/lib/docker

    I have been able to install two large docker images on Purple by just setting the volumes as I described above so I have not needed to do it, but you can relocate the docker root: 

    https://www.ibm.com/docs/en/z-logdata-analytics/5.1.0?topic=compose-relocating-docker-root-directory

    another good reference: https://www.freecodecamp.org/news/where-are-docker-images-stored-docker-container-paths-explained/

    Also, be sure to read up and use docker prune appropriately. 

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    I just thought about it and I suppose if you do it right, you could link /mnt/docker to /var/lib/docker be sure to move anything you have in /var/lib/docker first. Something like: 

    1. stop all docker images and docker itself. 
    2. cp everything in /var/lib/docker to /mnt/docker 
    3. remove /var/lib/docker
    4. then create a link from /mnt/docker to /var/lib/docker

    If you do this, I would put the volumes of your containers in like, /mnt/docker_images or something different than /mnt/docker. I think you want them separated. 

    I have not tried this so do it at your own risk. :) 

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    That IBM article is what I ended up doing. The HomeAssisstant image alone is 1.2 gigs. Portainer isn't nearly as large, but it obviously takes up space, too. Since I hadn't relocated the volumes I had made (including a Webthings volume), that 2 gig was getting mighty full.

    Everything I read about moving the docker stuff out of /var/lib/docker recommended against doing it via a link, though that would have been easier for me to grasp. 😏

    I'll definitely keep those docker prune commands handy. I played with them while trying to minimize /var/lib/docker usage, and running them from time to time will help keep things heathy.

    Thanks for your help and feedback. The Firewalla community is pretty awesome.

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    Hi David, 

    I put this together based on your excellent work. Thought I'd share. 

    #!/bin/bash 
    if [ -z "$(mount | grep mnt )" ] ; then
            sudo mount /dev/mmcblk1p1 /mnt
            sudo chown 1000:1000 /mnt
    else
            echo "/mnt is already mounted..."
    fi

    if [ -z $(grep foo /etc/docker/daemon.json) ] ; then
            echo -e '{\n"data-root": "/mnt/docker"\n}' > /etc/docker/daemon.json
            sudo systemctl restart docker
    else
            echo "modified docker location already set"
    fi 
    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    What little bash knowledge I had, it was rusty, so thank you for that updated script.

    One issue is that the pi user can't write to /etc/docker, so I had to update that line with
    sudo bash -c "echo -e '{\n\"data-root\": \"/mnt/docker\"\n}' > /etc/docker/daemon.json"

    Now, to figure out how to write a script to show ports on the host which were opened via a container (in host networking mode, in my case).

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    I think 

     

    sudo echo -e '{\n"data-root": "/mnt/docker"\n}' > /etc/docker/daemon.json

    should work.

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    That's the first thing I tried, and got permission denied. I'm guessing the sudo doesn't pass through to the output redirection.

    I made a few tweaks to the conditionals of the script. Using findmnt to determine if the filesystem is mounted, and bash testing to see if the file exists.

    #!/bin/bash
    if [ -z "$(findmnt -n /mnt)" ] ; then
    sudo mount /dev/mmcblk1p1 /mnt
    sudo chown 1000:1000 /mnt
    else
    echo "/mnt is already mounted..."
    fi


    if [[ ! -f /etc/docker/daemon.json ]] ; then
    sudo bash -c "echo -e '{\n\"data-root\": \"/mnt/docker\"\n}' > /etc/docker/daemon.json"
    sudo systemctl restart docker
    else
    echo "modified docker location already set"
    fi
    0
    Comment actions Permalink
  • Avatar
    Steve

    When I try to save to I get permission denied. Any Idea why that is happening? Also the default setting is only Root has RW access to the card. Did you make any changes to include pi as well?

     

     

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    @steve, what are you trying to save when getting that error? The sdcard_docker.sh script?

    0
    Comment actions Permalink
  • Avatar
    Steve

    @David, Thanks for your help. Yes, It was the startup script. I did not create that folder, Once that was done I was able to save. Thanks!

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    @Steve, BTW, I updated the OP to include the mkdir.

    0
    Comment actions Permalink
  • Avatar
    Lynk

    Was looking at Michaels post about changing the location of the volume. I changed up my pihole yaml to this and wondering if this looks accurate? i didn't change 2nd and 3rd line under volume and thinking i could leave as is? 

    pi@firewalla:~/.firewalla/run/docker/pi-hole (Firewalla) $ cat docker-compose.yaml
    version: "3"

    # More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
    services:
      pihole:
        container_name: pihole
        image: pihole/pihole:latest
        environment:
          # set a secure password here or the default will be firewalla
          WEBPASSWORD: 'firewalla'
        # Volumes store your data between container upgrades
        volumes:
          - '/mnt/docker_images/data/pi-hole/etc-pihole/:/etc/pihole/'
          - './etc-dnsmasq.d/:/etc/dnsmasq.d/'
          - '/etc/localtime:/etc/localtime:ro'
        restart: unless-stopped
        cap_add:
          - NET_ADMIN
        networks:
          default:
            # static IP address for pi-hole
            ipv4_address: 172.16.0.2
    networks:
      default:
        driver: bridge
        ipam:
         config:
           # your chosen docker network here
           - subnet: 172.16.0.0/24

    Then:

    1. stop all docker images and docker itself. 
    2. cp everything in /var/lib/docker to /mnt/docker 
    3. remove /var/lib/docker
    4. then create a link from /mnt/docker to /var/lib/docker

      For the link, is it as simple as 'docker run -d -p --name /mnt/docker --link /var/lib/docker' ?
    0
    Comment actions Permalink
  • Avatar
    Lynk

    Thanks will try this. Can't seem to get my docker containers to persist through reboot. Have the updated startup script in place as well and manually running shows mnt already mounted as expected.

    0
    Comment actions Permalink
  • Avatar
    Lynk

    Thanks, put this in and got the output below: 

    pi@firewalla:~ (Firewalla) $ sudo su
    root@firewalla:/home/pi# systemctl enable docker-compose@pi-hole
    Created symlink /etc/systemd/system/multi-user.target.wants/docker-compose@pi-hole.service → /etc/systemd/system/docker-compose@.service.

    0
    Comment actions Permalink
  • Avatar
    Lynk

    I think i'm doing something wrong with new containers, created traefik and portainer after moving pihole over to SSD /mnt/docker (I should rename docker to docker_containers maybe? my goal is to have any new container i create have it's volumes associated to /mnt/docker_containers/<docker name>

    docker root is under /mnt/docker

    pihole data volume not under /mnt and under default path still
    traefik docker-compose and other files under /mnt/traefik

    portainer docker-compose and other files under /mnt/portainer

    pi@firewalla:/mnt/portainer/data (Firewalla) $ df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            3.9G     0  3.9G   0% /dev
    tmpfs           790M   91M  699M  12% /run
    /dev/mmcblk0p3  3.4G  2.8G  492M  85% /media/root-ro
    tmpfs-root      200M   27M  174M  14% /media/root-rw
    overlayroot     200M   27M  174M  14% /
    tmpfs           3.9G  4.0K  3.9G   1% /dev/shm
    tmpfs           5.0M  8.0K  5.0M   1% /run/lock
    tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
    /dev/mmcblk0p2  477M   77M  371M  18% /boot
    /dev/mmcblk0p9  253M  242K  252M   1% /boot/efi
    /dev/mmcblk0p4  3.4G   15M  3.2G   1% /var/lib/docker
    /dev/mmcblk0p8  3.9G  600M  3.1G  16% /data
    /dev/mmcblk0p7  976M  477M  432M  53% /log
    /dev/mmcblk0p5  2.0G  947M  887M  52% /media/home-ro
    /dev/mmcblk0p6  2.0G  790M  1.1G  44% /media/home-rw
    overlay         2.0G  790M  1.1G  44% /home
    tmpfs            20M  792K   20M   4% /alog
    /dev/sda1       469G  878M  444G   1% /mnt
    tmpfs            30M  404K   30M   2% /bspool
    tmpfs           790M     0  790M   0% /run/user/1000

    root is was looks concerning to me which multiple overlays and limited space for each container. Am i accurate in my concern and doing this wrong? i'd expect not to see such a small available space per container. Maybe not understanding this layout. 

    root@firewalla:/mnt/portainer/data# df
    Filesystem     1K-blocks    Used Available Use% Mounted on
    udev             4011396       0   4011396   0% /dev
    tmpfs             808308   93020    715288  12% /run
    /dev/mmcblk0p3   3546848 2843428    503536  85% /media/root-ro
    tmpfs-root        204800   27572    177228  14% /media/root-rw
    overlayroot       204800   27572    177228  14% /
    tmpfs            4041540       4   4041536   1% /dev/shm
    tmpfs               5120       8      5112   1% /run/lock
    tmpfs            4041540       0   4041540   0% /sys/fs/cgroup
    /dev/mmcblk0p2    487634   78217    379721  18% /boot
    /dev/mmcblk0p9    258095     242    257853   1% /boot/efi
    /dev/mmcblk0p4   3546848   15180   3331784   1% /var/lib/docker
    /dev/mmcblk0p8   4062912  613560   3223256  16% /data
    /dev/mmcblk0p7    999320  488280    442228  53% /log
    /dev/mmcblk0p5   1998672  969612    907820  52% /media/home-ro
    /dev/mmcblk0p6   1998672  808716   1068716  44% /media/home-rw
    overlay          1998672  808716   1068716  44% /home
    tmpfs              20480     804     19676   4% /alog
    /dev/sda1      491207920  898232 465287976   1% /mnt
    overlay        491207920  898232 465287976   1% /mnt/docker/overlay2/7c6ea681f22daded                             51a20cdb90e5443a7a8660fb9611178492295c5f0d399c88/merged
    shm                65536    2852     62684   5% /mnt/docker/containers/f3f14d7128646a                                         13af15ed20882439c310dbd5b4fcdd9d1f05fd0c253983a3d2/mounts/shm
    tmpfs              30720     304     30416   1% /bspool
    tmpfs             808308       0    808308   0% /run/user/1000
    overlay        491207920  898232 465287976   1% /mnt/docker/overlay2/7b8bbc4ed55cafc9                            4386a7a051d09baac5b015bc3c3bc6a3fb3eb8cf2461322b/merged
    shm                65536       0     65536   0% /mnt/docker/containers/3891817a3d9762                                           2925ce2533becda0fbc6e61dfe9c090b457f844a94283aef5e/mounts/shm
    overlay        491207920  898232 465287976   1% /mnt/docker/overlay2/3ba60f35c929b345                             b659fb13736223e930dc69245fea2d0a864f77b3522523d7/merged
    shm                65536       0     65536   0% /mnt/docker/containers/9627a12ea0e5c9                                           95875e2053aeb338e456444495e3eb3c3062a3deb675072e5b/mounts/shm

    0
    Comment actions Permalink
  • Avatar
    Lynk

    Trying to isolate why my /var/lib/docker path is showing half used out of the blue after i had previously moved everything over to /mnt using your guide. I think this is happening after a restart of the FWG (had some recent power outages).

    'sudo docker info' shows correct = '/mnt/docker' as i expect after previously moving everything over before the unexpected power outage. 

    Decided to run 'df -h' today to double check and sure enough

    (Firewalla) $ df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            3.9G     0  3.9G   0% /dev
    tmpfs           790M   92M  699M  12% /run
    /dev/mmcblk0p3  3.4G  2.8G  492M  85% /media/root-ro
    tmpfs-root      200M   26M  175M  13% /media/root-rw
    overlayroot     200M   26M  175M  13% /
    tmpfs           3.9G  4.0K  3.9G   1% /dev/shm
    tmpfs           5.0M  8.0K  5.0M   1% /run/lock
    tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
    /dev/mmcblk0p2  477M   77M  371M  18% /boot
    /dev/mmcblk0p9  253M  242K  252M   1% /boot/efi
    /dev/mmcblk0p7  976M  359M  551M  40% /log
    /dev/mmcblk0p4  3.4G  1.6G  1.7G  48% /var/lib/docker
    /dev/mmcblk0p8  3.9G  705M  3.0G  19% /data
    /dev/mmcblk0p5  2.0G  947M  887M  52% /media/home-ro
    /dev/mmcblk0p6  2.0G  805M  1.1G  44% /media/home-rw
    overlay         2.0G  805M  1.1G  44% /home
    /dev/sda1       469G  3.1G  442G   1% /mnt
    tmpfs            20M  756K   20M   4% /alog
    tmpfs            30M  456K   30M   2% /bspool
    tmpfs           790M     0  790M   0% /run/user/1000

    0
    Comment actions Permalink
  • Avatar
    Lynk

    Thanks Michael, i do remember doing this after i did the initial xfer of data and saw /var/lib/docker at 0%. It's my understanding moving the path to /mnt/docker stops space being consumed in /var/lib/docker if done correctly and pruned following the xfer? 

    Will do this again now and monitor. 

    0
    Comment actions Permalink
  • Avatar
    Lynk

    (Firewalla) $ cat daemon.json
    {
    "data-root": "/mnt/docker"
    }

     

    pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $ cat sdcard_docker.sh
    #!/bin/bash
    if [ -z "$(findmnt -n /mnt)" ] ; then
        sudo mount /dev/sda1 /mnt
        sudo chown 1000:1000 /mnt
    else
        echo "/mnt is already mounted..."
    fi

    if [[ ! -f /etc/docker/daemon.json ]] ; then
        sudo bash -c "echo -e '{\n\"data-root\": \"/mnt/docker\"\n}' > /etc/docker/daemon.json"
        sudo systemctl restart docker
    else
        echo "modified docker location already set"
    fi
    pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $ ls -lrth
    total 12K
    -rwxr-xr-x 1 root root 308 Nov 20 09:26 start_pi_hole.sh
    -rwxrwxr-- 1 pi   pi   384 Nov 21 21:09 sdcard_docker.sh
    -rwxrwxr-x 1 pi   pi   318 Nov 25 16:43 start_unifi.sh
    pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $

    0
    Comment actions Permalink
  • Avatar
    Lynk


    pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $ sudo docker info
    Client:
     Debug Mode: false

    Server:
     Containers: 4
      Running: 4
      Paused: 0
      Stopped: 0
     Images: 6
     Server Version: 19.03.6
     Storage Driver: overlay2
      Backing Filesystem: extfs
      Supports d_type: true
      Native Overlay Diff: true
     Logging Driver: json-file
     Cgroup Driver: cgroupfs
     Plugins:
      Volume: local
      Network: bridge host ipvlan macvlan null overlay
      Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
     Swarm: inactive
     Runtimes: runc
     Default Runtime: runc
     Init Binary: docker-init
     containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
     runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
     init version: fec3683
     Security Options:
      apparmor
      seccomp
       Profile: default
     Kernel Version: 4.15.0-70-generic
     Operating System: Ubuntu 18.04.3 LTS
     OSType: linux
     Architecture: x86_64
     CPUs: 4
     Total Memory: 7.709GiB
     Name: firewalla
     ID: NNWA:Z5XN:R3FX:X7YJ:OWWM:7WWH:YRHD:6J3N:RCNT:JXFX:72Y4:F7Z7
     Docker Root Dir: /mnt/docker
     Debug Mode: false
     Registry: https://index.docker.io/v1/
     Labels:
     Experimental: false
     Insecure Registries:
      127.0.0.0/8
     Live Restore Enabled: false

    WARNING: No swap limit support
    WARNING: bridge-nf-call-iptables is disabled
    WARNING: bridge-nf-call-ip6tables is disabled
    pi@firewalla:~/.firewalla/config/post_main.d (Firewalla) $

    0
    Comment actions Permalink
  • Avatar
    Lynk

    Just so i understand correctly about how this old directory is filling up post doing a prune in the past, if i have pointed docker to /mnt/docker and create new containers in the future, this would not cause this old directory to start filling up right? Trying to think why it's filling up when i had previously performed a prune on /var/lib/docker following moving docker dir. I remember seeing /var/lib/docker going to 0% after the prune was done. Only difference i can see is that the FWG went through a reload from power outage. 

    0
    Comment actions Permalink

Please sign in to leave a comment.