Moving docker to SD card on Firewalla Purple

Comments

22 comments

  • Avatar
    Michael Bierman

    Why not update the yaml file? 

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    Hi Michael,

    I don't understand what you mean. Update my docker-compose.yml files to make those apps use the SD card? I wasn't aware that was an option.

    I'm hardly a docker expert; everything I know, I've learned in the past few weeks. When I searched for information about using something other than /var/lib/docker, I found info about changing the service itself, or using the /etc/docker/daemon.json file.

    Also, I may not always use docker-compose, and would like those components to use the external location as well.

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    If you look at this example,

    https://github.com/mbierman/homebridge-installer/blob/main/docker-compose.yaml

    The part under `volumes`. The left side would be out mount point /mnt/docker or whatever leaving the right side of the statement alone. This permanently tells this is your config and assuming the SD card is mounted, docker will use that file system.

     

    if you are using command line and not yaml, then adjust the “-v” parameter in the same way. https://docs.portainer.io/v/ce-2.11/start/install/server/docker/linux

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    I _do_ like the idea of explicitly specifying where the volumes are.

    My question is whether containers (and images) would go on the SD card with this method. The issue I was facing is that my partition was running out of space because of image sizes, though moving the user data volumes off the internal partition to the card would help the issue to some extent.

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    What I suggested puts the all of the data for the docker image where you specify. So it does address disk space to an extent. Using: 

    sudo docker info

    You can find where containers and images are stored. The default on Ubuntu is: /var/lib/docker

    I have been able to install two large docker images on Purple by just setting the volumes as I described above so I have not needed to do it, but you can relocate the docker root: 

    https://www.ibm.com/docs/en/z-logdata-analytics/5.1.0?topic=compose-relocating-docker-root-directory

    another good reference: https://www.freecodecamp.org/news/where-are-docker-images-stored-docker-container-paths-explained/

    Also, be sure to read up and use docker prune appropriately. 

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    I just thought about it and I suppose if you do it right, you could link /mnt/docker to /var/lib/docker be sure to move anything you have in /var/lib/docker first. Something like: 

    1. stop all docker images and docker itself. 
    2. cp everything in /var/lib/docker to /mnt/docker 
    3. remove /var/lib/docker
    4. then create a link from /mnt/docker to /var/lib/docker

    If you do this, I would put the volumes of your containers in like, /mnt/docker_images or something different than /mnt/docker. I think you want them separated. 

    I have not tried this so do it at your own risk. :) 

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    That IBM article is what I ended up doing. The HomeAssisstant image alone is 1.2 gigs. Portainer isn't nearly as large, but it obviously takes up space, too. Since I hadn't relocated the volumes I had made (including a Webthings volume), that 2 gig was getting mighty full.

    Everything I read about moving the docker stuff out of /var/lib/docker recommended against doing it via a link, though that would have been easier for me to grasp. 😏

    I'll definitely keep those docker prune commands handy. I played with them while trying to minimize /var/lib/docker usage, and running them from time to time will help keep things heathy.

    Thanks for your help and feedback. The Firewalla community is pretty awesome.

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    Hi David, 

    I put this together based on your excellent work. Thought I'd share. 

    #!/bin/bash 
    if [ -z "$(mount | grep mnt )" ] ; then
            sudo mount /dev/mmcblk1p1 /mnt
            sudo chown 1000:1000 /mnt
    else
            echo "/mnt is already mounted..."
    fi

    if [ -z $(grep foo /etc/docker/daemon.json) ] ; then
            echo -e '{\n"data-root": "/mnt/docker"\n}' > /etc/docker/daemon.json
            sudo systemctl restart docker
    else
            echo "modified docker location already set"
    fi 
    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    What little bash knowledge I had, it was rusty, so thank you for that updated script.

    One issue is that the pi user can't write to /etc/docker, so I had to update that line with
    sudo bash -c "echo -e '{\n\"data-root\": \"/mnt/docker\"\n}' > /etc/docker/daemon.json"

    Now, to figure out how to write a script to show ports on the host which were opened via a container (in host networking mode, in my case).

    0
    Comment actions Permalink
  • Avatar
    Michael Bierman

    I think 

     

    sudo echo -e '{\n"data-root": "/mnt/docker"\n}' > /etc/docker/daemon.json

    should work.

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    That's the first thing I tried, and got permission denied. I'm guessing the sudo doesn't pass through to the output redirection.

    I made a few tweaks to the conditionals of the script. Using findmnt to determine if the filesystem is mounted, and bash testing to see if the file exists.

    #!/bin/bash
    if [ -z "$(findmnt -n /mnt)" ] ; then
    sudo mount /dev/mmcblk1p1 /mnt
    sudo chown 1000:1000 /mnt
    else
    echo "/mnt is already mounted..."
    fi


    if [[ ! -f /etc/docker/daemon.json ]] ; then
    sudo bash -c "echo -e '{\n\"data-root\": \"/mnt/docker\"\n}' > /etc/docker/daemon.json"
    sudo systemctl restart docker
    else
    echo "modified docker location already set"
    fi
    0
    Comment actions Permalink
  • Avatar
    Steve

    When I try to save to I get permission denied. Any Idea why that is happening? Also the default setting is only Root has RW access to the card. Did you make any changes to include pi as well?

     

     

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    @steve, what are you trying to save when getting that error? The sdcard_docker.sh script?

    0
    Comment actions Permalink
  • Avatar
    Steve

    @David, Thanks for your help. Yes, It was the startup script. I did not create that folder, Once that was done I was able to save. Thanks!

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    @Steve, BTW, I updated the OP to include the mkdir.

    0
    Comment actions Permalink
  • Avatar
    Lynk

    Was looking at Michaels post about changing the location of the volume. I changed up my pihole yaml to this and wondering if this looks accurate? i didn't change 2nd and 3rd line under volume and thinking i could leave as is? 

    pi@firewalla:~/.firewalla/run/docker/pi-hole (Firewalla) $ cat docker-compose.yaml
    version: "3"

    # More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
    services:
      pihole:
        container_name: pihole
        image: pihole/pihole:latest
        environment:
          # set a secure password here or the default will be firewalla
          WEBPASSWORD: 'firewalla'
        # Volumes store your data between container upgrades
        volumes:
          - '/mnt/docker_images/data/pi-hole/etc-pihole/:/etc/pihole/'
          - './etc-dnsmasq.d/:/etc/dnsmasq.d/'
          - '/etc/localtime:/etc/localtime:ro'
        restart: unless-stopped
        cap_add:
          - NET_ADMIN
        networks:
          default:
            # static IP address for pi-hole
            ipv4_address: 172.16.0.2
    networks:
      default:
        driver: bridge
        ipam:
         config:
           # your chosen docker network here
           - subnet: 172.16.0.0/24

    Then:

    1. stop all docker images and docker itself. 
    2. cp everything in /var/lib/docker to /mnt/docker 
    3. remove /var/lib/docker
    4. then create a link from /mnt/docker to /var/lib/docker

      For the link, is it as simple as 'docker run -d -p --name /mnt/docker --link /var/lib/docker' ?
    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    Since you have a docker-compose.yaml file, I think you'd start it with something like sudo systemctl start docker-compose@pi-hole, not "docker run"

    2
    Comment actions Permalink
  • Avatar
    Lynk

    Thanks will try this. Can't seem to get my docker containers to persist through reboot. Have the updated startup script in place as well and manually running shows mnt already mounted as expected.

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    I think if you enable the service, it'll recover upon reboot.
    systemctl enable docker-compose@pi-hole

    1
    Comment actions Permalink
  • Avatar
    Lynk

    Thanks, put this in and got the output below: 

    pi@firewalla:~ (Firewalla) $ sudo su
    root@firewalla:/home/pi# systemctl enable docker-compose@pi-hole
    Created symlink /etc/systemd/system/multi-user.target.wants/docker-compose@pi-hole.service → /etc/systemd/system/docker-compose@.service.

    0
    Comment actions Permalink
  • Avatar
    David Koppenhofer

    I can't remember exactly the output, but that looks reasonable for enabling the service.

    1
    Comment actions Permalink
  • Avatar
    Lynk

    I think i'm doing something wrong with new containers, created traefik and portainer after moving pihole over to SSD /mnt/docker (I should rename docker to docker_containers maybe? my goal is to have any new container i create have it's volumes associated to /mnt/docker_containers/<docker name>

    docker root is under /mnt/docker

    pihole data volume not under /mnt and under default path still
    traefik docker-compose and other files under /mnt/traefik

    portainer docker-compose and other files under /mnt/portainer

    pi@firewalla:/mnt/portainer/data (Firewalla) $ df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            3.9G     0  3.9G   0% /dev
    tmpfs           790M   91M  699M  12% /run
    /dev/mmcblk0p3  3.4G  2.8G  492M  85% /media/root-ro
    tmpfs-root      200M   27M  174M  14% /media/root-rw
    overlayroot     200M   27M  174M  14% /
    tmpfs           3.9G  4.0K  3.9G   1% /dev/shm
    tmpfs           5.0M  8.0K  5.0M   1% /run/lock
    tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
    /dev/mmcblk0p2  477M   77M  371M  18% /boot
    /dev/mmcblk0p9  253M  242K  252M   1% /boot/efi
    /dev/mmcblk0p4  3.4G   15M  3.2G   1% /var/lib/docker
    /dev/mmcblk0p8  3.9G  600M  3.1G  16% /data
    /dev/mmcblk0p7  976M  477M  432M  53% /log
    /dev/mmcblk0p5  2.0G  947M  887M  52% /media/home-ro
    /dev/mmcblk0p6  2.0G  790M  1.1G  44% /media/home-rw
    overlay         2.0G  790M  1.1G  44% /home
    tmpfs            20M  792K   20M   4% /alog
    /dev/sda1       469G  878M  444G   1% /mnt
    tmpfs            30M  404K   30M   2% /bspool
    tmpfs           790M     0  790M   0% /run/user/1000

    root is was looks concerning to me which multiple overlays and limited space for each container. Am i accurate in my concern and doing this wrong? i'd expect not to see such a small available space per container. Maybe not understanding this layout. 

    root@firewalla:/mnt/portainer/data# df
    Filesystem     1K-blocks    Used Available Use% Mounted on
    udev             4011396       0   4011396   0% /dev
    tmpfs             808308   93020    715288  12% /run
    /dev/mmcblk0p3   3546848 2843428    503536  85% /media/root-ro
    tmpfs-root        204800   27572    177228  14% /media/root-rw
    overlayroot       204800   27572    177228  14% /
    tmpfs            4041540       4   4041536   1% /dev/shm
    tmpfs               5120       8      5112   1% /run/lock
    tmpfs            4041540       0   4041540   0% /sys/fs/cgroup
    /dev/mmcblk0p2    487634   78217    379721  18% /boot
    /dev/mmcblk0p9    258095     242    257853   1% /boot/efi
    /dev/mmcblk0p4   3546848   15180   3331784   1% /var/lib/docker
    /dev/mmcblk0p8   4062912  613560   3223256  16% /data
    /dev/mmcblk0p7    999320  488280    442228  53% /log
    /dev/mmcblk0p5   1998672  969612    907820  52% /media/home-ro
    /dev/mmcblk0p6   1998672  808716   1068716  44% /media/home-rw
    overlay          1998672  808716   1068716  44% /home
    tmpfs              20480     804     19676   4% /alog
    /dev/sda1      491207920  898232 465287976   1% /mnt
    overlay        491207920  898232 465287976   1% /mnt/docker/overlay2/7c6ea681f22daded                             51a20cdb90e5443a7a8660fb9611178492295c5f0d399c88/merged
    shm                65536    2852     62684   5% /mnt/docker/containers/f3f14d7128646a                                         13af15ed20882439c310dbd5b4fcdd9d1f05fd0c253983a3d2/mounts/shm
    tmpfs              30720     304     30416   1% /bspool
    tmpfs             808308       0    808308   0% /run/user/1000
    overlay        491207920  898232 465287976   1% /mnt/docker/overlay2/7b8bbc4ed55cafc9                            4386a7a051d09baac5b015bc3c3bc6a3fb3eb8cf2461322b/merged
    shm                65536       0     65536   0% /mnt/docker/containers/3891817a3d9762                                           2925ce2533becda0fbc6e61dfe9c090b457f844a94283aef5e/mounts/shm
    overlay        491207920  898232 465287976   1% /mnt/docker/overlay2/3ba60f35c929b345                             b659fb13736223e930dc69245fea2d0a864f77b3522523d7/merged
    shm                65536       0     65536   0% /mnt/docker/containers/9627a12ea0e5c9                                           95875e2053aeb338e456444495e3eb3c3062a3deb675072e5b/mounts/shm

    0
    Comment actions Permalink

Please sign in to leave a comment.