remote: Error response from daemon: Error processing tar file(exit status 1): write /app/.heroku/node/bin/node: no space left on device

[nelsonic]

When attempting to auto-deploy from CircleCI we are seeing the following error:

remote: Error response from daemon: Error processing tar file(exit status 1): write /app/.heroku/node/bin/node: no space left on device

Full Stack Trace:

#!/bin/bash -eo pipefail
sh bin/deploy.sh
CWD /root/repo/bin
BRANCH => close-modal
DOKKU_APP => ci
DOKKU_APP => ci-close-modal
URL => 138.68.163.126:ci-close-modal
CREATE => dokku apps:create ci-close-modal
Warning: Permanently added '138.68.163.126' (ECDSA) to the list of known hosts.

Name is already taken
 WARNING:  env2 was required to load an .env file:  /root/repo/.env  NOT FOUND!  Please see: http://git.io/vG3UZ
ENV => NODE_VERSION=8.11.3 HOSTNAME=a406bd28a9d6 CI_PULL_REQUESTS=https://github.com/charterindex/application/pull/320 SERVER_IP_ADDRESS=138.68.163.126 ENVIRONMENT-VARIABLES-REDACTED!
-----> Setting config vars
       YARN_VERSION:          1.6.0
       PORT:                  5000
       JWT_SECRET:            thisissecret
       HOSTNAME:              a406bd28a9d6
       SERVER_IP_ADDRESS:     138.68.163.126
       DOKKU_APP:             ci-close-modal
       CI:                    true
       NODE_VERSION:          8.11.3
       CI_PULL_REQUESTS:      https://github.com/charterindex/application/pull/320
       BASH_ENV:              /tmp/.bash_env-5b3bfdabb05ef50013c42a5b-0-build
       TRAVIS:                false
       CIRCLECI:              true
-----> Restarting app ci-close-modal
 !     App ci-close-modal has not been deployed
REMOTE => dokku dokku@138.68.163.126:ci-close-modal
fatal: No such remote 'dokku'
dokku	dokku@138.68.163.126:ci-close-modal (fetch)
dokku	dokku@138.68.163.126:ci-close-modal (push)
origin	git@github.com:charterindex/application.git (fetch)
origin	git@github.com:charterindex/application.git (push)
KILL_NGINX => pkill nginx
SYSTEMCTL_START_NGINX => systemctl start nginx
COMMIT_HASH => bedba5b4f404b4f94b7c69eb211eaf9844ba7f28
PUSH => git push dokku bedba5b4f404b4f94b7c69eb211eaf9844ba7f28:refs/heads/master
Counting objects: 3711, done.
Delta compression using up to 32 threads.
Compressing objects: 100% (1213/1213), done.
Writing objects:  26% (965/3711)   
Writing objects: 100% (3711/3711), 6.78 MiB | 5.13 MiB/s, done.
Total 3711 (delta 2415), reused 3698 (delta 2408)
remote: -----> Cleaning up...
remote: -----> Building ci-close-modal from herokuish...
remote: -----> Adding BUILD_ENV to build environment...
remote:        -----> Node.js app detected
remote:        
remote: -----> Creating runtime environment
remote:        
remote:        NPM_CONFIG_LOGLEVEL=error
remote:        YARN_VERSION=1.6.0
remote:        NODE_VERSION=8.11.3
remote:        NODE_VERBOSE=false
remote:        NODE_ENV=production
remote:        NODE_MODULES_CACHE=true
remote:        
remote: -----> Installing binaries
remote:        engines.node (package.json):  8.11.2
remote:        engines.npm (package.json):   5.6.0
remote:        
remote:        Resolving node version 8.11.2...
remote:        Downloading and installing node 8.11.2...
remote:        npm 5.6.0 already installed with node
remote:        
remote: -----> Restoring cache
remote:        Loading 2 from cacheDirectories (default):
remote:        - node_modules
remote:        - bower_components (not cached - skipping)
remote:        
remote: -----> Building dependencies
remote:        Installing node modules (package.json)
remote:        
remote:        > application@1.0.0 postinstall /tmp/build
remote:        > npm run elm-install && npm run elm-compile && npm run css-compile
remote:        
remote:        
remote:        > application@1.0.0 elm-install /tmp/build
remote:        > cd elm && ../node_modules/elm/binwrappers/elm-package install -y && cd ..
remote:        
remote:        Starting downloads...
remote:        
remote:        ● dwyl/elm-datepicker 2.2.0
remote:        ● NoRedInk/elm-decode-pipeline 3.0.0
remote:        ● elm-lang/core 5.1.1
remote:        ● elm-lang/dom 1.1.1
remote:        ● elm-lang/html 2.0.0
remote:        ● elm-lang/http 1.0.0
remote:        ● elm-lang/keyboard 1.0.1
remote:        ● elm-lang/navigation 2.1.0
remote:        ● elm-lang/virtual-dom 2.0.4
remote:        ● justinmimbs/elm-date-extra 3.0.0
remote:        
remote:        Packages configured successfully!
remote:        
remote:        > application@1.0.0 elm-compile /tmp/build
remote:        > cd elm && ../node_modules/elm/binwrappers/elm-make *.elm --output ../public/js/elm.js --yes --warn && cd ..
remote:        
remote:        Success! Compiled 77 modules.
remote:        Successfully generated ../public/js/elm.js
remote:        
remote:        > application@1.0.0 css-compile /tmp/build
remote:        > postcss routes/css/index.css --output public/stylesheets/style.min.css --config postcss.config.js --verbose
remote:        
remote:        Processing routes/css/index.css...
remote:        Finished routes/css/index.css in 3.41 s
remote:        up to date in 32.714s
remote:        
remote: -----> Caching build
remote:        Clearing previous node cache
remote:        Saving 2 cacheDirectories (default):
remote:        - node_modules
remote:        - bower_components (nothing to cache)
remote:        
remote: -----> Pruning devDependencies
remote:        Skipping because npm 5.6.0 sometimes fails when running 'npm prune' due to a known issue
remote:        https://github.com/npm/npm/issues/19356
remote:        
remote:        You can silence this warning by updating to at least npm 5.7.1 in your package.json
remote:        https://devcenter.heroku.com/articles/nodejs-support#specifying-an-npm-version
remote:        
remote: -----> Build succeeded!
remote:        -----> Discovering process types
remote:        Procfile declares types -> web
remote: Error response from daemon: Error processing tar file(exit status 1): write /app/.heroku/node/bin/node: no space left on device
To dokku@138.68.163.126:ci-close-modal
 ! [remote rejected] bedba5b4f404b4f94b7c69eb211eaf9844ba7f28 -> master (pre-receive hook declined)
error: failed to push some refs to 'dokku@138.68.163.126:ci-close-modal'
server.crt
server.key
-----> Unsetting DOKKU_PROXY_PORT
-----> Unsetting DOKKU_PROXY_SSL_PORT
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:5000
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:5000 https:443:5000
-----> Setting config vars
       DOKKU_PROXY_PORT:  80
-----> Setting config vars
       DOKKU_PROXY_SSL_PORT:  443
-----> No matching configured domains for ci-close-modal found in SSL certificate. Your app will show as insecure in a browser if accessed via SSL
-----> Please add appropriate domains via the dokku domains command
-----> Configured domains for app:
=====> A
=====> *.ademo.app
=====> 138.68.163.126
=====> ademo.app
-----> Domains found in SSL certificate:
=====> *.ademo.app
=====> ademo.app
grep: /home/dokku/ci-close-modal/VHOST: No such file or directory
-----> Configuring 138.68.163.126...(using built-in template)
-----> Configuring A...(using built-in template)
-----> Configuring *.ademo.app...(using built-in template)
-----> Configuring ademo.app...(using built-in template)
-----> App ci-close-modal has not been deployed. Skipping nginx config creation
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

[nelsonic]

still getting:

remote: Error response from daemon: Error processing tar file(exit status 1): write /app/node_modules/puppeteer/.local-chromium/linux-575458/chrome-linux/chrome: no space left on device

https://stackoverflow.com/questions/24671621/no-space-left-on-device

Such difference between the output of du -sh and df -h may happen if some large file has been deleted, but is still opened by some process. Check with the command lsof | grep deleted to see which processes have opened descriptors to deleted files. You can restart the process and the space will be freed.

Digital Ocean dashboard indicates that we are out of SSD and RAM space ... 🙄

so I logged into the machine and ran:

reboot -h now

Based on: https://serverfault.com/questions/620595/best-way-to-gracefully-restart-centos

Of the system RAM 1882872 (1.8Gb) total, 1299252 (1.29Gb) is used and only 268800 (268Mb) available.

That did not have the desired effect because there are still way too many running Docker Containers (52!)

So I decided to "Nuke" the droplet with a power cycle! (which is obviously not recommended!)

Obviously now there is no docker process running:

[root@centos-dokku-paas ~]# docker ps -a -q
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

After the power cycle:

[root@centos-dokku-paas ~]# sudo systemctl start docker
[root@centos-dokku-paas ~]# docker -D info
Containers: 52
 Running: 52
 Paused: 0
 Stopped: 0
Images: 270
Server Version: 18.05.0-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.2.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.796GiB
Name: centos-dokku-paas

I read: https://medium.com/the-code-review/clean-out-your-docker-images-containers-and-volumes-with-single-commands-b8e38253c271
and ran the command:

docker system prune --all --force --volumes

Which had the following output:

Deleted Containers:
5850620d78b98512276bd03816cd61c70eefd5c56f64a513edfc713188fa16f2
152487e3869812f40fb48f26a667d75959b950650c5a5fa257999a8561bbd081
61a40b85e3538dccf5bc5ad040f751fe1d0058db5a9e760fc2629aa680e3f847
0ec7301b697a24e758fe8f0193d2c2233008779f5ee49e652f95b5603c70b95b
dca74ebdb1f74a233c6fc6c8bfa9d080294f65f4bf6fea11955ea95b19bfbfce
755b3c5eca7fdba45bab39b9e05e4d9423cea9a8dea7444f4cc670161a9068c1

Deleted Images:
untagged: dokku/ci-master:latest
deleted: sha256:fce0a33a7768bc4bfa73ad324bc9ac546eb6bafad1a725e9c9118febbe8f556b
deleted: sha256:9efa1313d1bd4e12c2427c5715cd57db63d7a5236709f43a8c4938be9a634843
deleted: sha256:397980186cb30c1845f2b32f17f144723da417c135bc081d38c79cde5579617c
deleted: sha256:085bf52ee253902a5d7198143684d7138a7bad77052cbc1c05675e5765e18b38
deleted: sha256:26e4278b79a7ee649e9433fda844827e19b31eb3063c3d7890106dd3400b3d0e
deleted: sha256:f3efa2da50afd418fac27263a2a527c7ab46a2fb5655ee80080a5ca2237adfd4
untagged: dokku/ci-ui-tweaks:latest
deleted: sha256:614ec0e1e26a37d87f2611dbd3d44d7c37d265aff8c401bb2b87f33a7aca4042
deleted: sha256:11fb7656bb030b4ccb582540781e79994dbefdc24c2407713be18f613ea2d803
deleted: sha256:bfdeaf35596a52fc922050789fb0a0cba566ce3455e1e9050eeab020fe00eff6
deleted: sha256:3edbce420a30acb007e69490b303684ae7b36bc78b358d9d1dc05ad92356c38c
deleted: sha256:70b2518d2f72d6575845cf0159c97e9511e4163805a56528ad2bd7c64c7c5b00
deleted: sha256:c43535bc88aac739b558d13120c98cc2d59b8fe783c5ca2a73a88500f5bda6ac
untagged: dokku/ci-calendar-pill-yachts-page:latest
deleted: sha256:8f5759fb0d9ce975f6178ee2f39deb521f8a04808b2bd1b23e7c8b4885d8d568
deleted: sha256:4e60f958cb8856ed78a537434ff96b0d942caa937caf686104c78c5587c2defc
deleted: sha256:42fc1067a8571f92f2045d51aa5d742ff019f7624bcdc49df827f7b8cae7354c
deleted: sha256:b759b2257fd1537cb12273ee882b77b43900d50edf62b99571a5c08dd327015f
deleted: sha256:776b61880cd5cf6d77dca1f2d561d086a33e3d348569cf8b6059a6408273b60b
deleted: sha256:ecff6fcb5655d725da328a8041ead281bf236daec317142a66b6b544b39d7fb4
untagged: gliderlabs/herokuish:v0.4.2
untagged: gliderlabs/herokuish@sha256:32cbd10ca939e4fbfba2025a8d554b364f114af87531dc487ca25f4cab92a84e

Total reclaimed space: 3.356GB

Which is a start ... the RAM and Disk are no longer at 100% ...

But it's still not a "solution"!

Re-started the build for the app that we are testing ...

[nelsonic]

Now we have 420Mb of available RAM:

But we still have way too many running docker containers:

Fresh Start!

A complete Docker system clean can now be achieved by linking this with the full prune command covered earlier in this post.

docker container stop $(docker container ls -a -q) && docker system prune -a -f --volumes

Now when we run docker ps there are zero running docker containers.

let's rebuild all the dokku containers: dokku ps:rebuildall

[nelsonic]

Now the CPU, RAM and Disk are back to "normal" levels: