Infinite artifact re-deployment on a Windows 10 docker host

[MihaStemberger]

Hello!

I am having an issue that occurs only when running Docker on a Windows host machine. Artifacts I want to deploy to Wildfly never stop redeploying. The way deployment artifacts are passed to the container, is via a volume, that connects to /opt/jboss/wildfly/standalone/deployments.

docker-compose.yml file:

version: "3.7"
services:
  wildfly:
    image: "jboss/wildfly:20.0.1.Final"
    volumes:
      - ./_deployables:/opt/jboss/wildfly/standalone/deployments

I made an example project, that can be found here: https://github.com/MihaStemberger/docker.git

From what i have managed to debug, the issue is a wrongly set last modified timestamp because the last 3 digits of the timestamp round down thus making the timestamps different.

Here is an image of values while debugging wildfly-core-parent 12.0.3.Final version at FileSystemDeploymentService.java:1482 And this this is the stat result of the files that are made before and after the breakpoint.

sh-4.2$ cd /opt/jboss/wildfly/standalone/deployments/
sh-4.2$ ls -l
total 0
sh-4.2$ ls -l
total 4

---------------------
Before breakpoint
---------------------
-rwxrwxrwx 1 root  root  2494 Aug  7 13:25 wildfly_redeploy.war
-rw-r--r-- 1 jboss jboss   20 Aug  7 13:25 wildfly_redeploy.war.deployed
-rw-r--r-- 1 jboss jboss   20 Aug  7 13:25 wildfly_redeploy.war.isdeploying
sh-4.2$ stat wildfly_redeploy.war wildfly_redeploy.war.deployed  wildfly_redeploy.war.isdeploying

  File: 'wildfly_redeploy.war'
  Size: 2494            Blocks: 8          IO Block: 4096   regular file
Device: 33h/51d Inode: 1970324838626163  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-08-07 13:25:31.793848400 +0000
Modify: 2021-08-07 13:25:31.786287800 +0000
Change: 2021-08-07 13:25:31.786287800 +0000
 Birth: -

  File: 'wildfly_redeploy.war.deployed'
  Size: 20              Blocks: 0          IO Block: 4096   regular file
Device: 33h/51d Inode: 1970324838626165  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   jboss)   Gid: ( 1000/   jboss)
Access: 2021-08-07 13:25:35.867932900 +0000
Modify: 2021-08-07 13:25:35.559377700 +0000
Change: 2021-08-07 13:25:35.559377700 +0000
 Birth: -

  File: 'wildfly_redeploy.war.isdeploying'
  Size: 20              Blocks: 0          IO Block: 4096   regular file
Device: 33h/51d Inode: 1688849861915508  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   jboss)   Gid: ( 1000/   jboss)
Access: 2021-08-07 13:25:35.867932900 +0000
Modify: 2021-08-07 13:25:33.107215700 +0000
Change: 2021-08-07 13:25:33.107215700 +0000
 Birth: -

---------------------
After breakpoint
---------------------
sh-4.2$ stat wildfly_redeploy.war wildfly_redeploy.war.deployed  wildfly_redeploy.war.isdeploying

  File: 'wildfly_redeploy.war'
  Size: 2494            Blocks: 8          IO Block: 4096   regular file
Device: 33h/51d Inode: 1970324838626163  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-08-07 13:25:31.793848400 +0000
Modify: 2021-08-07 13:25:31.786287800 +0000
Change: 2021-08-07 13:25:31.786287800 +0000
 Birth: -

  File: 'wildfly_redeploy.war.deployed'
  Size: 20              Blocks: 0          IO Block: 4096   regular file
Device: 33h/51d Inode: 1970324838626165  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   jboss)   Gid: ( 1000/   jboss)
Access: 2021-08-07 13:25:35.000000000 +0000
Modify: 2021-08-07 13:25:31.000000000 +0000
Change: 2021-08-07 13:26:38.946329100 +0000
 Birth: -

  File: 'wildfly_redeploy.war.isdeploying'
  Size: 20              Blocks: 0          IO Block: 4096   regular file
Device: 33h/51d Inode: 1688849861915508  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   jboss)   Gid: ( 1000/   jboss)
Access: 2021-08-07 13:25:35.867932900 +0000
Modify: 2021-08-07 13:25:33.107215700 +0000
Change: 2021-08-07 13:25:33.107215700 +0000
 Birth: -
sh-4.2$

This causes FileSystemDeploymentService.java:865 to be true and invoke a re-deploy.

My docker info:

λ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-beta.6)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 12.38GiB
 Name: docker-desktop
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false

Any help would be greatly appreciated.

Edit1: this issue is also present if latest jboss/wildfly is used (as of Aug. 07.2021)

[tlf30]

I am having the same issue. If there is information I can provide to assist with fixing this, please let me know.

Edit: I am using WildFly 24.

[tlf30]

Just tested with WildFly 25, issue still exists.

[slominskir]

Issue is on Windows 11 too. With Docker Desktop 4.3.2 (72729) - currently the newest version available.

Also: Using WSL 2 based engine.

[tlf30]

I can confirm, I too can replicate the issue on Windows 10 and 11 with 4.3.2. I have not tested 4.4.2 yet.

[slominskir]

Same issue with Docker Desktop 4.4.3

[slominskir]

Probably worth mentioning Docker and Microsoft advise against mounting a Windows directory into a Linux Container using WSL 2. Neither go as far as to say it won't work, just that performance will be bad. See: https://docs.docker.com/desktop/windows/wsl/#best-practices. Using WSL 1 doesn't appear to be an option anymore on Windows 11 Home edition, so that leaves their other recommendation of installing projects in the WSL filesystem directly. This recommended approach involves a bit of indirection and assumes your IDE supports remote WSL directories (Microsoft Visual Studio Code and IntelliJ Idea both do). I've confirmed it does solve this problem though so is a reasonable, albeit annoying, workaround if your IDE supports it.

[davdr]

• I've encountered a similar problem. I believe the root cause is this WSL 2 issue: 9p client truncates modified time nanoseconds field to zero when setting it microsoft/WSL#7581 (it's confirmed but still open at the time of writing).
• You also be need to be running Java 9 or higher to be affected, because older versions of Java return file timestamps with second-precision only (details: https://stackoverflow.com/questions/24804618/get-file-mtime-with-millisecond-resolution-from-java). These older Java versions, as a side effect, thus make the deployment scanner's timestamp comparison "resilient" against the WSL 2 bug...

[brunocosta91]

As a workaround you can touch the generated file. (Declare this plugin after your maven-war-plugin/maven-ear-plugin/etc )

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-antrun-plugin</artifactId>
    <version>3.1.0</version>
    <executions>
        <execution>
            <id>update-modified-time</id>
            <phase>package</phase>
            <goals>
                <goal>
                    run
                </goal>
            </goals>
            <configuration>
                <target>
                    <touch file="target/${application.name}.${packaging}" datetime="now"/>
                </target>
            </configuration>
        </execution>
    </executions>
</plugin>

or run this command from bash before copy file to deployment folder:

touch -t $(date "+%Y%m%d%H%M.%S") <application-name.packaging>

[fjoubaud]

I got the same issue under Rancher Desktop (right now 1.20) and previously Docker for Windows 10 when mounting the volumes from Windows. We are on Java 8 with JBoss EAP 7.4.19. Wsl is 2.19.
According to the JBoss source code (org.jboss.as.server.deployment.scanner.FileSystemDeploymentService), last modified timestamp comparison is still used in Wildfly 38 so the issue is still here.

One fix in JBoss could be to store the Last Modified timestamp of the war file in the war file.deploy file and rely on it afterwards when the scanner performs the timestamp comparison to know if the war file must be reloaded.
That way there is no risk of millisecond truncation whatsoever.

To complete the list of workarounds, another one - little bit cumbersome but it does not impact your build - is to mount the volumes inside another container (which could therefore act as a file server for backups). Perform the touch timestamp (to set the 000 ms) upon container startup with the post_start instruction (requires docker-compose v30+) :

post_start:
  - command: touch -a -m -t 202512180130.000 <your first war file>
    user: root
  - command: touch -a -m -t 202512180130.000 <your other war file>
    user: root

Your Jboss container will be started afterwards thanks to an health check status (to make sure that the touch have been performed).