Lack of cache increases Kokoro flakiness

[ejona86]

This is a grab-bag for any time Kokoro fails during downloading resources or similar. We cache dependency downloads on our other CIs, so such problems should be more rare. This issue should only be for flakes where caches on Travis CI or Jenkins would have been able to circumvent the failure.

[ejona86]

https://kokoro.corp.google.com/job/grpc/job/java/job/master/job/windows/job/continuous/job/windows/51/

:grpc-compiler:generateTestLiteProto
Download https://repo1.maven.org/maven2/com/google/protobuf/protoc-gen-javalite/3.0.0/protoc-gen-javalite-3.0.0.pom
Download https://repo1.maven.org/maven2/com/google/protobuf/protoc-gen-javalite/3.0.0/protoc-gen-javalite-3.0.0-windows-x86_64.exe
:grpc-compiler:generateTestLiteProto FAILED
99 actionable tasks: 99 executed

FAILURE: Build failed with an exception.

* What went wrong:
Could not resolve all files for configuration ':grpc-compiler:protobufToolsLocator_javalite'.
> Could not download protoc-gen-javalite-windows-x86_64.exe (com.google.protobuf:protoc-gen-javalite:3.0.0)
   > Could not get resource 'https://repo1.maven.org/maven2/com/google/protobuf/protoc-gen-javalite/3.0.0/protoc-gen-javalite-3.0.0-windows-x86_64.exe'.
      > Connection reset

[ejona86]

https://kokoro.corp.google.com/job/grpc/job/java/job/v1.4.x/job/windows/job/release/job/windows/7/

Execution failed for task ':grpc-context:compileJava'.
> Could not resolve all files for configuration ':grpc-context:errorprone'.
   > Could not download javac.jar (com.google.errorprone:javac:9-dev-r3297-4)
      > Could not get resource 'https://repo1.maven.org/maven2/com/google/errorprone/javac/9-dev-r3297-4/javac-9-dev-r3297-4.jar'.
         > Connection reset

[ejona86]

https://kokoro.corp.google.com/job/grpc/job/java/job/master/job/windows/job/continuous/job/windows/92/

A problem occurred configuring project ':grpc-all'.
> Could not resolve all files for configuration ':grpc-all:classpath'.
   > Could not download xercesImpl.jar (xerces:xercesImpl:2.9.1)
      > Could not get resource 'https://repo1.maven.org/maven2/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar'.
         > Connection reset

[ejona86]

https://kokoro.corp.google.com/job/grpc/job/java/job/master/job/windows/job/continuous/job/windows/117/

Execution failed for task ':grpc-context:compileJava'.
> Could not resolve all files for configuration ':grpc-context:errorprone'.
   > Could not download error_prone_core.jar (com.google.errorprone:error_prone_core:2.0.21)
      > Could not get resource 'https://repo1.maven.org/maven2/com/google/errorprone/error_prone_core/2.0.21/error_prone_core-2.0.21.jar'.
         > Connection reset

[ejona86]

https://kokoro.corp.google.com/job/grpc/job/java/job/master/job/windows/job/continuous/job/windows/127/

A problem occurred configuring root project 'grpc'.
> Could not resolve all files for configuration ':classpath'.
   > Could not resolve net.ltgt.gradle:gradle-errorprone-plugin:0.0.11.
     Required by:
         project :
      > Could not resolve net.ltgt.gradle:gradle-errorprone-plugin:0.0.11.
         > Could not get resource 'https://plugins.gradle.org/m2/net/ltgt/gradle/gradle-errorprone-plugin/0.0.11/gradle-errorprone-plugin-0.0.11.pom'.
            > Could not GET 'https://plugins.gradle.org/m2/net/ltgt/gradle/gradle-errorprone-plugin/0.0.11/gradle-errorprone-plugin-0.0.11.pom'. Received status code 500 from server: Internal Server Error

[ejona86]

https://kokoro2.corp.google.com/job/grpc/job/java/job/master/job/windows/job/continuous/job/windows/40/

A problem occurred evaluating root project 'grpc'.
> Could not resolve all files for configuration ':grpc-core:detachedConfiguration2'.
   > Could not resolve io.grpc:grpc-core:1.6.1.
     Required by:
         project :grpc-core
      > Could not resolve io.grpc:grpc-core:1.6.1.
         > Could not get resource 'https://repo1.maven.org/maven2/io/grpc/grpc-core/1.6.1/grpc-core-1.6.1.pom'.
            > Could not GET 'https://repo1.maven.org/maven2/io/grpc/grpc-core/1.6.1/grpc-core-1.6.1.pom'.
               > Read timed out

[zpencer]

Would switching to a google operated mirror help a little with the connection flakiness?

[ejona86]

If the mirror is near GCP, maybe. To me the problem is assuming that networks and services are 100% reliable. But maybe swapping the mirror improves the situation even if it doesn't fix the problem.

[ejona86]

https://kokoro2.corp.google.com/job/grpc/job/java/job/master/job/macos/job/continuous/job/macos/31/

Could not resolve all files for configuration ':grpc-netty:compileClasspath'.
> Could not resolve io.netty:netty-codec-http2:[4.1.16.Final].
  Required by:
      project :grpc-netty
   > Could not resolve io.netty:netty-codec-http2:[4.1.16.Final].
      > Failed to list versions for io.netty:netty-codec-http2.
         > Unable to load Maven meta-data from https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml.
            > Could not get resource 'https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml'.
               > Could not GET 'https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml'.
                  > Read timed out

[ejona86]

https://kokoro2.corp.google.com/job/grpc/job/java/job/master/job/macos/job/continuous/job/macos/57/
https://kokoro2.corp.google.com/job/grpc/job/java/job/master/job/windows/job/continuous/job/windows/71/

Multiple errors in the form:

A problem occurred configuring root project 'grpc'.
> Could not resolve all files for configuration ':classpath'.
   > Could not resolve com.google.gradle:osdetector-gradle-plugin:1.4.0.
     Required by:
         project :
      > Could not resolve com.google.gradle:osdetector-gradle-plugin:1.4.0.
         > Could not get resource 'https://repo1.maven.org/maven2/com/google/gradle/osdetector-gradle-plugin/1.4.0/osdetector-gradle-plugin-1.4.0.pom'.
            > Could not GET 'https://repo1.maven.org/maven2/com/google/gradle/osdetector-gradle-plugin/1.4.0/osdetector-gradle-plugin-1.4.0.pom'.
               > Host name 'repo1.maven.org' does not match the certificate subject provided by the peer (CN=a.ssl.fastly.net, O="Fastly, Inc.", L=San Francisco, ST=California, C=US)

Although these are obviously misconfigurations of maven.org, this type of failure would have been less likely if we weren't doing I/O during the build.

[ejona86]

https://kokoro2.corp.google.com/job/grpc/job/java/job/master/job/windows/job/presubmit/job/windows/371/

Could not resolve all files for configuration ':grpc-gae-interop-testing-jdk7:compileClasspath'.
> Could not resolve io.netty:netty-codec-http2:[4.1.16.Final].
  Required by:
      project :grpc-gae-interop-testing-jdk7 > project :grpc-interop-testing > project :grpc-netty
   > Could not resolve io.netty:netty-codec-http2:[4.1.16.Final].
      > Failed to list versions for io.netty:netty-codec-http2.
         > Could not list versions using M2 pattern 'https://maven-central.storage.googleapis.com/[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]'.
            > Could not get resource 'https://maven-central.storage.googleapis.com/io/netty/netty-codec-http2/'.
               > Could not GET 'https://maven-central.storage.googleapis.com/io/netty/netty-codec-http2/'.
                  > Read timed out

[ejona86]

https://kokoro2.corp.google.com/job/grpc/job/java/job/master/job/macos/job/presubmit/job/macos/626/

* What went wrong:
Could not resolve all files for configuration ':grpc-interop-testing:compile'.
> Could not download google-http-client-jackson2.jar (com.google.http-client:google-http-client-jackson2:1.19.0)
   > Could not get resource 'https://repo1.maven.org/maven2/com/google/http-client/google-http-client-jackson2/1.19.0/google-http-client-jackson2-1.19.0.jar'.
      > Could not GET 'https://repo1.maven.org/maven2/com/google/http-client/google-http-client-jackson2/1.19.0/google-http-client-jackson2-1.19.0.jar'.
         > Connect to repo1.maven.org:443 [repo1.maven.org/151.101.52.209] failed: Read timed out

[ejona86]

https://kokoro2.corp.google.com/job/grpc/job/java/job/master/job/macos/16/

Could not resolve all files for configuration ':grpc-gae-interop-testing-jdk7:compileClasspath'.
> Could not resolve io.netty:netty-codec-http2:[4.1.17.Final].
  Required by:
      project :grpc-gae-interop-testing-jdk7 > project :grpc-interop-testing > project :grpc-netty
   > Could not resolve io.netty:netty-codec-http2:[4.1.17.Final].
      > Failed to list versions for io.netty:netty-codec-http2.
         > Unable to load Maven meta-data from https://jcenter.bintray.com/io/netty/netty-codec-http2/maven-metadata.xml.
            > Could not get resource 'https://jcenter.bintray.com/io/netty/netty-codec-http2/maven-metadata.xml'.
               > Could not GET 'https://jcenter.bintray.com/io/netty/netty-codec-http2/maven-metadata.xml'.
                  > jcenter.bintray.com: nodename nor servname provided, or not known