Skip to content
This repository was archived by the owner on Aug 31, 2023. It is now read-only.

Snapshot build (2022.01.15 ~ 2022.01.17) failed #267

Closed
Atry opened this issue Jan 17, 2022 · 12 comments
Closed

Snapshot build (2022.01.15 ~ 2022.01.17) failed #267

Atry opened this issue Jan 17, 2022 · 12 comments

Comments

@Atry
Copy link
Contributor

Atry commented Jan 17, 2022

No description provided.

@Atry
Copy link
Contributor Author

Atry commented Jan 17, 2022 •
edited
Loading

What is interesting is that the first failure log is build-failure-2022.01.14-debian-11-bullseye-.cloud-init-output.log, while the snapshot build 2022.01.14 was not failed according to https://hhvm.com/api/build-status/2022.01.14

Due to #264 ?

@Atry
Copy link
Contributor Author

Atry commented Jan 17, 2022 •
edited
Loading

The error in build-failure-2022.01.14-debian-11-bullseye-.cloud-init-output.log is different from build-failure-2022.01.17-debian-11-bullseye-.cloud-init-output.log.

build-failure-2022.01.14-debian-11-bullseye-.cloud-init-output.log

cd /tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/hphp/parser && /usr/bin/c++ -DDEFAULT_CONFIG_DIR=\"/etc/hhvm/\" -DENABLE_FASTCGI=1 -DHAVE_CURL_MULTI_WAIT -DHAVE_ELF_GETSHDRSTRNDX -DHAVE_FT2BUILD -DHAVE_INTTYPES_H -DHAVE_LIBDL -DHAVE_LIBSODIUM=1 -DHAVE_SYSTEM_TZDATA -DHHVM -DHHVM_DYNAMIC_EXTENSION_DIR=\"/opt/hhvm/2022.01.14/lib/x86_64-linux-gnu/hhvm/extensions\" -DHPHP_OSS=1 -DLIBDWARF_CONST_NAME -DNDEBUG -DNO_TCMALLOC=1 -DOPENSSL_NO_RAND_EGD -DPACKAGE=hhvm -DPACKAGE_VERSION=Release -DTHRIFT_MUTEX_EMULATE_PTHREAD_TIMEDLOCK -DUSE_CMAKE -DUSE_EDITLINE -DUSE_JEMALLOC=1 -DU_USING_ICU_NAMESPACE=1 -D_GNU_SOURCE -D_PTHREADS=1 -D_REENTRANT=1 -D__STDC_FORMAT_MACROS -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu -I/usr/include/libxml2 -I/usr/include/libdwarf -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/hphp -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/forks/libmbfl -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/forks/libmbfl/mbfl -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/forks/libmbfl/filters -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/fatal -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/libafdt/src/src -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/fastlz/src -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14 -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/brotli/bundled_brotli-prefix/include -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/third-party/folly/include -I/usr/include/libiberty -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -I/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -g -O2 -ffile-prefix-map=/tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wno-error=array-bounds -Wno-error=switch -Wno-attributes -Wno-deprecated -Wno-invalid-offsetof -Wno-register -Wno-sign-compare -Wno-strict-aliasing -Wno-unused-function -Wno-unused-local-typedefs -Wno-unused-result -Wno-write-strings -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-bool-compare -ffunction-sections -fno-delete-null-pointer-checks -DFOLLY_HAVE_MALLOC_H -std=gnu++1z -fno-omit-frame-pointer -fno-operator-names -Wall -Woverloaded-virtual -Werror=format-security -fdata-sections -fno-gcse -fno-canonical-system-headers -Wvla -Wno-misleading-indentation -mcrc32 -Wno-error=array-bounds -Wno-error=switch -Wno-attributes -Wno-deprecated -Wno-invalid-offsetof -Wno-register -Wno-sign-compare -Wno-strict-aliasing -Wno-unused-function -Wno-unused-local-typedefs -Wno-unused-result -Wno-write-strings -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-bool-compare -ffunction-sections -fno-delete-null-pointer-checks -DFOLLY_HAVE_MALLOC_H -std=gnu++1z -fno-omit-frame-pointer -fno-operator-names -Wall -Woverloaded-virtual -Werror=format-security -fdata-sections -fno-gcse -fno-canonical-system-headers -Wvla -Wno-misleading-indentation -mcrc32 -O2 -g -DNDEBUG -o CMakeFiles/hphp_parser.dir/parser.cpp.o -c /tmp/hhvmpkg.ojwXzgIv/hhvm-nightly-2022.01.14/hphp/parser/parser.cpp
make[5]: *** [Makefile.dune:58: build-hack] Error 1

build-failure-2022.01.17-debian-11-bullseye-.cloud-init-output.log

cd /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/cpp2 && /usr/bin/c++ -DFMT_LOCALE -DTHRIFT_HAVE_LIBSNAPPY=0 -I/tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift -I/tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -isystem /usr/include/libdwarf -isystem /usr/include/libiberty -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/wangle/bundled_wangle-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/fizz/bundled_fizz-prefix/include -I/usr/include -O2 -g -DNDEBUG -std=c++17 -o CMakeFiles/thriftprotocol.dir/__/thrift/gen-cpp2/reflection_types.cpp.o -c /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/thrift/gen-cpp2/reflection_types.cpp
The bug is not reproducible, so it is likely a hardware or OS problem.
make[7]: *** [thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/build.make:173: thrift/lib/cpp2/CMakeFiles/thriftprotocol.dir/protocol/TableBasedSerializer.cpp.o] Error 1

@Atry
Copy link
Contributor Author

Atry commented Jan 17, 2022

I successfully executed the failure command after loading the failure image.

cd /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/cpp2 && /usr/bin/c++ -DFMT_LOCALE -DTHRIFT_HAVE_LIBSNAPPY=0 -I/tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift -I/tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -isystem /usr/include/libdwarf -isystem /usr/include/libiberty -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/wangle/bundled_wangle-prefix/include -isystem /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/fizz/bundled_fizz-prefix/include -I/usr/include -O2 -g -DNDEBUG -std=c++17 -o CMakeFiles/thriftprotocol.dir/__/thrift/gen-cpp2/reflection_types.cpp.o -c /tmp/hhvmpkg.VycOBDbg/hhvm-nightly-2022.01.17/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/thrift/gen-cpp2/reflection_types.cpp

So this is not a reproducible issue.

@fredemmott
Copy link
Contributor

What is interesting is that the first failure log is build-failure-2022.01.14-debian-11-bullseye-.cloud-init-output.log, while the snapshot build 2022.01.14 was not failed according to https://hhvm.com/api/build-status/2022.01.14

We automatically retry on failure; the failure log shows an attempt failed, but we then tried again on another machine and it worked

It's worth checking the dmesg logs etc to see if there's anything correlating to the non-reproducible error (it does happen occassionally, especially on 11, but without a reliable repro)

But a single attempt failing isn't necessarily an issue worth investigating

@Atry
Copy link
Contributor Author

Atry commented Jan 19, 2022 •
edited
Loading

I killed the hanging EC2 instance and terminated all running step functions. However, the 2022.01.18 build still failed due to gcc crashing.

        "debian-11-bullseye": {
          "MakeBinaryPackage": {
            "failure": {
              "Error": null,
              "Cause": "{\"ec2\":\"i-072ac8413970a007d\",\"time_sec\":\"1824\"}"
            }
          }
        },


cd /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/cpp2 && /usr/bin/c++ -DFMT_LOCALE -DTHRIFT_HAVE_LIBSNAPPY=0 -I/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift -I/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -isystem /usr/include/libdwarf -isystem /usr/include/libiberty -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/wangle/bundled_wangle-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/fizz/bundled_fizz-prefix/include -I/usr/include -O2 -g -DNDEBUG -std=c++17 -o CMakeFiles/thriftprotocol.dir/__/thrift/gen-cpp2/reflection_types.cpp.o -c /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/thrift/gen-cpp2/reflection_types.cpp
Preprocessed source stored into /tmp/cc8x2P1d.out file, please attach this to your bugreport.
=== BEGIN GCC DUMP ===
139073: // Target: x86_64-linux-gnu
139073: // Configured with: ../src/configure -v --with-pkgversion='Debian 10.2.1-6' --with-bugurl=file:///usr/share/doc/gcc-10/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-10 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-10-Km9U7s/gcc-10-10.2.1/debian/tmp-gcn/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-mutex
139073: // Thread model: posix
139073: // Supported LTO compression algorithms: zlib zstd
139073: // gcc version 10.2.1 20210110 (Debian 10.2.1-6) 
139073: // 
139073: // /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp: In substitution of 'template<class T> constexpr T apache::thrift::detail::identity(T) [with T = <missing>]':
139073: // /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:37:1:   required from here
139073: // /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:32:67: internal compiler error: Segmentation fault
139073: //    32 |       reinterpret_cast<VoidFuncPtr>(identity(set<Type, ThriftType>)),  \
139073: //       |                                                                   ^
139073: // /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp:37:1: note: in expansion of macro 'THRIFT_DEFINE_PRIMITIVE_TYPE_TO_INFO'
139073: //    37 | THRIFT_DEFINE_PRIMITIVE_TYPE_TO_INFO(
139073: //       | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
139073: // Please submit a full bug report,
139073: // with preprocessed source if appropriate.
139073: // See <file:///usr/share/doc/gcc-10/README.Bugs> for instructions.
139073: 
139073: // /usr/lib/gcc/x86_64-linux-gnu/10/cc1plus -quiet -I /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift -I /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build -I /usr/include -imultiarch x86_64-linux-gnu -D_GNU_SOURCE -D FMT_LOCALE -D THRIFT_HAVE_LIBSNAPPY=0 -D NDEBUG -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/zstd/bundled_zstd-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/fmt/bundled_fmt-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/double-conversion/bundled_double-conversion-prefix/include -isystem /usr/include/libdwarf -isystem /usr/include/libiberty -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/folly/bundled_folly-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/wangle/bundled_wangle-prefix/include -isystem /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/fizz/bundled_fizz-prefix/include /tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp -quiet -dumpbase TableBasedSerializer.cpp -mtune=generic -march=x86-64 -auxbase-strip CMakeFiles/thriftprotocol.dir/protocol/TableBasedSerializer.cpp.o -g -O2 -std=c++17 -fasynchronous-unwind-tables -o - -frandom-seed=0 -fdump-noaddr
139073: 
139073: # 1 "/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp"
139073: # 1 "/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift-build/thrift/lib/cpp2//"
139073: # 1 "<built-in>"
139073: # 1 "<command-line>"
139073: # 1 "/usr/include/stdc-predef.h" 1 3 4
139073: # 1 "<command-line>" 2
139073: # 1 "/tmp/hhvmpkg.VYqs01Y6/hhvm-nightly-2022.01.19/obj-x86_64-linux-gnu/third-party/thrift/bundled_thrift-prefix/src/bundled_thrift/thrift/lib/cpp2/protocol/TableBasedSerializer.cpp"

@fredemmott
Copy link
Contributor

let's kick off another build just for 11, in case it's intermittent (to hopefully unblock this weeks' release), but also try and pull up the logs for what happened there. >2h is ok, > 3h is not.

@Atry
Copy link
Contributor Author

Atry commented Jan 19, 2022

Sorry, it's not a timeout. It's due to gcc crash.

@Atry
Copy link
Contributor Author

Atry commented Jan 19, 2022

Why didn't the step function retry?

@fredemmott
Copy link
Contributor

Is it possible that the step function did retry, but GCC crashed every time?

A GCC crash is not neccessarily a GCC bug - e.g. it could be the kernel killing it due to running out of memory. Needs correlating with syslog/dmesg

@Atry
Copy link
Contributor Author

Atry commented Jan 19, 2022 •
edited
Loading

Is it possible that the step function did retry, but GCC crashed every time?

I think so. The previous failure log file build-failure-2022.01.19-debian-11-bullseye-.cloud-init-output.log in S3 was overridden by the last retry, because there is a bug in file name generation, missing the EC2 instance name part.

@Atry
Copy link
Contributor Author

Atry commented Jan 20, 2022

I triggered bin/build-on-aws 2022.01.19 debian-11-bullseye MakeBinaryPackage and the step functions succeeded. I am not sure why the previous retry did not succeed.

@Atry
Copy link
Contributor Author

Atry commented Jan 20, 2022

Now triggering bin/build-on-aws 2022.01.19 debian-11-bullseye PublishBinaryPackages PublishDockerImages

@Atry Atry closed this as completed in f363a03 Jan 25, 2022
fredemmott pushed a commit that referenced this issue Mar 30, 2022
@Atry @fredemmott
Merge the init_script update-repos.sh into the the task script update…
5bf42e0 
…-repos (#269)

fixes #267
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fredemmott @Atry