Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build fails with custom directories #27922

Open
nalimilan opened this issue Jul 3, 2018 · 9 comments
Open

Build fails with custom directories #27922

nalimilan opened this issue Jul 3, 2018 · 9 comments
Labels
building Build system, or building Julia or its dependencies upstream The issue is with an upstream dependency, e.g. LLVM

Comments

@nalimilan
Copy link
Member

It appears that the build is completely broken when passing custom directories to prefix, libdir, etc. and build_*. This is from a clean git tree:

# After calling this command repeatedly to build things that don't fail
$ make prefix=/usr bindir=/usr/bin libdir=/usr/lib64 libexecdir=/usr/libexec datarootdir=/usr/share includedir=/usr/include sysconfdir=/etc build_prefix=~/build build_bindir=~/build/usr/bin build_libdir=~/build/usr/lib64 build_private_libdir=~/build/usr/lib64/julia build_libexecdir=~/build/usr/libexec build_datarootdir=~/build/usr/share build_includedir=~/build/usr/include build_sysconfdir=~/build/etc 
make: Circular /home/nalimilan/build/etc/julia <- /home/nalimilan/build/etc/julia dependency dropped.
make: Circular /home/nalimilan/build/tools <- /home/nalimilan/build/tools dependency dropped.
make: Circular /home/nalimilan/build/usr/bin <- /home/nalimilan/build/usr/bin dependency dropped.
make: Circular /home/nalimilan/build/usr/include <- /home/nalimilan/build/usr/include dependency dropped.
make: Circular /home/nalimilan/build/usr/include/julia <- /home/nalimilan/build/usr/include/julia dependency dropped.
make: Circular /home/nalimilan/build/usr/lib64 <- /home/nalimilan/build/usr/lib64 dependency dropped.
make: Circular /home/nalimilan/build/usr/lib64/julia <- /home/nalimilan/build/usr/lib64/julia dependency dropped.
make: Circular /home/nalimilan/build/usr/libexec <- /home/nalimilan/build/usr/libexec dependency dropped.
make: Circular /home/nalimilan/build/usr/share/julia <- /home/nalimilan/build/usr/share/julia dependency dropped.
make: Circular /home/nalimilan/build/usr/share/julia/stdlib <- /home/nalimilan/build/usr/share/julia/stdlib dependency dropped.
make: Circular /home/nalimilan/build/usr/share/man/man1 <- /home/nalimilan/build/usr/share/man/man1 dependency dropped.
make: Circular /home/nalimilan/build/usr/share/julia/base <- /home/nalimilan/build/usr/share/julia/base dependency dropped.
make: Circular /home/nalimilan/build/usr/share/julia/test <- /home/nalimilan/build/usr/share/julia/test dependency dropped.
make: Circular /home/nalimilan/build/usr/share/julia/stdlib/v0.7 <- /home/nalimilan/build/usr/share/julia/stdlib/v0.7 dependency dropped.
make[1]: Circular /home/nalimilan/build/manifest <- /home/nalimilan/build/manifest dependency dropped.
[  4%] Building C object src/CMakeFiles/libssh2.dir/agent.c.o
In file included from /data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:140:0,
                 from /data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/agent.c:40:
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:61:22: error: unknown type name ‘libssh2_rsa_ctx’; did you mean ‘libssh2_socket_t’?
 int _libssh2_rsa_new(libssh2_rsa_ctx ** rsa,
                      ^~~~~~~~~~~~~~~
                      libssh2_socket_t
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:77:30: error: unknown type name ‘libssh2_rsa_ctx’; did you mean ‘libssh2_socket_t’?
 int _libssh2_rsa_new_private(libssh2_rsa_ctx ** rsa,
                              ^~~~~~~~~~~~~~~
                              libssh2_socket_t
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:81:30: error: unknown type name ‘libssh2_rsa_ctx’; did you mean ‘libssh2_socket_t’?
 int _libssh2_rsa_sha1_verify(libssh2_rsa_ctx * rsa,
                              ^~~~~~~~~~~~~~~
                              libssh2_socket_t
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:86:28: error: unknown type name ‘libssh2_rsa_ctx’; did you mean ‘libssh2_socket_t’?
                            libssh2_rsa_ctx * rsactx,
                            ^~~~~~~~~~~~~~~
                            libssh2_socket_t
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:91:41: error: unknown type name ‘libssh2_rsa_ctx’; did you mean ‘libssh2_socket_t’?
 int _libssh2_rsa_new_private_frommemory(libssh2_rsa_ctx ** rsa,
                                         ^~~~~~~~~~~~~~~
                                         libssh2_socket_t
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:123:26: error: unknown type name ‘_libssh2_cipher_ctx’; did you mean ‘libssh2_socket_t’?
 int _libssh2_cipher_init(_libssh2_cipher_ctx * h,
                          ^~~~~~~~~~~~~~~~~~~
                          libssh2_socket_t
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:124:26: error: unknown type name ‘_libssh2_cipher_type’; did you mean ‘libssh2_socket_t’?
                          _libssh2_cipher_type(algo),
                          ^~~~~~~~~~~~~~~~~~~~
                          libssh2_socket_t
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:128:27: error: unknown type name ‘_libssh2_cipher_ctx’; did you mean ‘libssh2_socket_t’?
 int _libssh2_cipher_crypt(_libssh2_cipher_ctx * ctx,
                           ^~~~~~~~~~~~~~~~~~~
                           libssh2_socket_t
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/crypto.h:129:27: error: unknown type name ‘_libssh2_cipher_type’; did you mean ‘libssh2_socket_t’?
                           _libssh2_cipher_type(algo),
                           ^~~~~~~~~~~~~~~~~~~~
                           libssh2_socket_t
In file included from /data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/agent.c:40:0:
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:157:28: error: ‘SHA256_DIGEST_LENGTH’ undeclared here (not in a function); did you mean ‘MAX_SHA_DIGEST_LEN’?
 #define MAX_SHA_DIGEST_LEN SHA256_DIGEST_LENGTH
                            ^
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:245:30: note: in expansion of macro ‘MAX_SHA_DIGEST_LEN’
     unsigned char h_sig_comp[MAX_SHA_DIGEST_LEN];
                              ^~~~~~~~~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:250:5: error: unknown type name ‘_libssh2_bn_ctx’
     _libssh2_bn_ctx *ctx;
     ^~~~~~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:251:5: error: unknown type name ‘_libssh2_bn’
     _libssh2_bn *x;
     ^~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:252:5: error: unknown type name ‘_libssh2_bn’
     _libssh2_bn *e;
     ^~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:253:5: error: unknown type name ‘_libssh2_bn’
     _libssh2_bn *f;
     ^~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:254:5: error: unknown type name ‘_libssh2_bn’
     _libssh2_bn *k;
     ^~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:272:5: error: unknown type name ‘_libssh2_bn’
     _libssh2_bn *p;             /* SSH2 defined value (p_value) */
     ^~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:273:5: error: unknown type name ‘_libssh2_bn’
     _libssh2_bn *g;             /* SSH2 defined value (2) */
     ^~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:609:39: error: ‘SHA_DIGEST_LENGTH’ undeclared here (not in a function); did you mean ‘SHA256_DIGEST_LENGTH’?
     unsigned char server_hostkey_sha1[SHA_DIGEST_LENGTH];
                                       ^~~~~~~~~~~~~~~~~
                                       SHA256_DIGEST_LENGTH
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/libssh2_priv.h:905:7: error: expected specifier-qualifier-list before ‘_libssh2_cipher_type’
       _libssh2_cipher_type(algo);
       ^~~~~~~~~~~~~~~~~~~~
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/agent.c: In function ‘agent_connect_unix’:
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/agent.c:150:12: warning: implicit declaration of function ‘getenv’; did you mean ‘getopt’? [-Wimplicit-function-declaration]
     path = getenv("SSH_AUTH_SOCK");
            ^~~~~~
            getopt
/data/nalimilan/julia/deps/srccache/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/src/agent.c:150:10: warning: assignment makes pointer from integer without a cast [-Wint-conversion]
     path = getenv("SSH_AUTH_SOCK");
          ^
src/CMakeFiles/libssh2.dir/build.make:62: recipe for target 'src/CMakeFiles/libssh2.dir/agent.c.o' failed
make[5]: *** [src/CMakeFiles/libssh2.dir/agent.c.o] Error 1
CMakeFiles/Makefile2:85: recipe for target 'src/CMakeFiles/libssh2.dir/all' failed
make[4]: *** [src/CMakeFiles/libssh2.dir/all] Error 2
CMakeFiles/Makefile2:97: recipe for target 'src/CMakeFiles/libssh2.dir/rule' failed
make[3]: *** [src/CMakeFiles/libssh2.dir/rule] Error 2
Makefile:197: recipe for target 'libssh2' failed
make[2]: *** [libssh2] Error 2
/data/nalimilan/julia/deps/libssh2.mk:44: recipe for target 'scratch/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/build-compiled' failed
make[1]: *** [scratch/libssh2-30e9c1347e3b8baa2951db612f05e6d87fc8e2f2/build-compiled] Error 2
Makefile:60: recipe for target 'julia-deps' failed
make: *** [julia-deps] Error 2

When working around the problem by using system libraries, another problem happens with LLVM:

$ make prefix=/usr bindir=/usr/bin libdir=/usr/lib64 libexecdir=/usr/libexec datarootdir=/usr/share includedir=/usr/include sysconfdir=/etc build_prefix=~/build build_bindir=~/build/usr/bin build_libdir=~/build/usr/lib64 build_private_libdir=~/build/usr/lib64/julia build_libexecdir=~/build/usr/libexec build_datarootdir=~/build/usr/share build_includedir=~/build/usr/include build_sysconfdir=~/build/etc USE_SYSTEM_LIBSSH2=1 USE_SYSTEM_MBEDTLS=1
[...]
    LINK /home/nalimilan/build/usr/lib64/libjulia.so.0.7.0
    CC ui/repl.o
    LINK /home/nalimilan/build/usr/bin/julia
/usr/bin/ld: warning: libLLVM-6.0.so, needed by /home/nalimilan/build/usr/lib64/libjulia.so, not found (try using -rpath or -rpath-link)
/home/nalimilan/build/usr/lib64/libjulia.so: undefined reference to `llvm::raw_fd_ostream::~raw_fd_ostream()@JL_LLVM_6.0'
/home/nalimilan/build/usr/lib64/libjulia.so: undefined reference to `llvm::FunctionPass::getPotentialPassManagerType() const@JL_LLVM_6.0'
[...]

I haven't noticed this before because I use system libraries when building the RPM package, but I suspect a new problem has recently been introduced when custom directories are used. Unfortunately it's hard to debug because of these failures.
@nalimilan nalimilan added the building Build system, or building Julia or its dependencies label Jul 3, 2018
@vtjnash
Copy link
Member

vtjnash commented Jul 3, 2018

USE_SYSTEM_LIBSSH2=1

From the build logs, it looks it doesn't like the copy of libssh2 you provided. Maybe just let it build this?

@nalimilan
Copy link
Member Author

Actually, I only passed USE_SYSTEM_LIBSSH2=1 in the second command, i.e. the one which fails because of LLVM. The first one, which fails because of libssh2, has no USE_SYSTEM_LIBSSH2.

BTW, I ran this on Anubis, so it should be easy to replicate.

@nalimilan
Copy link
Member Author

Bump. This makes it impossible to update distribution packages to Julia 0.7/1.0.

@nalimilan
Copy link
Member Author

Actually that's due to the fact that LLVM completely ignores environment variables to choose libdir and includedir. I worked around it by setting LLVM_CMAKE += -DLLVM_LIBDIR_SUFFIX=64 on 64-bit. This can only be fixed upstream, so I've tried to revive an old PR of mine there: https://reviews.llvm.org/D28234

@vtjnash vtjnash added the upstream The issue is with an upstream dependency, e.g. LLVM label Jan 1, 2019
vtjnash added a commit that referenced this issue Jan 1, 2019
@vtjnash
make: fix out-of-tree build
c259cbe 
This rule was unused, and wouldn't work for an out-of-tree build anyways.

ref #27922
vtjnash added a commit that referenced this issue Jan 1, 2019
@vtjnash
make: fix out-of-tree build
0c36201 
This rule was unused, and wouldn't work for an out-of-tree build anyways.

ref #27922
vtjnash added a commit that referenced this issue Jan 2, 2019
@vtjnash
make: fix out-of-tree build
178dd2d 
This rule was unused, and wouldn't work for an out-of-tree build anyways.

ref #27922
vtjnash added a commit that referenced this issue Jan 2, 2019
@vtjnash
make: fix out-of-tree build
324154f 
This rule was unused, and wouldn't work for an out-of-tree build anyways.

ref #27922
vtjnash added a commit that referenced this issue Jan 4, 2019
@vtjnash
make: fix out-of-tree build
c1dafb1 
This rule was unused, and wouldn't work for an out-of-tree build anyways.

ref #27922
@cdluminate
Copy link
Contributor

I encountered the same problem too when trying to switch to Julia's vendored LLVM for the Debian package. The failure stems from the lack of MULTIARCH* support from LLVM's cmake build... In this case the compiler fails to find libLLVM-6.0.so in either usr/lib/<triplet>/ or usr/lib/<triplet>/julia when linking programs against usr/lib/<triplet>/libjulia.so.* because llvm is installed to usr/lib/libLLVM*.so.

@nalimilan
Copy link
Member Author

Could it be related to #29720 then?

FWIW, I'm not using any triplet in the reproducer above.

@cdluminate
Copy link
Contributor

cdluminate commented Jan 16, 2019 •
edited
Loading

Not sure. #29720 looks like a conda packaging bug. And for Debian this simple hack just works

--- julia.orig/deps/llvm.mk
+++ julia/deps/llvm.mk
@@ -560,6 +560,7 @@ ifeq ($(OS),Darwin)
 # https://github.com/JuliaLang/julia/issues/29981
 LLVM_INSTALL += && ln -s libLLVM.dylib $2$$(build_shlibdir)/libLLVM-$$(LLVM_VER_SHORT).dylib
 endif
+LLVM_INSTALL += && find $2 -type f,l -name 'libLLVM*so*' -print -exec install -Dv '{}' $(shell pwd)/../$$(libdir)/julia/ \;
 
 $(eval $(call staged-install,llvm,llvm-$$(LLVM_VER)/build_$$(LLVM_BUILDTYPE), \
 	LLVM_INSTALL,,,))

@ViralBShah
Copy link
Member

Is it ok to close this one?

@nalimilan
Copy link
Member Author

The build still fails when running the command from the original description on a clean git master (though errors are different now).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
building Build system, or building Julia or its dependencies upstream The issue is with an upstream dependency, e.g. LLVM
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vtjnash @ViralBShah @nalimilan @cdluminate