initial version of Julia support

[fdiblen]

See issue #624

[bouweandela]

After PR #652 is merged with the version2_development branch, a test should be added to this PR demonstrating that it is possible to run a julia diagnostic without data.

Test diagnostic code should be added in tests/integration/test_diagnostic_run.py

CircleCI configuration (in .circleci/config.yml) should be updated so 'python2_install', 'python3_install' and 'develop' and 'conda_install' installation test jobs also install julia.

[mattiarighi]

@fdiblen can you please finalize this PR? We need to merge it soon since other PRs depends on it.
👍

[fdiblen]

@mattiarighi I think the R support is more or less ready. I will work on this now to finalize it.

[bouweandela]

And don't forget to update the conda build in meta.yml.

[jhardenberg]

I would like to specify a recent julia version (1.0.2) in environment.yml and to update setup.jl to allow installing also not-registered packages from github (which requires Pkg.clone instead of Pkg.add). Should I commit directly to the version2_development_Julia_support branch or make a separate branch?

[bouweandela]

@jhardenberg You can specify the version in environment.yml and meta.yml. Make sure to do it in both files. It is fine to do it in this branch. I agree that we need at least version 1.

I strongly prefer to only install packages from the standard package repository, because we then we can expect the quality to be reasonable. Installing directly from github is not allowed for other languages either for this reason.

[jhardenberg]

I strongly prefer to only install packages from the standard package repository, because we then we can expect the quality to be reasonable. Installing directly from github is not allowed for other languages either for this reason.

In principle I totally agree, but it could be a feature which is useful for development. There is no need then to actually include any packages from github in julia_requirements.txt.

The reason is actually my RainFARM package ( https://github.com/jhardenberg/RainFARM.jl ) which in the past I never bothered to release. I will do it very soon, but I need a few days to complete the documentation first. In the meantime we need it urgently for Magic.
If you prefer, maybe I could add this change only to theREFACTORING_rainfarm branch, so that it works correctly and can be used for Magic for now, but as I said, I see no harm in having the capability.

[bouweandela]

In the meantime we need it urgently for Magic.
If you prefer, maybe I could add this change only to theREFACTORING_rainfarm branch, so that it works correctly and can be used for Magic for now,

Yes, that should be fine. You can then create a release and change it back before it is merged into the version2_development branch.

I see no harm in having the capability.

Having the capability will make people think that it is ok to use it and encourage pull requests that make esmvaltool unreliable. If you need a different version of a package for development, you can install it manually. The final pull request should depend on released packages.

[jhardenberg]

Should the directory esmvaltool/Julia not be in esmvaltool/install/Julia for consistency with R ?

I have locally done a merge of the latest version2_development branch into version2_development_Julia_support and resolved the conflicts listed below. Should I push it to the repository ?

[jhardenberg]

Adding julia 1.0 to the environment requirements seems to conflict with the requirement for gcc which was added with the R support. In fact conda info reveals that
gcc depends on mpfr >=2.4.2,<4 while julia> 1.0.0 depends on mpfr >=4.0.1,<5.0a0
Is gcc actually crucial for R support ?

[fdiblen]

Adding julia 1.0 to the environment requirements seems to conflict with the requirement for gcc which was added with the R support. In fact conda info reveals that
gcc depends on mpfr >=2.4.2,<4 while julia> 1.0.0 depends on mpfr >=4.0.1,<5.0a0
Is gcc actually crucial for R support ?

There is no gcc package in #631. Compilation of R packages need c/c++ compiler. To compile these packages I used r-rcpp package. When the lintr was being added I also raised an issue about gcc. Maybe we should remove it and add the one Julia needs.

[bouweandela]

Please see this issue #734

The conda gcc package is deprecated and should be replaced by these three packages:

• gcc_linux-64
• gxx_linux-64
• gfortran_linux-64

[jhardenberg]

I repeat this question: Should the directory esmvaltool/Julia not be in esmvaltool/install/Julia for consistency with R ?

[fdiblen]

I repeat this question: Should the directory esmvaltool/Julia not be in esmvaltool/install/Julia for consistency with R ?

@jhardenberg Yes, it should. We had a lot of changes in 'R support branch' since we started Julia support. As you mentioned, it should be moved to esmvaltool/install/Julia.

Please go ahead.

[jhardenberg]

@jhardenberg Yes, it should. We had a lot of changes in 'R support branch' since we started Julia support. As you mentioned, it should be moved to esmvaltool/install/Julia.

Please go ahead.

Ok, done in 802030c

[mattiarighi]

@fdiblen what is the status of this PR?

[bouweandela]

I will probably finish this sometime in the next month or so.

[jhardenberg]

I suggest to stick with julia 0.6.1 for now. While developing the RainFARM diagnostic (see #619 ) and trying to switch to the latest julia v1.0.3 I found that the JuliaCall R library is not compatible with julia >= 1.0.0 from conda-forge. The problem I run into is discussed here JuliaLang/julia#29720 , and has something to do with the conda-forge julia build. I fixed setup.jl so that it works also with julia < v0.7 ( using Pkg would not work, since Pkg was internal before v0.7)

[jhardenberg]

Builds are failing because lintr does not install. In #619 I was able to solve the problem by adding r-curl and r-igraph to environment.yml