Extra directories added to include path for CTU ast generation

[daverigby]

HI,

I'm trying to get a reasonably large (~3000 source file) project working with codechecker in CTU mode. I've built the compile-commands.json using CMake's built-in support; and things are broadly working, however it appears some additional include paths are being added to the command-line which cause my build to fail (as it ends up picking up the wrong header files.

Example of a failing command-line:

[ERROR 2021-05-13 10:11] - Error generating function map.

command:

/usr/lib/llvm-12/bin/clang-extdef-mapping /home/daver/repos/couchbase/server/kv_engine/engines/ep/src/dcp/producer.cc \
    -- --target=x86_64-pc-linux-gnu -isystem /usr/include/c++/8 -isystem /usr/include/x86_64-linux-gnu/c++/8 \
    -isystem /usr/include/c++/8/backward -isystem /usr/local/include -isystem /usr/include/x86_64-linux-gnu \
    -isystem /usr/include -c -x c++ \
    ... cut extra defines ... \
    -isystem /home/daver/repos/couchbase/server/build-codechecker/tlm/deps/boost.exploded/include \
    ... cut
 -pthread -fcolor-diagnostics -Qunused-arguments -Wall -Wextra -Wno-unused-parameter -pedantic -fno-strict-aliasing -ftemplate-depth=900 -fsized-deallocation -Wno-unused-lambda-capture -Wno-gnu-zero-variadic-macro-arguments -O3 -DNDEBUG -g -fPIC -ftls-model=initial-exec -std=c++17

stderr:

In file included from /home/daver/repos/couchbase/server/kv_engine/engines/ep/src/dcp/producer.cc:12:
In file included from /home/daver/repos/couchbase/server/kv_engine/engines/ep/src/dcp/producer.h:16:
In file included from /home/daver/repos/couchbase/server/kv_engine/engines/ep/src/ep_engine.h:17:
In file included from /home/daver/repos/couchbase/server/kv_engine/engines/ep/src/stats.h:23:
In file included from /home/daver/repos/couchbase/server/platform/include/platform/non_negative_counter.h:24:
In file included from /home/daver/repos/couchbase/server/platform/include/platform/exceptions.h:70:
In file included from /usr/include/boost/exception/all.hpp:10:
In file included from /usr/include/boost/exception/diagnostic_information.hpp:10:
In file included from /usr/include/boost/exception/get_error_info.hpp:13:
In file included from /usr/include/boost/exception/detail/shared_ptr.hpp:13:
In file included from /usr/include/boost/shared_ptr.hpp:17:
In file included from /usr/include/boost/smart_ptr/shared_ptr.hpp:28:
In file included from /usr/include/boost/smart_ptr/detail/shared_count.hpp:29:
In file included from /usr/include/boost/smart_ptr/detail/sp_counted_base.hpp:45:
/usr/include/boost/smart_ptr/detail/sp_counted_base_clang.hpp:31:5: error: address argument to atomic operation must be a pointer to _Atomic type ('boost::detail::atomic_int_least32_t *' (aka 'atomic<int> *') invalid)
    __c11_atomic_fetch_add( pw, 1, __ATOMIC_RELAXED );

Note the prefixing of -isystem /usr/include - this isn't present in the compilation database for that translation unit:

{
  "directory": "/home/daver/repos/couchbase/server/build-codechecker",
  "command": "/usr/bin/clang++-12 -DCB_DEVELOPMENT_ASSERTS -DCOUCHBASE_ENTERPRISE_EDITION=1 -DDESTINATION_ROOT=\\\"/home/daver/repos/couchbase/server/install-codechecker\\\" -DEP_USE_MAGMA=1 -DEP_USE_ROCKSDB=1 -DGLOG_NO_ABBREVIATED_SEVERITIES -DGOOGLE_GLOG_DLL_DECL=\"\" -DHAVE_BREAKPAD -DHAVE_JEMALLOC -DHAVE_JEMALLOC_SDALLOCX -DHAVE_LIBNUMA=1 -DLIBCOUCHSTORE_NO_VISIBILITY -DMEMCACHED_VERSION=\\\"1d90572086c2695e129f659901bc8656396b1073\\\" -DOBJECT_ROOT=\\\"/home/daver/repos/couchbase/server/build-codechecker/kv_engine\\\" -DPRODUCT_VERSION=\\\"0.0.0-0000\\\" -DSOURCE_ROOT=\\\"/home/daver/repos/couchbase/server/kv_engine\\\" -DSPDLOG_LEVEL_NAMES=\"{\\\"TRACE\\\", \\\"DEBUG\\\", \\\"INFO\\\", \\\"WARNING\\\", \\\"ERROR\\\", \\\"CRITICAL\\\", \\\"OFF\\\"}\" -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE=1 -D_POSIX_PTHREAD_SEMANTICS -D__EXTENSIONS__=1 -D__STDC_FORMAT_MACROS -Dgsl_CONFIG_CONTRACT_VIOLATION_THROWS -Itlm/deps/prometheus-cpp.exploded/include -Ikv_engine/engines/ep -Icouchstore/include -I../couchstore/include -I../kv_engine/include -I../kv_engine/tracing -I../kv_engine/utilities -I../kv_engine -I../platform/external -Itlm/deps/flatbuffers.exploded/include -Itlm/deps/snappy.exploded/include -Ikv_engine/engines/ep/src -I../kv_engine/engines/ep/src -I../kv_engine/engines/ep -I../install-codechecker/include -Ikv_engine/include -Iplatform/include -I../magma/include -I../kv_engine/engines/ep/tools/couchfile_upgrade -isystem ../third_party/HdrHistogram_c/src -isystem ../phosphor/include -isystem ../platform/include -isystem tlm/deps/boost.exploded/include -isystem tlm/deps/json.exploded/include -isystem ../third_party/gsl-lite/include -isystem ../third_party/spdlog/include -isystem ../third_party/googletest/googletest/include -isystem ../third_party/googletest/googlemock/include -isystem ../subjson -isystem tlm/deps/libevent.exploded/include -isystem tlm/deps/openssl.exploded/include -isystem tlm/deps/jemalloc.exploded/include -isystem tlm/deps/rocksdb.exploded/include -isystem tlm/deps/folly.exploded/include -isystem tlm/deps/double-conversion.exploded/include -isystem tlm/deps/glog.exploded/include  -fvisibility=hidden -pthread -fcolor-diagnostics -Qunused-arguments -Wall -Wextra -Wno-unused-parameter -pedantic -fno-strict-aliasing -Werror=switch -Werror=redundant-decls -Werror=missing-braces -ftemplate-depth=900 -fsized-deallocation -Wno-unused-lambda-capture -Wno-gnu-zero-variadic-macro-arguments -O3 -DNDEBUG -g -fPIC -ftls-model=initial-exec -std=c++17 -o kv_engine/engines/ep/CMakeFiles/ep_objs.dir/src/dcp/producer.cc.o -c /home/daver/repos/couchbase/server/kv_engine/engines/ep/src/dcp/producer.cc",
  "file": "/home/daver/repos/couchbase/server/kv_engine/engines/ep/src/dcp/producer.cc"
}```

This results in `boost/shared_ptr.hpp` being incorrectly picked up from `/usr/include`, when it should be found in the local header directory `/home/daver/repos/couchbase/server/build-codechecker/tlm/deps/boost.exploded/include`.

Could someone point me to where in CodeChecker these extra include paths are being added?

[csordasmarton]

@daverigby Without ctu do you get any similar errors during analysis?

[daverigby]

No - without ctu it all works correctly.

[csordasmarton]

The command related to the clang-extdef-mapping function can be found here:

codechecker/analyzer/codechecker_analyzer/analyzers/clangsa/ctu_manager.py

Lines 155 to 162 in 46cca7e

	def get_extdef_mapping_cmd(action, config, source, func_map_cmd):
	""" Get command to create CTU index file. """
	cmd = ctu_triple_arch.get_compile_command(action, config)
	cmd[0] = func_map_cmd
	cmd.insert(1, source)
	cmd.insert(2, '--')
	return cmd

codechecker/analyzer/codechecker_analyzer/analyzers/clangsa/ctu_triple_arch.py

Lines 18 to 44 in 46cca7e

	def get_compile_command(action, config, source='', output=''):
	""" Generate a standardized and cleaned compile command serving as a base
	for other operations. """
	cmd = [config.analyzer_binary]
	compile_lang = action.lang
	if not has_flag('--target', cmd) and \
	action.target[compile_lang] != "":
	cmd.append("--target=" + action.target[compile_lang])
	cmd.extend(prepend_all('-isystem', action.compiler_includes[compile_lang]))
	cmd.append('-c')
	if not has_flag('-x', cmd):
	cmd.extend(['-x', action.lang])
	cmd.extend(config.analyzer_extra_arguments)
	cmd.extend(action.analyzer_options)
	if output:
	cmd.extend(['-o', output])
	if source:
	cmd.append(source)
	if not has_flag('-std', cmd) and not has_flag('--std', cmd):
	cmd.append(action.compiler_standard.get(compile_lang, ""))
	return cmd

I think this is the place where we add those extra options to the compiler:

codechecker/analyzer/codechecker_analyzer/analyzers/clangsa/ctu_triple_arch.py

Line 30 in 46cca7e

cmd.extend(prepend_all('-isystem', action.compiler_includes[compile_lang]))

[daverigby]

Thanks - I commented out line 30 of ctu_triply_arch.py and the clang-extdef-mapping commands all pass.

I'm assuming they are needed for certain configurations - in my case the "main" compiler is also clang so I'm assuming minimal manipulation of the original compiler arguments is needed.