[clang][Dependency Scanning] Cherry-pick Commits To Support Compiler Instance Sharing (#11631)

* [clang][deps] Remove dependency on `tooling::ToolAction` (llvm#149904)

The dependency scanner was initially using a fair amount of
infrastructure provided by the `clangTooling` library. Over time, the
needs for bespoke handling of command lines grew and the overlap with
the tooling library kept shrinking. I don't think the library provides
any value anymore.

I decided to remove the dependency and only reimplement the small bits
required by the scanner.

This allowed for a nice simplification, where we no longer need to
create temporary dummy `FileManager` instances (mis-named as
`DriverFileMgr` in some parts) and `SourceManager` instances to attach
to the `DiagnosticsEngine`. That code was copied from the tooling
library to support `DiagnosticConsumers` that expect these to exist. The
scanner uses a closed set of consumers and none need these objects to
exist.

The motivation for this (hopefully NFC) patch are some new restrictions
to how VFS's can be propagated in Clang that I'm working on.

(cherry picked from commit aa1b416)

* Reland "[clang] Delay normalization of `-fmodules-cache-path` (llvm#150123)"

This reverts commit 613caa9, essentially
reapplying 4a4bdde after moving
`normalizeModuleCachePath` from clangFrontend to clangLex.

This PR is part of an effort to remove file system usage from the
command line parsing code. The reason for that is that it's impossible
to do file system access correctly without a configured VFS, and the VFS
can only be configured after the command line is parsed. I don't want to
intertwine command line parsing and VFS configuration, so I decided to
perform the file system access after the command line is parsed and the
VFS is configured - ideally right before the file system entity is used
for the first time.

This patch delays normalization of the module cache path until
`CompilerInstance` is asked for the cache path in the current
compilation context.

(cherry picked from commit 55bef46)

* NFC: Clean up of IntrusiveRefCntPtr construction from raw pointers. (llvm#151545)

Handles clang::DiagnosticsEngine and clang::DiagnosticIDs.

For DiagnosticIDs, this mostly migrates from `new DiagnosticIDs` to
convenience method `DiagnosticIDs::create()`.

Part of cleanup llvm#151026

(cherry picked from commit c7f3437)

 Conflicts:
	clang/tools/driver/cc1_main.cpp
	clang/unittests/Driver/DXCModeTest.cpp
	clang/unittests/Driver/SimpleDiagnosticConsumer.h
	clang/unittests/Frontend/SearchPathTest.cpp
	clang/unittests/Lex/HeaderSearchTest.cpp
	clang/unittests/Tooling/RewriterTestContext.h

* NFC: Clean up of IntrusiveRefCntPtr construction from raw pointers. (llvm#151782)

This commit handles the following types:
- clang::ExternalASTSource
- clang::TargetInfo
- clang::ASTContext
- clang::SourceManager
- clang::FileManager

Part of cleanup llvm#151026

(cherry picked from commit 4205da0)

 Conflicts:
	clang/lib/Frontend/ASTUnit.cpp
	clang/lib/Frontend/ChainedIncludesSource.cpp
	clang/lib/Frontend/CompilerInstance.cpp

* Merge commit '30633f308941' from llvm.org/main into next

(cherry picked from commit 95ea104)

 Conflicts:
	clang/include/clang/Frontend/CompilerInstance.h
	clang/lib/Frontend/CompilerInstance.cpp

* Merge pull request #11450 from swiftlang/jan_svoboda/cas-fix-early-vfs

[clang] Fix CAS initialization after upstream llvm#158381

(cherry picked from commit 6d73002)

* [clang] Avoid reparsing VFS overlay files for module dep collector (llvm#158372)

This PR uses the new-ish `llvm::vfs::FileSystem::visit()` interface to
collect VFS overlay entries from an existing `FileSystem` instance
rather than parsing the VFS YAML file anew. This prevents duplicate
diagnostics as observed by `clang/test/VFS/broken-vfs-module-dep.c`.

(cherry picked from commit 4957c47)

* [clang] Don't fail `ExecuteCompilerInvocation()` due to caller errors (llvm#158695)

This PR changes the behavior of `clang::ExecuteCompilerInvocation()` so
that it only returns early when the `DiagnosticsEngine` emitted errors
**within** the function. Handling errors emitted before the function got
called is a responsibility of the caller. Necessary for llvm#158381.

(cherry picked from commit f33fb0d)

* [clang] Only set non-empty bypass to scan VFS (llvm#159605)

Normalizing an empty modules cache path results in an incorrect
non-empty path (the working directory). This PR conditionalizes more
code to avoid this. Tested downstream by swift/llvm-project and the
`DependencyScanningCAPITests.DependencyScanningFSCacheOutOfDate` unit
test.

(cherry picked from commit 5a339b0)

* Merge commit '0e35f56d40d3' from llvm.org/main into next

(cherry picked from commit 3efcc0f)

 Conflicts:
	clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp

* [clang] NFCI: Clean up `CompilerInstance::create{File,Source}Manager()` (llvm#160748)

The `CompilerInstance::createSourceManager()` function currently accepts
the `FileManager` to be used. However, all clients call
`CompilerInstance::createFileManager()` prior to creating the
`SourceManager`, and it never makes sense to use a `FileManager` in the
`SourceManager` that's different from the rest of the compiler. Passing
the `FileManager` explicitly is redundant, error-prone, and deviates
from the style of other `CompilerInstance` initialization APIs.

This PR therefore removes the `FileManager` parameter from
`createSourceManager()` and also stops returning the `FileManager`
pointer from `createFileManager()`, since that was its primary use. Now,
`createSourceManager()` internally calls `getFileManager()` instead.

(cherry picked from commit b86ddae)

 Conflicts:
	clang/lib/Tooling/DependencyScanning/DependencyScannerImpl.cpp

* Merge commit '436861645247' from llvm.org/main into next

(cherry picked from commit 286ea7d)

 Conflicts:
	clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp

* [clang] Pass VFS into `ASTUnit::LoadFromASTFile()` (llvm#159166)

This PR makes the `VFS` parameter to `ASTUnit::LoadFromASTFile()`
required and explicit, rather than silently defaulting to the real file
system. This makes it easy to correctly propagate the fully-configured
VFS and load any input files like the rest of the compiler does.

(cherry picked from commit cda542d)

* Fix a line missing when merging 30633f3

* [clang][deps] Fix a use-after-free from expanding response files (llvm#164676)

In 4368616 we accidentally moved uses of command-line args saved
into a bump pointer allocator during response file expansion out of
scope of the allocator. Also, the test that should have caught this (at
least with asan) was not working correctly because clang-scan-deps was
expanding response files itself during argument adjustment rather than
the underlying scanner library.

rdar://162720059
(cherry picked from commit 3e6f696)

---------

Co-authored-by: Jan Svoboda <[email protected]>
Co-authored-by: James Y Knight <[email protected]>
Co-authored-by: git apple-llvm automerger <am@git-apple-llvm>
Co-authored-by: Ben Langmuir <[email protected]>

Author: 5 people

clang-tools-extra/clang-apply-replacements/tool/ClangApplyReplacementsMain.cpp

@@ -97,8 +97,7 @@ int main(int argc, char **argv) {
   cl::ParseCommandLineOptions(argc, argv);
 
   DiagnosticOptions DiagOpts;
-  DiagnosticsEngine Diagnostics(
-      IntrusiveRefCntPtr<DiagnosticIDs>(new DiagnosticIDs()), DiagOpts);
+  DiagnosticsEngine Diagnostics(DiagnosticIDs::create(), DiagOpts);
 
   // Determine a formatting style from options.
   auto FormatStyleOrError = format::getStyle(FormatStyleOpt, FormatStyleConfig,

clang-tools-extra/clang-change-namespace/tool/ClangChangeNamespace.cpp

@@ -128,9 +128,8 @@ int main(int argc, const char **argv) {
   LangOptions DefaultLangOptions;
   DiagnosticOptions DiagOpts;
   clang::TextDiagnosticPrinter DiagnosticPrinter(errs(), DiagOpts);
-  DiagnosticsEngine Diagnostics(
-      IntrusiveRefCntPtr<DiagnosticIDs>(new DiagnosticIDs()), DiagOpts,
-      &DiagnosticPrinter, false);
+  DiagnosticsEngine Diagnostics(DiagnosticIDs::create(), DiagOpts,
+                                &DiagnosticPrinter, false);
   auto &FileMgr = Tool.getFiles();
   SourceManager Sources(Diagnostics, FileMgr);
   Rewriter Rewrite(Sources, DefaultLangOptions);

clang-tools-extra/clang-include-fixer/IncludeFixer.cpp

@@ -53,7 +53,7 @@ class Action : public clang::ASTFrontendAction {
 
     Compiler->createSema(getTranslationUnitKind(), CompletionConsumer);
     SemaSource->setCompilerInstance(Compiler);
-    Compiler->getSema().addExternalSource(SemaSource.get());
+    Compiler->getSema().addExternalSource(SemaSource);
 
     clang::ParseAST(Compiler->getSema(), Compiler->getFrontendOpts().ShowStats,
                     Compiler->getFrontendOpts().SkipFunctionBodies);

clang-tools-extra/clang-include-fixer/tool/ClangIncludeFixer.cpp

@@ -454,7 +454,7 @@ int includeFixerMain(int argc, const char **argv) {
 
   // Set up a new source manager for applying the resulting replacements.
   DiagnosticOptions DiagOpts;
-  DiagnosticsEngine Diagnostics(new DiagnosticIDs, DiagOpts);
+  DiagnosticsEngine Diagnostics(DiagnosticIDs::create(), DiagOpts);
   TextDiagnosticPrinter DiagnosticPrinter(outs(), DiagOpts);
   SourceManager SM(Diagnostics, tool.getFiles());
   Diagnostics.setClient(&DiagnosticPrinter, false);

clang-tools-extra/clang-move/tool/ClangMove.cpp

@@ -178,9 +178,8 @@ int main(int argc, const char **argv) {
 
   DiagnosticOptions DiagOpts;
   clang::TextDiagnosticPrinter DiagnosticPrinter(errs(), DiagOpts);
-  DiagnosticsEngine Diagnostics(
-      IntrusiveRefCntPtr<DiagnosticIDs>(new DiagnosticIDs()), DiagOpts,
-      &DiagnosticPrinter, false);
+  DiagnosticsEngine Diagnostics(DiagnosticIDs::create(), DiagOpts,
+                                &DiagnosticPrinter, false);
   auto &FileMgr = Tool.getFiles();
   SourceManager SM(Diagnostics, FileMgr);
   Rewriter Rewrite(SM, LangOptions());

clang-tools-extra/clang-reorder-fields/tool/ClangReorderFields.cpp

@@ -74,9 +74,8 @@ int main(int argc, const char **argv) {
   LangOptions DefaultLangOptions;
   DiagnosticOptions DiagOpts;
   TextDiagnosticPrinter DiagnosticPrinter(errs(), DiagOpts);
-  DiagnosticsEngine Diagnostics(
-      IntrusiveRefCntPtr<DiagnosticIDs>(new DiagnosticIDs()), DiagOpts,
-      &DiagnosticPrinter, false);
+  DiagnosticsEngine Diagnostics(DiagnosticIDs::create(), DiagOpts,
+                                &DiagnosticPrinter, false);
 
   auto &FileMgr = Tool.getFiles();
   SourceManager Sources(Diagnostics, FileMgr);

clang-tools-extra/clang-tidy/ClangTidy.cpp

@@ -96,8 +96,7 @@ class ErrorReporter {
                 llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> BaseFS)
       : Files(FileSystemOptions(), std::move(BaseFS)),
         DiagPrinter(new TextDiagnosticPrinter(llvm::outs(), DiagOpts)),
-        Diags(IntrusiveRefCntPtr<DiagnosticIDs>(new DiagnosticIDs), DiagOpts,
-              DiagPrinter),
+        Diags(DiagnosticIDs::create(), DiagOpts, DiagPrinter),
         SourceMgr(Diags, Files), Context(Context), ApplyFixes(ApplyFixes) {
     DiagOpts.ShowColors = Context.getOptions().UseColor.value_or(
         llvm::sys::Process::StandardOutHasColors());
@@ -570,7 +569,7 @@ runClangTidy(clang::tidy::ClangTidyContext &Context,
 
   ClangTidyDiagnosticConsumer DiagConsumer(Context, nullptr, true, ApplyAnyFix);
   auto DiagOpts = std::make_unique<DiagnosticOptions>();
-  DiagnosticsEngine DE(new DiagnosticIDs(), *DiagOpts, &DiagConsumer,
+  DiagnosticsEngine DE(DiagnosticIDs::create(), *DiagOpts, &DiagConsumer,
                        /*ShouldOwnClient=*/false);
   Context.setDiagnosticsEngine(std::move(DiagOpts), &DE);
   Tool.setDiagnosticConsumer(&DiagConsumer);

clang-tools-extra/clang-tidy/ExpandModularHeadersPPCallbacks.cpp

@@ -71,7 +71,7 @@ ExpandModularHeadersPPCallbacks::ExpandModularHeadersPPCallbacks(
       InMemoryFs(new llvm::vfs::InMemoryFileSystem),
       Sources(Compiler.getSourceManager()),
       // Forward the new diagnostics to the original DiagnosticConsumer.
-      Diags(new DiagnosticIDs, DiagOpts,
+      Diags(DiagnosticIDs::create(), DiagOpts,
             new ForwardingDiagnosticConsumer(Compiler.getDiagnosticClient())),
       LangOpts(Compiler.getLangOpts()), HSOpts(Compiler.getHeaderSearchOpts()) {
   // Add a FileSystem containing the extra files needed in place of modular

clang-tools-extra/clang-tidy/plugin/ClangTidyPlugin.cpp

@@ -41,7 +41,7 @@ class ClangTidyPluginAction : public PluginASTAction {
         new ClangTidyDiagnosticConsumer(*Context, &Compiler.getDiagnostics());
     auto DiagOpts = std::make_unique<DiagnosticOptions>();
     auto DiagEngine = std::make_unique<DiagnosticsEngine>(
-        new DiagnosticIDs, *DiagOpts, DiagConsumer);
+        DiagnosticIDs::create(), *DiagOpts, DiagConsumer);
     Context->setDiagnosticsEngine(std::move(DiagOpts), DiagEngine.get());
 
     // Create the AST consumer.

clang-tools-extra/clangd/Preamble.cpp

@@ -659,7 +659,7 @@ buildPreamble(PathRef FileName, CompilerInvocation CI,
   WallTimer PreambleTimer;
   PreambleTimer.startTimer();
   auto BuiltPreamble = PrecompiledPreamble::Build(
-      CI, ContentsBuffer.get(), Bounds, *PreambleDiagsEngine,
+      CI, ContentsBuffer.get(), Bounds, PreambleDiagsEngine,
       Stats ? TimedFS : StatCacheFS, std::make_shared<PCHContainerOperations>(),
       StoreInMemory, /*StoragePath=*/"", CapturedInfo);