Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flatpak-cargo-builder fails because Cargo.toml in git submodule is not found #322

Closed
mfschumann opened this issue Oct 7, 2022 · 8 comments · May be fixed by #323
Closed

flatpak-cargo-builder fails because Cargo.toml in git submodule is not found #322

mfschumann opened this issue Oct 7, 2022 · 8 comments · May be fixed by #323
Labels
bug

Comments

@mfschumann
Copy link

flatpak-builder version

1.2.2

Linux distribution and version

Fedora Silverblue 36

Affected flatpak-builder tool

cargo/flatpak-cargo-generator.py

flatpak-builder tool cli args

Cargo.lock

Source repository URL

https://github.com/flathub/eu.betterbird.Betterbird/tree/58b020362586ad089249282a3e019705f310f8ff

Flatpak-builder manifest URL

No response

Description

I got the following error when trying to regenerate a sources json from an updated Cargo.lock for eu.betterbird.Betterbird:

Traceback (most recent call last):
  File "/var/home/martin/Projekte/eu.betterbird.Betterbird/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 356, in <module>
    main()
  File "/var/home/martin/Projekte/eu.betterbird.Betterbird/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 349, in main
    generated_sources = asyncio.run(generate_sources(load_toml(args.cargo_lock),
  File "/usr/lib64/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/var/home/martin/Projekte/eu.betterbird.Betterbird/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 303, in generate_sources
    for pkg in await asyncio.gather(*pkg_coros):
  File "/var/home/martin/Projekte/eu.betterbird.Betterbird/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 256, in get_package_sources
    return await get_git_package_sources(package, git_repos)
  File "/var/home/martin/Projekte/eu.betterbird.Betterbird/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 203, in get_git_package_sources
    git_repo['commits'][commit] = await get_git_repo_packages(repo_url, commit)
  File "/var/home/martin/Projekte/eu.betterbird.Betterbird/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 161, in get_git_repo_packages
    await get_dep_packages(pkg_toml, subpkg)
  File "/var/home/martin/Projekte/eu.betterbird.Betterbird/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 143, in get_dep_packages
    dep_toml = load_toml(os.path.join(git_repo_dir, dep_dir, 'Cargo.toml'))
  File "/var/home/martin/Projekte/eu.betterbird.Betterbird/flatpak-builder-tools/cargo/flatpak-cargo-generator.py", line 99, in load_toml
    with open(tomlfile, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/var/home/martin/.cache/flatpak-cargo/https_github.com_mozilla_application-services/components/external/glean/glean-core/ffi/Cargo.toml'

The folder /var/home/martin/.cache/flatpak-cargo/https_github.com_mozilla_application-services/components/external only contains glean, which is empty.

The issue seems to be caused by a submodule in the https://github.com/mozilla/application-services repo not being checked out when cloning the repo without --recursive:

flatpak-builder-tools/cargo/flatpak-cargo-generator.py

Line 114 in 4cdd030

subprocess.run(['git', 'clone', git_url, clone_dir], check=True)

Also, the exclude list in Cargo.toml at mozilla/application-services seems to be ignored by flatpak-cargo-generator, causing the missing submodule issue to surface in the first place.
@mfschumann mfschumann added the bug label Oct 7, 2022
@gasinvein
Copy link
Member

Thanks for detailed issue. You seem more proficient with Cargo then I am; what do you think would be the optimal solution here - clone git repos recursively, acknowledge excluded paths, or both?

@mfschumann
Copy link
Author

Well, for my particular situation acknowledging the excluded paths should be enough. But I can imagine that cloning recursively could be needed in other situations.

@gasinvein
Copy link
Member

Hmm, I'm not sure I understand how exclude paths are handled by Cargo. The docs say

The exclude key can be used to prevent paths from being included in a workspace. This can be useful if some path dependencies aren't desired to be in the workspace at all, or using a glob pattern and you want to remove a directory.

So, is a path listed in workspace.exclude excluded only from workspace members, or from member package dependencies?
In this specific case, application-services has megazords/ios/rust member, which, in turn, has components/external/glean/glean-core/ffi dependency, which is listed in the workspace exclude. So, should we just ignore this dependency? It doesn't feel right.

@mfschumann
Copy link
Author

From The Cargo Book:

[workspace] — Defines a workspace.
members — Packages to include in the workspace.
exclude — Packages to exclude from the workspace.

To me it seems that exclude does the opposite of members, i.e. any packages listed in exclude will not be added to the workspace. The text you quoted indicates that this applies both to packages that got selected via a glob pattern in members and to dependencies. At least that is my understanding of the documentation; I am no Cargo expert either.

@gasinvein gasinvein mentioned this issue Oct 9, 2022
Open
@gasinvein
Copy link
Member

@mfschumann Can you please try flatpak-cargo-generator.py from #323 and tell if it works for you (including at least cargo --offline fetch passing at build time, or complete build)?

@mfschumann
Copy link
Author

With this version the error does not occur anymore. Thanks for the quick fix!

@gasinvein
Copy link
Member

Did you manage to build the app (or at least get Cargo past "fetch" stage and start compiling)?

@mfschumann
Copy link
Author

Well, not really. The error in flatpak-cargo-generator.py was gone, but the generated .json produced an error during Cargo fetch. I then realized that I was using the wrong Cargo.lock for my project, so I am quite confident that this mistake is the actual cause for the Cargo error I was getting after your bugfix.

The correct Cargo.lock for my project does not provoke the issue I reported here in the first place, because it does not use submodules or exclude definitions. Building my project works fine when using a .json generated using flatpak-cargo-generator.sh from this PR and the correct Cargo.lock.

@zecakeh zecakeh mentioned this issue Apr 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cargo: Handle excluded paths in workspaces gasinvein/flatpak-builder-tools
2 participants
@gasinvein @mfschumann