feat(key on repo url): support git hosts other than GitHub + multiple forks

[kriswest]

resolves #950
resolves #511
resolves #66
resolves #1107
resolves #1028

Refactor (api, proxy & UI) to remove the assumption of GitHub as the git repository host and the use of the repository name field as the id of the repository (as this prevents git-proxy instances from supporting multiple forks of a project or projects from multiple hosts with the same name).

This PR:

• Replaces the use of the repo name field in the API with the _id field generated by the database adaptors,
  • Using the repository URL as a key does not work well with express routing, but _id does in both mongo and neDb
  • allows names to be repeated (multiple forks or clashing names from different organisations/repository hosts)
  • UI and CLI were updated accordingly
• Replaces the use of organisation/repoName.git in the proxy URLs with the repository url
  • i.e. https://myGitProxyInstance.com:8443/finos/git-proxy.git
    becomes
    https://myGitProxyInstance.com:8443/github.com/finos/git-proxy.git
  • Protocol is omitted from the embedded URL as thats assumed to be https://
  • Refactors the proxy to be able to proxy traffic to multiple different hosts
    • may require a restart after adding a host - further work may need to resolve that by restarting the proxy
  • Backwards compatibility is provided for old proxy urls by assuming github.com as the default
• Disables GitHub specific functionality in the UI if the host is not Github
• Completes application of Typescript to the database classes
  • Duplicated code reduced
  • A number of minor differences in behaviour (particularly return types) between the DB adaptors were resolved
  • Does NOT refactor all usages of the DB client to use typescript (still many requires to eliminate)
• Deprecates and ignores the config property proxyUrl as the proxied host(s) are now determined from the configured repositories
• Expands the tests for proxy routes and the Repo route of the API

To Do:

• Annotate PR for review
• Check test coverage
• Implement additional tests for the proxy and fallback
  • implement tests for new proxy URLs for github.com
  • implement tests for fallback with legacy proxy urls for github.com
  • implement tests for gitlab.com
  • implement tests for non-github/non-gitlab repo
  • implement tests for multiple forks
• Add support for GitLab API where repo is hosted at GitLab

(contributed as part of a GitLab CoCreate collaboration with help from @StingRayZA)

[netlify]

✅ Deploy Preview for endearing-brigadeiros-63f9d0 canceled.

Name	Link
🔨 Latest commit	837b49c
🔍 Latest deploy log	https://app.netlify.com/projects/endearing-brigadeiros-63f9d0/deploys/688894515580d00008f5ace8

[codecov]

Codecov Report

❌ Patch coverage is 83.24958% with 100 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.72%. Comparing base (d4d9239) to head (837b49c).

Files with missing lines	Patch %	Lines
src/db/mongo/repo.ts	37.14%	22 Missing ⚠️
src/service/routes/repo.js	85.04%	16 Missing ⚠️
src/db/mongo/users.ts	37.50%	15 Missing ⚠️
src/db/file/users.ts	65.21%	8 Missing ⚠️
src/proxy/index.ts	80.95%	8 Missing ⚠️
src/db/file/repo.ts	86.36%	3 Missing and 3 partials ⚠️
src/db/index.ts	93.40%	4 Missing and 2 partials ⚠️
src/db/mongo/pushes.ts	60.00%	4 Missing ⚠️
src/proxy/routes/index.ts	94.93%	4 Missing ⚠️
src/proxy/routes/helper.ts	94.23%	3 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1043      +/-   ##
==========================================
+ Coverage   77.40%   80.72%   +3.32%     
==========================================
  Files          56       63       +7     
  Lines        2288     2620     +332     
  Branches      258      310      +52     
==========================================
+ Hits         1771     2115     +344     
+ Misses        487      458      -29     
- Partials       30       47      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sam-holmes2]

LGTM after an initial scan through :) thanks for your contribution!

[kriswest]

Picked up a couple of test failures after merging main - will resolve (and start working on the additional tests needed).

[jescalada]

I went through the approval/rejection flows with a pre-existing repo, and things work well!

There is an issue with backwards compatibility with older, invalid databases from previous versions of GitProxy (unique URL enforcement with repos). This may also cause issues with the other files (pushes, users).

I also tested the Add Repo flow which caused my server to crash, maybe because of something wrong on my end (invalid input maybe?).

[jescalada]

I can release the unique constraint on the index to avoid this - however, I put it in to catch invalid data as, like the use of the repository name as an ID, it will result in the selection of the wrong repo project at times (although in this case the multiple records would have to be the same repo) and I thought it better for tests to fail etc. if data wasn't cleaned up from a previous run. Where do you think we should go with this @jescalada - an automated migration seems difficult as you'd just have to delete or programmatically edit one of the duplicate records...

I think catching and displaying a simple error message with the invalid entry/entries could be enough - so that the GitProxy administrator can quickly identify the issue and fix it manually. Thankfully, the error seems to occur on backend (db) startup, so end users wouldn't really have the app suddenly blowing up.

[jescalada]

Thank you for fixing the merge conflicts! 🙏🏼 Those were mostly caused by our team's PR...

I'll check out your recent updates before the next community meeting.

[jescalada]

The tests look good so far, and test coverage is excellent! 😃 Just wondering if the merge conflicts with main have been resolved correctly as I wasn't able to do the Add Repo flow...

[jescalada]

Everything looks good to me @kriswest! 🚀 Do you think we should merge this after #973?

[kriswest]

Everything looks good to me @kriswest! 🚀 Do you think we should merge this after #973?

Yes #973 should go first - I'll prob have a few further conflicts to sort thereafter, but hopefully they will be limited!

[kriswest]

Resolved the issue with admin permissions post login, and resolved both typing and layout issues in the RepoDetails view. No idea whats was up with the types for the MUI Box component, but we don't actually need it so its been replaced with Grid (which doesn't have the same issue).

[jescalada]

Looks fantastic! This is ready to merge for the v2 release candidates - just waiting on the remaining PRs.