New feature: per-group client repository definition

[fdbastionamio]

Resolves #51
This introduces the notion of 'scope' which is either 'Project' (default) or 'Group'

To pull dependencies from a group package manager, clients use an url with the following cargo/config.toml registries

[registries]
# Per-project registry:
# internal_project_registry = { index = "ssh://my.gitlab:2222/my_group/my_subgroup/my_project" } # per-project repo

# Per-group registry: set the path to the group path, add ?scope=group at the end
internal_group_registry = { index = "ssh://my.gitlab:222/my_group/my_subgroup/?scope=group" }

Limitations

• The project name and the crate file must match or this will fail on the client's side when performing the http fetch.
• Packages cannot be fetched for subgroups under the given group.
• Untested on multi-crates workspaces

This PR also filters packages by type so that only generic packages are being considered as some groups publish heterogenous packages (python / maven / generic)

[w4]

Thanks for the MR, last time I attempted this I ran into some issues with the download portion as there's no group-level download API - I attempted to contribute something for that in gitlab!82663 but it was ultimately rejected.

I don't see anything group-level in https://docs.gitlab.com/ee/user/packages/generic_packages/ - are these undocumented APIs?

[fdoyon]

@w4 I am not querying the package per-group but instead asking the users to follow the following convention : have a package name that matches the project name. Then the cargo url template can point to the generic package download URL.

It is very suboptimal, but there is indeed no API to download packages at the group level - I will see if I can submit a patch to gitlab to include an actual package manager.

[w4]

The way I've worked around this limitation in our internal fork is a HTTP server that can serve redirects to known crates. Could be another one to upstream, but does mean a greater attack surface for the application.

[loyd]

@w4, thanks for the shim!

Could be another one to upstream, but does mean a greater attack surface for the application.

Has anything changed since the creation of this PR?

[w4]

Hi @loyd, our current implementation uses the original approach from this repo which relies on this patch to Gitlab -- which they ultimately rejected to take into their upstream. Another problem is that CI tokens cannot call group-level endpoints, which would force the use of a root token - we are currently working around this with the GraphQL API, but Gitlab are actively working against us here because 16.11.7 recently put this behind a disabled-by-default feature flag with intent to remove 😬