Add Bitbucket enumeration support

[Coruscant11]

Hello,
Since my company is actually working with Bitbucket, I would be glad to work in order to add the same features for Github to Bitbucket. For example listing Bitbucket project repositories, listing user repositories, and ensure that each feature of noseyparker is working on it.

I could take inspiration from the existing code for Github and make a Bitbucket version.
Why not GitLab too in the future?
😄

[bradlarsen]

Hello again @Coruscant11 :)

Yes, having support for enumerating repositories from Bitbucket would be welcome.

I'm not all that familiar with Bitbucket. It looks like they have a cloud-hosted offering as well as self-hosted options. Which type of deployment is most relevant to you? I would start with getting that working.

I took a quick look at things, and have some high-level notes and pointers for you:

• In Nosey Parker, the GitHub repository enumeration code is used in just two places currently: in the scan command, and in the github command. In both cases, the high-level wrapper function is noseyparker::github::enumerate_repo_urls, which is a synchronous API that takes the set of requested users/orgs as input and simply returns a vector of repo URLs. Internally, it uses async functions, but that's all hidden for now.

• I didn't see an existing Bitbucket API library on crates.io, so building something simple for the Bitbucket APIs you need using the reqwest libraryis a reasonable path. (That's what I ended up doing for GitHub support, because the existing octocrab library was pretty big and didn't actually support the APIs I needed.)

• Nosey Parker's GitHub support has a ClientBuilder, used to specify how to initialize an actual Client object that makes the API requests. You could introduce similar types in a new bitbucket module.

• Nosey Parker's GitHub code only currently supports bearer token-based authentication. In Bitbucket, it looks like many different authentication schemes are supported, though it looks like in the examples that simpler bearer token-based authentication should work. I'd start with that, unless your use case needs something different.

• In Nosey Parker, if someone wants to use a GitHub token, they must specify it via the NP_GITHUB_TOKEN environment variable. You could introduce a similar NP_BITBUCKET_TOKEN variable.

• It looks like this Bitbucket API endpoint may be the most relevant. It can take filter parameters to restrict the scope to particular users or workspaces. (A "workspace" is a Bitbucket version of a GitHub organization?)

• The pagination in Bitbucket responses seems to use the same kind of mechanism as GitHub's APIs, so you could probably adapt Nosey Parker's GitHub code for that. Specifically, this and this.

• Nosey Parker's GitHub support detects and reports rate-limiting failures, but currently makes no attempt to retry failing requests.

[bradlarsen]

Again, I'm happy to answer any questions you may have about this!

[Coruscant11]

Great answer as always!

Yes, the main goal would be to allow people to also scan their private Bitbucket server. Like in my case for my company. Normally, the cloud and the self-hosted has the same API scheme, so just giving the Bitbucket URL to Nosey Parker should be enough.
As you said, we only need to call the API in order to find the git url, and reqwest should do the work perfectly.

We can generate authentication token with Bitbucket, and I think that it should be the way to use authentication on tools like scanners. Users should use read-only tokens for this kind of stuff and that is why I think that only supporting bearer token-based authentication is a good thing.

I will try to work on it and keep you updated if I need something!
Thank you so much 😃

[Coruscant11]

Hello, sorry I was quite busy the past months. I will work on this issue very soon 😄

[bradlarsen]

No problem at all :)

[amuessig]

I am also curious about the progress of that feature. Is there any news?

[Coruscant11]

Hello @amuessig, I didn't work on it unfortunately.. I could in fact maybe try to look at it again, just that finally, I didn't need it anymore at my company. If someone want to take this issue, feel free to!

Otherwise, I will give an update this week end. In fact I want to do some Rust these days so this would be a good opportunity 😋