Make the repo more open source and easy to work with

[ThatOneOcto]

Since the way releases are being handled (single commits) is pretty inconvenient to work with I'd imagine instead of committing one at a time, use 2 branches, main being the current state of the code and release branch being version releases

I'd imagine there would have to be a third branch for being commits that may be accepted into the main release branch since the CLA agreement for contributing means that NVIDIA owns the code but I am probably overthinking it but 2 branches instead of one commit per release would probably be appreciated by everybody

[mtijanic]

Hi there. We understand that the current workflow is very frustrating, and we appreciate all suggestions for help.

However, there is a common theme in all the suggestions we are getting (here and elsewhere) that seems to come from an incorrect model of how development happens internally. Not to put any words in your mouth, but I have a feeling the common assumption is that we have an internal repo where development happens, and then we occasionally squash all commits and push to this one.

For starters, we do not develop in git at all. The source of truth for code internally is Perforce. While Perfoce changelists do resemble git commits, they don't map 1:1, so even just translating those is not trivial.

Secondly, we effectively have a monorepo for all software, and changes often modify both the published and not-published parts of the codebase. These changes sometimes make no sense in isolation.

Then, we have multiple overlapping release "branches", which in a monorepo are practically just sibling folders. With git, these would be git branches, meaning a single internal commit would be multiple git commits.

Then, we have ABI-coupled firmware and usermode. Which means that we would have to publish valid firmware/usermode for every intermediate state. We keep these binaries in the monorepo, but git can't really deal with gigabytes of binary history per day.

Then, the file layout in this repo is not the same as internally. We have a mapping script that copies the files into locations that make more sense for the open kernel modules. Reasons this is done is because some files are owned by different teams, or they are used by other projects, etc. None of this is important for end users, but it helps internal development.

Lastly, the internal workflow has about 25 years of process and tooling built on top of it, and in some cases thousands of people working within that framework. It is.. difficult to change that.

--

So, we really do appreciate the suggestions, but keep in mind that things are not nearly as easy to do as one might think from the outside.

[mtijanic]

Yep, that's the one. Without the "git client" Wikipedia mentions, just plain old p4.

[ghtesting2020]

Jesus, Perforce. Everything they touch turns to garbage. Zend Server hasn't been the same since they bought it. Perforce has been doing such a bad job we are leaving Zend Server for an in house solution, lmao. For version control we use GitHub Enterprise (no cloud, local only for security reasons).

[throwaway1037]

There is a solution: publish all the software which has already been published as proprietary software, but as free software.
From an ethical standpoint, this is a no-brainer.
From a technical standpoint, this is as easy as hosting a gigantic compressed archive and making it available to download.
The community can help out and even take it from there if necessary.
From a legal standpoint, this is certainly a difficult task.

[hramrach]

But the git client does exist so the claim that p4 changelists don't map to git commits is not really true.

Maybe you don't have the tools to map them which is understandable but they are not fundamentally different.

[mtijanic]

To clarify, I'm making two separate claims there:

(1) A p4 changelist is not the same as a git commit. I found the closest equivalent to a changelist to be a git branch. As a consequence, you cannot losslessly convert between p4 and git repo histories. There are concepts that don't map cleanly and you need to make tradeoffs in how to represent them.
You can convert any subdirectory of a Perforce repo into a git repo with an entirely linear history - not exactly idiomatic git usage, but works well enough. It gets much murkier, however, if you need to do it for multiple sibling directories but not their common parent.

(2) Even if the theoretical problems from (1) are 'solved', there is currently no adequate tool that can do it for our specific repo.