Can't add repo to extra-deps if repo contains symlink to git submodule

[brandon-leapyear]

General summary/comments (optional)

Relevant links:

• Can't use Stack's github as an extra-dep: Tar error #4580
• Git submodules are not cloned #5536
• extra-deps submodules #5236
• Pantry does not work with git annex #4579
• https://github.com/commercialhaskell/pantry/blob/c7923fc963e36efd179ba301ac426139a9dc182d/src/Pantry/Archive.hs#L406

Steps to reproduce

extra-deps:
- github: google/proto-lens
  commit: 5ee7ac14686cf8d0668b8b3053b055303fc3ce86
  subdirs:
  - proto-lens-protoc

Then run stack build

Expected

It should checkout the repo, git submodule update, then use it.

Actual

It downloads the tar archive from GitHub and tries untarring it, with git submodules not included + symlinks not resolving.

Unsupported tarball from ...: Symbolic link dest not found from ... to ...
This may indicate that the source is a git archive which uses git-annex

Stack version

2.7.3

[qrilka]

Isn't it a duplicate of #5536?

[brandon-leapyear]

✨ _**This is an old work account. Please reference @brandonchinn178 for all future communication**_ ✨ <!-- updated by mention_personal_account_in_comments.py -->

…

--- I'm not sure. This issue is because theres a symlink pointing to a directory containing the submodule. I'm not sure the other issue has the symlink aspect to it.

On Sat, Aug 21, 2021, 2:24 PM Kirill Zaborsky ***@***.***> wrote: Isn't it a duplicate of #5536 <#5536>? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5603 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGUC75IQ6YPZAI3FQ2BYIA3T6AKPJANCNFSM5CRB4KMQ> .

[brandon-leapyear]

✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨

---

To be clear, this isn't a duplicate of #5536 (or rather, this could be solved without solving #5536) because the proto-lens-protoc package I'm trying to build doesn't use the symlink at all; the symlink just exists in the repo and Stack throws an error.

i.e. if this line were simply changed to

[] -> Right []

My build would work. So the git-annex failure is a false positive here

[jorpic]

I had a similar issue with submodules and spent some time trying to understand what is going on.

The problem is that with

extra-deps:
- github: google/proto-lens

stack does not clone the repo and can't update submodules, it just downloads an archive from https://github.com/google/proto-lens/archive/<hash>.tar.gz.

On the other hand, this version executes git clone and then git submodule update --init --recursive:

extra-deps:
- git: https://github/google/proto-lens

So, if some dependency uses git submodules you should replace github: user/repo with git: https://github.com/user/repo.

Seems that this is not stated explicitly in documentation but probably it should.

[mpilgrem]

I am going to close this issue, because (a) it was based on the false premise that github: checks out a repository and (b) the behaviour of github: (and the difference from git:) is documented online at https://docs.haskellstack.org/en/stable/pantry/.

[brandonchinn178]

@mpilgrem can you elaborate? The crux of this issue is that using github or git on a repo with a symlink immediately fails, even if the symlink isn't actually used. In other words, if I create a repo with just

stack new test-project
cd test-project
git init
ln -sf nonexistent-file link-to-nonexistent-file
git add .
git commit -m "Initial commit"

And use that repo as an extra-dep, Stack will immediately fail.

[mpilgrem]

@brandonchinn178, I am happy to elaborate. I'll refer to a Haskell package repository archive that contains a symlink other than to a file within the archive as 'malformed'. It seems to me that there are two broad categories of 'issue':

1. Stack/Pantry wrongly rejects repository archives that are not malformed; and
2. Stack/Pantry should tolerate repository archives that are malformed (strictly, a feature request).

I understood your post first raising the issue here as being in the first category. However, it was mistaken. Stack/Pantry was not rejecting a repository archive that was not malformed. Rather, it was rejecting an archive that was malformed because it included symlinks to submodule files not included in the archive. The solution being to use git: not github: (as the former does not use a repository archive, but clones the repository and updates submodules).

I understand your recent example as being in the second category. That category is covered by #4913.

[brandonchinn178]

Thanks for the explanation, that was helpful. Two things:

1. Using git: still doesn't work:

resolver: nightly-2023-08-10
packages: []
extra-deps:
  - git: [email protected]:google/proto-lens.git
    commit: 1378fbfbf44a3b529253aee1897d0e571f6c0ace
    subdirs:
      - proto-lens-protoc

$ stack --version
Version 2.11.1, Git revision d06506a16289c6b90e6ffc0dcc8be7f80487ac07 (dirty) aarch64 hpack-0.35.2

$ stack build
Cloning 1378fbfbf44a3b529253aee1897d0e571f6c0ace from [email protected]:google/proto-lens.git

Error: [S-760]
       Unsupported tarball from
       /private/var/folders/4w/tywh16w5777ds8kzfzdm0_l40000gn/T/with-repo-archive59100/foo.tar:

       Symbolic link dest not found from proto-lens-protobuf-types/proto-src to ../google/protobuf/src/, looking for google/protobuf/src.
       This may indicate that the source is a git archive which uses git-annex.
       See https://github.com/commercialhaskell/stack/issues/4579 for further information.

2. The reason I was using github: is because github: supports specifying a SHA from a fork while git: does not. i.e. you can reference the parent repo with a SHA from a fork with a PR open into the parent repo. I like this feature because if I fork a repo, make a PR, and merge it, I'd like to delete my fork without stack.yaml breaking immediately.

   At the very least, this behavior should probably be documented as a difference between the two. But it would be nice if the two behaviors were more closely aligned

[mpilgrem]

@brandonchinn178, I'm sorry I was mistaken about git: - I'll look into that.

On SHA from a fork, that sounds a bit like commercialhaskell/pantry#73 (SHA from an unmerged pull request from a fork) (now fixed).