New issues with installing under Apple Silicon

[ceesem]

This is in OS X 13.0.1 on a 2021 MacBook Pro M1.

A simple pip install isn't possible right now due to two issues I see in dependent libraries. I'm adding this as an issue both as feedback and also to lay out what worked for me in case other people come across the same thing.

• simplejpeg : The more recent versions of simplejpeg require an older version of cmake that unfortunately fails the OS X developer signing check and the OS won't let it run. The most recent version that worked for me is 1.6.3, before they added this version requirement as a workaround for some other issue. I would suggest you pin that one until they can fix things. As a current workaround, pip install simplejpeg==1.6.3 before installing cloudvolume worked.

• The zfpy issue mentioned late in the closed thread for Issue with Import in Anaconda on Apple Silicon (M1) #518 (and current thread zfpy prevents cloud-volume from working out of the box on macOS arm64 #561) happened for me as well. The conda workaround does work, although it installs 0.5.5. The version itself isn't an issue, I don't think, because pip installing zfpy 0.5.5 fails (albeit for compiler reasons rather than on import). Maybe there is a more elegant approach, but the version that seemed to work best for me required me to first pip install cloudvolume, then pip uninstall zfpy and then conda install zfpy.

[william-silversmith]

My recommendation for using CloudVolume on an M1 until these issues can be resolved is to use a Rosetta2 terminal. I got an M1 when it came out and support for the arm64 side of things was pretty sparse at the time. I got set up using this trick:

https://www.byran.tech/html/how-to-make-a-rosetta-2-emulated-x86-terminal-on-arm-apple-silicon-chips.html

I use the rosetta2 terminal as my main terminal right now and use arm for brew install and C++ time profiles.

It's difficult to resolve the above issues because CI providers do not have Apple Silicon in stock. Once that happens, dependency authors will be much more amenable to provide first class support.

It seems like pinning simplejpeg 1.6.3 might be acceptable. If you have a moment, it would be good to open an issue with that package maintainer. I'm a bit stuck on zfpy b/c they do not have a good way to compile their binaries as none of their maintainers have an M1. One possibility is to make zfpy/zfpc an optional dependency.

[ceesem]

That's a great tip, thanks! And yeah, the simplejpeg issue here is an OS X issue with code signing, I think, not arm64 vs. intel.

[william-silversmith]

Do you mean that you can't install the x86 binary due to the code signing issue or is it only at compile time? Let me try...

Update: seemed to work for me in rosetta2 mode

[jefferis]

@scachero and I ran into the simplejpeg & cmake issue earlier in the week after he updated the OS. I think he ended up suspending code signing for the terminal and then everything else bar zfpy worked. For zfpy conda install was the only way he could find.

[william-silversmith]

Interesting, I don't think I've run into the code signing issue (or if I did it was a long time ago and I disabled something 😅). What version of the OS are you using?

[scachero]

That’d be ventura S

…

On 29 Nov 2022, at 04:24, William Silversmith ***@***.***> wrote: Interesting, I don't think I've run into the code signing issue (or if I did it was a long time ago and I disabled something 😅). What version of the OS are you using? — Reply to this email directly, view it on GitHub <#570 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHYI26IBDUHE57QLAC6XOTTWKWAPFANCNFSM6AAAAAASMKYAPQ>. You are receiving this because you were mentioned.

[ceesem]

Yeah, it's related to OS X 13. This stack overflow post mentions a code signing issue for cmake versions prior to 3.24.2. Although I haven't seen reference to it anywhere else, it matches the behavior perfectly. Unfortunately, simplejpeg recently added a build requirement of cmake<=3.22.5 due to some other issue.

[ceesem]

Btw, using a Rosetta terminal wasn't quite sufficient for me when working with conda. I had to also follow the advice from this post to create a conda environment, which basically amounts to adding the architecture variable when creating and adding a config afterwards:

CONDA_SUBDIR=osx-64 conda create -n myenv_x86 python=3.9
conda activate myenv_x86
conda config --env --set subdir osx-64

[william-silversmith]

What happened when you used the Rosetta2 terminal? Here's what happens for me using a clean py39 and py310 env.

[ceesem]

@william-silversmith

Hi Will, these install issues remain and it's increasingly concerning given that cloudvolume is a fundamental client package for analysis and yet currently requires a number of manual hacks and/or workarounds to function on a chipset that has been around for more than 2 years. Would you consider a design pattern based around more optional installs, particularly for compression formats that may not be used by many or even most cloudvolume users who, for example, may just want to download a mesh or be working on projects that only use certain formats?

As an aside, a better simplejpeg workaround is to pip install from the source linked to here: https://gitlab.com/jfolz/simplejpeg/-/issues/15 . It's not clear to me why this hasn't been incorporated into a new version, however.

[william-silversmith]

Hi Casey,

Despite this chipset being available for 2 years, it seems that the wider ecosystem is still having problems, which is why the zfp people ran into issues with their CI build system - Github Actions doesn't provide this chipset as an automated runner. The reason most CloudVolume libraries support Apple Silicon natively is due to the happy coincidence that I got this machine type around the time it came out and control most libraries. However, you are a key user, so it makes sense to make your installs smooth.

I re-raised the issue with zfpy and pointed them towards my pre-compiled binaries. I'd be happy to make zfpy an optional dependency. Is that okay without you @nkemnitz and @macrintr? I believe you guys just started using zfpc more heavily for alignment vectors: ZettaAI/corgie#6

I will pin simplejpeg==1.6.3. Looking through the release history for libjpeg-turbo, it seems like we shouldn't be running into any problems based on our use case.

Will

[william-silversmith]

Hi Casey, it looks like the zfpy guys have uploaded my universal2 wheels to PyPI, but they have to clean up the PyPI first to remove the non-working wheels which are taking priority. Once they do that, the zfpy problem should be resolved. I will release CloudVolume with the simplejpeg pin today.

[william-silversmith]

Hi Casey, the zfpy issues should be resolved now. Would you mind trying a clean install of CloudVolume and let me know if it works for you?

LLNL/zfp#175 (comment)

[ceesem]

It worked on a fresh Python 3.9 conda environment! Thank you!

I still feel like having more modular requirements would have benefits across a number of use cases, but this definitely fixes one of the biggest current issues.

[ceesem]

Hit the wrong button there!

This worked for me on a clean python 3.9 environment. Thank you!

I still think a more modular approach would have benefits in use cases where cloudvolume is used as part of a service to just receive a narrow range data. but this solves probably the biggest problem we had at the moment.

[william-silversmith]

Let's reopen that as a separate issue as this is centered around Apple Silicon compatibility.

#576