This page will give more detailed setup instructions for Caiman than the main readme in the source tree. This is the place you should visit first if you run into problems and need to troubleshoot. It includes info on initial setup, as well as updating with new releases. There is a Table of Contents on the left-hand side of this page, so if you need help with a particular task, that should help you get oriented.
If the information on this page doesn't help, please ask us for help at Gitter, or open an issue.
Caiman setup consists of two main steps:
- Install Caiman
- Use caimanmanager to download sample code and data to a
caiman_datadirectory.
We will discuss each of these steps for different operating systems below. In a separate section, we will discuss how to upgrade once you've already installed.
If you do not already have conda installed, first install a 3.x version for your platform here. When installing, allow conda to modify your PATH variable. If you are using an M1-based Mac, please ignore the ARM builds of conda; install an x86 version instead (ignore any warnings you get while doing so; it will work fine).
We recommend you familiarise yourself a little bit with Conda before going further, if you are not already (https://docs.conda.io/projects/conda/en/latest/user-guide/index.html).
There are two main ways to install Caiman: you can install using conda (package-based install), or you can install in development-mode. The first will be appropriate for almost all users -- those that want to get started quickly using Caiman to analyze data. The second, development-mode installation, is for those who want to install an editable version of Caiman in order to tweak to the code base, and make contributions to Caiman.
These are basically the Quickstart instructions on the main readme page at the repo: <https://github.com/flatironinstitute/CaImAn>, so if you follow those, you should be good to go.
Details for conda install
This process is the same on every operating system, and is what most users will need who just want to use Caiman to get things running quickly. You will install mamba into your base environment, create a new environment with the caiman package from conda-forge, and then activate the new environment.
conda install -n base -c conda-forge mamba
mamba create -n caiman -c conda-forge caiman
conda activate caimanNote if you are installing on Windows, run the above commands in the anaconda prompt rather than powershell or the dos shell:
Known issues
If you are on Windows, have used Caiman before using our github repo and now want to use the conda-forge package,
you might encounter some errors with Python reading the files from the wrong directory. In this case, rename
(or remove) the caiman directory that contains the source of the repo and the caiman_data folder and then proceed
with setting up the caiman_data folder as explained below in Section 2.
Dev mode install allows you to modify the source files of Caiman and makes it easier to contribute to the project, fix bugs etc.
If you install in dev mode you will likely need to set some environment variables manually (it is done automatically when you do the conda install): this is discussed in Section 4C.
Dev Mode Installation on Windows
The Windows installation process differs quite a bit from installation on Linux or MacOSX. If you can work on Linux, it will be easier. Everything below should be from a Conda-prompt rather than from Powershell or any other shell.
- Install Microsoft Build Tools for Visual Studio 2019, which you can download from (https://visualstudio.microsoft.com/vs/older-downloads/). Check the “Build Tools” box, and in the detailed view on the right check the “C/C++ CLI Tools” component too. The specifics of this occasionally change as Microsoft changes its products and website; you may need to go off-script.
- Remove any associations you may have made between .py files and an existing python interpreter or editor
- If you have less than 64GB of RAM, increase the maximum size of your pagefile to 64G or more (http://www.tomshardware.com/faq/id-2864547/manage-virtual-memory-pagefile-windows.html). The Windows memmap interface is sensitive to the maximum setting and leaving it at the default can cause errors when processing larger datasets
At the conda prompt:
git clone [email protected]:flatironinstitute/CaImAn.git
cd CaImAn
mamba env create -f environment.yml -n caiman
conda activate caiman
mamba install -n caiman vs2019_win-64
pip install -e .A couple of things to note:
- If you don't want to develop code then replace the second-to-last command with
pip install . - If any of these steps gives you errors do not proceed to the following step without resolving it.
- If the environment doesn't active properly, there may be
batfiles that need to be removed. Use the Windows find-file utility (under the Start Menu) to look forvs2019_compiler_vars.batunder your home directory. If a copy shows up, delete the version that has yourcaimanenvironment name as part of its location. You may then continue the installation.
Installation on MacOS and Linux
git clone [email protected]:flatironinstitute/CaImAn.git
cd CaImAn/
mamba env create -f environment.yml -n caiman
source activate caiman
pip install -e .If you don't want to develop code then replace the last command with
pip install . If any of these steps gives you errors do not
proceed to the following step without resolving it
Known issues
If you recently upgraded to OSX Mojave you may need to perform the following steps before your first install:
xcode-select --install
open /Library/Developer/CommandLineTools/Packages/and install the package file you will find in the folder that pops up
Once Caiman is installed, you will likely want to set up a working directory with code samples and datasets.
The installation step in Section 1 produced a command caimanmanager that handles this. caimanmanager will
place demos and data in a caiman_data folder in your home directory. Install using:
caimanmanager install
if you used the conda-forge package or the pip install . option.
If you installed using the developer-mode option (installing with pip install -e .) then run caimanmanager with:
caimanmanager install --inplace
If you prefer to manage this information somewhere other than your home directory, the
CAIMAN_DATA environment variable can be set to customise it. The caimanmanager tool
and other libraries will respect that.
Upgrading can mean a couple of things. First, it typically means there has been a new release of Caiman, so you need
to install the new version of Caiman. Second, it could mean you need to upgrade changes to the demos in caiman_data
using caimanmanager. We'll discuss both options.
Updating the conda-forge package
From within your caiman environment type `conda update caiman -c conda-forge`. In most cases this should be enough.
If not, you may want to create a new environmrent from scratch.
- Remove your conda environment:
conda env remove -n caiman(or whatever you called the conda environment you used) - Remove or rename your ~/caiman_data directory
- Repeat the install instructions from above.
Updating in development mode
If you already have Caiman installed with the pip installer (development mode), but want to upgrade, please follow the procedure below. If you reinstall Caiman frequently, you can try skip deleting and recreating your Conda environment. In this case you can do only steps 1, 5, and 7 below to update the code. However, if the environment file has changed since your last update this may lead to you not the latest version. None of this applies to the conda-forge route (for which instructions are given above).
From the conda environment you used to install Caiman:
pip uninstall caiman- Remove your conda environment:
conda env remove -n caiman(or whatever you called the conda environment you used) - Close and reopen your shell (to clear out the old conda environment)
- Do a
git pullfrom inside your CaImAn folder. - Recreate and reenter your conda environment as you did in the installation instructions
- Do a
pip install .inside that code checkout - Run
caimanmanager installto reinstall the data directory (use--inplaceif you used thepip install -e .during your initial installation).
- If you used the
pip install -e .option when installing, then you can try updating by simply doing agit pull. Again, this might not lead to the latest version of the code if the environment variables have changed. - The same applies if you want to modify some internal function of
Caiman. If you used the
pip install -e .option then you can directly modify it (that's why it's the editable developer mode). If you used thepip install .option then you will need topip uninstall caimanfollowed bypip install .for your changes to take effect. Depending on the functions you're changing so you might be able to skip this step.
Upgrade the demos
When you upgrade Caiman, sometimes the underlying APIs change. When this happens and it impacts a demo (or otherwise
requires changes to files in caiman_data), we update the demo and data. This means that upgrading Caiman works
best if you also replace the caiman_data directory with a new version.
To check if the demos or datafiles have changed since your last install, you can run caimanmanager check. If they have not,
you may keep using them. If they have, we recommend moving your old caiman data directory out of the way (or just remove them if you have no
precious data in caiman_data) and updating caiman_data as described below.
However, you may also have made your own changes to the demos (e.g. to work with your data). If you have done this, you may need to massage your changes into the new versions of the demos. For this reason, we recommend that if you modify the demos to operate on your own data to save them as a different file to avoid losing your work when updating the caiman_data directory.
To update caiman_data you can follow the following procedure:
- If there are no new demos or files in the new Caiman distribution, then you can leave it as is.
- If you have not modified anything in
caiman_databut there have been changes in the new Caiman release, then removecaiman_datadirectory before upgrading and havecaimanmanagermake a new one after the upgrade, by running caimanmanager as discussed in Section 2. - If you have extensively modified things in
caiman_data, rename yourcaiman_datadirectory, and havecaimanmanagermake a new one after the upgrade, and then massage your changes back in. E.g., if you have extensively modifieddemo_pipeline.ipynbfor personal use, then change the name of this notebook before folding it back intocaiman_data.
32G RAM is required for a good experience, and depending on datasets, more may be necessary. Caiman is optimized for use by multiple CPUs, so workstations or clusters with multiple CPU cores are ideal (8+ logical cores). GPU computation is not used heavily by Caiman (though see Section 4D).
Right now, Caiman works and is supported on the following platforms:
- Linux on 64-bit x86 CPUs
- MacOS on 64-bit x86 CPUs or ARM CPUs
- Windows on 64-bit x86 CPUs
Support for Linux on ARM (e.g. AWS Graviton) is not available (but it may work with the port of conda, if you compile Caiman yourself - we do not have binary packages and this is untested). If you care about this, please let us know.
Caiman installs through the conda-forge conda channel. Some packages are available on multiple conda channels, and in this case it is important that you use the conda-forge channel if possible. To do this, when installing new packages inside your environment, use the following command:
mamba install -c conda-forge --override-channels NEW_PACKAGE_NAME
You will notice that any packages installed this way will mention, in their listing, that they are from conda-forge, with none of them having a blank origin. If you don't do this, differences between how packages are built in different channels could lead to some packages failing to work (e.g., OpenCV).
This is only important for people who are doing the dev-mode install. If you installed using the conda packaged-based install, this is done automatically.
To make the package work efficiently and eliminate "crosstalk" between different processes, some multithreading operations need to be turned off This is for Linux and Windows and is not necessary in OSX.
For Linux run these commands before launching Python:
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export VECLIB_MAXIMUM_THREADS=1For Windows run the same commands, replacing the word `export` with the word `set`.
The commands should be run every time before launching python. It is recommended that you save these values inside your environment so you don’t have to repeat this process every time. You can do this by following the instructions here.
See also: