resourceForSoftMatter

free online resource for soft matter (colloid) research

Contents

• Particle tracking
• Movie/Image
• Big data visualization
• Crystal defect identification
• Thesis
• Online learning materials
• Linux
• Latex and PDF files
• Useful data structures and algorithms

Particle tracking

• trackpy provides a mature version of python particle tracking implementation of the famous IDL tracking code from John Crocker and David Grier (www.physics.emory.edu/~weeks/idl/). Furthermore, it implements a few more advanced trackers to handle complicated situations. However, it seems that it lacks an image noise filter (noise filter is now supported in the latest version).

• Particle Identification and Tracking: Particle Identification and Tracking (http://tacaswell.github.io/tracking/html/) This is an implementation in C++ of the particle tracking algorithms developed by Croker and Grier.

• Matlab Particle Tracking

• circletracking: python toolkits for tracking circles and ellipses in 2D or 3D images

• EllipsoidFit can handle 3D ellipsoid

• Rectangle_fit can handle rod

• TubeTK is an open-source toolkit for the segmentation, registration, and analysis of tubes and surfaces in images, developed by Kitware, Inc.

• MathieuLeocmach/colloids is a C++ and python library to process both experimental and simulation data of colloidal particles. The main features are particle tracking from confocal microscopy images and analysis of local structure and dynamics of particles (from simulations or experiments).

• feature particles with multiscale sizes (jupyter-notebook)

• Colloid+polymer mixture phase diagram obtained from free volume theory (jupyter-notebook)

• Some comments and info on 2D feature finding: https://github.com/scikit-beam/scikit-beam/wiki/2-D-Feature-Finding

• Hough transform (1972), which started out as a technique to detect lines in image, has been generalized and extended to detect curves in 2D and 3D. It has been implemented in scikit-image

Movie/Image

• ffmpeg for making movies, and converting movie to images. It is a cross-platform solution to record, convert and stream audio and video. It includes libavcodec - the leading audio/video codec library. h.264 encoder is not included in this library. Hence if h.264 is the desired format, it is necessary to install x264 and compile ffmpeg with x264 enabled. Here is an example to use ffmpeg (10 fps, bitrate: 1MB/s) in bash in Linux

ffmpeg -i figs/%04d.png -vcodec mpeg4 -r 10 -b:v 1M output.avi

• html files can embed videos, so that one can use browser to show movie. Media formats supported by HTML (Browser compatibility) should be

  • Theora and Vorbis in Ogg
  • H.264 and MP3 in MP4
  • H.264 and AAC in MP4
  • WebM

• In PowerPoint 2013 and later, for the best video playback experience, it is recommended to use .mp4 files encoded with H.264 video (a.k.a. MPEG-4 AVC) and AAC audio. For audio, use .m4a files encoded with AAC audio.

• ImageMagick: Convert, Edit, Or Compose Bitmap Images

• import tool from ImageMagick can be used to capture the screenshot. Run

import screenshot.png

and select the window you want to capture or select a region by pressing the left mouse button and dragging. Ref: http://askubuntu.com/questions/194427/what-is-the-terminal-command-to-take-a-screenshot

• Inkscape is professional quality vector graphics software which runs on Linux, Mac OS X and Windows desktop computers. It can be used to extract and edit figures from scientific papers in PDF format. Its default format is SVG. To convert pdf figures to SVG, one can use pdftocairo which has been installed on most Linux computers:

pdftocairo -svg input.pdf

• https://en.wikipedia.org/wiki/Phi shows the Unicode for different forms of greek letter phi. The most important one is U+03D5, which shows the phi letter similar to the latex font, commonly used in math and technical contexts.

• pims: Python Image Sequence: Load video and sequential images in many formats with a simple, consistent interface. It has the same authors as trackpy.

• moviepy is a Python module for video editing. It can read and write all the most common audio and video formats, including GIF, and runs on Windows/Mac/Linux

• neural-enhance Super Resolution for images using deep learning.

Big data visualization

• Bokeh is a python project which targets browser-based graphics, and recent releases are beginning to do big data in the browser the right way.

• VisPy is another effort to provide easy visualization of large datasets with python. It is based on OpenGL, with plans to add a WebGL backend.

• mpld3 brings together Matplotlib, the popular Python-based graphing library, and D3js, the popular Javascript library for creating interactive data, which can output SVG files. The library may require a little knowledge on javascript to enable flexible functions. It provides an interactive way to data visualization, rather than aiming on big data. d3.js gallery: https://github.com/d3/d3/wiki/Gallery. An example can be found: Visualizing MBTA Data: http://mbtaviz.github.io/

• OVITO is a scientific visualization and analysis software for atomistic simulation data developed by Alexander Stukowski at Darmstadt University of Technology, Germany. It can be also used to visualize experimental data if the data is saved as the one of the following formats: LAMMPS, XYZ, IMD, CFG, POSCAR, AMBER/NetCDF, PDB, GSD/HOOMD, and VTK. See the description of those formats: http://ovito.org/manual/usage.import.html

• https://github.com/jbmouret/matplotlib_for_papers uses matplotlib to plot statistical figures. Good example for box plot, Stars (statistical significance).

• pubplot is a python module for making publication-quality figures with matplotlib.

• Practical Data Visualization (ppt)

Crystal defect identification

• OVITO implemented dislocation detection (DXA).

• BiDef is a python package using supervised machine learning for crystal defect identification. See details in the author's paper. It also seems that the package has something to do with ovito.

• Atoman provides a python wrapper (written by C extension) on voro++, and algorithms to detect point defects by voronoi.

Thesis

• Theses from Arjun G. Yodh's group in UPenn

• Tutorial and thesis from group of David A. Weitz

• W.L. Miller's thesis: Janus particles and aspherical particles MD simulation

• This thesis gives a note on FMT of hard spheres.

Online learning materials

• The Feynman Lectures on Physics (high resolution, read online)

• SklogWiki is an open-edit encyclopedia dedicated to thermodynamics and statistical mechanics, especially that of simple liquids, complex fluids, and soft condensed matter.

• Lecture notes on statistical mechanics

• Theoretical model for a binary mixture of oppositely charged colloids with counterions and salt (jupyter-notebook) It contains functions to create binary PY hardsphere RDFs using analytical results.

• Guideline for data sharing

• Reading academic papers

• Reviewing academic papers (as a referee)

• Lecture slides for Coursera's Data Analysis class

• List of free programming books

• Good coding practices for scientific programming (see the wiki)

• UC Berkeley Course on Deep Learning

• Notebooks on regression using python (jupyter-notebook)

• Example Python notebooks for data science

• Good tutorial on python for scientific computation

• Some interesting small student projects written in MATLAB on game theory, dynamical system, and other social systems

• Deep Learning Papers Reading Roadmap

• dataquest blog

• Statistics on github repos Rank-star distribution follows a power-law.

• A next-generation curated knowledge sharing platform for data scientists and other technical professions. Developed by Airbnb. Now in beta version.

• A machine learning approach to classify songs by mood Music are getting sad for several decades ⭐

• simple overview of python, numpy, scipy, matplotlib functions that are useful for scientific work

• A list of tools for general research

Linux

• To extract rpm package

   rpm2cpio package.rpm | cpio -idmv

  ref: http://stackoverflow.com/questions/18787375/how-do-i-extract-the-contents-of-an-rpm

• To extract deb package

   ar p package.deb data.tar.gz | tar xvz --strip-components 2

• Use the repoquery tool from the yum-utils package to check rpm package dependence

   repoquery --requires <package>
  

  To see local rpm package dependence (e.g. already downloaded)

   rpm -qp --requires <package file>
  

  ref: http://superuser.com/questions/294662/how-to-get-list-of-dependencies-of-non-installed-rpm-package

Latex and PDF files

• Latex Wikibook is a very good start point to learn latex.

• Overleaf is an online LaTeX and Rich Text collaborative writing and publishing tool that makes the whole process of writing, editing and publishing scientific documents much quicker and easier.

• An interactive introduction to LaTeX using Overleaf.

• Overleaf templates: Start your projects with quality LaTeX templates for journals, CVs, resumes, papers, presentations, assignments, letters, project reports, and more.

• zotero is a free, easy-to-use tool to help you collect, organize, cite, and share your research sources. It is a useful reference manager which can export bibtex.

• How to use ZOTERO

• Official repository for Citation Style Language (CSL) citation styles

• merge PDF files:

• pdfunite is a part of poppler, however, someone pointed out that the output.pdf may be very large while gs packs a small size.

pdfunite in1.pdf in2.pdf out.pdf

• use gs

gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=out.pdf in1.pdf in2.pdf

• Markdown is a lightweight and easy-to-use syntax for styling all forms of writing, which can be converted to various formats, such as Latex, html, pdf, etc.

  • Mastering Markdown 3 minutes read

• Pandoc filters list

• A two-page cheatsheet for restructured text

Useful data structures and algorithms

• https://github.com/jwasham/google-interview-university gives a lots of well-organized links and learn materials on commonly used data structure, as well as many other basic knowledge on programming.

• Path to a free self-taught education in Computer Science!

• Path to a free self-taught education in Data Science!

• k-d tree is a space-partitioning data structure for organizing points in a k-dimensional space. It is useful to obtain nearest neighbors with an O(log n) complexity. scipy.spatial.KDTree and scipy.spatial.cKDTree are two very fast implementations in python and C. (https://github.com/prody/ProDy/tree/master/prody/kdtree) has an implementation of kd-tree which can handle periodic boundary condition. A more general algorithm is cover tree which can handle any distance metric in arbitrary high dimensional space. There are some C++ cover-tree implementations on github. The speed of the algorithm is very fast: one of implementations claimed that it takes 250 seconds for inserting 10^6 1000-dimensional vectors, and achieves 300 queries per second.

• Some C/C++ single-file libraries useful for making parts of calculation fast.

• libccd is a library for collision detection between two convex shapes.

• NVT-GJK NVT-MC Hard Particle Simulation

• multistate Bennett acceptance ratio (MBAR) estimating expectations and free energy differences. Ref: Shirts MR and Chodera JD. Statistically optimal analysis of samples from multiple equilibrium states. J. Chem. Phys. 129:124105 (2008).

• A gallery of interesting IPython Notebooks

• Network analysis on Python package Dependency (data available for 20k packages)

• https://gist.github.com/nzjrs/990493 shows ways to interface python + C via ctypes or python module extensions.

• https://github.com/fengwangPhysics/ctypes-example is an example module to interface python + C via ctypes or python module extensions.

• boost-python is a common way to interface C++ to Python.

• boost-python-examples

• Boost.Python interface for NumPy

• pybind11 is a simple interface to port C++11 to Python and vice versa.

• Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation. It has already been used in many python packages.

• Center of mass of systems with periodic boundary condition. The algorithm used here is from https://en.wikipedia.org/wiki/Center_of_mass#Systems_with_periodic_boundary_conditions Here is an example implementation in Python

def centerOfMass(x, L):
    """Suppose x is in interval [0,L]"""
    theta = x/L*2.*np.pi
    xi = np.cos(theta)
    zeta = np.sin(theta)
    thetaBar = np.arctan2(-zeta.mean(), -xi.mean()) + np.pi
    return L*thetaBar/2./np.pi

• Regular expression cheat sheet useful in data wrangling

• tsfresh Automatic extraction of relevant features from time series

• Time Series analysis tsa

• EasyOpenCL Examples for using OpenCL with C++. It uses OpenCL for GPU-computing.

• TensorFlow Computation using data flow graphs for scalable machine learning

• Fast Style Transfer in TensorFlow Add styles from famous paintings to any photo in a fraction of a second! Training takes a much longer time (4-6 hours on a Maxwell Titan X). ⭐

• Image style transfer: history and future Learning and Combining Multiple Styles

• kmpfit provides a complete tutorial on curve fitting, uncertainty estimation of parameters, confidence and prediction intervals for fittings. kmpfit is part of the kapteyn package, a Python module for astronomical applications.

• Matrix exponential by Pade approximation

• Invert Pade approximation back to Taylor series with python

• Codes for growing soft solids and crumpling, wrinkling and tearing of thins sheets, developed by Tuomas Tallinen