SMR-Use_Mindat_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qrMhTOpOyjAP"
   },
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/ChuBL/How-to-Use-Mindat-API/blob/main/How_to_Use_Mindat_API.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u65PipxXuNbO"
   },
   "source": [
    "# How to Use OpenMindat Data API to Query and Download Datasets\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OOT8B2v7odok"
   },
   "source": [
    "## 0. Access Your Mindat API Token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "59gsuIxcoiIU"
   },
   "source": [
    "[How to Get My Mindat API Key or Token?](https://www.mindat.org/a/how_to_get_my_mindat_api_key)\n",
    "\n",
    "[Mindat API doc](https://api.mindat.org/schema/redoc/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YmvOE3ClX207"
   },
   "source": [
    "## 1. Dependencies (Run this section first)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UublqFDuYxeU"
   },
   "source": [
    "This set of examples runs with files that are located in the file directory with the python notebook .ipynb file, for running with Jupyter or similar host in your local desktop environment with an active internet connection. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "gpxJUYWHaPCw"
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "import sys\n",
    "import json\n",
    "import re\n",
    "import pprint\n",
    "import requests\n",
    "# import google"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RhAUoA9pcIQv"
   },
   "source": [
    "You should **avoid** placing your API token explicitly in your code if you plan to share it. That would include working with a notebook that is in a public Github repo, like this one.\n",
    "\n",
    "The solution here is to have the api key saved in a file accessible from your notebook environment (e.g. in the same directory), and adding that file to your github .ignore file so it is not copied to the public Github repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "nW9tqw7ScEs9"
   },
   "outputs": [],
   "source": [
    "\n",
    "api_key_file_dir = \"local/api_key.txt\"\n",
    "try:\n",
    "    with open(api_key_file_dir, 'r') as f:\n",
    "        YOUR_API_KEY = f.read()\n",
    "except FileNotFoundError:\n",
    "    print(\"API key file not found. Please create a text file containing your api key and place it in the correct directory.\")\n",
    "    \n",
    "try:\n",
    "    assert 0 != len(YOUR_API_KEY)\n",
    "except AssertionError:\n",
    "    raise Exception(\"Please set a valid API token before the start!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UEMu5YSJwdC2"
   },
   "source": [
    "### Basic access pattern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "RrIOSXhLwdC2"
   },
   "outputs": [],
   "source": [
    "# API root entry point\n",
    "MINDAT_API_URL = \"https://api.mindat.org\"\n",
    "\n",
    "# authorization header that must be included with each request.\n",
    "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': 48408,\n",
       "  'name': 'Dunite',\n",
       "  'synid': 0,\n",
       "  'ima_status': 0,\n",
       "  'ima_approved': 0},\n",
       " {'id': 51341,\n",
       "  'name': 'Metadunite',\n",
       "  'synid': 0,\n",
       "  'ima_status': 0,\n",
       "  'ima_approved': 0}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# see https://api.mindat.org/schema/redoc/ for documentation on using the Mindat API\n",
    "\n",
    "\n",
    "# using the geomaterials_search endpoint. this executes a text search using the term in the \n",
    "#  q parameter, and returns a fixed set of fields. \n",
    "# Apparently these are id','name': ,'synid','ima_status', 'ima_approved'\n",
    "# other parameters besides 'q' are ignored\n",
    "\n",
    "filter_dict = {\n",
    "    'q':'dunite',\n",
    "}\n",
    "\n",
    "params = filter_dict\n",
    "\n",
    "# use python requests package to GET results from mindat\n",
    "response = requests.get(MINDAT_API_URL+\"/geomaterials_search/\",\n",
    "                    params=params,\n",
    "                    headers=headers)\n",
    "# assume the query succeeds. See later examples for using the response.status_code\n",
    "#  to check if the request worked.\n",
    "\n",
    "#  another handy requests package function to cast the results to JSON.\n",
    "json_file = response.json()\n",
    "\n",
    "# this will echo the json file containing the query results.\n",
    "json_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OQ0gnOdPnlZX"
   },
   "source": [
    "### Get the Items with Selected Fields\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vgRLWpJooBhx"
   },
   "source": [
    "#### List of Mindat API fields: \n",
    "~all\n",
    "a\n",
    "aboutname\n",
    "aerror\n",
    "alpha\n",
    "alphaerror\n",
    "approval_year\n",
    "b\n",
    "berror\n",
    "beta\n",
    "betaerror\n",
    "c\n",
    "cclass\n",
    "cerror\n",
    "cim\n",
    "cim\n",
    "cleavage\n",
    "cleavagetype\n",
    "colour\n",
    "commentbreak\n",
    "commentcolor\n",
    "commentcrystal\n",
    "commentdense\n",
    "commenthard\n",
    "commentluster\n",
    "csmetamict\n",
    "csystem\n",
    "dana8ed1\n",
    "dana8ed2\n",
    "dana8ed3\n",
    "dana8ed4\n",
    "dcalc\n",
    "dcalcerror\n",
    "description_short\n",
    "diapheny\n",
    "discovery_year\n",
    "dmeas\n",
    "dmeas2\n",
    "dmeaserror\n",
    "electrical\n",
    "elements\n",
    "entrytype\n",
    "entrytype_text\n",
    "epitaxidescription\n",
    "fracturetype\n",
    "gamma\n",
    "gammaerror\n",
    "groupid\n",
    "guid\n",
    "hardtype\n",
    "hmax\n",
    "hmin\n",
    "id\n",
    "ima_formula\n",
    "ima_history\n",
    "ima_notes\n",
    "ima_status\n",
    "impurities\n",
    "industrial\n",
    "ir\n",
    "key_elements\n",
    "longid\n",
    "luminescence\n",
    "lustre\n",
    "lustretype\n",
    "magnetism\n",
    "meteoritical_code\n",
    "mindat_formula\n",
    "mindat_formula_note\n",
    "morphology\n",
    "name\n",
    "nolocadd\n",
    "occurrence\n",
    "optical2vcalc\n",
    "optical2vcalc2\n",
    "optical2vcalcerror\n",
    "optical2vmeasured\n",
    "optical2vmeasured2\n",
    "optical2vmeasurederror\n",
    "opticalalpha\n",
    "opticalalpha2\n",
    "opticalalphaerror\n",
    "opticalanisotropism\n",
    "opticalbeta\n",
    "opticalbeta2\n",
    "opticalbetaerror\n",
    "opticalbireflectance\n",
    "opticalbirefringence\n",
    "opticalcolour\n",
    "opticalcomments\n",
    "opticaldispersion\n",
    "opticalepsilon\n",
    "opticalepsilon2\n",
    "opticalepsilonerror\n",
    "opticalextinction\n",
    "opticalgamma\n",
    "opticalgamma2\n",
    "opticalgammaerror\n",
    "opticalinternal\n",
    "opticaln\n",
    "opticaln2\n",
    "opticalnerror\n",
    "opticalomega\n",
    "opticalomega2\n",
    "opticalomegaerror\n",
    "opticalpleochorismdesc\n",
    "opticalpleochroism\n",
    "opticalr\n",
    "opticalsign\n",
    "opticaltropic\n",
    "opticaltype\n",
    "other\n",
    "otheroccurrence\n",
    "parting\n",
    "polytypeof\n",
    "publication_year\n",
    "rock_bgs_code\n",
    "rock_parent\n",
    "rock_parent2\n",
    "rock_root\n",
    "shortcode_ima\n",
    "sigelements\n",
    "spacegroup\n",
    "spacegroupset\n",
    "specdispm\n",
    "streak\n",
    "strunz10ed1\n",
    "strunz10ed2\n",
    "strunz10ed3\n",
    "strunz10ed4\n",
    "synid\n",
    "tenacity\n",
    "thermalbehaviour\n",
    "tlform\n",
    "tranglide\n",
    "twinning\n",
    "type_specimen_store\n",
    "updttime\n",
    "uuid2mindat\n",
    "uv\n",
    "va3\n",
    "varietyof\n",
    "vhnerror\n",
    "vhng\n",
    "vhnmax\n",
    "vhnmin\n",
    "vhns\n",
    "weighting\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Mindat IDs from a list of html links in a Microsoft word document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KUus01VVnrLn"
   },
   "outputs": [],
   "source": [
    "#repeating for convienence\n",
    "\n",
    "MINDAT_API_URL = \"https://api.mindat.org\"\n",
    "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "You might need to install docx\n",
    "\n",
    "pip install docx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "\n",
    "from docx import Document\n",
    "from docx.opc.constants import RELATIONSHIP_TYPE as RT \n",
    "# Usage example  This file just contains a list of material types, each is a link with a URL \n",
    "  # behind the label text you see in the document.\n",
    "document_path = 'MeteoriteExtraterrestrial.docx'\n",
    "hyperlinks = []\n",
    "\n",
    "print(\" This program extracts hyperlinks detected in a word .docx file \\n\")\n",
    "\n",
    "document = Document(document_path)\n",
    "\n",
    "rels = document.part.rels\n",
    "\n",
    "for rel in rels:\n",
    "   if rels[rel].reltype == RT.HYPERLINK:\n",
    "      #print(\"\\n Original link id -\", rel, \"with detected URL: \", rels[rel]._target)\n",
    "      hyperlinks.append(rels[rel]._target)\n",
    "    \n",
    "idlist = []\n",
    "# Print the extracted URLs\n",
    "for url in hyperlinks:\n",
    "#    print(url)\n",
    "#    print(url[27:32])\n",
    "    idlist.append(url[27:32])\n",
    "    \n",
    "print (idlist)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xpu636_dpIWs"
   },
   "outputs": [],
   "source": [
    "# set your selected fields here\n",
    "# fields_str = 'id','longid','guid','name','entrytype','entrytype_text','description_short','rock_parent','rock_parent2','rock_root','rock_bgs_code','meteoritical_code'\n",
    "\n",
    "fields_str = 'id','longid','name', 'updttime'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "idlist = ['11263','48145','49089','49091','49093','49504','49505','49506','49507','49508','49509','49510','49511','49512','49513','49514','49517','49518','49519','49520','49521','49522','49523','49524','49525','49526','49527','49528','49529','49530','49531','49532','49533','49534','49535','49536','49537','49538','49539','49540','49541','49542','49543','49544','49545','49546','49547','49548','49549','49550','49551','49552','49553','49554','49556','49557','49558','49559','49560','49561','49562','49563','49564','49565','49566','49567','49568','49569','49570','49571','49572','49573','49574','49575','49576','49577','49578','49579','49580','49581','49584','49585','49586','49587','49588','49589','49590','49591','49592','49593','49595','49596','49597','49599','49600','49603','49604','49606','49607','49608','49609','49610','49612','49614','49615','49616','49617','49618','49619','49620','49621','49622','49623','49625','49626','49627','49628','49629','49630','49631','49632','49633','49634','49635','49636','49637','49638','49639','49640','49641','49642','49643','49644','49645','49646','49647','49648','49649','49650','49651','49652','49653','49654','49655','49656','49657','49658','49659','49660','49661','49662','49663','49664','49665','49666','49667','49668','49669','49670','49671','49672','49673','49674','49675','49676','49677','49678','49679','49680','49682','49683','49684','49685','49686','49688','49689','49690','49691','49692','49693','49695','49696','49697','49698','49699','49700','49702','49703','49705','49707','49708','49709','49710','49711','49712','49713','49715','49716','49717','49718','49719','49720','49721','49722','49723','49724','49725','49726','49727','49728','49729','49730','49731','49732','49733','49734','49735','49736','49737','49738','49739','49740','49741','49742','49743','49744','49745','49746','49750','49751','49752','49753','49754','49755','49756','49757','49758','49760','49761','49762','49763','49764','49765','49767','49768','49769','49770','49772','49773','49774','49776','49777','49778','49779','49780','49781','49782','49783','49784','49785','49786','49787','49788','49789','49790','49791','49792','49793','49794','49795','49796','49797','49798','49799','49800','49802','49803','49804','49805','49806','49807','49808','49809','49810','49811','49812','49813','49814','49815','49816','49817','49818','49819','49820','49821','49822','49823','49824','49825','49826','49827','49828','49829','49831','49832','49833','49835','49836','49837','49838','49839','49840','49841','49842','49843','49844','49845','49846','49847','49848','49849','49850','49851','49852','49853','49854','49856','49857','49859','49860','49861','49862','49863','49864','49865','49866','49867','49869','49872','49873','49876','49877','49878','49879','49880','49881','49882','49883','49884','49885','49886','49887','49888','49889','49890','49891','49892','49893','49894','49895','49896','49898','49899','49901','49902','49904','49905','49906','49907','49908','49909','49910','49911','49912','49913','49914','49915','49916','49917','49918','49919','49920','49921','49922','49923','49924','49925','49926','49927','49928','49929','49930','49931','49932','49933','49934','49935','49936','49937','49938','49939','49941','49942','49944','49945','49947','49950','49951','49952','49953','49954','49955','49956','49957','49958','49959','49960','49961','50269','50270','50444','50445','50446','50447','50448','50449','50450','50451','50452','50453','50454','50455','50456','50457','50458','50459','50460','50461','50462','50463','51453','52197','52198','52199','52200','52201','52202','52203','52204','52205','52206','52207','52208','52209','52210','52211','52212','52213','52214','52215','52216','52217','52218','52219','52220','52221','52222','52223','52224','52225','52226','52227','52228','52229','52237','52238','52239','52240','52241','52360','52368','52369','52370','52371','52372','52373','52374','52393','52394','52395','52396','52720','52787','54116','54117']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "query for items by id in an id list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "j7z0PF1WpVC1",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "import time\n",
    "\n",
    "json_array = []\n",
    "\n",
    "# idlist = ['11263','48145']\n",
    "\n",
    "for idstr in idlist:\n",
    "\n",
    "    params = {\n",
    "        'fields': fields_str, # put your selected fields here\n",
    "        'id__in': idstr, # set the item amount for each page\n",
    "        'format': 'json'\n",
    "    }\n",
    "\n",
    "    response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
    "                    params=params,\n",
    "                    headers=headers)\n",
    "\n",
    "    if 200 <= response.status_code <= 299:\n",
    "        json_out = response.json()\n",
    "    #    print (json_out)\n",
    "        json_array = json_array + json_out[\"results\"]\n",
    "    else:\n",
    "        print ('problem-- ', idstr)\n",
    " \n",
    "    time.sleep(3)\n",
    "    \n",
    "    \n",
    "print ('Done')    \n",
    "json_array\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "use the text search interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# get all records that have a mereoritical_code value\n",
    "# have to use cursor pagination\n",
    "\n",
    "import pandas\n",
    "import time\n",
    "\n",
    "MINDAT_API_URL = \"https://api.mindat.org\"\n",
    "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
    "\n",
    "#fields_str = 'id','longid','guid','name'\n",
    "\n",
    "json_array = []\n",
    "\n",
    "params = {\n",
    "    'fields': fields_str, # put your selected fields here\n",
    "    'format': 'json',\n",
    "    'meteoritical_code_exists':'true'\n",
    "}\n",
    "\n",
    "response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
    "                params=params,\n",
    "                headers=headers)\n",
    "\n",
    "if 200 <= response.status_code <= 299:\n",
    "    json_out = response.json()\n",
    "#    print (json_out)\n",
    "    json_array = json_array + json_out[\"results\"]\n",
    "else:\n",
    "    print ('problem')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_array\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {}\n",
    "\n",
    "while json_out[\"next\"] is not None :\n",
    "    response = requests.get(json_out[\"next\"],\n",
    "                params=params,\n",
    "                headers=headers)\n",
    "    \n",
    "    print (response.status_code)\n",
    "    if 200 <= response.status_code <= 299:\n",
    "        json_out = response.json()\n",
    "    #    print (json_out)\n",
    "        json_array = json_array + json_out[\"results\"]\n",
    "    else:\n",
    "        print ('problem-- ', json_out[\"next\"])\n",
    "\n",
    "print ('Done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_result = pandas.DataFrame(json_array)\n",
    "\n",
    "df_result.to_csv('timestamp.csv') \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Load the JSON array\n",
    "# Create a DataFrame from results\n",
    "df_nested_list = pandas.json_normalize(json_array, record_path =['results'])\n",
    "\n",
    "# Display the DataFrame\n",
    "print(df_nested_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_nested_list.to_csv('49089.csv') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "extract a list of ids and long-id for all minerals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fields_str = 'id','longid'\n",
    "\n",
    "params = {\n",
    "    'fields': fields_str, # put your selected fields here\n",
    "    'format': 'json'\n",
    "}\n",
    "\n",
    "response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
    "                params=params,\n",
    "                headers=headers)\n",
    "\n",
    "if 200 <= response.status_code <= 299:\n",
    "    json_out = response.json()\n",
    "#    json_array.append(json_out)\n",
    "else:\n",
    "    print ('problem ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "same operation, but iterate through allpages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a DataFrame from results\n",
    "df_nested_list = pandas.json_normalize(json_out, record_path =['results'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_out\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_nested_list.to_csv('id-longid.csv') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mindat check sum algorithm\n",
    "# from Jolyon,2023-06-18\n",
    "\n",
    "def mindat_longid(authority, type, id):\n",
    "    out = \"{}:{}:{}:\".format(authority, type, id)\n",
    "    out2 = \"{}{}{}\".format(authority, type, id)\n",
    "    t = 0\n",
    "    for i in range(len(out2)):\n",
    "        if i % 2 == 1:\n",
    "            t += int(out2[i]) * 3\n",
    "        else:\n",
    "            t += int(out2[i])\n",
    "    ck = t % 10\n",
    "    if ck:\n",
    "        ck = 10 - ck\n",
    "    out += str(ck)\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run checksum function\n",
    "mindat_longid(1,1,49602)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}