1_SHARE_API_Basics_py3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calling the SHARE API\n",
    "----\n",
    "Here are some working examples of how to query the current SHARE database for individual results, metrics, and statistics.\n",
    "\n",
    "These particular queries are just examples, and the data is open for anyone to use, so feel free to make your own and experiment!\n",
    "\n",
    "Soon, we'll need a URL to access the SHARE Search API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "SHARE_API = 'https://staging-share.osf.io/api/search/abstractcreativework/_search'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The SHARE Search Schema\n",
    "\n",
    "The SHARE search API is built on a tool called elasticsearch. It lets you search a subset of SHARE's normalized metadata in a simple format.\n",
    "\n",
    "Here are the fields available in SHARE's elasticsearch endpoint:\n",
    "\n",
    "    - 'title'\n",
    "    - 'language'\n",
    "    - 'subject'\n",
    "    - 'description'\n",
    "    - 'date'\n",
    "    - 'date_created'\n",
    "    - 'date_modified\n",
    "    - 'date_updated'\n",
    "    - 'date_published'\n",
    "    - 'tags'\n",
    "    - 'links'\n",
    "    - 'awards'\n",
    "    - 'venues'\n",
    "    - 'sources'\n",
    "    - 'contributors'\n",
    "\n",
    "You can see a formatted version of the base results from the API by visiting the [SHARE Search API URL](http://localhost:8000/api/search/abstractcreativework/_search)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Service Names for Reference\n",
    "----\n",
    "Each provider harvested from has a specific . Let's make an API call to generate a table to get all of those \"internal\" names, along with the official name of the repository that they represent.\n",
    "\n",
    "The SHARE API has different endpoints. One of those endpoints returns a list of all of the providers that SHARE is harvesting from, along with their internal names, official names, links to their homepages, and a simple version of an icon representing their service, in a parsable format called json.\n",
    "\n",
    "Let's make a call to that API endpoint using the requests libarary, get the json data, and print out all of the shortnames and longnames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "from IPython.display import Image, display\n",
    "\n",
    "SHARE_PROVIDERS = 'https://staging-share.osf.io/api/providers/'\n",
    "\n",
    "data = requests.get(SHARE_PROVIDERS).json()\n",
    " \n",
    "print('Here are the first 10 Providers:')\n",
    "for source in data['results']:\n",
    "    print(\n",
    "        '{}\\n{}\\n{}\\n'.format(\n",
    "            source['long_title'],\n",
    "            source['home_page'],\n",
    "            source['provider_name']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### SHARE Schema\n",
    "\n",
    "You can make queries against any of the fields defined in the [SHARE Schema](https://github.com/CenterForOpenScience/SHARE-Schema/blob/master/share.yaml). If we were able to harvest the information from the original source, it should appear in SHARE. However, not all fields are required for every document. \n",
    "\n",
    "Required fields include:\n",
    "- title\n",
    "- contributors\n",
    "- uris\n",
    "- providerUpdatedDateTime\n",
    "\n",
    "We add some information after each document is harvested inside the field shareProperties, including:\n",
    "- source (where the document was originally harvested)\n",
    "- docID  (a unique identifier for that object from that source)\n",
    "\n",
    "These two fields can be combined to make a unique document identifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get the first 3 results from the most basic query - the first page of the most recently updated research release events in SHARE.\n",
    "\n",
    "We'll use the URL parsing library furl to keep track of all of our arguments to the URL, because we'll be modifying them as we go along. We'll print the URL as we go to take a look at it, so we know what we're requesting.\n",
    "\n",
    "We'll print out the result's title and sources where it appears."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import furl\n",
    "\n",
    "search_url = furl.furl(SHARE_API)\n",
    "search_url.args['size'] = 3\n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "print('----------')\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's limit that query to only documents mentioning \"giraffes\" somewhere in the title, description, or in any of the metadata. We'd do that by adding a query search term."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "search_url.args['q'] = 'giraffes'\n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "print('---------')\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's search for documents from the source mit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "search_url.args['q'] = 'sources:providers.edu.mit'\n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "print('---------')\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's combine the two and find documents from MIT that mention giraffes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "search_url.args['q'] = 'sources:providers.edu.asu AND giraffes'\n",
    "recent_results = requests.get(search_url.url).json()\n",
    "\n",
    "recent_results = recent_results['hits']['hits']\n",
    "\n",
    "print('The request URL is {}'.format(search_url.url))\n",
    "print('---------')\n",
    "for result in recent_results:\n",
    "    print(\n",
    "        '{} -- from {}'.format(\n",
    "            result['_source']['title'],\n",
    "            result['_source']['sources']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complex Queries\n",
    "The SHARE Search API runs on elasticsearch - meaning that it can accept complicated queries that give you a wide variety of information.\n",
    "\n",
    "Here are some examples of how to make more complex queries using the raw elasticsearch results. You can read a [lot more about elasticsearch queries here](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "search_url.args = None  # reset the args so that we remove our old query arguments.\n",
    "search_url.url # Show the URL that we'll be requesting to make sure the args were cleared"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query Setup\n",
    "\n",
    "We can define a few functions that we can reuse to make querying simpler. Elasticsearch queries are passed through as json blobs specifying how to return the information you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def query_share(url, query):\n",
    "    # A helper function that will use the requests library,\n",
    "    # pass along the correct headers,\n",
    "    # and make the query we want\n",
    "    headers = {'Content-Type': 'application/json'}\n",
    "    data = json.dumps(query)\n",
    "    return requests.post(url, headers=headers, data=data, verify=False).json()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Some Queries\n",
    "The SHARE schema has many spots for information, and many of the original sources do not provide this information. We can do a query to find out if a certain field exists or not within certain records. The SHARE API is set up to show an empty list if the field is empty.\n",
    "\n",
    "Let's query for the counts of documents that have a content in their tags field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tags_query = {\n",
    "    \"query\": {\n",
    "        \"exists\": {\n",
    "            \"field\": \"tags\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "\n",
    "missing_tags_query = {\n",
    "    \"query\": {\n",
    "        \"bool\": {\n",
    "            \"must_not\": {\n",
    "                \"exists\": {\n",
    "                    \"field\": \"tags\"\n",
    "                }\n",
    "            }\n",
    "        }      \n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with_tags = query_share(search_url.url, tags_query)\n",
    "missing_tags = query_share(search_url.url, missing_tags_query)\n",
    "\n",
    "total_results = requests.get(search_url.url).json()['hits']['total']\n",
    "\n",
    "with_tags_percent = (float(with_tags['hits']['total'])/total_results)*100\n",
    "missing_tags_percent = (float(missing_tags['hits']['total'])/total_results)*100\n",
    "\n",
    "\n",
    "print(\n",
    "    '{} results out of {}, or {}%, have tags.'.format(\n",
    "        with_tags['hits']['total'],\n",
    "        total_results,\n",
    "        format(with_tags_percent, '.2f')\n",
    "    )\n",
    ")\n",
    "\n",
    "print(\n",
    "    '{} results out of {}, or {}%, do NOT have tags.'.format(\n",
    "        missing_tags['hits']['total'],\n",
    "        total_results,\n",
    "        format(missing_tags_percent, '.2f')\n",
    "    )\n",
    ")\n",
    "\n",
    "print('------------')\n",
    "print('As a little sanity check....')\n",
    "print('{} + {} = {}%'.format(with_tags_percent, missing_tags_percent, format(with_tags_percent + missing_tags_percent, '.2f')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using SHAREPA for SHARE Parsing and Analysis\n",
    "\n",
    "While you can always pass raw elasticsearch queries to the SHARE API, there is also a pip-installable python library that you can use that makes elasticsearch aggregations a little simpler. This library is called [sharepa - short for SHARE Parsing and Analysis](https://github.com/CenterForOpenScience/sharepa#sharepa)\n",
    "\n",
    "### Basic Actions\n",
    "\n",
    "A basic search will provide access to all documents in SHARE in 10 document slices.\n",
    "\n",
    "#### Count\n",
    "You can use sharepa and the basic search to get the total number of documents in SHARE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sharepa import basic_search\n",
    "\n",
    "basic_search.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Iterating Through Results\n",
    "Executing the basic search will send the actual basic query to the SHARE API and then let you iterate through results, 10 at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results = basic_search.execute()\n",
    "\n",
    "for hit in results:\n",
    "    print(hit.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we don't want 10 results, or we want to offset the results, we can use slices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results = basic_search[20:25].execute()\n",
    "for hit in results:\n",
    "    print(hit.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Advanced Search with sharepa\n",
    "\n",
    "You can make your own search object, which allows you to pass in custom queries for certain terms or SHARE fields. Queries are formed using [lucene query syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax), just like we used in the above examples.\n",
    "\n",
    "This type of query accepts an exists field. Other options include a query_string, a match query, a multi-match query, a bool query, and any other query structure available in the elasticsearch API.\n",
    "\n",
    "We can see what that query that we're about to send to elasticsearch by using the pretty print helper function. You'll see that it looks very similar to the queries we defined by hand earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sharepa import ShareSearch\n",
    "from sharepa.helpers import pretty_print\n",
    "\n",
    "my_search = ShareSearch()\n",
    "\n",
    "my_search = my_search.query(\n",
    "    'exists', # Type of query, will accept a lucene query string\n",
    "    field='tags', # This lucene query string will find all documents that don't have tags\n",
    ")\n",
    "\n",
    "pretty_print(my_search.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you execute that query, you can then iterate through the results the same way that you could with the simple search query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "new_results = my_search.execute()\n",
    "for hit in new_results:\n",
    "    print(hit.tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debugging and Problem Solving\n",
    "\n",
    "Not everything always goes as planned when querying an unfamillar API. Here are some debugging and problem solving strategies when you're querying the SHARE API.\n",
    "\n",
    "### Schema issues\n",
    "The SHARE schema has a lot of parts, and much of the information is nested within sections. Making a query isn't always as straight forward as you might think, if you're not looking in the right part of the schema.\n",
    "\n",
    "Let's say you were trying to query for all SHARE documents that specify the language as not being in English.\n",
    "\n",
    "We'll guess as to what that query might be, and try to make it using sharepa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "language_search = ShareSearch()\n",
    "\n",
    "language_search = language_search.query(\n",
    "    'query_string', # Type of query, will accept a lucene query string\n",
    "    query='NOT languages=english', # This lucene query string will find all documents that don't have tags\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results = language_search.execute()\n",
    "\n",
    "for hit in results:\n",
    "    print(hit.languages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the result does not have an attribute called languages! Let's try to figure out what went wrong here.\n",
    "\n",
    "Step one could be that we are trying to find something that does NOT match a given parameter. Since languages is not required, this is returning results that do not include the languages result at all!\n",
    "\n",
    "So let's fix this up a bit to make sure that we're querying for items that specify language in the first place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "language_search = ShareSearch()\n",
    "\n",
    "language_search = language_search.filter(\n",
    "    'exists',\n",
    "    field=\"language\"\n",
    ")\n",
    "\n",
    "language_search.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results = language_search.execute()\n",
    "\n",
    "# Let's see how many documents have language results.\n",
    "print('There are {} documents with languages specified'.format(language_search.count()))\n",
    "\n",
    "print('Here are the languages for the first 10 results:')\n",
    "\n",
    "# Check out the first few results\n",
    "for hit in results:\n",
    "    print(hit.language)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we're better equipped to add on to this filter, and then narrow down to results that are not in English.\n",
    "\n",
    "When we printed out the first few results, we might have noticed a second problem with our query -- going back to the [SHARE Schema](https://github.com/CenterForOpenScience/SHARE-Schema/blob/master/share.yaml), we might notice that there is a restriction on how languages are captured - as a three letter lowercase representation. Instead of \"english\" let's look for the three letter abbreviation - \"eng\"\n",
    "\n",
    "We can modify our new and improved language query by adding on another query to our started language_search. We'll use the elasticsearch query object Q, and invert it with a ~ symbol, and search for the term \"eng.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from elasticsearch_dsl import Q\n",
    "\n",
    "language_search = language_search.query(~Q(\"term\", language=\"eng\"))\n",
    "\n",
    "results = language_search.execute()\n",
    "\n",
    "# Let's see how many documents have language results that aren't eng\n",
    "print('There are {} documents that do not have \"eng\" listed.'.format(language_search.count()))\n",
    "\n",
    "print('Here are the languages for the first 10 results:')\n",
    "\n",
    "# Check out the first few results, make sure \"eng\" isn't in there\n",
    "for hit in results:\n",
    "    print(hit.language)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}