samples/caso-uso-pydatajson-xlsx-validando-dev.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Caso de uso - Validación, transformación y harvesting con el catálogo dev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Caso 1: catálogo válido\n",
    "\n",
    "En esta prueba se realiza el proceso completo de validación, transformación y harvesting a partir de un archivo xlsx que contiene los metadatos pertenecientes al catálogo del Ministerio de Justicia. \n",
    "\n",
    "**Nota**: Se trata de un catálogo conocido y  válido en cuanto a su estructura y metadatos. Archivo utilizado: catalogo-justicia.xlsx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importación de metodos y clases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import arrow\n",
    "import os, sys\n",
    "sys.path.insert(0, os.path.abspath(\"..\"))\n",
    "from pydatajson import DataJson #lib y clase\n",
    "from pydatajson.readers import read_catalog # lib, modulo ... metodo Lle el catalogo -json o xlsx o (local o url) dicc- y lo transforma en un diccionario de python\n",
    "from pydatajson.writers import write_json_catalog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Declaración de variables y paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#completar con lo que corresponda\n",
    "ORGANISMO = 'justicia'\n",
    "catalogo_xlsx = os.path.join(\"archivos-tests\", \"excel-validos\", \"catalogo-justicia.xlsx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#NO MODIFICAR\n",
    "\n",
    "#Creo la estructura de directorios necesaria si no existe\n",
    "if not os.path.isdir(\"archivos-generados\"):\n",
    "    os.mkdir(\"archivos-generados\")\n",
    "\n",
    "for directorio in [\"jsons\", \"reportes\", \"configuracion\"]:\n",
    "    path = os.path.join(\"archivos-generados\", directorio)\n",
    "    if not os.path.isdir(path):\n",
    "        os.mkdir(path)\n",
    "\n",
    "# Declaro algunas variables de interés\n",
    "HOY = arrow.now().format('YYYY-MM-DD-HH_mm')\n",
    "catalogo_a_json = os.path.join(\"archivos-generados\",\"jsons\",\"catalogo-{}-{}.json\".format(ORGANISMO, HOY))\n",
    "reporte_datasets = os.path.join(\"archivos-generados\", \"reportes\", \"reporte-catalogo-{}-{}.xlsx\".format(ORGANISMO, HOY))\n",
    "archivo_config_sin_reporte = os.path.join(\"archivos-generados\", \"configuracion\", \"archivo-config_-{}-{}-sinr.csv\".format(ORGANISMO, HOY))\n",
    "archivo_config_con_reporte = os.path.join(\"archivos-generados\", \"configuracion\", \"archivo-config-{}-{}-conr.csv\".format(ORGANISMO, HOY))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validación del archivo xlsx y transformación a json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validación del catálogo en xlsx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#catalogo = read_catalog(catalogo_xlsx)\n",
    "# En el caso que quiera trabajarse con un archivo remoto:\n",
    "catalogo = read_catalog(\"http://186.33.211.253/data.json\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformación del catálogo, de xlsx a json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "write_json_catalog(catalogo, catalogo_a_json)\n",
    "\n",
    "##write_json_catalog(catalog, target_file)  escrie un dicc a un archivo json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validación del catalogo en json y harvesting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validación del catálogo en json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "source": [
    "Instanciación de la clase DataJson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dj = DataJson()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Validación -V/F- del catálogo en json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "dj.is_valid_catalog(catalogo)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validación detallada del catálogo en json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{u'error': {u'catalog': {u'errors': [],\n",
       "   u'status': u'OK',\n",
       "   u'title': u'Datos Argentina'},\n",
       "  u'dataset': [{u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 0],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Base de datos legislativos Infoleg'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 1],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Entidades constituidas en la Inspecci\\xf3n General de Justicia'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 2],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Declaraci\\xf3n Jurada Patrimonial Integral de car\\xe1cter p\\xfablico'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 3],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Registro, Sistematizaci\\xf3n y Seguimiento de Hechos de Violencia Institucional'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 4],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Magistrados de la Justicia Federal y de la Justicia Nacional'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 5],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Sistema de Consulta Nacional de Rebeld\\xedas y Capturas - Co.Na.R.C.'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 6],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Solicitudes de Condici\\xf3n de Leg\\xedtimo Usuario de Armas de Fuego'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 7],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Estad\\xedstica de mediaciones prejudiciales y judiciales'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 8],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Solicitudes de Autorizaci\\xf3n de Portaci\\xf3n de Armas de Fuego'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 9],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Listado de Registros Seccionales de la Direcci\\xf3n Nacional de Registros Nacionales de la Propiedad Automotor y Cr\\xe9ditos Prendarios'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 10],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Estad\\xedstica de Inscripciones Iniciales de Automotores y Motoveh\\xedculos'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 11],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Internos del Servicio Penitenciario Federal - SPF'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 12],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Sistema Nacional de Estad\\xedsticas sobre Ejecuci\\xf3n de la Pena \\u2013 SNEEP'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 13],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Registro Unificado de V\\xedctimas del Terrorismo de Estado -RUVTE-'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 14],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Registro, Sistematizaci\\xf3n y Seguimiento de Femicidios y Homicidios Agravados por el G\\xe9nero'},\n",
       "   {u'errors': [{u'error_code': 1,\n",
       "      u'instance': None,\n",
       "      u'message': \"u'superTheme' is a required property\",\n",
       "      u'path': [u'dataset', 15],\n",
       "      u'validator': u'required',\n",
       "      u'validator_value': [u'title',\n",
       "       u'description',\n",
       "       u'publisher',\n",
       "       u'superTheme',\n",
       "       u'distribution',\n",
       "       u'accrualPeriodicity',\n",
       "       u'issued']}],\n",
       "    u'status': u'ERROR',\n",
       "    u'title': u'Centros de Acceso a la Justicia -CAJ-'}]},\n",
       " u'status': u'ERROR'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.validate_catalog(catalogo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Harvesting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generación del archivo de reporte de datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dj.generate_datasets_report(catalogo, harvest='valid',export_path=reporte_datasets)\n",
    "# proceso el repote, 0 y 1s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generación del archivo de configuración para el harvester"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#  usando el reporte\n",
    "dj.generate_harvester_config(harvest='report', report=reporte_datasets, export_path=archivo_config_con_reporte)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# sin usar el reporte\n",
    "dj.generate_harvester_config(catalogs=catalogo, harvest='valid', export_path=archivo_config_sin_reporte)\n",
    "#(catalogs=None, harvest=u'valid', report=None, export_path=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "toc": {
   "nav_menu": {
    "height": "233px",
    "width": "253px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}