Skip to content

Package dictionary .. how it varies

Jump to bottom Edit New page
David Read edited this page Aug 12, 2013 · 10 revisions

Form to database

Here we follow the package in its various formats, from form submission to storing it in the database.

Form submission (new dataset)

  • Parameters are serialized
  • add-resource-url is only in the CKAN 1.7.1 form because you first say what sort of resource it is
UnicodeMultiDict(
   [('title', u'OpenStreetMap'),
    ('name', u'openstreetmap'),
    ('url', u''),
    ('license_id', u'odc-pddl'),
    ('notes', u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.'),
    ('tag_string', u' 'map, geo spatial'),
    ('add-resource-url', u''),
    ('add-resource-url', u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'),
    ('resources__0__name', u'Planet dump'),
    ('resources__0__description', u'Full dump of the planet'),
    ('resources__0__url', u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'),     
    ('resources__0__format', u'OSM'), 
    ('resources__0__resource_type', u'file'), 
    ('resources__0__webstore_url', u''), 
    ('resources__0__last_modified', u''), 
    ('resources__0__size', u'234555'), 
    ('resources__0__mimetype', u''), 
    ('resources__0__mimetype_inner', u''), 
    ('resources__0__id', u''), 
    ('resources__0__hash', u''), 
    ('resources__0__KEY', u'VALUE'), 
    ('author', u''), 
    ('author_email', u''), 
    ('maintainer', u''), 
    ('maintainer_email', u''), 
    ('version', u''), 
    ('extras__0__key', u'location'), 
    ('extras__0__value', u'uk'), 
    ('extras__1__key', u''), 
    ('extras__1__value', u''), 
    ('extras__2__key', u''), 
    ('extras__2__value', u''), 
    ('extras__3__key', u''), 
    ('extras__3__value', u''), 
    ('log_message', u'Created new dataset.'), 
    ('save', u'Add Dataset')
   ])

Tuplized (package controller)

  • Dictionary
  • Keyed by a tuple containing the index and whether it is key or value, to make it easier to group them in the next 'unflatten' stage.
{
 ('title',): u'OpenStreetMap',
 ('name',): u'openstreetmap',
 ('url',): u'',
 ('license_id',): u'odc-pddl',
 ('notes',): u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
 ('tag_string',): u'map, geo spatial',
 ('add-resource-url',): [u'',
                         u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'],
 ('resources', 0, 'KEY'): u'VALUE',
 ('resources', 0, 'description'): u'Full dump of the planet',
 ('resources', 0, 'format'): u'OSM',
 ('resources', 0, 'hash'): u'',
 ('resources', 0, 'id'): u'',
 ('resources', 0, 'last_modified'): u'',
 ('resources', 0, 'mimetype'): u'',
 ('resources', 0, 'mimetype_inner'): u'',
 ('resources', 0, 'name'): u'Planet dump',
 ('resources', 0, 'resource_type'): u'file',
 ('resources', 0, 'size'): u'234555',
 ('resources', 0, 'url'): u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
 ('resources', 0, 'webstore_url'): u'',
 ('author',): u'',
 ('author_email',): u'',
 ('maintainer',): u'',
 ('maintainer_email',): u'',
 ('version',): u''
 ('extras', 0, 'key'): u'location',
 ('extras', 0, 'value'): u'uk',
 ('extras', 1, 'key'): u'',
 ('extras', 1, 'value'): u'',
 ('extras', 2, 'key'): u'',
 ('extras', 2, 'value'): u'',
 ('extras', 3, 'key'): u'',
 ('extras', 3, 'value'): u'',
 ('log_message',): u'Created new dataset.',
 ('save',): u'Add Dataset',
}

unflattened (package controller)

  • converts to lists of dicts: extras, resources
{
 'title': u'OpenStreetMap',
 'name': u'openstreetmap',
 'url': u'',
 'license_id': u'odc-pddl',
 'notes': u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
 'tag_string': u'map, geo spatial',
 'add-resource-url': [u'',
                      u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'],
 'resources': [{'KEY': u'VALUE',
                'description': u'Full dump of the planet',
                'format': u'OSM',
                'hash': u'',
                'id': u'',
                'last_modified': u'',
                'mimetype': u'',
                'mimetype_inner': u'',
                'name': u'Planet dump',
                'resource_type': u'file',
                'size': u'234555',
                'url': u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
                'webstore_url': u''}],
 'author': u'',
 'author_email': u'',
 'maintainer': u'',
 'maintainer_email': u'',
 'version': u''
 'extras': [{'key': u'location', 'value': u'uk'},
            {'key': u'', 'value': u''},
            {'key': u'', 'value': u''},
            {'key': u'', 'value': u''}],
 'log_message': u'Created new dataset.',
 'save': u'Add Dataset',
}

Cleaned (package controller)

  • Removes blank values
  • This is the format of the package returned by package_show if context{'validate': False} which is the one used by ckan.lib.search:SynchronousSearchPlugin for SOLR indexing.
{
 'title': u'OpenStreetMap',
 'name': u'openstreetmap',
 'url': u'',
 'license_id': u'odc-pddl',
 'notes': u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
 'tag_string': u'map, geo spatial',
 'add-resource-url': [u'',
                      u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'],
 'resources': [{'KEY': u'VALUE',
                'description': u'Full dump of the planet',
                'format': u'OSM',
                'hash': u'',
                'id': u'',
                'last_modified': u'',
                'mimetype': u'',
                'mimetype_inner': u'',
                'name': u'Planet dump',
                'resource_type': u'file',
                'size': u'234555',
                'url': u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
                'webstore_url': u''}],
 'author': u'',
 'author_email': u'',
 'maintainer': u'',
 'maintainer_email': u'',
 'version': u''
 'extras': [{'key': u'location', 'value': u'uk'}],
 'log_message': u'Created new dataset.',
 'save': u'Add Dataset',
}

And 'type': None (implies default - 'package') is inserted (for example) before sending to package_create.

Validated (package_create)

  • 'validate' also does various little transforms, like converting tag_string to tags. These are all defined in the schema. Default schema is ckan.logic.schema.default_create_package_schema
  • add-resource-url and 'save' are removed finally
{
 'title': u'OpenStreetMap',
 'name': u'openstreetmap',
 'url': u'',
 'license_id': u'odc-pddl',
 'notes': u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
 'tag_string': u'map, geo spatial',
 'tags': [{'name': u'map'}, {'name': u'geo spatial'}],
 'resources': [{'KEY': u'VALUE',
                'description': u'Full dump of the planet',
                'format': u'OSM',
                'hash': u'',
                'last_modified': None,
                'mimetype': u'',
                'mimetype_inner': u'',
                'name': u'Planet dump',
                'resource_type': u'file',
                'size': 234555,
                'url': u'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
                'webstore_url': u''}],
 'author': u'',
 'author_email': u'',
 'maintainer': u'',
 'maintainer_email': u'',
 'extras': [{'key': u'location', 'value': u'uk'}],
 'version': u''
 'log_message': u'Created new dataset.',
}

Database

package:

id: 'e3f0339e-e548-42e3-a194-2f5b09fe3c2d'
name: 'openstreetmap'
title: 'OpenStreetMap'
version: ''
url: ''
notes: 'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.'
license_id: 'odc-pddl'
revision_id: 'bc088d17-7826-4d75-afb7-eea464704daf'
author: '' author_email: '' maintainer: '' maintainer_email: ''
state: 'active'
type: ''

resource:

id: 'd5a3ca4b-10b7-401d-8152-f7e69a818582'
resource_group_id: '3ea44f6e-ac7d-4fcf-8a82-0da1e72a6898'
name: 'Planet dump'
url: 'ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2'
format: 'OSM'
description: 'Full dump of the planet'
extras: {"KEY": "VALUE"}
position: 0
revision_id: 'c2b583aa-1ab6-4e33-80f6-da29a78ffe63'
hash: ''
state: 'active'
resource_type: 'file'
size: 234555
mimetype: '' mimetype_inner: '' last_modified: Null cache_url: '' cache_last_updated: Null webstore_url: '' webstore_last_updated: ''
created: 2013-08-02 20:57:06.359635

package_extra:

id: '43f3cbb8-f7f2-428d-b998-11a51aafd985'
package_id: 'e3f0339e-e548-42e3-a194-2f5b09fe3c2d'
key: 'location'
value: "uk"
revision_id: 'c2b583aa-1ab6-4e33-80f6-da29a78ffe63'
state: 'active'

Non-validating form

  • If the validation stage has errors, then the form is to be redrawn with the previously submitted values in the boxes. This is the package dict which is passed to the template, which is the same as the Cleaned Dict above.

{
 'title': u'OpenStreetMap',
 'name': u'openstreetmap',
 'url': u'',
 'license_id': u'odc-pddl',
 'notes': u'OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.',
 'tag_string': u'map, geo spatial',
 'add-resource-url': [u'', u''],
 'resources': [{'description': u'Full dump of the planet',
                'format': u'OSM',
                'hash': u'',
                'id': u'',
                'last_modified': u'',
                'mimetype': u'',
                'mimetype_inner': u'',
                'name': u'Planet dump',
                'resource_type': u'api',
                'size': 234555,
                'url': u'http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2',
                'webstore_url': u''}],
 'author': u'',
 'author_email': u'',
 'maintainer': u'',
 'maintainer_email': u'',
 'version': u''
 'extras': [{'key': u'location', 'value': u'uk'}],
 'log_message': u'Created new dataset.',
 'save': u'Add Dataset',
 'type': None,
}

package_show

If you call package_show on the API (/api/action/package_show?id=openstreetmap) then you get:

  • usual API fields 'success' and 'help' with the package in 'result'
  • 'license_title', 'license_url' and 'isopen' added, based on license_id
  • resources now have 'position', 'tracking_summary'
  • 'relationships_as_object', 'relationships_as_subject', 'id', 'state', 'tracking_summary', 'revision_timestamp' added
{
   "success" : true,
   "help" : null,
   "result" : {
      "title" : "OpenStreetMap",
      "name" : "openstreetmap",
      "url" : "",
      "license_id" : "odc-pddl",
      "license_url" : "http://www.opendefinition.org/licenses/odc-pddl"
      "license_title" : "Open Data Commons Public Domain Dedication and Licence (PDDL)",
      "notes" : "OpenStreetMap (OSM) is aimed at creating and providing free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive or unexpected ways.",
      "tags" : [
         {
            "name" : "geo spatial",
            "display_name" : "geo spatial",
            "id" : "5bbb3684-e8a2-47a2-b61c-3991239d1927",
            "revision_timestamp" : "2013-08-02T20:44:41.421427",
            "vocabulary_id" : null,
            "state" : "active"
         },
         {
            "name" : "map",
            "display_name" : "map",
            "id" : "5b97bbba-02bb-4416-aa7f-453336e96218",
            "revision_timestamp" : "2013-08-02T20:44:41.421427",
            "vocabulary_id" : null,
            "state" : "active"
         }
      ],
      "resources" : [
         {
            "position" : 0,
            "state" : "active",
            "size" : 234555,
            "cache_last_updated" : null,
            "url" : "http://ftp://ftp.spline.de/pub/openstreetmap/planet/planet-latest.osm.bz2",
            "id" : "c83244dd-2089-4f2b-8a8e-50b0c04ff7bd",
            "resource_type" : "api",
            "revision_timestamp" : "2013-08-02T20:36:35.453823",
            "tracking_summary" : {
               "recent" : 0,
               "total" : 0
            },
            "resource_group_id" : "aa82bbce-e84b-462b-b366-cbda6682c57c",
            "webstore_last_updated" : null,
            "cache_url" : null,
            "last_modified" : null,
            "name" : "Planet dump",
            "description" : "Full dump of the planet",
            "revision_id" : "6ca47e2d-04e0-4a95-b472-45e999cdc716",
            "created" : "2013-08-02T21:36:35.491343",
            "hash" : "",
            "format" : "OSM",
            "webstore_url" : "",
            "mimetype_inner" : "",
            "mimetype" : ""
         }
      ],
      "author" : "",
      "author_email" : "",
      "maintainer" : "",
      "maintainer_email" : "",
      "version" : "",
      "extras" : [
         {
            "key" : "location",
            "value" : "\"uk\"",
            "id" : "dff97ec1-66eb-4167-9fef-b30296b0fa7e",
            "revision_id" : "9833f60b-5038-42ea-b20f-14976c0e64f9",
            "revision_timestamp" : "2013-08-02T20:44:41.421427",
            "package_id" : "57323b25-af96-484c-b5ba-2714257042e8",
            "state" : "active"
         }
      ],
      "isopen" : true,
      "groups" : [],
      "id" : "57323b25-af96-484c-b5ba-2714257042e8",
      "tracking_summary" : {
         "recent" : 0,
         "total" : 0
      },
      "revision_timestamp" : "2013-08-02T20:36:35.453823",
      "revision_id" : "6ca47e2d-04e0-4a95-b472-45e999cdc716",
      "relationships_as_subject" : [],
      "relationships_as_object" : [],
      "metadata_modified" : "2013-08-02T20:36:35.453823",
      "type" : null,
      "metadata_created" : "2013-08-02T20:36:35.453823",
      "state" : "active",
   }
}
  • If you do context['for_edit'] = True then you don't get tracking_summary.
  • If you do context['for_view'] = True then it will call IPackageController.before_view
Add a custom footer
Add a custom sidebar
Clone this wiki locally