Importing Data

Karma allows you to import data from a variety of sources using the buttons at the top of the Karma screen.

You can import data from

Databases using Import Database Table
Files using Import File ...
Web APIs directly from a Karma worksheet
Specialized Services using Import from Service

Importing Data from Databases

Karma can import data from relational databases (MySQL, SQL Server, Oracle and PostGIS). When Karma imports data from a relational database it will load the first 1000 rows only. You can work with this sample of data to define and save a script that you can use later in batch mode (see Batch Mode).

When you click the Import Database Table button, Karma shows a dialogue like the one below. First, fill in the appropriate parameters to connect to your database and click the OK button. Then Karma will show you panel to browse the tables in your database where you can do the following:

Filter the list of tables shown by entering a keyword in the Choose Table: option.
Show a preview of the contents of a table by clicking on the Preview button.
Import the table into your workspace by clicking on the Import button.

When you import a table, the dialogue to import database tables remains on the screen so that you can import additional tables. Karma will show the imported table in your workspace as shown below. You can continue to import additional tables and when you are done click the Cancel or OK button to dismiss the window for importing databases (yes, we know, why have both OK and Cancel buttons if they simply dismiss the dialogue? We haven't gotten around to fix this).

Importing Files

Use the Import File ... command to import data from a file. Karma will prompt you to select the file you want to import and then show a menu where you can specify the type of file you are importing. Karma guesses the file type based on the extension of the file; correct it if necessary and click Submit. Karma will import the file or prompt you for additional information, depending on the file type.

Import CSV and Other Delimited Text Files

To import a CSV or delimited text file, select CSV Text File in the Confirm File Format menu. Karma will show the following dialogue box to enable you to specify options to tell Karma how your file is delimited, whether it has a heading row, where your data starts and how is text specified in each cell.

When you adjust the options, the preview will change accordingly. When the preview shows your data correctly, click on the Import button to import your data.

When importing text delimited files, Karma will load the whole file in memory. If your file is very large (over 100,000 rows), Karma may run out of memory. To overcome this problem, create a smaller file out of your data, load it in Karma, use it to define the script you want to apply to the whole dataset, and then use the Batch Mode to apply the script to the whole file.

Import XML and JSON files

Karma can import data from XML and JSON files, preserving the hierarchical structure of the data. Karma supports a nested relational data model, which means that a worksheet consists of a list of records, each record can have several attributes. The values of the attributes can be simple values, or can be lists of other records, very similar to JSON. Karma assumes the data is regular in the sense that the nested records in a column have the same attributes. If your file does not have this regularity, Karma will make it regular by adding additional attributes to some of the records (leaving them with empty values) or by nesting records in lists. Karma does not throw away any of the data in your file.

Karma handles XML and JSON files the same way. Internally, to import XML data, Karma uses a public library to first convert the XML to a JSON representation and then import the JSON.

Here is how XML data looks when imported in Karma (partial view):

The XML file contains many artwork records like the following one:

<artwork title="Joy of the Waters">
	<sourceURL>http://www.imamuseum.org/art/collections/artwork/joy-waters-frishmuth-harriet-whitney</sourceURL>
	<imageURL>imaimg:b4/b42f9a2c-2c39-4f1b-bd1a-5283cec26a29.jpg</imageURL>
	<artist>Frishmuth, Harriet Whitney</artist>
	<nationality>American</nationality>
	<birthDeath>1880-1980</birthDeath>
	<materials>bronze</materials>
	<dimensions>H: 61 in.</dimensions>
	<creditLine>Gift of the Friends of American Art</creditLine>
	<accessionNumber>26.1</accessionNumber>
	<relatedArtworks id="0">
		<relatedArtworksURL>http://www.imamuseum.org/art/collections/artwork/crest-wave-frishmuth-harriet-whitney</relatedArtworksURL>
		<relatedArtworksTitle>Crest of the Wave</relatedArtworksTitle>
	</relatedArtworks>
	<relatedArtworks id="1">
		<relatedArtworksURL>http://www.imamuseum.org/art/collections/artwork/diana-saint-gaudens-augustus</relatedArtworksURL>
		<relatedArtworksTitle>Diana</relatedArtworksTitle>
	</relatedArtworks>
</artwork>

Import MS Excel Files

You can directly import MS Excel files in Karma. When you do so, Karma will create a separate Karma worksheet for each sheet in your spreadsheet.

Get Data from Web APIs

Karma allows you to invoke Web APIs on data stored in a Karma worksheet. To invoke a Web API, you need to have a column in your worksheet with the invocation URLs. The screen below shows a simple example with a worksheet containing one column with two URLs that invoke a weather service. To invoke the service, click on the menu on the column header and select Invoke Service. Karma will invoke the service as many times as you have URLs.

Note: the name of your worksheet is irrlelevant, your worksheet can have more than one column, and the name of the column is irrelevant too.

Karma can invoke services that return XML or JSON and populate the worksheet with the results. The screen below shows some of the results produced in our weather example.

Karma's service invocation facility works with Web APIs, but does not work with true REST APIs that represent different resources using different URLs. For example, if your worksheet contains the following URLs, Karma will not be able to invoke them because it requires the URLs in the different rows to be identical, differing only on the values of their parameters. In the example below the URLs are different (and have no parameters).

https://projects.propublica.org/free-the-files/committees/warren-for-senate-2012.json
https://projects.propublica.org/free-the-files/committees/planned-parenthood-issue.json

An additional restriction is that Karma only supports GET method invocations. We are working to extend Karma to provide more complete support for REST services.

Load Ontologies

You can load ontologies in Karma using the Import File ... command, selecting an OWL or RDF file, and choosing the OWL Ontology option in the Confirm File Format menu.

If Karma can process the ontology correctly, it will give a confirmation notice:

Karma can load ontologies stored in XML format, but does not currently support any of the other formats such as N3 or Manchester syntax. You can store the ontology in a file with an RDF extension (so Karma can load RDF Schema files). Make sure to select OWL Ontology in the Confirm File Format menu.

See also: Automatically Loading Commonly-Used Ontologies for a way to tell Karma to automatically load ontologies when it starts so that you don't have to load them one by one using the Import File ... command.