Importing Data

Karma allows you to import data from a variety of sources using the buttons at the top of the Karma screen.

You can import data from

Databases using Import Database Table
Files using Import File ...
Web APIs directly from a Karma worksheet
Specialized Services using Import from Service

Importing Data from Databases

Karma can import data from relational databases (MySQL, SQL Server, Oracle and PostGIS). When Karma imports data from a relational database it will load the first 1000 rows only. You can work with this sample of data to define and save a script that you can use later in batch mode (see Batch Mode).

When you click the Import Database Table button, Karma shows a dialogue like the one below. First, fill in the appropriate parameters to connect to your database and click the OK button. Then Karma will show you panel to browse the tables in your database where you can do the following:

Filter the list of tables shown by entering a keyword in the Choose Table: option.
Show a preview of the contents of a table by clicking on the Preview button.
Import the table into your workspace by clicking on the Import button.

When you import a table, the dialogue to import database tables remains on the screen so that you can import additional tables. Karma will show the imported table in your workspace as shown below. You can continue to import additional tables and when you are done click the Cancel or OK button to dismiss the window for importing databases (yes, we know, why have both OK and Cancel buttons if they simply dismiss the dialogue? We haven't gotten around to fix this).

Importing Files

Use the Import File ... command to import data from a file. Karma will prompt you to select the file you want to import and then show a menu where you can specify the type of file you are importing. Karma guesses the file type based on the extension of the file; correct it if necessary and click Submit. Karma will import the file or prompt you for additional information, depending on the file type.

Import CSV and Other Delimited Text Files

To import a CSV or delimited text file, select CSV Text File in the Confirm File Format menu. Karma will show the following dialogue box to enable you to specify options to tell Karma how your file is delimited, whether it has a heading row, where your data starts and how is text specified in each cell.

When you adjust the options, the preview will change accordingly. When the preview shows your data correctly, click on the Import button to import your data.

When importing text delimited files, Karma will load the whole file in memory. If your file is very large (over 100,000 rows), Karma may run out of memory. To overcome this problem, create a smaller file out of your data, load it in Karma, use it to define the script you want to apply to the whole dataset, and then use the Batch Mode to apply the script to the whole file.

Import XML and JSON files

Karma can import data from XML and JSON files, preserving the hierarchical structure of the data. Karma supports a nested relational data model, which means that a worksheet consists of a list of records, each record can have several attributes. The values of the attributes can be simple values, or can be lists of other records, very similar to JSON. Karma assumes the data is regular in the sense that the nested records in a column have the same attributes. If your file does not have this regularity, Karma will make it regular by adding additional attributes to some of the records (leaving them with empty values) or by nesting records in lists. Karma does not throw away any of the data in your file.

Here is how XML data looks when imported in Karma (partial view).

The XML file contains many artwork records like the following one:

<artwork title="Joy of the Waters">
	<sourceURL>http://www.imamuseum.org/art/collections/artwork/joy-waters-frishmuth-harriet-whitney</sourceURL>
	<imageURL>imaimg:b4/b42f9a2c-2c39-4f1b-bd1a-5283cec26a29.jpg</imageURL>
	<artist>Frishmuth, Harriet Whitney</artist>
	<nationality>American</nationality>
	<birthDeath>1880-1980</birthDeath>
	<materials>bronze</materials>
	<dimensions>H: 61 in.</dimensions>
	<creditLine>Gift of the Friends of American Art</creditLine>
	<accessionNumber>26.1</accessionNumber>
	<relatedArtworks id="0">
		<relatedArtworksURL>http://www.imamuseum.org/art/collections/artwork/crest-wave-frishmuth-harriet-whitney</relatedArtworksURL>
		<relatedArtworksTitle>Crest of the Wave</relatedArtworksTitle>
	</relatedArtworks>
	<relatedArtworks id="1">
		<relatedArtworksURL>http://www.imamuseum.org/art/collections/artwork/diana-saint-gaudens-augustus</relatedArtworksURL>
		<relatedArtworksTitle>Diana</relatedArtworksTitle>
	</relatedArtworks>
</artwork>