Open data would be of little use if it wasn't easy to automatically publish and update and subsequently find, download, and integrate with other open data, straight from the primary sources.

The Comprehensive Knowledge Archive Network (CKAN) [1], a key component of the Open Knowledge Foundation's mission to promote open data [2], is a collection of tools for implementing open data hubs and portals. If you have a dataset that you'd like to make available as open data, CKAN gives you a head start on implementing the necessary infrastructure.

The CKAN tools help you manage and federate the data. It also comes with advanced geospacial features, search capabilities, metadata, and visualization options. Using CKAN tools and best practices, you can build a portal for your open data with the least possible effort.

How CKAN Works

CKAN is based on open standards and free/open source software. At its core, most of CKAN is Python code, released under an aGPL license. Other main components are the PostgreSQL database engine and the Apache Solr search server [3].

CKAN websites make it easy to find, browse, and analyze open data manually. CKAN's real value, however, is its consistent support for creating federated catalogs of open data and, importantly, finding and reusing the same data automatically.

CKAN does this through application programming interfaces (APIs) [4] that support REST-style data exchange and announce and export entire datasets according to the Data Catalogue Vocabulary Application Profile (DCAT-AP) [5]. DCAT-AP is an open standard for describing public sector datasets, developed in Europe specifically to make those datasets easier to search. DCAT-AP rules standardize the description of datasets, defining which metadata should be published and the corresponding machine-readable formats.

For this reason, many organizations (mainly, but not exclusively, public administrations) use CKAN to collaboratively build a worldwide, coherent catalog of open data. CKAN's tools and methods completely automate not just the initial online publication of open data with all its metadata, but also its constant updating, straight from the original sources, whether from databases or spreadsheet files.

When loading one or more sets of open data onto a CKAN website, its administrator may enable its end users (both human and other software) to immediately download current dataset snapshots in JSON format or browse and visualize the raw data directly in several ways.

Automatic usage is possible thanks to other CKAN functions that, once a dataset is loaded, announce its existence with Resource Description Framework (RDF) declarations. Reading the RDF declarations, other portals may then expose the corresponding datasets, as well as datasets of many other CKAN-compatible sources, as part of one single catalog of their choosing. Even better, that catalog will always automatically contain complete, current versions of all the datasets it lists.

In Italy, for example, some regional administrations' CKAN catalogs automatically feed into the national data portal [6], which then automatically feeds into the European portal [7]. Worth noting here, the national data portal is not a CKAN portal, but it can still federate successfully by supporting and using the DCAT-AP standards and software tools. That's the beauty of open standards.

For this to work in practice, two conditions must be met. First, a website must announce that it's running CKAN (or at least following the compatible interfaces and standards) and be configured to make it discoverable by other CKAN (or CKAN-compatible) websites (which I am told is not always the case). Second, each dataset's metadata must always exactly correspond to its actual scope, freshness, and content. Otherwise, the whole thing either breaks or becomes practically useless. While this synchronization requires some effort, CKAN remains the fastest path to standardized, collaborative publishing of open data.

CKAN in Action!

To demonstrate CKAN's usefulness, I interviewed two of the main Italian CKAN experts: Maurizio Napolitano [8] and Vincenzo Patruno [9]. Napolitano works for Fondazione Bruno Kessler (FBK, Bruno Kessler Foundation) in Trento, Italy, as coordinator of the FBK Digital Commons Lab. Patruno is the IT/data manager for the Italian National Institute of Statistics and the vice president of onData APS [10] (a nonprofit association that promotes open data). Both Napolitano and Patruno got involved with open data because they believe "the most powerful form of revolution we have today [is] to give more opportunities to everybody with good ideas to improve our society."

Trento Tourist Spots

Napolitano provides the first of example of CKAN's power with a dataset that lists the tourist points of interest in the Valle dei Laghi (Valley of Lakes) near Trento in Northern Italy. To access this dataset (shown in Figure 1), you can use the federated OPENdata Trentino portal to search for "Valle dei Laghi" [11].

Figure 1: With CKAN, each dataset gets its own web page, containing all the related information, as well as several ways to access the data.

On the right, each Esplora (explore) drop-down menu has two options: Scaricare (download the raw dataset) and Anteprima (preview). Choosing Anteprima opens the interface shown in Figure 2.

Figure 2: A tabular view of the restaurants, transit services, and other tourist services, which users may sort and search in several ways.

The default Grid view lists the several entries in columns, allowing users to filter or sort them as needed. Clicking on the Map tab shows each entry on the map. However, in Figure 3, the map is empty. The reason for this is because the latitude and longitude columns have not been labeled correctly in the Grid view in Figure 2. CKAN expects these columns to be named Latitude and Longitude to recognize and use the geographic coordinates, but Figure 2 uses the Italian column names Latitudine and Longitudine. To get around this problem, you can tell CKAN explicitly where to find those values as shown in Figure 4. Once you have assigned the latitude and longitude fields, all the entries appear on the map (Figure 5).

Figure 3: If properly labeled, CKAN can automatically show the location of each entry. Here, CKAN doesn't recognize the Latitudine and Longitudine labels, so the map is blank.

Figure 4: When coordinates are not labeled as CKAN expects, you can still help CKAN recognize them.

Figure 5: After assigning the correct fields for latitude and longitude, CKAN now displays a map with the points of interest.