Data Dog

Our world of web applications and fast, interactive mobile devices calls for the free exchange of data in easily accessible forms. Standard formats promote interoperability and minimize development time. Open formats also make it easy to import data into other applications. Over the years, several popular alternatives have emerged. CSV, XML, and YAML are well known and easy to adapt to different applications (see the box entitled "Comparing Formats" and Listings 1-3 for examples). One format that is used extensively for web applications, mobile applications, and even some conventional desktop tools is JavaScript Object Notation (JSON) [1].

Stephen Fry; The Hippopotamus; 1994
Ian Rankin; Set In Darkness; 2009
Ken Follett; The Pillars of the Earth; 1989

<inventory>
  <book>
    <author>Stephen Fry</author>
    <title>The Hippopotamus</title>
    <publication>1994</publication>
  </book>
  <book>
    <author>Ian Rankin</author>
    <title>Set In Darkness</title>
    <publication>2009</publication>
  </book>
  <book>
    <author>Ken Follett</author>
    <title>The Pillars of the Earth</title>
    <publication>1989</publication>
  </book>
</inventory>

---
book:
- author: Stephen Fry
  title: The Hippopotamus
  publication: '1994'
- author: Ian Rankin
  title: Set In Darkness
  publication: '2009'
- author: Ken Follett
  title: The Pillars of the Earth
  publication: '1989'

JSON is wildly popular as a tool for passing information between web apps – for instance, it is currently the de facto standard for REST services – yet for many users, the details of JSON format are shrouded in mystery. This article takes a close look at JSON and some of the tools available for reading, manipulating, and importing JSON data.

Understanding JSON

The notation of JSON is analogous to objects, records, or dictionaries – depending on what that structure is currently called in your favorite programming language. Even though JSON format is based on JavaScript, parsers exist in almost all programming languages. In addition to Awk and C/C++, you can integrate JSON with Fortran, Go, Lisp, Lua, Python, and Visual Basic.

In everyday life, you will find the format in the data-sharing Jupyter Notebook app [2], in geographical specifications like GeoJSON [3] (Listing 4), and even in databases like MongoDB.

{
  "type": "Feature",
  "geometry": {
    "type": "Point",
    "coordinates": [125.6, 10.1]
  },
  "properties": {
    "name": "Dinagat Islands"
  }
}

Taking a closer look at the JSON data structure, you will see that it is in an easy-to-read, text-based format. Parentheses, colons, and commas separate the individual elements; the data can be nested as desired. This means, for example, that you can map lists, arrays, or objects. Table 1 summarizes the elementary data types that JSON supports.

Mark the individual structure levels with brackets and indentation for better readability (Listing 5). Pairs of curly braces ({ and }) each form a unit. Square brackets ([ and ]) are used to indicate fields (also known as arrays). Individual field elements follow the form of an enumeration and are separated by commas. Each field element consists of a key-value pair separated by a colon (:).

{"book": [
  {
    "author": "Stephen Fry",
    "title": "The Hippopotamus",
    "publication": "1994"
  },
  {
    "author": "Ian Rankin",
    "title": "Set In Darkness",
    "publication": "2009"
  },
  {
    "author": "Ken Follett",
    "title": "The Pillars of the Earth",
    "publication": "1989"
  }
]}

JSON was originally created in the early 2000s to exchange data between web applications. JSON worked quite well in the web context (even though you cannot always parse it unambiguously). To structure data, JSON falls back on conventions familiar to anyone who has programmed in a C-based language (C, C++, C#, Java, JavaScript, Perl, Python, and others).

JSON is specified according to RFC 8259 [4] and ECMA-404; common extensions are JSONP (JSON with padding), JSONPP (JSON with padding and parameters), and JSONML, which combines XML and JSON together. The character set for all JSON formats is Unicode (UTF-8), which eliminates the character-set guessing game that you will be familiar with from CSV.

You can use JSON to exchange smaller volumes of data between applications in an agile way. However, if the transferred data volume increases (e.g., if you have millions of measurements from a sensor), JavaScript-based Python libraries like Ipywidgets, Bokeh, and Plotly often fail. In the face of large data volumes, binary transport mechanisms are a better option for handling the load.

Tool Overview

Several command-line tools are available for parsing, processing, and outputting JSON data. Table 2 summarizes some of the available tools. All of these tools are available as packages for Debian GNU/Linux, Ubuntu, Devuan, Linux Mint, and macOS.

Not all of the tools in Table 2 are intuitive, and some of them only develop their full impact in a specific context. You will find more information on these tools in the various documents and cheat sheets available online [12].

Easily Readable JSON Output

When it comes to pretty printing, aeson-pretty, jc, jo, jq, and Jshon all have something to say. Some of the tools have a command-line parameter for printing, for example, -p for jo.

In Listing 6, cat and aeson-pretty work together for readable output via a pipe. Jq delivers the same results with the next call, but the output is in color (Figure 1):

$ jq . book_inventory.json

$ cat book-inventory.json | aeson-pretty
{
  "book": [
    {
      "publication": "1994",
      "author": "Stephen Fry",
      "title": "The Hippopotamus"
    },
    {
      "publication": "2009",
      "author": "Ian Rankin",
      "title": "Set In Darkness"
    },
    {
      "publication": "1989",
      "author": "Ken Follett",
      "title": "The Pillars of the Earth"
    }
  ]
}
$

Figure 1: The Jq tool puts in your JSON output and keeps the output readable.

The dot in the call to jq is not immediately understandable. It stands for the expression to be processed; in this case, it denotes all objects specified as parameters in the JSON file. You can define colorizing of the output using two options, -C (--colour-output) and -M (--monochrome-output).

Some users prefer compact output with as few (space) characters as possible. In Listing 7, see aeson-pretty with the -c (short for --compact) option. This option reduces the number of characters in the output by 45 percent, from 428 to 236 bytes. Compared to Listing 5 and Listing 6, the results still convey the same information, but with only half the amount of data.

$ cat book-inventory.json | aeson-pretty -c
{"book":[{"publication":"1994","author":"Stephen Fry","title":"The Hippopotamus"},
{"publication":"2009","author":"Ian Rankin","title":"Set In Darkness"},
{"publication":"1989","author":"Ken Follett","title":"The Pillars of the Earth"}]}