Chapter 1. JSON Overview

The JavaScript Object Notation (JSON) data format enables applications to communicate over a network, typically through RESTful APIs. JSON is technology-agnostic, nonproprietary, and portable. All modern languages (e.g., Java, JavaScript, Ruby, C#, PHP, Python, and Groovy) and platforms provide excellent support for producing (serializing) and consuming (deserializing) JSON data. JSON is simple: it consists of developer-friendly constructs such as Objects, Arrays, and name/value pairs. JSON is not limited to Representational State Transfer (REST); it also works with the following:

Node.js (which stores project metadata in package.json)
NoSQL databases such as MongoDB (see Chapter 9)
Messaging platforms such as Kafka (see Chapter 10)

JSON Is a Standard

In the early days, REST’s detractors derided RESTful Web Services as being non-standard, but (just like HTTP) JSON is in fact a standard. Both the Internet Engineering Task Force (IETF) and Ecma International (formerly the European Computer Manufacturers Association, or ECMA) have recognized JSON as a standard. Douglas Crockford originally created JSON in 2001, and initially standardized it in 2006 under RFC 4627 through the IETF; see the JSON specification. In the fall of 2013, Ecma International also standardized JSON under ECMA 404; see their JSON specification. With Ecma recognition (per Douglas Crockford; see his Google+ page), JSON is now considered a formal international data processing standard.

In March 2014, Tim Bray published an updated version of Douglas Crockford’s original standard as IETF RFC 7158 and RFC 7159 to correct errata with the original IETF 4627 standard (thus rendering it obsolete).

A Brief Sample

Before we go further, let’s look at a small JSON sample. Example 1-1 shows a simple JSON document.

Example 1-1. firstValidObject.json

{ "thisIs": "My first JSON document" }

A valid JSON document can be either of the following:

An Object surrounded by curly braces, { and }
An Array enclosed by brackets, [ and ]

The preceding example shows an Object that contains a single key/value pair, where the key, "thisIs", has a value of "My first JSON document".

Just to keep us honest, let’s validate this document by using JSONLint. Just paste the text into the text area, click the Validate button, and you should see the page in Figure 1-1.

Example 1-2 presents a simple JSON Array.

Example 1-2. firstValidArray.json

[
  "also",
  "a",
  "valid",
  "JSON",
  "doc"
]

In JSONLint, paste the JSON Array into the text area, and click the Validate button, and you should get the result shown in Figure 1-2.

But we’re getting ahead of ourselves. We’ll cover JSON syntax more thoroughly in “Core JSON”.

Why JSON?

Although standardization through Ecma International and the IETF has helped JSON gain industry acceptance, other factors have popularized JSON:

The explosive growth of RESTful APIs based on JSON
The simplicity of JSON’s basic data structures
The increasing popularity of JavaScript

JavaScript’s resurgence is boosting JSON’s popularity. Over the past several years, we have seen the rise of JavaScript as a first-class development language and environment. This ecosystem includes platforms such as Node.js, and Mode/View/Controller (MVC) frameworks such as AngularJS, React, Backbone, and Ember. There has also been a tremendous increase in the number of books and websites showing best practices in JavaScript Objects and Patterns. According to Douglas Crockford, JSON is a subset of JavaScript’s Object Literal notation, and fits seamlessly into JavaScript development.

Thousands of RESTful APIs leverage JSON. A sample list of popular JSON-based RESTful APIs includes the following:

LinkedIn
Twitter
Facebook
Salesforce
GitHub
DropBox
Tumblr
Amazon Web Services (AWS)

To see the thousands of available JSON-based REST APIs available, visit ProgrammableWeb, and do a search on REST and JSON. Then, take several weeks to review the results.

JSON is simple and is gradually replacing XML as the primary data interchange format on the internet. JSON is easy to read, and its structures easily translate to concepts well understood by software developers—Arrays, Objects, and name/value pairs. We don’t have to scratch our heads or argue anymore about what should be an Element or an Attribute. Objects and their data members are a much better fit for Object-Oriented (OO) design and development. A document formatted in JSON is usually smaller than its XML equivalent, because JSON has less overhead and is more compact. This is due to the lack of begin and end tags surrounding each data element. So, at an enterprise level, JSON is more efficient to process than XML, because JSON documents can be transmitted over a network and processed faster than their XML counterparts.

Although Douglas Crockford initially intended JSON to be a data interchange format (typically with REST), JSON is now finding a home in configuration files for widely used products such as Node.js and Sublime Text. Node.js has a package.json file that it uses to define its standard npm package structure; we’ll cover this in Chapter 2. Sublime Text, a popular IDE in the web development community, uses JSON to configure its appearance along with its package managers.

Core JSON

The Core JSON data format includes JSON Data and Value Types. We’ll also cover versions, comments, and File/MIME Types.

JSON Data Types

JSON has the following core Data Types:

Name (or Key)/value pair: Consists of a key (a data attribute) and a value.
Object: An unordered collection of name/value pairs.
Array: A collection of ordered values.

Now that we’ve covered basic definitions, let’s dig deeper into each Data Type.

Name/value pairs

Example 1-3 shows some sample name/value pairs.

Example 1-3. nameValue.json

{
  "conference": "OSCON",
  "speechTitle": "JSON at Work",
  "track": "Web APIs"
}

Name/value pairs have the following characteristics:

Each name (e.g., "conference")
- Is on the left side of the colon (:)
- Is a String, and must be surrounded by double quotes
The value (e.g., "OSCON") is to the right of the colon. In the preceding example, the value type is a String, but there are several other Value Types.

We’ll cover Strings and other valid Value Types further in “JSON Value Types”.

Objects

Objects consist of name/value pairs. Example 1-4 shows a sample Object that represents an address.

Example 1-4. simpleJsonObject.json

{
  "address" : {
    "line1" : "555 Any Street",
    "city" : "Denver",
    "stateOrProvince" : "CO",
    "zipOrPostalCode" : "80202",
    "country" : "USA"
  }
}

Example 1-5 shows an Object with a nested Array.

Example 1-5. jsonObjectNestedArray.json

{
  "speaker" : {
    "firstName": "Larson",
    "lastName": "Richard",
    "topics": [ "JSON", "REST", "SOA" ]
  }
}

Example 1-6 shows an Object that contains another Object.

Example 1-6. jsonObjectNestedObject.json

{
  "speaker" : {
    "firstName": "Larson",
    "lastName": "Richard",
    "topics": [ "JSON", "REST", "SOA" ],
    "address" : {
      "line1" : "555 Any Street",
      "city" : "Denver",
      "stateOrProvince" : "CO",
      "zipOrPostalCode" : "80202",
      "country" : "USA"
    }
  }
}

Objects have the following characteristics:

Are enclosed within a beginning left curly brace ({) and an ending right curly brace (})
Consist of comma-separated, unordered, name/value pairs
Can be empty, { }
Can be nested within other Objects or Arrays

Arrays

Example 1-7 shows an Array (containing nested Objects and Arrays) that describes conference presentations, including title, length, and abstract.

Example 1-7. jsonArray.json

{
  "presentations": [
    {
      "title": "JSON at Work: Overview and Ecosystem",
      "length": "90 minutes",
      "abstract": [ "JSON is more than just a simple replacement for XML when",
                    "you make an AJAX call."
                  ],
      "track": "Web APIs"
    },
    {
      "title": "RESTful Security at Work",
      "length": "90 minutes",
      "abstract": [ "You’ve been working with RESTful Web Services for a few years",
                     "now, and you’d like to know if your services are secure."
                  ],
      "track": "Web APIs"
    }
  ]
}

Arrays have the following characteristics:

Are enclosed within a beginning left brace ([) and an ending right brace (])
Consist of comma-separated, ordered values (see the next section)
Can be empty, [ ]
Can be nested within other Arrays or Objects
Have indexing that begins at 0 or 1

JSON Value Types

JSON Value Types represent the Data Types that occur on the righthand side of the colon (:) of a Name/Value Pair. JSON Value Types include the following:

object
array
string
number
boolean
null

We’ve already covered Objects and Arrays; now let’s focus on the remaining Value Types: string, number, boolean, and null.

String

Example 1-8 shows valid JSON Strings.

Example 1-8. jsonStrings.json

[
  "fred",
  "fred\t",
  "\b",
  "",
  "\t",
  "\u004A"
]

Strings have the following properties:

Strings consist of zero or more Unicode characters enclosed in quotation marks (""). Please see the following list for additional valid characters.
Strings wrapped in single quotes (') are not valid.

Additionally, JSON Strings can contain the following backslash-escaped characters:

\": Double quote
\\: Backslash
\/: Forward slash
\b: Backspace
\f: Form feed
\n: Newline
\r: Carriage return
\t: Tab
\u: Trailed by four hex digits

Number

Example 1-9 shows valid numbers in JSON.

Example 1-9. jsonNumbers.json

{
  "age": 29,
  "cost": 299.99,
  "temperature": -10.5,
  "unitCost": 0.2,
  "speedOfLight": 1.23e11,
  "speedOfLight2": 1.23e+11,
  "avogadro": 6.023E23,
  "avogadro2": 6.023E+23,
  "oneHundredth": 10e-3,
  "oneTenth": 10E-2
}

Numbers follow JavaScript’s double-precision floating-point format and have the following properties:

Numbers are always in base 10 (only digits 0–9 are allowed) with no leading zeros.
Numbers can have a fractional part that starts with a decimal pont (.).
Numbers can have an exponent of 10, which is represented with the e or E notation with a plus or minus sign to indicate positive or negative exponentiation.
Octal and hexadecimal formats are not supported.
Unlike JavaScript, numbers can’t have a value of NaN (not a number for invalid numbers) or Infinity.

Boolean

Example 1-10 shows a Boolean value in JSON.

Example 1-10. jsonBoolean.json

{
  "isRegistered": true,
  "emailValidated": false
}

Booleans have the following properties:

Booleans can have a value of only true or false.
The true or false value on the righthand side of the colon(:) is not surrounded by quotes.

null

Although technically not a Value Type, null is a special value in JSON. Example 1-11 shows a null value for the line2 key/property.

Example 1-11. jsonNull.json

{
  "address": {
    "line1": "555 Any Street",
    "line2": null,
     "city": "Denver",
        "stateOrProvince": "CO",
        "zipOrPostalCode": "80202",
        "country": "USA"
    }
}

null values have the following characteristics:

Are not not surrounded by quotes
Indicate that a key/property has no value
Act as a placeholder

JSON Versions

According to Douglas Crockford, there will never be another version of the core JSON standard. This isn’t because JSON is perfect; nothing is perfect. The purpose of a sole JSON version is to avoid the pitfalls of having to support backward compatibility with previous versions. Crockford believes that a new data format should replace JSON when the need arises within the development community.

But as you’ll see in subsequent chapters, this “no versions” philosophy applies only to the core JSON data format. For example, in Chapter 5, that specification is currently at version 0.5 as of this writing. Please note that these JSON-related specifications were created by others in the JSON community.

JSON Comments

There are no comments in a JSON document. Period.

According to his postings on the Yahoo! JSON group and Google+, Crockford initially allowed comments, but removed them early on for the following reasons:

He believed that comments weren’t useful.
JSON parsers had difficulties supporting comments.
People were abusing comments. For example, he noticed that comments were being used for parsing directives, which would have destroyed interoperability.
Removing comments simplified and enabled cross-platform JSON support.

JSON File and MIME Type

According to the core JSON specification, .json is the standard JSON file type when storing JSON data on filesystems. JSON’s Internet Assigned Numbers Authority (IANA) media (or MIME) type is application/json, which can be found at the IANA Media Types site. RESTful Web Service Producers and Consumers use a technique known as content negotiation (which leverages the JSON MIME type in HTTP Headers) to indicate that they are exchanging JSON data.

JSON Style Guidelines

JSON is all about interoperability, and it’s important to provide JSON data feeds in a way that Consumers expect. Google has published a JSON Style Guide to support maintainability and best practices.

The Google JSON Style Guide is extensive, and here are the most important things for an API designer and developer:

Property Names
Date Property Values
Enum Values

Property Names

Property Names (in Google parlance) are on the left side of the colon in a name/value pair (and Property Values are on the righthand side of the hyphen). Two main styles can be used to format a JSON Property Name:

lowerCamelCase
snake_case

With lowerCamelCase, a name is created by joining one or more words to look like a single word, and the first letter in each word is capitalized (except for the first word). Both the Java and JavaScript communities use lowerCamelCase in their coding guides. With snake_case, all letters are lowercase, and words are separated with an underscore (_). But the Ruby on Rails community prefers snake_case.

Google, along with the majority of RESTful APIs, uses lowerCamelCase for its Property Names, as shown in Example 1-12.

Example 1-12. jsonPropertyName.json

{
  "firstName": "John Smith"
}

Date Property Values

You may think that Date formats aren’t that important, but they are. Imagine exchanging date information between a Producer and Consumer who come from different countries or continents. Even within a single enterprise, two development groups will likely use different date formatting conventions. It is important to consider the semantics of how to interpret timestamps so that we have consistent date/time processing and interoperability across all time zones. The Google JSON Style Guide prefers that dates follow the RFC 3339 format, as shown in Example 1-13.

Example 1-13. jsonDateFormat.json

{
  "dateRegistered": "2014-03-01T23:46:11-05:00"
}

The preceding date provides a Coordinated Universal Time (UTC) offset (from UTC/GMT—Greenwich Mean Time) of -5 hours, which is US Eastern Standard Time. Note that RFC 3339 is a profile of ISO 8601. The main difference is notably that the International Standards Organization’s ISO 8601 allows the replacement of the T (which separates the date and time) with a space, and RFC 3339 does not allow this.

Latitude/Longitude Values

Geographical APIs (e.g., Google Maps) and APIs related to a geographical information system (GIS) use latitude/longitude data. To support consistency, the Google JSON Style Guide recommends that latitude/longitude data follows the ISO 6709 standard. According to Google Maps, the coordinates for the Empire State Building in New York City are 40.748747° N, 73.985547° W, and would be represented in JSON as shown in Example 1-14.

Example 1-14. jsonLatLon.json

{
  "empireStateBuilding": "40.748747-73.985547"
}

This example follows the ±DD.DDDD±DDD.DDDD format, with the following conventions:

Latitude comes first.
North (of the equator) latitude is positive.
East (of the prime meridian) longitude is positive.
The latitude/longitude is represented as a String. It can’t be a Number because of the minus sign.

Indentation

Although the Google JSON Style Guide is silent on this topic, here are a few rules of thumb:

JSON is a serialization format, not a presentation format. So, indentation is meaningless to an API Producer or Consumer.
Many JSON Formatters let the user choose between two, three, or four spaces when beautifying a JSON document.
JSON originated from JavaScript (as part of the ECMA 262 standard), but unfortunately there is no single consensus throughout the JavaScript community. Many people and coding style guides prefer two spaces, so this is the convention used in this book for consistency. It’s OK if you prefer another style here, but be consistent.

Our Example—MyConference

Our examples throughout this book cover conference-related data, including the following:

Speakers
Sessions

Our Technical Stack

We’ll start by creating a simple JSON data store for speakers and publishing it to a Stub RESTful API by taking the following steps:

Model JSON data with JSON Editor Online
Generate sample JSON data with JSON Generator
Create and deploy a Stub API (for future testing)

Our Architectural Style—noBackEnd

Our architectural style is based on the concept of noBackend. With noBackend, the developer doesn’t have to worry about the nuts and bolts of application servers or databases at the early stages of application development.

The first seven chapters of this book use noBackEnd architecture to maintain focus on our application from a business perspective (services and data first) so that we can support not only UI-based (e.g., mobile, tablet, and web) clients, but APIs and non-web-based client applications as well. We’ll deploy JSON data with simple tools such as json-server to emulate a RESTful API.

By using this approach, we take an interface-first approach to designing and building an API, which provides the following:

More Agile, rapid, iterative frontend development due to the decoupling from the backend.
Faster feedback on the API itself. Get the data and URI out there quickly for rapid review.
A cleaner interface between the API and its Consumers.
A separation of concerns between the Resource (e.g., speakers as JSON data) exposed by the API and its (eventual) internal implementation (e.g., application server, business logic, and data store). This makes it easier to change implementation in the future. If you create and deploy a real API with Node.js/Rails/Java (or other framework) too early, you’ve already made design decisions at a very early stage that will make it difficult to change after you start working with API Consumers.

A Stub API does the following:

Eliminates the initial need to work with servers and databases
Allows API Producers (those developers who write the API) to focus on API Design, how best to present the data to the Consumers, and initial testing
Enables API Consumers (e.g., UI developers) to work with the API at an early stage and provide feedback to the API development team

By using the lightweight tools in this book, you’ll see that you can go a long way before writing code and deploying it on a server. Of course, you’ll eventually need to implement an API, and we’ll show how to do that when we cover JavaScript, Ruby on Rails, and Java in Chapters 2–4.

Model JSON Data with JSON Editor Online

Creating a valid JSON document of any real size or complexity is tedious and error-prone. JSON Editor Online is a great web-based tool that does the following:

Enables you to model your JSON document as Objects, Arrays, and name/value pairs
Makes it easier to rapidly generate the text for a JSON document in an iterative manner

JSONmate is another solid editor on the web, but we don’t cover it further in this book.

JSON Editor Online features

In addition to JSON modeling and text generation, JSON Editor Online provides the following features:

JSON validation

Validation occurs as you type JSON data in the JSON text area on the left side of the page. If you forget a closing double quote for a value (e.g., "firstName": "Ester,), an X will show next to the following line of JSON text along with hover text that explains the validation error.

JSON pretty-printing

Click the Indent button at the upper-left corner of the JSON text area.

Full roundtrip engineering between the model and JSON text

After creating some Objects and key/value pairs (with the Append (+) button) in the JSON model on the right side of the page, generate JSON text by clicking the left-arrow button (in the upper-middle portion of the page). You should see the changes reflected in the JSON text area on the left side of the page.

Modify some data in the JSON text area and click the right-arrow button, and you should see the changes in the JSON model on the righthand side of the page.

Save JSON document to disk

You can save a JSON document to your local machine by selecting the Save to Disk option under the Save menu.

Import JSON document

You can import a JSON document from your computer by choosing the Open from Disk option from the Open menu.

Please remember that JSON Editor Online is publicly available, which means that any data you paste into this app is visible to others. So don’t use this tool with sensitive information (personal, proprietary, and so forth).

Speaker data in JSON Editor Online

After you’re finished modeling Speaker data, click the right-arrow button to generate a pretty-printed (indented) JSON document that represents the model. Figure 1-3 shows JSON Editor Online with our initial Speakers model.

This is just a rough model, but this initial sketch is a decent starting point. Use the initial model to visualize JSON data, get early feedback, and iterate quickly on the design. This approach enables you to refine the JSON data structure throughout the development life cycle without investing heavily in implementation and infrastructure.

Generate Sample JSON Data with JSON Generator

JSON Editor Online provides a decent start, but we want to generate lots of test data quickly. Test data can be problematic because of the sensitivity of the data, and the data volume needed to do any meaningful testing. Even with JSON Editor Online, it will take a great deal of effort to create the volume of test data we’re looking for. We need another tool to help create the data we need to create our first version of the API, and that’s where JSON Generator comes in. This excellent tool was used to create our speakers.json test data file. The template used to generate the speakers.json file is available on GitHub. Chapter 5 covers JSON Generator in more detail.

Create and Deploy a Stub API

To create the Stub API, we’ll use the Speaker data we just created and deploy it as a RESTful API. We’ll leverage the json-server Node.js module to serve up the speakers.json file as a Web API; this enables us to prototype quickly. You can find more information on the json-server GitHub page.

Before going further, please set up your development environment. Refer to Appendix A to do the following:

Install Node.js. json-server is a Node.js module, so you need to install Node.js first. Refer to “Install Node.js”.
Install json-server. See “Install npm Modules”.
Install JSONView and Postman. See “Install JSON Tools in the Browser”. JSONView pretty-prints JSON in Chrome and Firefox. Postman can also run as a standalone GUI application on most major operating systems.

Open a terminal session and run json-server on port 5000 from your command line:

cd chapter-1

json-server -p 5000 ./speakers.json

You should see the following:

Visit http://localhost:5000/speakers in your browser, and (with JSON pretty-printing provided by JSONView) you should see all the speakers from our Stub API as shown in Figure 1-4.

You can also get a single speaker by adding the id to the URI as follows: http://localhost:5000/speakers/0.

This is a good start, but a web browser has limited testing functionality; it can only send HTTP GET requests. Postman provides the ability to fully test a RESTful API. It can send HTTP GET, POST, PUT, and DELETE requests and set HTTP Headers.

Let’s use Postman to delete the first speaker in the API as follows:

Enter the http://localhost:5000/speakers/0 URL.
Choose DELETE as the HTTP verb.
Click the Send button.

You should see that the DELETE ran properly in Postman with a 200 (OK) HTTP Status, as shown in Figure 1-5.

Now, ensure that the first speaker has truly been deleted by revisiting http://localhost:5000/speakers/0 in your browser. You should now see the empty response shown in Figure 1-6.

You can stop json-server by pressing Ctrl-C at the command line.

With the Stub API in place, we can now invoke it from any HTTP client (e.g., JavaScript, Ruby, or Java) to consume the data from an external application. Although most of our examples in subsequent chapters use an HTTP GET, rest assured that json-server can handle all the core HTTP verbs (GET, POST, PUT, DELETE). Although not covered in this book, Mountebank is an alternative server that provides more robust functionality for stubbing and mocking APIs and protocols.

The main point here is that an API Producer can use JSON-based tools to prototype a testable RESTful API without having to write any code. This technique is powerful because it enables the API Consumer to test without having to wait for the API to be 100 percent complete. At the same time, the API development team can iteratively upgrade the design and prototype.

What We Covered?

We started by covering the basics of JSON. We modeled JSON data with JSON Editor Online, and deployed it with a Stub API.

What’s Next?

The next three chapters show how to use JSON with the following core platforms:

JavaScript
Ruby on Rails
Java

In Chapter 2, you’ll learn how to use JSON in JavaScript with the Stub API we just created with json-server.

Get JSON at Work now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial