قالب وردپرس درنا توس
Home / Tips and Tricks / How to parse JSON files on the Linux command line with jq

How to parse JSON files on the Linux command line with jq



  A terminal prompt on a Linux PC.
Fatmawati Achmad Zaenuri / Shutterstock

JSON is one of the most popular formats for transferring text-based data over the internet. It is everywhere and you will certainly encounter it. We show you how to do this from the Linux command line with the command jq .

JSON and jq

JSON stands for JavaScript Object Notation. It is a scheme that allows data to be coded in plain text files into plain text files. There are no comments in a JSON file ̵

1; the content must be self-explanatory. Each data value has a text string called a "name" or "key". This tells you what the data value is. Together they are known as name: value pairs or key: value pairs. A colon (: ) separates a key from its value.

An "object" is a collection of key: value pairs. In a JSON file, an object starts with an open brace ( {) and ends with a closing brace (} ). JSON also supports "arrays", ordered lists of values. An array starts with an opening hook ( [) and ends with a closing bracket (] ).

Of course, random complexity can arise from these simple definitions. For example, objects can be nested in objects. Objects can contain arrays and arrays can also contain objects. All of which can have an open nest level.

In practice, however, if the layout of JSON data is complicated, the design of the data layout should probably use a reconsideration. Of course, if you do not generate the JSON data, but simply try to use it, you have no control over the layout. In those cases, you unfortunately have to deal with it.

Most programming languages ​​have libraries or modules that they can use to parse JSON data. Unfortunately, the Bash shell does not have such functionality.

Need as the mother of the invention, however, the jq program was born! With jq we can easily parse JSON into the Bash shell. And it doesn't matter if you have to work with well-designed, elegant JSON, or what the nightmares are made of.

How to install jq

We had to jq on all the Linux distributions that we used to research this article.

If you want to install jq on Ubuntu, type this command:

  sudo apt-get install jq 

  The

Install jq on Fedora, type this command:

  sudo dnf install jq 

  The

To install jq on this command, type this command:

  sudo pacman -Sy jq [19659021] The "sudo pacman -Sy jq" command in a terminal window." width="646" height="57" src="/pagespeed_static/1.JiBnMqyl6S.gif" onload="pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);" onerror="this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);"/> 

How to make JSON readable

JSON does not care about white space and the layout has no influence on this. As long as it follows the rules of JSON grammar, systems that process JSON can read and understand it. As a result, JSON is often transferred as a simple, long series, without regard to the layout. This saves a little space because tabs, spaces and characters with a new line do not have to be included in the JSON. The disadvantage of all this is of course when a person tries to read it.

Let's get a short JSON object from the NASA site that tells us the position of the international space station. We will use curl which can download files to get the JSON object for us.

We usually don't care about the status messages curl usually generates, so we type the following, using the option -s (silent):

  curl -s http: //api.open-notify.org/iss-now.json cialis19659027BuchThe "curl -s http://api.open-notify.org/iss-now.json" command in a terminal window. "width =" 646 " height = "97" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfayibleAndVayibleAndBaycon (this) > 

Now you can read this with a little effort, you have to choose the data values, but it is not easy or convenient, let's repeat this, but this time we will guide it through jq . [19659006] jq uses filters to parse JSON, and the simplest of these filters is a period (. ), which means "print the entire object." Standard prints jq the output nicely finished.

We all put it together and type the following:

  curl -s http://api.open-notify.org/iss-now.json | jq. 

  The

That is much better! Now we can see exactly what is going on.

The entire object is wrapped in braces. It contains two key: name pairs: message and timestamp . It also contains an object named iss_position that contains two keys: value pairs: longitude and latitude .

We will try this again. This time we will type the following and redirect the output to a file named "iss.json":

  curl -s http://api.open-notify.org/iss-now.json | jq. > iss.json 
  cat iss.json 

  The iss.json "and" cat iss.json "commands in a terminal window." width = "646" height = "262" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" />

This gives us a well-formatted copy of the JSON object on our hard drive.

RELATED: [19659041] How to use curl to download files from the Linux command line

Access to data values ​​

As we saw above, jq Extract data values ​​that are forwarded from JSON. It can also work with JSON stored in a file. We are going to work with local files, so the command line is not full of curl commands. This should make tracking a bit easier.

The easiest way to extract data from a JSON file is to specify a key name to get the data value. Type a period and the key name without a space between them. Creates a filter based on the key name. We must also tell jq which JSON file to use.

We type the following to retrieve the value message :

  jq .message iss.json 

  The

jq pressed the text of the message value in the terminal window.

If you have a key name that contains spaces or punctuation marks, you must wrap the filter in quotation marks. It is usually ensured that only characters, numbers, and underscores are used so that the JSON key names are not a problem.

First we type the following to get the value timestamp :

  jq .timestamp iss .json 

  The

The time stamp value is retrieved and printed in the terminal window .

But how can we access the values ​​in the object iss_position ? We can use the JSON point notation. We include the object name iss_position in the & # 39; path & # 39; to the key value. To do this, the name of the object in which the key is located will precede the name of the key itself.

We type the following, including the key name latitude (note that there are no spaces between ". Iss_position" and ".latitude"):

  jq .iss_position.latitude iss. json 

  The

To extract multiple values, you must do the following:

  • Enter the key names on the command line.
  • Separate them with commas & # 39; s ().
  • Put them in quotation marks ( ") or apostrophes ( & # 39; ).

With that in mind, we type the following:

  jq" .iss_position .latitude, .timestamp "iss.json 

  The

The two values ​​are printed to the terminal window. [19659011] Working with arrays

Let's get another JSON object from NASA. [19659006] This time we use a list of astronauts currently in space:

  curl -s http: // api. Open notification y.org/astros.json cialis19659070 accommodate "curl -s http: // api .open-notify.org / astros.json "command in a terminal window." width = "646" height = "147" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" /> 

Okay, that worked, so let's do it again.

We type the following to implement it jq and to redirect to a file named "astro.json":

  curl -s http: //api.open-notify .org / astros.json | jq. > astro.json 

  The astros.json "command in a terminal window." width = "646" height = "77" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" [>19659006] Now let's type the following to check our file:

  less astro.json 

  The [19659006] As shown below, we now see the list of astronauts in space, as well as their spaceships.

 Output from

This JSON object contains an array called people . We know that it is an array because of the opening hook ( [) (highlighted in the screenshot above). It is a series of objects that each contain two keys: value pairs: name and craft .

As we did before, we can use the JSON point notation to access the values. We must also include the brackets ([]) in the name of the array. With all this in mind, we type the following:

  jq ".people [] .name" astro.json [19659085] The "jq" .people [] .name "astros.json" command in a terminal window. "width =" 646 "height =" 197 "src =" / pagespeed_static / 1.JiBnMqyl6S.gif "onload =" pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this); "onerror =" this.onerror = null; pagespeed .lazyLoadImages.loadIfVisibleAndMaybeBeacon (this); "/> 

This time all name values ​​are printed to the terminal window. What we asked jq was to print the name value for each object in the array. Pretty neat, right ?

We can retrieve the name of a single object if we place its position in the array in parentheses ([]) on the command line.The array uses zero offset indexing, which means that the object is on the first The position of the array is zero.

To access the last object in the array you can use -1; bject in the array, you can use -2 and so on.

Sometimes the JSON object provides the number of elements in the array, which is the case with this. Together with the array, it contains a key: name pair named number with a value of six.

The following number of objects is in this array:

  jq ".people [1]. Name" astro.json 
  jq ".people [3] .name" astro.json 
  jq " .people [-1] .name "astro.json 
  jq" .people [-2] .name "astro.json 

  The

You can also specify a start and end object within the array. This is called "sticking" and it can be a little confusing. Remember that the array uses a zero offset.

To retrieve the objects from index position two, to (but not including) the object at index position four, we type the following command:

  jq ".people [2:4]" astro.json 

  The

This prints the objects on array index two (the third object in the array) and three (the fourth object in the array). Processing is stopped at array index four, the fifth object in the array.

The way to better understand this is by experimenting on the command line. You will soon see how it works.

Using pipes with filters

You can filter the output from one filter to the other and you don't have to learn a new symbol. Same as the Linux command line, jq uses the vertical bar ( | ) to represent a pipe.

We tell jq to pass the pipe people array in the filter .name in which the names of the astronauts should be listed in the terminal window.

We type the following:

  jq ".people [] | .name" astro.json 

  The

RELATED: How to use pipes on Linux

Creating arrays and changing results

We can use jq to create new objects, such as arrays. In this example, we will extract three values ​​and create a new array that contains these values. Note that the opening ( [) and closures (] ) are also the first and last characters in the filter series.

We type the following:

  jq "[.iss-position.latitude, iss_position.longitude, .timestamp]" iss.json 

  The

The output is enclosed in brackets and separated by commas, making it a correctly formed array.

Numeric values ​​can also be manipulated while being retrieved. Let's take the time stamp from the ISS position file and then unpack it again and change the returned value.

To do this, we type the following:

  jq ".timestamp" iss. json 
  jq ".timestamp - 1570000000" iss.json 

  The

This is useful when you need to add or remove a standard offset from a range of values.

Let us type the following to remind ourselves of what the file iss.json contains:

  jq. iss.json 

  The

Let's say we want to remove the message key: value pair. It has nothing to do with the position of the international space station. It is just a flag indicating that the location has been successfully retrieved. If there are too many requirements, we can abandon them. (You can also just ignore it.)

We can use the delete function of jq del () to delete a key: value pair. To delete the message key: value pair, we type this command:

  jq "del (.message)" iss.json 

  The

Note that this is not really removed from the "iss. json ”file; it only removes it from the command output. If you need to create a new file without the message : value pair in it, execute the command and send the output to a new file.

More complicated JSON objects

Let's get some NASA data. This time we will use a JSON object that contains information about meteor impact locations from around the world. This is a larger file with a much more complicated JSON structure than the ones we discussed earlier.

First, we type the following to redirect it to a file called "strikes.json":

  curl - s https://data.nasa.gov/resource/y77d-th95.json | jq. > strikes.json 

  The strikes.json "command in a terminal window." width = "646" height = "77" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" />

To see what JSON looks like, we type the following:

  fewer strikes.json 

  The

As shown below, the file starts with an opening hook ( [), so the entire object is an array. The objects in the array are collections of key: value pairs and there is a nested object named geolocation . The geolocation object contains further key: value pairs and an array named coordinates .

 Output of the

Let's retrieve the names of the meteor strikes from the object at index position 995 to the end of the series.

We type the following to guide the JSON through three filters:

  jq ". [995:] |. [] | .name" strikes.json 

  The

The filters work in the following ways:

  • . [995:]: This tells jq to process the objects from the array, index 995 to the end of the series. No number after the colon (: ) is what jq tells to proceed to the end of the array.
  • . []: This iterator array tells jq to process every object in the array.
  • .name : This filter retrieves the name value.

With a small change we can remove the last 10 objects from the array. A "-10" instructs jq to process objects 10 back from the end of the array.

We type the following:

  jq ". [-10:] |. [] |. Name" strikes.json 

  The

As in earlier examples, we can type the following to select a single object:

  jq ". [650] .name" strikes.json [19659158] The "jq". [650] .name "strikes.json" command in a terminal window. "width = "646" height = "97" src = "/ pagespeed_static / 1.JiBnMqyl6S.gif" onload = "pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);" onerror = "this.onerror = null; pagespeed.lazyLoadImagesealoadIaybeayconayIaybeayconayIaybeayconayImayeaconconstruction ; "/> 

We can also apply cuts to strings. To do this, we type the following to request the first four characters of the object's name at array index 234:

  jq". [234] .name [0:4] "strikes.json 

  The

We can also see a specific object in its entirety. To do this, we type the following and include an array index without a key: value filters :

  jq ". [234] "strikes.json 

  The

If you only enter the values, you can do the same without the key names.

For our example, we type this command:

  jq". [234][] "strikes.json 

  The

to retrieve multiple values ​​from each object, we separate them with commas in the following command:

  jq". [450:455] | [] | .name, .mass "strikes.json 

  The

To retrieve nested values, you must identify the objects that form the" path "to them.

For example, to refer to the values ​​of coordinates we must include the all-embracing matrix, the geolocation nested object, and the nested coordinates array, as shown below.

 The path to array in a nested JSON object marked in a terminal window.

To see the coordinates values ​​for the object at index position 121 of the array, we type the following command:

  jq " [121]. Geolocation. coordinates [] "strikes.json 

  The

The length Function

The jq length function gives different measurement values ​​based on what it is used for, such as:

  • Strings : The length of the string in bytes.
  • Objects : The number of keys: value pairs in the object.
  • Arrays : The number of array elements in the array.

The following command returns the length of the value name in 10 of the objects in the JSON array, starting at index position 100:

  jq ". [100:110] | . [] .name | length "strikes.json 

  De

To see how many key: value pairs are in the first object in the array, we type this command:

  jq". [0] | length "strikes.json 

  The

The keys Function

You can use the keys function to find out more about the JSON you need to work with. He can tell you the names of the keys and how many objects there are in an array.

To find the keys in the people object in the "astro.json" file, we type this command:

  jq ". people. [0] | keys "astro.json 

  The

To see how many elements there are in the people array, we type this command:

  jq" .people | keys "astro.json 

  The

This shows that there are six, zero-offset array elements, numbered from zero to five.

The has () function

You can use the Using has () function to interrogate the JSON and see if an object has a specific key name. Please note that the key name must be enclosed in quotes. We filter the filter command in single quotes ( & # 39; ), as follows:

  jq & # 39 ;. [] | has ("nametype") & # 39; strikes.json 

  The

Any object in the array is checked, as shown below. [19659006]   Output of the "jq & # 39 ;. [] | has ("nametype") & # 39; strikes.json & # 39; assignment in a terminal window.

If you want to check a specific object, include the index position in the array filter as follows:

  jq & # 39; . [678] | has ("nametype") & # 39; strikes.json 

  The

Don't go near JSON Without It

The utility jq is the perfect example of the professional, powerful, fast software that makes life in the Linux world so enjoyable.

This was only a brief introduction to the general functions of this command - there is much more to it. jq manual if you want to dig deeper.




Source link