Shell命令行处理JSON

博主：想长胖的阿鹏
发布时间：2021 年 11 月 02 日
441次浏览
暂无评论
3929字数
分类：随便写写

Loading...

> 转载自：https://www.cnblogs.com/sheng-247/p/14120462.html

### 基本介绍

经常用python或java处理json数据，其实在Shell命令行，我们可以用开源JQ工具来处理Json数据。会更加简洁和高效。

### CentOS 安装JQ

```
wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
chmod +x ./jq
cp jq /usr/bin
```

### JQ常用举例

#### 1、原始JSON数据

```
cat jsonData.json
{"productId":"2723","click":60,"view":300,"deal":2,"day":"20130919"}
{"productId":"2728","click":130,"view":800,"deal":10,"day":"20130919"}
{"productId":"3609","click":50,"view":400,"deal":3,"day":"20130919"}
{"productId":"3783","click":375,"view":1200,"deal":50,"day":"20130919"}
{"productId":"3522","click":87,"view":573,"deal":15,"day":"20130919"}
```

#### 2、格式化输出

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
jq -r . jsonData.json
{
  "productId": "2723",
  "click": 60,
  "view": 300,
  "deal": 2,
  "day": "20130919"
}
{
  "productId": "2728",
  "click": 130,
  "view": 800,
  "deal": 10,
  "day": "20130919"
}
...
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

#### 2、提取json指定字段转csv

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
这里我们需要得到productId_day,click,view,deal这种格式的csv文件。

jq -r '[.productId+"_"+.day,(.click|tostring),(.view|tostring),(.deal|tostring)]|join(",")' jsonData.json
结果：
2723_20130919,60,300,2
2728_20130919,130,800,10
3609_20130919,50,400,3
3783_20130919,375,1200,50
3522_20130919,87,573,15
注意：
  .productId:取productId字段的值
　.click|tostring:取click字段的值并转化为string类型
　[]:数组
　join(","):数组中的元素通过","连接。
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

#### 3、复杂json数据查询

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
complexJson='{"uid":"2378","nameInfo":[{"firstName":"nova","lastName":"jos"}]}'
echo ${complexJson} |jq
{
  "uid": "2378",
  "nameInfo": [
    {
      "firstName": "nova",
      "lastName": "jos"
    }
  ]
}
echo ${complexJson} |jq -r '.nameInfo[].lastName'
结果:jos
注意:.nameInfo[].lastName是获取.nameInfo[] json array中每个json object的lastName字段。
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

#### 4、提取指定字段形成新的json

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
jq -c '{productId:.productId,click:.click,dayYmd:(.day|tonumber)}' jsonData.json >>newjson.json
结果:
cat newjson.json
{"productId":"2723","click":60,"dayYmd":20130919}
{"productId":"2728","click":130,"dayYmd":20130919}
{"productId":"3609","click":50,"dayYmd":20130919}
{"productId":"3783","click":375,"dayYmd":20130919}
{"productId":"3522","click":87,"dayYmd":20130919}
注意:
 -c/--compact-output:jq 默认是pretty-prints JSON 输出，加上这个参数，一行一个json对象。
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

#### 5、csv文件转json文件

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
cat csvData.csv
1478,john,38
1529,lucy,25
1673,iris,22
jq -R -c 'split(",")|{"uid":.[0],"name":.[1],"age":.[2]|tonumber}' csvData.csv > csv2json.json
cat csv2json.json
{"uid":"1478","name":"john","age":38}
{"uid":"1529","name":"lucy","age":25}
{"uid":"1673","name":"iris","age":22}
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

### 主要参考文档

### 1、*[https://programminghistorian.org/lessons/json-and-jq](https://programminghistorian.org/lessons/json-and-jq) *

*2、JQ使用手册:[https://stedolan.github.io/jq/manual/#Basicfilters](https://stedolan.github.io/jq/manual/#Basicfilters)
3、JQ Cookbook:[https://github.com/stedolan/jq/wiki/Cookbook](https://github.com/stedolan/jq/wiki/Cookbook)*

最后修改：2021 年 11 月 02 日 05 : 15 PM

© 转载自他站

要多恰饭才能长胖

发表评论取消回复

评论 *

私密评论

名称 *

🎲

邮箱

地址

Shell命令行处理JSON

想长胖的阿鹏 • 2021 年 11 月 02 日

Loading...

> 转载自：https://www.cnblogs.com/sheng-247/p/14120462.html

### 基本介绍

经常用python或java处理json数据，其实在Shell命令行，我们可以用开源JQ工具来处理Json数据。会更加简洁和高效。

### CentOS 安装JQ

```
wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
chmod +x ./jq
cp jq /usr/bin
```

### JQ常用举例

#### 1、原始JSON数据

```
cat jsonData.json
{"productId":"2723","click":60,"view":300,"deal":2,"day":"20130919"}
{"productId":"2728","click":130,"view":800,"deal":10,"day":"20130919"}
{"productId":"3609","click":50,"view":400,"deal":3,"day":"20130919"}
{"productId":"3783","click":375,"view":1200,"deal":50,"day":"20130919"}
{"productId":"3522","click":87,"view":573,"deal":15,"day":"20130919"}
```

#### 2、格式化输出

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
jq -r . jsonData.json
{
  "productId": "2723",
  "click": 60,
  "view": 300,
  "deal": 2,
  "day": "20130919"
}
{
  "productId": "2728",
  "click": 130,
  "view": 800,
  "deal": 10,
  "day": "20130919"
}
...
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

#### 2、提取json指定字段转csv

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
这里我们需要得到productId_day,click,view,deal这种格式的csv文件。

jq -r '[.productId+"_"+.day,(.click|tostring),(.view|tostring),(.deal|tostring)]|join(",")' jsonData.json
结果：
2723_20130919,60,300,2
2728_20130919,130,800,10
3609_20130919,50,400,3
3783_20130919,375,1200,50
3522_20130919,87,573,15
注意：
  .productId:取productId字段的值
　.click|tostring:取click字段的值并转化为string类型
　[]:数组
　join(","):数组中的元素通过","连接。
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

#### 3、复杂json数据查询

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
complexJson='{"uid":"2378","nameInfo":[{"firstName":"nova","lastName":"jos"}]}'
echo ${complexJson} |jq
{
  "uid": "2378",
  "nameInfo": [
    {
      "firstName": "nova",
      "lastName": "jos"
    }
  ]
}
echo ${complexJson} |jq -r '.nameInfo[].lastName'
结果:jos
注意:.nameInfo[].lastName是获取.nameInfo[] json array中每个json object的lastName字段。
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

#### 4、提取指定字段形成新的json

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
jq -c '{productId:.productId,click:.click,dayYmd:(.day|tonumber)}' jsonData.json >>newjson.json
结果:
cat newjson.json
{"productId":"2723","click":60,"dayYmd":20130919}
{"productId":"2728","click":130,"dayYmd":20130919}
{"productId":"3609","click":50,"dayYmd":20130919}
{"productId":"3783","click":375,"dayYmd":20130919}
{"productId":"3522","click":87,"dayYmd":20130919}
注意:
 -c/--compact-output:jq 默认是pretty-prints JSON 输出，加上这个参数，一行一个json对象。
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

#### 5、csv文件转json文件

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

```
cat csvData.csv
1478,john,38
1529,lucy,25
1673,iris,22
jq -R -c 'split(",")|{"uid":.[0],"name":.[1],"age":.[2]|tonumber}' csvData.csv > csv2json.json
cat csv2json.json
{"uid":"1478","name":"john","age":38}
{"uid":"1529","name":"lucy","age":25}
{"uid":"1673","name":"iris","age":22}
```

[![复制代码](https://blog.zpzp.xyz/usr/uploads/2021/11/315291722.gif)](javascript:void(0); "复制代码")

### 主要参考文档

### 1、*[https://programminghistorian.org/lessons/json-and-jq](https://programminghistorian.org/lessons/json-and-jq) *

*2、JQ使用手册:[https://stedolan.github.io/jq/manual/#Basicfilters](https://stedolan.github.io/jq/manual/#Basicfilters)
3、JQ Cookbook:[https://github.com/stedolan/jq/wiki/Cookbook](https://github.com/stedolan/jq/wiki/Cookbook)*

財

發