0%

tcpdump

​ 使用tcpdump + wireshark 可以很好的帮助我们解决线上抓包问题!这里同时也推荐一下我写的一个工具,tcpdump decoder, 因为线上我们并没有wireshark的可视化工具,往往需要tcpdump -w tcpdump.pcap 然后本地通过wireshark调试,是的太过于麻烦和冗余,因此我个人开了一个工具,可以在线解析 HTTP/Thrift流量!

tcpdump 命令介绍

  1. 一般来说 tcpdump 命令有三部分组成, tcpdump -i [interface] '[pacp expression]' [options]
  1. 获取网卡. 一般你需要确定你要抓取的ip,然后再去确认网卡, 例如我要抓取10.248.166.215 则需要使用 eth0的网卡!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
fanhaodong.516:~/ $ ip addr                                                                                                                                            [14:23:51]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether fa:16:3e:34:9d:82 brd ff:ff:ff:ff:ff:ff
inet 10.248.166.215/22 brd 10.248.167.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fdbd:dc03:ff:1:1:248:166:215/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe34:9d82/64 scope link
valid_lft forever preferred_lft forever
  1. 抓包
1
2
3
4
5
6
7
8
9
➜  tcpdump -i lo0  'port 8080 and host ::1' -l -n -v
tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
14:11:38.539268 IP6 (flowlabel 0x20900, hlim 64, next-header TCP (6) payload length: 212) ::1.8080 > ::1.59678: Flags [P.], cksum 0x00dc (incorrect -> 0x72c5), seq 633202846:633203026, ack 3120721099, win 6164, options [nop,nop,TS val 1356080951 ecr 3287837093], length 180: HTTP, length: 180
HTTP/1.1 200 OK
Content-Encoding: gzip
Date: Wed, 31 Aug 2022 06:11:38 GMT
Transfer-Encoding: chunked

3d
  1. 高级过滤
1
2
3
4
5
6
7
8
9
10
11
12
13
# 区分 src 和 dst 
tcpdump -i lo0 'src port 8080 and dst host ::1' -X

# 抓取sync包,也就是握手包
➜ tcpdump -i lo0 'tcp[13]=0x02' -l -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
14:20:24.679509 IP 127.0.0.1.63788 > 127.0.0.1.9229: Flags [S], seq 220694465, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 3788980622 ecr 0,sackOK,eol], length 0
14:20:24.680149 IP 127.0.0.1.63790 > 127.0.0.1.9229: Flags [S], seq 3821191416, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 278963318 ecr 0,sackOK,eol], length 0
14:20:25.682374 IP 127.0.0.1.63792 > 127.0.0.1.9229: Flags [S], seq 1726308526, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 818205733 ecr 0,sackOK,eol], length 0
14:20:25.683262 IP 127.0.0.1.63794 > 127.0.0.1.9229: Flags [S], seq 3871219645, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 1976538711 ecr 0,sackOK,eol], length 0
14:20:26.685361 IP 127.0.0.1.63797 > 127.0.0.1.9229: Flags [S], seq 2836663358, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 2399073465 ecr 0,sackOK,eol], length 0
14:20:26.686232 IP 127.0.0.1.63799 > 127.0.0.1.9229: Flags [S], seq 1158021919, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 837560897 ecr 0,sackOK,eol], length 0
  1. 特殊case,我们很多时候请求的是 域名,那么怎么解决呢?可以通过下面两种方式获取
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#方式1. 获取a记录
➜ test git:(fix/bug/1_0_4) ✗ dig www.baidu.com
## ....
www.baidu.com. 134 IN CNAME www.a.shifen.com.
www.a.shifen.com. 134 IN A 220.181.38.149
www.a.shifen.com. 134 IN A 220.181.38.150

#方式2. curl 获取ip
➜ curl www.baidu.com -v
* Trying 220.181.38.150:80...
* Connected to www.baidu.com (220.181.38.150) port 80 (#0)


#方式3. 直接过滤host
tcpdump -i en0 'host www.baidu.com' -l -n -v

抓取HTTP服务流量

  1. 我们需要创建一个HTTP Server, 这个接口有点复杂,是Chunked编码+压缩的!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
package main

import (
"fmt"
"log"

"github.com/anthony-dong/go-sdk/commons"

"github.com/anthony-dong/go-sdk/commons/codec/http_codec"

"github.com/gin-gonic/gin"
)

func main() {
router := gin.Default()
router.GET("/api/v1/:compress", func(context *gin.Context) {
resp := map[string]interface{}{
"data": commons.NewString('A', 1024*8),
}
data := commons.ToJsonString(resp)
context.Writer.Header().Set("Transfer-Encoding", "chunked")
compress := context.Param("compress")
if !http_codec.CheckAcceptEncoding(context.Request.Header, compress) {
compress = ""
}
if err := http_codec.EncodeHttpBody(context.Writer, context.Writer.Header(), []byte(data), compress); err != nil {
context.JSON(500, map[string]interface{}{
"error": err.Error(),
})
return
}
})
fmt.Println("Listen: http://localhost:8080\nTest: http://localhost:8080/api/v1/gzip")
if err := router.Run(":8080"); err != nil {
log.Fatal(err)
}
}
  1. 启动并且抓取流量
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
➜  test git:(fix/bug/1_0_4) ✗ tcpdump -i lo0 'port 8080' -l -v -n
tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
14:31:22.066978 IP6 (flowlabel 0x80f00, hlim 64, next-header TCP (6) payload length: 814) ::1.50327 > ::1.8080: Flags [P.], cksum 0x0336 (incorrect -> 0x2a59), seq 3927208909:3927209691, ack 3528028485, win 6369, options [nop,nop,TS val 4277916016 ecr 1002452734], length 782: HTTP, length: 782
GET /api/v1/gzip HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Cache-Control: max-age=0
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Google Chrome";v="104"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9
Cookie: Hm_lvt_899b2a5c34078209c5f30853eaaa7846=1650731778,1652082863,1652948029

14:31:22.067020 IP6 (flowlabel 0x10e00, hlim 64, next-header TCP (6) payload length: 32) ::1.8080 > ::1.50327: Flags [.], cksum 0x0028 (incorrect -> 0x68da), ack 782, win 6347, options [nop,nop,TS val 1002463298 ecr 4277916016], length 0
14:31:22.067407 IP6 (flowlabel 0x10e00, hlim 64, next-header TCP (6) payload length: 189) ::1.8080 > ::1.50327: Flags [P.], cksum 0x00c5 (incorrect -> 0xdce5), seq 1:158, ack 782, win 6347, options [nop,nop,TS val 1002463298 ecr 4277916016], length 157: HTTP, length: 157
HTTP/1.1 200 OK
Content-Encoding: gzip
Date: Wed, 31 Aug 2022 06:31:22 GMT
Transfer-Encoding: chunked

26
14:31:22.067429 IP6 (flowlabel 0x80f00, hlim 64, next-header TCP (6) payload length: 32) ::1.50327 > ::1.8080: Flags [.], cksum 0x0028 (incorrect -> 0x682a), ack 158, win 6366, options [nop,nop,TS val 4277916016 ecr 1002463298], length 0

我们可以发现并不能拿到chunked编码的内容,这里只展示了第一行 26,如何解决呢? 通过 gtool tcpdump

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
➜  test git:(fix/bug/1_0_4) ✗ tcpdump -i lo0 'port 8080' -l -X -n | gtool tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
14:32:08.549007 IP6 ::1.8080 > ::1.50327: Flags [.], ack 3927210473, win 6335, length 0
14:32:08.549037 IP6 ::1.50327 > ::1.8080: Flags [.], ack 1, win 6364, options [nop,nop,TS val 4277962497 ecr 1002494741], length 0
14:32:08.570433 IP6 ::1.50327 > ::1.8080: Flags [P.], seq 1:783, ack 1, win 6364, options [nop,nop,TS val 4277962519 ecr 1002494741], length 782: HTTP: GET /api/v1/gzip HTTP/1.1
GET /api/v1/gzip HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Cache-Control: max-age=0
sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Google Chrome";v="104"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: zh-CN,zh;q=0.9
Cookie: Hm_lvt_899b2a5c34078209c5f30853eaaa7846=1650731778,1652082863,1652948029

14:32:08.570456 IP6 ::1.8080 > ::1.50327: Flags [.], ack 783, win 6323, options [nop,nop,TS val 1002509801 ecr 4277962519], length 0
14:32:08.571189 IP6 ::1.8080 > ::1.50327: Flags [P.], seq 1:158, ack 783, win 6323, options [nop,nop,TS val 1002509801 ecr 4277962519], length 157: HTTP: HTTP/1.1 200 OK
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Encoding: gzip
Date: Wed, 31 Aug 2022 06:32:08 GMT

{"data":"AAAAAAAAAAAAAAAAAAAA"}
14:32:08.571208 IP6 ::1.50327 > ::1.8080: Flags [.], ack 158, win 6362, options [nop,nop,TS val 4277962519 ecr 1002509801], length 0

配合wireshark使用

  1. 介绍

wireshark 是一个完全开源的项目,出身也比较早,九十年代的产物,核心思想就是利用pcap去解析数据包,并支持强大的过滤语法,源码: https://gitlab.com/wireshark/wireshark, 文档: https://www.wireshark.org/docs/man-pages/wireshark-filter.html

  1. 其实wireshark 提供了很多cli,帮助dump数据,具体可以看: https://www.wireshark.org/docs/man-pages/

  2. 只需要tcpdump -w [filename] 就OK了,是的还可以 tcpdump -r [filename] 也可以读的!

1
2
➜ tcpdump -i lo0 'port 8080' -w ~/data/tcpdump.pcap
tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
  1. 读取文件(一般抓包文件的文件后缀是 .pcap) , 或者直接监听网卡

image-202208311435538535. 使用 flow 可以更好的查看连接上的请求!

image-20220831144055147

使用 gtool tcpdump

  1. 介绍: https://github.com/Anthony-Dong/go-sdk/blob/master/gtool/tcpdump

本项目只是为了了解 pcap+流量解析,虽然wireshark是一个非常好的GUI工具,但是线上并不能直接使用,其次就是使用 tcpdump大部分可以满足能力!所以我工具并没有造轮子,更多的是在线解析业务流量,关注于解析应用层流量!

  • 支持解析TCP的包
  • 支持解析HTTP包(HTTP/1.1 & HTTP/1.0),支持自动根据content-encoding类型进行解析!
  • 支持解析Thrift包,支持多种协议,包含Kitex的TTHeader 和 Thrift 官方协议(Framed、THeader、Unframed)!
  • 支持解析大包(粘包问题)
  • 支持tcpdump在线/离线流量解析
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
➜  gtool tcpdump -h
Name: decode tcpdump file, help doc: https://github.com/Anthony-Dong/go-sdk/tree/master/gtool/tcpdump

Usage: gtool tcpdump [-r file] [-v] [-X] [--max dump size] [flags]

Examples:
1. step1: tcpdump 'port 8080' -w ~/data/tcpdump.pcap
step2: gtool tcpdump -r ~/data/tcpdump.pcap
2. tcpdump 'port 8080' -X -l -n | gtool tcpdump


Options:
-X, --dump Enable Display payload details with hexdump.
-r, --file string The packets file, eg: tcpdump_xxx_file.pcap.
-h, --help help for tcpdump
--max int The hexdump max size
-v, --verbose Enable Display decoded details.

Global Options:
--config-file string set the config file (default "/Users/bytedance/.gtool.yaml")
--log-level string set the log level in "fatal|error|warn|info|debug" (default "debug")

To get more help with gtool, check out our guides at https://github.com/Anthony-Dong/go-sdk
  1. 使用 (注意管道符读取,需要-X输出,因为其他会丢失原数据)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
fanhaodong.516:go-sdk/ $ sudo tcpdump -i eth0 'port 8888' -l -n -X | bin/gtool tcpdump                     [14:08:10]
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:08:51.426622 IP 10.225.xx.196.28284 > 10.248.xx.215.8888: Flags [S], seq 174152772, win 28200, options [mss 1410,sackOK,TS val 445695352 ecr 0,nop,wscale 10], length 0
14:08:51.426726 IP 10.248.xx.215.8888 > 10.225.xx.196.28284: Flags [S.], seq 2533803252, ack 174152773, win 28960, options [mss 1460,sackOK,TS val 2106570914 ecr 445695352,nop,wscale 10], length 0
14:08:51.430699 IP 10.225.xx.196.28284 > 10.248.xxx.215.8888: Flags [.], ack 1, win 28, options [nop,nop,TS val 445695357 ecr 2106570914], length 0
14:08:51.430790 IP 10.225.xx.196.28284 > 10.248.xxx.215.8888: Flags [P.], seq 1:987, ack 1, win 28, options [nop,nop,TS val 445695357 ecr 2106570914], length 986
{
"method": "SimpleTestRPC",
"seq_id": 324,
"protocol": "UnframedBinary",
"message_type": "call",
"payload": {
"1_STRUCT": {
"255_STRUCT": {
"1_STRING": "111",
"2_STRING": "xxxx.xxx.xx",
"3_STRING": "10.xxx.xxx.xxx",
"4_STRING": "",
"6_MAP": {
"cluster": "test",
"env": "prod",
"idc": "xxx",
"tracestate": "_sr=1",
"user_extra": ""
}
},
"1_STRING": "hello world"
}
},
"meta_info": {}
}
14:08:51.430803 IP 10.248.xxx.215.8888 > 10.225.xxx.196.28284: Flags [.], ack 987, win 32, options [nop,nop,TS val 2106570918 ecr 445695357], length 0
14:08:51.432384 IP 10.248.xxx.215.8888 > 10.225.xxx.196.28284: Flags [P.], seq 1:890, ack 987, win 32, options [nop,nop,TS val 2106570919 ecr 445695357], length 889
{
"method": "SimpleTestRPC",
"seq_id": 324,
"protocol": "UnframedBinary",
"message_type": "reply",
"payload": {
"0_STRUCT": {
"1_STRING": "hello world",
"2_STRING": "hello, world!",
"255_STRUCT": {
"2_I32": 0,
"1_STRING": "",
"3_MAP": {
"_CUSTOM_CLUSTER": "default",
"_CUSTOM_ENV": "prod",
"_CUSTOM_IDC": "xxx",
"_CUSTOM_IP": "10.xxx.xx.xx",
"_CUSTOM_IP_V4": "xxx.xxx.xxx.xxx",
"_CUSTOM_IP_V6": "xxxx",
}
}
}
},
"meta_info": {}
}
14:08:51.436438 IP 10.225.xx.196.28284 > 10.248.xxx.215.8888: Flags [.], ack 890, win 31, options [nop,nop,TS val 445695362 ecr 2106570919], length 0
14:08:51.637336 IP 10.225.xx.196.28284 > 10.248.xx.215.8888: Flags [F.], seq 987, ack 890, win 31, options [nop,nop,TS val 445695563 ecr 2106570919], length 0
14:08:51.637552 IP 10.248.xx.215.8888 > 10.225.xx.196.28284: Flags [F.], seq 890, ack 988, win 32, options [nop,nop,TS val 2106571125 ecr 445695563], length 0
14:08:51.641670 IP 10.225.xx.196.28284 > 10.248.xxx.215.8888: Flags [.], ack 891, win 31, options [nop,nop,TS val 445695568 ecr 2106571125], length 0
本人坚持原创技术分享,如果你觉得文章对您有用,请随意打赏! 如果有需要咨询的请发送到我的邮箱!