diff --git a/cats.md b/cats.md index 0259e61..93fd5a1 100644 --- a/cats.md +++ b/cats.md @@ -158,7 +158,7 @@ Overall, while CATS shows some improvement in Pareto testing, it is less pronoun Database performance testing is inherently complex and error-prone [5]. It cannot be judged by data alone and requires thorough investigation to ensure logical consistency. -## References： +## References [1] B. Tian, J. Huang, B. Mozafari, and G. Schoenebeck. Contention-aware lock scheduling for transactional databases. PVLDB, 11(5), 2018. diff --git a/images/1725756951079.png b/images/1725756951079.png deleted file mode 100644 index d7dcc3e..0000000 Binary files a/images/1725756951079.png and /dev/null differ diff --git a/images/tcpcopy.png b/images/tcpcopy.png new file mode 100644 index 0000000..5310880 Binary files /dev/null and b/images/tcpcopy.png differ diff --git a/tcpcopy_introduction.md b/tcpcopy_introduction.md deleted file mode 100644 index 9282438..0000000 --- a/tcpcopy_introduction.md +++ /dev/null @@ -1,49 +0,0 @@ -# TCPCopy Introduction - -TCPCopy [1] is an open-source TCP traffic replication tool, commonly used for reusing and replaying online traffic to assist in performance testing, stress testing, or debugging of server applications. It can replicate production traffic and send it to a test environment without impacting the production environment, allowing for testing of new applications, features, or architectures. - -Below are some key concepts and mechanisms of TCPCopy's architecture: - -### 1 Traffic Replication Mechanism - -The core function of TCPCopy is to capture traffic from the production environment and replicate it to another designated target server (typically a test server). The specific steps are as follows: - -- **Capturing Production Traffic:** The TCPCopy client (deployed on the production server) listens on a specific port for TCP traffic (such as HTTP requests) and captures the TCP packets from the traffic. -- **Forwarding Traffic:** The captured data packets are sent by the TCPCopy client to the target test server. The test server, usually simulating applications in the production environment, receives this traffic for testing purposes. - -### 2 Traffic Replay on the Target Server - -The traffic sent to the target server by TCPCopy is similar to the production traffic, but when processing this traffic, the test server's configuration needs to be isolated from the production server to avoid application-level overlaps and ensure no impact on the production environment, such as accessing the same backend MySQL database. This process is mainly used to verify the system's performance under production loads, such as: - -- Verifying the correctness of the new version of the code. -- Assessing system performance and scalability. -- Examining how new configurations, architectures, or hardware behave under production-level traffic. - -### 3 Multi-Node Architecture - -In large-scale deployments, TCPCopy often uses a multi-node architecture to distribute the load of traffic replication. Each TCPCopy client can handle a portion of the production server's traffic and, based on a predefined load balancing strategy, replicate the traffic and send it to different test servers. This helps avoid single-point performance bottlenecks. - -### 4 Real-Time and Accuracy - -TCPCopy replicates traffic in near real-time, but not exactly in real-time. - -### 5 Isolation - -Traffic is transmitted between the production and test environments, but TCPCopy does not directly modify production environment data or state. This isolation allows large-scale testing without impacting the stability of the production system. - -### 6 Compatibility and Scalability - -TCPCopy supports a variety of network protocols, especially common application protocols like HTTP and MySQL. It is designed to support large traffic replication tasks and can work efficiently in high-concurrency environments. Additionally, users can combine TCPCopy with other monitoring and analysis tools, such as MySQL Proxy or Wireshark, to further analyze and process the traffic. - -### 7.0 Use Cases - -- **Performance Regression Testing**: Before the new version of a system goes live, replay production traffic using TCPCopy to verify if there are any performance regressions. -- **Fault Diagnosis**: Replay production traffic in a test environment to reproduce anomalous behavior observed in production for debugging and troubleshooting. -- **Stress Testing**: Amplify production traffic by replaying it multiple times to assess the system's performance under high load conditions. - -## **Summary** - -TCPCopy is a simple yet powerful tool, ideal for testing scenarios that require traffic replay from production environments. Its efficient and non-intrusive design makes it an important tool for developers conducting performance tests, fault diagnosis, and system evaluations. - -[1] https://github.com/session-replay-tools/tcpcopy. - diff --git a/tcpcopy_introduction_for_english_user.md b/tcpcopy_introduction_for_english_user.md deleted file mode 100644 index a3f14e8..0000000 --- a/tcpcopy_introduction_for_english_user.md +++ /dev/null @@ -1,304 +0,0 @@ -# Overview - -With the rapid development of internet technology, server-side architectures have become increasingly complex. It is now difficult to rely solely on the personal experience of developers or testers to cover all possible business scenarios. Therefore, real online traffic is crucial for server-side testing. TCPCopy [1] is an open-source traffic replay tool that has been widely adopted by large enterprises. While many use TCPCopy for testing in their projects, they may not fully understand its underlying principles. This article provides a brief introduction to how TCPCopy works, with the hope of assisting readers. - - - -# Architecture - -The architecture of TCPCopy has undergone several upgrades, and this article introduces the latest 1.0 version. As shown in the diagram below, TCPCopy consists of two components: *tcpcopy* and *intercept*. *tcpcopy* runs on the online server, capturing live TCP request packets, modifying the TCP/IP header information, and sending them to the test server, effectively "tricking" the test server. *intercept* runs on an auxiliary server, handling tasks such as relaying response information back to *tcpcopy*. - -![1725756951079](images/1725756951079.png) - -Figure 1. Overview of the TCPCopy Architecture. - -The simplified interaction process is as follows: - -1. *tcpcopy* captures data packets on the online server and parses the IP segments. -2. *tcpcopy* modifies the IP and TCP headers, fakes the source IP and port, and sends the packet to the test server. The fake IP address is determined by the *-x* and *-c* parameters set at startup. -3. The test server receives the request and returns the response packet, with the IP and port being those faked by *tcpcopy*. -4. The response packet is routed to the *intercept* server, where *intercept* captures and parses the IP and TCP headers, typically returning only empty response data to *tcpcopy*. -5. *tcpcopy* receives and processes the returned data. - - - -# Technical Principles - -TCPCopy operates in two modes: online and offline. The online mode is primarily used for real-time capturing of live request packets, while the offline mode reads request packets from pcap-format files. Despite the difference in working modes, the core principles remain the same. This section provides a detailed explanation of TCPCopy's core principles from several perspectives. - - - -## **1. **Packet Capturing and Sending - -The core functions of *tcpcopy* can be summarized as "capturing" and "sending" packets. Let's begin with packet capturing. How do you capture real traffic from the server? Many people may feel confused when first encountering this question. In fact, Linux operating systems already provide the necessary functionality, and a solid understanding of advanced Linux network programming is all that's needed. The initialization of packet capturing and sending in *tcpcopy* is handled in the `tcpcopy/src/communication/tc_socket.c` file. The following section explains the two methods *tcpcopy* uses for packet capturing. - - - -### Raw Socket - -Raw socket, also known as a raw socket, is capable of receiving data frames or packets directly from the network interface on the local machine. This is particularly useful for monitoring and analyzing network traffic. The code for initializing raw socket packet capturing in *tcpcopy* is shown below, and this method supports capturing packets at both the data link layer and the IP layer. - -```c -int -tc_raw_socket_in_init(int type) -{ - int fd, recv_buf_opt, ret; - socklen_t opt_len; - - if (type == COPY_FROM_LINK_LAYER) { - /* copy ip datagram from Link layer */ - fd = socket(AF_PACKET, SOCK_DGRAM, htons(ETH_P_IP)); - } else { - /* copy ip datagram from IP layer */ -#if (TC_UDP) - fd = socket(AF_INET, SOCK_RAW, IPPROTO_UDP); -#else - fd = socket(AF_INET, SOCK_RAW, IPPROTO_TCP); -#endif - } - - if (fd == -1) { - tc_log_info(LOG_ERR, errno, "Create raw socket to input failed"); - fprintf(stderr, "Create raw socket to input failed:%s\n", strerror(errno)); - return TC_INVALID_SOCK; - } - - recv_buf_opt = 67108864; - opt_len = sizeof(int); - - ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &recv_buf_opt, opt_len); - if (ret == -1) { - tc_log_info(LOG_ERR, errno, "Set raw socket(%d)'s recv buffer failed"); - tc_socket_close(fd); - return TC_INVALID_SOCK; - } - - return fd; -} -``` - -The code for initializing the raw socket for sending packets is shown below. First, it creates a raw socket at the IP layer and informs the protocol stack not to append an IP header to the IP layer. - -```c -int -tc_raw_socket_out_init(void) -{ - int fd, n; - - n = 1; - - /* - * On Linux when setting the protocol as IPPROTO_RAW, - * then by default the kernel sets the IP_HDRINCL option and - * thus does not prepend its own IP header. - */ - fd = socket(AF_INET, SOCK_RAW, IPPROTO_RAW); - - if (fd == -1) { - tc_log_info(LOG_ERR, errno, "Create raw socket to output failed"); - fprintf(stderr, "Create raw socket to output failed: %s\n", strerror(errno)); - return TC_INVALID_SOCK; - } - - /* - * tell the IP layer not to prepend its own header. - * It does not need setting for linux, but *BSD needs - */ - if (setsockopt(fd, IPPROTO_IP, IP_HDRINCL, &n, sizeof(n)) < 0) { - tc_socket_close(fd); - tc_log_info(LOG_ERR, errno, - "Set raw socket(%d) option \"IP_HDRINCL\" failed", fd); - return TC_INVALID_SOCK; - } - - - return fd; -} - -``` - - - -Construct the complete IP datagram and send it to the target server. - -- `dst_addr` is filled with the target IP address. -- The IP header is populated with the source and destination IP addresses. -- The TCP header is filled with the source port, destination port, and other relevant information. - - - -### Pcap - -Pcap is an application programming interface (API) provided by the operating system for capturing network traffic, with its name derived from 'packet capture.' On Linux systems, pcap is implemented via libpcap, and most packet capture tools, such as *tcpdump*, use libpcap for capturing traffic. - -The code below shows how to initialize pcap for packet sending. - -```c -int -tc_pcap_socket_in_init(pcap_t **pd, char *device, - int snap_len, int buf_size, char *pcap_filter) -{ - int fd; - char ebuf[PCAP_ERRBUF_SIZE]; - struct bpf_program fp; - bpf_u_int32 net, netmask; - - if (device == NULL) { - return TC_INVALID_SOCK; - } - - tc_log_info(LOG_NOTICE, 0, "pcap open,device:%s", device); - - *ebuf = '\0'; - - if (tc_pcap_open(pd, device, snap_len, buf_size) == TC_ERR) { - return TC_INVALID_SOCK; - } - - if (pcap_lookupnet(device, &net, &netmask, ebuf) < 0) { - tc_log_info(LOG_WARN, 0, "lookupnet:%s", ebuf); - return TC_INVALID_SOCK; - } - - if (pcap_compile(*pd, &fp, pcap_filter, 0, netmask) == -1) { - tc_log_info(LOG_ERR, 0, "couldn't parse filter %s: %s", - pcap_filter, pcap_geterr(*pd)); - return TC_INVALID_SOCK; - } - - if (pcap_setfilter(*pd, &fp) == -1) { - tc_log_info(LOG_ERR, 0, "couldn't install filter %s: %s", - pcap_filter, pcap_geterr(*pd)); - pcap_freecode(&fp); - return TC_INVALID_SOCK; - } - - pcap_freecode(&fp); - - if (pcap_get_selectable_fd(*pd) == -1) { - tc_log_info(LOG_ERR, 0, "pcap_get_selectable_fd fails"); - return TC_INVALID_SOCK; - } - - if (pcap_setnonblock(*pd, 1, ebuf) == -1) { - tc_log_info(LOG_ERR, 0, "pcap_setnonblock failed: %s", ebuf); - return TC_INVALID_SOCK; - } - - fd = pcap_get_selectable_fd(*pd); - - return fd; -} -``` - -The code for initializing packet sending with pcap is as follows: - -```c -int -tc_pcap_snd_init(char *if_name, int mtu) -{ - char pcap_errbuf[PCAP_ERRBUF_SIZE]; - - pcap_errbuf[0] = '\0'; - pcap = pcap_open_live(if_name, mtu + sizeof(struct ethernet_hdr), - 0, 0, pcap_errbuf); - if (pcap_errbuf[0] != '\0') { - tc_log_info(LOG_ERR, errno, "pcap open %s, failed:%s", - if_name, pcap_errbuf); - fprintf(stderr, "pcap open %s, failed: %s, err:%s\n", - if_name, pcap_errbuf, strerror(errno)); - return TC_ERR; - } - - return TC_OK; -} -``` - - - -### Raw Socket vs. Pcap - -Since *tcpcopy* provides two methods for packet capture, which one is better? - -When capturing packets, we are primarily concerned with the specific packets we need. If the capture configuration is not set correctly, the system kernel might capture too many irrelevant packets, leading to packet loss, especially under high traffic pressure. After extensive testing, it has been found that when using the pcap interface to capture request packets, the packet loss rate in live environments is generally higher than when using raw sockets. Therefore, *tcpcopy* defaults to using raw sockets for packet capture, although the pcap interface can also be used (with the `--enable-pcap` option), which is mainly suited for high-end pfring captures and captures after switch mirroring. - -For packet sending, *tcpcopy* uses the raw socket output interface by default, but it can also send packets via pcap_inject (using the `--enable-dlinject` option). The choice of which method to use can be determined based on performance testing in your actual environment. - - - -## **2. TCP Protocol Stack** - -We know that the TCP protocol is stateful. Although the packet sending mechanism was explained earlier, without establishing an actual TCP connection, the sent packets cannot be truly received by the testing service. In everyday network programming, we typically use the TCP socket interfaces provided by the operating system, which abstract away much of the complexity of TCP states. However, in *tcpcopy*, since we need to modify the source IP and destination IP of the packets to deceive the testing service, the APIs provided by the operating system are no longer sufficient. - -As a result, *tcpcopy* implements a simulated TCP state machine, representing the most complex and challenging aspect of its codebase. The relevant code, located in `tcpcopy/src/tcpcopy/tc_session.c`, handles crucial tasks such as simulating TCP interactions, managing network latency, and emulating upper-layer interactions. - -![](D:/github/The-Art-of-Problem-Solving-in-Software-Engineering_How-to-Make-MySQL-Better/media/ff831787f6acc1e38a7f70dc47925ee0.gif) - -Figure 2. Classic TCP state machine overview. - -In *tcpcopy*, a session is defined to maintain information for different connections. Different captured packets are processed accordingly: - -- **SYN Packet:** Represents a new connection request. *tcpcopy* assigns a source IP, modifies the destination IP and port, then sends the packet to the test server. At the same time, it creates a new session to store all states of this connection. -- **ACK Packet:** - - **Pure ACK Packet:** To reduce the number of sent packets, *tcpcopy* generally doesn't send pure ACKs. - - **ACK Packet with Payload (indicating a specific request):** It finds the corresponding session and sends the packet to the test server. If the session is still waiting for the response to the previous request, it delays sending. -- **RST Packet:** If the current session is waiting for the test server's response, the RST packet is not sent. Otherwise, it's sent. -- **FIN Packet:** If the current session is waiting for the test server's response, it waits; otherwise, the FIN packet is sent. - - - -## **3. Routing** - -After *tcpcopy* sends the request packets, their journey may not be entirely smooth: - -- The IP of the request packet is forged and not the actual IP of the machine running *tcpcopy*. If some machines have rpfilter (reverse path filtering) enabled, it will check whether the source IP address is trustworthy. If the source IP is untrustworthy, the packet will be discarded at the IP layer. -- If the test server receives the request packet, the response packet will be sent to the forged IP address. To ensure these response packets don't mistakenly go back to the client with the forged IP, proper routing configuration is necessary. If the routing isn't set up correctly, the response packet won't be captured by *intercept*, leading to incomplete data exchange. -- Once *intercept* captures the response packet, it extracts the IP datagram, discards the actual data, and only returns the response headers and essential information to *tcpcopy*. When necessary, it also merges the return information to reduce the impact on the network of the machine running *tcpcopy*. - - - -## **4. Intercept** - -For those new to *tcpcopy*, it might be puzzling—why is *intercept* necessary if we already have *tcpcopy*? While *intercept* may seem redundant, it actually plays a crucial role. You can think of *intercept* as the server-side counterpart of *tcpcopy*, with its name itself explaining its function: an "interceptor." But what exactly does *intercept* need to intercept? The answer is the response data from the test service. - -Without *intercept*, the test service's response data would be sent directly back to *tcpcopy*. Since *tcpcopy* operates in a live production environment, this would mean the response data would be sent directly to the online servers, significantly increasing their network load and potentially impacting the normal operation of the online services. By using *intercept*, and by forging the source IP, the test service is tricked into "believing" the requests are meant for it. *intercept* also merges and optimizes the response data packet information, further ensuring that the live production environment is unaffected by the testing environment on the network level. - -*intercept* is an independent process that, by default, captures packets using the pcap method. During startup, the `-F` parameter needs to be passed, for example, "tcp and src port 8080," following libpcap's filter syntax. This means that *intercept* does not connect directly to the test service but listens on the specified port, capturing the return data packets from the test service and interacting with *tcpcopy*. - - - -## **5. Performance** - -*tcpcopy* uses a single-process, single-thread architecture based on an epoll/select event-driven model, with related code located in the `tcpcopy/src/event` directory. By default, epoll is used during compilation, though you can switch to select with the `--select` option. The choice of method can depend on the performance differences observed during testing. Theoretically, epoll performs better when handling a large number of connections. - -In practical use, *tcpcopy*'s performance is directly tied to the amount of traffic and the number of connections established by *intercept*. The single-threaded architecture itself is usually not a performance bottleneck (for instance, nginx and redis both use single-threaded + epoll models and can handle large amounts of concurrency). Since *tcpcopy* only establishes connections directly with *intercept* and does not need to connect to the test machines or occupy port numbers, *tcpcopy* consumes fewer resources, with the main impact being on network bandwidth consumption. - -```c -static tc_event_actions_t tc_event_actions = { -#ifdef TC_HAVE_EPOLL - tc_epoll_create, - tc_epoll_destroy, - tc_epoll_add_event, - tc_epoll_del_event, - tc_epoll_polling -#else - tc_select_create, - tc_select_destroy, - tc_select_add_event, - tc_select_del_event, - tc_select_polling -#endif -}; -``` - - - -# Conclusion - -TCPCopy is an excellent open-source project. However, due to the author's limitations, this article only covers the core technical principles of TCPCopy, leaving many details untouched. Nevertheless, I hope this introduction provides some inspiration to those interested in TCPCopy and traffic replay technologies! - -# References： - -[1] https://github.com/session-replay-tools/tcpcopy. - -[1] Bin Wang (2024). The Art of Problem-Solving in Software Engineering:How to Make MySQL Better. diff --git a/tcpcopy_introduction_for_user.md b/tcpcopy_introduction_for_user.md deleted file mode 100644 index b76a2d5..0000000 --- a/tcpcopy_introduction_for_user.md +++ /dev/null @@ -1,293 +0,0 @@ -# 简述 - -随着互联网技术的迅速发展，服务端架构变得日益复杂，单靠开发或测试人员的个人经验已难以全面覆盖各种业务场景。因此，真实的线上流量对于服务端测试至关重要。TCPCopy是一款开源的流量回放工具，已广泛应用于多家大型企业。虽然许多人在项目中使用TCPCopy进行测试，但对其底层原理可能不甚了解。本文将简要介绍TCPCopy的工作原理，希望能为读者提供帮助。 - - - -# 架构 - -TCPCopy的架构经历了多次升级，本文介绍的是最新的1.0版本架构。正如下图所示，TCPCopy由两部分组成：*tcpcopy*和*intercept*。*tcpcopy*在线上服务器上运行，负责捕获在线TCP请求数据包，修改TCP/IP头部信息，并将其发送至测试服务器，巧妙地实现对测试服务器的“欺骗”。*intercept*在辅助服务器上运行，执行一些辅助任务，例如将响应信息传递回*tcpcopy*。 - -![1725756951079](images/1725756951079.png) - -其简化的交互流程如下： - -①. *tcpcopy*在线上服务器抓取数据包。 - -②. *tcpcopy*修改IP头和TCP头，伪造源IP和端口，并将数据包发送到测试服务器。伪造的IP地址依据启动时的*-x*和*-c*参数设置。 - -③. 测试服务器接收请求并返回响应包，响应包中的目的IP和端口为*tcpcopy*伪造的IP和端口。 - -④. 响应包被路由到*intercept*服务器，由*intercept*捕获并解析IP头和TCP头，通常只返回空响应数据给*tcpcopy*。 - -⑤. *tcpcopy*接收并处理返回数据。 - - - -# 技术原理 - -TCPCopy有在线和离线两种工作模式。在线模式主要用于实时捕获在线请求数据包，离线模式则从pcap格式文件中读取请求数据包。尽管工作方式不同，其核心原理相同。下文将从多个方面详细介绍TCPCopy的核心原理。 - -## **1. 抓包与发包** - -*tcpcopy* 的核心功能可以简单概括为“抓包”和“发包”，那我们就从“抓包”开始谈起。如何捕获服务器的真实流量？许多人在刚接触这个问题时可能感到困惑。其实，Linux 操作系统已经提供了相关功能，只需掌握 Linux 高级网络编程的知识即可。*tcpcopy* 的抓包和发包代码都初始化在 `tcpcopy/src/communication/tc_socket.c` 文件中。接下来介绍 *tcpcopy* 抓发包的两种方式： - - - -### Raw socket - -Raw socket，即原始套接字，能够接收本机网卡上的数据包，对于监听和分析网络流量非常有用。*tcpcopy* 中用于初始化抓包的 raw socket 代码如下，这种方式支持数据链路层和 IP 层的抓包。 - -```c -int -tc_raw_socket_in_init(int type) -{ - int fd, recv_buf_opt, ret; - socklen_t opt_len; - - if (type == COPY_FROM_LINK_LAYER) { - /* copy ip datagram from Link layer */ - fd = socket(AF_PACKET, SOCK_DGRAM, htons(ETH_P_IP)); - } else { - /* copy ip datagram from IP layer */ -#if (TC_UDP) - fd = socket(AF_INET, SOCK_RAW, IPPROTO_UDP); -#else - fd = socket(AF_INET, SOCK_RAW, IPPROTO_TCP); -#endif - } - - if (fd == -1) { - tc_log_info(LOG_ERR, errno, "Create raw socket to input failed"); - fprintf(stderr, "Create raw socket to input failed:%s\n", strerror(errno)); - return TC_INVALID_SOCK; - } - - recv_buf_opt = 67108864; - opt_len = sizeof(int); - - ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &recv_buf_opt, opt_len); - if (ret == -1) { - tc_log_info(LOG_ERR, errno, "Set raw socket(%d)'s recv buffer failed"); - tc_socket_close(fd); - return TC_INVALID_SOCK; - } - - return fd; -} -``` - -初始化发包的 raw socket 代码如下，首先创建IP层的 raw socket 套接字，并告知协议栈不再为IP层追加IP头。 - -```c -int -tc_raw_socket_out_init(void) -{ - int fd, n; - - n = 1; - - /* - * On Linux when setting the protocol as IPPROTO_RAW, - * then by default the kernel sets the IP_HDRINCL option and - * thus does not prepend its own IP header. - */ - fd = socket(AF_INET, SOCK_RAW, IPPROTO_RAW); - - if (fd == -1) { - tc_log_info(LOG_ERR, errno, "Create raw socket to output failed"); - fprintf(stderr, "Create raw socket to output failed: %s\n", strerror(errno)); - return TC_INVALID_SOCK; - } - - /* - * tell the IP layer not to prepend its own header. - * It does not need setting for linux, but *BSD needs - */ - if (setsockopt(fd, IPPROTO_IP, IP_HDRINCL, &n, sizeof(n)) < 0) { - tc_socket_close(fd); - tc_log_info(LOG_ERR, errno, - "Set raw socket(%d) option \"IP_HDRINCL\" failed", fd); - return TC_INVALID_SOCK; - } - - - return fd; -} - -``` - -构建完整的数据包并发送给目标服务器。 - -- `dst_addr` 填充目标IP地址 -- IP头部填充源IP和目标IP等信息 -- TCP头部填充源端口、目标端口等信息 - - - -### Pcap - -Pcap 是操作系统提供的用于捕获网络流量的应用程序接口（API），名称源于“抓包”（packet capture）。在 Linux 系统中，pcap 通过 libpcap 实现，大多数流量抓取工具（如 *tcpdump* 等）也都是通过 libpcap 来进行抓包。 - -下面展示了初始化抓包 pcap 的代码。 - -```c -int -tc_pcap_socket_in_init(pcap_t **pd, char *device, - int snap_len, int buf_size, char *pcap_filter) -{ - int fd; - char ebuf[PCAP_ERRBUF_SIZE]; - struct bpf_program fp; - bpf_u_int32 net, netmask; - - if (device == NULL) { - return TC_INVALID_SOCK; - } - - tc_log_info(LOG_NOTICE, 0, "pcap open,device:%s", device); - - *ebuf = '\0'; - - if (tc_pcap_open(pd, device, snap_len, buf_size) == TC_ERR) { - return TC_INVALID_SOCK; - } - - if (pcap_lookupnet(device, &net, &netmask, ebuf) < 0) { - tc_log_info(LOG_WARN, 0, "lookupnet:%s", ebuf); - return TC_INVALID_SOCK; - } - - if (pcap_compile(*pd, &fp, pcap_filter, 0, netmask) == -1) { - tc_log_info(LOG_ERR, 0, "couldn't parse filter %s: %s", - pcap_filter, pcap_geterr(*pd)); - return TC_INVALID_SOCK; - } - - if (pcap_setfilter(*pd, &fp) == -1) { - tc_log_info(LOG_ERR, 0, "couldn't install filter %s: %s", - pcap_filter, pcap_geterr(*pd)); - pcap_freecode(&fp); - return TC_INVALID_SOCK; - } - - pcap_freecode(&fp); - - if (pcap_get_selectable_fd(*pd) == -1) { - tc_log_info(LOG_ERR, 0, "pcap_get_selectable_fd fails"); - return TC_INVALID_SOCK; - } - - if (pcap_setnonblock(*pd, 1, ebuf) == -1) { - tc_log_info(LOG_ERR, 0, "pcap_setnonblock failed: %s", ebuf); - return TC_INVALID_SOCK; - } - - fd = pcap_get_selectable_fd(*pd); - - return fd; -} -``` - -初始化发包pcap的代码如下： - -``` -int -tc_pcap_snd_init(char *if_name, int mtu) -{ - char pcap_errbuf[PCAP_ERRBUF_SIZE]; - - pcap_errbuf[0] = '\0'; - pcap = pcap_open_live(if_name, mtu + sizeof(struct ethernet_hdr), - 0, 0, pcap_errbuf); - if (pcap_errbuf[0] != '\0') { - tc_log_info(LOG_ERR, errno, "pcap open %s, failed:%s", - if_name, pcap_errbuf); - fprintf(stderr, "pcap open %s, failed: %s, err:%s\n", - if_name, pcap_errbuf, strerror(errno)); - return TC_ERR; - } - - return TC_OK; -} -``` - - - -### Raw socket VS. pcap - -既然 *tcpcopy* 提供了两种抓发包方式，那么哪一种更好呢？ - -对于抓包，我们关注的仅仅是需要的包。如果抓包设置不当，系统内核可能会捕获过多的无关数据包，从而导致丢包现象，尤其是在高流量压力下更为明显。经过长时间的测试，发现利用 pcap 接口抓取请求数据包时，在线上环境中的丢包率通常高于使用 raw socket 的方式。因此，*tcpcopy* 默认使用 raw socket 进行抓包，尽管也可以选择使用 pcap 接口（通过 `--enable-pcap` 选项），这种方式主要适用于高端 pfring 抓包和交换机镜像后的抓包。 - -对于发包，*tcpcopy* 默认使用 raw socket 输出接口，也可以通过 *pcap_inject*（使用 `--enable-dlinject` 选项）来发包。具体选择哪种方式，可以根据实际环境中的性能测试结果来决定。 - - - -## **2. tcp协议栈** - -我们知道 TCP 协议是有状态的协议。尽管上述介绍了发包的原理，但如果不建立真正的 TCP 连接，发送的包也无法被测试服务真正接收。在日常网络编程中，我们通常使用操作系统提供的封装好的 TCP socket 接口编程，无需过多关注 TCP 状态。然而，在 *tcpcopy* 中，由于需要修改数据包的源 IP 和目的 IP 以欺骗测试服务，操作系统提供的 API 已经无法满足需求。因此，*tcpcopy* 实现了一个模拟的 TCP 有效状态机，这也是 *tcpcopy* 源码中最复杂、最具挑战性的部分，具体代码位于 `tcpcopy/src/tcpcopy/tc_session.c` 文件中。 - -![](D:/github/The-Art-of-Problem-Solving-in-Software-Engineering_How-to-Make-MySQL-Better/media/ff831787f6acc1e38a7f70dc47925ee0.gif) - -Figure 1. Classic TCP state machine overview. - -在 *tcpcopy* 中，定义了一个 session 用于维护不同连接的信息。对不同的抓取数据包会做不同的处理： - -- **SYN包**：表示新的连接请求。*tcpcopy* 会分配源 IP，修改目的 IP 和端口，然后将其发送到测试机。同时，新建一个 session 来保存此次会话中的所有状态。 -- **ACK包**： - - 纯 ACK 包：为了降低发包数量，一般不发送。 - - 带有 payload 的 ACK 包（表示具体请求）：找到对应的 session 并发送到测试机。如果本 session 正在等待上次请求的响应，则暂缓发送。 -- **RST包**：如果当前 session 正在等待测试机的结果返回，则不发送 RST 包，否则发送。 -- **FIN包**：如果当前 session 正在等待测试机的结果返回，则等待；否则发送 FIN 包。 - - - -## **3. 路由** - -*tcpcopy* 发送请求数据包后，其路途可能不会是“一帆风顺”的： - -- 请求数据包的 IP 是伪造的，而非 *tcpcopy* 所在机器的真实 IP。如果某些机器启用了 rpfilter（反向路径过滤技术），它会检查源 IP 地址是否可信。如果源 IP 不可信，数据包将在 IP 层被丢弃。 -- 如果测试服务接收到了请求数据包，返回的数据包会被发送到伪造的 IP 地址。为了确保这些响应数据包不会错误地返回到伪造 IP 的客户端，需要正确配置路由。如果路由配置不正确，响应包将无法被 *intercept* 捕获，导致数据交互不完整。 -- *intercept* 捕获到响应包后，会提取响应数据包，并丢弃具体数据，只返回响应头等必要信息给 *tcpcopy*。必要时，还会对返回信息进行合并，以减少对 *tcpcopy* 所在机器网络的影响。 - - - -## **4. Intercept** - -刚开始接触 *tcpcopy* 的同学可能会感到困惑，既然有了 *tcpcopy*，为何还需要 *intercept* 呢？*intercept* 看似多余，但实际上扮演了重要角色。可以把 *intercept* 理解为 *tcpcopy* 的服务端，其名字本身就解释了它的作用：“拦截器”。那么，什么需要 *intercept* 来拦截呢？答案是测试服务的响应数据。 - -如果没有 *intercept*，测试服务的响应数据包将直接返回给 *tcpcopy*。由于 *tcpcopy* 部署在线上环境，这意味着测试服务的响应数据包会直接发送到线上服务器，极大地增加了线上服务器的网络负载，甚至可能影响在线服务的正常运行。有了 *intercept* 后，通过伪造源 IP，可以让测试服务“误以为”这些请求是发送给它的。*intercept* 还对返回的响应数据包信息进行了合并优化，进一步确保了网络层面线上环境不受测试环境的影响。 - -*intercept* 是一个独立进程，其默认使用 pcap 方式进行抓包。在启动时，需要传入 `-F` 参数，例如 “tcp and src port 8080”，以符合 libpcap 的过滤语法。这意味着 *intercept* 不直接与测试服务连接，而是监听指定端口，抓取测试服务的返回数据包，并与 *tcpcopy* 进行交互。 - -## **5. 性能** - -*tcpcopy* 采用了单进程单线程架构，基于 epoll/select 事件驱动，其相关代码位于 `tcpcopy/src/event` 目录。默认情况下，编译时使用了 epoll，也可以通过 `--select` 选项切换到 select。可以根据测试结果的性能差异选择不同的方式。理论上，在连接数较多的情况下，epoll 的性能会更优。 - -在实际使用中，*tcpcopy* 的性能直接与流量大小以及 *intercept* 建立的连接数相关。单线程本身通常不会成为性能瓶颈（例如，nginx 和 redis 都采用了单线程 + epoll 的方式，并能处理大量并发）。由于 *tcpcopy* 仅与 *intercept* 直接建立连接，而不需要与测试机器建立连接或占用端口号，因此 *tcpcopy* 的资源消耗较少，主要体现在网络带宽的消耗上。 - -``` -static tc_event_actions_t tc_event_actions = { -#ifdef TC_HAVE_EPOLL - tc_epoll_create, - tc_epoll_destroy, - tc_epoll_add_event, - tc_epoll_del_event, - tc_epoll_polling -#else - tc_select_create, - tc_select_destroy, - tc_select_add_event, - tc_select_del_event, - tc_select_polling -#endif -}; -``` - - - -# 结语 - -TCPCopy是一款十分优秀的开源项目，但因为笔者能力有限，本文只介绍了TCPCopy的核心技术原理，很多细节并未涉及。不过还是希望通过本文的介绍，能够对TCPCopy和流量回放技术感兴趣的同学有所启发！ -