August Feng

A study on tcp sequence numbers

Relative sequence numbers

Let's simulate a some TCP traffic between a server (nc -l 1234) and client (echo foobar | nc localhost 1234).

If we dump this with tcpdump -i lo port 1234, we'll see that the sequence numbers reset to 1 after the initial handshake is completed.

1. 22:57:07.921614 IP localhost.47162 > localhost.1234: Flags [S], seq 934227076, win 65495, options [mss 65495,sackOK,TS val 2756153644 ecr 0,nop,wscale 7], length 0
2. 22:57:07.921626 IP localhost.1234 > localhost.47162: Flags [S.], seq 2435073417, ack 934227077, win 65483, options [mss 65495,sackOK,TS val 2756153644 ecr 2756153644,nop,wscale 7], length 0
3. 22:57:07.921636 IP localhost.47162 > localhost.1234: Flags [.], ack 1, win 512, options [nop,nop,TS val 2756153644 ecr 2756153644], length 0
4. 22:57:07.921688 IP localhost.47162 > localhost.1234: Flags [P.], seq 1:8, ack 1, win 512, options [nop,nop,TS val 2756153644 ecr 2756153644], length 7
5. 22:57:07.921692 IP localhost.1234 > localhost.47162: Flags [.], ack 8, win 512, options [nop,nop,TS val 2756153644 ecr 2756153644], length 0
6. 22:57:07.921700 IP localhost.47162 > localhost.1234: Flags [F.], seq 8, ack 1, win 512, options [nop,nop,TS val 2756153644 ecr 2756153644], length 0
7. 22:57:07.921758 IP localhost.1234 > localhost.47162: Flags [F.], seq 1, ack 9, win 512, options [nop,nop,TS val 2756153644 ecr 2756153644], length 0
8. 22:57:07.921781 IP localhost.47162 > localhost.1234: Flags [.], ack 2, win 512, options [nop,nop,TS val 2756153644 ecr 2756153644], length 0

Note that I've enumerated the packets to help with illustration in the following sections.

Absolute tcp sequence numbers

This is actually just tcpdump using relative sequence numbers to make it easier. We can get the actual sequence number values by providing the S (-absolute-tcp-sequence-number) flag:

1. 22:57:39.363197 IP localhost.35954 > localhost.1234: Flags [S], seq 625186957, win 65495, options [mss 65495,sackOK,TS val 2756185086 ecr 0,nop,wscale 7], length 0
2. 22:57:39.363207 IP localhost.1234 > localhost.35954: Flags [S.], seq 960599189, ack 625186958, win 65483, options [mss 65495,sackOK,TS val 2756185086 ecr 2756185086,nop,wscale 7], length 0
3. 22:57:39.363216 IP localhost.35954 > localhost.1234: Flags [.], ack 960599190, win 512, options [nop,nop,TS val 2756185086 ecr 2756185086], length 0
4. 22:57:39.363263 IP localhost.35954 > localhost.1234: Flags [P.], seq 625186958:625186965, ack 960599190, win 512, options [nop,nop,TS val 2756185086 ecr 2756185086], length 7
5. 22:57:39.363267 IP localhost.1234 > localhost.35954: Flags [.], ack 625186965, win 512, options [nop,nop,TS val 2756185086 ecr 2756185086], length 0
6. 22:57:39.363275 IP localhost.35954 > localhost.1234: Flags [F.], seq 625186965, ack 960599190, win 512, options [nop,nop,TS val 2756185086 ecr 2756185086], length 0
7. 22:57:39.363329 IP localhost.1234 > localhost.35954: Flags [F.], seq 960599190, ack 625186966, win 512, options [nop,nop,TS val 2756185086 ecr 2756185086], length 0
8. 22:57:39.363349 IP localhost.35954 > localhost.1234: Flags [.], ack 960599191, win 512, options [nop,nop,TS val 2756185086 ecr 2756185086], length 0

Acknowledgements during the handshake

The second packet is the server that acknowledges the client's sequence number (625186958 = 625186957 + 1), and the third packet is client that acknowledges the server's sequence number (960599190 = 960599189 + 1).

Next sequence number

The tcpdump's output describes the fourth packet as having a sequence number: seq 625186958:625186965.

The first number, 625186958, is the client's sequence number after the handshake. The second number, 625186965 is calculated by adding the length of the payload onto the client's sequence number.

The 625186965 number does not appear anywhere in the packet, it's just tcpdump that's assisting us.

Immediately afterwards, we can see the server acknowledging the 625186965 sequence number that was computed ahead of time by tcpdump.

Acknowledgments during the connection termination

On the sixth packet, the client's FIN + ACK signals the beginning of a TCP connection termination.

I expected this to only be a FIN as per all the diagrams illustrated online. I guess the client just wants to re-acknowledge the most recent sequence number received from the server.

Afterwards, the server responds to the client's FIN (+ ACK) flags with an acknowledgment number that bumps the sequence number received earlier from the client.

Finally, the client confirms the server's FIN+ ACK with an ACK with an acknowledgement number that bumps the sequence number received earlier from the server.

Can't send, but should read (aka TIME-WAIT)

Once the client terminated the connection, it can no longer send data but can still receive data.

It should continue reading data until the other side terminates as well.

This means that we'll see a connection with TIME-WAIT when we run ss -ta on the client side even though the application has already exited. Since the application has already exited, it won't be able to receive any data so I guess the operating system will just eat the data.

MacOS weirdness

I originally ran the experiment on MacOS and observed some really weird behaviors. These behaviors did not appear when I ran the experiments in a docker container.

Nevertheless, I'm going to preserve the original experiment's weirdness here.

1.  18:10:18.422576 IP 127.0.0.1.60566 > 127.0.0.1.1234: Flags [S], seq 1198051244, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 337147874 ecr 0,sackOK,eol], length 0
2.  18:10:18.422685 IP 127.0.0.1.1234 > 127.0.0.1.60566: Flags [S.], seq 1838975788, ack 1198051245, win 65535, options [mss 16344,nop,wscale 6,nop,nop,TS val 135178069 ecr 337147874,sackOK,eol], length 0
3.  18:10:18.422705 IP 127.0.0.1.60566 > 127.0.0.1.1234: Flags [.], ack 1838975789, win 6379, options [nop,nop,TS val 337147874 ecr 135178069], length 0
4.  18:10:18.422717 IP 127.0.0.1.1234 > 127.0.0.1.60566: Flags [.], ack 1198051245, win 6379, options [nop,nop,TS val 135178069 ecr 337147874], length 0
5.  18:10:18.422735 IP 127.0.0.1.60566 > 127.0.0.1.1234: Flags [P.], seq 1198051245:1198051252, ack 1838975789, win 6379, options [nop,nop,TS val 337147874 ecr 135178069], length 7
6.  18:10:18.422746 IP 127.0.0.1.60566 > 127.0.0.1.1234: Flags [F.], seq 1198051252, ack 1838975789, win 6379, options [nop,nop,TS val 337147874 ecr 135178069], length 0
7.  18:10:18.422765 IP 127.0.0.1.1234 > 127.0.0.1.60566: Flags [.], ack 1198051252, win 6379, options [nop,nop,TS val 135178069 ecr 337147874], length 0
8.  18:10:18.422777 IP 127.0.0.1.1234 > 127.0.0.1.60566: Flags [.], ack 1198051253, win 6379, options [nop,nop,TS val 135178069 ecr 337147874], length 0
9.  18:10:18.422818 IP 127.0.0.1.1234 > 127.0.0.1.60566: Flags [F.], seq 1838975789, ack 1198051253, win 6379, options [nop,nop,TS val 135178069 ecr 337147874], length 0
10. 18:10:18.422854 IP 127.0.0.1.60566 > 127.0.0.1.1234: Flags [.], ack 1838975790, win 6379, options [nop,nop,TS val 337147874 ecr 135178069], length 0

On macOS, there's a fourth packet after the tcp handshake

For some reason, there's a fourth packet right after the Three-Way TCP handshake. It repeats the acknowledgement done in the second packet.

On macOS, there's a bit of goof when acknowedging the FIN of a connection termination

On the seventh packet, the server is acknowledging the 545858608 sequence number, which is a little bit weird since it's supposed to do a + 1. On the eight packet it does what I expected.