NFS client tcpdump analysis: 3 common failure scenarios
Updated
Introduction
This document describes a few common failure scenarios between an NFSv4 client and NFSv4 server, as seen from the perspective of an NFSv4 client tcpdump. The scenarios were simulated by various means, and are meant to give a sample to compare to a real-world tcpdump on the NFS client. This can often be used to assist with diagnosing a communication problem between an NFS client and server, such as is seen by "nfs: server ... not responding, still trying" messages.
The analysis is done using tshark, and assume the /var/log/messages file as well as the tcpdump.pcap file has been gathered on just the NFS client, with proper filters to isolate the NFS client and server connection. In all scenarios, the NFS server is at 192.168.122.50 and the NFS client is at 192.168.122.5. Note if your tcpdump contains multiple TCP connections or other traffic, you may need to use additional filters to isolate the one NFS client to NFS server communications. In addition, all of the following scenarios involve NFSv4. If NFSv3 is being analyzed, different filters may be required (for example, you may want to look at traffic for the MOUNT protocol / port, rather than just the NFS port 2049).
Often the goal when gathering a tcpdump is to isolate the problem into one of 3 categories:
- A problem between the NFS client and NFS server
- A problem on the NFS server
- A problem on the NFS client
Once the problem is isolated, further troubleshooting is required to fix the problem, and is beyond the scope of this article.
Contents
- Scenario 1, NFS client tcpdump: the NFS server does not respond at all
- Scenario 2, NFS client tcpdump: the NFS server responds with a TCP-level ACK, but not an NFS response
- Scenario 3, NFS client tcpdump: the NFS server responds both with a TCP-level ACK and NFS reply, but NFS client re-sends the same NFS / RPC Call
Scenario 1, NFS client tcpdump: the NFS server does not respond at all
This scenario was simulated by blocking the NFS port at the NFS server's input firewall.
This scenario unfortunately does not narrow the problem to any of the 3 categories, but is useful to determine that further information is definitely required. To further troubleshoot, you must gather a tcpdump on the NFS server, or use a port mirror to see the traffic from the NFS server perspective.
In a tcpdump, look for the presence of 'RPC retransmission' and 'TCP Retransmission', as well as the lack of any packets coming from the NFS server IP. This is the main distinguishing characteristic of this scenario - the total lack of response from the NFS server. In the example below, the communication looks fine between the NFS client and NFS server from frames 1 until frame 46. Then in frame 47 the NFS client transmits a NFSv4 GETATTR Call but does not receive a response. We see the first retransmission from the NFS client in frame 48 which occurred at 13:35:40 and the retransmissions continue until the 'not responding, still trying" message is printed at 13:39:04.
# tshark -ntad -r tcpdump.pcap.gz -R "tcp.port == 2049"
1 2015-02-04 13:35:24.956860 192.168.122.5 -> 192.168.122.50 TCP 74 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21210556 TSecr=0 WS=64
4 2015-02-04 13:35:24.957765 192.168.122.50 -> 192.168.122.5 TCP 74 2049 > 747 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1818037730 TSecr=21210556 WS=64
5 2015-02-04 13:35:24.957822 192.168.122.5 -> 192.168.122.50 TCP 66 747 > 2049 [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=21210557 TSecr=1818037730
6 2015-02-04 13:35:24.957866 192.168.122.5 -> 192.168.122.50 NFS 226 V4 Call GETATTR FH:0xc8972378
7 2015-02-04 13:35:24.958106 192.168.122.50 -> 192.168.122.5 TCP 66 2049 > 747 [ACK] Seq=1 Ack=161 Win=15552 Len=0 TSval=1818037731 TSecr=21210557
8 2015-02-04 13:35:24.959321 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 6) GETATTR
9 2015-02-04 13:35:24.959344 192.168.122.5 -> 192.168.122.50 TCP 66 747 > 2049 [ACK] Seq=161 Ack=225 Win=15680 Len=0 TSval=21210559 TSecr=1818037732
10 2015-02-04 13:35:24.966459 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call ACCESS FH:0xc8972378, [Check: RD LU MD XT DL]
11 2015-02-04 13:35:24.966761 192.168.122.50 -> 192.168.122.5 NFS 306 V4 Reply (Call In 10) ACCESS, [Allowed: RD LU MD XT DL]
12 2015-02-04 13:35:24.966856 192.168.122.5 -> 192.168.122.50 NFS 226 V4 Call GETATTR FH:0xc8972378
13 2015-02-04 13:35:24.967042 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 12) GETATTR
14 2015-02-04 13:35:24.967173 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0xbcebd514
15 2015-02-04 13:35:24.967345 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 14) GETATTR
16 2015-02-04 13:35:24.967498 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x75bf082c
17 2015-02-04 13:35:24.967759 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 16) GETATTR
18 2015-02-04 13:35:24.967877 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0xe2a20b7d
19 2015-02-04 13:35:24.968060 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 18) GETATTR
20 2015-02-04 13:35:24.968138 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x766b1bb5
21 2015-02-04 13:35:24.968297 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 20) GETATTR
22 2015-02-04 13:35:24.968403 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x60f96b17
23 2015-02-04 13:35:24.968578 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 22) GETATTR
24 2015-02-04 13:35:24.968700 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x2fc36e66
25 2015-02-04 13:35:24.968910 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 24) GETATTR
26 2015-02-04 13:35:24.968995 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x5e3dbcdd
27 2015-02-04 13:35:24.969142 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 26) GETATTR
28 2015-02-04 13:35:24.969257 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x9e47d443
29 2015-02-04 13:35:24.969502 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 28) GETATTR
30 2015-02-04 13:35:24.969590 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x50db6501
31 2015-02-04 13:35:24.969837 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 30) GETATTR
32 2015-02-04 13:35:24.969925 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x8ec4ca71
33 2015-02-04 13:35:24.970106 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 32) GETATTR
34 2015-02-04 13:35:24.970192 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0xdd9ab069
35 2015-02-04 13:35:24.970347 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 34) GETATTR
36 2015-02-04 13:35:24.970433 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x9dba001a
37 2015-02-04 13:35:24.970594 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 36) GETATTR
38 2015-02-04 13:35:24.970686 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x3dfa900f
39 2015-02-04 13:35:24.970898 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 38) GETATTR
40 2015-02-04 13:35:24.971008 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x14ef0cec
41 2015-02-04 13:35:24.971176 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 40) GETATTR
42 2015-02-04 13:35:24.971263 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x010cf990
43 2015-02-04 13:35:24.971436 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 42) GETATTR
44 2015-02-04 13:35:24.971517 192.168.122.5 -> 192.168.122.50 NFS 234 V4 Call GETATTR FH:0x40587b33
45 2015-02-04 13:35:24.971669 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 44) GETATTR
46 2015-02-04 13:35:25.011698 192.168.122.5 -> 192.168.122.50 TCP 66 747 > 2049 [ACK] Seq=3177 Ack=4273 Win=35008 Len=0 TSval=21210612 TSecr=1818037745
=== NFS server stops responding ===
47 2015-02-04 13:35:40.460949 192.168.122.5 -> 192.168.122.50 NFS 226 V4 Call GETATTR FH:0xc8972378
48 2015-02-04 13:35:40.661690 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #47][TCP Retransmission] V4 Call GETATTR FH:0xc8972378
49 2015-02-04 13:35:41.063678 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #47][TCP Retransmission] V4 Call GETATTR FH:0xc8972378
50 2015-02-04 13:35:41.867689 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #47][TCP Retransmission] V4 Call GETATTR FH:0xc8972378
51 2015-02-04 13:35:43.475706 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #47][TCP Retransmission] V4 Call GETATTR FH:0xc8972378
52 2015-02-04 13:35:46.691705 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #47][TCP Retransmission] V4 Call GETATTR FH:0xc8972378
53 2015-02-04 13:35:53.124069 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #47][TCP Retransmission] V4 Call GETATTR FH:0xc8972378
54 2015-02-04 13:36:05.987729 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #47][TCP Retransmission] V4 Call GETATTR FH:0xc8972378
57 2015-02-04 13:36:31.715700 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #47][TCP Retransmission] V4 Call GETATTR FH:0xc8972378
58 2015-02-04 13:36:40.460873 192.168.122.5 -> 192.168.122.50 TCP 66 747 > 2049 [FIN, ACK] Seq=3337 Ack=4273 Win=35008 Len=0 TSval=21286061 TSecr=1818037745
59 2015-02-04 13:36:55.460780 192.168.122.5 -> 192.168.122.50 TCP 66 747 > 2049 [RST, ACK] Seq=3338 Ack=4273 Win=35008 Len=0 TSval=21301061 TSecr=1818037745
60 2015-02-04 13:36:55.460856 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Port numbers reused] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21301061 TSecr=0 WS=1
61 2015-02-04 13:36:56.460714 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21302061 TSecr=0 WS=1
62 2015-02-04 13:36:58.460695 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21304061 TSecr=0 WS=1
65 2015-02-04 13:37:02.460736 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21308061 TSecr=0 WS=1
66 2015-02-04 13:37:10.460719 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21316061 TSecr=0 WS=1
67 2015-02-04 13:37:26.460702 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21332061 TSecr=0 WS=1
68 2015-02-04 13:38:01.460782 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Port numbers reused] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21367061 TSecr=0 WS=1
69 2015-02-04 13:38:02.460695 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21368061 TSecr=0 WS=1
70 2015-02-04 13:38:04.460705 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21370061 TSecr=0 WS=1
73 2015-02-04 13:38:08.460741 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21374061 TSecr=0 WS=1
74 2015-02-04 13:38:16.460731 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21382061 TSecr=0 WS=1
75 2015-02-04 13:38:32.460716 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 747 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=21398061 TSecr=0 WS=1
# grep "not responding" /var/log/messages | tail -1
Feb 4 13:39:04 rhel6u6-node1 kernel: [21730.061122] nfs: server rhel6u2-node2 not responding, still trying
Scenario 2, NFS client tcpdump: the NFS server responds with a TCP-level ACK, but not an NFS response
This scenario was simulated by causing the NFS server to drop NFS requests by use of the reproducer in RHEL6: NFS mounts become unresponsive when NFS server experiences repeated hardware.py python backtrace / abrt
This scenario narrows the problem to the NFS server side. Proceed to troubleshoot the NFS server.
This scenario is often easy to recognize by noting the absence of NFS-level responses from the NFS server as well as the repeated connection teardown (FIN) followed by re-establishment (SYN) cycle. If you see this in a tcpdump, go back and look for the presence of the TCP-level ACKs and if the server is acking the NFS client's data but not responding at the NFS level, your tcpdump matches this scenario.
In the example below, note the NFS client transmitting a GETATTR request in frame 6, and the NFS server is responding in frame 7 with a TCP-level ACK and the 'Ack=' number has increased in size. However, the NFS server never responds after this. After 2 minutes, in frame 8, the NFS client issues a FIN and then the following frames it re-establishes the TCP connection, then the sequence repeats. This shows the NFS server has received the NFS request at the TCP layer, but for some reason is not responding at the NFS layer.
$ tshark -ntad -r tcpdump.pcap.gz -R "tcp.port == 2049"
3 2015-02-04 14:26:43.255451 192.168.122.5 -> 192.168.122.50 TCP 74 976 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=24288854 TSecr=0 WS=64
4 2015-02-04 14:26:43.255681 192.168.122.50 -> 192.168.122.5 TCP 74 2049 > 976 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1821116029 TSecr=24288854 WS=64
5 2015-02-04 14:26:43.255738 192.168.122.5 -> 192.168.122.50 TCP 66 976 > 2049 [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=24288856 TSecr=1821116029
6 2015-02-04 14:26:43.255790 192.168.122.5 -> 192.168.122.50 NFS 226 V4 Call GETATTR FH:0xc8972378
7 2015-02-04 14:26:43.256103 192.168.122.50 -> 192.168.122.5 TCP 66 2049 > 976 [ACK] Seq=1 Ack=161 Win=15552 Len=0 TSval=1821116029 TSecr=24288856
8 2015-02-04 14:28:43.255817 192.168.122.5 -> 192.168.122.50 TCP 66 976 > 2049 [FIN, ACK] Seq=161 Ack=1 Win=14656 Len=0 TSval=24408856 TSecr=1821116029
9 2015-02-04 14:28:43.256266 192.168.122.50 -> 192.168.122.5 TCP 66 2049 > 976 [FIN, ACK] Seq=1 Ack=162 Win=15552 Len=0 TSval=1821236029 TSecr=24408856
10 2015-02-04 14:28:43.256313 192.168.122.5 -> 192.168.122.50 TCP 66 976 > 2049 [ACK] Seq=162 Ack=2 Win=14656 Len=0 TSval=24408856 TSecr=1821236029
11 2015-02-04 14:28:43.256413 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Port numbers reused] 976 > 2049 [SYN] Seq=0 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=24408856 TSecr=1821236029 WS=1
12 2015-02-04 14:28:43.256662 192.168.122.50 -> 192.168.122.5 TCP 74 2049 > 976 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1821236030 TSecr=24408856 WS=64
13 2015-02-04 14:28:43.256735 192.168.122.5 -> 192.168.122.50 TCP 66 976 > 2049 [ACK] Seq=1 Ack=1 Win=14480 Len=0 TSval=24408857 TSecr=1821236030
14 2015-02-04 14:28:43.256779 192.168.122.5 -> 192.168.122.50 NFS 226 V4 Call GETATTR FH:0xc8972378
15 2015-02-04 14:28:43.257012 192.168.122.50 -> 192.168.122.5 TCP 66 2049 > 976 [ACK] Seq=1 Ack=161 Win=15552 Len=0 TSval=1821236030 TSecr=24408857
16 2015-02-04 14:31:43.258919 192.168.122.5 -> 192.168.122.50 TCP 66 976 > 2049 [FIN, ACK] Seq=161 Ack=1 Win=14480 Len=0 TSval=24588859 TSecr=1821236030
17 2015-02-04 14:31:43.260224 192.168.122.50 -> 192.168.122.5 TCP 66 2049 > 976 [FIN, ACK] Seq=1 Ack=162 Win=15552 Len=0 TSval=1821416033 TSecr=24588859
18 2015-02-04 14:31:43.260265 192.168.122.5 -> 192.168.122.50 TCP 66 976 > 2049 [ACK] Seq=162 Ack=2 Win=14480 Len=0 TSval=24588860 TSecr=1821416033
19 2015-02-04 14:31:43.260381 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Port numbers reused] 976 > 2049 [SYN] Seq=0 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=24588860 TSecr=1821416033 WS=1
20 2015-02-04 14:31:43.263996 192.168.122.50 -> 192.168.122.5 TCP 74 2049 > 976 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1821416037 TSecr=24588860 WS=64
21 2015-02-04 14:31:43.264079 192.168.122.5 -> 192.168.122.50 TCP 66 976 > 2049 [ACK] Seq=1 Ack=1 Win=14480 Len=0 TSval=24588864 TSecr=1821416037
22 2015-02-04 14:31:43.264293 192.168.122.5 -> 192.168.122.50 NFS 226 V4 Call GETATTR FH:0xc8972378
23 2015-02-04 14:31:43.264472 192.168.122.50 -> 192.168.122.5 TCP 66 2049 > 976 [ACK] Seq=1 Ack=161 Win=15552 Len=0 TSval=1821416038 TSecr=24588864
# grep "not responding" /var/log/messages | tail -1
Feb 4 14:31:43 rhel6u6-node1 kernel: [24888.857184] nfs: server rhel6u2-node2 not responding, still trying
Scenario 3, NFS client tcpdump: the NFS server responds both with a TCP-level ACK and NFS reply, but NFS client re-sends the same NFS / RPC Call
This scenario was simulated by blocking the NFS port at the NFS client's input firewall.
This scenario narrows the problem to the receive side of the NFS client. Proceed to troubleshoot the NFS client.
Note the presence of "RPC duplicate" in the tcpdump, and the fact that the NFS client's tcpdump does show replies from the NFS server. However, the NFS client re-sends the same RPC Call, indicating it did not actually process the NFS server's reply.
$ tshark -ntd -r tcpdump.pcap.gz -R "tcp.port == 2049"
1 2015-02-04 14:44:59.870237 192.168.122.5 -> 192.168.122.50 NFS 226 V4 Call GETATTR FH:0xc8972378
2 2015-02-04 14:44:59.870623 192.168.122.50 -> 192.168.122.5 NFS 290 V4 Reply (Call In 1) GETATTR
3 2015-02-04 14:45:00.070693 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #1][TCP Retransmission] V4 Call (Reply In 2) GETATTR FH:0xc8972378
4 2015-02-04 14:45:00.070902 192.168.122.50 -> 192.168.122.5 TCP 78 [TCP Dup ACK 2#1] 2049 > 924 [ACK] Seq=225 Ack=161 Win=562 Len=0 TSval=1822212844 TSecr=25385671 SLE=1 SRE=161
5 2015-02-04 14:45:00.075265 192.168.122.50 -> 192.168.122.5 NFS 290 [RPC duplicate of #2][TCP Retransmission] V4 Reply (Call In 1) GETATTR
6 2015-02-04 14:45:00.472715 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #1][TCP Retransmission] V4 Call (Reply In 2) GETATTR FH:0xc8972378
7 2015-02-04 14:45:00.472998 192.168.122.50 -> 192.168.122.5 TCP 78 [TCP Dup ACK 5#1] 2049 > 924 [ACK] Seq=225 Ack=161 Win=562 Len=0 TSval=1822213246 TSecr=25386073 SLE=1 SRE=161
8 2015-02-04 14:45:00.485315 192.168.122.50 -> 192.168.122.5 NFS 290 [RPC duplicate of #2][TCP Retransmission] V4 Reply (Call In 1) GETATTR
9 2015-02-04 14:45:01.276738 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #1][TCP Retransmission] V4 Call (Reply In 2) GETATTR FH:0xc8972378
10 2015-02-04 14:45:01.277003 192.168.122.50 -> 192.168.122.5 TCP 78 [TCP Dup ACK 8#1] 2049 > 924 [ACK] Seq=225 Ack=161 Win=562 Len=0 TSval=1822214050 TSecr=25386877 SLE=1 SRE=161
11 2015-02-04 14:45:01.305342 192.168.122.50 -> 192.168.122.5 NFS 290 [RPC duplicate of #2][TCP Retransmission] V4 Reply (Call In 1) GETATTR
12 2015-02-04 14:45:02.884710 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #1][TCP Retransmission] V4 Call (Reply In 2) GETATTR FH:0xc8972378
13 2015-02-04 14:45:02.885002 192.168.122.50 -> 192.168.122.5 TCP 78 [TCP Dup ACK 11#1] 2049 > 924 [ACK] Seq=225 Ack=161 Win=562 Len=0 TSval=1822215658 TSecr=25388485 SLE=1 SRE=161
14 2015-02-04 14:45:02.945377 192.168.122.50 -> 192.168.122.5 NFS 290 [RPC duplicate of #2][TCP Retransmission] V4 Reply (Call In 1) GETATTR
17 2015-02-04 14:45:06.100714 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #1][TCP Retransmission] V4 Call (Reply In 2) GETATTR FH:0xc8972378
18 2015-02-04 14:45:06.101038 192.168.122.50 -> 192.168.122.5 TCP 78 [TCP Dup ACK 14#1] 2049 > 924 [ACK] Seq=225 Ack=161 Win=562 Len=0 TSval=1822218874 TSecr=25391701 SLE=1 SRE=161
19 2015-02-04 14:45:06.225432 192.168.122.50 -> 192.168.122.5 NFS 290 [RPC duplicate of #2][TCP Retransmission] V4 Reply (Call In 1) GETATTR
22 2015-02-04 14:45:12.532702 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #1][TCP Retransmission] V4 Call (Reply In 2) GETATTR FH:0xc8972378
23 2015-02-04 14:45:12.533003 192.168.122.50 -> 192.168.122.5 TCP 78 [TCP Dup ACK 19#1] 2049 > 924 [ACK] Seq=225 Ack=161 Win=562 Len=0 TSval=1822225306 TSecr=25398133 SLE=1 SRE=161
24 2015-02-04 14:45:12.785352 192.168.122.50 -> 192.168.122.5 NFS 290 [RPC duplicate of #2][TCP Retransmission] V4 Reply (Call In 1) GETATTR
25 2015-02-04 14:45:25.396744 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #1][TCP Retransmission] V4 Call (Reply In 2) GETATTR FH:0xc8972378
26 2015-02-04 14:45:25.397050 192.168.122.50 -> 192.168.122.5 TCP 78 [TCP Dup ACK 24#1] 2049 > 924 [ACK] Seq=225 Ack=161 Win=562 Len=0 TSval=1822238170 TSecr=25410997 SLE=1 SRE=161
27 2015-02-04 14:45:25.905318 192.168.122.50 -> 192.168.122.5 NFS 290 [RPC duplicate of #2][TCP Retransmission] V4 Reply (Call In 1) GETATTR
28 2015-02-04 14:45:51.124692 192.168.122.5 -> 192.168.122.50 NFS 226 [RPC retransmission of #1][TCP Retransmission] V4 Call (Reply In 2) GETATTR FH:0xc8972378
29 2015-02-04 14:45:51.124997 192.168.122.50 -> 192.168.122.5 TCP 78 [TCP Dup ACK 27#1] 2049 > 924 [ACK] Seq=225 Ack=161 Win=562 Len=0 TSval=1822263898 TSecr=25436725 SLE=1 SRE=161
30 2015-02-04 14:45:52.145363 192.168.122.50 -> 192.168.122.5 NFS 290 [RPC duplicate of #2][TCP Retransmission] V4 Reply (Call In 1) GETATTR
33 2015-02-04 14:45:59.869802 192.168.122.5 -> 192.168.122.50 TCP 66 924 > 2049 [FIN, ACK] Seq=161 Ack=1 Win=547 Len=0 TSval=25445470 TSecr=1822151195
34 2015-02-04 14:45:59.870178 192.168.122.50 -> 192.168.122.5 TCP 66 2049 > 924 [FIN, ACK] Seq=225 Ack=162 Win=562 Len=0 TSval=1822272643 TSecr=25445470
37 2015-02-04 14:46:14.869774 192.168.122.5 -> 192.168.122.50 TCP 66 924 > 2049 [RST, ACK] Seq=162 Ack=1 Win=547 Len=0 TSval=25460470 TSecr=1822151195
38 2015-02-04 14:46:14.869849 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Port numbers reused] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25460470 TSecr=0 WS=1
39 2015-02-04 14:46:14.870176 192.168.122.50 -> 192.168.122.5 TCP 74 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822287643 TSecr=25460470 WS=64
40 2015-02-04 14:46:15.869717 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25461470 TSecr=0 WS=1
41 2015-02-04 14:46:15.869958 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822288643 TSecr=25460470 WS=64
42 2015-02-04 14:46:17.869691 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25463470 TSecr=0 WS=1
43 2015-02-04 14:46:17.870089 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822290643 TSecr=25460470 WS=64
44 2015-02-04 14:46:18.269408 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822291043 TSecr=25460470 WS=64
45 2015-02-04 14:46:21.869690 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25467470 TSecr=0 WS=1
46 2015-02-04 14:46:21.869974 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822294643 TSecr=25460470 WS=64
47 2015-02-04 14:46:24.669376 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822297443 TSecr=25460470 WS=64
50 2015-02-04 14:46:29.869694 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25475470 TSecr=0 WS=1
51 2015-02-04 14:46:29.870074 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822302643 TSecr=25460470 WS=64
52 2015-02-04 14:46:37.469388 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822310243 TSecr=25460470 WS=64
53 2015-02-04 14:46:45.869741 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25491470 TSecr=0 WS=1
54 2015-02-04 14:46:45.870103 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822318643 TSecr=25460470 WS=64
57 2015-02-04 14:47:01.469363 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822334243 TSecr=25460470 WS=64
60 2015-02-04 14:47:20.869750 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Port numbers reused] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25526470 TSecr=0 WS=1
61 2015-02-04 14:47:20.870192 192.168.122.50 -> 192.168.122.5 TCP 54 2049 > 924 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
62 2015-02-04 14:47:21.869719 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25527470 TSecr=0 WS=1
63 2015-02-04 14:47:21.870135 192.168.122.50 -> 192.168.122.5 TCP 74 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822354643 TSecr=25527470 WS=64
64 2015-02-04 14:47:23.869698 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25529470 TSecr=0 WS=1
65 2015-02-04 14:47:23.870039 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822356643 TSecr=25527470 WS=64
66 2015-02-04 14:47:26.069306 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822358843 TSecr=25527470 WS=64
67 2015-02-04 14:47:27.869696 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25533470 TSecr=0 WS=1
68 2015-02-04 14:47:27.870041 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822360643 TSecr=25527470 WS=64
69 2015-02-04 14:47:32.469350 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822365243 TSecr=25527470 WS=64
70 2015-02-04 14:47:35.869703 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25541470 TSecr=0 WS=1
71 2015-02-04 14:47:35.870083 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822368643 TSecr=25527470 WS=64
74 2015-02-04 14:47:45.269493 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822378043 TSecr=25527470 WS=64
75 2015-02-04 14:47:51.869715 192.168.122.5 -> 192.168.122.50 TCP 74 [TCP Retransmission] 924 > 2049 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=25557470 TSecr=0 WS=1
76 2015-02-04 14:47:51.869971 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822384643 TSecr=25527470 WS=64
79 2015-02-04 14:48:09.269459 192.168.122.50 -> 192.168.122.5 TCP 74 [TCP Retransmission] 2049 > 924 [SYN, ACK] Seq=744558590 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=1822402043 TSecr=25527470 WS=64
# grep "not responding" /var/log/messages | tail -1
Feb 4 14:48:23 rhel6u6-node1 kernel: [25889.470095] nfs: server rhel6u2-node2 not responding, still trying
Product(s)
Category
Components
Tags
Article Type