After an NFS client's TCP connection goes idle, or disconnects due to a network problem, does it reconnect using the same source port or a different port?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 10
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • NFS client (v3 or v4)

Issue

  • I've noticed the Linux NFS client using TCP may drop a connection due to an NFS mount not being accessed for a while (5 minute idle timer), or if certain errors occur on NFSv4 mounts. When it becomes active again and the TCP connection is re-established, will the NFS client re-use the same source TCP port or a different one?
  • The NFS client may lose a connection to the NFS server during a network or server failure. When it becomes active again and the TCP connection is re-established, will the NFS client re-use the same source TCP port or a different one?

Resolution

  • NOTE: If the NFS client drops a TCP connection as a result of an umount request, a different source port will always be used for a subsequent mount. The below table only applies to an existing NFS mount that either goes idle (5-minute idle timer), or goes through a TCP teardown / reconnect sequence due to an error.

  • NOTE: In all instances, the NFS client should be re-using the same port in order to not break NFS server duplicate reply cache (DRC) semantics. This applies to NFS versions 4.0 and below. The DRC will avoid that calls get processed a second time, instead using the reply to the first call for all subsequent calls. For instance, processing a WRITE call a second time may corrupt data on disk, which may cause malfunctions on the applications. The DRC uses the client port as part of the identification of a client, so changing the client port may cause data corruptions.

  • Whether the NFS client reconnects the TCP connection with the same source port or not depends on the kernel version, as some kernel versions had certain bugs where a different source port was used. The behavior has changed with NFSv4.1 and above from RHEL 8.6 or higher, due to the way NFS sessions are maintained, while reconnecting. These would use a different source port, during reconnection.

  • The following table lists the RHEL kernel versions / minor releases where the NFS client reconnects with the same or different source port:

    RHEL Minor Version | Kernel version(s) | Same / Different TCP source port |
    -|-|-
    6.0 | 2.6.32-71.*.el6 | Same
    6.1 | 2.6.32-131.*el6 | Same
    6.2 - 6.8 | 2.6.32-220.*el6 - 2.6.32-642.*.el6 | Different
    6.9 | 2.6.32-696.*el6 or above | Same
    7.0 - 7.2 | 3.10.0-123*.el7 - 3.10.0-327*.el7 | Different
    7.3 | 3.10.0-514*.el7 or above | Same
    8.0 | 4.18.0-80*el8 or above | Same
    8.6 | 4.18.0-372*el8 or above | NFS v3 & v4.0 - Same; NFS v4.1 & v4.2 - Different
    9.0 | 5.14.0-70*el9 or above | NFS v3 & v4.0 - Same; NFS v4.1 & v4.2 - Different
    10.0 | 6.12.0-55.9.1.el10_0 or above | NFS v3 & v4.0 - Same; NFS v4.1 & v4.2 - Different

Root Cause

Red Hat Enterprise Linux 10

  • In all RHEL 10 releases, NFS v4.1 and v4.2 mounts will use a different source port number when reconnecting. This is only possible for NFS v4.1+, due to the way that NFS sessions are maintained; NFS v3 and v4.0 mounts do not maintain session in the same way, so must still use the same source port number.

Red Hat Enterprise Linux 9

  • In all RHEL 9 releases, NFS v4.1 and v4.2 mounts will use a different source port number when reconnecting. This is only possible for NFS v4.1+, due to the way that NFS sessions are maintained; NFS v3 and v4.0 mounts do not maintain session in the same way, so must still use the same source port number.

Red Hat Enterprise Linux 8

  • Beginning with RHEL 8.6, NFS v4.1 and v4.2 mounts will use a different source port number when reconnecting. This is only possible for NFS v4.1+, due to the way that NFS sessions are maintained; NFS v3 and v4.0 mounts do not maintain session in the same way, so must still use the same source port number.

Red Hat Enterprise Linux 7

  • The first few minor versions of RHEL7 contained the older behavior of the NFS client, where the source port was different upon reconnect. After RHEL7.3 a series of patches was introduced to re-use the source port after reconnect of an existing mount as described in the following bug:
    • Bug 1270038 - SUNRPC: add SO_REUSEPORT socket option for TCP connections

Red Hat Enterprise Linux 6

  • Originally, in RHEL6.0 GA and RHEL6.1 GA, the NFS client used the same source port. In RHEL6.2-6.8 kernels, this changed to use a different source port. This remained in place through the RHEL6.8 kernels, but changed back to the same port in RHEL6.9 kernels.
  • The following private Red Hat bug changed the port re-use logic in RHEL6.9 kernels 2.6.32-696.*el6 or above back to using the same source port
    • Bug 1321366 - SUNRPC: add SO_REUSEPORT socket option for TCP connections

Diagnostic Steps

  • Mount a NFS share, monitor the source port, and wait for the connection to disappear. In this case, the source port is 740:

      # mount -o vers=3 nfs-server:/exports/test /mnt/test

  # while true; do netstat -an | grep 2049; grep /mnt/test /proc/mounts; sleep 30; done
  tcp        0      0 192.168.122.18:740          192.168.122.35:2049         TIME_WAIT   
  nfs-server:/exports/test /mnt/test nfs rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.122.35,mountvers=3,mountport=48247,mountproto=udp,local_lock=none,addr=192.168.122.35 0 0
  tcp        0      0 192.168.122.18:740          192.168.122.35:2049         TIME_WAIT   
  nfs-server:/exports/test /mnt/test nfs rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.122.35,mountvers=3,mountport=48247,mountproto=udp,local_lock=none,addr=192.168.122.35 0 0
  nfs-server:/exports/test /mnt/test nfs rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.122.35,mountvers=3,mountport=48247,mountproto=udp,local_lock=none,addr=192.168.122.35 0 0

  • Access the share again, and look at the source port. In this instance, the source port of 740 is the same after the TCP connection is re-established, so the source port is the same:

      # ls /mnt/test

  # netstat -an | grep 2049; grep /mnt/test /proc/mounts
  tcp        0      0 192.168.122.18:740          192.168.122.35:2049         ESTABLISHED 
  nfs-server:/exports/test /mnt/test nfs rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.122.35,mountvers=3,mountport=48247,mountproto=udp,local_lock=none,addr=192.168.122.35 0 0
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.