RHEL NFS Client mounting QNAP NFS export: "Too many levels of symbolic links" / "directory [...] contains a readdir loop."

Environment

• Red Hat Enterprise Linux 6.7 (kernel 2.6.32-573.8.1.el6.x86_64) (NFS Client)
• QNAP TS-269 Pro NFS Server, firmware 4.0.7 (Build 20140410)
• NFS

Issue

• While running rsync to a NFS share, we are experiencing a error/warning message (on the NFS client).

cp: cannot access `X/Y': Too many levels of symbolic links

• NFS: directory X/Y contains a readdir loop.Please contact your server vendor. The file: Z has duplicate cookie 1845644159

Resolution

• Upgrading QNAP NFS Server firmware to version 4.2.0 (Build 20151118) solved the issue.

Root Cause

• Bug in QNAP NFS Server firmware

Diagnostic Steps

• Wait for the issue to reoccur and grab the following data by reproducing the issue with the ls command:
  On the client execute the command:
  [a] Open a terminal and run the tcpdump as shown below:

# tcpdump -s0 -i <interface> host <SERVER> -w /tmp/nfs-client.pcap 

Where: Replace with actual interface name i.e eth0, eth1 or bond* which is used for communication between NFS client and NFS server.

[b] Once the above has started, start NFS debugging on the client and strace the ls command to reproduce the issue.
Enable the nfs and rpc debug logging in the nfs client where we are facing the issue, using the below command.

# rpcdebug -m nfs -s all 
# rpcdebug -m rpc -s all 
# strace -T -tt -f -v -q -s 4096 -o /tmp/strace.out <ls command and args> 

[c] Now on another terminal reproduce the issue i.e. execute rsync to NFS share and try accessing the share and listing it.
For instance try accessing the share and listing it:

$ cd /backup/example
$ ls -l | wc -l
ls: reading directory .: Too many levels of symbolic links

• Once the error Too many levels of symbolic links is received,press CTRL+C on the terminal (i) where tcpdump and strace is running.
  Note: The debuging needs to be disabled once the error is encountered using the below syntax:

# rpcdebug -m nfs -c all 
# rpcdebug -m rpc -c all 

• The debugging log would help us identify the file that is affected. All this data needs to be captured during the time issue is reported.
  Upload the following files in the case for our analysis.

* /tmp/nfs-client.pcap 
* /tmp/strace.out 
* /var/log/messages 

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.