Troubleshooting NFS

General Recommendations

NFS RPC daemons should be started from the boot scripts and not from the inetd, due to the overhead of the spawning process from the inetd server. Another problem would be when two mount daemons, one started in the boot scripts and one configured into inetd.conf, compete for the same RPC service number.
Symbolic links could cause problems when exporting and mounting file systems. When exporting a symbolic link you must be certain the file system pointed to by the link is exported and when mounting a file system on a symbolic link you must remember that the actual mount occurs on the directory pointed to by the link.

Exporting

Use the ro (read-only) option unless clients must write to file.
In secure installations, set anon to -1 to disable root access on any client, except those specified in the root option.

Mounting

Use conventional mounting for clients that are inoperable without NFS directories (such as diskless or dataless clients) and for directories frequently used.
If directories are mounted with rw (read-write) option or if they are executable files they should be mounted with hard option. If the NFS server fails while a directory is hard-mounted, the client keeps trying to complete the current NFS operation until the server responds.
The intr option is recommended when using a hard mount. It allows the user to break transmission attempts if the server becomes unavailable for an extended period of time.
The bg option should be used to expedite the boot process if a server is unavailable when the client is booting. In other words, a client hangs until the server comes back up unless bg option is specified.
Do not put NFS mount points in the root '/' directory of a client. Mount points in a root directory can slow the performance of the client and can cause the client to be unavailable when the server becomes unavailable.

Automounting

Use the automounter when the overhead of a mount operation is not important, when a file system is used more often than the automount time limit, or when the file system is used infrequently.
Plan and test maps on a small group of clients before using them for larger groups.
Use indirect maps whenever possible. With indirect maps, mounts occur when a process references a subdirectory of the daemon or map mount point. With direct maps, when a process reads a directory containing one or more direct mount points, all of the file systems are mounted at the mount points, so the system overhead is increased.
Use direct map when directories cannot be grouped, but must be distributed throughout the local file system.
Avoid mounting direct map mount points into routinely accessed directories. This can cause unexpected mount activity and slow down system performance.

Common NFS Errors

"No such host" - Name of the server is specified incorrectly
"No such file or directory" - Either the local or remote file system is specified incorrectly.
"No such device" - NFS is not configured into the client's kernel.
"NFS server is not responding" message followed by "NFS server xxxx OK" - Server is heavily loaded causing RPC timeouts, or server has crashed.
"Stale file handle" - The file is no longer available.
"MOUNT_PROG not registered" - rpc.mountd daemon never started up and registered.
"Too many levels of remote in path" - Attempting to mount a file system which is already an NFS mounted file system.
"Permission denied" - Accessing as root on the client and root is mapped to nobody. Or the user on the client does not have corresponding UID on the server.
"No space" - The server is out of space on the file system.

Troubleshooting

The showmount command may be used to display server-side mount information. The option -a displays all remote mounts showing the name of the client and the directory, separated by a colon. The -d option displays only the names of the directories mounted by the clients. And the -e option displays the list of file systems exported by the server.

     # showmount -a
     All mount points on local host:
     edcert20.ucs.indiana.edu:/home
     edcert21.ucs.indiana.edu:/usr/local

     # showmount -d
     Directories on local host:
     /home
     /usr/local

     # showmount -e
     Export list on local host
     /home           edcert21.ucs.indiana.edu edcert20.ucs.indiana.edu     
     /usr/local      edcert21.ucs.indiana.edu

The df command may be used to display information on the file systems mounted remotely, the mount point and the amount of available space. The -F option may be specified to list only a specified file system type.

     # df -F nfs
     Filesystem                      Type  blocks     use   avail %use  Mounted on
     edcert21.ucs.indiana.edu:/home  nfs    68510   55804   12706  81%  /usr/share/help

BSD systems use -t option to specify the fstype. The output from the df command also varies among the different operating systems.

df also resolves the symbolic links and determines the file system mounted at the link's target. For example:

     # ls -l /usr/local/man
     lrwxr-xr-x    1 root     sys            6 Mar 17  1995 /usr/local/man -> catman/
     # df /usr/local/man
     Filesystem              Type  blocks     use   avail %use  Mounted on
     orange:/usr/local        nfs   68510   55804   12706  81%  /usr/local

Use the command nfsstat -s to display NFS activity on the server side. For example:

     # nfsstat -s

     Server RPC:
     calls      badcalls   nullrecv   badlen     xdrcall    duphits    dupage
     50852      0          0          0          0          0          0.00    

     Server NFS:
     calls        badcalls     
     50852        0            
     null         getattr      setattr      root         lookup       readlink  
     1  0%        233  0%      0  0%        0  0%        1041  2%     0  0%  
     read         wrcache      write        create       remove       rename    
     49498 97%    0  0%        0  0%        0  0%        0  0%        0  0% 
     link         symlink      mkdir        rmdir        readdir      fsstat 
     0  0%        0  0%        0  0%        0  0%        75  0%       4  0%

The output may be interpreted using the following guidelines.

badcalls > 0 - RPC requests are being rejected by the server. This could indicate authentication problems caused by having a user in too many groups, attempts to access exported file systems as root, or an improper Secure RPC configuration.
nullrecv > 0 - NFS requests are not arriving fast enough to keep all of the nfsd daemons busy. Reduce the number of NFS server daemons until nullrecv is not incremented.
symlink > 10% - Clients are making excessive use of symbolic links that are on file systems exported by the server. Replace the symbolic link with a directory, and mount both the underlying file system and the link's target on the client.
getattr > 60% - Check for non-default attribute caching (noac mount option) on NFS clients.

On the client side use the command nfsstat -c to display the client statistics. For example:

     # nfsstat -c

     Client RPC:
     calls      badcalls   retrans    badxid     timeout    wait       newcred
     369003     62         1998       43         2053       0          0 

     Client NFS:
     calls        badcalls     nclget       nclsleep     
     368948       0            368948       0            
     null         getattr      setattr      root         lookup       readlink  
     0  0%        51732 14%    680  0%      0  0%        95069 25%    542  0% 
     read         wrcache      write        create       remove       rename 
     210187 56%   0  0%        2259  0%     1117  0%     805  0%      337  0%   
     link         symlink      mkdir        rmdir        readdir      fsstat    
     120  0%      0  0%        7  0%        0  0%        5510  1%     583  0%

This output may be interpreted using the guidelines given below.

timeout > 5% - The client's RPC requests are timing out before the server can answer them, or the requests are not reaching the server. Check badxid to determine the problem.
badxid ~ timeout - RPC requests are being handled by the server, but too slowly. Increase timeo parameter value for this mount, or tune the server to reduce the average request service time.
badxid ~ 0 - With timeouts greater than 3%, this indicates that packets to and from the server are getting lost on the network. Reduce the read and write block sizes (mount parameters rsize and wsize) for this mount.
badxid > 0 - RPC calls on soft-mounted file systems are timing out. If the server is running, and badcalls is growing, then soft mounted file systems should use a larger timeo or retrans value.