NcFTPd : Frequently Asked Questions : Troubleshooting

 

How do I put NcFTPd into verbose mode so it generates debugging output in the logs?

There are different ways to enable verbose logging depending on the version of the software.

Verbose logging is now enabled, and additional diagnostic information is being written to the NcFTPd misc logs. If you forgot where the misc logs are at, look in the general.cf configuration file for the log-misc option.

How can I use verbose logging mode to diagnose a problem?

By default, NcFTPd doesnt log anything except critical errors to the NcFTPd misc logs (as specified in the general.cf for the log-misc option; usually /var/log/ncftpd/misc.*).  There are also xfer and session logs, but those are intended for accounting rather than problem-solving.  By enabling verbose mode, you can see the exactly how client programs are communicating with NcFTPd since you see each line of communication between client and server.

For example, this sample shows why a users uploads were failing:

Unfortunately, like many of todays client programs, this particular program failed to recognize the numerous errors that had occurred and proclaimed success. 

Users are unable to login.  How do I diagnose the problem?

First, put NcFTPd into both verbose mode and user authentication debug mode and then try a login, using the regular ftp client on the same machine as the server processes. The resulting misc log entries for the user session should hopefully pinpoint the problem for you.

Often the problem has nothing to do with the login process.  Some, but not all, common problems include:

If you think the problem is that NcFTPd isnt validating the users password correctly, you can try using the tspwd diagnostic utility, which is in the extra directory of the NcFTPd tarball that you downloaded.  As superuser, run it as ./tspwd l the-user-name, then try to validate the password.  The utility dumps lots of debugging output which is helpful for locating the problem; send the output of the utility to tech support if you suspect a problem with NcFTPd rather than this particular user. 

This example shows how an administrator used tspwd to determine why joeuser could not login:

bash# ./tspwd -l joeuser
SpwdLibraryInit
-------------------------------------------------------------------------------
This test is used to verify that:
(a) The encrypted passwords can be accessed; and
(b) Passwords can be authenticated.

crypt = [pYw5pQf1/i11w]
encrypted guess == encrypted answer
DES crypt works.
GetSpwdName name=joeuser wantC=1 wantSupp=1 wantClose=1
InitSpwdStruct
PwdToSpwd pw->u=joeuser wantC=1 wantSupp=1
DES crypt password detected.
DES crypt password not long enough.
PwdToSpwd: pw->pw = [x] len=1
getspnam
PwdToSpwd: Getspnam's spwptr->sp_pwdp was [*], alg=0
        PwdToSpwd: Pp->username = [joeuser]
        PwdToSpwd: Pp->passwd = [*], alg = 0
        PwdToSpwd: Pp->uid = 6
GetUserGroups maxg=32 u=joeuser
        PwdToSpwd: Pp->gid #0 = 0
        PwdToSpwd: Pp->homedir = [/sbin]
        PwdToSpwd: Pp->shell = [/sbin/joeuser]
        Endspent
                endspent
GetSpwdByName rc=0
Password for joeuser:
crypt = [**kKHnP/IhsG.]
encrypted guess did not equal encrypted answer
Password was NOT authenticated.
This is not surprising since it appears that the encrypted password was not 
looked up correctly.

The answer is that the encrypted password that NcFTPd read was * which means that the entry in the shadow password file was invalid.  (Also note that the regular /etc/passwd file entry had its entry for the encrypted password field as an x, so NcFTPd then consulted the shadow password file.)

The * notation in the shadow or passwd file is often used to denote that the users account is disabled, since the old DES crypt algorithm could never produce an encrypted password containing a * character.  The administrator fixed this problem by setting the password for joeuser manually, i.e., as root, passwd joeuser.

Some of the entries in the xfer logs are denoted INCOMPLETE.  What does this mean?

The INCOMPLETE keyword denotes a transfer that definitely did not finish (for example, NcFTPd received a disconnection notification after only sending 30,000 bytes of a 200,000 byte download) or that the file was reputedly transferred in full, but the remote FTP client did not acknowledge that the transfer concluded successfully.

The first case is a simpler problem to understand; it indicates that the remote user disconnected before the transfer completed, or that the connection to the user was lost due to network problems.

The second case is a bit more difficult to explain. By default, NcFTPd requires implicit acknowledgment of downloads (and uploads) before creating a log entry. The remote user implicitly acknowledges a successful download by doing something else in the FTP session, such as changing to another directory or downloading another file, or even issuing a QUIT command to close the session. This allows NcFTPd to differentiate between transfers which were definitely received in entirety by the user, and those where NcFTPd sent the entire data file to the remote client's machine but the FTP client program closes the session abruptly. Such behavior often indicates that the data was sent to the remote client machine's TCP layer, but did not reach the FTP client program because it exited. The end result is that if NcFTPd does not receive an implicit acknowledgment (and the requirement for implicit acknowledgment has not been disabled), then NcFTPd must consider the transfer status as INCOMPLETE.

You may see higher than expected numbers of files marked as INCOMPLETE. For the first case, it indicates a problem with your network connectivity, or more likely, that your remote FTP client programs are using so-called download accelerators. These programs open multiple sessions to the server, and download chunks of the file in parallel. For example, if the user opens 10 simultaneous sessions, you will see only one of these sessions marked as OK, while the other 9 will be marked as INCOMPLETE. You can restrict users to one session using the max-domain-users-per-ip option.

For the second case, you may also see elevated numbers of INCOMPLETE transfers. Regrettably, there are many poorly-written FTP client implementations out there which do not follow the requirements dictated by the FTP protocol. These noncompliant client programs violate protocol closing the session without sending a QUIT command. Unfortunately, when the client closes the session after a successful transfer without sending a QUIT command, NcFTPd has to conclude that the transfer is incomplete. Internet Explorer is an example of a program which does not send the QUIT command.

Im getting permission denied when I try to transfer a file.  I have checked the permissions and everything appears OK.  What could be wrong?

You will also get permission denied for many other errors.  For example, you can't overwrite an existing file in /incoming, filenames must also not contain any illegal characters, etc.

If you really want to know the exact reason, you may be able to find out by putting NcFTPd into verbose mode, repeating the steps you did to produce the problem, and then checking the log output of the session in the NcFTPd misc logs.

NcFTPd takes a very long time to startup!  Specifically, it seems to hang while "Preloading users and groups"

The reason for pre-loading users and groups for printing the user and group names in ls output.  Thats it.  If you dont care about that (and it really doesnt matter much in the days where GUI client programs dont show it anyway), you can disable that by setting both a-ls-names and u-ls-names to no in the general.cf (and restart NcFTPd).

The reason it is taking so long for you is that your site is apparently using NIS (or perhaps LDAP), and it takes time to read each user record from your NIS server.

I want to use TCP Wrappers access control with NcFTPd so I tried to run NcFTPd from the inetd.conf file.  It doesnt work.

NcFTPd does not run from inetd.  Period.

It does support TCP Wrappers, however.  It just reads the same files that tcpd does.  Install NcFTPd like you normally would (i.e. not in inetd.conf) and see the documentation for the tcp-wrappers option.  

I'm getting poor performance when doing large numbers of small transfers.

There is a feature which sleeps a few seconds between batches of FTP command primitives to prevent people from trying to DoS the server.  You can tune it, however, by changing the value of the throttle.  If you're experiencing the throttle, most likely you're working over a fast network and/or doing large numbers of small transfers.  External users are most likely not going to be throttled.

I'm getting "respawning too fast" errors.  What does this mean?

The purpose of the entry in the /etc/inittab is to automatically re-run NcFTPd in case it exits.  For example, the standard procedure for restarting NcFTPd is to kill the processes and let "init" respawn a new batch.

The problem is that your inittab entry is repeatedly exiting/respawning, which means that NcFTPd is failing to start up and is immediately exiting.  Typically that means you already had an instance of NcFTPd running when you added the inittab entry, or there's a fatal error being logged by NcFTPd.

First, disable your inittab entry by editing /etc/inittab and commenting out the line that runs NcFTPd.  You do that by prefixing the line with a # character, followed by running "init q" to tell "init" to reload the inittab.

Next use "ps" to list all the processes running on the system (usually "ps -ef" or "ps aux").  If you see any "ncftpd" processes running, that was the problem.  Kill those processes and re-enable the inittab entry (uncomment, and "init q" which will start up NcFTPd again) and you're finished.

Otherwise, from a shell prompt, run NcFTPd manually using the same command line you're using in the inittab, except remove any occurances of "-q" and make sure you add a "-v" if necessary. For example, your command line at the shell prompt might be something like "/usr/local/etc/ncftpd/ncftpd -v /usr/local/etc/ncftpd/general.cf /usr/local/etc/ncftpd/domain.cf".  Since the problem wasn't that NcFTPd had already been running, you should see an error logged to the screen (and also the "misc" logs).  Fix the problem and then re-enable your inittab entry.

Directory listings fail with Only ls flags accepted are -C, -F, -l, -t, -r, -1, -S, -L, -R, and p.

The FTP client program is using flags to ls that NcFTPd doesnt understand or support.  Most often this is because youre using NcFTP client version prior to version 3, such as version 2.4.3.  You can fix this by upgrading to NcFTP (client) version 3.0.

I noticed that NcFTPd logged the error shared memory pool [key 0x77c0ffee] already exists!  Removed it.  What does this mean?

It almost always means that you forcefully terminated NcFTPd the time before, using kill 9.  Always do a proper shutdown of NcFTPd (i.e. use kill 15 instead) so that NcFTPd can clean up after itself before it exits.

One side effect of a forceful shutdown is that NcFTPds shared memory pool persists.  This error message means that NcFTPd removed the old shared memory pool and created a new one, but other than that, it is harmless.

Im running NcFTPd on Solaris and it doesnt startup when I have a large number of users (> 300).  I also see shared memory pool creation failed: Invalid argument in the logs.  

Usually this means that the system is not configured to support a large enough shared memory pool, and usually this problem comes up when youre running Solaris.  The problem is that the system by default places a limit on the maximum size of a shared memory pool, and in this case, this maximum is set too conservatively. When your max-users is around 306, on Solaris that causes the size of the shared-memory pool to exceed 1 megabyte, which is the default limit for Solaris.  

To fix this problem on Solaris, you need to tune your system by changing (or adding, if not present) a setting to your /etc/system file.  Add "set shmsys:shminfo_shmmax=10485760" to /etc/system and reboot the system.  This will cause Solaris to use 10 megabytes as it's limit instead of 1 megabyte, and will increase the limit of users to about 3000.  Note that NcFTPd itself has no such user limitation, it is only limited by the operating system.  

When I try to startup NcFTPd I get the error set-name variable on line 39 of /etc/ncftpd/general.cf needs to be set in individual domain configuration files.

You need to run it as ncftpd /etc/ncftpd/general.cf /etc/ncftpd/domain.cf rather than  ncftpd /etc/ncftpd/domain.cf /etc/ncftpd/general.cf.  In other words, youve reversed the order of the parameters on the command line.

I try to run NcFTPd but it just exits after a few seconds.

NcFTPd dumps errors to stderr until the logging process is started up, and at that point, all other errors are logged to the misc logs as specified in the general.cf.  When you startup NcFTPd you may want to redirect stderr into a file, such as ncftpd 2> startup.errors.  If that file is empty after it exits, then you need to look in the misc logs.  Also, until you've solved this problem, when you start it up only use the "-v" flag and not "-q" nor "-q -v".

What do "bind failed: Address in use" errors mean when I start the server?

There is already a process that is waiting for FTP connections. Most likely that is /etc/inetd, and you can fix that by commenting out the ftp service line in /etc/inetd.conf and restarting /etc/inetd. It could also mean that there is another instance of NcFTPd running already. Kill that one off first.

Trying to connect to the server errs out with "connection refused."

That means no process is currently listening for the connection; make sure NcFTPd is running.

Be sure NcFTPd isn't erring out when it starts up and is prematurely exiting.  To do that, from a shell prompt, run NcFTPd manually using the same command line you're using in the inittab, except remove any occurances of "-q" and make sure you add a "-v" if necessary. For example, your command line at the shell prompt might be something like "/usr/local/etc/ncftpd/ncftpd -v /usr/local/etc/ncftpd/general.cf /usr/local/etc/ncftpd/domain.cf".  If you see see an error logged to the screen (and also the "misc" logs), fix the problem then start NcFTPd (if you're using the /etc/inittab, all you should need to do is "init q" to start it up).

Does NcFTPd work with NFS-mounted filesystems?  Users get kicked out immediately after logging in.

Yes, but for home directories on an NFS mount, the FTP server machine needs to have root privileges translate accordingly on the NFS server.  The reason is that NcFTPd runs as root, but often the default for NFS mounts is to translate root to nobody when accessing the NFS mounted directory.  That causes NcFTPd to fail because it tries to access the users home directory while running as root, and the privileges have been mapped to nobody which does not have permission to access the home directory.

If you really want to allow FTP over an NFS mounted filesystem (which is bad news for performance anyway), you can change the NFS server so that the root user on the FTP server is trusted for the home directory export on the NFS server (i.e. not changed to nobody when the FTP server machine accesses the share).

Usually that involves changing the exports file and adding the appropriate option.  Try the manual pages for exports, dfstab, share, nfsd, etc to see how to do that, since the exact process varies per platform.

Users can login, but directory listings and file transfers fail.

More than likely, you have a firewall or NAT (network address translation) device which isn't handling FTP correctly.  FTP uses separate connections for the protocol's conversation channel between the client and server (called the control connection) and additional connections for data transfers (FTP directory listings are also data transfers).

The problem is that the data transfer connections use port numbers and IP addresses which are negotiated by the client and server on-the-fly.  Since they do not use pre-defined port numbers or addresses for these connections, firewalls and NAT devices must take special care to handle these connections.

If your timeouts are occurring while the FTP client is using "passive" (PASV) mode data connections, then there is a firewall on the sever side which is the problem.  While that's a problem, at least it's something on your side which you can attempt to fix.  Odds are your firewall is configured too strictly with respect to incoming data connections, or it isn't sending the data connections to the same server machine where the control connection terminates.

If your timeouts occur when the FTP client uses "active" (PORT) mode data connections, then the problem is with a firewall on the client side.  Unfortunately, you can't do anything about this since you cannot configure the firewall on the client's side, but you can ask your clients to have their FTP client program use "passive" (PASV) mode instead.

Please see our article titled "The FTP and Your Firewall / NAT / Load-Balancing Router" for more details on the problems and what you can do about them.

Established connections which have been alive for a long time are timing-out before they complete.  Could my firewall be to blame?

Firewalls like to violate protocol and just discard connections when possible to facilitate network address translation. For example, if you've got 10 machines behind the firewall and each machine has 10 connections open through the firewall and to the outside world, that's 100 connections that the firewall has to manage. The absolute limit usually 65535 connections (for IPv4), so you can see how as the number of client machines increases how the firewall machine might get closer to that limit.

So, firewalls cheat a little and re-use connections a little sooner when they think it's possible. Instead of waiting for the connection to complete according to TCP between the client machine and the remote machine, they just discard it and mark that connection as available for re-use to some other client machine. That's why you're getting the hanging, because the connection hasn't been closed.

For FTP, there's two connections. The control connection acts as the conversation channel where the client machine communicates with the server on what the client wants to do on the server. The data connections are separate connections when the client chooses to transfer a file. When a data connection is initiated, that means the control connection is going to idle until the data connection is finished (i.e. idle until the server machine comes back with "226 Transfer completed.").

Unfortunately, if the data transfer lasts a long time, that means the firewall might think the control connection is no longer being used and it discards it. That means to work-around this you'll need to tune your firewall so it's idle timeout is longer. The problem is you don't want to make that too long so the firewall can't do network address translation effectively, yet you don't want it too short so a majority of FTP data connections leave the FTP control connection hung.

Similarly, your data connection could be discarded if the firewall is configured to always terminate connections who exceed a configured lifetime.  The firewall might also drop the "least important" connection if it has to make room for a new connection; this might mean that the oldest active connection might be terminated to make room for the new one.

 

Up
 NcFTPd Home