Running a High-Performance Web Server for BSD

Edit the following two files:

/usr/include/sys/socket.h
/usr/src/sys/sys/socket.h

/*
 * Maximum queue length specifiable by listen.
 */
#define SOMAXCONN       5

After the edit, recompile the kernel and recompile the Apache server then reboot.

FreeBSD 2.1 seems to be perfectly happy, with SOMAXCONN set to 32 already.

Addendum for very heavily loaded BSD servers
from Chuck Murcko <chuck@telebase.com>

If you're running a really busy BSD Apache server, the following are useful things to do if the system is acting sluggish:

• Run vmstat to check memory usage, page/swap rates, etc.
• Run netstat -m to check mbuf usage
• Run fstat to check file descriptor usage

maxusers        256

• Maximum # of processes
• Maximum # of processes per user
• System wide open files limit
• Per-process open files limit
• Maximum # of mbuf clusters
• Proc/pgrp hash table size

# Network options. NMBCLUSTERS defines the number of mbuf clusters and
# defaults to 256. This machine is a server that handles lots of traffic,
# so we crank that value.
options         NMBCLUSTERS=4096        # mbuf clusters at 4096

#
# Misc. options
#
options         CHILD_MAX=512           # maximum number of child processes
options         OPEN_MAX=512            # maximum fds (breaks RPC svcs)

In many cases, NMBCLUSTERS must be set much larger than would appear necessary at first glance. The reason for this is that if the browser disconnects in mid-transfer, the socket fd associated with that particular connection ends up in the TIME_WAIT state for several minutes, during which time its mbufs are not yet freed. Another reason is that, on server timeouts, some connections end up in FIN_WAIT_2 state forever, because this state doesn't time out on the server, and the browser never sent a final FIN. For more details see the FIN_WAIT_2 page.

Some more info on mbuf clusters (from sys/mbuf.h):

/*
 * Mbufs are of a single size, MSIZE (machine/machparam.h), which
 * includes overhead.  An mbuf may add a single "mbuf cluster" of size
 * MCLBYTES (also in machine/machparam.h), which has no additional overhead
 * and is used instead of the internal data area; this is done when
 * at least MINCLSIZE of data must be stored.
 */

CHILD_MAX and OPEN_MAX are set to allow up to 512 child processes (different than the maximum value for processes per user ID) and file descriptors. These values may change for your particular configuration (a higher OPEN_MAX value if you've got modules or CGI scripts opening lots of connections or files). If you've got a lot of other activity besides httpd on the same machine, you'll have to set NPROC higher still. In this example, the NPROC value derived from maxusers proved sufficient for our load.

To increase the size of the listen() queue, you need to adjust the value of SOMAXCONN. SOMAXCONN is not derived from maxusers, so you'll always need to increase that yourself. We use a value guaranteed to be larger than Apache's default for the listen() of 128, currently. The actual value for SOMAXCONN is set in sys/socket.h. The best way to adjust this parameter is run-time, rather than changing it in this header file and thus hardcoding a value in the kernel and elsewhere. To do this, edit /etc/rc.local and add the following line:

/usr/sbin/sysctl -w kern.somaxconn=256

We used 256 but you can tune it for your own setup. In many cases, however, even the default value of 128 (for later versions of FreeBSD) is OK.

Caveats

Be aware that your system may not boot with a kernel that is configured to use more resources than you have available system RAM. ALWAYS have a known bootable kernel available when tuning your system this way, and use the system tools beforehand to learn if you need to buy more memory before tuning.

RPC services will fail when the value of OPEN_MAX is larger than 256. This is a function of the original implementations of the RPC library, which used a byte value for holding file descriptors. BSDI has partially addressed this limit in its 2.1 release, but a real fix may well await the redesign of RPC itself.

Finally, there's the hard limit of child processes configured in Apache.

For versions of Apache later than 1.0.5 you'll need to change the definition for HARD_SERVER_LIMIT in httpd.h and recompile if you need to run more than the default 150 instances of httpd.

From conf/httpd.conf-dist:

# Limit on total number of servers running, i.e., limit on the number
# of clients who can simultaneously connect --- if this limit is ever
# reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.
# It is intended mainly as a brake to keep a runaway server from taking
# Unix with it as it spirals down...

MaxClients 150

Thanks to Tony Sanders and Chris Torek at BSDI for their helpful suggestions and information.

"M. Teterin" <mi@ALDAN.ziplink.net> writes:

It really does help if your kernel and frequently used utilities are fully optimized. Rebuilding the FreeBSD kernel on an AMD-133 (486-class CPU) web-server with
-m486 -fexpensive-optimizations -fomit-frame-pointer -O2
helped reduce the number of "unable" errors, because the CPU was often maxed out.

More welcome!

Apache HTTP Server Version 1.3

Index Home