Server operation

his chapter describes the basic operation of the server. The server accepts a client request and returns a response, which can range from a simple error diagnostic to a sizeable document. The server operation is governed by server application functions (SAFs). Each SAF is configured to execute in a certain step of a pre-ordained sequence. When you install the server, a number of built-in server application functions are provided. When you understand how the server operates, you may want to create your own server application functions. We term them server plug-in functions, and they are described in the chapter that follows this one.

How the server handles client requests

When the server is started, it loads the magnus.conf file, which contains information the server uses to configure itself. For example, the file tells the server the port to bind to, the name of the server, the user account to use after start-up, and so on. You can find a sample of the magnus.conf file on page 74.

Once it has started running, the server waits for requests to come in from client applications (such as Netscape Navigator). When a request comes in, the server uses another file, obj.conf, to determine if and how it should service the request. You can find a sample obj.conf file on page 77.

For example, the obj.conf file contains directives that can be set up to tell the server whether a user has access to the document they're requesting, and if they have access, to determine (among other things) whether the server should attach information to the document when it sends it to the client. This process is much more detailed, as described here and in the following sections.

The Netscape server responds to a client request by following a prescribed process in which each step is done once for all objects matching the request, then another step is done for all objects, and so on. For example, authorization translation (the first step) is performed on all objects in obj.conf, and then name translation (the second step) is done for all objects.

How the server uses directives and functions in an object

For more information on creating functions for these directives, see page 80. The process steps are as follows:

  1. Authorization translation (AuthTrans step)

    Translate any authorization information sent by the client into a user and a group. If necessary, decode the message to get the actual request. Also, user authorization is available through user databases.

  2. Name translation (NameTrans step)

    A URL can simply be kept intact and retrieved as-is (the typical server case), or it can be translated into one of the following:

  3. Path checks (PathCheck step)

    Perform various tests on the resulting path. Typically, this is done to make sure that it's safe for the given client to retrieve the document.

  4. Object types (ObjectType step)

    Identify the MIME type information for the given document. MIME types can be registered document types such as text/html and image/gif, or they can be internal document identification types.

    Internal types always begin with magnus-internal/, and are used to select a server function to use to decode the document. (Only used within the server itself; the server system calls these routines automatically when necessary.)

  5. Service (Service step)

    Select an internal server function to send the result back to the client. This function can be the normal server-service routine, it can return a custom document, or it can run a CGI program.

  6. Add Log (AddLog step)

    Record information about the transaction.

  7. Error

    Controls how the server responds to the client when it encounters an error.

    The first six of these steps map directly to six of the configuration directives allowed for each object. The seventh configuration directive (error) controls how the server responds to the client when it encounters an error.

    If at any time one of these steps fails, another step must be taken to handle the error and inform the client about what happened. In this case, the Netscape server lets you customize the response that is sent with more site-specific information about the error.

What are server application functions?

To accomplish each of the process steps, a set of server application functions (SAFs) are used in each step. These functions take the client's request and the server's configuration variables as input and return a response to the server as output. The Netscape FastTrack and Enterprise servers provide a set of built-in SAFs, and you can use the API to write your own plug-in SAFs.

Each server application function belongs to a particular class that corresponds to the directive that is used to call it in the configuration files. The functions appear after the directive type in obj.conf. You use the syntax:

directive fn="ourfunction" name1=value1 ... nameN=valueN
The directive is the directive the function belongs to (for example, the AuthTrans directive). You can then send any number of optional parameters to your function as arguments you specify on a directive line in obj.conf. For example, you could create a function for the NameTrans directive called myfunc1 that takes a URL path as a value and maps it to a hard-coded path specified in your function source code. The line in obj.conf might look like this:

NameTrans fn="myfunc1" URLpath="/special/docs"
The request-response code returned by the function can tell the server to do the following (REQ stands for request):

There is an additional class of application function, called init-class functions, that run when the server starts. They perform static data initialization for the various server modules. Each server application function (both the build-in functions Netscape provides and the plug-in functions you create) work with a single directive.

Note:
Before you can use any of your plug-in functions, you must configure the server to load the functions. You do this with the Init-class function load_modules in obj.conf. The full process is described later in this chapter.

Configuring the server

When you use the Server Manager, the forms act as an interface to the configuration files. The changes you make in the Server Manager are saved in these configuration files. Although you can use these files to manually configure the server, you should use the Server Manager to configure your server.

For various reasons, you might need to manually configure the server. If, for example, you accidentally lock your hosts out of the administrative forms or if you forget your administrative password, you'll have to manually change information in the server's configuration files.

Before you can edit any of the configuration files, you must have permission to read and write to the files. This probably means you need to log in as root. The magnus.conf, obj.conf, and mime.types files are kept in the config directory in your server root directory; the admpw file is in the admserv directory in your server root directory.

The magnus.conf file

The global configuration file, called magnus.conf, controls all server operations not related to handling of documents or directories. (The obj.conf file handles these.) All of the items in magnus.conf are global and apply to the entire server, as opposed to affecting only one document or set of documents.

Directive syntax

Every command line in the magnus.conf file has the format:

Directive Value

Comment lines begin with a # character with no leading white space. Directive lines can contain white space at the beginning of the line and between the directive and value, but do not use trailing white space after the value. You can continue long lines with a backslash () character before the linefeed.

Caution
If you are using the Server Manager, you shouldn't use continuation lines in the magnus.conf file. Instead, put each configuration entirely on a single line. If you are absolutely sure you will never use the Server Manager, you can use the backslash character.

Sample magnus.conf file

# Sample magnus.conf file for Netscape server 2.0 
# The server's home--its root directory 
ServerRoot /usr/ns-home/https-server1 
# The server's name 
ServerName www.netscape.com 
# Which port? 
Port 443 
# This tells the server to get its objects from obj.conf, and use  
# the "default" object as the default. 
LoadObjects obj.conf 
RootObject default 
# The logfile for errors, and the file where it should keep 
# the pid of the master server process. 
ErrorLog /usr/ns-home/https-server1/logs/errors 
PidLog /usr/ns-home/https-server1/logs/pid 
# Which user should the server run as? This is the UNIX user 
# account name. 
User http 
# Processes - number of processes to spawn 
MaxProcs 1 
MinThreads 4 
MaxThreads 128 
# Use DNS? (turn this off for performance reasons) 
DNS on 
# Security directives: is security on, where is my keyfile, 
# which ciphers should I support (Ciphers directive is on the 
# US version only) 
Security on 
Keyfile ServerKey.db 
Certfile ServerCert
For the details about the directives, see "Directives in magnus.conf" on page 201.

The obj.conf file

The object configuration file, called obj.conf, uses objects to control how the server handles documents.

Objects (also referred to as resources) are settings that tell the server how to treat documents, CGI programs, directories, imagemap files, and so on. You can define objects in two ways:

The structure of obj.conf

The obj.conf file must have certain specific objects in it. You can add other objects to this file. To specify an object, you use the following format:

</Object> 
<Object name=cgi> 
Directives 
... 
</Object>
You use wildcard patterns to control what is grouped in the object, or you use a name to create a configuration style.

<Object ppath=wildcardpattern> Directives ... <Client dns=wildcardpattern> Directives ... </Client>

You then specify one or more directives to control what the server does when it encounters anything that uses the configuration style or that matches the wildcard pattern specified with ppath. For more on the concept of ppath ( "partial path") see "NameTrans" on page 81.

You can also set options for specific client hosts. This is a powerful feature because, unlike other servers where a host either can or cannot access a document, you make the server act differently for a client depending on the document they access. Although <Client> sections are not required in an </Object> section, you can specify one or more--so the server acts differently based both on who requests something and what they request.

Directive syntax
Each command line in the file has the format:

Directive fn=function [parameter1=value1]...[parameterN=valueN]
Directive identifies an aspect of server operation. This string is case insensitive and must appear at the beginning of a line.

Function is the name of a function. It can be a built-in function (see Appendix B, "Server configuration files") or a plug-in function (see Chapter 5, "Creating server plug-in functions"). Its format depends on the directive.

The parameter and value pairs represent parameters given to the directive.

Comment lines begin with a # character with no leading white space. Directive lines can contain white space at the beginning of the line and between the directive and value, but trailing white space after the value might confuse the server. Long lines (which should only occur with the Init directive) can be continued with a space before the linefeed.

A sample object
The following sample object applies to the /user/public/ directory.

When the server receives a request for a document in this directory, it doesn't send the document. Instead, it denies the existence of any files or subdirectories and displays the "not found" error message.

<Object ppath="/user/public/*"> 
		PathCheck		fn=deny-existence 
		Service		fn=server-retrieve 
</Object>
The Service directive tells the server to get the documents by default.

Sample obj.conf file

# 
# Sample obj.conf file for Netscape server 2.0. 
# 
# This file was automatically generated by the server. 
# Edit at your own risk. 
# The default object. This is what the server uses if none of the other 
# objects fit. 
# 
# This one has a CGI directory specified in /usr/local/bin/cgi, 
# a directory mapping to a bigger disk in /gig-drive/sales, 
# and a document root of /usr/http-docs 
# 
# Initializations, such as log files and loading NSAPI libraries 
Init fn="flex-init" format.access="%Ses->client.ip% - %Req->vars.auth-user% 
[%SYSDATE%] "%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status% 
 %Req->srvhdrs.content-length%" access="/usr/ns-home/httpd-80/logs/access" 
Init fn=load-types mime-types=mime.types 
Init fn="dns-cache-init" cache-size="512" expire="1200" 
 
<Object name="default"> 
NameTrans from="/ns-icons" fn="pfx2dir" dir="/usr/ns-home/ns-icons" 
NameTrans from="/cgi-bin" dfn="pfx2dir" ir="/usr/local/bin/cgi" name="cgi" 
NameTrans from="/sales" fn="pfx2dir" dir="/gig-drive/sales" 
NameTrans root="/usr/http-docs" fn="document-root"  
PathCheck fn="unix-uri-clean" 
PathCheck fn="find-pathinfo" 
PathCheck index-names="index.html,home.html" fn="find-index"  
ObjectType fn="type-by-extension" 
ObjectType fn="force-type" type="text/plain" 
Service fn="imagemap" method="(GET|HEAD)" type="magnus-internal/imagemap" 
Service fn="index-common" method="(GET|HEAD)" type="magnus-internal/directory" 
Service fn="send-cgi" type="magnus-internal/cgi" 
Service fn="send-file" method="(GET|HEAD)" type="*~magnus-internal/*" 
AddLog fn="flex-log" name="access" 
</Object> 
 
# All CGI directories have these configuration options set 
<Object name="cgi"> 
ObjectType fn="force-type" type="magnus-internal/cgi" 
Service fn="send-cgi" 
</Object> 

Required objects for obj.conf

There are two objects that must be in the obj.conf file to make the Server Manager work for your server. These functions control local file access and CGI execution.

The following sections describe the objects that must be in obj.conf.

The default object
<Object name="default"> 
NameTrans fn="pfx2dir" from="/mc-icons" dir="/usr/home/mc-icons" 
NameTrans fn="document-root" root="/usr/http-docs" 
PathCheck fn="unix-uri-clean" 
PathCheck fn="find-pathinfo" 
ObjectType fn="type-by-extension" 
ObjectType fn="force-type" type="text/plain" 
Service fn="imagemap" method="(GET|HEAD)"  
        type="magnus-internal/imagemap" 
Service fn="index-common" method="(GET|HEAD)"  
        type="magnus-internal/directory" 
Service fn="send-file" method="(GET|HEAD)"  
        type="*~magnus-internal/*" 
</Object>
CGI Object
This object controls the admin form handler scripts and should read exactly as follows:

  <Object name="cgi"> 
  ObjectType fn="force-type" type="magnus-internal/cgi" 
  Service fn="send-cgi" 
  </Object>

Data in the server: the parameter block

The server stores variables in name-value pairs. The parameter block, or pblock (see page 194), is a hash table keyed on the name string. The parameter block maps these name strings onto their value character strings. Basically, server application functions use parameter block to get, change, add, and remove name-value pairs of data.

All server application functions (regardless of class) are described by the following prototype:

int function(pblock *pb, Session *sn, Request *rq);

The parameter pb is the parameter block containing the parameters given by the site administrator for this function invocation.

The parameter sn is a Session data structure (see page 193) that holds variables that apply throughout an entire session (the time between the opening and closing of the connection between the client and the server).

The parameter rq is a Request data structure (see page 195) contains the server's working variables.

Details for each directive/step

The following sections list the special conditions that exist in the parameter-block variables of the Request structure, and the request-response codes for each class of server function.

AuthTrans

The server splits authorization into two phases.

  1. Any authorization data sent by the client is decoded and verified against internal user-provided databases. If the data is valid, it is stored into the rq->vars parameter block.
  2. This data is compared against the required authorization for the requested path (performed in a PathCheck-class function). There is no reason that both of these operations could not be performed in one phase. (The Access Control List functions do perform both in one phase.) The split provides for flexibility and for possible future support of the SHTTP protocol.

    There are no variables in the Request->vars parameter block upon entry to an AuthTrans-class function. Upon successful completion of an AuthTrans-class function, the following parameter block variables are customarily set:

Of course, the flexibility of the pblock structure lets AuthTrans-class functions define their own custom variables that can later be retrieved by a PathCheck-class function that implements the second step.

Upon completion, AuthTrans-class functions should return:

  • REQ_PROCEED if successful.
  • REQ_NOACTION if the authorization was unsuccessful.
  • REQ_ABORTED if the request is to be aborted.

    NameTrans

    The NameTrans-class functions are used to translate a virtual path sent in the URI by the client into a physical path as used by the server.

    The following variables exist in the rq->vars parameter block for NameTrans-class functions when the NameTrans step begins:

    Upon return from a NameTrans-class function, the request-response codes have the following meaning:

  • REQ_PROCEED: Indicates that a name translation was performed, and that name translation for this object should not continue. This means the server moves to the next directive type in the object as defined in obj.conf (usuallyPathCheck, either in a user-defined object or the default object). When REQ_PROCEED is returned by a NameTrans-class function, path is automatically created from ppath. These two persist until the request is completed.
  • REQ_NOACTION indicates that no final name translation was applied by the function. This doesn't mean that ppath was not changed. The server continues applying the rest of the object's NameTrans functions.
  • REQ_ABORTED indicates that the request has encountered an error and that an error message should be sent to the client. No functions are called after the function returns this, except any functions that are designated error handlers.
  • REQ_EXIT indicates that an I/O error occurred while talking with the client. The request should be aborted and no error message should be sent.

    PathCheck

    The PathCheck-class functions are used to verify that a given path is safe to return to a given client. This class of functions performs various system-specific URL filtering and screening actions, authorization checks, access control, searches for CGI extra path information, and other functions.

    The server only defines one variable for PathCheck-class functions in the Request->vars parameter block: the value of path is the path resulting from the execution of all NameTrans directives. Any variables that have been created by previous directives are also available.

    Upon return from a PathCheck-class function, a request-response code of REQ_PROCEED and REQ_NOACTION indicates success.

    ObjectType

    The ObjectType-class functions take the path resulting from the previous directives and locate a file resource for the path. If none exists and none of the PathCheck directives have looked yet, the current ObjectType routines return a default type to the client.

    The ObjectType directives receive no special variables aside from path in Request->vars and the variables defined by the previous directives. Upon return from an ObjectType-class function, a request-response code of REQ_PROCEED and REQ_NOACTION indicates success.

    The server's built-in functions provide mechanisms for typing files natively to the machine's operating system. On most systems, this consists of file name extension recognition code.

    Service

    The Service-class functions send the server's response to the client. The function is generally selected according to the directory the file resides in or the type of the file being sent. Most files are simply mapped into memory and sent to the client. Some types of documents run as system programs and others are parsed before being sent to the client.

    Service directives receive no special variables aside from path in Request->vars and any variables defined by the previous directives.

    Upon return from a Service-class function, the request-response codes have the following meaning:

    The Service-class functions need to call the protocol_status function, then start the protocol-specific response process for the server. They do this through the following function:

    int protocol_start_response(Session *sn, Request *rq);
    
    If the protocol_start_response function returns REQ_NOACTION, then the actual response should be skipped and the application function should return successfully. The only other value that protocol_start_response returns is REQ_PROCEED.

    If cross platform considerations are not required, then operating-system specific I/O calls can be made by Service-class functions.

    Initialization functions

    This special Init-class of server application functions initialize global data for the server to use on startup. These functions are called by the base server daemon process, and their data, sockets, or file descriptors are inherited by child processes or threads. This class of function initializes the logging, file typing, and some of the name translation functions.

    Init-class functions receive their parameters just like the other functions, except that the Session and Request parameters are NULL. The parameter pblock is filled with information from the server's configuration file (obj.conf).

    Static data should be limited to read-only data if possible. If the program can be sure that no race conditions exist, it can make the data read/write. Other data media (such as databases) will need to employ some method of locking to ensure that only one process or thread is accessing the data at a time. The routines in the base/sem.h header file are used to coordinate processes, and those in the base/crit.h file are used to coordinate threads.

    Upon failure, plug-in Init-class functions should return REQ_ABORTED and insert into their parameter block a variable named error that contains a string describing the error, for logging Any other request-response code is considered success.

    If the server is restarted, the Init-class functions might need to be terminated before they are started again by the server code. For this purpose, the server provides the following function:

    void daemon_atrestart(void (*fn)(void *), void *data)
    
    The given callback fn is called by the server when the server is restarted. The data pointer is given to the function as its argument.

    Note:
    The Netscape servers do not call the callbacks upon server termination, only upon restart.

    Equivalents to CGI Environment Variables

    Table 4.1 shows the common environment variables used by the CGI and their server plug-in API equivalents. When processing begins on a message, the environment variables and their values are all available by a call to the getenv routine, but as processing continues, the values are stored in the data structures as shown here.
    CGI context

    Server plug-in API context (x representing the value)

    SERVER_VERSION

    x=MAGNUS_VERSION_STRING;

    MAGNUS_VERSION_STRING is a macro.

    SERVER_PORT

    x=port;

    port is a global.

    SERVER_HOSTNAME

    x=server_hostname;

    server_hostname is a global.

    SERVER_URL

    x=http_uri2url ("","");

    REMOTE_HOST

    x=session_dns(pblock_findval("ip", sn->client));

    REMOTE_ADDR

    x=pblock_findval("ip", sn->client);

    REMOTE_USER

    x=pblock_findval("auth-user", rq->vars);

    AUTH_TYPE

    x=pblock_findval("auth-type", rq->vars);

    HTTPS

    if (security_active) x=TRUE else x=FALSE:

    security_active is a global.

    HTTPS_KEYSIZE

    x=pblock_findval("keysize", sn->client);

    HTTPS_SECRETSIZE

    x=pblock_findval("secret-keysize", sn->client);

    SERVER_PROTOCOL

    x=pblock_findval("protocol", rq->reqpb);

    REQUEST_METHOD

    x=pblock_findval("method", rq->reqpb);

    PATH_INFO

    x=pblock_findval("path-info", rq->vars);

    QUERY_STRING

    x=pblock_findval("query", rq->vars);

    USER_AGENT

    x=pblock_findval("user-agent", rq->headers);

    Some new features in version 2.0

    The Version 2.0 servers provide several new features, including multi-threading (both on Unix and on Windows NT), caching of requests, and multihoming.

    Threading

    A thread is a single independent stream of execution within a program. A critical section is a section of program code that must be executed by only one thread at a time. Portions of code that perform such tasks as switching between threads or updating a record in a database are critical sections. A condition variable is a thread-synchronization variable that lets the user specify arbitrary conditions on which to block.

    In addition to the Windows NT servers being multithreaded, all of the 2.0 Unix Web servers are multithreaded. There are abstractions in base/systhr.h and base/crit.h for threads, critical sections, and condition variables.

    The Unix servers are still capable of running in multiple-process mode. Each process then has a pool of threads that it uses to answer requests. If your server plug-in modules are not thread-safe, you can run the server with one thread per process to simulate the 1.1 server architecture; however this can result in a performance loss. You can run one process with multiple threads to simplify programming. This lets you share context without using shared memory. However, high traffic sites that use CGI should use several processes (2-8) with fewer maximum threads per process. For example, if you normally run one process with 128 threads, run 4 processes with 32 maximum threads each if your site serves a very high volume. This avoids large virtual address spaces for a single process.

    New thread manipulation calls exist in base/systhr.h. The following functions can manipulate threads:

    Note that creating a new thread in an Init-class function results in undefined behavior under Unix. If the thread does something periodically, that action will happen in both the base server process and all of its children.

    The functions defined in base/sem.h remain the same and provide server-wide mutual exclusion. Under Windows NT, these are simply critical sections that apply to all threads in the server process. Under Unix, these primitives operate on all threads in all processes of the server, so they are very heavyweight. They should only be used to protect access to process-shared resources such as shared memory and files.

    New calls exist in base/crit.h that provide process-local critical sections and condition variables for Windows NT and Unix. These calls only affect threads in the current process. Critical sections are simply a lock that can be owned by only one thread at a time. They can be recursively entered. Condition variables are used in conjunction with a critical section to provide "wait" and "notify" primitives.

    On IRIX, Solaris 2.x, and OSF/1, the operating systems provide threading support, and multithreaded versions of libc. On those systems, you can call most libc routines safely. One notable exception is that you must use the localtime_r and gmtime_r functions instead of the localtime and gmtime functions respectively. Many other functions also must be called in their reentrant form. Check your system's documentation for details.

    On systems that don't have threading support, we provide the threading via user-level context switches. The server doesn't currently try to perform thread preemption, which means that you can still use libc calls that aren't re-entrant. Thread switching is performed when a lock is used, when the API I/O functions such as net_read are called, and when a systhread_yield function is explicitly called. This also means that if your server plug-in function makes a system call that will block, you will block the process that your thread is running in (thus blocking all threads in that process). DNS calls such as gethostbyaddr are an example of a call which will block the process, however the API function session_dns will not block the process.

    Under IRIX, you must compile with -D_SGI_MP_SOURCE.

    Under Solaris, you must compile with -D_REENTRANT.

    Under OSF/1, you must compile with -threads.

    Caching

    The 2.0 server attempts to cache requests that will create the same response when requested by multiple clients at different times. That is, if a client requests /mfg/proc/item.txt, and then another client requests /mfg/proc/item.txt, the server's response will be the same as long as /mfg/proc/item.txt doesn't change between the requests. When the server can avoid calling the NSAPI functions for a request, it can return the responses faster.

    There is a new variable called directive_is_cachable in the Request structure in 2.0. By default, this variable is set to 0 when calling your NSAPI functions. If you do not set this variable to 1 before your function returns, the server will not try to cache the request, and each subsequent request will call your function again. If you set it to 1, the server may not call your function when the next client makes the same request.

    You should cache requests that do not depend on some aspect of the client to determine how they are returned. If your function performs access control, logging, switches data based on user-agent, or anything similar, you should not mark your directive as cachable. If your function is doing something which does not depend on the client IP address or the headers the client sends, then you can mark your directive as cachable.

    Software multihoming

    Netscape Navigator Version 2.0 and some other browsers now send a Host: header telling the server which URL host and port is being accessed. In addition to the Version 1.1 functionality of making your obj.conf directives conditional on the client's IP address, you can also specify that they only be executed when the URL host is a specific pattern. For example,

    <Client urlhost=www.(any|such).com>
    NameTrans fn=my-nametrans from=/anysuchdir
    </Client>

    placed in the default object will call my-nametrans only when the browser thinks it is accessing www.any.com or www.such.com. Similarly,

    <Client urlhost=*~www.(any|such).com>
    NameTrans fn=my-nametrans from=/anysuchdir
    </Client>

    will apply my-nametrans to every except those who access www.any.com and www.such.com including browsers that don't send the Host: header.