Communicating with an NNTP server
by André Pinheiro (adlp@camoes.rnl.ist.utl.pt)

The NNTP (Network News Transfer Protocol) server is the backbone of the Usenet newsgroups. NNTP servers interact with two kinds of clients: newsreaders and other news servers. The interaction with a NNTP server is done through text-based commands and responses.

We'll develop an application that communicates with an NNTP server and requests either a list of all available newsgroups or some newsgroups' articles.

As I mentioned before, the interaction is done through text-based commands and responses. Here are a few commands that a news server understands: LIST, GROUP, STAT, HEAD, BODY, NEXT, and POST. But not all these commands go just by themselves. Neither all of them need to be covered by this article, as our main purpose is to develop an NNTP client that lists all newsgroups and displays some articles - no post operations involved here, so let's not despair so early.

Here are some rules regarding commands and parameters:

- Commands may be followed by a parameter.
- Parameters are separated by one or more space or tab characters.
- Command lines must be complete with all required parameters, and may not contain more than one command.
- Commands and parameters are case insensitive.
- Each command line must be terminated by a CR-LF (Carriage Return - Line Feed) pair.

There are two kinds of responses, status and textual:

Status Responses - These are status reports from the server and indicate the response to the last command received from the client.

Status response lines begin with a 3 digit numeric code. The first digit of the response indicates the success, failure, or progress of the previous command.

1xx - Informative message.
2xx - Command ok.
3xx - Command ok so far, send the rest of it.
4xx - Command was correct, but couldn't be performed for some reason.
5xx - Command unimplemented, or incorrect, or a serious program error occurred.

Let's create a very simple method for sending commands:

  void sendCommand(String cmd)
    throws IOException, ZNewsCommandException
  {
    System.out.println("Sending command \"" + cmd + "\"...");
    out.write(cmd + "\r\n");
    out.flush();

    if ((buffer = in.readLine()) == null)
      throw new IOException();
    else
    {
      if (buffer.length() < 1)
        throw new IOException();

      if (buffer.charAt(0) != '2')
        throw new ZNewsCommandException("Command \"" + cmd + "\" failed.");
    }
  }

Text Responses - Text is sent only after a numeric status response line has been sent indicating that text will follow.

Text is sent as a series of successive lines, each terminated with a CR-LF pair. And a single line containing only a period (".") is sent to indicate the end of the text.

If the text contained a period as the first character of the text line in the original, that first period is doubled. Therefore, the client must examine the first character of each line received, and for those beginning with a period, determine either that this is the end of the text or whether to collapse the doubled period to a single one.

Here's a method for handling text responses:

  String getTextResponse()
    throws IOException
  {
    String text = "";

    while(true)
    {
      buffer = in.readLine();

      if (buffer == null)
        throw new IOException("End of text not expected.");

      if (buffer.length() < 1)
        continue;

      buffer += "\n";

      // check for end of text
      if (buffer.charAt(0) == '.')
      {
        if (buffer.charAt(1) != '.')
          break; // no doubt, text ended

        // we've got a doubled period here
        // collapse the doubled period to a single one
        buffer = buffer.substring(1, buffer.length());
      }

      text += buffer;
    }

    return text;
  }

This is the easy part, we just have to create a stream socket and connect it to the well-know port #119 on the specified host (i.e. the news server).

  Socket connect()
    throws IOException
  {
    int port = 119;

    System.out.println("Connecting to " + host + "...");
    Socket socket = new Socket(host, port);
    System.out.println("Connected.");

    out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
    in = new BufferedReader(new InputStreamReader(socket.getInputStream()));

    return socket;
  }

There are still no intelligent news servers, so I guess the above sentence wouldn't work. Hum... where's the quick reference? Ah! Found it:

Returns a list of valid newsgroups and associated info.

Each newsgroup is sent as a line of text in the following format: "<group> <last> <first> <p>", where <group> is the name of the newsgroup, <last> is the number of the last known article currently in that newsgroup, <first> is the number of the first article currently in the newsgroup, and <p> is either "y" or "n" indicating whether posting to this newsgroup is allowed ("y") or prohibited ("n").

The following code should do it:

  void listNewsgroups()
    throws IOException
  {
    try
    {
      sendCommand("list");
    }
    catch(Throwable e)
    {
      System.out.println(e.getMessage());
      return;
    }

    // get names of all newsgroups
    while(true)
    {
      buffer = in.readLine();

      if (buffer == null)
        throw new IOException("End of text not expected.");

      if (buffer.length() < 1)
        continue;

      if (buffer.charAt(0) == '.') // check for end of listing
        break;

      StringTokenizer st = new StringTokenizer(buffer, " ");
      newsgroups.addElement(st.nextToken());
    }

    for(int i = 0; i < newsgroups.size(); i++)
      System.out.println(newsgroups.elementAt(i).toString());
  }

To get the contents of all articles from a given newsgroup, we need to:

- select the newsgroup;
- set the current article pointer to the first article;
- get the article's header;
- get the article's body;
- set the current article pointer to the next article.

These steps can be accomplished using four commands: GROUP, HEAD, BODY, and NEXT.

GROUP <group name>

<group name> is the name of the newsgroup to be selected (e.g. "comp.lang.java.tech").

When a valid group is selected with this command, the internally maintained current article pointer is set to the first article in the group.

Responses:

<n> is the estimated number of articles in the newsgroup;
<f> is the first article's number;
<l> is the last article's number;
<s> is the name of the newsgroup.

Returns the header text of the current article.

Responses:

<n> is the article's number;
<a> is the message id.

Returns the body text of the current article.

Responses:

<n> is the article's number;
<a> is the message id.

The current article pointer is advanced to the next article in the current newsgroup. If no more articles remain in the current group, an error message is returned and the current article remains selected.

Responses:

<n> the article number
<a> the unique article id

Given a Vector newsgroups, holding the newsgroups we want to look at, these two methods shall do the job:

  void displaySelectedNewsgroups()
    throws IOException
  {
    for(int i = 0; i < newsgroups.size(); i++)
      displayNewsgroup(newsgroups.elementAt(i).toString());
  }


  void displayNewsgroup(String group)
    throws IOException
  {
    try
    {
      sendCommand("group " + group);
    }
    catch(ZNewsCommandException e)
    {
      System.out.println("Error: could not access newsgroup \"" + group +"\".");
      return;
    }
    catch(IOException e)
    {
      System.out.println(e.getMessage());
      return;
    }

    StringTokenizer st = new StringTokenizer(buffer, " ");
    st.nextToken();
    st.nextToken();
    int firstArticleNumber = (new Integer(st.nextToken())).intValue(),
        lastArticleNumber = (new Integer(st.nextToken())).intValue();
    System.out.println("First article #" + firstArticleNumber);
    System.out.println("Last article #" + lastArticleNumber);

    while(true)
    {
      try
      {
        sendCommand("head");
        System.out.println(getTextResponse()); // display header text
        sendCommand("body");
        System.out.println(getTextResponse()); // display body text
      }
      catch(Throwable e)
      {
        System.out.println(e.getMessage());
        // there's been a problem with the current article, let's try to get the next one
      }

      try
      {
        sendCommand("next");
      }
      catch(ZNewsCommandException e)
      {
        // "next" command failed
        // looks like there are no more articles in the current newsgroup
        System.out.println(e.getMessage());
        return;
      }
      catch(IOException e)
      {
        System.out.println(e.getMessage());
        // an IO error, let's get out of here!
        return;
      }
    }
  }

Here's the full source code for the ZNewsReader application: javazine/znetutil/ZNewsReader.java.

And, if you're like me, you'll want to read the NNTP specification:

Network News Transfer Protocol,
A Proposed Standard for the Stream-Based Transmission of News
Brian Kantor and Phil Lapsley, February 1986