Java Web Fetch

By Darrin Koltow, for TopHost.gr, a Greek web hosting company specializing in shared and reseller hosting.

Among Java's many strengths are its classes used to send and receive web content. Among these classes is an Authenticator class, which you can use to programmatically submit a login id and password before processing content returned by the server.

The sample program in this article fetches a web page and displays its content in a text area. The program includes functionality to submit credentials (AKA login id and password) if the server requests such.

Working with the NetBeans IDE

We're going to use the free NetBeans IDE (integrated development environment) to create this app. Download NetBeans from NetBeans.org.

Open NetBeans after installing it, then select "File>New Project," followed by "Java application." Name the project "webpuller," and uncheck the "Create Main Class" checkbox. Press the "Finish" button on the "New Java application" dialog box.

Right-click on the "Source packages" node in the "Projects" window and select "New>Java package." Name the package "webpuller," then press "Finish." Right-click the project window's "webpuller" package node that you just created and add a new "Jframe form" with the name "webpuller."

Design the form

In the Design view, drag a text field from the Palette onto the blank form, and situate it at the form's top. Resize the field to extend the width of the form.

Drag a text area from the palette to the form, under the text field. Extend its width to the form's width. Extend the text area's height from the bottom of the text field to the bottom of the form.

Right-click the text area and select "Properties," then check the "linewrap" and "wrapStyleWord" properties. Close the Properties dialog box.

Write the program code

Now that you've designed the form, it's time to make it _do_ something. In the form designer window, double-click the text box. The NetBeans IDE will move you to an event function for that text box. Specifically, when the user presses the return key after typing into this box, code execution will jump to this function.

NetBeans provides just the skeleton of the function, which we must fill in with code to make a request to the server, formatted with the text of this text box. Here's the code to make that request:

private void jTextField1ActionPerformed(
    java.awt.event.ActionEvent evt) {
    String content = getContentFromServer(jTextField1.getText());
    JTextArea1.setText(content);
}

Let's write the code to actually fetch the content. First, paste this statement under the "package..." statement at the top of the file:

import java.net.*;

Paste this method after the last curly brace of the jTextField1ActionPerformed method just listed:

String getContentFromServer(String strRequest) {
    String str = "";
    String str2 = "";
    try {
        URL url = new URL(strRequest);
        InputStreamReader rdr = new
                InputStreamReader(url.openStream());
        BufferedReader bufrdr = new BufferedReader(rdr);
        while ((str2 = bufrdr.readLine()) != null) {
            str += str2;
        }
    } catch(Exception e) {
        System.out.println(e.toString());
    }
    return str;
}

Run the program

Run the program by pressing "Shift-F6." Type "http://www.java.sun.com" or another valid, full URL into the text box. Press "return" after typing the URL, and allow a moment to pass before watching the text area, which should fill with HTML code.

Adding authentication

Our text browser works fine for URLs that don't require authentication (i.e. login id and password), but will be stopped cold by URLs that do. Let's write the code to automatically provide authentication when the server asks for it.

First, you'll need a password-protected URL that you have the password (and login id) for. Once you have that, return to your program in NetBeans and enter the following additional code. Place the code just after the line that reads "public class webpuller..." (the start of the webpuller class definition.)

public class Cmyauth extends Authenticator {
    protected PasswordAuthentication getPasswordAuthentication() {
        String slogin = "joetrece";
        String spword = "joetrece";
        return (new PasswordAuthentication(slogin,
            spword.toCharArray()));
    }
}
/* Creates new form webpuller */
Cmyauth myauth;

Next, change the existing webpuller constructor ("public webpuller() {...") to this:

public webpuller() {
    initComponents();
    myauth = new Cmyauth();
    Authenticator.setDefault(myauth);
}

Re-run the program ("shift-f6"), entering the password-protected URL in the text box, followed by hitting "enter." After a moment, the HTML of the page you requested should appear in the text area.

References

Java World: Use Java's Authenticator Class