The CGI Interface

What is CGI?

CGI stands for Common Gateway Interface. It is essentially a mechanism which is used to connect programs written on a server in any language to the web.

During the 1990s, NASA and the Russian space program conducted several missions in which American space shuttles were docked to the Mir space station. What is remarkable about this is when these vehicles were designed, the US and the then Soviet Union were cold war enemies. It was unthinkable that the two vehicles would ever actually connect together. When the NASA - Mir missions came up, the engineers created a special device which was essentially an airlock with a Russian door on one end, and an American door on the other. This device could be attached to the two different systems simultaneously, and would allow transactions between the two.

The CGI interface is a lot like that docking module. It is designed to work with web servers (and through them web browsers) on one end, and standard programming languages on the other. It is not a programming language, but a technique for attaching programming languages to the web.

CGI access is part of the server daemon, and is controlled by the server's system administrator. It allows a specific subset of programs that reside on the server computer to be accessed through the web.

The communication goes two ways- a web page can send information on a form to a program through the CGI, and a CGI-activeated program can send its output to a web page.

Some examples of CGI in action

Take a look at the following segment of a web page:

You will see that it contains a simple form, with a textbox and a button. We can enter text into the textbox, and press the button, and we will be given a special page, which is the result of a yahoo search regarding whatever we typed in. For example, If I type "Motorcycle," I will find a number of pages in the Yahoo system dedicated to motorcycles. This is the equivalent of going to the Yahoo page and using the search engine there. The difference is that this page is on my own site. Essentially, my own site can be a front end to Yahoo!!

Here is part of the result page I got back from my yahoo search:

As you can see, it has returned a set of pages that match my request. Take a special look at the location (URL) area of the browser. You can see that the URL we have gone to is not exactly the same as we have seen before. The URL looks like this:

http://search.my.yahoo.com/search?p=motorcycle&n=25

Much of the URL looks familiar, but there is a question mark followed by some other values. The word 'motorcycle' is in there. As an experiment, let's modify the URL by hand, and replace the word 'motorcycle' with some other word, like 'movie', and see what happens. Here is the result:

By editing the part of the URL that comes after the question mark, I was able to completely change the search.

This information on its own is not very interesting, but it does point out some interesting possibilities. Clearly it is possible to send information to certain sites on the internet, and these sites will respond differently based on what information is sent. What we need now is a way to control how to send the information, and where to send it.

Another look at HTML forms

HTML forms are useful for grouping together input elements, but we haven't yet seen why the form tag itself is necessary. If all you have done is client-side scripting with a language such as javascript or vbscript, there doesn't seem to really be much need for forms. Forms were actually designed with CGI in mind, and they were only later converted for use in the client-side languages. Let's take a look at the HTML code that creates the Yahoo lookup.

 <form action="http://search.my.yahoo.com/search">
 <h1>Search Yahoo for...</h1>

 <input size=25 name=p> 
 <font size="-1">  
 <input type=submit value=Search>
 <input type=hidden name=n value="25">   
 <a href="http://search.my.yahoo.com/search/options">Options</a>
 </font>
 </form>

As you can see, there is a form here, and some standard elements. The text box is completely normal, but there are some other things you may not have seen before.

action

First, the form tag has an action attribute. The action attribute points to the URL of a CGI program somewhere on the web. This is NOT a standard web page, but some kind of executable program that lives on a server somewhere. In this case, I'm pointing to the search program at yahoo.

hidden field

There is a hidden input object on the form. These are used frequently in CGI programming so that an HTML author can send some kind of information to the CGI program without the user being involved. For example, on a page that you use to check your grades online, you might expect to find some inofrmation about your particular class and section 'hard-coded' into the HTML, so you don't have to worry about it.

submit button

The submit button looks much like a regular button, but it has an important difference in CGI. When this button is pressed, it connects the browser to the program identified in the action attribute of the form tag, and it sends all the values of the other input elements of the form to that program.

named elements

You can see that there are two named elements in the form. One is called p, and it appears to contain the subject being searched for. The other named element is a hidden tag called n. By experimenting with this value, it appears that it stands for the number of elements to return per page.

The client's responsibility

The entire goal of an HTML form is to set up a collection of name/value pairs to send to a particular server program. The server program is designed to 'catch' this series of name/value pairs, and to do some kind of manipulation with these values.

How the server fits in

On the server side, CGI accepts the name/value pairs, and passes them to the program running on the server. Any output the program generates is redirected back to the browser that originally called the program. If a program is called through CGI, the program's normal output will be read by the browser. If that output is HTML code, it will be read by the browser just as if the code were coming from a plain HTML file on the server.

How it works

As you might guess, server-side programming can be much more powerful than programing on the client, as the programmer has access to most of the resources on the machine, including other programs, and the ability to load and store files. This power comes with some risk, so system administrators do not usually allow just anyone access to cgi programming features.

First, the web server must be configured to allow cgi programming. All the major web daemons have some capability to do this, although it is not always turned on as a default.

Any programs which are going to be accessed through CGI are usually placed in a special directory, often called cgi-bin. Any program not here will NOT be executed through the web. This is done so that malicious users cannot break into a system and run all the executable programs from the web.

In addition to being in the cgi-bin directory, a cgi program must of course also be an executable program. In the unix world, this means it must have the execute bits of its permission set. Usually a CGI program has 755 or 711 permission.

Unless you are the system administrator, you probably do not have direct access to the cgi-bin directory. There are three major workarounds to this problem. First, you might be allowed only to access programs that have already been written, tested for safety, and placed there by the system administrator. Of course, this is not very interesting for us, as we want to write our own. Another common solution is to submit all programs to the administrator and ask him or her to place the program in the cgi-bin directory. While this strategy works, it can take a lot of time for the administrator, and it makes it difficult to modify your programs. A third option is the use of redirection with a program like cgi-wrap.

Cgi-wrap is a program that resides in the machine's cgi-bin directory. It has a number of functions, but primarily it accepts all cgi requests, grabs the cgi program from a particular user's account, and checks to ensure the program is reasonably secure before running it. This has two important side effects: Each user effectively has his or her own cgi-bin, and the system administrator does not have to personally scan each program to ensure it is safe.

Converting a program to CGI

It is sometimes easier to look at a straight command line program first, and then see how it can be converted to deal with CGI input and output. We will start by looking at a very simple program that does some input and output:

#!/usr/local/bin/perl
#where's wally?
#A very simple example in command line perl

print "What is your name?";
$userName = <STDIN>;
chop $userName;

if (uc($userName) eq "WALLY"){
    print "I found you at last!!!\n";
} else {
    print "Sorry.  I was looking for Wally.\n";
} # end if

This is a very simple program, and it is easy to understand. It contains one input, a condition, and one of two outputs. Since CGI is an IO interface, all we need to do is ensure we can do input and output, and all the other work will be just like normal command-line perl.

In CGI, all input comes from HTML forms, so the first thing we need to do is devise a form that will 'set the stage' and ask the user for any data we need (in this case the user's name) before the program runs. Such a page might look like this:

Where's Wally?

The source code for this page is as follows:

<html>
<head>
<title>Where's Wally?</title>
</head>

<body bgcolor = "white">
<center>
<h1>Where's Wally?<hr></h1>
</center>
<form action = http://www.cs.iupui.edu/cgi-bin/cgiwrap/aharris/wally.cgi
      method = post>
<h3>What is your name?</h3>
<input type = text
       name = txtName>
<br>
<input type = submit>
</form>

</body>
</html>

As you can see, there are very few surprises here. The only part that is new is the action attribute of the form tag. Exactly what goes in there will depend on how your server is configured. On the machine that is running this example, we have cgiwrap set up, so I send my cgi requests to the cgiwrap program in the server's cgi-bin directory. The rest of the path ("/aharris/wally.cgi") is used by the cgiwrap program to figure out that it is supposed to go to the directory structure of a user called 'aharris' and look in his cgi-bin for a file called 'wally.cgi'. When the user presses the submit button, her browser will attempt to find that program, and will send it the name 'txtName' and an associated value that the user has typed into the text box. Presumably the program will know what to do with the values, and will return something sensible back.

Of course the CGI author is responsible for ensuring that the program is able to work with the data that is sent to it. Let's look at the wally.cgi program, and see how it works:

#!/usr/local/bin/perl
#where's Wally converted to a cgi program

use CGI ':standard';

print header();

print start_html("results...");

$userName = param('txtName');
if (uc($userName) eq "WALLY"){
    print "I found you at last!!!\n";
} else {
    print "Sorry.  I was looking for Wally.\n";
} # end if

As you look over this program, you will recognize that most of the program has not changed at all. The changes are just in a few areas...

The use statement

use CGI ':standard';

First, the use command is used to import an external library. In this case, we are bringing in a library called CGI.pm, which contains several functions for streamlining the CGI process. It is not absolutely necessary to use a library, but the code is much more difficult to write and debug if we do not use the CGI routines. CGI.pm is part of the standard perl 5.0 installation. If you are using perl 4.0, cgi-lib.pl gives similar functionality, but is not as powerful.

We will usually use this form of the use command to bring in the standard parts of the cgi module. You might encounter other forms of the statement that are designed to bring in special parts of the module, but if you need to do that, you should be carefully reading the documentation that comes with CGI.pm.

print header();

This line calls a function of the CGI.pm module which automates making a header for your file. When a browser receives a file, it first expects some kind of header, explaining what kind of data will be in the file. For standard HTML pages, the header must be

content-type: text/html/n/n

It is very easy to get this wrong, so we will usually just rely on the header function to create the appropriate string for us. you must print the header before you print anything else, or the browser will not know how to read the page.

The content-type line allows CGI to re-route anything you print to the screen (standard output) to go to the browser that called the program. The browser will read the data as if it is coming from a standard file. Any HTML tags you embed in the output will be read by the browser. Once we have used this line, we can generate HTML as our output.

If there is an error in our code, the error message will come to the screen before the content-type message, so the browser will not get a valid header, and you will get a '500 server error.' The last line of the message will probably look like this:

Error: HTTPd: malformed header from script
/opt/www/httpd/cgi-bin/cgiwrap

That very unhelpful error message should be interpreted to mean "There's something wrong with your script!!!" go back and debug it some more!!" See later in this chapter for some debugging hints.

print start_html("results...");

This line shows one of the nicer features of CGI.pm. It sends the following code to the browser:

<html>
<head>
<title>results...</title>
</head>
<body>

Of course, you can just use a series of print statements to do the same thing, but this is very easy.

$userName = param('txtName');

This line is extremely important, because it illustrates how we do input in cgi.pm. The param function looks at the name-value pairs sent by the form, and it returns back the value associated with the name we sent it. Since we have an attribute called 'txtName' on the form which called this program, the value of the txtName text box will be sent to the local variable $userName.

Almost all cgi programs you write will follow the same general format. You will call the cgi.pm module, write the header to set up output, print the start-html() function to start up your page, and use the param function to grab any variables that were in the original form. It's easiest to load the values into perl variables, because then you won't have to worry about the form again. Then, most of your programs will involve some kind of processing, and finally you will output HTML back to the browser.

Changing it to one form

CGI was originally thought of as involving at least two entirely seperate files, the HTML page that sends the request, and the CGI program that responds to it. It can be extremely awkward to maintain this 'long distance relationship' (as it is called by Lincoln Stein, CGI.pm's author). Recently, it has become popular to write all the code in the same file. CGI.pm was designed to make this reasonably easy to do. Watch closely the following example, and see if you can figure out how it replaces BOTH the HTML file and the script that reads the HTML:

#!/usr/local/bin/perl
#wallyOne.cgi

#uses ONE FILE to handle writing the wally HTML and to decipher the values
#Andy Harris, 6/99

use CGI ':standard';

print header();

#check to see if there are any parameters
if (!param){
  #there's no parameters, so this is the first time through.
  #set up the web page to ask for the user's name

  #note the multiline quote.  Be careful of double quotes inside the quote
  print "
<html>
<head>
<title>Where's Wally?</title>
</head>

<body bgcolor = 'white'>
<center>
<h1>Where's Wally?<hr></h1>
</center>
<form action = http://www.cs.iupui.edu/cgi-bin/cgiwrap/aharris/webprog/wallyOne.cgi
      method = post>
<h3>What is your name?</h3>
<input type = text
       name = txtName>
<br>
<input type = submit>
</form>
</html>";

} else {
  #there is a parameter, so they must have answered
  #Let's figure out what they said
  # this is copied from the regular "wally.cgi" program

  print start_html("results...");

  $userName = param('txtName');
  if (uc($userName) eq "WALLY"){
      print "I found you at last!!!\n";
  } else {
      print "Sorry.  I was looking for Wally.\n";
  } # end 'who is it?' if
  print end_html();

} # end 'any parameters?' if

When the web author wants to use this program, he or she will put an anchor that points directly at the script, rather than an html document. The script will determine whether it is coming from a form automatically. This works by first examining the parameters with the param function. param returns back name - value pairs normally, but if the program was called without any name - value combinations, it will return back the value false (0). The first time we call the program, it will be coming from an anchor tag, NOT a form, so the value of param will be false.

We check for this in the line that says:

if (!param){

Remember, the ! means 'not.'

dynamically writing the page

If the condition is true, this is our first time through the program with this user, so they need to be asked their name. We will simply copy and paste the HTML from the original document here, to make exactly the same screen. Notice the special way we did the print statement. Perl allows multi-line print statements like this, but you have to be very careful about quotes in the HTML. If your HTML must have quotes, try using single quotes, or you can use the backslash (/") to tell perl that it should write the quote, but not interpret it as the end of the string we are printing. It is easy to forget the ending quote mark. Be careful about this.

The only other thing about this part of the program that I changed from the original HTML is the action attribute of the form. I made sure that it returned back to exactly the same program. (!)

Dealing with the input

When this program is run the first time, it creates the form for user input, and sets up that form with some values. It also tells that form to return to this program when it is submitted. Once the page is sent to the browser, the CGI connection will be completely lost, and will need to be re-established by the user submitting the form. This is actually good. If, for example, we have some kind of online survey, and a user takes 20 minutes to fill out the survey, we do not want to leave the connection open that long.

It is important to see that we will call the program once for each transaction, so for this example, the user will have two completely different transactions with exactly the same program. In more complex programs we might have dozens of different connections.

This time, though, the user is coming from a form, so param will have something in it. Specifically, it will have a txtUser object with some kind of string value (whatever the user typed into the box) in it. We run the same (!param) condition, and this time it is false, because param is NOT empty. Now we have something in param, so we can write the code to work with it.

The rest of the code is identical to the wally.cgi we saw before. In fact, it was copied and pasted into this code.

Implementation Hints

It makes a lot of sense to plan ahead when you are writing cgi programs. You are working on at least two languages (html and perl) at once, and you have all kinds of opportunities for confusion. It is a very good idea to sketch out all the screens you expect your program to have, and to write them out in HTML first. You might be able to re-use the HTML directly, and you will definitely want to refer to it.

Learn the cgi.pm html shortcuts. The library has a number of shortcuts that automate the creation of html for you. The start_html() function is a good example. Most of the HTML tags you already know have support functions. As an example, look how the the wally program that creates the form looks when heavily relying on these functions:

if (!param){
  print start_html("Where's Wally?");
  print h1("Where's Wally<hr>");
  print start_form();
  print h3("What is your name?");
  print textfield("txtName");
  print "
\n";
  print submit();
  print end_form();
  print end_html();
} else {
  ...

This is easier to read and write. Most of the standard html tags are included, and you simply write the tag name in lower-case letters. You can mix actual text tags with these special functions, as we did above with the <hr> tag. The form objects use a different naming convention, and they are a bit more cumbersome. You might find yourself writing some tags in text or multi-line text, and others using these shortcut functions. That is fine, but make sure you develop a style that makes sense. In this book, we will generally only use the shortcut tags where there meaning is obvious.

Debugging with cgiwrap

If your server is using cgiwrap, it also has a special version of the wrapper called cgiwrapd. When you first write your programs, you should use cgiwrapd rather than plain cgiwrap. The main advantage is if you make a syntax error, that error message will be passed on in readable form, rather than a meaningless 500 server error. You will get a printout of several environment variables which you probably will not need, and the source code of any HTML your program produces. Once the code is functioning well, you can take the d out, and the user will see the output of the page rather than the source.

Debugging with cgi.pm cgi.pm also allows you to debug from the command line. You can run a program that uses cgi.pm from the command line, and it will allow you to type in the parameters. You then use control-d (in unix) or control-z (in DOS) to tell perl you are done entering parameters. In this way, you can test a cgi program independant of the forms. Usually, the cgiwrapd is better, but this is a nice technique to know about.

Laboratory assignment

Take the program you wrote in the last unit, and convert it to cgi.
NOTES:

Although the algorithm will not change much, the input and output will change quite a bit.
It is almost always better to start from the algorithm again, rather than trying to just modify the code.
You will need to think very carefully about your input and output screens.
For this project, you will NOT need a seperate HTML page. Make all the code happen in a cgi/perl program.
All input and output should be in HTML pages created by the program

© Andy Harris
Indiana University / Purdue University, Indianapolis
email: aharris@.cs.iupui.edu
homepage: www.cs.iupui.edu/~aharris