Web Services Overview

A web service is just a machine -- the server -- connected to the Internet, running a computer program that listens to HTTP requests from client programs, such as browsers or other web servers.

For our purposes, the difference between a web service and a web server is that a web service sends data, while a web server sends HTML.

We'll limit ourselves here to web services that send data to clients using JSON (JavaScript Object Notation). JSON is a relatively readable, compact notation for data containing arbitrarily nested numbers, strings, arrays, and key-value tables, as shown in these examples.

Note that JSON is more restricted than JavaScript objects. For example, { name: "John", age: 10 } is fine in JavaScript, but in JSON you have to write { "name": "John", "age": 10 }.

Web services are, in my opinion, a great way to go, even when implementing a web site. That's the focus on this page.

Web Sites with Web Services

When the web began, it had a very simple architecture: clients, e.g., browsers, sent URL's to servers and servers returned HTML files.

static html

A major factor in the expansion of the web into all kinds of areas was the fact that a client program doesn't need to where or how a server got the HTML it sent out. So the web soon expanded to include HTML dynamically constructed by applying HTML templates to data.

server-side dynamic html

In the past few years, there's been an additional decoupling, by having servers send data separately from the HTML. This turns the servers into more general web services. It also allows web design to be done purely in web technologies, i.e., HTML, CSS and JavaScript.

client-side dynamic html

A number of very powerful JavaScript libraries have been built for generating HTML from templates on the client side, getting data as needed from the server via AJAX calls.

Lisp client-side technologies

The disadvantages of doing this include:

The advantages are many, however, including:

Lisp Technologies for Web Services

There are a number of Lisp web servers. In this course, we recommend Franz, Inc.'s AServe library for Allegro and Lispworks users, because:

For those students not using Allegro, there is a portable AServe library that runs in a number of Common Lisp's. I use it in LispWorks Personal Edition. It's loaded automatically by the course-provided initialization file using Quicklisp.

For SBCL users, where portable Aserve appears not to work, at least on Windows or MacOS, you can use Hunchentoot.

The EECS 325 Simple Server

Much of the documentation for AllegroServe, Hunchentoot, and other Lisp web servers focuses on how to generate HTML in Lisp. This is irrelevant to us. The only HTML we're going to send will be static HTML assets, i.e., HTML, CSS, JavaScript, image files, etc. All the dynamic data will be sent using JSON.

To focus on this, I have developed the Simple Server framework for AllegroServe and Hunchentoot. It supports just the few operations that we need to do single-page web apps.

Installation

To install Simple Server,

Loading into Lisp

To use in Lisp, do (ql:quickload "simple-server"). By default, this uses AllegroServe for Allegro and Lispworks, and Hunchentoot for SBCL. If you wish to change this, edit the file simple-server.asd.

Hunchentoot only: Hunchentoot tries to load the openSSL library. If you don't have it, loading Simple Server will get an error about not finding libSSL. You can either install openSSL, or just select the option in the error display to act as if the file cl+ssl loaded successfully. This will mean you can't run the server with HTTPS requests. SSL and HTTPS are beyond the scope of this course.

The Simple Server API

The Simple Server API is deliberately very small, with little configuration. To create a JSON-based web app:

Here's a picture of how the directory folders are laid out:

To load and run a web app:

The default port is 8000. You can specify a different port, e.g., (simple-server:start-server :port 8001). You can have servers running on several ports. You can stop a specific server with (simpler-server:stop-server :port 8001). If no server is given, all are stopped.

It's clearest to put your web app's home page and ASDF file at the top level of your web app's folder. That will make for the simplest URL and an easy place to see what code the web app needs. You can organize all the other files however you wish.

In your Lisp, put as much as you can in regular code that knows nothing about web services or even JSON. Write small wrapper functions to decode URL data, call normal Lisp code, then encode the Lisp results as a JSON string. Use defroute, described below, to map URLs to those wrapper functions.

(defroute http-method url function) [macro]

This macro specifies what function to call when a client sends url via http-method. The relevant HTTP methods are :get, :put, or :post. Url is a string path. If you want to run multiple apps, use URLs that include the the app name, e.g., "/roman/encode".

Function should evaluate to a function object or a function name. The function will be called with two arguments: data and params. data will be a string with JSON data if a web page did an AJAX call with PUT or POST. params will be an alist of the request query parameters, if any, for a GET or form POST.

A handy utility function defined by SimpleServer is param-value.

(param-value key alist &optional default) [function]
This function returns the value for key in alist, if present, else default, if given, else NIL. The lookup uses a case-insensitive string match, so (param-value "number" ...) and (param-value :number ...) do the same thing.

Example Web App: The JSON Demo Server

The JSON demo code shows how to build a JSON-based web app with Simple Server.

To try the demo:

> (ql:quickload "json-demo")
...
> (in-package #:json-demo)
...
> (start-server)

Assuming there were no errors, point your browser at http://localhost:8000/json-demo/demo.html. A form should appear with several simple examples that use JavaScript to communicate with the server.

If you get an error that start-server is not defined, you probably forgot to do the in-package.

Some notes on the demo code in the file json-demo.lisp follow.

Location

The JSON demo is in the folder ~/quicklisp/local-projects/simple-server/apps/json-demo. It has a home page demo.html in the top level of that folder, so the URL to get to the home page is http://localhost:8000/json-demo/demo.html.

SAVE-DATA example

(defroute :post "/save-data" 'save-data)

This makes any HTTP POST to the URL /save-data call the function SAVE-DATA. We use a function name, rather than a LAMBDA. In general, the best way to define routes is with a named function, so that you can trace the function in case of errors.

The body of this POST request from a browser should be a JSON object of the form

{ key: value }

SAVE-DATA saves the key and value in an alist on the server and return a JSON object with all the saved keys and values.

(defun save-data (data params)
  (encode-json-alist-to-string (update-alist (get-json-from-string data))))

(defun get-json-from-string (str)
  (and str (stringp str) (> (length str) 0)
       (decode-json-from-string str)))

get-json is typical code for parsing a string to get a JSON object. You need to make sure the string is not empty, or a JSON parsing error will occur. get-json uses the CL-JSON function decode-json-from-string to parse the JSON string into an equivalent alist. You can use any JSON library if you wish. There's a good comparison of them here.

update-alist, defined in the demo code, merges an alist with an alist stored in an internal special variable. It returns the merged result. By design, update-alist is a normal Lisp function that knows nothing about JSON.

The returned alist is then encoded by the CL-JSON function encode-json-alist-to-string, and returned to the browser.

EXPORT example

(defroute :get "/exports" 'exports)

This makes any HTTP GET to the URL /exports call the EXPORTS function. The full URL should have the form /exports?name=...&package=.... The package is optional, and defaults to COMMON-LISP.

(defun exports (data params)
  (let ((name (param-value "name" params ""))
        (package (param-value "package" params "COMMON-LISP")))
    (encode-json-plist-to-string 
     (list :name name :package package
           :symbols (exports-list name package)))))

This shows typical code for getting data from URL request parameters.

The function EXPORTS-LIST is defined in the demo code. It is a normal Lisp function that knows nothing about JSON. It takes two strings and returns a list of those Lisp symbols that package exports that contain name as a substring.

The code above creates a plist with this data for encoding in the equivalent JSON object to return to the client. It could have just as easily created an alist.

Debugging web apps

In most systems, it's a little tricky to debug web applications because they run as separate processes, and error messages and trace output are often sent to a console stream, not to your Lisp Listener window.

To see error and trace output that occur in web request handling

See all my web tips page.

Faculty: Chris Riesbeck
Time: Monday, Wednesday, Friday: 1pm - 2pm
Location:Annenberg G15

Contents

Important Links