Web Services Overview

A web service is just a machine -- the server -- connected to the Internet, running a computer program that listens to HTTP requests from client programs, such as browsers or other web servers.

For our purposes, the difference between a web service and a web server is that a web service sends data, while a web server sends HTML.

We'll limit ourselves here to web services that send data to clients using JSON (JavaScript Object Notation). JSON is a relatively readable, compact notation for data containing arbitrarily nested numbers, strings, arrays, and key-value tables, as shown in these examples.

Note that JSON is more restricted than JavaScript objects. For example, { name: "John", age: 10 } is fine in JavaScript, but in JSON you have to write { "name": "John", "age": 10 }.

Web services are, in my opinion, a great way to go, even when implementing a web site. That's the focus on this page.

See my web tips page for some technical notes on web services.

Web Sites with Web Services

When the web began, it had a very simple architecture: clients, e.g., browsers, sent URL's to servers and servers returned HTML files.

static html

A major factor in the expansion of the web into all kinds of areas was the fact that a client program doesn't need to where or how a server got the HTML it sent out. So the web soon expanded to include HTML dynamically constructed by applying HTML templates to data.

server-side dynamic html

In the past few years, there's been an additional decoupling, by having servers send data separately from the HTML. The client requests data via AJAX calls.

This turns the web servers into more general web services. It also allows web design to be done purely in web technologies, i.e., HTML, CSS, JavaScript, and the many modern frameworks, such as Angular, React, and Vue.

Lisp client-side technologies

The disadvantages of doing this include:

The advantages are many, however, including:

Lisp Technologies for Web Services

There are a number of Lisp web servers. One is Franz, Inc.'s AServe library for Allegro and Lispworks users. Students using Allegro already have AServe available and ready to load.

Another popular server is Hunchentoot. This is commonly used with SBCL and LispWorks.

These provide the basis for implementing Lisp-based web sites. But to avoid writing the same boilerplate code over and over, a web services framework can be helpful.

The 325 Simple Server

Much of the documentation for AllegroServe, Hunchentoot, and other Lisp web servers focuses on how to generate HTML in Lisp. This is irrelevant to us. The only HTML we're going to send will be static HTML assets, i.e., HTML, CSS, JavaScript, image files, etc. All the dynamic data will be sent using JSON.

To focus on this, I have developed the Simple Server framework for AllegroServe and Hunchentoot. It supports static assets and passing JSON data. The in-class quizzes and Boggle server are implemented as Simple Server apps.


To install Simple Server,

Loading into Lisp

To use in Lisp, do (ql:quickload "simple-server"). By default, this uses AllegroServe for Allegro, and Hunchentoot for SBCL and Lispworks. If you wish to change this, edit the file simple-server.asd.

Hunchentoot only: Hunchentoot tries to load the openSSL library. If you don't have it, loading Simple Server will get an error about not finding libSSL. You can either install openSSL, or just select the option in the error display to act as if the file cl+ssl loaded successfully. This will mean you can't run the server with HTTPS requests. SSL and HTTPS are beyond the scope of this course.

The Simple Server API

The Simple Server API is deliberately very small, with little configuration. To create a JSON-based web app:

Here's a picture of how the directory folders are laid out:

For a typical app, you will want at least two Lisp files:

For example, suppose you wanted to build a service to convert numbers to Roman numerals or English text, using CL-JSON to generate JSON from lists. (Hint: Number conversion is a snap in Lisp.) You might create the following:

To load and run a web app:

The default port is 8000. You can specify a different port, e.g., (simple-server:start-server :port 8001). You can have servers running on several ports. You can stop a specific server with (simpler-server:stop-server :port 8001). If no server is given, all are stopped.

It's clearest to put your web app's home page and ASDF file at the top level of your web app's folder. That will make for the simplest URL and an easy place to see what code the web app needs. You can organize all the other files however you wish.

In your Lisp, put as much as you can in regular code that knows nothing about web services or even JSON. Write small wrapper functions to decode URL data, call normal Lisp code, then encode the Lisp results as a JSON string. Use defroute, described below, to map URLs to those wrapper functions.

(defroute http-method url function) [macro]

This macro specifies what function to call when a client sends url via http-method. The relevant HTTP methods are :get, :put, or :post. Url is a string path. If you want to run multiple apps, use URLs that include the the app name, e.g., "/numbers/to-roman".

Function should evaluate to a function object or a function name. The function will be called with two arguments: data and params. data will be a string with JSON data if a web page did an AJAX call with PUT or POST. params will be an alist of the request query parameters, if any, for a GET or form POST.

A handy utility function defined by SimpleServer is param-value.

(param-value key alist &optional default) [function]
This function returns the value for key in alist, if present, else default, if given, else NIL. The lookup uses a case-insensitive string match, so (param-value "number" ...) and (param-value :number ...) do the same thing.

Example Web App: The JSON Demo Server

The JSON demo code shows how to build a JSON-based web app with Simple Server.

To try the demo:

> (ql:quickload "json-demo")
> (in-package #:json-demo)
> (start-server)

Assuming there were no errors, point your browser at http://localhost:8000/json-demo/demo.html. A form should appear with several simple examples that use JavaScript to communicate with the server.

If you get an error that start-server is not defined, you probably forgot to do the in-package.

Some notes on the demo code in the file json-demo.lisp follow.


The JSON demo is in the folder ~/quicklisp/local-projects/simple-server/apps/json-demo. It has a home page demo.html in the top level of that folder, so the URL to get to the home page is http://localhost:8000/json-demo/demo.html.

SAVE-DATA example

(defroute :post "/save-data" 'save-data)

This makes any HTTP POST to the URL /save-data call the function SAVE-DATA. We use a function name, rather than a LAMBDA. In general, the best way to define routes is with a named function, so that you can trace the function in case of errors.

The body of this POST request from a browser should be a JSON object of the form

{ key: value }

SAVE-DATA saves the key and value in an alist on the server and return a JSON object with all the saved keys and values.

(defun save-data (data params)
  (encode-json-alist-to-string (update-alist (get-json-from-string data))))

(defun get-json-from-string (str)
  (and str (stringp str) (> (length str) 0)
       (decode-json-from-string str)))

get-json is typical code for parsing a string to get a JSON object. You need to make sure the string is not empty, or a JSON parsing error will occur. get-json uses the CL-JSON function decode-json-from-string to parse the JSON string into an equivalent alist. You can use any JSON library if you wish. There's a good comparison of them here.

update-alist, defined in the demo code, merges an alist with an alist stored in an internal special variable. It returns the merged result. By design, update-alist is a normal Lisp function that knows nothing about JSON.

The returned alist is then encoded by the CL-JSON function encode-json-alist-to-string, and returned to the browser.

EXPORT example

(defroute :get "/exports" 'exports)

This makes any HTTP GET to the URL /exports call the EXPORTS function. The full URL should have the form /exports?name=...&package=.... The package is optional, and defaults to COMMON-LISP.

(defun exports (data params)
  (let ((name (param-value "name" params ""))
        (package (param-value "package" params "COMMON-LISP")))
     (list :name name :package package
           :symbols (exports-list name package)))))

This shows typical code for getting data from URL request parameters.

The function EXPORTS-LIST is defined in the demo code. It is a normal Lisp function that knows nothing about JSON. It takes two strings and returns a list of those Lisp symbols that package exports that contain name as a substring.

The code above creates a plist with this data for encoding in the equivalent JSON object to return to the client. It could have just as easily created an alist.

Faculty: Chris Riesbeck
Time: Monday, Wednesday, Friday: 11am - 11:50am
Location: Tech LR5


Important Links