Making HTTP requests with Lisp
There are several libraries for making HTTP calls to web servers, to get HTML, XML, or JSON. We will use Drakma.
See this page for a quick intro, with links, to JSON, web services, and CL-JSON. Don't worry about the later parts on SimpleServer yet.
Install Open SSL
You need to have SSL working in Lisp in order to access HTTPS URLs. On Windows, that means you need to have OpenSSL installed. These instructions should work. Some notes:
- You can ignore the instructions about
changing the
PATHvariable. We don't use the OpenSSL client program, just the library DLLs. - Choose the option to install into the Windows directories.
- For Lispworks Personal 6.0, choose Win32 1.0.2 Light. Do not use 1.1.
- For SBCL, choose Win64 1.1 Light.
Note: both the Win32 and Win64 versions can be installed if you want to run several Lisps.
If you get an error in Lisp about a missing SSL library, check your Windows directories for the files libssl32.dll and
libeay32.dll.
- Lispworks: look in
C:\Windows\SysWoW64\ - SBCL: look in
C:\Windows\system32\
Install Drakma
This should be easy.
(ql:quickload "drakma")
Using Drakma
Get data with URLs in Drakma is simple:
(drakma:http-request url)
url needs to be a string containing a complete URL. Drakma will send a request for that URL to the server indicated, just as a browser or other user client would.
drakma:http-request returns seven values, as
Lisp multiple values. If you need to save or use
values other than the first, use
multiple-value-bind or a similar form. The values
returned, in order, are
- the body of the reply, either a string, when getting HTML or plain text, a binary array for images, audio, and JSON, or a file stream, if requested using a keyword parameter
- the HTTP status code
- an alist of the headers sent by the server
- the URI the reply comes from, which might be different than the request when redirects occur
- the stream the reply was read from
- a boolean indicating whether the stream should be closed
- the server's status text, e.g., "OK"
Examples Calls
If the URL is for an HTML file, like this:
(drakma:http-request "http://www.lisp.org/index.html")
you will see a string containing the raw HTML.
If the URL is for data not labelled as text, including a call for JSON, like this:
(drakma:http-request "https://docs-examples.firebaseio.com/rest/quickstart/users.json")
you will get an array of byte codes.
Allegro CL Express:
If you get an "SSL not supported" error, replace
two files in ~/quicklisp/dists/quicklisp/software/drakma-2.0.2/
with their current versions in the Drakma repository:
drakma.asd
and
util.lisp.
An array of byte codes can be converted to a string using normal Lisp functions, e.g.,
(map 'string 'code-char (drakma:http-request "https://docs-examples.firebaseio.com/rest/quickstart/users.json"))
A better way, that doesn't create an intermediate string, is used in the testing below.
Yet another approach is to tell Drakma that HTTP response content labeled application/json is plain text. You can do that with this:
(push (cons "application" "json") drakma:*text-content-types*)
Test getting and parsing HTML
If you are not using Allegro Lisp, get the open source version of the HTML parser using QuickLisp.
(ql:quickload "cl-html-parse")
Now do
(net.html.parser:parse-html (drakma:http-request ...))
to test parsing the HTML returned by the URL you tried before. You should get back a list version of the page, i.e., a version you can process with standard Lisp list functions.
If you want to have the HTTP client and HTML parser available whenever you start Lisp, put thequickloadcommands above into yourcs325.lispfile.
Testing getting and parsing JSON
Get the CL-JSON parser.
(ql:quickload "cl-json")
Now test getting parsed JSON using the Drakma JSON example code. Change it to use the CL-JSON parser instead of the YASON parser.
Test on at least two different JSON sites. See here for some simple examples.
Test. You should get back a list version of the JSON data, similar to the one shown in the example using the YASON parser.