Making HTTP requests with Lisp
There are several libraries for making HTTP calls to web servers, to get HTML, XML, or JSON. We will use Drakma.
See this page for a quick intro, with links, to JSON, web services, and CL-JSON. Don't worry about the later parts on SimpleServer yet.
Install Open SSL
You need to have SSL working in Lisp in order to access HTTPS URLs.
On Windows, that means you need to have OpenSSL installed.
should work. For our purposes, you can skip the part at the end of those instructions about
This should be easy.
Get data with URLs in Drakma is simple:
url needs to be a string containing a complete URL. Drakma will send a request for that URL to the server indicated, just as a browser or other user client would.
drakma:http-request returns seven values, as
Lisp multiple values. If you need to save or use
values other than the first, use
multiple-value-bind or a similar form. The values
returned, in order, are
- the body of the reply, either a string, when getting HTML or plain text, a binary array for images, audio, and JSON, or a file stream, if requested using a keyword parameter
- the HTTP status code
- an alist of the headers sent by the server
- the URI the reply comes from, which might be different than the request when redirects occur
- the stream the reply was read from
- a boolean indicating whether the stream should be closed
- the server's status text, e.g., "OK"
If the URL is for an HTML file, like this:
you will see a string containing the raw HTML.
If the URL is for data not labelled as text, including a call for JSON, like this:
you will get an array of byte codes.
Allegro CL Express: If you get an "SSL not supported" error, replace two files in
~/quicklisp/dists/quicklisp/software/drakma-2.0.2/with their current versions in the Drakma repository: drakma.asd and util.lisp.
An array of byte codes can be converted to a string using normal Lisp functions, e.g.,
(map 'string 'code-char (drakma:http-request "https://docs-examples.firebaseio.com/rest/quickstart/users.json"))
A better way, that doesn't create an intermediate string, is used in the testing below.
Yet another approach is to tell Drakma that HTTP response content labeled application/json is plain text. You can do that with this:
(push (cons "application" "json") drakma:*text-content-types*)
Test getting and parsing HTML
If you are not using Allegro Lisp, get the open source version of the HTML parser using QuickLisp.
(net.html.parser:parse-html (drakma:http-request ...))
to test parsing the HTML returned by the URL you tried before. You should get back a list version of the page, i.e., a version you can process with standard Lisp list functions.
If you want to have the HTTP client and HTML parser available whenever you start Lisp, put the
quickloadcommands above into your
Testing getting and parsing JSON
Get the CL-JSON parser.
Now test getting parsed JSON using the Drakma JSON example code. Change it to use the CL-JSON parser instead of the YASON parser.
Test on at least two different JSON sites. See here for some simple examples.
Test. You should get back a list version of the JSON data, similar to the one shown in the example using the YASON parser.