The goal of the Bug Explainer is to provide a web service for novice programmers encountering bugs in their code when learning a new language. Novices don't know what's going wrong, and they often describe their problems in vague or invalid terms, e.g., "member doesn't work!" Hence their attempts to find answers via Google or FAQ searches often fail to find relevant answers.
To get around the ambiguity problem,
- the front-end lets novices enter concrete details of their problem, rather than their (often incorrect) interpretation of the problem
- the back-end searches for matching previously explained bugs
This is called case-based explanation, one of many applications of case-based reasoning (CBR).
Tasks
Bug Analyzer
The analyzer needs to look at the bug report and
find possible explanations for the bug. The clues to
common bugs can be subtle. For example, novices often
report that deleting doesn't work, and their example
involves
deleting the first element of a list. But another
reason it doesn't work is because they're trying to delete
a list and they're not telling DELETE
to use
EQUAL
. And yet another possibility is that they
use REMOVE
and didn't realize that REMOVE
doesn't modify the list.
If we think about how a Lisp expert understands these bugs, there are two parts. First, the expert has lots of explanations for common bugs. Some are specific to a single function or data type, others are more general. This is the kind of thing that frame hierarchies are good at representing.
Second, the expert has rules for analyzing the user's report for specific clues that a particular explanation may apply. This is where pattern matching works well. So let's build a system where the user's report is analyzed by a rule matcher, which produces a set of observations, which are then used to find explanations.
The standard way to look for such clues is pattern matching. The matcher used in the Lisp Critic is a good fit, since it's been tested on matching many Lisp code constructs. One pattern that recognizes the "deleting first element" situation is:
Input | (?CONTAINS (DELETE ?*)) |
---|---|
Expected | (?X ?*) |
Actual | (?Y ?X ?*) |
Note the use of variables across several fields of the user input.
Another common mistake is using any function that searches a container,
and searching for a list but not giving an appropriate :test
parameter. We could write a pattern like this:
Input | (?CONTAINS ((?OR ASSOC DELETE FIND MEMBER ...) ? ?)) |
---|
But there are a lot of such functions, and new ones might be defined in a library. A better way would be to create a concept in our memory for that subclass of functions that do equality testing, organize the relevant functions under that concept, and then use that concept in the pattern.
Input | (?CONTAINS ((?IS ISA-P EQUALITY-USING-FUNCTION) ? ?)) |
---|
The goal of the pattern matching is to generate one or more descriptions of the code in the form of frames. Then we can search for bug reports that explain bugs in similar pieces of code.
A bug report would be something like this:
Code | (delete-call-15 (delete-call) (target-position 0)) |
---|---|
Explanation | Although DELETE can return a list without the first element, the structure of Lisp lists make it impossible to destructively remove the first element of a list. For more, see http://www.cs.cmu.edu/Groups/AI/html/faqs/lang/lisp/part3/faq-doc-4.html |
Coding Tasks
Create a set of frames
- Define a macro
DEFBUG
such that (defbug name pattern text) defines a bug explanation rule. Each rule has an internal name, a pattern to match against the parts of a bug report, and a text string with a brief explanation of the problem. The string might contain HTML, specifically a link to the Lisp FAQ or other useful pages. - Define at least the two DELETE bug explanations referred to above.
- Use
DEFINE-TEST
to define some test cases for the (undefined!) functionGET-EXPLANATIONS
that verify that (get-explanations bug-report) returns the correct explanation rules for some input patterns. Be sure to test for inputs that match 0, 1 or several rules.
Verify that the tests fail. - Define a function
GET-EXPLANATIONS
and get it to pass your test cases.. - Modify your your bug explanation page
to use
GET-EXPLANATIONS
to find and display the texts of any explanation rules that apply.
DEFBUG
is similar to DEFINE-LISP-PATTERN
used in defining Lisp Critic rules in
lisp-rules.lisp. You need
some format that identifies which fields are being matched against, i.e.,
input versus expected versus actual.
GET-EXPLANATIONS
should return rules, not just the text strings.
There's no additional work to do this, and it leaves your options
open for extensions, such as editing rules in the next task.
Submit: the Webactions functions you defined to implement to create and store the bug reports. Do not submit the CLP pages, but do submit functions called by CLP tags to display the bug reports, if any were defined.
Bug Report Interface
The input interface is where the end user specifies a problem. The interface needs to be easy to use and not require any expert knowledge from the user.
Use SimpleServer to create a simple app with a form that lets a user specify the following information:
Summary | A short text summary of bug. Not useful for the computer. |
---|---|
Input | Sample code that produces the bug |
Expected | the result(s) the user expected to see |
Actual | the result(s) the user actually saw |
Operating System | the operating system the user is on |
Language | the language, including implementation version the user has |
Here are two example bug reports. I'm leaving out the summaries and platform information, since they're not important here:
Input | (LET ((L '(A B C))) (DELETE 'A L) L) |
---|---|
Expected | (B C) |
Actual | (A B C) |
Input | (DELETE '(B) '((A) (B) (C)) |
---|---|
Expected | ((A) (C)) |
Actual | ((A) (B) (C)) |
Clicking submit on the form should send an AJAX request to the server with the form data. The server should search its library of existing bug reports and return in JSON form a list of matching cases, most similar ones first.
When the data is returned, the cases should be displayed below the user input.
Implement and test with at least the two examples above.
Submit: The server-side code for processing and responding to the request, and the ontology for explained bug reports.
Bug Rule Editor
Modify your your bug explanation page to allow people to edit an existing explanation rule, or to add a new one. The exact interface is up to you, but either option should open the same rule editing form, to let people change the pattern and explanation text, but not the name.
Submit: the Webactions functions you defined to implement to back-end editing and updating. Do not submit the CLP pages, but do submit functions called by CLP tags to display the bug reports, if any were defined.