Let's try RESTler on a simple example of REST API and service.
The directory restler\demo_server in this repo contains a self-contained example of service you can run locally on your machine. See the README.md file in that directory to install and start this demo_server service. Make sure your demo server is running before starting this tutorial.
Let's create a fresh directory C:\demo-server-test where will run RESTler on this example. Let's cd into that dir: cd C:\demo-server-test.
Let's copy the Swagger spec swagger.json from restler\demo_server into C:\demo-server-test and let's compile it with RESTler:
C:\restler_bin\restler\Restler.exe compile --api_spec C:\demo-server-test\swagger.json
This command creates a new sub-directory Compile where the results of the compilation are saved in several files:
grammar.pyandgrammar.jsonare the RESTler grammarsdict.jsonis a default dictionary associating parameters types with a set of default values (you can edit this file if you want more or different values)engine_settings.jsonis a default engine_settings file with options for the RESTler test engine (you can edit this file to turn on other options)config.jsonis a default compiler configuration file (you can edit this file and re-run the compiler with other options)
For this simple tutorial, we will use the default values automatically generated by the RESTler compiler and move directly to the next step.
Let's run RESTler in test mode to see what specification coverage we get with this default RESTler grammar:
C:\restler_bin\restler\restler.exe test --grammar_file C:\demo-server-test\Compile\grammar.py --dictionary_file C:\demo-server-test\Compile\dict.json --settings C:\demo-server-test\Compile\engine_settings.json --no_ssl
(For help, run C:\restler_bin\restler\restler.exe --help)
The results are saved in a new sub-directory Test. In the sub-directory C:\demo-server-test\Test\RestlerResults\experiment<...>\logs, we see a file named main.txt which lists one-by-one all the API requests in the Swagger spec and whether the request has been succesfully executed (VALID) or not (INVALID). A VALID request means RESTler was able to execute the request and get a 20x HTTP status code as a response. The total spec coverage is given at the bottom of that file:
Final Swagger spec coverage: 5 / 6
The exact requests executed by RESTler and the service response are all logged in the file network.testing.<...>.txt.
The file speccov.json is a machine-readable version of main.txt, which is easier to parse by other tools (for instance, for regression testing and automatically comparing coverage data).
RESTler has a garbage-collector (gc) that attempts to delete all resources ever created by RESTler during a test run.
The garbage-collector logs are in the file network.gc.<...>.txt and the raw HTTP/S traffic from the gc is in garbage_collector.gc.<...>.txt.
In this example, coverage is 5 / 6 and the only INVALID request is
2022-06-24 11:32:24.360: Rendering INVALID - restler_static_string: 'GET ' - restler_static_string: '' - restler_static_string: '/' - restler_static_string: 'api' - restler_static_string: '/' - restler_static_string: 'blog' - restler_static_string: '/' - restler_static_string: 'posts' - restler_static_string: '?' - restler_static_string: 'page=' - restler_fuzzable_int: '1' - restler_static_string: '&' - restler_static_string: 'per_page=' - restler_fuzzable_int: '1' - restler_static_string: ' HTTP/1.1\r\n' - restler_static_string: 'Accept: application/json\r\n' - restler_static_string: 'Host: localhost\r\n' - restler_static_string: '\r\n'
By looking at network.testing.<...>.txt, we can see that RESTler attempts to execute this request with a value 1 for the per_page= and page=. It turns out that the per_page= must be 2 minimally, but RESTler was not able to infer this automatically. (One way to fix this is to edit dict.json and add the value 2 in the list for restler_fuzzable_int.)
Let's now try to run restler in Fuzz-lean mode.
C:\restler_bin\restler\restler.exe fuzz-lean --grammar_file C:\demo-server-test\Compile\grammar.py --dictionary_file C:\demo-server-test\Compile\dict.json --settings C:\demo-server-test\Compile\engine_settings.json --no_ssl
The results are in a new FuzzLean directory and the experiment results can be found in C:\demo-server-test\FuzzLean\RestlerResults\experiment<...>\. The logs\ directory should contain the same coverage results as the previous Test run, but you should also now see a new bug_buckets directory.
Inside the bug_buckets directory there should be four files:
- bug_buckets.txt
- InvalidDynamicObjectChecker_20x: 2
- PayloadBodyChecker_500: 2
- UseAfterFreeChecker_20x: 1
The bug_buckets.txt file contains a list of each unique sequence that found a bug. Any request that receives a '500' status code as a response will be treated as a bug. Additionally, some checkers also record bugs if an unexpected '20x' status code is received. Bugs are bucketized by only including one bug per unique request sequence even if that same sequence produces more than one bug throughout the fuzzing run. The exception to this is if a new bug was detected due to a different status code, e.g. the first bug was a 500 and the second was a 503.
Each unique sequence in the bug_buckets.txt file also has a corresponding individual log associated with that bug. This log shows the sequence of requests and their responses exactly as they were sent and received from the server. These particular bugs were planted in the demo server for this example.
Looking at the InvalidDynamicObjectChecker_20x_1.txt log you can see that the sequence of requests includes a POST request that creates a blog post and a GET request that attempts to get the post that was just created. Here the Invalid Dynamic Object Checker replaced the blog post id of 14 with 14?injected_query_string=123, which was accepted by the demo server because the unexpected query string of ?injected_query_string=123 was ignored, which resulted in a 200 response code. Because the Invalid Dynamic Object Checker assumes that any 'invalid dynamic object' should cause the request to fail (not return a 20x status code), it flags this as a bug.
Next, the PayloadBodyChecker_500_1.txt file contains a sequence that includes a POST request followed by a PUT request. The body used during this fuzz can be seen in the sequence log, but, for Payload Body Checker bugs only, you can also find the body used to fuzz the request at the top of the log file in the header. In this case, the body was {"body":"my first blog post"}.
Right above the body in the header you should also see StructMissing_/id/checksum. This is a Payload Body Checker specific tag that helps identify how the body was fuzzed in order to help identify what could have caused the bug in the service. What this particular tag is telling us is that the body was fuzzed by removing a piece of the body's structure (StructMissing) and the pieces that were missing were 'id' and 'checksum'. Because the bug was planted, we know that this bug was, in fact, triggered by the missing 'id' field in the body.
Finally, the UseAfterFreeChecker_20x_1.txt file contains a POST followed by a DELETE, then a GET. The Use After Free checker detected that a resource (in this case, blog post with ID 24) was successfully deleted, but could still be accessed after deletion. In this case, the cause of the failure is a planted bug in the GET validation logic.
Now let's try to fuzz:
C:\restler_bin\restler\restler.exe fuzz --grammar_file C:\demo-server-test\Compile\grammar.py --dictionary_file C:\demo-server-test\Compile\dict.json --settings C:\demo-server-test\Compile\engine_settings.json --no_ssl --time_budget 1
The new time_budget parameter is the length, in hours, to perform the fuzzing run.
The results are in a new Fuzz directory and the experiment results can be found in C:\demo-server-test\Fuzz\RestlerResults\experiment<...>\.
Like the FuzzLean experiment above, you will see three unique bugs in bug_buckets\bug_buckets.txt as well as their corresponding bug logs.
If you were to closely inspect the logs\network.testing.####.1.txt file
you may notice that there were several more instances where these same bugs were triggered.
Because RESTler bucketizes the bugs based on request type and response code,
the duplicate bugs were not reported.
Because the demo server is a very small and uncomplicated API, the one hour fuzz was unable to detect any additional unique bugs that FuzzLean did not already find.