ngx-http-log-json: release v0.0.2

nginx: upgrade to nginx 1.11.10 support
module: rename to formal name ngx_http_log_json
module: parse recipe with pcre regex
module: refactoring functions
config: support array values
config: support literal boolean values
config: support literal null values
config: if condition argument
tests: initial unit tests
build: travis integration

Author: fooinha

.gitignore

@@ -0,0 +1,2 @@
+tags
+t/servroot

.travis.yml

@@ -0,0 +1,43 @@
+sudo: required
+dist: trusty
+
+os: linux
+
+language: c
+
+compiler:
+  - gcc
+
+cache:
+  apt: true
+  directories:
+
+env:
+  global:
+    - NGINX_PREFIX=/opt/nginx
+    - JOBS=4
+    - PATH=$PATH:$NGINX_PREFIX/sbin
+
+before_install:
+  - sudo apt-get install -qq -y software-properties-common
+  - sudo add-apt-repository "deb http://us.archive.ubuntu.com/ubuntu/ xenial main universe"
+  - sudo apt-get update -qq -y --fix-missing
+  - sudo apt-get install -qq -y --fix-missing cpanminus librdkafka-dev libjansson-dev mercurial build-essential make clang valgrind libjson-perl
+
+install:
+  - if [ ! -d /opt ]; then mkdir /opt; fi
+  - git clone https://github.com/openresty/test-nginx.git
+  - hg clone http://hg.nginx.org/nginx
+
+script:
+  - cd test-nginx/ && sudo cpanm . && cd ..
+  - sudo cpanm install JSON
+  - cd nginx
+  - auto/configure --with-debug --prefix=$NGINX_PREFIX --add-module=$PWD/.. > build.log 2>&1 || (cat build.log && exit 1)
+  - make -j$JOBS > build.log 2>&1 || (cat build.log && exit 1)
+  - sudo make install > build.log 2>&1 || (cat build.log && exit 1)
+  - cd ..
+  - export PATH=$NGINX_PREFIX/sbin:$PATH
+  - /opt/nginx/sbin/nginx -V
+  - ldd /opt/nginx/sbin/nginx
+  - prove

CHANGES

@@ -1,3 +1,15 @@
+Changes with ngx-http-log-json 0.0.2                                26 Mar 2017
+
+    *) upgrade to nginx 1.11.10 support
+    *) parse recipe with pcre regex
+    *) support literal boolean values
+    *) support literal null values
+    *) support array values
+    *) refactoring functions
+    *) unit tests
+    *) travis script integration
+    *) if= condition argument
+
 Changes with ngx-kasha 0.0.1                                        14 Dec 2016
 
     *) Log output to Kafka topic.

README.md

@@ -1,34 +1,58 @@
-# ngx-kasha
+# ngx-http-log-json  [![Build Status](https://travis-ci.org/fooinha/nginx-http-log-json.svg?branch=master)](https://travis-ci.org/fooinha/nginx-http-log-json)
 
-
-nginx module for advanced per location logging - aka kasha (🍲)
+nginx http module for logging in custom json format - aka kasha (🍲)
 
 ## Description
 
 This module adds to nginx the ability of advanced JSON logging of HTTP requests per location.
+
 It's possible to log to a destination ouput any request made to a specific nginx location.
+
 The output format is configurable.
 
+It also allows to log complex and multi-level JSON documents.
+
+It supports logging to text file or to a kafka topic.
+
+## Use cases
+
+That are many use cases.
+
+Many things can be done by using the access log data.
+
+Having it in JSON format makes easier for integration with other platforms and applications.
+
+A quick example:
+
+![](docs/use-case-kafka-logs.png?raw=true)
+
+
 ### Configuration
 
-Each logging configuration is based on a kasha_recipe. (🍲)
+Each logging configuration is based on a http_log_json_format. (🍲)
 
-A kasha recipe is a ';' separated list of items to include in the logging preparation.
+A http_log_json_format is a ';' separated list of items to include in the logging preparation.
 
 The left hand side part of item will be the JSON Path for the variable name
-The left hand side part can be prefixed with 's:', 'i:' or 'r:', so the JSON encoding type can be controlled.
+The left hand side part can be prefixed with 's:', 'i:', 'r:', 'b:' or 'n:', so, the JSON encoding type can be controlled.
 
 * 's:' - JSON string ( default )
 * 'i:' - JSON integer
 * 'r:' - JSON real
+* 'b:' - JSON boolean
+* 'n:' - JSON null
+
+Additional prefix:
+
+* 'a:' - JSON Array - MUST be used before other prefixes. All keys with same name and defined as array will be its values grouped together in an array. ( see example below )
 
 
 The right hand side will be the variable's name or literal value.
 For this, known or previously setted variables, can be used by using the '$' before name.
 
 Common HTTP nginx builtin variables like $uri, or any other variable set by other handler modules can be used.
 
-The output is sent to the location specified by the first kasha_recipe argument.
+The output is sent to the location specified by the first http_log_json_format argument.
 The possible output locations are:
 
 * "file:" - The logging location will be a local filesystem file.
@@ -40,7 +64,7 @@ The possible output locations are:
 ##### A simple configuration example
 
 ```yaml
-     kasha_recipe file:/tmp/log '
+     http_log_json_format file:/tmp/log '
         src.ip                      $remote_addr;
         src.port                    $remote_port;
         dst.ip                      $server_addr;
@@ -49,12 +73,15 @@ The possible output locations are:
         r:_real                     1.1;
         i:_int                      2016;
         i:_status                   $status;
+        b:_notrack                  false;
         _literal                    root;
         comm.proto                  http;
         comm.http.method            $request_method;
         comm.http.path              $uri;
         comm.http.host              $host;
         comm.http.server_name       $server_name;
+        a:i:list                    1;
+        a:list                      string;
      ';
 ```
 
@@ -68,6 +95,7 @@ To ease reading, it's shown here formatted with newlines.
   "_literal": "root",
   "_real": 1.1,
   "_status": 200,
+  "_notrack": false,
   "comm": {
     "http": {
       "host": "localhost",
@@ -84,7 +112,11 @@ To ease reading, it's shown here formatted with newlines.
   "src": {
     "ip": "127.0.0.1",
     "port": "52136"
-  }
+  },
+  "list": [
+    1,
+    "string"
+  ]
 }
 ```
 
@@ -100,7 +132,7 @@ To ease reading, it's shown here formatted with newlines.
                         return "";
         }';
 
-       kasha_recipe file:/tmp/log '
+       http_log_json_format file:/tmp/log '
         comm.http.server_name       $server_name;
         perl.bar                    $bar;
        ';
@@ -125,7 +157,7 @@ To ease reading, it's shown here formatted with newlines.
 ### Directives
 
 ---
-* Syntax: **kasha_recipe** _location_ { _recipe_ };
+* Syntax: **http_log_json_format** _location_ { _format_ } _if_=...;
 * Default: —
 * Context: http location
 
@@ -139,61 +171,64 @@ For a **file:** type the value part will be a local file name. e.g. **file:**/tm
 
 For a **kafka:** type the value part will be the topic name. e.g. **kafka:** topic
 
-The kafka output only happens if a list of brokers is defined by **kasha_kafka_brokers** directive.
+The kafka output only happens if a list of brokers is defined by **http_log_json_kafka_brokers** directive.
 
-###### _recipe_ ######
+###### _format_ ######
 
 See details above.
 
 ---
 
-* Syntax: **"kasha_kafka_partition** _compression_codec_;
+* Syntax: **"http_log_json_kafka_partition** _partition_;
 * Default: RD_KAFKA_PARTITION_UA
 * Context: http local
 
 ---
 
-* Syntax: **kasha_kafka_brokers** list of brokers separated by spaces;
+* Syntax: **http_log_json_kafka_brokers** list of brokers separated by spaces;
 * Default: —
 * Context: http main
 
 ---
 
-* Syntax: **kasha_kafka_client_id** _id_;
-* Default: kasha
+* Syntax: **http_log_json_kafka_client_id** _id_;
+* Default: http_log_json
 * Context: http main
 
 ---
 
-* Syntax: **"kasha_kafka_compression** _compression_codec_;
+* Syntax: **"http_log_json_kafka_compression** _compression_codec_;
 * Default: snappy
 * Context: http main
 
 ---
 
-* Syntax: **"kasha_kafka_log_level** _numeric_log_level_;
+* Syntax: **"http_log_json_kafka_log_level** _numeric_log_level_;
 * Default: 6
 * Context: http main
 
 ---
 
-* Syntax: **"kasha_kafka_max_retries** _numeric_;
+* Syntax: **"http_log_json_kafka_max_retries** _numeric_;
 * Default: 0
 * Context: http main
 
 ---
 
-* Syntax: **"kasha_kafka_buffer_max_messages** _numeric_;
+* Syntax: **"http_log_json_kafka_buffer_max_messages** _numeric_;
 * Default: 100000
 * Context: http main
 
 ---
 
-* Syntax: **"kasha_kafka_backoff_ms** _numeric_;
+* Syntax: **"http_log_json_kafka_backoff_ms** _numeric_;
 * Default: 10
 * Context: http main
 
 
+###### _if_=... ######
+
+Works the same way as _if_ argument from http [access_log](http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log) directive.
 
 ### Build
 
@@ -212,7 +247,7 @@ $ sudo apt-get install libjansson-dev librdkafka-dev
 Build as a common nginx module.
 
 ```bash
-$ ./configure --add-module=/build/ngx-kasha
+$ ./configure --add-module=/build/ngx-http_log_json
 $ make && make install
 
 ```
@@ -223,7 +258,29 @@ $ make && make install
 
 **THIS IS NOT PRODUCTION** ready.
 
-This was done over the weekend as a proof of concept, and it also lacks unit tests.
-
 So there's no guarantee of success. It most probably blow up when running in real life scenarios.
 
+#### Unit tests
+
+The unit tests use https://github.com/openresty/test-nginx framework.
+
+
+```
+$ git clone https://github.com/openresty/test-nginx.git
+$ cd test-nginx/
+$ cpanm .
+$ export PATH=$PATH:/usr/local/nginx/sbin/
+```
+
+At project root just run the prove command:
+
+```
+$ prove
+
+t/0001_simple_file_log.t .. ok
+All tests successful.
+Files=1, Tests=8,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.15 cusr  0.00 csys =  0.18 CPU)
+Result: PASS
+
+```
+

config

@@ -1,11 +1,11 @@
-ngx_addon_name=ngx_kasha_module
+ngx_addon_name=ngx_http_log_json_module
 ngx_module_incs=$ngx_addon_dir/src
 
-HTTP_MODULES="$HTTP_MODULES ngx_kasha_module"
+HTTP_MODULES="$HTTP_MODULES ngx_http_log_json_module"
 
 CORE_INCS="$CORE_INCS $ngx_module_incs"
 NGX_ADDON_SRCS="$NGX_ADDON_SRCS \
- $ngx_addon_dir/src/ngx_kasha.c \
- $ngx_addon_dir/src/ngx_kasha_kafka.c \
- $ngx_addon_dir/src/ngx_kasha_str.c"
+ $ngx_addon_dir/src/ngx_http_log_json_module.c \
+ $ngx_addon_dir/src/ngx_http_log_json_kafka.c \
+ $ngx_addon_dir/src/ngx_http_log_json_str.c"
 CORE_LIBS="$CORE_LIBS -ljansson -lrdkafka"