Use modern Clojure for this task.

Author: davejacobs

.gitignore

@@ -1,21 +1,23 @@
 Background
 ==========
 
-Tgrep is a commandline utility meant to search log files per Reddit's specifications [0].
+`tgrep` is a commandline utility meant to search 100 GB log files.
 
-Setup
-=====
+### Prerequisites ###
 
-Tgrep is written in Clojure and, as such, requires a JVM and the Clojure language [1] to be installed on your computer. What's more, it manages its project dependencies using Leiningen [2], so lein is required to run the utility, as well. All other libraries are included with the executable.
+`tgrep` is written in Clojure and, as such, requires a JVM to be installed on your computer. Leiningen is required to install Clojure and library dependencies.
 
-Ruby is used instead of Bash to actually parse arguments in the executable and then run the Clojure code as a script. Any version of Ruby after 1.8.6 should do.
+### Running ###
 
-To run the executable, first give it permission to execute via chmod +x bin/tgrep.
+To run:
 
-Then run the program (bin/tgrep) according to the Reddit specifications. /log/haproxy.log is the default file, if none other is specified.
+    lein search 00:05
+    lein search -f logfile 00:05
+    lein search -f logfile 00:05-00:10
+    lein search -f logfile 00:05:01-00:10
+    lein search -f logfile 00:05:01-00:10:01
 
-Edge cases
-==========
+### Edge cases ###
 
 There were several different potential edge cases and bugs that could've cropped up with this utility. These are mainly the product of:
 
@@ -29,18 +31,19 @@ There were several different potential edge cases and bugs that could've cropped
    a 24 hour timepoint. This means that a specified date interval range
    might appear twice in a log file.
 
-I've tried to remedy edge cases up front by normalizing all incoming dates
-and immediately turning them into intervals. Any precise values simply turn into one boundary for the time interval we're looking at, and if we're only looking for one timestamp, I simply make an interval out of two identical date values.
-
-To combat (3), I simply decided that the first range within the valid bounds of the log file would be the target range.
-
-Performance
-===========
-
-One note about performance. This code runs as fast as vanilla Java, and it will self-optimize the more you run it. One drawback about using the JVM is that the VM must load before the script can execute. I've talked with jedberg about this problem, and he said that the JVM load time would not be an issue. To give you an indication of the actual run time of my script, I've included an "Elapsed time" tag after the script runs. To mediate the JVM load time, a nailgun Java server (which gives you a persistent JVM) may also be used.
+I've tried to remedy edge cases up front by normalizing all incoming
+dates and immediately turning them into intervals. Any precise values
+simply turn into one boundary for the time interval we're looking at,
+and if we're only looking for one timestamp, I simply make an interval
+out of two identical date values.
 
-[0] http://www.reddit.com/r/blog/comments/fjgit/reddit_is_doubling_the_size_of_its_programming/
+To combat (3), I simply decided that the first range within the valid
+bounds of the log file would be the target range.
 
-[1] http://clojure.org/getting_started
+### Performance ###
 
-[2] https://github.com/technomancy/leiningen
+This code runs nearly as fast as vanilla Java, and it will self-optimize
+the more you run it. One drawback about using the JVM is that the VM
+must load before the script can execute. A persistent JVM can solves
+this problem. To give you an indication of the actual run time of my
+script, I've included an "Elapsed time" tag after the script runs.

README.md

@@ -0,0 +1,23 @@
+#!/Users/David/.cljr/bin/jark
+
+(ns tgrep
+  (:require [tgrep.search :as ts])
+  (:use clojure.contrib.command-line))
+
+(defn invoke
+  ([logfile start-time] (invoke logfile start-time start-time))
+  ([logfile start-time end-time]
+   (time (ts/process-file logfile start-time end-time))))
+
+(defn -main 
+  "Search log files in O(log n) in time"
+  [& args]
+  (println "here")
+  (with-command-line args
+    [[logfile "log file"]
+     [start   "start time, to arbitrary precision"]
+     [end     "end time, to arbitrary precision"]]
+    (println "start: " start)
+    (println "end: " end)
+    (println "log file: " logfile)
+    (invoke (logfile start end))))

bin/generate

@@ -1,8 +1,7 @@
 (defproject tgrep "0.0.1"
-  :description "FIXME: write"
-  :dependencies [[org.clojure/clojure "1.2.0"]
-                 [org.clojure/clojure-contrib "1.2.0"]
-                 [clj-time "0.3.0-SNAPSHOT"]]
-  :run-aliases {:tgrep tgrep.search
-                :generate tgrep.generate}
-  :main tgrep.search)
+  :description "A speedy log parser"
+  :dependencies [[org.clojure/clojure "1.6.0"]
+                 [org.clojure/tools.cli "0.3.1"]
+                 [clj-time "0.7.0"]]
+  :aliases {"search" ["run" "-m" "tasks.search"]
+            "generate" ["run" "-m" "tasks.generate"]})

bin/search.clj

@@ -0,0 +1,23 @@
+(ns tasks.generate
+  (:require [tgrep.generate :as generate]
+            [clojure.tools.cli :as cli]
+            [clojure.string :as string])
+  (:gen-class))
+
+(def cli-options
+  [["-f" "--file FILE" 
+    "File to generate"
+    :default "haproxy.log"]
+   ["-s" "--start TIME" 
+    "Start time for start of log filel"
+    :default "00:00"]
+   ["-n" "--number NUMBER" 
+    "Number of lines to generate"
+    :parse-fn #(Integer/parseInt %)
+    :default 1000000]
+   ["-v" "--verbose" "Verbose"]])
+
+(defn -main [& args]
+  (let [{:keys [options]} (cli/parse-opts args cli-options)
+        {:keys [file start number]} options]
+    (generate/create-file!  file start number)))

project.clj

@@ -0,0 +1,29 @@
+(ns tasks.search
+  (:require [tgrep.search :as search]
+            [clojure.tools.cli :as cli]
+            [clojure.string :as string]))
+
+(def cli-options
+  [["-f" "--file FILE" "Logfile to search"
+    :default "haproxy.log"]
+   ["-v" "--verbose" "Verbose"]])
+
+(defn time-elapsed [start-time-ns end-time-ns]
+  (/ (double (- end-time-ns start-time-ns))
+     1000000.0))
+
+(defn process-file-and-print-lines [file start-time end-time]
+  (let [lines (search/process-file file
+                                   start-time 
+                                   (or end-time start-time))]
+    (doseq [line lines]
+      (println line))))
+
+(defn -main [& args]
+  (let [{:keys [options arguments]} (cli/parse-opts args cli-options)
+        [start-time end-time] (string/split (first arguments) #"[-]")
+        script-start-time (System/nanoTime)
+        _ (process-file-and-print-lines (options :file) start-time end-time)
+        script-end-time (System/nanoTime)
+        queried-in (time-elapsed script-start-time script-end-time)]
+    (println "Queried in" queried-in "ms")))

src/tasks/generate.clj

@@ -1,9 +1,20 @@
 (ns tgrep.generate
-  (:require [tgrep.search :as ts])
-  (:use clojure.contrib.string
-        clojure.contrib.duck-streams
-        clj-time.core
-        clj-time.coerce))
+  (:require [tgrep.search :as ts]
+            [clojure.string :as string]
+            [clj-time.core :as clj-time]
+            [clj-time.coerce :as coerce]
+            [clojure.java.io :refer [writer]])
+  (:import java.io.BufferedWriter))
+
+;; Copied from old clojure-contrib
+(defn write-lines
+  "Writes lines (a seq) to f, separated by newlines. f is opened with
+   writer, and automatically closed at the end of the sequence."
+  [filename lines]
+  (with-open [w (writer filename)]
+    (binding [*out* w]
+      (doseq [line lines]
+        (println line)))))
 
 (def start-time (ts/parse-date "11/Feb/2011:23:55:00.000" ts/date-formatter-1))
 
@@ -12,11 +23,11 @@
 (defn entry-for
   "Generates an entry for the given date (in millis) based on template."
   [template millis]
-  (let [date (from-long millis)
+  (let [date (coerce/from-long millis)
         formatted-1 (ts/unparse-date date ts/date-formatter-1)
         formatted-2 (ts/unparse-date date ts/date-formatter-2)
-        subst-1 (replace-first-re ts/date-re-1 formatted-1 template)
-        subst-2 (replace-first-re ts/date-re-2 formatted-2 subst-1)]
+        subst-1 (string/replace-first template ts/date-re-1 formatted-1)
+        subst-2 (string/replace-first subst-1 ts/date-re-2 formatted-2)]
     subst-2))
 
 (defn next-entry
@@ -26,17 +37,16 @@
    (let [next-date (ts/inc-date (ts/get-date curr-entry) millis)
          formatted-1 (ts/unparse-date next-date ts/date-formatter-1)
          formatted-2 (ts/unparse-date next-date ts/date-formatter-2)
-         subst-1 (replace-first-re ts/date-re-1 formatted-1 curr-entry)
-         subst-2 (replace-first-re ts/date-re-2 formatted-2 subst-1)]
+         subst-1 (string/replace-first curr-entry ts/date-re-1 formatted-1)
+         subst-2 (string/replace-first subst-1 ts/date-re-2 formatted-2)]
      subst-2)))
 
-(defn write-entries
+(defn create-file!
   "Writes all entries to logfile via lazy map, to avoid heavy memory
-  footprint but still harness the power of functional programming."
-  [start-date n]
-  (let [millis (to-long start-date)
+   footprint but still harness the power of functional programming."
+  [filename start-date n]
+  (let [millis (coerce/to-long start-date)
         entry (entry-for example millis)]
-    (write-lines ts/*logfile* (map #(next-entry entry (* 1000 %)) 
-                           (range 0 n)))))
-
-(defn -main [] (write-entries start-time 1000000))
+    (write-lines filename 
+                 (map #(next-entry entry (* 1000 %)) 
+                      (range 0 n)))))