Issue #454 - Add shell execution ALU functions

.github/workflows/release.yml

@@ -141,10 +141,7 @@ jobs:
 
     - name: Substitute version and date into guidebook
       run: |
-        DATE=$(date '+%B %d, %Y')
-        VERSION=$(specs/exe/specs '@version' 1 </dev/null)
-        sed -e "s|XXDATE|${DATE}|g" -e "s|XXVERSION|${VERSION}|g" \
-            specs/docs/guidebook.md > specs/docs/guidebook_tmp.md
+        specs/exe/specs -i specs/docs/guidebook.md -o specs/docs/guidebook_tmp.md -f specs/docs/guidebook_prepare
 
     - name: Build guidebook PDF
       run: |

manpage

@@ -1760,6 +1760,51 @@ are specified, the first
 words and last 
 .I ftr 
 words are omitted.
+.IP "exec(cmd)" 3
+Runs the shell command in
+.I cmd
+and returns whatever the command wrote to its standard output. A single
+trailing newline is stripped from the output. The command's return code is
+saved and can be retrieved with
+.B excrc(),
+while whatever the command wrote to its standard error is saved and can be
+retrieved with
+.B excerr().
+.IP "exc1(cmd [,lineNo])" 3
+Like
+.B exec(cmd),
+but returns only the contents of line
+.I lineNo
+of the standard output. The
+.I lineNo
+argument must be a positive integer and defaults to 1, which returns the first
+line. If the requested line does not exist, an empty string is returned.
+.IP "excrc()" 3
+Returns the return code of the last shell command run by
+.B exec()
+or
+.B exc1().
+If no such command has been run, returns
+.B NaN.
+This reflects only the most recent run, so use it before running any other
+shell command. Take special care when
+.B exec()
+or
+.B exc1()
+are run inside an
+.B IF
+or
+.B WHILE
+block.
+.IP "excerr()" 3
+Returns whatever the last shell command run by
+.B exec()
+or
+.B exc1()
+wrote to its standard error. If no such command has been run, returns an empty
+string. As with
+.B excrc(),
+this reflects only the most recent run.
 
 
 .SS "Built-In Statistical Pseudo-Functions" 

specs/docs/TOC.md

@@ -50,6 +50,7 @@ Advanced Topics
   * [Table of REXX-Derived Functions](alu_adv.md#table-of-other-rexx-derived-functions)
   * [Table of Statistical and Frequency Map Pseudo-Functions](alu_adv.md#table-of-statistical-and-frequency-map-pseudo-functions)
   * [Table of Record Access Functions](alu_adv.md#table-of-record-access-functions)
+  * [Table of Shell Command Functions](alu_adv.md#table-of-shell-command-functions)
   * [Table of Special Functions](alu_adv.md#table-of-special-functions)
 * [Local Python Functions](pyfuncs.md)
 * [Examples](examples.md)

specs/docs/alu_adv.md

@@ -221,6 +221,19 @@ The parameters for the `fmap_dump` functions are as follows:
 * *showPct* - evaluated as boolean. If *true* causes the textual formats to print out a percentage. Causes the CSV and JSON formats to add a fraction. Default is *false*.
 
 
+## Table of Shell Command Functions
+
+These functions run a shell command and let you process its output and return code.
+
+| Function | Description |
+| -------- | ----------- |
+| `exec(cmd)` | Runs the shell command `cmd` and returns its standard output. A single trailing newline is stripped. The return code is saved for `excrc()` and the standard error is saved for `excerr()` |
+| `exc1(cmd,[lineNo])` | Like `exec(cmd)`, but returns only the content of line `lineNo` of the standard output. `lineNo` must be a positive integer and defaults to `1` (the first line); an empty string is returned if that line does not exist |
+| `excrc()` | Returns the return code of the last shell command run by `exec()` or `exc1()`, or **NaN** if no such command has been run |
+| `excerr()` | Returns the standard error of the last shell command run by `exec()` or `exc1()`. If no such command has been run, returns an empty string |
+
+The `excrc()` and `excerr()` functions reflect **only the last run** of a shell command, so use them before any other shell command is run. Take special care when `exec()` or `exc1()` are used inside an `IF` or `WHILE` block, where the order and number of runs may not be obvious.
+
 ## Table of Special Functions
 | Function | Description |
 | -------- | ----------- |

specs/docs/guidebook.md

@@ -22,15 +22,15 @@ header-includes: |
 include-before: |
   \begin{titlepage}
   \centering
-  {\Huge\bfseries specs\par}
-  \vspace{2cm}
-  {\Large The complete guidebook\par}
-  \vspace{4cm}
-  {\textit\underline\color{blue}https://github.com/yoavnir/specs2016\par}
+  {\Huge\bfseries\itshape specs\par}
+  \vspace{0cm}
+  {\Huge The complete guidebook\par}
+  \vspace{5cm}
+  {\Large\itshape\color{blue}\underline{https://github.com/yoavnir/specs2016}\par}
   \vfill
-  {\Large Version: XXVERSION\par}
+  {\Large Version XXVERSION\par}
   \vspace{1cm}
-  {\large XXDATE\par}
+  {\Large XXDATE\par}
   \end{titlepage}
   \clearpage
   \chapter*{Preface}
@@ -1731,6 +1731,55 @@ Example — word frequency counter:
 cat file.txt | specs a: w1 . EOF PRINT "fmap_dump(a,'csv','cd',1)" 1
 ```
 
+## Shell Command Functions
+
+These functions run a shell command and let you process its standard output, standard error, and return code:
+
+| Function | Description |
+|----------|-------------|
+| `exec(cmd)` | Run shell command c; returns its standard output (one trailing newline stripped) |
+| `exc1(cmd, [lineNo])` | Like `exec`, but returns only the content of output line `lineNo`; `lineNo` must be a positive integer and defaults to 1; empty string if absent |
+| `excrc()` | Return code of the last exec/exc1 run; NaN if none has run |
+| `excerr()` | Standard error of the last exec/exc1 run; empty string if none has run |
+
+`exec` returns whatever the command wrote to standard output. The command's return code and standard error are saved for the `excrc()` and `excerr()` functions:
+
+```
+specs PRINT "exec('echo hello')"
+hello
+```
+
+`exc1` is the same as `exec`, except it returns just one line of output:
+
+```
+specs PRINT "exc1('echo hello')"
+hello
+```
+
+### Reading the return code and standard error
+
+`excrc()` and `excerr()` always reflect **only the last** shell command that was run. Read them right after the relevant `exec`/`exc1` call, and before any other shell command runs:
+
+```
+specs PRINT "exec('grep specs *')" WRITE "stderr" WRITE PRINT "excerr()"
+```
+
+Be especially careful when `exec` or `exc1` appear inside an `IF` or `WHILE` block, where the order and number of runs may not be obvious.
+
+### Efficiency
+
+`exec` and `exc1` launch a new shell for every record they are evaluated on. The following runs `ls | wc` once **per input record**, which is wasteful:
+
+```
+specs -C ls "File" 1 w8 NW "is one of the" NW PRINT "exc1('ls | wc')" NW "files in this directory"
+```
+
+A more efficient version runs the command only once — on the first record — and stores the result in a field identifier for reuse:
+
+```
+specs -C ls IF "first()" THEN SET "#0:=exc1('ls | wc')" ENDIF "File" 1 w8 NW "is one of the" NW PRINT "#0" NW "files in this directory"
+```
+
 ## Special Functions
 
 | Function | Description |

specs/docs/guidebook_prepare

@@ -0,0 +1,13 @@
+# Process `guidebook.md`:
+#   Replace instances of XXDATE with the current date
+#   Replace instances of XXVERSION with the current version
+# This makes things ready for generating the PDF
+
+   IF "first()" THEN 
+      SET "#0:=exc1('date \"+%B %d, %Y\"')" 
+   ENDIF 
+
+   PRINT "substitute(@@,XXDATE,#0,'U')" 
+   REDO 
+   PRINT "substitute(@@,XXVERSION,@version,'U')"
+

specs/src/setup.py

@@ -318,7 +318,7 @@ def python_search(arg):
 book: $(DOCS_DIR)/guidebook.pdf
 
 $(DOCS_DIR)/guidebook.pdf: $(DOCS_DIR)/guidebook.md $(DOCS_DIR)/header.tex
-	sed -e "s|XXDATE|$$(date '+%B %d, %Y')|g" -e "s|XXVERSION|$$($(EXE_DIR)/specs '@version' 1 </dev/null)|g" $(DOCS_DIR)/guidebook.md > $(DOCS_DIR)/guidebook_tmp.md
+	$(EXE_DIR)/specs -i $(DOCS_DIR)/guidebook.md -o $(DOCS_DIR)/guidebook_tmp.md -f $(DOCS_DIR)/guidebook_prepare
 	pandoc $(DOCS_DIR)/guidebook_tmp.md -o $(DOCS_DIR)/guidebook.pdf --pdf-engine=xelatex -H $(DOCS_DIR)/header.tex
 	/bin/rm $(DOCS_DIR)/guidebook_tmp.md
 """

specs/src/test/ALUUnitTest.cc

@@ -1706,6 +1706,19 @@ int runALUUnitTests16(unsigned int onlyTest)
 	VERIFY_EXPR_RES("not(includes(raid,'i'))", "0");  // 835
 	VERIFY_EXPR_RES("not(includes(team,'i'))", "1");  // proving that there is really no 'i' in team
 
+	// Shell command functions: exec, exc1, excrc, excerr
+	// 'echo' is used because it works on Linux, Mac OS, and Windows.
+	// excrc()/excerr() are exercised within the same expression as the
+	// exec()/exc1() call so each test is self-contained when run in isolation.
+	printHeader(onlyTest, "\nShell command functions\n======================\n\n");
+	VERIFY_EXPR_RES("exec('echo hello')", "hello");          // 837
+	VERIFY_EXPR_RES("exc1('echo hello')", "hello");
+	VERIFY_EXPR_RES("exc1('echo hello',1)", "hello");
+	VERIFY_EXPR_RES("exc1('echo hello',2)", "");             // line out of range -> empty
+	VERIFY_EXPR_RES("exc1('echo hello',0)", "exc1: Argument must be a positive integer, but got 0: #2 (lineNo)");
+	VERIFY_EXPR_RES("exc1('echo hello',-3)", "exc1: Argument must be a positive integer, but got -3: #2 (lineNo)");
+	VERIFY_EXPR_RES("exec('echo hello')||'/'||excrc()", "hello/0");
+	VERIFY_EXPR_RES("exc1('echo hello')||'/'||excerr()", "hello/");
 
 	if (countFailures) {
 		if (onlyTest == 0) {

specs/src/utils/aluFunctions.cc

@@ -15,6 +15,12 @@
 #include <iomanip>
 #include <algorithm> // for std::reverse
 #include <cstdlib>   // for std::getenv
+#include <cstdio>    // for popen/_popen
+#include <array>     // for the shell command read buffer
+#ifndef WIN64
+#include <sys/wait.h>  // for WIFEXITED/WEXITSTATUS
+#include <unistd.h>    // for mkstemp and close
+#endif
 
 #define PAD_CHAR ' '
 
@@ -3018,6 +3024,138 @@ PValue AluFunc_pset(PValue pName, PValue pValue)
 	return ret;
 }
 
+/*
+ *
+ * Shell command functions: exec, exc1, excrc, excerr
+ * ==================================================
+ * These functions run a shell command and expose its standard output, its
+ * standard error, and its return code. The standard error and return code of
+ * only the *last* run are retained in the globals below.
+ */
+static PValue      g_execRc = mkValue0();   // NaN until a shell command has run
+static std::string g_execErr = "";
+
+#define EXEC_READ_BUFFER_SIZE 65536
+
+// Creates an empty temporary file and returns its name. The file is left in
+// place so the shell can reopen it via redirection. On POSIX we use mkstemp
+// (the safe, race-free way); on Windows, where mkstemp is unavailable, we
+// fall back to tmpnam.
+static std::string makeTempFileName()
+{
+#ifdef WIN64
+	const char* name = std::tmpnam(nullptr);
+	if (nullptr == name) {
+		MYTHROW("exec: Failed to generate a temporary file name");
+	}
+	return std::string(name);
+#else
+	const char* tmpdir = std::getenv("TMPDIR");
+	std::string templ = std::string((tmpdir && *tmpdir) ? tmpdir : "/tmp") + "/specsXXXXXX";
+	int fd = mkstemp(&templ[0]);  // std::string storage is contiguous (C++11+)
+	if (fd < 0) {
+		MYTHROW("exec: Failed to create a temporary file for capturing standard error");
+	}
+	close(fd);
+	return templ;
+#endif
+}
+
+// Runs 'cmd' through the shell, returns its standard output, and updates
+// g_execRc (return code) and g_execErr (standard error).
+static std::string runShellCommand(const std::string& cmd)
+{
+	std::string errFileName = makeTempFileName();
+	std::string fullCmd = cmd + " 2>\"" + errFileName + "\"";
+
+	std::array<char, EXEC_READ_BUFFER_SIZE> buffer;
+	std::string output;
+
+#ifdef WIN64
+	FILE* pipe = _popen(fullCmd.c_str(), "r");
+#else
+	FILE* pipe = popen(fullCmd.c_str(), "r");
+#endif
+	if (nullptr == pipe) {
+		std::string err = "exec: Failed to run command: " + cmd;
+		MYTHROW(err);
+	}
+
+	while (nullptr != fgets(buffer.data(), int(buffer.size()), pipe)) {
+		output += buffer.data();
+	}
+
+#ifdef WIN64
+	int status = _pclose(pipe);
+	g_execRc = mkValue(ALUInt(status));
+#else
+	int status = pclose(pipe);
+	g_execRc = mkValue(ALUInt(WIFEXITED(status) ? WEXITSTATUS(status) : status));
+#endif
+
+	g_execErr = "";
+	FILE* errFile = fopen(errFileName.c_str(), "r");
+	if (nullptr != errFile) {
+		while (nullptr != fgets(buffer.data(), int(buffer.size()), errFile)) {
+			g_execErr += buffer.data();
+		}
+		fclose(errFile);
+	}
+	remove(errFileName.c_str());
+
+	return output;
+}
+
+PValue AluFunc_exec(PValue pCmd)
+{
+	ASSERT_NOT_ELIDED(pCmd,1,command);
+	std::string output = runShellCommand(pCmd->getStr());
+	// Strip a single trailing newline (\n or \r\n)
+	if (!output.empty() && output.back()=='\n') {
+		output.erase(output.length()-1);
+		if (!output.empty() && output.back()=='\r') {
+			output.erase(output.length()-1);
+		}
+	}
+	return mkValue(output);
+}
+
+PValue AluFunc_exc1(PValue pCmd, PValue pLine)
+{
+	ASSERT_NOT_ELIDED(pCmd,1,command);
+	ALUInt lineNo = ARG_INT_WITH_DEFAULT(pLine,1);
+	if (lineNo < 1) {
+		// The below dereference of pLine is safe because if !pLine, lineNo = 1
+		std::string err = "Argument must be a positive integer, but got " + pLine->getStr();
+		THROW_ARG_ISSUE(2,lineNo,err);
+	}
+	std::string output = runShellCommand(pCmd->getStr());
+
+	std::istringstream iss(output);
+	std::string line;
+	ALUInt idx = 0;
+	while (std::getline(iss, line)) {
+		if (++idx == lineNo) {
+			// std::getline strips the \n; strip a trailing \r if present
+			if (!line.empty() && line.back()=='\r') {
+				line.erase(line.length()-1);
+			}
+			return mkValue(line);
+		}
+	}
+	return mkValue(std::string(""));
+}
+
+PValue AluFunc_excrc()
+{
+	return g_execRc;  /* NaN until a shell command has run */
+}
+
+PValue AluFunc_excerr()
+{
+	return mkValue(g_execErr);
+}
+
 PValue AluFunc_getenv(PValue pName)
 {
 	ASSERT_NOT_ELIDED(pName,1,variableName);

specs/src/utils/aluFunctions.h

@@ -322,7 +322,16 @@
 	X(split,          3, ALUFUNC_REGULAR,     true,   \
 			"([sep], [hdr], [ftr]) - Returns on multiple lines the fields (separated by the 'sep' character), discarding the first 'hdr' and last 'ftr' records.","The separator defaults to the current field separator.\n'hdr' and 'ftr' both default to zero.") \
 	X(splitw,         3, ALUFUNC_REGULAR,     true,   \
-			"([sep], [hdr], [ftr]) - Returns on multiple lines the words (separated by the 'sep' character), discarding the first 'hdr' and last 'ftr' records.","The separator defaults to the current word separator.\n'hdr' and 'ftr' both default to zero.")
+			"([sep], [hdr], [ftr]) - Returns on multiple lines the words (separated by the 'sep' character), discarding the first 'hdr' and last 'ftr' records.","The separator defaults to the current word separator.\n'hdr' and 'ftr' both default to zero.") \
+	H(Shell Command Functions,22) \
+	X(exec,           1, ALUFUNC_REGULAR,     false,  \
+			"(cmd) - Runs the shell command 'cmd' and returns its standard output.","One trailing newline is stripped from the output.\nThe command's return code is saved and is available through excrc(),\nwhile its standard error is saved and is available through excerr().") \
+	X(exc1,           2, ALUFUNC_REGULAR,     false,  \
+			"(cmd,[lineno]) - Runs the shell command 'cmd' and returns the content of line 'lineno' of its standard output.","'lineno' must be a positive integer and defaults to 1, returning the first line. Returns an empty string if that line does not exist.\nLike exec(), it saves the return code and standard error for excrc() and excerr().") \
+	X(excrc,          0, ALUFUNC_REGULAR,     false,  \
+			"() - Returns the return code of the last shell command run by exec() or exc1().","Returns NaN if no shell command has been run yet.\nReflects only the last run, so use it before running any other shell command.\nTake care when exec() or exc1() are run within an IF or WHILE block.") \
+	X(excerr,         0, ALUFUNC_REGULAR,     false,  \
+			"() - Returns the standard error of the last shell command run by exec() or exc1().","Initialized to an empty string if no shell command has been run yet.\nReflects only the last run, so use it before running any other shell command.\nTake care when exec() or exc1() are run within an IF or WHILE block.")
 
 #define ALU_DEBUG_FUNCTION_LIST                       \
 	X(testfunc,       4, ALUFUNC_REGULAR,     false,  \

specs/tests/valgrind_unit_tests.py

@@ -1,6 +1,6 @@
 import sys, memcheck, argparse, platform
 
-count_ALU_tests = 844
+count_ALU_tests = 852
 count_processing_tests = 277
 count_token_tests = 17