Issue #396 - harmonize record retrieval in run-out cycle

manpage

@@ -574,7 +574,7 @@ sorted by department name. You can print this out without repeating the departme
           BREAK c
               ID c 1
 
-.SS "RunIn and RunOut Cycles"
+.SS "Run-In and Run-Out Cycles"
 A 
 .B cycle 
 is a single run of the specification on the current active input record. A cycle may read additional input records, produce zero output records, or produce multiple output records. If the specification contains 
@@ -584,16 +584,16 @@ or
 tokens, a single cycle can consume more than one input record.
 
 The 
-.B runin 
-cycle is the first cycle. In the runin cycle, the function 
+.B run-in 
+cycle is the first cycle. In the run-in cycle, the function 
 .B first()
 returns 1. This can be used for initial processing such as printing of headers or setting initial values. 
 
 The 
-.B runout 
+.B run-out 
 cycle happens 
 .I after 
-the last line has been read, but only when the specification requires a runout cycle. It consists of the spec items that follow the 
+the last line has been read, but only when the specification requires a run-out cycle. It consists of the spec items that follow the 
 .B EOF 
 token, or (when 
 .I select second 
@@ -612,6 +612,21 @@ function. Example:
         /==========/ 1 write
         /Total:/ 1
         print #0 nw
+
+
+Note that any use of the
+.B eof()
+function anywhere in the specification forces a run-out cycle. This is called a
+.I forced run-out cycle
+and any use of record-reading spec units, such as a 
+.I data field 
+or
+.B ALU
+functions that access records, like
+.B record()
+or
+.B word(i)
+will result in an empty string returned.
 
 .SS "Input Streams"
 The keyword 
@@ -913,12 +928,12 @@ Returns a binary representation of the unsigned integer in x. The field length i
 Returns the length of the argument when viewed as a string. For example, len(37) is 2; len('hello') is 5.
 .IP "first()" 3
 Returns 1 during the 
-.B runin
+.B run-in
 cycle, and zero otherwise.
 .IP "eof()" 3
 Returns 1 during the
-.B runout 
-cycle, and zero otherwise.
+.B run-out 
+cycle, and zero otherwise. Its use in a specification forces a run-out cycle.
 .IP "number()" 3
 Returns the number of times the specification has so far been run on different records.
 .IP "recno()" 3

specs/docs/onepage.md

@@ -150,13 +150,13 @@ Without **while-guard** this specification will loop forever. To solve this, **s
 
 **While-Guard** is not perfect. To disable it, you can use the command-line switch `--no-while-guard` or you can override the maximum iteration count at which the program exist by setting the `while-guard-limit` to some integer value.
 
-RunIn and RunOut Cycles
+Run-In and Run-Out Cycles
 =========================
 A **cycle** is defined as a single run of the specification, which includes reading an input record, processing it, and outputting one or more records. If the specification contains **read** or **readstop** tokens, a single cycle can consume more than one input records.
 
-The **runin** cycle is the first one to run. In the runin cycle, the function **first()** returns 1. This can be used for initial processing such as printing of headers or setting initial values. 
+The **run-in** cycle is the first one to run. In the run-in cycle, the function **first()** returns 1. This can be used for initial processing such as printing of headers or setting initial values. 
 
-The **runout** cycle happens *after* the last line has been read.  It consists of the spec items that follow the **EOF** token, or (when **select second** is used) conditional specifications with the **eof()** function. Example:
+The **run-out** cycle happens *after* the last line has been read.  It consists of the spec items that follow the **EOF** token, or (when **select second** is used) conditional specifications with the **eof()** function. Example:
 ```
             if first() then
                 /Item/  1  /Square/ nw write
@@ -170,6 +170,9 @@ The **runout** cycle happens *after* the last line has been read.  It consists o
                 /Total:/ 1
                 print #0 nw
 ```
+
+Note that there are two kinds of **run-out** cycle: the **explicit run-out cycle**, where there are spec units after an **EOF** token, and the **forced run-out cycle** which is triggered by the use of the `eof()` function anywhere in the specification. A forced run-out cycle will force an extra run of the specification with apparently an empty input record.
+
 Configuration File
 ==================
 

specs/docs/struct.md

@@ -107,7 +107,7 @@ specs if "first()" then
 
 ### Run-Out
 The Run-Out cycle runs *after* the last record is processed. It is only run if it has something to do. There are two ways to do things on the run-out cycles:
-1. Using the boolean function `eof()`.
+1. Using the boolean function `eof()`. We call this a **forced run-out cycle**
 2. Using the `EOF` keyword.
 
 The following enhancement of the run-in example will demonstrate both:
@@ -138,7 +138,7 @@ specs
       set #0+=a
    eof
       /Total:/   1
-      print #0   Next
+      print #0   NEXTWORD
 ```
 | Input | Output |
 | ----- | ------ |
@@ -148,6 +148,19 @@ specs
 | 4 | 4 |
 | | Total: 10 |
 
+Here's an alternate implementation with a **forced run-out cycle**:
+```
+# Summing
+specs
+   IF "eof()" THEN
+      /Total:/   1
+      print #0   NEXTWORD
+   ELSE
+   a: WORD 1     1
+      SET #0+=a
+   ENDIF
+```
+
 ### Control Breaks
 **Field identifiers** can be used for conditional execution when their value changes from record to record. Consider the following example CSV file containing personnel records:
 ```

specs/src/gdb/specs_gdb.py

@@ -830,6 +830,7 @@ def invoke(self, arg, from_tty):
             print(f"DataField:")
             val = gdb.parse_and_eval(arg)
             label = chr(int(val["m_label"])) if int(val["m_label"]) > 0 else "none"
+            tail_label = chr(int(val["m_tailLabel"])) if int(val["m_tailLabel"]) > 0 else "none"
             out_start = int(val["m_outStart"])
             max_len = int(val["m_maxLength"])
             strip = bool(val["m_strip"])
@@ -839,14 +840,57 @@ def invoke(self, arg, from_tty):
             conv_str = STRING_CONVERSIONS.get(conv, f"Unknown({conv})")
             align_str = OUTPUT_ALIGNMENT.get(align, f"Unknown({align})")
 
-            print(f"  Label: {label}")
+            print(f"  Label: {label}/{tail_label}")
             print(f"  Output Start: {out_start}")
             print(f"  Max Length: {max_len}")
             print(f"  Strip: {strip}")
             print(f"  Conversion: {conv_str}")
             print(f"  Alignment: {align_str}")
+
+            # m_InputPart (shared_ptr<InputPart>)
+            try:
+                ip_ptr = val["m_InputPart"]["_M_ptr"]
+                if int(ip_ptr) == 0:
+                    print(f"  Input Part: <nullptr>")
+                else:
+                    result = gdb.parse_and_eval(
+                        f'((InputPart*)({int(ip_ptr)}))->Debug()')
+                    debug_str = std_string_to_str(result)
+                    print(f"  Input Part: {debug_str}")
+            except Exception as ex:
+                print(f"  Input Part: <error: {ex}>")
+
+            # AluVec fields
+            self._dump_alu_vec_field(val, "m_outputStartExpression", "Output Start Expression")
+            self._dump_alu_vec_field(val, "m_outputWidthExpression", "Output Width Expression")
+            self._dump_alu_vec_field(val, "m_outputAlignmentExpression", "Output Alignment Expression")
         except Exception as e:
             print(f"Error: {e}")
+
+    def _dump_alu_vec_field(self, val, field_name, label):
+        """Print an AluVec field on one line, or 'empty' if it has no elements."""
+        try:
+            vec = val[field_name]
+            size = std_vector_size(vec)
+            if size == 0:
+                print(f"  {label}: empty")
+            else:
+                start = vec["_M_impl"]["_M_start"]
+                items = []
+                for i in range(size):
+                    try:
+                        ptr = start[i]["_M_ptr"]
+                        if int(ptr) != 0:
+                            result = gdb.parse_and_eval(
+                                f'((AluUnit*)({int(ptr)}))->_identify()')
+                            items.append(std_string_to_str(result))
+                        else:
+                            items.append("<null>")
+                    except:
+                        items.append("?")
+                print(f"  {label}: {'; '.join(items)}")
+        except Exception as ex:
+            print(f"  {label}: <error: {ex}>")
 
 class DumpTokenItem(gdb.Command):
     """Dump a TokenItem."""

specs/src/processing/ProcessingState.cc

@@ -357,14 +357,14 @@ int ProcessingState::getWordEnd(int idx) {
 	return m_wordEnd[idx-1];
 }
 
-// Convention: returns NULL for an empty string
 // Convention: from=0 means from the start (same as 1)
 // Convention: to=0 means to the end
 // Convention: from=0 and to=0 -- empty string.
 PSpecString ProcessingState::getFromTo(int from, int to)
 {
-	if (m_inputStation != STATION_SECOND) {
-		MYASSERT_WITH_MSG(nullptr!=m_ps,"Tried to read record in run-out cycle");
+	// In the run-out cycle, return an empty string
+	if (m_inputStation != STATION_SECOND && nullptr==m_ps) {
+		return std::make_shared<std::string>();
 	}
 	int slen = (int)(currRecord()->length());
 

specs/src/test/ProcessingTest.cc

@@ -990,6 +990,19 @@ int main(int argc, char** argv)
 	spec = "CONTEXT 1 PRINT 'cfrecord()' 1 PRINT 'record()' NW";
 	VERIFY2(spec, "alpha\nbeta\ngamma", "alpha beta\nbeta gamma\ngamma"); // TEST #255
 
+	// record(), word(), field(), range() return empty string during forced run-out cycle
+	spec = "PRINT 'record()' 1 PRINT 'eof()' NEXTWORD";
+	VERIFY2(spec, "hello\nworld", "hello 0\nworld 0\n1"); // TEST #256
+
+	spec = "PRINT 'word(1)' 1 PRINT 'eof()' NEXTWORD";
+	VERIFY2(spec, "hello world", "hello 0\n1"); // TEST #257
+
+	spec = "PRINT 'field(1)' 1 PRINT 'eof()' NEXTWORD";
+	VERIFY2(spec, "hello\tworld", "hello 0\n1"); // TEST #258
+
+	spec = "PRINT 'range(1,3)' 1 PRINT 'eof()' NEXTWORD";
+	VERIFY2(spec, "abcdef", "abc 0\n1"); // TEST #259
+
 	if (errorCount) {
 		std::cout << '\n' << errorCount << '/' << testCount << " tests failed.\n";
 		std::cout << "Failed tests: ";

specs/tests/valgrind_unit_tests.py

@@ -1,7 +1,7 @@
 import sys, memcheck, argparse
 
 count_ALU_tests = 832
-count_processing_tests = 255
+count_processing_tests = 259
 count_token_tests = 17
 
 # Parse the one command line options