manpage

.\" Manpage for specs.
.\" Open an issue at https://github.com/yoavnir/specs2016 to correct errors or typos
.mso www.tmac
.TH man 1 "11 Sep 2026" "1.0.0" "specs man page"
.SH NAME
specs \- a text processing tool
.SH SYNOPSIS
specs [switches] spec-units
.SH DESCRIPTION
.B specs
is a command line utility for parsing and re-arranging text input. It allows re-alignment of fields, some format conversion, and re-formatting multiple lines into single lines or vice versa. Input comes from standard input, and output flows to standard output.
.P 
This is a re-writing of the specs pipeline stage from CMS, only changed quite a bit.
.P 
This version is liberally based on the CMS Pipelines User's Guide and Reference
.URL "https://www.vm.ibm.com/library/740pdfs/74625200.pdf"
-- especially chapters 16, 24, and 20.

.SS "Spec Units"
.B Spec Units 
are the building blocks of a 
.B specs
.I specification.
Each spec unit tells the 
.B specs
engine to perform some action. The most common spec unit is a 
.I data field
which consists of five arguments, three of which may be omitted:
.P
.RS 5
[fieldIdentifier] InputPart [conversion] OutputPart [alignment]
.RE
.P

A 
.B fieldIdentifier 
is a single letter followed by a colon (like
.I a:
), that maps to the input or output of a single data field unit for later reference such as
in later data fields or in 
.I expressions. 
If the fieldIdentifier is at the start of the data field unit, it contains the input. If it
is the OutputPart, it contains the output.  For example:
.RS 5
a: w1 ucase b:
.RE
.I a
contains the first word of the input, while 
.I b
contains an uppercase version of 
.I a

The 
.B InputPart
argument may be any of the following:
.RS 3
.P
A range of characters, such as `5`, `3-7`, or `5.8`, the last one indicating 8 characters starting in the 5th position. Note that the indexing of characters is 1- rather than 0-based.
.P
A range of words, such as `w5` or `words 5-7`, where words are separated by one or more `wordseparator` characters -- locale-defined whitespace by default. The word indexing is 1-based.
.P
A range of fields, such as `fields 5` or `f5-7`, where fields are separated by exactly one `fieldseparator` character -- a tab by default. The field indexing is 1-based.
.P 
.B TODclock
-- A 64-bit formatted timestamp, giving microseconds since the Unix epoch.
.P 
.B DTODclock
-- A 64-bit formatted timestamp, giving microseconds since the Unix epoch. The difference is that TODclock shows the time when this run of
.I specs
begun, while DTODclock gives the time of producing the current record.
.P
.B NUMBER
or 
.B RECNO
-- A record counter as a 10-digit decimal number.
.P
.B TIMEDIFF
-- A 12-digit decimal number indicating the number of microseconds since the invocation of the program.
.P
The 
.B ID
keyword followed by a previously defined 
.B FieldIdentifier.
.P
The 
.B PRINT
keyword followed by a calculated expression.
.B PRINT
can be replaced by a question mark (
.B ?
) either as its own command-line token or prefixing the calculated expression.
.P
A string literal, optionally enclosed by delimiters, such as `/TODclock/` or `'NUMBER'`. Note that to include the single quotes on the command line requires you to enclose them in double quotes.
.P
A 
.B SUBSTring
of another InputPart.
.RE
    
The 
.B OutputPart
argument specifies where to put the source:
.RS 3
.P
Absolute position (such as `1`) with no limit on output length.
.P
A range (such as `1-5` or `1.5`)
.P
.B N 
or
.B NEXT
for placing the output immediately after the previous output.
.P
.B NW 
or
.B NEXTWORD
for placing the output following a space character after the previous output.
.P
.B NF
or
.B NEXTFIELD
for placing the output following a tab character after the previous output.
.P 
A field identifier specification such as 
.B a:
or
.B C:
\&.
.RE
    
The alignment argument can be `l`, `c`, or `r`, for
.B left
,
.B center
, and
.B right
respectively. Yes, you can use the entire word. In fact, for compatibility with other countries, you can even spell it
.B centre.
.RE
    
The 
.B OutputPart
argument can also be specified as a 
.B Composed Output Placement
argument. It is enclosed in non-optional parenthesis and contains 1, 2, or 3 comma-separated expressions. The expressions are evaluated at each cycle to produce the starting column, the field width, and the alignment of the field.
.P
With one expression, that expression is the starting column. For example:
.P
    specs w1 (@cols)
.P
places the first word at the column given by the
.I @cols
label.
.P
With two expressions, they are the starting column and the width. A width of zero is a special value denoting the natural width of the output. For example:
.P
    specs w1 (1,10)
.P
places the first word in columns 1-10.
.P
With three expressions, the third expression is evaluated as a string and determines the alignment. If that value begins with a capital or lower-case
.I c
the field is centered; if it begins with a capital or lower-case
.I r
the field is right-aligned; otherwise it is left-aligned. For example:
.P
    specs /hello/ (1,@cols,'R')
.P
right-aligns the string in the available display width.
.P
If the alignment field is two characters in length, the second character is a digit between
.I 1-5,
and the content is longer than the output field, the truncation will use an ellipsis as follows: For a value of
.I 1, 
.B specs 
will output an ellipsis followed by the suffix of the content; for
.I 5,
.B specs
will output a prefix followed by an ellipsis; for 
.I 2, 3, 
and
.I 4,
.B specs
will output a prefix, an ellipsis and a suffix, with the prefix being one third, one half, or two thirds of the output string respectively.  If a composed output placement argument appears in a particular data field, neither a regular Output Placement nor an Alignment argument may appear.  The first arguments in a composed output placement can be elided, as in 
.I (,,'R'). 
If that is done, the default for the output start is the next place, equivalent to the function 
.I next(), 
while the default for the second argument is the rest of the line, based on the 
.I @cols
label. In other words, omitted arguments default the same way they would if the field were placed at the next available position and allowed to use the remaining width.

The conversion argument can specify any of the following conversions:
.IP "rot13" 3
Encrypts the bytes using the ROT-13 cipher.
.IP "C2B" 3
Converts characters to binary: "AB" --> "0010000100100010".
.IP "C2X" 3
Converts characters to hexadecimal: "AB" --> "4142".
.IP "B2C" 3
Converts binary to characters: "0010000100100010" --> "AB". Will throw an exception if called with an invalid character.
.IP "X2CH" 3
Converts hexadecimal to characters: "4142" --> "AB". Will throw an exception if called with an invalid character.
.IP "b2x" 3
Converts binary data to hex.
.IP "D2X" 3
Convert decimal to hex: "314159265" --> "12b9b0a1".
.IP "X2D" 3
Convert hex to decimal: "12b9b0a1" --> "314159265".
.IP "ucase" 3
Converts text to uppercase.
.IP "lcase" 3
Converts text to lowercase.
.IP "BSWAP" 3
Byte Swap. reverses the order of bytes. "AB" --> "BA"
.IP "ti2f" 3
Convert internal time format (8-byte microseconds since the epoch) to printable format using the conventions of strftime, plus %xf for fractional seconds, where x represents number of digits from 0 to 6.
.IP "tf2i" 3
Convert printable time format to the internal 8-byte representation. 
.IP "s2tf" 3
Convert a decimal number with up to six decimal places, representing seconds since the epoch, to printable format using the conventions of strftime, plus %xf for fractional seconds, where x represents number of digits from 0 to 6.
.IP "tf2s" 3
Convert printable time format to a decimal number, representing seconds since the epoch. 
.IP "mcs2tf" 3
Convert a number, representing microseconds since the epoch, to printable format using the conventions of strftime, plus %xf for fractional seconds, where x represents number of digits from 0 to 6.
.IP "tf2mcs" 3
Convert printable time format to a number, representing microseconds since the epoch. 

.SS "Other Spec Units"
There are also other spec units, that may be used:
.IP "READ" 3
Causes the program to read the next line of input. If we have already read the last line, the read line is taken to be the empty string. When a
.B READ
or
.B READSTOP
spec unit is applied, the context offset is reset to zero (the current record).
.IP "READSTOP" 3
Causes the program to read the next line of input. If we have already read the last line, no more processing is done for this iteration. When a
.B READ
or
.B READSTOP
spec unit is applied, the context offset is reset to zero (the current record).
.IP "UNREAD" 3
Causes the program to push back the current active record back to the reader, so that the next iteration of the specification or the next 
.I READ
or
.I READSTOP
spec unit will read it back.  This is useful when looping with READ searching for the next line to run the specification on.
.IP "WRITE" 3
Immediately writes the current output record and resets the output buffer to empty so that subsequent spec units in the same cycle can build a new output record.
.IP "NOWRITE" 3
Suppresses the output record for the current processing cycle. The specification continues to execute normally, but no output record is written at the end of the cycle.
.I NOPRINT
is a synonym for 
.I NOWRITE
.IP "NOPRINT" 3
A synonym for 
.I NOWRITE
.IP "ASSERT" 3
Followed by a condition, this keyword halts the specification immediately if the condition evaluates to 
.B false
or zero. The normal use of assertions is to make sure that the read data or the internal state makes sense. 
.IP "ABEND" 3
The 
.I ABnormal END 
keyword is used to abruptly end the run of the specification. The token that follows the ABEND token is output in the error message. ABEND units make sense only within conditionals as they are never part of the normal processing or a record.
.IP "WORDSEPARATOR and FIELDSEPARATOR" 3
Declares a string of characters to be the word of field separators respectively, which affects word and field ranges. For 
.I WORDSEPARATOR 
it is possible to use the special value 
.I default
to make all whitespace defined by the locale work as a word separator. This keyword can also be used locally in a 
.I SUBSTRING
spec unit.
.IP "REDO" 3
Causes the current output line to become the new input line.
.IP "SPLITW" 3
Splits the current input record by words and produces one output record for each word. 
Any spec units before the 
.B SPLITW
form a prefix that is replicated in each output record.
Any spec units after the 
.B SPLITW
(such as
.B REDO
) are applied to each output record individually.
An optional 
.B WORDSEPARATOR
may follow the 
.B SPLITW
keyword to specify a custom word separator for splitting.
An optional 
.B OF
clause may follow to specify which part of the input record to split. The 
.B OF
clause accepts the same input parts as 
.B SUBSTRING:
a character range, a word range, or a field range.
.P
Example:
.P
    echo "one two three" | specs splitw 1
.P
produces three output records: "one", "two", "three".
.P
    echo "one two three" | specs 'prefix:' 1 splitw nextword
.P
produces: "prefix: one", "prefix: two", "prefix: three".
.P
    echo "a:b:c are items" | specs splitf fs : of word 1 1
.P
produces: "a", "b", "c" (splitting only the first word by field separator).
.P
Nested 
.B SPLITW
or
.B SPLITF
units in the same specification are not allowed.
.IP "SPLITF" 3
Splits the current input record by fields and produces one output record for each field. 
Works like 
.B SPLITW
but splits by field separator instead of word separator. Empty fields are preserved.
An optional 
.B FIELDSEPARATOR
may follow the 
.B SPLITF
keyword to specify a custom field separator for splitting.
An optional 
.B OF
clause may follow to specify which part of the input record to split, accepting the same input parts as 
.B SUBSTRING.
.IP "SET" 3
Followed by a single assignment operation or multiple assignment operations separated by semicolons, SET implements this and assigns a value to the system counters. See the Expressions and Assignments section.
.IP "CONTINUE" 3
Stops execution of the current cycle immediately and goes on to the next record. Any output already accumulated for the current cycle remains in the output buffer, and is handled by the normal end-of-cycle logic: it is written, unless suppressed by 
.B NOWRITE
or by 
.B PRINTONLY,
and if 
.B PRINTONLY KEEP
is in effect, the buffered output may be carried forward instead of being discarded. This is typically used within an 
.B IF
block.
.IP "SKIP-WHILE and SKIP-UNTIL" 3
These statements are one-time gates for the beginning of a specification. They cause the specification to skip input lines
.B while
a condition is true, or
.B until
the condition becomes true. More precisely, each of them stops the current cycle and goes on to the next record, similar to the 
.B CONTINUE
unit, but once the pass condition occurs once, the condition is never evaluated again and future records are passed unchecked. It usually only makes sense to place 
.B SKIP-WHILE
and
.B SKIP-UNTIL
units at the beginning of the specification.
.P
For example:
.P
    specs w1 a: SKIP-WHILE "a<20200701" 1-* 1
.P
skips records until the first record whose first word is at least 20200701, then processes that record and all subsequent records normally.
.P
Similarly:
.P
    specs w1 a: SKIP-UNTIL "a=BEGIN" 1-* 1
.P
skips records until the first record whose first word is BEGIN, then processes that record and all subsequent records normally.
.IP "CONTEXT" 3
Temporarily changes the working string to a nearby record without consuming it.
.B CONTEXT
is followed by an integer offset:
.B CONTEXT 0
resets the working string to the current record,
.B CONTEXT 1
sets it to the next record,
.B CONTEXT -1
sets it to the previous record, and so on.
The required buffer sizes are computed at parse time from the largest positive and negative offsets used. If the offset refers to a record beyond the beginning or end of the input, the working string is set to the empty string. Note that reading beyond the input with
.B CONTEXT
does not cause processing to stop, even if a
.B READSTOP
token is present in the specification.
.P
.B CONTEXT
must not be abbreviated. It is not supported with threading or multiple input streams.
.P
Example:
.P
    echo -e "A\\nB\\nC" | specs 1-* 1 CONTEXT 1 1-* NW
.P
produces: "A B", "B C", "C" -- each line shows the current record followed by the next.
.P
In expressions, the syntax
.B @+n
and
.B @-n
provides access to nearby records without changing the working string. For example,
.B @+1
refers to the next record and
.B @-1
refers to the previous record.
.B @+0
and
.B @-0
are equivalent to
.B @@
(the current record).
.P
Example:
.P
    echo -e "A\\nB\\nC" | specs print @+1 1
.P
produces: "B", "C", "" -- each line shows the next record (empty for the last).
.P
Note that reading beyond the input with
.B @+n
or
.B @-n
does not cause processing to stop, even if a
.B READSTOP
token is present in the specification.

.SS "Out-of-Bounds (OOB) Records"
When
.B CONTEXT
or an
.B @\(+-n
expression refers to a record beyond the beginning or end of the input, or when a
.B READ
runs dry, the working string becomes an
.I out-of-bounds
(OOB) record. An OOB record behaves as an empty string, but it additionally carries a hidden flag marking it as out of bounds. This flag lets you tell the difference between a record that is genuinely empty and one that is empty only because it lies past the edge of the input. The flag is queried with the
.B ctxoob()
function (see
.B "Built-In Functions" ).
.P
The OOB flag travels with the working string only as long as the OOB record is read
.I directly.
It is preserved by the following:
.RS
.IP \(bu 3
.B ctxoob()
with no argument (it inspects the current working string).
.IP \(bu 3
The record-access functions
.B record(),
.B range(),
.B substr(),
.B word(),
.B wordrange(),
.B field()
and
.B fieldrange()
when they operate on the current (OOB) record.
.IP \(bu 3
The
.B @@,
.B @!
and
.B @\(+-n
input-record expressions.
.IP \(bu 3
Character, word and field range labels (for example
.B "1-5 a:" ,
.B "w1-3 x:"
or
.B "f1 y:" )
that capture the OOB record. The captured label, e.g.
.B a,
remains OOB and can be tested with
.B ctxoob(a).
.RE
.P
The OOB flag is
.B not
preserved once the value is copied into a plain text value that no longer refers to the working string. Known cases where the flag is lost include:
.RS
.IP \(bu 3
Assigning into a numbered counter with a
.B SET
spec unit (or any
.B :=
assignment). For example, after
.B "CONTEXT 1 set #5:=record()" ,
the expression
.B ctxoob(#5)
returns 0, because the counter stores only the (empty) text and not the OOB flag.
.IP \(bu 3
Values that pass through an output-producing spec unit such as
.B PRINT
before being re-captured. For example, in
.B "CONTEXT 1 1-5 a: PRINT \(dqa\(dq b: PRINT \(dqctxoob(b)\(dq" ,
the label
.B b
is no longer OOB and
.B ctxoob(b)
returns 0.
.IP \(bu 3
Other operations that materialize the working string as ordinary text may likewise drop the flag.
.RE
.P
In short: query OOB status as close as possible to where the OOB record is read, and do not expect it to survive a round trip through a counter or other plain-text storage.

.SS "MainOptions"
These are optional spec units that appear at the beginning of the specification and modify the behavior of the entire specification.
.IP "STOP" 3
This option is followed by either the keyword
.B ALLEOF,
the keyword
.B ANYEOF,
or a number indicating an input stream. This indicates when the specification stops. The default is
.B ALLEOF
which means that the specification terminates when every input stream is exhausted. When some but not all of the streams are exhausted, those that are get treated as if they emit empty records. With 
.B ANYEOF
the specification terminates when any of the streams is exhausted. With a numeric value the specification terminates when the specified stream is exhausted. Other streams, if exhausted are treated as if they emit empty records.
.IP "PRINTONLY" 3
This option instructs 
.B specs
to suppress output records unless a specified
.I break level
is established. The break level is either a field identifier (case matters) or it can be the keyword
.B EOF 
which specifies records are suppressed until the input is exhausted or the condition specified with 
.B STOP 
is satisfied.
.IP "KEEP" 3
This option, always following 
.B PRINTONLY
instructs
.B specs
not to reset the output buffer when a record in not output due to break level not being established. This allows the content from several records to be aggregated into a single output record.

.SS "Conditions and Loops"
A specification can include conditions and loops. 

Conditions begin with the word 
.B if
followed by an 
.B expression 
that evaluates to true of false, followed by the token
.B then,
followed by some 
.B Spec Units.
Those will be executed only if the condition evaluates to true. They may be followed by an
.B else
token followed by more
.B Spec Units 
that will be executed if the expression is not true, or they may be followed by an
.B elseif
token with its own condition, 
.B then 
token, and set of 
.B Spec Units.
The chain of 
.B elseif
tokens may be arbitrarily long, but there may only be at most one 
.B else
token. The conditional block ends with an
.B endif
token.  For example:

.RS 5
.B if
#2 > 5
.B then 
.RS 4
/big/ 1
.RE 
.B elseif
#2 > 3
.B then 
.RS 4
/medium/ 1
.RE
.B else 
.RS 4
/small/ 1
.RE 
.B endif
.RE

The loop available in 
.B specs
is a 
.B while
loop.  It begins with the 
.B while
token, followed by an 
.B expression 
that evaluates to true of false, followed by the token
.B do,
and a series of 
.B Spec Units
that will be executed as long as the expression evaluates to true. The series of 
.B Spec Units is terminated by the token
.B done.
Example:

.RS 5
.B while
#2 > 0
.B do
.RS 4
print /#2/ 1
.RS 1
.RE 
write
.RS 1
.RE 
set /#2 -= 1/
.RE
.B done
.RE

.SS "While-Guard"
The
.I While-Guard
feature, available from version 0.9.5 makes an effort to prevent
.B specs
from entering endless loops. For example, consider this specification:

.RS 5
.B set
"#1 := 5"
.RS 1
.RE 
.B while
#1 > 3
.B do
.RS 4
.B set
"#1 += 1"
.RE
.B done
.RE

Without
.I while-guard
this specification will loop forever. To solve this, 
.B specs
keeps a counter for each 
.I while
statement that it increments each time the loop is entered. This counter is reset to zero if a record is read from any input.  The 
.B specs
program exists when the counter reaches 5000.

.I While-Guard
is not perfect. To disable it, you can use the command-line switch
.B --no-while-guard
or you can override the maximum iteration count at which the program exist by setting the 
.B while-guard-limit
to some integer value.

.SS "Control Breaks"
The 
.B Field Identifiers
have another use. When used with the
.B BREAK
keyword or the 
.B break()
function they act as flow control. A 
.B break level 
is 
.I established
when the value of the corresponding field identifier changes from the previous iteration. For
example, consider a three-field CSV file where the first field is the department name, the second is 
the employee's first name, and the third is his or her last name.  Suppose further that the file is 
sorted by department name. You can print this out without repeating the department name like this:

          FIELDSEPARATOR ,
       c: FIELD 1   .
          FIELD 3  10
          /,/      NEXT
          FIELD 2  NEXTWORD
          BREAK c
              ID c 1

.SS "Run-In and Run-Out Cycles"
A 
.B cycle 
is a single run of the specification on the current active input record. A cycle may read additional input records, produce zero output records, or produce multiple output records. If the specification contains 
.B read 
or 
.B readstop 
tokens, a single cycle can consume more than one input record.

The 
.B run-in 
cycle is the first cycle. In the run-in cycle, the function 
.B first()
returns 1. This can be used for initial processing such as printing of headers or setting initial values. 

The 
.B run-out 
cycle happens 
.I after 
the last line has been read, but only when the specification requires a run-out cycle. It consists of the spec items that follow the 
.B EOF 
token, or (when 
.I select second 
is used) conditional specifications with the 
.B eof() 
function. Example:

    if first() then
        /Item/  1  /Square/ nw write
        /====/  1  /======/ nw write
    endif
    a:  w1 1.4 right
        print "a*a" 6.6 right
        set '#0+=(a*a)'
    EOF
        /==========/ 1 write
        /Total:/ 1
        print #0 nw


Note that any use of the
.B eof()
function anywhere in the specification forces a run-out cycle. This is called a
.I forced run-out cycle
and any use of record-reading spec units, such as a 
.I data field 
or
.B ALU
functions that access records, like
.B record()
or
.B word(i)
will result in an empty string returned.
        
.SS "Input Streams"
The keyword 
.B SELECT
along with the keywords 
.B FIRST
and
.B SECOND 
can be used to select between two input stations. The 
.B FIRST 
input station is the regular primary input. The
.B SECOND 
input station holds the previous record from the primary input.

Imagine an input stream that has the natural numbers. Let's run the following specification:

    specs w1 1 /- 1 =/ NW SELECT SECOND w1 nw
    
The result will be the following:
    1 - 1 =
    2 - 1 = 1
    3 - 1 = 2
    4 - 1 = 3
    5 - 1 = 4

and so on.

The keyword
.B SELECT
can also be used with a number between 1 and 8 to denote different input streams. These extra streams are specified with the
.B --is2 - --is8
command-line switches to specify the files that contain the other input streams. Every cycle of the specification begins with 
.B specs
set to stream #1.  For example, suppose we have two files: f1 and f2, both of which contain lists of numbers, and we want to add them:

    specs -i f1 --is2 f2 a: W1 1 '+' NW  SELECT 2  b: W1 NW  '=' NW  "?a+b" NW
    
If both files contained the natural numbers we would get the following output:
    1 + 1 = 2
    2 + 2 = 4
    3 + 3 = 6
    4 + 4 = 8
    5 + 5 = 10

and so on.

.SS "Output Streams"
The keyword
.B OUTSTREAM
can be used to select between multiple output streams. Output stream number #1 is the primary default output stream. Output streams #2 through #8 can be files specified with the
.B --os2 - --os8
command-line switches.  There is one additional output stream, which is the standard error.  It is selected with the keyword
.B STDERR
for a total of up to 9 output streams.  Example:

    ls -l | specs --os2 filesizes --os3 owners 
                W9 1 WRITE 
                OUTSTREAM 2 W5 1 WRITE
                OUTSTREAM 3 w3 1
                
This specification will print the list of owners to file 
.I owners
the file sizes to file
.I filesizes
and the file names to standard output.

.SS "Expressions and Assignments"
Expressions are used in PRINT data units as well as in assignments. Assignments are used in SET Spec Units.

Expressions are made up of numbers, field identifiers (no colon needed), and counter numbers preceded by a hash mark. They allow ordinary arithmetic and logical operations as well as function calls for pre-defined functions:
.IP "Unary Operators" 3
+ (plus - does nothing), - (minus), ! (logical NOT - equivalent to the 
.I not
built-in function)
.IP "Binary Arithmetic Operators" 3
+, -, *. / (division), // (integer division), % (remainder)
.IP "Binary String Operator" 3
|| (string append)
.IP "Binary Arithmetic/String Logical Operators" 3
<, >, <=, >=, =, !=
.IP "Binary Strict Logical Operators" 3
<<, >>, <<=, >>=, ==. !==
.IP "Binary Logical Operators" 3
& (AND), | (OR)

.P 
Assignments assign the result of an expression into a 
.I numbered counter. 
For example:

    specs SET #3:=b+3
.P
.I SET
statements can also be compound. For example:

    specs SET "#3:=b+3 ; #3:=b-3"
.P
Assignment can also be used as expressions. When used as such, the value returned is the content of the counter after executing the assignment. For example, the following specification:

    specs print "#0:=2+2" 1
    
OR

    specs "?#0:=2+2" 
    
will output 4.  The assignment is performed and the value stored in the counter.
	
.IP "Assignment Operators" 3
:=, +=, -=, *=, /=, //=, %=, ||=

.IP "At-sign (@) Operator" 3
The at-sign allows the inclusion of user-defined and system-defined labels as strings in expressions. The double at-sign (@@) substitutes for the entire input record.
The syntax
.B @+n
and
.B @-n
(where n is a natural number) accesses nearby records from the rolling context buffer.
.B @+1
is the next record,
.B @-1
is the previous record.
.B @+0
and
.B @-0
are equivalent to
.B @@.
The
.B @!
syntax substitutes for the current record as affected by
.B CONTEXT.
Without
.B CONTEXT
in effect,
.B @!
is equivalent to
.B @@.
When
.B CONTEXT
is active,
.B @@
still returns the original input record, while
.B @!
returns the context-affected record.
Out-of-bounds offsets return the empty string. The forward and backward buffer sizes are computed automatically at parse time. This feature requires non-threaded mode and a single input stream.

.SS "Built-In Functions"
.IP "abs(x)" 3
The absolute function returns the absolute value of the number passed to it. Will return an integer if the parameter is integer, or float otherwise.
.IP "ceil(x), floor(x), round(x,d)" 3
These function return the closest numbers to 
.I x
:
.B ceil
returns a whole number no smaller than 
.I x
;
.B floor
returns a whole number no greater than 
.I x
;
.B round
returns a the number closest to 
.I x
with up to
.I d
decimal places. If 
.I d
is omitted, 
.B round 
returns the closest whole number.
.IP "fact(n)" 3
The
.B fact
function returns the factorial of 
.I n
.IP "combinations(n,k) and permutations(n,k)" 3
These function give the number of ways in which 
.I k
elements can be chosen from a set of 
.I n
elements. The difference is that with 
.B combinations
the order of the elements within the subgroup (or the order in which they were selected) 
does not matter, while with
.B permutations,
it does. For example, if we are choosing 2 elements from a group of 6,
.B permutations
will return 30, because there are 6 ways to choose the first element and 5 to choose the 
second element. On the other hand, 
.B combinations
will return 15, because it doesn't matter which of the two chosen elements was chosen first.
.IP "fmt(value,format,digits,decimal,separator)" 3
The
.B fmt
function formats a floating-point
.B value
as a string. The 
.B format 
argument can be omitted, or it can begin with 
.I f
for a 
.I fixed 
number of 
.B digits
after the decimal point, or
.I s
for 
.I scientific
notation. When omitted, the 
.B digits
argument sets the total number of digits displayed. The 
.B decimal
argument sets the character used for the decimal point (default is a period), while the 
.B separator
argument sets the character used as thousands separator (default is none).
.IP "pretty(value,flimit,ilimit,locale)" 3
The
.B pretty
function formats the number in 
.B value
as a printable string with commas.  The optional 
.B flimit
and
.B ilimit
parameters determine how many digits the floating-point or integer number should have (to the left of the decimal point) before the function switches to scientific notation. These parameters default to 10 and infinite digits respectively.  The number is formatted according to the locale specified in the 
.B locale 
configuration string. The optional
.B locale
parameter overrides that configuration.
.IP "pow(x,y)" 3
The power function returns x to the power of y. Much like arithmetic operations, if either operand is float, the result is a float. If both operands are integers, the result is an integer. Otherwise, if both operands are whole numbers, the result is integer, otherwise it is float.
.IP "rand(x)" 3
The random function returns a random integer value between 0 up to and not including 
.B x.
If 
.B x
is omitted, returns a random
.I real
value no smaller than 0.0 and smaller than 1.0.
.IP "sin(x), cos(x), tan(x), dsin(x), dcos(x), dtan(x)" 3
These trigonometric functions return the sine, cosine and tangent of an angle.
.B sin, cos,
and
.B tan
accept an argument in radians, while
.B dsin, dcos,
and
.B dtan 
accept an argument in degrees.
.IP "arcsin(x), arccos(x), arctan(x), arcdsin(x), arcdcos(x), arcdtan(x)" 3
These trigonometric functions return the inverse sine, cosine and tangent.
.B arcsin, arccos,
and
.B arctan
return an angle in radians, while
.B arcdsin, arcdcos,
and
.B arcdtan 
return an angle in degrees.
.IP "exp(x) and log(x,base)" 3
These return the 
.B exponent 
of x or the 
.B logarithm 
of x. By default, the logarithm function returns the 
.I natural 
log.
.IP "string(x)" 3
Returns the same value as the argument, but forced to be stored as a string. There is nothing preventing this from later evaluating as a number, so
.I string(3)+2
will evaluate to 5.
.IP "substitute(haystack,needle,subst,max)"
Returns the string in 
.I haystack
where occurences of 
.I needle
are replaced by the string 
.I subst
for at most 
.I max
(by defualt: 1) times. The special value 
.B """U"""
for 
.I max
is used to indicate that all occurrences should be replaced.
.IP "sqrt(x)" 3
The square root function always returns a float.
.IP "c2u(x)" 3
Returns an unsigned decimal number from the binary representation of x.
.IP "c2d(x)" 3
Returns a signed decimal number from the binary representation of x.
.IP "c2f(x)" 3
Returns a floating point number assuming that x is a binary representation of a floating point number in the native encoding of the platform. On most platforms valid lengths are 4, 8, and 16 bytes (32, 64, and 128 bits).
.IP "frombin(x)" 3
Returns a decimal number from the binary representation in x.
.IP "tobine(x,d)" 3
Returns a binary representation of the unsigned integer in x with field of length d bits. Valid values for d are 8, 16, 32, and 64.
.IP "tobin(x)" 3
Returns a binary representation of the unsigned integer in x. The field length is automatically determined by the value of x, but will be 1, 2, 4, or 8 characters in length.
.IP "length(x)" 3
Returns the length of the argument when viewed as a string. For example, len(37) is 2; len('hello') is 5.
.IP "first()" 3
Returns 1 during the 
.B run-in
cycle, and zero otherwise.
.IP "eof()" 3
Returns 1 during the
.B run-out 
cycle, and zero otherwise. Its use in a specification forces a run-out cycle.
.IP "number()" 3
Returns the number of times the specification has so far been run on different records.
.IP "recno()" 3
Returns the number of input records that have been read so far. If the specification contains no 
.I READ
or
.I READSTOP
spec units, then this number will be equal to the result of 
.B number().
Otherwise, it will be higher.
.IP "ctxrecno()" 3
Returns the record number of the record that input parts work on. This is similar to
.B recno(),
but considers rolling context, which
.B recno()
does not. When no
.I CONTEXT
is active, returns the same value as
.B recno().
.IP "ctxoffset()" 3
Returns the current effective context offset. When no
.I CONTEXT
is active, returns 0.
.IP "ctxoob()" 3
Returns 1 if the argument string came from out-of-bounds input (either via
.I CONTEXT
or
.I @±n
reading past the input edges, or via a
.I READ
that ran dry), and 0 otherwise. With no argument, checks the current (context-affected) record. The out-of-bounds property is not preserved by all operations; see the subsection
.B "Out-of-Bounds (OOB) Records"
for details and limitations.
.IP "record()" 3
Returns the entire input record. Equivalent to 
.B range(1,-1)
.IP "cfrecord()" 3
Returns the entire input record, disregarding rolling context. When
.I CONTEXT
is active,
.B record()
returns the context-affected record, while
.B cfrecord()
returns the original input record. Equivalent to
.B @@.
.IP "range(s,e)" 3
Returns the range of characters from
.B s
(default first) to 
.B e 
(default last) from the input record. If the end of the range exceeds the end of the input, the result is truncated at the end. If the start of the range exceeds the end of the input, the result is the empty string. If the start is greater than the end, the function returns NaN.
.IP "wordcount(s,p)" 3
Returns the number of words in the string 
.B s, 
or in the current record if 
.B s 
is not specified. The separator used is
.B p. 
If 
.B p 
is not specified, the separator is the current word separator if processing the current record, or a 
.I blank space
if processing 
.B s.
.IP "word(i)" 3
Returns the i-th word in the input record.
.IP "wordstart(i)" 3
Returns the starting position of the i-th word in the input record.
.IP "wordend(i)" 3
Returns the ending position of the i-th word in the input record. 
.I range(wordstart(i),wordend(i))
is equivalent to 
.I word(i)
but less efficient.
.IP "wordrange(s,e)" 3
Returns the string from the start of the 
.B s-th
word (default first) to the end of the 
.B e-th 
word (default last). It is also equivalent to 
.I range(wordstart(s),wordend(e))
but less efficient.
.IP "fieldcount(s,p)" 3
Returns the number of fields in the string 
.B s, 
or in the current record if 
.B s 
is not specified. The separator used is
.B p. 
If 
.B p 
is not specified, the separator is the current field separator if processing the current record, or a 
.I tab 
if processing 
.B s.
.IP "field(i)" 3
Returns the i-th field in the input record.
.IP "fieldstart(i)" 3
Returns the starting position of the i-th field in the input record.
.IP "fieldend(i)" 3
Returns the ending position of the i-th field in the input record. 
.I range(fieldstart(i),fieldend(i))
is equivalent to 
.I field(i)
but less efficient.
.IP "fieldrange(s,e)" 3
Returns the string from the start of the 
.B s-th
field (default first) to the end of the 
.B e-th 
field (default last). It is also equivalent to 
.I range(fieldstart(s),fieldend(e))
but less efficient.
.IP "splus(s,o,[l])" 3
Returns a substring of the current record, starting with the first character at offset 
.B o
from the first instance of the string
.B s,
and having a length of 
.B l
which defaults to 1.
.B o 
may be non-positive. If the search string 
.B s
is not found or if the start of the substring underflows or overflows the input record, an empty string is returned.
.IP "wplus(s,o,[l])" 3
Returns a substring of the current record, starting with the first word at offset 
.B o
(measured in words) from the first instance of the word
.B s,
and having a length of 
.B l
words, which defaults to 1.
.B o 
may be non-positive. If the search word 
.B s
is not found or if the start of the substring underflows or overflows the input record, an empty string is returned.
.IP "fplus(s,o,[l])" 3
Returns a substring of the current record, starting with the first field at offset 
.B o
(measured in fields) from the first instance of the field
.B s,
and having a length of 
.B l
fields, which defaults to 1.
.B o 
may be non-positive. If the search field 
.B s
is not found or if the start of the substring underflows or overflows the input record, an empty string is returned.
.IP "tf2mcs(string,format)" 3
Converts a string containing a date and time representation using the format specified in the 
.I format
argument to a floating point or integer number representing the number of microseconds since the UNIX epoch. Formatting is as described for the similarly named conversion.
.IP "mcs2tf(microseconds,format)" 3
Converts a number (integer or float) representing the number of microseconds since the UNIX epoch to a printable format.
.IP "tf2s(string,format)" 3
Converts a string containing a date and time representation using the format specified in the 
.I format
argument to a floating point or integer number representing the number of seconds since the UNIX epoch. Formatting is as that of tf2mcs.
.IP "s2tf(seconds,format)" 3
Converts a number (integer or float) representing the number of seconds since the UNIX epoch to a printable format.
.IP "x2d(string,length)" 3
Returns the value in 
.I string
converted to decimal. If 
.I length is missing or non-positive, the resulting decimal is signed, otherwise it is unsigned.
.IP "next()" 3
Returns the column of the next character to print if a
.I spec unit
specifies the 
.B NEXT
position.
.IP "exact(expression)" 3
Returns whether the evaluation of
.I expression
results in an exact result (1) or not (0). 
.B NOTE:
Python functions use a heuristic where integer and string returns are exact, while float returns are inexact. This may not always be correct. Python functions can override this by returning a 2-tuple
.B (value, exactness)
to explicitly specify exactness. When unsure, the
.B exact
function errs on the side of returning 0.

.SS "Built-In String Functions"
.IP "abbrev(information, info, len)" 3
Returns
.I 1
if 
.B information
begins with the first
.B len
characters of 
.B info
or 
.I 0 
otherwise. If
.B len
is omitted, all of 
.B info 
is considered.
.IP "substr(bigstring, start, length)" 3
Returns a substring of 
.I bigstring
starting at offset 
.I start
with a length of 
.I length
. If the length exceeds the end of the string, it is truncated. As always in 
.B specs
the indexes are 1-based. Negative indexes can be used to indicate offsets from the end of the string.
.IP "left(bigstring, length)" 3
Returns
.I length
characters starting at the beginning of 
.I bigstring.
If the length exceeds the length of bigstring, the result will be right-padded with blanks. If the 
length is negative, the length of the string + 1 will be added to it, so 
.B left(bigstring,-3)
is the entire string less the last two characters.
.IP "right(bigstring, length)" 3
Returns the last
.I length
characters of 
.I bigstring.
If the length exceeds the length of bigstring, the result will be left-padded with blanks. If the 
length is negative, the length of the string + 1 will be added to it, so 
.B right(bigstring,-3)
is the entire string less the first two characters.
.IP "center(bigstring, length)" 3
Returns
.I length
consecutive characters from
.I bigstring
as centered as possible in the string. If the length exceeds the length of bigstring, the result will be right-padded and left-padded with blanks. If the 
length is negative, the length of the string + 1 will be added to it, so 
.B center(bigstring,-3)
is the entire string less the first and last characters. This function can also be spelled 
.B centre
because the CMS Pipelines version also does so.
.IP "pos(needle, haystack)" 3
Returns the 1-based index of the 
.B first 
position of 
.I needle
in 
.I haystack. Note that if the 
.I haystack
argument is omitted, it defaults to the entire input line.
.IP "lastpos(needle, haystack)" 3
Returns the 1-based index of the 
.B last 
position of 
.I needle
in 
.I haystack.
Note that if the 
.I haystack
argument is omitted, it defaults to the entire input line.
.IP "includes(haystack, needle1 [,needle2, needle3, needle4])" 3
Returns
.B TRUE
(1) if the string 
.I haystack
includes the string
.I needle1
or one of the other optional 
.I needle2, needle3,
or
.I needle4
as a substring, or 
.B FALSE
(0) otherwise. To clarify, the function returns true if
.B any
of the needles match. Note that if the 
.I haystack
argument is omitted, it defaults to the entire input line.
.IP "includesall(haystack, needle1 [,needle2, needle3, needle4])" 3
Returns
.B TRUE
(1) if the string 
.I haystack
includes the string
.I needle1
and all of the other defined optional
.I needle2, needle3,
or
.I needle4
as substrings, or 
.B FALSE
(0) otherwise. To clarify, the function returns true if
.B all
of the needles match. Note that if the 
.I haystack
argument is omitted, it defaults to the entire input line.
.IP "rmatch(string, regEx [,matchFlags])" 3
Returns 
.B TRUE
(1) if the regular expression
.I regEx
matches the string
.I string,
or 
.B FALSE
(0) otherwise. See the special section below for more info on regular expressions. Note that if the 
.I string
argument is omitted, it defaults to the entire input line.
.IP "rsearch(string, regEx [,matchFlags])" 3
Returns 
.B TRUE
(1) if the regular expression
.I regEx
matches some substring of the string
.I string,
or 
.B FALSE
(0) otherwise. See the special section below for more info on regular expressions. Note that if the 
.I string
argument is omitted, it defaults to the entire input line.
.IP "rreplace(string, regEx, fmt [,matchFlags])" 3
Returns the string 
.I string,
with all matches of the regular expression 
.I regEx
replaced by what's in
.I fmt.
See the special section below for more info on regular expressions. Note that if the 
.I string
argument is omitted, it defaults to the entire input line.
.IP "conf(key,default)" 3
Returns the configured string from either the built-in configured strings such as 
.I version
or from the 
.B $HOME/.specs
file. If the string is not defined, returns the
.I default
value. If that is not defined, returns
.B NaN.
.IP "defined(key)" 3
Returns 
.B TRUE
(1) if the configured string 
.I key
is defined, or 
.B FALSE
(0) otherwise.
.IP "getenv(name)" 3
Returns the content of the environment variable 
.I name
or
.B NaN
if the environment variable of that name is not defined.
.IP "pset(var,value)" 3
Sets the
.B persistent variable
with the name in 
.I var
to the value
.I value
The value will be set in future invocations of 
.B specs.
The function returns the value in 
.I value. 
.IP "pget(var,default)" 3
Returns the value of the
.B persistent variable
with the name in
.I var
If the variable is not defined, it returns the 
.I default
value. If that is unspecified, returns 
.B NaN.
It is also possible to use the variable name preceded by a hash mark, as in 
.I #var
as an abbreviated alternative to
.I pget(var)
.IP "pdefined(var)" 3
Returns 
.B TRUE
(1) if the
.B persistent variable
.I var
is defined, or 
.B FALSE
(0) otherwise.
.IP "pclear(var)" 3
Clears the 
.B persistent variable
.I var.
Returns the value before clearing it. If the variable was not defined, returns
.B NaN.
.IP "sfield(str,n,sep)" 3
Returns the n-th field of string
.I str
counting from the start or end of the string depending on the sign of n. Separator is the first char of 
.I sep
if specified and non-empty, or 
.B tab
otherwise.
.IP "lvalue(str,sep)" 3
Returns the left hand side of strings such as 
.B pi=3.14159265. 
For this input it will return the string 
.B pi. 
.I sep 
defaults to the equals sign, but can be overridden for input strings such as 
.B idnum:543
.IP "rvalue(str,sep)" 3
Returns the right hand side of strings such as 
.B pi=3.14159265. 
For this input it will return the string 
.B 3.14159265. 
.I sep 
defaults to the equals sign, but can be overridden for input strings such as 
.B idnum:543
.IP "sword(str,n,sep)" 3
Returns the n-th word of string
.I str
counting from the start or end of the string depending on the sign of n. Separator is the first char of 
.I sep
if specified and non-empty, or 
.B space
otherwise.

.SS "Built-In REXX-Derived Functions"
.IP "bitand(x,y)" 3
Returns the bit-wise 
.B AND
of 
.I x
and 
.I y
If the length is not equal, the return string has the minimum length.
.IP "bitor(x,y)" 3
Returns the bit-wise 
.B OR
of 
.I x
and 
.I y
If the length is not equal, the return string has the minimum length.
.IP "bitxor(x,y)" 3
Returns the bit-wise 
.B XOR
of 
.I x
and 
.I y
If the length is not equal, the return string has the minimum length.
.IP "compare(s1,s2,pad)" 3
Returns the index of the first mis-matched character in strings 
.I s1
and
.I s2.
If the strings are equal, the function returns
.B zero.
If the strings are of unequal length, the shorter string is padded with the pad character (by default - space).
.IP "copies(string,times)" 3
Returns the content of
.I string 
repeated
.I times
times.
.IP "delstr(string,start,length)" 3
Deletes the substring of 
.I string 
that starts at position 
.I start 
for the specified 
.I length. 
If 
.I length 
is zero, the rest of the string is deleted from position start to the end.
.IP "delword(string,start,length)" 3
Deletes the substring of 
.I string 
that starts at position 
.I start 
and is of length 
.I length 
blank-delimited words. If 
.I length 
is zero, it defaults to removing the rest of the words in 
.I string.
.IP "find(string,phrase)" 3
Returns the word number of the first occurrence of 
.I phrase 
in 
.I string. 
Returns 0 if 
.I phrase 
is not found. Multiple blanks between words are treated as one in comparisons.
.IP "index(haystack,needle,start)" 3
Returns the character position of 
.I needle
within string
.I haystack. 
Returns 0 if 
.I needle 
is not found. If positive, 
.I start tells where in
.I haystack 
to initiate the search. It defaults to 1 if not specified. The standard
.B pos 
function should be used instead of 
.B index 
if possible.
.IP "insert(string,target,position,length,pad)" 3 
Inserts 
.I string
into 
.I target
at position 
.I position 
and truncated or padded with 
.I pad
characters to length 
.I length. 
With default or zero values, 
.I position 
inserts the 
.I string 
at the start of 
.I target, 
and the length of the 
.I string
is kept as is. The 
.I pad
defaults to a space.
.IP "justify(string,length,pad)" 3
Evenly justifies words within 
.I string. 
The 
.I length
specifies the length of the returned string, while
.I pad
specifies what padding (by default a space) to insert (if necessary).
.IP "overlay(string1, string2 ,start ,length ,pad)" 3
Returns a copy of 
.I string2, 
partially or fully overwritten by 
.I string1. start 
specifies the starting position of the overlay. 
.I length 
truncates or pads 
.I string1 
prior to the operation, using 
.I pad 
as the pad character.
.IP "reverse(string)" 3
Returns a copy of a 
.I string 
with its characters reversed.
.IP "sign(number)" 3
Returns 1 if the 
.I number 
is positive, 0 if the 
.I number 
is 0, and -1 if the 
.I number 
is negative.
.IP "not(expression)" 3
Returns 1 if the 
.I expression
is an integer zero, or 0 if it is anything else.  It serves as an alternative to the
.B unary logical not
operator (!).
.IP "space(string,length,pad)" 3
Formats a 
.I string 
by replacing internal blanks with 
.I length 
occurrences of the 
.I pad 
character. The default pad character is blank and the default length is 1. Leading and trailing blanks are always removed. If 
.I length 
is 0, all blanks are removed.
.IP "strip(string,option,pad-chars)" 3
Returns 
.I string 
stripped of leading and/or trailing blanks or any other 
.I char 
specified in the 
.I pad-chars
argument. 
.I Option 
values determine the action:
.B L
for leading, 
.B T
for trailing, and
.B B
for both (the default).
.IP "subword(string,start,length)" 3
Returns the substring that begins at blank-delimited word 
.I start. 
If
.I length is omitted, it defaults to the remainder of the string.
.IP "translate(string,tableout,tablein,pad)" 3
Returns a translated copy of 
.I string. 
Characters are translated according to the input translation table 
.I tablein 
and its output equivalent, 
.I tableout. 
If 
.I tablein 
and 
.I tableout 
are not coded, all characters in 
.I string 
are translated to uppercase. If 
.I tableout 
is shorter than 
.I tablein, 
it is padded with the 
.I pad 
character or its default, blanks.
.IP "verify(string, reference ,option ,start)" 3
Verifies that all characters in 
.I string 
are members of the 
.I reference 
string. Returns the position of the first character in 
.I string
that is not in 
.I reference, 
or 0 if all characters in 
.I string 
are in 
.I reference. 
.P
.I start 
specifies where in 
.I string 
to start the search, the default is 1. The option may be:
.RS 3
.P
.B N 
or Nomatch — Default. Works as described earlier.
.P
.B M
or Match — Returns the position of the first character in string that is in reference.
.IP "wordindex(string,wordno)" 3
Returns the character position of the first character of the blank-delimited word given by word number 
.I wordno 
within 
.I string. 
Returns 0 if the word numbered 
.I wordno 
does not exist in the 
.I string.
.IP "wordlength(string,wordno)" 3
Returns the length of the blank-delimited word given by word number 
.I wordno 
within 
.I string. 
Returns 0 if the word numbered 
.I wordno 
does not exist in the 
.I string.
.IP "wordpos(phrase, string [,start])" 3
If 
.I phrase 
is a substring of 
.I string, 
returns the word number position at which it begins. Otherwise returns 0. 
.I start 
is an optional word number within 
.I string 
at which the search starts. It defaults to 1.
.IP "words(string)" 3
Returns the number of blank-delimited words within the 
.I string.
.IP "wordwith(substr)" 3
Returns the first word of the input string that contains
.I substr
or an empty string if not found.
.IP "wordwithidx(substr)" 3
Returns the index of the first word of the input string that contains
.I substr
or zero if not found.
.IP "fieldwith(substr)" 3
Returns the first field of the input string that contains
.I substr
or an empty string if not found.
.IP "fieldwithidx(substr)" 3
Returns the index of the first field of the input string that contains
.I substr
or zero if not found.
.IP "xrange(start,end)" 3
Returns a string composed of all the characters between 
.I start 
and 
.I end 
inclusive. 
.I start 
defaults to 0x00, and 
.I end
defaults to 0xff.
.IP "split([sep], [hdr], [ftr])" 3
Returns all fields in the input line, one on each output line.  If 
.I sep 
is specified, it will be used instead of the active field separator. If 
.I hdr 
and/or 
.I ftr 
are specified, the first 
.I hdr 
fields and last 
.I ftr 
fields are omitted.
.IP "splitw([sep], [hdr], [ftr])" 3
Returns all words in the input line, one on each output line.  If 
.I sep 
is specified, it will be used instead of the active word separator. If 
.I hdr 
and/or 
.I ftr 
are specified, the first 
.I hdr 
words and last 
.I ftr 
words are omitted.


.SS "Built-In Statistical Pseudo-Functions" 
All of the below functions work on the values stored in field identifiers. You cannot place a proper expression inside the parenthesis.
.IP "present(a)" 3
Returns 
.I 1
if the
.B Field Identifier
a is set, or
.I 0
otherwise. 
.IP "sum(a)" 3
Returns the sum of all values in
.B Field Identifier
a. 
.IP "min(a)" 3
Returns the minimum of all values in
.B Field Identifier
a. 
.IP "max(a)" 3
Returns the maximum of all values in
.B Field Identifier
a. 
.IP "average(a)" 3
Returns the average (mathematical mean) of all values in
.B Field Identifier
a. 
.IP "variance(a)" 3
Returns the 
.I variance 
(the expectation of the squared deviation of a random variable from its mean) of the values in
.B Field Identifier
a. 
.IP "stddev(a)" 3
Returns the 
.I standard deviation 
(square root of the variance) of the values in
.B Field Identifier
a. 
.IP "stderrmean(a)" 3
Returns the 
.I standard error 
(standard deviation divided by sample size minus 1) of the values in
.B Field Identifier
a. 
.IP "fmap_nelem(a)" 3
Returns the number of distinct values of
.B Field Identifier
a. 
.IP "fmap_nsamples(a)" 3
Returns the number of samples collected of
.B Field Identifier
a. 
.IP "fmap_common(a)" 3
Returns the string value with most occurrences of
.B Field Identifier
a. In case of a tie, one of the values is returned.
.IP "fmap_rare(a)" 3
Returns the string value with least but non-zero occurrences of 
.B Field Identifier
a. In case of a tie, one of the values is returned.
.IP "fmap_count(a,s)" 3
Returns the number of occurences of the 
.B string 
s in
.B Field Identifier
a. 
.IP "fmap_frac(a,s)" 3
Returns the fraction of
.B Field Identifier
a values that are equal to the
.B string 
s.
.IP "fmap_pct(a,s)" 3
Returns the fraction of
.B Field Identifier
a values that are equal to the
.B string 
s.
.IP "fmap_sample(a,s)" 3
Treats the value of the 
.B string 
s as a new sample for
.B Field Identifier
a. Returns the count of occurences of s. 
.IP "fmap_dump(a,format,sortOrder,showPct)" 3
Returns a string containing a dump of the frequency map for
.B Field Identifier
a. If the frequency map is empty, returns the content of the configuration string
.B EmptyFrequencyMapMessage,
and if that is not defined, returns an empty string.
.IP "countocc(n,h)" 3
Returns the number of matches for this particular 
.B needle 
so far.
.IP "countocc_get(n)" 3
Returns the number of matches for this particular 
.B needle 
so far. No match is made here.
.IP "countocc_dump(format,sortOrder,showPct)" 3
Returns a string containing a dump of the 
.I frequency map 
used in 
.B countocc`.


.SH Regular Expressions
Regular expressions can be used with the 
.I rmatch, rsearch,
and
.I rreplace
functions. You use the 
.B --regexType
command line switch to select from a list of syntax options. The supported options are:
.IP "icase" 3
.B Case insensitive
.IP "nosubs" 3
.B No sub-expressions:
Not meaningful, but included here for completeness.
.IP "optimize" 3
.B Optimize matching:
Matching efficiency is preferred over efficiency constructing regex objects.
.IP "collate" 3
.B Locale sensitiveness:
Character ranges like 
.I [a-b] are affected by locale
.IP "ECMAScript" 3
.B ECMAScript grammar
The regular expression follows this grammar. Do not specify more than one of these.
.IP "basic" 3
.B Basic POSIX grammar
The regular expression follows this grammar. Do not specify more than one of these.
.IP "extended" 3
.B Extended POSIX grammar
The regular expression follows this grammar. Do not specify more than one of these.
.IP "awk" 3
.B Awk POSIX grammar
The regular expression follows this grammar. Do not specify more than one of these.
.IP "grep" 3
.B Grep POSIX grammar
The regular expression follows this grammar. Do not specify more than one of these.
.IP "egrep" 3
.B Egrep POSIX grammar
The regular expression follows this grammar. Do not specify more than one of these.
.P
.RE 3
The last parameter in all three functions is 
.I matchFlags.
This optional parameter is also a comma-separated list of flags. The supported flags are:
.IP "not_bol" 3
Not beginning of line
.IP "not_eol" 3
Not end of line
.IP "not_bow" 3
Not beginning of word
.IP "not_eow" 3
Not end of word
.IP "any" 3
Any match is acceptable if more than one is possible
.IP "not_null" 3
An empty 
.I string
does not match
.IP "continuous" 3
The expression must match a sub-sequence that begins at the first character. Sub-sequences must begin at the first character to match.
.IP "prev_avail" 3
Previous Available - One or more characters exist before the first one
.IP "sed" 3
.B (for rreplace only) 
Replaces using the same rules as the 
.B sed
POSIX utility
.IP "no_copy" 3
.B (for rreplace only) 
Sections that do not match are not copied
.IP "first_only" 3
.B (for rreplace only) 
Only the first occurrence is matched.

.SH DIRECTIVES
.B specs
specifications that are specified in a 
.B specFile
can include some directives that influence the processing. Directives
.I MUST
begin with a plus (+) sign on the left-most column of the line in the specification file. All directives 
.I MIUST
precede all spec units. Below are the supported directives:
.IP "+SET" 3
This directive followed by a single word and then a command-line command, sets a 
.B Configured String
to the output of the command. If the command issues multiple lines of output, only the first line is stored in the configured string. For example, the following directive may lead the configured string
.B ex1
to contain a value such as "total 120" on Unix-like systems:

    +SET ex1 ls -l
.IP "+IN" 3
This directive followed by a command runs the command and uses its output as the input stream for the specification.

.SH ELISIONS
.B specs
specifications can be made a little shorter by eliding some obvious tokens at the end.
.SS "Final Output Placement"
If the final
.B spec-unit
is missing its output placement, then 
.B specs
will assume a 
.I NEXTWORD
.P
    specs w1 1 w3
.P
will print the first word followed by a space and then the third word.
.SS "Final ENDIF / DONE"
Final
.I IF
or
.I WHILE
clauses can omit their terminators, 
.I ENDIF
or
.I DONE
respectively.  So
.P
    specs IF "word(1)=='UU'" THEN w2 1 "has a conflict" NEXTWORD
.P
will work just fine, as long as that 
.I IF
clause is the last thing in the specification.  In fact, in view of the previous item, you could just shorted it to 
.P
    specs IF "word(1)=='UU'" THEN w2 1 "has a conflict" 
.SS "grep-like elision of the THEN clause"
If the specification ends with an 
.I IF
clause that covers that outputs the entire record if the condition occurs, you can omit the 
.I THEN
clause.  The following two are equivalent:
.P
    specs IF "word(1)=='UU'" THEN 1-* 
.P
    specs IF "word(1)=='UU'"
    

.SH PYTHON FUNCTIONS
When compiled with Python support,
.I specs
can load user-defined functions from a Python file named
.B localfuncs.py.
The file is searched on
.B SPECSPATH,
the same path used for specification files.
It should be a normal importable Python module.
.PP
Functions whose names do not start with an underscore are made available to
.I specs
expressions. Helper functions can be kept private by starting their names with
an underscore.
.PP
For example, a
.B localfuncs.py
file can define a small formatting function:
.RS
.nf
def commas(value):
    value = int(value)
    return f"{value:,}"
.fi
.RE
.PP
It can then be called from a specification expression:
.RS
.nf
specs print "commas(word(1))" 1.12 RIGHT
.fi
.RE
.PP
Python functions may return an integer, floating-point number, string, or
.B None.
By default, integer and string returns are marked as exact, while floating-point
returns are marked as inexact. To override this heuristic, a function may instead
return a 2-tuple
.B (value, exactness)
where
.I value
is the return value (integer, float, string, or None) and
.I exactness
is a Python boolean
.B True
or
.B False.
For example,
.B return (3.14159, True)
returns an exact floating-point value.
.PP
By default, Python functions receive their arguments as plain values with exactness information stripped.
To allow a function to receive exactness information with its arguments, set the
.B arg_type
attribute on the function to the string
.B \[dq]exact\[dq].
When
.B arg_type = \[dq]exact\[dq]
is set, every argument is passed as a 2-tuple
.B (value, exactness)
where
.I value
is the argument value and
.I exactness
is a Python boolean
.B True
or
.B False
indicating whether the value is exact. This allows functions to inspect and propagate exactness information.
For example:
.RS
.nf
def lowindex(x):
    return (int(x[0]) % 65536, x[1])

lowindex.arg_type = "exact"
.fi
.RE
.PP
Docstrings are used by
.B specs --help pyfuncs
when listing loaded Python functions. Use
.B --pythonFuncs on
to force loading Python functions,
.B --pythonFuncs off
to disable them, or
.B --pythonFuncs auto
to load them only after an unknown function is encountered.

.SH OPTIONS
.B specs 
supports the following switches:
.IP "--toASCII" 3
Causes output to be translated into ASCII if it's outside the range.
.IP "--force-read-input" 3
Forces specs to read every input line even if none of the spec units use it. By default it won't.
.IP "--specFile or -f" 3
Makes the program read the specification from a file rather than the command line. This is useful both for reusing specifications and for long specifications. It also relieves the plumber from having to escape a lot of characters in the specification to avoid creative interpretation by the Unix shell.
You can specify the full path of the files, or 
.B specs 
will search the 
.I SPECSPATH 
for them. The 
.I SPECSPATH 
can be set from either the environment variable 
.B SPECSPATH 
or the configuration string 
.B SPECSPATH. 
In both cases the syntax is just like the OS 
.B PATH: 
a list of directories separated by colons on Linux and Mac OS, or semicolons on Windows. When neither the environment variable nor the configuration string are set, the 
.I SPECSPATH
defaults to 
.B $HOME/specs
.IP "--verbose or -v" 3
Outputs more information when something goes wrong.
.IP "--stats"
Output statistics on run time, and records read, and on records written to standard error.
.IP "--threaded or -t" 3
Run 
.B specs 
in separate threads for processing, for readers, and for writers. This was the default until version 0.9.5. Now the default is to run it all in a single thread.
.IP "--inFile or -i" 3
Makes the program read the input records from a file rather than from standard input.
.IP "--outFile or -o" 3
Makes the program write the output records to a file rather than to standard output.
.IP "--shell or -X" 3
Executes the output records as shell commands rather than emitting them to standard output.
.IP "--spaceWS" 3
Makes the program treat the only spaces as the default word separator. Otherwise all locale-defined whitespace is treated as the default word separator.
.IP "--debug-alu-comp" 3
Prints out detailed information about the parsing and compiling of expressions (
.I only in debug build
).
.IP "--debug-alu-run" 3
Prints out detailed step-by-step information about the evaluation of expressions (
.I only in debug build
). 
.IP "--no-while-guard" 3
Disables 
.I while-guard, 
allowing specifications to enter endless loops.
.IP "--progress" 3
While
.B specs
is running, will print once a second a message to stderr telling the user how many records have already been processed.
.IP "--config or -c" 3
Changes the configuration file from the default 
.I ~/.specs
to whatever the parameter says
.IP "--set" or "-s" 3
The following argument sets a configured string. For example: 
.B specs --set hello=goodbye
will set the configured string
.B hello
to the value
.B goodbye
.IP "--timezone" 3
Changes the timezone from the default which is the current timezone to the value specified.
.IP "--recfm" 3
Sets the record format for the primary input stream. Supported options are 
.B D (delimited)
for records delimited by a character in the input stream (this is the default); 
.B F (fixed)
for records or a fixed length (see --lrecl); 
.B FD (fixed-delimited)
for records delimited as in 
.B fixed,
but truncated or padded to a fixed length.
.IP "--lrecl" 3
Sets the record length for 
.B fixed
and
.B fixed-delimited
records.
.IP "--linedel" 3
For 
.B delimited
and
.B fixed-delimited
records, sets the character that delimits the string. The OS-specific delimiter (CR+LF for Unix-like) is used if this is not specified.
.IP "--regexType" 3
Sets the syntax option for regular expressions. The parameter is a comma-separated list of syntax options. See the 
.B Regular Expressions
section for a list of valid syntax options.
.IP "--pythonFuncs" 3
Disables or enables the use of Python functions. The parameter can be: 
.B on
to force the loading of Python functions,
.B off
to disable Python functions, or
.B auto,
which is the default, and signifies that Python functions are loaded only when the parser encounters an unknown function. Note that setting the 
.I pythonDisabled 
configured literal to 
.B 1 
will disable Python functions and cannot be overridden from the command line.
.IP "--pythonErr" 3
Determines what happens when a called Python function throws an exception. The default, 
.B throw 
is for 
.I specs 
to throw its own exception and terminate. The alternatives, 
.B NaN, zero, 
and
.B nullstr 
make 
.I specs
behave as if the function returned, NaN, the integer zero, or an empty string respectively.
.IP "--inCmd or -C" 3
Makes the program read the input records from the output of the command that follows this command line parameter. Similar to the 
.B +IN
directive for specification specified in a
.B specfile
The command must be specified in a single command line parameter, so enclose it in single or double quotes to include spaces or pipes.
.IP "--help" 3
Does not run specs. Instead it displays a help message according to the parameter: 
.B help
prints general help, 
.B pyfuncs
prints the name, arguments, and docstring of all loaded python functions, 
.B builtin
prints the name, arguments, and a line of help for the built-in functions, 
.B specs 
prints a list of all specifications on the path and the comment, and a particular python function name makes 
.I help
display the arguments and docstring of that function only.
.IP "--info" 3
Does not run specs. Instead it displays some information about this build of specs: compiler version, watermarks for queues, 
random provider, python and floating point precision.

.SH CONFIGURATION FILE
.B specs
reads a configuration file by default from 
.B $HOME/.specs
.P 
This file contains some configuration options. For now, it only allows you to set user-predefined labels and four specific configuration options: 
.B timezone, 
.B locale, 
.B while-guard-limit,
and 
.B SPECSPATH. 
For example, an entry such as this:
.P
    timezone: "Europe/Belfast"
    locale: es_ES
    SPECSPATH: /home/myself/specs:/Library/specs
    while-guard-limit: 8000
    myCA: "Honest Abe's Used Cars & Certificates"
.P
allows you to issue a 
.B specs
command like this:
.P
    specs /My certificates come from/ 1 @myCA nw 
.SS "Special configuration labels"
There is currently just one user-settable configuration label with special meaning:
.IP "NO_WARN_REDEFINED_FID" 3
If this is set to any value, then 
.B specs
suppresses the warning message if a 
.I field identifier
is re-defined within a specification.   
.SS "System-Defined Labels"
The system also defines some labels:
.IP "version" 3
This returns the version of 
.B specs, 
for example "1.0.0"
.P
To find out the version of specs that you are using, use the following command:
.P 
    specs @version
.IP "cols" 3
contains the number of columns in the display. You can override this in the configuration file. For example, the following prints a right-justified string. 
.P
    specs /hello/ (1,@cols,'R')
.IP "rows" 3
contains the number of rows in the display. This can also be overridden in the configuration file.
.IP "python" 3
contains either 
.B Enabled
or
.B Disabled
depending on whether support for python function is available.
.IP "platform" 3
contains a string describing this build. For example:
.p
    POSIX (darwin) system using the g++ compiler and Python 3.9.6 - release variation
.IP "build-commit" 3
contains the git commit hash (short form) of the build. For example:
.p
    b74fcb1
.IP "build-branch" 3
contains the git branch name of the build. May be empty if not available. For example:
.p
    dev-1.0.0
.IP "build-time" 3
contains the timestamp of when the build was created. Format is
.B yyyy-MM-ddTHH:mm:ss.
It has local time for local builds, and UTC for GitHub builds. For example:
.p
    2026-06-08T08:22:52
.IP "build-source" 3
contains either
.B local
or
.B github
depending on where the build was created.
.IP "build-number" 3
contains the build number from GitHub Actions (github.run_number). Empty for local builds.
.IP "build-runid" 3
contains the 11-digit build id from GitHub Actions (github.run_id). Empty for local builds.
.IP "build-url" 3
contains the build URL for the GitHub build. Empty for local builds.
.IP "build-info" 3
contains a composite string with all build information. For example:
.p
    Built locally from commit 8bd11da on branch dev-1.0.0 at 2026-06-08T13:01:18 local
.p
or
.p
    Built on github (id 27334782912; build 217) from commit 8bd11da of version 1.0.0 at 2026-06-08T10:01:18 UTC
.p
    
.SH EXAMPLES
`ls -l` yields this:

    total 96
    -rw-rw-r-- 1 sio sio 16432 Jun  1 12:49 dataField.cc
    -rw-rw-r-- 1 sio sio  6187 Jun  4 12:16 InputPart.cc
    -rw-rw-r-- 1 sio sio 11717 Jun  4 12:16 item.h
    -rw-rw-r-- 1 sio sio 35974 Jun  4 12:16 specItems.cc
    -rw-rw-r-- 1 sio sio  1144 Jun  4 12:16 specItems.h
    -rw-rw-r-- 1 sio sio 13953 Jun  1 12:49 splitItem.cc

Let's run it though a spec:

    ls -l | specs 12-* 1 REDO IF "wordcount()>=8" THEN w2 1 w4 d2x 8.8 RIGHT w8 17
    
The first spec unit converts it to this:

    1 sio sio 16432 Jun  1 12:49 dataField.cc
    1 sio sio  6187 Jun  4 12:16 InputPart.cc
    1 sio sio 11717 Jun  4 12:16 item.h
    1 sio sio 35974 Jun  4 12:16 specItems.cc
    1 sio sio  1144 Jun  4 12:16 specItems.h
    1 sio sio 13953 Jun  1 12:49 splitItem.cc

Then after the REDO arm, we get this:

    sio        4030 dataField.cc
    sio        182b InputPart.cc
    sio        2dc5 item.h
    sio        8c86 specItems.cc
    sio         478 specItems.h
    sio        3681 splitItem.cc
      
Alternatively, let's arrange this on multiple lines:

    specs -C "ls -l" w9 1 write "Owner:" 3 w3 10 write "Size:" 3 w5 10-20 r

    Owner:
    Size:              r
    dataField.cc
    Owner: sio
    Size:  16432       r
    InputPart.cc
    Owner: sio
    Size:  6187        r
    item.h
    Owner: sio
    Size:  11717       r
    specItems.cc
    Owner: sio
    Size:  35974       r
    specItems.h
    Owner: sio
    Size:  1144        r
    splitItem.cc
    Owner: sio
    Size:  13953       r

Finally, let's make our own version of the multi-column display:

    specs -C "ls -l" w9 1 read w9 26 read w9 51

                            dataField.cc             InputPart.cc
    item.h                  specItems.cc             specItems.h
    splitItem.cc

.SH SEE ALSO
sed(1), awk(1)
.SH BUGS
Likely some. Open a ticket in github if you find them.
.SH AUTHOR
Yoav Nir (yoav.nir@gmail.com)