PIPE( command, recorddef [, CSV | XML ] )
PIPE( recordset, command [, recorddef ] [, REPEAT] [, CSV | XML ] [, OUTPUT( CSV | XML ) ] [, GROUP] [, UNORDERED | ORDERED( bool ) ] [, STABLE | UNSTABLE ] [, PARALLEL [ ( numthreads ) ] ] [, ALGORITHM( name ) ] )
| command | The name of a program to execute, which must take any input data through stdin and produce its output through stdout. This program must have already been deployed on the HPCC Systems cluster in the Thor instance directory (such as: /var/lib/HPCCSystems/mythor/) but that can be overridden by the externalProgDir environment setting for the Thor cluster). |
| recorddef | The RECORD structure format for output. If omitted, output is the same as the input format. |
| CSV | Optional. In form 1 (and as the parameter to the OUTPUT option), specifies the output data format is CSV. In form 2, specifies the input data format is CSV. If omitted, the format is raw. |
| XML | Optional. In form 1 (and as the parameter to the OUTPUT option), specifies the output data format is XML. In form 2, specifies the input data format is XML. If omitted, the format is raw. |
| recordset | The input dataset. |
| REPEAT | Optional. Specifies a new instance of the command program is created for each row in the recordset. |
| OUTPUT | Optional. Specifies CSV or XML result data format. |
| GROUP | Optional. Specifies each result record is generated in a separate GROUP (only if REPEAT is specified). |
| UNORDERED | Optional. Specifies the output record order is not significant. |
| ORDERED | Specifies the significance of the output record order. |
| bool | When False, specifies the output record order is not significant. When True, specifies the default output record order. |
| STABLE | Optional. Specifies the input record order is significant. |
| UNSTABLE | Optional. Specifies the input record order is not significant. |
| PARALLEL | Optional. Try to evaluate this activity in parallel. |
| numthreads | Optional. Try to evaluate this activity using numthreads threads. |
| ALGORITHM | Optional. Override the algorithm used for this activity. |
| name | The algorithm to use for this activity. Must be from the list of supported algorithms for the SORT function's STABLE and UNSTABLE options. |
| Return: | PIPE returns a record set. |
The PIPE function allows ECL code to launch an external command program on each node, effectively parallelizing a non-parallel processing program. PIPE has two forms:
Form 1 takes no input, executes the command, and produces its output in the recorddef format. This is an "input" pipe (like the PIPE option on a DATASET definition).
Form 2 takes the input recordset, executes the command, producing output in the recorddef format. This is a "through" pipe.
Example:
namesRecord := RECORD
STRING10 forename;
STRING10 surname;
STRING2 nl := '\r\n';
END;
d := PIPE('pipeRead 200', namesRecord); //form 1 - input pipe
t := PIPE(d, 'pipeThrough'); //form 2 - through pipe
OUTPUT(t,,PIPE('pipeWrite \\thordata\\names.all')); //output pipe
//Form 2 with XML input:
namesRecord := RECORD
STRING10 Firstname{xpath('/Name/FName')};
STRING10 Lastname{xpath('/Name/LName')};
END;
p := PIPE('echo <Name><FName>George</FName><LName>Jetson</LName></Name>', namesRecord, XML);
OUTPUT(p);