Components

      Departition components combine multiple flow partitions of data records into a single flow as follows

Concatenate:

                     Concatenate  appends multiple flow partitions of data records one   after another

1. Reads all the data records from the first flow connected to the in port (counting from top to bottom on the graph) and copies them to the out port.

1. Then reads all the data records from the second flow connected to the in port and appends them to those of the first flow, and so on

GATHER:

 Gather combines data records from multiple flow partitions arbitrarily

  Gather is used to:

• Reduce data parallelism, by connecting a single fan-in flow to the in port
• Reduce component parallelism, by connecting multiple straight flows to the in port

The Gather component:

1. Reads data records from the flows connected to the in port.Combines the records arbitrarily.

1. Writes the combined records to the out port.

INTERLEAVE:                  
Interleave combines blocks of data records from multiple flow partitions in round-robin fashion

Parameter :
blocksize
(integer, required)

Number of data records Interleave reads from each flow before reading the same number of data records from the next flow.

Default is 1.

1. Reads the number of data records specified in the blocksize parameter from the first flow connected to the in port

1. Reads the number of data records specified in the blocksize parameter from the next flow, and so on

1. Writes the records to the out port

MERGE:
Merge combines the data records from multiple flow partitions that have been sorted based on  the same key specifier, and maintains the sort order

Parameter:

Key :Name of the key(primary key or unique key) field for combine the data .the key maybe more than one .

  1 .Read the records from the in port and combine the records based on the sorting order

---

Sort components

Checkpoint sort:

It sorts and merges data records, inserting a checkpoint between the sorting and merging phases

FIND SPLITTERS:

Find splitters  sorts data records according to a key specifier, and then finds the ranges of key values that divide the total number of input data records approximately evenly into a specified number of partitions.
Parameter:
key
(key specifier, required)

 

Name(s) of the key field(s) and the sequence specifier(s) required to  Find Splitters to use when it orders data records and sets splitter points.
Number  of  partitions

(Integer, required)

Number of partitions  into which you want to divide the total number of data records evenly.

  The output from the out port of  Find Splitters to the split port of PARTITION BY RANGE.

Rules to use Find splitter:

• Use the same key specifier for both components.
• Make the number of partitions on the flow connected to the out port of Partition by Range the same as the value in the num_partitions parameter of Find Splitters.

If n represents the value in the num_partitions parameter, Find Splitters generates n-1 splitter points. These points specify the key values that divide the total number of input records approximately evenly into n partitions.

You do not have to provide sorted input data records for Find Splitters. Find Splitters sorts internally.

Find Splitters:

1. Reads records from the in port

1. Sorts the records according to the key specifier in the key parameter

1. Writes a set of splitter points to the out port in a format suitable for the split port of Partition by Range

Partition by KEY and Sort:

A partition by key component is generally followed by a sort component. If the partitioning key and sorting key is the same instead to using those two components partition by key and sort component should be used
In this component also key and max-core value has be mentioned as per same rule of sort component

Sort                
Sort component sort the data in ascending or descending order according to the key specified.

By default sorting is done in ascending order. To make the flow in descending order the descending radio button has to be clicked.
In the parameter max-core value is required to be specified. Though there is a default value, it recommended to use $ variable which is defined in the system [$MAX_CORE, $MAX_CORE_HALF etc].

1. Sort stores temporary files Reads the records from all the flows connected to the in port until it reaches the number of bytes specified in the max-core parameter
2. Sorts the records and writes the results to a temporary file on disk
3. Repeats this procedure until it has read all records
4. Merges all the temporary files, maintaining the sort order
5. Writes the result to the out port .  Sort stores temporary files in the working        directories specified by its layout

Sort within Groups:

Sort within Groups refines the sorting of data records already sorted according to one key specifier: it sorts the records within the groups formed by the first sort according to a second key specifier

In parameter part there are two sort keys
1) major key: it is the main key on which records are already sorted.
2) minor key : If the records are already sorted according to major key, according to minor key records are resorted within major key group.

Sort within Groups assumes input records are sorted according to the major-key parameter.

Sort within Groups reads data records from all the flows connected to the in port until it either reaches the end of a group or reaches the number of bytes specified in the max-core parameter.

When Sort within Groups reaches the end of a group, it does the following:

1. Sorts the records in the group according to the minor-key parameter
2. Writes the results to the out port

Repeats this procedure with the next group

---

Transform  components

Aggregate
Aggregate generates data records that summarize groups of data records ( similar to rollup).             But it has lesser control over data

    Parameter :the following are the parameter for aggregate are sorted-input,key,max-   core,transform,select,reject threshold ,ramp,error log,reject log

              The input must be sorted before using aggregate .we can use max core parameter or sort component for sort the data .

  The Aggregate component:

1.      Reads the records from the in port

2.     Does one of the following:

•         If you do not supply an expression for the select parameter, processes all the records on the in port.

•         If you have defined the select parameter, applies the select expression to the records:

3.     Aggregates the data records in each group, using the transform function as follows:

a.       For the first record of a group, Aggregate calls the transform function with two arguments: NULL and the first record.

Aggregate saves the return value of the transform function in a temporary aggregate record that has the record format of the out port.

b.      For the rest of the data records in the group, Aggregate calls the transform function with the temporary record for that group and the next record in the group as arguments.

Again, Aggregate saves the return value of the transform function in a temporary aggregate record that has the record format of the out port.

4.      If the transform function returns NULL, Aggregate:

a.       Writes the current input record to the reject port.

Aggregate stops execution of the graph when the number of reject events exceeds the result of the following formula:

limit + (ramp * number_of_records_processed_so_far)

For more information, see "Setting limits and ramps for reject events".

b.      Writes a descriptive error message to the error port.

5.      Aggregate writes the temporary aggregate records to the out port in one of two ways, depending on the setting of the sorted-input parameter:

a.       When sorted-input is set to Input must be sorted or grouped, Aggregate writes the temporary aggregate record to the out port after processing the last record of each group, and repeats the preceding process with the next group.

b.      When sorted-input is set to In memory: Input need not be sorted, Aggregate first processes all the records, and then writes all the temporary aggregate records to the out port.

Denormalize sorted:
                             Denormalize Sorted consolidates groups of related data records into a single output record with a vector field for each group, and optionally computes summary fields in the output record for each group. Denormalize Sorted requires grouped input
Filter by Expression:  Filter by Expression filters data records according to a specified DML expression.
Basically it can be compared with the where clause of sql select statement.
Different functions can be used in the select expression of the filter by expression component even look up can also be used. In this filter by expression there is reject-threshold parameter
The value of this parameter specifies the component's tolerance for reject events. Choose one of the following:
       • Abort on first reject — Write Multiple Files stops the execution of the graph at the first reject event it generates.
      • Never abort — the component does not stop the execution of the graph, no matter how many reject events it generates.
      • Use ramp/limit — the component uses the settings in the ramp and limit parameters to determine how many reject events to allow before it stops the execution of the graph.
The default is Abort on first reject.

Filter by Expression:

1.      Reads data records from the in port.
2.      Applies the expression in the select_expr parameter to each record. If the expression returns:
   •         Non-0 value — Filter by Expression writes the record to the out port.
   •         0 — Filter by Expression writes the record to the deselect port. If you do not connect a flow to the deselect port, Filter by Expression discards the records.
   •         NULL — Filter by Expression writes the record to the reject port and a descriptive error message to the error port

     FUSE:        Fuse combines multiple input flows into a single output flow by applying a transform function to corresponding records of each flow

       Parameter :the following are the parameter for fuse are count,transform,select,reject threshold ,ramp,error log,reject log

        Fuse applies a transform function to corresponding records of each input flow. The first time the transform function executes, it uses the first record of each flow. The second time the transform function executes, it uses the second record of each flow, and so on. Fuse sends the result of the transform function to the out port.

fuse works as follows:

1.      Fuse tries to read from each of its input flows): If all of its input flows are finished, fuse exits.
Otherwise, Fuse reads one record from each still-unfinished input port and a NULL from each finished input port.
2.   If Fuse reads a record from at least one flow, Fuse uses the records as arguments to the select function if the select function is present.
   •         If the select function is not present, Fuse uses the records as arguments to the fuse function.
   •         If the select function is present, fuse discards the records if select returns zero and uses the records as arguments to the fuse function if select returns non-zero.
3.      Fuse sends to the out port the record returned by the fuse function

JOIN:

     Join reads the records from multiple ports, operates on the records with matching keys using a multiinput transform function and writes the result into output ports

                 In join the key parameter has to be specified from input flow (either of the flow) ascending or descending order. If all the input flows do not have any common field, override-key must be specified to map the key specified

 It has the following ports

    in0: the input file is connected to this port

  in1:the second input file is connected to this port .this in port will increase based on the no of input file

   out :the output of join component is going to this out port

   Unused0:In this file contains the unused data (unmatched data) from the input file0

   Unused1:In this file contains the unused data (unmatched data) from the input file1

   Reject port :it contains the rejected record due to some error in the data from the file

   Error port : It contains about detail description of rejection of the data .it write what is error in the file

   Log port:It writes the process status based on the time until the process end.     

     It has three types of join as follows

1.                   Inner join (default) — Sets the record-requiredn parameters for all ports to True. The GDE does not display the record-requiredn parameters, because they all have the same value.

2.                  Outer join — Sets the record-requiredn parameters for all ports to False. The GDE does not display the record-requiredn parameters, because they all have the same value.

3.                   Explicit — Allows you to set the record-requiredn parameter for each port individually

REFORMAT:

                    Reformat changes the record format of data records by dropping fields, or by using DML expressions to add fields, combine fields, or transform the data in the records
      By default reformat has got one output port but incrementing value of count parameter number. But for that two different transform functions has to be written for each output port.
       If any selection from input ports is required the select parameter can be used instead of using ‘Filter by expression’ component before reformat

The Reformat component:

1.    Reads records from the in port.

2.    If you supply an expression for the select parameter, the expression filters the records on the in port:

a)    If the expression evaluates to 0 for a particular record, Reformat does not process the record, which means that the record does not appear on any output port.

b)  If the expression produces NULL for any record, Reformat writes a descriptive error message and stops execution of the graph.

c)    If the expression evaluates to anything other than 0 or NULL for a particular record, Reformat processes the record.

3.    If you do not supply an expression for the select parameter, Reformat processes all the records on the in port.

4.    Passes the records to the transform functions, calling the transform function on each port, in order, for each record, beginning with out port 0 and progressing through out port count - 1.

5.    Writes the results to the out ports

ROLLUP:
        Rollup generates data records that summarize groups of data records on the basis of key specified.

Parts of Aggregate
• Input select (optional)
• Initialize
• Temporary variable declaration
• Rollup (Computation)
• Finalize
• Output select (optional)


Input_select : If it is defined , it filters the input records.

Initialize: rollup passes the first record in each group to the initialize transform function.

Temporary variable declaration:The initialize transform function creates a temporary record for the group, with record type temporary_type.

Rollup (Computation): Rollup calls the rollup transform function for each record in a group, using that record and the temporary record for the group as arguments. The rollup transform function returns a new temporary record.

Finalize:
If you leave sorted-input set to its default, Input must be sorted or grouped:

• Rollup calls the finalize transform function after it processes all the input records in a group.
• Rollup passes the temporary record for the group and the last input record in the group to the finalize transform function.
• The finalize transform function produces an output record for the group.
• Rollup repeats this procedure with each group.


Output select: If you have defined the output_select transform function, it filters the output records.

SCAN:             

1.    For every input record, Scan generates an output record that includes a running, cumulative summary for the data records group that input record belongs to. For example, the output records might include successive year-to-date totals for groups of data records

2.     The input should be sorted before the scan else it produce error .

3.     The main difference between Scan and Rollup is Scan generates intermediate (cumulative) result and Rollup summarizes

Partition Components
 

Broadcast:

           broadcast arbitrarily combines all the data records it receives into a single flow and writes a copy of that flow to each of its output flow partitions 1.                  Reads records from all flows on the in port

2.                 Combines the records arbitrarily into a single flow

3.                  Copies all the records to all the flow partitions connected to the out port

Partition by Expression:

                     Partition by Expression distributes data records to its output flow partitions according to a specified DML expression. The output port for Partition by Expression is ordered

Partition by Key :

                    Partition by Key distributes data records to its output flow partitions according to key values.            Reads records in arbitrary order from the in port

Distributes them to the flows connected to the out port, according to the key parameter, writing records with the same key value to the same output flow

   Partition by percentage:

                          Partition by Percentage distributes a specified percentage of the total number of input data records to each output flow

            Reads records from the in port

Writes a specified percentage of the input records to each flow on the out port

Partitions by Round robin:

                             1.                   Partition by round-robin distributes blocks of data records evenly to each output flow in round-robin fashion. Partitioning key is not required.The difference between Partition by Key and Partition by Round Robin is the 1st one may not distribute data uniformly across the all partition in a multi file system but the latter does                     

2.                    It first Reads records from the input file.

3.                  Then distributes them in block_size (specified no.of records)chunks to its output flows according to the order in which the flows are connected

Partition by Range:
                                     Partition by range partition or divide the record based on the specified range in the file.for example consider the file contains 50records and it has four output file .i want 20 records in output 1and 5 records in output 2 & 3 remaining records in output4 then we can specify the range in the parameter tab.

              Use the same key specifier for both components.
Make the number of partitions on the flow connected to the out port of Partition by Range the same as the value (n) in the num_partitions parameter of Find Splitters.

This component
Reads splitter records from the split port, and assumes that these records are sorted according to the key parameter.
Determines whether the number of flows connected to the out port is equal to n (where n-1 represents the number of splitter records).If not, Partition by Range writes an error message and stops the execution of the graph.
Reads data records from the flows connected to the in port in arbitrary order.
Distributes the data records to the flows connected to the out port according to the values of the key field(s), as follows:
a) Assigns records with key values less than or equal to the first splitter record to the first output flow.
b) Assigns records with key values greater than the first splitter record, but less than or equal to the second splitter record to the second output flow, and so on.

Partition by load balance:
Partition with Load Balance distributes data records to its output flow partitions by writing more records to the flow partitions that consume records faster.

The output port for Partition with Load Balance is ordered

1. Reads records in arbitrary order from the flows connected to its in port
2. Distributes those records among the flows connected to its out port by sending more records to the flows that consume records faster

3.     Partition with Load Balance writes data records until each flow's output buffer fills up

Miscellaneous Components:

 

 

 

 

 

 

GATHER LOG

Gather log used to collects the output from the log ports of components for analysis of a graph after execution

1. Collects log records generated by components through their log ports.

1. Writes a record containing the text from the StartText parameter to the file specified in the LogFile parameter.

1. Writes any log records from its in port to the file specified in the LogFile parameter.

1. Writes a record containing the text from the End Text parameter to the file specified in the LogFile parameter

LEADING RECORDS:
              Leading record is used to copies the specified no of record from the in port and write into the out port counting from the first record of the file                   

REPLICATE:
Replicate arbitrarily combines all the data records it receives into a single flow and writes a copy of that flow to each of its output flows

1. Arbitrarily combines the data records from all the flows on the in port into a single flow
2. Copies that flow to all the flows connected to the out port

GENERATE RECORDS:                           

     Generate Records generates a specified number of data records with fields of specified lengths and types.

 Generate Records generate random values within the specified length and type for each field, or you can control various aspects of the generated values. Typically, you would use the output of Generate Records to test a graph

1. Generates the number of data records specified in the num_records parameter.

2.      The values of the records depend on the record format of the out port, and the optional command_line parameter.

1. Writes the records to its out port.