How can I create a Pipes project?

Our Pipes product is unlike our extractor and crawler. Pipes can control extractors and crawlers, and can retrieve external information from APIs and databases. Pipes do not extract information from Web sites as the crawler and extractor do. Rather, Pipes combine crawlers, extractors, APIs, and data sets to create a single workflow for data extraction and processing.

To create a Pipes project:

- Navigate to the Projects page.

- Click the New Robot button.

- Choose the Pipes option.

- Enter a name for the project.

- Click the Create New Robot button.

The Pipes editor opens to display a grid upon which nodes will be laid out with connectors between. The relationships between nodes represent the Pipes workflow.

Clicking on a node opens the node menu at the top of the page. Multiple nodes may be selected by Shift-clicking or by dragging a box to surround the nodes.

Clicking the Connect button will connect all selected nodes with the same name and type.

To add a node:

- Drag a node option from the Add Node menu on the right side of the screen.

- Enter a name for the node and configure any other options as appropriate.

There are many node types available to provide a comprehensive array of data processing possibilities. Here is a list of nodes under the category, just to give you a taste of the possibilities you’re about to discover with Pipes.

- Execute robot: Starts a crawler or extractor. A worker is required for Pipes, plus one worker for each robot executed from Pipes.

- Data type: A custom data type created by the account.

- For each row in data set: Iterates over each row in an existing data set.

- Save row to data set: Saves input to a data set at the node’s output.

- Dictionary lookup: Uses a key to reference an existing dictionary for a value.

Did this answer your question?