What should I know about Pipes?

Pipes is unlike the extractor and crawler. Pipes doesn’t extract data from Web sites. Rather, Pipes ties a number of resources together into a single data extraction and processing workflow.

What can you do with Pipes? Well, a lot. For example, Pipes can:

- Execute multiple extractors or crawlers and manage the results

- Transform text and images

- Perform calculations

- Join separate data sets

- Parse JSON and XML

- Send HTTP requests

- Send data to an SQL database

In real world use, this means easier price comparisons, simple image resizing, monolithic data sets from multiple Web sites, simple export of data to a relational database, and more.

A Pipes project is composed of nodes, each of which performs its own action. The list of available node types is a long one, so you’re sure to find a solution to the data problems you have been wrestling with thus far.

See How can I create a Pipes project?

Did this answer your question?