What should I know about Pipes?
Pipes is unlike the extractor and crawler. Pipes doesn’t extract data from Web sites. Rather, Pipes ties a number of resources together into a single data extraction and processing workflow.
What can you do with Pipes? Well, a lot. For example, Pipes can:
- Execute multiple extractors or crawlers and manage the results
- Transform text and images
- Perform calculations
- Join separate data sets
- Parse JSON and XML
- Send HTTP requests
- Send data to an SQL database
In real world use, this means easier price comparisons, simple image resizing, monolithic data sets from multiple Web sites, simple export of data to a relational database, and more.
A Pipes project is composed of nodes, each of which performs its own action. The list of available node types is a long one, so you’re sure to find a solution to the data problems you have been wrestling with thus far.