What are AutoBots?

Introduction and guide to using AutoBots

Henrik Hofmeister avatar
Written by Henrik Hofmeister
Updated over a week ago

AutoBots are - in addition to being our allies in the struggle against the Decepticons,  a way of organising many extractor robots that should extract the same type of data - given the same type of input. An example would be that you'd want to extract product details from a wide range of sites using the product urls on the individual sites - AutoBots will map these urls to the proper Extract robots and provide a single endpoint for your integrations to get any product extracted.
Additionally AutoBots allow you to query for data from urls that are not yet created for the AutoBot - which will automatically create a new entry - allowing you to create the proper Extractor at a later point. 

Input

The first thing you'll need to determine is what input your AutoBot - and thus all Extractor robots within - will require. The url field is required as that is what the AutoBot uses for mapping urls to robots but you can add additional fields like query if your AutoBot should search on various pages for a search string.
You can also use a data type for input - as long as it contains a url field as well.
All extractor robots within the AutoBot will have the exact same input fields as the AutoBot itself.

Output

Next you'll determine what output fields the robot should have - the same applies here as with input fields - all robots within this AutoBot will have the exact same output fields as the AutoBot itself. Unlike the input fields - output fields does not require any specific field to be available.

Sites

The sites section shows you the mapping between urls (domains) and robots. Any request to the AutoBot for a specific domain will go to the same Extractor robot. If an entry in the mapping is not marked as Ready the execution will fail.
As you execute the AutoBot with various urls it will automatically populate the Sites mapping with new empty entries for unknown entries - and it will also automatically add more examples to the existing domains - allowing you to test your robot with more urls - also when a site changes.

You can also add urls manually to the AutoBot through the Add urls button which opens a dialog that allows you to paste in several urls - one per line.

Did this answer your question?