What is Datapool?
Datapool is an open-source web application for efficient automated data processing. Processes are configurated graphically as a data flow throught processing blocks.
Following the principle of Divide-and-Conquer multiple Datapool instances (e.g. hosted in a cloud) can interact with eachother or legacy software to handle complex problems.
This approach keeps complexity under control, responsability can be shared and the overall processing speed can be adjusted.
Data exchange is done in a transparant human readble form through lists, emails, pdf-documents or SMS improving debugging on system level.
Calendar-based trigger can be used for time-based flow control. All calendar entries or properties such as the number of data records and their changes, generate signals. Trigger can be derived from these signals. Trigger can initiate data processing as well as the creation of messages, e.g. e-mail or SMS.
The design goal for a single instance of Datapool is maximum configurability, not processing speed.
For example, a calendar date is saved as an array in all relevant formats from which mapping can choose:
Configuration is done by selection, not by conversion!
Cooperative approach
Datapool is an open source software project managed on Github SourcePot/datapool. Datappol provides interfaces for adding processors, data receiver and transmitter.
Datapool content such as dataflows can be easily be exported and imported, i.e. shared within the organization or with others or stored as backup file. A user role infrastructure provides the different levels of access control, i.e. import and export is restricted to the "Admin" and "Content admin".
Graphical data flow builder (DataExplorer-class)
A dataflow consists of two types of (canvas) elements: "connecting elements" and "processing blocks" The connecting elements have no function other then helping to visualize the data flow.
The processing blocks contain all functionallity, i.e. "providing a database table view", "storing settings" and "linking a processor". The settings define the target or targets canvas elements for the result data. There are basic processor, e.g. for data acquisition, mapping, parsing or data distribution. In addition, user-defined processor can be added.
