Lecture 15: Data Flow Systems: Spark, MapReduce shortcomings.
On August 21, 2003, we released Ptolemy II 3.0.2, a block diagram editor and simulator for continuous-time, hybrid and data flow systems.
RDDs are motivated by two types of applications that current data flow systems handle inefficiently: iterative algorithms, which are common in graph applications and machine learning, and interactive data mining tools.
Data flow systems are related to functional systems and do not assign a strict sequence of operations.
Although some of the models may seem to perform a small number of operations, we argue that the main nature of Simulink-like models is the implementation of data flow systems in different levels of abstraction.
The data-flow system is responsible for reading out, formatting and conveying the event data, eventually saving the selected events into the mass storage.
Through this alternative perspective, GraphX integrates graph computation into a general purpose distributed data-flow system, enabling users to view data as tables or graphs without data movement or duplication and efficiently execute graph algorithms at performance parity with specialized graph-processing systems.
Because a combination of factors-cost, reliability, uniformity and convention-made it virtually impossible to obtain historical options data, we developed a data-flow system to capture, process and analyze streaming data over a period of over a year.
In this paper we will report on the operation and the performance of the ATLAS data-flow system during the 2011 physics run of the Large Hadron Collider at 7 TeV.
Test shows that this system is better than the traditional data flow management system in aspects of efficiency, scalability and reliability.
