WINGS' ability to execute workflows under a variety of modes — either in a local host, across a network of local machines, or across large scale distributed data processing environments, such as clusters or cloud services — is an invaluable tool in processing large omics data sets.
A new distributed data processing and analysis environment has been developed, which has generic functionalities for neutron scattering experiments.
Alternatively, WINGS can generate execution-ready workflows that can be submitted to either Apache OODT [ 39] or the Pegasus/Condor execution engine [ 40], which are designed for large-scale distributed data processing in a variety of environments, such as local clusters, shared infrastructure, or cloud resources.
However, current MapReduce implementations are developed to operate on single cluster environments and cannot be leveraged for large-scale distributed data processing across multiple clusters.
● Embedded and distributed: data processing and reasoning are tasks which do not necessarily have to be centralized and given an intelligent environment is supported by a number of interconnected devices part of this responsibility can be passed to the increasingly computationally capable devices physically distributed in the environment.
Convergent approach to distributed computing is the convergence of distributed data processing technologies (GRID, cloudy, foggy, and mobile computing).
Distributed estimation is commonly utilized for distributed data processing over sensor networks, which exhibits increased robustness, flexibility, and system efficiency compared to centralized processing.
For example, the MapReduce programming model along with Hadoop, introduced initially for massive distributed data processing, was explored [ 21– 23].
