uu.seUppsala University Publications
Change search
ReferencesLink to record
Permanent link

Direct link
Scalable Scientific Stream Query Processing
Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Division of Computing Science. Uppsala University, Disciplinary Domain of Science and Technology, Mathematics and Computer Science, Department of Information Technology, Computing Science.
2005 (English)Doctoral thesis, monograph (Other academic)
Abstract [en]

Scientific applications require processing of high-volume on-line streams of numerical data from instruments and simulations. In order to extract information and detect interesting patterns in these streams scientists need to perform on-line analyses including advanced and often expensive numerical computations. We present an extensible data stream management system, GSDM (Grid Stream Data Manager) that supports scalable and flexible continuous queries (CQs) on such streams. Application dependent streams and query functions are defined through an object-relational model.

Distributed execution plans for continuous queries are specified as high-level data flow distribution templates. A built-in template library provides several common distribution patterns from which complex distribution patterns are constructed. Using a generic template we define two customizable partitioning strategies for scalable parallel execution of expensive stream queries: window split and window distribute. Window split provides parallel execution of expensive query functions by reducing the size of stream data units using application dependent functions as parameters. By contrast, window distribute provides customized distribution of entire data units without reducing their size. We evaluate these strategies for a typical high volume scientific stream application and show that window split is favorable when expensive queries are executed on limited resources, while window distribution is better otherwise. Profile-based optimization automatically generates optimized plans for a class of expensive query functions. We further investigate requirements for GSDM in Grid environments.

GSDM is a fully functional system for parallel processing of continuous stream queries. GSDM includes components such as a continuous query engine based on a data-driven data flow paradigm, a compiler of CQ specifications into distributed execution plans, stream interfaces and communication primitives. Our experiments with real scientific streams on a shared-nothing architecture show the importance of both efficient processing and communication for efficient and scalable distributed stream processing.

Place, publisher, year, edition, pages
Uppsala: Acta Universitatis Upsaliensis , 2005. , 137 p.
Uppsala Dissertations from the Faculty of Science and Technology, ISSN 1104-2516 ; 66
Keyword [en]
data stream management systems, parallel stream processing, scientific stream query processing, user-defined stream partitioning
National Category
Computer Science
Research subject
Computer Systems Sciences
URN: urn:nbn:se:uu:diva-6024ISBN: 91-554-6351-7OAI: oai:DiVA.org:uu-6024DiVA: diva2:167178
Public defence
2005-11-07, 1211, Hus 1, MIC, Polacksbacken, Lägerhyddsvägen 2, Uppsala, 13:15
Available from: 2005-10-14 Created: 2005-10-14 Last updated: 2011-02-17Bibliographically approved

Open Access in DiVA

No full text
Buy this publication >>

By organisation
Division of Computing ScienceComputing Science
Computer Science

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 3651 hits
ReferencesLink to record
Permanent link

Direct link