(Guest post by Omar Mukhtar)

I presented the paper Efficient Computation of Frequent and Top-k Elements in Data
Streams.

I presented the SpaceSaving data structure and algorithms for finding the Top- and Frequent elements in an insert-only setting. The key element of the paper is a unified approach to solving the very common Top- and Frequent element problem as well as guaranteeing the results. The main hurdle to a unified approach is that a Top- query can’t preprocess a Frequent element query and vice versa, so the authors use a common data structure, SpaceSaving, that is used to maintain stream statistics, and then answer queries. Thus, only one-pass is made over data to answer the Top- and Frequent element query.

The main idea is to have a set of counters which keep an exact frequency of individual elements. Each counter also keeps track of the over-estimation error in the elements frequency. The SpaceSaving data structure has two parts, a counter which has a field representing the element, and the over estimation error, . The second part is a sorted doubly linked of buckets, each element is attached to a bucket which represents that elements frequency. Once the pass over the Data Stream is made and the SpaceSaving data structure constructed, a pass over the counters is made to find the Top- and the Frequent elements. The over estimation field is used to guarentee the accuracy of the Top- and Frequent element query.

The space bounds of the algorithm are lower than other algorithms which solve the Top- and Frequent element query separately. Experimental results with synthetic as well as Zipfian data showed that the algorithm in the paper provides a very efficient way to track frequencies in small space with guaranteed results.

(Guest Post by Karthik Dwarakanath)

I presented the paper Streaming algorithms for Robust, real-time detection of DDoS attacks. I presented two data structures and algorithms for the efficient tracking of potentially malicious connections. The key element of this approach is a novel data-streaming algorithm for efficiently tracking, in guaranteed small space and time, destination IP addresses that are large with respect to the number of distinct source IP addresses that access them.

The main idea in the basic estimator is to employ the distinct-count sketch synopsis to build a (appropriately-sized) distinct sample of the observed active source – destination pairs. Once such a distinct sample is available, the top- estimation process is fairly straightforward: the destinations with the highest occurrence frequencies in the distinct sample are identified, and are returned with their frequencies appropriately scaled by the distinct-sampling rate. This algorithm can incur a high overhead for producing the top- frequency estimates and corresponding destination addresses. The update time for this algorithm is and the query time is , where and are the number of “first level” and “second level” hash tables.

They provide a second algorithm the “Tracking-DCS-based top-k estimation algorithm” which offers guaranteed poly-logarithmic update and query times, while increasing the overall storage space by only a small constant factor over thebaseline distinct-count sketch synopsis. The update time for this algorithm is and the query time is . Experimental results with synthetic data showed that the tracking version of the algorithm in the paper provides a very efficient way to track large distinct frequencies in small space and guaranteed small update/query time.

(Guest post by Denis Mindolin)

In my presentation, I showed the results from the following paper: Stabbing the Sky: Efficient Skyline Computation over Sliding Windows, ICDE ‘05.

The paper deals with the problem of computing skylines over data streams. Given a data set of elements with attributes , a skyline of is a set of all undominated elements of . An element dominates another element if for all , , and for at least one , . This problem is a topic of active research for the past thirty years. However, the paper I presented is one of the first ones considering skylines in the data stream framework.

The stream model considered in the paper is append only. A stream element is a vector having components each of which is a number. It also assumed that there’s a sliding window of the most recent elements, and the queries issued are

1. Compute the skyline of the most recent elements (), and
2. Compute the skyline of the elements between the most recent and ().

To answer queries of the first type (-of-), the authors propose to keep in main memory a set of nonredundant elements . The size of is for some restricted streams. In order to avoid the quadratic size of the dominance graph over , it is proposed to keep at most two dominance relationships per element in which are enough to answer -of- queries. This makes the dominance graph size linear in the size of . To answer skyline queries in time, the authors propose to store the graph as a set of intervals ( space) which reduces the skyline problem to the already solved problem of answering stabbing queries.

The second query model ( -of- ) is handled in a similar way with an exception that here the entire set of the most recent elements should be kept in main memory. As a result, the space requirement here is higher than in the first approach.

• The report is due today by midnight.
• If you haven’t submit the blog entry for your presentation, please do so by tomorrow.

I would like to take you guys out for lunch this Wednesday or Thursday at noon. Use the comments section to let us know if you can make it (and if you prefer one day over the other).

Update (04/29) Based upon the comments Thursday looks like the better option. So let’s meet at noon on Thursday, May 1 in my office (Bell 123).

(Guest post by Yang Wang)

In my presentation, I showed results from the following three papers.

1. On the streaming model augmented with a sorting primitive
2. Trading off space for passes in graph streaming problems,
3. Adapting Parallel Algorithms to the W-Stream Model, with Applications to Graph Problems

The three papers were developed in sequence. In the first paper, the authors proposed using multiple passes with the ability to write onto an intermediate stream and also sort on that stream (as some difficult problems such as graph problems, the data stream model is too weak). Thus this improves the power of the “basic” data stream model. Define a class to be the problems that can be solved in polylog passes and polylog (inner memory) space. Then they showed that graph connectivity, max spanning tree and many other problems are in this class. However there do exists problems that are not in the polylog class. In particular, they showed an example that need at least passes, and space with where is the input length.

In the second paper, the authors consider the case when we can write to intermediate streams but cannot sort them. This new model is called the -stream. They use ideas from parallel algorithms that solve the transitive closure problem to attack the connected component problems and single source shortest paths problem. Later they extended the idea to more general cases in the third paper. They found the following relationship between parallel algorithm and -stream. Every processor memory time PRAM algorithm can be simulated on the -stream with passes with space . In the next step, they claimed that though the -stream model can be used to simulate PRAM, it is still more powerful that PRAM in that in the -stream model every memory cell in the entire stream can be accessed while in PRAM only a constant number of cells can be accessed. Thus the strategy is to change a PRAM algorithm to a RPRAM algorithm and then to the -stream. In this way, the authors solve many problems such as Maximum independent set and biconnected components.

(Guest post by Xi Zhang)

I presented [JKV07] on how to compute aggregates , where , over a probabilistic data stream. Only those five aggregates are considered as they are the major concerns in databases.

First, I introduced the probabilistic data stream model, where in contrast to the classical (deterministic) stream model, we have pdfs instead of elements from some domain. In a probabilistic data stream , each pdf is over the base domain , where the special symbol indicates no element is produced. Any deterministic stream which is a possible outcome of is a possible stream of . And the semantics of over is the expected value of over all the possible streams of .

Then, I went on to talk about how to evaluate those five aggregates over a probabilistic stream one by one.

For and , [BDJ05] shows exact algorithms to compute those two aggregates in one-pass with constant space and update time.

is an interesting case. The paper presents algorithms from the generating function point of view.
It shows a generating function such that . is in the form of a high-degree polynomial. The paper exhibits a data stream algorithm for estimating this definite integral to a relative error in passes over the data with space and update time per pdf. The approximation here focuses on the approximation of a definite integral, where rectangle approximation is used.

The paper proves the lower bound of any one-pass exact algorithm for computing is . However, complementing this result, this paper also presents an exact algorithm for computing , which runs in time and space. It is still better than the previously best-known exact algorithm which uses the same amount of space but runs in time.

For and , the paper presents a one-pass data stream algorithm with relative accuracy , using space and constant update time per pdf (in fact, it is , where is the size of maximum support of all pdfs. Whether it is really a “constant” is arguable). The approximation technique used here is “binning”.

(Guest post by Steve Uurtamo)

I presented a few algorithms for the detection of superspreaders, where superspreaders are defined to be sources in a networking data stream that contact many unique destinations. The paper was New algorithms for fast detection of superspreaders, by S. Venkataraman, D. Song, P. Gibbons, and A. Blum.

The basic algorithm is to store a uniformly chosen fraction of the pairs, using a hash function to make sure that we don’t count any pair more than once. Assuming uniformity of the hash function, if we want to detect a -superspreader, holding onto of the resulting datastream with elements should be sufficient, where, if superspreaders are as rare and as chatty as expected, the storage will be modest.

Modifications of this algorithm were presented for the sliding window model, where we are continuously monitoring a datastream over a fixed length window, and for parallel monitoring, where we need to combine the information in several different datastreams to detect superspreaders.

Experimental results with synthetic data showed that the algorithms in the paper were much more space-efficient than simply combining existing approximate counting and distinct counting stream algorithms together.

A 2-3 page report is due on May 5 (midnight). When you are done, email it to both Hung and me. We basically want you to summarize your thoughts on some problem related to the seminar that you thought about. If you have any questions and/or comments, please use the comments section of this post.

Update (4/14) When you are suggesting open problem, please also summarize what is already known about the problem.

(Guest post by Demian Lessa)

I presented the main results and algorithms of the SODA08 paper “Declaring Independence via the Sketching of Sketches,” by Indyk and McGregor.

The authors consider the problem of computing the correlation- that is, the degree of independence, of data streams. In particular, if are pairs appearing in a data stream, the frequencies of such pairs define a joint distribution , and the goal is to compute the correlation between and the product of the marginals. In the centralized model, the coordinates and appear together in the stream, while in the distributed model each coordinate may appear separately. Three measures of closeness are used to approximate the correlation- the norm, the norm, and the mutual information between and . All positive results in the paper are obtained in the centralized model.

In order to obtain a -space, -approximation for the distance between the joint distribution and the product of the marginals, the authors explore the techniques in [AMS] for the computation of . In particular, they utilize two -wise independent vectors and (of size ), constructed using the parity check matrix of BCH codes, to compute vector (of size ) defined as the outer product of and . They show that, even though is not -wise independent, the variance can still be bounded as necessary by observing the geometric relationship on the indices of . The algorithm is defined naturally by composing sketches and producing an estimate for the square of the distance. A set of weak estimates followed by an application of the “median trick” yields the claimed approximation and space bounds.

In order to obtain a -space, -approximation for the distance between the joint distribution and the product of the marginals, the authors explore the techniques in [I06] for the computation of . In particular, they utilize vectors drawn from the Cauchy and -Truncated Cauchy distributions to produce an estimate for . Differently from the algorithm for , the estimate for is produced by the median of estimators. By repeating the process times, the claimed approximation and space bounds follow.

The remaining results for the centralized model consist of extensions to the algorithm, and an approximation of the distance by the mutual information between and . Finally, negative results were presented for the distributed model.