The new year is here and so is the next release of EventQL! New features and fixes include:

• Added a native transport layer (a framed TCP-based protocol). See doc/internals/protocol.txt for the specification

• Added a c client driver library (libevqlclient)

• Added a new table config option that allows to set a finite partition size (called a "partition size hint" in the user facing documentation). A table with a finite partition size will initially have zero partitions but instead add (finite) partitions as it receives inserts for the respective chunks of the keyspace (without splitting the new partition from a larger partition)

• Added the DROP TABLE statement

• Added a thread-local storage and retrieval mechanism for the current session. This also adds a new requirement that all operations must be executed from within a valid database thread.

• Added support for loading configuration files using the '-c' flag and for setting/overring individual configuration parameters using the '-C' flags in the EventQL client (evql) binary

• Added the following new configuration options: server.c2siotimeout, server.c2sidletimeout, server.s2siotimeout, server.s2sidletimeout, server.heartbeatinterval, cluster.allowedhosts, server.httpiotimeout, server.queryprogressratelimit, cluster.allowanonymous, cluster.allowdroptable, server.querymaxconcurrentshards, server.querymaxconcurrentshardsperhost, server.queryfailedshardpolicy, client.timeout, server.diskcapacity, server.loadlimit{soft,hard}, server.partitionsloadinglimit{soft,hard}, server.loadinfopublishinterval, server.noalloc, server.s2spoolmaxconnections, server.s2spoolmaxconnectionsperhost, server.s2spoollingertimeout, cluster.allowcreatedatabase, server.replicationthreadsmax

• Added the evqlslap benchmark and load testing tool

• Added caching for metadata lookups

• Added internal (server-to-server) connection pooling for native TCP connections

• Added a new "user defined partitioning" mode that can be enabled by setting userdefinedpartitioning=true on a table and allows to explicitly specify the partition key for each row

• Added a new monitor thread that publishes disk usage and other load information to the coordination service

• Added new frames to native protocol: ACK, INSERT, REPLINSERT, METAGETFILE{RESULT}, METACREATEFILE, METAPERFORMOP{RESULT}, METADISCOVER{RESULT}, METALISTPARTITIONS{RESULT}, METAFINDPARTITION{RESULT}

• Added the DESCRIBE PARTITIONS table statement to obtain information about the partitions of a table

• Added the USE statement to switch databases

• Added new sql functions: usleep, fnv32

• Added the CREATE DATABASE statement

• Added the CLUSTER SHOW SERVERS statement and the cluster-list command to obtain information about the servers of the current cluster.

• Added the table-import command to the evqlctl util

• Added a basic ruby driver (drivers/ruby)

• Changed the parallel GROUP BY operation to use the new native protocol and to schedule all remote partial group bys from a a single thread using asynchronous I/O (currently via poll()).

• Changed the evql client to use the new native protocol

• Changed the connection acceptor code to read the first byte of an incoming connection and switch protocols (currently HTTP and native) based on this first byte

• Changed the http server code to execute incoming requests from within a database thread (previously this was a generic thread pool)

• Changed the internal cluster auth to require all internal hosts to be whitelisted using the cluster.allowed_hosts option. In standalone mode, the whitelist is defaulted to 0.0.0.0/0

• Changed the HTTP transport to run fully within the database thread (i.e. removed the dependency on an external event loop) and added http timeouts

• Changed the SQL engine to use the new native transport

• Changed the zookeeper backend to explicitly reconnect on session timeouts and not rely on the built-in reconnect mechanism

• Changed the DESCRIBE TABLE statement to return a new "encoding" column that contains information about the column's encoding settings

• Changed the server allocator to assign new partitions to servers using a weighted random scheme, taking into a account the load facotr and the number of available bytes on disk for each server

• Changed all metadata operations to use the native protocol

• Removed the obsolete "sha1_tokens" field from the cluster/server config

• Removed the PCRE dependency

We hope you enjoy working with the new release and please let us know of any issues.

-- The EventQL Team

In the last post we saw that in order to execute interactive queries on a large data set we have to split the data up into smaller partitions and put each partition on it's own server. This way we can utilize the combined processing power of all servers to answer the query rapidly.

The problem we'll discuss today is how exactly we're going to split up a given dataset into partitions and distribute them among servers.

Say we have a table containing a large number of rows, a couple billion or so. The total size of the table is around 100TB. Our task is to distribute the rows uniformly among 20 servers, i.e. put roughly 5TB of the table on each server.

Of course, solving that task is trivial: We read in our 100TB source table, write the first 5TB of rows to the first server, the next 5TB to the second server and so on.

While this simplistic scheme works well for a static dataset, we'll have to be more clever if we are to implement an entire database that supports adding and modifying rows.

Why? Consider this: If we want to modify a row in our naively partitioned table, we first have to figure out on which server we have put the row when splitting the table into pieces. Now, to find the row, we have to search through all rows until we hit the correct one. In the worst case we would have to examine all rows on all servers to find any single row - the whole 100TB of data.

In more technical terms: Locating a row has linear complexity [3]: Finding the row in a table containing one thousand rows would take one thousand times longer than finding the row in a table containing just one row. It gets slower and slower as we add more rows.

Clearly, our simplistic solution will not scale: We need a more efficient way to figure out on which server a given row is stored.

To quickly tell the location of a specific row, we could store an index file somewhere that records the location of each row. We could then do a quick lookup into our index to find the correct server instead of searching through all the data.

Sadly, this doesn't solve the problem. The index file would still have linear complexity. Allthough this time it wouldn't be linear in time, but in space: If we were storing a bazillion rows in our table, our index file would also have a bazillion rows.

Essentially, we would just be rephrasing the problem statement from "How do we partition a large table?" to "How do we partition a large index file?".

We'll have to come up with a partitioning scheme that allows us to find rows with less than linear complexity. I.e. we have to find an algorithm that can correctly compute the location of any single row, but doesn't get slower and slower as we add more rows to the table.

Modulo Hashing

One such algorithm is called modulo hashing. The good thing about modulo hashing is that it's not only very efficient but also extremely simple to implement.

If we want to partition our input table using modulo hashing, we first have to assign an identifier ID to every row in the table. This ID is usually dervied from the row itself, for example by designating one of the table's columns as the primary key. For illustration purposes, we will use numeric identifiers, but the same works with strings. [4]

The only piece of informationthat modulo hashing keeps is a single variable N. This variable N contains the number of servers among which the table should be partitioned.

Now, to figure out on which server a given row belongs, we simply compute the remainder of the division of the row's ID over N. This use of the modulo operation is also where the algorithm derives its name from.

find_row(row_id) {
  server_id := row_id % N;
  return server_id;
}

If, for example, we wanted to locate the row with ID=123 in a table partitioned among 8 servers (N=8), the row would be stored on server number 3 (123 % 8 = 3).

Of course, this is assuming we have also used the algorithm to decide on which server to put each row while loading the input table in the first place.

Modulo hashing works out so that every possible ID is consistently mapped to a single server. The distribution of the rows among servers will be approximately uniform, i.e. every server will get roughly the same number of rows. [5]

The modulo hashing algorithm is a huge improvement over our naive approach as it is constant in space and time: Locating a row takes the same amount of time regardless of the total number of rows in the table. And the only pieces of information we need to tell where a row goes are the row's ID and the total number of servers N, which will only take a few bytes to store. Not bad.

So are we done? I'm afraid we're not.

One thing modulo hashing can't handle well is a growing dataset. If we're continually adding more rows to the table, we will eventually have to add more capacity and increase the number of servers N. However, once we do that the locations of almost all of the rows would change, since changing N also changes the result of all modulo operations.

This means that in order to increase the number of servers N and still keep the rows where they belong, we would have to copy every single row in the table to it's new location every time we add or remove a server.

That's not exactly ideal: Even if we did not care about the massive overhead of copying every single row, we would eventually reach a point where our table grows faster than we can rebalance it.

Consistent Hashing

Consistent hashing is a more involved version of modulo hashing. The main improvement of consistent hashing is that it allows to add and remove servers without affecting the locations of all rows.

At the heart of the consistent hashing algorithm is a so called circular keyspace. Before we discuss what exactly that means let's define the term keyspace:

As with modular hashing we need to assign an identifier ID to each row. Now, the keyspace is the range of all valid ID values. If the ID is numeric, the keyspace is the range from negative infinity to positive infinity.

Within the keyspace, identifiers are well-ordered [6]. That means each ID has a successor and a predecessor. The successor of an ID is the ID that goes immediately after it and the predecessor is the one that goes immediately before it.

In the illustration above, the successor of green (ID=123) is red (ID=856), the successor of red (ID=856) is yellow (ID=923) and so on.

But what is the successor of yellow (ID=923)? Does it have one? The answer is not entirely clear. However, we will later have to come up with a successor for each possible position in the keyspace, so we will have to define what the successor of the last position in our keyspace is.

Imagine, we glued one end of our keyspace to the other end:

Now we can clearly say that, in clockwise order, yellow's (ID=923) successor is green (ID+123). Also, it finally looks like a circular keyspace!

Back to consistent hashing. Like we did with modulo hashing, we will choose an initial number of servers N. For each of the N servers, we will put a marker at a random position in the circular keyspace.

To decide on which server a given row ID belongs, we first locate the position of the ID in the circular keyspace and then search clockwise for the next server marker. In other words: each row goes onto the server whose marker immediately succeeds the row-id's position in the keyspace.

The illustration below shows a circular keyspace with eight server markers. In the illustration, the succeeding server marker for row 123 is server 1, while the suceeding server marker for rows 856 and 923 is server 3.

So row 123 gets stored on server 1 while rows 856 and 923 get stored on server 3.

Alike the modulo hashing scheme, we still end up with a uniform distribution of rows among servers [7] and can quickly locate each row. The only information we have to store are the positions of each server's marker in the keyspace.

Additionally, any server marker that we add or remove will only affect the rows immediately between it and the previous marker: The location of all other rows remain consistent, hence the name consistent hashing.

This means we can add or remove servers and only have to move a small subset of the rows into their new locations. The ratio of rows that needs to be moved is 1/N where N is the number of servers. So as we add more servers, the percentage of rows that need to be moved actually gets smaller.

Can we still do better? It depends on your usecase. Consistent hashing is successfully employed in a number of popular key/value databases [8] such as DynamoDB [9], Cassandra [10] or memcache [11]. Nevertheless, here are two things that we could improve about consistent hashing:

Firstly, consistent hashing only supports an exact lookup operation. That is, we can only find a row quickly if we already know it's ID. If we want to find the locations of all rows in a range of IDs, for example all rows with an ID between 100 and 200, we're back to scanning the full table.

Because of this, consistent hashing is particularly well suited for key/value databases where range scans are not required but less so for OLAP [12] systems like EventQL.

The other possible improvement is that we still have to copy roughly 1/N of the table's rows after changing the number of servers. If we had 100TB on 20 servers, that would mean we're - realistically - still copying at least 15TB for every server addition [13]. Not too bad, but still a lot of overhead network traffic.

The BigTable Algorithm

The last algorithm we will look at today is best known for it's publication in Google's BigTable paper [14].

The bigtable algorithm takes a completely different approach to the problem, but it also starts by defining a keyspace. Except this time it's not a circular keyspace, but a linear one.

The illustration below shows a bigtable keyspace and the position of three rows with the identifiers 123, 856 and 923 within the keyspace.

The next thing bigtable does is to split up the keyspace into a number of partitions that are defined by their start and end positions, i.e. by the lowest and highest row identifiers that will still be contained in the partition.

The illustration below shows a keyspace that is split into five parititions A-E. In the illustration, the row with ID=123 goes into partition B and the rows with ID=856 and ID=923 both go into partition D.

Now, the clever bit about the bigtable algorithm is how it comes up with the partition boundaries. To see why it's clever we have to understand why we can't simply divide the keyspace into equal parts without knowing the exact distribution of the input data:

One reason for that is that if you split up the range from negative to positive infinity into a list of discrete partitions, you end up with an infinite number of partitions.

The other reason is that it's highly likely that the row identifiers will all be piled up in a small area of the keyspace. After all, the identifiers might be user-supplied so we can't nessecarily guarantee anything about their distribution.

So it could be, that even though we have split up the keyspace into a large number of partitions, all rows actually end up in the same partition. And we can't solve the problem by making the partitions infinitesimally small either - that would be like going to back to keeping track of every row's location individually, just a bit worse.

Here's how bigtable solves the problem: Initially the table starts out with a single partition that covers the whole keyspace - from negative infinity to positive infiity.

As soon as this first partition has become too large, it will be split in two. The split point will be chosen so that it roughly halves the data in the partition into equal parts. This continues recursively as partitions become too large. At a basic level, it's an application of the classic divide and conquer principle [15].

This way, you always end up with a number of partitions that are roughly equal in size. Even though the distribution of row identifiers in the keyspace is initially unknown.

And since each partition is defined in terms of it's lowest and highest contained row identifier, we can easily implement efficient range scans: To find all rows in a given range of identifiers, we only have to scan the partitions with overlapping ranges.

Lastly, the bigtable scheme does require a second allocation layer to assign partitions to servers that we didn't discuss here. Suffice to say that this second allocation layer allows us to add new servers to a cluster without physically moving a single row. Of course, we still need to move around some rows every time we split a partition.

Conclusion

So is the bigtable algorithm really "better" than consistent hashing? Again, it depends on the usecase.

The upsides of the bigtable scheme are that it supports range scans and that we can add capacity to a cluster without copying rows. The major downside is that implementing the algorithm in a masterless system requires a fair amount of code and synchronization.

For EventQL, we still chose the bigtable algorithm as the clear winner. After reading this post, go have a look at the debug interface of EventQL where you can see the partition map for a given table. Hopefully it will make a lot more sense now:

That's all for today. In the next post we will discuss how to handle streaming updates on columnar files. You can subscribe to email updates for upcoming posts or the RSS feed in the sidebar.

Next up in the series:

• Part 1: Parallel I/O and Columnar Storage
• Part 2: Dividing Infinity - Distributed Partitioning Schemes

We begin with a high level overview of the system while follow up posts will discuss specific components in more detail. The target audience are software and systems engineers with an interest in databases and distributed systems.

The Challenge

Let's start off with a challenge: We're given a table with 100TB of web tracking data collected over a period of a few days. Here is a small sample of rows from the table:

Our goal is to answer queries like "How many people visited the page '/account/signup' in the last 10 days?". The are only two rules: The query is not known beforehand and we must answer it in less than a second. Here is the same example query in SQL:

SELECT count(1)
FROM tracking_data
WHERE url = '/account/signup' AND time > time_at("-10d");

If that doesn't sound like an interesting challenge yet, consider this quick back-of-the-envelope calculation: Assuming the average hard disk can read roughly 200MB per second (sequentially), loading 100TB from disk will take 500k disk-seconds or about 138 disk hours.

Oops. 138 hours is almost 6 days. A long way from "less than a second". We're seven orders of magnitude off and that's before we even started to process any of the data — just to read it from disk.

There is only one small problem with that scheme — half a million disks cost a lot of money.

So even if we use a lot of machines, reading the full data set from disk in less than a second is utterly out of reach.

Can we still solve the challenge? Yes, but I'm afraid we'll have to cheat — maybe we can answer the query without actually reading all the data from disk.

If we could come up with an algorithm which computes the query result after reading only .01% of the data (or 10GB) from disk, then we could return an answer in one second using just fifty disks. Fifty disks could be hosted in a dozen servers. Finally, that sounds reasonable!

But how do we compute the answer for the full dataset while reading only .01% of the data from disk? One approach would be to use sampling and probabalistic algorithms. However, sampling would give us approximately correct results. Approximate results are great for some use cases and not so great to unworkable for others.

There is another trick we can use to minimze the amount of data to be read from disk while always giving correct results. The technique is called "column-oriented" or "columnar" storage.

Columnar Storage

To really understand the benefit of columnar storage for data anlytics we first have to look at how regular row-oriented databases store tables on disk:

It turns out they do it pretty much like you'd expect them to. They basically keep a file somwhere which contains all the rows in the table. One row after another. Hence the name "row-oriented".

What's problematic about row-oriented databases is that to execute a query, they always have to read the full rows from disk. Even if just a small part of each row is required to answer the query, the database still has to read every row in full. [3]

The not completely intuitive reason for this is that hard disks are only fast if you read a file sequentially, i.e. only if you read one byte after another. Jumping around within a file performs very poorly. So poorly in fact, that reading whole rows is practically always faster than reading partial rows. [4]

Consider the following example query which calculates the number of page views per minute:

SELECT time, count(1)
FROM tracking_data
GROUP BY date_trunc("1min", time);

To compute the answer, we only need to know the value of the time column of each row. We're not interested in the session_id, url or any of the potentially hundreds of other columns of the table.

Ideally, we would only load the time column of each row from disk when executing the query. But we just saw that row-oriented storage can't do that efficiently. We always have to read the full rows no matter what.

Depending on the table and query, we could be spending 99% of the time reading data from disk which we're not going to need to answer the query.

This is the problem column-oriented storage tries to solve. The basic idea is that instead of storing one row after another, we can break up the row into the individual columns and then store one column after another.

If, for example, our table contained one thousand rows with each three columns time, session_id and url, we would first store an array of a thousand time values, then another array of a thousand session_id values and finally an array of a thousand url values.

You might have come across the concept before under a different name: What we're doing is basically vectorization [5].

Storing each column seperately has two desirable properties. The first and most obvious one is that it allows us to also read each column separately — we don't have to load all the extraneous columns from disk anymore.

The second and less obvious upside of columnar storage is that we can compress the data very efficiently. The compression further reduces the number of bytes we actually have to fetch from disk to read a row.

To see why compression in columnar files can be very significant, imagine our table had a fourth is_customer column. Sadly, we don't have any customers yet so the field is always false.

In a row-oriented database, storing the is_customer field for one million rows would take at least 1MB (one byte per boolean). In a columnar database we can store all one million values in a single byte — a 1000000x improvement. [6]

Lastly, it should be noted that columnar storage also has a big downside: It's less efficient to perform updates on columnar files than on row-oriented files. So you probably won't see the classical OLTP databases like MySQL switching to columnar any time soon. Still, for analytical queries on large data sets columnar is clearly the way to go.

Are we there yet?

Looks like we have finally put together a scheme which will allow us to excute a SQL query on 100TB of data in less than a second, even though just reading the data from disk would have taken many days.

Let's recapitulate our approach: We're going to split up the data into many small pieces, then distribute the pieces among a dozen or so machines. On each machine, we will cheat by storing the rows in columnar format and only actually reading a small subset of the compressed data to answer the query.

Of course, we have only scratched the surface of the problem so far. In the next post we will discuss how exactly we're going to split up the data into pieces.

You subscribe to email updates for upcoming posts or the rss feed in the sidebar.

Next up in the series:

• Part 1: Parallel I/O and Columnar Storage
• Part 2: Dividing Infinity - Distributed Partitioning Schemes