Machine Learning and Big Data with kdb+/q

CHAPTER 9Data Cleaning and Filtering

We shall briefly cover some practices and rules around data filtering and ways to extract the data we want from a table and a historical partitioned database.

9.1 PREDICATE FILTERING

9.1.1 The Where Clause

The where function is the q king of filters; we were first introduced to it in Chapter 1. It acts as a natural filter on a vector:

It is also our constraint filter in a select statement:

We recall from Chapter 2 that we separate multiple constraints using a ',' and these are applied from left to right, an exception to the usual q parsing. Note that for multiple where clauses, the filtering is cascading, i.e. the second filter is applied after the initial filter has already reduced our data set. This is a built-in optimisation in q:

Now in ...