R’s data.table package provides fast methods for handling large tables of data with simplistic syntax. The following is an introduction to basic join operations using data.table.
A semijoin is a nestjoin plus a filter where you check that every element of data has at.
Suppose you have two data.tables – a table of insurance policies
and a table of insurance claims.
If you want to see the policy data for each claim, you need to do a join on the policy number. In SQL terms, this is a right/left outer join. That is, you want the result to include every row from the claims table, and only rows from the policy table that are associated with a claim in the claims table. Right outer joins are the default behavior of data.table’s join method.
First we need to set the key of each table based on the column we want to use to match the rows of the tables.
Note: Technically we only need to specify the key of the policies table for this join to work, but the join runs quicker when you key both tables.
Now do the join.
Since claim 126’s policy number, 4, was not in the policies table its effective and expiration dates are set as NA.
The important thing to remember when doing a basic X[Y] join using data.table is that the table inside of the brackets will have all of its rows in the resultant table. So, doing
claims[policies] will return all policies and any matching claims.
If you want to return only claims that have a matching policy (i.e. rows where the key is in both tables), set the nomatch argument of data.table to 0.
(This is equivalent to
claims[policies, nomatch = 0] and is referred to as an inner join.)
If you want to return rows in the claims table which are not in the policies table, you can do
Or, for policies with no claims…
Now suppose we add a field, Company, to each table and set all the values to “ABC”.
What would the result be if we try to join policies and claims based on the new Company field?
data.table throws an error in this situation because our resultant table has more rows than the combined number of rows in each of the tables being joined. This is a common sign of a mistake, but in our case it’s desired. In this situation we need to tell data.table that this isn’t a mistake by specifying
allow.cartesian = TRUE.
Next to come – rolling joins.