JuliaDB is a package for working with persistent data sets.
We recognized the need for an all-Julia, end-to-end tool that can
Load multi-dimensional datasets quickly and incrementally.
Index the data and perform filter, aggregate, sort and join operations.
Save results and load them efficiently later.
Readily use Julia's built-in parallelism to fully utilize any machine or cluster.
We built JuliaDB to fill this void.
JuliaDB provides distributed table and array datastructures with convenient functions to load data from CSV. JuliaDB is Julia all the way down. This means queries can be composed with Julia code that may use a vast ecosystem of packages.
JuliaDB works on Julia 0.6. To install it, run:
To use JuliaDB, you may start Julia with a few worker processes (
julia -p N) or, alternatively, run
addprocs(N) before running
julia> using JuliaDB
Multiple processes may not be benificial for datasets with less than a few million rows. Communication costs are eliminated on a single process, but of course you will be using a single CPU.