Favourite Tool/Package of 2019

Introduction

When you are dealing with relatively large datasets on a regular basis, the computation time required to process your data becomes an issue. We always want more speed. If you can cut run time from 30 minutes to 10 minutes, then that is a huge gain.

In python there’s the usual performance gain coming from removing native loops and vectorising with NumPy, using pandas and then there’s even the new kid on the block, Numba (not actually that new now, but still newer!) which requires a bit more effort rewriting some code into functions that can be Just-In-Time (JIT) compiled. However, not everything can be compiled.

But there are also some other very simple performance tweaks that can be made and that can have a significant effect on runtime. That’s where Bottleneck comes in.

Bottleneck

Bottleneck is a:

collection of fast, NaN-aware NumPy array functions written in C.

Bottleneck is basically a drop-in replacement for some popular NumPy functions such as sum, mean, min, max, etc. Please refer to the documentation for the full list.

Here’s a very simple example of bottleneck at work:

import numpy as np
import bottleneck as bn

a = np.array([1, 2, np.nan, 4, 5])

np.nanmean(a)
bn.nanmean(a)

Bottleneck also provides some really useful rolling window functions that work along a single axis. Super easy and useful for calculating moving averages. The output will be the same shape as the input, but with the first few values smaller than the windo size returned as nan.

b = np.random.random(100)

rolling_avg_3 = bn.move_mean(b, window=3)
rolling_avg_10 = bn.move_mean(b, window=10)

Performance

Bottleneck comes with a built in benchmarking suite that you can run on your machine to see what your performance will be like. Simply run bn.bench()

Here’s the results output from my machine:

Bottleneck performance benchmark
    Bottleneck 1.3.2; Numpy 1.20.3
    Speed is NumPy time divided by Bottleneck time
    NaN means approx one-fifth NaNs; float64 used

              no NaN     no NaN      NaN       no NaN      NaN    
               (100,)  (1000,1000)(1000,1000)(1000,1000)(1000,1000)
               axis=0     axis=0     axis=0     axis=1     axis=1  
nansum         37.4        1.8        2.3        3.3        3.4
nanmean        99.5        2.2        3.0        4.3        4.2
nanstd        167.6        2.3        3.4        5.1        3.4
nanvar        165.2        2.2        2.5        4.1        2.9
nanmin         25.5        0.6        1.8        1.0        3.3
nanmax         24.2        0.8        1.9        0.8        3.1
median        113.3        1.3        3.8        1.1        3.8
nanmedian     113.3        6.5        7.1        5.5        5.9
ss             13.0        1.9        2.2        3.6        3.5
nanargmin      58.4        3.4        4.6        2.5        5.6
nanargmax      59.8        3.4        5.3        2.4        5.7
anynan          7.8        0.5       37.5        0.3       26.0
allnan         10.2      158.2      118.7       82.9       89.2
rankdata       23.5        1.3        1.3        2.6        2.6
nanrankdata    23.6        1.5        1.4        2.8        2.8
partition       4.6        1.0        1.3        0.9        1.3
argpartition    9.1        1.2        1.4        1.1        1.5
replace         7.6        0.9        1.0        0.9        0.9
push         1091.2        4.5        5.5        9.4        8.9
move_sum     2973.9       52.0      104.7      201.4      253.1
move_mean    7566.5       60.3      112.3      286.9      286.2
move_std     8564.2       73.8      146.0      190.7      302.7
move_var    11758.3       81.3      172.4      253.6      392.3
move_min     1361.0       15.6       31.8       22.8       53.8
move_max     1463.0       15.7       33.1       26.9       52.0
move_argmin  3319.4       67.4      102.3       77.0      128.4
move_argmax  3476.7       61.9       75.8       71.2      119.1
move_median  1704.8      137.2      160.7      171.2      185.9
move_rank     911.9        1.5        1.9        1.9        2.4

In the vast majority of cases bottleneck provides close to a 2x increase in performance of some of the most frequently used NumPy functions which can really help reduce that runtime and increase productivity. It’s definitely noticeable for me in my work.