Reduce the number of CI test jobs

[jessegrabowski]

Now that we're testing numba and cvm, we probably have too many jobs between 2 versions of python, 2 OS, and 2 backends. We should trim it down.