Many months ago, I first read “Searching 20 GB/sec: Systems Engineering Before Algorithms” , an excellent post by Steve at Scalyr. I re-read it six months ago when I was on winter break between semesters of college. I was traveling, working on Cistern , and thinking about time series storage.
By “thinking,” I mean “annoyed.” I was using Bolt  (the B+tree-based, transactional storage engine implemented in Go) to store Cistern’s time series. The implementation I had was pretty inefficient. There was one key-value pair per time series point without any compression. There was a significant amount of overhead with this approach, especially considering my time series points were 12-bytes at most (8-byte timestamp plus 4-byte float value). I could have implemented compression by packing multiple points into a single value, and then compressing afterwards, but then I’d have to implement batching and all of the compression logic myself.
At some point I just went, “forget B-trees, forget log-structured merge, forget everything!” and decided to try to brute force it. I just need to store a bunch of arrays. How hard could that be, right?
Building a ridiculously simple, "brute force" time series engine pic.twitter.com/g2cPnvVxwp— Preetam Jinka (@PreetamJinka) January 3, 2015
Fast forwarding to the present, you may have seen my blog post introducing Catena , heard me talk about it at a meetup or watched a webinar , or heard about it from some other source. There are so many topics that get covered when I talk about Catena, and you may think that it took a lot of researching time to come up with its design. That’s isn’t the case. The truth is that it has mainly been brute forced! Problems and challenges came up as I was implementing it, and they needed to be solved.
As my work on this storage engine progresses… pic.twitter.com/k0MWIDwJte— Preetam Jinka (@PreetamJinka) January 9, 2015
The really interesting thing now is that the current design is not unique at all. In fact, it could probably be replicated with RocksDB. How about that? I started from scratch and now it seems like the path leads to RocksDB.
I’d like to remind you that I didn’t want to write a time series storage engine. I guess it’s what I’m mostly known for now, but that was a complete accident. I never expected this to actually work.
HOLY **** IT WORKS A WAL is getting checked twice for some reason though xD pic.twitter.com/Dk3bYhadcD— Preetam Jinka (@PreetamJinka) January 10, 2015
But it does. Cistern no longer uses Bolt and has been using Catena for a while. I now have time series storage with compression, and it works well enough for me to build a dashboard that I can use to monitor my systems.
Yay, I can make a dashboard with a custom storage engine! pic.twitter.com/53D1tSHvHJ— Preetam Jinka (@PreetamJinka) January 11, 2015
I guess the moral of the story is that brute force can yield some interesting results. You may end up finding a solution to a problem, like I did when I finally got a better time series storage engine for Cistern. Or, more importantly, you may find that it leads you in a direction that may not have been so obvious when you started off. So, what are the next steps? I’ve already solved my own time series storage problem, but if I had something that needed to be “production ready,” I’d start looking at RocksDB.