Back in December 2014, we had a long-standing issue at work. It was about our sampling problem. We had a GitHub issue open, and people were discussing various strategies for weeks. Finally someone mentioned the count-min sketch, but after that there was silence. After about two weeks, I finally took a look. After a few hours, I had an approach that was very similar to the count-min sketch. That became the last-seen sketch.

It seems like such an obvious solution. Surely someone else could’ve come up with the same thing, I thought. I just took one logical step. Maybe I was just thinking the right things at the right time. It ended up being a great idea, but I didn’t expect myself to come up with an idea that good again.

A little over a year ago, I got so annoyed with how I was storing time series with Cistern that I decided to write my own storage engine. It ended up being a really neat idea, and after blogging about it, I found that lots of other people thought it was a neat idea as well.

Wow… I guess I got lucky again, I thought. This idea didn’t seem novel either. It was pretty straightforward considering the steps I took to get there. But no one else, as far as I knew, did the same thing. That made a big difference.

A few weeks ago, I was getting a little bored at work so I decided to spend some time reading papers about ranking. Somehow I ended up coming up with a technique to improve the way we did our ranking. It wasn’t really close to anything I read. Even if it is, I’m not smart enough to see it :). In other words, no one directly told me about this idea. It came up organically. It’s the SketchRank system that I wrote about recently.

This time, I definitely don’t think I just got lucky. There wasn’t a single step to take, nor was there a clear, logical path to the solution. It just came out of nowhere. I think one word describes this perfectly: serendipity. I recommend reading Mastery by Robert Greene, which describes serendipity and its importance in more detail.

One of the reasons why I’m writing this is to understand where good ideas come from. Obviously there is no simple trick. I wish there was! I think there are, however, certain behaviors, activities, and perhaps a mindset that helps me generate good ideas.

Here are some “pro tips” I’m giving myself to keep this idea generation pattern going:

1. Be confident in your ability to generate good ideas.
2. Constantly give your mind new information to crunch on.
3. Keep tackling the hard problems.
4. Remember: there’s always room for something new.

Finally, I’d like to share something I definitely don’t need to tell myself: for every good idea there are tons of bad ideas that never make it anywhere.