Cistern: The Vision of Reinvented Network Monitoring

Background As a hosting provider, I’ve had my fair share of DDoS attacks. My company doesn’t do any peering with transit providers. We just have a single upstream provider at our Ashburn data center. My provider has an automated DDoS detection system, which is made from scratch, that detects anomalous flows and either automatically blocks traffic or sends email alerts. I sometimes get alerts forwarded to me in case it’s an outbound anomaly originating from one of my clients’ VMs.

Optimizing Concurrent Map Access in Go

One of the more contentious sections of code in Catena, my time series storage engine, is the function that fetches a metricSource given its name. Every insert operation has to call this function at least once, but realistically it will be called potentially hundreds or thousands of times. This also happens across multiple goroutines, so we’ll have to have some sort of synchronization. The purpose of this function is to retrieve a pointer to an object given its name.

State of the State Part III

First, I suggest reading Baron’s “Time-Series Database Requirements” blog post to get some more context for this post. I read that and, as I usually do, had my mind set on low-level thoughts. I wrote the following comment: I took this screenshot a few months ago, so it has actually been almost a year since I wrote that. Time flies! Cistern’s graphs Cistern had graphs back in October 2014. I think I used my metricstore package.

Break-in

Someone broke into my server. I was at beSwarm yesterday with my “social networking” setup. Social networking! pic.twitter.com/fdApIwlKyy — Preetam Jinka (@PreetamJinka) February 7, 2015 I was demoing Cistern in some form. Cistern doesn’t expose much to the user right now since most of my time was spent on very core features. So, what most people usually saw was the terminal log output. It’s still a little interesting because you can see it do some basic host discovery using SNMP, and it prints flow data as it arrives.

Personal projects, knowledge, and intuition

I had a short conversation with someone recently about having personal projects and applying to internships. The short version of what he said is, I do my school work well and good grades. Besides that, I also do well during internships. Why do I need personal projects? (The following represent my own views. This is a personal blog after all, and this is just, like, my opinion, man.) Just for fun, I did a quick search on personal projects to see what kind of links would show up.

Observium Annoys Me

I first started using Observium in 2011 or 2012. I was a senior in high school. I wasn’t that good at programming. I mean, I could write code in a few languages, knew the basic data structures, Big-O, etc. but I was not familiar with many higher level concepts like monitoring. I knew about SNMP, but I didn’t know anything at all about the implementation. As Bitcable’s infrastructure grew to include network switches and more hardware, I needed a monitoring tool.

A New Design

There used to be a period of time when I kept changing Misframe’s design. I was also changing what it was running on quite often. I spent most of my time thinking about how it ran rather than what I was going to write. After hopping around from using WordPress, Node.js, and Tumblr, I’ve landed on something so simple that I can’t get distracted from the writing. With today’s tweak, Misframe is mostly in serif.

SNMP Part II

This post is more about the details of how SNMP is currently implemented in Cistern. The code at the latest commit as I write this is here. SNMP communication itself is pretty simple. There are requests and responses. PDU stands for “protocol data unit.” I think of them as structs. If you look in the RFCs you’ll see them defined like this, in the ASN.1 language: GetResponse-PDU ::= [2] IMPLICIT SEQUENCE { request-id RequestID, error-status ErrorStatus, error-index ErrorIndex, variable-bindings VarBindList } That’s a pretty dense way of describing a GetResponse.

SNMP Part I

SNMP stands for Simple Network Management Protocol. In case you haven’t heard, it’s not simple. SNMP is older than me, and it’s used everywhere in networking. I need SNMP support for Cistern. sFlow is great for statistics but it’s not useful to get general information. You can easily get interface statistics from sFlow datagrams, but you may want to know what the interface description strings are. You may want to know which VLANs those interfaces are assigned to.

Custom Router Part II

Welcome back! In my previous post I described this interesting idea of writing a router. I had no idea whether or not it would work. I knew it was possible, of course. I run two OpenBSD routers in a failover setup with CARP (this blog is routed through them, FYI). Setup Turns out my BeagleBone Black is great for testing this out! When you plug in a factory default BeagleBone Black into a computer, it sets up a network over USB.

Custom Router

So this clever new idea has been floating around in my head for a couple of days now. If you consider a router, it’s basically connecting directing packets to different subnets. The simple case is when you have two subnets and a router that’s in the middle. Let’s call the router R1, with the two subnets being S1 and S2. Let’s say you’re managing a bunch of servers in a data center rack and they’re all on S2.

The Crazy One

We should be celebrating the ones who take time to slow down and better understand themselves, the ones who are shaken by life’s beauty, the ones who aren’t afraid to be alone, the ones who play the baby grand piano when no one’s listening, the ones who don’t follow the money, the ones who practice compassion, the ones who admit the only thing they know is that they know nothing at all.

State of the State Part II

The original post was written last December (almost 10 months ago). To summarize, I mentioned my time series logging program, Adversaria, and I wrote about my little toy key-value store, Fickle. I wrote that I would try to add transactions, and then I ended with a goal to use it to power this blog. Since then, I wrote a skip list in C, implemented basic transactions in Go, and played around with chunks of ordered byte slices, and wrote a memory-mapped linked list.

BeagleBone Black

I’m a huge fan of the BeagleBone Black. It’s an alternative to the popular Raspberry Pi. I’m such a fan that I got my second one a few days ago. This website runs on a BeagleBone Black. It’s powered via USB, and it’s connected to my switch at a data center. For some reason I choose to use this credit card-sized ARM computer to host stuff instead of a VM. It reminds me of the little VMs I used to work with in middle school.

Overdue invoices

I used to subscribe to CloudLinux but then stopped. I do not use CloudLinux anymore. I’m still getting invoiced every month even though I’m not using my license at all. And so, every month, my balance goes up. I will not pay it. Why should I? I didn’t use their service! It’s too much of a hassle, in my opinion, to contact them about it. Eh. Here’s the thing: I can’t get another license from them unless I cough up the $42.00, right?

Nontrivial pipes

I’m going to take a simple concept, UNIX pipes, and basically frame (or misframe?) a nontrivial scenario. $ cat /proc/cpuinfo | grep CPU | wc -l 4 We all know pipes, of course. You pipe stuff in and pipe stuff out. Easy enough. I think the piping in the example above is context-free. Each pipe has a stream of continuous bytes for its input and output. This is the simplest case.

Thoughts on Career/Job/Internship Fairs

This is just my opinion. I don’t like career fairs. I don’t think they are of any value to me. Most people might disagree with what I have to say, and that’s totally cool. I’m not like most people. Career fairs are way too competitive. It’s probably the worst place to differentiate yourself. Everyone’s in a suit. Everyone’s carrying around their resumes, and everyone basically gives the same rehearsed intro, right?

Cursive ‘f’

I was sitting in one of my math classes the other day (how specific!) and noticed that my professor had an interesting way of writing his lowercase cursive ‘f’. You know how the bottom loop thing should go to the right? He wrote his going to the left. I thought it was weird because I thought it wasn’t correct. Then I thought, “wait, have I been writing my ‘f’ wrong?!” Later that day, I started doing homework for another class.

LLVM

For some reason, instead of studying for my math quiz today (it’s past midnight right now), I decided to play around with LLVM. One of my homework exercises was to solve the Fibonacci difference equation, F(n) = F(n-1) + F(n-2) for n >= 2, so that’s why I wrote a recursive Fibonacci function. I think it’s pretty cool considering I never formally learned assembly. declare i32 @printf(i8* noalias nocapture, ...) @numPrintStr = constant [27 x i8] c"#%d Fibonacci number is %d\00" define void @printNumber(i32 %a) { %f = call i32 @fib(i32 %a) call i32 (i8*, ...)* @printf( i8* getelementptr([27 x i8]* @numPrintStr, i32 0, i32 0), i32 %a, i32 %f) ret void } define i32 @fib(i32 %a) { entry: switch i32 %a, label %recur [ i32 1, label %base i32 2, label %base ] base: ret i32 1 recur: %prev1 = sub i32 %a, 1 %prev2 = sub i32 %a, 2 %prev1val = call i32 @fib(i32 %prev1) %prev2val = call i32 @fib(i32 %prev2) %sum = add i32 %prev1val, %prev2val ret i32 %sum } define i32 @main() { entry: call void @printNumber(i32 40) ret i32 0 } Output: $ llc fib.ll -o fib.s $ gcc fib.s -o fib $ ./fib #40 Fibonacci number is 102334155 There’s probably a way to generate a binary without using gcc, but it’s late and I’m too lazy to figure it out.

Streaming HTTP

I just came back from this month’s PyCHO meetup, and one of the things that we discussed was how to handle many simultaneous HTTP clients that send data in streams. Someone had a project, written in Python of course, that accepted large file uploads from clients, computed a hash, and sent it to S3. The goal is to do everything concurrently with minimal resource usage. I’m not a Python programmer, but as a Go programmer this is very simple to do.

Rotring Rapid Pro Review

This is the Rapid Pro pencil by Rotring. In case you don’t know German, “rotring” basically means “red ring.” I’ve been using this pencil since March. It was the only thing I used when I was doing school work, so that basically means I used it (a lot!) for notes, homeworks, essays, etc. This one takes 0.5 mm lead, but Rotring has other versions as well, including a ballpoint. I think they should all roughly be the same.

What I Do Alone

When no-one’s looking, when the house is empty, when the afternoon is yours alone — what you choose to do says a lot about you. Pay close attention to where your mind wanders in the shower. Your natural wanderings are your compass to what’s truly interesting to you. — https://medium.com/things-ive-written/thirty-things-ive-learned-482765ee3503 I mentioned something about this before when I wrote about thought experiments. I still do this stuff. These days, it happens mostly right before I go to sleep.

Retrakt Pen Review

This is the Retrakt Pen by Karas Kustoms. It’s machined out of aluminum and uses the popular Pilot G-2 gel refill. While this pen is made out of solid metal, machined in the US, and looks neat, it’s not a good pen. The clip is made of stainless steel and feels quite durable. It comes with two springs (one shown here) and a spacer. The pen also accepts a Parker ballpoint refill, so that’s what the smaller spring and spacer are for.

Lessons learned.

I’ve finished another summer of interning. Here are some incredibly useful “lessons learned.” Some of these came from hours of frustration. Some are related to others. And by the way, just because you can read this list does not mean you’re “learning” these lessons. ;) Intuition is incredibly powerful. Having the right tools is the difference between trying for hours or days and failing, and getting the answer within minutes. Tools can either empower you or burden you.

Workaholic.

It’s been over three weeks since my last post? Wow. I know I’ve been throwing some thoughts around in my head (and my notebook) for a while but they’re still only a paragraph long, I guess. They’ll end up here eventually. Onto the good stuff. I’m pretty sure I’m a workaholic. A workaholic is a person who is addicted to work. While the term generally implies that the person enjoys their work; it can also imply that they simply feel compelled to do it.

Fun to obligation.

A few months ago I was sitting in the back seat of a minivan on the way to New Jersey to visit family. I had music playing through my headphones and I was staring at a bunch of sFlow specification documents that I had downloaded onto my laptop. sFlow is a packet and counter sampling specification, and I wanted to write a decoder in Go. It’s pretty neat to have the entire history on GitHub.

Taking notes.

I took a lot of notes in high school. I remember color coding stuff in freshman biology to help me memorize. Other than memorization, I didn’t really understand the point of notes. Everyone else was doing it, so I figured I should too. I never really looked back on my notes unless I was memorizing something. I think math was the exception. I think the examples we did in class were better than what the textbooks offered, so I depended on my notes.

I have a dedicated bagger at Kroger.

I have a dedicated bagger at Kroger. Kroger’s a grocery store, if you’re unfamiliar with it. Whenever I go to Kroger to get groceries, the same person (probably still in high school) bags my stuff. And no, it’s not me because I don’t use the self-checkout ;). It all started last month… The bagger at the grocery store asked, "are you from India?" I think the whole wheat bread gave it away.

Holding back.

(This is basically a brain dump. No revisions or drafts!) For many people, their GitHub profile is their coding portfolio. I wanted mine to be like a portfolio too. I guess you can say the same thing about Misframe. I don’t write about everything I think about. Sometimes I let things simmer around in my head, and when I’m not feeling lazy I just collect those thoughts together and digest it into a post.

People-taught.

I started programming in 9th grade. It wasn’t completely new to me. I knew how to make websites with simple HTML and CSS, and I knew how to code simple things like menus in Flash using Actionscript. I also spent a lot of time working with VMs, content management systems (Joomla, etc), and LAMP in middle school. I was not new to doing stuff with Bash, either. I went to a magnet high school.

Syntax Highlighting.

A couple of years ago, for whatever reason, I wrote a color scheme. I called it “DarkFruit” because it has some fruity-looking colors on a dark background. The “LightFruit” version has the same colors on a light background. I used it for a long time, and I got used to looking at it, but there are many things obviously wrong with it. The colors basically look randomly chosen. Comments are bright yellow and bolded, which was a little annoying.

ARP 101.

ARP ARP stands for Address Resolution Protocol. It’s a protocol to translate network layer (layer 3) IP addresses to link layer (layer 2) MAC addresses. Why would you need this? Let’s look at a simple example. Here we have a couple of hosts. We need them to communicate with each other and they both have IP addresses on the same subnet. They’re connected to a layer 2 switch. We want to have communication between these two servers over TCP, which is a layer above IP.

Halfway.

This post is way overdue :-/. I finished my second year at UVa early this month. Overall, it was much more different compared to the previous year. My first year was all about finding my place and what I wanted to do. That required trying lots of different things and seeing what I liked. I think it worked out well. I really like what I do these days. Confidence I think I had confidence issues last year.

Appending to a file instead.

In the previous post, I wrote about inserting records into a file and keeping them ordered. The trivial solution of shifting records in a file is horrible in terms of write amplification, because we rewrite most, if not all, records on every single write that does not occur at the end. So let’s make that better by writing to the end. In order to keep things in order, we need to know to move from the newly inserted record back to the top of the file.

Prepending to a file.

Prepending to a file is relatively expensive, but to what extent? I decided to check. I wrote 2^23 integers to a file, in ASCII, separated by a newline. On my ThinkPad with spinning rust, it takes a little longer than 12 seconds. The file size is around 31 MB. In order to “prepend” an integer to the beginning, we essentially have to shift all of the other elements forward and then write what we need at the beginning.

Batching channel values.

This was an interesting use of channels in Go from today. The goal was to listen on a channel for a stream of values as they arrived one-by-one. We then had to process and batch them up in order to get them into a certain format, and then send them on another channel. The following code shows a simpler example: listening for a stream of ints and batching them into slices.

“Too young to be serious.”

I was at Bodo’s one morning last week getting a bagel before work. I had a few minutes to spare because I had to wait for the bus. As I finished ordering a bagel, the lady behind the counter said, “Smile! You’re too young to be serious!” I was a bit surprised. Usually people tell me that I smile a lot. Then again, it was early in the morning and I had not had my caffeine yet.

Laplace and Fourier transforms.

Let’s talk math because it’s cool. Last year I took a course on ordinary differential equations. We were basically taught to identify different types of ordinary differential equations and use an appropriate method to solve them. Certain problems could be solved in multiple ways, but some methods are better than others. One interesting method was the application of the Laplace transform (1). The Laplace transform helps when dealing with convolutions of functions.

Grades.

This past semester was relatively more difficult for me. I decided to get all of my humanities requirements done in a single semester. I cleverly picked courses such that they each satisified at least one requirement. It was difficult, considering I had a relatively late enrollment time, but I managed to work it out. I’m not much of a humanities guy. I have to work harder, and it’s a bit more stressful.

Time-wasting conversations.

On December 30 last year, I decided to delete my Facebook account. I was regularly deactivating my account and finally decided that I did not need it anymore. Facebook chat was one of the sources of time-wasting conversations. These conversations took up time, were generally about things I did not really care about, and I did not get a lot out of them. They unfortunately stole time that I could have used for other things.

On advice.

I’m rarely going to give advice anymore. Here’s why. It’s sometimes unsolicited. Unsolicited advice is sometimes good and sometimes bad, both for the person giving it and the person receiving it. If someone doesn’t ask for advice, they either don’t need it or they don’t know any better. Regardless, I feel like more often than not it’s bad. They don’t act on it and I feel like I’ve wasted my time.

Scanner.

I got a scanner recently and have started to scan documents. For example, the following are my “cheat sheets” for a STAT 3120 (Introduction to Mathematical Statistics) final exam. It was three hours of fun! I’ll be uploading more occasionally. I’m a fan of graphology so it is interesting to see how my handwriting changes.

Because I had to.

I gave an extremely short demo of a pair of OpenBSD routers in a redundant setup using CARP. Someone asked me afterwards why I learned about this and my answer was, essentially, because I had to. I thought it was an interesting question since I never get asked why I learned something. I learn a bunch of stuff. Why? Why learn C? It was taught in school. I had to. Why learn JavaScript?

Powered By

A long, long time ago, Misframe used to run on Wordpress. That didn’t last very long :-). Then, in 2011, I learned enough about Node.js and CouchDB to build my first blogging “platform” from scratch. That was a really fun project. I essentially wrote most of it in one evening during a Thanksgiving break. Occasionally I browse through the source code to see how horrible I coded back then. Eventually it became a hassle.

Disconnected.

Yesterday I attended a couple of talks at this year’s Tom Tom Founders Festival. The first was titled “Why Charlottesville? Founding Stories,” and the second “Scaling Talent from a Small Town.” I occasionally scanned the audience to see who was attending the sessions. Other than myself, I think there was only one person who looked like a student. In the second session, I didn’t see any students. The majority of people around me were at least a decade older than me.

On Regret.

A few years ago, one of the things I decided to do was to live without regret. I never say, “I wish I had done …” or “I shouldn’t have …” If you should have done something, why didn’t you do it? If you, through reason, choose to do something you believe is the best choice, why would you have to regret it in the future? You did nothing wrong! If you didn’t choose the best option, well, that’s something to fix.

Little things.

I’ve always liked getting nicer things of objects that I use most often. I like to use nice keyboards, for example, because I always use a keyboard. I like to use a nice mouse for the same reason. I like having nice office supplies in general. It is these little things in life that I use constantly, day after day, and I like to make sure I use nice things. I’m a fan of subtle aesthetics, I guess.

There’s no rush.

I’ve just finished my first day of Fluent 2014. I’ve talked and listened to people about jobs, careers, internships, startups, and businesses. Also, by quick observation, I think I’m the youngest person here. It’s pretty weird when I’m in conversations where people talk about their jobs or careers. I haven’t been working that long – I think I’ve worked for a grand total of less than 2 years. I barely know what it means to have a job!

Write-ahead logs and shadow paging

There are at least two ways to provide atomicity and durability in databases. The first, and (I think) more common approach, is write-ahead logging. The other is shadow paging. The only system that I’m aware of that uses shadow paging is lmdb. Write-ahead log (WAL) This is pretty easy to understand. Every transaction is first appended to a log as a set of mutations to the state. Eventually, these mutations will make their way into the primary data structure (usually some kind of B-tree).

Making skip lists faster

First off, watch this video: Bjarne Stroustrup: Why you should avoid Linked Lists. To summarize, Stroustrup states that using vectors is significantly better than using linked lists because linked lists are horrible at cache hits. (From http://daviddeley.com/programming/docs/page_faults_and_array_addressing.htm) Notice that reading sequentially from a vector means sequential reads in memory, but sequential reads from a linked list is comparable to random (bad for cache) reads in memory. How would you make a linked list faster?

Getting confused.

After working with C for a while, I feel like I understand C relatively less than before. I’m going to put this in the context of my vlmap ordered map data structure. Here’s a code snippet to start off: uint8_t* key = "foo"; int keylength = 3; uint8_t* val = "bar"; int vallength = 3; vlmap* m = vlmap_create(); vlmap_insert(m, key, keylength, val, vallength); I’ve modified the original code for simplicity.

Managing memory

I found libgc, a garbage collector library written in C, a few months ago and wanted to write something like it. The usage is really simple: replace all mallocs with gc_malloc and delete all occurrences of free in your code. Now, a few months later, I’m thinking that it’s not going to happen :). I’m not giving up, but rather changing my focus. I learned some more about other garbage collectors, especially those with persistent data structures.

Service down or server down?

How do you check if a server (an entire physical server or VM) is down? If it’s running some sort of web server, I’ll usually open up a browser and check if a page loads. This isn’t very useful, since a web server process could be down but the server is still up. In that case, I try pinging it. If that doesn’t work, I’ll try SSHing into it. If none of those work, I am pretty sure that the entire server is down.

Mechanical keyboards

(Tux says “Hi.”) I purchased a Monoprice mechanical keyboard almost a year ago (Jan. 30). It was around $60 and has the loud, clicky Cherry MX Blue keys. It’s on the cheaper-end of the mechanical keyboard range, but it has literally the same switches as the more expensive keyboards, like the Das Keyboard. I use laptops a lot these days (perhaps exclusively), and they all have the Apple-style chiclet keys. I don’t have a problem with these keyboards – I’ve been using them for around 7 years now.

Gambling with cheap RAM

I got an email from 123Systems about a week ago: First off, I got that email because I used to be a customer of 123Systems. I would not recommend them if you’re looking for a reliable VPS host. I’ve had issues with them and their support isn’t the best. They are, however, really cheap. Read that promotion carefully – a 2 GB RAM VM for $25 per year. That’s a little more than $2 a month.

Thoughts on lists, trees, and other data structures

This past spring, I used to go to a study room in Clemons after dinner to work on problem sets. Those rooms have white boards, which I often used to check over work or write some notes as I did my homework. After I was done with my homework, I’d start drawing trees and come up with rotation algorithms. Sure, there are algorithms out there for balanced trees, but I always got the feeling that they were really generic and hard to improve.

C is weird.

Falling off the end of a function that is declared to return a value (without explicitly returning a value) leads to undefined consequences. — http://stackoverflow.com/questions/293499/what-happens-if-you-dont-return-a-value-in-c Yeah. I forgot to write a return line in a function that was supposed to return a pointer. I didn’t notice it for days since the program worked fine – the right thing was being returned! I tried it out on my BeagleBone Black and it looked like calloc() was returning NULL.

Haste.

I guess people call these types of posts “post mortems.” Here we go. I got this email this morning: I generally don’t trust alerts with 100% confidence, so I started to check on it. It was weird: it was responding to pings and establishing TCP connections, but it wouldn’t respond to anything. SSH wouldn’t work either. I disabled and enabled its ethernet port from the switch. It didn’t fix it. The next step was to give it a power cycle.

Don't panic! Type assertion safety.

Go’s container/heap package documentation has an example of an integer heap. Here’s an interesting part of it: func (h *IntHeap) Push(x interface{}) { // Push and Pop use pointer receivers because they modify the slice's length, // not just its contents. *h = append(*h, x.(int)) } There’s a little problem here. The point is, you shouldn’t copy/paste that integer heap example into a package that you’re going to use. The last line (ignoring the brace) has a type assertion.

State of the... state!

Three months ago, I started working on fickle, a key-value store written in Go. The point of this post is to just jot down some thoughts on where the project is right now and where I’d like to take it. Adversaria This all goes back to Adversaria. It’s that small program I wrote in Java to help me log traffic. I use it in combination with a little Flot code to make pretty graphs: It’s really simple, which is great, but it’s also limited.

Types...

What does this print? package main import "fmt" type Foo interface { foo() } type A int type B A func (a A) foo() {} func (b B) foo() {} func main() { hashmap := make(map[Foo]string) a := A(1) b := B(1) hashmap[a] = "foo" hashmap[b] = "bar" fmt.Println(hashmap) } Let’s see… there’s a map that takes keys of type Foo, which is an interface. a and b both implement foo(), so they’re of the Foo interface.

Magic.

A friend was talking over dinner about when he was asked, during an interview, what happens when you type in a URL into an address bar in a browser. Lots of stuff happens! I’m just going to start listing stuff. It’s practically impossible for me to get everything, so I’ll just do a brain dump: A DNS request over UDP to get an IP address A TCP connection gets established (SYNs and ACKs!) A GET request is created by the browser and sent over that connection Tons of different things can happen after that, including proxying, database operations, cache reads, whatever.

Rejecting customers

When I first started Bitcable, I used to get really excited whenever I had a new order. It sounds great, right? A new customer, more revenue, getting bigger! Yay! And it was great. I thought it was perfect… until there were trouble clients. My “trouble client” classification is pretty diverse, in fact. I’ve had people open up support tickets constantly. Some have no idea how to manage servers and ask for help on trivial tasks.

Router-on-a-stick

(This is an old post from July 2012.) Attribution: Patrick Stahl This is just a quick configuration example to setup a “router-on-a-stick.” I’m using Vyatta for the router and a Brocade switch. I’ve changed the real addresses. Brocade switch configuration Ethernet 16 is the trunk port, in the Cisco lingo. It’s hooked up to the Vyatta router. It has to be tagged because we need to tag each frame with a VLAN ID so the switch knows where to send it.

Thoughts on garbage collection

A few days ago, I was in the shower thinking about garbage collectors. I think some of my best ideas come from the shower, and I think it’s because I don’t have anything to distract me. I was thinking about easy it was to use the Hans Boehm garbage collector: Empirically, this collector works with most unmodified C programs, simply by replacing malloc with GC_malloc calls, replacing realloc with GC_realloc calls, and removing free calls.

Non-blocking I/O thoughts

I started using Node.js a long time ago. It was the first time I saw async execution outside of the browser. It was cool – still is. I worked with Node internals for a significant part of a summer. It doesn’t take long before you realize that the event loop is one of the most important components of Node. Node uses libuv for its evented I/O, and I sort of got into it, but not really.

Digging deeper

I’m starting to find kernel internals, debugging, tracing, and compilers really interesting. Wait no, I’m finding interesting things about them. Regardless, I’m digging deeper into stuff. It’s messy and ugly, but that’s what you get when you look under abstractions.

What not to do...

A web hosting provider called “Easy Hosting” shut down recently. Here’s what their website says right now: We’re Sorry. After 5 years of service(rlshosting.com,easyhosting.pw), we are unable to provide service period and the backups were on the same datacenter too. We ran out of money and we cannot pay our server bills. Our Servers are suspended. You can believe what you want but we didn’t spend your money for fun. It wasn’t fun at all.

Hinting

Can you tell the difference between these two images? Click on them to see a full-size version. Hint: there is a difference, and it's font hinting! The first image has hinted text, and it's terrible! It's morphed. The dimensions are funky. Why destroy a perfectly fine typeface (created by artists) by hinting (using algorithms and fancy math)? I turn hinting off everywhere. I suggest you do it too. If you use Ubuntu, you can install Unity Tweak Tool: Sublime Text will still hint.

Three years.

On March 11th 2012, a Sunday, it was just past midnight and I was closing up tabs and getting ready to go to sleep. I had school the next day and I had already set myself up for less than 6 hours of sleep. My phone beeped to let me know that I had received an email. Usually it’s just a matter of taking a peek and archiving it. This one was a little different.

Thoughts on Go.

At the beginning of the year, I told myself I would learn Go. Stuff about Go seems to pop up on the front page of Hacker News frequently. In fact, it’s gotten to the point where I think people upvote articles that mention Go in their titles. People really like it, or they just like talking about it. In any case, I think it’s worth trying out. I’ve been using Go for a few months now.

Where do I stand?

I do many different things. When I was younger (around 11 years old), I used to make 3D models with Blender and play around with raytracing, caustics, bump maps, shaders, and all that good stuff. At the same time, I was getting into GNU/Linux, MySQL, and Apache. A few years later, I started playing guitar. Eventually, I became fascinated by cameras and photography. I also became the founder of a company.

Giving Back.

I just want to preface this by saying that I didn’t make a draft before publishing this. It’s been a while since I’ve written something so I just wanted to get something out as quickly as I could. I’m expecting this post to be of a lower quality than usual. I remember my first GNU/Linux distribution years ago. It was Fedora Core 3. I think we still have the CDs back home.

8 Hours

I sleep for 8 hours (overestimate). I intern for another 8. Since there are 24 hours in a day (right?), I’m left with 8 hours. With those 8 hours, I need to eat, cook, go to the bathroom, brush my teeth, get around, and so on. Let’s see… I’m a university student, I’m interning at a database startup, and I run a web hosting company. As a student, I’m surrounded by people my age.

Callback Magic with Go?

Go sounds great. You can write concurrent programs using goroutines and channels without ever touching that callback nonsense, right? Yeah! I love FoundationDB. On the surface it’s extremely easy to use — it’s just a key-value store. All of the complexities are so well abstracted that you don’t have to worry about them. I wanted to try to use FoundationDB with Go. However, there’s no Go driver for it yet. I decided I should try to fix that.

Potential

Remember my success equation? Success = potential + opportunities. I wanted to share my thoughts on potential. As I mentioned before, potential is everywhere in an academic setting. It’s hard for me to not think about it. Potential is necessary for success. A simple example is a math test. You build up potential by studying the material, doing practice problems, and understanding key concepts. You’re given the opportunity to succeed when you’re given the test, and you’ll succeed given that you had enough potential to do so.

Thought Experiments

I’ve been doing this for a few years now, and it’s an interesting process. It’s pretty much exactly what a thought experiment is: In its broadest usage, thought experimentation is the process of employing imaginary situations to help us understand the way things really are (or, in the case of Herman Kahn’s “scenarios”, understand something about something in the future). The understanding comes through reflection upon this imaginary situation. – Wikipedia In my “thought experiments,” which generally occur in the shower or when I’m shaving, I assume I make a certain decision and then I fast forward into the future.

Majoring in Math

It’s May. I have a few finals left before I can say that I’ve completed my first year of undergrad. You could say that I’ve changed my mind a lot about what I wanted to accomplish over these four years, but you could also say that I haven’t. As for my major, I think I’ve landed on something. I want to major in mathematics. I applied to the School of Engineering and Applied Science (SEAS) at UVa, also known as the E-school.

Mastering the art of context switching

In computing, a context switch is the process of storing and restoring the state (context) of a process so that execution can be resumed from the same point at a later time. – Wikipedia Essentially, context switching means you stop what you’re doing, do something else, and then go back to what you were originally doing. Context switching is important in computing, but it’s also important for people. We do many different things in our lives, and that involves playing different roles.

/etc/shadow passwords with Go

Here’s how you can create hashed + salted passwords for use in /etc/shadow. I’m using libcrypt. package main /* #cgo LDFLAGS: -lcrypt #include */ import "C" import ( "fmt" ) func main() { fmt.Println("Hashed:", C.GoString(C.crypt(C.CString("password!!!"), C.CString("$6$Vi.DuMQS")))); } The output should be… Hashed: $6$Vi.DuMQS$3hoKGTZ4ym8W3VHhLith2rGnChBtEobC3h07MVfdzk/0GxnWlkAUZ7/msJ1t93ekA8qc7jzVfP./8fnkfk/e6/

Working and growing up

My first internship. In July 2011, I joined a company called 25K Digital as a software engineering intern. I just finished my junior year of high school, and I was excited. It was my first internship and I was ready to apply those impressive skills I had acquired over the years. I was ready to make some cool things! 25K Digital is a digital media agency. Most, if not all, of their work is client-based.

Why I don't like to use my Macbook or Windows

My Thinkpad running Ubuntu 12.04 is my workhorse machine. I get most of my work done on it since my desktop is back home in Ashburn. Here’s the thing about looking at good font rendering continuously: switching to a worse display gets really annoying really quickly. People have varying reasons why they don’t use Windows or Mac OS X. I think I have a less common reason. It’s not about usability or software compatibility.

Writing a database.

Early last month, I wrote Adversaria. I don’t have a precise definition for it, but I like to think of it as a tool to store time series data in key-value format. It’s really simple but useful. Using Adversaria and a little Flot magic, I make graphs like this: A few weeks ago, I decided to take it one step further. I felt that it was time to use whatever I learned to make that ad-hoc data storage solution to write a distributed key-value store.

Cursive and handwriting.

In 9th grade, our English teacher would assign us journal entries for homework. Sometimes we had prompts, but usually we would have the option of writing whatever we wanted. Poetry, short stories, haikus, whatever. I remember writing about my handwriting a few times. It’s interesting. I noticed in middle school that my handwriting always changed somehow after summer vacation. My handwriting would even change as I wrote a journal entry. I wrote about that too.

Using C libraries with Go.

Here’s a simple example of how to use a C library from Go. I made a small library called libfoo. libfoo.c #include "libfoo.h" int foo() { return 1; } libfoo.h int foo(); Of course, you have to compile that as a shared object with GCC. $ gcc -shared libfoo.c -o libfoo.so Make sure you copy over the binary to /usr/lib/ and copy the header to /usr/include/. The Go file. package main /* #cgo LDFLAGS: -lfoo #include <libfoo/libfoo.h> */ import "C" import "fmt" func main() { fmt.Println("From libfoo: ", C.foo()) } …and that’s it.

Opportunities.

As I mentioned in this post, I think success is the product of both potential and opportunity. I’ve started noticing people with great potential who struggle to succeed because they don’t receive the necessary opportunities. It’s annoying to think about, but there’s not much to do about it. Given enough potential, it’s not hard to find opportunities. I think sometimes opportunities just come to you. That’s what it feels like to me.

New theme...

I got bored with the old one.

Stripes.

I put racing stripes back on my guitars. I took them off temporarily because I messed up the first time and didn’t bother to replace them. Why the stripes? I’m not sure. I’m definitely not the first to put stripes on a guitar. I haven’t seen many others do it though. Still, I think they look nice. Suddenly a couple of strips of black electrical tape make these guitars mine. I can look at them for a millisecond and tell that they’re my guitars.

Time machine.

I really want a time machine. I want to meet past-Preetam. I’m different from him, and in some ways it’s incredibly significant. I forget what he was like. I forget how he thought. I want to hear his stories. I felt the same way two years ago, I think. That’s why I started a blog. It’s gone now, unfortunately. There’s nothing I can do about it. Misframe I’m terrible with names.

Design.

Jay (MiB): You know what the difference is between you and me? I make this look good. Think about that quote for a second. The Men in Black agents are interesting. They all look essentially the same. Even their names are bland (Jay/J, K, Zed/Z). Let’s be honest here: it’s hard to be unique. Whether you’re a high school senior finishing up your college applications or a recent college graduate updating that resume to a web hosting company in a saturated market.

Adversaria.

Everyone (or almost everyone) uses RRDtool to store traffic data. I wanted something a little different. Couchbase woes! I started using Couchbase to log traffic. I had a document for each data point and was able to run map-reduce queries with Couchbase views. It worked great for a couple of months and then for some reason… weird stuff happened. Over 5,000 documents stopped getting mapped. That’s not good. I used a 4-node Couchbase cluster, because why not?

Starting over... again!

Alright. This is the what… third or fourth time I started over? There’s a reason why I call this blog misframe. I’m constantly framing (or planning) poorly and doing things wrong in general. That’s not bad, though. I love what I do. I love being annoyed at things. Seeing exactly why things don’t work convinces me better than anyone else could. Tumblr, don’t let me down!

Success, potential, and opportunity.

Success I think success is the product of work. Not hard work, necessarily. Just work. The rest is about two very important things: potential and opportunity. Potential I see potential all the time. Academic environments do that. There’s a continuous expansion of human capital. Amazing. For some reason, I think I forgot about my own potential. The past two years of my life have been incredible—filled with many changes, experiences, and people.

Thoughts on working.

This was my 2-inch response to a talk by Richard Hamming called “You and Your Research.” The actual talk is great, and I believe there’s a recording on YouTube. I’ve linked to a transcription in case it a video doesn’t exist. I think one of the most obvious yet important statements in the entire talk is “you have to neglect things if you intend to get what you want done.” As students, we’re constantly dealing with situations where we just have too much to do.

Frustration.

I like frustration. I like being annoyed by how things work. I like it when I find something wrong about the status quo. Frustration is what drives me. It’s my “engine of creation.” It’s my source of inspiration. Creation isn’t just something that’s done—it’s a solution to a problem. It’s easy for me to get productive. I just have to get slightly furious.

Looking up.

How often do you just look up on a clear night? I’m starting to do it as often as I can. It fascinates me that photons have been traveling for perhaps thousands of years through space to pierce my cornea. It fascinates me that there is so much more to these dots in the sky. There are red supergiants (Betelgeuse), open clusters (Pleides), planets (occasionally), and galaxies (Andromeda). They’re out there.

The Journey to Perfection.

I start many things. I never finish anything. For some reason, I think of “finishing” as perfecting. I never think of something I create to be perfect. That’s just unrealistic. So here I am with a mountain of works in progress. Writing I think about writing all the time. I’ve been thinking about a certain blog post for months. I think about it when I’m walking to a class, in the bathroom (seriously), or when I’m trying to fall asleep.

Ampless

Attribution: Marco Raaphorst Introduction For years I had a beginner’s electric guitar amp. After a while I wasn’t satisfied with the sound that I was getting from it. After looking at (and listening to) products like Native Instrument’s Guitar Rig I decided to transition to a digital rig. Why not just get a better amp? Good amps are expensive! Tube amps sound the best with their warmth and rich distortion, but they need to be cranked to a high volume.

Text

It’s been a long time since Johannes Gutenberg revolutionized the printing process by developing movable type printing. I’m guessing most people haven’t even seen a printing press in real life—I definitely haven’t. Typesetting back in those days is completely different from what it is now. Typesetters had to physically place movable type into letter cases. Things have changed...well, for most people. Attribution: Willi HeidelbachAardvark Letterpress in Los Angeles is different. It’s a company that still does typesetting the “old” way.

Goods and services on the web

I don't really have an introduction for this one. I honestly just wrote everything/anything I thought about. Hopefully you'll get something out of this. I introduce my forecast about business on the web, so let's see if I'm right. Two examples. I’ll start with something people can’t really relate to: fonts. In my case, there were two choices to make. FontSpring or Typekit. FontSpring sells fonts with a one-time fee. You buy specific fonts, use them, and that’s it.