Engineering vs Problem Solving

A lot of people say that I'm not really a programmer. There are other things that are interesting to me, like data science, data visualization, and math. Maybe you're like me too: maybe you're comfortable writing whatever code to solve whatever problems you're having, whether it involves web programming, javascript, python, or just a single line of awk, but you don't have a "proper" background in computer science.

But recently I realized that this wasn't what they meant. What people really meant to say was that I am not an engineer. An engineer is someone who builds things, and building software quickly often means churning out a lot of code. There are many friends of mine who can do this very well, and it's very useful. Sure I can build things too, but not at their speed.

I'm a different kind of programmer. I'm a problem solver. I like to look at a nontrivial problem on software, and come up with ways to solve it with a minimal amount of code change. If you're looking at a really hard problem that people around you would write a lot of code to solve, and you wake up one morning with a much easier solution, then perhaps this is your style too.

A lot of programmers I know falls into one "type" or another. The two aren't really mutually exclusive (especially since good Engineering is supposed to involve good abstraction and thus less code). It's probably also determined by people's preference about the kind of things they like to do, how they approach problems, and what they find more rewarding. Building new things is really fun and writing lots of code makes you feel productive, while doing problem solving makes you feel clever when you count the lines of code you didn't write.

I happened to find the latter more rewarding. So, I'm a programmer who counts the lines of code that I didn't have to write.

End of Entry

Summer 2011: personal reflection

To say that I had a wonderful time last term in San Francisco would be an understatement. I haven't been happier in all my life. Granted I haven't lived for a long time, but nevertheless it was a good term. The reasons it was good is pretty straightforward:

• interesting problems to work on at work -- some things were like puzzles, where the solution came to me on a Saturday morning as I woke up,
• side projects -- launching DataInColour was on the agenda for about half a year, and doing so while supporting DonorsChoose by hacking with the data was a great late evening/weekend project to be doing,
• meeting people doing what I love -- going to data insight sf meant I finally had a chance to work with David and Fravic, and lead me back in touch with the person who originally taught me ggplot2 and seeing what's up at the Square office,
• generally being active -- walking about 40-50 min per day, doing some hiking on the weekends, and learning how to swim probably helped in increasing happiness,
• plenty of down time -- I slept a lot more than usual this term, and lazed around more than a certain friend of mine would be comfortable with,
• and of course, being surrounded by great people -- no explanations needed here.

Trying to reproduce some of these in Waterloo would be difficult, but hopefully there will be enough going on at school to make this term just as fulfilling.

End of Entry

The Creative Process

This happens with about anything that I do that lasts more than a few days. The funny thing is that every time I press that "publish" button, something good happens. I honestly don't think that any of the work I published here, over at the Facebook data blog or at DataInColour is particularly great, but for some reason, posts that I am more hesitant to publish end up being the better ones.

(And yes, while working on the "Hacking Educations" series I did accidentally run "rm -rf *" while attempting to move code into git. It might have ended up being a good thing. People say that knowing what to remove is half of writing well, and I try to do that in my data posts: i.e. remove "random facts" that just aren't part of the story. In the "Hacking Educations" series and (less so) in the "Mining Jobmine" series, there were a lot of "random facts" than didn't really fit anywhere. Losing half of my code made it easier to remove those.)

PS: Sorry about the bad art and terrible "scanning".

End of Entry

New Data Blog: Data in Colour

I've been waiting for a while to announce my new data blog, but here it is now.

http://www.dataincolour.com/

All my data projects will now be posted there. Non-statsy posts will appear here as per usual.

End of Entry

Chinese and Entropy

I happened to have a cold that day, and was trying to write a brief note to a friend canceling an appointment that day. I found that I couldn't remember how to write the character 嚔, as in da penti 打喷嚔 "to sneeze". I asked my three friends how to write the character, and to my surprise, all three of them simply shrugged in sheepish embarrassment. Not one of them could correctly produce the character. Now, Peking University is usually considered the "Harvard of China". Can you imagine three Ph.D. students in English at Harvard forgetting how to write the English word "sneeze"?? Yet this state of affairs is by no means uncommon in China. -- David Moser

I came across David Moser's essay on why Chinese is hard a while ago, and found it quite entertaining. Moser points out that Chinese is not only difficult for non-native speakers, it is also difficult for native-born Chinese as well. Among the reasons Moser thinks Chinese is hard are that the language is not phonetic, it has no alphabet, and it has a god-awful dictionary system.

All of these reasons are somewhat consequences to the fact that Chinese is much denser than English. I would hypothesize a Chinese text to have a higher entropy than its English equivalent (although this is probably pretty difficult to measure). Even in speech, one can convey the same information in fewer syllables in Chinese than in English.

I think one reason why Chinese is so terse is that historically, Chinese people commended terseness. Terseness is associated with wisdom. One can imagine followers of an old master asking a long list of questions, only to receive a single, one-syllable word in response.

End of Entry