Wednesday, July 13, 2011

The Creative Process


This happens with about anything that I do that lasts more than a few days. The funny thing is that every time I press that "publish" button, something good happens. I honestly don't think that any of the work I published here, over at the Facebook data blog or at DataInColour is particularly great, but for some reason, posts that I am more hesitant to publish end up being the better ones.

(And yes, while working on the "Hacking Educations" series I did accidentally run "rm -rf *" while attempting to move code into git. It might have ended up being a good thing. People say that knowing what to remove is half of writing well, and I try to do that in my data posts: i.e. remove "random facts" that just aren't part of the story. In the "Hacking Educations" series and (less so) in the "Mining Jobmine" series, there were a lot of "random facts" than didn't really fit anywhere. Losing half of my code made it easier to remove those.)

PS: Sorry about the bad art and terrible "scanning".

End of Entry

Thursday, June 23, 2011

New Data Blog: Data in Colour

I've been waiting for a while to announce my new data blog, but here it is now.


All my data projects will now be posted there. Non-statsy posts will appear here as per usual.

End of Entry

Sunday, June 19, 2011

Chinese and Entropy

I happened to have a cold that day, and was trying to write a brief note to a friend canceling an appointment that day. I found that I couldn't remember how to write the character , as in da penti 打喷嚔 "to sneeze". I asked my three friends how to write the character, and to my surprise, all three of them simply shrugged in sheepish embarrassment. Not one of them could correctly produce the character. Now, Peking University is usually considered the "Harvard of China". Can you imagine three Ph.D. students in English at Harvard forgetting how to write the English word "sneeze"?? Yet this state of affairs is by no means uncommon in China. -- David Moser

I came across David Moser's essay on why Chinese is hard a while ago, and found it quite entertaining. Moser points out that Chinese is not only difficult for non-native speakers, it is also difficult for native-born Chinese as well. Among the reasons Moser thinks Chinese is hard are that the language is not phonetic, it has no alphabet, and it has a god-awful dictionary system.

All of these reasons are somewhat consequences to the fact that Chinese is much denser than English. I would hypothesize a Chinese text to have a higher entropy than its English equivalent (although this is probably pretty difficult to measure). Even in speech, one can convey the same information in fewer syllables in Chinese than in English.

I think one reason why Chinese is so terse is that historically, Chinese people commended terseness. Terseness is associated with wisdom. One can imagine followers of an old master asking a long list of questions, only to receive a single, one-syllable word in response.

End of Entry

Friday, April 15, 2011

Education Inflation, Technological Advancement, and Cognitive Surplus

The question of whether or not to pursue higher education is a relevant one among my almost-graduating peers. Especially with all the attention this issue has received recently, I can't help but engage in a bit of amateur speculation myself. Most of the articles on this issue study the nature of higher education directly. I don't wish to do that. Instead, I'd like to speculate about why it makes sense economically for there to be education inflation, and why education inflation might have an effect analogous to producing a generation of sitcom fans.

Education Inflation

The amount of education one needs to obtain a particular job has increased over the years. Positions that used to require high school level education now require college/university education. One can say that technological advances demand more skilled individuals in the workforce, but that's not the whole story: Asian countries such as China aren't exactly more technologically sophisticated than countries in North America, but in China one would need to have a PhD, have decent grades, and have spent at least a year studying abroad in order to even apply for an entry position at a bank.

So education inflation is real, and overpopulated Asian countries are having it much worse. This makes sense considering the natural explanation for education inflation that I always hear: as population increases, the number of candidates that apply for a given job increases, forcing employers to raise the bar on what is considered to be a "qualified" candidate. Admitting only candidates that have achieved a certain level of education is a quick and easy way of reducing the resume pile, albeit it is somewhat arbitrary.

This is a nice explanation, but I think we can dig deeper: why are the number of qualified candidates per job increasing?

Technological Advancement

Back in the day of subsistence farming, every person in a society needed to contribute his or her efforts in order for the group to survive. With technological advancement, one person's effort is now sufficient enough to feed many other people. The advancement allowed us to move from an agriculture-based economy to a service-based economy, meaning that instead of 50 people working on food production, 10 people can take care of food production while the 40 others find other ways to enrich each others' lives.

This is supposed to be a good thing. Technological advancement gives us, as a society, free time to do whatever we wish: build better shelter, design better clothing, invent the internet, speculate about the meaning of life, or even just take a break! Unfortunately, we have a different name for prolonged free time: unemployment.

We produce enough food to feed everyone, yet in order to live well and be respected, you need a job. In order to obtain a job, you need to satisfy an employer's stringent and sometimes arbitrary requirements. In order to do so, you must spend extra years of your life in school. Thus, as a society, instead of using the free time we gain from technological advancement to our advantage, we end up forcing people to spend this time in school pursuing education that they might not necessarily enjoy.

Cognitive Surplus

I first saw the idea that "free time" might be an important aspect of society from an article by Clay Shirky. Shirky hypothesized that as we initially introduced the five-day work week and 8-hour work days, we didn't know what to do with all the extra free time. As a consequence, we spent the extra time watching sitcoms. "The interesting thing about a surplus like that is that society doesn't know what to do with it at first" – he says. Shirky's article is about how sites like Wikipedia are finally taking advantage of the cognitive surplus, a resource that can do so much for us.

Back to the original question: when we don't need every single person in society to work to supply everyone's basic needs, what should we do with the extra time available? This is by no means an easy question to answer. I think that in some cases, pursuing higher education now is like what watching sitcoms were back in the day. It is society's first answer to the question, perhaps a decent first-order approximation but definitely not the best answer.

To be fair, the question of what to do with extra resources isn't a new question. We've had to answer this question many times over as our technology became better and better. We started by moving from an agricultural economy to an industrial economy to a service economy. I'd like to ask an economical historian whether the move was as smooth as we'd like to think it is.

Now what?

"Now what?" is the final thing the fishes say in the ending scene of "Finding Nemo" when they've finally escaped from the aquarium into the ocean. It's never an easy question to answer, yet it is one that many of us will need to answer in the next year or so. In this economy, higher education is the default answer, much like watching television was back in the 50's. After all, when we ask a senior student what they plan on doing after graduation, we expect to hear either the name of a prospective employer or that of a university.

Defaulting to a job or a masters program gives us a certain social standing along with a support structure and an illusion of certainty (some money or a degree at the end). It's certainly not the only choice, though. For those of us that can live off of savings or parents for the next few years, we have a tremendous amount of freedom to do whatever we want. Freedom in excess is terribly scary and has inherent risks, but if we can accept the uncertainty, we can work through it. We've done it before: we eventually watched less sitcoms and created Wikipedia. We can do something better here, too.


End of Entry

Friday, April 1, 2011

Canadian J1 visa data: sanity check failed

Being sick was an excuse for me to playing around with R and this dataset from data.gov. If you're a Waterloo student and you did an internship in the US, you might find the below plot to not quite add up...

If I recall correctly, both in 2008 and 2009 the number of people attending the US Visa info session was well over 100. In fact, the summer 2009 contact sheet for Waterloo students interning in California alone contains 41 names -- that's NOT even including students doing their co-ops in other places in the US and in the fall/winter terms.

Alas, I wonder how trustworthy other numbers listed in this and other datasets are. It's too bad that we can't sanity check other datasets in a similar, direct fashion.

End of Entry