Thursday, September 20, 2018

Candy Japan's A/B Testing

The idea behind A/B testing is very simple: randomly assign users to groups A or B, change one thing between the groups, then see whether the change affects some metric of interest.

Candy Japan recently wrote an article on "A/B testing how to ask YouTubers for product reviews". I really like articles like these, where an author breaks down a real experiment and discuss what they learned. In this case, the experiment was emailing Youtubers to see whether they would be interested in making an "unboxing" video to review Candy Japan's product. The author varied the words used in the email and analyzed how the variations affected the rate of a positive response. In total, 180 YouTubers were emailed.

Considering the amount of effort required to message each YouTuber, the author took the opportunity to A/B test several different "splits" simultaneously. The author, unfortunately, does not discuss his methodology regarding the splits, so I'll assume that each split was even and independent of the others.

Some of the results presented were great. For example, including a call to action:
If you would like to receive a review box, please let me know your mailing address.
had the most impact, with the message reducing positive responses by more than 10%. It is a well-known fact in advertising technology that including the price of an item decreases the click-through rate. Here, asking for something as sensitive as a person's mailing address in the first email can come off as creepy.

I did find some of the other results less than convincing. There was only a less than 2% difference in positive response rates difference between including the following "elevator pitch" vs not.
I run a site called Candy Japan, which sends surprise boxes of Japanese sweets to members twice a month.
Since the 180 YouTubers would need to be split into two groups, with each group having around 90 YouTubers, the difference in positive responses between the two groups must be no more than 2. This is a very small difference, too small to justify even the author's toned-down conclusion that "including an elevator pitch of your service may help."

Moreover, the difference between offering viewers a discount vs not:
I can also give your viewers a discount coupon.
was also around 1-2%. It is interesting (but understandable) that the author's takeaway there was that discounts don't really matter -- at least for getting YouTubers interested in making a video. Of course, including a discount might encourage video viewers to become new CandyJapan customers, which is the real goal.

I'm as surprised as the author that the positive response rate was over a quarter. The success is a testament to the author's effort in targetting the right channels. I wonder how results would change if he targetted a wilder group of YouTubers, without the initial selection. I also wonder how he assigned channels into groups, and whether there were any correlations between the groups. Experiments like this are so difficult because there are so many features to test, and getting a large sample size is a lot of work.

The author promised a part 2, to test whether this method of advertising yields better results than buying YouTube ads. I'm very curious to see the results.


This blog post was written so that I could follow my own prompt at https://www.cs.toronto.edu/~lczhang/290/blog.html#b2 

Tuesday, September 18, 2018

A (re)introduction

It is surreal that this blog has been around for almost a decade. I'm almost afraid to write here, lest someone dig up something unexpected from my undergraduate days. (Please don't. I don't know how I got away with such terrible writing.)

The reason I'm back is because I am teaching a communications skills course to undergraduate computer science students. I am asking the students to write weekly blog posts, and figured I should follow my own prompts once in a while. I want to feel the same blank screen and blinking cursors as my students.

So here I am, following this week's prompt to introduce myself to everyone in the class.

I am Lisa, currently an "Assistant Professor, Teaching Stream (CLTA)" at the University of Toronto Mississauga. The term "Teaching Stream" means that I am what is traditionally called a "Lecturer". The acronym "CLTA" means that I'll be at UTM for the next 2 years.

My path to teaching was long, winding, and full of surprises. I was once on a roller coaster called a startup. I spend several years as a data scientist, building models to make people click on ads. I published a few papers during my masters, and learned to write (more) properly. I hope that traversing the winding path of life made me a better teacher, so I that can bring together startup, industry, and academic experience to the courses I teach.

I told my students to email me if any part of my background interests them. A few students took me up on the offer. They asked, for example, about how to get an internship at Facebook. Maybe I'll share some advice here too, someday. Until then, email me with your questions about getting into grad school, getting started with machine learning, and applying for internships! It is much easier for me to share resources via email than in person.

The truth is, I never expected teaching to be a possible career option. Pursuing this career is more risky than one might realize. To add to the fun, I don't have a PhD and don't (yet) intend to get one. But what would life be if we don't take chances to do what we find meaningful?

So, if you know of universities that are hiring full time teaching staff in Computer Science, especially Machine Learning, please let me know.

Tuesday, November 8, 2016

The Fundamentals

What changed me from an amateur into a professional was getting a really firm grip on the fundamentals -- Toshiro Kageyama (7d Professional Go Player)

It was pretty much the only advise I remember my dad repeatedly give me, both before starting undergrad and before starting my masters just a few months ago: focus on the fundamentals. He may have said other things, but focus on the fundamentals is the only phrase that really stuck.

You might have heard of the story of Da Vinci and the egg that illustrates this concept:
It has been said that when Da Vinci first came to Verrocchio’s workshop, he was told to draw eggs. Day after day, Da Vinci was told to draw eggs all the time. One day, Da Vinci finally got tired of drawing eggs so that he came to his master Verrocchio for complaint. However, his master explained to him of profound significance “Drawing eggs is not a simple thing to do, even for the same egg, if you change the observation angle, the light will change as well, and you will find the different shape of it.” Da Vinci suddenly understood the purpose of his master. After then, Da Vinci accepted drawing eggs with an open mind which actually helped him built the foundation of further achievement.
The fundamentals can often be less interesting than whatever is new and shiny. It requires patience and honest self-assessment about how much one actually understands. It's always so tempting to "move fast and break things". But a firm grasp on the fundamentals is necessary to be able to intuit deep connections and do meaningful work.

There's a second part to the argument for focusing on the fundamentals. Popular research will move on, and that whatever is new and shiny now will cease to be important in a few years. The skills that will stay relevant for a long time will end up being the fundamentals, the things that won't change or go out of favour. This is even more crucial in a field like machine learning that moves lightning fast.

But what exactly are the fundamentals of machine learning? There are the obvious tools like linear algebra and multivariate calculus. There's regression and its generalizations, gradient descent and its second-order extensions, back-propagations and the like. What about all the types of neural networks with the many acronyms like CNN, RNN, LSTM, ...? At what point do we break away from the fundamentals and find ourselves in the arena of popular research, the kind of things that will cease to be important in a few years?

What's been smelling most like fundamentals in the last few months have been variational inference and variational autoencoders (VAE's). They come up everywhere in recent research, but VAE's were only introduced in 2014!

It's possible that something else will take its place in another few years. If so, is it still a fundamental? Perhaps machine learning is just such a young field that the fundamentals are still being built?

Thursday, October 20, 2016

Less boring

You've become boring. -- Jee (~mid 2015)

The truth is, I had become boring. I had become boring because happiness is boring and comfort is boring. The formula is really quite simple: a loving relationship, engaging and well-paid work, a nice place to live, good health and lack of serious responsibility. It's all one can ask for.

So, instead, here I am, back in school to study Machine Learning.

...and then when you graduate, you can get a job as a Senior Data Scientist! Oh wait... -- Matt (May 2016)

It might not make sense financially. It might not make sense as a career move. It might not even make me a "better" person.

All I can say is, it's making me less boring.

I might even have more to say now.

I might even say it here.

Maybe.

Wednesday, February 24, 2016

Education is entertainment (and vice versa)

The Lampsilis mussel is an interesting creature: it imitates another fish in order to lure a bass. You really have to see a video to understand how amazing it is:


Did you learn something new just now? Would you consider this video "educational"? Can you imagine one situation where knowing about the Lampsilis mussel would somehow help you in life?

"That's a very high standard", you may say. Quite right. A lot of things that we learn would be considered "educational" but would otherwise not be useful on a day-to-day life. Take calculus, or introductory cell biology, or the difference between a catabolic and an anabolic reaction. Even if they are not referenced daily, they may be very interesting and deeply satisfying to master.

But wait, "interesting"? You mean, "interesting" the way some will find learning Middle Earth geography in Elvish would be "interesting"? "Deeply satisfying" like knowing the truth about Snape and Lily?

What's really the difference here?

"Maybe", you may interject. "It's just that some very smart people find very serious uses of calculus." Certainly more so than Harry Potter. But then what would you say about philosophy and art history? What about the Lampsilis mussels?

"Maybe you're forgetting something," you may say. "Information is just information. It's the context in which we consume it that determines whether it's educational or not. If you watched the video about the Lampsilis mussel while writing a paper on evolution, then it was educational. Otherwise, it was entertainment."

Yet even that's unsatisfying. Why would you be researching evolution? Why would you be taking that course? Why are you getting a Ph.D. in marine biology? Somewhere along those lines of questioning, an answer should pop up that sounds awfully like "because I find it very interesting".

And so, education is entertainment.

"Wow, hold your horses," another person (who is not you) might interject. "I'm studying because I have to graduate to get a job."

Well Bob (because those people are always named Bob), it seems like society doesn't need you right now. Not in a bad way -- you're still very special, just that this particular moment, there are enough people producing enough stuff to sustain all of our lives. There's a cognitive surplus, so to speak. It looks like forcing you to be entertained educated is one way to "deal" with this surplus.

And so, education is still entertainment.

Let's not talk about the Lampsilis mussel. Let's talk about sports. Has knowing about the outcomes of your local sports team somehow helped you in life? I know it had in mine -- from the perspective of building rapport. Then again, you could really extend that to any piece of information! Watching Star Wars has been useful -- very educational about, um, culture, yes -- culture. And binging on Game of Thrones? That, uh, brought me closer with, uh, people.

You learn things about human nature from film. You learn teamwork in games. I learned about the different types of alcohol from a text-based MMORPG. Friend of mine swore that he used to learn much more from anime than anything else.

And so, entertainment is education.