My Top Conference Attendance Tip

One of my PhD students is on his way to his first academic conference. Conferences are one of my favourite parts of research: I've met so many interesting people and started so many fun collaborations that way.

Just today I saw this great advice from Michael Ernst about attending conferences. His advice is so good, that I've only got one thing to add.

Here's my tip. When a big group students from the same institution go to the same conference, they'll often hang out together all the time: at meals, coffee breaks, etc. This isn't a great strategy on their part, but you can exploit this. Once you make friends with one person in the group, now you can go with your new friend to lunch with everyone else in the group. Voila, now you have a network!

This is a general principle of networking (horrible name, but incredibly important if done honestly and well): Cool people will help you to meet other cool people. And once you've been around long enough, you can return the favour!

Viva la voce

One of my favourite aspects of academia in the UK is the final oral examination for the PhD --- formally called a viva voce, which everyone seems to call a viva (VEYE-vah). The viva is an oral examination that typically consists of the student and two examiners, one from within the University (the internal examiner), and one from outwith the university (the external examiner).

Both examiners read the thesis carefully, and ask the student detailed questions. The traditional way to do this is that all three have a paper copy of the thesis, and the examiners go through the thesis page by page with their questions. Some questions are high level ("Why did you choose technique X rather than technique Y?") and some can be very detailed ("In your proof of Theorem 4.3 on page 176, I'm not sure that the third step is correct. What if the matrix A is non-singular?").

This discussion commonly takes 2-3 hours. Longer and shorter vivas are not unheard of, though if you ask me, a one-hour viva is a bit of a rip-off for the student, and a five-hour viva isn't kind, unless the length is caused by the student being exceptionally argumentative or loquacious.

At the end of the viva, the student is asked to wait outside, and the examiners decide whether the student should be awarded a PhD, and if so what corrections to the thesis are required. The most common outcome is a pass subject to minor corrections, which the student is allowed a few months to complete.

Essentially, in my experience a viva is a detailed technical discussion of the content of the thesis. Most students start out nervous, sometimes exceedingly so, but relax after a few questions as they realize that this is just a technical discussion of the sort that they have had many times before. That said, even if your research career is long, it is rare that a trusted colleague will provide you with several hours of detailed feedback on your work. To be part of a discussion like this, on either side of the table, is a privilege: most work is ignored, so any criticism is a compliment.

I am sure that the process is more tense if the examiners believe that the quality of the thesis is borderline --- fortunately, I haven't yet been asked to examine a thesis like that. If portions of your thesis have already been published in prestigious venues, and whether this is possible at all varies greatly across disciplines, then you can be fairly sure that your thesis is not near the borderline.

A colleague suggested to me once that a viva is like a negotiation. If your thesis represents a sufficient amount of research of acceptable quality (and if it was indeed you that wrote it), then you will pass. The negotiation is over which corrections will be required. Responsible examiners do not want to require additional experiments that will require months of work, if the thesis as submitted is of excellent quality. But they also don't want a thesis to be passed with gaping holes in its argumentation. The purpose of the discussion is to sort out which potential concerns are which, and your voice in this discussion matters --- you wrote the thesis, so you are the expert in the room.

Sometimes an examiner asks a question with an eye to a correction being required. Maybe you think, "yeah, that's a good point, I should add a paragraph on that" --- in this case, don't hesitate to say so. On the other hand, if providing a good answer to the examiner's question would require months of additional research, politely explain why, while also giving the best answer you can given what you do know. What you don't want to do is argue every point strongly, even when the examiners are clearly right... that is not good negotiation strategy.

A bit more about who attends the viva. The internal examiner is not the supervisor, in fact, they will not have been involved with the thesis research at all. It is not unusual for the internal examiner to be a bit of generalist with respect to the thesis topic, although when I have served as internal, I have usually been able to make out the thesis reasonably well. Presumably this is because the School of Informatics is large enough that there are many theses in machine learning and natural language processing that need to be examined.

The external examiner is chosen specifically for their expertise in the subject matter, and to serve as an external is generally seen as a minor indicator of prestige. A certain amount of deference is paid to the external in the culture of the process. Even so, I have the sense that part of the role of the internal is to be accountable to the University (for following correct procedures) and to the student and the supervisor (to make sure that the examination is fair to the student). I have read and heard horror stories of aggressive external examiners but never witnessed one; to the contrary, the examiners who I have witnessed have all gone out of their way to be kind to the student.

The role of the supervisor in the viva, I think that this may vary slightly across institutions. At Edinburgh, the supervisor is allowed to attend the viva, if the student permits, but not to participate in any way. In my experience as internal examiner, the supervisor has attended about half the time. One supervisor silently took notes to share with the student, which I think is quite a kind thing to do.

Finally, although the description is written for a US audience, British academia also fetaures the snake fight portion of your viva.

(Written in honour of my first PhD student graduating. Congratulations Yichuan!)

Taste in research, and the paradox of deciding what not to work on

A large part of taste in research is deciding what not to work on. You might choose not to apply method X, even though you don't really understand it, because it has a reputation for being fiddly and difficult to get right. You might choose not to work on topic Y because you think that even though there's a lot of people writing papers about it, its goals are too ambitious to ever be met. This extends all the way to entire fields of research. I could name a few popular fields within computer science — with active research communities, large amounts of external research funding, leading researchers with fancy prestigious awards — that I suspect are being investigated in entirely the wrong way, and that I personally think are currently pointless.

I could name them. Will I? No.

Why not? To protect my career? If I am honest, probably in part yes. But what I tell myself is different. The real answer, I think, is that my opinion of these areas is poorly informed. Because I think these areas are uninteresting, I haven't studied them carefully, and so I don't know how they've attempted to address my naive objections. It would be arrogant and professionally irresponsible to publicly denigrate the hard work of many people without having even bothered to read it.

This leads to a paradox. It's impossible by definition for me to become better informed about these areas, unless I decide to actually start researching them. In order to be fully confident that an area is uninteresting, you need to study it — and that study itself is part of doing research! But you can't do careful reading on every research area that seems bogus at first impression, because then you would do nothing else. Instead, you have to take intellectual shortcuts, and do the best you can with limited time to think. Those research areas that smell a bit off, you ignore them until either they die out, or a major success forces you to reevaluate. Part of taste in research is deciding what to study, and what to ignore.

This is the paradox of taste in research. Your decision of what not to work on is, by definition, always ill-informed.

Ubiquitous capture and the ideas file

Ubiquitous capture is a great term from Getting Things Done. Like the best ideas from GTD, it is simple, obvious in retrospect, but changes everything. Ubiquitous capture means: When you think of something, you should write it down, right away, in some place where you will check it later.

This is especially good for keeping track of ideas for new research projects. I tend to find ideas for new projects while I'm walking to work, when I'm sitting in a talk, or when I'm working intensely for a paper deadline. Hardly ever can I work on them right away, but I know that I will need them later. So, whenever I have an idea for a new project, I stop whatever I'm doing and write in down in my ideas list. If I have to stop in the street or pause a one-on-one meeting to pull out my phone, well, a benefit of being an academic is that you get to be eccentric.

I keep my ideas list in Evernote, but it doesn't matter what you use, as long as all your ideas are on one list.

Later, usually many months later, a student will ask me for suggestions for an undergraduate, master's, or PhD project. I go back to my ideas list and look. I also tag each idea "ug", "msc", or "phd", if I think it would work well for one of those degrees.

I also look back through the list periodically to pull out ones that are especially exciting. Every idea is exciting when you first have it; the ones that are still exciting a week later are the ones to keep.

Of course I use a similar system for blog ideas.

And can you teach me how to talk real slow?

A switch flipped in my head at the beginning of my lectures last spring. At that point I had lectured something like 5 full university courses and maybe something like 50 research seminars. I was an experienced speaker.

But I was fast.

You'llnoticethisifyouaskmeaboutsomethingresearchrelatedthatI'llstarttogetexcitedandtalkfaster. In my personal life, I'm much more laid back, but at work, I talk fast. It is what it is, I suppose, but it's not the best attribute for an effective lecturer.

And then last term something happened. I walked into class and started speaking twice as slow as I ordinarily did. I liked it. I felt that I still had the amount of energy that I should have, just... slower. I don't know what I did, so I couldn't tell you how to do it if you wanted to, but now I can turn on the slow mode whenever I want.

Actually, I just thought of a theory about what I might have been doing. In every sentence when you're speaking, there are few key words that you emphasize. When you get to those---and you should try to anticipate those words before you say them---exaggerate your emphasis and focus on slowing those words down. Then the other words in the sentence, and the length of your pauses, will follow. That might be how I learned to switch, maybe. Let me know if you try this and it works for you.

Another nice thing about speaking slower is that it gives me more time to plan my sentences. This reduces the number of times that I get halfway through the sentence, think of a better way to say the sentence, and start the whole thing over from the beginning---a bad habit of mine.

It's nice to know that you always have more to learn. This change sure messed up my lecture plans for that term---everything took longer than I expected---but overall a positive change, I think.

Future Work

It seems customary for computer science research papers to list directions for future work at the end. This custom is immensely strange. If your idea for future work is really good, the last thing you want to do is tell everyone about it. Literally the last thing: right after you've done the research and written it up! On other hand, if the idea for future work is bad, why do you want other people to see it?

My belief is that the "future work" discussions are not in fact lists of future work. In fact, it is perhaps safest if beginning students ignore these sections altogether. But they do serve a purpose, or rather, one of several:

1 Delimit the scope of current work. You have to stop somewhere, so listing an obvious idea for future work is a way of saying, "Yes, we know that this is an obvious extension, but we didn't have time for it, and its not as interesting as the stuff we did do." These are the research ideas that you want to stay far away from; if they were that interesting, the authors would have written that paper instead.

2 Stake an early claim. You've written a good, coherent paper, but there's another idea that's an obvious but still exciting follow on from what you did. It's a bad idea to put too much in one paper, so you mention the follow on to acknowledge that it's obvious, in case somebody else gets to the follow on first.

As an aside, if someone else does the follow on, don't feel bad. They'll be citing your paper prominently, which is very good both for your career, but more important intellectually: the point of you doing the work was for other people to use it. 

3 Convince readers that the work is useful. If you've built machinery (whether code or a proof technique) that you want other people to use, you might give some potential directions to encourage people to build on your work.

A simple trick to encourage lecture participation

It's the time of year when teaching is very much on my mind. In an essay about his teaching style, Michael Scott says something about encouraging student participation that stuck with me:

I’ve found that the very first class period sets the tone for the whole semester. If I don’t get students to participate on day one, they probably won’t participate at all, and the course ends up dreadfully dull. My first lecture in any class thus begins with a brainstorming exercise, in which I get as many different students as possible to voice a suggestion or opinion. 

Last year I tried something like this in my undergraduate machine learning class. I don't want to go into details in case I use it again, but it wasn't a brainstorming exercise (I couldn't think of one), but a simple quiz question that introduced part of the material. I had the students vote on the correct answer---and everyone voted wrong, because it was of course a trick question.

To my delight, I found that year's class asked many more questions than the one before, even though it was significantly larger. This may be due to random variation, or to the fact that I was better at teaching the course the second time, but it's enough that I'll keep trying it.

About to graduate with your PhD? One more tip.

A rite of passage for US PhD students is the title page of their dissertation. The way that faculty indicate their approval of the final dissertation is by signing the title page, and students are required to leave space on the title page for this purpose. It's up to the student to run around to all their committee members (mine had 5) and get them to sign. Holding the final title page, with all the signatures, this bland sheet of acid-free paper that signifies that your hard work has come to something... it's a heady feeling.

Often people go to a bookbinder to get bound copies made as gifts for their parents and PhD supervisors. I had a copy bound for myself as well (boy was that a mistake). So here's my tip: Keep a photocopy of your signed title page. Then, when you get your thesis bound, you can include the signed title page with all the bound copies. This looks much nicer than a title page with blank signature lines, which gives the faint impression that you're trying to pull something over on someone.

Congratulations!

Principles of Research Code

Ali Eslami has just writen a terrific page on organizing your experimental code and output. I pretty much agree with everything he says. I've thought quite a bit about this and would like to add some background.
Programming for research is very different than programming for industry. There are several reasons for this, which I will call Principles of Research Code. These principles underly all of the advice in Ali's post and in this post. These principles are:

1. As a researcher, your product is not code. Your product is knowledge. Most of your research code you will completely forget once your paper is done.
2. Unless you hit it big. If your paper takes off, and lots of people read it, then people will start asking you for a copy of your code. You should give it to them, and best to be prepared for this in advance.
3. You need to be able to trust your results. You want to do enough testing that you do not, e.g., find a bug in your baselines after you publish. A small amount of paranoia comes in handy.
4. You need a custom set of tools. Do not be afraid to write infrastructure and scripts to help you run new experiments quickly. But don't go overboard with this.
5. Reproducability. Ideally, your system should be set up so that five years from now, when someone asks you about Figure 3, you can immediately find the command line, experimental parameters, and code that you used to generate it.

Principle 1 implies that the primary thing that you need to optimise for in research code is your own time. You want to generate as much knowledge as possible as quickly as possible. Sometimes being able to write fast code gives you a competitive advantage in research, because you can run on larger problems. But don't spend time optimising unless you're in a situation like this.
Also, I have some more practical suggestions to augment what Ali has said. These are

1. Version control: Ali doesn't mention this, probably because it is second nature to him, but you need to keep all of your experimental code under version control. To not do this is courting disaster. Good version control systems include SVN, git, or Mercurial, etc. I now use Mercurial, but it doesn't really matter what you use. Always commit all of your code before you run an experiment. This way you can reproduce your experimental results by checking out the version of your code form the time that you ran an experiment.
2. Random seeds: Definitely take Ali's advice to take the random seed as a parameter to your methods. Usually what I do is pick a large number of random seeds, save them to disk, and use them over and over again. Otherwise debugging is a nightmare.
3. Parallel option sweeps: It takes some effort to get set up on a cluster like ECDF, but if you invest this, you get some nice benefits like the ability to run a parameter sweep in parallel.
4. Directory trees: It is good to have your working directory in a different part of the directory space from your code, because then you don't get annoying messages from your version control system asking you why you haven't committed your experimental results. So I end up with a directory structure like
   

   ~/hg/projects/loopy_crf/code/synth_experiment.py
       ~/results/loopy_crf/synth_experiment/dimensions_20_iterations_300
   

   Notice how I match the directory names to help me remember what script generated the results.
5. Figures list. The day after I submit a paper, I add enough information to my notebook to meet Principle 5. That is, for every figure in the paper, I make a note of which output directory and which data file contains the results that made that figure. Then for those output directories, I make sure to have a note of which script and options generated those results.
6. Data preprocessing. Lots of times we have some complicated steps to do data cleaning, feature extraction, etc. It's good to save these intermediate results to disk. It's also good to use a text format rather than binary, so that you can do a quick visual check for problems. One tip that I use to make sure I keep track of what data cleaning I do is to use Makefiles to run the data cleaning step. I have a different Makefile target for each intermediate result, which gives me instant documentation.

If you want to read even more about this, I gave a guest lecture last year on a similar topic (slides, podcast).