I’ve been in a fair few interviews in my time, that’s for sure. Some good, some bad, some horrific. Sometimes that horrific situation has been my fault, and sometimes it’s been the interviewing party.
Anyway, over the years, I’ve learned what I like and what I dislike in a technical interview, and I thought I’d share some these thoughts. As a jobbing engineer, I like to think I’m kind of everyman. I have some math – I understand trig and all the rest of it, but I don’t spend that much time thinking about Big O notation (for a good reason – I wait for the profiler to tell me I’m spending too much time in a function before I worry about it’s speed) or stuff like that.
This is a long post – a big wall of text -, but hopefully it’s worth it.
Obviously some of this stuff is wish list and in some situations isn’t possible – preparation, for example, can only happen if you are forewarned. But most people are, and still don’t do it and it just smacks of disrespect.
General Observations
1) Prepare. Don’t walk into an interview without having looked at the resume of the person, and at least looked at the websites of places he’s worked. If there are things to see – a blog, a set of lecture notes, whatever, then look at them. From the interviewee position, there is nothing worse than the guy you need to impress rolling into the interview and desperately looking at a resume that it’s obvious he’s never even seen before.
Far too many interviews are basically just the interviewer reading a resume and groping for questions to ask about each of the jobs you’ve had. It smacks of disrespect, and that’s a killer in terms of the person trying to get the job actually wanting it.
2) Have some questions prepared. And I don’t mean of the technical variety, I mean of “Who are you?” variety. You are likely to be working in close proximity to this person – and may even be working with them directly – and as such, you need to understand if this person is someone you can work with. Whether they are likely to arrive stoned, or drunk. Whether they have flash points regarding C++ Code notation layout. Whether they are going to drone on and one about the band Gwar etc.
Basically, everyone need to get a handle on cultural fit for the individual, way beyond the root question of “Can they do the job?”
Some companies break interviews down into separating out different parts – person A does technical acumen, person B does cultural fit. This is not a good approach. What it does is concentrate all of the potential elements that go towards a “yes” answer in one persons hands. If one person doesn’t like that the interviewee doesn’t know dot product, he’s now in a position to completely torpedo that application, regardless of what everyone else thinks.
Now obviously not everyone can do technical acumen – you need to be an engineer to judge that – but everyone can ask some questions about what the candidate is into, what they like to do on their own time, what their favorite repository is, biggest mistake they’ve ever made and why they thought it was, and what they’d do about it now etc. Questions that talk to what that candidate is – or at least thinks they are – rather than just “Can you tell me the code for Dot Product?”
NOTE: Be careful that your personal questions aren’t too personal. Run them by other people (or HR) first, just to be sure you aren’t contravening a law when you ask about the candidates family (in actual fact, that’s a no no anyway).
3) Do some of your technical questioning before the person comes into the office. By the time they get to the office – and are taking up resources time to sit and talk to them – you should already be fairly confident they can do the job from a technical point of view. Make them do a test – and don’t ever do that test while they are in the office. That’s just wasting everyone’s time. The fact is, you’ll probably get one day to determine if this person is going to fit into your team. One day. So the last thing you want to do is waste it, putting that individual in a room by themselves, while they knaw on a pencil trying to remember the syntax for using STL.
It’s also worth pointing out that tests like this are entirely artificial. The fact is, if you employ them, they are not going to be working like this day in day out. They aren’t going to be standing and writing code on a grease board in front of three board mid level engineers. They’ll be looking stuff up on the internet, trying stuff, refactoring it and so on. The only thing that on site tests really tell you about a candidate is whether or not they do well in high stress situations where you’ve got a bunch of other engineers standing watching you and judging you, and where you can’t look up stuff. If that’s how your company operates as a norm, then you’ve probably got more problems than worrying about how your interviews go.
4) Your technical questions are not a competition, nor are they an excuse to hand out “Find me a pebble” questions.
I’ve had situations where I’ve walked into the interview and it’s become instantly obvious that the person on the other side of the table has their own self worth issues and needs to prove some degree of mastery over me instantly in order that they dominate in some way. I literally had one guy correct the very first thing I said – where I was being generic – and I just looked at him. In that instant I knew I wouldn’t take the job even if I was offered it, if it meant working with this guy. Any human who feels the need to establish a pecking order in the first sentence is not someone I want to work for/with.
Also, when it comes to technical questions, a lot of engineers either have their own pet questions to ask – where in reality what they want to know is “do you think like me?” They have a question that can have many answers and they have a favorite answer and if you don’t get it, well, you’ve failed. The reality is you haven’t read their mind or come to the same conclusion they would have, ergo you aren’t as good as them.
The bottom line is, I probably know as many really-hard-to-answer questions as you do, and I can ask them back, because my time is valuable too. If all you want to do is establish a pecking order, well, I can do that too, and make you look as stupid as you are set on making me look. Then what?
For many engineers – particular those with fragile egos – interviews are about proving they are better than the candidate (mainly because a lot have a worry that they are not, even though they’ve already got the job) – even more about discovering if the candidate is good in his own right.
The fact is, even if he gives you a different solution, that’s ok too. Explore that. Don’t get frustrated that he hasn’t come to the solution you would. Thinking differently is ok, and will almost certainly get you out of a jam some day, when you can’t see a solution and someone else can.
Also, be sure that your questions can be worked out by the candidate, and don’t require an either “you know it or you don’t” question. Examples of that are asking someone to write an STL vector prototype. If you don’t remember that off the top of your head, then you’ve failed – because there’s no way to work it out. This is why some of the questions from Microsoft in the 90′s were such a hit – asking “How many ping pong balls fit in a 747?” for example, or “How many gas stations are there in Seattle?” – these are all questions where it’s more about working out ways to get an answer than it is about what the answer actually is.
The more a question relies on some esoteric knowledge, the less valuable it is, because what it’s really testing is “Has this candidate come across this before” and not “Can this candidate work it out if he’s not come across this before”, which is what you should want to know.
And while we are on that subject, the more obscure the knowledge you ask for is, the more you as an interviewer look like a prick for asking it – this goes along with with point 1), where I’ve seen a lot of engineers, struggling to retain coder dominance in the interview, resort to more and more weird and specific edge case questions in a way of tripping up an interviewee. And lets not kid ourselves, this is exactly what it is – it’s not “exploring the depth of this candidates knowledge” – it’s a way of the interviewing coder feeling better about himself because he knows some obscure fact that the interviewee does not.
Also, asking for equations sucks. It just does. No one writes equations in code. If you need to know if someone knows the code for line intersection, don’t ask for the equation. Ask them to write the code.
Which leads me on to…
5) Ask for what you want. Don’t be clever and ask a leading question and hope to gently push the individual into what you really want. I had a graphics engineer once ask me for an early box discard algorithm – which is 4 if-then’s and that’s it; can’t get much quicker than that – but he wanted something else. In the end, what he wanted was a binary search algorithm. It took half an hour of me trying to figure out what it was he actually wanted. When I finally worked out what it was, I gave him his binary search algorithm, and then said “Look, if that’s what you wanted, then ask me for it. Don’t make me spend half an hour wondering what it is you want.”
It was done purely for the sake of the engineer in question doing it, and I have to say, it pissed me off no end.
Convoluted and obscure questions just waste everyone’s time and you may think you are clever for asking it; more than likely you are making the candidate roll his eyes internally.
6) Sometimes, knowing what the use you put something to is more important that knowing how it is generated. For example, Dot Product. If an engineer can describe what a dot product is (e.g., it’s the difference of angle between two vectors, described as from -1 to 1.) and give you four or five usage cases for it, that is way more important than being able to write the code from memory.
Some engineers put great stock by remembering formula by rote, and while there is some value to this, again, this is not normal day to day coding situations. If I don’t remember the code for a dot product, I go look it up. I spend some time comprehending what it is, write the code in my way, and forget about it until I need to do it again.
There is an argument that if you remember the source code, you ‘know’ the code, but that’s not actually real. Most coders who remember the source code couldn’t tell you what happens if you feed a dot product two vectors that are *not* normalized. If you can’t do that, then you don’t understand the root math.
Questions like this tend to show up a lot because they are easy to validate – either you know it or you don’t. Questions that really plumb how someone works are far harder – and require a lot more effort on the part of the interviewer – to set up.
The answer “I’d look that up” IS a valid response, even if you are mortally offended that someone doesn’t know the exact same things you do – or worse still, doesn’t associate the same value to it that you do.
7) Understand that your value system – of what’s important or not – is not going to be the same for everyone. Value systems are generated via experience, and by definition, everyone’s experiences are different.
Therefore, quite a lot of people will have very different reactions to what they see based on their experience – and sometimes the situation they are in too. Reacting to finding an incomplete API is very different when you have a deadline to get something done by, and find it’s blocked by this, than it is if you are just perusing the API docs and find something not quite right in it.
The problem with engineers is that they tend to be very black and white in what they believe. Way X is the “one true path” and everything else is just not. Any one who comes at it from a different point of view is therefore wrong, and to be argued with, in order that they see the error of their ways. Missing the point entirely that a) no one is likely to be swayed by your superior knowledge (just pissed off instead) and b) sometimes there is more than one way to skin a cat, and that’s – god forbid – ok.
8) Understand the guy in front of you is sweating, and wondering what it is that you want to hear, and will attempt to give it to you, regardless of the reality of their beliefs.
It’s an unfortunate statement that most people who are interviewing need the job, and therefore will contort themselves to be anything they can discern that you need. If you ask them leading questions, they’ll pick up on it and then try and mold the answers to whatever it is that they think you want.
E.g. “How do you feel about unit tests?” – that’s a leading question, and the right answer here is obviously “They are great, I use them all the time“. And that’s almost certainly the answer you will get, regardless of whether the candidate hates them with a passion or not.
You just aren’t going to get the truth by asking questions like this – that’s the reality here.
It’s taken me 25 years of interviewing to understand that I need to be myself in an interview, and not try and be what the interviewer necessarily wants. If I’m hired, then I’m going to end up being me anyway, and if that’s not what they want or can deal with, then I’m better off knowing that immediately rather than uprooting my family and moving across the world to take this job, and then find out that I’m really not a fit.
I don’t want to give anyone a false impression of me, so I have to be myself. If it’s not a fit then a) either I’m really not a fit or b) the people interviewing me really don’t have a good understanding of what they want. Either way, I’m probably not going to be happy there, and neither are they, so bullet dodged by both parties.
But most people aren’t like that, unfortunately (at least, *I* think it’s unfortunate). They want / need the job too much, so they’ll twist themselves into whatever pretzel they think the interviewer wants and then get the job and have a disastrous time because they are now going to revert to type. Life is too short for that I’m afraid – but most people haven’t gotten to the point of having it happen to them repeatedly, and haven’t learned that lesson.
9) encourage them to ask questions.
And what I mean by that is not “Do you do crunch here?” The point I am trying to make is that they have as much right to test your technical acumen as you have to test yours.
A lot of engineers don’t get that an interview is a two way street. They think “Hey, you are trying to get a job here. You have to impress me and not the other way around. I’m the one asking the questions.” Well, I’ve taken two jobs in my life where I’ve not done a technical due diligence on whether the people I am going to work with can actually get the job done, and in one case, it resulted in me getting stuck in Arizona for five years (and counting). Sure, I want to ask you about your technical abilities, what your development process is, how you QA stuff, what some of the biggest hurdles you’ve hit have been. If only to get an idea if some of the experience I have will be helpful.
Some engineers have their noses put out of joint when the candidate starts asking penetrating questions – this is an attitude that needs to be beaten out of people. They have as much right to know that you are going to work well with them as you do to know if they will work with you. The more that right is denied, the more you look bad as a potential employer.
Some thoughts on things that might be helpful.
I heard a story once about an engineer who, in his technical interview, sat down opposite the candidate, and then declared “I am a debugger. There is code that is crashing. It’s supposed to do X, but crashes when we do Y. What do you do?”
The interviewee then asked for more details, and the interviewer said “I’m a debugger. I can’t answer those questions. You can’t ask Visual Studio those questions and get answers – what would you do with Visual Studio?”
Eventually, the interviewee got the hang of it, and started asking things of the debugger that the debugger was able to answer – “What’s the stack trace?”, “What are the local variable values?”, “What does the source code look like?” and so on.
This is an excellent way of assessing how a candidate thinks, but while it’s great, it’s not without it’s problems too. The interviewer has to have all this stuff in memory and be able to reproduce it accurately, based on questions he cannot predict, and that’s difficult. One wrong answer and the candidate might not be able to solve the issue at all.
But, what it does tell you is exactly what the candidate would do – and it’s done in such a way that the candidate is more interested in what is going in in the interview than they are sweating out what they think the interviewer wants to hear.
One of the techniques I use now is to sit with a candidate, and describe a current problem I am having – what I know about the problem space, what the solution needs to provide and then say “Ok, what do we do about this? Lets work together on this.”
The idea is to mirror the experience he’ll actually have if the two of you collaborated on some design or implementation aspect, and get to see what he suggests. See if it’s off the wall, or what you were thinking or if he see’s issues you didn’t.
At root, you are attempting to see what happens in the real world of development at your studio, rather than the mostly artificial situation that an interview really is. How they work, how they think and how they collaborate.
Some great questions I’ve seen in my time include this gem – “Tell me, if I hired you, what would I most regret about that decision in six months time?” It’s a terribly leading question and really puts people on the spot because I guarantee you, they won’t have a prepared answer for that one.
The interesting aspect of that question is not what the actual answer is, it’s generally how long they take in answering it. Too long, and they are trying to find something that’s positive that they think you’ll accept “Oh, I’m too much of a perfectionist”. Or too little, and they are riddled with faults and are very quick to point that out.
Speaking of questions, the questions “Describe your greatest success” questions are interesting, but I’m not sure what they tell you about a candidate vis-a-viz the job in hand? “Describe your greatest failure” questions are more useful, particularly when followed up by the relevant “What would you do to avoid this in future” and “How would you fix that situation now?”
Something else I try and do is relax the candidate a bit. Most are pretty uptight – they need the job and want you to like / respect them. They aren’t at their best in this situation – almost no one is, so its your job to get them to relax.
At some point you need to say to the candidate “Look, there are no right or wrong answers here. There is no answer that is going to get you definitely hired or not hired. We are here to see how you tick and where your technical limits are. It’s not a problem that you have them – everyone does, and it’s all to do with experience. I’m just interested to know where they are. If you don’t know something, that’s ok – you aren’t being judged on that – I’m more interested in how you deal with something when you don’t understand it. It’s more a combination of everything. So relax, lets get some work done!”
Asking their opinion on something – asking why they did something, not in an accusing way but in way that asks them about their experience that brings out their conclusion – will grant you far greater insights into the candidate.
Also, ask them if they have questions and give them time to ask them.
Ask them if they want water or the bathroom. People do need to pee.
Last, remember, while you are giving this person a job and it means you’ll see them every day – which in itself is a big deal – for them it may well mean selling a house and moving a family across or continent, or even country. This is a big fucking deal for them. Please treat it as such.
The bottom line is, have some respect for the people who are sweating bullets in front of you, hoping to god they don’t say something you violently disagree with. Because they have no clue what you expect, what you want to hear or what is going to make you drop that black ball in the sack.
The Internet and Social Justice
So yeah, been an interesting few days. Well, “interesting” is not really the word. “Bloody scary” is probably closer to the mark.
I refer, of course, to the whole Social Justice Warrior scene between Zoe Quinn and her ex boyfriend.
To recap, her ex – Eron Gjoni – put up a blog where he basically accused her of sleeping around, while making sure he knew of her feelings on infidelity. This then got picked up by 4Chan and Reddit, who made it out to be more than it was – a simple case of loose morality and bad judgement on her part – and made it into “Game dev sleeps with journo to get coverage”. Which it wasn’t and never was, as Eron himself pointed out.
But then, the avalanche of social warriors, who are fearless behind their anon email addresses, start harassing Zoe in unbelievable ways. Those women who come to her defense also get attacked – one even having to leave her home after the death threats.
Now, lots of people have had a lot to say about this – mostly picking sides. Some are all about the fact that Zoe invited this on herself, behaving as she did. Some are all about the over reaction of the internet based social warrior, who appears to be consist of mostly white guys who are deeply misogynistic and have anger issues.
I have my own thoughts, and I’m going to get them out here.
Firstly, this is not about “Slut Shaming” as some people describe it. Lets deal with this. Firstly, the word Slut is bullshit. It means nothing. It’s a derogatory term for a woman who sleeps around. The word for that for men is “Stud”. It’s a double standard that is horseshit. Women are looked down on because they sleep around, while men are venerated for it. It’s just so apparent crap, yet it perpetuates itself. The least said about this the better. We can all just agree it’s shit and move on. So lets remove the word Slut from this.
When Eron did was “Hypocrisy shaming”. Zoes behavior was, at best, hypocritical. Lets be clear on that too. She made great capital about fidelity, then didn’t live up to those statements. That’s at best hypocritical and at worst, extremely damaging to your significant other – to the point where the relationship fails, as is the case here.
Now I don’t really want to harp on about morality and/or sleeping around, suffice to say, whatever a relationship is between two people, it’s up to them and no one else. Common western civilization dictates that a relationship between two people that is romantic tends to be monogamous. That’s the default state, as agreed by pretty much everyone out there. It’s what we are brought up with and defined as. Now that doesn’t make it right at all – just what you would reasonably expect of a relationship. Now if two people don’t want to do that, then it’s entirely their decision to alter those parameters and make it whatever they want it to be. They get to define what their relationship actually is, no one else, and more power to them. If they want to open up their relationship, then that’s fine.
But it does take two to tango in that regard. One person cannot do that and expect the other to just go along. Both have to know and agree, or it doesn’t work, as it hasn’t here.
I totally get how Eron feels about this – it’s very apparent in his blog posts. He’s upset, bewildered and probably very angry, and making this behavior public is probably his only recourse here.
I would imagine that he feels he is just making her behavior public, holding it up to the light so others can make their own judgements. Of course it’s rendered in such a way as that he controls the interpretation, but still, I would imagine that’s his mindset.
But.
The fact is, The internet has no middle ground. It just doesn’t. The reaction from anonymous commentators has to been to vilify Zoe, his Ex and to basically eviscerate anyone who disagrees with that analysis. Nasty emails, threats and so on.
What Zoe did was bad, but it in no way justified the backlash she is getting. It’s outrageous. It’s life threatening. It’ll destroy her in ways that she in no way deserves.
But there it is. The crux. The word “Deserve”.
What do we do here? As a society? Her behavior was crap, there’s no real getting around that. But proportionate to this response? Not even remotely. She’d have had to kill people to deserve this reaction. This is just angry boys, who have finally realized that life isn’t like an 80′s romantic comedy, and they aren’t going to get the prom queen to suddenly come to the realization that they are “the one” and are pissed off about it.
It’s made worse because it’s video game related, and by definition, lots of the audience is angry 12 year olds who have too much testosterone and have no outlets to let it out; we’ve done a great job as a civilization removing those, over the past few years, via concerned mothers. So your audience for this starts out being skewed, which just fuels the fire.
There are those who say “She doesn’t deserve to be even mentioned” – which is a valid view point; her behavior was to one person – her significant other – not the rest of us. Why should we be involved in this? It doesn’t effect us?
Others would say it does, because what she does in her personal life apparently spills out in to her professional one. Hypocrisy is hypocrisy, and as such, when it’s discovered, it should be brought out into public. Other wise the behavior goes on. This is the driving force for things like Watergate and Snowden.
The question is, what do we do when the behavior is bad, but the only punishment for it – the unleashing of the Social Justice Warriors is far worse than the crime?
That’s where we are right now. The internet, as it’s structured now, gives these cowards (and I use the word advisedly) a place to strike and hide. Society, as a whole, develops these kinds of people because parents are generally too busy to actually instil some social values in their kids (or, worse still, have a wry smile when it happens). This issue is not about to go away and regrettably, loose groupings of people who oppose it are far outnumbered by those perpetrating it. Or to put it another way, those who do it have already justified it in their minds. Missives like this are not going to change anyone’s core beliefs that it’s their right to vilify people.
So what is the solution? We shouldn’t be hiding this kind of behavior, but equally, the response is too outrageous to even contemplate?
My personal feeling is that what this particular situation has taught us is that we will HAVE to hide this kind of thing. Or at least, not make it public in the way it has been. The internet Beast is just a thing of evil, with a mind of it’s own, and it’s simply too aggressive and too random to be unleashed.
I don’t like it; I feel justice as much as the next person – but I don’t honestly see what alternatives there are.
Thoughts?