Rather than continuing to talk about how hardcore developers should have hardcore math fu, I’d like to shift the focus a little to softer skills (although, as I said, I will come back to math stuff often). I believe that technical people should place even more of an emphasis on soft/people skills than non-techies (I mentioned this in one of the comments to my last post). I will expand on my reasons for this in a later post. In the meantime let’s just take it as a given and look at the value of these skills in a technical interview situation.

The thing with a programming interview, or any interview for that matter, is the fact that it is more of a conversation rather than a grilling. The company, and by extension the interviewer, has just as much invested in the process as the interviewee and therefore want to get the most information from the time they spent talking to you as they possibly can. Of course, in a programming interview the main objective is to assess your skills, but people rarely make decisions regarding other people (e.g. hiring decisions) based on hard factual data:

Only 70% of your friends rate you as a ‘nice guy’ and therefore we will be unable to pursue a friendship with you at this time.

Missing a semicolon on the second line of the last question, clearly puts you below the historical 95-th percentile of all candidates, which unfortunately means we’re unable to hire you.

It just doesn’t happen – people mostly go by how they ‘feel’ about another person based on what is important to them. They may try to get an objective picture of your skills, but no matter how well you think you did, if they don’t  feel like they want to work with you themselves (i.e. they don’t really like you), you will not get a positive report, at best the interviewer will be ambivalent:

He clearly knew his stuff but was a bit of a cold fish. I guess he was ok overall.

Not really the best outcome, ambivalence usually means – no hire.

This does not at all mean that you can bluff your way through a technical interview just by creating ‘good vibes’. Most technical people value technical skills highly and if you don’t measure up in that regard you won’t succeed. What it does mean, is that you can be in the middle of the pack as far as pure technical skill is concerned, but still come out as the preferred candidate at the end of your programming interview. Some developers (i.e. highly technical/analytical people) may not like it, but that’s just the way it is, dealing with people is not an exact science.

The Objective Of A Technical Interview

A regular interview is all about psychology – can you leave the interviewer with a good impression, can you ‘click’ with the person in the short time that you have. A technical interview should be all about technical skill, but people are involved, which means regardless of what you might like, it is still as much about psychology as it is about technology. The objective is to answer the questions to the best of your ability, while at the same time giving the interviewer as much positive clues about yourself as you can, to allow them to form a good overall impression. The impression is just as important as the answers you give. To that end here is the process I would use when answering a programming interview question which will allow you to not only give yourself the best chance of answering the question correctly, but should also give the interviewer a decent feel for who you are and how you work. If you disagree with any of this, or have other tips, do leave a comment.

The Process

The majority of technical interviews onsite, consist of answering a programming question on the whiteboard. I don’t like it either, but we work with what we have, so this process is geared towards that.

1. When you’re asked a question, don’t try and jump straight into writing code unless it is truly trivial. It’s not a race, sit and think about the question for a little bit. This shows the interviewer that you’re not a “cowboy” and don’t go off half-cocked to try and solve a problem you don’t yet understand. To help you understand the problem…
2. Use the whiteboard. A picture is worth 1000 words so use the whiteboard to draw the problem as you understand it. At this point you should be talking to the interviewer to make sure you both have the same view of the problem; the picture will help with this. This also gives the interviewer a chance to clarify the question if necessary. From your point of view, you’re talking and eliciting requirements, the value of this should be self-evident. Don’t fill up the whole whiteboard with pictures, you don’t want to have to wipe your diagram as it may be useful later on for debugging purposes.
3. Don’t be afraid to start over and keep talking. You should be chatting with your interviewer throughout, to keep them engaged and to allow them to understand your thought process. As you’re coming up with a solution you may start writing it up, but if you find that you’ve gone down the wrong path, or come up with a better solution, don’t be afraid to start again. Keep talking, tell your interviewer what you’re going to do – this gives them a chance to point you in the right direction if they think it necessary, or stop you before you waste too much time (if you’re completely off base). All of this should not only demonstrate the fact that you have decent coding skills, but show that you’re a good communicator, can reason logically and can recognise when you’re wrong (and take the appropriate steps). All of it helps the interviewer form a positive overall image of you. If you just stand there silently writing or thinking, you get none of those benefits. Infact you do the silence thing for long enough and it can get a little awkward, which is the last thing you want.
4. Take your time, like I said, it’s not a race. If you’re taking too long the interviewer will let you know, otherwise assume you have all the time you need (this should also help with stress if you’re prone to panicking during interviews). As you’re thinking, don’t forget to do some of it out loud, I can’t stress the importance of walking the interviewer through your thought process enough. When you’re actually writing the code, take the time to consider things like boundary conditions. Many technical interviewers are extremely pedantic about stuff like null checks, I personally wouldn’t be, but it takes little time and better safe than sorry.

5. Don’t be afraid to ask questions or even ask for hints if you’re stuck. The interviewer will often have some prepared anyway and will be happy to give them to you (i.e. you won’t lose too many ‘points’). If you don’t remember specific API details, let the interviewer know, they will either tell you or let you know that it is not important. This demonstrates that you know what you’re talking about while at the same time don’t get hung up on the details.

   If you’re stuck (but think you should know the answer), don’t panic. Stressing-out is one of the worst things you can do in any interview, it just isn’t the best image to project. Iterate over steps 2-5 again. It will probably be obvious that you’re under a bit of pressure but equally obvious that you’re taking it well – not a bad thing for an interviewer to know about you. It also shows that you don’t give up easily :).

6. When you’re finished, you’re not really finished. Remember what I said about interviewers being pedantic about null checks and boundary conditions, now is the time to make sure you don’t get caught out by this. Walk through your code and find any potential bugs, use the examples you drew in step 2. This demonstrates thoroughness and – chances are – you will find a little bug or two (which even the interviewer may have missed) which looks good. If you’re short on time, at least tell the interviewer that you would have liked to walk through the code.

7. Demonstrate that you know what you’re on about and didn’t just stumble on to the solution by accident. You can do this by telling the interviewer a little more about your code, consider things like, strengths, weaknesses, running time (big Oh). You may also mentioned other potential ways to solve the problem and any improvements you would make if you had more time. Chances are the interviewer was going to ask about this stuff anyway, but preempting displays professionalism (you’re aware of the issues and what is required) and the ability to think about the problem at more than just a superficial level.

Some Extra Tips

• Don’t write too small on the whiteboard, that can be really annoying (too large is usually not an issue, but watch for that too). If you’re writing is of a decent size and you’ve run out of whiteboard while still being nowhere near the end of the solution, you’re probably missing something, go back to step 1.
• If you’re into TDD it may help to think about the problem in a test first fashion. What would you test in order to solve – and go from there, it may help get you into your normal flow.
• Books like How Would You Move Mount Fuji? may be a bit of a cliche, but are worth reading for the insights into the mind of an interviewer (and it certainly won’t hurt to be prepared if someone ever does ask you THOSE questions) and how to leave them feeling positive towards you. Another good one is Programming Interviews Exposed for more technically-minded questions/tips.
• If you don’t know the answer and none of the hints help (in short, if you have no idea), don’t try to bluff it. Just admit it and move on, you can ask the interviewer to explain it later if you have time. As far as I am concerned, it is unfortunate that you couldn’t answer the question, but you get points for being man enough to admit it, after all, noone can know everything. It is obviously not as good as solving it, but is better than nothing and a damn sight better than trying to bluff it, the interviewer is not an idiot (and even if they are, pretend like they’re not) they’ll see through it and that’s an automatic fail.

This is how I would go about answering a coding interview question these days. Looking at it from the other side, if I was the interviewer and someone went through this kind of process I would rate them pretty highly (unless they couldn’t actually answer ANY of the questions :)). Doing all this allows you to demonstrate your coding skills, shows that you’re a good communicator, have decent people skills, work well under pressure are thorough and generally personable. Compare that with – “He was ok. He solved all the questions.” – big difference.

Images by urban don, pvera and Sugarmonster

A little while ago I started thinking about math. You see, I’ve been writing software for quite a few years now and to be totally honest, I haven’t yet found a need for math in my work. There has been plenty of new stuff I’ve had to learn/master, languages, frameworks, tools, processes, communication skills and library upon library of stuff to do just about anything you can think of; math hasn’t been useful for any of it. Of course this is not surprising, the vast majority of the work I’ve been doing has been CRUD in one form or another, that’s the vast majority of the work most developers do in these interweb times of ours. You do consulting – you mostly build websites, you work for a large corporates – mostly build websites, you freelance – you mostly build websites. I am well aware that I am generalising quite a bit, but do bear with me, I am going somewhere.

Eventually you get a little tired of it, as I did. Don’t get me wrong it can be fun and challenging work, providing opportunities to solve problems and interact with interesting people – I am happy to do it during work hours. But the thought of building yet more websites in my personal time has somewhat lost its luster – you begin to look for something more interesting/cool/fun, as – once again – I did. Some people gravitate to front-end technologies and graphical things – visual feedback is seductive – I was not one of them (I love a nice front-end as much as the next guy, but it doesn’t really excite me), which is why, when I was confronted with some search-related problems I decided to dig a little further. And this brings me back to the start of this story because as soon as I grabbed the first metaphorical shovel-full of search, I ran smack-bang into some math and realized exactly just how far my skills have deteriorated. Unlike riding a bike – you certainly do forget (although I haven’t ridden a bike in years so maybe you forget that too :)).

Broadening Horizons

Learning a little bit about search exposed me to all sorts of interesting software-y and computer science-y related things/problems (machine learning, natural language processing, algorithm analysis etc.) and now everywhere I turn I see math and so feel my lack of skills all the more keenly. I’ve come to the realization that you need a decent level of math skill if you want to do cool and interesting things with computers. Here are some more in addition to the ones I already mentioned – cryptography, games AI, compression, genetic algorithms, 3d graphics etc. You need math to understand the theory behind these fields which you can then apply if you want to write those libraries and tools that I was talking about – rather than just use them (_be a producer rather than just a consumer – to borrow an OS metaphor :)_). And even if you don’t want to write any libraries, it makes for a much more satisfying time building software, when you really understand what makes things tick, rather than just plugging them in and hoping they do whatever the hell they’re supposed to.

The majority of developers will tell you that they’ve never needed math for their work (like I did a couple of paragraphs above :)), but after musing on it for a while, I had a though. What we might have here is a reverse Maslow’s hammer problem. You know the one – when you have a hammer, everything looks like a nail. It is a metaphor for using a favourite tool even when it may not be best for the job at hand. Math is our hammer in reverse. We know the hammer exists but don’t quite know how to use it, so even when we meet a problem where our hammer would be the perfect tool, we never give it serious consideration. The screwdriver was good enough for my granddaddy, it was good enough for my daddy and it is good enough for me, who needs a hammer anyway? The trick with math is – people are afraid of it – even most programmers, you’d think we wouldn’t be, but we are. So, we turn our words into a self-fulfilling prophecy. It’s not that I don’t need math for my work it’s just that I don’t really know it and even if I do, I don’t know how to apply it. So I get by without it and when you make-do without something for long enough, after a while you don’t even notice it’s missing and so need it even less – self-fulfilling prophecy.

Here is some food for thought about something close to all our hearts – learning new skills. As a developer in the corporate world, you strive to be a generalizing specialist (_read this book if you don’t know what I am talking about_). You try to be decent at most things and really good at some. But what do you specialize in? Normally people choose a framework or two and a programming language and go with that, which is fine and worthwhile. But consider the fact that frameworks and to a lesser extent languages have a limited shelf life. If you’re building a career on being a Hibernate, Rails or Struts expert (the struts guys should really be getting worried now :)), you will have to rinse and repeat all over again in a few years when new frameworks come along to supersede the current flavour of the month. So is it really the best investment of your time – maybe, but then again maybe not. Math, on the other hand is not going away any time soon. Everything we do in our field is built upon solid mathematical principles at its root (algorithms and data structures being a case in point), so time spent keeping up your math skills is arguably never wasted. And it, once again, comes down to really understanding something rather than just using it by rote – math can help you understand everything you do more deeply, when it comes to computers. Infact, as Steve Yegge said, what we do as programmers is so much like math we don’t even realise it.

What/Who Makes A Difference

You don’t believe me, then consider this. Most of the people who are almost universally respected  in our field as great programmers are also great mathematicians. I am talking people like Donald Knuth, Edsger W. Dijkstra, Noam Chomsky, Peter Norvig. But then again these guys weren’t really developers, they were computer scientists, so it doesn’t really count right? I guess, but then again, maybe we shouldn’t really talk until our output in pure lines of code even begins to approach 10% of what these people have produced. Of course, you can be successful and famous without being a boffin, everyone has heard of Gavin King or DHH. That’s kinda true (although it’s an arguable point whether or not many people have heard of Gavin or DHH outside their respective niches), but “heard of” and universally respected are different things, about as different as creating a framework and significantly advancing the sum-total of human knowledge in your field (don’t get me wrong, I respect Gavin And David, they’ve done a hell of a lot more than I have, but that doesn’t make what I said any less of a fact). How is all of this relevant? I dunno, it probably isn’t, but I thought I’d throw it in there anyway since we’re being introspective and all.

The world is getting filled up with data, there is more and more of it every day and whereas before we had the luxury of working with relatively small sets of it, these days the software we write must operate efficiently with enormous data sets. This is increasingly true even in the corporate world. What this means is that you will be less and less likely to be able to just “kick things off” to see how they run, because with the amount of data you’ll be dealing with it will just grind to a halt unless you’re smart about it. My prediction is that algorithm analysis will become increasingly important for the lay-programmer, not that it wasn’t before, but it will become more so. And what do you need to be a decent algorist – you guessed it, some math skills.

So, what about me? Well, I’ve decided to build up/revive my math skills a little bit at a time, there are still plenty of books to read and code to write, but I will try to devote a little bit of my time to math at least once in a while, because like exercise, a little bit once in a while, is better than nothing (_to quote Steve Yegge yet again_). Of course I have a bit of an ace up my sleeve when it comes to math, which is good for me, but luckily with this blog, we might all benefit (I know you’re curious, I’ll tell you about it soon :)).

Where Do You See Yourself In 5 Years

So, is all this math gonna be good for anything? It’s hard to say in advance, I am pretty happy with where I am at right now and so might you be, but it’s all about potential. End of the day, if you’re a developer in the corporate world you don’t really need any math. If you’re happy to go your entire career doing enterprise CRUD apps during work hours and paragliding or wakeboarding (or whatever trendy ‘sport’ the geeky in-crowd is into these days) during your off time then by all means, invest some more time into Spring or Hibernate or Visual Studio or whatever. It will not really limit your potential in that particular niche; you can become extremely valuable – even sought after. But if you strive for diversity in your career and want to have the ability to try your hand at almost any activity that involves code, from information retrieval to Linux kernel hacking. In short if you want to be a perfect mix of developer, programmer and computer scientist, you have to make sure you math skills are up to scratch (and hell, you can still go wakeboarding if you really want :)). Long story short, if you grok math, there are no doors that are closed to you in the software development field, if you don’t – it’s going to be all CRUD (pun intended)!

Images by Beni Ishaque Luthor, Scott Laird and arturodonate

There comes a time in every developers life where they need a data file for testing purposes and there are none handy. Rather than searching around for a file that fits your needs, the easiest thing to do is simply to generate one. There are a number of reasons why you might want to generate a data file. For example, recently we needed to test the file upload functionality of a little application we were writing at work, using a whole range of files of different sizes (from <1Mb up to >100Mb). Rather than hunt around for files that would fit the bill, it was a lot easier to just generate some. Another reason might be when you need to test some functionality (e.g. algorithm) to see how it would handle very large sets of data. Since you normally don’t have files that are 1Gb or more in size just lying around, generating some is probably a good way to go.

Fortunately the Linux command line has all the tools we need to quickly and easily generate any kind of data file that we require (I am of course assuming that as a self-respecting developer you’re using or at least have access to a Linux system :)). Let us examine some of the options.

Firstly, to get the obvious out of the way. Solaris has a command called mkfile which will allow you to generate a file of a particular size, but we don’t have this command on Linux (or at the very least I don’t have it on Ubuntu), so I’ll leave it at that. If you’re on Solaris feel free to investigate.

When You Don’t Care At All About The Contents Of The File

You just want a file of a particular size, and don’t really care what’s in it or how many lines it contains – use /dev/zero. This is a special file on Linux that provides a null character every time you try to read from it. This means we can use it along with the dd command to quickly generate a file of any size.

dd if=/dev/zero of=file.txt count=1024 bs=1024

This command will create a file of size count*bs bytes, which in the above case will be 1Mb. This file will not contain any lines i.e.:

alan@alan-ubuntu-vm:~/tmp$ dd if=/dev/zero of=file.txt count=1024 bs=1024
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.00510162 s, 206 MB/s
alan@alan-ubuntu-vm:~/tmp$ ls -al file.txt
-rw-r--r-- 1 alan alan 1048576 2010-03-21 22:25 file.txt
alan@alan-ubuntu-vm:~/tmp$ wc -l file.txt
0 file.txt

The advantages of this approach are as follows:

• it is blazingly fast taking around 1 second to generate a 1Gb file (dd if=/dev/zero of=file.txt count=1024 bs=1048576 where 1048576 bytes = 1Mb)
• it will create a file of exactly the size that you specified

The disadvantage is the fact that the file will only contain null characters and as a result will not seem to contain any lines.

When You Don’t Care About The Contents But Want Some Lines

You want a file of a particular size but don’t want it to just be full of nulls, other than that you don’t really care. This is a similar case to the above, use /dev/urandom. This is another special file in Linux, it is a partner of /dev/random which serves as a random number generator on a Linux system. I don’t want to go into the mechanics of it, but essentially /dev/random will eventually block unless your system has a lot of activity, /dev/urandom in non-blocking. We don’t want blocking when we’re creating our files so we use /dev/urandom (the only real difference is that /dev/urandom is actually less random but for our purposes it is random enough :)). The command is similar:

dd if=/dev/urandom of=file.txt bs=2048 count=10

This will create a file with bs*count random bytes, in our case 2048*10 = 20Kb. To generate a 100Mb file we would do:

dd if=/dev/urandom of=file.txt bs=1048576 count=100

The file will not contain anything readable, but there will be some newlines in it.

alan@alan-ubuntu-vm:~/tmp$ dd if=/dev/urandom of=file.txt bs=1048576 count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 13.4593 s, 7.8 MB/s
alan@alan-ubuntu-vm:~/tmp$ wc -l file.txt
410102 file.txt

The disadvantages here are the fact that the file does not contain anything readable and the fact that it is quite a bit slower than the** /dev/zero** method (around 10 seconds for 100Mb). The advantage is the fact that it will contain some lines.

You Want Readable Contents But Don’t Care If It Is Duplicated

In this case you want to create a file with a particular number of human-readable lines, but you don’t really care if the lines are duplicated and don’t need the size of the file to be precise. The best way I have found of doing this is as follows.

• create a file with two lines in it
• concatenate the file with itself and output to a different file
• copy the new file over the original file
• keep doing this until you get a file of a size you desire

Here are the specifics. Firstly create a file with two lines in it:

cat - > file.txt

This commands redirects STDIN to a file, so you will need to enter two lines and then press Ctrl+D. Then you will need to run the following command:

for i in {1..n}; do cat file.txt file.txt > file2.txt && mv file2.txt file.txt; done

Where n is an integer. This will create a file with 2^(n+1) lines in it, by duplicating your original two lines. So to create a file with 16 lines you would do:

for i in {1..3}; do cat file.txt file.txt > file2.txt && mv file2.txt file.txt; done

Here are some more numbers to get you started:

• n=15 will give you 65536 lines (if the original two lines were ‘hello’ and ‘world’ the file will be 384Kb)
• n=20 will give you 2097152 lines (12Mb file with ‘hello’ and ‘world’ as the two starting lines)
• n=25 will give you 67108864 lines (384Mb file with ‘hello’ and ‘world’ as the two starting lines)

Up to n=20 the file is generated instantly, after that there will be some noticeable lag, n=25 takes a few seconds.

Here is a handy tip. If you want to quickly empty a file without deleting it, redirect /dev/null into it:

cat /dev/null > file.txt

alan@alan-ubuntu-vm:~/tmp$ ls -ltrh file.txt
-rw-r--r-- 1 alan alan 100M 2010-03-21 22:56 file.txt
alan@alan-ubuntu-vm:~/tmp$ cat /dev/null > file.txt
alan@alan-ubuntu-vm:~/tmp$ ls -ltrh file.txt
-rw-r--r-- 1 alan alan 0 2010-03-21 22:57 file.txt

You Want Readable Contents And No Duplicate Lines

Similar to above, but duplicate lines are an issue for you. You want a file with a certain number of lines but don’t need the size to be precise. In this situation it is a bit of a tall order to do this as a one-liner using pure shell tools (_although it is quite possible if you don’t mind writing a little script_), so we need to turn to one of our beefier friends, Perl or Ruby (whichever one you prefer). I pick Ruby – of course :). The idea is as follows.

• Linux has a dictionary of words which is located at /usr/share/dict/words
• we want to randomly pick a number of words from there to make up into a line then output the line
• keep doing this until we get the number of lines we were looking for

The command will look like this:

```ruby

Where X is the number of lines in the file you want to generate and Y is the number of words in each line. So, to create a file with 100 lines and 4 words in each line you would do:

ruby -e 'a=STDIN.readlines;100.times do;b=[];4.times do; b << a[rand(a.size)].chomp end; puts b.join(” “); end' < /usr/share/dict/words > file.txt```


This is getting a little complex for a one-liner, but once you understand what’s going on, it is fairly easy to put together. We basically read in the dictionary into an array, then we randomly select words to form a line (getting rid of newlines while we’re at it), we then output our newly created line and we put a loop around the whole thing –  pretty simple.

With this method you’re very unlikely to ever get repeated lines (although it is technically possible). It takes about 10 seconds to generate a 100Mb file (around 1 million lines with 12 words per line), which is comparable to some of our other methods. The lines we produce will not make any semantic sense but will be made up of real words and will therefore be readable.

There you go, you’re now able to generate any kind of data file you want (large or small) for your testing purposes. If you know of any other funky ways to generate files in Linux, then please share and don’t forget to grab my RSS feed if you haven’t already. Enjoy!

Image by MGSpiller