I am feeling a little existential and metaphorical today, so forgive me if this post has more than just a slight tinge of existential metaphor about it. I do hope it still makes some sense. 

There are two fundamental ways of looking at software development. One is all about the final product and the direct benefits you can get from it. The other is all about the lessons you learn from the process of building the software. I call them software as a destination and software as a journey. Historically speaking and even into the present day, the majority of companies that build any kind of software are ‘software as a destination’ companies. They want to extract some direct financial benefit from the software they build (which is fair enough), until this benefit is realised, the process of building the software is just a resource (time, money) drain. On the other hand, most individuals who build software for those companies would instinctively like to be (and in many cases ARE) ‘software as a journey’ people. We want to enjoy our work, we want to learn as much as we can, we want to interact with those around us, we want to build code that we can be proud of and we want to get paid. All of this will happen throughout the project rather than just at the end, and so it is all about the journey, the final success of the product is almost icing on the cake. It seems that there is a fundamental disconnect between what a company wants to get out of building software and what the developers will try to extract from the experience.

This is why the software development industry is in a sort-of screwed-up state, or at least it has been for many years (things are getting better). Companies see the destination and want to arrive there as quickly as they can, the sooner they get there the sooner they see the benefits from all that money they spend and the sooner they can start getting their money back, either directly or indirectly. Anything that stands in the way of this goal must be swept aside and this includes the developers who want to enjoy the journey and ‘smell the roses’ on their way to the final product. Of course, the companies hold all the cards (they employ you, pay you – which indirectly, feeds and clothes’ your children) and so can usually get their way if they really want to. Developers can try to make noise about the value of paying attention to the journey, but if the company is dead set on doing things their way, that’s the way things will be done, developers can either fall into line or there will be other developers to replace them. However, just like a real, physical journey, if you have your eyes set firmly on the destination and are not looking where you’re putting your feet, there is a good chance you will trip over some roots, or a snake will bite you, or you won’t notice that cliff which is right between you and where you’re going. This is how we get failed projects.

Once the project and the company are at the bottom of the cliff, you can kinda tell that everyone is in trouble. But it is difficult to figure out exactly how you ended up where you are, you had your eyes firmly on the destination and forgot to pay attention to what was happening around you, so it is hard to pinpoint the mistakes that landed you at the bottom of a cliff. Companies will do a post-mortem, come up with some vague reasons for why things went to hell, everyone feels better that they have an explanation (leaving everyone none the wiser in reality). And so next time they go out and do the same thing all over again. All the while, the answer is staring everyone right in the face. If only they had listened to the developers who told them to pay attention to the journey and not worry so much about the destination things would have been different. Even novice programmers soon learn, from personal experience, that when we slow down and pay attention to what we are doing right now, we get a lot more out of it, we learn more and the final product is improved as a result. There are only benefits, no downsides, alright, you lose some time in the short term, but gain it all back with interest in the long term. But, at the macro level where committees make decisions, this same lesson takes years and decades to permeate the hive-mind. Not to mention the fact that seeing only the destination is much easier than constantly paying attention, and so even the occasional developer gets seduced by the ‘software as a destination’ mentality (it’s really more than just the occasional), which sets everyone back by introducing conflicting opinion and muddying the waters unnecessarily.

There was a bunch of people who were all ‘software as a journey’ people who basically got sick of all the back and forth. They recognised the fact that the journey is what really matters not just at the personal level, where each individual developer is concerned, but at the team, project and company level as well. They decided to give a united voice to the message that as long as you take care and extract the maximum benefit along the way, the destination will take care of itself – you will arrive somewhere you want to be. It may not be exactly the same place where you thought you would end up, but it will be a nice place nevertheless and best of all, you will have a very good idea of precisely how you got where you are, despite not having a narrow focus on the final objective from the start. More like-minded people joined the initial bunch and the movement got more organised. Some of these people started companies of their own; others brought their message to larger organisation. And now that the ‘software as a journey’ message had a united voice (albeit still a comparatively small one), the success of looking at software development this way could be demonstrated, but more importantly, disseminated to the industry at large.

It is a major undertaking for a company to develop any significant piece of software. Not just from a financial perspective, but as far as airing out the dirty baggage and discovering the process inefficiencies that every significant company will have. After all, in order to put something logically into code you need to figure out exactly what that something is and how it works. This is a golden opportunity for the company to streamline and improve how their business works, figure out if some processes are unnecessary. You’re building software, but you’re paying attention to what you’re doing not just at the technical level but at the organisational level as well. As a result, not only do you get more robust software that better fits your needs, but you also improve the day to day operations of the company; all because you were paying attention to the journey rather than focusing on the destination.

But, it is so much easier to simply look towards the goals and on paper it seems like it SHOULD be more efficient. If you’re in management you want to seem more efficient and focused rather than being wishy-washy and touchy-feely, preaching self-awareness and a less-rapid pace just isn’t strategic enough. And so companies create artificial deadline pressures (like end of financial cycle of some sort), to make everything neater and make the people in charge look good and it feels like for every step the agile movement takes towards running software projects better, some company somewhere, takes two steps back.

Robert A Heinlein once said:

“The supreme irony of life is that hardly anyone gets out of it alive.”

Infact, he had many other quotes which I think are quite relevant to the software profession, maybe I’ll do a post about that at some point. To me this quote says that it’s the moment that really matters, enjoy the life you have now rather than constantly looking forward to something better. I am not sure if there is any irony to be found in software, but if you direct all your focus towards your goal without paying due attention to the nitty gritty of what you’re doing every day, you’re likely to not get any useable software out of it. As long as you have a reasonable idea of where you want to end up, you just need to get the details right and the bigger picture will tend to sort itself out. On the other hand, you can have the clearest possible goal in mind, but if you let the details slide, bad things will almost certainly happen.

Images by Norma Desmond and alicepopkorn

If you do your development work in Linux, there are certain commands that you owe it to yourself to master fully. There are a number of these with the main ones being grep, find and sort. Just about everyone has at least a passing familiarity with these commands, but with most people the knowledge is superficial, they don’t even realise how powerful those commands can be. So, if you really put in the effort to master them, not only will you make your own life much easier, but you will also be able to impress all you friends with your elite Linux skills when you pair with them :). I will cover grep and find (as well as other valuable commands) in subsequent posts – here we will concentrate on sort. 

Note: I am using bash, so your mileage might vary if you’re using a different shell.

Sorting is a fundamental task when it comes to programming, if you have a decent knowledge of various sorting algorithms, their advantages and disadvantages, you will be a better software developer for it. However, often enough you just don’t need to draw on this deeper knowledge. Whether you’re answering an interview question about sorting or simply need to quickly sort some data in you day to day work – the Linux sort command is your friend.

The extent of most people’s knowledge ends with:

sort some_file.txt

Which is fair enough, you rarely need to dig deeper, the default behaviour will usually do what you need and when it doesn’t – we have Ruby or Perl, we can hack something together. Well I hope that we’re somewhat more curious than your average developer :). As much is we like hacking things together, if a tool can already do all the work for us, we want to know about it. We can look at the man page for sort and discover all sorts of interesting bits, but even the man page is not really clear on the more advanced aspects of sort usage. It helps to have examples to truly grok the kinds of stuff you can do with sort, so let’s have a look.

Sorting Basics

To start with we have the following file:

alan@alan-ubuntu-vm:~/tmp/sort$ cat letters.txt
b
D
c
A
C
B
d
a

We’ll do the most basic sort first:

alan@alan-ubuntu-vm:~/tmp/sort$ sort letters.txt
a
A
b
B
c
C
d
D

Looks good, how about doing it in reverse:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -r letters.txt
D
d
C
c
B
b
A
a

Also easy, but what if we want to be case insensitive? Hang on a sec, according to the output it’s already case insensitive. But the man page has an option for this:

-f, --ignore-case
              fold lower case to upper case characters

If sort is case insensitive by default, what is this option for. We’ll that one is a bit of a gotcha, it looks like GNU sort is case insensitive by default, but the man page also contains the following:

*** WARNING *** The locale specified by the  environment  affects  sort
       order.  Set LC_ALL=C to get the traditional sort order that uses native
       byte values.

What this means that we need to set the LC_ALL environment variable to get the behaviour that we would expect from sort (i.e. capital letters before non-capitals). Let’s try that:

alan@alan-ubuntu-vm:~/tmp/sort$ export LC_ALL=C
alan@alan-ubuntu-vm:~/tmp/sort$ sort letters.txt
A
B
C
D
a
b
c
d

That’s better, and now our -f option is actually useful:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f letters.txt
A
a
B
b
C
c
D
d

That looks ok, but something still seems a little funny, all the capitals appear before all the non-capitals every time. That’s because the sort is not stable, but we can make it stable:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s letters.txt
A
a
b
B
c
C
D
d

Now that’s exactly what we wanted, it’s case insensitive and stable, i.e. if the small letter appeared before the capital when unsorted (and the letters are the same), this order will be the same in the sorted list.

Ok, but what if we have numbers:

alan@alan-ubuntu-vm:~/tmp/sort$ cat numbers.txt
5
4
12
1
3
56

A normal sort is not what we want:

alan@alan-ubuntu-vm:~/tmp/sort$ sort numbers.txt
1
12
3
4
5
56

But we can fix that:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -n numbers.txt
1
3
4
5
12
56

And, if our lines happen to have some leading blanks:

alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters.txt
b
D
   c
A
C
    B
d
a

We can easily ignore those and still sort correctly (using the -b flag):

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s -b blank_letters.txt
A
a
b
    B
   c
C
D
d

Of course none of this actually writes the sorted output back to the file, we only get it on standard output. If we want to write it back to the file, we have to redirect the output to a new file and then replace the old file with the new file:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s -b blank_letters2.txt > blank_letters2.sorted
alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters2.sorted
A
a
b
    B
   c
C
D
d
alan@alan-ubuntu-vm:~/tmp/sort$ mv blank_letters2.sorted blank_letters2.txt
alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters2.txt
A
a
b
    B
   c
C
D
d

As an alternative to redirection, sort also has the -o option:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s -b blank_letters2.txt -o blank_letters2.sorted
alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters2.sorted
A
a
b
    B
   c
C
D
d

Alright, this is all pretty standard stuff, let’s see how we can do something a little bit more fancy.

Advanced Sort Usage

One of the most common use cases when it comes to sort is to pipe its output to uniq, in order to remove any duplicate lines, but this is often not necessary as sort has a uniq-type option built right in (-u):

alan@alan-ubuntu-vm:~/tmp/sort$ cat blank_letters.txt
b
D
c
A
C
B
d
a
alan@alan-ubuntu-vm:~/tmp/sort$ sort -f -s -u blank_letters.txt
A
b
c
D

It even took into account the fact that we wanted to be case insensitive. Compare that to a case-sensitive sort with a uniq option set:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -s -u blank_letters.txt
A
B
C
D
a
b
c
d

Nothing is removed as there are no duplicate lines.

But what if my lines are a little bit more complicated than just a single letter, what if there are multiple columns and I want to sort by one of those columns (not the first one). This is also possible:

alan@alan-ubuntu-vm:~/tmp/sort$ ls -al | sort -k5
total 36
-rw-r--r-- 1 alan alan   14 May  9 00:12 numbers.txt
-rw-r--r-- 1 alan alan   16 May  9 00:00 letters.txt
-rw-r--r-- 1 alan alan   16 May  9 00:40 blank_letters.txt
-rw-r--r-- 1 alan alan   20 May  9 00:33 blank_letters2.sorted
-rw-r--r-- 1 alan alan   20 May  9 00:33 blank_letters2.txt
-rw-r--r-- 1 alan alan   20 May  9 00:37 blank_letters3.txt
-rw-r--r-- 1 alan alan   84 May  8 23:15 file1.txt
drwxr-xr-x 3 alan alan 4096 May  8 23:13 ..
drwxr-xr-x 2 alan alan 4096 May  9 00:40 .

As you can see we sorted the output of ls by the size (the 5th column). This is what the -k option is for. Basically -k tells sort to start sorting at a particular column given a particular column separator. The column separator is, by default, any blank character. So in the above example, we told sort to sort by the 5th column given the fact that the column separator is blanks.

But we don’t have to be restricted by the default separator, we can specify our own using the -t option. Let’s sort the first 10 lines of my /etc/passwd file (promise you won’t hack my machine since I am giving it away like that :)) by the 4th column – the group id. As you know the /etc/passwd file uses the : (colon) character as the separator (_for more info on the format see here_). Here is the output unsorted:

alan@alan-ubuntu-vm:~/tmp/sort$ cat /etc/passwd | head                 
root❌0:0:root:/root:/bin/bash
daemon❌1:1:daemon:/usr/sbin:/bin/sh
bin❌2:2:bin:/bin:/bin/sh
sys❌3:3:sys:/dev:/bin/sh
sync❌4:65534:sync:/bin:/bin/sync
games❌5:60:games:/usr/games:/bin/sh
man❌6:12👨/var/cache/man:/bin/sh
lp❌7:7:lp:/var/spool/lpd:/bin/sh
mail❌8:8:mail:/var/mail:/bin/sh
news❌9:9:news:/var/spool/news:/bin/sh

There is a group id of 65534 in there which should appear last. Let’s sort it:

alan@alan-ubuntu-vm:~/tmp/sort$ cat /etc/passwd | head | sort -t: -k4 -n
root❌0:0:root:/root:/bin/bash
daemon❌1:1:daemon:/usr/sbin:/bin/sh
bin❌2:2:bin:/bin:/bin/sh
sys❌3:3:sys:/dev:/bin/sh
lp❌7:7:lp:/var/spool/lpd:/bin/sh
mail❌8:8:mail:/var/mail:/bin/sh
news❌9:9:news:/var/spool/news:/bin/sh
man❌6:12👨/var/cache/man:/bin/sh
games❌5:60:games:/usr/games:/bin/sh
sync❌4:65534:sync:/bin:/bin/sync

We had to do a numeric sort since we’re dealing with numbers, and we specified : (colon) as the column separator. The output is sorted correctly with 65534 being on the last line. Pretty cool! But the fun doesn’t stop there, we can sort by multiple columns, one after the other. Consider this list of IP addresses:

alan@alan-ubuntu-vm:~/tmp/sort$ cat ips.txt
192.168.0.25
127.0.0.12
192.168.0.1
127.0.0.3
127.0.0.6
192.168.0.5

Let’s sort it by the first column, so that all the addresses starting with 127 go together, and then sort it by the 4th column, to make sure that the IPs are sorted by the last column within each range.

alan@alan-ubuntu-vm:~/tmp/sort$ cat ips.txt | sort -t. -k 2,2n -k 4,4n
127.0.0.3
127.0.0.6
127.0.0.12
192.168.0.1
192.168.0.5
192.168.0.25

We specified the dot as the separator. The -k 2,2n syntax has the following meaning. Do a sort by column (-k), start at the beginning of column 2 and go to the end of column 2 (2,2). The n on the end is to indicate that we want to do a numeric sort since we are dealing with numbers. That is some powerful stuff, wouldn’t you agree?

Cool/Useful Stuff

There is still more we can do with the sort command. Have you ever wanted to randomize the lines in a file? It is not a common use case, but does come in handy once in a while (if only for testing purposes, sometimes). Well, the sort command has you covered here also with the -R option (that’s capital R):

alan@alan-ubuntu-vm:~/tmp/sort$ cat numbers.txt
5
4
12
1
3
56
alan@alan-ubuntu-vm:~/tmp/sort$ cat numbers.txt | sort -R
5
4
1
3
12
56
alan@alan-ubuntu-vm:~/tmp/sort$ cat numbers.txt | sort -R
3
4
1
56
5
12

We get a different order every time, which is what we would expect from randomizing the lines.

If you give sort multiple files on the command line, it will combine the contents of all the files and sort it as a whole:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -n numbers.txt numbers2.txt
1
1
3
4
4
5
7
8
10
12
22
23
26
56
56
68

This is really handy, but sometimes, the files you have are already sorted, you just want to merge them. Sort provides the** -m** options just for this purpose. The output of using sort on two files will be exactly the same whether you use -m or not, but merging should be faster:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -n -m numbers1.sorted numbers2.sorted
1
1
3
4
4
5
7
8
10
12
22
23
26
56
56
68

Lastly, if you just want to check if a file is sorted or not, without actually performing the sort, you have the -c option:

alan@alan-ubuntu-vm:~/tmp/sort$ sort -n -c numbers.txt
sort: numbers.txt:2: disorder: 4
alan@alan-ubuntu-vm:~/tmp/sort$ sort -n -c numbers1.sorted
alan@alan-ubuntu-vm:~/tmp/sort$

There you go, the total awesomeness of the sort command laid bare. If you know any other handy things you can do with sort, do leave a comment. And remember – only use your new-found sort powers for good instead of evil :).

Image by SewPixie (so far behind!)

A lot of people have asked me recently how/where I get my ideas (_e.g. like here) – in reference to the diverse topics that I write about. After what must have been the 10th time someone asked me that question, I thought to myself “…gee, what a good idea, why not write about that…_“, so perhaps I am not the best person to answer the question after all :), but I do have some thoughts, so I’ll give it a whirl regardless.

Ideas are an interesting beast, the more of them you have; the more you tend to come up with. That might seem somewhat counterintuitive or even circular, “…so, ummm, to have more ideas you need to come up with more ideas? Why, that’s pure diabolical genius! …” but bear with me for just a minute and I’ll explain. Here is the way it goes with most people. You need to come up with an idea – business idea, article idea, book idea, conversational topic – whatever. So, you strain for a few seconds trying to have a flash of brilliance and when the flash inevitably fails to materialize, you shrug, say something self-deprecating like, “… man, I really wish I was better at thinking up ideas. If only I could have a brilliant business idea, I could become a millionaire. Nay, a billionaire! Oh well …”. At this point you go back to doing whatever you were doing before and life goes back to normal – which is precisely the worst thing you could have done.

The Idea-Generating Muscle

When I first started blogging, there were a few opinions I wanted to share, I was excited. It seemed like I had hundreds of ideas, but in reality, it was more like half a dozen (the brain is good making stuff seem completely out of proportion when you’re excited). Once my initial idea pool was exhausted, I found myself stuck for things to write about. Me, stuck for something to say – completely unacceptable, I have a reputation to uphold :). So, I just started writing about crazy stuff that popped into my head, all the while doing my best to pay attention to the random thoughts that I was having and examining each one for potential (to be turned into a post). This had the effect of keeping my need for ideas at the forefront of my mind and you know what, after a while my brain started to respond.

Here is how I see it. The needs that you have send signals to your brain, like the need to have a good idea about something. When your mind senses the need, it will do its best to respond, but its first try will likely be a bit of a failure. It’s understandable, have you ever been brilliant at something the very first time YOU tried it? If at this point you remove the need (i.e. you give up since you couldn’t get a good idea instantly) your mind will just go back to its default state (of not generating ideas). But, if the need remains, if it is persistent, your mind will keep trying to fulfil it and will likely generate another idea for you after a while (which will likely also be pretty dismal). The difference here is that you’re now training your mind to give you what you require, by maintaining your need for it. Try visualising your brain as a set of muscles, there is an idea-generating muscle in there somewhere. But, like any muscle, if you don’t consistently exercise it, you won’t be able to get much out of it and are likely to overstrain it at the merest effort. So, keep your need for ideas constant, force yourself to come up with stuff, even if it is complete trash.  Keep sending the messages to your brain and over a period of days, weeks and months you will find yourself having more and more ideas about your topic of interest. More than that, as you keep going, your ideas WILL get better – that’s a promise. Your mind will adapt, as you discard the bad ideas and adopt the good ones, by producing more ideas along the lines of what you consider “good”. The more ideas you have, the better “trained” your mind becomes, the more chance it has to adapt to your style; which means you’ll keep having better and better ideas (and more of them as well). It is really quite a fascinating process to observe in yourself.

What effect did it have on me? Well these days, my biggest problem, at least when it comes to my writing, is a kind of analysis paralysis where I often have several ideas that I am equally keen to write about at any given moment, it is surprisingly difficult to choose :). Of course, I do make a decision eventually – one can’t procrastinate forever, but my list of “stuff/thoughts/ideas to write about” just gets longer and longer – kinda like my reading list.

Confirming The Principle

The interesting thing is that this training doesn’t seem to work for ideas in general, only ideas regarding the actual need that you focus your mind on. When I developed an interest in how to conduct a good interview to find decent coders, I needed to come up with some interesting coding interview questions. And you know what, I was drawing blanks once more, it was like starting the mental training process all over again. I forced myself to come up with anything and produced some crazy ideas at first, then had some better thoughts and now (a few months later) am coming up with what I consider to be some pretty decent questions and ideas (which end up on that long list of things I want to write about :)). I’ve also gone through the same exercise for personal projects to practice your coding – with similar results. You start seeing the pattern after a while. Start thinking about whatever interests you; immerse yourself in it to create a personal need for ideas. Force yourself to examine everything your mind throws at you – even if it seems like nonsense – to allow your “mental-muscle” to adapt. Given enough time, good ideas will follow.

What About Business Ideas?

You know those guys who always seem to have hundreds of brilliant business ideas while you struggle to come up with even one that seems viable. Well, you can BE one of those guys; more specifically you can train your mind into making you one of those guys. You just need to persist and not give up, like most things in life that are worth striving for. The only issue is the fact that the cost of trying out a business idea (to really evaluate its worth), is potentially way too high for most people to be able to afford to do it every time. However once you have trained yourself to have some good ideas in several areas, you start to get a feel for how long it takes for your brain to start coming up with some great stuff more or less regularly. That time frame will likely be similar when it comes to ideas in other areas – like business ideas. The key is to pay attention and be a little bit more self-aware than your average bear :).

Finally, don’t forget to write down your ideas when you have them, it doesn’t matter how you do it, dictate them into your phone, carry a pen and paper, send yourself an SMS. Just make you sure you don’t lose them; there is really no point in training yourself to have ideas if you’re just going to let them flow away into the ether. Especially considering that given enough time, any one of them could be a real winner.

Images by Cayusa and sarah …