I have strong opinions regarding software consulting and consulting companies, which is probably no surprise as I seem to have a strong opinion about most things :). It’s not a bad thing, makes for some lively conversation over drinks, but it is also a bit of a pain as it makes it hard to choose what to write about. I’ll regale you with my serious thoughts regarding consulting in another post (don’t forget to subscribe to my feed so you don’t miss it) – today we’ll try to maintain a bit of a lighter tone.

A little while ago I read an interesting book called “The Nomadic Developer”, if you are considering going into consulting I recommend you read it (although you would appreciate it more if you’ve already worked in software consulting). The book has some decent advice and information for budding/existing software consultants, but the thing that really caught my eye was the humorous classification of consulting firms found at the beginning of the book. It was titled “The Seven Deadly Firms” and I thought I would share it with you for a bit of fun as well as adding one more classification of my own to bring the total up to eight. I will quote and paraphrase as necessary – here we go.

BOZO Consulting

BOZO is a place with well-intentioned people who really want to please their clients. The only downside to BOZO is it doesn’t have the slightest clue about how to achieve that goal. BOZO says yes to everything: “Yes, of course we will cut the estimate.” “Yes, of course, we will lower our bill rate.” The idea is to get the deal at all costs. To summarize, BOZO has a sales-driven culture that lacks the ability to leverage any sort of delivery capability it accidentally hires (thought from Skorks: that would have to be one of the most buzzword driven sentences ever, how can we “leverage” that).

FEAR Consulting

Firm founded on its ability to motivate software developers by fear. Micromanagement, abuse and/or Machiavellian management techniques are the tools this firm uses to generate mundane and uninspired results for clients. The best way to describe how FEAR stays in business despite such deplorable working conditions and treatment is to reference what is known as the Stockholm Syndrome (thought from Skorks: the funny/sad thing is that any consultancy can accidentally/purposely take on aspects of FEAR Consulting in certain situation (financial/project pressures etc.)).

The Body Shop

Contracting firm that pretends to be a consultancy. Sadly, not all consultants involved have been informed of this “minor” detail. The Body Shop has zero corporate culture. No events, no infrastructure and no interaction with fellow “consultants”. Working for a Body Shop you run significant risk of getting the sack after completing your first gig if there is no work immediately available (_thought from Skorks: does anyone else find it funny that the body shop is an actual company, unfortunately all they do is sell skin care products, but who knows software consulting could be in their future_).

CHEAP Consulting

Some firms compete on quality. CHEAP consulting isn’t one of those firms. This is a great place to work as long as eating and having a roof over your head are not huge priorities for you. CHEAP Consulting firms execute entire projects using hordes of low-cost developers, rather than small numbers of good ones. Ironically, this makes their projects, after all is said and done, come in far more over budget than the firms they competed against in the first place. The main feature of CHEAP is the heavy use of the word resource. (_thought from Skorks: and you know how I feel about the word resource, anyway, probably not too many of these around in the first world even cheap developers are reasonably expensive if the cost of living is high enough_).

Personality Cult Consulting

The only drink that this firm has in the refrigerator is kool-aid. And if you refuse to drink it – that is, decide not to indulge in the cult of local hero worship – you might not have a future here. Critical thinking skills are not the highest priority. Frequently, a person who combines charisma with a good measure of talents finds himself able to start a consulting company that capitalizes on these traits. What starts out as a small group of like-minded people with a mission grows into a creature called PC Consulting. Discussions in such a place are rife with appeals to authority; that is, if the cult leader says something, it not only is given a lot of weight, but is treated as gospel (thought from Skorks: you know what I said about me having strong opinions, I cringe at how quickly I, and most decent developers I know, would get their ass handed to them in a place like this).

Smelzer and Melzer Accounting

Even the owners of this place, an accounting partnership, wonder how they ended up in the “computers” business. They know they make money from it but are unsure why, and frankly, they are not sure they trust these “kids” who have a lot of scary ideas. In most cases, accounting firms are run by accountants. And accountants frequently have significant issues with the way software tends to be developed. Software development methodology is chosen, based on what is perceived as predictable and developers often become the scapegoats when the expectations of predictability fail to materialize (_thought from Skorks: this one is easy, Big Four anyone?_).

“Push the SKU” Consulting

“Services” arm of a product company that ends up acting as a de-facto sales force for the product, while at the same time, getting paid by the hour from the client to sell…ahem, I mean… “provide independent advice”. The incentive system tends to make this not the place for development of software that works independent of a particular product. Such an organization is not always a good place for software developers who have the interest of the client at heart first and foremost (thought from Skorks: does your software do X? “Sure, sure it does, it just needs some minor configuration …“).

Band Of Freelancers (My Contribution)

A bunch of friends decide to form a company to build the next great product. But of course building products takes time and you have to live in the meantime, so everyone picks up some occasional consulting work to “keep the lights on”. The only problem is, occasional means pretty much permanent and it is difficult to write a product while working full time on something else. So, all the “founders” keep on consulting and no product ever materializes, so the company turns into a perpetual band-of-freelancers consulting company. The real danger is that the “founders” actually do well as consultants and are able to hire more people at which point there is a real danger of this company turning into BOZO, Body Shop or Personality Cult consulting.

There you go, humorous – yes, but also rather accurate, wouldn’t you agree? Of course all those are examples of potentially dysfunctional consulting companies. There are plenty of big and small consultancies (admittedly mostly smaller) that are great places to work, treat their people well and approach their software development with skill and wisdom. The thing to be aware of is that no matter how good a consulting company is, most will exhibit at least some aspects of the above 8 at one point or another. The trick is to know whether or not, it is just a bad patch – soon corrected and forgotten – or whether it is becoming more of a permanent situation. Anyways, go check out the book if you want to get the full descriptions as well as tips for recognising if you might, infact, be working for one of these companies. If you do work for a software consultancy, how would you classify the company you work for? Does it by any chance fall into one of the 8 categories above?

On an even lighter note, is it just me or does anyone else ever misspell the word consulting as consluting, can be a little embarrassing if you do it in any kind of semi-official document. The other one is conlusting which also works but is not quite as good :).

Images by orkboi and Venn Diagram

I have recently written about the value of fundamentals in software development. I am still firmly of the opinion that you need to have your fundamentals down solid, if you want to be a decent developer. However, several people made a valid point in response to that post, in that it is often difficult to know what the fundamentals actually are (be they macro or micro level). So, I thought it would be a good idea to do an ongoing series of posts on some of the things that I consider to be fundamental – this post is the first instalment.

Being a developer this day and age, it would be almost impossible for you to avoid doing some kind of web-related work at some point in your career. That means you will inevitably have to deal with URLs at one time or another. We all know what URLs are about, but there is a difference between knowing URLs like a user and knowing them like a developer should know them.

As a web developer you really have no excuse for not knowing everything there is to know about URLs, there is just not that much to them. But, I have found that even experienced developers often have some glaring holes in their knowledge of URLs. So, I thought I would do a quick tour of everything that every developer should know about URLs. Strap yourself in – this won’t take long :).

The Structure Of A URL

This is easy, starts with HTTP and ends with .com right :)? Most URLs have the same general syntax, made up of the following nine parts:

<scheme>://<username>:<password>@<host>:<port>/<path>;<parameters>?<query>#<fragment>

Most URLs won’t contain all of the parts. The most common components, as you undoubtedly know, are the scheme, host and path. Let’s have a look at each of these in turn:

• scheme – this basically specifies the protocol to use to access the resource addressed by the URL (e.g. http, ftp). There are a multitude of different schemes. A scheme is official if it has been registered with the IANA (like http and ftp), but there are many unofficial (not registered) schemes which are also in common use (such as sftp, or svn). The scheme must start with a letter and is separated from the rest of the URL by the first : (colon) character. That’s right, the // is not part of the separator but is infact the beginning of the next part of the URL.
• username – this along with the password, the host and the port form what’s known as the authority part of the URL. Some schemes require authentication information to access a resource this is the username part of that authentication information. The username and password are very common in ftp URLs, they are less common in http URLs, but you do come across them fairly regularly.

• password – the other part of the authentication information for a URL, it is separated from the username by another : (colon) character. The username and password will be separated from the host by an @ (at) character. You may supply just the username or both the username and password e.g.:

  ftp://[email protected]/
  ftp://some_user:[email protected]/
  

  If you don’t supply the username and password and the URL you’re trying to access requires one, the application you’re using (e.g. browser) will supply some defaults.

• host – as I mentioned, it is one of the components that makes up the authority part of the URL. The host can be either a domain name or an IP address, as we all should know the domain name will resolve to an IP address (via a DNS lookup) to identify the machine we’re trying to access.

• port – the last part of the authority. It basically tells us what network port a particular application on the machine we’re connecting to is listening on. As we all know, for HTTP the default port is 80, if the port is omitted from an http URL, this is assumed.

• path – is separated from the URL components preceding it by a / (slash) character. A path is a sequence of segments separated by / characters. The path basically tells us where on the server machine a resource lives. Each of the path segments can contain parameters which are separated from the segment by a ; (semi-colon) character e.g.:

  http://www.blah.com/some;param1=foo/crazy;param2=bar/path.html

  The URL above is perfectly valid, although this ability of path segments to hold parameters is almost never used (I’ve never seen it personally).

• parameters – talking about parameters, these can also appear after the path but before the query string, also separated from the rest of the URL and from each other by ; characters e.g.:

  http://www.blah.com/some/crazy/path.html;param1=foo;param2=bar

  As I said, they are not very common

• query – these on the other hand are very common as every web developer would know. This is the preferred way to send some parameters to a resource on the server. These are key=value pairs and are separated from the rest of the URL by a ? (question mark) character and are normally separated from each other by & (ampersand) characters. What you may not know is the fact that it is legal to separate them from each other by the ; (semi-colon) character as well. The following URLs are equivalent:

  http://www.blah.com/some/crazy/path.html?param1=foo&param2=bar
  http://www.blah.com/some/crazy/path.html?param1=foo;param2=bar

• fragment – this is an optional part of the URL and is used to address a particular part of a resource. We usually see these used to link to a particular section of an html document. A fragment is separated from the rest of the URL with a # (hash) character. When requesting a resource addressed by a URL from a server, the client (i.e. browser) will usually not send the fragment to the server (at least not where HTTP is concerned). Once the client has fetched the resource, it will then use the fragment to address the relevant part.

That’s it, all you need to know about the structure of a URL. From now on you no longer have any excuse for calling the fragment – “that hash link thingy to go to a particular part of the html file”.

Special Characters In URLs

There is a lot of confusion regarding which characters are safe to use in a URL and which are not, as well as how a URL should be properly encoded. Developers often try to infer this stuff from general knowledge (i.e. the / and : characters should obviously be encoded since they have special meaning in a URL). This is not necessary, you should know this stuff solid – it’s simple. Here is the low down.

There are several sets of characters you need to be aware of when it comes to URLs. Firstly, the characters that have special meaning within a URL are known as reserved characters, these are:

";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" | "$" | ","

What this means is that these characters are normally used in a URL as-is and are meaningful within a URL context (i.e. separate components from each other etc.). If a part of a URL (such as a query parameter), is likely to contain one of these characters, it should be escaped before being included in the URL. I have spoken about URL encoding before, check it out, we will revisit it shortly.

The second set of characters to be aware of is the unreserved set. It is made up of the following characters

"-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"

The characters can be included as-is in any part of the URL (note that they may not be allowed as part of a particular component of a URL). This basically means you don’t need to encode/escape these characters when including them as part of a URL. You CAN escape them without changing the semantics of a URL, but it is not recommended.

The third set to be aware of is the_ ‘unwise’ set_, i.e. it is unwise to use these characters as part of a URL. It is made up of the following characters

"{" | "}" | "|" | "\" | "^" | "[" | "]" | "`"

These characters are considered unwise to use in a URL because gateways are known to sometimes modify such characters, or they are used as delimiters. That doesn’t mean that these characters will always be modified by a gateway, but it can happen. So, if you include these as part of a URL without escaping them, you do this at your own risk. What it really means is you should always escape these characters if a part of your URL (i.e. like a query param) is likely to contain them.

The last set of characters is the excluded set. It is made up of all ASCII control characters, the space character as well the following characters (known as delimiters)

"<" | ">" | "#" | "%" | '"'

The control characters are non-printable US-ASCII characters (i.e. hexadecimal 00-1F as well as 7F). These characters must always be escaped if they are included in a component of a URL. Some, such as # (hash) and % (percent) have special meaning within the context of a URL (they can really be considered equivalent to the reserved characters). Other characters in this set have no printable representation and therefore escaping them is the only way to represent them. The <, > and “ characters should be escaped since these characters are often used to delimit URLs in text.

To URL encode/escape a character we simply append its 2 character ASCII hexadecimal value to the % character. So, the URL encoding of a space character is %20 – we have all seen that one. The % character itself is encoded as %25.

That’s all you need to know about various special characters in URLs. Of course aside from those characters, alpha-numerics are allowed and don’t need to be encoded :).

A few things you have to remember. A URL should always be in its encoded form. The only time you should decode parts of the URL is when you’re pulling the URL apart (for whatever reason). Each part of the URL must be encoded separately, this should be pretty obvious, you don’t want to try encoding an already constructed URL, since there is no way to distinguish when reserved characters are used for their reserved purpose (they shouldn’t be encoded) and when they are part of a URL component (which means they should be encoded). Lastly you should never try to double encode/decode a URL. Consider that if you encode a URL once but try to decode it twice and one of the URL components contains the % character you can destroy your URL e.g.:

http://blah.com/yadda.html?param1=abc%613

When encoded it will look like this:

http://blah.com/yadda.html?param1=abc%25613

If you try to decode it twice you will get:

1. http://blah.com/yadda.html?param1=abc%613

   Correct

• http://blah.com/yadda.html?param1=abca3

  Stuffed

By the way I am not just pulling this stuff out of thin air. It is all defined in RFC 2396, you can go and check it out if you like, although it is by no means the most entertaining thing you can read, I’d like to hope my post is somewhat less dry :).

Absolute vs Relative URLs

The last thing that every developer should know is the difference between an absolute and relative URL as well as how to turn a relative URL into its absolute form.

The first part of that is pretty easy, if a URL contains a scheme (such as http), then it can be considered an absolute URL. Relative URLs are a little bit more complicated.

A relative URL is always interpreted relative to another URL (hence the name :)), this other URL is known as the base URL. To convert a relative URL into its absolute form we firstly need to figure out the base URL, and then, depending on the syntax of our relative URL we combine it with the base to form its absolute form.

We normally see a relative URL inside an html document. In this case there are two ways to find out what the base is.

1. The base URL may have been explicitly specified in the document using the HTML tag.
2. If no base tag is specified, then the URL of the html document in which the relative URL is found should be treated as the base.
   

Once we have a base URL, we can try and turn our relative URL into an absolute one. First, we need to try and break our relative URL into components (i.e. scheme, authority (host, port), path, query string, fragment). Once this is done, there are several special cases to be aware of, all of which mean that our relative URL wasn’t really relative.

• if there is no scheme, authority or path, then the relative URL is a reference to the base URL
• if there is a scheme then the relative URL is actually an absolute URL and should be treated as such
• if there is no scheme, but there is an authority (host, port), then our relative URL is likely a network path, we take the scheme from our base URL and append our “relative” URL to it separating the two by ://
  

If none of those special cases occurred then we have a real relative URL on our hands. Now we need to proceed as follows.

• we inherit the scheme, and authority (host, port) from the base URL
• if our relative URL begins with /, then it is an absolute path, we append it to the scheme and authority we inherited from the base using appropriate separators to get our absolute URL
• if relative URL does not begin with / then we take the path of from base URL, discarding everything after the last / character
• we then take our relative URL and append it to the resulting path, we now need to do a little further processing which depends on the first several characters of our relative URL
• if there is a ./ (dot slash) anywhere in a resulting path we remove it (this means our relative URL started with ./ i.e. ./blah.html)
• if there is a ../ (dot dot slash) anywhere in the path then we remove it as well as the preceding segment of the path i.e. all occurrences of “/../” are removed, keep doing this step until no more ../ can be found anywhere in the path (this means our relative path started with one or more ../ i.e. ../blah.html or ../../blah.html etc.)
• if the path ends with .. then we remove it and the preceding segment of the path, i.e. “/..” is removed (this means our relative path was .. (dot dot))
• if the path ends with a . (dot) then we remove it (this most likely means our relative path was . (dot))
  

At this point we simply append any query string or fragment that our relative URL may have contained to our URL using appropriate separators and we have finished turning our relative URL into an absolute one.

Here are some examples of applying the above algorithm:

1)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: rel1





final absolute: http://www.blah.com/yadda1/yadda2/rel1

2)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: /rel1

final absolute: http://www.blah.com/rel1

3)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: ../rel1

final absolute: http://www.blah.com/yadda1/rel1

4)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: ./rel1?param2=baz#bar2

final absolute: http://www.blah.com/yadda1/yadda2/rel1?param2=baz#bar2

5)
base: http://www.blah.com/yadda1/yadda2/yadda3?param1=foo#bar
relative: ..

final absolute: http://www.blah.com/yadda1/

1
2
3
4
5

        Now you should be able to confidently turn any relative URL into an absolute one, as well as know when to use the different forms of relative URL and what the implications will be. For me this has come in handy time and time again in my web development endeavours.

        There you go that's really all there is to know about URLs, it's all relatively simple (_forgive the pun :)_) so no excuse for being unsure about some of this stuff next time. Talking about next time, one of the most common things you need to do when it comes to URLs is recognise if a piece of text is infact a URL, so next time I will show you how to do this using regular expressions (_as well as show you how to pull URLs out of text_). It should be pretty easy to construct a decent regex now that we've got the structure and special characters down. Stay tuned.

        <span style="font-size: 10px; font-family: trebuchet ms;">Images by <a href="http://www.flickr.com/photos/jfreund1/4188863140/" target="_blank">jackfre2</a>, <a href="http://www.flickr.com/photos/beautifulsorta/4515587293/" target="_blank">saucebomb</a> and Steve Nilsen</span>

I love shell productivity hacks. I believe that if you work in a Linux environment, you owe it to yourself to become as good as you can be at working with your shell of choice (e.g. bash). You see, most people who have worked with Linux for any length of time have some level of skill with the shell, that level is usually mediocre at best. But, you do meet the occasional person who wows you with what they can accomplish and I will tell you right now that their skill does not come from superior knowledge (or at least not fully from superior knowledge). It is all about maximizing the effect of the knowledge you do have, finding little tricks and hacks that will save you, potentially less than a second, every time you perform a particular action. Thing is, some actions you might do hundreds of times per day, so those seconds really start to add up, especially once you have accumulated dozens of these hacks. Anyway, here is one such tip.

Running Multiple Commands

Whenever you work with the shell you almost always need to run several commands in a row. I am not talking about piping commands to each other, but just running several in sequence. Surprisingly, people usually tend to wait for each command to execute before running the next one. Why not execute all the commands at once? Bash has some decent support for this, and if you train yourself to do it this way, not only will it potentially save you some time, but it will also force you to think further ahead; all of which will make you that little bit more productive. Let’s have a look.

Firstly we need to create some commands to make it easier to see what is happening. The first one will be called cmd1, and all it will do is sleep for 2 seconds then output something to the screen.

#!/bin/bash
sleep 2
echo "cmd1"

The second command will be called long-cmd and it will sleep for 30 seconds before outputting something to the screen.

#!/bin/bash
sleep 30
echo "long-cmd"

The third command will be called cmd-fail and it will exit with a non-zero exit status to simulate a failing command.

#!/bin/bash
echo "Failing..."
exit 2

We’re now ready to run some commands in sequence by kicking them off at the same time. All you need to do is separate the commands with semicolons and they will execute one after another.

alan@alan-ubuntu-vm:~/tmp$ ./cmd1 ; ./cmd1 ; date
cmd1
cmd1
Sat May  1 23:29:49 EST 2010

As you can see, the two cmd1 commands executed first and then the date command ran – as expected. The only problem here is this, if a preceding command fails, the subsequent ones will still run:

alan@alan-ubuntu-vm:~/tmp$ ./cmd1 ; ./cmd-fail ; date
cmd1
Failing...
Sat May  1 23:32:33 EST 2010

This may be the behaviour we desire, in which case all is well, but what if we do care about the success of the preceding commands. In that case, separate the commands with double ampersand instead of semicolon.

alan@alan-ubuntu-vm:~/tmp$ ./cmd1 && ./cmd-fail && date
cmd1
Failing...

As soon as one of the commands fails, no subsequent command will be executed – handy.

Here is another thought, if you don’t need to run multiple commands in sequence, but simply want to kick off multiple commands, why not execute all of them at once by putting them all in the background. All you need to do is separate each command with a single ampersand:

alan@alan-ubuntu-vm:~/tmp$ ./long-cmd & ./long-cmd & ./long-cmd &
(http://feeds.feedburner.com/softwaretechandmore) 2643
[2] 2644
[3] 2645
alan@alan-ubuntu-vm:~/tmp$

All three of our long commands have kicked off in the background. This works because as soon as we background the first command by putting an ampersand after it, the shell gives us another prompt and is ready to accept more input, so we can keep going and execute as many commands as we want on the same line, by backgrounding them all. It is a slightly quicker way (than waiting for each command to finish, or backgrounding each one separately) to execute multiple commands if they don’t rely on each other. If the commands produce output, all the output will still go to the screen, so the output from all the commands can potentially get mixed up, but you can, of course, still redirect the output of each command somewhere else. You can also easily check on the status of each of the backgrounded commands:

alan@alan-ubuntu-vm:~/tmp$ jobs
(http://feeds.feedburner.com/softwaretechandmore)   Running                 ./long-cmd &
[2]-  Running                 ./long-cmd &
[3]+  Running                 ./long-cmd &

The number in square brackets is the id of the job, and you can use it to perform actions on the running job, such as bringing it to the foreground, or killing it:

alan@alan-ubuntu-vm:~/tmp$ ./long-cmd & ./long-cmd & ./long-cmd &
(http://feeds.feedburner.com/softwaretechandmore) 2679
[2] 2680
[3] 2681
alan@alan-ubuntu-vm:~/tmp$ kill %2
alan@alan-ubuntu-vm:~/tmp$ jobs
(http://feeds.feedburner.com/softwaretechandmore)   Running                 ./long-cmd &
[2]-  Terminated              ./long-cmd
[3]+  Running                 ./long-cmd &
alan@alan-ubuntu-vm:~/tmp$  fg %1
./long-cmd

If you foreground a job by using fg, and want to put it back in the background after, all you need to do is press Ctrl-Z and then background the job again by using its id.

alan@alan-ubuntu-vm:~/tmp$ fg %1
./long-cmd
^Z
(http://feeds.feedburner.com/softwaretechandmore)+  Stopped                 ./long-cmd
alan@alan-ubuntu-vm:~/tmp$ bg %1
(http://feeds.feedburner.com/softwaretechandmore)+ ./long-cmd &
alan@alan-ubuntu-vm:~/tmp$ jobs
(http://feeds.feedburner.com/softwaretechandmore)   Running                 ./long-cmd &
[2]-  Running                 ./long-cmd &
[3]+  Running                 ./long-cmd &
alan@alan-ubuntu-vm:~/tmp$

There you go, not revolutionary, but some handy tips to keep in mind to shave off a few seconds here and there. And believe you me, those seconds really do start to add up after a while. More about shell stuff soon.

Image by Roger Smith