Debugging programs into existence

One day I was playing around with putting together a document and I realised something; when I started the spec as a blank page, I did not think of a plan or decide on what I needed to do, I was in a rush and staring at the page, almost angry with it's blankness. I looked at the page as if the emptiness was a bug and I needed to put something there to fix it. So I started with a paragraph along the lines of what I had to say, wasn't the best but it did the job. From those few lines, I added some text above and then below. Did some refactoring and pasted a section from another document. Slowly but surely, the document took shape until it was ready.

The document was a simple thing, just a high-level overview but the way I put it together reminded me of the type of code that I often see, code that works but looks like it was the result of some nuclear explosion and it dawned on me; that code was not written so much as debugged into existence. The programmer started with some code, then added some code around that, fixing bugs as they go along until the program works. Not quite TDD but its cousin from several incestuous unions.

Now don't get me wrong, these programs do work, but the code quality depends on the mind of the person creating it (ie is the mind organised enough to plan in realtime in the right direction?), the size of the solution and the complexity of the problem space.

It is possible to produce quality code without planning but that is very difficult to do and very few people can pull it off. A better way is to think about and solve the major issues but implement using agile methods so no big planning but not going in blind. I personally like that way of doing things but it does not always work out as I expected (some corner cases and gotchas that my simple plan did not uncover). For certain problem spaces, nothing short of a full plan will do; YMMV.

Do you produce programs by debugging them into existence?

Whiners vs Entrepreneurs

What do you do when the smelly stuff hits the fan? What goes through your mind when users are having problems? Do you whinge, whine, and say things like: "Damn those pesky lusers", or something equally insulting? Perhaps you see an opportunity in the making to shine, or make the user's day better?

It seems that the IT camp, much like American politics, is a two party state.

On the one side are the “Whiners”, those people who think the whole world should operate according to their rhythm. They hate having to deal with problems, especially if they are the cause. A problem is something that must go away, either by dropping it on someone else's lap, or redefining it as a “system” issue that cannot be fixed, or just ignoring it. And when it cannot be ignored, then whine, whine, whine and whine some more. Maybe, they will be forced to do something about it, in that case, something gets done; quickly, dirty and not necessarily fixing the problem but the annoying boss, user, or whoever wielding the whip is satisfied. For goodness sake, they have work to do, and interesting problems to work with, like that Rails 2.0 application, or optimising the database to squeeze an extra 10ms out of it. Important stuff. Whiners do not always whine out loud, but they will do anything to avoid having to work on stuff that is “boring”.

In the other camp, are the “Entrepreneurs”. These guys love problems. Every problem is an opportunity to make their life and the lives of all those around them better. By making the client happy, they gain a good reputation. Problem happens, opportunity happens, and stuff gets done. These people realise that without problems, there cannot be solutions and IT is all about solutions. There is a reason that entrepreneurs do well, they are not afraid of getting the job done, and are willing to take the time to do it well (even though that is boring). Theirs is a career, not a “job”.

Which are you?

Thoughts on Exception Design

Working on my library, I came across an issue that has plagued me in the past. In my quest to provide the most information on failures, I construct my exceptions with detailed messages. This works where book sized messages can be displayed or stored but what happens when the volume of the message hides the relevant details for those that want at it easily? Also, what about exception hierachies, what happens if the same information must be associated with more than one hierarchy?

Here is an example of my dilema - binding values to a database query parameter; three possible error hierarchies can be invoked:

The programmer provided an incorrect index or bind name
SqlException->SqlDatabaseException->SqlProgrammingException

Some database problem prevented the bind
SqlException->SqlDatabaseException->SqlOperationalException

The database interface library caused a problem
SqlException->SqlInterfaceException

I want to supply the following values - Bind index, upper bound for arguments, value to be bound.

Normally, when there is only one hierarchy, I would create a subclass with the required instance variables and a custom toString (or equivalent) implementation. With three hierarchies, I would have to create three subclasses for the same variables, a bit of a waste.

So I decided to do something a bit more generic. I created my base exception class with the concept of properties so you have something like this (pseudo-ish D code):

class Exception {
  typeof(this) setProperty (T)(string key, T value)
  Box          getProperty (key)
  bool         hasProperty (key)
  Box[string]  propertyBag
}

The setProperty returns the type of the exception to allow for chained method calling.

Since index range errors are common developer errors, I decided to create a exception class for that case so I have:

SqlException
  ->SqlDatabaseException
      ->SqlProgrammingException->SqlBindException
      ->SqlOperationalException
  ->SqlInterfaceException

For the standard bind error, I instantiate the SqlBindException as such:

throw (new SqlBindException ("Bind index out of range"))
  .setProperty ("BindPosition", idx)
  .setProperty ("BindType", "TEXT")
  .setProperty ("BindUBound", ubound)
  .setProperty ("BindValue", value)
  .setProperty ("BindValueType", valueType)
  .setProperty ("SQL", sqlText);

If there is a database error, then I can do the following:

throw (new SqlOperationalException ("Something went wrong"))
  .setProperty ("BindPosition", idx)
  .setProperty ("BindType", "TEXT")
  .setProperty ("BindUBound", ubound)
  .setProperty ("BindValue", value)
  .setProperty ("BindValueType", valueType)
  .setProperty ("SQL", sqlText)
  .setProperty ("VendorCode", vendorCode)
  .setProperty ("VendorMsg", vendorMsg);

or if the interface is in an inconsistent state with the database:

throw (new SqlInterfaceException ("The interface is broken"))
  .setProperty ("BindPosition", idx)
  .setProperty ("BindType", "TEXT")
  .setProperty ("BindUBound", ubound)
  .setProperty ("BindValue", value)
  .setProperty ("BindValueType", valueType)
  .setProperty ("SQL", sqlText)
  .setProperty ("VendorCode", vendorCode)
  .setProperty ("VendorMsg", vendorMsg);

With the modified toString, I get the following:

SqlBindException: Bind index out of range

BindPosition: 4
BindType: TEXT
BindUBound: 3
BindValue: testval
BindValueType: string
SQL: select ... from ... where x=?, y=?, z=?;

By separating the properties from the message, I have absolute control over how I will display, or log, the information.

And that helps with the other problem: long messages versus terse description.

The above SqlBindException has a very terse description, the one that is common to sequence type errors. I have seen many instances of 'array bounds out of range' type errors. Of course, with the properties, the message can be terse but the information necessary to track the bug is available.

Let's say, I have the following terse error message:

"Failed to bind parameter"

Not very descriptive. Now let's say that I wanted to provide a default user facing error message for developers to use, or a more descriptive message aimed at developers.

"Failed to bind "~bindType~" parameter, either the statement is finalized, or the database is in an inconsistent state. If the statement is finalized, then the interface is out of sync with the database. If the database is in an inconsistent state, then this may be an interface error or a problem with SQLite. If the problem can be replicated, please log an issue with a test case."

To support long messages, I have added them to the base class:

class Exception {
  typeof(this) setProperty (T)(string key, T value)
  Box          getProperty (key)
  bool         hasProperty (key)
  Box[string]  propertyBag

  typeof(this) setLongMsg (string longMsg)
  string       longMsg;
}

Besides the length of message, long messages differ from the normal exception messages in that they support properties with the "$()" syntax. The above message is set as a long message as follows:

throw (new SqlOperationalException ("Failed to bind parameter"))
  .setLongMsg (
    "Failed to bind $(BindType) parameter, either the statement "
    "is finalized, or the database is in an inconsistent "
    "state. If the statement is finalized, then the interface "
    "is out of sync with the database. If the database is in "
    "an inconsistent state, then this may be an interface "
    "error or a problem with SQLite. If the problem can be "
    "replicated, please log an issue with a test case.")
  .setProperty ("BindPosition", idx)
  .setProperty ("BindType", "TEXT")
  .setProperty ("BindUBound", ubound)
  .setProperty ("BindValue", value)
  .setProperty ("BindValueType", valueType)
  .setProperty ("SQL", sqlText)
  .setProperty ("VendorCode", vendorCode)
  .setProperty ("VendorMsg", vendorMsg);

When the instance property longMsg is invoked, the message is returned as follows:

Failed to bind TEXT parameter, either the statement is finalized, or the database is in an inconsistent state. If the statement is finalized, then the interface is out of sync with the database. If the database is in an inconsistent state, then this may be an interface error or a problem with SQLite. If the problem can be replicated, please log an issue with a test case.

Properties allow me to do something a little different with long error messages. When chaining exceptions, I can construct a long message that uses properties from the chained exception. Eg:

try {
  throw (new SomeException ("Inner exception"))
    .setProperty ("A", aValue);

} catch (SomeException e) {
    throw (new OtherException ("Outer exception", e))
      .setLongMsg (
          "Blah blah blah $(A) blah blah $(B).")
      .setProperty ("B", bValue);
}

I have found that the concept of exception properties to be a valuable tool for exception flexibility; something that is making my life a bit easier.

Preparing For Winter

Recently, I have begun to think of all the things I do for development, as preparing for winter. My rationale is simple.

I live in South Africa and, except for a few parts. the weather here is mild compared to Europe. Imagine the land four thousand years ago, wide open spaces, plentiful food, water, and little fear of freezing to death. The easy life. Can you imagine yourself in this place, working your behind off doing anything? In short, there is no necessity for exerting much effort except for fun and profit.

Contrast Europe. At least once a year, it gets so cold, that you can die unless you find some way to retain heat. No loin-cloths for these people, only the real deal will pass the muster. In this really cold period, food is hard to come by. In fact, if you did not think ahead, you are likely to perish. In summer, the trees are full, food is plentiful, water flows and life is good. Do you sit and enjoy what life has, or do you prepare for winter?

The clever ones get it over and done with as quickly as possible, and optimise the process to require the least amount of time, and energy, leaving the rest to enjoy what life offers. The stupid ones intend on doing all that but would rather do it tomorrow, maybe the next day, at least some time before it becomes necessary but then they usually end up spending more effort, taking more time and having a poorer result because of it (crunch time). The really stupid ones work hard all day, all week, never enjoy themselves but cannot seem to get it right and then have a miserable winter (they usually survive but don't really want to).

Preparing for winter is an analogy of things that we should do, but have better things that we want to do. Needs versus desires. What it comes down to is a simple question: Do you want to just survive this winter, or do you want to pass the time in comfort and have the means to enjoy it?

I have my own terminology for this. The people who do just what they think is enough to survive, are slackers. The people who work hard but don't make life easier for themselves, are gears. The people who plan and work smartly to make life easier for themselves (and generally for others), are farmers.

Preparing for winter, is a life decision whether for an individual, group, or organisation. It sets a vibe and like magnets, it has polarity; slackers and farmers don't mix, gears are neutral and can mix with either slackers or farmers. Just to make things more interesting, this is not an all or nothing type; most people are a mix of slacker, gear and farmer.

Preparing for winter and making the most of it is about effort, planning and the right tools for the job. All things that I hope to do right in my IT projects.