A second hazard arises in that, by looking for language-specific experience, this practice contains an implicit assumption that programming languages stand alone, with little relation to one another, and that learning a new one is a major undertaking.

These issues are two sides of the same coin, both flawed in their failure to recognize that, while programming languages are the means used to develop software, they are not the primary skills of software development.

The Right Tool For The Job

A commonly-used metaphor, and one which would have been appropriate to the points made in Why Do You Hire Programmers?, is that of a carpenter’s toolbox. “When you want to build a house, you look for a carpenter who builds houses, not a carpenter who has 5 years of experience using hammers and 2 years using nailguns with preference given to candidates who are also familiar with screwdrivers.”

While this metaphor does make a good point, that the craftsman’s skills go beyond the specific tools used and that you should trust him to use the right tool at the right time, I still have minor issues with it in that it preserves the misconception that each software development tool is fundamentally different from the others. Although there is some overlap between the skills of using hammers, nailguns, or screwdrivers, they are, by and large, three distinct activities. The skills of writing software in C, Java, or Smalltalk, on the other hand, overlap by much more than they differ.

Something A Little Less Prosaic

A much more apt metaphor would be to compare the programmer to the poet.

Skilled poets can write in many forms. A master of the sonnet can also be expected to competently produce ballads or limericks when called for and would likely be able to write a mean haiku with very little practice. The core skills of handling meter and rhyme or choosing the right words to paint a vivid image apply broadly across all poetic forms.

Similarly, a well-rounded developer is proficient in a variety of programming languages and can apply his core skills in logic and program structure to new languages with a minimum of difficulty. The techniques do vary from one language to another, with some placing more emphasis on one type of logic and others emphasizing structure more heavily, just as rhyme is everything to a limerick and irrelevant to a haiku, but this is generally a minor obstacle, if any at all.

Learning The Language

There is a common case where this falls apart, however: Programmers who only know a single language. Although it’s not inevitable, this often indicates someone who has solely learned that language without also learning the more general techniques of software development and how they apply beyond the one language.

The primary skills needed in software development are ways of thinking, of finding solutions for problems, and of structuring those solutions. They are not the vocabulary or syntax of particular programming languages. You will often be much better off with someone who has extensive experience with a wide range of technologies, but doesn’t know your particular language, than with someone who has extensive experience with your language, but has never used anything else.

1. A Correction To Last Tuesday’s Post

In Off the Record: Passwords, I recommended the use of SHA1 rather than MD5 hashes when storing passwords. Since then, I have encountered a persuasive argument in favor of abandoning both of them and using bcrypt instead, as it’s designed to be less time-efficient, thus dramatically reducing the number of potential passwords that a brute-force attack can attempt in a given amount of time.

As this can make your passwords much more secure against attackers while still keeping single-hash generation running at a reasonable pace, I have revised that post to recommend the use of bcrypt over SHA1 where available.

2. I’m Not Disappearing

New posts to this blog have slowed down substantially over the last couple weeks because the stockpile which I prepared prior to launching the blog have all been published and my new writing process, which I expect to work out much better in the long run, is taking longer than the old one to produce completed posts, ready to publish. If all goes as it appears that it will, the rate of new posts should pick up again by the end of next week if not sooner.

Although I didn’t say a word, I just about died on the spot. Although he wasn’t aware of it, he now knew the PIN for my bank account!

Granted, I should have known better than to reuse a PIN (and I do know better now!), but it’s a very common practice nonetheless, especially as the number of PINs and passwords that each of us needs to keep track of seems to be increasing daily.

Back To The Present

More recently, there has been discussion on the debian-user mailing list of ways to log the passwords entered on unsuccessful login attempts. While this seems innocuous at first glance, and an effective way of seeing what passwords are being tried by brute-force attackers, it will also catch passwords entered incorrectly by legitimate users. These mistyped passwords will generally only be off by a letter or two, thus giving anyone with access to these logs the ability to much more easily guess the correct version.

Despite the potential advantages in being able to revise your password requirements to improve their strength against the latest dictionary attacks, the issue of password reuse remains. These kinds of system logs are generally locked down well enough that they’re only accessible to system administrators whose legitimate powers are broad enough that knowing other users’ passwords will not allow them to do any additional damage locally beyond what they can already do using their own account, but granting them knowledge of others’ passwords also potentially allows them to impersonate the user on other systems.

Trusting them with this information is not necessary for them to perform their legitimate duties. It allows them to do additional harm, but does not expand their powers for good. Therefore, it should be kept from them and any security-conscious sysadmin or developer should recognize this. (Personally, I have, on several occasions, cut off users who were about to tell me their passwords and explained that I would not allow them to do so.)

Avoiding Password Leakage

To maintain secrecy, user-entered passwords should never be recorded in plaintext. I can’t think of any situations in which there would be a legitimate need to be able to recover an actual user password, but, should such a case exist, there are plenty of good, reversible encryption methods which can be used to store it, safe from prying eyes or casual glances.

The proper and generally-accepted method of dealing with passwords is to pass them through a non-reversible cryptographic hashing function, such as SHA1¹ bcrypt², then storing the hash rather than the password itself. Use of reversible encryption would still leave the password vulnerable to anyone with access to the encryption key and the will to use it. A SHA1 bcrypt hash allows you to determine whether the entered password matches the correct password without ever knowing (or being able to know) what the correct password is.

Afterword: Honeypots and Testing

I did mention, above, that there are potential benefits to being able to see what passwords are being attempted by attackers. If this information is needed, it can be obtained without compromising the security of your actual users’ passwords by setting up a honeypot system with no valid user accounts (or at least none which accept password-based logins over the network). Since no legitimate logins will be attempted, you can safely record the passwords without giving away information about legitimate users’ accounts.

The other situation in which I’ve seen it suggested that it is appropriate to record passwords is when testing or debugging software under development to ensure that input (including the password) is being received and processed correctly. Safeguarding actual user credentials is again easily done by means of not having any actual user data (or at least not any actual passwords) in the testing environment.

² Shortly after making this post, I ran across this article, which makes an excellent case for using bcrypt rather than MD5 or SHA1. If a bcrypt library is available in the environment where you’re operating, use it. If it’s not available, see if you can get it added.

The First Rule of Program Optimization: Don’t do it.
The Second Rule of Program Optimization (for experts only!): Don’t do it yet.”
- Michael A. Jackson

If you spend much time with people who have any involvement with software development, you’re going to run across a conversation about optimizing software. Either the program is too big or too slow or too chatty on the network or too something and somebody wants to do a little optimization to improve on that.

It usually starts with the natural impulse of geeks to tinker with their systems, but project leads, marketing departments, and managers are often drawn in by the promise of a better product.

Should you find yourself in that position, contemplating either undertaking an optimization project or authorizing one, stop and consider what the proposed optimization is and how much it will actually improve things.

Optimizing Line-By-Line Is Rarely Worth It

Micro-optimizations are generally not worth the time it takes to make them. Shaving off a couple milliseconds from a function that only runs once is entirely pointless - it will produce no perceptible improvement, yet it comes at a potentially high cost in development time. Simple optimizations of this type are often already done automatically by the programming language behind the scenes anyhow, while more complex attempts may backfire.

The primary exception to this is when the code in question will be repeated many, many times in rapid succession. (Programmers often call this a “tight loop”.) If the code will run 10,000 times, then making it take a millisecond less each time will save a total of 10 seconds. If your user is sitting and waiting for the task to complete, that 10 seconds is an eternity. On the other hand, if it’s part of a 6-hour non-interactive process to close out your monthly books, the 10 seconds saved is meaningless despite the repetition.

Optimizing Algorithms Works Much Better

Several years ago, as I was just starting my programming career, I had a job doing data entry on a system which needed to run a check for duplicate records before posting completed tasks into its archival database. For performance reasons, it checked only those active records which were marked complete - and it still took nearly half an hour to run.

Eventually, the programmer who wrote the system left the company and I inherited his responsibilities for it. One of the first things I did was take a hard look at the duplicate checking code.

The duplicate check, as originally written, looked at every single record in the archival database for each active record that was being checked. Testing 10 active records for duplication against a 10,000-record archive database required 100,000 record-level comparisons. No wonder it was slow. It used an extremely inefficient algorithm.

Within a day, I had rewritten it with a better algorithm which only needed to make one pass over each database. With 1,000 active records and 10,000 archived, it could test every active record (not just the completed ones) with only around 11,000 record-level comparisons. It also used the databases more efficiently, bringing total run time down to roughly 15 seconds. Vastly improved performance, plus a more thorough check. A truly worthwhile optimization!

The Most Important Optimization

If I were to revise the original version of that duplicate check today, I could do even better. I’m confident that I could get it under 5 seconds and probably down into the 1-2 second range, while still running on the same, now horribly outdated, hardware and software.

Why are such extreme improvements possible? Because, at the time, I was able to devise a better algorithm than the original programmer had and because I now have several more years experience behind me than I did then.

If presented with my revised version, though, I would argue against further optimization of that code. Since it was just run once a week, it wouldn’t be worth the effort involved in bringing it down from 15 seconds to 5, maybe not even if it could get down to 1. Rewriting to eliminate the need to run the check at all might be worthwhile, but making the check faster would not.

The most important optimization is to optimize the skills and experience of your developers. Software development is not a commodity product. Getting someone with the skill to choose the right algorithms and the experience to know when an optimization wouldn’t produce sufficient improvement to justify the time invested will get you better results, in less time, and often for a lower overall cost.

Incorrect Validation Is Worse Than No Validation

A recurrent question I’ve seen on a couple different programming sites is “What regular expression can I use to validate an email address?” They’ve gotten a lot of responses explaining ways to examine an email address and reject it if it’s “invalid”. The only problem is that every one of them that tried to be more restrictive than just checking for the presence of an “@” was wrong.

The most common flaw was failing to accept “+” in the local-part of an address, causing many perfectly valid, deliverable email addresses to be rejected. There were also several which failed on top-level domains (TLDs) which were four or more characters long, such as .info or .museum.

On the flip side, there were users accustomed to using disposable email addresses of the form username+foo@domain.com or with addresses automatically generated from their surname of O’Malley - including the apostrophe - complaining about sites refusing to accept their “invalid” (but syntactically correct and perfectly functional) email addresses.

Rejecting these addresses may only shut out a percent or two of the world, but is that really an acceptable cost to pay in an attempt to shut out truly bogus addresses? Particularly given that…

Valid Does Not Imply Correct

Let’s say that you’ve managed to find yourself a perfect validation algorithm which is 100% accurate at determining whether email addresses are syntactically valid, even for the O’Malleys. You’re getting only good email addresses, right?

Wrong.

abcde@fghi.jkl is syntactically valid, but utterly bogus. There isn’t even a .jkl TLD!

“OK,” you say, “I’ll just add a DNS lookup to my perfect validator so that it rejects domains that don’t exist or don’t have a mail exchanger (MX) record defined.”

Still no good. dave.sherohman@whitehouse.gov is syntactically valid. It points to a domain that exists and accepts mail. It’s still no good.

“No problem. I’ll connect to the mail server and validate that the user account exists.”

I can still give you a bad email address. Try president@whitehouse.gov on for size. The account exists, but it’s certainly not mine.

If There’s Any Chance Of An Incorrect Rejection, Don’t Pre-Validate

The only way to authoritatively validate the correctness of an email address is to send email to the address and request that the user respond to it in some manner to confirm receipt.

You may still benefit from doing a preliminary validation to avoid the trouble of sending out mail that could never possibly be received, but this pre-validation cannot reasonably be considered authoritative and should, therefore, only be used to filter out the most blatantly incorrect cases. Attempting to make it more restrictive will only introduce an unnecessary risk of rejecting genuine, correct, deliverable addresses.

The cost of performing the authoritative validation is low and the cost of rejecting an address which is incorrectly flagged as invalid by a flawed pre-validation is high, so don’t pre-validate beyond the point at which you are absolutely, 100% certain that it will not fail anything which might pass the authoritative validation.

Be liberal in what you accept, and conservative in what you send.
- Postel’s Prescription

Previous posts here have discussed reducing the burden of data entry on your application’s users by cutting down on the number of items that they are required to provide and, even if something is required, allowing it to remain incomplete for as long as possible, only enforcing that “required” status when the time comes that it is actually needed.

But what about the values that are entered, whether required or not?

The next step in minimizing the burden placed upon your users is to “be liberal in what you accept”. Do not impose validation rules which must be satisfied unless they are unavoidable. If you really must require them, make sure that the rules themselves are complete and correct.

Don’t Validate Formatting

The biggest, most common, and most annoying case of pointless validation rules are those which impose restrictions on how the user must format his entry. Any programming language worth its salt can effortlessly remove spaces from a string or insert a space after every fourth digit, so there is absolutely no good reason for credit card validations to care whether you enter the card number as “1234 5678 9012 3456″ or as “1234567890123456″, or even “12 3 45678901 234 56″.

Phone numbers can trip you up here, too. Does the program want them entered as (555)555-1212 or 555-555-1212 or +15555551212 or what? The correct answer, again, is that it should accept any of these formats (and more), which is easily implemented by simply removing any non-numeric characters and then adding or removing a leading “1″, depending on whether you want the country code in your database. This will also accommodate international phone numbers, which may have their digits grouped in patterns you don’t expect.

As long as it’s the right set of digits, let the user group them however he wants to.

Remember The Rest Of The World

When I moved from the US to Sweden, I paid my bank a visit to change my address with them. It went reasonably well and their system was smart enough to recognize that, since my new address was outside the US, the “state” could be left blank.

But it still required a ZIP code. Beyond that, it required the ZIP code to be formatted as a string of five digits with no intervening characters. While it does just so happen that my Swedish postal code is 5 digits, they’re grouped “226 47″. Also, the postal code goes before the city here, so we ended up having to enter the city as “226 47 Lund”, with a ZIP code of “00000″. This is obviously incorrect, but it was the only option for satisfying the US-centric validation rules.

Trust the User

Speaking of ZIP codes, ZIP code-to-city databases are pretty nice. They can be great for letting users enter their ZIP code and then automatically filling in a default city for them.

Note the word “default“. The last US city I lived in was Eagan, MN, ZIP code 55122. About half the ZIP code databases out there correctly list 55122 as Eagan. The other half list it as St. Paul, which is a good 10 miles north of Eagan. I knew I lived in Eagan, and the post office knew I lived in Eagan, but several websites insisted I was in St. Paul and provided no way for me to correct their mistake.

Looking up data instead of making the user enter it is good, but your database will be wrong sometimes. Allow the user to correct the lookup results when this happens.

 
I do have a bit more to say on ways that data validation can go wrong, but that’s a topic for another post. Until next time, what issues have you run into where too-strict or too-lax data validation has caused you problems?

Going open source can bring major benefits for your software and your company, but it can also invite disaster if you’re not familiar with both the concepts and the culture behind it. So let’s clear up some of the common misconceptions surrounding open source:

You Still Own Your Code

Whatever code you write (or pay someone to write for you as a “work for hire”) belongs to you, unless other arrangements are made. As the owner of the copyright on that code, you will never be legally bound by the terms of the code’s license, so you will still be able to use it in whatever non-open way you might desire.

Until you accept someone else’s patch.

If you release a FOSS project and I submit code for it, then I still own the copyright on my submission. By incorporating my changes into your project, you become bound by the project’s licensing terms because you no longer own the whole thing. The one exception to this is if I assign rights over my contribution to you above and beyond those in the FOSS license, most often through the use of a Contributor License Agreement.

You Can Sell FOSS

FOSS licenses place no restrictions on who can distribute the software covered by the license or how. Their restrictions apply to those who make changes to it. Depending on the license, anyone making modifications may be required to do such things as:

• make the source code available to anyone they distribute the software to
• provide attribution to the original author
• submit the changes back to the original author

 
Even if you are subject to the FOSS license, you can still sell or give the software to whoever you like. But they can also sell or give it to whoever they like, so the cost will tend to drop to zero rather quickly unless you can provide some additional value to encourage people to pay you for your version when they can get it free (and legally) from someone else.

One common way of doing this is to make use of a Contributor License Agreement, as mentioned above, which allows the original copyright holder to sell the project under a commercial license in addition to free distribution under a FOSS license.

Open Source Is Sharing

Open source is not a gimmick for getting programmers around the world to write a program for free which you can then turn around and sell. Unless the programmers gain some benefit from contributing to your project, they are unlikely to do so and, worse, you may get bad press in the FOSS community for trying to take advantage of them.

For this reason, FOSS licensing tends to work better for software which potential developers are likely to run on their own computers, not for the custom portions of an online service or other “unique” offering that’s intended to only run in one place.

I recently advised someone who wanted to set up a web-based service and sell subscriptions for access, but also to make it open source so that outside developers would contribute and improve on it. Although basing the commodity portions of such a system on existing FOSS projects works extremely well - that’s the heart of LAMP, after all - trying to present the service itself as FOSS would have gone quite poorly.

I explained that developers would have no reason to make free contributions to his for-pay service and, even if they did, they would need to set up their own working installations of the service to develop against, at which point they would be in a position to compete against him by offering the same service at no cost. This was a case for which FOSS was simply not well-suited to his objectives.

 
Have you considered using FOSS licenses for any of your projects or do you have further questions about FOSS which have not been addressed here?

If you’ve done much software development, that statement alone should be enough to have you thinking “concurrency issue” or, more specifically, “race condition”. Given that the application involved both a web-based management interface and a once-a-minute scheduled task to handle accounting, the obvious point of conflict was for the problems to result when an operator submitted an update through the web interface while the accounting task was running.

So off I went to ensure that such a conflict wouldn’t cause issues, then sent the revised code off to the client. He reported back the next day that it hadn’t had any effect.

After looking harder at the code, I came up with a way that this conflict could conceivably have still occurred under some contrived circumstance, eliminated that possibility, and sent it off. Still no improvement.

We went through maybe half a dozen rounds of this before I ran out of ways that the web application and accounting task could conflict, no matter how far I stretched the bounds of probability, and finally took a closer look at the application’s logs. Once I started looking at them as a whole, instead of just records of single users, I quickly noticed that these mischarges came in clusters, affecting several users at once, not random individuals. These clusters also tended to hit right on the hour.

Checking in with the client, it turned out that the specific times when the problems occurred matched up with the times when the per-minute rates changed.

To keep the accounting simple, the application deals with rate changes by closing out each active user’s session, charging it at the old rate, and then opening a new session at the new rate. With larger numbers of users logged in, this process was taking more than a minute to complete. The conflict wasn’t between the web interface and the scheduled accounting task, it was between two (or more…) copies of the accounting task! This also explained why I was never able to reproduce the problem in my own development and testing environment, as I had never simulated enough concurrent users in my tests to produce the problem.

Once the correct culprit had been identified, it was a simple enough matter to resolve by adding a check to prevent multiple instances of the accounting task from running concurrently.

The moral of the story:

Even if the cause of an issue seems clearly obvious, don’t forget to fully examine the system to verify that you’re solving the right problem before you spend too much time solving the wrong one.

It doesn’t much matter whether your project is the latest addition to the world’s surplus of data-entry systems or the next killer application which will revolutionize our lives, it’s going to involve a lot of common functions either way. Common functions which have already been implemented, dozens if not hundreds of times over. How much of that can and should you make use of?

There are existing frameworks which handle these common functions in broadly generic ways which can generally be customized to get an application based on them up and running relatively quickly and at a low cost. Most good frameworks also allow for plug-ins to expand on their base feature set similarly easily.

The other major option is to build a fully-custom application. This route is more expensive and takes longer to complete, but it ensures that you can get any desired features without being restricted by a framework’s underlying design or the availability of plug-ins. In the long term, it also can provide greater flexibility for your application if it is done correctly.

In some cases, it can be appropriate to combine both approaches, first building a framework-based prototype, then replacing it with a fully-custom final version. Due to the additional time and expense involved, this is generally only an option for large or business-defining projects, but the improved quality of the final result can justify those costs in such cases.

Characteristics of Framework-Based Development

• Faster initial implementation
• Lower cost
• Provides a solid, well-tested foundation for basic functionality
• Potentially more secure, due to lessons learned from attacks against other applications based on the same framework
• Many good frameworks are available for free under Open Source licenses, but some licenses may place requirements on how you use framework-based code
• Some frameworks include a “standard” look and feel, which is good for getting something that looks decent built quickly, but may ultimately limit design options

Characteristics of Fully-Custom Development

• Can produce whatever you desire
• Higher performance, as it will contain only what you need
• Potentially more secure, due to not being a well-known and thoroughly-analyzed target, provided the developer pays attention to security
• You will fully own the end result and can set your own licensing terms

 
What other advantages or disadvantages of either approach are there beyond what I’ve listed? Which do you tend to prefer?

I’m often surprised at how readily people will forget this.

In late January, I ran across a question on LinkedIn which related to a website the asker had developed which was 100% AJAX-driven. The site itself looked nice enough, but it was a disaster technically. Unless JavaScript was enabled, it didn’t even display its front-page content, just a (non-functional) navigation bar.

With JavaScript on, the content did appear, but it turned out to be nothing more than a standard six-page site which could have been done statically with plain HTML pages and regular links to navigate among them without using JavaScript at all. The only thing lost would have been having the old page’s content fade out, then the new page fade in, when you navigate from one page to another.

Yes, the fade in/out is a nice effect. Yes, it looks cool. And, yes, AJAX is a very buzzwordy “Web 2.0″ technology. But it was used all wrong on this site. It added nothing to the functionality or usability of the site, at the cost of making it inaccessible to a substantial segment of potential users¹. If you’re into SEO, this choice of technology also killed that by making the site essentially invisible to search engines.

I’m sure you’ve also seen sites which use Flash, SilverLight, or another “more interactive” technology in a way which does nothing beyond what traditional HTML can provide and makes no use of interactivity at all. Once again, such designs limit the ability of users to access the site and its content, shut out search engines from indexing the site, and add no actual value.

This is not to say that AJAX or Flash or SilverLight or whatever are intrinsically evil, or even suspect, but rather that you need to know what you’re trying to accomplish with your site or software and then choose appropriate technology - and an appropriate use of that technology - to support that goal. Just using technology for its own sake or because it’s the hottest thing at the moment will, in 99% of cases, do nothing to support the software’s purpose and is likely to detract from it.

What’s the most useless (mis-)use of technology you’ve seen lately?

If your answer is “to write code”, then think about it some more. I don’t know what your reason is, but I’m pretty sure that’s not it. Writing code is merely a means to an end, not an end in itself.

Do you want to create internal systems that keep your company running? Or are you looking to produce a product for sale to customers? Maybe you’re doing something else entirely?

I’ve seen this question presented from the marketing side a few times, usually in terms of “features” vs. “benefits”¹, but the same principle applies here. Just as you need to present a benefit to your customers when selling to them, you also need to consider the benefit that you want to obtain when you’re buying.

Probably the most common issues I see in this area are companies looking for a programmer who knows a specific language, or uses a specific database, or has experience with some other particular tool rather than focusing on the actual objective of the project: the functionality that they require.

This is not to say that you should never look for someone who can use specific tools. If you’re adding on to an existing Java application or if you have an existing team of Java developers that you mean to add to, then you’re probably going to want someone who can work with Java, and for good reason.

If you’re building something new, though, then you’re likely to be better served by finding someone with broad experience and the ability to create what you’re looking for, then asking him what the most appropriate tools would be. Since you’re hiring an expert, use his expertise. Any genuine expert should be able to clearly explain the advantages and disadvantages of different options to you in terms you can understand, no matter how technical or non-technical those terms may be.

Just don’t be surprised if the best option ends up not being the one that you would have chosen if you had started by picking the tool.

What have you hired programmers for in the past and what would you want to hire one for today?

¹ Basically, the concept is that you don’t want to sell the feature “our product has X”, but instead you should sell the benefit that “our product allows you to do Y”.

Looking back over it, I have to say that I really enjoyed this project. Not only was CyberPenguin’s owner great to work with and provided some of the best specs I’ve seen from any of my clients, but he was also always willing to discuss their content and to consider any suggestions I had for improving on his design.

Features

The case study itself provides fairly complete lists of the original functionality and what was added in the expansion project. The only thing that stands out as missing was the one part that I was least sure of being able to work in smoothly: Promotions.

The original spec called for an extremely flexible promotions structure, allowing operators to define various combinations of times of day, days of the week/month, minimum purchase amounts, etc. under which a user receives either a flat bonus amount or a percentage bonus to purchases of credit on his account.

Previous experience with calendars and scheduling software gave me a pretty good idea of how to handle the infrastructure behind the promotions functionality, and even to make it more flexible, allowing the criteria for promotion eligibility to be mixed and matched in any way desired rather than only the specified combinations. But I expected designing an interface to control this flexibility without overwhelming the user to be a somewhat more daunting task.

In the end, I was able to come up with a clear, simple way for users to specify a promotion to be, say, “Top up account by P10.00 and get 15% extra from midnight until 6:00 every 4th Monday, Wednesday, Friday.” And, yes, the interface is also able to describe the promotion details in English - that sentence was copied and pasted directly from it.

Technology

CyberPenguin runs on Ubuntu, which is derived from Debian GNU/Linux - my preferred operating system - so I was already very familiar with the server environment that I’d be working with. The workstations also run Ubuntu, under the control of DRBL (”Diskless Remote Boot in Linux”), which was new to me, but I was already familiar with other Linux-based thin client and diskless workstation setups, so it didn’t present any issues.

In addition to the Linux operating system, the rest of the application was built on Apache 2, MySQL 5, and Perl 5.8, making up a standard LAMP (Linux/Apache/MySQL/Perl) environment. In addition to LAMP for the web interface, the application makes use of Perl programs run as scheduled tasks (by cron) each minute and each day to handle the billing and accounting details.

While that may be a good negative reason (”there’s no reason not to”), there are also positive reasons for allowing this:

It makes your users’ lives easier.

Are you perfectly organized at all times? I know I’m not.

Requiring that saved data must be complete and valid (or at least valid-looking) at all times demands perfect organization from your users. It makes them collect all of the information you want up front and have it all available at the same time when they enter it. Removing that requirement means that they can enter as much as they know right now, then find the rest and come back later to finish.

It provides a record of user actions which were started, but not completed.

Suppose you have an online store for which users go through a series of three pages in the course of placing an order. Do you persistently store the partial data at each stage of the process or do you just keep it in the active session and write it all to the database at once when the final form is complete?

If you’re smart, you’ll make a permanent record of it at each stage. This information about what items a user started the purchase process for, but didn’t actually buy, can provide valuable insight into buying habits. If you’re using a recommendation engine (”people who bought X also liked Y”), then information about what users almost bought is almost as valuable as information about what they actually bought.

More importantly, though, this incomplete transaction data can highlight issues with your site’s effectiveness. If 90% of your users are starting a transaction, but abandoning it after the second step, then perhaps you should take a hard look at the third step (and, while you’re at it, make sure that the server is completing step two in a timely fashion) to determine why they’re not continuing at that point.

Incomplete or invalid data can show what mistakes your users are making.

Users will make mistakes and they will submit invalid data. That’s as inevitable as death and taxes.

Most software ignores invalid submissions, beyond throwing it back at the user along with an error message.

If yours saves it anyhow, though, then these invalid submissions give you a window into your users’ experience that will show you what mistakes they’re making. Once you know what mistakes are being made, you can attempt to determine the cause of the mistakes and make it easier for the users to get it right the first time instead of just scolding them for doing it wrong.

Like the data I’ve been talking about, I’m sure that this post is also incomplete. What other benefits or drawbacks are there to saving partial or incorrect data instead of rejecting it?

But the latest, of course, is also the least stable. The most likely to have problems. The biggest risk.

Staying on top of the latest-and-greatest tech works out well on your own time, at least for those of us who like to tinker and don’t mind doing a bit of troubleshooting when the inevitable problems arise. As a professional, though, that’s rarely what’s best for clients.

On the other hand, stagnation isn’t good for anyone. Newer technology is needed to take advantage of the improved tools and techniques which are constantly being developed. Fifty-year-old mainframes running code in languages just as old are not a strong target for new development today, no matter how stable they may be.

This creates a constant tension between the desire to make use of the new and the need to retain the reliability of the not-so-new. It’s a tricky balance to get right and I see a lot of software developers who give in to the lure of the latest version, only to get burned when it crashes.

I even succumbed recently myself…

About two months ago, I was debating with myself whether to clean up the web application framework that has grown up out of several projects I’ve done and continue using it or to switch over to an existing framework. (I had previously looked at the major existing frameworks and found them wanting, but I make a habit of checking in on them once or twice a year to see if either they or I have changed enough that I’d want to use them.) As it turned out, I found that there was a new framework in town, created by a well-known developer with experience in such things, which avoided many of the pitfalls of the existing frameworks, so I decided to pitch in and help out with development and debugging.

It turned out to be much less complete than I had initially believed and, a month later - practically to the day - it pretty much disappeared.

During the time I was initially evaluating it, I had also been discussing a project with a potential new client and had mentioned it to her. She also thought it sounded great and was very interested in having me base her project on this new framework. Fortunately for me, the project ultimately didn’t happen, because, if I had committed myself to build it based on this shiny new framework, I would not have been able to do so without having to invest far more work than expected into completing the framework first.

How do you prefer to maintain a balance between the latest-and-greatest and the tried-and-true? Have you ever been bitten by this preference?

This is the wrong approach. As application designers and developers, we should be minimizing the effort we require from our users, not adding to it.

I recently encountered a blog post on Rethinking the Comment Form which, among other things, suggested that there’s no reason for a blog comment form to require - or even request, really - an email address unless the user requests to be notified of additional comments. This is a step in the right direction and provides my first principle for minimizing requirements:

If you will never use the information, don’t request it.

In most cases, that can be expanded slightly to “if you will never use the information for the user’s benefit“. If your only use for the information is to sell it to spammers or “marketing partners”, that doesn’t count.

The second principle expands slightly on the first:

If you won’t use the information immediately, don’t require it.

The first was pretty conventional, at least if you’re not talking to marketers. The second is a bit bigger of a step. It means that, if you’re not going to be sending someone postal mail today, then it’s perfectly OK for them to enter an address with no ZIP code. Or a phone number with no area code.

Yes, yes, I know… “What about data integrity?!?”

Which is a good question. What about data integrity? If you’re not going to use the information yet, then why does it matter? Sure, you definitely do need to get that ZIP code before sending them a letter, but, until you have a letter to send? So what if you don’t have it?

This then leads to:

Even if you do need the information, accept its omission.

Let the user leave it blank. Really. You can remind them each time they log in about important information that needs to be completed, but there is no excuse for refusing to let users enter partial information now and then come back and finish it up later.

The only potential exceptions are a username and either a password or an email address (so you can send them a password). This is sufficient to allow them to return and provide whatever other information may be needed at a later time.

Of course, if you use the email address as their username, then that leaves exactly one piece of required information. You can’t get much more minimal than that.

Do you have any other suggestions for good ways to minimize the load on your users of providing information?

There was a recent post on Freelance Switch titled Questions & Answers: How to Describe What You Do on this general topic, so apparently it’s a problem which is shared by others with closer-to-mainstream specialties. Here are my answers to that article’s questions:

1. What kind of work do you do?

I write software to help clients manage information, technology, and/or business processes. Given the current state of the business world, this very frequently means writing web-based applications, but this does not mean I am a web developer. I primarily create software which sometimes happens to have a web interface, not websites which happen to have functionality behind them.

It can be a subtle (and, arguably, pedantic) distinction, but it means that I am equally comfortable setting up websites or creating software which just sits in the background, doing its thing (e.g., monitoring access logs or auditing user activity) with no human direction required and no user interface at all aside from writing to a log or sending out email notifications to tell you when it finds something amiss.

This distinction also means that web layout and graphic design are not really my specialty. If you’re building a website and know what you want it to do and what you want it to look like, I can make that happen, but, if you ask me to decide what it should look like, you’re likely to get something that looks fairly spartan (some might even say “boring”).

2. What makes you unique?

For starters, my preference to specialize in back-end processing rather than user interface design seems rather unique.

I’ve been developing software a long time - as I write this, I’ve been programming for 30 years total, 15 years professionally - and I have a breadth and depth of experience that few can match. I’ve worked in several different industries over the course of my career and learned not only how to write software, but also how to assess a situation and identify what software should be written. On top of that, I’ve worked some years in systems and network administration, giving a solid understanding of the environments in which software runs and how to ensure that it will do so reliably.

I also write software with a distinct eye towards future maintainability and expandability. I know that someone will have to fix or change it within the next year or two and I know that, more often than not, that someone will be me. This isn’t anything unique within the realm of traditional software development projects, but it is often sorely lacking in the “results now, don’t worry about tomorrow” world of far too many online projects.

3. What kind of clients do you have?

Mostly small-to-medium businesses. Organizations which are big enough to afford custom software development, but not large enough to have the available technical manpower to do such projects in-house.

I also do quite a bit of work with web designers who need an expert partner to help them out with projects that require heavier or more in-depth back-end development.

4. How much do you charge?

I normally prefer to bill hourly for my time, but I am willing to do fixed-bid projects if provided with a sufficiently-detailed specification of the project. Even with a good spec, though, I have found that hourly rates help to avoid potential conflicts over what work is or isn’t within the project scope.

Reduced rates are available for projects which are to be released under a Free/Open Source license or for which I will retain ownership of intellectual property rights after project completion.

5. What kind of business relationship will we have?

I primarily work independently and off-site, but will consider other arrangements provided that travel and other expenses are covered, including payment of my normal hourly rate for time spent in transit.

To avoid confusion caused by conflicting information, you will be expected to designate a single point of contact who will provide all direction on your project.

6. How do I pay you?

I am currently billing clients through Aurenav, LLC, as this allows me to focus on my clients’ projects without being distracted by invoicing, collections, or payroll issues. We will agree to a schedule for billing as part of the initial project contract and Aurenav will issue invoices according to that schedule. Payment on each invoice will be due within 30 days after it is issued.

7. How long have you been in business?

NomadNet, Inc. was founded in October, 2004.

8. Where are you located, and what are your hours?

I am a U.S. citizen currently resident in Lund, Sweden, but I typically work remotely, so you won’t have to worry about the expenses involved in getting me from Sweden to your site! On the contrary, the substantial majority of my clients have never met me in person, but instead have dealt with me entirely via email, phone, and/or Skype.

I am a bit of a night owl, but, thanks to the magic of time zones, this puts my most common working hours roughly in sync with standard business hours in the U.S.

I’ve seen some very good, very successful company blogs out there which have led me to the decision to start this one. The blogs I have in mind are those which visibly contribute to the success of the (small) business which runs them. These businesses have the common factor that the blog itself can serve to showcase the owner’s skills in that business - writers, business coaches, self-help gurus…

But I don’t do any of that. I write software. Writing software on the blog or blogging about how to write software would be a great way to contribute to the software development community, but it would have no relevance to anyone who might require my services. Conversely, a blog that was all about marketing would produce very little actual value, not to mention that it would be largely inauthentic of me to do so.

To resolve this, I’ve settled on creating a two-pronged blog. (All the pro bloggers who write about focusing on a single topic and never straying from one well-defined niche may now collectively gasp in dismay.) All posts will be categorized as either “Technical” or “Non-Technical”.

• “Technical” posts will be those which directly relate to the actual process of crafting software. The first few in this category will be design- and usability-focused, but coding techniques and the like will appear there, too.
• “Non-Technical” posts will cover anything else.

Feel free to subscribe to only one category’s RSS feed if the other doesn’t interest you. It won’t bother me in the least. Honest.

That still doesn’t address the issue of showcasing my skills on the blog, though.

I’ve decided that the way to accomplish that is to make the blog itself a demonstration of my skills by writing my own blog software, to be known (somewhat unimaginatively) as NomadNet::Blog. It’s not going quickly, as I don’t have that much free working time to do it in, so I’ve decided to start out with using WordPress rather than waiting until NN::Blog is ready to face the world. Going this route will also pretty much force me to write an import tool which will convert a WordPress blog’s data into something usable by NN::Blog; I expect that having this capability will greatly increase the odds of anyone other than me using it.

So that’s where we’re starting from. Stick around and I hope you enjoy your stay.
 
 
 
P.S. Getting this blog to match the look of the rest of my site involved a bit of ad hoc tweaking to WordPress beyond simply editing the base Sandbox theme. If you happen to notice any dark corners of the blog which don’t look quite right, please contact me and let me know so that I can get them cleaned up.