1. A Correction To Last Tuesday’s Post

In Off the Record: Passwords, I recommended the use of SHA1 rather than MD5 hashes when storing passwords. Since then, I have encountered a persuasive argument in favor of abandoning both of them and using bcrypt instead, as it’s designed to be less time-efficient, thus dramatically reducing the number of potential passwords that a brute-force attack can attempt in a given amount of time.

As this can make your passwords much more secure against attackers while still keeping single-hash generation running at a reasonable pace, I have revised that post to recommend the use of bcrypt over SHA1 where available.

2. I’m Not Disappearing

New posts to this blog have slowed down substantially over the last couple weeks because the stockpile which I prepared prior to launching the blog have all been published and my new writing process, which I expect to work out much better in the long run, is taking longer than the old one to produce completed posts, ready to publish. If all goes as it appears that it will, the rate of new posts should pick up again by the end of next week if not sooner.

Although I didn’t say a word, I just about died on the spot. Although he wasn’t aware of it, he now knew the PIN for my bank account!

Granted, I should have known better than to reuse a PIN (and I do know better now!), but it’s a very common practice nonetheless, especially as the number of PINs and passwords that each of us needs to keep track of seems to be increasing daily.

Back To The Present

More recently, there has been discussion on the debian-user mailing list of ways to log the passwords entered on unsuccessful login attempts. While this seems innocuous at first glance, and an effective way of seeing what passwords are being tried by brute-force attackers, it will also catch passwords entered incorrectly by legitimate users. These mistyped passwords will generally only be off by a letter or two, thus giving anyone with access to these logs the ability to much more easily guess the correct version.

Despite the potential advantages in being able to revise your password requirements to improve their strength against the latest dictionary attacks, the issue of password reuse remains. These kinds of system logs are generally locked down well enough that they’re only accessible to system administrators whose legitimate powers are broad enough that knowing other users’ passwords will not allow them to do any additional damage locally beyond what they can already do using their own account, but granting them knowledge of others’ passwords also potentially allows them to impersonate the user on other systems.

Trusting them with this information is not necessary for them to perform their legitimate duties. It allows them to do additional harm, but does not expand their powers for good. Therefore, it should be kept from them and any security-conscious sysadmin or developer should recognize this. (Personally, I have, on several occasions, cut off users who were about to tell me their passwords and explained that I would not allow them to do so.)

Avoiding Password Leakage

To maintain secrecy, user-entered passwords should never be recorded in plaintext. I can’t think of any situations in which there would be a legitimate need to be able to recover an actual user password, but, should such a case exist, there are plenty of good, reversible encryption methods which can be used to store it, safe from prying eyes or casual glances.

The proper and generally-accepted method of dealing with passwords is to pass them through a non-reversible cryptographic hashing function, such as SHA1¹ bcrypt², then storing the hash rather than the password itself. Use of reversible encryption would still leave the password vulnerable to anyone with access to the encryption key and the will to use it. A SHA1 bcrypt hash allows you to determine whether the entered password matches the correct password without ever knowing (or being able to know) what the correct password is.

Afterword: Honeypots and Testing

I did mention, above, that there are potential benefits to being able to see what passwords are being attempted by attackers. If this information is needed, it can be obtained without compromising the security of your actual users’ passwords by setting up a honeypot system with no valid user accounts (or at least none which accept password-based logins over the network). Since no legitimate logins will be attempted, you can safely record the passwords without giving away information about legitimate users’ accounts.

The other situation in which I’ve seen it suggested that it is appropriate to record passwords is when testing or debugging software under development to ensure that input (including the password) is being received and processed correctly. Safeguarding actual user credentials is again easily done by means of not having any actual user data (or at least not any actual passwords) in the testing environment.

² Shortly after making this post, I ran across this article, which makes an excellent case for using bcrypt rather than MD5 or SHA1. If a bcrypt library is available in the environment where you’re operating, use it. If it’s not available, see if you can get it added.

Incorrect Validation Is Worse Than No Validation

A recurrent question I’ve seen on a couple different programming sites is “What regular expression can I use to validate an email address?” They’ve gotten a lot of responses explaining ways to examine an email address and reject it if it’s “invalid”. The only problem is that every one of them that tried to be more restrictive than just checking for the presence of an “@” was wrong.

The most common flaw was failing to accept “+” in the local-part of an address, causing many perfectly valid, deliverable email addresses to be rejected. There were also several which failed on top-level domains (TLDs) which were four or more characters long, such as .info or .museum.

On the flip side, there were users accustomed to using disposable email addresses of the form username+foo@domain.com or with addresses automatically generated from their surname of O’Malley - including the apostrophe - complaining about sites refusing to accept their “invalid” (but syntactically correct and perfectly functional) email addresses.

Rejecting these addresses may only shut out a percent or two of the world, but is that really an acceptable cost to pay in an attempt to shut out truly bogus addresses? Particularly given that…

Valid Does Not Imply Correct

Let’s say that you’ve managed to find yourself a perfect validation algorithm which is 100% accurate at determining whether email addresses are syntactically valid, even for the O’Malleys. You’re getting only good email addresses, right?

Wrong.

abcde@fghi.jkl is syntactically valid, but utterly bogus. There isn’t even a .jkl TLD!

“OK,” you say, “I’ll just add a DNS lookup to my perfect validator so that it rejects domains that don’t exist or don’t have a mail exchanger (MX) record defined.”

Still no good. dave.sherohman@whitehouse.gov is syntactically valid. It points to a domain that exists and accepts mail. It’s still no good.

“No problem. I’ll connect to the mail server and validate that the user account exists.”

I can still give you a bad email address. Try president@whitehouse.gov on for size. The account exists, but it’s certainly not mine.

If There’s Any Chance Of An Incorrect Rejection, Don’t Pre-Validate

The only way to authoritatively validate the correctness of an email address is to send email to the address and request that the user respond to it in some manner to confirm receipt.

You may still benefit from doing a preliminary validation to avoid the trouble of sending out mail that could never possibly be received, but this pre-validation cannot reasonably be considered authoritative and should, therefore, only be used to filter out the most blatantly incorrect cases. Attempting to make it more restrictive will only introduce an unnecessary risk of rejecting genuine, correct, deliverable addresses.

The cost of performing the authoritative validation is low and the cost of rejecting an address which is incorrectly flagged as invalid by a flawed pre-validation is high, so don’t pre-validate beyond the point at which you are absolutely, 100% certain that it will not fail anything which might pass the authoritative validation.

If you’ve done much software development, that statement alone should be enough to have you thinking “concurrency issue” or, more specifically, “race condition”. Given that the application involved both a web-based management interface and a once-a-minute scheduled task to handle accounting, the obvious point of conflict was for the problems to result when an operator submitted an update through the web interface while the accounting task was running.

So off I went to ensure that such a conflict wouldn’t cause issues, then sent the revised code off to the client. He reported back the next day that it hadn’t had any effect.

After looking harder at the code, I came up with a way that this conflict could conceivably have still occurred under some contrived circumstance, eliminated that possibility, and sent it off. Still no improvement.

We went through maybe half a dozen rounds of this before I ran out of ways that the web application and accounting task could conflict, no matter how far I stretched the bounds of probability, and finally took a closer look at the application’s logs. Once I started looking at them as a whole, instead of just records of single users, I quickly noticed that these mischarges came in clusters, affecting several users at once, not random individuals. These clusters also tended to hit right on the hour.

Checking in with the client, it turned out that the specific times when the problems occurred matched up with the times when the per-minute rates changed.

To keep the accounting simple, the application deals with rate changes by closing out each active user’s session, charging it at the old rate, and then opening a new session at the new rate. With larger numbers of users logged in, this process was taking more than a minute to complete. The conflict wasn’t between the web interface and the scheduled accounting task, it was between two (or more…) copies of the accounting task! This also explained why I was never able to reproduce the problem in my own development and testing environment, as I had never simulated enough concurrent users in my tests to produce the problem.

Once the correct culprit had been identified, it was a simple enough matter to resolve by adding a check to prevent multiple instances of the accounting task from running concurrently.

The moral of the story:

Even if the cause of an issue seems clearly obvious, don’t forget to fully examine the system to verify that you’re solving the right problem before you spend too much time solving the wrong one.