How to store passwords

[nilamo]

There was a thread recently about checking passwords against previously known values.  Passwords (and verification answers, email addresses sometimes, ssn, phone numbers etc) are information which you use to verify that someone is who they say they are, but which you never actually need their answer for.  You need to verify that what they type matches what they typed previously, but you don't ever need to see what that is.  So you NEVER store the actual value anywhere.  

Think about all the information you have across all the varied websites you use, and how bad it would be if that information was freely given to other people.  Think about the worst case scenario: your database has been cracked wide open, and anyone can see any of the information stored there.  The goal here, is to store things nobody would want to see anyway, so even in the face of a worst case scenario, nothing bad actually happens.

In order to accomplish this, we use one-way hashing algorithms.  In software and various types of documents, you'll see something called an "md5 hash", so after you download the file you can hash it yourself and compare the hashes to make sure what you actually downloaded is what they made available (to protect yourself from man-in-the-middle attacks, or, more likely, just to make sure the downloaded files weren't corrupted).

Let's take a side trip, and pretend we're a l33t kracker with mad skillz, and we want to pwn some systmz.  Understanding what an attacker has to do will help to understand why we store passwords the way we do.  So, we've gotten into a database, and in the User table, we see there's a column called "Password".  Bingo.  But then you see the value of the column looks like this:

> select Password from User;
0c18c2a2cf4c2bca864f948409c7295ce784a97d4c9e27bb69cdfcdf41dfa8a7051480b8ac6c8ea87d89c611af321ab2bc2e95aded966099feab6c3e19bde182:8090a9c58052ccfae4e0720341401ccde410af6213b1d197abd3ddffd3a47a9df8f8fe10a7227c1d5c4f8560a3c26efd69f84356eeb0ae6b6c1426ccb418dfd8
2a2586aeb1d2d6a035a9b49d84268970e420bcdc7ce28faaeb555fe55bf9f9df3f422e3a8559d09a6a1329eb1dbd86a89cb61aa259d8e301f41de23eab0fbfc6:c0b879f6c97e8a9899492915bf3e32bea6ce983ac97e8c4870fc74aae70471fb20d4e989e2cb1a4e5b35382eab6da7be08a5a17115005f037ae7288c877fc8eb
a0a5d0c8391dfd784b91f261d1d03c1a5a9fdf576e98dbd0669526506d620a7e7b84a44c04890cf8492e05a2acfd61d6fb121d4c30be775d0aca58f18443f624:e0fa51466e340ee777bf2fda8640377cf11ecf58ed36935b6d05cc22860735b4414152cb468aae766038c8deb01d79ab950a2205fa60e1c05432376a3fec0557
544bed47b1300f1162a2ed59fbd16af3da780136bab72e059ea81d1d0be149798a5800af7b60811219c93bb52d6c89ea9000943f3efb46da5a1c51a668591c8f:47e3098d4412bd61704d064a00a7b4fab8bc37390d0c423f90d5e043df395ea7cb69fbf4645312252b0c83a3c02d28e0823ab44760cc6e130121199be7ebf86d
731caaaa977ec88f8c39a358879747227698c50fd60802506b9666936fd4029fc69a5afdefef754d6faa49b8f9731ab25393d2817277570efb23d13525937d8c:30379336403acdb6817d7d12f7eb0191ebaf395a8cfec8bd963fd01cac861be0685dfb1b23174e8dce2db596d745b7e19e9132e9871ace51553303f59bd5fcd1
5cfc045c6d12b0f11134f68baae887086bd9c3019f9d6dc3a74a1f26b9e895716926432bd07570aa2957dfa5c53b0deaed9570e40c6446cdd98fd7155876babf:efb57b110165edc3b369e4a7c66c1d1a8e3b51c5a7d12e98c8b4c522dda6bd30d4b5ab9c0e4d44b13dcb12bbfa9a1d32233cb5f9016d03191457047461f5272a
d45be497332193978fc1ff399465a4249eee3bc9a91ee40aa8f73b4a03ca11f92b2b878b05714bbfa7b676df0e6b63f1e8f7c469c9ed99770f4f51dbb599320d:4224cea2edf604cabd636667c07360c67816d9bf800fa25ad591abd887d223496f8b733548a94c31d68fcfbcac39405eb5837dcae160f3e25398f6de14d4837a
f55fd0c26f686658dcfdc27b62fcc19e34f6e0da0d967431525d3b24eecbeb1f97d499704f035f45be48949aed04ed82b15e694c6416f0ff5a8a938341fc95d4:9a84259756300a7ce8ea8b60d1619096d0f78f2950ca171953c1f704e40e971037fea5dfe95ccae7494e35e6be2825dba608316d00d3ec8ab79e2f0ea7f33adc
d08093ae54de27423063b590fd986db0bc2feb21ea8cd9f6aa516c1a5e80b2580f93c140292f7ca270f2e8436f70749963b2efb26f07c2ab5fd2d24b6e243ede:cbf6e85e52308fdd28ab4c6d2bde03b59c3de534882c8070c08048994c1e3de47aa70ab377c67a5e0ca7bf64e46fb24e5efe73b813e721c55bbca1efc9a380fd
acc8a2ee65853e29e768a3d14a4ca598615521d3ea8b1ae11d421b8403355ea11b942632d76de33d3a3923c77bba3526276afee3f3307082897c5173cf7880f6:b580d7c0ebb0e97ca2259102a1d287ae58991f6caeb3e31e7de0012cf726db3f5746184e0407a1f55779fea356e2e463291dcd8c85f71586b96025129b47e182
9431232f4e2b40b8a06fab7207ab45a51bd43f3d00fb546e99fdeda8408ff4458a78ac9658b7b8ee526fefdb3d367f49a5504d21beb909e25b10171067d8dbd8:a110b74388a6b5378e7837de2cc9184d0a4887ee373a2297c30015032de3db30f32a01e56b8faa60111620d4483226a2e7008c09a17f99ff47beb9f03dc2c2fd
1d6bc8a35bbcde6b3a5a5e7e67690a2ab882fc1d1c1a4657506f1d8237d1de5c28243e6fbe3d8b72f2fbb63340fa73f6d96db7155ae89b82fec638b47cf6eb6a:1db82c674c9cb19090166765a92e56c002e7509e619acdbb4d046532a566b25942bbfc6f5380ce40cdd157fd6ffca76aabca7197b11afcd75cc82cc173dbedb5
c89a35833525e3715e94754736bd7b64f2d5ede02491e24394e71863f3e5fdfef493e46825789c6d0c0e7129eadf92647ca29a6fc966b5faf4cb18d8c10feeab:5e817e12a4833e1a4318bbacbc2775f5aae7e640d5dd62eac65015006eb217ed9ce4c911db382b45f01285dd909684a4c8051d000d820cc3a31263f5e6223e7f
07ff4c5d0cfc1229e2ea4d817cf3b7adde528ff27c014ef51715dbab76573c84baedcbb2c3b0a042f7c5779f30d09f1a753ba57a10908f6ae03df66abe89abf0:fec16d6fe85458111a50fff95cfdbe9872b608aa6973df03035d8fd17801656743bdac7a120d27d1dc60dc9e7785d220ad3e7a6f40efe3248d11be485a9c5431
eb2eacceafe92180ab1ce9597bb15280039e3f2241eaf8a67d4ccd2f6383bbd46894304f3ffc7592cad36fb21965f1bd8c7e6a8cfb3f643c8eb0fe841b919086:311be4612a2d1eb8fda7f61c983940369908896896df727e71d041ebe383fc77e40310f302667ddce20123f5b1c88628cc61bbac44db507342782b658312dd0b
7682bb90f64ff99363b0646bdd913680ac48347f18fd24612f7e409712dbea4e25ceb159451117cf4d7c8bdf006d6abb8062f480aab1e2d9dd5f4a9940c7ed29:653cbcce3f5947d678c72146d9e9899e9bd865b7cf2c5fb4edb8443880c702066f5360f9dcd7b5e4344eb2f2579e6268805389cfcac0845b07858ff74faf734b
6f4b56761ea58672c11023a8b0848eb7b53b22308801ba48eeb4f8293f4e7864f32256179775c648fd400e3e857a03e2ed92839aa95c0e7b9afee4badfa7ab48:00f965e54d65916d1a4079dcc1ba7b7bbe413e413046c616f66df8638f146c624b2e3d7960dd71875f24758196e352b8677f1fb336af0d77d7bd4490125c107a
fe092faf9a3bfb62e1b37081a292303dae1965f3ae2d43a0ac7665ccefcc4fb4ce4313b964f96046d9f6147c9e3c12e28638c8fce4aa09ca7e2f5bbb49f341f0:851cf1d771c2e1ca913c97343c8f3dd27de6b41067008640da70960ea5fb1ba909a10e688761d75cfc60a71ba94ba74afb556be2fde2149dd17883738ccf364e
d78ca3a1ddc6a599307484e278bcfe8a3b10d55bb94e7670e7096f1169df1cea9825869e43da27a3f7262bd863ec3a82927ca154240b53b4c8f8b6e2fe69b5f3:8c91987e236274eed5957d0ea453764d8f9a3d111591ee9b7118ae31ed891c20163b6240b7859613e62933568d75605ce49b6e17b2f772e5178344e4c7f66985
c3c3ec7a7d9e9126233e498e092776bace142f66219e5e89472cfe8222e51d1331a6294a68d75eebab1035b0e02ed69ec9451712fc3c3e20dd6320b11618811f:966f4fb9cd80af0aa108df47da1271137ebe81a3f7282518063b2d241ab3588d00c2141c8e39a795a12d23a3a0732b3c6b3de1274e3630d6ec7571a2ef8c7afd

Most n00bs would stop right there, because wtf amirite?  But you've got mad skillz, and access to the source code (databases are harder to get into than the server itself, so it is not unreasonable to assume an attacker can read your source code), so you can build a dictionary to test these values against.  Go ahead, I'll wait.  Each password has it's own salt, which means you'd need to re-build your rainbow table for *every* user.  Which would take roughly 3 days for a password 5 characters long.... per user (for sha512.  Using the proper hash algo, it'll take significantly longer.  Longer than "heat death of the universe" kind of significant).  

md5 and sha512 aren't actually good enough for passwords for a variety of reasons.  They're too fast.  Well, and md5 is broken (two different passwords can yield the same hash).  Time is the enemy.  If you give an attacker enough time, they can eventually unwind whatever insane algorithm you're using to encrypt your password, and then apply the same process to a rainbow table, then compare it with your database to get the passwords.  So the longer something takes, the better it is.  sha512 and md5 are designed to be fast.

Luckily, this is 2016 (almost 2017!).  We have more than just some random loon proclaiming things are bad and other things are good.  We have many loons.  Science has come to the rescue, and delivered upon you a simple solution: use bcrypt.  It's been tested hard for years, it's designed for passwords, it's slow on-purpose, and it's available for almost all languages.  In python, it's as simple as "pip install bcrypt".

>>> import bcrypt
>>> user_password = "spam spam eggs and spam"
>>> hashed = bcrypt.hashpw(user_password.encode(), bcrypt.gensalt())
>>> # this is what you store in your database.
...
>>> hashed
b'$2b$12$fZLMiMQrovVwRQt9vO7hW.6FNW1NU/1LIYVLpG1LGaT7kWAmlBDy2'
>>> # later...
...
>>> bcrypt.hashpw(user_password.encode(), hashed) == hashed
True

That's very simple, easy to understand, and almost guaranteed to be more secure than anything you can dream up on your own.  Use it.  Embrace it.  Love it.

[micseydel]

Getting into a server hosting a service doesn't imply access to the source code. It's (largely) true of Python, but imagine, hypothetically, a company keeping their source code in private Github repos and then pushing compiled binaries to AWS. You could crack one or the other without necessarily getting both.

[Skaperen]

what do you recommend instead of sha512?  maybe sha384?  anything hashlib has?

[nilamo]

Getting into a server hosting a service doesn't imply access to the source code. It's (largely) true of Python, but imagine, hypothetically, a company keeping their source code in private Github repos and then pushing compiled binaries to AWS. You could crack one or the other without necessarily getting both.

While true, I do still think it's far more common for the source to be available than the database.

---

what do you recommend instead of sha512?  maybe sha384?  anything hashlib has?

All of the "sha" algorithms are really the same thing, but with different sized output (...basically).  I'd recommend bcrypt, but if you're in an environment where you can't install packages, then I'd like to reference the docs: 

Key derivation and key stretching algorithms are designed for secure password hashing. Naive algorithms such as sha1(password) are not resistant against brute-force attacks. A good password hashing function must be tunable, slow, and include a salt.

Starting with python 3.6, hashlib will include scrypt, but it has also included pbkdf2_hmac since at least 2.7.8.  The NIST recommends pbkdf2 as being "good enough", while scrypt has a few minor issues.  So if you can't use bcrypt, then use pbkdf2_hmac.  Using it is fairly similar to bcrypt:

import hashlib, os
salt = os.urandom(100)
password = 'get this from the user'
hash = hashlib.pbkdf2_hmac('sha512', password.encode(), salt.encode(), 100000)
print(hash.hex())

The difference between bcrypt and pbkdf2 being minor: bcrypt's output includes the salt so you only store whatever value it spits out, while pbkdf2 spits out only the hash, so you need to store both the salt AND the hash.

[micseydel]

While true, I do still think it's far more common for the source to be available than the database.

Why do you believe this? Hosts for databases must necessarily be externally accessible, which makes them more vulnerable than source code, which doesn't necessarily need to be so accessible.

[nilamo]

Do databases need to be externally available? I guess it depends on the service, but most of the databases I've seen don't have public ip addresses, and are only accessible once you're inside the intranet, at which point you'd already have access to the source code. The sorts of things like our forum right here. The only entrypoint for average_joe is the one server running the php frontend (or the Squid server, or the load balancer), and you'd have to spaghetti your way to other things from there.

[micseydel]

What I mean to say is, a database storing user password hashes must, in some way, be accessible from the internet for any websites. The database isn't itself publicly visible, but there still must be a path to it. But a company can keep their source on an intranet, and actually can be stored entirely offline. The opposite cannot be said. Having access to a database does not entail in any way having access to source code. You almost certainly have binaries, but even with Python there might be obfuscated .pyc files without nice source files.

[nilamo]

Ok, I see what you're saying now. I just don't see that as being all that likely.
Either way, with or without the source, the passwords shouldn't be stored in any sort of reversible or easily reproducible fashion.

[micseydel]

What exactly do you not see as being likely?

I didn't mean to suggest this tutorial is incorrect, just that the conclusion of this quote is a non sequitur: "databases are harder to get into than the server itself, so it is not unreasonable to assume an attacker can read your source code."

[nilamo]

Maybe I phrased it poorly. I guess what I meant was, if you start with the assumption that your code and data are both publicly visible, and encrypt important things to account for that, then any additional security to the servers is icing on the cake.

And aside from things like asp.net, I guess I assumed very few websites use complied binaries or obfuscated code.