Embracing Chaos

A couple weeks ago, Dan Kaminsky found a flaw in DNS.  Without getting into details, this flaw enables a malicious attacker to fool your web browser into connecting to the wrong computer to get your web pages.  So when you type www.facebook.com into your browser, you might actually go to Joe Hacker’s site, even though your browser says http://www.facebook.com/ in its address bar just like it should.

Dan, being a "good guy", tried to keep the details of this hack quiet for long enough for network operators to patch their systems and close the loophole.  He wanted everybody running a DNS server to do this before the "bad guys" figured out what the bug is and started to take advantage of it.  He was hoping for 30 days of time to prepare, but somebody spilled the beans after 13 days, and now the hackers are off and running.

It’s a jungle out there

You might be asking, So what? What are the dangers of being directed to the wrong website?  Of course, you could read incorrect news and that’s not great.  More likely you’re going to have your password stolen for whatever site you log into.

The obvious attacks are to sites like paypal or banks, but they’re actually safe from such attacks if you use your browser properly.  Any financial site will use a secure connection.  You can tell because of the https:// at the beginning of their address.  These sites use a digital certificate that your browser checks to verify their authenticity.  All this happens independently of the DNS system.

But you can still connect to a hacked site with https.  Your browser will probably warn you saying something about a certificate not matching.  More often than not these errors occur because of a lazy sysadmin or something.  But right now, I strongly advise you to take all HTTPS warnings seriously.

Protect yourself

If you want to be sure you’re safe, manually connect your machine to OpenDNS, as Dan recommends.  We know they’re patched and can take the traffic.  I’ll give you the steps to do this on Windows:

1. Start menu
2. Control Panel
3. Network connections  (might have to switch to "classic view")
4. Select the one you’re actually using.  It’s likely called "local area connection".
5. Click Properties on the status dialog
6. Scroll down in the list of checkboxes, and select "Internet Protocl (TCP/IP)" so that it’s highlighted.  (Leave it checked!)
7. Click Properties
8. In the first General tab, change the second radio-button from "Obtain DNS server address automatically" to "Use the following DNS server addresses:"
9. For Preferred DNS server, enter: 208.67.222.222
10. For Alternate DNS server, enter: 208.67.220.220

That’s mostly it, but to be safe, you should reboot, restart your browsers, and/or:

11. (Windows key+R).  In the dialog type  "ipconfig /flushdns" (without the quotes) and hit okay.

It was pretty late at night at my friend Miller’s birthday party last week.  She had asked everybody to do something good for the world in lieu of birthday presents.  The awake were discussing options as I was dozing off.  I overheard somebody say "If you’ve got an old linux box that you’re using as a firewall drawing 400 watts continuously, consider spending $30 on a dedicated router."  I thought about the headless Pentium 3 box in my office closet which is running the IP Cop Linux firewall distro.  I thought about the four matching ethernet cards I’d put in it and the rainbow of color-coded cat-5 coming off it: red for untrusted outside world, green for safe, orange for servers and blue for wifi.  I thought about all the time I’d spent configuring the thing perfectly and routing cables throughout the house and I thought, yeah it draws a lot of power, but I NEED all that.

When I sobered up the next afternoon it occured to me that I’d pulled my file server off the orange DMZ network for performance and simplicity, and that the other server box had long since been virtualized into the file server.  I moved my local public wifi off the blue network onto the red to make its security brain-dead simple.  So despite all the pretty color-coded cables and corresponding hubs, all I really had was a big loud NAT box with a few key port holes in it.  And since I’ve switched from outlook to Gmail, I never even RAS into my home XP boxes any more.  And since I do all my personal development on EC2 or some other host, I never use my home dev servers any more.  So in fact, I don’t need to tunnel home for anything.  Cloud computing.  For real.  All this stuff I used to need I don’t any more.  I could replace that old linux box with a cheap low-power firewall.

But that got me thinking.  There’s this li’l XP box sitting next to the printer that I have configured never to go to sleep because otherwise I can’t print from my laptops.  Print servers are similarly small and low-power and sometimes come in the same box as the firewall.  Then my eye turned to the terabyte file server in the corner and next thing you know I’ve got an Apple Time Capsule in the mail to replace all three permanently powered-on PCs in my house.

Happy BEarthday, Miller!

Every time we sign up for a new service on the net, we have to make a new account and pick a password to go with this account.  We’re going to be stuck like this until something like OpenID becomes dominant which is going to take years.  Until then we’re stuck remembering which password goes with which site, which is an age-old problem.  I think very few of us actually use a unique password for every account we have.  For sanity’s sake, we re-use passwords, or at least password themes.  Personally, a couple of times I’ve had throw-away passwords get used in so many places that I realize they are actually now quite important because they grant access to many services.  Some of these services seemed stupid at first so I created an account with a throw-away password and then they became important to me later.

The problem that’s bothering me today is that I have no idea what the new site is going to do with the password I give it.  Few things irk me more than seeing one of my good passwords sent to me in plain-text, say in a password reminder mail.  Why are you storing my password in plaintext!?  Some of my most valuable passwords I’ve never even seen written out, and the first time I do see them say in a username field it’s a little surprising and I feel like they’re a bit compromised.

I’d like to see sites post their password handling policies.  I’m going to start doing this on my own sites.  Here are some examples of levels of security to consider:

• Your password is stored in plaintext
  • Forgotten passwords will be emailed to you in plaintext
• Your password is stored using a one-way cryptographic hash form
  • Forgotten passwords cannot be recovered.  If you forget your
    password you will have to create a new password after we verify your
    identity.
  • using MD5 (not considered very strong cryptographically anymore)
  • using SHA1 (better)
• Your password is stored using a strong hash with a salt
  • Without a salt, somebody looking through the password database could see that two users have the same password since they hash to the same value.  Obtaining one password thus grants access to the other account.
• Your password is never transmitted or stored in the clear. 
  • All auth events happen over SSL
  • Your password will not appear in any server logs.  (Rails devs take note.)
• Your password is never stored in any form in a browser cookie.
  • Some sites might see it fitting to store a hashed password in a cookie for re-auth.  Ugh.

Responsible sites will follow the last three rules.  We could come up with a simplified grading system that explains in simple terms whether you can use a valuable password here or not.  We know we can trust the smart folks to do that.  But with so many features being launched as stand-alone sites, understanding these policies is important.

Why it’s good to break the MVC pattern

Bruce Perens hit on a really good thing when he wrote a package for Ruby on Rails called Model Security.  It’s too bad the project is gathering dust.  But even if you don’t use the whole thing (I haven’t been able to) there are some really valuable ideas and chunks of code in there.

The idea behind Model Security is to centralize security rules in the model classes.  Certain objects can only be accessed by certain users.  Perens talks about multi-layered security.  But in my mind the real benefit is that you can just write the basic rules in one place and not worry about it everywhere else.

An apparent problem with this strategy is that it violates the encapsulation of the MVC pattern.  The only way to put security into the Model part of the pattern is for the Model to know who is trying to access it.  The concept of the user is generally localized to the controllers in an MVC pattern.  Maybe the view.  But definitely not the model.  In MVC, the model is supposed to stand entirely on its own and not depend on anything except maybe the persistence mechanism (i.e. the database).  So in this way Model Security violates the basic MVC pattern.  Violating well-known design patterns is bad, right?

Absolutely not!  In this case it’s actually a really good thing.  Developers who blindly follow the MVC pattern end up copying and pasting the same security code all over their controllers.  Every place that could possibly modify data needs to check security rights.  Any place the developer forgets to do this represents a security hole.  By putting security rules in your models, you know everything is secure against hackers.   Then in your controllers you just need to worry about preventing your users from accidentally seeing security exceptions that would confuse and distress them.  The result is cleaner, more maintainable, more secure code.

Unit tests

"What about unit tests?" I hear you cry.  Good
question!  For good reasons, we like having unit tests that run on the
models without the web framework in place.  But with ModelSecurity, the models depend on the user object, which is generally a part of the web session.  So we’re kinda stuck.  Encapsulation is broken, and thus follow our unit tests.  The easy answer is to use a
global configuration setting that turns the model security checking on
and off.  When you’re processing a web request, turn it on.  When
you’re running unit tests, leave it off.  I’m thinking this should be pretty easily done in application.rb.  Or perhaps through an IOC method in the tests themselves.  But I haven’t actually revived the unit tests in this project so I couldn’t tell you for sure.  Sloppy, I know, but it’s a lot easier to justify when there’s only one coder on the project.  I’ll post an update when I dive back into this project.

Problems with Peren’s ModelSecurity gem

I’ve experienced some bizarre interactions with FCGI at least on dreamhost.  The ModelSecurity subsystem seems to crash at some point and then opens everything up to allow free access for everybody until I restart the FCGI process.  This is absolutely not acceptable.  On a somewhat similar note, sometimes basic functions will fail on first execution claiming things like "NoMethodError" but will work fine on subsequent reloads.  Having very little interest in debugging this interaction, I have given up on using Perens’ fine-grained rules.  The ModelSecurity allows you to specify very carefully which data fields can be accessed by which users under which conditions.

In my app, and many others I can imagine, it’s enough to set security at the row or object level.  This is relatively straightforward with ActiveRecord’s own callbacks. 

class SecureObject < ActiveRecord::Base  has_one :user

  #Implement model-based security  before_save :check_is_me  def after_find       # For performance reasons, you have to explicitly define an after_find method.       # You can't link it in with "after_find :check_is_me" like other AR callbacks.       check_is_me  end

  def check_is_me      if !is_me?          raise "Security exception.  Not your object!"      end  end

  def is_me?      return (User.current) && (User.current.id == self.user_id)  end

I still use a valuable construct from the ModelSecurity package,
which is the User.current class method which keeps track of who is
currently logged in thread local storage.  This global variable is what enables us to break the MVC pattern by giving the Model access to information about the User from the Controller.  Here’s a relevant snippet from Perens’ user_controller.rb:

  def User.current    # This does not refer to the session because the application has set    # this from the session in user_setup.    Thread.current[:user]  end

  def User.current=(u)    Thread.current[:user] = u

    session = Thread.current[:session]

    if session.nil?      message = "Programming error: Please add \"before_filter :user_setup\" to your application controller. See the ModelSecurity documentation."

      raise RuntimeError.new(message)    end

    # Don't cause a session store unnecessarily    if session[:user] != u      session[:user] = u    end  end

The missing ModelSecurity migration

Another problem is the lack of a migration to add the tables required by Perens’ code.  Fortunately, it’s not hard to reverse-engineer using schema.rb and the .sql files that Perens provides.  Here’s db/migrate/###_add_modelsecurity_tables.rb: (the filename is important — read on)

class AddModelsecurityTables < ActiveRecord::Migration  def self.up    create_table "user_configurations", :force => true do |t|      t.column "email_confirmation", :integer,   :limit => 3, :default => 1,  :null => false      t.column "email_sender",       :text,                   :default => "", :null => false      t.column "created_on",         :timestamp      t.column "updated_on",         :timestamp    end

    create_table "users", :force => true do |t|      t.column "login",        :string,    :limit => 40,  :default => "", :null => false      t.column "name",         :string,    :limit => 128, :default => "", :null => false      t.column "admin",        :integer,   :limit => 1,   :default => 0,  :null => false      t.column "activated",    :integer,   :limit => 1,   :default => 0,  :null => false      t.column "email",        :string,    :limit => 80,  :default => "", :null => false      t.column "cypher",       :text,                     :default => "", :null => false      t.column "salt",         :string,    :limit => 40,  :default => "", :null => false      t.column "token",        :string,    :limit => 10,  :default => "", :null => false      t.column "token_expiry", :timestamp      t.column "created_on",   :timestamp      t.column "updated_on",   :timestamp      t.column "lock_version", :integer,                  :default => 0,  :null => false    end

    add_index "users", ["login"], :name => "login"    add_index "users", ["email"], :name => "email"  end

  def self.down    drop_table :users    drop_table :user_configurations  endend

In classically annoying Rails style, if the class name of your migration doesn’t perfectly "match" the  filename then rake migrate will fail mysteriously with an unhelpful error message and a mile-long stack-trace with none of your code in it.  E.g. if you name the above file 005_add_model_security_tables.rb (note the extra underscore between "model" and "security") you’ll get an error message like this:

rake aborted!uninitialized constant AddModelSecurityTables

or if you run rake migrate --trace you’ll get this stack trace:

** Invoke migrate (first_time)** Invoke db:migrate (first_time)** Invoke environment (first_time)** Execute environment** Execute db:migraterake aborted!uninitialized constant AddModelSecurityTables/usr/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/dependencies.rb:266:in `load_missing_constant'/usr/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/dependencies.rb:452:in `const_missing'/usr/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/dependencies.rb:464:in `const_missing'/usr/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/inflector.rb:250:in `constantize'/usr/lib/ruby/gems/1.8/gems/activesupport-1.4.1/lib/active_support/core_ext/string/inflections.rb:148:in `constantize'/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.2/lib/active_record/migration.rb:366:in `migration_class'/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.2/lib/active_record/migration.rb:346:in `migration_classes'/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.2/lib/active_record/connection_adapters/mysql_adapter.rb:248:in `inject'/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.2/lib/active_record/migration.rb:342:in `each'/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.2/lib/active_record/migration.rb:342:in `inject'/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.2/lib/active_record/migration.rb:342:in `migration_classes'/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.2/lib/active_record/migration.rb:330:in `migrate'

Then you might grep your code for "AddModelSecurityTables" and find that it’s not there because you have "AddModelsecurityTables" (difference in upper- vs. lower-case S).  This kind of thing is why Rails is still a bad choice for complex projects — a small hard-to-see typo results in the system not running and providing almost no useful feedback about what’s wrong.  And yet we keep trying to use Rails.  Because it seems to have so much potential.

I’ve often heard that "security through obscurity is no security."  I don’t agree with this one bit.  After all a password is only useful in that its value is obscure.  DRM schemes rely on obscurity — they are only effective because the public doesn’t know where or how the secure media’s decryption keys are stored.  Similarly, it actually does help to use a non-standard port number for exposing a service to the public net that you really don’t want outsiders using — it significantly reduces the number of people who will notice that the service is exposed and even think to try to get in.  The key difference is what kind of hacker are you protecting against?  Casual or determined?  Protecting against casual hackers is a totally valid and useful technique, especially for those of us who nobody really cares about hacking.  But if foreign teenagers can win big bragging rights by finding a hole in your security, you’d better lock things down really well.

Inconvenience is another security technique that doesn’t get talked about as much, but shares many qualities with obscurity.  An example of security through inconvenience is streaming video using flash — the content isn’t encrypted at all, but is protected by the fact that there aren’t any common tools for saving streaming flash video to a persistent (non-streaming) format.  You could also put this into the category of legal security since Adobe can send cease-and-desist letters to anybody distributing such a tool.

I use inconvenience security to protect my private wifi network at my house.  I use WPA which still seems to be not installed by default on most windows machines.  For a long time (probably still) there’s been a bug in the installer for the WPA patch that Microsoft distributes so that it wouldn’t install on XP machines with Service Pack 2.  (It complains that you need Service Pack 1 or later installed in order to use the patch, and good luck trying to convince it that you do.)  This inconvenience security provides effective protection against casual war drivers who are just looking for a little bandwidth to borrow.  Very few of these casual hackers can even try to use my network, so they just drive on.  I also employ a second security technique to protect my private wifi that is based on a carrot rather than a stick.  I offer a free, open wifi network that is outside my firewall. 

On the other hand, inconvenience security is totally inappropriate for something like a corporate VPN.  An IT manager that chose a VPN because it’s not installed by default on client machines is only making things more difficult for legitimate users.  Corporate VPN’s need to do a rock-solid job of protecting themselves from foreign teenagers as well as hired corporate espionageers.  So while making it inconvenient to even try to connect will dissuade casual hackers.  If you’re doing your job at all well, the casual hacker wouldn’t have gotten in anyway, and you’re also making your actual customers’ lives more difficult.

Here’s the best reason I’ve heard to upgrade your browser in a long time.  Apparently a number of people are finding their gmail accounts wiped clean.  This is potentially linked to a known security flaw in Firefox 2.0 which was fixed in 2.0.0.1 related to cross-site scripting (XSS).  So upgrade your firefox, yo.

I’ve heard from security experts that FF has way more problems than IE does, but it just doesn’t get attacked as much.  I hate to say it, but we’re probably already living in a world where all software not only has bugs but has security bugs too.  We just gotta learn to live with it.

We might say “google should keep backups.” But then again, for a lot of
things, we probably don’t want google to keep backups. The gubment can
read backups, afterall. Data security vs. privacy is an intrinsic
trade-off that people don’t seem to think of much yet.

Here’s a little food for thought about hacking into a development system.  If you wanted to gain control of somebody’s network how would you do it?  Well, you’d probably try to figure out a way to get one of the computers on the inside of their firewall to run some code for you.  If you could get it to run an arbitrary block of code that you wrote, then you’re probably pretty close to 0wning it.

Now think about the continuous integration server in your development farm.  What does it do?  Whenever anybody checks in new code, it runs all the unit tests to make sure they still pass.  Or, look at it this way: it takes whatever code anybody checks into the source control system and … compiles it and runs it.  This means that unless you’re being really careful, anybody who has write access to your source control system has control over your CI server and complete access to your network.

Old strategies for containing this mess included running the CI daemon as a limited authority user or chroot’ing the process.  These days, I think putting the CI server in a dedicated virtual machine is the way to go.  VMWare’s newly free Server product is perfect for this. 

These strategies can limit what somebody can do to the CI server itself.  But regardless of that, they’ve got open access to your network from inside your firewall.  So if you’re being really paranoid (a fine quality in a system administrator IMHO) cut it off from the rest of the network except the source control server.  The CI machine generally needs to be able to send e-mail to let folks know when things break, which means it also needs outbound SMTP access.  The smart hacker will use this to impersonate somebody within your org and get deeper in through social-engineering.  The best way I can see around that is to have a process on the email server poll the status of the last build on the CI system (say over HTTP, perhaps checking an RSS feed that many CI systems support) and send e-mail as appropriate.  Remember — your network firewall rules isolating this box don’t have to be symmetric.  It shouldn’t be able to see out, but other boxes can still get in.

This should make you think seriously about how accessible your SVN server is.  What kinds of passwords do your users have on it?  Do you require HTTPS?  Do you require client certs?  What about cached SVN credentials on all those dev boxes?  Remember — if you’re running a CI server, SVN write access in the wrong hands translates pretty quickly into a whole lot more access.