My housemate Ellery created these mac ‘n’ cheese cupcakes for dinner the other night. The frosting is mashed potatoes, and they’re topped with a cherry tomato. Inside is a meatless meaty macaroni and cheese combination surrounded by a savory dough. They were super fun and tasty.

I can take very little credit for these beyond the photo. And helping to consume them. But it’s a great example of why I love living with fun creative people! I’ve heard many requests for the recipe — stay tuned! It’s not mine to share, but when Ellery writes it down I’ll be sure to let you know. (And update this page.)

It was quite a chore to get Apple to admit that the cause was a hardware problem, and fix it.  But I finally succeeded, so I thought I’d share some of my experiences.  I’ll explain what a Kernel Panic is, how they sometimes can be caused by faulty software but often indicate hardware problems, how they differ from other kinds of crashes, and provide a guide on how to read a Mac OS X kernel panic report.

Dealing with the “Genius” Bar staff

“Genius” is what Apple calls its first tier of technical support.  I find the brand unfortunate and insulting for everybody involved.  There is no intelligence test required to work as a “genius” — just some minimal training on how to follow Apple customer service scripts like an obedient robot.  Knowing Apple, I wouldn’t be surprised if the “Genius” staff are required to follow these scripts verbatim and face not only termination but punitive lawsuits for deviating from the party line.  Keep this in mind when dealing with them.  Also know that they have some discretion in the outcome of your visit, but the discretion exists within guidelines that they cannot control.

Some tips on getting past the “genius” from my limited experience.  Print out your kernel panic reports and bring them in.  The more the better.  Highlight the relevant parts.  I’m not sure if bringing a bad attitude with you helps or not — they want to make their customers happy, but they don’t like their “genius” title challenged with logic.  I also recommend persistence.  Following their stupid advice and showing them that it did no good will help.  I’m not sure if understanding what’s going on will or not.  But if you’d like to understand more about why your Mac is crashing, read on…

Kernel panics and hardware failures vs regular software failures

There are two basic ways your Mac can crash.  First, an application might lock up on you and become unresponsive.  You get the spinning beachball of death, and eventually have to Force Quit your application, losing whatever work you hadn’t saved.  This kind of user mode failure is very common with buggy software.  If the beachball is getting you down, the problem is almost certainly caused by bad software, not by a hardware problem.  In OS 9 and before, this kind of failure could have taken down your entire machine, but since the introduction of the BSD kernel in OS X, the system is designed to allow one application to fail while protecting all the other applications.

Sometimes though your entire Mac will crash hard.  Without warning your system displays a full-screen message saying “You need to restart your computer. Hold down the Power button for several seconds or press the Restart button.” in several languages.  This is OS X’s last ditch attempt to tell you something about what happened before it goes completely teets up.  It’s formally known as a kernel panic.  Sometimes the system is so screwed it can’t even get that error message onto the screen before it dies.

Kernel panics indicate a serious problem, either with the computer’s hardware, or the low-level software in the operating system. In fact there are only three things that can cause a kernel panic:

1. Faulty hardware causes a problem that the OS doesn’t know how to deal with
2. A bug in OS X itself
3. A bug in an OS plugin called a kernel extension or kext

Firstly, if the hardware itself has problems, then kernel panics are a common way they manifest themselves.  Similarly, if the operating system itself has any bugs, they could take down the entire system.  The third option could be caused by third-party software, while the first two are entirely Apple’s responsibility.  So when it comes to dealing with the “Genius” behind the bar, the first two are fairly straightforward.  If you’re seeing this problem a lot, and nobody else is, then it’s probably a hardware problem, and they should replace your hardware. Here’s a thought experiment I tried unsuccessfully with the Apple “geniuses” I had to deal with: Imagine you have a hundred Macs all running the same software, and one of them crashes periodically, but the other 99 don’t.  Would you classify that Mac as having a hardware problem or a software problem?  In my case, the genius insisted that it was a software problem.  In fact he claimed he was certain that if I uninstalled Adobe Flash, the problem would be fixed.  Read on, and you’ll learn how the kernel panic reports themselves show that this explanation is impossible.

Understanding and interpreting Kernel Panic reports

First a bit about what a Kernel Panic is.  Very simply, it’s when something unexpected goes wrong in the operating system kernel.  What’s the kernel?  The kernel is the lowest level of the operating system — the part that’s closest to the hardware.  In modern operating systems, there’s a fairly arbitrary line between what functionality lives in the kernel and what functionality lives in the user space.  The key difference is that when something goes wrong with software in the user space, you get a beachball on the app, but the system survives.  When something goes wrong in the kernel, you get a kernel panic, and the whole system goes bye bye fast.  So it’s critical that any code running in the kernel space be ultra reliable.  You don’t change kernel code quickly or lightly, and you test the hell out of it before you release it.  But code runs faster in the kernel, so most modern operating systems put important things like networking and graphics into the kernel.  The BSD kernel which powers OS X allows the installation of “kernel extensions” or “kexts” which add functionality.  More about these soon.  But suffice to say that when anything goes wrong with any kext, it’s a big deal problem because there’s nothing to fall back on (e.g. can’t display an error dialog if the problem is with the display system), so the system’s reaction is called a panic.  Thus “kernel panic.”

Immediately after a KP, your computer does two things: it stores a bunch of information to help diagnose what caused the problem, and puts up the error screen, if it can.  When you reboot, your computer asks if you want to send the KP report to Apple.  You should do this.  The smarter of the “genius” staff can look these reports up and see that your Mac is actually crashing, but they’ll admit that the contents are too technical for a mere “genius” to understand.  Well I’m going to explain to you what the reports contain and what it means about what’s wrong with your computer.

Here’s a typical crash report from my computer.  In my case, these panics weren’t even accompanied by the “restart your computer message” because as I’ll explain, the problem originated in the graphics system.  My computer just suddenly went black and non-responsive.  I’ve highlighted a few key sections for explanation below.

Interval Since Last Panic Report:  420 sec
Panics Since Last Report:          1
Anonymous UUID:                    8A09F455-1039-4696-8479-xxxxxxxxxxxx
Thu Apr 21 09:00:51 2011
panic(cpu 3 caller 0x9cdc8f): NVRM[0/1:0:0]: Read Error 0x00000100: CFG 0xffffffff 0xffffffff 0xffffffff, BAR0 0xc0000000 0xa734e000 0x0a5480a2, D0, P2/4
Backtrace (CPU 3), Frame : Return Address (4 potential args on stack)
0xbc001728 : 0x21b510 (0x5d9514 0xbc00175c 0x223978 0x0)
0xbc001778 : 0x9cdc8f (0xbe323c 0xc53840 0xbf23cc 0x0)
0xbc001818 : 0xae85d3 (0xe0cfc04 0xe5c9004 0x100 0xb83de000)
0xbc001868 : 0xadf5cc (0xe5c9004 0x100 0xbc001898 0x9bd76c)
0xbc001898 : 0x16c8965 (0xe5c9004 0x100 0x438004ee 0x28)
0xbc0019d8 : 0xb07250 (0xe5c9004 0xe5ca004 0x0 0x0)
0xbc001a18 : 0x9d6e23 (0xe5c9004 0xe5ca004 0x0 0x0)
0xbc001ab8 : 0x9d3502 (0x0 0x9 0x0 0x0)
0xbc001c68 : 0x9d4aa0 (0x0 0x600d600d 0x704a 0xbc001c98)
0xbc001d38 : 0xc89217 (0xbc001d58 0x0 0x98 0x2a358d)
0xbc001df8 : 0xc8ec1d (0xe8e5404 0x0 0x98 0x45e8d022)
0xbc001f18 : 0xc8f0b4 (0xe8e5404 0x124b6204 0x6d39d1c0 0x0)
0xbc001f78 : 0xc8f39f (0xe8e5404 0x124b6204 0x6d39d1c0 0xbc0021e0)
0xbc002028 : 0xca3691 (0xe8e5404 0x1f80d8e8 0xbc00239c 0xbc0021e0)
0xbc002298 : 0xc84d09 (0x6d0b7000 0x1f80d8e8 0xbc00239c 0x0)
0xbc0023f8 : 0xc84f47 (0x6d0c6000 0x1f80d800 0x1 0x0)
0xbc002428 : 0xc87a04 (0x6d0c6000 0x1f80d800 0x0 0x97c6c4fc)
0xbc002468 : 0xca9d40 (0x6d0c6000 0x1f80d800 0x6d09f274 0x140)
0xbc0024f8 : 0xc9b5a9 (0xde94bc0 0x1f80d800 0x0 0x1)
0xbc002558 : 0xc9b810 (0x6d09f000 0x6d09f77c 0x1f80d800 0x0)
0xbc0025a8 : 0xc9bce4 (0x6d09f000 0x6d09f77c 0xbc0028cc 0xbc00286c)
0xbc0028e8 : 0xc98aaf (0x6d09f000 0x6d09f77c 0x1 0x0)
0xbc002908 : 0xc605a1 (0x6d09f000 0x6d09f77c 0x1956a580 0x0)
0xbc002938 : 0xc9a572 (0x6d09f000 0xbc002a7c 0xbc002968 0x5046b1)
0xbc002978 : 0xc648de (0x6d09f000 0xbc002a7c 0x0 0xc000401)
0xbc002ab8 : 0xc9dee6 (0x6d09f000 0x0 0xbc002bcc 0xbc002bc8)
0xbc002b68 : 0xc60c93 (0x6d09f000 0x0 0xbc002bcc 0xbc002bc8)
0xbc002be8 : 0x56a738 (0x6d09f000 0x0 0xbc002e3c 0xbc002c74)
0xbc002c38 : 0x56afd7 (0xcef020 0x6d09f000 0x129bab88 0x1)
0xbc002c88 : 0x56b88b (0x6d09f000 0x10 0xbc002cd0 0x0)
0xbc002da8 : 0x285be0 (0x6d09f000 0x10 0x129bab88 0x1)
0xbc003e58 : 0x21d8be (0x129bab60 0x1ec235a0 0x1fd7e8 0x5f43)
      Backtrace continues...

      Kernel Extensions in backtrace (with dependencies):
         com.apple.GeForce(6.2.6)@0xc55000->0xd0afff
            dependency: com.apple.NVDAResman(6.2.6)@0x967000
            dependency: com.apple.iokit.IONDRVSupport(2.2)@0x95a000
            dependency: com.apple.iokit.IOPCIFamily(2.6)@0x927000
            dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x938000
         com.apple.nvidia.nv50hal(6.2.6)@0x1592000->0x19a6fff
            dependency: com.apple.NVDAResman(6.2.6)@0x967000
         com.apple.NVDAResman(6.2.6)@0x967000->0xc54fff
            dependency: com.apple.iokit.IOPCIFamily(2.6)@0x927000
            dependency: com.apple.iokit.IONDRVSupport(2.2)@0x95a000
            dependency: com.apple.iokit.IOGraphicsFamily(2.2)@0x938000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
10J869
Kernel version:
Darwin Kernel Version 10.7.0: Sat Jan 29 15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386
System model name: MacBookPro6,2 (Mac-F22586C8)
System uptime in nanoseconds: 35829130822125

unloaded kexts:
com.apple.filesystems.msdosfs 1.6.3 (addr 0xbc1e5000, size 0x53248) - last unloaded 12216461868115

loaded kexts:
com.parallels.kext.prl_vnic 6.0 11992.625164
com.parallels.kext.prl_netbridge 6.0 11992.625164
com.parallels.kext.prl_usb_connect 6.0 11992.625164
com.parallels.kext.prl_hid_hook 6.0 11992.625164
com.parallels.kext.prl_hypervisor 6.0 11992.625164
com.apple.filesystems.smbfs 1.6.6 - last loaded 12151022138289
com.apple.driver.AppleHWSensor 1.9.3d0
com.apple.driver.AGPM 100.12.19
com.apple.driver.AppleMikeyHIDDriver 1.2.0
com.apple.driver.AppleHDA 1.9.9f12
com.apple.driver.AppleUpstreamUserClient 3.5.4
com.apple.driver.AppleMCCSControl 1.0.17
com.apple.driver.AppleMikeyDriver 1.9.9f12
com.apple.driver.AudioAUUC 1.54
com.apple.driver.AppleIntelHDGraphics 6.2.6
com.apple.driver.AppleIntelHDGraphicsFB 6.2.6
com.apple.driver.SMCMotionSensor 3.0.0d4
com.apple.kext.AppleSMCLMU 1.5.0d3
com.apple.Dont_Steal_Mac_OS_X 7.0.0
com.apple.iokit.CHUDUtils 201
com.apple.iokit.CHUDProf 216
com.apple.driver.AudioIPCDriver 1.1.6
com.apple.driver.AppleGraphicsControl 2.8.68
com.apple.driver.ACPI_SMC_PlatformPlugin 4.5.0d5
com.apple.GeForce 6.2.6
com.apple.driver.AppleLPC 1.4.12
com.apple.filesystems.autofs 2.1.0
com.apple.driver.AppleUSBTCButtons 200.3.2
com.apple.driver.AppleUSBTCKeyboard 200.3.2
com.apple.driver.AppleIRController 303.8
com.apple.driver.AppleUSBCardReader 2.5.8
com.apple.iokit.SCSITaskUserClient 2.6.5
com.apple.BootCache 31
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.iokit.IOAHCIBlockStorage 1.6.3
com.apple.driver.AppleUSBHub 4.1.7
com.apple.driver.AppleFWOHCI 4.7.1
com.apple.driver.AirPortBrcm43224 427.36.9
com.apple.iokit.AppleBCM5701Ethernet 2.3.9b6
com.apple.driver.AppleEFINVRAM 1.4.0
com.apple.driver.AppleSmartBatteryManager 160.0.0
com.apple.driver.AppleUSBEHCI 4.1.8
com.apple.driver.AppleAHCIPort 2.1.5
com.apple.driver.AppleACPIButtons 1.3.5
com.apple.driver.AppleRTC 1.3.1
com.apple.driver.AppleHPET 1.5
com.apple.driver.AppleSMBIOS 1.6
com.apple.driver.AppleACPIEC 1.3.5
com.apple.driver.AppleAPIC 1.4
com.apple.driver.AppleIntelCPUPowerManagementClient 105.13.0
com.apple.security.sandbox 1
com.apple.security.quarantine 0
com.apple.nke.applicationfirewall 2.1.11
com.apple.driver.AppleIntelCPUPowerManagement 105.13.0
com.apple.driver.DspFuncLib 1.9.9f12
com.apple.driver.AppleProfileReadCounterAction 17
com.apple.driver.AppleProfileTimestampAction 10
com.apple.driver.AppleProfileThreadInfoAction 14
com.apple.driver.AppleProfileRegisterStateAction 10
com.apple.driver.AppleProfileKEventAction 10
com.apple.driver.AppleProfileCallstackAction 20
com.apple.driver.AppleSMBusController 1.0.8d0
com.apple.iokit.IOFireWireIP 2.0.3
com.apple.iokit.IOSurface 74.2
com.apple.iokit.IOBluetoothSerialManager 2.4.0f1
com.apple.iokit.IOSerialFamily 10.0.3
com.apple.iokit.CHUDKernLib 208
com.apple.iokit.IOAudioFamily 1.8.0fc1
com.apple.kext.OSvKernDSPLib 1.3
com.apple.driver.AppleHDAController 1.9.9f12
com.apple.iokit.IOHDAFamily 1.9.9f12
com.apple.iokit.AppleProfileFamily 41
com.apple.driver.AppleSMC 3.1.0d3
com.apple.driver.IOPlatformPluginFamily 4.5.0d5
com.apple.driver.AppleSMBusPCI 1.0.8d0
com.apple.nvidia.nv50hal 6.2.6
com.apple.NVDAResman 6.2.6
com.apple.iokit.IONDRVSupport 2.2
com.apple.iokit.IOGraphicsFamily 2.2
com.apple.driver.BroadcomUSBBluetoothHCIController 2.4.0f1
com.apple.driver.AppleUSBBluetoothHCIController 2.4.0f1
com.apple.iokit.IOBluetoothFamily 2.4.0f1
com.apple.driver.AppleUSBMultitouch 206.6
com.apple.iokit.IOUSBHIDDriver 4.1.5
com.apple.iokit.IOSCSIBlockCommandsDevice 2.6.5
com.apple.iokit.IOUSBMassStorageClass 2.6.5
com.apple.driver.AppleUSBMergeNub 4.1.8
com.apple.driver.AppleUSBComposite 3.9.0
com.apple.iokit.IOSCSIMultimediaCommandsDevice 2.6.5
com.apple.iokit.IOBDStorageFamily 1.6
com.apple.iokit.IODVDStorageFamily 1.6
com.apple.iokit.IOCDStorageFamily 1.6
com.apple.driver.XsanFilter 402.1
com.apple.iokit.IOAHCISerialATAPI 1.2.5
com.apple.iokit.IOSCSIArchitectureModelFamily 2.6.5
com.apple.iokit.IOUSBUserClient 4.1.5
com.apple.iokit.IOFireWireFamily 4.2.6
com.apple.iokit.IO80211Family 314.1.1
com.apple.iokit.IONetworkingFamily 1.10
com.apple.iokit.IOUSBFamily 4.1.8
com.apple.iokit.IOAHCIFamily 2.0.4
com.apple.driver.AppleEFIRuntime 1.4.0
com.apple.iokit.IOHIDFamily 1.6.5
com.apple.iokit.IOSMBusFamily 1.1
com.apple.kext.AppleMatch 1.0.0d1
com.apple.security.TMSafetyNet 6
com.apple.driver.DiskImages 289
com.apple.iokit.IOStorageFamily 1.6.2
com.apple.driver.AppleACPIPlatform 1.3.5
com.apple.iokit.IOPCIFamily 2.6
com.apple.iokit.IOACPIFamily 1.3.0

The first line is fairly clear — how long has your system been running since its last crash?  If this is less than an hour, as it was for my computer, then your machine is completely FUBAR.  Less than a day and you’ve still got a seriously unstable computer.  (Hint for any “genius” that might be reading this article: take the number of seconds, divide it by 60 using the Calculator app on your store-issued-iPad, and that will give you the number of minutes.  Divide that new smaller number by 60 again to get an even smaller number which is hours.  If you can figure out how to get to number of days by yourself, it’s time to apply for the “Genius Lead” job.)

The Anonymous UUID is an effectively random code that allows Apple to lookup the crash reports for your computer when you go into the store.  Then there’s the date.  Straightforward.

The line which starts “panic” is the closest thing you’ll find to a concise explanation of what went wrong. In all likelihood this will be a jumble of words and numbers that make no sense, but it’s a great string to Google.  If you’re having a hardware problem, this message will probably stay about the same with each KP.  Googling my error message “NVRM[0/1:0:0]: Read Error 0×00000100” turns up a bunch of people with similar problems — computer going black without warning, often while playing World of Warcraft.

The next section titled “backtrace” is worthless unless you’re actually diving into the source code that caused the problem.  Skip over it.  But the section after it is extremely interesting and relatively easy to interpret.

The section titled “Kernel Extensions in backtrace (with dependencies)” actually tells you what part of the system failed.  Read this one closely and try to make sense of it. In the case of my example, there are three kernel extensions involved with the crash.  They are called “com.apple.GeForce” and “com.apple.nvidia.nv50hal” and “com.apple.NVDAResman”.  The first one is fairly obvious — GeForce is the kind of graphics chip in the macbook.  The second one is also pretty clear — NVidia is the company that makes GeForce, and nv50hal I would guess means “NVidia 5.0 Hardware Abstraction Layer” or something similar.  I’m not sure what NVDAResman is but looking down a bit I see it’s related to “IOGraphicsFamily”.  This paints a really clear picture that the failure is in the graphics system.  Moreover, since every line here starts with “com.apple” we know the failure is entirely in code written by Apple.  There is no third-party software involved in this crash.

For my particular crash, it’s important to know something about the graphics hardware of these MacBooks, since all evidence points to the graphics hardware.  This generation of macbooks have two graphics chips — a faster one from Nvidia, and a more battery-friendly one from Intel.  The nvidia chip which is apparently having problems is always used when the computer has an external monitor plugged in, or when something fancy is happening on the built-in screen.  A nice utility called gfxCardStatus can help you understand this complexity, and will definitely give you a leg up on the “genius.”

The following line starting with “BSD process name” can also be important.  This will sometimes tell you which user-level app originated the call into the kernel which failed.  In my case it was “kernel_task” which provides no additional information.

The next section gives some basic info about the Mac — hardware and OS versions.  What follows is a complete list of kernel extensions (kexts) installed.  This gives you a bit more ammo in dealing with the “genius” who is probably ignoring you at this point anyway.  You can look through this list and see everything that might possibly contribute to a kernel panic.  In my case, the only software modules that aren’t from Apple are some drivers from Parallels for running my Windows virtual machine.  So the only reasons my Mac might kernel panic are because of a hardware problem, a bug in OS X itself, or something going wrong with Parallels.  Understanding this should, in theory, be very helpful when talking to your local neighborhood “genius” but unfortunately they are simple bots that only run scripts authored in Cupertino and are not permitted to listen to logic.

Apple’s Propaganda about Flash

When the “genius” told me my Mac’s problem was that I had Adobe Flash installed, I just laughed at first.  Flash is installed on something like 97% of desktop computers, and very few of them regularly turn themselves off for no reason.   Moreover, the kernel panic report lists every piece of software that could possibly contribute to the kernel panic, and neither the word “flash” nor “adobe” appear anywhere in the list.  But then I realized he wasn’t joking.

Apple’s ongoing arguments with Adobe over Flash are well publicized.  The root of the issue, in very brief summary, is that Apple sees Adobe’s Flash as a strategic threat to their incredibly profitable iPhone platform.  The poor “genius” I’m stuck with has become a pawn in Apple’s PR battle, throwing himself on the grenade of propaganda just to spread FUD about Flash.  I tried reasoning with him, explaining that Adobe’s software doesn’t run in the kernel, and therefore cannot cause a kernel panic.  The job of the kernel is to protect users from badly written software crashing the whole machine. But he would not budge.  I imagined a “genius” script which read as follows:

Mac is crashing…

1. Run hardware diagnostic tests.

2. Address any identified hardware problems.

3. If hardware tests come back clean, tell customer that the problem (whatever it is) is caused by Flash.  Tell them to uninstall it, and see if that helps.

Here I imagine the Dantesque trap of the rare “genius” who actually understands how OS X works.  I’m telling the customer something which is impossible on its face, and he knows it.  He’s arguing with me telling me I’m being stupid.  But I signed a contract with Apple saying I would defame Adobe, and deviation from this contract will bring the wrath of Steve’s legal team on me.  I just have to smile and say things like “yeah, that’s the really strange thing about this particular software problem — it only affects certain computers.  But it’s definitely caused by Flash.”

One might reason that Flash could cause kernel panics because it makes more extensive use of the graphics system than other applications.  But in this case, Flash isn’t the actual problem.  Flash is exposing the underlying problem, as would any software which works the graphics system hard.  Thus lots of people with the same problem as me who play World of Warcraft.  If the “genius” advice ever works, it’s just because Flash is the most graphics intensive software that many people use on their Macs.  The actual problem is still either a bug in OS X, or a hardware problem.

Consider the advice not to use Flash on your Mac in analogy to a car.  (A high-end MacBook actually costs as much as some cars.)  Imagine that your car sometimes just turned its engine off while you were in the middle of driving it – catastrophic failure with no warning or apparent reason.  You go to the dealership and they can’t find anything wrong with it, but ask if you ever listen to electronic music?  Well, yes, sometimes.  That’s the problem!  It’s the electronic music which is causing your car to malfunction.  So stop listening to it, and the problem will be fixed.  Umm, what?  The closest thing to the truth, by analogy, would be that any bass-heavy music (graphics-intensive application) is stressing out some weak connection in the electronics.  But because the car dealership is owned by the local philharmonic, they’re blaming it on that awful music the kids listen to.   Using your misfortune and their incompetence to push an unrelated political agenda.

It’s an interesting glimpse into how Apple is using their retail presence to advance a strategic PR goal.  Evidence that Apple has grown up as a company to the point where their own motives are more important than doing what actually helps customers.  *sigh*  At least I got my MacBook fixed.

The simple way to counter buffer bloat is simply to reduce the size of the transmit buffer in each piece of your network gear.  Most linux systems default to a transmit buffer of 1,000 packets, each of which can be 1.5 kilobytes, meaning that 1.5 megabytes of data can get queued up waiting for a chance to go across the network.  Any application that is trying to move a lot of data through a clogged network will fill this buffer.  That’s fine for the buffer-filling application, but any other application will suffer.  So, for example, if you’re watching youtube and your roommate is trying to surf the net, your roommate’s web page requests will suffer very long latency, because their small web pages must get in this megabyte-long line along with your youtube video before they can be delivered.  If your DSL line runs at say 10 mbps, then it’ll take 1.2 seconds for that 1.5 MB buffer to work fit through your pipe.  Since it takes at least 2 round-trips to get a web page that means your roommate’s web page will take at least 2.4 seconds to show up, no matter how small it is!

Gettys quotes Kleinrock that the ideal size of a network buffer is (bandwidth) x (latency).  Say your bandwidth is 10 mbps.  Latency to any web page you’re likely to visit in the US should be less than 100ms, so let’s use that.  This puts your ideal buffer size 125k.  Buffer sizes are usually configured in terms of maximum number of packets.  Typically the maximum packet size (MTU) is 1500 bytes, resulting in the ideal theoretical buffer size of 83 packets for a typical fastish home network line.  Please redo these calculations yourself and experiment with how different numbers affect your system. (Be careful not to set your buffer size to zero as it could lock up the device’s network.)  Remember that linux (which is likely what your wifi router is running) defaults to 1,000 packets!

[Update shortly after posting: a reader suggested I try setting my buffer to be much smaller still.  So I went down to just 2 packets, and noticed that my ping times are much more reliable now when the network has more than one thing.  His caveat which I will echo is that this will mess with your system if your router is trying to do any kind of traffic shaping, i.e. QoS.  But otherwise protocols like TCP will keep everything running fine.]

If your home wifi access points are using DD-WRT as mine are, here’s how you set them to use a more sane buffer size:

1. Log in to your router’s admin web page.

2. Select the Administration tab and the Commands sub-tab

3. Type in the following commands into the box:

ifconfig eth0 txqueuelen 2

ifconfig eth1 txqueuelen 2

4. Click the “Save Startup” button at the bottom.

There — you’re done!  For alternate techniques to configure your dd-wrt router for this kind of thing, see the wiki page on Startup Scripts.

I’m sharing this information because it took me a while to figure out.  This problem is not well documented.  I’m trying this out now on my house’s network now.  In some controlled tests it seems like it might be somewhat better.  But my tests have not been able to replicate the really horrible situations I’ve seen on our network which I suspect come from lots of simultaneous users.  So it’ll be a while before we know for sure if this was a good change.  To be clear, I don’t know if this advice is good or not. It could reduce your network’s maximum effective bandwidth, but hopefully it will do so by reducing the maximum latency, which is often a very good trade-off.  This advice is consistent with the advice Gettys offers in terms of optimizing buffer sizes, and make sense to me.  YMMV.  If you try it out, please leave a comment on whether or not it helps you.

I was patiently waiting for the feature to be turned for my account, which wasn’t happening.  I kept getting the same old spreadsheet program.  But then I found the setting to enable it.  So I’m sharing with y’all an explanation of how you too can get these great new features.

Go to http://docs.google.com/ and click the select “Settings” link in the upper-right hand corner, and choose “Documents Settings”.

Then choose the “Editing” tab and check the checkbox next to “New version of Google documents”.

Now any new documents you create will use the new real-time editor.  And anybody you share them with will get the new features without having to set this up for their account.

• Keep all blog posts and comments
• Keep all posts at their original URLs
• Maintain all category pages at the same URLs

The first one’s easy.  Google released some migration tools which cover that quite well.  But, at least when importing from typepad / movable type, they don’t preserve permalink URL’s.  So anybody who followed a linked to a specific page on my site would get a 404 page.  Weak.

I spent a lot of time on this.  Basic problem is that Typepad doesn’t include URL information in their export file format. It would be very easy for them to do this, but then why would they want to make it easy for you to leave?  Actually the answer there is easy.  Because by trying to lock in users, they create angry vocal opponents of their service.  I’m not angry, but I would advise against anybody considering Typepad as a blog host, specifically because of their tendency to lock people in.

<rant> Don’t keep my data hostage.  It’s my content.  I created it.  You’re just delivering it.  Do not try to lock me into using you as a service provider.  You might get some more money out of me, but every dollar I give you after I want to leave will contribute to my dis-liking you.  As the internet matures and consumers become more sophisticated and better able to share their experiences with each other, they will increasingly choose the service providers who are open.  (Echoing Jonathan Rosenberg’s recent diatribe on openness.)  I really appreciate Google’s commitment to Data Liberation.  My current provider, Dreamhost, also does a splendid job of giving me control over my data.</rant>

It turns out that getting a full-fidelity export out of typepad is possible with some work.  I followed these instructions from FolioVision which provides a custom export template that does include URL’s.  If your blog has more than 100 posts, then you need to change the first line to

<MTEntries lastn="100">

<MTEntries lastn="100" offset="100">

<MTEntries lastn="100" offset="200">

etc. and merge all these files together into one big export file that has URL’s.  Then I tried to get blogger to honor the import file with permalinks but I couldn’t.  I do believe blogger is capable of doing this, but what ultimately turned me away from it was that it doesn’t seem to offer any way to honor links like www.embracingchaos.com/humor for category listings.  Which I like and get a lot of visitors on.  So I went with wordpress.

FolioVision helpfully posted a custom wordpress import plugin to match their typepad output template, which makes it all go.  Once that’s done, you have to move all the attachments hosted at typepad, and then there’s a bunch of wordpress configuration, and moving your analytics and favicons and finally switching DNS.

So here we are.  Please tell me if you notice anything amiss with the new site.

• http://movabletype2blogger.appspot.com/
• http://wordpress2blogger.appspot.com/
• http://livejournal2blogger.appspot.com/

Well I tried this with my blog, and the resulting file it spit out was almost empty.  I think my blog is just a bit too large, since when I ran it on my local machine it came out to 1.03 megs.  So if your blog is smaller than mine you can probably use the online tool.

After a couple of bX-xji785 errors, the file imported into blogger about as well as could be expected, which is to say okay but not great.  The blog is mostly there.  Feel free to take a peak at http://leo-embracingchaos.blogspot.com/ but please don’t make any permanent links to that URL as it’s really just for testing.  The posts and drafts all made it with the right dates and times, along with the comments and tags.  But as previously noted, the TypePad export format does not include URLs.  So if I were to actually use this conversion, all the inbound links to pages other than the homepage of my blog would break, which is totally unacceptable for me.

I started a thread on the discussion group if you’d like to follow along.

• Better WYSIWYG editing in Blogger.  (I can’t change any font characteristics in Typepad without going into raw HTML. Ugh.)
• Typepad’s lack of support for Chrome, Google’s awesome new browser
• The awkward and limiting way Typepad assigns human-readable URLs to posts
• Blogger is free, while Typepad costs >$100 per year
• Cool new features being added to Blogger like Followers
• Adding new features in Typepad is a p.i.t.a. (i.e. I’m too old to be messing with perl-based templating languages.  Did more than my share of that in 1996.)

Typepad has long supported an export feature, which dumps out all the posts, drafts, comments, etc into a text file.  And now blogger has an import feature, which takes an XML file that looks kinda like an Atom feed.  I looked around for a tool to convert between the two and found only others asking the same question. 

So I thought here’s a chance for me to contribute something to
society and help myself at the same time.  Converting one text file to
another shouldn’t be that hard.  So I rolled up my sleeves and started
playing with it.  I realized one problem fairly quickly — the typepad export format
doesn’t include the URL for each post.  I really don’t want to break
all the previous inbound links because
that’s how people get to my content.  But I’m going to need to crawl
the old blog to get those old links.

Since the blogger format looks like a regular Atom feed, I thought I’d try to just grab the Atom feed off the blog and import it into Blogger.  Well to that, blogger said:

Blogger does not currently support import files generated by TypePad.

Fine.  But what are the differences?  The blogger format includes a bunch of fake "entry" elements which are really configuration — like a giant "layout" node and a bunch of other settings.  So I tried grafting the two together — just taking the legitimate "entry" nodes from the typepad feed and putting them into the blogger export that includes all the layout and settings.  By converting everything to look like the blogger format, it imports, but that also loses information.  So I set about looking for the minimal set of changes that will work.

One that definitely breaks it is a side-efect of being hooked-up to feedburner.  There’s extra stuff in there like the feedburner:origLink tags:

    <feedburner:origLink>http://www.embracingchaos.com/2008/12/repairing-a-deg.html</feedburner:origLink>

These break the import, which shouldn’t be much of a surprise since the feedburner: namespace isn’t defined in the Blogger export.

I gave up looking for today because blogger was having too many server errors on import.  They’re intermittent, so I was happy to refresh through them for a while.  But at some point 90% of my imports gave me funny error codes like bX-gxxw6w so I’m giving up for now. 

I wrote a stub of a python program to do the merge.  I called it typepad2blogger.py and happily donate it to the public domain.  It’s definitely not done and is useful only insofar as it shows one way to approach the problem and has enough stuff in place that somebody else should be able to get a running start.  I’ll hopefully continue work on this at some point.  But even a couple hours of work on it has triggered my RSI so I’ll have to take a break.  If you want to pick this up, let me know and we can coordinate efforts.

To convince myself that the repair was successful, I unplugged one of the previously functional drives, and saw that all my files were still there when the array was running just on the new drive and the other previous drive. I recommend this to anybody who thinks they’re running a RAID system — until you’ve seen the RAID array work with a drive removed, how can you be sure it’s really working? If your system is set up better than mine is, you’ll get some kind of warning message too.

My advice for people seeking reliable storage: go with a hosted solution.  Understanding the arcane nuances of these software systems is an extremely specific skill that doesn’t translate well to many real-life necessities.  If you’re smart, you can figure it out, but it doesn’t teach you much of anything except how to do exactly that.  Each person who understands this stuff should be keeping petabytes of data happy, rather than one couple’s pictures and music collections.  I hear Microsoft’s "home server" actually makes this pretty easy, but I can’t recommend anybody willingly lock themselves into Microsoft’s business model.

Background

So I bought a new drive, following my own advice about picking drives from different manufacturers when building a raid array, and plugged it in to the mobo and booted the machine.  After futzing with /etc/fstab to get it to find the boot disk and load up, I logged into evms and got these messages:

MDRaid5RegMgr: RAID5 array md/md1 is miissing the member  with RAID index 0.  The array is running in degrade mode.

and

MDRaid5RegMgr: Region md/md1 is currently in degraded mode.  To bring it back to normal state, add 1 new spare device to replace the faulty or missing device.

Conceptually easy.  I’ve got a new 500 GB drive in the system.  Linux sees it.  It didn’t take me too long to figure out it’s called /dev/sda, while the previous 2 disks in the array are sdb and sdc, with a small boot drive at sdd.  Now the fun part is figuring out EVMS terminology enough to tell it to use the new disk.

The hierarchy of the array in EVMS land seems to be as follows:

• Logical Volume teraraid (contains)
• Region md/md1 (which contains)
• Segments sdb1 and sdc1 (which are built on)
• Logical disks sdb, sdc.

What I tried, and what seems to have worked

I see that logical disk sda has no segments.  So I try Action -> Create -> Segment.  It only gives me one choice for "Segment Manager" which is "GPT Segment Manager."  But when I choose it, it doesn’t let me make a segment on sda.  Only the tiny free space on sdb and sdc.  So sda needs something else done to it before we can use it.  What?

sda also shows up in the list of Logical Volumes, next to Teraraid and the formatted boot partition.  Hmmm.

Well I tried converting it to an EVMS Volume.  It complained that sda does not have a File System Interface Module (FSIM) associated with it, but it made the new logical volume anyway.  This wasn’t getting me anywhere.  So I erased it.

Next I tried "Add" -> "Segment Manager to Storage Object".  I noticed that all of the Disk Segments associated with the array were listed as using "Plug-in" "GptSegM" and this gave me the choice of adding Gpt Segment Manager to sda.  W00t.  I said "No" to make this a system disk.  This seems to be working.  Now I see a bunch of Disk Segments starting with sda, including a big one (465 GB) labelled sda_freespace1. 

Now when I tried to Create -> Segment, it let me use GPT Segment Manager on sda_freespace1 and allocate a 450 GB disk segment to match the others.  (I left 15 GB off each disk with the idea I could put a boot segment in that space, but I’ve never gotten around to it.)

Now in "Available Objects" there is sda1 with 450.0 ready for me.  Alrighty we’re getting there.

Now I look at "Storage Regions" and in the context menu for md/md1 I see an option that says "Add spare to fix degraded array…"  I didn’t see it there before — it might have not shown up when there weren’t any spares, or maybe I was just being thick.  In any case, selecting it now gives me a menu with one choice — sda1. 

Now in details of md/md1 it shows:

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                    Value                                      │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number            9                                          │ md│     Minor Number            1                                          │   │     Name                    md/md1                                     │   │     State                   Discovered, Degraded, Active               │   │     Personality             RAID5                                      │   │ +   Working SuperBlock                                                 │   │     Number of disks         3                                          │   │ +   Disk 1                  sdb1                                       │   │ +   Disk 2                  sdc1                                       │   │     Number of stale disks   1                                          │   │ +   Stale disk 0            sda1                                       │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

That last line about the Stale disk is new.

Actions -> Save commits these changes to disk.  Now looking at Detailed information for md/md1 shows

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                    Value                                      │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number            9                                          │ md│     Minor Number            1                                          │   │     Name                    md/md1                                     │   │     State                   Discovered, Degraded, Active, Syncing =  0 │   │     Personality             RAID5                                      │   │ +   Working SuperBlock                                                 │   │     Number of disks         3                                          │   │ +   Disk 1                  sdb1                                       │   │ +   Disk 2                  sdc1                                       │   │     Number of stale disks   1                                          │   │ +   Stale disk 0            sda1                                       │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

Emotionally I feel like I should be done now.  But I don’t hear the thrashing noise of a half-terabyte of of checksums being unwound and copied onto a fresh disk.  And it says "Syncing = 0".  Hmmm.

I quit evmsn and reload it to see two new messages.  One familiar:

MDRaid5RegMgr: Region md/md1 is currently in degraded mode.  To bring it
back to normal state, add 1 new spare device to replace the faulty or missing device.

And one novel:

MDRaid5RegMgr: RAID5 array md/md1 is missing the member  with RAID index 0.  The array is running in degrade mode.  The MD recovery process is running, please wait…

But this novel message saying it’s recovering is "Number 0" implying that it came before the other message (Number 1) which tells me I need to take action for it to fix itself.  And the drives are not thrashing.  Again I look at the details for md/md1 and now I see:

 Na┌──────────────────── Detailed Information - md/md1 ─────────────────────┐ ──│                                                                        │── lv│     Name                 Value                                         │ lv│ ────────────────────────────────────────────────────────────────────── │ lv│     Major Number         9                                             │ md│     Minor Number         1                                             │   │     Name                 md/md1                                        │   │     State                Discovered, Degraded, Active, Syncing =  0.3% │   │     Personality          RAID5                                         │   │ +   Working SuperBlock                                                 │   │     Number of disks      3                                             │   │ +   Disk 1               sdb1                                          │   │ +   Disk 2               sdc1                                          │   │ +   Disk 3               sda1                                          │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │                                                                        │   │    Use spacebar on fields marked with "+" to view more information     │   │                                                                        │   │ [Help]                                                          [OK]   │   │                                                                        │   └────────────────────────────────────────────────────────────────────────┘

Which really seems to say its doing its thing.  Maybe I don’t hear the disks because it’s formating the disk first, which is a linear process.  Or maybe the whole copy process is very linear and I won’t hear it thrashing.  Its progress implies it’s going to take a couple/few days to finish, which is what I’d expect.  So maybe it’s working. I’ll let it run for a while and see what happens to the array if I try to unplug one of the previously working drives.

Pretty cool that I didn’t even need to unmount the array to do this.

Now if I could just figure out why my laser printer periodically decides it needs to print it internal test page, I’d be even happier.

Last week I went to a different kind of Dork Bot meeting here in Seattle called the Catastrophic Cacophony Workshop.  It was a maker-style event where we broke up into teams of people and in 90 minutes had to build a robotic musical instrument out of one supplied motor and whatever other parts we could scrounge together. 

Our team started with the basic idea of a repetitive percussive instrument with multiple hammers hitting multiple objects.  So we constructed a platform to hold the motor and used tennis rackets to make a rotating frame that held a series of arms that struck objects as they went around.  We then built frames to hold up things like pot lids or copper pipes to be struck, and attached everything from small chains to bundles of spaghetti to the rotating frame to strike them.  The real enabling insight came from the women on the team who realized that by spacing the arms un-evenly we could create an interesting rhythm.  The one-two-and-three rhythm across a variety of instruments ended up sounding much like a drum circle that actually got people dancing.

Thanks to Espressobuzz for capturing our creation on video.  Thanks to Josh Kopel for bringing the event to Seattle, and Shelly Farnham for organizing.  And to my wonderful teammates whose energy and creativity made the Spinning Spaghetti Monster possible.

The Prius’s High-Tech Safety Features

The Prius has at least a couple of smart traction control systems which use technology
to help maintain control of the car in difficult situations.  Of course
there’s anti-lock brakes (ABS).  In addition to ABS the braking system
does things called Electronic Brake-force Distribution (EBD) and Brake
Assist.  Beyond that, it’s got Traction Control, and in all but the
stripped-down cars something called Vehicle Stability Control (VSC).
What each of these features does, or how they differ from each other I
don’t know and don’t frankly care.  They help the car do what I’d want
it to do under good driving conditions when the road is slick or uneven
or I’m driving too fast or what have you.  All this is great.  But
there’s one subtle and very bad flaw in the implementation of at least
one of these systems.

The Stupid UX for these Features

When the Prius needs to engage one of its TLA control systems, an
orange light appears on the otherwise dark dashboard.  Sometimes a
sound will beep.  This is presumably done to let you know that the
fancy feature that you paid for is working on your behalf.  Call it a
marketing feature if you will, because there’s nothing you can do with
this information, or way to react to it within the vehicle’s controls.
The stupid thing about this feature is that it’s distracting to have
lights flashing and sounds beeping.  When you’re having trouble
controlling the car is the last time you want something distracting
you.  So the engineers at Toyota have done something completely moronic
– a complex system senses when you’re having trouble controlling your
car, and chooses that time to throw more distractions at you.  I’ve
observed the light and beeping  on when driving over bumpy roads, or
down
something very steep that has a rut in it, or when the back-right wheel
accidentally jumps up the curb.  None of these situations I’ve gotten
into have been serious or very dangerous.  But even still, they were a
bit confusing, and the confusion was confounded by having the car meow
at me demanding attention.  When the car is not doing what you expect it to, the last thing you want is additional distractions.  Bad choice, Toyota.  These features should be absolutely silent.

What you can do about it

If anybody knows a cheat code to disable this light or beeping, please
post it in a comment.  Until then, you can help a little bit by putting
a small piece of electrical tape over the orange light with the symbol
for "I think you’re losing control of the car."

Dan, being a "good guy", tried to keep the details of this hack quiet for long enough for network operators to patch their systems and close the loophole.  He wanted everybody running a DNS server to do this before the "bad guys" figured out what the bug is and started to take advantage of it.  He was hoping for 30 days of time to prepare, but somebody spilled the beans after 13 days, and now the hackers are off and running.

It’s a jungle out there

You might be asking, So what? What are the dangers of being directed to the wrong website?  Of course, you could read incorrect news and that’s not great.  More likely you’re going to have your password stolen for whatever site you log into.

The obvious attacks are to sites like paypal or banks, but they’re actually safe from such attacks if you use your browser properly.  Any financial site will use a secure connection.  You can tell because of the https:// at the beginning of their address.  These sites use a digital certificate that your browser checks to verify their authenticity.  All this happens independently of the DNS system.

But you can still connect to a hacked site with https.  Your browser will probably warn you saying something about a certificate not matching.  More often than not these errors occur because of a lazy sysadmin or something.  But right now, I strongly advise you to take all HTTPS warnings seriously.

Protect yourself

If you want to be sure you’re safe, manually connect your machine to OpenDNS, as Dan recommends.  We know they’re patched and can take the traffic.  I’ll give you the steps to do this on Windows:

1. Start menu
2. Control Panel
3. Network connections  (might have to switch to "classic view")
4. Select the one you’re actually using.  It’s likely called "local area connection".
5. Click Properties on the status dialog
6. Scroll down in the list of checkboxes, and select "Internet Protocl (TCP/IP)" so that it’s highlighted.  (Leave it checked!)
7. Click Properties
8. In the first General tab, change the second radio-button from "Obtain DNS server address automatically" to "Use the following DNS server addresses:"
9. For Preferred DNS server, enter: 208.67.222.222
10. For Alternate DNS server, enter: 208.67.220.220

That’s mostly it, but to be safe, you should reboot, restart your browsers, and/or:

11. (Windows key+R).  In the dialog type  "ipconfig /flushdns" (without the quotes) and hit okay.

Last week Seattle had some crazy electrical storms.  In recent years’ storms, my block has done better than most with respect to power failures making me think we’re either lucky or in a particularly robust section of the grid.  So I was a little surprised to find my whole house offline on Wednesday morning.  After a bit of debugging I figured out that the small UPS that runs all my networking gear got toasted, and for some reason the file server was down.

I left it alone for several days, and when I got around to turning it back on, I was happy that the whole stack through the samba server came up by itself.  (It doesn’t always!)  But when I started looking around I quickly realized things were amiss.  The media/video directory normally has 4 subdirectories: movies, episodic TV, imake and other.  But today it listed:

leo@elephant:/raid/shares/media/video$ lsdpisndic TV  hmakd  movies  nther

WTF!?  A few bits had been scrambled in the directory names.  This sounds really bad.  Moreover, even though the first couple levels of the directory hierarchy were there, but no files were to be found.  Definitely a problem.

Step 1: As soon as you suspect your RAID array has a problem, stop writing to it until you know what’s going on.  Writing changes can make things worse.  Stop the bleeding.   

I unmounted the drive from my mac, not trusting Finder or Spotlight to sprinkle damaging meta-files over the array.  Once I remembered how to ssh into the box, I stopped the samba daemon,

leo@elephant:/$ sudo /etc/init.d/samba stop

unmounted the filesystem

leo@elephant:/$ sudo umount /raid

and changed fstab so it would be read-only when it comes back, and that it wouldn’t come back without me asking.

leo@elephant:/$ sudo vi /etc/fstab

changing

/dev/evms/teraraid500 /raid ext3 defaults  0 0

to

/dev/evms/teraraid500 /raid ext3 ro,noauto  0 0

I tried poking around in EVMS by running

leo@elephant:/$ evmsn

But it hung during initialization with blue dialog saying "Discovering segments…"  I’m thinking EVMS can’t help me.  After a bit of googling I thought I should try e2fsck or some such.  First, I tried to mount it again read-only and see what’s there.

mount: wrong fs type, bad option, bad superblock on /dev/evms/teraraid500,       missing codepage or other error       In some cases useful info is found in syslog - try       dmesg | tail  or so

Bad superblock.  Uh oh.  Well this guy managed to recover a drive with a bad superblock.  Lots of things were pushing me in this direction — fix the filesystem.  But I realized that was a mistake.

Step 2: Do not make changes at the filesystem level until you’re confident that the RAID array is working properly.  You set up RAID for a reason.  You’ve still got a chance to recover everything, but if you start
making changes to it in a broken state, you’re almost certainly going
to make things worse.

Me to self: Think about it.  EVMS is confused.  Linux is confused.  Ext2 and ext3 are messed up complaining about bad superblocks.  The problem was caused by lightning.  When the drive was mounted there were wierd bit-level corruptions in the data that were still there.  Maybe one of the drives in the array got data scrambled, but didn’t get totally fragged so it went offline.  RAID 5 is designed to survive total loss of a single drive.  But if a drive gets corrupted, who knows what will happen.  So I came up with this plan:

Step 3: Try physically disconnecting the drives in your array, one at a time.  If only one of them is scrambled, disconnecting it should restore all the data in the array.

Having followed my own advice, it’s easy for me to tell the drives in my array apart since each drive in the RAID array is from a different manufacturer (which makes array failure due to manufacturing defects far less likely). 

This plan actually worked perfectly!  Removing a drive caused a bit of a hassle in getting the machine back up, because when I booted it couldn’t find the /boot partition complaining

 * Starting Enterprise Volume Management System...[42949392.340000] raid5: raid level 5 set md1 active with 2 out of 3 devices, algorithm 0

 * Checking all filesystems...fsck.ext3: No such file or directory while trying to open /dev/sdd5/dev/sdd5:The superblock could not be read or does not describe a correct ext2 filesystem.  If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck with an alternate superblock:    e2fsck -b 8193 <device>

Notice the complaint about the superblock again — don’t trust it, and don’t do what it says!  What really happened was that the boot drive letter had been changed from /dev/sdd to /dev/sdc, so I had to change /etc/fstab to mount /boot from  /dev/sdc5 instead of /dev/sdd5.  In my system, I boot off a non-RAID disk attached to the mobo, which for some annoying reason gets the last drive letter after all the drives no the SATA card.

But once I got past this, it quickly turned out that the Samsung drive was the culprit.  With it removed, the software RAID kicked in and plugged all the whole.  Everything the array looked
completely normal again.  All the directories.  All the files.  Hooray!

1. Call 1-800-333-9956
2. Enter your 10-digit mobile number
3. Enter your 5-digit billing zip-code.  (This one threw me for a loop, since Verizon thinks I live in South Dakota.  Long story.)
4. Enter the last 4 digits of your SSN
5. Press 9 to opt out

I’m rebroadcasting a post from Ars Technica.
Normally I don’t like blogging unless I have something interesting and
useful to add, (and the blabber about location-based advertising hardly counts as insightful original analysis) but this one’s annoying and timely enough that I thought it would be useful to repeat it.

Update: The cellfish blog also has a nice article about this Verizon problem.

I’m talking about my good friend Barry’s new float tank.  It’s called the Lilypod for hopefully obvious reasons.  Setting this thing up and maintaining it is sure to be an adventure.  And adventures deserve blogs.  This one’s lives at http://thelilypod.blogspot.com/

It’s an interesting idea.  Six 28-hour days make up a week.  6 x 28 = 7 x 24 = 168 hours.  {math corrected} If you sleep 8 hours out of 28, then you’re getting 48 hours of sleep a week, which is equivalent to just under 7 hours of sleep a night on a regular schedule.  (6 hours 51 minutes).  Personally, I’m optimal at about 7 hours 25 minutes a night on a long-term basis, but I can go 6 hours a night for quite a while and stay upright.

To see what this might be like with my schedule, I put XKCD’s schedule into a public Google calendar.  It doesn’t fit with my school schedule next quarter, so I won’t be trying it.  But if you give it a try, let me know how it works.

Or here’s the ical version.

I like to group sets of related browser tabs together into OS windows.  This organizational structure makes it easy to multi-task.  I might be researching something I want to buy, and I’ll have lots of tabs open in a single window with the various options I’m considering and the related research.  When I’ve made my choice and have purchased something, I can easily close all those tabs together with a single click, and not interrupt any of my other activities.  Or if I get interrupted before I can make a decision, I will minimize this window and the whole set of activities is put aside until I’m ready to work on it again. 

The same strategy works for research — if I’m trying to figure out a specific something, I will typically open a half dozen windows before finding what I want.  But once I have the answer, none of those windows are useful anymore.

Suggestions / Requests

Here are a couple of things I’d like to see change about tab management in firefox:

I want the ability to take an open tab and pull it into its own window frame.  Sometimes a new tab will open in the current FF window and I’ll realize it’s going to be the root of an exploration and should really have its own window.  Right now I have to copy the URL, close the tab, open a new window and paste the URL.  It would be great to have a context menu option to do this for me.

Now some might question why that tab opened in the wrong window in the first place.  Sometimes it’s because I don’t realize when I open something that it will blossom into a whole research task.  So I open it in the same window, but then later want it in its own window.  More often though it’s because I click on a hyperlink in another application.  I have firefox set to open new links in new tabs instead of new windows.  Normally this is the correct behavior when browsing — anything I click on that wants to open a new window is related to what I’m doing.  But when I click a new link in another application, it’s generally not related to whichever FF window I happened to be using last.  I want a firefox option which allows me to change the new tab/new window behavior for OS links from other applications because normally I want those to open in new windows.

Update: Ask and you shall receive

It turns out most of what I want is already available in the fabulous Tab Mix Plus plugin.  Thanks to Stuart for pointing this out.  It’s got a lot of options and I haven’t explored all of them.  But I think it still doesn’t give me the level of control I’d really like to move tabs between windows.  I think what I really want is to be able to drag & drop between browser windows.

• Press Start
• Select Settings
• Choose the System tab
• Run Memory
• Choose the Running Programs tab

Happy terminating!

My girlfriend’s condition has gotten worse in recent months, to the point where immuno-suppresents seem reasonable.  But instead, she’s opted to do something much more difficult and follow the advice of a naturopath.  She’s agreed to eliminate basically all tasty foods from her diet for some unreasonably long period of time.  After weeks of eating nothing but rice and steamed vegetables (I’m exaggerating, but not much) she’ll slowly start adding foods in one at a time to see what might be causing an negative reaction.  It’s an elimination diet — a fairly common practice which is pretty easy to visualize but takes care and dedication to do properly.

Why suffer through this process instead of just taking some pills and getting better?  Because it promises
to understand and solve the cause of the problem, rather than just cover up the symptoms.  I admire her strength and wisdom in this choice.  Until then, we’ll be making lots of use of the veggie steamer.  (My veggie steamer actually looks a lot more like this one, but mine has the fabulous retro-luddite feature of a knob to set how long to cook for instead of digital controls.)