As our technology becomes more complex, it is often shipped with flaws and missing features. The evolution of the Internet coupled with a “must ship” attitude has led to a number of interesting business practices. One in particular, remote updates/patching, presents some interesting reliability issues.

One of the best known versions of remote patching is the software update function, currently found in many computer applications, and in most common operating systems. In its usual form, this is a system that can download patches or whole new software artifacts to address a newly-discovered security vulnerability. Some systems are automated, but most require manual intervention. The current systems generally only involve security fixes and no functionality improvements—the functionality improvements may or may not be bundled in less frequent updates (service packs), or they may be deferred into a major revision that requires additional payment.

Many of us working in security and reliability have expressed concern about these updates, because if the update mechanism is somehow hijacked by an attacker, it can be used to quickly distribute malware to large numbers of systems at once. There have also been examples where updates accidentally deleted critical files or provided faulty configurations, thus disabling or degrading many, many machines at once (for example this one and this one). Most vendors have elaborate systems in place to test and verify such patches, and they have plans in place to quickly respond if something goes wrong.

Now, we’re seeing the same concerns begin to occur with consumer goods that aren’t primarily intended as interactive computers. I can speak from personal (unfortunate) experience that at least one major vendor appears clueless and not customer-friendly.

I recently purchased 2 Samsung Blu-Ray DVD players: a BD-P2500, and a BD-P1500. Both have Internet connections for firmware updates and Blu-Ray Live. The BD-P2500 also supports live streaming of Netflix content.

A couple of days after Christmas, the 2500 froze up. I could not get it to respond to anything, including the factory reset code. I contacted Samsung and was given information to send the player in for service—it was still within warranty. They’ve had it for nearly 2 weeks with a status of “waiting for parts.” It has now been broken longer than it was working, with still no prognosis about when it might be returned.

No problem—I still have the other player I can use, right?

The 1500 came up with an on-screen message early in the week that a firmware update was available. Having had experience with downloads and upgrades of OS components, I waited a couple of days before doing anything. When I initiated the download, it completed without error, according to the display. However, after completion, it too was dead—no response to anything, including reset codes. So, I called Samsung again. The problem was escalated in customer service. This is what I was told:

 
1. There was a bad update put on the servers, and many players that got the download have frozen up
 
2. They do not have a fix for it at the current time and have no idea when one will be available
 
3. I should check their WWW site once a week to see when an update is available. “It should almost certainly be within a month.”
 
4. Even though it is their fault for putting up a bad firmware update, if I am required to send in the player, it is now out of warranty for service so it is my own expense.

It seems fairly clear that Samsung has a major problem in testing and assurance, and a surprising lack of concern for customer support. It also sounds like they don’t have much of a handle on what it will take to fix a locked-up player.

I wonder how many other people around the world are stuck with non-functional players and a vague answer about the fix? It could well be in the thousands. And the best they can offer us is to check the WWW site once a week to see when they are ready for us to pay to install a fix to a problem they caused in the first place!

As someone who works in security and reliability, I can see all sorts of interesting problems here involving updates to consumer appliances. They problems are magnified with incomplete or incompetent responses from the vendors. It certainly suggests that consumers should press vendors to issue things that work correctly and don’t require updates—or at least have a fail safe state that allows recovery! Imagine losing use of your TV, phone, refrigerator or car indefinitely because of a faulty update caused by the vendor, with an indefinite fix. For those with malice in mind, this would be a great thing to do to harm a company—and maybe to extort some money as “protection.”

As a consumer, I’m rather angry. I don’t expect to buy anything else made by Samsung, and I certainly won’t recommend them to anyone else. You may choose to use this as a cautionary tale in your own pursuit of consumer items and choose another vendor that is more careful with their updates, and more considerate of customers who have paid for their products. And if you have one of the frozen players with some idea how to recover it to working condition, I’d be interested in hearing about it.

Sadly, caveat emptor.

Update 01/19/09: Samsung is shipping me a replacement for my bricked P2500. It left their plant on Friday, surface UPS. So, that will be a 3-week turnaround.

Meanwhile, I called the service number again about the P1500 and pressed until they escalated me to “executive response.” (Third or fourth level customer service, I guess.) I kept reminding them that it was their firmware update that caused the problem. After 30 minutes on the phone, I must have worn them down sufficiently: they extended the warranty through this week, and are providing me the shipping information to send it in for service under warranty. Hooray!

Unlike last week, the personnel I talked with today were uniformly helpful and informative. I wonder if they have had enough complaints that there has been a change in policy? Or did I just get two really bad service reps in a row last week?

Nonetheless, the bad updates and the lack of a failsafe are really poor design.