A lot of people are intrigued with using “cloud” applications and storage for personal health data. This week we’re seeing what I think is the final nail in the coffin of “cloud only” for anything important. You gotta have offline backups: two huge cloud vendors – Amazon and now Google – have demonstrated that even they can go down, leaving their users absolutely powerless.
Cloud computing (Wikipedia) is hugely attractive to software developers and businesses. As shown in this diagram from Wikipedia, the idea is that you do your computing using storage or tools that are on some computer somewhere out there “in the cloud.” You don’t know or care where, because somebody out there takes care of things. As your business or database grows, “they” take care of it.
And it’s real – it works.
But when “they” screw up, you could be screwed.
Last month Amazon Web Services went down for a couple of days. PC Magazine posted a good summary, and many of us learned that well known companies like Hootsuite and Foursquare don’t actually own the computers that deliver their product: they rent services from Amazon Web Services (AWS).
So when AWS went down, there was nothing they could do to help their customers.
Now the same problem has happened with Google’s Blogger.com (“blogspot”) blogs. For at least 48 hours the back end of their blogging system has been dead. For instance you can read my old blog (patientdave.blogspot.com), but when I try to log in to create a new post, here’s what I get:
ZDNet reports that all posts and comments added since the problem started have “been removed,” a euphemism for “lost forever.” ZDNet asks, what keeps this from happening to other cloud products, like Google Docs? (What if you’d stored your business documents in Google Docs? What if your last two days of Gmail were lost?)
What’s the relevance of this to patient engagement? Well, a lot of people talk about cloud-based personal health records (PHR), and cloud-based medical tools. Here’s the lesson: For anything you can’t afford to be without, you gotta have non-cloud backups.
- You gotta have offline data backups.
- My personal website is built using WordPress; every day they email me a backup file of my complete database. I can’t lose more than a day’s work. It’s vital, it’s obviously not complicated software, and it’s free.
- At my old day job, I used to manage a pretty big cloud-based database for our sales and marketing needs: Salesforce.com. Every week they emailed us a link to download a zip file of our entire database, plus a daily update.
- For any “can’t afford to be down” situation, you gotta have ironclad availability.
- Here at e-patients.net, we don’t rely on the cloud. Our WordPress blog is hosted on the highly reliable servers of one of our board members.
- Salesforce.com has a huge number of redundant server farms, and they’re totally transparent about outages and even degradation. To earn the trust of the corporate world, they published trust.salesforce.com. It displays the uptime, performance degradation and outages of every system around the world. Click on any symbol and see the root cause, how it happened, and what they did about it. (And you can subscribe to any RSS feed for all updates! Hey Google and Amazon, you gonna offer that?)
People I talk to tend not to “get” this unless it’s expressed as their kids’ medical record. A couple of years ago I spotted this message on the Google Health user forums:
Help – leaving for hospital – data won’t print.
I do believe in the cloud – it makes sense for many situations. It makes innovation far faster and less risky. But for anything important – which health is – you gotta have offline backups and you gotta be sure you won’t go down.
Of course, there’s an ultimate trap for anything online: if the internet goes down, the whole cloud goes down. That happens sometimes, and it could very well happen if an enemy hacked the Web. For anything mission-critical you have to consider whether you could afford to be offline for hours or days, and think out what you’d do meanwhile. (Same for power outages, which is why hospitals etc. have backup generators.)
All this has been debated for years in the IT community, but once again the health IT world seems to be naive. As we consumer/patients (and health workers) start to acquire IT tools, we must insist that tools we rely on have sufficient reliability – even in a disaster.
e-Patient Dave (Dave deBronkart, MD) is one of the leading spokesperson for the e-Patient movement. A high tech executive and online community leader for many years, he was diagnosed in 2007 with Stage IV kidney cancer, with a median survival of just 24 weeks. Today he is well. In 2008 he discovered the e-patient movement, and began studying, blogging, and speaking at conferences, and in 2009 was elected founding co-chair of the new Society for Participatory Medicine.
Categories: Uncategorized
magnificent issues altogether, you just won a logo new reader. What would you recommend in regards to your publish that you simply made some days ago? Any sure?
Excellent post, I will be checking back regularly to look for updates.
Really Interesting …:)
I wish u all about seeing of ur blog, As for health records, I fully agree with Dave de Bronkart, it is not wise to trust only one deposit place for sensitive data, they should be backed up as it best suits the user. Besides cloud going down, we have witnessed recently so many other dangers (hackers, data hacker thiefs, etc.) that make the offline availability of sensitive data a must.
———————————–
sweety
Any one of these problems could have happened in a companies’ own data center. I work in IT in a Fortune 100 company and our main portal went down for over a day due to a bad change. Made the WSJ. Having a good disaster recovery plan based on the critically of your services is important regardless of where your data is hosted.
BTW, i use Google Heath and every time i update it, i download a PDF that is saved to a Dropbox account that is synced to local storage on 3 devices. That example of the family not being able to print out from Google health could have easily been that they could not print out the Excel spreadsheet because the printer was out of ink.
Ugh. Once again I wish i’d known this was going to be cross posted so I could have said “PLEASE be sure to include the prefatory note I addded on the original post on e-patients.net.”
I know this blog has a practice of cross posting without notice but as I’ve said before I think it’s rude, and in this instance has the effect of misquoting me, because what it says here is no longer an accurate snapshot of my view. I could easily have corrected that.
What the hell is so hard about letting me know about an upcoming cross post, as KevinMD and BetterHealth do? Am I just web meat, to be harvested however you feel like? Throw sh#t around, it’s all just an SEO game anyway, eh? (Deja vu, now.)
Anyway, loyal commenters, I encourage you to go read the convo over there, to get a better picture of my current view.
Ugh.
Private and hybrid clouds provide a model that provides the flexibility of the cloud but the security, availability, and performance required for specific areas including “medical-grade”, critical infrastructure, and mission-critical government applications.
You are right! Moreover, in many regions in Europe, maybe in other areas, there is no broadband or at very high cost/low speed. I have experienced this more than once. Therefore, I always backup all my data in an external hard drive so that I can use them with my netbook.
As for health records, I fully agree with Dave de Bronkart, it is not wise to trust only one deposit place for sensitive data, they should be backed up as it best suits the user. Besides cloud going down, we have witnessed recently so many other dangers (hackers, data hacker thiefs, etc.) that make the offline availability of sensitive data a must.
You are right! Moreover, in many regions in Europe, maybe in other areas, there is no broadband or at very high cost/low speed. I have experienced this more than once. Therefore, I always backup all my data in an external hard drive so that I can use them with my netbook.
As for health records, I fully agree with Dave de Bronkart, it is not wise to trust only one deposit place for sensitive data, they should be backed up as it best suits the user. Besides cloud going down, we have witnessed recently so many other dangers (hackers, data hacker thiefs, etc.) that make the offline availability of sensitive data a must.
You are right! Moreover, in many regions in Europe, maybe in other areas, there is no broadband or at very high cost/low speed. I have experienced this more than once. Therefore, I always backup all my data in an external hard drive so that I can use them with my netbook.
As for health records, I fully agree with Dave de Bronkart, it is not wise to trust only one deposit place for sensitive data, they should be backed up as it best suits the user. Besides cloud going down, we have witnessed recently so many other dangers (hackers, data hacker thiefs, etc.) that make the offline availability of sensitive data a must.
We recently experiences severe problems with our own hosting solution. Guess what, we used real servers not cloud.
And no, a backup is not a solution. Backups has nothing to do with the question of “going cloud” or not.
And now we have transitioned to a pure cloud solution. I feel better. Why?
– When servers go down (cloud or not) is mainly because of human errors.
– The probability that your local IT staff is of higher quality than the IT staff at Amazon AWS is very very low.
– I trust that AWS staff will do fewer human errors than local IT staff. And your individual experiences will probably confirm this.
– Hosting your own server cost a lot of resources. The machines and equipment is not the biggest expense. A high quality IT admin will cost you above 150K a year. Many health organizations simply can not afford this.
Not all clouds are created equal.
blogspot.com is a free service. Expect it to perform likewise. You get what you pay for.
I agree, backups are key. I heard this week about the launch of the new cloud computing laptops: I don’t have the details, but as some who travels a great deal I would never be able to use them. There is no internet connection on flights and the services which are available on trains is shocking.
With all due respect, Dave, I think that “backup” and “cloud computing” are separate issues.
Of course all data should be backed up, whether they are stored in the “cloud” or on a server.
The beauty of cloud computing is that it makes complex applications available to anyone with internet access. And computers in the cloud, like computers in my hospital, can (and do) routinely go down.
Most of us in healthcare are used to getting regular emails announcing the temporary failure of one application or another. (Honestly, I’d rather trust most of my applications to the IT team at AWS than the IT team in my hospital.)
The key is in the selection of appropriate applications for the cloud (and the FDA can help with this). Cloud cardiac monitoring? Heck no! Cloud EHR with appropriate security? Sure.
Going back to the Amazon Cloud issue, there was a cardio monitoring company who was on the AWS boards asking for help as they were down. Now I gave this a little impact here as they included their account number and other vital information on their SOS to get help from Amazon who was nowhere to be found. At the very end one commenter said it was a hoax but whether it was or not, it was a wake up call.
I posted it on my blog and as we all know there were many who could not get customer service when the failure happened and even Robert Scoble looked at it as well and made the same comment, how customer service these days is not easy. Has anyone experienced cloud failure and data hosting recently? If you were on blogger you did with having to repost data that was lost and “read only” blogs for a number of hours, even up to 30 hours in some instances.
Cloud services are using “virtual hard drives” which have been around for quite a while and once Server 2008 baked them in a whole new opportunity opened and has been named the cloud. If you take the time to read the full explanation from the CEO of Amazon on what happened, there was nothing out of the ordinary other than some normal routines that caused failure and a few other configurations that needed to be adjusted to avoid this in the future as well as a better back up. We all learn all the time.
Nobody can anticipate everything in the data world but you want you see the AWS message board and again with a home cardio monitoring system being down with nowhere to turn, well there goes the service and we hope that nobody who was being monitored had an event by all means.
http://ducknetweb.blogspot.com/2011/04/what-happens-when-cloud-server-goes.html
The patient could pay for the med rec app directly. Who needs EHR?
I’m on my way to a conference right now. My itinerary is available in the cloud. But I’ve learned to download it. To all my devices, and to keep a paper copy in case I run out of power (which happens frequently on long flights). If it wer my health data, you bet I’d have it equally protected.
“…but once again the health IT world seems to be naive”
We don’t do “cloud” in health IT. The overwhelming majority of health IT is deployed locally in industrial strength data centers owned and controlled by hospitals. A majority of independent ambulatory health IT is sitting on servers located in the clinic. The newest trend is to have vendors maintain the servers in their own data centers while users log in remotely (ASP). Some smaller vendors who offer web based health IT also deploy everything in their own data centers. For the large ASP vendors, there is always a secondary data center at a meaningful geographic distance from the primary.
I am not aware of any significant health IT application deployed in the public cloud. An the incumbent advocates of “cloud” in health IT are really talking about vendor data centers. They are just using the wrong terminology, probably because “cloud” makes for sexier marketing. I bet they will be adjusting the message really soon….. 🙂
Redundancy, failover and disaster recovery have always been at the forefront of considerations of any serious health IT deployment.
I think the naivete belongs to the new crop of tiny entrepreneurs, with no health care experience, presuming that developing health IT is as simplistic (and cheap) as developing a 99 cents game for the app store.
After a proper risk analysis, which one should always do, there will emerge a set of responsibilities and accountabilities for any vendor — cloud or not — entrusted with important (to you) data. To the extent that they are unwilling or unable to meet those requirements, the burden falls on you to supply a solution. That could be frequent backups. It could also be having trustworthy medical resources that don’t require an available EHR or PHR to meet immediate health care needs.
Most people don’t have the skills to do a risk analysis, so they will rely on anecdotal experience from friends, blogs, etc. Most of those stories will document interesting failures, since systems that quietly do their jobs are unexciting.
Case in point: Although we have had quite good security and privacy protections for many years, the spectacular failure reports make the truth irrelevant to public opinion.
Bottom line: We don’t need better systems, just a better PR for the few times that screw-ups occur.
I agree with the principle, but the devil is (always) in the details.
The only data I do not have backed up in two (available) places is data I consider unnecessary. And I’m not even talking about medical records!
There is, however, a problem with applying this principle to life. At what point is one over reacting?
Should the local backup be printed (nightly) to guard against the inability to access electronic data in the server farm, down the block? Should every practice with a local server keep current hard copies for a power failure? In a community with two neurosurgeons, should they never operate at the same time, in case one of them collapses during surgery? How extensive a supply of materials should a hospital have, to guard against a problem with the supply chain of medications or oxygen or disposable catheters? What should accompany a patient being transported in an elevator in the event that there is a power outage or mechanical malfunction and the elevator is locked between floors for 3 hours?
Perhaps a unit like hours of patient data availability/100% availability would help us assess the relative risks of local and cloud data, and compare both of these to other risks?
We live in a world of limited resources but unlimited opportunities for malfunction.
Peter