Beginning on December 30th we had an issue with Windows Live Hotmail that impacted 17,355 accounts. Customers impacted temporarily lost the contents of their mailbox through the course of mailbox load balancing between servers. We identified the root cause and restored mail to the impacted accounts as of yesterday evening, January 2nd. As with all incidents like this, we will fully investigate the cause and will take steps to prevent this from happening again. We’re very sorry for the inconvenience this may have caused to you, our customers and partners.
Chris Jones Windows Live
Very good post. I agree with your premise: no computer system is 100% reliable and it is up to us (the users) to take the appropriate steps to backup our important data.
However, not every Hotmail user has 30 years of experience using computers. They (mistakenly) assume that their email service provider is taking sufficient precautions to backup their critical email. So, in order to better educate their customers, perhaps Microsoft should post a warning to anyone signing up for Hotmail ... a warning that tells the customer that Microsoft's email service is not 100% reliable and for safety sake, each customer should protect themselves by backing up their critical email.
Perhaps Microsoft actually does include such a warning, it's been more than a decade since I signed up for a Hotmail account. However, I doubt that the marketing folks would allow such a warning to be displayed. Those with lots of computer experience know about reliability issues and the need for backup. However, I suspect that a large portion of Hotmail's 350+ million customers are not aware of the reliability issues.
And even if one has backed up their critical email messages, that does not address the problems some people face when they cannot access their Hotmail account for days at a time. People waiting for email with medical test results from their physicians, people waiting for grades and assignments from their teachers, people waiting for information about job interviews, etc. Yes, unreliability is a fact of life, but not having access to ones email account for days at a time can be traumatic (and costly). What can the uninitiated customer do about that? They depend upon Microsoft having an email service that does not result in unavailability being measured in days.
You mentioned the Internet and ARPANET in your response as valid examples of systems that are not 100% reliable ... although the problem that caused the 3+ day Hotmail problem in early January had nothing to do with network failures. As described by Mike Schackwitz (Windows Live Product Manager) in his blog, the root cause of the outage was: "On December 30th, we had an error in a script that inadvertently removed the directory records of a small number of real user accounts along with a set of test accounts." Of course, scripts are also not always 100% reliable .... but that's where thorough software testing and backup come in (Microsoft's responsibilities). Unfortunately, it took Microsoft 3+ days to find and fix the problem, and during that time 17,000 customers were without access to their email account. 17,000 may be a small percentage of Hotmail's total customer base, but I contend that 17,000 is not a "small number" when it comes to customers being without access to their email account.
I agree with your position on Cloud Storage. It is but one of many tools/services available. I would never entrust critical data to cloud storage without also keeping my own backup. But the Hotmail problem that occurred in early January was not with Cloud Storage. It was a problem with a Cloud Service (i.e., web-based Hotmail). Data kept in Hotmail's cloud storage was not lost. Mike Schackwitz stated "email messages and folders of impacted users were not deleted; only their inbox location in the directory servers was removed". There was a software (script) failure in Microsoft's cloud service (Hotmail) that resulted in the deletion of valid inbox locations. Microsoft is trying to convince the world that they can be a major player in the Cloud Services market. To do that, the reliability and availability of their existing cloud service (Hotmail) needs to be improved. Regardless of the number of servers/clusters in Hotmail's "server farm", outages measured in days should be exceedingly rare ... yet not a month goes by without customers posting in the Windows Live Solution Center about being without access their account for days.
Again, I agree that no complex system can be made 100% reliable. However, in the case of Hotmail, I believe that Microsoft could do a better job of improving reliability and availability (and we customers should insure that any critical information entrusted to any complex system is thoroughly backed up). It's a partnership with responsibilities on both sides.
Thanks for a welcome voice of sanity in a wilderness of despair.
I have read every comment and there are many valid points.
I challenge the complainers to read and understand this entire post.
Oddly enough, I experienced Hotmail problems about 6 weeks before this recent problem you are all discussing. My problem was that suddenly some email from outside the Hotmail system did not come to me and the sender was not alerted nor did I have any indication. Additionally, some percentage of email did arrive but it was 8 days late. I have been using computers for personal and business communications for over 30 years now, so this comes as no surprise to me. I have seen these things before.
Something I try to impress on my clients who often fail to listen is that computers fail and telephone wires fail and fiber optic fails and satellite communications fail. In fact, everything is subject to failure. Cars stop running, houses burst into flames. Nothing is failure proof. That is why we have backup systems for computer related information and some of us use them. While I am sure that Hotmail backup systems are in place, I can not speak to how available they are. I can only state that email servers are much more complicated than most people realize and in a large installation that complication is multiplied because of the various failover components and redundancy required. It is also not practical for any organization to offer every blow by blow detail of investigation of issues or resolution of those issues as they occur to the bottom rung of its customer service department. For those of you with a family, compare this situation to explaining the details of an exploratory surgery one parent is currently having from the surgeon's perspective to your 2 year old child and your 5 year old child and so on. It is not practical or effective. It wastes time and leads to confusion and fear. It might even be impossible for you to understand completely if you were standing over the shoulder of the surgeon as the surgery was being performed.
Nor is it practical to tell the initial contact staff to eliminate the possibility that some unrelated issue is being experienced by an email user.
Imagine if you had a browser with some flaw in the settings or a corrupt file in the cache (you can look that word up in the browser context if you do not understand it) and you contacted Hotmail support and because some other users are experiencing a particular issue they simply LUMP YOUR PROBLEM with unrelated problems and tell you it will resolve when repairs or maintenance is complete. That might cause you to wait long past the completion date for a correction that will never come because it is local to your computer.
So in support of computers it is proper for the 1st level support person to be engaged in the MOST COMMON types of troubleshooting. This involves making recommendations based on the MOST COMMON causes of certain problems and helping users work through basic corrective procedures. If the basic corrective procedures are performed and the results are not positive, then a user is moved along to another resource whose job is to focus on "things that are not as common but typically resolve issues that were not resolved by the most common techniques of resolution."
Running a support system is quite complicated and must be handled in a manner that allows the most experienced people do what they are most experienced in.
As for cloud storage, I could not agree more. I would never trust someone else to manage all of my storage needs. I can not trust that I can hire employees who will not be subverted to steal from my company much less place my trust in some unseen outsider to do that for me. Cloud storage is a tool. It is one of many that might be used simultaneously to provide reliable storage and retrieval under the widest range of circumstances.
It is naive to think that the Internet will be up 100% of the time even if the remote storage system being accessed is so perfect that there is no way it will ever be down. Read the original ARPANET information to understand why the Internet exists. This will shed a little light on the purpose of the Internet and how it is organized. Ask yourself if your cog in the current machine fits into that picture perfectly. If the answer is "no" , and it will be, then you have to take a little responsibility and quit expecting the world to take care of you. You must have mutiple backups of anything important and they must be on different types of media and in multiple locations and you must be able to access any of those locations in the timeframe necessary to satisfy YOUR REQUIRMENTS if you expect to be protected.
I don't know how long the Windows Team monitors threads, but just in case (because of the importance of the wedding related emails), you might want to re-post your issue on the more current blog that addresses this issue:
Be sure to apologize for the double post. ;-)
I hope you get the help you need !
My fiance and I created a hotmail account about a year and a half ago to keep all correspondence about our wedding in Feb. 2011 organized. It all disappeared some time around new years. I posted a couple of messages in Windows Live Hotmail feedback on Jan 2nd and 3rd and never got a response of any kind. Can you help us recover our emails or refer us to someone who can?
Would it be possible for you to determine why there are hundreds (possible thousands!) of Hotmail users who at the moment have been unable to access their accounts for several days due to a "server maintenance" outage? These people have been posting the issue on the solution center, but have not received any meaningful responses. Could you let us know the root cause of this rather extended service interruption and let us know what steps are being taken to prevent anything like ths from ever happening again? Thanks.
Customer bbarker65 stated that for two days he/she was unable to access Hotmail because of the dreaded "we're doing maintenance" message. Obviously, bbarker65 did not receive much help from the Windows Live Solution Center, and posted here.
Thankfully, you were able to do something to fix the problem.
I'm sure you would agree that not being able to access ones account for two days due to "maintenance" is excessive. Yet, if you browse posts in the Windows Live Solution Center, you will see that this is a very common problem ... customers frequently cannot access their account for days due to "maintenance".
Can you answer a few questions:
1. What did you do in order to fix bbarker65's problem (i.e., what caused his/her two day maintenance problem and how was it fixed).
2. Since extended maintenance outages are such a commonly reported problem on the Windows Live Solution Center, what is Microsoft doing to significantly reduce the frequency and duration of extended outages.
The reason customers are posting their problems here is because the Windows Live Solution Center cannot solve such strategic issues. It takes intervention from Microsoft.
@bbarker65, your account should be working now.
All -- from this point forward, if you have concerns about missing email or account access, please submit a request in the Support Forum: windowslivehelp.com/forums.aspx
@notahappycamper8, I sent email to the account registered with your user name to ask for more information. @bbarker65, I've asked team members to investigate why your account isn't available.
The issue is still not resolved. For the last two days I have received the following error:
We're doing some maintenance to improve the service. We apologize for getting in your way. Please try again in a few minutes. If it's been more than an hour, tell us you're still waiting.
Regarding your lovely, everso well written response above and sincere apologies... Not so, Mr. Jones, not so...Our emails still remain "unrecovered" a week later and getting no response from your "people".
@cesamela, we'll follow up with you offline using the account that's registered to your profile here.
My hotmail access issue is NOT resolved.??? How do I fix this.
To cut down on targeted advertisements tracking your browsing, disable third party cookies in your browser.
If I understand right there are many different servers that work with the editing of emails, what controls your advertisements on the hotmail editor page? How does Microsoft/Live Windows/Hotmail agree to let other companies represent Microsoft in their advertisements? When I say that- when you put an Advertisement on the side your companies van- that add reflects what your companies' values are. And in that light- the adds that hotmail let’s sit next to its email editor represents Microsoft's values.
I know this is off subject- but My antivirus/antiMalware program is alerting me that some of the advertisements that show up in hotmail's editor contain malware. And it took about 4 times before I could convince the windows live help team I was actually receiving malware via the advertisements. The good news is my AntiMalware program will refuse to let that code on to my computer. Tonight Hotmail's- 5/8 of advertisements were triggering malware alerts.
I have had things that I have looked at on Amazon follow me around while editing emails on hotmail, but this latest problem just takes the cake.
@Langeware- hard to find you, but good to see you’re keeping MS on their toes.
@tsiskchicago, we've done some fixes to your account, so please try to log in again. I'll send an email to your account, and you can reply to that (or private message me here) if things don't look right. I apologize for this not working right the first time.
p.s. - I'm bad at math: 20,000 x 10 is 200,000, not 2,000,000. Nevertheless, my points remain somewhat valid. FIX THE PROBLEMS.
Thanks for the refreshing, if difficult to find, insight as to the scope of problem #1 encountered by Hotmail users. I was one of the claimed 17,355 impacted accounts where, in my case, 8 years worth of e-mails simply disappeared from my account beginning on December 30. However, since January 1 I have been receiving "Cannot Access Hotmail - Account under maintenance" errors when I log into my account, and cannot view my inbox or access the mail server. The notice says that this should be a temporary outage. It is now January 5th. Neither the private moderators, nor the public forum moderators, are able to give any indication of when this will be resolved. I am a hotmail plus subscriber and I find it criminal that Microsoft is willing to take my money but not provide services in return. If this were a retail business, I would call the police to investigate why the store just took $20 from me without providing services in return. But unfortunately, the lack of a clear-cut technical support system to fix the problem - and the length that it has now persisted - leave me wondering if the Justice Department should be involved to investigate why Microsoft cannot provide a promised or delivered service. I also wonder if Microsoft's advertisers are being billed full rates when nearly 20,000 promised "eyeballs" are unable to view ads.
Can you please, please, please find a way to resolve this situation? It's becoming unconscionable that Microsoft, which prides itself on being the world's leading technology company, cannot fix a problem affecting less than 20,000 users. And with all due respect to @ thljcl, who notes that the number of impacted users is less than 0.00014 percent of all users, it's not a stretch to assume that roughly 20,000 users will tell 10 people each of their access issues with hotmail, thereby convincing 2,000,000 people NOT to use hotmail or to be wary of Microsoft's online services. And I can't imagine Microsoft, or any corporation, would turn away 2 million potential customers or viewers of advertising.
Please address this ASAP.
@winter - must be the purge of the 17355 accounts on your server ;-)
After the new year, my live mail becomes much faster than before. Has microsoft turned on more servers for email? Hopefully the email can always be this fast!
What was the reason for the outage the previous weekend as I had the same issues then as well.
As much as it is the amount of downtime it is how the situation was handled. Agreed the percentage of mailboxes affected was miniscule BUT it still affected people and businesses alike and the issues are that
a) no acknowledgement of a problem
b) timing of problem and length of outage
c) wasting peoples time with BS solutions to rectify client issues when it was a server issue
d) lack of resilience in the cloud - this years big push from Microsoft
YES people should backup offline, YES people should maybe have other means of communication as backups BUT even as a free service Microsoft make ad revenue from the service so it's not like they are not making money from the Hotmail/Live userbase - they are just not charging up front.
I'm not going to go on about Google or other cloud providers as being better services as they are not - they all have outages. It is how these outages are managed and communicated to clients that will show who is the service to go with. Open and honest communication and a quick resolution.
Data going up in a puff of smoke does not promote confidence just as not communicating or responding to people with blatantly useless support steps not even related to the issue at hand.
@langware, as I have noted in the forums and in our email exchange on the topic, we will post an summary of the issue and steps we are taking to address ;such incidents in the future. This will happen once our investigation is complete.
The issue is not that there was an outage in Hotmail service. As you said, there is no such thing as a perfect system.
The issue is that there was an outage, that affected 17,000 customers, and it took over THREE DAYS to fix it.
We need a full disclosure of Microsoft's post mortem .... what specifically caused the problem (load balancing between servers should not result in a three day outage), why did it take so long to fix it, and what specific actions will be taken to insure that similar extended outages do not occur in the future.
For a company who held over 1.2 billion email inboxes, the number 17000 represents approximately 0.0014 percent or less of all inboxes which are hosted Microsoft. Any logical person will never expect that perfect ‘ideal’ situation ever exists. In fact, in thermodynamics, the second law of thermodynamics specifically tells that the irreversibility of time means that the conversion of energy will never reach 100% efficiency.
Still, when people are affected by issues personally, the reactions that people have largely reflect the degree of emotional control and rationality that a person can have. In fact, in the last year, I did experience several occasions when I cannot use one of my Windows Live account services properly. Relatively speaking, while it did happen, the frequency was extremely rare.
Having using 5 Windows Live IDs personally, I used to see certain Windows Live ID of mine encounters some issues. Objectively speaking, while I saw a few bloggers were ready to bash Microsoft and tout Google, these rarely happened incidents, while affecting some users, do show that Microsoft does provide excellent cloud services.
While there are over 1.2 billion email inboxes at Microsoft’s server, according to Microsoft, there are over 360 million active users because many users have multiple email inboxes. Averagely, each Hotmail user has over 3 email inboxes. By far, Microsoft is the largest web mail provider on earth.
Windows Live services are more than just email. Online storage service which is known as Windows Live SkyDrive (with Office Web App), Windows Live Calendar and Windows Live Messenger are among the services which are included in Windows Live.
Did Google and other companies suffer outage? Yes, they did. People were skeptical about the cloud services. However, is there a perfect solution?
The recent Hotmail email access issues, however, did not affect me at all. None of my Hotmail accounts suffered the same issues. It’s not surprising since it’s very unlikely for a Hotmail user to ever notice that will happen. Still, interestingly, people can report the same event from different perspective.
That is to bash Microsoft for the impermanence which is defined by the Laws of Nature, assuming that perfect system exists, so it must be Microsoft’s fault. Of course, Software Giant cannot escape the blame. It must take the responsibility to please its customers, no matter what.
That’s said, rare occurrences did occur at times. When I encountered issues with cloud services, I wouldn’t be hesitated to file my complaints. After all, we do count on Software Giant in our lives in the Cloud. It’s not a bad idea to further pressure Microsoft to benefit all of us to have better services in the future.
@westcoastborn, I just sent an email to the account you registered with this forum to investigate.
All, many complaints about the loss of data have been caused by POP clients with the "Leave a copy of messages on server" option disabled. If you find your emails missing, and you've recently set up a phone or client to access Hotmail, that could be the cause.
This thread contains directions for how to check this for most email clients:
I think you are missing a critical point .... Microsoft wants to sell us Cloud Services. With Cloud Services, all ones data is kept "in the cloud" ... no backups are kept locally. If reliability is so poor that we must keep our critical data for Microsoft's services (email in this case) backed up locally, then we don't need Microsoft's Cloud Services (and can just run client applications instead).
I posted yesterday in France, after having put on my web site, blogs, Relatively important Advice !
I see it here and I really feel obliged to say it to, sharing the advice of moonman ...
10 years FREE using without spending 1 support to archive the important datas, with all the abilities offered by the tool.
Thank to Microsoft to let some Billion people using with no fees such technologies for several years ...
Must Microsoft and all the technical teams support the Expression of Deseases of customers without hearing from them while EVERY THING IS OK !
Congratulations to the teams and for there work ! What a pity for those who had not a copy of messages on there phones, on other stations, or who had not thought that buying Outlook for example was intelligent. Or using FREE Microsoft Included mail client softwares ...
Microsoft shared a lot and offers a lot of thing and Products, Services ... DO THEY HAVE TO Offer BRAINS ?
Sorry for this intervention But I feel Blessed by the purposes i hear for several days .
See U ...
Can We know from a technical point of view wich Kind of Issue it was relative to ?
Must say that I learned IT on Twitter, because we don't have to live this issue in France, but Technical Explainations can Help me understanding and progressing ...
TY Chris ...
On the morning of Jan 3, you stated this ...
"We identified the root cause and restored mail to the impacted accounts as of yesterday evening, January 2nd."
But on the evening of Jan 4, moderators on the Windows Live Solution Center stated this (in response to customers complaining about their email and folders still missing) ...
"The issue you have reported is currently under investigation. We'll inform you once the issue has been resolved."
These are conflicting reports ... either the problem has been resolved and all accounts restored, or some accounts have not yet been restored and the problem is still under investigation.
Would be nice if customers received consistent status.
Think it is very possible that all of MSFT has their "heads in the cloud" if not up a posterior body part!
@mulan8575 I wonder if there is any chance he reads here or simply just pushes out updates...? Bit like the support system on Windows Live. Maybe we need to optimize our browsers in order to get a response, or maybe his reply is in the cloud........ :-)
.....and my trust in "cloud storage" is now even less existant than it was before.
I have to wonder if you have actually read all of the pleas for help regarding the missing emails that are still being posted as of Jan. 4th on the Windows Live Help Desk board. These are people who still have not had their emails restored and have not even heard anything from Microsoft.
I am among those who lost their emails and contacted Microsoft immediately on Dec 2nd. It only took over 24 hours for someone from Microsoft to get back to me and that was only after posting a second plea. I was asked for my email address and I have not heard from anyone since. This seems to be what has happened to hundreds of others also. I honestly have yet to read a post online from anyone who has RECOVERED their emails since they were deleted by Microsoft. There has been no solution for what appears to be hundreds of people who had their emails deleted as of Jan1st and if as you have posted there are some people who have recovered their emails, what can be done to prevent this from occurring again?
As of Jan 4, some customers are still posting on the Windows Live Solution Center that their lost email and contacts have not been restored. Can someone be assigned to followup with the customers who state that their emails/contacts are still missing (all the moderators say is that "we are investigating this problem").
As a result of this problem, were there any permanently lost data?
You mentioned that "we will fully investigate the cause and will take steps to prevent this from happening again."
Will you share the details of your investigation and the steps being taken to prevent this from happening again?
I would recommend using Windows Live Desktop Mail or another offline mail client that will at lease keep your existing messages stored on your PC - that way you can use any regular backup tool/regime to keep things backed up and safe.
If you have Outlook then you can always store your email in PST files and back those up.
Hope that helps!
This was a serious issue and impacted not just regular users but Office Live business users too.
This was the *second* outage in just over a week - how can server maintenance be scheduled over probably the most busiest times of the year for people. Who makes decisions like this?
What is worse is that the only means of support was a complete failure. There was no official acknowledgement of an issue just support staff asking people to optimize their browsers. Why blatantly lie about the issue and waste peoples time? The forums were littered with complaints and cries for help - people having spent hours trying to optimize their browsers when basically they had no hope of getting logged into their mailboxes.
Businesses lost customers, peoples plans were affected due to not being able to communicate with friends and family, this was a huge outage and no-one had the decency to post that there was an actual problem.
How can Microsoft behave in this manner ESPECIALLY when they are promoting their cloud as THE place for people to store and use their data. How can anyone have confidence in Microsoft's ability to keep things running and maintain uptime on servers when they cannot handle and communicate with their customers on what was effectively (in terms of number of mailboxes and servers) a minor issue?
To the cloud? I think not - not Microsoft's one until someone can
a) take ownership and responsibility for outages and report to customers in a timely and honest fashion
b) prove a level of enterprise resilience and uptime
c) have support mechanisms in place to actually help customers
Very poor performance Microsoft.
Of course I have some stuff backed up or printed, but at least in this past year the majority of tax receipts, travel statements, etc for my business, personal things,are all gone.....I basically go thru most in March during tax season and print out etc.....
I guess I never felt the need to back up my Hotmail constantly, I have relied on it for many years without issue....And what would be the best way to back up thousands of emails?
@westcoastborn: 10 years of email and you have no backup? Please...
edit: all microsoft web pages are extremely slow or do not load at all (e.g. microsoft.com). what's going on there?
I did not have the problem with the deleted e-mails. however, since yesterday, the hotmail pages are very (!) slow and i can't sync my windows live mail client with hotmail (can't connect to server). additionally, the solution center (www.windowslivehelp.com) is compleately down!
Having read 482 pages of the Windows Live Solution Center posts, is it not time to close this thing down or rename it? It is very far from being "Live" and completely useless for Solutions! Live Mail received a considerable amount of negative press concerning this issue because your Live Solution Center was giving generalizations & canned replies in lieu of adequate responses to inquiries from users.
It was quite obvious that a server issue existed!
My inbox of over 8000 emails from the start of this address over 10 years ago is STILL GONE....I only noticed it last week, but my emails were completely gone from late Oct 10 and prior...I AM DEVASTATED BY THIS LOSS, MY LIFE, BUSINESS, TAX INFO ETC , WAS ALL IN THIS EMAIL I ACCESSED DAILY!!!
I WAS GIVEN SOME INADEQUATE RESPONSES AND THEN IGNORED FROM HOTMAIL SUPPORT.
THERE HAS TO BE A WAY OF RETRIEVING!!
A 10 year client.
I'm thankful I wasn't affected, but I'm also thankful for the team effort in working to restore the affected users' mailboxes (as I'm sure they are as well).