In our war on spam, we’re making real progress. We’ve cut spam in Hotmail inboxes by 90% from its peak. We’ve played a key role in reducing spam on the Internet by 15% from its peak. And we’ve made it harder for spammers to use Hotmail to send spam – reducing “outbound spam” from Hotmail by 75%.
Last year, we wrote about how Hotmail was fighting a war on spam with our SmartScreen™ technology. This post gives an update on the latest and greatest features and innovations that we’ve brought to bear against the spammers. We’ve made it so hard on the spammers that they have now turned to a technique called “reputation hijacking.” I’ll explain how spammers use reputation hijacking across all email services and how Hotmail is shutting them down.
As you’ll recall from our earlier posts, spam is a huge problem that continues to plague the Internet. Historically, more than 90% of all email sent has been spam, and spam affects every email provider. Spammers do what they do because it’s profitable; they need only a few people to click on the spam messages in order to make money.
Way back in 2006, Hotmail had a big spam problem, and we got a deservedly bad reputation for it. Since then, we’ve made amazing advances, and over the last few years, we’ve wrestled the spammers to the ground. Here’s a chart that shows the amount of Spam In The Inbox (SITI) for Hotmail users over the last several years, compared with the amount of spam on the Internet (expressed as a percentage of all email that is sent on the Internet).
The chart shows two things:
Hotmail keeps spam out of your Inbox We’ve reduced the level of spam in Hotmail by 90% since its peak in 2006. Since last year, we’ve reduced what was left by another 40% (from 5% true SITI to 3% true SITI).
We’ve helped to reduce overall spam on the Internet The percentage of spam on the Internet has actually declined 15% from its peak in 2008, due to a number of factors including the legal and technical disruptive action Microsoft has helped drive in the prosecution of spammers and the takedowns of botnets used to send spam. Botnets – collections of people’s malware-infected computers covertly operating under the remote control of a cybercriminal – are often used to send spam (and commit other online crimes). This video explains a little more about how botnets are used to send spam.
Microsoft is working with law enforcement and others in the industry to proactively take down and dismantle botnets, including our recent takedowns of the Waledac and Rustock botnets. These disruptive actions are proving to be important in the fight against spam by taking away the tools and infrastructure cybercriminals use to spam the world. These efforts are paying off: before we took them down, Rustock was known as one of the largest single sources of spam on the Internet, capable of sending up to 30 billion spam messages a day. Global spam levels have gone down and stayed down since we took them out.
Our relentless pursuit and prosecution of spammers helps not only Hotmail, but all email users on the Internet. In fact, Microsoft has established a Digital Crimes Unit whose sole mission is to disrupt cybercrime like this. Spammers may keep developing new tactics and tools, but Hotmail and the Microsoft Digital Crimes Unit are going to keep working together on disruptive actions to help protect our customers and make the Internet safer for everyone.
Between 2006 and 2009, we dropped true SITI from 35% to under 5% with a variety of investments including connection-time filtering, content filtering, blocklist and safelist preferences, and more. Of course, the spammers continue to come and continue to get more and more clever. But we’ve not only held the spammers at bay, we’ve actually reduced SITI even more. Over the last year, we’ve dropped SITI to historically low rates – below 3%. Here are a couple of the new tools we’ve created to help us keep winning this fight:
Personalization Our spam filters are great at filtering out spam for the general population. However, we knew we could do better. So we created personalized spam filters that work based on how you use email – using information about the people you send email to and receive it from and also which email messages you actually read.
Trusted sender Hotmail helps you to visually identify trusted senders in your inbox, particularly banks and other institutions commonly used for phishing scams. We put safety logos next to only those senders that we recognize as legitimate so that you can more easily spot malicious imitators. It’s important to note that this also helps us take more aggressive spam-prevention action on email that is attempting to imitate a legitimate trusted sender.
These two tools augment the efficiency of our SmartScreen™ filters. But of course, we’re also continuously tuning the other SmartScreen™ features – like Time Travelling filter, IP reputation, URL reputation and more – to get additional gains in spam prevention.
Almost nothing is more frustrating for us than knowing that the spammers use Hotmail, too. Of course, spammers use all the major email services to send spam, and all mail providers must battle the problem we call “outbound spam.” Outbound spam is a form of “reputation hijacking.” After all, Hotmail maintains a good reputation among all email providers; simply put, email from Hotmail gets delivered, and the spammers know that.
Just as we’ve made great strides battling inbound spam (SITI), we have also made it increasingly difficult for the spammers to use Hotmail as a spam-sending tool. In fact, over the last year, we’ve reduced the volume of outbound spam from Hotmail by 75%.
Here are a few of the innovative features that helped us get it done:
Account reputation As you use your account, you gain a “reputation.” Good behavior (receiving email from the same people you sent email to, for example) gains you a good reputation. Bad behavior (sending a bunch of email and getting only delivery errors, for example) gains you a bad reputation, as these behaviors are indicative of spammers and other service abusers. Gain a bad enough rep, and we change the way your account works. For example, we will prevent accounts with bad reputations from sending mail.
Account creation limits We have a variety of ways that we throttle account creation in order to prevent spammers from getting an unlimited number of free accounts to use in sending spam. For example, we limit the number of accounts that can be created per day from a particular IP address.
Outbound content filters Just like we filter incoming mail to remove spam, we now filter outbound mail as well. For example, we look for suspicious content that matches known spam campaigns.
In the old days (you know, two years ago), the spammers just opened email accounts at one of the major providers to send spam. After all, accounts at Yahoo!, Gmail, AOL, and Hotmail are free and can send email, which is pretty much all you need to start a spam campaign.
But with the advances we’ve made in account reputation, these accounts have become less and less useful to spammers. Unfortunately, our success in preventing new accounts from sending spam had a tragic side-effect: Spammers turned to using existing customer accounts to send spam. This is a second form of reputation hijacking, in which the spammers are hijacking your reputation as a good customer of Hotmail. How? By hijacking your account.
In fact, most outbound spam now comes from hijacked accounts.
As the problem of account hijacking has grown over the past few years, we’ve invested more and more energy into protecting your accounts and, in doing so, making this avenue of sending spam less and less attractive to the bad guys. We fight account hijacking by focusing on three key activities:
Detection When a spammer hijacks an account, we have many ways of detecting that hijacking. We look for unusual behavior from the account, including access from unusual IP addresses, sending an unusual volume of mail, sending mail that triggers our outbound spam filters, etc. We even introduced a feature that lets you report your friends if their accounts get hijacked.
Remediation Once we’ve identified an account as compromised, we want to block the hijacker from accessing the account and then return the account to the rightful owner as painlessly as possible. We typically block the account and then send the real account owner through an account recovery flow that the bad guys will have difficulty getting through. We provide many ways for you to protect your account by setting up “proofs” that only you will be able to use to prove account ownership. We strongly encourage all our users to set up these proofs on all their email accounts. Proofs include:
Prevention Of course, the best way to fight hijacking is to prevent it from happening in the first place. The problem is that hijacking is fairly straightforward in many cases – it’s just a matter of getting your password. Hijackers get your password through several methods:
We’re fighting all of these not only in Hotmail, but in Windows and all the Windows Live services. For example, we’ve made IE more secure by detecting URLs with bad reputation, and we’ve added phishing and social engineering detection to SmartScreen™.
We’ve made tremendous progress in our battle against spam, but we know that spam and hijacking will continue to be a big problem for all service providers as long as there is economic incentive for the bad guys to do what they do. So we’re not letting up. We continue to invest in research and development to find ways to make it even harder for the spammers to get spam into your Inbox and to use Hotmail as a way of sending spam.
In my next post, I’ll go a bit deeper on one of the most insidious ways that spammers compromise your account: Phishing attacks. See you then.
Dick Craddock Group Program Manager, Hotmail
Follow-up question (assuming this is being read): if an account is hacked, then the hacker can change all of the information that hotmail uses to validate the original owner, such as alternate e-mail, security questions, etc. So how would this ever work in practice? The user is at the mercy of a full account restoration which apparently takes some indeterminate of time (5+ days) to begin processing.
What is the support flow after an account recovery request? I submitted a recovery request with the required information on Sunday and have not heard a peep from any support personnel. To make matters more frustrating than they already are, the automated e-mail that I cannot reply to contains a ticket number that I can't enter anywhere (so why send it?). The e-mail indicates it takes up to 24 hrs for a response, but it has been 4 days...that's pretty much untenable and has resulted in me starting to transfer everything to gmail.
My Account is Blocked :
The account Hotmail is using to send the Validation code to is an invalid domain "hothail.com" How may I change the alternate email address or add a phone number? I have had no response from the Hotmail support forum and it seems they are just picking the low laying fruit and bypassing anything witch needs a thought. I value my hotmail account and run my business via hotmail....
Anything would help!
The account Hotmail is useing to send the Validation code to is an invalid domain "hothail.com" How may I change the alternate email address? I have had no response from the Hotmail support form and it seems they are just picking the low fruit and bypassing anything with needs a thought. I value my hotmail account and run my business via hotmail and have had to resort to ur competitor just to use email.
I have had my account hacked and have asked for a validation code to be sent through to my alternative email but nothing has come through for the last 4 days. Any help would be grateful. I have set up a new email to log in to send this.
A responsible email service provider, Hotmail has to attach great importance to security issues. As we know, spam is one of the most popular threats. In my opinion, the spam emails do not only refer to the junk e-mails or the unsolicited emails; they are also including the emails which contain worms, virus and Trojans. Compare with junk emails, the emails with worms, virus and Trojans may have higher risks. The junk emails may only show people the contents which they do not want to read. However, emails with malware are able to steal people’s personal information and damage people’s computer system. I am glad to see that hotmail has reduced 90% spam. As a regular hotmail user, I have felt that spam emails within hotmail are not as many as before. However, I found I always receive spam from my friends; because my friends’ computer may be toxic. All these spam have attachments. If I click the attachment, my system will be toxic as well. As a result, I think hotmail also needs to pay attention on computer viruses spread via emails. The malware may be not from mass-mailers but your friend; so it will be difficult to monitor.
I haven't had spam or Junk in my inbox for months... :-)
JUST ONE WORD
MY INBOX IS SAFE AND CLEAN...
Problem seems to be solved
When sending/receiving mail in Hotmail via Outlook Connector (https always on) I get the following errors:
- Task ‘@hotmail.com’ reported error (ox80004005): ‘Network operation failed’
- Task ‘@hotmail.com’ reported error (ox8004102A): ‘Error with Send/Receive. There was an error synchronizing a contacts folder. The network connection is unavailable or interrupted. Please try again later.’
Outlook also warns that that there is Limited Connectivity (a yellow triangle in the right bottom corner).
Could you please solve the problem. Thanks in advance
@trulyindian, that's why I would say never trust the word that comes from MS when when say they are the best on.... and the first on....Even though I like and use many of their products despite the fact of many flaws, I hate it when MS makes self proclamation. Besides this, others are the right click feature (they should have been shamed to make this bold claim, for making late delivery than yahoo and google and still claiming as something out of this world), claim of loading hotmail faster than yahoo and google. So, the users should trust their own result on what they find instead of what MS claims. Well, we like and always appreciate the good things to come from MS but putting its loyal users on delusion makes me mad so bad.
I do not agree with this post by Dick Craddock. I use two hotmail accounts. one for Completely Professional use and another for personal use. But I must admit I get more spam on My professional ID even when I never use that address except for my professional communication. And my second address: I have used that to register on many sites, newsletters etc. and still it gets less spams as comapared to first one. I created professonal ID just some time back whereas am using personal ID since my school days even then I get more spam in my Professional ID. The problem is not only with my account. Even my father and my brother have the same problem.
As am using all the popular email services (yahoo, hotmail and Gmail), I must admit that Gmail gets least spam messages, second is hotmail and yahoo is third.
One hidden fact: I get less spam in hotmail than yahoo not because they are better than yahoo but because I have blocked messages from innumerable senders in hotmail and have not did that in yahoo.
Every now and then I get an e-mail in my Hotmail account from the Junk Mail Classification Service. It is a service set-up by Microsoft a loooong time ago which sends to your inbox copies of some of the mail you have received recently and which I guess the filters had a hard time determining whether it was junk or not. Then, at the top of each e-mail it asks you to classify it and tell it whether it is junk or not. It could be a useful service as it explicitly asks you to classify mail in order to improve the junk mail filters. However, it hasn't been updated in years. It still talks about MSN Hotmail for example. And the links do not delete the e-mail notifications it sends when clicked from Outlook even though it says they would. This makes me wonder if it even works and if any of the classifications made through this service are recorded, or if it is simply a discontinued service that some Microsoft servers, forgotten behind in one of the Hotmail update cicles, still run without the company even knowing about it. How else can one explain the reference to MSN Hotmail a name that was droppe 5 years ago. How can any company be running a service that is so outdated and even perhaps broken and which nobody at the company even would care to update at least a tiny bit to fix the branding or to make it compatible with the new Hotmail. How can one trust that company with their more important stuff such as their e-mails if they show this attitude of "I doo not care"?
This is the general atmosphere that I feel exists around Hotmail. Innovation is slow and updates are so slow in coming that one wornders if Hotmail is still run by live programmers who care about their users and who ffeel motivated to work hard. Promises are met after years, whilst improvements are not frequent and continuous but tend instead to follow a major versions pattern with long gaps between versions. The problem with that is until the next major version of Hotmail rolls out, it is usually behind the times and although the new features might have looked great a year ago while they were planned, the approach of releasing new features in slow and non-frequent cycles destroys any value they would have had if they had been released immediately. Hotmail management seems to have no new ideas or exciting features and even the name Hotmail reminds of the older generation and the past.
Could you address your false-positive rate? By that I mean I get a number of reports from people who aren't receiving email that they've signed up for, or email from people that that are legitimate.
I see more spam in my Hotmail account now from people with msn.com and hotmail.com accounts.
Perhaps this is why I’ve started seeing less junk email. Just yesterday Smart Screen correctly identified a phishing email (junk box), which at first glance would have appeared to be from well known Bank. A closer look at the email I found at least 2 problems wrong with it, especially the Log On link (nope didn’t use it). Keep up the good work!
On my Hotmail account, I got a notification from Microsoft Answer, opening the email had some blocked content. When I clicked on 'Show Content' option, it shows me this message (in a high-alert pink box):
"This message looks suspicious to our SmartScreen filters."
What should we conclude from this, MS Answers is generating ill-formatted/malign email content or SmartScreen is not smart enough?
FWIW, the sender's email address under this discussion is:
Microsoft Answers (MicrosoftAnswers@microsoft.com)
I assume, there is some specific "content" in the email itself that SmartScreen filters dont like. At first it says:
"This message looks very suspicious to our SmartScreen filters, so we've blocked attachments, pictures, and links for your safety. Show content"
When I click on Show content button, it shows the underlying (blocked) content but a short-form of that message keep sticking:
P.S. Another point I didn’t get, if such emails look suspicious to SmartScreen filter then why they are even in my inbox and not sent to the spam folder in first place ?
@ian.aldrighetti - The normal default is to report the mails, we've got a bug at the moment and we're working to correct it.
I have a question... I guess.
I have noticed that whenever I have created a Hotmail account that Hotmail does not default the option to report emails marked as spam to Microsoft (or anyone else). I even recently made (and setup) a Hotmail account for my grandmother and the option does not default to "Report Junk - Help us keep junk out of everyone's inboxes when you use the Junk button."
I highly doubt that many people enable this option, in fact I didn't even know about it for years, and I mess with options on every service I encounter because I like to check that stuff out.
Do you think that by enabling this by default, or at least asking a user when they report email as junk (and the option is set to "Don't Report"), would improve the accuracy of the spam filter even more?
Just a thought.
@cvdonato i'd love to take a look at your spam folder. if you're interested, send me a private message here with your hotmail account name. i do not need your password, and no one from msft will ever ask you for it, either ;)
What are you folks doing about the Windows Live Alert Account Spams and Windows Live account spam?
My spam folder still continues to receive a lot ( a LOT ) of e-mail type @81l14n_.com, @breakfreedev.net, cocoastillcreek.info, etc, etc...Even falling into the spam folder is very inconvenient, and the configuration option to automatically delete the contents of the spam folder is disabled and no longer has the option to leave it active.I think hotmail should know (detect) these e-mails are spam and do not let them appear even more in the spam folder.
I didn't know about Account Reputations. :-)
Can you please explain (I know this is off topic, sorry) why you removed the Linked IDs feature?
thats really cool...but I doubt if hotmail can still bit Gmail.... :)