Think Information. Think Security.
As Spring cracked the Moscow frosts and March rains doused the streets, a computer in an innocuous server farm somewhere in the heart of the city winked to life. It was 2007, a year when many people became truly invested in online life. Twitter was a year old and the most popular smartphone was the Blackberry Curve – a pure email machine. It was a year ripe with promise for cyber-everything. And a group of hackers, unnamed to this day, wanted to grab their piece.

The server first sent a blast of emails containing a link to a piece of software that many around the world wanted to download. Once they grabbed the Trojan Horse, the infected program took over computer after computer, creating something security experts call a botnet – a collection of infected machines controlled by a central command and control unit (CnC).

This Moscow server, hidden behind IP address, began receiving incoming messages from a number of computers around the world. A whole swath of California lit up as communicating programs came online. Then New York, then London, Berlin, and Minsk. Computer after computer began chirping out requests to the mothership. The infection spread thanks to a mixture of gullibility and trust seen time and time again in the annals of computer security. The Grum botnet was born.

It took a few days, but ultimately 120,000 machines spoke to the Command and Control server in Moscow and the server messaged back. Some machines dropped out of the network thanks to vigilant users but others quickly took their place. It was like a mold grown over the globe, spores spreading through various networks. Grum sent over a quarter of the world’s spam and was one of the most ingenious botnets ever created. But, with savvy, a lot of luck, and cooperative ISPs, the Grum botnet dried up and died last month.
The Vector

That March, Internet users began receiving emails from with the subject line “Internet Explorer 7 Downloads.” A click later and they were at a bright splash page purporting to offer a fresh download of the latest Microsoft web browser, Internet Explorer 7. The download was a dud. Clicking on the link brought nothing but a small file called ie7.0.exe. Running it revealed nothing – just a little gibbering in the hard drive and then silence. Users could click all they want – IE 7.0 wouldn’t appear.

To many, this was just another bum link on the Internet. But inside their computers, something was happening. The skittering meant something had been installed on the hard drive, within a temporary Windows directory. The file was winlogin.exe, an innocuous enough name that might have been familiar to slightly savvy PC users. In less than a second, however, the program burrowed its way into the computer’s registry – a database of information about the machine – and added itself to the list of programs run when the computer begins to boot.

Eventually, the program was identified as the Grum-A aka Tedroo and Reddyb. It was probably written somewhere in Russia and carried a payload called a “rootkit” – a program that gave an outside user administrator access to the hard drive. Grum listened for a set of commands sent by the CnC servers. The simple commands came through a standard HTTP port and could “update” itself automatically. Initial reports saw the worm as fairly harmless. One security firm described it this way: The execution of this virus leads to an attack on all executable files that it can find stored in the hard drive of the infected computer system. The presence of the W32.Grum.A will also allow the installation of a rootkit which is used to conceal the fact that the system has already been compromised. The user normally is led to a false sense of security believing that the computer system has maintained its integrity.
The primary locus of infection was a program that ran every time Windows booted. By adding code to a kernel library called ntdll.dll, the virus was able to hide and run itself automatically every time the user started his or her computer. Deleting ntdll would be catastrophic and because it was a high-level, privileged file it was nearly impossible to pull it off. More importantly, however, is the way Grum worked internally. Each copy of the virus spoke with a set of CnC nodes and the CnC system could segregate infected computers into different secondary groups. However, the program had a fatal flaw.

The virus contained a set of hard-coded master IP addresses. Instead of sending commands to, say,, the program sent messages to a set of two or more CnC IP addresses. Like a biological virus primed to thrive in a certain type of medium, the Grum virus was susceptible to defeat if someone knocked out each of those CnC IP addresses. The commands weren’t human readable – there was no “SEND SPAM” command – but it was fairly easy to see what was going on with a bit of effort.

Grum’s creator’s foresaw this problem and placed their CnC servers in countries that had, in many cases, lax or nonexistent, cybercrime laws. The initial IP addresses were in Russia but others popped up in Panama, the Netherlands, and the Ukraine. To be clear, there was nothing inherently bad about these ISPs. They weren’t about to practice Internet censorship and given the distributed nature of the CnC system, the Grum botnet kept a low profile even as it sent its commands out to various parts of the network.

As the botnet spread, its creators sent out periodic updates that fixed bugs and identified new CnC servers. If a CnC server went down, the coders would update a new binary with the new IPs. These binaries would spread slowly because not every infected machine would check back in with the mothership every day. Like Microsoft or Apple pushing out OS patches, the Grum makers were upgrading their virus regularly, adding new features and fixing problems. The Grum botnet was one of the most robust and powerful in the world. Aside from its single, glaring flaw, the system worked without peer and slowly began spamming the world, mostly with poorly worded pharmaceutical emails. Every time someone pulled the plug on a CnC server, a new one popped up somewhere else.
CnC Virus Factory

Spamming isn’t very lucrative. Brian Krebs, a security reporter, notes that while businesses spend $40 billion per year for anti-Spam technology, the estimated revenues of most major spammers hover at around $150 million in a good year. In the bell curve of spammers, however, most end up on the side of making very little.

In an excellent series, Krebs was about to track down the creator of Grum and its leader, a hacker name Ger@ or Gera/GeRa. By tracing money back to the source, Krebs was able to assess who, specifically, was making the most money from spamming. Gera’s affiliate account, gleaned from a list of payments for the pharmaceutical sales program SpamIt and Glavmed, showed that his efforts brought in $6 million in 2010. This data suggested that Gera was a very prolific spammer. Further leaked documents showed repeated conversations between SpamIt leader Dmitry Stupin and Gera. Stupin called Gera out for his practices, saying that he was beyond compare when it came to “trouble with hosting providers.”

Grum was probably run by a small team led by Gera and that, even given its reach and relative lucrativeness, the entire operation was streamlined. While it could move fast, this could also mean the organization wouldn’t be able to react to a massive shutdown. Other botnets had ways to dynamically reassign CnC servers very quickly. Grum did not. Gera was also not particularly beloved by ISPs or even the affiliates that used Grum’s botnet to send pharmaceutical Spam. It was, in other words, a nearly perfect target for some dedicated anti-spam researchers.
The Bot Fighters

In 1998, a former Pink Floyd production manager and songwriter, Steve Linford, realized his computer-consulting clients had a huge problem: spam. Over the course of about a year, Linford began collecting the source of most of the spam circulating on the Internet and created a list called ROKSO – Register of Known Spam Operations. This living list, updated regularly with new and reformed spammers, has long been the first line of defense for most spam fighters. For years, Linford lived on a houseboat in the Thames but now the organization has grown, with headquarters in Switzerland and the UK. Linford’s organization, Spamhaus, went on to become an anti-spam powerhouse, garnering respect and fear from ISPs around the world. ROKSO itself blocked the 100 known spam operators responsible for 80% of spam and systems that used its data were able to reduce spam considerably. However, some still got through, and the worst of the spam came from the relatively anonymous botnets.

One researcher, Carel van Straten, worked from Amsterdam and watched botnets rise and fall. A cheerful senior spam researcher, he was very well-versed on the ins and outs of rogue server hosting.

The soft spoken Senior Staff Scientist for FireEye in San Francisco, Atif Mushtaq, had that target. Mushtaq studied computer science at the University of Management And Technology Lahore and worked as a network architect for Palmchip in 2008. He moved from Pakistan to the Bay Area where he began writing a series of concise, sometimes breathless, posts about his efforts to find and shut down popular botnets. None of these security experts enjoyed the limelight. Anti-spam researchers have been harassed, threatened, and their websites have been shut down by angry spammers. Spamhaus, for example, rarely publishes photographs of its researchers in order to protect their privacy online.

Meanwhile, in Moscow, a computer security rapid response team was also following the Grum virus. In 2011 the botnet remained stable and strong but in the spring and early summer of 2012, researchers noticed that the number of CnC servers was falling slightly and that multiple servers were in only three countries – the Netherlands, Russia, and Panama. Perhaps all it would take was a few good taps to shut it down? “Grum was the world’s number one spam botnet back in January 2012,” said Mushtaq.”  Then in the last six months, there were less command and control servers and it was sending less spam.  I didn’t know why it was happening but I told myself ‘Okay, this is the right time to do it.’”

Mushtaq began by assessing the list of CnC servers for holes. Immediately, a few things stuck out. Panamaserver SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA SteepHost DC-UA GazInvestProekt ltd. ECATEL LTD ECATEL LTD PROEKTPROFDEVELOPMENT-NET

Although it looked like a large list, most of them were in the same location and some even in the same building. SteepHost DC-UA, for example, was based in Kharkiv, Ukraine in a building by the main train line. ECATEL was a Dutch ISP and Panamaserver was, as expected, in Panama. The rest were in Russia, including GazInvestProekt, a small ISP in Pskov. None of these ISPs were “rogue,” per se. It was generally bad business for an ISP to shut down a server or IP address based on complaints by security firms – he-said-she-said back and forths were rarely constructive. However, they did respond quickly whenever someone reported true abuse.

“ECATEL does have a very long history of hosting shady things,” said van Straten. Seeing Mushtaq’s detailed posts, van Straten reached out to FireEye to see if they could help take down some of the servers. As luck would have it, Mushtaq was ready to move on his first decisive attack.
Killing The HydraOn July 9, 2012, Mushtaq began musing on a Grum takedown.

“For a successful takedown attempt, we need to clearly identify Grum’s command and control coordinates. We also need to find out what would happen if the master CnC servers become unavailable during a takedown attempt. If Grum has a fallback mechanism, then we need to disrupt the secondary CnC structure as well and so on. The most important of all is the geo location of active command and control servers. Historically, it has been relatively easy to shutdown CnC servers located inside of the U.S. as compared to countries like Ukraine, Russia, and China,” he wrote on his blog. “Keeping all of this information in mind, I am getting mixed feelings. I can see a few factors that can go in favor of the Grum botnet. At the same time, Grum has some obvious architecture-level weaknesses.”

However, as he examined the servers, he noticed Spam levels were dropping precipitously – down 30% over the last year at least – and the thought the time might be ripe to pull the plug. However, that was just the beginning. With the Dutch servers down, the botnet creators had a few days in which to bring up new servers and send out updates to all of the infected computers. At that point, time was against him. He began to reach out the other providers. One developer, Isidro Gonzalez, told Atif that he could try to help shut down the botnet in Panama.

Van Straten began working more intensely with Atif and the pair was able to convince SteepHost in the Ukraine to shut down their servers. The worked closely with a response team in Russian, Group ID, to hit the servers quickly and quietly. They took down most of the servers – the Netherlands servers were gone and Panama was about to wink out. However, Spamhaus’ “big hammer” worked. The Ukraine servers were toast. And then one more came back up again. As the Grum “bot-herders” saw their servers die one after the other, they continued to try to bring up new servers. 5 years, 3 months, and 17 days after the first emails began spewing out of the Grum botnet, the last server was dead. The Internet got just a bit quieter. 
The After Party

Mushtaq was stunned. The bot was dead. 120,000 Grum IP addresses dried up to about 21,505. These zombies, unable to communicate with their CnC nodes, would eventually disappear, unable to send out any more spam. The only way to restart Grum would be to reassign the dead IP addresses, and Spamhaus would make sure all of those were on a watch-list. Spamhaus allowed them to attack with a purpose and not needle ISPs with random requests. But Mushtaq wasn’t stopping there.

Cross-posted from: Tech Crunch

Leave a Reply.