Email Server Vulnerability And Anti-Spam Protection Techniques

Importance of email server security

The role of cyber security in modern business is hard to overstate. Almost all business processes are automated to a degree, and thus need to be thoroughly protected from any potential tampering. Vendors use anti malware and anti reverse engineering techniques to protect their products, but they can’t possibly weed out every vulnerability.

One particularly vulnerable area is communications. Your company probably intensively uses emails in your business communications, just like most other companies out there. It means that crucially important data gets stored on email servers and transferred via the internet. Thus, the problem of securing those email servers becomes extremely important.

However, this issue is not as simple as it sounds. One part of the problem is to protect outgoing data, which can be done by using encryptions and making sure that you’re sending it to the correct recipient.

But what even more important is to protect your server from incoming emails – spam with malware-ridden attachments, as well as denial of service attacks.

In this article we will look at various ways to protect email servers from spam and other cyber security threats and will give you some particle tips on how to detect, evaluate and fix vulnerabilities.

Dealing with vulnerabilities

Every cyber security breach is a result of particular vulnerability. As mentioned above, it’s impossible to weed out all of them, but nevertheless, we should do what we can.

A great way to limit the number of vulnerabilities is to actively follow the best cyber security practices for email server setup and maintenance. This will allow you to avoid the most common issues and make sure there are no obvious holes in your defenses:

• Store the minimum amount of data – any unnecessary data, stored on the server, simply widens potential attack surface and contributes to damage costs in case of an attack. Make sure that you aren’t using any unnecessary software and that all opened ports are in use and thoroughly protected (for example, via authorization requirements).
• Make sure that your server is up to date – software always contains vulnerabilities and when one is discovered, vendors usually issue a patch over the course of couple of days. You need to make sure that all the components of the server are always up to date with all the latest security patches and fixes.
• Employ a strong authentication procedure – set a complex password requirements for any account used to access the server. This will prevent a brute-force attack, which is one the easiest ways to crack a password. Other security measures depend on your specific hardware and software configuration, such as type of the server and OS in use, etc.

One more way to protect a server from unauthorized login is to use SMTP authentication. This is something that we will cover in more detail down the line.

Another basic cyber security measure is to make sure that all your emails are thoroughly encrypted, so as to protect the data from being intercepted via man-in-the-middle-type attack. You must encrypt SMTP, POP3, and IMAP protocols with SSL/TLS type encryption.

The problem of spam emails

Apart from general vulnerabilities, described above, probably the biggest problem email servers face today is spam emails.

This problem can be further divided into two categories’:

• Incoming spam messages – spam messages from outside, sent to the server’s own clients
• Outgoing spam messages – spam, sent from clients to other parties, where server acts as an Open Relay.

The best way to fight spam is to use content filtering. Such filters should be configured either on the server itself or via a proxy application, such as firewall, that protects access to the server. Besides filtering, you can also blacklist known spam sending servers. There are a number of local IP black lists, as well as DNS based lists, such as DNSBL, and SURBL out there.

And to prevent an Open Relay, you should configure Mail Relay parameters for email server.

One of the biggest problems of spam is that it often carries malware via attachments or links in the body of the email that infect the whole system when clicked. Infected email server is a threat to the stability of the whole system, not to mention, the risk of getting customers private data compromised.

There are, however, myriad of tools, both built-in and third party, designed to protect email server from malicious software.

Server stability and performance

Another concern with regards to email servers is their stability and server performance. And when we think about performance, the first thing that comes to mind is load balancing.

With this regard, Denial of Service (or DoS) attacks can prove extremely damaging, as they can render the whole service out of commission for long periods of time. This can have double costs – both in remediation, as well as lost reputation and customer loyalty.

To prevent DoS attacks, you need to limit the amount of both general over time and simultaneous connections to SMTP server.

Another type of DoS attack is sending high number of Send requests. To protect from it, you may want to enable SMTP authentication. When enabled, each time someone wants to send an email to the server a set of credentials would be required.

Other ways to protect the server from large quantities of send messages include Mail Relay and Reverse DNS. While the former allows you to specify IP addresses from which the server can send mail, the latter allows to compare IP addresses with domain and host names.

Also, as a general rule of thumb, if your server doesn’t work, regardless of the reason, you need to have a reserve server ready. You can do this by having two MX records for each domain.

Security assessment

First, you need to make sure that you have a way to assess server security. Often times the best thing to do is to take to solutions that are already there, such as cyber security audit services. However, if this is not an option, then you need to design your own process, choosing yourself how formal and flexible it should be.

Initial preparations

First thing first, you need to determine the scope of your audit. To do this, you need to answer three simple questions:

• What you need to check: list every piece of data (e.g., user names, attachments, contacts, etc.) and every parameter (e.g., uptime, performance, etc.) that you consider important. Subdivide it into a several separate checklists within each area of responsibility (such as operating system, server, network). Weight each entry on your lists according to the potential impact that the problem with this entry may cause.
• How you need to check it: there are two approaches to this question:
  • Find the tools necessary to check whether the components on your list are vulnerable or not. Each entry on the list should correspond to a specific way of checking its vulnerability.
  • Derive additional controls from your list of potential vulnerabilities and add them to the list of monitored objects. Repeated entries are market rather than deleted.
• Why you need to check it: target priority is derived from its weight and from the effectiveness and scope of the check. We assign priorities for every entry, and remove repeated ones only when they are fully covered by another control or a combination of controls.

The next step is to assess the worth of each security check with regards to resources necessary to run it. This includes estimating the costs of buying software, hiring or training personnel and conducting the checks themselves. If the cost of a low priority entry is too high, it can be moved further down the list, but not fully removed, because situation can always change.

Checklist from NIST SP 800-45 is a great asset for creating your own list of objects to check.

When the list is fully formed, all that’s left to do is to set a scope and designate resources for an audit. As a part of this preparation process, you can also make a detailed plan covering each procedure from top to bottom up to the point when the data is included into final report.

Checking for vulnerabilities

Now all you need to do is to conduct all the necessary checks. When a strict time limit is involved, you should get high priority items out of the way first. However, if there is no time limit, it may often be best to group checks out of convenience based on their scope – this approach can help you save both time and money.

Sometimes, executing a check can take more time than was initially designated. In this case it is often better to skip a check and move it into a separate group, while trying to find a way to optimize the process.

Any incident regarding data and server settings should be logged. In this early stages you don’t need to investigate each and every detail, but instead make sure that your checks cover as much as possible in the designated time.

Analysis of detected problems

The main formula for assessing risks is the following:

“Exposure*Likelihood*Impact”

As such, each control can be scored based on the severity of issue, for example, a 1 to 5 scale where 5 indicates a critical problem that needs to be solved ASAP, while 1 indicates that it may be to inefficient to try and solve the problem.

• Impact – whether an impact is small (e.g., server not working for 100 ms) or large (e.g., database has been corrupted) is entirely depends on the component in question. Our checklists should give us clear information on the impact of each entry.
• Likelihood – this depends on the repeatability of the problem – whether it can frequently and easily reoccur. Problems range from very rare (e.g., running out of memory when sending exceedingly large amounts of messages, something that happens no often than once a year), and stable (e.g., running out of memory each time while getting 2²⁵⁶ udp packets via open tcp port).
• Exposure – this depends on how frequent a problem occurs during the regular operation of the server, and how hard it is to detect. The range vary from unavoidable (e.g., using “admin-admin” as a login-password pair, or server crashing due to receiving a specific large number of emails), to impossible (e.g., several highly unlikely problem occurring simultaneously).

Based on the formula (Exposure*Likelihood*Impact), problems are sorted by the level of risk. Then you need to discuss each problem greater than a certain risk value (for example, 8, if you’re using a 1 to 5 scale for each of the three parts of the formula). As a result of a discussion, you should be able to divide all the problems into security vulnerabilities, flaws, and ignored problems. Then you need to assign priority to each flaw and vulnerability.

Fixing vulnerabilities

There are three main ways in which vulnerabilities can be fixed:

• Start using the version of the product without the vulnerability (usually by applying the latest official fix)
• Eliminate problem by installing other software
• Disable the feature with a vulnerability

However, you need to consider your budget, risks, as well as associated costs, as these are all the things that will ultimately impact your schedule.

It is also often beneficial to start fixing smaller problems before you get to bigger ones. Large fixes can take up to several days, and through all this time small and important problems are put on the backburner, which introduces additional risks.

Sometimes you can also use a single fix to deal with several vulnerabilities. This can both be a great time and money saver, however, you need to makes sure that the fix is truly reliable and will not cause problems in the future. While such situations are ideal, exceeding corner cutting can lead to problems later down the line, so it’s best to not focus on it.

 Final tips

The field of cyber security is constantly evolving and email server security is no exception. However, as a conclusion we wanted to list a set of basic tips that should definitely be followed by everyone who wants to secure an email server.

First, you need to make sure that security is on the table as early as possible. Many problems can be solved by setting up a server initially with security in mind, not to mention that this is probably the most cost-effective way to do things. The things you need to consider include:

1. The type of data that will go through the server and the type of services it will support
2. What level of security is required for the server
3. Who will use the server and what level of privilege will they have
4. What method of authentication are you planning to employ
5. How the server will integrate into existing network infrastructure
6. What other software needs to be installed
7. How the server will be maintained and managed

You also need to consider the required cyber security level and potential vectors from which your email server can be attacked. Another important security factor is an operating system the server is installed on.

So, to sum things up, here are the most basic recommendations on email server security:

• Make sure that attack surface of your server is as small as possible. The best way to do this is to establish a network perimeter that will protect your corporate network. A proxy application within the perimeter (for Exchange Server it can be Edge Transport server) can be linked to an email server and used to transfer emails from and within your corporate network.
• Always apply encryption on any stage of data transfer. Never use self-written certificates, and instead carefully select SSL certificate for each component of the server.
• Don’t forget about basics – the fat that email server has built-in anti-malware capabilities is not a reason to drop third-party anti-viruses and anti-malware solutions. Using them will only help to reinforce your protection.
• Don’t forget about updates. Microsoft, for example, has a Security Bulletins with all the latest patches.
• And last, but not least, set two MX DNS records and don’t forget to backup your data.

[su_box title=”About Dennis Turpitka” style=”noise” box_color=”#336588″][short_info id=’102375′ desc=”true” all=”false”][/su_box]