Delivery

From Halon, SMTP software for hosting providers
Jump to: navigation, search

The Halon SMTP software is used by many as an outbound anti-spam. Successfully delivering email (measured as "deliverability") to external parties (email servers on the Internet) is important in many cases, such as outbound email services, forwarding and even VPS (cloud) providers. Halon's scripting language HSL lets you design and tailor the logic to handle compromised accounts and abusive users; in order to avoid blacklisting without bothering legitimate users.

Source address

We recommend having multiple (warmed up) IPs per cluster node in order to do

  • Source hashing different customers (domains)
  • Sending suspect spam out though a bulk IP

Since each node have different IPs, it's convenient to address the IPs using their ID (netaddr:X). The example pre-delivery script below uses IP 1-3 for normal traffic, and IP 4 for suspect spam. The metadata "spam" variable should be set in the DATA script.

if (GetMetaData()["spam"] == "yes") {
    
// Bulk IP
    
SetSourceIP("netaddr:4");
} else {
    
// Source hash
    
$addrs = ["netaddr:1""netaddr:2""netaddr:3"];
    
$sourcehash number("0x".md5($senderdomain)[0:6]);
    
SetSourceIP($addrs[$sourcehash count($addrs)]);

It's also possible to set the correct HELO message based on the PTR record:

$ptr cache ["ttl" => 3600dnsptr($sourceip, ["extended_result" => true]);
if (isset(
$ptr["result"][0]))
    
SetHELO($ptr["result"][0]); 

Anti-spam and rate limits

With the right tools, outbound anti-spam can be extremely effective because you know who the sender is; their $saslusername if authenticated directly with the Halon system, $sender address (if enforced by the email server), $senderip (in the case of a VPS provider), a header in the message (enforced by PHP), etc. Therefore, we recommend that you create a deferring, rate-limit based script such as this:

// Identify the sender as accurately as possible 
$customer $senderdomain;
if (
GetHeader("X-PHP-Originating-Script"))
    
$customer GetHeader("X-PHP-Originating-Script");
if (
$saslusername)
    
$customer $saslusername;

// Defer high volumes of suspect spam
if ((ScanRPD() == 50 or ScanSA() > 4) and rate("outbound-bulk"$customer30028800) == false)
    
Defer("You are only allowed to send 300 bulk messages per 8 hours, try later $messageid");
if ((
ScanRPD() == 100 or ScanSA() > 6) and rate("outbound-spam"$customer10028800) == false)
    
Defer("You are only allowed to send 100 spam messages per 8 hours, try later $messageid");

// Suspect spam flag to the pre-delivery script
if (ScanRPD() > or ScanSA() > 3)
    
SetMetaData(["spam" => "yes"]); 

which should be accompanied with a upper limit in for example the RCPT TO script

// Max messages per hour
if (rate("outbound"$senderdomain2503600) == false)
    
Defer("You may only send 250 messages per hour, try later"); 

Act on deliverability

Reacting to automatically measured deliverability is a powerful way to avoid blacklisting. By counting the delivery failure rate in the post-delivery script

// Counts up to a rate of 1000 failures per hour, you can use another ID than $senderdomain
if ($errorcode >= 400)
    
rate("delivery-failures"$senderdomain10003600); 

which is read in for example the the RCPT TO or DATA script

if (rate("delivery-failures"$senderdomain03600) > 999)
    
Defer("$senderdomain has more than 1000 failed deliveries during the last hour");
if (
GetMailQueueMetric(["filter" => [ "senderdomain" => $senderdomain ]]) > 500)
    
Defer("$senderdomain has exeeded the max queue limit of 500 messages"); 

Connection concurrency

We recommend having a sane limit for the number of concurrent connections per destination server or recipient domain. This is even more important for common (high volume) recipient domains such as gmail.com. Again, the scripting language enables you to create a powerful, fine-grained logic. Our pre-delivery script example below however, is pretty basic;

$concurrent 3;
if (
$recipientdomain == "gmail.com")
    
$concurrent 10;
if (
CurrentConnections("to-domain"$recipientdomain$concurrent) == false)
    
Reschedule(rand(1300), ["reason" => "More than $concurrent connections for $recipientdomain""increment_retry" => false]); 

Reporting

The scriptable reporting enables you to design just the right metrics for you. For example, the post-delivery script below

if ($errorcode >= 400 and $errorcode 500)
    
stat("delivery-failures", ["500" => 0"400" => 1]);
if (
$errorcode >= 500)
    
stat("delivery-failures", ["500" => 1"400" => 0]); 

creates both a line chart (over time) and a pie chart for the delivery failures.

Forwards

If you deliver (inbound) email directly to your email storage server using for example LMTP, it's common to also do forwarding and auto responses in the inbound script. Forwards should typically be treated in the same way as outbound email in terms of outbound anti-spam filtering and rate limiting. We also recommend using SRS.

DKIM

While not necessarily a deliverability feature, many customers has DKIM as part of their outbound deliverability plan. The exceptionally lightweight and heavily optimised DKIM signer in the DATA context is based on our libdkim++ project, and enables you to deploy per-domain DKIM for all your outbound traffic with very little overhead.

// Override the deliver function
function Deliver() {
    global 
$apiurl$senderdomain;
    
$dkim cache ["ttl" => 86400"size"=> 32768"ttl_override" => ["" => 60]]
        
http("$apiurl/dkim/$1", ["timeout" => 10], [$senderdomain]);
    
$dkim json_decode($dkim);
    if (
is_array($dkim))
        
DeliverWithDKIM($dkim["selector"], $dkim["domain"], $dkim["rsakey"]);
    
builtin Deliver();

Transparent proxy

Most of the examples above works equally good in a transparent proxy installation, suitable for VPS providers that (for whatever reason) have chosen not to enforce the usage of an SMTP relay.

Feedback loops

Receiving Feedback Loop Reports (FBLs) is an important step in monitoring your deliverability rate. By analysing the complaints you receive for emails sent through your system you can identify common causes and take action. Below is a list of some of the biggest FBLs but it's by no means exhaustive.