HSL code examples

From Halon, SMTP software for hosting providers
Jump to: navigation, search

Below is a smorgasbord with some script examples you may find useful. For more information about HSL, please see the reference manual.

Flow

These scripts may be used in all flows.

Notification limit

On certain events it may be useful to eg. send a mail notification, however if these messages aren't rate-limited you may end up with huge amounts of notifications. Here are two ways of limiting notifications by time, non of which is better but serves different purposes.

if (true) {
     
// this example sends one mail per hour for every unique function arguments.
     
cache "ttl" => 3600 ]
          
mail("""support@halon.se""Event occurred""An event occurred"); 
if (true) {
     
// this example sends one mail per hour (based on the event1 rate).
     
if (rate("notification""event1"13600))
          
mail("""support@halon.se""Event occurred""An event occurred"); 

Clustered lookup cache

It's possible to query the rate function for the number of entries (implementing a lookup cache), which could be used to implement eg. a SASL authentication cache shared in a cluster.

$ttl 3600;
if (
rate("sasl-cache"$saslusername ":" sha1($saslpassword), 0$ttl))
    
Accept();
if (
$saslusername == "username" and $saslpassword == "secret") {
    
rate("sasl-cache"$saslusername ":" sha1($saslpassword), 1$ttl);
    
Accept();

IP flow

These scripts may be used in IP flows.

Black and white list

// White list: check if $senderip is in any of the networks..
$network = ["10.0.0.0/8""192.168.0.0/16"];
foreach (
$network as $net) {
    if (
in_network($senderip$net)) {
        
Allow();
    }
}

// Black list: check if $senderip is in any of the networks..
$network = ["172.16.0.0/12"];
foreach (
$network as $net) {
    if (
in_network($senderip$net)) {
       
Block();
    }

DNSBL

if (count(dnsbl($senderip,"zen.spamhaus.org")))
   
Block("$senderip blocked by Spamhaus");
if (
count(dnsbl($senderip,"bl.spamcop.net")))
   
Block("$senderip blocked by SpamCop"); 

RCPT TO flow

These examples can be used in the flows executing MAIL FROM/RCPT TO combinations.

External black/whitelist using API

This example shows how to implement a black and whitelist as a web service in PHP and use that data in the SP as a black/whitelist passing the data using JSON.

bwlist.php:

<?php
 $blacklist 
= [];
 
$whitelist = ["john@example.com""example.org"];
 echo 
json_encode(["blacklist" => $blacklist"whitelist" => $whitelist]); 

json data:

{"blacklist":[], "whitelist":["john@example.com", "example.org"]}

rcptflow:

$list json_decode(http("http://api.example.com/bwlist.php", ["timeout" => 10])); // TODO: use cache

if (is_array($list) and is_array($list["whitelist"])) {
    foreach (
$list["whitelist"] as $item) {
        if (
$senderip == $itemAccept();
        if (
$senderdomain == $itemAccept();
        if (
$sender == $itemAccept();
    }
}

if (
is_array($list) and is_array($list["blacklist"])) {
    foreach (
$list["blacklist"] as $item) {
        if (
$senderip == $itemReject("Blacklisted");
        if (
$senderdomain == $itemReject("Blacklisted");
        if (
$sender == $itemReject("Blacklisted");
    }

Greylisting

Greylisting is a technique to prevent custom/non-compliant spam software/bots to send messages on the first try, thus requiring a server implementations with proper queuing.

function greylist($triplet$time) {
    return 
$time;
}

// $sendernet (network/24) can be replaced with $senderip
$sendernet implode("."explode("."$senderip)[0:3]);
$triplet "$sendernet:$sender:$recipient";

// time recommendations from http://en.wikipedia.org/wiki/Greylisting
$windowopen 24;                   // Greylist time (24 minutes)
$windowclose 60 4;              // Greylist time end (4 hours)
$windowtime 60 24 14;         // Skip greylist time (14 days)
    
$slot cache ["size" => 1000000,
        
"ttl" => ($windowclose 60),
        
"argv_filter" => [1] ]
            
greylist(md5($triplet), uptime() + ($windowopen 60));
    
if (
$slot == 0)
    echo 
"Greylist, ok";
else if (
$slot <= uptime()) {
    echo 
"Greylist, pass";
    
cache ["size" => 1000000,
        
"ttl" => ($windowtime 60),
        
"argv_filter" => [1],
        
"force" => true]
            
greylist(md5($triplet), 0);
} else
    
Defer("Greylisted, please try again in ".round($slot-uptime(), 0)." seconds"); 

Outbound rate control based on failed deliveries

One indication of outbound messages being spam, is that other server reject them. Put the following code in the post-delivery (transport) flow

if ($transportid == "mailtransport:2"// whichever is the outbound (lookup-mx) transport
   
if ($errorcode >= 500)
      
rate("delivery-fail"$sender10000003600); 

and the following in the outbound RCPT TO flow:

if (rate("delivery-fail"$sender03600) > 100)
   
Defer("You have more than 100 failed deliveries the last hour, try again later"); 

Concurrency control for smtp_lookup_rcpt

function smtp_lookup_rcpt(...$argv) {
    
barrier "smtp_lookup_rcpt" => $var {
        if (!isset(
$var)) {
            
$var = [];
        }
        if (
is_string($argv[0]))
            
$unique $argv[0];
        else
            
$unique $argv[0]["host"];
        if (!isset(
$var[$unique])) {
            
$var[$unique] = 0;
        }
        if (
$var[$unique] >= 3) {
            if (
$argv[3]["error_code"] == true) {
                return [
                    
"error_code" => 400,
                    
"error_message" => "Too many concurrent lookups"
                
];
            }
            return -
1;
        }
        
$var[$unique] += 1;
    }
    
$result builtin smtp_lookup_rcpt(...$argv);
    
barrier "smtp_lookup_rcpt" => $var {
        
$var[$unique] -= 1;
    }
    return 
$result;

DATA flow

These scripts may be used in DATA flows; processing the actual mail message.

Distribution list

This example shows how to create a simple mail distribution list with JSON API integration. Add a custom/fictional domain (eg. notify). Add a custom flow

<?php
if ($_GET['key'] === 'secretkey') {
    
$mail = array();
    
$mail[] = 'john@hotmail.com'// fetch from database
    
$mail[] = 'john.doe@hotmail.com'// fetch from database
    
die(json_encode($mail));
if ($senderip == "192.168.0.2" and $sender == "john@example.com") {
    
$mail json_decode(http("https://api.example.com/notifications/?key=secretkey"));
    
SetSender("support@example.com");
    
SetDelayedDeliver(300); // "whops"-protection
    
foreach ($mail as $e) {
        
DiscardMailDataChanges();
        
DelHeader("Received");
        
SetHeader("From""Example company <support@example.com>");
        
SetHeader("To"string($e));
        
CopyMail(string($e));
    }
    
Reject("Message sent to ".count($mail)." users!");
}
Reject("Not allowed"); 

Prevent getting blacklisted by using a dedicated "spam" source IP

  1. Add an extra IP address (for each cluster node)
  2. Add an outbound (lookup-mx) transport with that source address
  3. Add the script below to the outbound DATA flow
if (ScanRPD() > or ScanSA() > 2)
    
SetMailTransport(“mailtransport:X”); // with X being the transport added above 

DSN bypass (rate)

Deliver (possible without scanning) DSN messages (in this example, if receiving more than 10 DSN / 5 minutes, only do a Deliver). Good if you do regularly large send out and are expecting bounces.

if ($sender == "" and rate("dsn-bypass"$recipient10300) == false) {
    echo 
"DSN bypass";
    
Deliver();

Manually review high-volume email

Place this script in the very end of the DATA flow, to delay the 20th+ delivery (during one hour) of messages with identical subject lines. These can be listed on the Activity > Rate limit page, and reviewed by searching for them using the filter

rawsubject~"TEXT_FROM_RATE_PAGE"

on the Activity > Tracking page. In some occasions, this can be spam, and thus deleted.

function is_news($senderdomain) {
    
$news = [
        
"epmail.se",
        
"anpdm.com",
        
"carmamail.com",
        
"exacttarget.com",
        
"anppub1.com",
        
"clickredirect.com"
    
];
    foreach (
$news as $domain)
        if (
is_subdomain($senderdomain$domain))
            return 
true;
    return 
false;
}

$subject GetHeader("subject"false);
if (
$subject and
    !
cache ["ttl" => 3600"size" => 20000is_news($senderdomain) and 
    
rate("delayed-review"$subject[0:40], 20003600) and
    
rate("delayed-review"$subject[0:40], 03600) > 20) {
    if (
rate("delayed-review-report""junk"13600))
        
mail("""review@example.com""Review delayed messages""Please do it");
    
SetDelayedDeliver(3600);

Check total process time of a message

executiontime() only tells the current recipients process time, to check total process time of a message (for all recipients). The following script may be used.

$beginprocess cache ["per_message" => trueuptime();
// do processing...
echo (uptime() - $beginprocess) . " total process time"

Check if sender domain exists

Check if Sender Domain exists; if not delete the mail.

if ($senderdomain != "" and !dns($senderdomain) and !dnsmx($senderdomain))
    
Reject("$senderdomain doesn't exist"); 

White list sender domain

This is how a white list could be written. Below follows two examples, the first using string compare and second using regular expressions.

$whitelist = ["halonsecurity.com""halon.se""example.org"];
if (
in_array($senderdomain$whitelist))
   
Deliver(); 

Example using regular expressions against $sender.

$whitelist = ["@halonsecurity\\.com$""@halon\\.se$""@example\\.org$"];
foreach (
$whitelist as $host)
   if (
$sender =~ $host)
      
Deliver(); 

Be aware of that you have to escape the dots in the host with \\., or else the dot will match any character.

Add headers

AddHeader("X-Halon-ID"$messageid);
AddHeader("X-Halon-RPD"ScanRPD());
AddHeader("X-Halon-RefID"ScanRPD(["refid"=>true])); 

Limit number of recipient addresses allowed per domain

We generally recommend using the http() function to query external data sources when implementing business logic such as this, but in case you want to design a pure HSL script which limits the number of users to scan on a domain basis, the following script should work.

$maxusers in_file($recipientdomain"file:1");
if (
$maxusers)
    
$maxusers $maxusers[1];
else
    
$maxusers 5;
echo 
$maxusers;

barrier "user-count" => $var {
    if (!isset(
$var))
        
$var = [];
    if (!isset(
$var[$recipientdomain]))
        
$var[$recipientdomain] = [];
    
$mbox explode("@"$recipient)[0];
    echo 
$mbox;
    if (!
in_array($mbox$var[$recipientdomain]))
        
$var[$recipientdomain][] = $mbox;
    if (
count($var[$recipientdomain]) > number($maxusers))
        echo 
"Exceeding user count";
    echo 
$var;

Support/collaboration script

This script makes sure support or sales addresses always reaches everyone in the distribution group.

// Inbound
function DoInCC($address) {
    global 
$recipient;
    global 
$messageid;
    if (
$recipient == $address and GetHeader("References") == "")
        
SetHeader("References""<$messageid-cc-$address>");
}
DoInCC("support@example.com");
DoInCC("support@example.org");

// Outbound
function DoOutCC($address$name$transport) {
    if (
GetHeader("References") =~ "-cc-$address>") {
        
SetHeader("from""$name  <$address>");
        
SetSender($address);
           
CopyMail($address$transport);
    }
}
DoOutCC("support@example.com""Example Sales""mailtransport:X");
DoOutCC("support@exampe.org""Example Sales""mailtransport:X"); 

Verify sender "HELO" hostname

This example will check if the senders HELO message does resolve back to his IP.

if (!in_array($senderipdns($senderhelo)))
   echo 
"Provided HELO message does not resolve to sender IP, this is suspicious"

Log all mail messages sent out not during office hours

Since you don't expect anyone to send out mail during the night, you may want to explicit log all sent mail. This of course only makes sense when using the flow for outgoing traffic.

// Code executed between 17:00 and 6:59
$time number(strftime("%H"));
if (
$time 16 or $time 7)
    echo 
"$senderip tried to send a mail to $recipient (from $sender)"

DSN Spam Protection

Delete the message if it is a DSN (delivery status notification) but not sent directly from your outgoing mail server (*.example.com).

$dsn GetDSN();
if (
$dsn["route"][1] =~ "\\.example\\.com") {
  echo 
"DSN Spam; Deleted Message from $sender to $recipient";
  
Delete();