The Conservative Cat

Every incoming comment or ping in Movable Type is analyzed by three SpamLookup plugins, and each one returns a negative number called the junk score. The scores are combined, and if the result is less than 0, the comment or trackback is placed in the junk folder for disposal. It's important to discuss each one in detail because Movable Type's recommended defaults are totally wrong.

1. Lookup checks the IP address and domain name against known spammer IDs. It also junks trackbacks that appear to be from suspicious sources. This latter test (known as Basil's Bug) almost always generates a false positive, so be sure to turn it OFF in your configuration.
2. Links examines the links in the comment or trackback. It will moderate or junk a comment when it has too many links. Almost all spam contains at least 3 links, so you want to set this to junk a comment with 3 links and turn moderation off. This is also the module that gives people a bonus if their EMAIL address or URL has already been published, so it's possible for this one to come back with a positive score rather than a negative one.
3. Keyword Filter looks in the entire text of the trackback or comment (including the URL, email address, and user name) and penalizes it for each junk word. My list is shown below. The numbers indicate the penalty for each item, the slashes mean that the string is actually a regular expression instead of a word. Inside the slashes, a dot (.) is a wild card character. After the slashes an i indicates a case-insensitive match. So, for example, "cialis" must appear as a word surrounded by punctuation and its junk score is 2. "big boobs", however, is caught no matter where it occurs, and the character between "big" and "boobs" can be anything-- a space, a period, an underscore, or even a hypen.

   cialis 2
   /casino/i 2
   phentermine
   /poker/i 4
   /you may find it/i 2
   /<h/i 2
   /update your site soon/i 3
   favourits 4
   payday loans 3
   foo 2
   /foosh/i 3
   /big.boobs/i 3

   The "/<h/" pattern is particularly important. Google looks for heading tags when it does its ranking. Approximately 80% of the comment spam we get has <h1> tags in it.

Now here's the extremely weird part: the junk score is the average (arithmetic mean) of the non-zero scores from the three modules. So, if a comment contains "poker" it has a score of 4, but if the comment contains poker and comes from a banned domain, it has a score of 2.5, because there are 2 results (4 and 1) and (4+1)/2 is 2.5.. This is why the weights on the keywords are so high.

The module that controls junk filtering is \cgi-bin\mt\plugins\spamlookup\lib\spamlookup.pm. The hack below will cause a duplicate ping to be recognized as a junk trackback. This prevents the ping from appearing on your blog or causing a rebuild. More important, it still counts as a successful ping, so the sender is told he's done and there's no need to send it any more. The junk score returned below is 5, which is pretty high. That enables it to overcome the weirdness of the way the weights are combined.

This code replaces the elsif clause beginning at line 148 of the un-hacked version, which is inside the subroutine link_memory.

            } elsif (UNIVERSAL::isa($obj, 'MT::TBPing')) {
                my $url = $obj->source_url;
                $url =~ s/^\s+|\s+$//gs;
                # Look for a ping with the same URL and target.
                my $t = MT::TBPing->load({ source_url => $url,
                    blog_id => $obj->blog_id,
                    tb_id => $obj->tb_id,
                    visible => 1 });
                if ($t) {
                    # We found one. Give it a junk score of -5.
                    return (-5, "Duplicate of ping " . $t->id . ".");
                } else {
                    # Not a duplicate. Check to see if this link was previously published.
                    # If so, we consider it safer than an unknown link.
                    $t = MT::TBPing->load({ source_url => $url,
                        blog_id => $obj->blog_id,
                        visible => 1});
                    if ($t) {
                        return ((int($config->{priorurl_weight}) || 1),
                            "Link was previously published (TrackBack id " . $t->id . ").");
                    }
                }
            }

Typepad uses the Movable Type software, so some of these hints and tips will apply to Typepad as well. If someone with a Typepad account and spam problems would like to let Bruce help tune your system, send him an EMAIL and he will get in touch with you to work out the details.

Respectfully submitted,

Ferdinand T. Cat