Broadcom BCM4328 wifi on Ubuntu 9.04 Jaunty

Ubuntu now includes support for this wireless chip in its ‘Restricted Driver Manager’. Sadly the activation silently fails on a fresh install of Ubuntu. The solution? Install b43-fwcutter. Then disable and re-enable the driver in Ubuntu’s Restricted Driver Manager and it will work after a restart.

Processing email feedback loops for fun & profit, part 2

Part 1 here.

So we know what a feedback loop is and how to use it to identify customers complaining about our mails. The next step is to automate removal of these people.

It turns out many (most?) feedback loop emails you get from ISPs follow the same format. The message is made up of three sections:

• The first section is a generic message from the feedback loop provider, “this is an email abuse report, yadda yadda yadda”.
• The second section is basically useless. It contains some information from the abuse report, but is usually heavily censored.
• The third section is a copy of the original email you sent. Some information is redacted, like the recipient, and the sender address.

To process this we need two pieces of information:

• Which ISP our feedback loop email comes from.
• Who we sent the original email to.

This is relatively easy with Python, which includes the email library module and will do most of the heavy lifting for you.

The ’standard’ feedback loop format is ARF, and is used by (at least) AOL, Comcast and Yahoo. To process an email in this format:

import email
message = email.message_from_string(foo)
 
messagefrom = message['From'] # This is how you pull headers out.
                              # Note the keys are not case sensitive
 
messagebody = message.get_payload() # This returns three sections, as described above
 
originalmessage = messagebody[2].get_payload()[0] # This returns the original message.
                                       # We need to explicitly get the single payload.
 
originalmessagesender = originalmessage['Return-Path'] # Ta-da! The original sender.

You can then process originalmessagesender as required to retrieve your UID. Once you have that, send a request through to the appropriate interface and you’re done.

Bonus Notes

• Some providers also strip the return path from the original message. We solved this by hand setting the message-id header on all outgoing mail. If you also do that, make sure you set each message-id to a unique value. We use the same string as our sender address, but change the generic ‘@noreply.ourcompany.com’ domain to the real machine involved in sending and also include the output of time.time(), which gives us useful troubleshooting info too.
• The original plan was to process feedback loop messages as they come in by spawning a script for every returned mail. This turned out to be a bad idea because abuse messages seem to come in bursts, making it possible we’d accidentally DOS our own mailserver with tens of python scripts. Now every feedback email will be delivered to a Maildir. We iterate over every message in the Maildir with Python’s mailbox module (which returns an email object as above for each message).
• If your feedback loop is in another format, it’s pretty easy to work out the structure by playing with a sample message in the Python interpreter (IDLE).

3 redheaded songstresses you should listen to

Clare Bowditch

‘You Looked So Good’. Off the acoustic bonus disc of ‘The Moon Looked On‘.

Neko Case

‘The Next Time You Say “Forever”‘. Off ‘Middle Cyclone‘.

Florence & The Machine

‘You Got The Love’. The B-Side off the ‘Dog Days‘ single.

Processing email feedback loops for fun & profit, part 1

Warning: this blog post expects you to know your email envelope recipient from your ‘To’ header address. If you don’t, here’s a good primer.

The company I currently work for sends out periodic mailouts to paying customers of our site. We comply with the CAN-SPAM act and give people a simple way to unsubscribe from our mailings, but some people still mark them as spam. This is a problem, because when people do this from the webmail interface of some ISPs, it records this, and if there are enough spam markings, starts refusing all connections from our mail servers until they process our removal request.

Many of the larger US ISPs allow you to sign up for an email ‘feedback loop’ where they notify you whenever they detect a ’spam’ mail coming from your nominated IPs. We decided to try and use these feedback loops to automatically unsubscribe people from our mailing lists when they mark us as spam. There are two advantages:

• We have a lower chance of getting unilaterally blocked by some ISPs,
• People who don’t want our messages won’t get them.

All we need to do is pass the feedback loop messages to a script, extract the recipient address and send an unsubscribe request to the relevant part of our website. Sounds simple, right? I should be able to write something up using Python and procmail in an hour. Unfortunately life is a little more complex.

The first stumbling block is that most (all?) feedback loops remove some information from their notification emails. Of the two I’ve played with so far (AOL & Comcast), both remove the destination address. They don’t do it sloppily either: every reference to the destination address is replaced with <redacted>. How can you tell who the message was sent to? In our case, we decided to modify our mailouts. Now, the envelope sender address we use in our mailouts includes a UID we can use to identify the sender. Previously our ‘Return-Path’ header looked like this:

Return-Path: <blah@ourdomain.com>

Now it looks like:

Return-Path: <blah+UID@ourdomain.com>

So now the feedback loop emails we get are useful! Next post I’ll describe the simple procedure of parsing feedback loop emails with Python. If you want to skip ahead, here’s a hint.

Part 2 here.

How not to launch your new multiplayer game…

[UPDATE: After the day 1 patch it's actually possible to get a 3v3 game going without too much trouble now. Buyer Beware warning downgraded from 'Avoid' to 'Check it out'. Hopefully in a few more days they will fix some of the other more egregious issues. This is the nice part about Stardock published games -- they really do keep patching them forever.]

…when it feels like a paid beta. Demigod – a DotA clone – was released yesterday. This game is one of many games that is only fun when played against other people. There are bots, but they basically suck. So the multiplayer functionality of this game is critical: if it doesn’t work properly, there are serious problems. Guess what Demigod’s multiplayer support is like so far:

Yes, the multiplayer is completely broken. So broken that one of the game developers has said on the support forums that there’s less than a 50% chance for games with more than two people from even starting properly, let alone playing smoothly.

So what’s the problem? The game went through multiple months of private beta testing, how could it be this bad? Two words: “UDP p2p”. Seemingly introduced in an attempt to combat the evils of lag, so far it appears to be an expensive exercise in not letting people play the game at all, laggy or not. Demigod requires a UDP connection to every other player in your game. If one of these connections doesn’t work, the game doesn’t start. For anyone.

The big problem seems to be how the game handles NAT, or other situations where incoming UDP connections are blocked. How do you handle NAT traversal? Should you try STUN or some other punchthrough technique? Should you try and use UPnP to negotiate this? What happens if the game has two (or more) people who just can’t accept incoming connections? Do you proxy their communication through other players? What about personal firewalls? What about firewalls elsewhere on the network?

The whitepaper written about the backend multiplayer tech suggests NAT traversal is implemented, but so far in game it appears to fail more often than it succeeds, when it really should be succeeding well over 99% of the time for a task like this.

Demigod is a promising game, and I’d really like to play it against other humans. So far though, it’s been an expensive reminder that you should read reviews of games before you buy them.