Making Apple Mail, Gmail, IMAP and AppleScript work together

If you’ve tried AppleScripting Mail (e.g. to archive mail messages) and you use Gmail, you’ve probably run into issues. The reason is that Gmail by default does some things that are non-IMAPpy.

Apple’s been trying to work nice with Gmail as of Lion (something I’m not sure is a good idea, but that’s beside the point).

Here’s how to make Gmail work like a normal IMAP service, which also makes Mail behave like a normal IMAP client (and makes your AppleScripts work as you’d expect).

  • Gmail > Settings > Labels
    • Turn off “Show in IMAP” for:
      • Important
      • Chats
      • All Mail

Quit and restart Mail (or Mailbox > Synchronize > your Gmail accounts).  It’ll churn for a while, and your gmail accounts should disappear from the “Archive”  mailbox on the left sidebar in Mail.

This simple change to your settings makes Gmail act like a normal IMAP server.  The basic premise is that you’re turning off labels that are subsets or duplicates of other mailboxes (well, and Chats, which is just something you don’t want showing up in mail unless you want all your google chats showing up as email messages).  IMAP requires that a mail message be in one, and only one, folder.  Gmail’s “labels” apply zero or more labels to a message (zero means it’s “archived”, and still shows up in “all mail”). This makes IMAP messy.  The fix above cleans up that mess.

If you’re curious about details, and about how to archive mail using AppleScript, read on.

All Mail is the real problem here: it’s every message you ever sent (and, depending on your settings, every message you ever deleted).  If you’re using AppleScript, say to archive messages, then “mailbox of message …” will always be “All Mail”.  If you “move theMessage to theArchiveMailbox” (assuming “theMessage” is a message and “theArchiveMailbox” is a mailbox), Mail will misbehave and add a label for theArchiveMailbox, while still leaving the old label.  So, it’ll appear in your inbox (for example) AND your archive mailbox (in Gmail, you’ll see it has 2 labels).  This isn’t really Apple’s fault (except that they’re pandering to consumer’s use of Gmail): since IMAP by design puts a message in a folder, and only one folder, “mailbox of message” can only return one folder.  If you’re using Gmail though, which “folder” is the message in if you can only return one folder?  “All Mail” is the logical answer.

So, if you stop Gmail from being a bad IMAP citizen by turning off “All Mail”, Mail can (and does) behave like a normal IMAP client again. “mailbox of message” will properly show the mailbox the message is in again.

So then if you want to archive your mail message from AppleScript, you do this:

tell application "Mail"
    set theMessage to selection --- Get the selected messages
    set accountName to name of account of mailbox of theMessage --- To find an archive mailbox.
    --- This gets a mailbox that contains "Archive".  Uses "last" so you could have
    --- "Archive 2014-02" and "Archive 2014-03" and it'd use "Archive 2014-03".
    set theArchiveMailbox to last item of (mailboxes of account accountName whose name contains "Archive")
    set theMessageId to message id of theMessage --- So we can find it again once it's moved.
    move theMessage to theArchiveMailbox --- Moves the message to the archive
end tell

Notice that line where we get the message id? That’s because theMessage is a reference to a message in an IMAP mailbox. Well, you just moved that message to a new mailbox. But, your reference still points to the old mailbox (e.g. ‘message 123 of mailbox “INBOX” of account “[email protected]” ‘). The “message id of theMessage” however is unique across all messages (everywhere in the world). So if you want to find your message in the archive (e.g. to mark it read, prove it’s moved, etc) you can do this:

set movedMessage to item 1 of (messages of theArchiveMailbox whose message id is theMessageId)

If your message is there, movedMessage will contain a reference to it. If it’s not, that’ll generate an error (because you’ll be referencing item 1 of an empty list).

btw, don’t confuse “id of theMessage” (a number that has limited meaning only within Mail) with “message id of theMessage” (which is the unique message ID that the RFCs require all email messages to have).  We use the “message id” here to uniquely identify and find the moved message.

Best practices: handling a line of customers

Maybe it’s just years of computer experience, but I’m always surprised that only a few places handle lines correctly. Even McDonald’s, one of the most efficient food places, doesn’t do it well (although I guess their approach is to try to eliminate the line, which is admirable).

Fry’s electronics, on the other hand, does it beautifully:

  • Have a single line
  • Have multiple registers (or tellers, or customs agents, TSA scanners, etc). I’ll call them “processors” from now on, as they perform whatever process people are lining up for.
  • The person at the front of the line goes to the next available processor.
  • Don’t split the line at any point: there should never be a line for any given processor. The line should be for the entire group of processors.  The reason for this is that if a single customer takes a long time and hold up a processor, it doesn’t hold up the rest of the line, or a small segment of a line.  If you let a small line occur for each processor, a slow customer will hold up that line – unnecessarily.  Don’t split the line.

This approach is fast, fair, simple to implement and simple for customers to understand.  It also lets your customer get out (and pay) faster, decreases processing costs, and gives you an opportunity to form the line with walls of candy and other impulse buys like Rite Aid and Fry’s do.

Places that do it well (if not fast):

  • DMV (scary, huh – uses a ticket system to generate a line, at least in Santa Clara)
  • Social Security Administration (also uses a ticket system, in Mountain View at least)
  • Santa Clara County Recorder (ticket system)
  • Fry’s Electronics
  • Most airport ticket/baggage check lines
  • Rite Aid (usually via a makeshift line)

Places that fail:

  • Grocery Stores – almost always, and horribly
  • Target
  • Walmart
  • Many customs lines (they fail by splitting the lines near the end)
  • Many airport security lines (they split the line at some point near the end)
  • Some fast food restaurants (e.g. McDonald’s)
  • Most airport security lines, although you need a little line for each conveyer/scanner by necessity due to the length of the screening process.

How to write rock-solid code

There are 4 simple things I’ve learned over the years (and try to do…) to write solid code:

  • Write modular code
  • Document as you write
  • Have each code chunk complete its task or throw an exception
  • Write unit tests as you debug

These are compiled here largely for my re-reference, but also because I was just discussing it with a friend (also a software developer).  Now to the details, which include a really handy technique I’ve picked up for manual testing.

  • Write modular code.
    • Each piece of code should be short and have a well-defined purpose.  It takes specific inputs (and doesn’t rely on global variables unless appropriate), performs a specific function, and returns specific values.  If you have more than a pageful of code in one method, for example, you need to split it into more methods.
  • Document as you write.
    • Every class, method, handler, subroutine, etc should have documentation appropriate for the language you’re using (e.g. POD for Perl).  If you don’t know what that is, go Google “standard documentation format for <insert_language_here>” right now and learn it.
    • Document:
      • What your code-chunk does in a one-sentence summary.  Your user is looking for modules to use and should be able to read that line in a list and pick the module they need.
      • What parameters (input) does it take?
      • What does it do?
      • What does it return?
    • In a Wiki or similar (I use Google Docs for personal stuff), document the system itself.  If you get to more than 5 classes, you’re going to start forgetting what calls what and where your data flows.  Document data flows and sequence diagrams at least.
  • Have each code chunk complete its task or throw an exception.
    • Each method/handler/subroutine should either complete what it’s asked to do or throw an exception.  As you’re coding, you can start with a general exception at first, then as time passes and you get specific errors happening, you can update the code to throw specific exceptions and possibly catch and deal with them.  This makes several things possible:
      • Calling code can just call the method, not have to check obscure return values.
        • Makes calling code cleaner:
          • getWidget();
          • moveWidget();
          • deleteWidget();
          • Instead of:
          • if getWidget() = -1 then { dealWithBadWidget(); }, etc.
    • Validate inputs to the method. Throw an exception if the input’s invalid. Assuming your caller may give you bad values helps prevent obscure bugs.  Note here that distinguishing between user input vs programmatic input is handy: e.g. input’s the wrong class is a program problem, input’s got a semicolon in “fist name” is a user error.  Throw a different type of exception for each – program problems should alert the developer, user errors should alert the user.
  • Write unit tests as you debug.
    • You’re going to write code to debug anyway, just make them into unit tests.  Follow the format appropriate for the environment/language you’re writing in. Don’t worry about making super-comprehensive tests, just make sure you have (safe) tests you can run that test the things you’re working on.
    • Unit Testing Frameworks:
      Perl: Test::More, but really see “man perlnewmod”
      PHP: PHPUnit
      JavaScript: QUnit
      Java: JUnit
      Python: Unittest
    • If it’s not practical to write automated tests (e.g. for a user flow on a web site, although see Selenium), make a QA script (AKA QA test procedure):
      • Make a 2-column spreadsheet (Google Docs is great for this).  Column one is “Step”, column 2 is “Result”.  I like to keep this really simple and have the result be “PASS” or “FAIL”.  If there are any notes about the failure, I add them as a note attached to the cell.
      • In the “Step” column, list the steps to test the feature, one step and result per line.  Make sure each step and test can pass or fail.  So a test for Google search might look like: “Go to www.google.com.  Page loads and displays a search box.”, “Type ‘pizza pie’ in the search box, hit return. Search results display.”, “Top 3 search results contain the words ‘pizza’ and ‘pie’ in the title.”, “Ads display on the top and right side of the page.” etc.
      • You can also use conditional formatting to make the backgrounds of the Results cells red or green so you get a really clear-at-a-glance picture of whether your code works or not.
      • After you make significant changes, before you push to production, run through the script yourself or have an intern or assistant do it.
      • Tips:
        • Split the tests into multiple tabs (or spreadsheet docs if appropriate) based on feature (think separate unit tests), and name the tabs/docs by feature.
        • Keep the scripts short enough to quickly test a feature, and as long as necessary to include all steps needed to test that feature (e.g. if you’re testing a shopping cart’s order confirmation page, your test will need to include adding items to the cart, signing in and/or registering, entering test payment info, and confirming your order).
        • Write the steps very clearly.  Pretend you’re writing a program.  A person with no technical knowledge should be able to follow your steps without thinking and mark them “PASS” or “FAIL”.  e.g. say “go to http://www.google.com, verify a search box displays” and not “load the web site” (which web site?), or “see if it loads correctly” (what’s “correct”? If I’m testing, I may think the error message is “correct” because it’s grammar is accurate).
        • “PASS” or “FAIL” makes debugging easy.  If your steps and verifications can’t pass or fail, break them into smaller steps or separate test sheets.  Your automated unit tests don’t leave vague notes – your manual ones shouldn’t either.

There.  Solid code in 4 steps.

My 3 top productivity tools

GTD helps organize my life, goals, projects, and actions and determine where and when things get done.  4-hour workweek helps with concepts like eliminating useless tasks, focusing on what’s important, work/life balance, and getting things done quickly.  Pomodoro technique helps focus on the task at hand, finish it quickly, and stave off interruptions (almost any interruption can wait 25 minutes or less, and usually doesn’t need to be done once that 25 minutes is up).

See, kept it short so you can go be productive.

Best navigation app for iPhone: Apple Maps

Sorry, I know it’s more popular to bash on Apple Maps, but it’s just become my everyday Navigation app.  Here’s why.

I need an app that has:

  • Accurate maps so that it can guide me when I’m somewhere I don’t know
  • Good traffic routing – I need to know the real fastest route, and in California that includes other cars in my way, offramp closures, road closures, carmageddon, etc.
  • A good UI

My go-to app before iOS 6 was MotionX Drive.  I’d also tried Waze, Garmin, and a few free apps I don’t remember.  I didn’t get to try the TomTom app.

I’ve now switched to Apple Maps, because:

  • Despite a lot of bad press, I haven’t had a single significant problem with map inaccuracy.
  • The navigation screen is clean, clear, and easy to understand when driving.
  • It routes around traffic, and does it intelligently with decently accurate time estimates.
    • Waze gives really good time estimates, but doesn’t know about ramp closures.  Also, its maps are too unreliable.  I can’t tell if it’s routing me a weird way because of traffic or because it thinks one freeway doesn’t connect to another.
    • MotionX routes around traffic very well, but its estimates on surface streets are overly optimistic.  As such, it’ll take you onto surface streets when the freeway with traffic might be faster, and its estimates are far too optimistic to be useful.
    • Apple Maps has decent traffic information (although it appears to be delayed by about 15-20 minutes, which led to me sitting in an accident-caused traffic jam last night that Waze had immediately).  It routes around bad traffic, but doesn’t send me onto semi-crowded side streets when there’s a section of red on the freeway.
  • The voice prompts are incredibly well-timed.  I haven’t seen any other system that’s as well timed as Apple Maps.
  • Re-routing is instantaneous when I miss a turn.
  • I can tell Siri “Give me directions to Weinerschnitzel”, and it’ll find the closest one, switch to maps, and start giving directions.

I’m willing to pay for a good nav app.  Apple Maps is free, and is only version 1.0.

P.S.:

Garmin was really nice, but routed me straight through heavy traffic with an option to detour.  That made it a FAIL for me – of course I want to detour!

TomTom was too expensive for me to try, and the write-up didn’t convince me that it met the criteria above.