How to back up and archive your tweets (and why you should)

by Laurie Anstis on May 9, 2011

Why would anyone want to back up their tweets? For me, there are a number of reasons:

1. Search

Twitter search function is getting better, but I still find it very difficult or impossible to search through my own tweets and timeline. Questions such as “what was the music @legalbizzle recommended a month ago?” or even “where was the Law Society’s guidance note on compromise agreements and the Equality Act” are impossible to answer using Twitter’s own search function, but relatively easy to find out when I have my own local copy of my tweets and timeline to search.

2. The “fail whale

Twitter’s notorious fail whale is less common these days, but it still makes the occasional appearance – most recently this afternoon – and is a reminder that Twitter’s infrastructure is not infallible. If Twitter does suffer a major attack or loss of data, I would like to know that many months worth of useful tweets are not lost.

3. Regulation and other professional concerns

In case of any trouble it would be good to have an authoritative local backup/archive of what has been said. This is particularly so in a professional context where there are multiple people with access to a particular account, and multiple accounts.

In my professional life I look after 12 Twitter accounts, with around 18 people having access to the accounts. This is not unusual when tweeting in a business context, and makes it valuable to have a local copy of what was said when.

What I am looking for

Any backup or archiving service has to automatically and regularly backup tweets. It should also backup any mentions or replies.

It has to be reliable, and ideally be held locally under my control, rather than in a cloud service. If it is not held locally, it must have the ability for the backup to be exported. A bonus would be some form of analytics, showing how many times something has been retweeted, or keeping tabs on the number of followers.

I am willing to pay for the service, but not much.

How?

I have been through three services so far:

1. Backupify

One of the original cloud backup services, Backupify can back up many different cloud services, including Facebook, Gmail and Twitter. I only use it for Twitter. Its free version will back up five Twitter accounts, weekly. Any more than that needs a paid plan, which start at $4.99/month. Data can be exported as a .pdf.

I have found Backupify to be thoroughly reliable, but weekly is not often enough for a backup. The major issue (which is not remedied with a paid account) is that the data is stored in a format that makes it very difficult to make any sense of. The backup is essentially of raw Twitter data, meaning that screen shows a Twitter code, “Node Type Display” and the time it was updated. Technically it is all getting backed up, but in a way that it almost meaningless. The .pdf export is little better. That is more readable, but the output does not seem to be in any sort of sensible order.

Backupify scores highly for its reliability (I have used it for a year or so without interruption) but the output is not user-friendly.

2. ThinkUp

ThinkUp is a web app with enormous potential. Started by Gina Trapani, it is still in beta format, and requires some technical knowledge to get it up and running. It runs using the LAMP (Linux, Apache, MySql, PHP) architecture used by much of the web, and has to be hosted on your own web server. You cannot just sign up online and use it. Running it will require a paid web hosting account, probably costing at least £5/month. Anyone who has installed their own WordPress service will be familiar with how this kind of thing works, and most people will be able to get it up and running with a free afternoon or evening and lots of coffee. If acronyms like FTP and SSH worry you, then it is best to steer clear, but otherwise installing can work fine.

ThinkUp has many virtues. It will store all your tweets, mentions, retweets of your tweets, and provides useful data on followers. Tweets and mentions can be searched. It is designed for large-scale application (there is a public example running on the White House account). Everything is stored under your control under your own web hosting account. It will run automatically at intervals of 20 minutes if you subscribe to an RSS feed, or as often as you want by using a cron job.

In many respects ThinkUp is ideal … but it does require some technical knowledge to operate. I have enough knowledge to operate it, but I can’t get the cron jobs to work, meaning that I have to clog up my RSS reader in order to get it to update automatically. For reasons that I don’t understand it does not seem to be 100% reliable in picking up on tweets (although it is comprehensive in picking up mentions). There is a log that runs every time it does an update, but I can’t understand or make sense of the log.

The most significant problem is that it ends up creating enormous MySQL databases. Most consumer-level web hosting accounts have limits around 150Mb on the size of the database. Running my twelve accounts I came to that limit within a couple of months. Given that one of Twitter’s virtues is its low bandwidth and memory requirements, this is surprising. It appears that ThinkUp stretches to include followers of your followers, which may explain why it takes up so much memory.  I can’t see why it would want to do this, but this memory issue limits its utility in a situation where there is not access to a costly dedicated web hosting account with unlimited memory.

I have split the twelve accounts across four different installations of ThinkUp, but even with that it is possible to see the upper memory limits being reached in the near future. Web host 1and1 have recently upgraded their MySQL service to 1Gb, which is much more suitable for this kind of installation.

3. Tweetstream

Tweetstream is another cloud-based web app which will backup your tweets across multiple accounts, and also collect any mentions and replies. It will keep track of followers, and has a simple, clean interface and full search facilities. A free account (not presently available) will back up three Twitter accounts weekly. A paid account (at the reasonable price of $12/year) will back up unlimited accounts daily. Data can be exported in various ways, including via .pdf.

This has a much clearer interface than Backupify, and the premium account is a much better price than Backupify. Dropbox integration is promised, which would be a major step forward, enabling you to keep a copy locally on your Dropbox account as well as in the cloud. This could be a winning feature if and when it is implemented.

My reservation about Tweetstream is that it does not seem to be robust. It is a small-scale start-up that seems to have been struggling with demand. Logging in takes a long time. The recent migration from Amazon’s storage services does not seem to have helped, and despite promising daily backups to premium customers, as at the time of writing (9 May) my last backup was 5 May. I would like to think that Tweetstream will come good, but for now I do not have 100% confidence in it.

Recommendations

For those who are tech-savvy, I would recommend ThinkUp every time, particularly if you can overcome the memory problems. For others, I would wait to see how Tweetstream settles down.

There are many other services available for backing up and archiving tweets, including Tweet Library, which does all of this on an iPad, but these are the ones that I have had experience of.

Do any commenters have particular recommendations or favourites?

4 comments

I’ve recently used Tweetbook to get my feed converted into a .pdf document. Just go to tweetbook.in and allow it to connect to your account and it’ll do everything for you, including a title page with your avatar picture on it!

by SmartUK on 14 May 2011 at 12:01 pm. #

Hi Laurie,

Great article. You are not alone in wanting a backup of tweets. We’ve spotted this as a key need, particularly as more regulated industries start to experiment with social media. Right now we already archive conversations you have with your contacts, but plan to add the full archive capability you are looking for in the near future. (Your list of requirements re. analytics etc. have been duly noted too!)

Do contact me if you have any more requirements you’d like to share. The more data I have the better. I am mark [at] Connectegrity [dot] com.

p.s. If you haven’t already, you can sign-up for our Beta at http://www.cubesocial.com.

Mark.

by Mark Bower on 13 June 2011 at 5:37 pm. #

Thanks Mark – yes – fully signed up to the beta.

by lanstis on 13 June 2011 at 6:01 pm. #

We are backing up our data with a company called SysCloudSoft that offers on premise backup at just $399. You can access all your data offline and view emails in Outlook/Thunderbird. They also offer a free Google Apps backup for all personal users. I backed up  Gmail,Google Docs,Contacts,Sites and my Google Calendar – http://www.syscloudsoft.com

by Judy D'cruz on 22 February 2012 at 10:04 am. #