Yesterday was also the day that another RSS based news app, Pulse, was removed from the store despite being praised & shown on stage at WWDC by Steve Jobs. Why? Because the New York Times complained that the app contained the URL for their RSS feed. Quoting from the letter Apple received from the Times:

I note that the app is delivered with the NYTimes.com RSS feed preloaded, which is prominently featured in the screen shots used to sell the app on iTunes.

The same argument was made by Apple to me for the recent rejection of an update to iNewz (and a few more news feeds were cited as problematic too).

Is It Copyright Violation?

From my discussion with Apple’s developer relations last night, it seems that these organizations are happy to have their readers use a commercial RSS reader where they must type in the RSS feed URL, but they are not happy to have the URLs pre-loaded for the user to select from. Aside from the obvious stupidity of making it harder for people to find your content, the only difference between these two app designs is that the actual RSS feed URL is embedded in one. Does that mean that the New York Times considers its actual URLs to be copyright material? Can you even claim that a URL is copyrighted? Seems like a stretch.

The content itself is clearly copyright protected, but since all RSS readers present this in roughly the same way, to complain about ones with pre-loaded feed URLs but be happy about ones where the feed URLs are entered by end users implies that it is not the content itself that is the problem. It will be the same in both cases.

Or are they just worried about the apps using the New York Times name (and perhaps some content via screenshots) in their marketing materials? If so, why not just ask the developers not to use their name and/or content in the app store copy?

More Readers Is Better, No?

One assumes that these companies include RSS feeds for their content in order to encourage more readers to visit their website. If they don’t want people to use RSS feeds, why have them? The feeds are published by the news organisations, and they control the amount of content inside them. Feeds are essentially a promotional vehicle. By including their feed URLs, a news reader application is essentially providing them with free advertising. Other industries actually pay for this type of referral!

Having more applications out there that promote your content, as long as it delivers the users to the content when they want to read more, would seem to be only beneficial. Unless they don’t really want people visiting their website. If they think that making it harder to read the news online is going to drive people back to their paper products, or force them to subscribe to an in-house electronic version, they are likely to be disappointed. And if they really do want to do that, then why have the RSS feeds at all?

Sharing

Even more frustrating for me, not only did the iNewz apps have the potential to bring new readers to a whole selection of publications they might not otherwise have even known about, it went further and allowed them to share articles with their followers on Twitter. And from there, out to even more people via the @iNewz Twitter feed that re-tweeted any article shared using an iNewz application.

That’s even more free promotion for the news organisations, and even more eyeballs on their content web pages.

Revenue

The claim that these news apps, because they are not free apps, are commercial use of the feeds is also missing the point. The app developer gets a one time payment of a few dollars for their work in writing the software and marketing it. There is no cost to the news organisations whose feeds are being included for that development work, for the marketing, nor for any updates necessary to keep the app working or improve the user experience. Essentially, as long as the app drives users to their site to read the full story always, they are getting free promotion of their content.

And more readers on their website, means more ad revenues. And those are ongoing revenues, not one-time micro-payments. They should be considering rev-share deals for that revenue with the folks writing news apps, not banning us from helping them boost their circulation!

The Rejection
Like most app developers, I’ve had rejection emails from Apple before (actually called “feedback”), and they have normally been for things that are simple to address, even if not always things I agree with the need for.

This one was different though. Here’s the key paragraph of the email:

Thank you for submitting iNewz to the App Store. We’ve reviewed iNewz and determined that we cannot post your application because it appears to contain features, namely, feeds, that bear a resemblance to well-known third-party marks, specifically well known news organizations including The New York TImes, The Washington Post, USA Today, and BBC.

Yes, iNewz does include feeds from major news organisations – it is essentially an RSS reader with a set of pre-configured feed URLs making it easier for people to select the news sources they are interested in reading content from.

Clarification
Not being entirely sure about the implication of the wording in that paragraph, I asked for clarification. Here’s what I received back:

Thank you for your response and prompt attention to the Trademark issue. While we cannot provide legal guidance on the specifics of copyrights, The New York TImes, The Washington Post, USA Today, and BBC have all previously objected to other applications that include preset feeds to their content, and believes that such features infringe their rights.

So, I guess that means iNewz cannot include at least those feeds, and by extension I am sure they will assume all other similar news organisations could feel the same way. I also see no reason why they would feel differently about the other iNewz applications, even though the feeds are from smaller news organisations and blogs.

The Inconsistency
To make it worse, iNewz has been on the app store for over a year, and had mostly the same set of feeds for that time. It has been through the review process numerous times (several times in the last month or so), and certainly makes no attempt to hide the fact that it gets the news from the feeds of major news organisations.

The Irony
To further emphasize just how random the Apple review process is, on the very same day, just a few hours after the initial rejection email, I received a second email stating that the same update (a fix for a Twitter OAuth problem) to iNewz for iPad had been approved.

The Conclusion
There were really two ways out of this dilemma:

1. I can try to find a way to get the application(s) approved;
2. I can pull the applications from the store.

This application was never really a commercially viable project; it has not even come close to paying for its development in hours. It was more a labour of love – I wanted a news application for my commute, and I thought other people might enjoy it too.

But looking back over the reviews in the app store, I realise that was perhaps a mistake. And the update it appears is disliked by almost all those who have written reviews so far.

Given that, I think I have no real choice to make here. Sadly, I think iNewz, in all its forms, will have to be removed from the app store soon.

Signing Up
First step in converting is to sign in to Twitter’s new developer site and register your app as an OAuth application. Once you have done that you need to send an email to api@twitter.com requesting permission to xAuth. Include in that email the client ID (the number at the end of the http://dev.twitter.com/apps/ URL once it’s registered as an OAuth app), and a link to your application’s website.

Until Twitter has approved your application for xAuth, you will not be able to use the xAuth protocol.

Once you register for OAuth though you will immediately have a consumer key and secret for your application. Make a note of these – you will need them for all the steps below!

xAuth Background
A useful starting point here is to get an understanding of how xAuth works. Some of this applies to OAuth as well, since after the initial steps both are pretty much the same thing.

There are two stages to xAuth/OAuth that a developer needs to think about:

1. Authorisation – the initial request to get permission to access the user’s Twitter account. Your application’s registration with Twitter will determine whether or not this is read-only or read/write access.
2. Identification – all subsequent requests using the permission must include something that proves to Twitter that you have permission.

In the basic auth model, the two were basically one step, achieved by sending the user’s username & password in each request. Now they’re two steps.

The authorisation step returns a token and a matching secret that you need to use for the second step. But unlike basic auth, you don’t pass them both back on each request, but instead use them to generate a signature. Generating that signature is something that the Twitter docs don’t help much with, so we’ll look at it in more detail later (to add to the fun, even the authorisation step requires a signature, but one that is generated slightly differently).

Authorisation
Unlike OAuth, where the authorisation process is a multi-step affair that involves sending the user to a Twitter web page, xAuth is a single request. That request is to the OAuth API endpoint that would normally be the last one in the OAuth sequence, but it includes some extra parameters.

The endpoint is https://api.twitter.com/oauth/access_token. To request your access token and secret on behalf of a user, you will need to POST a request to that URL with the following parameters:

• x_auth_username – the user’s Twitter username
• x_auth_password – the user’s Twitter password

x_auth_mode – always set to client_auth

Remember to URL encode the username and password values.

But you can’t just post those values; you also need to sign this request and add an HTTP Authorization header to your POST.

Signing The Access Token Request
First step in the signing process is to generate the base string. The base string contains all the required parameters for the request and is used as one of the inputs to the hashing function that will generate the signature.

This is one of the parts of OAuth that I feel is poorly designed, but it is what it is, so we need to jump through the necessary hoops to get the signature. The key to remember here is that Twitter is going to reconstruct this string and use that to verify your signature value; if anything is different at all, then the signatures will not match, and the request will be refused.

There are three parts to the base string, joined by ampersands (&) as follows:

method&url&parameters

For this request, method will be POST in upper case always. The URL is the URL encoded version of the access token endpoint URL:

https%3A%2F%2Fapi.twitter.com%2Foauth%2Faccess_token

The hard part is the parameter values (and this is where I think OAuth shows its first sign of poor design). The parameters must be added to the base string in strict order: alphabetic order of the parameter names for this request (requests where a parameter may be repeated must further sort those in order of the values).

The parameters you will need to get an access token, in the order they must be added to the base string, are:

• oauth_consumer_key – your application’s consumer key.
• oauth_nonce – a nonce (number used once) generated by your application (we’ll look at this more in a minute).
• oauth_signature_method – this is just set to HMAC-SHA1 (the Twitter docs say HMAC_SHA1 in some places, but that is wrong).
• oauth_timestamp – the current time in seconds since Jan 1, 1970 00:00:00 GMT. There are posts online claiming that Twitter violates the OAuth spec here by checking that this is within 5 minutes of current time. That means your app will fail on devices where the time is not set correctly, or not available.
• oauth_version – set this to ‘1.0′ (without the quotes).
• x_auth_mode – set this to ‘client_auth’ (without the quotes).
• x_auth_password – the user’s password.
• x_auth_username – the user’s Twitter username.

All the values should be URL encoded. Straightforward so far. To create a single parameters value though for substitution into our base string’s third component we need to join all that together.

Each parameter is added as key%3Dvalue (where %3D represents the URL encoding for ‘=’). So, using the oauth_signature_method as an example, for that parameter we would generate the string:

oauth_signature_method%3DHMAC-SHA1

Then we must join them all together to form a single parameter list. Do that by separating the individual parameter strings with %26 (the URL encoding for ‘&’). So the start of the list might look like this:

oauth_consumer_key%3DKEY%26oauth_nonce%3DNONCE

Where KEY and NONCE are the URL encoded values for those parameters (too long to include here and make it readable).

The Nonce
The nonce is something that your application should generate randomly ideally. The OAuth specification states:

The Consumer SHALL then generate a Nonce value that is unique for all requests with that timestamp. A nonce is a random string, uniquely generated for each request.

There is a suspicion that Twitter takes a stricter view of these values, but generating a large random number should be sufficient.

The Timestamp
The timestamp should be simple, the number of seconds since January 1, 1970 00:00:00 GMT. All the OAuth specification says about this value is:

The timestamp value MUST be a positive integer and MUST be equal or greater than the timestamp used in previous requests.

There is a belief (though I’ve not seen confirmation or denial from Twitter) that the timestamp must also be within 5 minutes of the current time. As well as being an obvious violation of the specification, it is also very poorly thought out design since it will be a problem for devices where the time is set incorrectly, or not available. Even the requirement in the OAuth specification could be problematic, and is unnecessary.

The Signature
The only algorithm Twitter currently supports is HMAC-SHA1, so this is what we need to use here. The trick (that is not easy to find in Twitter’s docs) is how you encode the ’secret’ for this process.

Firstly though, here’s an example of the code to generate the signature (in Objective-C for the iPhone, but should be easy to port to other platforms):


NSData *dSecret = [secret dataUsingEncoding:NSUTF8StringEncoding];
NSData *dBase = [base dataUsingEncoding:NSUTF8StringEncoding];
uint8_t result[CC_SHA1_DIGEST_LENGTH];
CCHmacContext hmacCtx;
memset(&hmacCtx, 0, sizeof(hmacCtx));
CCHmacInit(&hmacCtx, kCCHmacAlgSHA1, dSecret.bytes, dSecret.length);
CCHmacUpdate(&hmacCtx, dBase.bytes, dBase.length);
CCHmacFinal(&hmacCtx, result);


The secret is normally created by joining the consumer secret for your application (from the Twitter developer site’s page for your application) and the token secret using an ampersand (&):

consumer_secret&token_secret

In this case though, since we are still requesting the token, we don’t have a token secret. So the secret in this case is the consumer secret with a trailing ampersand:

consumer_secret&

Don’t miss the trailing ampersand or it won’t work!

The Authorization Header
Once you have all this information, you can build the string that will become the ‘Authorization’ HTTP header value. That value is OAuth followed by a sequence of comma separated key value pairs, the values being URL encoded and then wrapped in double quotes:

OAuth oauth_nonce="qfQ4", oauth_signature_method="HMAC-SHA1", ...

Unlike the base string values, the order of these values doesn’t seem to matter. Also, only the oauth_ prefixed values are included here:

• oauth_nonce
• oauth_signature_method
• oauth_timestamp
• oauth_consumer_key
• oauth_signature
• oauth_version

The x_auth prefixed values are sent in the body of the POST.

The POST Response
In response to posting this information you will receive one of two things:

• An (annoyingly) URL encoded list of values, including the oauth_token and oauth_token_secret that you will need to store and use on subsequent requests to the Twitter API;
• An error message (which is unlikely to help you much, and will tell a user even less – OAuth was designed by geeks with no idea about user experience it seems);

In the case of an error, you should also get an HTTP error (400 or 401 most likely), so you can tell the difference between a successful request and an error.

You don’t need to store the Twitter username (unless you want it for some other purpose, such as displaying which account the user authorised the application to use), and must not store the Twitter password. Here’s an example response (with the token and secret masked):

oauth_token=XXXX&oauth_token_secret=YYYY&user_id=14642644&
screen_name=bluedonkey&x_auth_expires=0

That will all be on one line in the response from Twitter.

Updating User Status
Updating the user’s status (i.e. posting a tweet into the user’s timeline) is an authenticated method. Using xAuth (or OAuth, since they’re the same), we must sign the request and include the Authorization HTTP header.

The process is the same, but here are some things to be careful of:

• Once you’ve acquired the access token and corresponding secret, remember to use the combined consumer secret and token secret when generating the signature (see above);
• You will also need to add the oauth_token parameter into the list of values in the HTTP Authorization header value.
• When creating the base string, the status value (i.e. the content of the tweet) needs to be included. In the body of the POST it will be URL encoded. In the base string it needs to be double URL encoded. So, ‘Test Tweet’ becomes ‘Test%20Tweet’ for the body of the POST, and it becomes ‘Test%2520Tweet’ for the base string.

Invalid / Used Nonce Error
Another indication that Twitter’s OAuth implementation is not really ready for production use is the simple fact that almost any problem with the contents of the Authorization header results in the inaccurate ‘invalid / used nonce’ error.

In all the examples I found online, as well as my own experiences, the problem is never related to the nonce value. This error seems to highlight the least likely cause of the problem. Possible real causes of this error include:

• Timestamp values that are more than 5 minutes from the current time (check the clock on your system!);
• Using the old ‘+’ encoding for spaces instead of ‘%20′ in values included in the base string (e.g. tweet message content);
• Incorrect/invalid token value;
• Not double encoding the values of POST body parameters that are being included in the base string.

If you see this misleading error, check everything else in your base string too – it is almost certainly not a problem with your nonce as long as you have done something to make sure they’re random!

Wasted Time
The IRS estimates that on average across all people filing any type of federal return, individuals will spend 17.3 hours “preparing” these forms. The IRS reported (pdf) that in 2007 there were almost 139 million individual tax returns filed. So, a quick calculation suggests that every year in the US, 274,509 person-years are wasted on collating and transcribing information that the recipient (the IRS) already had. And that doesn’t include the burden from the state tax filings.

Wasted Money
In addition to the ridiculous amount of time wasted on this activity, the IRS also estimates that it will cost individuals on average $225 to prepare these returns; that’s $31 billion every year (and, again, doesn’t take into account state tax filing costs). That could be spent on things that were a lot more beneficial than collating and transcribing information from one set of forms to another.

Wasted Resources
It doesn’t end with wasted time and money either. Every year, employers, banks and other companies mail out those tax details to all their employees/customers. Some now offer electronic delivery at least, but I wonder how many of those end up being printed at home, either for reference while preparing taxes, for providing to an accountant or just for records?

Estimates online range from 9,000 to 15,000 sheets of paper per tree (obviously, the type & size of the tree plays a big part in this!). Let’s assume the 15,000 sheets per tree number for the sake of this post. That means, assuming each person filing a return received at least 5 of these forms (probably lower than reality based on my experience), 46,000 trees were destroyed to print these forms (and that doesn’t include the envelopes used to mail them – probably close to as much again).

Then there are the tax forms themselves. The IRS reported (in that same data book PDF) that about 79 million of the 139 million returns were filed electronically (a great achievement, by the way). That still leaves 60 million that were mailed in, on paper, though. My federal return from last year was 13 pages. For the sake of round numbers though, let’s say that the average person mailing in a return is 10 pages. That means another 70,000 trees were cut down to send the information to the IRS a second time.

Every year, that means over 100,000 trees are being destroyed every year just to pass this information around multiple times. And that doesn’t include the rest of the carbon footprint of this process (converting trees to paper, the resources used in printing, and the delivery of the forms).

A Better Idea
I don’t expect the US to do the smartest thing and switch to a taxed at source model, where companies who pay people are responsible for deducting the correct amount of tax before sending out the money, any time soon. I just don’t believe it is in the American psyche. Especially not when I hear so many people who are ecstatic to receive a refund from the IRS! They don’t seem to realise that all those refunds mean is that they gave the IRS an interest free loan for the year.

There is another solution though that would make much more sense than the current scheme: just bill me!

Yes, that’s right, the IRS (and the state tax authorities too) have all the information that they need to be able to calculate how much tax I owe, or how much they owe me. They have the resources to process that data, and it is all keyed off of a single identifier. So, why can’t they just send me a statement, and either a bill for what I owe or my refund? That would reduce the time most people spend on taxes to just minutes (the time required to scan the statement, and pay the bill).

The paper waste is also reduced dramatically. Let’s assume an average of 2 pages, printed double sided, per person billed, and 139 million bills mailed out. That’s under 10,000 trees now. Add electronic statements as an option, and no-fee bill pay options, and you also reduce the paper wasted by the process even further – assuming the same proportion of people who e-file their taxes, it would drop to below 5,000 trees per year – a reduction of over 95%.

Tax Reform
How about it President Obama? Democrats? If you want to make a meaningful difference to the tax system in the US, how about reforming the collection mechanism rather than individual taxes? It makes sense from a fiscal perspective; it makes sense from an enforcement perspective (less avoidance); and it makes sense from an environmental perspective. What’s not to like here?

I’ve written a lot of software, both application and system level (right down to the lowest levels of an RTOS), and believe me, if it is written properly a background app does little or no harm to battery life.

Many of the applications that people would like to see running in the background would spend almost all of that time waiting for a system event. That waiting state doesn’t harm your battery life; only when the application is actually processing something does it really consume power. The push mechanism on the iPhone today might actually be worse since it has to load the app each time, a far more expensive operation (in CPU, and therefore battery) than just switching to one that is already “running.”

Consider the IM app example that is so often used to support the claim that background apps kill battery life. Sure, if you run the IM app (background or foreground) and stream messages at it continually, then it will reduce the battery life. If you just have it sitting there in case somebody tries to start a session though it isn’t doing anything most of the time (occasional presence messages perhaps). I ran an IM app all the time on my Nokia N95, connected over AT&T’s network 24/7. My battery life was unaffected, as expected.

Another example of a well behaved background app is the daemon that we wrote for the jailbroken version of Devicescape’s app (before the SDK and app store existed). It made no difference to battery life because it spent almost all the time blocked waiting for a system event. One that only happened when a new Wi-Fi connection was made. We run in the background on Nokia, Windows Mobile and Android (not to mention Windows XP/Vista/7 and Mac OS X) today without impacting battery life.

So what will affect battery life? Well, an app that continues to do something in the background, rather than waiting for an event, one that polls for an event rather than blocking until the OS tells it about the event, or one that requires a power-hungry piece of the hardware to be on all the time (e.g. GPS). But even those apps have their place. Imagine a background image uploader: it will do something in the background while it is needed, and then exit or wait for a new photo to be taken. Or an app that checks your location every 5 minutes. It is my choice to use the battery that way, so why restrict it? Just make sure it is reasonable for the application, explained to the user, and under my control (can be checked as part of the review process).

These types of apps don’t take any more power than they would if I left them running in the foreground, but letting me push them to the background allows me to choose if I want to watch them work, or read my email etc.

Above all, please stop spreading this myth that multitasking or background processes will harm battery life. Only badly written apps would do that.