aws logo

Automated AWS Re-billing


If you manage your clients' Amazon Web Services (AWS) platform, you're faced with the problem of how to bill for the AWS costs.

We typically tell clients to get an AWS account, give us a privileged IAM user and leave us to it. Because we can take it from there. However, sometimes clients want one commercial relationship, or they don't want to pay via a credit card. Here, we own the relationship with AWS on their behalf and bill on their usage at cost.

As you can imagine, this is a pain. As we don't want someone on our accounts team to pick through AWS billing reports and assemble invoices every month. But, AWS make this easier, by allowing you to group billing with AWS Organizations and permitting the automated generation of detailed billing reports. However, someone or something has to process the resulting CSV file.

This is where open-source apps come in

That's where our new open source Node app, Billder, comes in. It's a service that allows you to pull in your AWS billing data from S3, process it and generates an invoice for each linked account in your AWS Organizations setup.

At the moment, it only supports certain accountancy software providers, so pull requests proposing additional accountancy providers are most welcome!

Anyway, here's how I went about it, for those who are interested in the details:

Automated AWS Re-Billing Explained

I started out by hunting around for something that already reads the AWS billing data in a sane way. And I came across the existing 'aws-billing' app by a company called Segment on the npm website. Unfortunately, it worked on the basis you only cared about total billing for all accounts, with sales tax included. And, had no concept of "linked accounts". So I forked it (hurrah for open source!). Then, I added linked account support and an option for before tax totals, which is the version that gets included with Billder.

Here's the pull request I submitted for the feature.

So I have a means in Node of getting my linked account data back in a JavaScript object. Now to do something with it.

API authentication

Having started in Node, I decided to continue down that path and tackle API authentication at FreeAgent next. I tried a few OAuth 2.0 modules (which is the authentication method used by FreeAgent) and settled on 'simple-oauth2'. I used this to create a method for refreshing our access token so we can make API calls, which is a monthly task.

Having got authentication nailed I needed to figure out how to make some other API calls as well. I decided to follow the lead of the 'simple-oauth2' module and continue to use the popular 'request' module it had already installed for me.

To make the HTTP request building part as simple as possible, I wrote a little function I could pass some parameters into that would return an 'options' object all ready to pass straight into 'request'.

The first API call I needed to handle was one to get currency exchange rates. This is because regardless of your billing currency, AWS reports are in USD. We operate mostly in GBP, so some conversion was required. After looking at the Visa and XE.com API pricing, I decided to use Fixer.io instead, a free API wrapping the European Central Bank's daily currency feed. Not real-time, but good enough for my purposes.

So with some JavaScript files just exporting objects of configuration (one for general settings and one for a list of accounts to process) we were almost there. Now for the final piece of the puzzle, compiling an invoice per account and shipping it to our accountancy service provider.

Invoicing per account

I made a wrapper for service provider-specific invoice creation code, which I could consistently call from index.js - our main executable script here.

That pulled in the currency conversion bits, and any other massaging of the account data we might need to do in the future before the actual task of building an invoice, then handed the 'account' object (complete with necessary currency data) off to a provider-specific function that actually handles the formatting and submission of the invoice.

To achieve this and be sure previous API calls had succeeded before continuing, I also added in a popular extension to the 'request' module, 'request-promise'. Node is normally asynchronous, and 'request-promise' makes it simple to use JavaScript Promises to sequence HTTP request events that are need to be synchronous.

A bit of logging with 'console.log' and the excellent 'modern-syslog' module, for sensible logging to the Linux system log, and that's a wrap!

Hope some folk other than us find this useful for automated AWS re-billing

Written by

Greg Harvey