How to use AWS Lambda to build a tweetbot

Inspired by Vicki I decided to build a Tweetbot – the code is available here. You can follow the tweetbot online – the architecture that Vicki proposed is basically what I did, only I made a few changes in the code.

Time taken

It’s worth pointing out first that it took me approximately 32 hours of work to get this all to work. I think a big part of that is that I wasn’t familiar with AWS IAM. Like anything – I think the first time you build something it’s more tricky than normal.

 

dn34r8cxcaeg0_l
“Paddle steamer Britannie – Bristol Quay” Eric Ravilious, 1938

Technical goals of this project:

The technical goals of this project was to get my head around how to use AWS Lambdas – because before starting I didn’t know what they were. I also wanted to understand how to set up a CI system from scratch. I used Travis CI because of it’s integration with github.  The other reason for this project is that I wanted to stretch my software engineering skills. All in all I consider it a success.

Learnings from AWS:

  • Getting IAM to work, it’s quite tricky and I had to read lots of documentation and stackoverflow to get this to work.
  • Total time spent about 32 hours, about half of that was spent with errors – of connections between lambdas and S3 and all.
  • This is tricky and I still am not very good at all of this. The docs aren’t in my opinion super useful.
  • Don’t mix up lambda execution roles and IAM roles.
  • Make sure to configure your AWS CLI correctly – I got confused by different profiles.

 

Travis CI set up

 

TODO:

  • Still haven’t got SSM working – I’ve no idea about this.
  • It needs more tests like tests before CI runs, and more tests locally.

 

AWS Lambdas

AWS Lambdas are super useful. Here is a short list of what they can do(much more in the docs):

  • Run when a file is placed in an S3 bucket and move the file to another location
  • Process logs with Kinesis streams and store the logs in DynamoDB
  • Compile a static website based on a new code change and reload the site
  • Backup files and put them into cold storage

When you think about it – many business processes, fit into this sort of workflow. For example you may have someone uploading a pdf of a document to a server, an AWS Lambda could resize that document and then put it into cold storage after a certain period of time.

A good thing to keep in mind is the limitations of lambdas:

  • 512 MB disk space
  • Execution timeout after 5 minutes of activity
  • 128 to 1536 MB
  • Have boto included as a runtime library

I had some trouble uploading the AWS Lambda manually – see Vickis blog post for details of how to solve that.

Conclusion

All in all this was a successful side project. I learned a ton about AWS Lambdas and I probably need to spend more time getting my head around AWS IAM. Also investing in bash scripts and tools to get the zipping of AWS Lambda done is worth it.

I also want to add in more images, so there can be more varied tweets.

Something I need to get my head around is how to do this with Zappa.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s