Inspired by Vicki I decided to build a Tweetbot – the code is available here. You can follow the tweetbot online – the architecture that Vicki proposed is basically what I did, only I made a few changes in the code.
Time taken
It’s worth pointing out first that it took me approximately 32 hours of work to get this all to work. I think a big part of that is that I wasn’t familiar with AWS IAM. Like anything – I think the first time you build something it’s more tricky than normal.
Technical goals of this project:
The technical goals of this project was to get my head around how to use AWS Lambdas – because before starting I didn’t know what they were. I also wanted to understand how to set up a CI system from scratch. I used Travis CI because of it’s integration with github. The other reason for this project is that I wanted to stretch my software engineering skills. All in all I consider it a success.
Learnings from AWS:
- Getting IAM to work, it’s quite tricky and I had to read lots of documentation and stackoverflow to get this to work.
- Total time spent about 32 hours, about half of that was spent with errors – of connections between lambdas and S3 and all.
- This is tricky and I still am not very good at all of this. The docs aren’t in my opinion super useful.
- Don’t mix up lambda execution roles and IAM roles.
- Make sure to configure your AWS CLI correctly – I got confused by different profiles.
Travis CI set up
- It wasn’t too difficult to get a Travis CI setup working.
- https://stackoverflow.com/questions/37267916/how-to-run-aws-configure-in-a-travis-deploy-script was a useful link for helping me understand how to configure credentials.
TODO:
- Still haven’t got SSM working – I’ve no idea about this.
- It needs more tests like tests before CI runs, and more tests locally.
AWS Lambdas
AWS Lambdas are super useful. Here is a short list of what they can do(much more in the docs):
- Run when a file is placed in an S3 bucket and move the file to another location
- Process logs with Kinesis streams and store the logs in DynamoDB
- Compile a static website based on a new code change and reload the site
- Backup files and put them into cold storage
When you think about it – many business processes, fit into this sort of workflow. For example you may have someone uploading a pdf of a document to a server, an AWS Lambda could resize that document and then put it into cold storage after a certain period of time.
A good thing to keep in mind is the limitations of lambdas:
- 512 MB disk space
- Execution timeout after 5 minutes of activity
- 128 to 1536 MB
- Have boto included as a runtime library
I had some trouble uploading the AWS Lambda manually – see Vickis blog post for details of how to solve that.
Conclusion
All in all this was a successful side project. I learned a ton about AWS Lambdas and I probably need to spend more time getting my head around AWS IAM. Also investing in bash scripts and tools to get the zipping of AWS Lambda done is worth it.
I also want to add in more images, so there can be more varied tweets.
Something I need to get my head around is how to do this with Zappa.
