tompaton.com/blog

Posts for 2017/06

remoteStorage.io - Unhosted Web Apps

↫ 2017-06-28 bookmarks

remoteStorage is an open standard to enable unhosted web apps where users are in full control of their data and where it is stored, while app developers are freed of the burden of hosting, maintaining and protecting a central database.

Staging of blog updates

2017-06-17 posts

  1. Symlink generated output to next_release/
  2. Run ./release.sh to move next_release/ to released/ and released/ to prev_release/
  3. Run ./release.sh --rollback to revert to prev_release/

    4. Time-delay safety feature — ./release.sh will no-op until next_release/ is an hour old. Can then run this from a cron job and there's a window to catch issues before they are publicly visible. This will also effectively batch RSS/JSON feed updates to be kind to readers.

Blog architecture revisited

2017-06-16 posts aws

I'm bailing on the original blog architecture for now.

Using AWS Lambda is just more complicated than it should be and I'd rather just get this thing working.

I reverted a bunch of the complicated changes I had made so far in my static generation script and that was before I'd even figured out how to handle the mako templates or bundle the whole thing for deployment or hook up the API gateway...

Using Zappa seemed like a possible solution, but even that seems like overkill.

So the new process is:

  1. content in a github repo, initially directly in a single jsonfeed file
  2. posting to my bot would have a checkout of that repo and update the file, push it to github
  3. generate.py script pulls the latest version and processes the feed file into static html
  4. nginx serves static html
  5. profit
Nginx proxy to AWS S3 with url rewrites

2017-06-07 posts nginx aws

There seems to be plenty of information on how to use Nginx to proxy content stored in an AWS S3 bucket , but it took me a long time to figure out how to also get url rewriting to work in conjunction with this.

I wanted to rewrite the $uri to allow links to omit index.html from directories and also to omit .html extensions, and came up with the following (this goes in the server block, before the location block below): 

rewrite ^/somepath/(.+)\.(html|json|rss|css)$ /somepath/$1.$2 last;
rewrite ^/somepath(|/.+)/$ /somepath$1/index.html last;
rewrite ^/somepath/(.+)$ /somepath/$1.html last;

This works as follows:

  1. If $uri has a known extension, rewrite as itself and break out of the rewrite block.
  2. Add index.html to directories and break out of the rewrite block.
  3. If we've made it this far, it doesn't have an extension and isn't a directory, so assume a .html extension.

Note that the list of extensions in line 1 needs to include everything you serve from S3, otherwise a .html extension will be added and it will 404.

Once that's done, you can then pass any requests to /somepath/ through to S3 (I have the bucket permissions set to public-read, you may need to use a different proxy_pass url): 

location /somepath/ {
  proxy_intercept_errors on;
  proxy_hide_header x-amz-id-2;
  proxy_hide_header x-amz-request-id;
  proxy_pass https://s3.<region>.amazonaws.com/<bucket>/;
}

When debugging, it may help to enable logging of rewites: 

rewrite_log on;
error_log /var/log/nginx/<somepath>.error.log notice;
Python, Go and Rust bot comparison

2017-06-01 posts code

After implementing my bot in Python, I thought I'd try porting it to other languages as an opportunity to learn a bit of Go and Rust.

Both Go and Rust are statically typed and compile to a single self-contained binary, which is appealing as there is no need to maintain a virtualenv and install dependencies on the server.

The parsing code for "rain" messages is a good starting point to compare the bots. Each is using a "parser combinator" library to parse the messages.

Quick thoughts: Python & Go are both pretty sane. There's almost certainly a better way in Rust, but I found it considerably harder.

The tests are stored in different places in each version so the total line count is not comparable, and the Rust version includes "ignore case" code that is built into Python's parser combinators, and that I added into a fork of the Go parser library.

Blog architecture outline

2017-06-01 posts

  1. content in a github repo, initially directly in a single jsonfeed file
  2. posting to my bot would have a checkout of that repo and update the file, push it to github
  3. github webhook would call amazon lambda
  4. lambda would process the feed file into static html
  5. push html to s3
  6. set up nginx to proxy the blog/ path to s3
  7. profit