Screenshots of website BayWa GIF generator website shown on a MacBook Pro and an iPhone X

Engineering of a GIF Generator Micro-Site

Summary

Cloud Under developed front-end and all back-end services of a micro-site for its agency client, who provided concept, design and content. The core feature of the website is an interactive GIF generator for users to create their personalised, yet branded animated GIF to download and share on social media. This is the case of a full-stack web engineering project by Cloud Under.

From a user's perspective creating a personalised GIF happens in three steps:

  1. Select one of the available video loops.
  2. Personalise the video sequence by selecting a predefined text options or enter your own text.
  3. Download the rendered GIF file or share the GIF file or landing page URL containing the GIF directly with friends via Facebook, Twitter, WhatsApp or email.

Check out the demo video:

Screen recording of the microsite on a desktop browser

Implementation of the front-end

The easier part of this project was the decision on the front-end stack. We already had great success in previous projects with React for building fast user interfaces. We knew the micro-site would only be made up of very few pages and the most important part was the responsiveness of the interactive GIF creator in three steps.

Although not demanded by the project specifications, we thought, fully server-rendered HTML pages for better SEO can’t hurt. So, for the first time, to bring multiple parts, that we have already used in other projects, together, we gave React Starter Kit a go and it turned out to be a perfect fit to start with this project. Also, there was no longer an excuse not to try out GraphQL, which is quite a thing at the moment.

Tiny back-end server even during traffic spikes

As mentioned, we wanted server-rendered HTML pages for fast page performance and search engine optimisation. React Starter Kit gives us that for free – a few customisations and we were ready to go. Its build script even includes an option to package everything as Docker image.

With each release of our little web application we had pretty much everything bundled together in one Docker image – the server (Express on Node.js) as well as all static assets, packaged by Webpack.

We added some “Cache-Control” headers to all server responses and put the server behind AWS CloudFront, the content management network (CDN) of Amazon Web Services. This way, we could be sure that even during traffic spikes the origin server would only receive very little hits – a great start for scalability with the added benefit of saving cost.

Scripted GIF rendering

How do you render an animated GIF on a server? This was certainly a new challenge. We already had experience with server-side image rendering and we also developed quite a few projects with server-side video rendering and transcoding. GIFs are somewhat in-between. Generally GIFs are considered an image format, but with the ability to add frame animations, it is actually more like a video file. I won’t go into details about the challenges of the GIF format itself, which was first developed over 30 years ago.

After exploring a number of options, including JavaScript-based GIF encoders, we eventually decided to go with FFmpeg, a very flexible open-source tool for audio and video, which we have used in many other projects over many, many years. Thankfully FFmpeg is able to encode GIFs and also deal with the 8-bit colour palettes of GIF files.

Once we figured out how to overlay the video with branding a frame image and the custom text, entered by the user on the website, we were able to put everything together in a script, which ultimately takes two dynamic parameters: the source video template and the user’s text.

The challenge of server video encoding

If you ever encoded or transcoded a video file, you know how CPU demanding this process is. If you have a fast computer and encode one video at a time, the time it takes to complete might be acceptable and – more importantly – it is predictable.

There is nothing worse for a user to wait for an undefined amount of time for a result. If 2, 3, 25 or 100 users want to generate their personalised GIF at the same time on the same server, predictability goes out of the window. Apart from that, if you were restricted to a single server, you would have to come up with a queuing system in order to encode only a limited number of videos at any given time (e.g. one per available CPU), otherwise everything would slow down and the server could quickly run out of memory and fail. We implemented queues in previous video web projects, where due to hosting restrictions set by the client, we had no other choice.

The problem with a queue is, when a user hits the submit button, they want to see a result very soon and if they have to wait for longer than a regular page load time, they want to know how long they would have to wait. A fast moving progress bar improves the user experience immensely, yet if you can’t ensure that the result is actually available when the progress bar has filled up, it is understandable that the user might think that this thing isn’t working and they may move on by closing the tab.

Serverless GIF rendering

There are many potential solutions to the problem of how to render and encode a video, or an animated GIF in this instance, in a scalable and predictable manner and it all depends on the circumstances of the given project, including horizontal scaling and auto-scaling (load-balancing between multiple servers), vertical scaling (servers with lots of CPU and memory).

In our case, we had no idea about the demand to be expected. We kinda knew there was a limited target audience, but on the other hand, the website would be promoted online and at a trade show within a very limited time frame, so we had to expect traffic spikes.

To go serverless for this job seemed to be the perfect fit. In case you haven’t heard about the “serverless” craze: Of course, serverless doesn’t mean there are no servers involved. You still run your code on a server, but you don’t have to manage the server, the operating system or even the runtime environment and you are billed only for actual usage in 100 millisecond increments and for the memory reserved during execution. This means, in times when nobody requests a new GIF, e.g. in the middle of the night, there is absolutely no cost. But then, if 1000 users request a new GIF at the very same time, because they all saw this promotion at 9am, Lambda simply starts 1000 instances within milliseconds, each run for about 4 seconds and then shut down again.

So, what does our Lambda function actually do? Basically, our Lambda function is the GIF rendering and encoding script, that takes two arguments, mentioned above. The truth is, it does a few more things under the hood:

  1. Takes the arguments and validates them.
  2. Start downloading the source template video file as requested in one of the arguments from S3.
  3. Start querying a database to check if the requested GIF has already been produced (same template and same custom text).
  4. Number 2 and 3 run in parallel to save some valuable milliseconds. When both requests have completed and the database didn’t find a matching entry, execute FFmpeg. Otherwise skip the following steps and go straight to step 9.
  5. Add a new database entry with a “is still rendering” timestamp flag. This prevents multiple pointless renders of the very same GIF, but also allows to try again if a render fails for some reason. It’s always good to have as much resilience in your code as you can afford.
  6. When FFmpeg has completed, optimise the GIF file to reduce file size. This optimisation doesn’t write to disk, but returns the stream directly back to the Lambda function.
  7. The optimised GIF stream is uploaded as GIF file with a hashed file name to S3.
  8. When the upload is complete, remove the “is still rendering” flag from the database.
  9. Return the GIF’s filename hash.

Back-end and front-end communication

The invocation of the Lambda function is done by the server (remember, the small Express server running in a Docker container). Once the Lambda function returns the filename of the GIF, it is returned to the client via the still open XHR request. The React component eventually receives the GIF ID and displays the result by changing the route (the URL in the browser), but of course, without a full page load.

Each GIF with its unique ID (hash) has at least in theory its own URL, actually two. One for displaying the result GIF to the user who created the GIF with some social media share buttons. The second URL is the landing page for a shared link, which contains the relevant meta tags.

The (not so) secret trick here is, the server blindly generates those pages without even checking if a GIF with the given ID even exists. As long as the ID matches a regular expression, it’ll respond with a HTML page with links to a GIF file that most likely exists. This saves an almost pointless database request – lower cost, faster response.

Just in case, the GIF file is lazy-loaded via JavaScript and if the download fails, an error message is displayed. This is another example of some resilience, but ultimately, it is extremely unlikely that someone would share the link to a GIF page, which has never been created in the first place. If someone does this anyway, they probably do it on purpose. Yes, we could respond with HTTP 404 error to be totally correct, but in this case I doubt the additional database request is worth it. So far the server web application doesn’t even require a database.

Gallery

The website also contains a gallery showcasing manually selected creations. This finally requires the server to read from a database. As the dataset is expected to be very small – we need about 13 bytes per gallery entry plus some overhead for the data structure itself – we hold all the data on the server in memory and refresh it from the database no more than twice per hour. If a gallery admin makes changes to the entries, not only is the database updated, the in-memory dataset is also updated directly, which means the changes take effect immediately, without another database request.

The images on the gallery page are lazy-loaded via JavaScript. The number of images loaded at once depend on the client’s viewport size. GIFs are not exactly small in size and also quite hungry for resources on the client device, which can be a problem on older smart phones. If you’re loading the gallery page from a smaller device, only a small number of GIFs are loaded in one go and the number increases with the number of columns we fit into the viewport.

To be honest, the actual HTML source code of the gallery page doesn’t contain any <img> tags for the GIFs in the gallery. In fact, the list of GIFs is loaded via XHR from the server and the HTML never needs to be changed. Although a normal REST API endpoint returning a JSON with the gallery’s GIF IDs would be perfectly fine, we decided to experiment with GraphQL. In this instance, GraphQL doesn’t do the job any better than a REST endpoint. The end result exactly the same, while the implementation is slightly more complicated. It was just a good opportunity to try something new without risking to run into a dead end in a larger project.

What else did Cloud Under do in this project?

  • We also designed and developed a small, easy to use, and fast back-office for administering the gallery.
  • We implemented a bad language filter with a list of words provided by the client.
  • We edited the jumbotron video on the home page to make it look like a GIF and encoded it as web-optimised video for much better performance than a large GIF.
  • As an extra bonus we vectorised some graphics and integrated them as SVG to look crisp on all devices and we even CSS-animated the windmill.
  • We did some technical research in terms of GIF sharing options on social media.
  • We optimised the client-provided GIF templates in preparation for server-side rendering with branding.
  • We instructed and supported the client’s system administrators to get the tech stack designed by us up and running in their own Amazon Web Services environment.

What can we do for you?

If you work for a digital, creative or design agency or you’re a web designer or have an upcoming online project in any other company looking for tech expertise – look no further. Cloud Under is specialised on advising, planning, and implementing very unique projects – online, offline or a combination of both.