# Building A Weather App Using Go

- Published: December 10, 2013
- Author: William Kennedy
- URL: https://www.ardanlabs.com/blog/2013/12/building-weather-app-using-go.html

<a href="../../../images/goinggo/weather-main.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;" target="_blank"><span class="sr-only">View linked image: Screen Shot</span>![Screen Shot](../../../images/goinggo/weather-main.jpg)</a>

At <a href="http://www.ardanstudios.com/" target="_blank">Ardan Studios</a> we have spent the last 6 months, in our spare time and on weekends, building a consumer based mobile application called <a href="http://www.outcast.io/" target="_blank">OutCast</a>. The mobile application is tailored towards those who like spending time outdoors, whether that be fishing, hunting or any other type of activity.<br />
<br />

<a href="../../../images/goinggo/buoys.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;" target="_blank"><span class="sr-only">View linked image: Screen Shot</span>![Screen Shot](../../../images/goinggo/buoys.jpg)</a>

This first release of OutCast shows the conditions for the buoy stations and marine forecasts areas within the United States. All this information is updated every 10 minutes and there are map views with traditional grids and search.<br />
<br />
The backend processing for buoy and marine data is built using Go and running at <a href="http://www.iron.io/" target="_blank">Iron.IO</a> as a scheduled worker task. The buoy processing downloads a text file from the NOAA website and rips through it, updating MongoDB with any changes. The marine processing is a bit more complicated. This requires pulling down multiple web pages from the NOAA website and parsing out all the text. Go made building and running these tasks a breeze.<br />
<br />
Another important aspect of OutCast is real time weather radar for the last 50 minutes. This has been very challenging on multiple levels. Mainly because we needed a real good image library that would run on Linux and could be integrated with Go. We were fortunate to find ImageMagick's&nbsp;MagickWand C API and their Go package that provides the CGO bindings (<a href="https://github.com/gographics/imagick" target="_blank"><span class="sr-only">External resource on github.com: </span>https&#58;//github.com/gographics/imagick</a>).<br />
<br />
Processing images is an intense piece of work. Sometimes it takes 3 seconds to clean a single image. With 155 radar stations
that need to be processed every 5 minutes, it took us several refactors to get things working well. The MagickWand library can
only handle processing one image at a time. This restriction places a lot of stress on getting things done accurately within
an acceptable amount of time.<br />
<br />
Here is a sample of a radar image before and after processing: <br />
<br />
<div class="separator" style="clear: both; text-align: center;">

<a href="../../../images/goinggo/BYX_N0R_0.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" target="_blank"><span class="sr-only">View linked image: Screen Shot</span>![Screen Shot](../../../images/goinggo/BYX_N0R_0.gif)</a>

<a href="../../../images/goinggo/final.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;" target="_blank"><span class="sr-only">View linked image: Screen Shot</span>![Screen Shot](../../../images/goinggo/final.gif)</a>

</div>
There is another interesting constraint. NOAA updates the images every 120 seconds on different time boundaries. If the program can't download all the images very quickly, the application could have images out of sync when they are animated across the map. Many radar images cross each other like in Orlando, FL. On this area of the map we have 3 radar images overlapping each other.<br />
<br />
Sometimes the images are not available. The program goes out to get the image and it doesn't exist. This creates problems with gaps in the timeline. When this happens, there is an alternate image location the program attempts to use. If the image is still not available, then the image from the previous run is used. Unless you are looking for it, you usually can't tell.<br />
<br />
Then you have the issue of updating S3 storage and MongoDB for each image. This again needs to happen quickly to prevent image overlays from being out of sync. At the end of the day, you want to do your best to make sure that the images for all the radar stations are in sync. This will provide the best user experience.<br />
<br />
Radar image processing happens in three stages and runs on a VM at&nbsp;<a href="https://www.digitalocean.com/" target="_blank">Digital Ocean</a>&nbsp;with 2 Gig of memory and 2 Cores.<br />
<br />
So how does Go help make this all happen every 5 minutes all day and all night without fail?<br />
<br />
<b><span style="font-size: large;">Stage 1: Download Images</span></b><br />
There are 155 images that have to be downloaded. The images range from 1k to 20k in size depending on the activity of the weather at that moment. In this stage, the program spawns a Go routine for each image that needs to be downloaded. The Go program consistently does all 155 downloads in less than one second:<br />
<br />
<div style="background-color: #efefef; font-family: 'Courier New', Courier, monospace; font-size: small; overflow-x: scroll; padding: 10px; white-space: nowrap;">
15:15:02 radar.go:425: main : downloadImages : Started<br />
<br />
-- Spawn Go Routines<br />
15:15:02 radar.go:431: main : downloadImages : Info : Image [1] of [155]<br />
15:15:02 radar.go:431: main : downloadImages : Info : Image [155] of [155]<br />
<br />
-- Sample Download Per Image<br />
Worker-JAX : downloadImage : Info : HEADER : Url , N0R/JAX_N0R_0.gif<br />
Worker-JAX : downloadImage : Info : HEADER : Last-Modified , [Fri, 06 Dec 2013 20:12]<br />
Worker-JAX : downloadImage : Info : HEADER : Content-Type , [image/gif]<br />
Worker-JAX : downloadImage : Info : HEADER : Vary , [Accept-Encoding]<br />
Worker-JAX : downloadImage : Info : HEADER : Cache-Control , [max-age=180]<br />
Worker-JAX : downloadImage : Info : HEADER : Expires , [Fri, 06 Dec 2013 20:18:02 GMT]<br />
Worker-JAX : downloadImage : Info : HEADER : Date , [Fri, 06 Dec 2013 20:15:02 GMT]<br />
Worker-JAX : downloadImage : Info : HEADER : Connection , [keep-alive]<br />
Worker-JAX : downloadImage : Info : HEADER : Server , [Apache/2.2.15 (Red Hat)]<br />
Worker-JAX : downloadImage : Info : HEADER : Accept-Ranges , [bytes]<br />
Worker-JAX : downloadImage : Info : HEADER : Content-Length , -1<br />
Worker-JAX : downloadImage : Info : HEADER : Image-Length , 6873<br />
<br />
-- All Images Complete<br />
15:15:02 radar.go:445: main : downloadImages : Completed
</div>
<br />
<b><span style="font-size: large;">Stage 2: Image Cleanup</span></b><br />
Now that all 155 images have been downloaded, they need to be cleaned using the ImageMagick API. Unfortunately, this can only be done with a single Go routine. Trying to clean more than one image at a time causes the program to pull a lot of memory. It really slows things down and can cause the program to be terminated by the OS. I have also seen other very odd behavior. The program will consistently complete this work in 90 seconds or less:<br />
<br />
<div style="background-color: #efefef; font-family: 'Courier New', Courier, monospace; font-size: small; overflow-x: scroll; padding: 10px; white-space: nowrap;">
15:15:02 radar.go:453: main : cleanImages : Started<br />
15:15:02 radar.go:457: main : cleanImages : Info : Image [1] of [155]<br />
<br />
-- Sample Processing Per Image<br />
Worker-RIW : cleanImage : Started<br />
Worker-RIW : cleanImage : Info : ReadImageBlob<br />
Worker-RIW : cleanImage : Info : TransparentPaintImage<br />
Worker-RIW : cleanImage : Info : WaveImage<br />
Worker-RIW : cleanImage : Info : Crop<br />
Worker-RIW : cleanImage : Info : Resize<br />
Worker-RIW : cleanImage : Info : EqualizeImage<br />
Worker-RIW : cleanImage : Info : GaussianBlurImage<br />
Worker-RIW : cleanImage : Info : BrightnessContrastImage<br />
Worker-RIW : cleanImage : Info : ResetIterator<br />
Worker-RIW : cleanImage : Info : GetImageBlob<br />
Worker-RIW : cleanImage : Completed<br />
Worker-RIW : cleanImage : Info : Defer : PixelWand Destroy<br />
Worker-RIW : cleanImage : Info : Defer : MagicWand Destroy<br />
<br />
-- All Images Complete<br />
15:16:20 radar.go:477: main : cleanImages : Completed
</div>
<br />
<b><span style="font-size: large;">Stage 3: Upload To S3 and Update MongoDB</span></b><br />
The last stage requires uploading all the cleaned images to S3 storage, removing the old images from S3 and then updating MongoDB with the new list of available image files. I am using the <a href="https://wiki.ubuntu.com/goamz" target="_blank">goamz</a> package from Ubuntu for accessing S3 and the <a href="http://labix.org/mgo" target="_blank">mgo</a>&nbsp;package from <a href="http://labix.org/" target="_blank">Gustavo Neimeyer </a>to access MongoDB.<br />
<br />
Just like when we download the images, this stage spawns a Go routine for each image that needs to be uploaded and recorded.&nbsp;The Go program consistently performs this work in one second:
<br />
<br />
<div style="background-color: #efefef; font-family: 'Courier New', Courier, monospace; font-size: small; overflow-x: scroll; padding: 10px; white-space: nowrap;">
15:16:20 radar.go:485: main : updateImages : Started<br />
15:16:20 radar.go:491: main : updateImages : Info : Image [1] of [155]<br />
15:16:20 radar.go:491: main : updateImages : Info : Image [2] of [155]<br />
15:16:20 radar.go:491: main : updateImages : Info : Image [3] of [155]<br />
<br />
-- Sample Processing Per Image<br />
collateImage : Started : StationId[RIW] FileName[US/WY/RIW/20131206-2015.gif]<br />
collateImage : Info : Remove : Minutes[50.01] FileName[US/WY/RIW/20131206-1935]<br />
collateImage : Info : Keep : Minutes[45.00] FileName[US/WY/RIW/20131206-1940.gif]<br />
collateImage : Info : Keep : Minutes[40.00] FileName[US/WY/RIW/20131206-1945.gif]<br />
collateImage : Info : Keep : Minutes[35.00] FileName[US/WY/RIW/20131206-1940.gif]<br />
collateImage : Info : Keep : Minutes[30.01] FileName[US/WY/RIW/20131206-1945.gif]<br />
collateImage : Info : Keep : Minutes[25.02] FileName[US/WY/RIW/20131206-1950.gif]<br />
collateImage : Info : Keep : Minutes[20.01] FileName[US/WY/RIW/20131206-1955.gif]<br />
collateImage : Info : Keep : Minutes[15.01] FileName[US/WY/RIW/20131206-2000.gif]<br />
collateImage : Info : Keep : Minutes[10.00] FileName[US/WY/RIW/20131206-2005.gif]<br />
collateImage : Info : Keep : Minutes[5.01] FileName[US/WY/RIW/20131206-2010.gif]<br />
collateImage : Info : Keep - New : FileName[US/WY/RIW/20131206-2015.gif]<br />
collateImage : Completed<br />
storeImageMongoDB : Info : Updating Mongo<br />
storeImageMongoDB : Completed<br />
storeImageS3 : Started : Bucket[SRR-Dev] FileName[US/AK/APD/20131206-2015.gif]<br />
storeImageS3 : Info : Putting File Into S3 : FileName[US/AK/APD/20131206-2015.gif]<br />
storeImageMongoDB : Completed<br />
<br />
-- All Images Complete<br />
15:16:21 radar.go:505: main : updateImages : Completed
</div>
<br />
<b><span style="font-size: large;">Conclusion</span></b><br />
This project has taught me a lot about Go. It is exciting to see how fast the Go routines can download the images and perform all the S3 and MongoDB work. Thanks to CGO, I was able to leverage a powerful image processing library and make calls directly from my Go code.
<br />
<br />
Currently<span style="color: #f57900;"> </span>we<span style="color: #f57900;"> </span>are<span style="color: #f57900;"> </span>porting<span style="color: #f57900;"> </span>the<span style="color: #f57900;"> </span>web<span style="color: #f57900;"> </span>service<span style="color: #f57900;"> </span>that<span style="color: #f57900;"> </span>powers<span style="color: #f57900;"> </span>the<span style="color: #f57900;"> </span>mobile<span style="color: #f57900;"> </span>application<span style="color: #f57900;"> </span>to<span style="color: #f57900;"> </span>Go.<span style="color: #f57900;"> </span>It<span style="color: #f57900;"> </span>is<span style="color: #f57900;"> </span>currently<span style="color: #f57900;"> </span>written<span style="color: #f57900;"> </span>in<span style="color: #f57900;"> </span>Ruby.&nbsp;We are using the <a href="http://beego.me/" target="_blank">beego</a>&nbsp;package for our web framework, the&nbsp;<a href="http://smartystreets.github.io/goconvey/" target="_blank">goconvey</a>&nbsp;package for our tests and the&nbsp;<a href="http://godoc.org/github.com/kelseyhightower/envconfig" target="_blank">envconfig</a>&nbsp;package to handle our configuration needs.
<br />
<br />
Our goal for OutCast is to provide people the ability to know in advance that the weekend is going to be great. We plan on using Go and MongoDB to analyze outdoor condition data with user preferences and experiences to deliver relevant information and forecasting. In the future, users will interact with OutCast by providing an experience review after their outdoor activities have ended.
<br />
<br />
Currently OutCast is only available in the <a href="https://itunes.apple.com/us/app/outcast-noaa-radar-hourly/id739808997?mt=8" target="_blank">Apple App Store</a>. We will have our Android version complete in January 2014.