# Calculating Download MD5 Hash - Published: March 9, 2023 - Author: Miki Tebeka - URL: https://www.ardanlabs.com/blog/2023/03/calculating-download-md5-in-go.html ### Introduction One of the exercises I give to students is to download a single big file over HTTP concurrently using several goroutines using [HTTP Range](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests) requests. An extra part of the exercise is to validate the downloaded file from a known MD5 signature. This extra part turns out to be interesting, let's have a look. ### Getting Download Information Let's make an HTTP HEAD request to get information about a file located in a public dataset stored on Google Cloud Storage (GCS). https://storage.googleapis.com/gcp-public-data-landsat/LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/LC08_L1GT_044034_20130330_20170310_01_T2_B2.TIF **Listing 1: HEAD request** ``` $ curl -i -I https://storage.googleapis.com/gcp-public-data-landsat/LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/LC08_L1GT_044034_20130330_20170310_01_T2_B2.TIF HTTP/2 200 x-guploader-uploadid: ADPycdtU_fX5RkUlYHRaU4ajofN7LOXIdjzNJUzKWyKKIOtIhxyhhyY-0JZ1avs5T1ohCZD7_0jPurQ2ByB3YlCm2D1D0A expires: Thu, 02 Mar 2023 11:02:52 GMT date: Thu, 02 Mar 2023 10:02:52 GMT cache-control: public, max-age=3600 last-modified: Fri, 08 Sep 2017 09:10:25 GMT etag: "eec1fa5ce8077d7030e194eb5989c937" x-goog-generation: 1504861825662906 x-goog-metageneration: 1 x-goog-stored-content-encoding: identity x-goog-stored-content-length: 69962928 content-type: application/octet-stream x-goog-hash: crc32c=ir4fqg== x-goog-hash: md5=7sH6XOgHfXAw4ZTrWYnJNw== x-goog-storage-class: STANDARD accept-ranges: bytes content-length: 69962928 server: UploadServer alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" ``` Listing 1 shows a `curl` call where the `-I` flag denotes the use of the HEAD verb and the `-i` flag tells curl to include the response headers. By looking at the response (content-length header) we can see the size of the file is 69962928 bytes or about 66.7 MiB. Look for the `x-goog-hash` header which contains the hash information for the file. The `md5` prefix tells us the value represents an MD5 signature. The data that follows is the signature encoded in [base64 encoding](https://en.wikipedia.org/wiki/Base64). Note that we have two `x-goog-hash` HTTP headers, another one for the [CRC check](https://en.wikipedia.org/wiki/Cyclic_redundancy_check). ### Actual File Sum Let's check the file MD5 signature as reported by the `md5sum` utility. **Listing 2: MD5** ``` $ curl https://storage.googleapis.com/gcp-public-data-landsat/LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/LC08_L1GT_044034_20130330_20170310_01_T2_B2.TIF | md5sum eec1fa5ce8077d7030e194eb5989c937 - ``` Listing 2 shows the calculation of MD5 signature after piping the file curl downloads to the `md5sum` utility program. We can see the signature is `eec1fa5ce8077d7030e194eb5989c937`. Now that we have the data we need, let's start writing Go code to get a signature for a file hosted on GCS. The code can be found [here](https://github.com/ardanlabs/gotraining/blob/master/topics/go/algorithms/md5/dlhash.go) ### Making a `HEAD` Request **Listing 3: HEAD Request in Go** ``` 12 func urlSig(ctx context.Context, url string) (string, error) { 13 req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil) 14 if err != nil { 15 return "", err 16 } 17 18 resp, err := http.DefaultClient.Do(req) 19 if err != nil { 20 return "", err 21 } 22 if resp.StatusCode != http.StatusOK { 23 return "", fmt.Errorf("%q: bad status - %s", url, resp.Status) 24 } ``` Listing 3 shows the first part of the `urlSig` function which will return the MD5 signature for any file at the specified url. . On line 12 we define the `urlSig` function with two parameters: a context and the location of the file. It's always a good idea to use context for timeouts when dealing with the network. On line 13, we create a new `HEAD` request using the context and the URL parameters. The last `nil` parameter means we are not passing any data with the request. On line 18, we use the default HTTP client to make the call and on line 22, we check that the response status code is a 200. ### Finding the Signature **Listing 4: Finding MD5 Signature HTTP Header** ``` 26 // Find MD5 hash in HTTP headers. 27 const ( 28 header = "x-goog-hash" 29 prefix = "md5=" 30 ) 31 b64hash := "" 32 values := resp.Header.Values(header) 33 for _, v := range values { 34 if strings.HasPrefix(v, prefix) { 35 b64hash = v[len(prefix):] 36 break 37 } 38 } 39 40 if b64hash == "" { 41 return "", fmt.Errorf("can't find md5 hash %s: %v", header, values) 42 } ``` Listing 4 shows how to read the MD5 signature from the HTTP header. On lines 28 and 29, we define the header and the MD5 signature keys. On line 32, we get all the values for the specified key. **Listing 5: HEAD request** ``` x-goog-hash: crc32c=ir4fqg== x-goog-hash: md5=7sH6XOgHfXAw4ZTrWYnJNw== ``` Listing 5 is a reminder that two values are provided for the `x-goog-hash` key in the header. Then on lines 33-38 in listing 4, we look for the MD5 signature value from the list. Once found, on line 35 we remove the `md5=` prefix from the header value to get the signature. Finally on lines 40-42 we return an error if we can't find the signature. ### Decoding and Returning the Hash **Listing 6: Decoding the Hash Value** ``` 44 hash, err := base64.StdEncoding.DecodeString(b64hash) 45 if err != nil { 46 return "", err 47 } 48 49 // Convert hash to "eec1fa5ce8077d7030e194eb5989c937" format. 50 return fmt.Sprintf("%x", hash), nil 51 } ``` Listing 6 shows how to decode the base64 encoded hash value. On line 44 we use the standard encoder from the `encoding/base64` package to decode the signature into a `[]byte`. Then on line 50, we convert the signature from a `[]byte` to a hexadecimal string. ### Testing the Code **Listing 7: Testing** ``` 53 func main() { 54 url := "https://storage.googleapis.com/gcp-public-data-landsat/LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/LC08_L1GT_044034_20130330_20170310_01_T2_B2.TIF" 55 ctx, cancel := context.WithTimeout(context.Background(), time.Second) 56 defer cancel() 57 58 fmt.Println(urlSig(ctx, url)) 59 } ``` Listing 7 shows a call to the `urlSig` function. On line 55 we create a context and on line 58 we call the function and print the return values. Now, let’s run it: **Listing 8: Running the Code** ``` $ go run dlhash.go eec1fa5ce8077d7030e194eb5989c937 ``` Listing 8 shows how to run the code. We can see the signature matches the one we got using the `md5sum` program at the beginning of the post. ### Conclusion Even small tasks such as verifying download integrity can lead to many interesting places. We covered the following topics to make this happen: - Making an HTTP head request - HTTP headers - Base64 encoding - MD5 signatures - Converting `[]byte` to a string I hope you learned something useful from this blog post, feel free to reach me at miki@ardanlabs.com for comments and suggestions. And of course, you are more than welcome to join one of our [world class trainings](https://www.ardanlabs.com/training/).