Introduction
In part 1 we took a high-level view on serialization and JSON. In this part, we’ll roll our sleeves and start working with JSON, focused on emitting JSON. You might think this is a basic topic, but there is much more to it than just calling json.Marshal
.
json.Marshal
vs json.Encoder
The encoding/json
package has two main APIs: Marshal
and NewEncoder
. The Marshal
function returns a []byte
while the NewEncoder
function will write to an io.Writer
. The question is: When should you use one API over the other?
When you are building a web service, you’ll typically emit JSON back to the client inside an HTTP handler. Using the NewEncoder
function with the http.ResponseWriter
parameter is sometimes the best choice here.
The problem with using NewEncoder
is that if there are encoding errors when using the API, you cannot notify the client by setting the HTTP status code. This is because the status needs to be set before streaming starts and once you start streaming the API will set the status code to OK (200)
. If you get an error during streaming, make sure to log it.
Working with the Marshal
API will allow you to check for an encoding error before you send the data to the client. So you can send a failure status code. However, since the network is not reliable, writing the data to the http.ResponseWriter
can also fail.
Simple Marshaling
There are two ways to structure data that can be marshaled into JSON. One way is using a map[string]any
and the other is to use a struct.
Listing 1: Marshaling a map
25 vm := map[string]any{
26 "id": "b70229443f8d489bbc733f13a9268f63",
27 "cpus": 4,
28 "memory": 32,
29 }
30
31 json.NewEncoder(os.Stdout).Encode(vm)
Listing 1 shows how to use a map[string]any to marshal JSON
. On lines 25-29, we create the map and then on line 31, we marshal it to standard output.
This code will print:
{"cpus":4,"id":"b70229443f8d489bbc733f13a9268f63","memory":32}
Using a map is easy and quick, but using a struct allows you to define a schema for your outgoing message.
Here’s the example using a struct:
Listing 2: Marshaling a Struct
08 type VM struct {
09 ID string
10 CPUs int
11 Memory int
12 }
...
13
15 vm := VM{
16 ID: "b70229443f8d489bbc733f13a9268f63",
17 CPUs: 4,
18 Memory: 32,
19 }
20
21 json.NewEncoder(os.Stdout).Encode(vm)
Listing 2 shows struct marshaling. On lines 08-12, we define the VM
struct and then on lines 15-19, we create the vm
variable. Finally on line 21, we marshal the struct to standard output.
This code will print:
{"ID":"b70229443f8d489bbc733f13a9268f63","CPUs":4,"Memory":32}
You may have noticed the properties are not in lower case (e.g. ID
and not id
) as in the last example. If you want to change the name of the properties, you need to use field tags.
You can use field tags to tell the encoding/json
package how to map the struct fields with the emitted JSON properties.
Listing 3: Using Fields Tags
08 type VM struct {
09 ID string `json:"id"`
10 CPUs int `json:"cpus"`
11 Memory int `json:"memory"`
12 }
13
...
15 vm := VM{
16 ID: "b70229443f8d489bbc733f13a9268f63",
17 CPUs: 4,
18 Memory: 32,
19 }
20
21 json.NewEncoder(os.Stdout).Encode(vm)
Listing 3 shows how to use field tags to change the name of the output JSON properties. On lines 08-12, we define the VM
struct and add a field tag to each field in the struct. On lines 15-19, we create a vm
variable and on line 21 we marshal it to the standard output.
This code will print:
{"id":"b70229443f8d489bbc733f13a9268f63","cpus":4,"memory":32}
You can do more with field tags, see the json.Marshal function documentation for the full specification.
I’m going to focus on two things from the full specification: omitempty
and -
.
If you add omitempty
to a field tag, encoding/json
won’t write the field to the output if it has a zero value.
This allows you to save bandwidth, which eventually will save you money.
Listing 4: omitempty
08 type VM struct {
09 ID string `json:"id,omitempty"`
10 CPUs int `json:"cpus,omitempty"`
11 Memory int `json:"memory,omitempty"`
12 }
13
...
15 vm := VM{
16 ID: "",
17 CPUs: 4,
18 Memory: 32,
19 }
20
21 json.NewEncoder(os.Stdout).Encode(vm)
Listing 4 shows how to use omitempty
. On lines 08-12, we define VM
with omitempty
in the fields tags. On lines 15-19, we create a vm
variable and on line 21 we marshal it to the standard output.
The code will print:
{"cpus":4,"memory":32}
Without omitempty
this code will print:
{"id":"","cpus":4,"memory":32}
The use of omitempty
in this case saved 8 bytes, which might not seem a lot, but if you’re sending many messages it can amount to a lot of bandwidth.
Another struct tag that encoding/json
recognizes is -
, which tells it not to emit a specific field. The -
option is used to prevent leaking sensitive information, such as not wanting to serialize a Token
field in your User
struct.
I see -
as a code smell that you don’t have a good separation between your API layer and your business or data layers.
Custom Serialization
JSON has a limited set of types which means not all Go types can be mapped to JSON types. For example, JSON does not have a “timestamp” type, while Go has time.Time
.
Timestamps can be serialized into JSON in multiple ways: One way is using a string such as 2025-01-02T15:35:47.990Z
. Another way is to use a number which is usually the number of seconds since January 1, 1970 UTC (known as epoch).
Let’s try and see what the json package in the Go stdlib does:
Listing 5: Marshaling time.Time
09 type Log struct {
10 Time time.Time
11 Level string
12 Message string
13 }
...
16 l := Log{
17 Time: time.Now().UTC(),
18 Level: "ERROR",
19 Message: "divide by cucumber error",
20 }
21
22 if err := json.NewEncoder(os.Stdout).Encode(l); err != nil {
23 fmt.Println("ERROR:", err)
24 }
Listing 5 shows marshaling a struct with a time.Time
field. On lines 09-13, we define Log
struct and then on lines 16-20, we create a variable, then finally on line 22, we marshal l
to stdout.
This code produces the following output without any error:
{"Time":"2024-11-19T05:33:22.774425457Z","Level":"ERROR","Message":"divide by cucumber error"}
The encoding/json
package marshals time.Time
into an RFC3339. If you look at the documentation for the time.Time type, you’ll see it has a method called MarshalJSON
. This means that time.Time
implements the json.Marshaler interface.
Note: Another type that is missing from JSON is a binary type. encoding/json
will encode a []byte
into a base64 encoded string.
Using json.Marshaler
You can implement json.Marshaler
on your types to get custom JSON encoding.
Assume you have the following type:
Listing 6: Value Type
11 const (
12 Meter = "meter"
13 Inch = "inch"
14 )
15
16 type Value struct {
17 Unit string
18 Amount float64
19 }
By default, encoding/json
will marshal a Value
to a JSON object with a Unit
and Amount
properties. But say that you want a Value
to be encoded as 14.2inch
instead.
I use two steps when implementing json.Marshaler
. The first step is converting the type to a type that encoding/json
knows how to handle. The second step is to use json.Marshal
to return the result.
Do not try to construct the output JSON by hand unless you have a really good reason, there are many edge cases you might miss that way.
Listing 7: Implementing json.Marshaler
21 func (v Value) MarshalJSON() ([]byte, error) {
22 // Step 1: Convert to type known to encoding/json
23 s := fmt.Sprintf("%f%s", v.Amount, v.Unit)
24
25 // Step 2: Use json.Marshal
26 return json.Marshal(s)
27 }
...
49 v := Value{
50 Unit: Meter,
51 Amount: 2.1,
52 }
53
54 data, err := json.Marshal(v)
55 if err != nil {
56 return err
57 }
58
59 fmt.Println(string(data)) // "2.100000meter"
Listing 7 shows how to implement json.Marshaler
for Value
. On line 23, we convert v
to a string and on line 26, we use json.Marshal
to convert the string to a JSON compliant string. On lines 49-59, we create a Value
and then encode it to JSON.
Take a look at the output value: "2.100000meter"
, it is surrounded by quotes. If you try to construct the output by hand, there’s a good chance you’ll forget to add these quotes. And even if you remember the quotes, what happens if the string contains a quote? It’s best to let json.Marshal
do the work for you.
Note: that the MarshalJSON
method is defined with value semantics, but it will work with pointer semantics as well. If you’re not sure why both semantics will work, check out this awesome video by Bill.
Streaming JSON
The JSON specification does not natively support streaming - i.e.,sending one JSON object after another. If you want to stream JSON, the common way is to send one JSON object per line. This is known as jsonlines or ndjson.
Lucky for you, the json.Encoder
method can be used for this. Here’s an example:
Listing 8: Streaming JSON
09 type Event struct {
10 Type string `json:"type"`
11 X float64 `json:"x"`
12 Y float64 `json:"y"`
13 }
14
15 func work() error {
16 events := []Event{
17 {"click", 100, 200},
18 {"move", 101, 202},
19 }
20
21 enc := json.NewEncoder(os.Stdout)
22
23 for _, e := range events {
24 if err := enc.Encode(e); err != nil {
25 return err
26 }
27 }
28
29 return nil
30 }
Listing 8 shows how to stream JSON. On lines 09-13, we define the Event
type. On lines 16-19, we create a slice of two events. On line 21, we create a JSON encoder and on lines 23-29, we use the encoder to encode and stream all the events.
The output of this code is:
{"type":"click","x":100,"y":200}
{"type":"move","x":101,"y":202}
The encoder encoded each JSON object in a single line and added a newline between each JSON object. Of course, the receiving side should know to parse each line as a JSON object, which the JSON decoder does as well.
Streaming JSON with HTTP Chunked Transfer Encoding
HTTP version 1.1 added chunked transfer encoding. This allows an HTTP server to send a response in chunks. The server sets the HTTP header Transfer-Encoding
to chunked
and then for each chunk of data it writes the size in bytes followed by data.
In Go, you can send chunked data using a http.ResponseController.
Listing 9: Streaming JSON Over HTTP
41 func eventsHandler(w http.ResponseWriter, r *http.Request) {
42 ctrl := http.NewResponseController(w)
43
44 enc := json.NewEncoder(w)
45 for evt := range queryEvents() {
46 if err := enc.Encode(evt); err != nil {
47 // Can't set error
48 slog.Error("JSON encode", "error", err)
49 return
50 }
51
52 if err := ctrl.Flush(); err != nil {
53 slog.Error("flush", "error", err)
54 return
55 }
56 }
57 }
Listing 9 shows how to stream JSON in an HTTP handler. On line 42, we create an http.ResponseController
and on line 44, we create a json.Encoder
. On line 45, we iterate over the events and on line 46, we use enc
to encode the event and on line 52, we call Flush
that will send the current chunk of data.
You can use curl
to view the raw HTTP response:
Listing 10: Using curl to Call the Server
01 $ curl --raw -i http://localhost:8080/events
02 HTTP/1.1 200 OK
03 Date: Tue, 19 Nov 2024 17:15:21 GMT
04 Content-Type: text/plain; charset=utf-8
05 Transfer-Encoding: chunked
06
07 21
08 {"type":"click","x":100,"y":200}
09
10 20
11 {"type":"move","x":101,"y":202}
12
13 20
14 {"type":"move","x":102,"y":203}
15
16 20
17 {"type":"move","x":103,"y":204}
18
19 20
20 {"type":"move","x":104,"y":204}
21
22 21
23 {"type":"click","x":104,"y":204}
24
25 0
Listing 10 shows how to call the server and view the underlying chunked response. On line 01, we use curl
to call the server. The --raw
flag tells curl
to show the raw response and the -i
switch tells curl
to show the response HTTP headers. On line 05, we see that we get a chunked response and on lines 06-24 we see the chunks. Each chunk starts with the size in hexadecimal number and then the chunk content. One line 25, we see the sentinel value of 0
tell the HTTP client there are no more chunks.
Conclusion
Emitting JSON can be simple as json.Marshal
, but if you need more sophisticated methods, encoding/json
is there for you with field tags, the json.Marshaler
interface and streaming support.
Get to know the API and understand the pros and cons of using io.Writer
vs a []byte
in your code.