JSON Serialization With Golang

Overview

JSON is one of the most popular serialization formats. It is human readable, reasonably concise, and can be parsed easily by any web application using JavaScript. Go as a modern programming language has first-class support for JSON serialization in its standard library. 

But there are some nooks and crannies. In this tutorial you'll learn how to effectively serialize and deserialize arbitrary as well as structured data to/from JSON. You will also learn how to deal with advanced scenarios such as serialization enums.

The json Package

Go supports several serialization formats in the encoding package of its standard library. One of these is the popular JSON format. You serialize Golang values using the Marshal() function into a slice of bytes. You deserialize a slice of bytes into a Golang value using the Unmarshal() function. It's that simple. The following terms are equivalent in the context of this article:

  • Serialization/Encoding/Marshalling
  • Deserialization/Decoding/Unmarshalling

I prefer serialization because it reflects the fact that you convert a potentially hierarchical data structure to/from a stream of bytes.

Marshal

The Marshal() function can take anything, which in Go means the empty interface and return a slice of bytes and error. Here is the signature:

func Marshal(v interface{}) ([]byte, error)

If Marshal() fails to serialize the input value, it will return a non-nil error. Marshal() has some strict limitations (we'll see later how to overcome them with custom marshallers):

  • Map keys must be strings.
  • Map values must be types serializable by the json package.
  • The following types are not supported: Channel, complex, and function.
  • Cyclic data structures are not supported.
  • Pointers will be encoded (and later decoded) as the values they point to (or 'null' if the pointer is nil).

Unmarshal

The Unmarshal() function takes a byte slice that hopefully represents valid JSON and a destination interface, which is typically a pointer to a struct or basic type. It deserializes the JSON into the interface in a generic way. If the serialization failed, it will return an error. Here is the signature:

func Unmarshal(data []byte, v interface{}) error

Serializing Simple Types

You can easily serialize simple types like using the json package. The result will not be a full-fledged JSON object, but a simple string. Here the int 5 is serialized to the byte array [53], which corresponds to the string "5".

    // Serialize int
    var x = 5
    bytes, err := json.Marshal(x)
    if err != nil {
        fmt.Println("Can't serislize", x)
    }

    fmt.Printf("%v => %v, '%v'\n", x, bytes, string(bytes))

    // Deserialize int
    var r int
    err = json.Unmarshal(bytes, &r)
    if err != nil {
        fmt.Println("Can't deserislize", bytes)
    }

    fmt.Printf("%v => %v\n", bytes, r)
    
Output:

- 5 => [53], '5'
- [53] => 5

If you try to serialize unsupported types like a function, you'll get an error:

    // Trying to serialize a function
    foo := func() {
        fmt.Println("foo() here")
    }

    bytes, err = json.Marshal(foo)
    if err != nil {
        fmt.Println(err)
    }

Output:

json: unsupported type: func()

Serializing Arbitrary Data With Maps

The power of JSON is that it can represent arbitrary hierarchical data very well. The JSON package supports it and utilizes the generic empty interface (interface{}) to represent any JSON hierarchy. Here is an example of deserializing and later serializing a binary tree where each node has an int value and two branches, left and right, which may contain another node or be null.

The JSON null is equivalent to the Go nil. As you can see in the output, the json.Unmarshal() function successfully converted the JSON blob to a Go data structure consisting of a nested map of interfaces and preserved the value type as int. The json.Marshal() function successfully serialized the resulting nested object to the same JSON representation.

    // Arbitrary nested JSON
    dd := `
    {
        "value": 3,
        "left": {
            "value": 1,
            "left": null,
            "right": {
                "value": 2,
                "left": null,
                "right": null
            }
        },
        "right": {
            "value": 4,
            "left": null,
            "right": null
         }
    }`

    var obj interface{}
    err = json.Unmarshal([]byte(dd), &obj)
    if err != nil {
        fmt.Println(err)
    } else {
        fmt.Println("--------\n", obj)
    }


    data, err = json.Marshal(obj)
    if err != nil {
        fmt.Println(err)
    } else {
        fmt.Println("--------\n", string(data))
    }
}

Output:

--------
 map[right:map[value:4 
               left:<nil> 
               right:<nil>] 
     value:3 
     left:map[left:<nil> 
              right:map[value:2 
                        left:<nil> 
                        right:<nil>] 
              value:1]]
--------
 {"left":{
      "left":null,
      "right":{"left":null,"right":null,"value":2},
      "value":1},
  "right":{"left":null,
           "right":null,
           "value":4},
  "value":3}

To traverse the generic maps of interfaces, you'll need to use type assertions. For example:

func dump(obj interface{}) error {
    if obj == nil {
        fmt.Println("nil")
        return nil
    }
    switch obj.(type) {
    case bool:
        fmt.Println(obj.(bool))
    case int:
        fmt.Println(obj.(int))
    case float64:
        fmt.Println(obj.(float64))
    case string:
        fmt.Println(obj.(string))
    case map[string]interface{}:
        for k, v := range(obj.(map[string]interface{})) {
            fmt.Printf("%s: ", k)
            err := dump(v)
            if err != nil {
                return err
            }
        }
    default:
        return errors.New(
            fmt.Sprintf("Unsupported type: %v", obj))
    }

    return nil
}

Serializing Structured Data

Working with structured data is often the better choice. Go provides excellent support for serializing JSON to/from structs via its struct tags. Let's create a struct that corresponds to our JSON tree and a smarter Dump() function that prints it:

type Tree struct {
    value int
    left *Tree
    right *Tree
}

func (t *Tree) Dump(indent string) {
    fmt.Println(indent + "value:", t.value)
    fmt.Print(indent + "left: ")
    if t.left == nil {
        fmt.Println(nil)
    } else {
        fmt.Println()
        t.left.Dump(indent + "  ")
    }

    fmt.Print(indent + "right: ")
    if t.right == nil {
        fmt.Println(nil)
    } else {
        fmt.Println()
        t.right.Dump(indent + "  ")
    }
}

This is great and much cleaner than the arbitrary JSON approach. But does it work? Not really. There is no error, but our tree object is not getting populated by the JSON.

    jsonTree := `
    {
        "value": 3,
        "left": {
            "value": 1,
            "left": null,
            "right": {
                "value": 2,
                "left": null,
                "right": null
            }
        },
        "right": {
            "value": 4,
            "left": null,
            "right": null
         }
    }`


    var tree Tree
    err = json.Unmarshal([]byte(dd), &tree)
    if err != nil {
        fmt.Printf("- Can't deserislize tree, error: %v\n", err)
    } else {
        tree.Dump("")
    }

Output:

value: 0
left: <nil>
right: <nil>

The problem is that the Tree fields are private. JSON serialization works on public fields only. So we can make the struct fields public. The json package is smart enough to transparently convert the lowercase keys "value", "left", and "right" to their corresponding uppercase field names.

type Tree struct {
    Value int   `json:"value"`
    Left  *Tree `json:"left"`
    Right *Tree `json:"right"`
}


Output:

value: 3
left: 
  value: 1
  left: <nil>
  right: 
    value: 2
    left: <nil>
    right: <nil>
right: 
  value: 4
  left: <nil>
  right: <nil>

The json package will silently ignore unmapped fields in the JSON as well as private fields in your struct. But sometimes you may want to map specific keys in the JSON to a field with a different name in your struct. You can use struct tags for that. For example, suppose we add another field called "label" to the JSON, but we need to map it to a field called "Tag" in our struct. 

type Tree struct {
    Value int
    Tag string    `json:"label"`
    Left  *Tree
    Right *Tree
}

func (t *Tree) Dump(indent string) {
    fmt.Println(indent + "value:", t.Value)
    if t.Tag != "" {
        fmt.Println(indent + "tag:", t.Tag)
    }
    fmt.Print(indent + "left: ")
    if t.Left == nil {
        fmt.Println(nil)
    } else {
        fmt.Println()
        t.Left.Dump(indent + "  ")
    }

    fmt.Print(indent + "right: ")
    if t.Right == nil {
        fmt.Println(nil)
    } else {
        fmt.Println()
        t.Right.Dump(indent + "  ")
    }
}

Here is the new JSON with the root node of the tree labeled as "root", serialized properly into the Tag field and printed in the output:

    dd := `
    {
        "label": "root",
        "value": 3,
        "left": {
            "value": 1,
            "left": null,
            "right": {
                "value": 2,
                "left": null,
                "right": null
            }
        },
        "right": {
            "value": 4,
            "left": null,
            "right": null
         }
    }`


    var tree Tree
    err = json.Unmarshal([]byte(dd), &tree)
    if err != nil {
        fmt.Printf("- Can't deserislize tree, error: %v\n", err)
    } else {
        tree.Dump("")
    }

Output:

value: 3
tag: root
left: 
  value: 1
  left: <nil>
  right: 
    value: 2
    left: <nil>
    right: <nil>
right: 
  value: 4
  left: <nil>
  right: <nil>

Writing a Custom Marshaller

You will often want to serialize objects that don't conform to the strict requirements of the Marshal() function. For example, you may want to serialize a map with int keys. In these cases, you can write a custom marshaller/unmarshaller by implementing the Marshaler and Unmarshaler interfaces.

A note about spelling: In Go, the convention is to name an interface with a single method by appending the "er" suffix to the method name. So, even though the more common spelling is "Marshaller" (with double L), the interface name is just "Marshaler" (single L).

Here are the Marshaler and Unmarshaler interfaces:

type Marshaler interface {
        MarshalJSON() ([]byte, error)
}

type Unmarshaler interface {
        UnmarshalJSON([]byte) error
}

You must create a type when doing custom serialization, even if you want to serialize a built-in type or composition of built-in types like map[int]string. Here I define a type called IntStringMap and implement the Marshaler and Unmarshaler interfaces for this type.

The MarshalJSON() method creates a map[string]string, converts each of its own int keys to a string, and serializes the map with string keys using the standard json.Marshal() function.

type IntStringMap map[int]string

func (m *IntStringMap) MarshalJSON() ([]byte, error) {
    ss := map[string]string{}
    for k, v := range *m {
        i := strconv.Itoa(k)
        ss[i] = v
    }
    return json.Marshal(ss)
}

The UnmarshalJSON() method does the exact opposite. It deserializes the data byte array into a map[string]string and then converts each string key to an int and populates itself.

func (m *IntStringMap) UnmarshalJSON(data []byte ) error {
    ss := map[string]string{}
    err := json.Unmarshal(data, &ss)
    if err != nil {
        return err
    }
    for k, v := range ss {
        i, err := strconv.Atoi(k)
        if err != nil {
            return err
        }
        (*m)[i] = v
    }
    return nil
}

Here is how to use it in a program:

    m := IntStringMap{4: "four", 5: "five"}
    data, err := m.MarshalJSON()
    if err != nil {
        fmt.Println(err)
    }
    fmt.Println("IntStringMap to JSON: ", string(data))


    m = IntStringMap{}

    jsonString := []byte("{\"1\": \"one\", \"2\": \"two\"}")
    m.UnmarshalJSON(jsonString)

    fmt.Printf("IntStringMap from JSON: %v\n", m)
    fmt.Println("m[1]:", m[1], "m[2]:", m[2])

Output:

IntStringMap to JSON:  {"4":"four","5":"five"}
IntStringMap from JSON: map[2:two 1:one]
m[1]: one m[2]: two

Serializing Enums

Go enums can be pretty vexing to serialize. The idea to write an article about Go json serialization came out of a question a colleague asked me about how to serialize enums. Here is a Go enum. The constants Zero and One are equal to the ints 0 and 1.

type EnumType int

const (
    Zero     EnumType = iota
    One
)

While you may think it's an int, and in many respects it is, you can't serialize it directly. You must write a custom marshaler/unmarshaler. That's not a problem after the last section. The following MarshalJSON() and UnmarshalJSON() will serialize/deserialize the constants ZERO and ONE to/from the corresponding strings "Zero" and "One".

func (e *EnumType) UnmarshalJSON(data []byte) error {
    var s string
    err := json.Unmarshal(data, &s)
    if err != nil {
        return err
    }

    value, ok := map[string]EnumType{"Zero": Zero, "One": One}[s]
    if !ok {
        return errors.New("Invalid EnumType value")
    }
    *e = value
    return nil
}

func (e *EnumType) MarshalJSON() ([]byte, error) {
    value, ok := map[EnumType]string{Zero: "Zero", One:"One"}[*e]
    if !ok {
        return nil, errors.New("Invalid EnumType value")
    }
    return json.Marshal(value)
}

Let's try to embed this EnumType in a struct and serialize it. The main function creates an EnumContainer and initializes it with a name of "Uno" and a value of our enum constant ONE, which is equal to the int 1.

type EnumContainer struct {
    Name                string
    Value               EnumType
}


func main() {
    x := One
    ec := EnumContainer{
        "Uno",
        x,
    }
    s, err := json.Marshal(ec)
    if err != nil {
        fmt.Printf("fail!")
    }

    var ec2 EnumContainer
    err = json.Unmarshal(s, &ec2)

    fmt.Println(ec2.Name, ":", ec2.Value)
}

Output:

Uno : 0

The expected output is "Uno : 1", but instead it's "Uno : 0". What happened? There is no bug in the marshal/unmarshal code. It turns out that you can't embed enums by value if you want to serialize them. You must embed a pointer to the enum. Here is a modified version where that works as expected:

type EnumContainer struct {
    Name                string
    Value               *EnumType
}

func main() {
    x := One
    ec := EnumContainer{
        "Uno",
        &x,
    }
    s, err := json.Marshal(ec)
    if err != nil {
        fmt.Printf("fail!")
    }

    var ec2 EnumContainer
    err = json.Unmarshal(s, &ec2)

    fmt.Println(ec2.Name, ":", *ec2.Value)
}

Output:

Uno : 1

Conclusion

Go provides many options for serializing and deserializing JSON. It's important to understand the ins and outs of the encoding/json package to take advantage of the power.

This tutorial put all the power in your hands, including how to serialize the elusive Go enums.

Go serialize some objects!



from Envato Tuts+ Tutorials

Comments

Post a Comment

Popular posts from this blog

How to Build Self-Confidence: 10+ Steps to Improve Your Self-Esteem

Image
Do you have friends who can walk up to someone at a party and they’d instantly hit it off? These are the people who often get free drinks, invites to cool events, and make new friends whenever they go out. You can gain confidence in yourself and improve different aspects of your life. (Image source:  Envato Elements ) Meanwhile, there are people who feel anxious in almost every social encounter. Their first day on a job, a big company meeting, and even a housewarming party all make them feel vulnerable and unsure of themselves. Questions like the following often spring to mind during these events:  “ What will I say? ”  “Who do I talk to?” “ What will I do if there’s an awkward silence? ”  While social awkwardness is common, it’s not a lifetime thing. You can learn how to build self-confidence that doesn’t feel forced or unnatural. You can still be yourself and have the self-confidence to try new things, meet new people, and all the benefits that come with a confident aura.
Read more

Best of 2020: Great Landing Pages for Design Inspiration (Showcase Examples)

Image
A great landing page design is crucial when it comes to announcing new products or services as well as the launch of your company. Great landing pages help visitors focus on a singular action. That makes them great for driving conversions and signups. Zimed is one of our best landing page design examples on Envato Elements. Nowadays, you don’t have to be a web design expert to create a landing page. There's no shortage of  quality landing page samples on various marketplaces so all you need to do is find the one you like, buy it, and set it up. That's why we've set up this landing page showcase. In today's post, we'll share our best landing page examples from Envato Elements and ThemeForest. Find Great Landing Page Design Examples on Envato Elements for 2020 Envato Elements is a great starting point when it comes to the best  landing page design . Envato Elements is a subscription-based marketplace with dozens of landing page templates. When you start bro
Read more

18+ Best Free PowerPoint Puzzle Pieces Templates (Infographic PPT Slides)

Image
Many people are visual learners. The population of visual learners is why it helps to use visual infographics to explain an idea in your PowerPoint presentation.  One of the best examples is using a PowerPoint puzzle pieces template.  Each piece of the puzzle can fit together into a singular idea. Templates like the premium  PANDAWA  template have all the PowerPoint puzzle pieces you need to explain a complex idea. Using a PowerPoint puzzle pieces will help you explain concepts more clearly. You can quickly and easily change professionally designed PowerPoint puzzles to fit your needs. This round-up will show you some of the best free PowerPoint puzzle templates. You'll also see top PowerPoint puzzle pieces templates from Envato Elements, and more puzzle pieces templates from GraphicRiver. Download the Best PowerPoint Puzzle Templates From Envato Elements Later in this tutorial, you'll see free template options. But for now, let's focus on the best of the best: pr
Read more