Go Programming Tutorial: Golang by Example
Go is a relatively new language with a number of attractive features. It’s great for writing concurrent programs, thanks to an excellent set of low-level features for handling concurrency. In many cases, though, a handful of reusable abstractions over those low-level mechanisms makes life much easier. This introductory tutorial walks you through building one such abstraction: a wrapper that can turn any data structure into a transactional service in Go.
Go is a relatively new language with a number of attractive features. It’s great for writing concurrent programs, thanks to an excellent set of low-level features for handling concurrency. In many cases, though, a handful of reusable abstractions over those low-level mechanisms makes life much easier. This introductory tutorial walks you through building one such abstraction: a wrapper that can turn any data structure into a transactional service in Go.
Brendon is a software engineer who has worked in a variety of industries, including fintech and telecommunications. He has extensive experience implementing high-availability, high-traffic web applications, and has worked with a wide range of programming languages and technologies, including Go, Python, Erlang, and Elixir.
Previous Role
Chief ArchitectPREVIOUSLY AT
What is the Go Programming Language?
The relatively new Go programming language sits neatly in the middle of the landscape, providing lots of good features and deliberately omitting many bad ones. It compiles fast, runs fast-ish, includes a runtime and garbage collection, has a simple static type system and dynamic interfaces, and an excellent standard library. This is why so many developers are keen to learn Go programming.
OOP is one of those features that Go deliberately omits. It has no subclassing, and so there are no inheritance diamonds or super calls or virtual methods to trip you up. Still, many of the useful parts of OOP are available in other ways.
*Mixins* are available by embedding structs anonymously, allowing their methods to be called directly on the containing struct (see embedding). Promoting methods in this way is called *forwarding*, and it's not the same as subclassing: the method will still be invoked on the inner, embedded struct.
Embedding also doesn't imply polymorphism. While `A` may have a `B`, that doesn't mean it is a `B` -- functions which take a `B` won't take an `A` instead. For that, we need interfaces, which we'll encounter briefly later.
Meanwhile, Golang takes a strong position on features that can lead to confusion and bugs. It omits OOP idioms such as inheritance and polymorphism, in favor of composition and simple interfaces. It downplays exception handling in favour of explicit errors in return values. There is exactly one correct way to lay out Go code, enforced by the gofmt
tool. And so on.
Why Learn Golang?
Go is also a great language for writing concurrent programs: programs with many independently running parts. An obvious example is a webserver: Every request runs separately, but requests often need to share resources such as sessions, caches, or notification queues. This means skilled Go programmers need to deal with concurrent access to those resources.
While Golang has an excellent set of low-level features for handling concurrency, using them directly can become complicated. In many cases, a handful of reusable abstractions over those low-level mechanisms makes life much easier.
In today’s Go programming tutorial, we’re going to look at one such abstraction: A wrapper which can turn any data structure into a transactional service. We’ll use a Fund
type as an example – a simple store for our startup’s remaining funding, where we can check the balance and make withdrawals.
To demonstrate this in practice, we’ll build the service in small steps, making a mess along the way and then cleaning it up again. As we progress through our Go programming tutorial, we’ll encounter lots of cool Go language features, including:
- Struct types and methods
- Unit tests and benchmarks
- Goroutines and channels
- Interfaces and dynamic typing
Go by Example: Building A Simple Fund
Let’s write some Golang code to track our startup’s funding. The fund starts with a given balance, and money can only be withdrawn (we’ll figure out revenue later).
Go is deliberately not an object-oriented language: There are no classes, objects, or inheritance. Instead, we’ll declare a struct type called Fund
, with a simple function to create new fund structs, and two public methods.
fund.go
package funding
type Fund struct {
// balance is unexported (private), because it's lowercase
balance int
}
// A regular function returning a pointer to a fund
func NewFund(initialBalance int) *Fund {
// We can return a pointer to a new struct without worrying about
// whether it's on the stack or heap: Go figures that out for us.
return &Fund{
balance: initialBalance,
}
}
// Methods start with a *receiver*, in this case a Fund pointer
func (f *Fund) Balance() int {
return f.balance
}
func (f *Fund) Withdraw(amount int) {
f.balance -= amount
}
Testing With Benchmarks
Next we need a way to test Fund
. Rather than writing a separate program, we’ll use Go’s testing package, which provides a framework for both unit tests and benchmarks. The simple logic in our Fund
isn’t really worth writing unit tests for, but since we’ll be talking a lot about concurrent access to the fund later on, writing a benchmark makes sense.
Benchmarks are like unit tests, but include a loop which runs the same code many times (in our case, fund.Withdraw(1)
). This allows the framework to time how long each iteration takes, averaging out transient differences from disk seeks, cache misses, process scheduling, and other unpredictable factors.
The testing framework wants each benchmark to run for at least 1 second (by default). To ensure this, it will call the benchmark multiple times, passing in an increasing “number of iterations” value each time (the b.N
field), until the run takes at least a second.
For now, our benchmark will just deposit some money and then withdraw it one dollar at a time.
fund_test.go
package funding
import "testing"
func BenchmarkFund(b *testing.B) {
// Add as many dollars as we have iterations this run
fund := NewFund(b.N)
// Burn through them one at a time until they are all gone
for i := 0; i < b.N; i++ {
fund.Withdraw(1)
}
if fund.Balance() != 0 {
b.Error("Balance wasn't zero:", fund.Balance())
}
}
Now let’s run it:
$ go test -bench . funding
testing: warning: no tests to run
PASS
BenchmarkWithdrawals 2000000000 1.69 ns/op
ok funding 3.576s
That went well. We ran two billion (!) iterations, and the final check on the balance was correct. We can ignore the “no tests to run” warning, which refers to the unit tests we didn’t write (in later Go programming examples in this Golang tutorial, the warning is snipped out).
Concurrent Access in Go
Now let’s make the benchmark concurrent, to model different users making withdrawals at the same time. To do that, we’ll spawn ten goroutines and have each of them withdraw one tenth of the money.
Goroutines are the basic building block for concurrency in the Go language. They are green threads – lightweight threads managed by the Go runtime, not by the operating system. This means you can run thousands (or millions) of them without any significant overhead. Goroutines are spawned with the go
keyword, and always start with a function (or method call):
// Returns immediately, without waiting for `DoSomething()` to complete
go DoSomething()
Often, we want to spawn off a short one-time function with just a few lines of code. In this case we can use a closure instead of a function name:
go func() {
// ... do stuff ...
}() // Must be a function *call*, so remember the ()
Once all our goroutines are spawned, we need a way to wait for them to finish. We could build one ourselves using channels, but we haven’t encountered those yet, so that would be skipping ahead.
For now, we can just use the WaitGroup
type in Go’s standard library, which exists for this very purpose. We’ll create one (called “wg
”) and call wg.Add(1)
before spawning each worker, to keep track of how many there are. Then the workers will report back using wg.Done()
. Meanwhile in the main goroutine, we can just say wg.Wait()
to block until every worker has finished.
Inside the worker goroutines in our next example, we’ll use defer
to call wg.Done()
.
defer
takes a function (or method) call and runs it immediately before the current function returns, after everything else is done. This is handy for cleanup:
func() {
resource.Lock()
defer resource.Unlock()
// Do stuff with resource
}()
This way we can easily match the Unlock
with its Lock
, for readability. More importantly, a deferred function will run even if there is a panic in the main function (something that we might handle via try-finally in other languages).
Lastly, deferred functions will execute in the reverse order to which they were called, meaning we can do nested cleanup nicely (similar to the C idiom of nested goto
s and label
s, but much neater):
func() {
db.Connect()
defer db.Disconnect()
// If Begin panics, only db.Disconnect() will execute
transaction.Begin()
defer transaction.Close()
// From here on, transaction.Close() will run first,
// and then db.Disconnect()
// ...
}()
OK, so with all that said, here’s the new version:
fund_test.go
package funding
import (
"sync"
"testing"
)
const WORKERS = 10
func BenchmarkWithdrawals(b *testing.B) {
// Skip N = 1
if b.N < WORKERS {
return
}
// Add as many dollars as we have iterations this run
fund := NewFund(b.N)
// Casually assume b.N divides cleanly
dollarsPerFounder := b.N / WORKERS
// WaitGroup structs don't need to be initialized
// (their "zero value" is ready to use).
// So, we just declare one and then use it.
var wg sync.WaitGroup
for i := 0; i < WORKERS; i++ {
// Let the waitgroup know we're adding a goroutine
wg.Add(1)
// Spawn off a founder worker, as a closure
go func() {
// Mark this worker done when the function finishes
defer wg.Done()
for i := 0; i < dollarsPerFounder; i++ {
fund.Withdraw(1)
}
}() // Remember to call the closure!
}
// Wait for all the workers to finish
wg.Wait()
if fund.Balance() != 0 {
b.Error("Balance wasn't zero:", fund.Balance())
}
}
We can predict what will happen here. The workers will all execute Withdraw
on top of each other. Inside it, f.balance -= amount
will read the balance, subtract one, and then write it back. But sometimes two or more workers will both read the same balance, and do the same subtraction, and we’ll end up with the wrong total. Right?
$ go test -bench . funding
BenchmarkWithdrawals 2000000000 2.01 ns/op
ok funding 4.220s
No, it still passes. What happened here?
Remember that goroutines are green threads – they’re managed by the Go runtime, not by the OS. The runtime schedules goroutines across however many OS threads it has available. At the time of writing this Go language tutorial, Go doesn’t try to guess how many OS threads it should use, and if we want more than one, we have to say so. Finally, the current runtime does not preempt goroutines – a goroutine will continue to run until it does something that suggests it’s ready for a break (like interacting with a channel).
All of this means that although our benchmark is now concurrent, it isn’t parallel. Only one of our workers will run at a time, and it will run until it’s done. We can change this by telling Go to use more threads, via the GOMAXPROCS
environment variable.
$ GOMAXPROCS=4 go test -bench . funding
BenchmarkWithdrawals-4 --- FAIL: BenchmarkWithdrawals-4
account_test.go:39: Balance wasn't zero: 4238
ok funding 0.007s
That’s better. Now we’re obviously losing some of our withdrawals, as we expected.
Make it a Server
At this point we have various options. We could add an explicit mutex or read-write lock around the fund. We could use a compare-and-swap with a version number. We could go all out and use a CRDT scheme (perhaps replacing the balance
field with lists of transactions for each client, and calculating the balance from those).
But we won’t do any of those things now, because they’re messy or scary or both. Instead, we’ll decide that a fund should be a server. What’s a server? It’s something you talk to. In Go, things talk via channels.
Channels are the basic communication mechanism between goroutines. Values are sent to the channel (with channel <- value
), and can be received on the other side (with value = <- channel
). Channels are “goroutine safe”, meaning that any number of goroutines can send to and receive from them at the same time.
Buffering communication channels can be a performance optimization in certain circumstances, but it should be used with great care (and benchmarking!).
However, there are uses for buffered channels which aren't directly about communication.
For instance, a common throttling idiom creates a channel with (for example) buffer size `10` and then sends ten tokens into it immediately. Any number of worker goroutines are then spawned, and each receives a token from the channel before starting work, and sends it back afterward. Then, however many workers there are, only ten will ever be working at the same time.
By default, Go channels are unbuffered. This means that sending a value to a channel will block until another goroutine is ready to receive it immediately. Go also supports fixed buffer sizes for channels (using make(chan someType, bufferSize)
). However, for normal use, this is usually a bad idea.
Imagine a webserver for our fund, where each request makes a withdrawal. When things are very busy, the FundServer
won’t be able to keep up, and requests trying to send to its command channel will start to block and wait. At that point we can enforce a maximum request count in the server, and return a sensible error code (like a 503 Service Unavailable
) to clients over that limit. This is the best behavior possible when the server is overloaded.
Adding buffering to our channels would make this behavior less deterministic. We could easily end up with long queues of unprocessed commands based on information the client saw much earlier (and perhaps for requests which had since timed out upstream). The same applies in many other situations, like applying backpressure over TCP when the receiver can’t keep up with the sender.
In any case, for our Go example, we’ll stick with the default unbuffered behavior.
We’ll use a channel to send commands to our FundServer
. Every benchmark worker will send commands to the channel, but only the server will receive them.
We could turn our Fund type into a server implementation directly, but that would be messy – we’d be mixing concurrency handling and business logic. Instead, we’ll leave the Fund type exactly as it is, and make FundServer
a separate wrapper around it.
Like any server, the wrapper will have a main loop in which it waits for commands, and responds to each in turn. There’s one more detail we need to address here: The type of the commands.
We could have made our commands channel take *pointers* to commands (`chan *TransactionCommand`). Why didn't we?
Passing pointers between goroutines is risky, because either goroutine might modify it. It's also often less efficient, because the other goroutine might be running on a different CPU core (meaning more cache invalidation).
Whenever possible, prefer to pass plain values around.
In the next section below, we’ll be sending several different commands, each with its own struct type. We want the server’s Commands channel to accept any of them. In an OOP language we might do this via polymorphism: Have the channel take a superclass, of which the individual command types were subclasses. In Go, we use interfaces instead.
An interface is a set of method signatures. Any type that implements all of those methods can be treated as that interface (without being declared to do so). For our first run, our command structs won’t actually expose any methods, so we’re going to use the empty interface, interface{}
. Since it has no requirements, any value (including primitive values like integers) satisfies the empty interface. This isn’t ideal – we only want to accept command structs – but we’ll come back to it later.
For now, let’s get started with the scaffolding for our Go server:
server.go
package funding
type FundServer struct {
Commands chan interface{}
fund Fund
}
func NewFundServer(initialBalance int) *FundServer {
server := &FundServer{
// make() creates builtins like channels, maps, and slices
Commands: make(chan interface{}),
fund: NewFund(initialBalance),
}
// Spawn off the server's main loop immediately
go server.loop()
return server
}
func (s *FundServer) loop() {
// The built-in "range" clause can iterate over channels,
// amongst other things
for command := range s.Commands {
// Handle the command
}
}
Now let’s add a couple of Golang struct types for the commands:
type WithdrawCommand struct {
Amount int
}
type BalanceCommand struct {
Response chan int
}
The WithdrawCommand
just contains the amount to withdraw. There’s no response. The BalanceCommand
does have a response, so it includes a channel to send it on. This ensures that responses will always go to the right place, even if our fund later decides to respond out-of-order.
Now we can write the server’s main loop:
func (s *FundServer) loop() {
for command := range s.Commands {
// command is just an interface{}, but we can check its real type
switch command.(type) {
case WithdrawCommand:
// And then use a "type assertion" to convert it
withdrawal := command.(WithdrawCommand)
s.fund.Withdraw(withdrawal.Amount)
case BalanceCommand:
getBalance := command.(BalanceCommand)
balance := s.fund.Balance()
getBalance.Response <- balance
default:
panic(fmt.Sprintf("Unrecognized command: %v", command))
}
}
}
Hmm. That’s sort of ugly. We’re switching on the command type, using type assertions, and possibly crashing. Let’s forge ahead anyway and update the benchmark to use the server.
func BenchmarkWithdrawals(b *testing.B) {
// ...
server := NewFundServer(b.N)
// ...
// Spawn off the workers
for i := 0; i < WORKERS; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < dollarsPerFounder; i++ {
server.Commands <- WithdrawCommand{ Amount: 1 }
}
}()
}
// ...
balanceResponseChan := make(chan int)
server.Commands <- BalanceCommand{ Response: balanceResponseChan }
balance := <- balanceResponseChan
if balance != 0 {
b.Error("Balance wasn't zero:", balance)
}
}
That was sort of ugly too, especially when we checked the balance. Never mind. Let’s try it:
$ GOMAXPROCS=4 go test -bench . funding
BenchmarkWithdrawals-4 5000000 465 ns/op
ok funding 2.822s
Much better, we’re no longer losing withdrawals. But the code is getting hard to read, and there are more serious problems. If we ever issue a BalanceCommand
and then forget to read the response, our fund server will block forever trying to send it. Let’s clean things up a bit.
Make it a Service
A server is something you talk to. What’s a service? A service is something you talk to with an API. Instead of having client code work with the command channel directly, we’ll make the channel unexported (private) and wrap the available commands up in functions.
type FundServer struct {
commands chan interface{} // Lowercase name, unexported
// ...
}
func (s *FundServer) Balance() int {
responseChan := make(chan int)
s.commands <- BalanceCommand{ Response: responseChan }
return <- responseChan
}
func (s *FundServer) Withdraw(amount int) {
s.commands <- WithdrawCommand{ Amount: amount }
}
Now our benchmark can just say server.Withdraw(1)
and balance := server.Balance()
, and there’s less chance of accidentally sending it invalid commands or forgetting to read responses.
There’s still a lot of extra boilerplate for the commands, but we’ll come back to that later.
Transactions
Eventually, the money always runs out. Let’s agree that we’ll stop withdrawing when our fund is down to its last ten dollars, and spend that money on a communal pizza to celebrate or commiserate around. Our benchmark will reflect this:
// Spawn off the workers
for i := 0; i < WORKERS; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < dollarsPerFounder; i++ {
// Stop when we're down to pizza money
if server.Balance() <= 10 {
break
}
server.Withdraw(1)
}
}()
}
// ...
balance := server.Balance()
if balance != 10 {
b.Error("Balance wasn't ten dollars:", balance)
}
This time we really can predict the result.
$ GOMAXPROCS=4 go test -bench . funding
BenchmarkWithdrawals-4 --- FAIL: BenchmarkWithdrawals-4
fund_test.go:43: Balance wasn't ten dollars: 6
ok funding 0.009s
We’re back where we started – several workers can read the balance at once, and then all update it. To deal with this we could add some logic in the fund itself, like a minimumBalance
property, or add another command called WithdrawIfOverXDollars
. These are both terrible ideas. Our agreement is amongst ourselves, not a property of the fund. We should keep it in application logic.
What we really need is transactions, in the same sense as database transactions. Since our service executes only one command at a time, this is super easy. We’ll add a Transact
command which contains a callback (a closure). The server will execute that callback inside its own goroutine, passing in the raw Fund
. The callback can then safely do whatever it likes with the Fund
.
In this next example we're doing two small things wrong.
First, we're using a `Done` channel as a semaphore to let calling code know when its transaction has finished. That's fine, but why is the channel type `bool`? We'll only ever send `true` into it to mean "done" (what would sending `false` even mean?). What we really want is a single-state value (a value that has no value?). In Go, we can do this using the empty struct type: `struct{}`. This also has the advantage of using less memory. In the example we'll stick with `bool` so as not to look too scary.
Second, our transaction callback isn't returning anything. As we'll see in a moment, we can get values out of the callback into calling code using scope tricks. However, transactions in a real system would presumably fail sometimes, so the Go convention would be to have the transaction return an `error` (and then check whether it was `nil` in calling code).
We're not doing that either for now, since we don't have any errors to generate.
// Typedef the callback for readability
type Transactor func(fund *Fund)
// Add a new command type with a callback and a semaphore channel
type TransactionCommand struct {
Transactor Transactor
Done chan bool
}
// ...
// Wrap it up neatly in an API method, like the other commands
func (s *FundServer) Transact(transactor Transactor) {
command := TransactionCommand{
Transactor: transactor,
Done: make(chan bool),
}
s.commands <- command
<- command.Done
}
// ...
func (s *FundServer) loop() {
for command := range s.commands {
switch command.(type) {
// ...
case TransactionCommand:
transaction := command.(TransactionCommand)
transaction.Transactor(s.fund)
transaction.Done <- true
// ...
}
}
}
Our transaction callbacks don’t directly return anything, but the Go language makes it easy to get values out of a closure directly, so we’ll do that in the benchmark to set the pizzaTime
flag when money runs low:
pizzaTime := false
for i := 0; i < dollarsPerFounder; i++ {
server.Transact(func(fund *Fund) {
if fund.Balance() <= 10 {
// Set it in the outside scope
pizzaTime = true
return
}
fund.Withdraw(1)
})
if pizzaTime {
break
}
}
And check that it works:
$ GOMAXPROCS=4 go test -bench . funding
BenchmarkWithdrawals-4 5000000 775 ns/op
ok funding 4.637s
Nothing But transactions
You may have spotted an opportunity to clean things up some more now. Since we have a generic Transact
command, we don’t need WithdrawCommand
or BalanceCommand
anymore. We’ll rewrite them in terms of transactions:
func (s *FundServer) Balance() int {
var balance int
s.Transact(func(f *Fund) {
balance = f.Balance()
})
return balance
}
func (s *FundServer) Withdraw(amount int) {
s.Transact(func (f *Fund) {
f.Withdraw(amount)
})
}
Now the only command the server takes is TransactionCommand
, so we can remove the whole interface{}
mess in its implementation, and have it accept only transaction commands:
type FundServer struct {
commands chan TransactionCommand
fund *Fund
}
func (s *FundServer) loop() {
for transaction := range s.commands {
// Now we don't need any type-switch mess
transaction.Transactor(s.fund)
transaction.Done <- true
}
}
Much better.
There’s a final step we could take here. Apart from its convenience functions for Balance
and Withdraw
, the service implementation is no longer tied to Fund
. Instead of managing a Fund
, it could manage an interface{}
and be used to wrap anything. However, each transaction callback would then have to convert the interface{}
back to a real value:
type Transactor func(interface{})
server.Transact(func(managedValue interface{}) {
fund := managedValue.(*Fund)
// Do stuff with fund ...
})
This is ugly and error-prone. What we really want is compile-time generics, so we can “template” out a server for a particular type (like *Fund
).
Unfortunately, Go doesn’t support generics – yet. It’s expected to arrive eventually, once someone figures out some sensible Golang syntax and semantics for it. In the meantime, careful interface design often removes the need for generics, and when they don’t we can get by with type assertions (which are checked at runtime).
Are We Done?
Yes.
Well, okay, no.
For instance:
-
A panic in a transaction will kill the whole service.
-
There are no timeouts. A transaction that never returns will block the service forever.
-
If our Fund grows some new fields and a transaction crashes halfway through updating them, we’ll have inconsistent state.
-
Transactions are able to leak the managed
Fund
object, which isn’t good. -
There’s no reasonable way to do transactions across multiple funds (like withdrawing from one and depositing in another). We can’t just nest our transactions because it would allow deadlocks.
-
Running a transaction asynchronously now requires a new goroutine and a lot of messing around. Relatedly, we probably want to be able to read the most recent
Fund
state from elsewhere while a long-running transaction is in progress.
These issues and more will be covered in a future Go programming tutorial.
Further Reading on the Toptal Blog:
Understanding the basics
What is the Go language written in?
The Go Programming Language Specification is a document written in English, while Go’s standard library and compiler are written in Go itself.
What is Go used for?
Go is a general-purpose programming language and can be used for a variety of uses. Go can be used as a web-server in the back-end, and has been used to build Docker, Kubernetes, and the Heroku CLI.
What companies are using Go?
Go is used by a large number of reputable tech companies, most notably Google, Dropbox, Docker, Kubernetes, Heroku, and many more.
Taipei, Taiwan
Member since October 16, 2013
About the author
Brendon is a software engineer who has worked in a variety of industries, including fintech and telecommunications. He has extensive experience implementing high-availability, high-traffic web applications, and has worked with a wide range of programming languages and technologies, including Go, Python, Erlang, and Elixir.
Previous Role
Chief ArchitectPREVIOUSLY AT