Building Gokey A Key-Value Store on Raft

In my quest to learn distributed systems, I wanted to implement a simple key-value store on top of the Raft protocol. HashiCorp provides a solid production-ready Raft implementation in Go, hashicorp/raft, and Gokey builds on top of it.

Gokey is a small replicated key-value store. The storage layer is intentionally simple: it is a Go map[string]string protected by a mutex. The interesting part is not the map itself, but how writes reach the map.

In Gokey, SET and DELETE requests do not directly mutate the map. They are first converted into commands, submitted to Raft, replicated through the cluster, committed, and then applied to the key-value store’s finite state machine.

That was the main idea I wanted to understand while building this project: Raft does not know anything about key-value stores. It only knows how to replicate an ordered log. The application decides what each log entry means.

Gokey’s Architecture

Gokey has three main parts:

• HTTP API: Accepts requests for key-value operations and cluster membership changes.
• KeyStore: Exposes methods like Set, Get, Delete, and GetOrCreate.
• Raft adapter and FSM: Connects the application state to hashicorp/raft.

The entry point accepts four important flags:

• -node-id: unique ID of the Raft node.
• -address: TCP address used by Raft to communicate with other nodes.
• -http-port: HTTP port used by clients.
• -bootstrap: whether this node should create the initial cluster.

The bootstrap flag is only needed once for the lifetime of a cluster. In Gokey, the node started with -bootstrap creates the initial Raft configuration with itself as a voter.

The KeyStore

The core application type is KeyStore.

It contains:

• a raft *raft.Raft, which is Gokey’s wrapper around hashicorp/raft.
• a data map[string]string, which stores the actual key-value data.
• a mutex, which protects access to the map.

When a new KeyStore is created, Gokey also creates the FSM that Raft will call after log entries are committed.

store := &KeyStore{}
fsm := (*KeyStoreFSM)(store)
store.data = make(map[string]string)
 
raft := raft.NewRaft(nodeID, address, fsm)
store.raft = raft

This is the important connection: the same store that holds the map also implements the Raft FSM behavior through KeyStoreFSM.

Write Path

The most important path in Gokey is the write path.

When a client sends a SET request to /key-store, the HTTP handler validates the request and calls Keystore.Set.

Set first checks whether the current node is the leader:

if !store.raft.IsLeader() {
	return errors.New("request denied, not a leader")
}

If the node is not the leader, Gokey rejects the request. It does not currently forward the request to the leader.

If the node is the leader, Gokey builds a command:

cmd.Operation = SET
cmd.Key = key
cmd.Value = value

Then it marshals the command to JSON and calls raft.Apply.

raw_cmd, err := getRawCommand(cmd)
fut := store.raft.Apply(raw_cmd)

At this point, the key has still not been written to the local map. The command has only been submitted to Raft.

Raft then replicates the command to the other nodes. Once the log entry is stored by a quorum and considered committed, HashiCorp Raft calls Gokey’s FSM Apply method on the leader. The Apply future returns after the command has been applied to the leader’s FSM.

This does not mean every follower has already applied the command to its own FSM. Raft needs a quorum to commit, not every node. Followers that were slow, disconnected, or simply behind can catch up later through normal log replication.

The full write path looks like this:

HTTP request
  -> KeyStore.Set
  -> JSON command
  -> raft.Apply
  -> replicated Raft log
  -> committed log entry
  -> KeyStoreFSM.Apply
  -> map update

This is the part that made the Raft state machine model click for me. The map is not directly shared between nodes. Instead, each caught-up node has applied the same ordered commands to its own local map.

Commands

Gokey models client operations using a small command struct:

type Command struct {
	Operation Operation `json:"command"`
	Key       string    `json:"key"`
	Value     string    `json:"value,omitempty"`
}

The supported operations are:

• GET
• SET
• DELETE
• GET_OR_CREATE

Only SET and DELETE are applied through the FSM. GET reads from the local map. This is intentional. If every read had to go to the leader, replicas would mostly help with failover, but they would not help much with serving read traffic.

GET_OR_CREATE first tries a local GET; if the key does not exist, it performs a replicated SET.

This distinction matters. Gokey uses Raft to order and replicate mutations, but it lets any node serve reads from its local copy of the state.

The FSM

The FSM is the application-specific part of Raft.

HashiCorp Raft handles leader election, log replication, and committing entries. Gokey provides the state machine that knows what to do with committed entries.

In Gokey, the FSM decodes the log entry back into a command:

cmd, err := GetCommand(log.Data)

Then it applies the operation:

switch cmd.Operation {
case SET:
	store.applySet(cmd.Key, cmd.Value)
case DELETE:
	store.applyDel(cmd.Key)
}

applySet and applyDel are simple map operations protected by a mutex:

store.data[key] = value
delete(store.data, key)

The simplicity is the point. Raft gives caught-up nodes the same committed log entries in the same order. Since the FSM is deterministic, those nodes should end up with the same map state.

Raft Adapter

Gokey wraps hashicorp/raft behind a small adapter.

The adapter is responsible for:

• creating the Raft node.
• creating the TCP transport.
• bootstrapping the first cluster member.
• adding new voters.
• applying commands to Raft.

The Raft node is created with HashiCorp’s default config:

config := raft_lib.DefaultConfig()
config.LocalID = raft_lib.ServerID(raftID)

Gokey then creates a TCP transport and passes the FSM, log store, stable store, snapshot store, and transport into raft.NewRaft.

node, err := raft_lib.NewRaft(
	config,
	fsm,
	LogStore,
	StableStore,
	SnapshotStore,
	transport,
)

This means Gokey is not implementing Raft from scratch. It is implementing the application layer around Raft: commands, state machine behavior, HTTP API, and cluster wiring.

Storage Used By Raft

HashiCorp Raft needs a few storage components:

• Log store: stores Raft log entries.
• Stable store: stores durable Raft metadata.
• Snapshot store: stores compacted FSM snapshots.

Gokey uses raft-boltdb for the log store and stable store. It uses HashiCorp’s file snapshot store for snapshots.

In the current code, the file paths are generated with a random suffix:

LogStoreFilePath     = "logstore.db" + fmt.Sprintf("%d", rand.Int())
StableStoreFilePath  = "stablestore.db" + fmt.Sprintf("%d", rand.Int())
SnapshotStoreBaseDir = "snapshot_data/" + fmt.Sprintf("%d", rand.Int())

This is useful for local experimentation because each run gets separate files. For a production-style store, these paths should be explicit and stable so a node can restart with its previous Raft state.

Snapshots

Raft logs cannot grow forever. At some point, the current state should be snapshotted so older logs can be compacted.

Gokey implements snapshots by serializing the entire key-value map to JSON.

data, err := json.Marshal(data_map)

The snapshot object then writes those bytes into the Raft snapshot sink:

_, err := sink.Write(snap.data)

This is a simple snapshot strategy because the whole state is just a map. A more complex database would need a more careful snapshot format.

Adding Nodes

Gokey exposes cluster membership through the /add-replica API.

The handler accepts a node ID and Raft address:

{
  "node_id": "B",
  "address": "localhost:8001"
}

Then it calls:

Keystore.Replicate(nodeID, addr)

Internally, this calls HashiCorp Raft’s AddVoter:

future := r.node.AddVoter(serverID, serverAddress, prevIndex, AddVoterTimeoutDuration)

This must be done through the leader. Once the node is added as a voter, Raft is responsible for bringing it up to date with the cluster.

Local Reads And Consistency

Gokey’s write operations are replicated through Raft, but GET is a local read.

That is a reasonable design choice for a replicated key-value store. Once replicas maintain their own copies of the state machine, reads do not have to be routed through the leader for every request. This gives replicas a real purpose beyond just waiting to become leader after a failure.

But the exact guarantee matters.

A successful SET followed by a GET on the same leader should observe the write, because Set waits for Raft’s Apply future and HashiCorp Raft returns that future after the command has been applied to the leader’s FSM.

A GET sent to an arbitrary follower has a weaker guarantee. The follower reads whatever has already been applied to its local map. Usually that follower will be close to the leader, but Raft does not require every follower to apply every committed entry before a write returns to the client.

This means Gokey’s local reads are fast and useful, but they are not linearizable reads from arbitrary nodes.

If Gokey wanted stronger read semantics, there are a few options:

• send strongly consistent reads to the leader.
• have the leader verify it is still leader before serving a local read.
• use a Raft barrier or read-index style mechanism before reading local state.
• expose the current leader address so clients can retry writes and strong reads against the leader.

The lesson for me was that replication and read semantics are separate choices. Raft gives Gokey a consistent order for writes. The application still has to decide what kind of reads it wants to expose.

Current Limitations

Gokey is still a learning project, not production infrastructure.

Some important limitations and tradeoffs are:

• non-leader writes are rejected instead of being redirected to the leader.
• reads are intentionally local, which is good for read scaling, but they are not linearizable when served from arbitrary followers.
• GET_OR_CREATE is not atomic because it performs a local read before a replicated write.
• Raft storage paths are generated randomly, so node restarts are not durable in the way a real deployment would need.
• snapshot restore needs more care before it can be relied on.

These limitations are not failures of Raft. They are the parts around Raft that a real database still has to design carefully.

Gokey Usage

Starting The Cluster

Start nodes A, B, and C. Node A is started with the -bootstrap flag to create the initial Raft cluster.

Terminal 1:

go run cmd/gokey.go -node-id A -address localhost:8000 -http-port 9000 -bootstrap

Terminal 2:

go run cmd/gokey.go -node-id B -address localhost:8001 -http-port 9001

Terminal 3:

go run cmd/gokey.go -node-id C -address localhost:8002 -http-port 9002

Other nodes can be added through Gokey’s /add-replica API.

Adding Nodes To The Cluster

Add node B:

curl --location 'localhost:9000/add-replica' \
  --header 'Content-Type: application/json' \
  --data '{
    "node_id": "B",
    "address": "localhost:8001"
  }'

Add node C:

curl --location 'localhost:9000/add-replica' \
  --header 'Content-Type: application/json' \
  --data '{
    "node_id": "C",
    "address": "localhost:8002"
  }'

Using The Key Store API

Set a key:

curl --location 'localhost:9000/key-store' \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "SET",
    "key": "FOO",
    "value": "BAR"
  }'

Response:

{"created":true}

Get a key:

curl --location 'localhost:9000/key-store' \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "GET",
    "key": "FOO"
  }'

Response:

{"value":"BAR"}

Delete a key:

curl --location 'localhost:9000/key-store' \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "DELETE",
    "key": "FOO"
  }'

Get or create a key:

curl --location 'localhost:9000/key-store' \
  --header 'Content-Type: application/json' \
  --data '{
    "command": "GET_OR_CREATE",
    "key": "FOO",
    "value": "BAR"
  }'

Response when the key is created:

{"created":true,"value":"BAR"}

Gokey makes the write path consistent by routing mutations through Raft. The project helped me understand that using Raft is not just about leader election or replication. The important application work is designing the commands, the FSM, the read semantics, the snapshot format, and the operational behavior around the Raft library.

References

• Gokey
• In Search of an Understandable Consensus Algorithm: The Raft paper
• raft.tla: Diego Ongaro’s TLA+ spec for Raft
• HashiCorp’s Raft implementation in Go
• Raft gRPC example: example implementation of HashiCorp Raft using gRPC