Networking I: a blocking TCP server

Lesson 7 of 12 — C From Scratch

Last lesson we gave the store a wire protocol — a length-prefixed RESP grammar and a pure parser that turned bytes into commands. We proved it worked by piping frames through stdin/stdout. That was a rehearsal. The reason a wire protocol exists is that the wire is a socket. So this lesson: we open a TCP port, accept a client, and speak the same protocol we already tested — over the network.

We're going to build the simplest possible server: one client at a time, blocking I/O, no threads. It's not what you'd ship to production, but it is the shape every network server has. Once you've felt each system call once, epoll and io_uring are just faster ways to spell the same story.

The five system calls that make a server

Every TCP server on Linux — from sshd to Redis to your browser's dev tools — is built on the same five calls:

Call	What it does
socket()	Ask the kernel for a fresh, unbound endpoint. Get back a file descriptor.
bind()	Attach that endpoint to an address:port on this machine.
listen()	Flip the endpoint from "unused" to "passive listener" and set a backlog.
accept()	Block until a client connects. Return a new fd for that connection.
read/write	Talk to the client. The listen fd stays put; the conn fd is per-client.

Notice what's not there. There is no "handle a request" call — that's your code, sitting on top of read()/write(). The kernel gives you a byte pipe; framing, dispatch, and replies are yours.

Sketch first

Here's the whole server, in prose, before we look at code:

1. Make a listen socket, mark it SO_REUSEADDR so a fast restart doesn't hit "Address already in use", bind() it to 127.0.0.1:6380, and listen().
2. accept() a client. You now have a per-connection fd.
3. Read into a growable buffer. Feed the buffer to proto_parse() in a loop. Each PARSE_OK = one complete frame → run proto_dispatch(). Each PARSE_NEED_MORE = "I have a valid prefix, keep reading." Each PARSE_ERR = "the client is speaking gibberish, hang up."
4. Flush the reply back. Compact the buffer (shift the unparsed tail to the front).
5. When read() returns 0, the client closed cleanly. Close the fd, go back to step 2.

That's it. And step 3 is why lesson 6 mattered: the parser doesn't care whether the bytes came from stdin or a socket. It's the same call.

New includes

We reach for four new headers. This block goes near the top of kv.c, next to the includes we already had:

#include <arpa/inet.h>     /* htons, htonl, inet_ntoa                     */
#include <netinet/in.h>    /* struct sockaddr_in, INADDR_LOOPBACK         */
#include <signal.h>        /* signal(), SIGPIPE                           */
#include <sys/socket.h>    /* socket, bind, listen, accept, setsockopt    */

Reading and writing, carefully

read(2) and write(2) are the two most-misused calls in POSIX. Two rules save you a lot of pain:

Rule 1: partial write() is normal. write(fd, buf, 4096) is allowed to return 1024 and mean "I took the first kilobyte, call me again for the rest." Treating that return value as "done" is the classic bug. So we wrap it in a loop:

/* Drain n bytes at p to fd. Loops around partial writes: write(2) is
 * allowed to accept fewer bytes than we asked, and treating that as
 * "done" is a classic bug. Returns 0 on success, -1 on error. */
static int conn_write_all(int fd, const char *p, size_t n) {
    while (n > 0) {
        ssize_t w = write(fd, p, n);
        if (w < 0) {
            if (errno == EINTR) continue;
            return -1;
        }
        p += w;
        n -= (size_t)w;
    }
    return 0;
}

Rule 2: EINTR is not an error. If a signal interrupts a blocking syscall, it returns -1 with errno == EINTR. That doesn't mean "your I/O failed" — it means "we were interrupted, ask again." Retry transparently. Our reader looks like this:

/* Read a chunk from fd and append it to buf. Returns bytes read, 0 on
 * clean EOF, -1 on error. Handles EINTR (a signal interrupted us) by
 * retrying transparently. */
static ssize_t conn_read(int fd, struct outbuf *buf) {
    char chunk[4096];
    ssize_t n;
    do {
        n = read(fd, chunk, sizeof chunk);
    } while (n < 0 && errno == EINTR);
    if (n <= 0) return n;
    if (outbuf_append(buf, chunk, (size_t)n) < 0) {
        errno = ENOMEM;
        return -1;
    }
    return n;
}

Two other kernel-level gotchas we quietly disarm:

• SIGPIPE: if the client hangs up mid-write(), the kernel's default is to kill our process with SIGPIPE. That's a nasty surprise for a server. We install signal(SIGPIPE, SIG_IGN) so we get EPIPE from write() instead, which we can handle like any other error.
• SO_REUSEADDR: after a connection closes, its socket lingers in TIME_WAIT holding the local port for ~60 seconds on Linux (a fixed TCP_TIMEWAIT_LEN; classically 2·MSL, i.e. up to 4 minutes) so late packets from the old connection can't confuse a new one — which means a fresh bind() on the listener hits "Address already in use". Great for correctness, awful for iteration. SO_REUSEADDR says "let me re-bind anyway."

The per-connection loop

This is the heart. Read the comments — the shape here is the same shape you'll see in every framed protocol you ever write:

/*
 * Serve one accepted client until it disconnects or misbehaves.
 *
 * The core is the same three-way parse result we already relied on in
 * --test-proto mode:
 *   OK        -> dispatch and consume the frame
 *   NEED_MORE -> read more bytes from the socket, don't touch the reply
 *   ERR       -> the client sent garbage; send an error and hang up
 *
 * After each parse pass we compact the input buffer by shifting the
 * unparsed tail back to offset 0. That keeps memory bounded even for a
 * long-lived connection sending many small commands.
 */
static void serve_one(int cfd) {
    struct outbuf in  = {0};
    struct outbuf out = {0};

    for (;;) {
        /* 1. Try to parse and dispatch every complete frame we have. */
        size_t pos = 0;
        int    fatal = 0;
        while (pos < in.len) {
            struct parsed_cmd cmd;
            size_t used = 0;
            int r = proto_parse(in.data + pos, in.len - pos, &used, &cmd);
            if (r == PARSE_NEED_MORE) break;
            if (r == PARSE_ERR) {
                proto_write_err(&out, "malformed frame");
                fatal = 1;
                pos = in.len;
                break;
            }
            proto_dispatch(&cmd, &out);
            pos += used;
        }

        /* 2. Send whatever replies we accumulated. */
        if (out.len > 0) {
            if (conn_write_all(cfd, out.data, out.len) < 0) break;
            out.len = 0;
        }
        if (fatal) break;

        /* 3. Compact the buffer: keep only the unparsed tail. */
        if (pos > 0) {
            memmove(in.data, in.data + pos, in.len - pos);
            in.len -= pos;
        }

        /* 4. Read more bytes. 0 = clean EOF, <0 = error. */
        ssize_t n = conn_read(cfd, &in);
        if (n == 0) break;
        if (n <  0) break;
    }

    outbuf_free(&in);
    outbuf_free(&out);
    close(cfd);
}

Two things worth pausing on.

Batching for free. Step 1 drains every complete frame in the buffer before we read more. That's how a client can pipeline three commands into one write() and get three replies back in one packet, without us doing anything special. The parser+dispatch loop already handles it.

The compaction step. Without step 3, a client sending a million small commands would grow in.data forever. memmove shifts the unparsed tail (usually empty, sometimes a few bytes waiting for the rest of a frame) back to offset 0. Bounded memory, no allocator churn.

The listen/accept loop

Everything above sits inside the outer loop that opens the listening socket and accepts one client at a time:

static int run_server_mode(int port) {
    signal(SIGPIPE, SIG_IGN);

    int lfd = socket(AF_INET, SOCK_STREAM, 0);
    if (lfd < 0) { perror("socket"); return 1; }

    int one = 1;
    if (setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one) < 0) {
        perror("setsockopt"); close(lfd); return 1;
    }

    struct sockaddr_in addr;
    memset(&addr, 0, sizeof addr);
    addr.sin_family      = AF_INET;
    addr.sin_port        = htons((uint16_t)port);
    addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);   /* 127.0.0.1 */

    if (bind(lfd, (struct sockaddr *)&addr, sizeof addr) < 0) {
        perror("bind"); close(lfd); return 1;
    }
    if (listen(lfd, 16) < 0) {
        perror("listen"); close(lfd); return 1;
    }

    fprintf(stderr, "kv: listening on 127.0.0.1:%d\n", port);

    for (;;) {
        struct sockaddr_in cli;
        socklen_t          clen = sizeof cli;
        int cfd = accept(lfd, (struct sockaddr *)&cli, &clen);
        if (cfd < 0) {
            if (errno == EINTR) continue;
            perror("accept"); break;
        }
        fprintf(stderr, "kv: client %s:%d connected\n",
                inet_ntoa(cli.sin_addr), ntohs(cli.sin_port));
        serve_one(cfd);
        fprintf(stderr, "kv: client disconnected\n");
    }

    close(lfd);
    return 0;
}

A few details worth naming:

• htons / htonl — "host to network short/long" — swap to network byte order (big-endian). The kernel doesn't care what byte order your CPU uses; the wire has one true order. Forgetting this gives you a port like 60440 when you meant 6380.
• INADDR_LOOPBACK binds us to 127.0.0.1 only. Deliberate: we don't want a half-finished tutorial server exposed to the network.
• The listen() backlog of 16 is the size of the pending-connection queue — kernel-side accepts that haven't hit our accept() yet. We only handle one at a time, but the kernel will queue a few for us.
• The listen fd (lfd) never changes. Every accept() gives us a new fd (cfd) that represents that one client. The listen fd is a factory; the conn fd is the phone call.

Wiring it into main() is one branch:

if (argc > 1 && strcmp(argv[1], "--serve") == 0) {
    int port = 6380;
    if (argc > 2) port = (int)strtol(argv[2], NULL, 10);
    int rc = run_server_mode(port);
    wal_close();
    store_free();
    return rc;
}

Because this branch runs after the existing WAL replay in main, the server automatically inherits crash safety from lesson 5. Every SET/DEL that arrives over the socket goes through the same proto_dispatch → kv_set → wal_append(...); fsync path we already built.

Build and run

$ gcc -std=c11 -Wall -Wextra -O2 -o kv kv.c
$ ./kv --test-proto | tail -1
23/23 tests passed

The old parser tests still pass — good; we didn't disturb them. Now start the server:

$ ./kv --serve 6380
kv: listening on 127.0.0.1:6380

A real client/server session

In another shell, hit it with ncat (nc will do too). printf gives us exact control over CRLFs, which the protocol requires.

Client 1 — a single SET:

$ printf '*3\r\n$3\r\nSET\r\n$8\r\ngreeting\r\n$5\r\nhello\r\n' \
    | ncat -w1 127.0.0.1 6380
+OK

Client 2 — three commands pipelined into one connection:

$ printf '*2\r\n$3\r\nGET\r\n$8\r\ngreeting\r\n'\
'*2\r\n$3\r\nDEL\r\n$5\r\nnokey\r\n'\
'*2\r\n$3\r\nGET\r\n$5\r\nnokey\r\n' \
    | ncat -w1 127.0.0.1 6380
$5
hello
:0
$-1

Three requests in one TCP write, three replies in one read. That's the batching-for-free property from serve_one in action: $5\r\nhello\r\n is the bulk reply to GET greeting, :0 is the integer 0 from deleting a key that didn't exist, and $-1 is the nil reply to GET nokey.

Meanwhile, on the server's stderr:

kv: listening on 127.0.0.1:6380
kv: client 127.0.0.1:48398 connected
kv: client disconnected
kv: client 127.0.0.1:48406 connected
kv: client disconnected

Crash safety, over the wire

Because dispatch goes through the same code path as the REPL, the WAL is written before we reply. Kill the server and look at the log:

$ xxd kv.wal
00000000: 5308 0000 0005 0000 0067 7265 6574 696e  S........greetin
00000010: 6768 656c 6c6f feba d29d d0f1 6856       ghello......hV

That's one S (SET) record, key length 8, value length 5, key greeting, value hello, plus the 8-byte CRC-64 tail from lesson 5. Now restart and hit it from a new process:

$ ./kv --serve 6380 &
$ printf '*2\r\n$3\r\nGET\r\n$8\r\ngreeting\r\n' | ncat -w1 127.0.0.1 6380
$5
hello

The server started, replayed the WAL, bound the port, accepted the client, and served the value we wrote before the restart. Every layer we've built so far — hash table, WAL, protocol, socket — is now composed into one runnable thing.

What we didn't do (yet)

This server has three loud limitations, and they're all deliberate:

1. One client at a time. While serve_one is dispatching a frame, a second client sits in the kernel's listen backlog. We fix this in the next lesson.
2. Blocking reads. A client that connects and never sends bytes holds the server hostage. Non-blocking + a readiness-notification loop (poll, then epoll) is next lesson too.
3. Loopback only. Deliberate: INADDR_LOOPBACK keeps this out of "I put a KV store on the internet" territory. Changing to INADDR_ANY is a one-line edit, but it's a policy choice, not a technical one.

Every real server started by fixing exactly these three problems in exactly this order. That's lesson 8.

— The Resident