Writing an HTTP Server from Scratch in Zig
Zig is a modern systems programming language that aims to be a better C. To really understand both Zig and HTTP, I decided to build a web server from scratch using only the standard library - no frameworks, no external dependencies. I will be following the Codecrafters challenge.
Note: It's not really a course or a tutorial where everything is just served to you on a silver platter. These challenges are more like a guides that help you understand the design, concepts, provide relevant sources of information and hints on how to approach a task.
React bros right now (I'm one of them)
Reinventing the wheel
Because I'm chronically online and quite aware that people love to ask:
Why are you trying to solve something that has already been solved? ~ Super Smart AI Fork startup founder
My response is quite simple. Reinventing the wheel is not a bad thing, especially if you are learning. Sure, you can spin up an Express.js or GO server in minutes (seconds even perhaps), hell even this blog is built with React. However there's a huge value in building a really primitive versions of the solutions we use everyday. I'm not talking about a full GO http implementation re-write, but a simple piece that will help you understand the core fundamentals.
Why Zig?
Zig is a relatively new language that provides a fine control over memory like C, but with better safety measures. With that of course it introduces a more "modern" approach to memory management and one of it's biggest features Comptime.
I will not dive deep into Zig technical aspects in this post as this is not the agenda here, but Comptime is a feature in Zig that allows code to be executed at compile time. You can find more about Comptime here. Zig compiles to native code and has a really strong support for low-level networking primitives which is perfect for building an HTTP server.
Shape of the HTTP Request
Let's start with parsing requests. HTTP is a text-based protocol, which makes it conceptually simple but tricky to parse correctly. The server needs to handle request lines, headers, and bodies while being resilient to malformed input.
We need to establish a shape of our request. As mentioned at the beginning, this is a really simple implementation of an HTTP server so I'm keeping this simple. I read RFC 9112 Spec and ended up with a really simple type.
pub const Request = struct {
// []const u8 is essentially a string in Zig
request_line: []const u8,
headers: []const u8,
body: []const u8,
const Self = @This();
}
I defined three fields of type string. @This() in Zig allows you to refer to a generated type, in this case our Request struct.
Parsing Requests
Now that we know what the shape of our request is we can write a method to parse the request. But... How are we supposed to do that?
RFC 9112 Spec comes with help, specifically Section 2.1 that describes the message format.
HTTP-message = start-line CRLF
*( field-line CRLF )
CRLF
[ message-body ]
So our Request will look roughly like this:
GET /path HTTP/1.1\r\n Host: example.com\r\n Content-Type: text/plain\r\n \r\n This is the body content
My method receives the input and splits it on the double CRLF ("\r\n\r\n") to separate request line and headers from body. It returns an iterator which basically is just two items. Then I split rl_headers on single CRLF to separate request line and headers. Then I simply return our Request struct.
pub const Request = struct {
request_line: []const u8,
headers: []const u8,
body: []const u8,
const Self = @This();
pub inline fn parse(input: []const u8) Request {
var iter = std.mem.splitSequence(u8, input, "\r\n\r\n");
const rl_headers = iter.next().?;
const body = iter.rest();
var rl_headers_iter = std.mem.splitSequence(u8, rl_headers, "\r\n");
return Request{
.request_line = rl_headers_iter.first(),
.headers = rl_headers_iter.rest(),
.body = body,
};
}
}Extracting method and target
Now that we parsed our Request we need to extract two more things. HTTP Method and Target Path. Our server will handle only GET and POST requests, but we need to extract that information from our request line but also our target path so we know which endpoint has been hit, luckily it's pretty straight-forward. I just split request line from our struct on an empty character and for method I return the first element and for target path the second element.
pub inline fn method(self: Self) []const u8 {
var iter = std.mem.splitScalar(u8, self.request_line, ' ');
return iter.first();
}
pub inline fn target(self: Self) ?[]const u8 {
var iter = std.mem.splitScalar(u8, self.request_line, ' ');
_ = iter.first();
return iter.next();
}I also created a simple helper method that allows me to find a header by its key:
pub inline fn get_header(self: Self, needle: []const u8) []const u8 {
var iter = std.mem.splitSequence(u8, self.headers, "\r\n");
while (iter.next()) |header| {
var headerSplit = std.mem.splitSequence(u8, header, ": ");
if (std.mem.eql(u8, headerSplit.first(), needle)) {
return headerSplit.rest();
}
}
return "";
}Handling Connection
Before we handle specific HTTP methods and process incoming requests, let's consider what actually happens when an HTTP server receives a connection. First, when a client connects to our server, it initiates a TCP connection via a three-way handshake. The client and server exchange packets with specific TCP flags—primarily two:
- SYN (Synchronize): Initiates the connection request.
- ACK (Acknowledge): Confirms receipt of data.
The handshake unfolds in three steps:
- The client sends a SYN packet to the server.
- The server replies with SYN-ACK, acknowledging the request and sending its own SYN.
- The client sends ACK, confirming everything and establishing the connection.
At this point, the TCP connection is ready, and the actual HTTP data (e.g., request headers) can be exchanged.
Why bring this up? It's crucial to understand the low-level mechanics at the heart of every HTTP interaction. It also illustrates how much "under-the-hood" work occurs even for a simple request. This leads to a natural follow-up: What if the server receives more than one connection at a time?
Threads
Zig gives us a pretty easy way to utilize concurrency pattern by using its std.Thread.Pool. When our server receives a connection, instead of processing it in a main thread of our application we delegate this job to another thread. If we run this on a CPU with one core, the OS will use scheduler to manage multiple threads on a single core by utilizing context switching.
It rapidly switches between threads, giving each a "time slice" on the core, for example 50 milliseconds. During a switch, the OS saves the state of the current thread and loads another one's state and this way the OS creates the illusion of multiple threads running "at the same time" which is called Concurrency. Of course, if more cores are available, then our program will indeed process request simultaneously - Parallelism.
Since we are following the Codecrafters instructions I also created a Config struct that parses process arguments, specifically one "--directory" that just gives as a path to the file that will be used for I/O operations. Nothing fancy, parse method just sets the struct field to the parsed path.
pub const Config = struct {
fileDir: ?[]const u8,
pub inline fn parse() Config {
var fileDir: ?[]const u8 = null;
var args = std.process.args();
const alloc = std.heap.page_allocator;
while (args.next()) |arg| {
if (std.mem.eql(u8, arg, "--directory")) {
if (args.next()) |dir| {
fileDir = std.fs.cwd().realpathAlloc(alloc, dir) catch unreachable;
}
}
}
return Config{
.fileDir = fileDir,
};
}
};I decided to handle the main connection loop in the main function. I use Zig's std.net to bind a port to the TCP socket and create a listener. We need a thread pool to handle multiple connections at once, n_jobs allows as to set a maximum number of threads that can be spawned at once. Once it's done, we can create a while loop that will work unless the program is explicitly killed or crashes, inside we take the incoming connection, log it (just for debuging) and spawn a thread that will handle the connection.
const std = @import("std");
const stdout = std.io.getStdOut().writer();
const handlers = @import("handlers.zig");
const types = @import("types.zig");
const net = std.net;
const Config = types.Config;
const Request = types.Request;
const Response = types.Response;
pub fn main() !void {
const config = Config.parse();
const address = try net.Address.resolveIp("127.0.0.1", 4221);
var listener = try address.listen(.{
.reuse_address = true,
});
defer listener.deinit();
var gpa_alloc = std.heap.GeneralPurposeAllocator(.{}){};
const gpa = gpa_alloc.allocator();
defer {
if (gpa_alloc.deinit() == std.heap.Check.leak) {
std.debug.print("Memory leak", .{});
}
}
var pool: std.Thread.Pool = undefined;
try pool.init(.{ .allocator = gpa, .n_jobs = 8 });
defer pool.deinit();
while (true) {
const conn = try listener.accept();
try stdout.print("client {} connected!\n", .{conn.address.in.sa.port});
try pool.spawn(handlers.handleConn, .{ gpa, config, conn });
}
}Handling Requests
You might have noticed that I pass handlers.handleConn to the thread, this is our main connection handler that reads bytes from the request and where we will process them. For that I also created two helper functions - handleSingleRequest and findRequestLength. While the handleConn takes care of our connection and reads raw bytes, handleSingleRequest uses our methods defined in the Request struct to parse the request and dispatch a correct method handler or response with an appropriate error. Our second helper function is just looking for the Content-Length header and return the total length with padding.
pub fn handleConn(allocator: std.mem.Allocator, config: Config, conn: net.Server.Connection) void {
defer conn.stream.close();
const writer = conn.stream.writer();
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();
const alloc = arena.allocator();
var buf = std.ArrayList(u8).init(alloc);
defer buf.deinit();
while (true) {
var tmp_buf: [1024]u8 = undefined;
const read = conn.stream.read(&tmp_buf) catch |err| {
std.debug.print("Read error: {}\n", .{err});
return;
};
if (read == 0) break;
buf.appendSlice(tmp_buf[0..read]) catch return;
while (true) {
const req_len = findRequestLength(buf.items) orelse break;
if (buf.items.len < req_len) break;
const req = Request.parse(buf.items[0..req_len]);
const keep_alive = handleSingleRequest(allocator, req, config, writer);
buf.replaceRange(0, req_len, &.{}) catch unreachable;
if (!keep_alive) {
return;
}
}
}
}
pub inline fn handleSingleRequest(alloc: std.mem.Allocator, req: Request, config: Config, writer: net.Stream.Writer) bool {
var res = Response.init(alloc, writer);
defer res.deinit();
res.keep_alive = !std.mem.eql(u8, req.get_header("Connection"), "close");
if (!res.keep_alive) {
res.add_header("Connection", req.get_header("Connection"));
}
if (std.mem.eql(u8, req.method(), "GET")) {
handleGet(alloc, config, writer, req);
} else if (std.mem.eql(u8, req.method(), "POST")) {
handlePost(alloc, config, writer, req);
} else {
res.set_status("405 Method Not Allowed");
writer.writeAll("HTTP/1.1 405 Method Not Allowed\r\n\r\n") catch res.handleErr();
res.write(writer);
}
return res.keep_alive;
}
pub inline fn findRequestLength(data: []const u8) ?usize {
const headers_end = std.mem.indexOf(u8, data, "\r\n\r\n") orelse return null;
const headers = data[0..headers_end];
var content_length: usize = 0;
var headers_iter = std.mem.splitSequence(u8, headers, "\r\n");
while (headers_iter.next()) |header| {
if (std.mem.startsWith(u8, header, "Content-Length: ")) {
const content_length_str = header["Content-Length: ".len..];
content_length = std.fmt.parseInt(usize, content_length_str, 10) catch 0;
break;
}
}
const total_length = headers_end + 4 + content_length;
return if (data.len >= total_length) total_length else null;
}The server supports both GET and POST requests. GET requests are straightforward - parse the path and return the appropriate response. POST requests require reading and processing the request body, which involves careful buffer management.
GZIP Compression
Our HTTP server will support GZIP compression. When a client sends an Accept-Encoding: gzip header, the server compresses the response body before sending it. This significantly reduces bandwidth usage for large responses. But how does it look like?
# Example taken from Codecrafters HTTP/1.1 200 OK Content-Encoding: gzip Content-Type: text/plain Content-Length: 23 1F 8B 08 00 00 00 00 00 // This is the hexadecimal representation of the body. 00 03 4B 4C 4A 06 00 C2 // It should actually be sent as binary data. 41 24 35 03 00 00 00
As you can see the response body is compressed using gzip and the actual body is represented in hexadecimal. Luckily Zig's standard library provides a gzip.compress method. Let's create a set_body method (and a few helper methods) in our Request struct.
pub inline fn set_body(self: *Self, str: []const u8) void {
if (self.compressGzip) {
var buf = std.ArrayList(u8).init(self.allocator);
var strBuf = std.io.fixedBufferStream(str);
std.compress.gzip.compress(strBuf.reader(), buf.writer(), .{}) catch return self.handleErr();
self.body = buf.toOwnedSlice() catch return self.handleErr();
} else {
self.body = str;
}
const iStr = std.fmt.allocPrint(self.allocator, "{d}", .{self.body.len}) catch return self.handleErr();
defer self.allocator.free(iStr);
self.add_header("Content-Length", iStr);
}
pub inline fn write(self: *Self, writer: net.Stream.Writer) void {
if (!self.keep_alive) {
self.add_header("Connection", "close");
}
std.fmt.format(writer, "HTTP/1.1 {s}\r\n{s}\r\n{s}", .{ self.status, self.headers, self.body }) catch return self.handleErr();
}
pub inline fn notFound(self: Self) void {
self.writer.writeAll("HTTP/1.1 404 Not Found\r\n\r\n") catch return self.handleErr();
}
pub inline fn handleErr(self: Self) void {
self.writer.writeAll("HTTP/1.1 500 Internal Server Error\r\n\r\n") catch return;
}GET and POST Requests
All right, now that we have everything ready we can finally take a look at handleGet. We mostly check the request path, read and set headers, set body and once the response is ready we just write it into the stream. In case we receive a request to /files/, we try to read the path, read file and set body to the file content.
pub fn handleGet(allocator: std.mem.Allocator, config: Config, writer: net.Stream.Writer, req: Request) void {
var res = Response.init(allocator, writer);
defer res.deinit();
if (std.mem.containsAtLeast(u8, req.get_header("Accept-Encoding"), 1, "gzip")) {
res.add_header("Content-Encoding", "gzip");
res.compressGzip = true;
}
const t = req.target();
if (t == null) return res.handleErr();
const target = t.?;
res.keep_alive = !std.mem.eql(u8, req.get_header("Connection"), "close");
if (!res.keep_alive) {
res.add_header("Connection", req.get_header("Connection"));
}
if (std.mem.eql(u8, target, "/")) {
res.set_status("200 OK");
} else if (std.mem.startsWith(u8, target, "/echo/")) {
res.set_status("200 OK");
res.add_header("Content-Type", "text/plain");
const str = target[6..];
res.set_body(str);
} else if (std.mem.eql(u8, target, "/user-agent")) {
res.set_status("200 OK");
res.add_header("Content-Type", "text/plain");
const str = req.get_header("User-Agent");
res.set_body(str);
} else if (std.mem.startsWith(u8, target, "/files/")) {
if (config.fileDir) |dir| {
const path = std.fs.path.join(allocator, &.{ dir, target[7..] }) catch return res.handleErr();
defer allocator.free(path);
const file = std.fs.openFileAbsolute(path, .{}) catch |err| return {
if (err == error.FileNotFound) {
return res.notFound();
} else {
return res.handleErr();
}
};
res.set_status("200 OK");
res.add_header("Content-Type", "application/octet-stream");
const str = file.readToEndAlloc(allocator, 4096) catch return res.handleErr();
res.set_body(str);
} else return res.notFound();
} else return res.notFound();
res.write(writer);
}Our handlePost is way easier as it is used only for /files/ route. We make sure that the path is correct, check if the file exist (if not, we create it) and we write the request body to the file. Of course we set a correct status 201 Created and write the response to the stream.
Summary
Honestly, if someone got to this point, thanks! We successfully finished Codecrafters HTTP Server challenge in Zig. Perhaps it's not the best code quality - mediocre at best, but I really learned a lot about HTTP insides and honestly had a lot of fun doing that challenge. Who knows, maybe I will re-visit this one using a different language in the future, but for now, thanks again for reading and try to visit sometimes, I will probably post more articles, cya.