Basic HTTP server in Clojure

HTTP uses TCP, so naturally this is going to be built on top of the previous post.

Goal

To goal is to receive a HTTP request and respond with 200 OK where the body has information about the request received.

HTTP protocol

A HTTP request has three parts:

  1. Request line: which contains information of the method (e.g., GET, POST, PUT), the resource and the HTTP version.
  2. Headers: One by line and provide additional information.
  3. Body: Contains more data.

Example:

GET /index.html HTTP/1.1
Host: marcelofernandes.dev


{:hello "There"}

Previous TCP server

Convenient copy of the previous TCP server that we’re going to built on top of.

(ns user
  (:require [clojure.string :as str])
  (:import (java.net ServerSocket)
           (java.io BufferedReader InputStreamReader PrintWriter)))


(defn- handle-input
  [{:keys [in out handler]}]
  (loop []
    (when-let [line (.readLine in)]
      (println "Received:" line)
      (.println out (handler line))
      (recur))))


(defn- client-handler
  [client-socket handler]
  (future
    (try
      (with-open [in  (BufferedReader. (InputStreamReader. (.getInputStream client-socket)))
                  out (PrintWriter. (.getOutputStream client-socket) true)]
        (handle-input {:in      in
                       :out     out
                       :handler handler}))
      (catch Exception e
        (println "Client error:" (.getMessage e)))
      (finally
        (when-not (.isClosed client-socket)
          (.close client-socket)
          (println "Client disconnected by error"))))))


(defn create-tcp-server
  [{:keys [port handler]}]
  (let [state        (atom {:server-socket nil
                            :clients       []})
        start-server (fn []
                       (when (nil? (:server-socket @state))
                         (reset! state {:server-socket (ServerSocket. port)
                                        :clients       []})
                         (println "Server started on port" port)
                         (future
                           (try
                             (loop []
                               (let [client-socket (.accept (:server-socket @state))]
                                 (swap! state update :clients conj client-socket)
                                 (client-handler client-socket handler)
                                 (recur)))
                             (catch Exception e
                               (println "Server error:" (.getMessage e)))
                             (finally
                               (when-not (.isClosed (:server-socket @state))
                                 (.close (:server-socket @state))
                                 (println "Server stopped by error")))))))
        stop-server  (fn []
                       (when-let [server-socket (:server-socket @state)]
                         (doseq [client-socket (:clients @state)]
                           (.close client-socket)
                           (println "Client disconnected by server"))
                         (.close server-socket)
                         (println "Server stopped by user")
                         (reset! state {:server-socket nil
                                        :clients       []})))]
    {:start start-server
     :stop  stop-server}))


(let [{:keys [start stop]} (create-tcp-server {:port    1230
                                               :handler str/upper-case})]
  (start)
  (Thread/sleep 10000)
  (stop))

To start we can test the behavior of this server by making a request with curl:

Terminal window
curl localhost:1230

The logs are:

Server started on port 1230
Received: GET / HTTP/1.1
Received: Host: localhost:1230
Received: User-Agent: curl/8.4.0
Received: Accept: */*
Received:
Client error: Connection reset
Client disconnected by server
Server stopped by user
Server error: Socket closed

We can see that it reads one line at a time, first getting the request line, then the headers, followed by an empty line. That doesn’t say anything about the body. Let’s try with a POST:

Terminal window
curl --data '{:hello "There"}' --header 'Content-Type: application/edn' localhost:1230

The logs:

Server started on port 1230
Received: POST / HTTP/1.1
Received: Host: localhost:1230
Received: User-Agent: curl/8.4.0
Received: Accept: */*
Received: Content-Type: application/edn
Received: Content-Length: 16
Received:
Client error: Connection reset
Client disconnected by server
Server stopped by user
Server error: Socket closed

We got some extra headers but we’re not able to see the {:hello "There"} yet.

Parse request to a useful data structure

I want to transform that request into a data structure that I can work with. For the request above that would be:

{:method  :post
 :path    "/"
 :headers {"Host"           "localhost:1230"
           "User-Agent"     "curl/8.4.0"
           "Accept"         "*/*"
           "Content-Type"   "application/edn"
           "Content-Length" "16"}}
 :body    {:hello "There"}

Changing the handle-input function

The handle-input doesn’t need a handler function anymore. To keep the REPL flow going we can structure the code like this:

(ns user
  (:require [clojure.string :as str])
  (:import (java.net ServerSocket)
           (java.io BufferedReader InputStreamReader PrintWriter)))


(defn- handle-input
  [{:keys [in _out]}]
  (loop [request {}
         state   :idle]
    (cond
      (= state :idle)
      (if-let [line (.readLine in)]
        (let [[method path protocol] (str/split line #"\s+")]
          (recur (assoc request
                        :method   (-> method str/lower-case keyword)
                        :path     path
                        :protocol protocol)
                 :headers))
        (println "Client disconnected"))


      (= state :headers)
      (do
        (println request)
        (println "Still need to parse headers")))))


(defn- client-handler
  [client-socket]
  (future
    (try
      (with-open [in  (BufferedReader. (InputStreamReader. (.getInputStream client-socket)))
                  out (PrintWriter. (.getOutputStream client-socket) true)]
        (handle-input {:in      in
                       :out     out}))
      (catch Exception e
        (println "Client error:" (.getMessage e)))
      (finally
        (when-not (.isClosed client-socket)
          (.close client-socket)
          (println "Client disconnected by error"))))))


(defn create-tcp-server
  [{:keys [port]}]
  (let [state        (atom {:server-socket nil
                            :clients       []})
        start-server (fn []
                       (when (nil? (:server-socket @state))
                         (reset! state {:server-socket (ServerSocket. port)
                                        :clients       []})
                         (println "Server started on port" port)
                         (future
                           (try
                             (loop []
                               (let [client-socket (.accept (:server-socket @state))]
                                 (swap! state update :clients conj client-socket)
                                 (client-handler client-socket)
                                 (recur)))
                             (catch Exception e
                               (println "Server error:" (.getMessage e)))
                             (finally
                               (when-not (.isClosed (:server-socket @state))
                                 (.close (:server-socket @state))
                                 (println "Server stopped by error")))))))
        stop-server  (fn []
                       (when-let [server-socket (:server-socket @state)]
                         (doseq [client-socket (:clients @state)]
                           (.close client-socket)
                           (println "Client disconnected by server"))
                         (.close server-socket)
                         (println "Server stopped by user")
                         (reset! state {:server-socket nil
                                        :clients       []})))]
    {:start start-server
     :stop  stop-server}))


(comment
  (def server (create-tcp-server {:port 1230}))






  ((:start server))) ;; to start


  ((:stop server)) ;; to stop

Then we can just evaluate handle-input any time there is a change. When receiving a new request it will use the new version of the function.

When sending a request like this:

Terminal window
curl --data '{:hello "There"}' --header 'Content-Type: application/edn' localhost:1230

the logs show:

{:method :post, :path /, :protocol HTTP/1.1}
Still need to parse headers

Parsing the headers is a matter of reading line by line and splitting by :.

(defn- handle-input
  [{:keys [in _out]}]
  (loop [request {}
         state   :idle]
    (condp = state
      :idle
      (if-let [line (.readLine in)]
        (let [[method path protocol] (str/split line #"\s+")]
          (recur (assoc request
                        :method   (-> method str/lower-case keyword)
                        :path     path
                        :protocol protocol)
                 :headers))
        (println "Client disconnected"))


      :headers
      (if-let [line (.readLine in)]
        (if (empty? line)
          (recur request :done)
          (let [header-vector           (str/split line #": ")
                request-with-new-header (assoc-in request
                                                  [:headers (first header-vector)]
                                                  (second header-vector))]
            (recur request-with-new-header :headers)))
        (recur request :done))


      :done
      (do
        (println request)
        (println "Done parsing the request")))))

which logs

{:method :post, :path /, :protocol HTTP/1.1, :headers {Host localhost:1230, User-Agent curl/8.4.0, Accept */*, Content-Type application/edn, Content-Length 16}}
Done parsing the request

Writing the response back to client

To write the response back we just need to answer according to HTTP protocol:

HTTP/1.1 200 OK
{headers here}


{body here}

In our case we won’t sent any headers and just send our structure as the body.

(defn- handle-input
  [{:keys [in out]}]
  (loop [request {}
         state   :idle]
    (condp = state
      :idle
      (if-let [line (.readLine in)]
        (let [[method path protocol] (str/split line #"\s+")]
          (recur (assoc request
                        :method   (-> method str/lower-case keyword)
                        :path     path
                        :protocol protocol)
                 :headers))
        (println "Client disconnected"))


      :headers
      (if-let [line (.readLine in)]
        (if (empty? line)
          (recur request :done)
          (let [header-vector           (str/split line #": ")
                request-with-new-header (assoc-in request
                                                  [:headers (first header-vector)]
                                                  (second header-vector))]
            (recur request-with-new-header :headers)))
        (recur request :done))


      :done
      (do
        (.println out "HTTP/1.1 200 OK\r\n\r\n")
        (.println out (str request))
        (println "Done parsing the request")))))

When I open the browser and navigate to http://localhost:1230 I see this response (unformatted):

{:method   :get
 :path     "/"
 :protocol "HTTP/1.1"
 :headers  {"Sec-Fetch-Dest"            "document"
            "Sec-Fetch-Mode"            "navigate"
            "Upgrade-Insecure-Requests" "1"
            "Sec-Fetch-User"            "?1"
            "User-Agent"                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:128.0) Gecko/20100101 Firefox/128.0"
            "Priority"                  "u=0, i"
            "Sec-Fetch-Site"            "none"
            "Accept-Encoding"           "gzip, deflate, br, zstd"
            "Accept-Language"           "en-US,en;q=0.5"
            "Connection"                "keep-alive"
            "Accept"                    "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8"
            "Host"                      "localhost:1230"}}

Parsing the body

The body is a little trickier because we can’t just use readLine. It doesn’t have a line feed (\n) or carriage return (\r) or anything like that. Instead in this basic attempt we rely on Content-Length header.

This request:

Terminal window
curl --data '{:hello "There"}' --header 'Content-Type: application/edn' localhost:1230

Has a content length of 16.

Then we can check if the content length header is present, make a huge assumption that it’s telling the truth, and read that many characters:

(defn- handle-input
  [{:keys [in out]}]
  (loop [request {}
         state   :idle]
    (condp = state
      :idle
      (if-let [line (.readLine in)]
        (let [[method path protocol] (str/split line #"\s+")]
          (recur (assoc request
                        :method   (-> method str/lower-case keyword)
                        :path     path
                        :protocol protocol)
                 :headers))
        (println "Client disconnected"))


      :headers
      (if-let [line (.readLine in)]
        (if (empty? line)
          (recur request :body)
          (let [[key value] (str/split line #": ")]
            (recur (assoc-in request [:headers key] value) :headers)))
        (recur request :body))


      :body
      (let [content-length (get-in request [:headers "Content-Length"])]
        (if-not content-length
          (recur request :done)
          (let [size (parse-long content-length)
                body (char-array size)]
            (doseq [i (range size)]
              (aset body i (char (.read in))))
            (recur (assoc request :body (String. body)) :done))))


      :done
      (do
        (.println out "HTTP/1.1 200 OK\r\n\r\n")
        (.println out (str request))
        (println "Done parsing the request")))))

The response for the previous request is:

{:method   :post
 :path     "/"
 :protocol "HTTP/1.1"
 :headers  {"Host"           "localhost:1230"
            "User-Agent"     "curl/8.4.0"
            "Accept"         "*/*"
            "Content-Type"   "application/edn"
            "Content-Length" "16"}
 :body     "{:hello \"There\"}"}

Conclusion

The complete code is this:

(ns user
  (:require [clojure.string :as str])
  (:import (java.net ServerSocket)
           (java.io BufferedReader InputStreamReader PrintWriter)))


(defn- handle-input
  [{:keys [in out]}]
  (loop [request {}
         state   :idle]
    (condp = state
      :idle
      (if-let [line (.readLine in)]
        (let [[method path protocol] (str/split line #"\s+")]
          (recur (assoc request
                        :method   (-> method str/lower-case keyword)
                        :path     path
                        :protocol protocol)
                 :headers))
        (println "Client disconnected"))


      :headers
      (if-let [line (.readLine in)]
        (if (empty? line)
          (recur request :body)
          (let [[key value] (str/split line #": ")]
            (recur (assoc-in request [:headers key] value) :headers)))
        (recur request :body))


      :body
      (let [content-length (get-in request [:headers "Content-Length"])]
        (if-not content-length
          (recur request :done)
          (let [size (parse-long content-length)
                body (char-array size)]
            (doseq [i (range size)]
              (aset body i (char (.read in))))
            (recur (assoc request :body (String. body)) :done))))


      :done
      (do
        (.println out "HTTP/1.1 200 OK\r\n\r\n")
        (.println out (str request))
        (println "Done parsing the request")))))


(defn- client-handler
  [client-socket]
  (future
    (try
      (with-open [in  (BufferedReader. (InputStreamReader. (.getInputStream client-socket)))
                  out (PrintWriter. (.getOutputStream client-socket) true)]
        (handle-input {:in      in
                       :out     out}))
      (catch Exception e
        (println "Client error:" (.getMessage e)))
      (finally
        (when-not (.isClosed client-socket)
          (.close client-socket)
          (println "Client disconnected by error"))))))


(defn create-tcp-server
  [{:keys [port]}]
  (let [state        (atom {:server-socket nil
                            :clients       []})
        start-server (fn []
                       (when (nil? (:server-socket @state))
                         (reset! state {:server-socket (ServerSocket. port)
                                        :clients       []})
                         (println "Server started on port" port)
                         (future
                           (try
                             (loop []
                               (let [client-socket (.accept (:server-socket @state))]
                                 (swap! state update :clients conj client-socket)
                                 (client-handler client-socket)
                                 (recur)))
                             (catch Exception e
                               (println "Server error:" (.getMessage e)))
                             (finally
                               (when-not (.isClosed (:server-socket @state))
                                 (.close (:server-socket @state))
                                 (println "Server stopped by error")))))))
        stop-server  (fn []
                       (when-let [server-socket (:server-socket @state)]
                         (doseq [client-socket (:clients @state)]
                           (.close client-socket)
                           (println "Client disconnected by server"))
                         (.close server-socket)
                         (println "Server stopped by user")
                         (reset! state {:server-socket nil
                                        :clients       []})))]
    {:start start-server
     :stop  stop-server}))


(comment
  (def server (create-tcp-server {:port 1230}))






  ((:start server))
  ((:stop server)))

It’s full of problems and security issues. But it’s a nice start for creating a basic HTTP server using only the Socket interface.

Back to notes