This Just In: Yet Another Local Web Server

Fact is, I should not have built this code. I'm sure that somewhere, there's a suitable NPM or PyPi package (or something in some other language) which would have worked out just fine. But nothing I saw on the first page of my search results seemed like it was both the correct functionality AND light-weight enough.

I'm using GitPages to host the documentation and demos for my little Web Components library. The docs are Markdown files, which GitPages dynamically converts to HTML for me. So surely there's a local equivalent I can use to test my updates prior to publication, right?

Jekyll, et al, Not Really to the Rescue

The "official" method is to install Jekyll and a bunch of other tooling. Jekyll is written in Ruby, and I don't use Ruby myself (no offense), so it seemed like a bridge too far to install an entire language system just to test this one thing. I would spend more time configuring that then writing the docs. Bleh.

There are several "markdown web servers" out there too, but they uniformly seem to just want to server markdown, and not do that nifty GitPages translate-on-the-fly thing. Maybe I just missed the silver bullet. Too late if I did, because I wrote one in about 90 minutes.

I LOVE to tinker

I know JavaScript reasonably well, so why not just dive in? I legitimately spent less time building what I present here than I had researching a pre-built solution.

So let's talk code, eh?

First, as in any project a complete, well-thought-out plan was needed. Except I didn't make one, exactly. As is my habit, I start by writing code that just won't work, but expresses the basics:

I need an HTTP server
which sends files from the current directory
and processes Markdown into HTML on the fly
and if a request for an .html resource comes in but no such file exists, try the corresponding .md file
and if the URL looks like a directory, serve up index.html (which could actually be index.md)
and gives meaningful errors as needed

We'll set up a project

mkdir mdserver
cd mdserver
npm init -y

and add this code as index.js:

const http = require('http')

// create a simple HTTP server
const server = http.createServer((req, rsp) => {
  // convert the requested URL to a local pathname
  let pathname = resolveUrl(req.url)

  // send the content from that pathname, or an error
  return resolvePath(pathname)
})

// listen for requests.
server.listen(3000, "localhost", () => {
   console.log(`ehMDserver: running at http://localhost:3000/`)
})

This stub doesn't won't actually work, because I haven't defined resolveURL() and resolvePath(). That's OK, it's all part of my process of starting from a relatively high level, and then drilling down into the details.

Let's convert the URL to a local pathname

This one is currently super simple, but later we'll add some additional features, so hang tight:

const pathlib = require('path')
...
function resolveUrl(url) {
  return pathlib.resolve(`./${url}`)
}

Now you may be thinking, "how does he know we'll add features later?" Well, truthfully, I didn't. But the principle of mapping out the high-level steps means that if this never evolves, the main function is still readable. And if it does evolve, the main function is still readable.

Now let's send a file

For starters, let's just send, verbatim, whatever file is requested.

const fs= require('fs')
...
function resolvePath(path) {
  if (fs.existsSync(path)) {
    if (fs.lstatSync(path).isFile())  {
      return sendFile(path)
    }
  }
}

You'll note that I've once again called a non-existent function, but -- again -- you get what this thing is doing.

Now we'll need to implement sendFile():

function sendFile(path) {
  body = fs.readFileSync(path)
  rsp.writeHead(200,
    {
      'Content-Type': 'text/plain',
      'Content-Length': Buffer.byteLength(body)
    }
  )
  rsp.end(body)
}

We read the file from disk, send HTTP headers Content-Type and Content-Length and send the file contents to the client.

And now we have a problem: our rsp (response) object, which came into our main function, needs to be here too, so we'll pass it all down the chain. In fact, as long as we're at it, let's send the req (request object) as well. Could prove useful later, and for now we're not worried about a linter complaining.

function resolvePath(req, rsp, path) {
...
      return sendFile(req, rsp, path)
...
}
function sendFile(req, rsp, path) {
  body = fs.readFileSync(path)
  rsp.writeHead(200,
    {
      'Content-Type': 'text/plain',
      'Content-Length': Buffer.byteLength(body)
    }
  )
  rsp.end(body)
}

const server =  http.createServer((req, rsp) => {
...
  return resolvePath(req, rsp, pathname)
})

Now when we run node index.js and visit http://localhost:3000/index.js we'll get back the content of index.js unaltered.

But we'll need some special handling for Markdown files, so let's add that special case to our sendFile() function:

function sendFile(req, rsp, path) {
  if (path.endsWith('.md')) {
    return sendMarkdown(req, rsp, path)
  }
  body = fs.readFileSync(path)
  rsp.writeHead(200,
    {
      'Content-Type': 'text/plain',
      'Content-Length': Buffer.byteLength(body)
    }
  )
  rsp.end(body)
}

Commence Markdownification

Ah, Internet, how I love thee! There are plenty of libraries which will convert Markdown into HTML, all just a search-engine away. Let's add one to our project:

npm install --save showdown

And write our sendMarkdown() function:

const showdown  = require('showdown')
showdown.setFlavor('github')

function sendMarkdown(req, rsp, path) {
  body = fs.readFileSync(path, 'utf-8')
  converter = new showdown.Converter()
  html = '<html><body>'
    + converter.makeHtml(body)
    + "</body></html>";
  rsp.writeHead(200,
    {
      'Content-Type': 'text/html',
      'Content-Length': Buffer.byteLength(html)
    }
  )
  rsp.end(html)
}

Since the showdown converter doesn't wrap it's output in html tags, we do that inline, and set the content type correctly.

What about all those other file types?

So far, we've completed a few of the original goals. Let's modify sendFile() to handle more file types, including those we will likely encounter along the way.

function sendFile(req, rsp, path) {
  let ctype
  ext = path.split('/').slice(-1)[0].split('.').slice(-1)[0]
  switch (ext) {
    case 'css':
    case 'html':
      ctype = `text/${ext}`; break
    case 'gif':
    case 'jpeg':
    case 'png':
      ctype = `image/${ext}`; break
    case 'jpg':
      ctype = 'image/jpeg'; break
    case 'js':
      ctype = 'text/javascript'; break
    case 'md':
      return sendMarkdown(req, rsp, path)
    default:
      ctype = 'text/plain'
  }
  body = fs.readFileSync(path)
  rsp.writeHead(200,
    {
      'Content-Type': ctype,
      'Content-Length': Buffer.byteLength(body)
    }
  )
  rsp.end(body)
}

We capture the file extension (if any), and add a switch statement to map the usual gang of file extensions (with a special case for .jpg) the the correct MIME type. Yes, there are libraries for this, but this suffices for our very simple needs, and keeps the project lightweight.

Now we can serve up files which are named explicitly, and we are converting Markdown to HTML, but we are not handling directory indexes or mapping an .html URL to a corresponding .md file.

Handling directories and non-existent paths

If you visit http://localhost:3000/, the request will time out, because we're not sending any data unless we find a file, and / is NOT a file, per-se. What we really want is to serve an index file (either index.html or index.md) when a request maps to a directory. And we want to send back a 404 error if we can't find the requested file or the index for a directory.

Let's modify resolvePath() a bit:

function resolvePath(req, rsp, path) {
  path = pathlib.normalize(path)
  if (fs.existsSync(path)) {
    if (fs.lstatSync(path).isFile())  {
      return sendFile(req, rsp, path)
    } else if (fs.lstatSync(path).isDirectory()) {
      return resolvePath(req, rsp, `${path.replace(/\/$/, '')}/index.html`)
    }
  } else {
    return sendError(req, rsp, 404))
  }
}

First, we normalize the path to account for segments like ../../stuff and other variations.

Then, if the path exists and is a file, we call sendFile() as before. If it's a directory, we call resolvePath() (recursively) for a theoretical index.html in that directory. We do a little regex work to ensure have exactly one '/' between the original URL and index.html. If the path doesn't exits, we'll call sendError() to send back a 404-Not-Found. Probably should write that, too:

function sendError(req, rsp, code) {
  messages = {
    404: "Not Found"
  }
  rsp.writeHead(code, {'Content-Type': 'text/html'})
  rsp.end(`<html><body><h1>${code} ${messages[code]}</h1></body</html>`)
}

Serving Markdown where HTML was requested

If the URL ends in .html, but no such file exits, we should look for a file with the same name, but ending in .md. This, once again, is handled by resolvePath():

function resolvePath(req, rsp, path) {
  path = pathlib.normalize(path)
  if (fs.existsSync(path)) {
...
  } else if (path.endsWith('.html')) {
    // Maybe there's a corresponding .md file we can send instead?
    return resolvePath(req, rsp, path.slice(0, -5) + ".md")
  } else {
    return sendError(req, rsp, 404)
  }
}

We've wedged another if-clause between "the path exits" and "send a not-found" which will, if the path ends in .html, call resolvePath() with the updated pathname.

All the boxes ticked!

That wraps it up. We've completed all of the original objectives, to wit:

We have an HTTP server
which sends files from the current directory (with the correct MIME-types, mostly)
and processes Markdown into HTML on the fly
and if a request for an .html resource comes in but no such file exists, it tries the corresponding .md file
and if the URL looks like a directory, it serves up index.html (which could actually be index.md)
and gives meaningful errors as needed (just 404, but they're meaningful!)

You may recall I mentioned adding features to resolveURL(), but those aren't covered here. You can find a more complete version of this code in the eh-mdserver GitHub project, which has a few more tricks up it's sleeve.

The repository also contains the version we built here for your kind consideration.

As always, your constructive comments are very welcome!

This Just In: Yet Another Local Web Server

Comments (1)

More from this blog

Building a CLI in Node for DevPail

Building a Docker/Gulp-based development environment: Introducing DevPail

Events: Web Components Phone Home

Component Styling For Fun and Profit

Jekyll, et al, Not Really to the Rescue

I LOVE to tinker

Let's convert the URL to a local pathname

Now let's send a file

Commence Markdownification

What about all those other file types?

Handling directories and non-existent paths

Serving Markdown where HTML was requested

All the boxes ticked!

Command Palette

Comments (1)

More from this blog

Jekyll, et al, Not Really to the Rescue

I LOVE to tinker

Let's convert the URL to a local pathname

Now let's send a file

Commence Markdownification

What about all those other file types?

Handling directories and non-existent paths

Serving Markdown where HTML was requested

All the boxes ticked!