sgdev: Back to basics: A demo http server and the TCP/IP stack on Windows

Let us build a web server

At work, I spend a lot of time on trivial things like 'how to fix html/css to make this page work for IE7'. I realized it has been a long time since I had to look at how the web server which powers our product really works under the hood. The best way to refresh my memory was - implement a web server and learn how it works bottom up. I used the language and platform I am currently using at work: C# and Windows.

Webserver implementation

Source code in GitHub
I updated code someone left on pastebin. The link is gone now, so I cant reference it. This is a toy implementation, error handling or performance are not considered. It uses the .NET TCP library and implements a small subset of the http server side protocol. Features:
- Simple threaded handling of requests
- Logging
- URL re-write
- Implement GET and POST
- CGI meta-variables
- Powershell as a CGI extension
- Process Cgi and File GET
- Implement some error codes - 400, 403 and 404

TCP/IP

If I had to implement any of the protocol layers, the RFCs and standards corresponding to that layer would be a good starting point.

Diagram modified from: http://www.codeproject.com/KB/IP/serversocket.aspx

Windows TCP/IP stack

OSI Layers 1 and 2

The driver interface is generally implemented by a Network Interface Card (NIC) vendor. To program a driver, you need to get the Windows Dev Kit (WDK), ndis.lib (Network Driver Interface Specification library). Good places to start would be the WDK 'passthru' sample and source code for the WinPcap (Windows Packet Capture) library. C or C++ are the languages to program with the NDIS.

OSI Layers 3 and 4

TCP/IP is implemented in the protocol driver TCPIP.sys, as part of the Windows kernel. At this layer AFAIK there is limited programmability. Windows does provide several configuration settings in the registry: TcpNumConnections, TcpWindowsSize and others. There are command line programs which operate at this level: arp, route, netstat, tracert, netsh, nsloopup, ipconfig. I have read about people who actually patch the TCPIP.sys, but I dont know how this is done.

OSI Layers 5-7

At this layer we are above the kernel in the user space. Various dlls implement different services:
TCP and UDP sockets - ws2_32.dll, winsock.dll
WinInet (FTP, HTTP 1.0, HTTP 1.1) - wininet.dll
RPC - rpc*.dll
NetBIOS - netapi.dll
WNet - mpr.dll

Diagram modified from: http://i.msdn.microsoft.com/dynimg/IC72693.gif

Protocol

Notes on protocol specs for the web server implementation.

HTTP 1.1 – RFC 2616

- The HTTP protocol is a request/response protocol.
- A client sends a request to the server in the form of a request method, URI, and protocol version, followed by a MIME-like message containing request modifiers, client information, and possible body content over a connection with a server.
- The server responds with a status line, including the message's protocol version and a success or error code, followed by a MIME-like message containing server information, entity meta-information, and possible entity-body content.

MIME – RFC 2045 etc.

- Multipurpose Internet Mail Extension
- Defines how to send data over the internet
- Content-types: text, multipart, message, application, image, audio, video

Diagram from: http://docs.oracle.com/cd/E19957-01/816-6028-10/asd3j.htm

HTTP Communication

request chain ------------------------>
<---------------------- response chain
UserAgent------------- Origin server
example
Browser---------------- Webserver

request chain -------------------------------------->
<------------------------------------- response chain
UserAgent --------A-------B------- Origin server
example
Browser----Firewall----Router----- Webserver

Http Request:

Http Response:

CGI – RFC 3875

sgdev

Wednesday, September 4, 2013

Back to basics: A demo http server and the TCP/IP stack on Windows