17 KiB
17 KiB
for correction
man
- htons, htonl, ntohs, ntohl : converts the unsigned short or integer argument between host byte order and network byte order
- poll : waits for one of a set of file descriptors to become ready to perform I/O
- alternatives : select, epoll (epoll_create, epoll_ctl, epoll_wait), kqueue (kqueue, kevent)
- socket : creates an endpoint for communication and returns a file descriptor that refers to that endpoint
- listen : marks a socket as a passive socket, that is, as a socket that will be used to accept incoming connection requests using accept()
- accept : used with connection-based socket types. It extracts the first connection request on the queue of pending connections for the listening socket, creates a new connected socket, and returns a new file descriptor referring to that socket. The newly created socket is not in the listening state. The original socket is unaffected by this call
- send : (~write) used to transmit a message to another socket. May be used only when the socket is in a connected state (so that the intended recipient is known). The only difference between send() and write() is the presence of flags. With a zero flags argument, send() is equivalent to write()
- recv : (~read) used to receive messages from a socket. May be used to receive data on both connectionless and connection-oriented sockets. The only difference between recv() and read() is the presence of flags. With a zero flags argument, recv() is generally equivalent to read()
- bind : associate a socket fd to a local address. When a socket is created with socket(), it exists in a name space (address family) but has no address assigned to it. It is normally necessary to assign a local address using bind() before a socket may receive connections (see accept())
- connect : connects a socket fd to a remote address
- inet_addr : converts the Internet host address cp from IPv4 numbers-and-dots notation into binary data in network byte order. Use of this function is problematic because in case of error it returns -1, wich is a valid address (255.255.255.255). Avoid its use in favor of inet_aton(), inet_pton(), or getaddrinfo()
- setsockopt : manipulate options for a socket fd. Options may exist at multiple protocol levels; they are always present at the uppermost socket level
- getsockname : returns the current address to which a socket fd is bound
- fcntl : manipulate an open fd, by performing some actions, like duplicate it or changing its flags
correction
general
- launch the installation of siege with homebrew
- explain the basics of an HTTP server
- ask which function they used for I/O Multiplexing
- ask to get an explanation of how select (or equivalent) is working
- ask if they use only one select (or equivalent) and how they've managed the server accept and the client read/write
- the select (or equivalent) should be in the main loop and should check fd for read and write AT THE SAME TIME, if not please give a 0 and stop the evaluation
- there should be only one read or one write per client per selct (or equivalent). Ask to show the code that goes from the select (or equivalent) to the read and write of a client
- search for all read/recv/write/send on a socket and check that if an error returned the client is removed
- search for all read/recv/write/send and check if the returned value is well checked. (checking only -1 or 0 is not good, you should check both)
- if a check of errno is done after read/recv/write/send, please stop the evaluation and put a mark to 0
- writing ot reading ANY file descriptor withour going through the select (or equivalent) is strickly FORBIDDEN
configuration
- look for the HTTP response status codes list on internet and during this evaluation. if any status codes is wrong don't give related points.
- setup multiple servers with different port
- setup multiple servers with different hostname (use something like: curl --resolve example.com:80:127.0.0.1 http://example.com/)
- setup default error page (try to change the error 404)
- limit the client body (use curl -X POST -H "Content-Type: plain/text" --data "BODY IS HERE write something shorter or longer than body limit")
- setup routes in a server to different directories
- setup a default file to search for if you ask for a directory
- setup a list of method accepted for a certain route (ex: try to delete something with and without permission)
basic checks
Using telnet, curl, prepared files demonstrates that the following features work properly:
- GET requests -> should work
- POST requests -> should work
- DELETE requests -> should work
- UNKNOWN requests -> should not produce any crash
- For every test the status code must be good
- upload some file to the server and get it back
Check with a browser
- Use the reference browser of the team, open the network part of it and try to connect to the server with it
- Look at the request header and response header
- It should be compatible to serve a fully static website
- Try a wrong URL on the server
- Try to list a directory
- Try a redirected URL
- Try things
Port issues
- In the configuration file setup multiple ports and use different websites, use the browser to check that the configuration is working as expected, and show the right website.
- In the configuration try to setup the same port multiple times. It should not work.
- Launch multiple servers at the same time with different configurations but with common ports. Is it working? If it is working, ask why the server should work if one of the configurations isn't working. keep going
Siege & stress test
- Use Siege to run some stress tests.
- Availability should be above 99.5% for a simple get on an empty page with a siege -b on that page
- Check if there is no memory leak (monitor the process memory usage it should not go up indefinitely)
- Check if there is no hanging connection
- You should be able to use siege indefinitely without restarting the server (look at siege -b)
todo
- read the RFC and do some tests with telnet and NGINX
parsing config
- Your program has to take a configuration file as argument, or use a default path.
- Choose the port and host of each ’server’.
- Setup the server_names or not.
- The first server for a host:port will be the default for this host:port (that means it will answer to all the requests that don’t belong to an other server).
- Setup default error pages.
- Limit client body size.
- Setup routes with one or multiple of the following rules/configuration (routes wont be using regexp):
- Define a list of accepted HTTP methods for the route.
- Define a HTTP redirection.
- Define a directory or a file from where the file should be searched (for example, if url /kapouet is rooted to /tmp/www, url /kapouet/pouic/toto/pouet is /tmp/www/pouic/toto/pouet).
- Turn on or off directory listing.
- Set a default file to answer if the request is a directory.
- Execute CGI based on certain file extension (for example .php).
- Make the route able to accept uploaded files and configure where they should be saved.
connection basic
- You can’t execve another web server.
- Your server must never block and the client can be bounced properly if necessary.
- It must be non-blocking and use only 1 poll() (or equivalent) for all the I/O operations between the client and the server (listen included).
- poll() (or equivalent) must check read and write at the same time.
- You must never do a read or a write operation without going through poll() (or equivalent).
- Checking the value of errno is strictly forbidden after a read or a write operation.
- You don’t need to use poll() (or equivalent) before reading your configuration file. Because you have to use non-blocking file descriptors, it is possible to use read/recv or write/send functions with no poll() (or equivalent), and your server wouldn’t be blocking. But it would consume more system resources. Thus, if you try to read/recv or write/send in any file descriptor without using poll() (or equivalent), your grade will be 0.
- You can use every macro and define like FD_SET, FD_CLR, FD_ISSET, FD_ZERO (understanding what and how they do it is very useful).
- A request to your server should never hang forever.
- Your server must be compatible with the web browser of your choice.
parsing request HTTP (fields, ...)
- We will consider that NGINX is HTTP 1.1 compliant and may be used to compare headers and answer behaviors.
response HTTP (fields, ...)
- Your HTTP response status codes must be accurate.
- You server must have default error pages if none are provided.
- You can’t use fork for something else than CGI (like PHP, or Python, and so forth).
- You must be able to serve a fully static website.
upload files
- Clients must be able to upload files.
CGI
- You need at least GET, POST, and DELETE methods.
- Do you wonder what a CGI is?
- Because you won’t call the CGI directly, use the full path as PATH_INFO.
- Just remember that, for chunked request, your server needs to unchunked it and the CGI will expect EOF as end of the body.
- Same things for the output of the CGI. If no content_length is returned from the CGI, EOF will mark the end of the returned data.
- Your program should call the CGI with the file requested as first argument.
- The CGI should be run in the correct directory for relative path file access.
- Your server should work with one CGI (php-CGI, Python, and so forth).
write tests
- Stress tests your server. It must stay available at all cost.
- Do not test with only one program.
- Write your tests with a more convenient language such as Python or Golang, and so forth. Even in C or C++ if you want to
persistent connexion
- Your server must be able to listen to multiple ports (see Configuration file)
- Your server should never die.
cgi env variables
cgi env variables wikipedia variables environnements cgi cgi server variables on adobe
AUTH_TYPE : if the srcipt is protected, the authentification method used to validate the user
CONTENT_LENGTH : length of the request content
CONTENT_TYPE : if there is attached information, as with method POST or PUT, this is the content type of the data (e.g. "text/plain", it is set by the attribute "enctype" in html <form> as three values : "application/x-www-form-urlencoded", "multipart/form-data", "text/plain")
GATEWAY_INTERFACE : CGI version (e.g. CGI/1.1)
PATH_INFO : if any, path of the resquest in addition to the cgi script path (e.g. for cgi script path = "/usr/web/cgi-bin/script.cgi", and the url = "http://server.org/cgi-bin/script.cgi/house", the PATH-INFO would be "house")
PATH_TRANSLATED : full path of the request, like path-to-cgi/PATH_INFO, null if PATH_INFO is null (e.g. for "http://server.org/cgi-bin/prog/the/path", PATH_INFO would be : "/the/path" and PATH_TRANSLATED would be : "/usr/web/cgi-bin/prog/the/path")
QUERY_STRING : everything following the ? in the url sent by client (e.g. for url "http://server.org/query?var1=val2&var2=val2", it would be : "var1=val2&var2=val2")
REMOTE_ADDR : ip address of the client
REMOTE_HOST : host name of the client, empty if not known, or equal to REMOTE_ADDR
REMOTE_IDENT : if known, username of the client, otherwise empty, use for logging only
REMOTE_USER : username of client, if script is protected and the server support user authentification
REQUEST_METHOD : method used for the request (for http, usually POST or GET)
SCRIPT_NAME : path to the cgi, relative to the root, used for self-referencing URLs (e.g. "/cgi-bin/script.cgi")
SERVER_NAME : name of the server, as hostname, IP address, or DNS (e.g. dns : "www.server.org")
SERVER_PORT : the port number your server is listening on (e.g. 80)
SERVER_PROTOCOL : protocol used for the request (e.g. HTTP/1.1)
SERVER_SOFTWARE : the server software you're using (e.g. Apache 1.3)
REDIRECT_STATUS : for exemple, 200
ressources
- create an http server
- guide to network programming
- same, translated in french
- bind() vs connect()
- INADDR_ANY for bind
- hack with CGI
code architecture
functions action in this scenario :
______
sd = SOCKET() : create a listening socket descriptor
__________
SETSOCKOPT(sd) : allow socket descriptor to be reuseable
_____
IOCTL(sd) : set listening socket and all incoming socket to be non-blocking
_____
FCNTL(sd) : set listening socket and all incoming socket to be non-blocking
____
BIND(port) : associate listening socket to a port
______
LISTEN(nb_queue) : queue the incoming connections to listening socket
up to a chosen number
____
POLL(fds[]) : wait for event on files descriptors
______
SELECT(fds[]) : wait for event on files descriptors
FD_SET() : add a fd to a set
FD_CLR() : remove a fd from a set
FD_ZERO() : clears a set
FD_ISSET() : test to see if a fd is part of the set
______
new_sd = ACCEPT() : extract first connection request in queue of listening socket
and creates a new socket that is connected
____
RECV(new_sd) : read data in socket created by accept()
____
SEND(new_sd) : write data in socket created by accept()
_____
CLOSE(new_sd) : close open file (here socket) descriptor
compare architectures :
POLL SELECT
______ ______ ______
lstn_sd = SOCKET() | lstn_sd = SOCKET() | lstn_sd = SOCKET()
| __________ | __________
| SETSOCKOPT() | SETSOCKOPT()
| _____ | _____
| IOCTL() | IOCTL()
____ | ____ | ____
BIND(port) | BIND(port) | BIND(port)
______ | ______ | ______
LISTEN(nb_queue) | LISTEN(nb_queue) | LISTEN(nb_queue)
| |
| fds[1] = lstn_sd | FD_SET(lstn_fd)
| |
| | max_sd = lstn_sd
| |
loop | loop | loop
. ____ | . ____ | . ______
. POLL() | . POLL(fds[]) | . SELECT(fds[])
. | . | .
. | . loop through fds[] | . loop i++ < max_fd
. | . . | . .
. | . . POLLIN && lstn_sd ? | . . FD_ISSET(i) & lstn_fd ?
. | . . loop | . . loop
. ______ | . . . ______ | . . . ______
. ACCEPT() | . . . new_sd = ACCEPT() | . . . new_sd = ACCEPT()
. | . . . | . . .
. | . . . fds[] += new_sd | . . . FD_SET new_sd in fds[]
. | . . | . . .
. | . . | . . . max_sd = new_sd
. | . . | . .
. | . . or POLLIN ? | . . or FD_ISSET ?
. | . . loop | . . loop
. ____ | . . . ____ | . . . ____
. RECV() | . . . RECV() | . . . RECV()
. ____ | . . . ____ | . . . ____
. SEND() | . . . SEND() | . . . SEND()
. | |
. | loop through fds[] | loop through fds[]
. _____ | . _____ | . _____
. CLOSE(fds[]) | . CLOSE(fds[]) | . CLOSE(fds[])
first, create the socket, add some options, bind it, and listen to it.
then, do a loop starting with POLL or SELECT
if it's incomming connexions, accept and add them all to the list
if it's accepted connexions, read their datas and write new datas