work together
next commit
questions
- how should we handle a wrong url like
http://localhost/cgi-bin/wrong.phpp/good.php?- do we serve
./srcs/cgi-bin/good.php? - or do we return 404 "not found" ? -> - for now, execve would crash, but that doesn't produce a 404 error, rather a 500, is it bad ?
- could we use errno after execve to choose an appropriate http error ? subject says : "Checking the value of errno is strictly forbidden after a read or a write operation"
- do we serve
- if a url has a file with extension, but it's not a cgi extension, is it necessary to look further ?
- ex.
http://localhost/file.php/file.pyforcgi_ext py;?
- ex.
- the response page is received long after the cgi-script is done, why ?
man
- htons, htonl, ntohs, ntohl : converts the unsigned short or integer argument between host byte order and network byte order
- poll : waits for one of a set of file descriptors to become ready to perform I/O
- alternatives : select, epoll (epoll_create, epoll_ctl, epoll_wait), kqueue (kqueue, kevent)
- socket : creates an endpoint for communication and returns a file descriptor that refers to that endpoint
- listen : marks a socket as a passive socket, that is, as a socket that will be used to accept incoming connection requests using accept()
- accept : used with connection-based socket types. It extracts the first connection request on the queue of pending connections for the listening socket, creates a new connected socket, and returns a new file descriptor referring to that socket. The newly created socket is not in the listening state. The original socket is unaffected by this call
- send : (~write) used to transmit a message to another socket. May be used only when the socket is in a connected state (so that the intended recipient is known). The only difference between send() and write() is the presence of flags. With a zero flags argument, send() is equivalent to write()
- recv : (~read) used to receive messages from a socket. May be used to receive data on both connectionless and connection-oriented sockets. The only difference between recv() and read() is the presence of flags. With a zero flags argument, recv() is generally equivalent to read()
- bind : associate a socket fd to a local address. When a socket is created with socket(), it exists in a name space (address family) but has no address assigned to it. It is normally necessary to assign a local address using bind() before a socket may receive connections (see accept())
- connect : connects a socket fd to a remote address
- inet_addr : converts the Internet host address cp from IPv4 numbers-and-dots notation into binary data in network byte order. Use of this function is problematic because in case of error it returns -1, wich is a valid address (255.255.255.255). Avoid its use in favor of inet_aton(), inet_pton(), or getaddrinfo()
- setsockopt : manipulate options for a socket fd. Options may exist at multiple protocol levels; they are always present at the uppermost socket level
- getsockname : returns the current address to which a socket fd is bound
- fcntl : manipulate an open fd, by performing some actions, like duplicate it or changing its flags
todo
- read the RFC and do some tests with telnet and NGINX
parsing config
- Your program has to take a configuration file as argument, or use a default path.
- Choose the port and host of each ’server’.
- Setup the server_names or not.
- The first server for a host:port will be the default for this host:port (that means it will answer to all the requests that don’t belong to an other server).
- Setup default error pages.
- Limit client body size.
- Setup routes with one or multiple of the following rules/configuration (routes wont be using regexp):
- Define a list of accepted HTTP methods for the route.
- Define a HTTP redirection.
- Define a directory or a file from where the file should be searched (for example, if url /kapouet is rooted to /tmp/www, url /kapouet/pouic/toto/pouet is /tmp/www/pouic/toto/pouet).
- Turn on or off directory listing.
- Set a default file to answer if the request is a directory.
- Execute CGI based on certain file extension (for example .php).
- Make the route able to accept uploaded files and configure where they should be saved.
connection basic
- You can’t execve another web server.
- Your server must never block and the client can be bounced properly if necessary.
- It must be non-blocking and use only 1 poll() (or equivalent) for all the I/O operations between the client and the server (listen included).
- poll() (or equivalent) must check read and write at the same time.
- You must never do a read or a write operation without going through poll() (or equivalent).
- Checking the value of errno is strictly forbidden after a read or a write operation.
- You don’t need to use poll() (or equivalent) before reading your configuration file. Because you have to use non-blocking file descriptors, it is possible to use read/recv or write/send functions with no poll() (or equivalent), and your server wouldn’t be blocking. But it would consume more system resources. Thus, if you try to read/recv or write/send in any file descriptor without using poll() (or equivalent), your grade will be 0.
- You can use every macro and define like FD_SET, FD_CLR, FD_ISSET, FD_ZERO (understanding what and how they do it is very useful).
- A request to your server should never hang forever.
- Your server must be compatible with the web browser of your choice.
parsing request HTTP (fields, ...)
- We will consider that NGINX is HTTP 1.1 compliant and may be used to compare headers and answer behaviors.
response HTTP (fields, ...)
- Your HTTP response status codes must be accurate.
- You server must have default error pages if none are provided.
- You can’t use fork for something else than CGI (like PHP, or Python, and so forth).
- You must be able to serve a fully static website.
upload files
- Clients must be able to upload files.
CGI
- You need at least GET, POST, and DELETE methods.
- Do you wonder what a CGI is?
- Because you won’t call the CGI directly, use the full path as PATH_INFO.
- Just remember that, for chunked request, your server needs to unchunked it and the CGI will expect EOF as end of the body.
- Same things for the output of the CGI. If no content_length is returned from the CGI, EOF will mark the end of the returned data.
- Your program should call the CGI with the file requested as first argument.
- The CGI should be run in the correct directory for relative path file access.
- Your server should work with one CGI (php-CGI, Python, and so forth).
write tests
- Stress tests your server. It must stay available at all cost.
- Do not test with only one program.
- Write your tests with a more convenient language such as Python or Golang, and so forth. Even in C or C++ if you want to
persistent connexion
- Your server must be able to listen to multiple ports (see Configuration file)
- Your server should never die.
cgi rfc
output cgi script :
summary :
- the cgi-script will send back at least one header field followed by an empty line
- this header field will be one of three :
- "Content-Type"
- "Location"
- "Status"
- the cgi-script may send back more header fields
- the server must check and modify few things :
- there is no field duplicate (resolve conflicts)
- there is no space between filed name and ":"
- change all the '\n' by '\r\n'
- if no Location field && no Status field -> status code = 200
- handle Location field, either :
- local : start with '/' --> rerun the request with new uri
- client : start with ':' --> send back status code 302
- there is at least one header field followed by '\r\n\r\n' :
- "Content-Type"
- "Location"
- "Status"
- if status field, change server status for this one
- to pass the body-message to the cgi-script, we write it into the temporary fd on which the script read it's standard input
- The server [...] receives the request from the client
- selects a CGI script to handle the request
- converts the client request to a CGI request
- executes the script and converts the CGI response into a response for the client
- the 'Script-URI' [...] MUST have the property that if the client had accessed this URI instead, then the script would have been executed
4: how the server prepare the cgi requests
- the cgi receives 2 differents set of informations :
- the request meta-variables (in UNIX, by env variables)
- and the message-body
- a header field that spans multiple lines MUST be merged onto a single line
- unless defined otherwise, the script access request data by reading stdin
6: how the response from the script is returned to the server
- The response comprises 2 parts, separated by a blank line :
- a message-header
- and a message-body
- The message-header contains one or more header fields
- The body may be NULL
- four types of responses :
- document response
- local redirect response
- client redirect response
- client redirect response with document
- document response :
- it must return a Content-Type header field
- a Status-Header field is optional (200 is assumed if omited)
- the server must check the cgi-script output, and modifie it to comply with the protocol version
- local redirect response :
- it must return only a Location field
- it contains a local path URI and query string ('local-pathquery')
- the local path URI must start with a "/"
- the server must generate the response for this local-pathquery
- client redirect response :
- it must return only a Location field
- it contains an absolute URI path, to indicate the client that it should reprocess the request with this URI
- the absolute URI always start with the name of scheme followed by ":"
- the http server must generate a 302 'Found' message
- client redirect response with document
- it must return a Location field with an absolute URI path
- it must return the Status header field, with a value of 302 'Found'
- the server must check the cgi-script output, and modifie it to comply with the protocol version
- whitespace is permitted between the ":" and the field-value
- but not between the field-name and the ":"
- the CGI script can set three differents fields :
- Content-Type
- Location
- Status
- Content-Type :
- if there is a body in the response, a Content-Type field must be present
- if there is no Content-Type, the server must not attempt to determine one
- Location :
- the local URI path must be an absolut path, not a relative path, nor NULL
- the local URI path must, then, start with "/"
- the absolut URI start with ":"
- Status :
- a 3-digit integer code
- 4 standards :
- 200 'OK' indicates success, it's the default value
- 302 'Found' with Location header and response message-body
- 400 'Bad Request' an unknown request format, like missing CONTENT-TYPE
- 501 'Not Implemented' the script received unsupported REQUEST-METHOD
- construction:
Status:400 "explication of the error"\n
- the cgi-script can return other header fields, concerning the response message
- the server must translate cgi-headers syntax into http-header syntax
- for exemple, newline can be encoded in different ways
- the cgi-script must not return header fields concerning client-side communication
- the server can remove such fields
- (not sure : https://www.rfc-editor.org/rfc/rfc3875#section-6.3.4)
- the server must resolve conflicts between script-header fields and themselves
- the server must read it untill EOF
- the server must not modify it, except to convert charset if needed
7 and 8: usefull informations about implementation and security
cgi env variables
cgi env variables wikipedia variables environnements cgi cgi server variables on adobe
AUTH_TYPE : if the srcipt is protected, the authentification method used to validate the user
CONTENT_LENGTH : length of the request body-message
CONTENT_TYPE : (Content-Type field) if there is attached information, as with method POST or PUT, this is the content type of the data (e.g. "text/plain", it is set by the attribute "enctype" in html <form> as three values : "application/x-www-form-urlencoded", "multipart/form-data", "text/plain")
GATEWAY_INTERFACE : CGI version (e.g. CGI/1.1)
PATH_INFO : if any, path of the resquest in addition to the cgi script path (e.g. for cgi script path = "/usr/web/cgi-bin/script.cgi", and the url = "http://server.org/cgi-bin/script.cgi/house", the PATH-INFO would be "house")
PATH_TRANSLATED : full path of the request, like path-to-cgi/PATH_INFO, null if PATH_INFO is null (e.g. for "http://server.org/cgi-bin/prog/the/path", PATH_INFO would be : "/the/path" and PATH_TRANSLATED would be : "/usr/web/cgi-bin/prog/the/path")
QUERY_STRING : everything following the ? in the url sent by client (e.g. for url "http://server.org/query?var1=val2&var2=val2", it would be : "var1=val2&var2=val2")
REMOTE_ADDR : ip address of the client
REMOTE_HOST : host name of the client, empty if not known, or equal to REMOTE_ADDR
REMOTE_IDENT : if known, username of the client, otherwise empty, use for logging only
REMOTE_USER : username of client, if script is protected and the server support user authentification
REQUEST_METHOD : method used for the request (for http, usually POST or GET)
SCRIPT_NAME : path to the cgi, relative to the root, used for self-referencing URLs (e.g. "/cgi-bin/script.cgi")
SERVER_NAME : name of the server, as hostname, IP address, or DNS (e.g. dns : "www.server.org")
SERVER_PORT : the port number your server is listening on (e.g. 80)
SERVER_PROTOCOL : protocol used for the request (e.g. HTTP/1.1)
SERVER_SOFTWARE : the server software you're using (e.g. Apache 1.3)
REDIRECT_STATUS : for exemple, 200
http errors
HTTP Client Errors
- 400 Bad Request This error code indicates that the request cannot be processed because of wrong syntax usage by the client.
- 401 Unauthorized This error code indicates that the client is not authorized to receive the requested data, without authentication. A login name and password based authentication might be required to access the requested data.
- 403 Forbidden There is no way you can access the requested data. A 403 error announces that the data is off limits.
- 404 Not Found This error indicates that the resources requested by the client are currently unavailable.
- 405 Method Not Allowed This error indicates wrong usage of request method. Depending on the kind of data requested, the appropriate request method must be chosen.
- 406 Not Acceptable When the data provided by a web server does not match the specifications made in ‘Accept’ header of the client HTTP request, this error is the result.
- 407 Proxy Authentication Required This error clearly indicates that an authentication from the proxy server is required to gain access to requested resources.
- 408 Request Timeout This type of error indicates that the client was delayed in making a request, within the specified time allocated to it, by the server.
- 409 Conflict This error code is displayed when the server perceives a conflict between two requests made simultaneously by different clients, for the same resource.
- 410 Gone This error code indicates that the requested data is no longer hosted on the server and therefore further requests made for it, would be futile.
- 411 Length Required If the request made by the client does not include information about the length of the requested data or resource, this error code is displayed.
- 412 Precondition Failed Some requests made by clients come attached with conditions that need to be satisfied by the server, before data transaction may happen. If these conditions are not met, error 412 results.
- 413 Request Entity Too Large When a client makes a request which is too overwhelming for the server’s resources to handle, it presents this error code.
- 414 Requested URI Too Long A Uniform Resource Identifier (URI) is a character string used to describe a data stream or resource on a server. Error 414 occurs when the server is unable to process the URI, because of limited resources and long string length.
- 415 Unsupported Media Type A server may be designed to allow only certain formats for media files. When error 415 is displayed, it indicates that the format of file being uploaded through a client request, does not match the requisite format.
- 416 Request Range Not Satisfiable Sometimes, a client may request for only a small part of a file, instead of asking for the entire file. If this request is not specified properly and the part of the file requested does not exist, this error is displayed.
- 417 Expectation Failed This error code is displayed when the server cannot meet the specifications provided in the request.
- 422 Unprocessable Entity This error is displayed when the request made, cannot be processed due to an error in semantic structure.
- 423 Locked This error is displayed when a requested piece of data or resource has been locked, making it inaccessible for a server.
- 424 Failed Dependency A server may process a succession of requests from a client with the fulfillment of each, dependent on the one made before. This error is displayed when a request made before is not fulfilled, due to which the current request cannot be processed.
- 426 Upgrade Required This error signifies that the client may need to switch over to a secure protocol like TLS to get the request processed.
- 444 No Response This error signifies that the server has simply rejected the client request and terminated connection.
- 449 Retry With This is a request made by the server to the client, to make the request again after executing certain actions or making specific changes in request. This is an error code introduced by Microsoft.
- 499 Client Closed Request When client terminates a connection made with the server, while its processing the associated request, this error code is displayed.
- 450 Blocked By Windows Parental Controls Another error code introduced by Microsoft, this one is displayed when a URL is blocked by parental control settings on the web browsers.
HTTP Server Errors
- 500 Internal Server Error A generic message displayed by the server, when the problem with the request cannot be specified by any other appropriate code.
- 501 Not Implemented This error indicates the inability of the server to process a request, as it hasn’t been configured to respond to the request method used.
- 502 Bad Gateway Sometimes, hosted pages on web servers are transmitted to client via proxy servers. If the proxy server (to which a client has sent a request), fails connecting with the web server (known as the upstream server), error 502 results.
- 503 Service Unavailable When the server is already overloaded with multiple requests, it will temporarily stop entertaining new requests, by displaying a 503 error code.
- 504 Gateway Timeout When the request made by a proxy server to the web server hosting a resource times out, error 504 is reported.
- 505 HTTP Version Not Supported An error code seen rarely, it is displayed when the web server does not support the protocol version of the client request.