1148871Scperciva/*- 2148871Scperciva * Copyright 2005 Colin Percival 3148871Scperciva * All rights reserved 4148871Scperciva * 5148871Scperciva * Redistribution and use in source and binary forms, with or without 6148871Scperciva * modification, are permitted providing that the following conditions 7148871Scperciva * are met: 8148871Scperciva * 1. Redistributions of source code must retain the above copyright 9148871Scperciva * notice, this list of conditions and the following disclaimer. 10148871Scperciva * 2. Redistributions in binary form must reproduce the above copyright 11148871Scperciva * notice, this list of conditions and the following disclaimer in the 12148871Scperciva * documentation and/or other materials provided with the distribution. 13148871Scperciva * 14148871Scperciva * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 15148871Scperciva * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 16148871Scperciva * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 17148871Scperciva * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY 18148871Scperciva * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 19148871Scperciva * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 20148871Scperciva * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 21148871Scperciva * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, 22148871Scperciva * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING 23148871Scperciva * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 24148871Scperciva * POSSIBILITY OF SUCH DAMAGE. 25148871Scperciva */ 26148871Scperciva 27148871Scperciva#include <sys/cdefs.h> 28148871Scperciva__FBSDID("$FreeBSD$"); 29148871Scperciva 30148871Scperciva#include <sys/types.h> 31148871Scperciva#include <sys/time.h> 32148871Scperciva#include <sys/socket.h> 33148871Scperciva 34148871Scperciva#include <ctype.h> 35148871Scperciva#include <err.h> 36148871Scperciva#include <errno.h> 37148871Scperciva#include <fcntl.h> 38148871Scperciva#include <limits.h> 39148871Scperciva#include <netdb.h> 40150461Scperciva#include <stdint.h> 41148871Scperciva#include <stdio.h> 42148871Scperciva#include <stdlib.h> 43148871Scperciva#include <string.h> 44148871Scperciva#include <sysexits.h> 45148871Scperciva#include <unistd.h> 46148871Scperciva 47148871Scpercivastatic const char * env_HTTP_PROXY; 48150461Scpercivastatic char * env_HTTP_PROXY_AUTH; 49148871Scpercivastatic const char * env_HTTP_USER_AGENT; 50164057Scpercivastatic char * env_HTTP_TIMEOUT; 51148871Scpercivastatic const char * proxyport; 52150461Scpercivastatic char * proxyauth; 53148871Scperciva 54148871Scpercivastatic struct timeval timo = { 15, 0}; 55148871Scperciva 56148871Scpercivastatic void 57148871Scpercivausage(void) 58148871Scperciva{ 59148871Scperciva 60148871Scperciva fprintf(stderr, "usage: phttpget server [file ...]\n"); 61148871Scperciva exit(EX_USAGE); 62148871Scperciva} 63148871Scperciva 64150461Scperciva/* 65150461Scperciva * Base64 encode a string; the string returned, if non-NULL, is 66150461Scperciva * allocated using malloc() and must be freed by the caller. 67150461Scperciva */ 68150461Scpercivastatic char * 69150461Scpercivab64enc(const char *ptext) 70150461Scperciva{ 71150461Scperciva static const char base64[] = 72150461Scperciva "ABCDEFGHIJKLMNOPQRSTUVWXYZ" 73150461Scperciva "abcdefghijklmnopqrstuvwxyz" 74150461Scperciva "0123456789+/"; 75150461Scperciva const char *pt; 76150461Scperciva char *ctext, *pc; 77150461Scperciva size_t ptlen, ctlen; 78150461Scperciva uint32_t t; 79150461Scperciva unsigned int j; 80150461Scperciva 81150461Scperciva /* 82150461Scperciva * Encoded length is 4 characters per 3-byte block or partial 83150461Scperciva * block of plaintext, plus one byte for the terminating NUL 84150461Scperciva */ 85150461Scperciva ptlen = strlen(ptext); 86150461Scperciva if (ptlen > ((SIZE_MAX - 1) / 4) * 3 - 2) 87150461Scperciva return NULL; /* Possible integer overflow */ 88150461Scperciva ctlen = 4 * ((ptlen + 2) / 3) + 1; 89150461Scperciva if ((ctext = malloc(ctlen)) == NULL) 90150461Scperciva return NULL; 91150461Scperciva ctext[ctlen - 1] = 0; 92150461Scperciva 93150461Scperciva /* 94150461Scperciva * Scan through ptext, reading up to 3 bytes from ptext and 95150461Scperciva * writing 4 bytes to ctext, until we run out of input. 96150461Scperciva */ 97150461Scperciva for (pt = ptext, pc = ctext; ptlen; ptlen -= 3, pc += 4) { 98150461Scperciva /* Read 3 bytes */ 99150461Scperciva for (t = j = 0; j < 3; j++) { 100150461Scperciva t <<= 8; 101150461Scperciva if (j < ptlen) 102150461Scperciva t += *pt++; 103150461Scperciva } 104150461Scperciva 105150461Scperciva /* Write 4 bytes */ 106150461Scperciva for (j = 0; j < 4; j++) { 107150461Scperciva if (j <= ptlen + 1) 108150461Scperciva pc[j] = base64[(t >> 18) & 0x3f]; 109150461Scperciva else 110150461Scperciva pc[j] = '='; 111150461Scperciva t <<= 6; 112150461Scperciva } 113150461Scperciva 114150461Scperciva /* If we're done, exit the loop */ 115150461Scperciva if (ptlen <= 3) 116150461Scperciva break; 117150461Scperciva } 118150461Scperciva 119150461Scperciva return (ctext); 120150461Scperciva} 121150461Scperciva 122148871Scpercivastatic void 123148871Scpercivareadenv(void) 124148871Scperciva{ 125150461Scperciva char *proxy_auth_userpass, *proxy_auth_userpass64, *p; 126150461Scperciva char *proxy_auth_user = NULL; 127150461Scperciva char *proxy_auth_pass = NULL; 128164057Scperciva long http_timeout; 129148871Scperciva 130148871Scperciva env_HTTP_PROXY = getenv("HTTP_PROXY"); 131158301Scperciva if (env_HTTP_PROXY == NULL) 132158301Scperciva env_HTTP_PROXY = getenv("http_proxy"); 133150427Scperciva if (env_HTTP_PROXY != NULL) { 134148871Scperciva if (strncmp(env_HTTP_PROXY, "http://", 7) == 0) 135148871Scperciva env_HTTP_PROXY += 7; 136148880Scperciva p = strchr(env_HTTP_PROXY, '/'); 137148880Scperciva if (p != NULL) 138148880Scperciva *p = 0; 139148871Scperciva p = strchr(env_HTTP_PROXY, ':'); 140148871Scperciva if (p != NULL) { 141148871Scperciva *p = 0; 142148871Scperciva proxyport = p + 1; 143148871Scperciva } else 144148871Scperciva proxyport = "3128"; 145148871Scperciva } 146148871Scperciva 147150461Scperciva env_HTTP_PROXY_AUTH = getenv("HTTP_PROXY_AUTH"); 148150461Scperciva if ((env_HTTP_PROXY != NULL) && 149150461Scperciva (env_HTTP_PROXY_AUTH != NULL) && 150150461Scperciva (strncasecmp(env_HTTP_PROXY_AUTH, "basic:" , 6) == 0)) { 151150461Scperciva /* Ignore authentication scheme */ 152150461Scperciva (void) strsep(&env_HTTP_PROXY_AUTH, ":"); 153150461Scperciva 154150461Scperciva /* Ignore realm */ 155150461Scperciva (void) strsep(&env_HTTP_PROXY_AUTH, ":"); 156150461Scperciva 157150461Scperciva /* Obtain username and password */ 158150461Scperciva proxy_auth_user = strsep(&env_HTTP_PROXY_AUTH, ":"); 159156405Sume proxy_auth_pass = env_HTTP_PROXY_AUTH; 160150461Scperciva } 161150461Scperciva 162150461Scperciva if ((proxy_auth_user != NULL) && (proxy_auth_pass != NULL)) { 163150461Scperciva asprintf(&proxy_auth_userpass, "%s:%s", 164150461Scperciva proxy_auth_user, proxy_auth_pass); 165150461Scperciva if (proxy_auth_userpass == NULL) 166150461Scperciva err(1, "asprintf"); 167150461Scperciva 168150461Scperciva proxy_auth_userpass64 = b64enc(proxy_auth_userpass); 169150461Scperciva if (proxy_auth_userpass64 == NULL) 170150461Scperciva err(1, "malloc"); 171150461Scperciva 172150461Scperciva asprintf(&proxyauth, "Proxy-Authorization: Basic %s\r\n", 173150461Scperciva proxy_auth_userpass64); 174150461Scperciva if (proxyauth == NULL) 175150461Scperciva err(1, "asprintf"); 176150461Scperciva 177150461Scperciva free(proxy_auth_userpass); 178150461Scperciva free(proxy_auth_userpass64); 179150461Scperciva } else 180150461Scperciva proxyauth = NULL; 181150461Scperciva 182148871Scperciva env_HTTP_USER_AGENT = getenv("HTTP_USER_AGENT"); 183148871Scperciva if (env_HTTP_USER_AGENT == NULL) 184148871Scperciva env_HTTP_USER_AGENT = "phttpget/0.1"; 185164057Scperciva 186164057Scperciva env_HTTP_TIMEOUT = getenv("HTTP_TIMEOUT"); 187164057Scperciva if (env_HTTP_TIMEOUT != NULL) { 188164057Scperciva http_timeout = strtol(env_HTTP_TIMEOUT, &p, 10); 189164057Scperciva if ((*env_HTTP_TIMEOUT == '\0') || (*p != '\0') || 190164057Scperciva (http_timeout < 0)) 191164057Scperciva warnx("HTTP_TIMEOUT (%s) is not a positive integer", 192164057Scperciva env_HTTP_TIMEOUT); 193164057Scperciva else 194164057Scperciva timo.tv_sec = http_timeout; 195164057Scperciva } 196148871Scperciva} 197148871Scperciva 198148871Scpercivastatic int 199148871Scpercivamakerequest(char ** buf, char * path, char * server, int connclose) 200148871Scperciva{ 201148871Scperciva int buflen; 202148871Scperciva 203148871Scperciva buflen = asprintf(buf, 204148871Scperciva "GET %s%s/%s HTTP/1.1\r\n" 205148871Scperciva "Host: %s\r\n" 206148871Scperciva "User-Agent: %s\r\n" 207148871Scperciva "%s" 208150461Scperciva "%s" 209148871Scperciva "\r\n", 210148871Scperciva env_HTTP_PROXY ? "http://" : "", 211148871Scperciva env_HTTP_PROXY ? server : "", 212148871Scperciva path, server, env_HTTP_USER_AGENT, 213150461Scperciva proxyauth ? proxyauth : "", 214171120Scperciva connclose ? "Connection: Close\r\n" : "Connection: Keep-Alive\r\n"); 215148871Scperciva if (buflen == -1) 216148871Scperciva err(1, "asprintf"); 217148871Scperciva return(buflen); 218148871Scperciva} 219148871Scperciva 220148871Scpercivastatic int 221148871Scpercivareadln(int sd, char * resbuf, int * resbuflen, int * resbufpos) 222148871Scperciva{ 223148871Scperciva ssize_t len; 224148871Scperciva 225148871Scperciva while (strnstr(resbuf + *resbufpos, "\r\n", 226148871Scperciva *resbuflen - *resbufpos) == NULL) { 227148871Scperciva /* Move buffered data to the start of the buffer */ 228148871Scperciva if (*resbufpos != 0) { 229148871Scperciva memmove(resbuf, resbuf + *resbufpos, 230148871Scperciva *resbuflen - *resbufpos); 231148871Scperciva *resbuflen -= *resbufpos; 232148871Scperciva *resbufpos = 0; 233148871Scperciva } 234148871Scperciva 235148871Scperciva /* If the buffer is full, complain */ 236148871Scperciva if (*resbuflen == BUFSIZ) 237148871Scperciva return -1; 238148871Scperciva 239148871Scperciva /* Read more data into the buffer */ 240148871Scperciva len = recv(sd, resbuf + *resbuflen, BUFSIZ - *resbuflen, 0); 241152546Scperciva if ((len == 0) || 242152546Scperciva ((len == -1) && (errno != EINTR))) 243148871Scperciva return -1; 244148871Scperciva 245148871Scperciva if (len != -1) 246148871Scperciva *resbuflen += len; 247148871Scperciva } 248148871Scperciva 249148871Scperciva return 0; 250148871Scperciva} 251148871Scperciva 252148871Scpercivastatic int 253148871Scpercivacopybytes(int sd, int fd, off_t copylen, char * resbuf, int * resbuflen, 254148871Scperciva int * resbufpos) 255148871Scperciva{ 256148871Scperciva ssize_t len; 257148871Scperciva 258148871Scperciva while (copylen) { 259148871Scperciva /* Write data from resbuf to fd */ 260148871Scperciva len = *resbuflen - *resbufpos; 261148871Scperciva if (copylen < len) 262148871Scperciva len = copylen; 263148871Scperciva if (len > 0) { 264148871Scperciva if (fd != -1) 265148871Scperciva len = write(fd, resbuf + *resbufpos, len); 266148871Scperciva if (len == -1) 267148871Scperciva err(1, "write"); 268148871Scperciva *resbufpos += len; 269148871Scperciva copylen -= len; 270148871Scperciva continue; 271148871Scperciva } 272148871Scperciva 273148871Scperciva /* Read more data into buffer */ 274148871Scperciva len = recv(sd, resbuf, BUFSIZ, 0); 275148871Scperciva if (len == -1) { 276148871Scperciva if (errno == EINTR) 277148871Scperciva continue; 278148871Scperciva return -1; 279148871Scperciva } else if (len == 0) { 280148871Scperciva return -2; 281148871Scperciva } else { 282148871Scperciva *resbuflen = len; 283148871Scperciva *resbufpos = 0; 284148871Scperciva } 285148871Scperciva } 286148871Scperciva 287148871Scperciva return 0; 288148871Scperciva} 289148871Scperciva 290148871Scpercivaint 291148871Scpercivamain(int argc, char *argv[]) 292148871Scperciva{ 293148871Scperciva struct addrinfo hints; /* Hints to getaddrinfo */ 294148871Scperciva struct addrinfo *res; /* Pointer to server address being used */ 295148871Scperciva struct addrinfo *res0; /* Pointer to server addresses */ 296148871Scperciva char * resbuf = NULL; /* Response buffer */ 297148871Scperciva int resbufpos = 0; /* Response buffer position */ 298148871Scperciva int resbuflen = 0; /* Response buffer length */ 299148871Scperciva char * eolp; /* Pointer to "\r\n" within resbuf */ 300148871Scperciva char * hln; /* Pointer within header line */ 301148871Scperciva char * servername; /* Name of server */ 302148871Scperciva char * fname = NULL; /* Name of downloaded file */ 303148871Scperciva char * reqbuf = NULL; /* Request buffer */ 304148871Scperciva int reqbufpos = 0; /* Request buffer position */ 305148871Scperciva int reqbuflen = 0; /* Request buffer length */ 306148871Scperciva ssize_t len; /* Length sent or received */ 307148871Scperciva int nreq = 0; /* Number of next request to send */ 308148871Scperciva int nres = 0; /* Number of next reply to receive */ 309148871Scperciva int pipelined = 0; /* != 0 if connection in pipelined mode. */ 310171120Scperciva int keepalive; /* != 0 if HTTP/1.0 keep-alive rcvd. */ 311148871Scperciva int sd = -1; /* Socket descriptor */ 312148871Scperciva int sdflags = 0; /* Flags on the socket sd */ 313148871Scperciva int fd = -1; /* Descriptor for download target file */ 314148871Scperciva int error; /* Error code */ 315148871Scperciva int statuscode; /* HTTP Status code */ 316148871Scperciva off_t contentlength; /* Value from Content-Length header */ 317148871Scperciva int chunked; /* != if transfer-encoding is chunked */ 318148871Scperciva off_t clen; /* Chunk length */ 319148871Scperciva int firstreq = 0; /* # of first request for this connection */ 320190679Scperciva int val; /* Value used for setsockopt call */ 321148871Scperciva 322148871Scperciva /* Check that the arguments are sensible */ 323148871Scperciva if (argc < 2) 324148871Scperciva usage(); 325148871Scperciva 326148871Scperciva /* Read important environment variables */ 327148871Scperciva readenv(); 328148871Scperciva 329148871Scperciva /* Get server name and adjust arg[cv] to point at file names */ 330148871Scperciva servername = argv[1]; 331148871Scperciva argv += 2; 332148871Scperciva argc -= 2; 333148871Scperciva 334148871Scperciva /* Allocate response buffer */ 335148871Scperciva resbuf = malloc(BUFSIZ); 336148871Scperciva if (resbuf == NULL) 337148871Scperciva err(1, "malloc"); 338148871Scperciva 339148871Scperciva /* Look up server */ 340148871Scperciva memset(&hints, 0, sizeof(hints)); 341148871Scperciva hints.ai_family = PF_UNSPEC; 342148871Scperciva hints.ai_socktype = SOCK_STREAM; 343148871Scperciva error = getaddrinfo(env_HTTP_PROXY ? env_HTTP_PROXY : servername, 344148871Scperciva env_HTTP_PROXY ? proxyport : "http", &hints, &res0); 345148871Scperciva if (error) 346154909Scperciva errx(1, "host = %s, port = %s: %s", 347148871Scperciva env_HTTP_PROXY ? env_HTTP_PROXY : servername, 348148880Scperciva env_HTTP_PROXY ? proxyport : "http", 349148871Scperciva gai_strerror(error)); 350148871Scperciva if (res0 == NULL) 351148871Scperciva errx(1, "could not look up %s", servername); 352148871Scperciva res = res0; 353148871Scperciva 354148871Scperciva /* Do the fetching */ 355148871Scperciva while (nres < argc) { 356148871Scperciva /* Make sure we have a connected socket */ 357148871Scperciva for (; sd == -1; res = res->ai_next) { 358148871Scperciva /* No addresses left to try :-( */ 359148871Scperciva if (res == NULL) 360148871Scperciva errx(1, "Could not connect to %s", servername); 361148871Scperciva 362148871Scperciva /* Create a socket... */ 363148871Scperciva sd = socket(res->ai_family, res->ai_socktype, 364148871Scperciva res->ai_protocol); 365148871Scperciva if (sd == -1) 366148871Scperciva continue; 367148871Scperciva 368148871Scperciva /* ... set 15-second timeouts ... */ 369148871Scperciva setsockopt(sd, SOL_SOCKET, SO_SNDTIMEO, 370148871Scperciva (void *)&timo, (socklen_t)sizeof(timo)); 371148871Scperciva setsockopt(sd, SOL_SOCKET, SO_RCVTIMEO, 372148871Scperciva (void *)&timo, (socklen_t)sizeof(timo)); 373148871Scperciva 374190679Scperciva /* ... disable SIGPIPE generation ... */ 375190679Scperciva val = 1; 376190679Scperciva setsockopt(sd, SOL_SOCKET, SO_NOSIGPIPE, 377190679Scperciva (void *)&val, sizeof(int)); 378190679Scperciva 379148871Scperciva /* ... and connect to the server. */ 380148871Scperciva if(connect(sd, res->ai_addr, res->ai_addrlen)) { 381148871Scperciva close(sd); 382148871Scperciva sd = -1; 383148871Scperciva continue; 384148871Scperciva } 385148871Scperciva 386148871Scperciva firstreq = nres; 387148871Scperciva } 388148871Scperciva 389148871Scperciva /* 390148871Scperciva * If in pipelined HTTP mode, put socket into non-blocking 391148871Scperciva * mode, since we're probably going to want to try to send 392148871Scperciva * several HTTP requests. 393148871Scperciva */ 394148871Scperciva if (pipelined) { 395148871Scperciva sdflags = fcntl(sd, F_GETFL); 396148871Scperciva if (fcntl(sd, F_SETFL, sdflags | O_NONBLOCK) == -1) 397148871Scperciva err(1, "fcntl"); 398148871Scperciva } 399148871Scperciva 400148871Scperciva /* Construct requests and/or send them without blocking */ 401148871Scperciva while ((nreq < argc) && ((reqbuf == NULL) || pipelined)) { 402148871Scperciva /* If not in the middle of a request, make one */ 403148871Scperciva if (reqbuf == NULL) { 404148871Scperciva reqbuflen = makerequest(&reqbuf, argv[nreq], 405148871Scperciva servername, (nreq == argc - 1)); 406148871Scperciva reqbufpos = 0; 407148871Scperciva } 408148871Scperciva 409148871Scperciva /* If in pipelined mode, try to send the request */ 410148871Scperciva if (pipelined) { 411148871Scperciva while (reqbufpos < reqbuflen) { 412148871Scperciva len = send(sd, reqbuf + reqbufpos, 413148871Scperciva reqbuflen - reqbufpos, 0); 414148871Scperciva if (len == -1) 415148871Scperciva break; 416148871Scperciva reqbufpos += len; 417148871Scperciva } 418148871Scperciva if (reqbufpos < reqbuflen) { 419148871Scperciva if (errno != EAGAIN) 420148871Scperciva goto conndied; 421148871Scperciva break; 422148871Scperciva } else { 423148871Scperciva free(reqbuf); 424148871Scperciva reqbuf = NULL; 425148871Scperciva nreq++; 426148871Scperciva } 427148871Scperciva } 428148871Scperciva } 429148871Scperciva 430148871Scperciva /* Put connection back into blocking mode */ 431148871Scperciva if (pipelined) { 432148871Scperciva if (fcntl(sd, F_SETFL, sdflags) == -1) 433148871Scperciva err(1, "fcntl"); 434148871Scperciva } 435148871Scperciva 436148871Scperciva /* Do we need to blocking-send a request? */ 437148871Scperciva if (nres == nreq) { 438148871Scperciva while (reqbufpos < reqbuflen) { 439148871Scperciva len = send(sd, reqbuf + reqbufpos, 440148871Scperciva reqbuflen - reqbufpos, 0); 441148871Scperciva if (len == -1) 442148871Scperciva goto conndied; 443148871Scperciva reqbufpos += len; 444148871Scperciva } 445148871Scperciva free(reqbuf); 446148871Scperciva reqbuf = NULL; 447148871Scperciva nreq++; 448148871Scperciva } 449148871Scperciva 450148871Scperciva /* Scan through the response processing headers. */ 451148871Scperciva statuscode = 0; 452148871Scperciva contentlength = -1; 453148871Scperciva chunked = 0; 454171120Scperciva keepalive = 0; 455148871Scperciva do { 456148871Scperciva /* Get a header line */ 457148871Scperciva error = readln(sd, resbuf, &resbuflen, &resbufpos); 458148871Scperciva if (error) 459148871Scperciva goto conndied; 460148925Scperciva hln = resbuf + resbufpos; 461148871Scperciva eolp = strnstr(hln, "\r\n", resbuflen - resbufpos); 462148871Scperciva resbufpos = (eolp - resbuf) + 2; 463148871Scperciva *eolp = '\0'; 464148871Scperciva 465148871Scperciva /* Make sure it doesn't contain a NUL character */ 466148871Scperciva if (strchr(hln, '\0') != eolp) 467148871Scperciva goto conndied; 468148871Scperciva 469148871Scperciva if (statuscode == 0) { 470148871Scperciva /* The first line MUST be HTTP/1.x xxx ... */ 471148871Scperciva if ((strncmp(hln, "HTTP/1.", 7) != 0) || 472148871Scperciva ! isdigit(hln[7])) 473148871Scperciva goto conndied; 474148871Scperciva 475148871Scperciva /* 476148871Scperciva * If the minor version number isn't zero, 477148871Scperciva * then we can assume that pipelining our 478148871Scperciva * requests is OK -- as long as we don't 479148871Scperciva * see a "Connection: close" line later 480148871Scperciva * and we either have a Content-Length or 481148871Scperciva * Transfer-Encoding: chunked header to 482148871Scperciva * tell us the length. 483148871Scperciva */ 484148871Scperciva if (hln[7] != '0') 485148871Scperciva pipelined = 1; 486148871Scperciva 487148871Scperciva /* Skip over the minor version number */ 488148871Scperciva hln = strchr(hln + 7, ' '); 489148871Scperciva if (hln == NULL) 490148871Scperciva goto conndied; 491148871Scperciva else 492148871Scperciva hln++; 493148871Scperciva 494148871Scperciva /* Read the status code */ 495148871Scperciva while (isdigit(*hln)) { 496148871Scperciva statuscode = statuscode * 10 + 497148871Scperciva *hln - '0'; 498148871Scperciva hln++; 499148871Scperciva } 500148871Scperciva 501148871Scperciva if (statuscode < 100 || statuscode > 599) 502148871Scperciva goto conndied; 503148871Scperciva 504148871Scperciva /* Ignore the rest of the line */ 505148871Scperciva continue; 506148871Scperciva } 507148871Scperciva 508171120Scperciva /* 509171120Scperciva * Check for "Connection: close" or 510171120Scperciva * "Connection: Keep-Alive" header 511171120Scperciva */ 512176250Scperciva if (strncasecmp(hln, "Connection:", 11) == 0) { 513148871Scperciva hln += 11; 514176250Scperciva if (strcasestr(hln, "close") != NULL) 515148871Scperciva pipelined = 0; 516176250Scperciva if (strcasestr(hln, "Keep-Alive") != NULL) 517171120Scperciva keepalive = 1; 518148871Scperciva 519148871Scperciva /* Next header... */ 520148871Scperciva continue; 521148871Scperciva } 522148871Scperciva 523148871Scperciva /* Check for "Content-Length:" header */ 524176250Scperciva if (strncasecmp(hln, "Content-Length:", 15) == 0) { 525148871Scperciva hln += 15; 526148871Scperciva contentlength = 0; 527148871Scperciva 528148871Scperciva /* Find the start of the length */ 529148871Scperciva while (!isdigit(*hln) && (*hln != '\0')) 530148871Scperciva hln++; 531148871Scperciva 532148871Scperciva /* Compute the length */ 533148871Scperciva while (isdigit(*hln)) { 534148881Scperciva if (contentlength >= OFF_MAX / 10) { 535148871Scperciva /* Nasty people... */ 536148871Scperciva goto conndied; 537148871Scperciva } 538148871Scperciva contentlength = contentlength * 10 + 539148871Scperciva *hln - '0'; 540148871Scperciva hln++; 541148871Scperciva } 542148871Scperciva 543148871Scperciva /* Next header... */ 544148871Scperciva continue; 545148871Scperciva } 546148871Scperciva 547148871Scperciva /* Check for "Transfer-Encoding: chunked" header */ 548176250Scperciva if (strncasecmp(hln, "Transfer-Encoding:", 18) == 0) { 549148871Scperciva hln += 18; 550176250Scperciva if (strcasestr(hln, "chunked") != NULL) 551148871Scperciva chunked = 1; 552148871Scperciva 553148871Scperciva /* Next header... */ 554148871Scperciva continue; 555148871Scperciva } 556148871Scperciva 557148871Scperciva /* We blithely ignore any other header lines */ 558148871Scperciva 559148871Scperciva /* No more header lines */ 560148871Scperciva if (strlen(hln) == 0) { 561148871Scperciva /* 562148871Scperciva * If the status code was 1xx, then there will 563148871Scperciva * be a real header later. Servers may emit 564148871Scperciva * 1xx header blocks at will, but since we 565148871Scperciva * don't expect one, we should just ignore it. 566148871Scperciva */ 567148871Scperciva if (100 <= statuscode && statuscode <= 199) { 568148871Scperciva statuscode = 0; 569148871Scperciva continue; 570148871Scperciva } 571148871Scperciva 572148871Scperciva /* End of header; message body follows */ 573148871Scperciva break; 574148871Scperciva } 575148871Scperciva } while (1); 576148871Scperciva 577148871Scperciva /* No message body for 204 or 304 */ 578148871Scperciva if (statuscode == 204 || statuscode == 304) { 579148871Scperciva nres++; 580148871Scperciva continue; 581148871Scperciva } 582148871Scperciva 583148871Scperciva /* 584148871Scperciva * There should be a message body coming, but we only want 585148871Scperciva * to send it to a file if the status code is 200 586148871Scperciva */ 587148871Scperciva if (statuscode == 200) { 588148871Scperciva /* Generate a file name for the download */ 589148871Scperciva fname = strrchr(argv[nres], '/'); 590148871Scperciva if (fname == NULL) 591148871Scperciva fname = argv[nres]; 592148871Scperciva else 593148871Scperciva fname++; 594148871Scperciva if (strlen(fname) == 0) 595148871Scperciva errx(1, "Cannot obtain file name from %s\n", 596148871Scperciva argv[nres]); 597148871Scperciva 598148871Scperciva fd = open(fname, O_CREAT | O_TRUNC | O_WRONLY, 0644); 599148871Scperciva if (fd == -1) 600148871Scperciva errx(1, "open(%s)", fname); 601148871Scperciva }; 602148871Scperciva 603148871Scperciva /* Read the message and send data to fd if appropriate */ 604148871Scperciva if (chunked) { 605148871Scperciva /* Handle a chunked-encoded entity */ 606148871Scperciva 607148871Scperciva /* Read chunks */ 608148871Scperciva do { 609148871Scperciva error = readln(sd, resbuf, &resbuflen, 610148871Scperciva &resbufpos); 611148871Scperciva if (error) 612148871Scperciva goto conndied; 613148871Scperciva hln = resbuf + resbufpos; 614148871Scperciva eolp = strstr(hln, "\r\n"); 615148871Scperciva resbufpos = (eolp - resbuf) + 2; 616148871Scperciva 617148871Scperciva clen = 0; 618148871Scperciva while (isxdigit(*hln)) { 619148881Scperciva if (clen >= OFF_MAX / 16) { 620148871Scperciva /* Nasty people... */ 621148871Scperciva goto conndied; 622148871Scperciva } 623148871Scperciva if (isdigit(*hln)) 624148871Scperciva clen = clen * 16 + *hln - '0'; 625148871Scperciva else 626148871Scperciva clen = clen * 16 + 10 + 627148871Scperciva tolower(*hln) - 'a'; 628148871Scperciva hln++; 629148871Scperciva } 630148871Scperciva 631148871Scperciva error = copybytes(sd, fd, clen, resbuf, 632148871Scperciva &resbuflen, &resbufpos); 633148871Scperciva if (error) { 634148871Scperciva goto conndied; 635148871Scperciva } 636148871Scperciva } while (clen != 0); 637148871Scperciva 638148871Scperciva /* Read trailer and final CRLF */ 639148871Scperciva do { 640148871Scperciva error = readln(sd, resbuf, &resbuflen, 641148871Scperciva &resbufpos); 642148871Scperciva if (error) 643148871Scperciva goto conndied; 644148871Scperciva hln = resbuf + resbufpos; 645148871Scperciva eolp = strstr(hln, "\r\n"); 646148871Scperciva resbufpos = (eolp - resbuf) + 2; 647148871Scperciva } while (hln != eolp); 648148871Scperciva } else if (contentlength != -1) { 649148871Scperciva error = copybytes(sd, fd, contentlength, resbuf, 650148871Scperciva &resbuflen, &resbufpos); 651148871Scperciva if (error) 652148871Scperciva goto conndied; 653148871Scperciva } else { 654148871Scperciva /* 655148871Scperciva * Not chunked, and no content length header. 656148871Scperciva * Read everything until the server closes the 657148871Scperciva * socket. 658148871Scperciva */ 659148881Scperciva error = copybytes(sd, fd, OFF_MAX, resbuf, 660148871Scperciva &resbuflen, &resbufpos); 661148871Scperciva if (error == -1) 662148871Scperciva goto conndied; 663148871Scperciva pipelined = 0; 664148871Scperciva } 665148871Scperciva 666148871Scperciva if (fd != -1) { 667148871Scperciva close(fd); 668148871Scperciva fd = -1; 669148871Scperciva } 670148871Scperciva 671148871Scperciva fprintf(stderr, "http://%s/%s: %d ", servername, argv[nres], 672148871Scperciva statuscode); 673148871Scperciva if (statuscode == 200) 674148871Scperciva fprintf(stderr, "OK\n"); 675148871Scperciva else if (statuscode < 300) 676148871Scperciva fprintf(stderr, "Successful (ignored)\n"); 677148871Scperciva else if (statuscode < 400) 678148871Scperciva fprintf(stderr, "Redirection (ignored)\n"); 679148871Scperciva else 680148871Scperciva fprintf(stderr, "Error (ignored)\n"); 681148871Scperciva 682148871Scperciva /* We've finished this file! */ 683148871Scperciva nres++; 684148871Scperciva 685148871Scperciva /* 686148871Scperciva * If necessary, clean up this connection so that we 687148871Scperciva * can start a new one. 688148871Scperciva */ 689171120Scperciva if (pipelined == 0 && keepalive == 0) 690148871Scperciva goto cleanupconn; 691148871Scperciva continue; 692148871Scperciva 693148871Scpercivaconndied: 694148871Scperciva /* 695148871Scperciva * Something went wrong -- our connection died, the server 696148871Scperciva * sent us garbage, etc. If this happened on the first 697148871Scperciva * request we sent over this connection, give up. Otherwise, 698148871Scperciva * close this connection, open a new one, and reissue the 699148871Scperciva * request. 700148871Scperciva */ 701148871Scperciva if (nres == firstreq) 702148871Scperciva errx(1, "Connection failure"); 703148871Scperciva 704148871Scpercivacleanupconn: 705148871Scperciva /* 706148871Scperciva * Clean up our connection and keep on going 707148871Scperciva */ 708148871Scperciva shutdown(sd, SHUT_RDWR); 709148871Scperciva close(sd); 710148871Scperciva sd = -1; 711148871Scperciva if (fd != -1) { 712148871Scperciva close(fd); 713148871Scperciva fd = -1; 714148871Scperciva } 715148871Scperciva if (reqbuf != NULL) { 716148871Scperciva free(reqbuf); 717148871Scperciva reqbuf = NULL; 718148871Scperciva } 719148871Scperciva nreq = nres; 720148871Scperciva res = res0; 721148871Scperciva pipelined = 0; 722148871Scperciva resbufpos = resbuflen = 0; 723148871Scperciva continue; 724148871Scperciva } 725148871Scperciva 726148871Scperciva free(resbuf); 727148871Scperciva freeaddrinfo(res0); 728148871Scperciva 729148871Scperciva return 0; 730148871Scperciva} 731