1/*************************************************************************** 2 * _ _ ____ _ 3 * Project ___| | | | _ \| | 4 * / __| | | | |_) | | 5 * | (__| |_| | _ <| |___ 6 * \___|\___/|_| \_\_____| 7 * 8 * Copyright (C) 1998 - 2014, Daniel Stenberg, <daniel@haxx.se>, et al. 9 * 10 * This software is licensed as described in the file COPYING, which 11 * you should have received as part of this distribution. The terms 12 * are also available at http://curl.haxx.se/docs/copyright.html. 13 * 14 * You may opt to use, copy, modify, merge, publish, distribute and/or sell 15 * copies of the Software, and permit persons to whom the Software is 16 * furnished to do so, under the terms of the COPYING file. 17 * 18 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY 19 * KIND, either express or implied. 20 * 21 ***************************************************************************/ 22#include "tool_setup.h" 23 24#ifdef HAVE_FCNTL_H 25# include <fcntl.h> 26#endif 27 28#ifdef HAVE_UTIME_H 29# include <utime.h> 30#elif defined(HAVE_SYS_UTIME_H) 31# include <sys/utime.h> 32#endif 33 34#ifdef HAVE_LOCALE_H 35# include <locale.h> 36#endif 37 38#ifdef HAVE_NETINET_TCP_H 39# include <netinet/tcp.h> 40#endif 41 42#ifdef __VMS 43# include <fabdef.h> 44#endif 45 46#include "rawstr.h" 47 48#define ENABLE_CURLX_PRINTF 49/* use our own printf() functions */ 50#include "curlx.h" 51 52#include "tool_binmode.h" 53#include "tool_cfgable.h" 54#include "tool_cb_dbg.h" 55#include "tool_cb_hdr.h" 56#include "tool_cb_prg.h" 57#include "tool_cb_rea.h" 58#include "tool_cb_see.h" 59#include "tool_cb_wrt.h" 60#include "tool_dirhie.h" 61#include "tool_doswin.h" 62#include "tool_easysrc.h" 63#include "tool_getparam.h" 64#include "tool_helpers.h" 65#include "tool_homedir.h" 66#include "tool_libinfo.h" 67#include "tool_main.h" 68#include "tool_metalink.h" 69#include "tool_msgs.h" 70#include "tool_operate.h" 71#include "tool_operhlp.h" 72#include "tool_paramhlp.h" 73#include "tool_parsecfg.h" 74#include "tool_setopt.h" 75#include "tool_sleep.h" 76#include "tool_urlglob.h" 77#include "tool_util.h" 78#include "tool_writeenv.h" 79#include "tool_writeout.h" 80#include "tool_xattr.h" 81#include "tool_vms.h" 82#include "tool_help.h" 83#include "tool_hugehelp.h" 84 85#include "memdebug.h" /* keep this as LAST include */ 86 87#ifdef CURLDEBUG 88/* libcurl's debug builds provide an extra function */ 89CURLcode curl_easy_perform_ev(CURL *easy); 90#endif 91 92#define CURLseparator "--_curl_--" 93 94#ifndef O_BINARY 95/* since O_BINARY as used in bitmasks, setting it to zero makes it usable in 96 source code but yet it doesn't ruin anything */ 97# define O_BINARY 0 98#endif 99 100#define CURL_CA_CERT_ERRORMSG1 \ 101 "More details here: http://curl.haxx.se/docs/sslcerts.html\n\n" \ 102 "curl performs SSL certificate verification by default, " \ 103 "using a \"bundle\"\n" \ 104 " of Certificate Authority (CA) public keys (CA certs). If the default\n" \ 105 " bundle file isn't adequate, you can specify an alternate file\n" \ 106 " using the --cacert option.\n" 107 108#define CURL_CA_CERT_ERRORMSG2 \ 109 "If this HTTPS server uses a certificate signed by a CA represented in\n" \ 110 " the bundle, the certificate verification probably failed due to a\n" \ 111 " problem with the certificate (it might be expired, or the name might\n" \ 112 " not match the domain name in the URL).\n" \ 113 "If you'd like to turn off curl's verification of the certificate, use\n" \ 114 " the -k (or --insecure) option.\n" 115 116static bool is_fatal_error(CURLcode code) 117{ 118 switch(code) { 119 /* TODO: Should CURLE_SSL_CACERT be included as critical error ? */ 120 case CURLE_FAILED_INIT: 121 case CURLE_OUT_OF_MEMORY: 122 case CURLE_UNKNOWN_OPTION: 123 case CURLE_FUNCTION_NOT_FOUND: 124 case CURLE_BAD_FUNCTION_ARGUMENT: 125 /* critical error */ 126 return TRUE; 127 default: 128 break; 129 } 130 131 /* no error or not critical */ 132 return FALSE; 133} 134 135#ifdef __VMS 136/* 137 * get_vms_file_size does what it takes to get the real size of the file 138 * 139 * For fixed files, find out the size of the EOF block and adjust. 140 * 141 * For all others, have to read the entire file in, discarding the contents. 142 * Most posted text files will be small, and binary files like zlib archives 143 * and CD/DVD images should be either a STREAM_LF format or a fixed format. 144 * 145 */ 146static curl_off_t vms_realfilesize(const char * name, 147 const struct_stat * stat_buf) 148{ 149 char buffer[8192]; 150 curl_off_t count; 151 int ret_stat; 152 FILE * file; 153 154 file = fopen(name, "r"); 155 if(file == NULL) { 156 return 0; 157 } 158 count = 0; 159 ret_stat = 1; 160 while(ret_stat > 0) { 161 ret_stat = fread(buffer, 1, sizeof(buffer), file); 162 if(ret_stat != 0) 163 count += ret_stat; 164 } 165 fclose(file); 166 167 return count; 168} 169 170/* 171 * 172 * VmsSpecialSize checks to see if the stat st_size can be trusted and 173 * if not to call a routine to get the correct size. 174 * 175 */ 176static curl_off_t VmsSpecialSize(const char * name, 177 const struct_stat * stat_buf) 178{ 179 switch(stat_buf->st_fab_rfm) { 180 case FAB$C_VAR: 181 case FAB$C_VFC: 182 return vms_realfilesize(name, stat_buf); 183 break; 184 default: 185 return stat_buf->st_size; 186 } 187} 188#endif /* __VMS */ 189 190static CURLcode operate_do(struct GlobalConfig *global, 191 struct OperationConfig *config) 192{ 193 char errorbuffer[CURL_ERROR_SIZE]; 194 struct ProgressData progressbar; 195 struct getout *urlnode; 196 197 struct HdrCbData hdrcbdata; 198 struct OutStruct heads; 199 200 metalinkfile *mlfile_last = NULL; 201 202 CURL *curl = config->easy; 203 char *httpgetfields = NULL; 204 205 int res = 0; 206 unsigned long li; 207 208 /* Save the values of noprogress and isatty to restore them later on */ 209 bool orig_noprogress = global->noprogress; 210 bool orig_isatty = global->isatty; 211 212 errorbuffer[0] = '\0'; 213 214 /* default headers output stream is stdout */ 215 memset(&hdrcbdata, 0, sizeof(struct HdrCbData)); 216 memset(&heads, 0, sizeof(struct OutStruct)); 217 heads.stream = stdout; 218 heads.config = config; 219 220 /* 221 ** Beyond this point no return'ing from this function allowed. 222 ** Jump to label 'quit_curl' in order to abandon this function 223 ** from outside of nested loops further down below. 224 */ 225 226 /* Check we have a url */ 227 if(!config->url_list || !config->url_list->url) { 228 helpf(global->errors, "no URL specified!\n"); 229 res = CURLE_FAILED_INIT; 230 goto quit_curl; 231 } 232 233 /* On WIN32 we can't set the path to curl-ca-bundle.crt 234 * at compile time. So we look here for the file in two ways: 235 * 1: look at the environment variable CURL_CA_BUNDLE for a path 236 * 2: if #1 isn't found, use the windows API function SearchPath() 237 * to find it along the app's path (includes app's dir and CWD) 238 * 239 * We support the environment variable thing for non-Windows platforms 240 * too. Just for the sake of it. 241 */ 242 if(!config->cacert && 243 !config->capath && 244 !config->insecure_ok) { 245 char *env; 246 env = curlx_getenv("CURL_CA_BUNDLE"); 247 if(env) { 248 config->cacert = strdup(env); 249 if(!config->cacert) { 250 curl_free(env); 251 helpf(global->errors, "out of memory\n"); 252 res = CURLE_OUT_OF_MEMORY; 253 goto quit_curl; 254 } 255 } 256 else { 257 env = curlx_getenv("SSL_CERT_DIR"); 258 if(env) { 259 config->capath = strdup(env); 260 if(!config->capath) { 261 curl_free(env); 262 helpf(global->errors, "out of memory\n"); 263 res = CURLE_OUT_OF_MEMORY; 264 goto quit_curl; 265 } 266 } 267 else { 268 env = curlx_getenv("SSL_CERT_FILE"); 269 if(env) { 270 config->cacert = strdup(env); 271 if(!config->cacert) { 272 curl_free(env); 273 helpf(global->errors, "out of memory\n"); 274 res = CURLE_OUT_OF_MEMORY; 275 goto quit_curl; 276 } 277 } 278 } 279 } 280 281 if(env) 282 curl_free(env); 283#ifdef WIN32 284 else { 285 res = FindWin32CACert(config, "curl-ca-bundle.crt"); 286 if(res) 287 goto quit_curl; 288 } 289#endif 290 } 291 292 if(config->postfields) { 293 if(config->use_httpget) { 294 /* Use the postfields data for a http get */ 295 httpgetfields = strdup(config->postfields); 296 Curl_safefree(config->postfields); 297 if(!httpgetfields) { 298 helpf(global->errors, "out of memory\n"); 299 res = CURLE_OUT_OF_MEMORY; 300 goto quit_curl; 301 } 302 if(SetHTTPrequest(config, 303 (config->no_body?HTTPREQ_HEAD:HTTPREQ_GET), 304 &config->httpreq)) { 305 res = CURLE_FAILED_INIT; 306 goto quit_curl; 307 } 308 } 309 else { 310 if(SetHTTPrequest(config, HTTPREQ_SIMPLEPOST, &config->httpreq)) { 311 res = CURLE_FAILED_INIT; 312 goto quit_curl; 313 } 314 } 315 } 316 317 /* Single header file for all URLs */ 318 if(config->headerfile) { 319 /* open file for output: */ 320 if(!curlx_strequal(config->headerfile, "-")) { 321 FILE *newfile = fopen(config->headerfile, "wb"); 322 if(!newfile) { 323 warnf(config, "Failed to open %s\n", config->headerfile); 324 res = CURLE_WRITE_ERROR; 325 goto quit_curl; 326 } 327 else { 328 heads.filename = config->headerfile; 329 heads.s_isreg = TRUE; 330 heads.fopened = TRUE; 331 heads.stream = newfile; 332 } 333 } 334 else { 335 /* always use binary mode for protocol header output */ 336 set_binmode(heads.stream); 337 } 338 } 339 340 /* 341 ** Nested loops start here. 342 */ 343 344 /* loop through the list of given URLs */ 345 346 for(urlnode = config->url_list; urlnode; urlnode = urlnode->next) { 347 348 unsigned long up; /* upload file counter within a single upload glob */ 349 char *infiles; /* might be a glob pattern */ 350 char *outfiles; 351 unsigned long infilenum; 352 URLGlob *inglob; 353 354 int metalink = 0; /* nonzero for metalink download. */ 355 metalinkfile *mlfile; 356 metalink_resource *mlres; 357 358 outfiles = NULL; 359 infilenum = 1; 360 inglob = NULL; 361 362 if(urlnode->flags & GETOUT_METALINK) { 363 metalink = 1; 364 if(mlfile_last == NULL) { 365 mlfile_last = config->metalinkfile_list; 366 } 367 mlfile = mlfile_last; 368 mlfile_last = mlfile_last->next; 369 mlres = mlfile->resource; 370 } 371 else { 372 mlfile = NULL; 373 mlres = NULL; 374 } 375 376 /* urlnode->url is the full URL (it might be NULL) */ 377 378 if(!urlnode->url) { 379 /* This node has no URL. Free node data without destroying the 380 node itself nor modifying next pointer and continue to next */ 381 Curl_safefree(urlnode->outfile); 382 Curl_safefree(urlnode->infile); 383 urlnode->flags = 0; 384 continue; /* next URL please */ 385 } 386 387 /* save outfile pattern before expansion */ 388 if(urlnode->outfile) { 389 outfiles = strdup(urlnode->outfile); 390 if(!outfiles) { 391 helpf(global->errors, "out of memory\n"); 392 res = CURLE_OUT_OF_MEMORY; 393 break; 394 } 395 } 396 397 infiles = urlnode->infile; 398 399 if(!config->globoff && infiles) { 400 /* Unless explicitly shut off */ 401 res = glob_url(&inglob, infiles, &infilenum, 402 global->showerror?global->errors:NULL); 403 if(res) { 404 Curl_safefree(outfiles); 405 break; 406 } 407 } 408 409 /* Here's the loop for uploading multiple files within the same 410 single globbed string. If no upload, we enter the loop once anyway. */ 411 for(up = 0 ; up < infilenum; up++) { 412 413 char *uploadfile; /* a single file, never a glob */ 414 int separator; 415 URLGlob *urls; 416 unsigned long urlnum; 417 418 uploadfile = NULL; 419 urls = NULL; 420 urlnum = 0; 421 422 if(!up && !infiles) 423 Curl_nop_stmt; 424 else { 425 if(inglob) { 426 res = glob_next_url(&uploadfile, inglob); 427 if(res == CURLE_OUT_OF_MEMORY) 428 helpf(global->errors, "out of memory\n"); 429 } 430 else if(!up) { 431 uploadfile = strdup(infiles); 432 if(!uploadfile) { 433 helpf(global->errors, "out of memory\n"); 434 res = CURLE_OUT_OF_MEMORY; 435 } 436 } 437 else 438 uploadfile = NULL; 439 if(!uploadfile) 440 break; 441 } 442 443 if(metalink) { 444 /* For Metalink download, we don't use glob. Instead we use 445 the number of resources as urlnum. */ 446 urlnum = count_next_metalink_resource(mlfile); 447 } 448 else 449 if(!config->globoff) { 450 /* Unless explicitly shut off, we expand '{...}' and '[...]' 451 expressions and return total number of URLs in pattern set */ 452 res = glob_url(&urls, urlnode->url, &urlnum, 453 global->showerror?global->errors:NULL); 454 if(res) { 455 Curl_safefree(uploadfile); 456 break; 457 } 458 } 459 else 460 urlnum = 1; /* without globbing, this is a single URL */ 461 462 /* if multiple files extracted to stdout, insert separators! */ 463 separator= ((!outfiles || curlx_strequal(outfiles, "-")) && urlnum > 1); 464 465 /* Here's looping around each globbed URL */ 466 for(li = 0 ; li < urlnum; li++) { 467 468 int infd; 469 bool infdopen; 470 char *outfile; 471 struct OutStruct outs; 472 struct InStruct input; 473 struct timeval retrystart; 474 curl_off_t uploadfilesize; 475 long retry_numretries; 476 long retry_sleep_default; 477 long retry_sleep; 478 char *this_url = NULL; 479 int metalink_next_res = 0; 480 481 outfile = NULL; 482 infdopen = FALSE; 483 infd = STDIN_FILENO; 484 uploadfilesize = -1; /* -1 means unknown */ 485 486 /* default output stream is stdout */ 487 memset(&outs, 0, sizeof(struct OutStruct)); 488 outs.stream = stdout; 489 outs.config = config; 490 491 if(metalink) { 492 /* For Metalink download, use name in Metalink file as 493 filename. */ 494 outfile = strdup(mlfile->filename); 495 if(!outfile) { 496 res = CURLE_OUT_OF_MEMORY; 497 goto show_error; 498 } 499 this_url = strdup(mlres->url); 500 if(!this_url) { 501 res = CURLE_OUT_OF_MEMORY; 502 goto show_error; 503 } 504 } 505 else { 506 if(urls) { 507 res = glob_next_url(&this_url, urls); 508 if(res) 509 goto show_error; 510 } 511 else if(!li) { 512 this_url = strdup(urlnode->url); 513 if(!this_url) { 514 res = CURLE_OUT_OF_MEMORY; 515 goto show_error; 516 } 517 } 518 else 519 this_url = NULL; 520 if(!this_url) 521 break; 522 523 if(outfiles) { 524 outfile = strdup(outfiles); 525 if(!outfile) { 526 res = CURLE_OUT_OF_MEMORY; 527 goto show_error; 528 } 529 } 530 } 531 532 if(((urlnode->flags&GETOUT_USEREMOTE) || 533 (outfile && !curlx_strequal("-", outfile))) && 534 (metalink || !config->use_metalink)) { 535 536 /* 537 * We have specified a file name to store the result in, or we have 538 * decided we want to use the remote file name. 539 */ 540 541 if(!outfile) { 542 /* extract the file name from the URL */ 543 res = get_url_file_name(&outfile, this_url); 544 if(res) 545 goto show_error; 546 if((!outfile || !*outfile) && !config->content_disposition) { 547 helpf(global->errors, "Remote file name has no length!\n"); 548 res = CURLE_WRITE_ERROR; 549 goto quit_urls; 550 } 551#if defined(MSDOS) || defined(WIN32) 552 /* For DOS and WIN32, we do some major replacing of 553 bad characters in the file name before using it */ 554 outfile = sanitize_dos_name(outfile); 555 if(!outfile) { 556 res = CURLE_OUT_OF_MEMORY; 557 goto show_error; 558 } 559#endif /* MSDOS || WIN32 */ 560 } 561 else if(urls) { 562 /* fill '#1' ... '#9' terms from URL pattern */ 563 char *storefile = outfile; 564 res = glob_match_url(&outfile, storefile, urls); 565 Curl_safefree(storefile); 566 if(res) { 567 /* bad globbing */ 568 warnf(config, "bad output glob!\n"); 569 goto quit_urls; 570 } 571 } 572 573 /* Create the directory hierarchy, if not pre-existent to a multiple 574 file output call */ 575 576 if(config->create_dirs || metalink) { 577 res = create_dir_hierarchy(outfile, global->errors); 578 /* create_dir_hierarchy shows error upon CURLE_WRITE_ERROR */ 579 if(res == CURLE_WRITE_ERROR) 580 goto quit_urls; 581 if(res) { 582 goto show_error; 583 } 584 } 585 586 if((urlnode->flags & GETOUT_USEREMOTE) 587 && config->content_disposition) { 588 /* Our header callback MIGHT set the filename */ 589 DEBUGASSERT(!outs.filename); 590 } 591 592 if(config->resume_from_current) { 593 /* We're told to continue from where we are now. Get the size 594 of the file as it is now and open it for append instead */ 595 struct_stat fileinfo; 596 /* VMS -- Danger, the filesize is only valid for stream files */ 597 if(0 == stat(outfile, &fileinfo)) 598 /* set offset to current file size: */ 599 config->resume_from = fileinfo.st_size; 600 else 601 /* let offset be 0 */ 602 config->resume_from = 0; 603 } 604 605 if(config->resume_from) { 606#ifdef __VMS 607 /* open file for output, forcing VMS output format into stream 608 mode which is needed for stat() call above to always work. */ 609 FILE *file = fopen(outfile, config->resume_from?"ab":"wb", 610 "ctx=stm", "rfm=stmlf", "rat=cr", "mrs=0"); 611#else 612 /* open file for output: */ 613 FILE *file = fopen(outfile, config->resume_from?"ab":"wb"); 614#endif 615 if(!file) { 616 helpf(global->errors, "Can't open '%s'!\n", outfile); 617 res = CURLE_WRITE_ERROR; 618 goto quit_urls; 619 } 620 outs.fopened = TRUE; 621 outs.stream = file; 622 outs.init = config->resume_from; 623 } 624 else { 625 outs.stream = NULL; /* open when needed */ 626 } 627 outs.filename = outfile; 628 outs.s_isreg = TRUE; 629 } 630 631 if(uploadfile && !stdin_upload(uploadfile)) { 632 /* 633 * We have specified a file to upload and it isn't "-". 634 */ 635 struct_stat fileinfo; 636 637 this_url = add_file_name_to_url(curl, this_url, uploadfile); 638 if(!this_url) { 639 res = CURLE_OUT_OF_MEMORY; 640 goto show_error; 641 } 642 /* VMS Note: 643 * 644 * Reading binary from files can be a problem... Only FIXED, VAR 645 * etc WITHOUT implied CC will work Others need a \n appended to a 646 * line 647 * 648 * - Stat gives a size but this is UNRELIABLE in VMS As a f.e. a 649 * fixed file with implied CC needs to have a byte added for every 650 * record processed, this can by derived from Filesize & recordsize 651 * for VARiable record files the records need to be counted! for 652 * every record add 1 for linefeed and subtract 2 for the record 653 * header for VARIABLE header files only the bare record data needs 654 * to be considered with one appended if implied CC 655 */ 656#ifdef __VMS 657 /* Calculate the real upload site for VMS */ 658 infd = -1; 659 if(stat(uploadfile, &fileinfo) == 0) { 660 fileinfo.st_size = VmsSpecialSize(uploadfile, &fileinfo); 661 switch (fileinfo.st_fab_rfm) { 662 case FAB$C_VAR: 663 case FAB$C_VFC: 664 case FAB$C_STMCR: 665 infd = open(uploadfile, O_RDONLY | O_BINARY); 666 break; 667 default: 668 infd = open(uploadfile, O_RDONLY | O_BINARY, 669 "rfm=stmlf", "ctx=stm"); 670 } 671 } 672 if(infd == -1) 673#else 674 infd = open(uploadfile, O_RDONLY | O_BINARY); 675 if((infd == -1) || fstat(infd, &fileinfo)) 676#endif 677 { 678 helpf(global->errors, "Can't open '%s'!\n", uploadfile); 679 if(infd != -1) { 680 close(infd); 681 infd = STDIN_FILENO; 682 } 683 res = CURLE_READ_ERROR; 684 goto quit_urls; 685 } 686 infdopen = TRUE; 687 688 /* we ignore file size for char/block devices, sockets, etc. */ 689 if(S_ISREG(fileinfo.st_mode)) 690 uploadfilesize = fileinfo.st_size; 691 692 } 693 else if(uploadfile && stdin_upload(uploadfile)) { 694 /* count to see if there are more than one auth bit set 695 in the authtype field */ 696 int authbits = 0; 697 int bitcheck = 0; 698 while(bitcheck < 32) { 699 if(config->authtype & (1UL << bitcheck++)) { 700 authbits++; 701 if(authbits > 1) { 702 /* more than one, we're done! */ 703 break; 704 } 705 } 706 } 707 708 /* 709 * If the user has also selected --anyauth or --proxy-anyauth 710 * we should warn him/her. 711 */ 712 if(config->proxyanyauth || (authbits>1)) { 713 warnf(config, 714 "Using --anyauth or --proxy-anyauth with upload from stdin" 715 " involves a big risk of it not working. Use a temporary" 716 " file or a fixed auth type instead!\n"); 717 } 718 719 DEBUGASSERT(infdopen == FALSE); 720 DEBUGASSERT(infd == STDIN_FILENO); 721 722 set_binmode(stdin); 723 if(curlx_strequal(uploadfile, ".")) { 724 if(curlx_nonblock((curl_socket_t)infd, TRUE) < 0) 725 warnf(config, 726 "fcntl failed on fd=%d: %s\n", infd, strerror(errno)); 727 } 728 } 729 730 if(uploadfile && config->resume_from_current) 731 config->resume_from = -1; /* -1 will then force get-it-yourself */ 732 733 if(output_expected(this_url, uploadfile) && outs.stream && 734 isatty(fileno(outs.stream))) 735 /* we send the output to a tty, therefore we switch off the progress 736 meter */ 737 global->noprogress = global->isatty = TRUE; 738 else { 739 /* progress meter is per download, so restore config 740 values */ 741 global->noprogress = orig_noprogress; 742 global->isatty = orig_isatty; 743 } 744 745 if(urlnum > 1 && !global->mute) { 746 fprintf(global->errors, "\n[%lu/%lu]: %s --> %s\n", 747 li+1, urlnum, this_url, outfile ? outfile : "<stdout>"); 748 if(separator) 749 printf("%s%s\n", CURLseparator, this_url); 750 } 751 if(httpgetfields) { 752 char *urlbuffer; 753 /* Find out whether the url contains a file name */ 754 const char *pc = strstr(this_url, "://"); 755 char sep = '?'; 756 if(pc) 757 pc += 3; 758 else 759 pc = this_url; 760 761 pc = strrchr(pc, '/'); /* check for a slash */ 762 763 if(pc) { 764 /* there is a slash present in the URL */ 765 766 if(strchr(pc, '?')) 767 /* Ouch, there's already a question mark in the URL string, we 768 then append the data with an ampersand separator instead! */ 769 sep='&'; 770 } 771 /* 772 * Then append ? followed by the get fields to the url. 773 */ 774 if(pc) 775 urlbuffer = aprintf("%s%c%s", this_url, sep, httpgetfields); 776 else 777 /* Append / before the ? to create a well-formed url 778 if the url contains a hostname only 779 */ 780 urlbuffer = aprintf("%s/?%s", this_url, httpgetfields); 781 782 if(!urlbuffer) { 783 res = CURLE_OUT_OF_MEMORY; 784 goto show_error; 785 } 786 787 Curl_safefree(this_url); /* free previous URL */ 788 this_url = urlbuffer; /* use our new URL instead! */ 789 } 790 791 if(!global->errors) 792 global->errors = stderr; 793 794 if((!outfile || !strcmp(outfile, "-")) && !config->use_ascii) { 795 /* We get the output to stdout and we have not got the ASCII/text 796 flag, then set stdout to be binary */ 797 set_binmode(stdout); 798 } 799 800 if(config->tcp_nodelay) 801 my_setopt(curl, CURLOPT_TCP_NODELAY, 1L); 802 803 /* where to store */ 804 my_setopt(curl, CURLOPT_WRITEDATA, &outs); 805 if(metalink || !config->use_metalink) 806 /* what call to write */ 807 my_setopt(curl, CURLOPT_WRITEFUNCTION, tool_write_cb); 808#ifdef USE_METALINK 809 else 810 /* Set Metalink specific write callback function to parse 811 XML data progressively. */ 812 my_setopt(curl, CURLOPT_WRITEFUNCTION, metalink_write_cb); 813#endif /* USE_METALINK */ 814 815 /* for uploads */ 816 input.fd = infd; 817 input.config = config; 818 /* Note that if CURLOPT_READFUNCTION is fread (the default), then 819 * lib/telnet.c will Curl_poll() on the input file descriptor 820 * rather then calling the READFUNCTION at regular intervals. 821 * The circumstances in which it is preferable to enable this 822 * behaviour, by omitting to set the READFUNCTION & READDATA options, 823 * have not been determined. 824 */ 825 my_setopt(curl, CURLOPT_READDATA, &input); 826 /* what call to read */ 827 my_setopt(curl, CURLOPT_READFUNCTION, tool_read_cb); 828 829 /* in 7.18.0, the CURLOPT_SEEKFUNCTION/DATA pair is taking over what 830 CURLOPT_IOCTLFUNCTION/DATA pair previously provided for seeking */ 831 my_setopt(curl, CURLOPT_SEEKDATA, &input); 832 my_setopt(curl, CURLOPT_SEEKFUNCTION, tool_seek_cb); 833 834 if(config->recvpersecond) 835 /* tell libcurl to use a smaller sized buffer as it allows us to 836 make better sleeps! 7.9.9 stuff! */ 837 my_setopt(curl, CURLOPT_BUFFERSIZE, (long)config->recvpersecond); 838 839 /* size of uploaded file: */ 840 if(uploadfilesize != -1) 841 my_setopt(curl, CURLOPT_INFILESIZE_LARGE, uploadfilesize); 842 my_setopt_str(curl, CURLOPT_URL, this_url); /* what to fetch */ 843 my_setopt(curl, CURLOPT_NOPROGRESS, global->noprogress?1L:0L); 844 if(config->no_body) { 845 my_setopt(curl, CURLOPT_NOBODY, 1L); 846 my_setopt(curl, CURLOPT_HEADER, 1L); 847 } 848 /* If --metalink is used, we ignore --include (headers in 849 output) option because mixing headers to the body will 850 confuse XML parser and/or hash check will fail. */ 851 else if(!config->use_metalink) 852 my_setopt(curl, CURLOPT_HEADER, config->include_headers?1L:0L); 853 854 if(config->xoauth2_bearer) 855 my_setopt_str(curl, CURLOPT_XOAUTH2_BEARER, config->xoauth2_bearer); 856 857#if !defined(CURL_DISABLE_PROXY) 858 { 859 /* TODO: Make this a run-time check instead of compile-time one. */ 860 861 my_setopt_str(curl, CURLOPT_PROXY, config->proxy); 862 my_setopt_str(curl, CURLOPT_PROXYUSERPWD, config->proxyuserpwd); 863 864 /* new in libcurl 7.3 */ 865 my_setopt(curl, CURLOPT_HTTPPROXYTUNNEL, config->proxytunnel?1L:0L); 866 867 /* new in libcurl 7.5 */ 868 if(config->proxy) 869 my_setopt_enum(curl, CURLOPT_PROXYTYPE, (long)config->proxyver); 870 871 /* new in libcurl 7.10 */ 872 if(config->socksproxy) { 873 my_setopt_str(curl, CURLOPT_PROXY, config->socksproxy); 874 my_setopt_enum(curl, CURLOPT_PROXYTYPE, (long)config->socksver); 875 } 876 877 /* new in libcurl 7.10.6 */ 878 if(config->proxyanyauth) 879 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 880 (long)CURLAUTH_ANY); 881 else if(config->proxynegotiate) 882 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 883 (long)CURLAUTH_GSSNEGOTIATE); 884 else if(config->proxyntlm) 885 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 886 (long)CURLAUTH_NTLM); 887 else if(config->proxydigest) 888 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 889 (long)CURLAUTH_DIGEST); 890 else if(config->proxybasic) 891 my_setopt_bitmask(curl, CURLOPT_PROXYAUTH, 892 (long)CURLAUTH_BASIC); 893 894 /* new in libcurl 7.19.4 */ 895 my_setopt(curl, CURLOPT_NOPROXY, config->noproxy); 896 } 897#endif 898 899 my_setopt(curl, CURLOPT_FAILONERROR, config->failonerror?1L:0L); 900 my_setopt(curl, CURLOPT_UPLOAD, uploadfile?1L:0L); 901 my_setopt(curl, CURLOPT_DIRLISTONLY, config->dirlistonly?1L:0L); 902 my_setopt(curl, CURLOPT_APPEND, config->ftp_append?1L:0L); 903 904 if(config->netrc_opt) 905 my_setopt_enum(curl, CURLOPT_NETRC, (long)CURL_NETRC_OPTIONAL); 906 else if(config->netrc || config->netrc_file) 907 my_setopt_enum(curl, CURLOPT_NETRC, (long)CURL_NETRC_REQUIRED); 908 else 909 my_setopt_enum(curl, CURLOPT_NETRC, (long)CURL_NETRC_IGNORED); 910 911 if(config->netrc_file) 912 my_setopt(curl, CURLOPT_NETRC_FILE, config->netrc_file); 913 914 my_setopt(curl, CURLOPT_TRANSFERTEXT, config->use_ascii?1L:0L); 915 if(config->login_options) 916 my_setopt_str(curl, CURLOPT_LOGIN_OPTIONS, config->login_options); 917 my_setopt_str(curl, CURLOPT_USERPWD, config->userpwd); 918 my_setopt_str(curl, CURLOPT_RANGE, config->range); 919 my_setopt(curl, CURLOPT_ERRORBUFFER, errorbuffer); 920 my_setopt(curl, CURLOPT_TIMEOUT_MS, (long)(config->timeout * 1000)); 921 922 if(built_in_protos & CURLPROTO_HTTP) { 923 924 long postRedir = 0; 925 926 my_setopt(curl, CURLOPT_FOLLOWLOCATION, 927 config->followlocation?1L:0L); 928 my_setopt(curl, CURLOPT_UNRESTRICTED_AUTH, 929 config->unrestricted_auth?1L:0L); 930 931 switch(config->httpreq) { 932 case HTTPREQ_SIMPLEPOST: 933 my_setopt_str(curl, CURLOPT_POSTFIELDS, 934 config->postfields); 935 my_setopt(curl, CURLOPT_POSTFIELDSIZE_LARGE, 936 config->postfieldsize); 937 break; 938 case HTTPREQ_POST: 939 my_setopt_httppost(curl, CURLOPT_HTTPPOST, config->httppost); 940 break; 941 default: 942 break; 943 } 944 945 my_setopt_str(curl, CURLOPT_REFERER, config->referer); 946 my_setopt(curl, CURLOPT_AUTOREFERER, config->autoreferer?1L:0L); 947 my_setopt_str(curl, CURLOPT_USERAGENT, config->useragent); 948 my_setopt_slist(curl, CURLOPT_HTTPHEADER, config->headers); 949 950 /* new in libcurl 7.5 */ 951 my_setopt(curl, CURLOPT_MAXREDIRS, config->maxredirs); 952 953 /* new in libcurl 7.9.1 */ 954 if(config->httpversion) 955 my_setopt_enum(curl, CURLOPT_HTTP_VERSION, config->httpversion); 956 957 /* new in libcurl 7.10.6 (default is Basic) */ 958 if(config->authtype) 959 my_setopt_bitmask(curl, CURLOPT_HTTPAUTH, (long)config->authtype); 960 961 /* curl 7.19.1 (the 301 version existed in 7.18.2), 962 303 was added in 7.26.0 */ 963 if(config->post301) 964 postRedir |= CURL_REDIR_POST_301; 965 if(config->post302) 966 postRedir |= CURL_REDIR_POST_302; 967 if(config->post303) 968 postRedir |= CURL_REDIR_POST_303; 969 my_setopt(curl, CURLOPT_POSTREDIR, postRedir); 970 971 /* new in libcurl 7.21.6 */ 972 if(config->encoding) 973 my_setopt_str(curl, CURLOPT_ACCEPT_ENCODING, ""); 974 975 /* new in libcurl 7.21.6 */ 976 if(config->tr_encoding) 977 my_setopt(curl, CURLOPT_TRANSFER_ENCODING, 1L); 978 979 } /* (built_in_protos & CURLPROTO_HTTP) */ 980 981 my_setopt_str(curl, CURLOPT_FTPPORT, config->ftpport); 982 my_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 983 config->low_speed_limit); 984 my_setopt(curl, CURLOPT_LOW_SPEED_TIME, config->low_speed_time); 985 my_setopt(curl, CURLOPT_MAX_SEND_SPEED_LARGE, 986 config->sendpersecond); 987 my_setopt(curl, CURLOPT_MAX_RECV_SPEED_LARGE, 988 config->recvpersecond); 989 990 if(config->use_resume) 991 my_setopt(curl, CURLOPT_RESUME_FROM_LARGE, config->resume_from); 992 else 993 my_setopt(curl, CURLOPT_RESUME_FROM_LARGE, CURL_OFF_T_C(0)); 994 995 my_setopt_str(curl, CURLOPT_SSLCERT, config->cert); 996 my_setopt_str(curl, CURLOPT_SSLCERTTYPE, config->cert_type); 997 my_setopt_str(curl, CURLOPT_SSLKEY, config->key); 998 my_setopt_str(curl, CURLOPT_SSLKEYTYPE, config->key_type); 999 my_setopt_str(curl, CURLOPT_KEYPASSWD, config->key_passwd); 1000 1001 if(built_in_protos & (CURLPROTO_SCP|CURLPROTO_SFTP)) { 1002 1003 /* SSH and SSL private key uses same command-line option */ 1004 /* new in libcurl 7.16.1 */ 1005 my_setopt_str(curl, CURLOPT_SSH_PRIVATE_KEYFILE, config->key); 1006 /* new in libcurl 7.16.1 */ 1007 my_setopt_str(curl, CURLOPT_SSH_PUBLIC_KEYFILE, config->pubkey); 1008 1009 /* new in libcurl 7.17.1: SSH host key md5 checking allows us 1010 to fail if we are not talking to who we think we should */ 1011 my_setopt_str(curl, CURLOPT_SSH_HOST_PUBLIC_KEY_MD5, 1012 config->hostpubmd5); 1013 } 1014 1015 if(config->cacert) 1016 my_setopt_str(curl, CURLOPT_CAINFO, config->cacert); 1017 if(config->capath) 1018 my_setopt_str(curl, CURLOPT_CAPATH, config->capath); 1019 if(config->crlfile) 1020 my_setopt_str(curl, CURLOPT_CRLFILE, config->crlfile); 1021 1022 if(curlinfo->features & CURL_VERSION_SSL) { 1023 if(config->insecure_ok) { 1024 my_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); 1025 my_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); 1026 } 1027 else { 1028 my_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1L); 1029 /* libcurl default is strict verifyhost -> 2L */ 1030 /* my_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2L); */ 1031 } 1032 } 1033 1034 if(built_in_protos & (CURLPROTO_SCP|CURLPROTO_SFTP)) { 1035 if(!config->insecure_ok) { 1036 char *home; 1037 char *file; 1038 res = CURLE_OUT_OF_MEMORY; 1039 home = homedir(); 1040 if(home) { 1041 file = aprintf("%s/%sssh/known_hosts", home, DOT_CHAR); 1042 if(file) { 1043 /* new in curl 7.19.6 */ 1044 res = res_setopt_str(curl, CURLOPT_SSH_KNOWNHOSTS, file); 1045 curl_free(file); 1046 if(res == CURLE_UNKNOWN_OPTION) 1047 /* libssh2 version older than 1.1.1 */ 1048 res = CURLE_OK; 1049 } 1050 Curl_safefree(home); 1051 } 1052 if(res) 1053 goto show_error; 1054 } 1055 } 1056 1057 if(config->no_body || config->remote_time) { 1058 /* no body or use remote time */ 1059 my_setopt(curl, CURLOPT_FILETIME, 1L); 1060 } 1061 1062 my_setopt(curl, CURLOPT_CRLF, config->crlf?1L:0L); 1063 my_setopt_slist(curl, CURLOPT_QUOTE, config->quote); 1064 my_setopt_slist(curl, CURLOPT_POSTQUOTE, config->postquote); 1065 my_setopt_slist(curl, CURLOPT_PREQUOTE, config->prequote); 1066 1067#if !defined(CURL_DISABLE_HTTP) && !defined(CURL_DISABLE_COOKIES) 1068 { 1069 /* TODO: Make this a run-time check instead of compile-time one. */ 1070 1071 if(config->cookie) 1072 my_setopt_str(curl, CURLOPT_COOKIE, config->cookie); 1073 1074 if(config->cookiefile) 1075 my_setopt_str(curl, CURLOPT_COOKIEFILE, config->cookiefile); 1076 1077 /* new in libcurl 7.9 */ 1078 if(config->cookiejar) 1079 my_setopt_str(curl, CURLOPT_COOKIEJAR, config->cookiejar); 1080 1081 /* new in libcurl 7.9.7 */ 1082 my_setopt(curl, CURLOPT_COOKIESESSION, config->cookiesession?1L:0L); 1083 } 1084#endif 1085 1086 my_setopt_enum(curl, CURLOPT_SSLVERSION, config->ssl_version); 1087 my_setopt_enum(curl, CURLOPT_TIMECONDITION, (long)config->timecond); 1088 my_setopt(curl, CURLOPT_TIMEVALUE, (long)config->condtime); 1089 my_setopt_str(curl, CURLOPT_CUSTOMREQUEST, config->customrequest); 1090 my_setopt(curl, CURLOPT_STDERR, global->errors); 1091 1092 /* three new ones in libcurl 7.3: */ 1093 my_setopt_str(curl, CURLOPT_INTERFACE, config->iface); 1094 my_setopt_str(curl, CURLOPT_KRBLEVEL, config->krblevel); 1095 1096 progressbarinit(&progressbar, config); 1097 if((global->progressmode == CURL_PROGRESS_BAR) && 1098 !global->noprogress && !global->mute) { 1099 /* we want the alternative style, then we have to implement it 1100 ourselves! */ 1101 my_setopt(curl, CURLOPT_XFERINFOFUNCTION, tool_progress_cb); 1102 my_setopt(curl, CURLOPT_XFERINFODATA, &progressbar); 1103 } 1104 1105 /* new in libcurl 7.24.0: */ 1106 if(config->dns_servers) 1107 my_setopt_str(curl, CURLOPT_DNS_SERVERS, config->dns_servers); 1108 1109 /* new in libcurl 7.33.0: */ 1110 if(config->dns_interface) 1111 my_setopt_str(curl, CURLOPT_DNS_INTERFACE, config->dns_interface); 1112 if(config->dns_ipv4_addr) 1113 my_setopt_str(curl, CURLOPT_DNS_LOCAL_IP4, config->dns_ipv4_addr); 1114 if(config->dns_ipv6_addr) 1115 my_setopt_str(curl, CURLOPT_DNS_LOCAL_IP6, config->dns_ipv6_addr); 1116 1117 /* new in libcurl 7.6.2: */ 1118 my_setopt_slist(curl, CURLOPT_TELNETOPTIONS, config->telnet_options); 1119 1120 /* new in libcurl 7.7: */ 1121 my_setopt_str(curl, CURLOPT_RANDOM_FILE, config->random_file); 1122 my_setopt_str(curl, CURLOPT_EGDSOCKET, config->egd_file); 1123 my_setopt(curl, CURLOPT_CONNECTTIMEOUT_MS, 1124 (long)(config->connecttimeout * 1000)); 1125 1126 if(config->cipher_list) 1127 my_setopt_str(curl, CURLOPT_SSL_CIPHER_LIST, config->cipher_list); 1128 1129 /* new in libcurl 7.9.2: */ 1130 if(config->disable_epsv) 1131 /* disable it */ 1132 my_setopt(curl, CURLOPT_FTP_USE_EPSV, 0L); 1133 1134 /* new in libcurl 7.10.5 */ 1135 if(config->disable_eprt) 1136 /* disable it */ 1137 my_setopt(curl, CURLOPT_FTP_USE_EPRT, 0L); 1138 1139 if(global->tracetype != TRACE_NONE) { 1140 my_setopt(curl, CURLOPT_DEBUGFUNCTION, tool_debug_cb); 1141 my_setopt(curl, CURLOPT_DEBUGDATA, config); 1142 my_setopt(curl, CURLOPT_VERBOSE, 1L); 1143 } 1144 1145 /* new in curl 7.9.3 */ 1146 if(config->engine) { 1147 res = res_setopt_str(curl, CURLOPT_SSLENGINE, config->engine); 1148 if(res) 1149 goto show_error; 1150 my_setopt(curl, CURLOPT_SSLENGINE_DEFAULT, 1L); 1151 } 1152 1153 /* new in curl 7.10.7, extended in 7.19.4 but this only sets 0 or 1 */ 1154 my_setopt(curl, CURLOPT_FTP_CREATE_MISSING_DIRS, 1155 config->ftp_create_dirs?1L:0L); 1156 1157 /* new in curl 7.10.8 */ 1158 if(config->max_filesize) 1159 my_setopt(curl, CURLOPT_MAXFILESIZE_LARGE, 1160 config->max_filesize); 1161 1162 if(4 == config->ip_version) 1163 my_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4); 1164 else if(6 == config->ip_version) 1165 my_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V6); 1166 else 1167 my_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_WHATEVER); 1168 1169 /* new in curl 7.15.5 */ 1170 if(config->ftp_ssl_reqd) 1171 my_setopt_enum(curl, CURLOPT_USE_SSL, (long)CURLUSESSL_ALL); 1172 1173 /* new in curl 7.11.0 */ 1174 else if(config->ftp_ssl) 1175 my_setopt_enum(curl, CURLOPT_USE_SSL, (long)CURLUSESSL_TRY); 1176 1177 /* new in curl 7.16.0 */ 1178 else if(config->ftp_ssl_control) 1179 my_setopt_enum(curl, CURLOPT_USE_SSL, (long)CURLUSESSL_CONTROL); 1180 1181 /* new in curl 7.16.1 */ 1182 if(config->ftp_ssl_ccc) 1183 my_setopt_enum(curl, CURLOPT_FTP_SSL_CCC, 1184 (long)config->ftp_ssl_ccc_mode); 1185 1186#if defined(HAVE_GSSAPI) || defined(USE_WINDOWS_SSPI) 1187 { 1188 /* TODO: Make this a run-time check instead of compile-time one. */ 1189 1190 /* new in curl 7.19.4 */ 1191 if(config->socks5_gssapi_service) 1192 my_setopt_str(curl, CURLOPT_SOCKS5_GSSAPI_SERVICE, 1193 config->socks5_gssapi_service); 1194 1195 /* new in curl 7.19.4 */ 1196 if(config->socks5_gssapi_nec) 1197 my_setopt_str(curl, CURLOPT_SOCKS5_GSSAPI_NEC, 1198 config->socks5_gssapi_nec); 1199 } 1200#endif 1201 /* curl 7.13.0 */ 1202 my_setopt_str(curl, CURLOPT_FTP_ACCOUNT, config->ftp_account); 1203 1204 my_setopt(curl, CURLOPT_IGNORE_CONTENT_LENGTH, config->ignorecl?1L:0L); 1205 1206 /* curl 7.14.2 */ 1207 my_setopt(curl, CURLOPT_FTP_SKIP_PASV_IP, config->ftp_skip_ip?1L:0L); 1208 1209 /* curl 7.15.1 */ 1210 my_setopt(curl, CURLOPT_FTP_FILEMETHOD, (long)config->ftp_filemethod); 1211 1212 /* curl 7.15.2 */ 1213 if(config->localport) { 1214 my_setopt(curl, CURLOPT_LOCALPORT, (long)config->localport); 1215 my_setopt_str(curl, CURLOPT_LOCALPORTRANGE, 1216 (long)config->localportrange); 1217 } 1218 1219 /* curl 7.15.5 */ 1220 my_setopt_str(curl, CURLOPT_FTP_ALTERNATIVE_TO_USER, 1221 config->ftp_alternative_to_user); 1222 1223 /* curl 7.16.0 */ 1224 if(config->disable_sessionid) 1225 /* disable it */ 1226 my_setopt(curl, CURLOPT_SSL_SESSIONID_CACHE, 0L); 1227 1228 /* curl 7.16.2 */ 1229 if(config->raw) { 1230 my_setopt(curl, CURLOPT_HTTP_CONTENT_DECODING, 0L); 1231 my_setopt(curl, CURLOPT_HTTP_TRANSFER_DECODING, 0L); 1232 } 1233 1234 /* curl 7.17.1 */ 1235 if(!config->nokeepalive) { 1236 my_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L); 1237 if(config->alivetime != 0) { 1238#if !defined(TCP_KEEPIDLE) || !defined(TCP_KEEPINTVL) 1239 warnf(config, "Keep-alive functionality somewhat crippled due to " 1240 "missing support in your operating system!\n"); 1241#endif 1242 my_setopt(curl, CURLOPT_TCP_KEEPIDLE, config->alivetime); 1243 my_setopt(curl, CURLOPT_TCP_KEEPINTVL, config->alivetime); 1244 } 1245 } 1246 else 1247 my_setopt(curl, CURLOPT_TCP_KEEPALIVE, 0L); 1248 1249 /* curl 7.20.0 */ 1250 if(config->tftp_blksize) 1251 my_setopt(curl, CURLOPT_TFTP_BLKSIZE, config->tftp_blksize); 1252 1253 if(config->mail_from) 1254 my_setopt_str(curl, CURLOPT_MAIL_FROM, config->mail_from); 1255 1256 if(config->mail_rcpt) 1257 my_setopt_slist(curl, CURLOPT_MAIL_RCPT, config->mail_rcpt); 1258 1259 /* curl 7.20.x */ 1260 if(config->ftp_pret) 1261 my_setopt(curl, CURLOPT_FTP_USE_PRET, 1L); 1262 1263 if(config->proto_present) 1264 my_setopt_flags(curl, CURLOPT_PROTOCOLS, config->proto); 1265 if(config->proto_redir_present) 1266 my_setopt_flags(curl, CURLOPT_REDIR_PROTOCOLS, config->proto_redir); 1267 1268 if(config->content_disposition 1269 && (urlnode->flags & GETOUT_USEREMOTE) 1270 && (checkprefix("http://", this_url) || 1271 checkprefix("https://", this_url))) 1272 hdrcbdata.honor_cd_filename = TRUE; 1273 else 1274 hdrcbdata.honor_cd_filename = FALSE; 1275 1276 hdrcbdata.outs = &outs; 1277 hdrcbdata.heads = &heads; 1278 1279 my_setopt(curl, CURLOPT_HEADERFUNCTION, tool_header_cb); 1280 my_setopt(curl, CURLOPT_HEADERDATA, &hdrcbdata); 1281 1282 if(config->resolve) 1283 /* new in 7.21.3 */ 1284 my_setopt_slist(curl, CURLOPT_RESOLVE, config->resolve); 1285 1286 /* new in 7.21.4 */ 1287 if(curlinfo->features & CURL_VERSION_TLSAUTH_SRP) { 1288 if(config->tls_username) 1289 my_setopt_str(curl, CURLOPT_TLSAUTH_USERNAME, 1290 config->tls_username); 1291 if(config->tls_password) 1292 my_setopt_str(curl, CURLOPT_TLSAUTH_PASSWORD, 1293 config->tls_password); 1294 if(config->tls_authtype) 1295 my_setopt_str(curl, CURLOPT_TLSAUTH_TYPE, 1296 config->tls_authtype); 1297 } 1298 1299 /* new in 7.22.0 */ 1300 if(config->gssapi_delegation) 1301 my_setopt_str(curl, CURLOPT_GSSAPI_DELEGATION, 1302 config->gssapi_delegation); 1303 1304 /* new in 7.25.0 */ 1305 if(config->ssl_allow_beast) 1306 my_setopt(curl, CURLOPT_SSL_OPTIONS, (long)CURLSSLOPT_ALLOW_BEAST); 1307 1308 if(config->mail_auth) 1309 my_setopt_str(curl, CURLOPT_MAIL_AUTH, config->mail_auth); 1310 1311 /* new in 7.31.0 */ 1312 if(config->sasl_ir) 1313 my_setopt(curl, CURLOPT_SASL_IR, 1L); 1314 1315 if(config->nonpn) { 1316 my_setopt(curl, CURLOPT_SSL_ENABLE_NPN, 0L); 1317 } 1318 1319 if(config->noalpn) { 1320 my_setopt(curl, CURLOPT_SSL_ENABLE_ALPN, 0L); 1321 } 1322 1323 /* initialize retry vars for loop below */ 1324 retry_sleep_default = (config->retry_delay) ? 1325 config->retry_delay*1000L : RETRY_SLEEP_DEFAULT; /* ms */ 1326 1327 retry_numretries = config->req_retry; 1328 retry_sleep = retry_sleep_default; /* ms */ 1329 retrystart = tvnow(); 1330 1331#ifndef CURL_DISABLE_LIBCURL_OPTION 1332 res = easysrc_perform(); 1333 if(res) { 1334 goto show_error; 1335 } 1336#endif 1337 1338 for(;;) { 1339#ifdef USE_METALINK 1340 if(!metalink && config->use_metalink) { 1341 /* If outs.metalink_parser is non-NULL, delete it first. */ 1342 if(outs.metalink_parser) 1343 metalink_parser_context_delete(outs.metalink_parser); 1344 outs.metalink_parser = metalink_parser_context_new(); 1345 if(outs.metalink_parser == NULL) { 1346 res = CURLE_OUT_OF_MEMORY; 1347 goto show_error; 1348 } 1349 fprintf(config->global->errors, 1350 "Metalink: parsing (%s) metalink/XML...\n", this_url); 1351 } 1352 else if(metalink) 1353 fprintf(config->global->errors, 1354 "Metalink: fetching (%s) from (%s)...\n", 1355 mlfile->filename, this_url); 1356#endif /* USE_METALINK */ 1357 1358#ifdef CURLDEBUG 1359 if(config->test_event_based) 1360 res = curl_easy_perform_ev(curl); 1361 else 1362#endif 1363 res = curl_easy_perform(curl); 1364 1365 if(outs.is_cd_filename && outs.stream && !global->mute && 1366 outs.filename) 1367 printf("curl: Saved to filename '%s'\n", outs.filename); 1368 1369 /* if retry-max-time is non-zero, make sure we haven't exceeded the 1370 time */ 1371 if(retry_numretries && 1372 (!config->retry_maxtime || 1373 (tvdiff(tvnow(), retrystart) < 1374 config->retry_maxtime*1000L)) ) { 1375 enum { 1376 RETRY_NO, 1377 RETRY_TIMEOUT, 1378 RETRY_HTTP, 1379 RETRY_FTP, 1380 RETRY_LAST /* not used */ 1381 } retry = RETRY_NO; 1382 long response; 1383 if((CURLE_OPERATION_TIMEDOUT == res) || 1384 (CURLE_COULDNT_RESOLVE_HOST == res) || 1385 (CURLE_COULDNT_RESOLVE_PROXY == res) || 1386 (CURLE_FTP_ACCEPT_TIMEOUT == res)) 1387 /* retry timeout always */ 1388 retry = RETRY_TIMEOUT; 1389 else if((CURLE_OK == res) || 1390 (config->failonerror && 1391 (CURLE_HTTP_RETURNED_ERROR == res))) { 1392 /* If it returned OK. _or_ failonerror was enabled and it 1393 returned due to such an error, check for HTTP transient 1394 errors to retry on. */ 1395 char *effective_url = NULL; 1396 curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &effective_url); 1397 if(effective_url && 1398 checkprefix("http", effective_url)) { 1399 /* This was HTTP(S) */ 1400 curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response); 1401 1402 switch(response) { 1403 case 500: /* Internal Server Error */ 1404 case 502: /* Bad Gateway */ 1405 case 503: /* Service Unavailable */ 1406 case 504: /* Gateway Timeout */ 1407 retry = RETRY_HTTP; 1408 /* 1409 * At this point, we have already written data to the output 1410 * file (or terminal). If we write to a file, we must rewind 1411 * or close/re-open the file so that the next attempt starts 1412 * over from the beginning. 1413 * 1414 * TODO: similar action for the upload case. We might need 1415 * to start over reading from a previous point if we have 1416 * uploaded something when this was returned. 1417 */ 1418 break; 1419 } 1420 } 1421 } /* if CURLE_OK */ 1422 else if(res) { 1423 curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response); 1424 1425 if(response/100 == 4) 1426 /* 1427 * This is typically when the FTP server only allows a certain 1428 * amount of users and we are not one of them. All 4xx codes 1429 * are transient. 1430 */ 1431 retry = RETRY_FTP; 1432 } 1433 1434 if(retry) { 1435 static const char * const m[]={ 1436 NULL, "timeout", "HTTP error", "FTP error" 1437 }; 1438 warnf(config, "Transient problem: %s " 1439 "Will retry in %ld seconds. " 1440 "%ld retries left.\n", 1441 m[retry], retry_sleep/1000L, retry_numretries); 1442 1443 tool_go_sleep(retry_sleep); 1444 retry_numretries--; 1445 if(!config->retry_delay) { 1446 retry_sleep *= 2; 1447 if(retry_sleep > RETRY_SLEEP_MAX) 1448 retry_sleep = RETRY_SLEEP_MAX; 1449 } 1450 if(outs.bytes && outs.filename) { 1451 /* We have written data to a output file, we truncate file 1452 */ 1453 if(!global->mute) 1454 fprintf(global->errors, "Throwing away %" 1455 CURL_FORMAT_CURL_OFF_T " bytes\n", 1456 outs.bytes); 1457 fflush(outs.stream); 1458 /* truncate file at the position where we started appending */ 1459#ifdef HAVE_FTRUNCATE 1460 if(ftruncate( fileno(outs.stream), outs.init)) { 1461 /* when truncate fails, we can't just append as then we'll 1462 create something strange, bail out */ 1463 if(!global->mute) 1464 fprintf(global->errors, 1465 "failed to truncate, exiting\n"); 1466 res = CURLE_WRITE_ERROR; 1467 goto quit_urls; 1468 } 1469 /* now seek to the end of the file, the position where we 1470 just truncated the file in a large file-safe way */ 1471 fseek(outs.stream, 0, SEEK_END); 1472#else 1473 /* ftruncate is not available, so just reposition the file 1474 to the location we would have truncated it. This won't 1475 work properly with large files on 32-bit systems, but 1476 most of those will have ftruncate. */ 1477 fseek(outs.stream, (long)outs.init, SEEK_SET); 1478#endif 1479 outs.bytes = 0; /* clear for next round */ 1480 } 1481 continue; /* curl_easy_perform loop */ 1482 } 1483 } /* if retry_numretries */ 1484 else if(metalink) { 1485 /* Metalink: Decide to try the next resource or 1486 not. Basically, we want to try the next resource if 1487 download was not successful. */ 1488 long response; 1489 if(CURLE_OK == res) { 1490 /* TODO We want to try next resource when download was 1491 not successful. How to know that? */ 1492 char *effective_url = NULL; 1493 curl_easy_getinfo(curl, CURLINFO_EFFECTIVE_URL, &effective_url); 1494 if(effective_url && 1495 curlx_strnequal(effective_url, "http", 4)) { 1496 /* This was HTTP(S) */ 1497 curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response); 1498 if(response != 200 && response != 206) { 1499 metalink_next_res = 1; 1500 fprintf(global->errors, 1501 "Metalink: fetching (%s) from (%s) FAILED " 1502 "(HTTP status code %d)\n", 1503 mlfile->filename, this_url, response); 1504 } 1505 } 1506 } 1507 else { 1508 metalink_next_res = 1; 1509 fprintf(global->errors, 1510 "Metalink: fetching (%s) from (%s) FAILED (%s)\n", 1511 mlfile->filename, this_url, 1512 (errorbuffer[0]) ? 1513 errorbuffer : curl_easy_strerror((CURLcode)res)); 1514 } 1515 } 1516 if(metalink && !metalink_next_res) 1517 fprintf(global->errors, "Metalink: fetching (%s) from (%s) OK\n", 1518 mlfile->filename, this_url); 1519 1520 /* In all ordinary cases, just break out of loop here */ 1521 break; /* curl_easy_perform loop */ 1522 1523 } 1524 1525 if((global->progressmode == CURL_PROGRESS_BAR) && 1526 progressbar.calls) 1527 /* if the custom progress bar has been displayed, we output a 1528 newline here */ 1529 fputs("\n", progressbar.out); 1530 1531 if(config->writeout) 1532 ourWriteOut(curl, &outs, config->writeout); 1533 1534 if(config->writeenv) 1535 ourWriteEnv(curl); 1536 1537 /* 1538 ** Code within this loop may jump directly here to label 'show_error' 1539 ** in order to display an error message for CURLcode stored in 'res' 1540 ** variable and exit loop once that necessary writing and cleanup 1541 ** in label 'quit_urls' has been done. 1542 */ 1543 1544 show_error: 1545 1546#ifdef __VMS 1547 if(is_vms_shell()) { 1548 /* VMS DCL shell behavior */ 1549 if(!global->showerror) 1550 vms_show = VMSSTS_HIDE; 1551 } 1552 else 1553#endif 1554 if(res && global->showerror) { 1555 fprintf(global->errors, "curl: (%d) %s\n", res, (errorbuffer[0]) ? 1556 errorbuffer : curl_easy_strerror((CURLcode)res)); 1557 if(res == CURLE_SSL_CACERT) 1558 fprintf(global->errors, "%s%s", 1559 CURL_CA_CERT_ERRORMSG1, CURL_CA_CERT_ERRORMSG2); 1560 } 1561 1562 /* Fall through comment to 'quit_urls' label */ 1563 1564 /* 1565 ** Upon error condition and always that a message has already been 1566 ** displayed, code within this loop may jump directly here to label 1567 ** 'quit_urls' otherwise it should jump to 'show_error' label above. 1568 ** 1569 ** When 'res' variable is _not_ CURLE_OK loop will exit once that 1570 ** all code following 'quit_urls' has been executed. Otherwise it 1571 ** will loop to the beginning from where it may exit if there are 1572 ** no more urls left. 1573 */ 1574 1575 quit_urls: 1576 1577 /* Set file extended attributes */ 1578 if(!res && config->xattr && outs.fopened && outs.stream) { 1579 int rc = fwrite_xattr(curl, fileno(outs.stream)); 1580 if(rc) 1581 warnf(config, "Error setting extended attributes: %s\n", 1582 strerror(errno)); 1583 } 1584 1585 /* Close the file */ 1586 if(outs.fopened && outs.stream) { 1587 int rc = fclose(outs.stream); 1588 if(!res && rc) { 1589 /* something went wrong in the writing process */ 1590 res = CURLE_WRITE_ERROR; 1591 fprintf(global->errors, "(%d) Failed writing body\n", res); 1592 } 1593 } 1594 else if(!outs.s_isreg && outs.stream) { 1595 /* Dump standard stream buffered data */ 1596 int rc = fflush(outs.stream); 1597 if(!res && rc) { 1598 /* something went wrong in the writing process */ 1599 res = CURLE_WRITE_ERROR; 1600 fprintf(global->errors, "(%d) Failed writing body\n", res); 1601 } 1602 } 1603 1604#ifdef __AMIGA__ 1605 if(!res && outs.s_isreg && outs.filename) { 1606 /* Set the url (up to 80 chars) as comment for the file */ 1607 if(strlen(url) > 78) 1608 url[79] = '\0'; 1609 SetComment(outs.filename, url); 1610 } 1611#endif 1612 1613#ifdef HAVE_UTIME 1614 /* File time can only be set _after_ the file has been closed */ 1615 if(!res && config->remote_time && outs.s_isreg && outs.filename) { 1616 /* Ask libcurl if we got a remote file time */ 1617 long filetime = -1; 1618 curl_easy_getinfo(curl, CURLINFO_FILETIME, &filetime); 1619 if(filetime >= 0) { 1620 struct utimbuf times; 1621 times.actime = (time_t)filetime; 1622 times.modtime = (time_t)filetime; 1623 utime(outs.filename, ×); /* set the time we got */ 1624 } 1625 } 1626#endif 1627 1628#ifdef USE_METALINK 1629 if(!metalink && config->use_metalink && res == CURLE_OK) { 1630 int rv = parse_metalink(config, &outs, this_url); 1631 if(rv == 0) 1632 fprintf(config->global->errors, "Metalink: parsing (%s) OK\n", 1633 this_url); 1634 else if(rv == -1) 1635 fprintf(config->global->errors, "Metalink: parsing (%s) FAILED\n", 1636 this_url); 1637 } 1638 else if(metalink && res == CURLE_OK && !metalink_next_res) { 1639 int rv = metalink_check_hash(global, mlfile, outs.filename); 1640 if(rv == 0) { 1641 metalink_next_res = 1; 1642 } 1643 } 1644#endif /* USE_METALINK */ 1645 1646 /* No more business with this output struct */ 1647 if(outs.alloc_filename) 1648 Curl_safefree(outs.filename); 1649#ifdef USE_METALINK 1650 if(outs.metalink_parser) 1651 metalink_parser_context_delete(outs.metalink_parser); 1652#endif /* USE_METALINK */ 1653 memset(&outs, 0, sizeof(struct OutStruct)); 1654 hdrcbdata.outs = NULL; 1655 1656 /* Free loop-local allocated memory and close loop-local opened fd */ 1657 1658 Curl_safefree(outfile); 1659 Curl_safefree(this_url); 1660 1661 if(infdopen) 1662 close(infd); 1663 1664 if(metalink) { 1665 /* Should exit if error is fatal. */ 1666 if(is_fatal_error(res)) { 1667 break; 1668 } 1669 if(!metalink_next_res) 1670 break; 1671 mlres = mlres->next; 1672 if(mlres == NULL) 1673 /* TODO If metalink_next_res is 1 and mlres is NULL, 1674 * set res to error code 1675 */ 1676 break; 1677 } 1678 else 1679 if(urlnum > 1) { 1680 /* when url globbing, exit loop upon critical error */ 1681 if(is_fatal_error(res)) 1682 break; 1683 } 1684 else if(res) 1685 /* when not url globbing, exit loop upon any error */ 1686 break; 1687 1688 } /* loop to the next URL */ 1689 1690 /* Free loop-local allocated memory */ 1691 1692 Curl_safefree(uploadfile); 1693 1694 if(urls) { 1695 /* Free list of remaining URLs */ 1696 glob_cleanup(urls); 1697 urls = NULL; 1698 } 1699 1700 if(infilenum > 1) { 1701 /* when file globbing, exit loop upon critical error */ 1702 if(is_fatal_error(res)) 1703 break; 1704 } 1705 else if(res) 1706 /* when not file globbing, exit loop upon any error */ 1707 break; 1708 1709 } /* loop to the next globbed upload file */ 1710 1711 /* Free loop-local allocated memory */ 1712 1713 Curl_safefree(outfiles); 1714 1715 if(inglob) { 1716 /* Free list of globbed upload files */ 1717 glob_cleanup(inglob); 1718 inglob = NULL; 1719 } 1720 1721 /* Free this URL node data without destroying the 1722 the node itself nor modifying next pointer. */ 1723 Curl_safefree(urlnode->url); 1724 Curl_safefree(urlnode->outfile); 1725 Curl_safefree(urlnode->infile); 1726 urlnode->flags = 0; 1727 1728 /* 1729 ** Bail out upon critical errors 1730 */ 1731 if(is_fatal_error(res)) 1732 goto quit_curl; 1733 1734 } /* for-loop through all URLs */ 1735 1736 /* 1737 ** Nested loops end here. 1738 */ 1739 1740 quit_curl: 1741 1742 /* Reset the global config variables */ 1743 global->noprogress = orig_noprogress; 1744 global->isatty = orig_isatty; 1745 1746 /* Free function-local referenced allocated memory */ 1747 Curl_safefree(httpgetfields); 1748 1749 /* Free list of given URLs */ 1750 clean_getout(config); 1751 1752 hdrcbdata.heads = NULL; 1753 1754 /* Close function-local opened file descriptors */ 1755 if(heads.fopened && heads.stream) 1756 fclose(heads.stream); 1757 1758 if(heads.alloc_filename) 1759 Curl_safefree(heads.filename); 1760 1761 /* Release metalink related resources here */ 1762 clean_metalink(config); 1763 1764 return (CURLcode)res; 1765} 1766 1767CURLcode operate(struct GlobalConfig *config, int argc, argv_item_t argv[]) 1768{ 1769 CURLcode result = CURLE_OK; 1770 1771 /* Setup proper locale from environment */ 1772#ifdef HAVE_SETLOCALE 1773 setlocale(LC_ALL, ""); 1774#endif 1775 1776 /* Parse .curlrc if necessary */ 1777 if((argc == 1) || (!curlx_strequal(argv[1], "-q"))) { 1778 parseconfig(NULL, config); /* ignore possible failure */ 1779 1780 /* If we had no arguments then make sure a url was specified in .curlrc */ 1781 if((argc < 2) && (!config->first->url_list)) { 1782 helpf(config->errors, NULL); 1783 result = CURLE_FAILED_INIT; 1784 } 1785 } 1786 1787 if(!result) { 1788 /* Parse the command line arguments */ 1789 ParameterError res = parse_args(config, argc, argv); 1790 if(res) { 1791 result = CURLE_OK; 1792 1793 /* Check if we were asked for the help */ 1794 if(res == PARAM_HELP_REQUESTED) 1795 tool_help(); 1796 /* Check if we were asked for the manual */ 1797 else if(res == PARAM_MANUAL_REQUESTED) 1798 hugehelp(); 1799 /* Check if we were asked for the version information */ 1800 else if(res == PARAM_VERSION_INFO_REQUESTED) 1801 tool_version_info(); 1802 /* Check if we were asked to list the SSL engines */ 1803 else if(res == PARAM_ENGINES_REQUESTED) 1804 tool_list_engines(config->easy); 1805 else 1806 result = CURLE_FAILED_INIT; 1807 } 1808 else { 1809#ifndef CURL_DISABLE_LIBCURL_OPTION 1810 /* Initialise the libcurl source output */ 1811 result = easysrc_init(); 1812#endif 1813 1814 /* Perform the main operations */ 1815 if(!result) { 1816 size_t count = 0; 1817 struct OperationConfig *operation = config->first; 1818 1819 /* Get the required aguments for each operation */ 1820 while(!result && operation) { 1821 result = get_args(operation, count++); 1822 1823 operation = operation->next; 1824 } 1825 1826 /* Set the current operation pointer */ 1827 config->current = config->first; 1828 1829 /* Perform each operation */ 1830 while(!result && config->current) { 1831 result = operate_do(config, config->current); 1832 1833 config->current = config->current->next; 1834 } 1835 1836#ifndef CURL_DISABLE_LIBCURL_OPTION 1837 /* Cleanup the libcurl source output */ 1838 easysrc_cleanup(); 1839 1840 /* Dump the libcurl code if previously enabled */ 1841 dumpeasysrc(config); 1842#endif 1843 } 1844 else 1845 helpf(config->errors, "out of memory\n"); 1846 } 1847 } 1848 1849 return result; 1850} 1851