1Online: http://curl.haxx.se/docs/httpscripting.html 2Date: Jan 19, 2011 3 4 The Art Of Scripting HTTP Requests Using Curl 5 ============================================= 6 7 This document will assume that you're familiar with HTML and general 8 networking. 9 10 The possibility to write scripts is essential to make a good computer 11 system. Unix' capability to be extended by shell scripts and various tools to 12 run various automated commands and scripts is one reason why it has succeeded 13 so well. 14 15 The increasing amount of applications moving to the web has made "HTTP 16 Scripting" more frequently requested and wanted. To be able to automatically 17 extract information from the web, to fake users, to post or upload data to 18 web servers are all important tasks today. 19 20 Curl is a command line tool for doing all sorts of URL manipulations and 21 transfers, but this particular document will focus on how to use it when 22 doing HTTP requests for fun and profit. I'll assume that you know how to 23 invoke 'curl --help' or 'curl --manual' to get basic information about it. 24 25 Curl is not written to do everything for you. It makes the requests, it gets 26 the data, it sends data and it retrieves the information. You probably need 27 to glue everything together using some kind of script language or repeated 28 manual invokes. 29 301. The HTTP Protocol 31 32 HTTP is the protocol used to fetch data from web servers. It is a very simple 33 protocol that is built upon TCP/IP. The protocol also allows information to 34 get sent to the server from the client using a few different methods, as will 35 be shown here. 36 37 HTTP is plain ASCII text lines being sent by the client to a server to 38 request a particular action, and then the server replies a few text lines 39 before the actual requested content is sent to the client. 40 41 The client, curl, sends a HTTP request. The request contains a method (like 42 GET, POST, HEAD etc), a number of request headers and sometimes a request 43 body. The HTTP server responds with a status line (indicating if things went 44 well), response headers and most often also a response body. The "body" part 45 is the plain data you requested, like the actual HTML or the image etc. 46 47 1.1 See the Protocol 48 49 Using curl's option --verbose (-v as a short option) will display what kind 50 of commands curl sends to the server, as well as a few other informational 51 texts. 52 53 --verbose is the single most useful option when it comes to debug or even 54 understand the curl<->server interaction. 55 56 Sometimes even --verbose is not enough. Then --trace and --trace-ascii offer 57 even more details as they show EVERYTHING curl sends and receives. Use it 58 like this: 59 60 curl --trace-ascii debugdump.txt http://www.example.com/ 61 622. URL 63 64 The Uniform Resource Locator format is how you specify the address of a 65 particular resource on the Internet. You know these, you've seen URLs like 66 http://curl.haxx.se or https://yourbank.com a million times. 67 683. GET a page 69 70 The simplest and most common request/operation made using HTTP is to get a 71 URL. The URL could itself refer to a web page, an image or a file. The client 72 issues a GET request to the server and receives the document it asked for. 73 If you issue the command line 74 75 curl http://curl.haxx.se 76 77 you get a web page returned in your terminal window. The entire HTML document 78 that that URL holds. 79 80 All HTTP replies contain a set of response headers that are normally hidden, 81 use curl's --include (-i) option to display them as well as the rest of the 82 document. You can also ask the remote server for ONLY the headers by using 83 the --head (-I) option (which will make curl issue a HEAD request). 84 854. Forms 86 87 Forms are the general way a web site can present a HTML page with fields for 88 the user to enter data in, and then press some kind of 'OK' or 'submit' 89 button to get that data sent to the server. The server then typically uses 90 the posted data to decide how to act. Like using the entered words to search 91 in a database, or to add the info in a bug track system, display the entered 92 address on a map or using the info as a login-prompt verifying that the user 93 is allowed to see what it is about to see. 94 95 Of course there has to be some kind of program in the server end to receive 96 the data you send. You cannot just invent something out of the air. 97 98 4.1 GET 99 100 A GET-form uses the method GET, as specified in HTML like: 101 102 <form method="GET" action="junk.cgi"> 103 <input type=text name="birthyear"> 104 <input type=submit name=press value="OK"> 105 </form> 106 107 In your favorite browser, this form will appear with a text box to fill in 108 and a press-button labeled "OK". If you fill in '1905' and press the OK 109 button, your browser will then create a new URL to get for you. The URL will 110 get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the 111 previous URL. 112 113 If the original form was seen on the page "www.hotmail.com/when/birth.html", 114 the second page you'll get will become 115 "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK". 116 117 Most search engines work this way. 118 119 To make curl do the GET form post for you, just enter the expected created 120 URL: 121 122 curl "http://www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK" 123 124 4.2 POST 125 126 The GET method makes all input field names get displayed in the URL field of 127 your browser. That's generally a good thing when you want to be able to 128 bookmark that page with your given data, but it is an obvious disadvantage 129 if you entered secret information in one of the fields or if there are a 130 large amount of fields creating a very long and unreadable URL. 131 132 The HTTP protocol then offers the POST method. This way the client sends the 133 data separated from the URL and thus you won't see any of it in the URL 134 address field. 135 136 The form would look very similar to the previous one: 137 138 <form method="POST" action="junk.cgi"> 139 <input type=text name="birthyear"> 140 <input type=submit name=press value=" OK "> 141 </form> 142 143 And to use curl to post this form with the same data filled in as before, we 144 could do it like: 145 146 curl --data "birthyear=1905&press=%20OK%20" \ 147 http://www.example.com/when.cgi 148 149 This kind of POST will use the Content-Type 150 application/x-www-form-urlencoded and is the most widely used POST kind. 151 152 The data you send to the server MUST already be properly encoded, curl will 153 not do that for you. For example, if you want the data to contain a space, 154 you need to replace that space with %20 etc. Failing to comply with this 155 will most likely cause your data to be received wrongly and messed up. 156 157 Recent curl versions can in fact url-encode POST data for you, like this: 158 159 curl --data-urlencode "name=I am Daniel" http://www.example.com 160 161 4.3 File Upload POST 162 163 Back in late 1995 they defined an additional way to post data over HTTP. It 164 is documented in the RFC 1867, why this method sometimes is referred to as 165 RFC1867-posting. 166 167 This method is mainly designed to better support file uploads. A form that 168 allows a user to upload a file could be written like this in HTML: 169 170 <form method="POST" enctype='multipart/form-data' action="upload.cgi"> 171 <input type=file name=upload> 172 <input type=submit name=press value="OK"> 173 </form> 174 175 This clearly shows that the Content-Type about to be sent is 176 multipart/form-data. 177 178 To post to a form like this with curl, you enter a command line like: 179 180 curl --form upload=@localfilename --form press=OK [URL] 181 182 4.4 Hidden Fields 183 184 A very common way for HTML based application to pass state information 185 between pages is to add hidden fields to the forms. Hidden fields are 186 already filled in, they aren't displayed to the user and they get passed 187 along just as all the other fields. 188 189 A similar example form with one visible field, one hidden field and one 190 submit button could look like: 191 192 <form method="POST" action="foobar.cgi"> 193 <input type=text name="birthyear"> 194 <input type=hidden name="person" value="daniel"> 195 <input type=submit name="press" value="OK"> 196 </form> 197 198 To post this with curl, you won't have to think about if the fields are 199 hidden or not. To curl they're all the same: 200 201 curl --data "birthyear=1905&press=OK&person=daniel" [URL] 202 203 4.5 Figure Out What A POST Looks Like 204 205 When you're about fill in a form and send to a server by using curl instead 206 of a browser, you're of course very interested in sending a POST exactly the 207 way your browser does. 208 209 An easy way to get to see this, is to save the HTML page with the form on 210 your local disk, modify the 'method' to a GET, and press the submit button 211 (you could also change the action URL if you want to). 212 213 You will then clearly see the data get appended to the URL, separated with a 214 '?'-letter as GET forms are supposed to. 215 2165. PUT 217 218 The perhaps best way to upload data to a HTTP server is to use PUT. Then 219 again, this of course requires that someone put a program or script on the 220 server end that knows how to receive a HTTP PUT stream. 221 222 Put a file to a HTTP server with curl: 223 224 curl --upload-file uploadfile http://www.example.com/receive.cgi 225 2266. HTTP Authentication 227 228 HTTP Authentication is the ability to tell the server your username and 229 password so that it can verify that you're allowed to do the request you're 230 doing. The Basic authentication used in HTTP (which is the type curl uses by 231 default) is *plain* *text* based, which means it sends username and password 232 only slightly obfuscated, but still fully readable by anyone that sniffs on 233 the network between you and the remote server. 234 235 To tell curl to use a user and password for authentication: 236 237 curl --user name:password http://www.example.com 238 239 The site might require a different authentication method (check the headers 240 returned by the server), and then --ntlm, --digest, --negotiate or even 241 --anyauth might be options that suit you. 242 243 Sometimes your HTTP access is only available through the use of a HTTP 244 proxy. This seems to be especially common at various companies. A HTTP proxy 245 may require its own user and password to allow the client to get through to 246 the Internet. To specify those with curl, run something like: 247 248 curl --proxy-user proxyuser:proxypassword curl.haxx.se 249 250 If your proxy requires the authentication to be done using the NTLM method, 251 use --proxy-ntlm, if it requires Digest use --proxy-digest. 252 253 If you use any one these user+password options but leave out the password 254 part, curl will prompt for the password interactively. 255 256 Do note that when a program is run, its parameters might be possible to see 257 when listing the running processes of the system. Thus, other users may be 258 able to watch your passwords if you pass them as plain command line 259 options. There are ways to circumvent this. 260 261 It is worth noting that while this is how HTTP Authentication works, very 262 many web sites will not use this concept when they provide logins etc. See 263 the Web Login chapter further below for more details on that. 264 2657. Referer 266 267 A HTTP request may include a 'referer' field (yes it is misspelled), which 268 can be used to tell from which URL the client got to this particular 269 resource. Some programs/scripts check the referer field of requests to verify 270 that this wasn't arriving from an external site or an unknown page. While 271 this is a stupid way to check something so easily forged, many scripts still 272 do it. Using curl, you can put anything you want in the referer-field and 273 thus more easily be able to fool the server into serving your request. 274 275 Use curl to set the referer field with: 276 277 curl --referer http://www.example.come http://www.example.com 278 2798. User Agent 280 281 Very similar to the referer field, all HTTP requests may set the User-Agent 282 field. It names what user agent (client) that is being used. Many 283 applications use this information to decide how to display pages. Silly web 284 programmers try to make different pages for users of different browsers to 285 make them look the best possible for their particular browsers. They usually 286 also do different kinds of javascript, vbscript etc. 287 288 At times, you will see that getting a page with curl will not return the same 289 page that you see when getting the page with your browser. Then you know it 290 is time to set the User Agent field to fool the server into thinking you're 291 one of those browsers. 292 293 To make curl look like Internet Explorer 5 on a Windows 2000 box: 294 295 curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL] 296 297 Or why not look like you're using Netscape 4.73 on an old Linux box: 298 299 curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL] 300 3019. Redirects 302 303 When a resource is requested from a server, the reply from the server may 304 include a hint about where the browser should go next to find this page, or a 305 new page keeping newly generated output. The header that tells the browser 306 to redirect is Location:. 307 308 Curl does not follow Location: headers by default, but will simply display 309 such pages in the same manner it display all HTTP replies. It does however 310 feature an option that will make it attempt to follow the Location: pointers. 311 312 To tell curl to follow a Location: 313 314 curl --location http://www.example.com 315 316 If you use curl to POST to a site that immediately redirects you to another 317 page, you can safely use --location (-L) and --data/--form together. Curl will 318 only use POST in the first request, and then revert to GET in the following 319 operations. 320 32110. Cookies 322 323 The way the web browsers do "client side state control" is by using 324 cookies. Cookies are just names with associated contents. The cookies are 325 sent to the client by the server. The server tells the client for what path 326 and host name it wants the cookie sent back, and it also sends an expiration 327 date and a few more properties. 328 329 When a client communicates with a server with a name and path as previously 330 specified in a received cookie, the client sends back the cookies and their 331 contents to the server, unless of course they are expired. 332 333 Many applications and servers use this method to connect a series of requests 334 into a single logical session. To be able to use curl in such occasions, we 335 must be able to record and send back cookies the way the web application 336 expects them. The same way browsers deal with them. 337 338 The simplest way to send a few cookies to the server when getting a page with 339 curl is to add them on the command line like: 340 341 curl --cookie "name=Daniel" http://www.example.com 342 343 Cookies are sent as common HTTP headers. This is practical as it allows curl 344 to record cookies simply by recording headers. Record cookies with curl by 345 using the --dump-header (-D) option like: 346 347 curl --dump-header headers_and_cookies http://www.example.com 348 349 (Take note that the --cookie-jar option described below is a better way to 350 store cookies.) 351 352 Curl has a full blown cookie parsing engine built-in that comes to use if you 353 want to reconnect to a server and use cookies that were stored from a 354 previous connection (or handicrafted manually to fool the server into 355 believing you had a previous connection). To use previously stored cookies, 356 you run curl like: 357 358 curl --cookie stored_cookies_in_file http://www.example.com 359 360 Curl's "cookie engine" gets enabled when you use the --cookie option. If you 361 only want curl to understand received cookies, use --cookie with a file that 362 doesn't exist. Example, if you want to let curl understand cookies from a 363 page and follow a location (and thus possibly send back cookies it received), 364 you can invoke it like: 365 366 curl --cookie nada --location http://www.example.com 367 368 Curl has the ability to read and write cookie files that use the same file 369 format that Netscape and Mozilla do. It is a convenient way to share cookies 370 between browsers and automatic scripts. The --cookie (-b) switch 371 automatically detects if a given file is such a cookie file and parses it, 372 and by using the --cookie-jar (-c) option you'll make curl write a new cookie 373 file at the end of an operation: 374 375 curl --cookie cookies.txt --cookie-jar newcookies.txt \ 376 http://www.example.com 377 37811. HTTPS 379 380 There are a few ways to do secure HTTP transfers. The by far most common 381 protocol for doing this is what is generally known as HTTPS, HTTP over 382 SSL. SSL encrypts all the data that is sent and received over the network and 383 thus makes it harder for attackers to spy on sensitive information. 384 385 SSL (or TLS as the latest version of the standard is called) offers a 386 truckload of advanced features to allow all those encryptions and key 387 infrastructure mechanisms encrypted HTTP requires. 388 389 Curl supports encrypted fetches thanks to the freely available OpenSSL 390 libraries. To get a page from a HTTPS server, simply run curl like: 391 392 curl https://secure.example.com 393 394 11.1 Certificates 395 396 In the HTTPS world, you use certificates to validate that you are the one 397 you claim to be, as an addition to normal passwords. Curl supports client- 398 side certificates. All certificates are locked with a pass phrase, which you 399 need to enter before the certificate can be used by curl. The pass phrase 400 can be specified on the command line or if not, entered interactively when 401 curl queries for it. Use a certificate with curl on a HTTPS server like: 402 403 curl --cert mycert.pem https://secure.example.com 404 405 curl also tries to verify that the server is who it claims to be, by 406 verifying the server's certificate against a locally stored CA cert 407 bundle. Failing the verification will cause curl to deny the connection. You 408 must then use --insecure (-k) in case you want to tell curl to ignore that 409 the server can't be verified. 410 411 More about server certificate verification and ca cert bundles can be read 412 in the SSLCERTS document, available online here: 413 414 http://curl.haxx.se/docs/sslcerts.html 415 41612. Custom Request Elements 417 418 Doing fancy stuff, you may need to add or change elements of a single curl 419 request. 420 421 For example, you can change the POST request to a PROPFIND and send the data 422 as "Content-Type: text/xml" (instead of the default Content-Type) like this: 423 424 curl --data "<xml>" --header "Content-Type: text/xml" \ 425 --request PROPFIND url.com 426 427 You can delete a default header by providing one without content. Like you 428 can ruin the request by chopping off the Host: header: 429 430 curl --header "Host:" http://www.example.com 431 432 You can add headers the same way. Your server may want a "Destination:" 433 header, and you can add it: 434 435 curl --header "Destination: http://nowhere" http://example.com 436 43713. Web Login 438 439 While not strictly just HTTP related, it still cause a lot of people problems 440 so here's the executive run-down of how the vast majority of all login forms 441 work and how to login to them using curl. 442 443 It can also be noted that to do this properly in an automated fashion, you 444 will most certainly need to script things and do multiple curl invokes etc. 445 446 First, servers mostly use cookies to track the logged-in status of the 447 client, so you will need to capture the cookies you receive in the 448 responses. Then, many sites also set a special cookie on the login page (to 449 make sure you got there through their login page) so you should make a habit 450 of first getting the login-form page to capture the cookies set there. 451 452 Some web-based login systems features various amounts of javascript, and 453 sometimes they use such code to set or modify cookie contents. Possibly they 454 do that to prevent programmed logins, like this manual describes how to... 455 Anyway, if reading the code isn't enough to let you repeat the behavior 456 manually, capturing the HTTP requests done by your browers and analyzing the 457 sent cookies is usually a working method to work out how to shortcut the 458 javascript need. 459 460 In the actual <form> tag for the login, lots of sites fill-in random/session 461 or otherwise secretly generated hidden tags and you may need to first capture 462 the HTML code for the login form and extract all the hidden fields to be able 463 to do a proper login POST. Remember that the contents need to be URL encoded 464 when sent in a normal POST. 465 46614. Debug 467 468 Many times when you run curl on a site, you'll notice that the site doesn't 469 seem to respond the same way to your curl requests as it does to your 470 browser's. 471 472 Then you need to start making your curl requests more similar to your 473 browser's requests: 474 475 * Use the --trace-ascii option to store fully detailed logs of the requests 476 for easier analyzing and better understanding 477 478 * Make sure you check for and use cookies when needed (both reading with 479 --cookie and writing with --cookie-jar) 480 481 * Set user-agent to one like a recent popular browser does 482 483 * Set referer like it is set by the browser 484 485 * If you use POST, make sure you send all the fields and in the same order as 486 the browser does it. (See chapter 4.5 above) 487 488 A very good helper to make sure you do this right, is the LiveHTTPHeader tool 489 that lets you view all headers you send and receive with Mozilla/Firefox 490 (even when using HTTPS). 491 492 A more raw approach is to capture the HTTP traffic on the network with tools 493 such as ethereal or tcpdump and check what headers that were sent and 494 received by the browser. (HTTPS makes this technique inefficient.) 495 49615. References 497 498 RFC 2616 is a must to read if you want in-depth understanding of the HTTP 499 protocol. 500 501 RFC 3986 explains the URL syntax. 502 503 RFC 2109 defines how cookies are supposed to work. 504 505 RFC 1867 defines the HTTP post upload format. 506 507 http://curl.haxx.se is the home of the cURL project 508