README.ares
1 _ _ ____ _
2 ___| | | | _ \| |
3 / __| | | | |_) | |
4 | (__| |_| | _ <| |___
5 \___|\___/|_| \_\_____|
6
7 How To Build libcurl to Use c-ares For Asynch Name Resolves
8 ===========================================================
9
10c-ares:
11 http://c-ares.haxx.se/
12
13NOTE
14 The latest libcurl version requires c-ares 1.6.0 or later.
15
16 Once upon the time libcurl built fine with the "original" ares. That is no
17 longer true. You need to use c-ares.
18
19Build c-ares
20============
21
221. unpack the c-ares archive
232. cd c-ares-dir
243. ./configure
254. make
265. make install
27
28Build libcurl to use c-ares in the curl source tree
29===================================================
30
311. name or symlink the c-ares source directory 'ares' in the curl source
32 directory
332. ./configure --enable-ares
34
35 Optionally, you can point out the c-ares install tree root with the the
36 --enable-ares option.
37
383. make
39
40Build libcurl to use an installed c-ares
41========================================
42
431. ./configure --enable-ares=/path/to/ares/install
442. make
45
46c-ares on win32
47===============
48(description brought by Dominick Meglio)
49
50First I compiled c-ares. I changed the default C runtime library to be the
51single-threaded rather than the multi-threaded (this seems to be required to
52prevent linking errors later on). Then I simply build the areslib project (the
53other projects adig/ahost seem to fail under MSVC).
54
55Next was libcurl. I opened lib/config-win32.h and I added a:
56 #define USE_ARES 1
57
58Next thing I did was I added the path for the ares includes to the include
59path, and the libares.lib to the libraries.
60
61Lastly, I also changed libcurl to be single-threaded rather than
62multi-threaded, again this was to prevent some duplicate symbol errors. I'm
63not sure why I needed to change everything to single-threaded, but when I
64didn't I got redefinition errors for several CRT functions (malloc, stricmp,
65etc.)
66
67I would have modified the MSVC++ project files, but I only have VC.NET and it
68uses a different format than VC6.0 so I didn't want to go and change
69everything and remove VC6.0 support from libcurl.
70
README.curl_off_t
1
2 curl_off_t explained
3 ====================
4
5curl_off_t is a data type provided by the external libcurl include headers. It
6is the type meant to be used for the curl_easy_setopt() options that end with
7LARGE. The type is 64bit large on most modern platforms.
8
9Transition from < 7.19.0 to >= 7.19.0
10-------------------------------------
11
12Applications that used libcurl before 7.19.0 that are rebuilt with a libcurl
13that is 7.19.0 or later may or may not have to worry about anything of
14this. We have made a significant effort to make the transition really seamless
15and transparent.
16
17You have have to take notice if you are in one of the following situations:
18
19o Your app is using or will after the transition use a libcurl that is built
20 with LFS (large file support) disabled even though your system otherwise
21 supports it.
22
23o Your app is using or will after the transition use a libcurl that doesn't
24 support LFS at all, but your system and compiler support 64bit data types.
25
26In both these cases, the curl_off_t type will now (after the transition) be
2764bit where it previously was 32bit. This will cause a binary incompatibility
28that you MAY need to deal with.
29
30Benefits
31--------
32
33This new way has several benefits:
34
35o Platforms without LFS support can still use libcurl to do >32 bit file
36 transfers and range operations etc as long as they have >32 bit data-types
37 supported.
38
39o Applications will no longer easily build with the curl_off_t size
40 mismatched, which has been a very frequent (and annoying) problem with
41 libcurl <= 7.18.2
42
43Historically
44------------
45
46Previously, before 7.19.0, the curl_off_t type would be rather strongly
47connected to the size of the system off_t type, where currently curl_off_t is
48independent of that.
49
50The strong connection to off_t made it troublesome for application authors
51since when they did mistakes, they could get curl_off_t type of different
52sizes in the app vs libcurl, and that caused strange effects that were hard to
53track and detect by users of libcurl.
54
55SONAME
56------
57
58We opted to not bump the soname for the library unconditionally, simply
59because soname bumping is causing a lot of grief and moaning all over the
60community so we try to keep that at minimum. Also, our selected design path
61should be 100% backwards compatible for the vast majority of all libcurl
62users.
63
64Enforce SONAME bump
65-------------------
66
67If configure doesn't detect your case where a bump is necessary, re-run it
68with the --enable-soname-bump command line option!
69
README.curlx
1 _ _ ____ _
2 ___| | | | _ \| |
3 / __| | | | |_) | |
4 | (__| |_| | _ <| |___
5 \___|\___/|_| \_\_____|
6
7 Source Code Functions Apps Might Use
8 ====================================
9
10The libcurl source code offers a few functions by source only. They are not
11part of the official libcurl API, but the source files might be useful for
12others so apps can optionally compile/build with these sources to gain
13additional functions.
14
15We provide them through a single header file for easy access for apps:
16"curlx.h"
17
18 curlx_strtoofft()
19
20 A macro that converts a string containing a number to a curl_off_t number.
21 This might use the curlx_strtoll() function which is provided as source
22 code in strtoofft.c. Note that the function is only provided if no
23 strtoll() (or equivalent) function exist on your platform. If curl_off_t
24 is only a 32 bit number on your platform, this macro uses strtol().
25
26 curlx_tvnow()
27
28 returns a struct timeval for the current time.
29
30 curlx_tvdiff()
31
32 returns the difference between two timeval structs, in number of
33 milliseconds.
34
35 curlx_tvdiff_secs()
36
37 returns the same as curlx_tvdiff but with full usec resolution (as a
38 double)
39
40FUTURE
41======
42
43 Several functions will be removed from the public curl_ name space in a
44 future libcurl release. They will then only become available as curlx_
45 functions instead. To make the transition easier, we already today provide
46 these functions with the curlx_ prefix to allow sources to get built properly
47 with the new function names. The functions this concerns are:
48
49 curlx_getenv
50 curlx_strequal
51 curlx_strnequal
52 curlx_mvsnprintf
53 curlx_msnprintf
54 curlx_maprintf
55 curlx_mvaprintf
56 curlx_msprintf
57 curlx_mprintf
58 curlx_mfprintf
59 curlx_mvsprintf
60 curlx_mvprintf
61 curlx_mvfprintf
62
README.encoding
1
2 Content Encoding Support for libcurl
3
4* About content encodings:
5
6HTTP/1.1 [RFC 2616] specifies that a client may request that a server encode
7its response. This is usually used to compress a response using one of a set
8of commonly available compression techniques. These schemes are `deflate' (the
9zlib algorithm), `gzip' and `compress' [sec 3.5, RFC 2616]. A client requests
10that the sever perform an encoding by including an Accept-Encoding header in
11the request document. The value of the header should be one of the recognized
12tokens `deflate', ... (there's a way to register new schemes/tokens, see sec
133.5 of the spec). A server MAY honor the client's encoding request. When a
14response is encoded, the server includes a Content-Encoding header in the
15response. The value of the Content-Encoding header indicates which scheme was
16used to encode the data.
17
18A client may tell a server that it can understand several different encoding
19schemes. In this case the server may choose any one of those and use it to
20encode the response (indicating which one using the Content-Encoding header).
21It's also possible for a client to attach priorities to different schemes so
22that the server knows which it prefers. See sec 14.3 of RFC 2616 for more
23information on the Accept-Encoding header.
24
25* Current support for content encoding:
26
27Support for the 'deflate' and 'gzip' content encoding are supported by
28libcurl. Both regular and chunked transfers should work fine. The library
29zlib is required for this feature. 'deflate' support was added by James
30Gallagher, and support for the 'gzip' encoding was added by Dan Fandrich.
31
32* The libcurl interface:
33
34To cause libcurl to request a content encoding use:
35
36 curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, <string>)
37
38where <string> is the intended value of the Accept-Encoding header.
39
40Currently, libcurl only understands how to process responses that use the
41"deflate" or "gzip" Content-Encoding, so the only values for
42CURLOPT_ACCEPT_ENCODING that will work (besides "identity," which does
43nothing) are "deflate" and "gzip" If a response is encoded using the
44"compress" or methods, libcurl will return an error indicating that the
45response could not be decoded. If <string> is NULL no Accept-Encoding header
46is generated. If <string> is a zero-length string, then an Accept-Encoding
47header containing all supported encodings will be generated.
48
49The CURLOPT_ACCEPT_ENCODING must be set to any non-NULL value for content to
50be automatically decoded. If it is not set and the server still sends encoded
51content (despite not having been asked), the data is returned in its raw form
52and the Content-Encoding type is not checked.
53
54* The curl interface:
55
56Use the --compressed option with curl to cause it to ask servers to compress
57responses using any format supported by curl.
58
59James Gallagher <jgallagher@gso.uri.edu>
60Dan Fandrich <dan@coneharvesters.com>
61
README.hostip
1 hostip.c explained
2 ==================
3
4 The main COMPILE-TIME DEFINES to keep in mind when reading the host*.c
5 source file are these:
6
7 CURLRES_IPV6 - this host has getaddrinfo() and family, and thus we use
8 that. The host may not be able to resolve IPv6, but we don't really have to
9 take that into account. Hosts that aren't IPv6-enabled have CURLRES_IPV4
10 defined.
11
12 CURLRES_ARES - is defined if libcurl is built to use c-ares for asynchronous
13 name resolves. It cannot have ENABLE_IPV6 defined at the same time, as c-ares
14 has no ipv6 support. This can be Windows or *nix.
15
16 CURLRES_THREADED - is defined if libcurl is built to run under (native)
17 Windows, and then the name resolve will be done in a new thread, and the
18 supported asynch API will be the same as for ares-builds.
19
20 If any of the two previous are defined, CURLRES_ASYNCH is defined too. If
21 libcurl is not built to use an asynchronous resolver, CURLRES_SYNCH is
22 defined.
23
24 The host*.c sources files are split up like this:
25
26 hostip.c - method-independent resolver functions and utility functions
27 hostasyn.c - functions for asynchronous name resolves
28 hostsyn.c - functions for synchronous name resolves
29 hostares.c - functions for ares-using name resolves
30 hostthre.c - functions for threaded name resolves
31 hostip4.c - ipv4-specific functions
32 hostip6.c - ipv6-specific functions
33
34 The hostip.h is the single united header file for all this. It defines the
35 CURLRES_* defines based on the config*.h and setup.h defines.
36
README.httpauth
1
21. PUT/POST without a known auth to use (possibly no auth required):
3
4 (When explicitly set to use a multi-pass auth when doing a POST/PUT,
5 libcurl should immediately go the Content-Length: 0 bytes route to avoid
6 the first send all data phase, step 2. If told to use a single-pass auth,
7 goto step 3.)
8
9 Issue the proper PUT/POST request immediately, with the correct
10 Content-Length and Expect: headers.
11
12 If a 100 response is received or the wait for one times out, start sending
13 the request-body.
14
15 If a 401 (or 407 when talking through a proxy) is received, then:
16
17 If we have "more than just a little" data left to send, close the
18 connection. Exactly what "more than just a little" means will have to be
19 determined. Possibly the current transfer speed should be taken into
20 account as well.
21
22 NOTE: if the size of the POST data is less than MAX_INITIAL_POST_SIZE (when
23 CURLOPT_POSTFIELDS is used), libcurl will send everything in one single
24 write() (all request-headers and request-body) and thus it will
25 unconditionally send the full post data here.
26
272. PUT/POST with multi-pass auth but not yet completely negotiated:
28
29 Send a PUT/POST request, we know that it will be rejected and thus we claim
30 Content-Length zero to avoid having to send the request-body. (This seems
31 to be what IE does.)
32
333. PUT/POST as the last step in the auth negotiation, that is when we have
34 what we believe is a completed negotiation:
35
36 Send a full and proper PUT/POST request (again) with the proper
37 Content-Length and a following request-body.
38
39 NOTE: this may very well be the second (or even third) time the whole or at
40 least parts of the request body is sent to the server. Since the data may
41 be provided to libcurl with a callback, we need a way to tell the app that
42 the upload is to be restarted so that the callback will provide data from
43 the start again. This requires an API method/mechanism that libcurl
44 doesn't have today. See below.
45
46Data Rewind
47
48 It will be troublesome for some apps to deal with a rewind like this in all
49 circumstances. I'm thinking for example when using 'curl' to upload data
50 from stdin. If libcurl ends up having to rewind the reading for a request
51 to succeed, of course a lack of this callback or if it returns failure, will
52 cause the request to fail completely.
53
54 The new callback is set with CURLOPT_IOCTLFUNCTION (in an attempt to add a
55 more generic function that might be used for other IO-related controls in
56 the future):
57
58 curlioerr curl_ioctl(CURL *handle, curliocmd cmd, void *clientp);
59
60 And in the case where the read is to be rewinded, it would be called with a
61 cmd named CURLIOCMD_RESTARTREAD. The callback would then return CURLIOE_OK,
62 if things are fine, or CURLIOE_FAILRESTART if not.
63
64Backwards Compatibility
65
66 The approach used until now, that issues a HEAD on the given URL to trigger
67 the auth negotiation could still be supported and encouraged, but it would
68 be up to the app to first fetch a URL with GET/HEAD to negotiate on, since
69 then a following PUT/POST wouldn't need to negotiate authentication and
70 thus avoid double-sending data.
71
72 Optionally, we keep the current approach if some option is set
73 (CURLOPT_HEADBEFOREAUTH or similar), since it seems to work fairly well for
74 POST on most servers.
75
README.memoryleak
1 _ _ ____ _
2 ___| | | | _ \| |
3 / __| | | | |_) | |
4 | (__| |_| | _ <| |___
5 \___|\___/|_| \_\_____|
6
7 How To Track Down Suspected Memory Leaks in libcurl
8 ===================================================
9
10Single-threaded
11
12 Please note that this memory leak system is not adjusted to work in more
13 than one thread. If you want/need to use it in a multi-threaded app. Please
14 adjust accordingly.
15
16
17Build
18
19 Rebuild libcurl with -DCURLDEBUG (usually, rerunning configure with
20 --enable-debug fixes this). 'make clean' first, then 'make' so that all
21 files actually are rebuilt properly. It will also make sense to build
22 libcurl with the debug option (usually -g to the compiler) so that debugging
23 it will be easier if you actually do find a leak in the library.
24
25 This will create a library that has memory debugging enabled.
26
27Modify Your Application
28
29 Add a line in your application code:
30
31 curl_memdebug("dump");
32
33 This will make the malloc debug system output a full trace of all resource
34 using functions to the given file name. Make sure you rebuild your program
35 and that you link with the same libcurl you built for this purpose as
36 described above.
37
38Run Your Application
39
40 Run your program as usual. Watch the specified memory trace file grow.
41
42 Make your program exit and use the proper libcurl cleanup functions etc. So
43 that all non-leaks are returned/freed properly.
44
45Analyze the Flow
46
47 Use the tests/memanalyze.pl perl script to analyze the dump file:
48
49 tests/memanalyze.pl dump
50
51 This now outputs a report on what resources that were allocated but never
52 freed etc. This report is very fine for posting to the list!
53
54 If this doesn't produce any output, no leak was detected in libcurl. Then
55 the leak is mostly likely to be in your code.
56
README.multi_socket
1Implementation of the curl_multi_socket API
2
3 The main ideas of the new API are simply:
4
5 1 - The application can use whatever event system it likes as it gets info
6 from libcurl about what file descriptors libcurl waits for what action
7 on. (The previous API returns fd_sets which is very select()-centric).
8
9 2 - When the application discovers action on a single socket, it calls
10 libcurl and informs that there was action on this particular socket and
11 libcurl can then act on that socket/transfer only and not care about
12 any other transfers. (The previous API always had to scan through all
13 the existing transfers.)
14
15 The idea is that curl_multi_socket_action() calls a given callback with
16 information about what socket to wait for what action on, and the callback
17 only gets called if the status of that socket has changed.
18
19 We also added a timer callback that makes libcurl call the application when
20 the timeout value changes, and you set that with curl_multi_setopt() and the
21 CURLMOPT_TIMERFUNCTION option. To get this to work, Internally, there's an
22 added a struct to each easy handle in which we store an "expire time" (if
23 any). The structs are then "splay sorted" so that we can add and remove
24 times from the linked list and yet somewhat swiftly figure out both how long
25 time there is until the next nearest timer expires and which timer (handle)
26 we should take care of now. Of course, the upside of all this is that we get
27 a curl_multi_timeout() that should also work with old-style applications
28 that use curl_multi_perform().
29
30 We created an internal "socket to easy handles" hash table that given
31 a socket (file descriptor) return the easy handle that waits for action on
32 that socket. This hash is made using the already existing hash code
33 (previously only used for the DNS cache).
34
35 To make libcurl able to report plain sockets in the socket callback, we had
36 to re-organize the internals of the curl_multi_fdset() etc so that the
37 conversion from sockets to fd_sets for that function is only done in the
38 last step before the data is returned. I also had to extend c-ares to get a
39 function that can return plain sockets, as that library too returned only
40 fd_sets and that is no longer good enough. The changes done to c-ares are
41 available in c-ares 1.3.1 and later.
42
43 We have done a test runs with up to 9000 connections (with a single active
44 one). The curl_multi_socket_action() invoke then takes less than 10
45 microseconds in average (using the read-only-1-byte-at-a-time hack). We are
46 now below the 60 microseconds "per socket action" goal (the extra 50 is the
47 time libevent needs).
48
49Documentation
50
51 http://curl.haxx.se/libcurl/c/curl_multi_socket_action.html
52 http://curl.haxx.se/libcurl/c/curl_multi_timeout.html
53 http://curl.haxx.se/libcurl/c/curl_multi_setopt.html
54
README.pingpong
1Date: December 5, 2009
2
3Pingpong
4========
5
6 Pingpong is just my (Daniel's) jestful collective name on the protocols that
7 share a very similar kind of back-and-forth procedure with command and
8 responses to and from the server. FTP was previously the only protocol in
9 that family that libcurl supported, but when POP3, IMAP and SMTP joined the
10 team I moved some of the internals into a separate pingpong module to be
11 easier to get used by all these protocols to reduce code duplication and ease
12 code re-use between these protocols.
13
14FTP
15
16 In 7.20.0 we converted code to use the new pingpong code from previously
17 having been all "native" FTP code.
18
19POP3
20
21 There's no support in the documented URL format to specify the exact mail to
22 get, but we support that as the path specified in the URL.
23
24IMAP
25
26SMTP
27
28 There's no official URL syntax defined for SMTP, but we use only the generic
29 one and we provide two additional libcurl options to specify receivers and
30 sender of the actual mail.
31
README.pipelining
1HTTP Pipelining with libcurl
2============================
3
4Background
5
6Since pipelining implies that one or more requests are sent to a server before
7the previous response(s) have been received, we only support it for multi
8interface use.
9
10Considerations
11
12When using the multi interface, you create one easy handle for each transfer.
13Bascially any number of handles can be created, added and used with the multi
14interface - simultaneously. It is an interface designed to allow many
15simultaneous transfers while still using a single thread. Pipelining does not
16change any of these details.
17
18API
19
20We've added a new option to curl_multi_setopt() called CURLMOPT_PIPELINING
21that enables "attempted pipelining" and then all easy handles used on that
22handle will attempt to use an existing pipeline.
23
24Details
25
26- A pipeline is only created if a previous connection exists to the same IP
27 address that the new request is being made to use.
28
29- Pipelines are only supported for HTTP(S) as no other currently supported
30 protocol has features resemembling this, but we still name this feature
31 plain 'pipelining' to possibly one day support it for other protocols as
32 well.
33
34- HTTP Pipelining is for GET and HEAD requests only.
35
36- When a pipeline is in use, we must take precautions so that when used easy
37 handles (i.e those who still wait for a response) are removed from the multi
38 handle, we must deal with the outstanding response nicely.
39
40- Explicitly asking for pipelining handle X and handle Y won't be supported.
41 It isn't easy for an app to do this association. The lib should probably
42 still resolve the second one properly to make sure that they actually _can_
43 be considered for pipelining. Also, asking for explicit pipelining on handle
44 X may be tricky when handle X get a closed connection.
45
46- We need options to control max pipeline length, and probably how to behave
47 if we reach that limit. As was discussed on the list, it can probably be
48 made very complicated, so perhaps we can think of a way to pass all
49 variables involved to a callback and let the application decide how to act
50 in specific situations. Either way, these fancy options are only interesting
51 to work on when everything is working and we have working apps to test with.
52