class: center, middle
# LibCurl and Perl 6 Curt Tilmes *Curt.Tilmes@nasa.gov* *Philadelphia Perl Mongers* 2017-06-12 --- # Introduction * **`curl`** is a popular command line tool for transferring data with URLs -- * **`libcurl`** is a C library encompassing the functionality: libcurl is a free and easy-to-use client-side URL transfer library, supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP. libcurl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, Kerberos), file transfer resume, http proxy tunneling and more! libcurl is free, thread-safe, IPv6 compatible, feature rich, well supported, fast, thoroughly documented and is already used by many known, big and successful companies. -- * **`LibCurl`** is a Perl 6 module using the NativeCall capability of Perl 6 to interface directly with `libcurl` --- # Overview `libcurl` supports two interfaces, both of which are mirrored into `LibCurl`: * **Easy** - "The easy interface is a synchronous, efficient, quickly used and... yes, easy interface for file transfers." - The libcurl bindings are available via a low-level interface in `LibCurl::EasyHandle`. A friendly Perl 6 class makes them available through a high level interface as `LibCurl::Easy`. -- * **Multi** - "The multi interface is the asynchronous brother in the family and it also offers multiple transfers using a single thread and more." - The multi bindings are similarly available via a low-level interface in `LibCurl::MultiHandle`, and wrapped in a high level interface as `LibCurl::Multi`. --- # LibCurl::Easy examples ``` use LibCurl::Easy; print LibCurl::Easy.new(URL => 'http://example.com').perform.content; ``` -- Let's break it up into three phases: 1 .block[.new()] Construct a new LibCurl::Easy Object You can pass in many options (~80 so far implemented) to control the nature of the desired transfer. The only option that is required is `URL`. You can also add options to the handle later. -- 2 .block[.perform()] Perform the transfer -- 3 .block[.content()] Examine the handle after the transfer, and, in this case, return the content. This is optional. For an upload (PUT/POST, or FTP upload), you might check status. -- Most methods perform some action, then return the same handle, so you can easily chain methods as in this example. --- # Shortcuts Because those basic actions are so frequent, there are some shortcuts which perform them: ``` use LibCurl::HTTP :subs; print get 'http://example.com'; ``` -- There is also a version that decodes as JSON into a data structure: ``` use LibCurl::HTTP :subs; print jget('http://example.com/something-that-returns-json')
; ``` --- # Setting options on an Easy handle * At construction: ``` my $curl = LibCurl::Easy.new(someoption => 'something'); ``` -- * Explicitly with `.setopt()` ``` $curl.setopt(someoption => 'something'); ``` -- * Directly with `FALLBACK` method: ``` $curl.someoption('something'); ``` -- You can shortcut with `:someoption` or `:!someoption` for Boolean options. These all return the handle, so you can chain them. For the most part, these are identical to `libcurl` options `CURLOPT_SOMETHING`, just remove the `CURLOPT_` and lowercase. --- # Options: [CAinfo](https://curl.haxx.se/libcurl/c/CURLOPT_CAINFO.html) [CApath](https://curl.haxx.se/libcurl/c/CURLOPT_CAPATH.html) [URL](https://curl.haxx.se/libcurl/c/CURLOPT_URL.html) [accepttimeout-ms](https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPTTIMEOUT_MS.html) [accept-encoding](https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html) [address-scope](https://curl.haxx.se/libcurl/c/CURLOPT_ADDRESS_SCOPE.html) [append](https://curl.haxx.se/libcurl/c/CURLOPT_APPEND.html) [autoreferer](https://curl.haxx.se/libcurl/c/CURLOPT_AUTOREFERER.html) [buffersize](https://curl.haxx.se/libcurl/c/CURLOPT_BUFFERSIZE.html) [certinfo](https://curl.haxx.se/libcurl/c/CURLOPT_CERTINFO.html) [cookie](https://curl.haxx.se/libcurl/c/CURLOPT_COOKIE.html) [cookiefile](https://curl.haxx.se/libcurl/c/CURLOPT_COOKIEFILE.html) [cookiejar](https://curl.haxx.se/libcurl/c/CURLOPT_COOKIEJAR.html) [cookielist](https://curl.haxx.se/libcurl/c/CURLOPT_COOKIELIST.html) [customrequest](https://curl.haxx.se/libcurl/c/CURLOPT_CUSTOMREQUEST.html) [dirlistonly](https://curl.haxx.se/libcurl/c/CURLOPT_DIRLISTONLY.html) [failonerror](https://curl.haxx.se/libcurl/c/CURLOPT_FAILONERROR.html) [followlocation](https://curl.haxx.se/libcurl/c/CURLOPT_FOLLOWLOCATION.html) [forbid-reuse](https://curl.haxx.se/libcurl/c/CURLOPT_FORBID_REUSE.html) [fresh-connect](https://curl.haxx.se/libcurl/c/CURLOPT_FRESH_CONNECT.html) [ftp-skip-pasv-ip](https://curl.haxx.se/libcurl/c/CURLOPT_FTP_SKIP_PASV_IP.html) [ftp-use-eprt](https://curl.haxx.se/libcurl/c/CURLOPT_FTP_USE_EPRT.html) [ftp-use-epsv](https://curl.haxx.se/libcurl/c/CURLOPT_FTP_USE_EPSV.html) [ftpport](https://curl.haxx.se/libcurl/c/CURLOPT_FTPPORT.html) [header](https://curl.haxx.se/libcurl/c/CURLOPT_HEADER.html) [http-version](https://curl.haxx.se/libcurl/c/CURLOPT_HTTP_VERSION.html) [httpauth](https://curl.haxx.se/libcurl/c/CURLOPT_HTTPAUTH.html) [httpget](https://curl.haxx.se/libcurl/c/CURLOPT_HTTPGET.html) [httpproxytunnel](https://curl.haxx.se/libcurl/c/CURLOPT_HTTPPROXYTUNNEL.html) [infilesize](https://curl.haxx.se/libcurl/c/CURLOPT_INFILESIZE_LARGE.html) [low-speed-limit](https://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html) [low-speed-time](https://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html) [maxconnects](https://curl.haxx.se/libcurl/c/CURLOPT_MAXCONNECTS.html) [maxfilesize](https://curl.haxx.se/libcurl/c/CURLOPT_MAXFILESIZE_LARGE.html) [maxredirs](https://curl.haxx.se/libcurl/c/CURLOPT_MAXREDIRS.html) [max-send-speed](https://curl.haxx.se/libcurl/c/CURLOPT_MAX_SEND_SPEED_LARGE.html) [max-recv-speed](https://curl.haxx.se/libcurl/c/CURLOPT_MAX_RECV_SPEED_LARGE.html) [netrc](https://curl.haxx.se/libcurl/c/CURLOPT_NETRC.html) [nobody](https://curl.haxx.se/libcurl/c/CURLOPT_NOBODY.html) [noprogress](https://curl.haxx.se/libcurl/c/CURLOPT_NOPROGRESS.html) [nosignal](https://curl.haxx.se/libcurl/c/CURLOPT_NOSIGNAL.html) [password](https://curl.haxx.se/libcurl/c/CURLOPT_PASSWORD.html) [post](https://curl.haxx.se/libcurl/c/CURLOPT_POST.html) [postfields](https://curl.haxx.se/libcurl/c/CURLOPT_COPYPOSTFIELDS.html) [postfieldsize](https://curl.haxx.se/libcurl/c/CURLOPT_POSTFIELDSIZE_LARGE.html) [protocols](https://curl.haxx.se/libcurl/c/CURLOPT_PROTOCOLS.html) [proxy](https://curl.haxx.se/libcurl/c/CURLOPT_PROXY.html) [proxyauth](https://curl.haxx.se/libcurl/c/CURLOPT_PROXYAUTH.html) [proxyport](https://curl.haxx.se/libcurl/c/CURLOPT_PROXYPORT.html) [proxytype](https://curl.haxx.se/libcurl/c/CURLOPT_PROXYTYPE.html) [proxyuserpwd](https://curl.haxx.se/libcurl/c/CURLOPT_PROXYUSERPWD.html) [range](https://curl.haxx.se/libcurl/c/CURLOPT_RANGE.html) [redir-protocols](https://curl.haxx.se/libcurl/c/CURLOPT_REDIR_PROTOCOLS.html) [referer](https://curl.haxx.se/libcurl/c/CURLOPT_REFERER.html) [resume-from](https://curl.haxx.se/libcurl/c/CURLOPT_RESUME_FROM_LARGE.html) [ssl-verifyhost](https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html) [ssl-verifypeer](https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html) [timecondition](https://curl.haxx.se/libcurl/c/CURLOPT_TIMECONDITION.html) [timeout](https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT.html) [timeout-ms](https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT_MS.html) [timevalue](https://curl.haxx.se/libcurl/c/CURLOPT_TIMEVALUE.html) [unrestricted-auth](https://curl.haxx.se/libcurl/c/CURLOPT_UNRESTRICTED_AUTH.html) [use-ssl](https://curl.haxx.se/libcurl/c/CURLOPT_USE_SSL.html) [useragent](https://curl.haxx.se/libcurl/c/CURLOPT_USERAGENT.html) [username](https://curl.haxx.se/libcurl/c/CURLOPT_USERNAME.html) [userpwd](https://curl.haxx.se/libcurl/c/CURLOPT_USERPWD.html) [verbose](https://curl.haxx.se/libcurl/c/CURLOPT_VERBOSE.html) [wildcardmatch](https://curl.haxx.se/libcurl/c/CURLOPT_WILDCARDMATCH.html) --- # Options Some fun ones: ``` my $curl = LibCurl::Easy.new(`:verbose`, URL => "http://..."); ``` When debugging always set `:verbose` and you'll get a dump of the actual HTTP transaction, very handy. -- **`:followlocation`** = automatically follow a `Location:` header as part of an HTTP 3xx response. -- **`:nobody`** = Doesn't transfer the body of the request, for HTTP this is equivalent to a `HEAD` request. -- **`proxy`** = Set a proxy to use for the transfer. -- **`private`** = Store any Perl object you want access to later. --- # Header Options There are a few special options that set headers ([useragent](https://curl.haxx.se/libcurl/c/CURLOPT_USERAGENT.html), [referer](https://curl.haxx.se/libcurl/c/CURLOPT_REFERER.html), [cookie](https://curl.haxx.se/libcurl/c/CURLOPT_COOKIE.html)), there are some extra options for headers: `Content-MD5`, `Content-Type`, `Content-Length`, `Host`, `Accept`, `Expect`, `Transfer-Encoding`. ``` $curl.Host('somewhere.com'); # or $curl.setopt(Host => 'somewhere.com') $curl.Content-MD5('...'); # or $curl.setopt(Content-MD5 => '...') ``` -- You can also add any other headers you like: ``` $curl.set-header(X-My-Header => 'something', X-something => 'foo'); ``` -- Clear all ***except*** the libcurl special headers: ``` $curl.clear-header; ``` --- # Upload/Download * .block[download => 'myfile'] * .block[upload => 'myfile'] * .block[send => 'something'] * .block[send => $mybuf] If you *don't* specify a download filename, it will stash all incoming content in `$curl.buf`. You can also access that content decoded as a `UTF-8` `Str` with `$curl.content`. You can change encoding if you want `$curl.content('utf-16')`. --- # Info * After a transfer completes (successfully or otherwise), you can access a lot of information about the transfer. Similar to options, there are several methods to get that information: ``` say $curl.getinfo('effective-url'); say $curl.getinfo('response-code'); say $curl.getinfo(
); # Hash with those keys say $curl.getinfo; # Hash of all info fields say $curl.response-code; ``` -- Fields currently defined are: [appconnect_time](https://curl.haxx.se/libcurl/c/CURLINFO_APPCONNECT_TIME.html) [certinfo](https://curl.haxx.se/libcurl/c/CURLINFO_CERTINFO.html) [condition-unmet](https://curl.haxx.se/libcurl/c/CURLINFO_CONDITION_UNMET.html) [connect-time](https://curl.haxx.se/libcurl/c/CURLINFO_CONNECT_TIME.html) [content-type](https://curl.haxx.se/libcurl/c/CURLINFO_CONTENT_TYPE.html) [cookielist](https://curl.haxx.se/libcurl/c/CURLINFO_COOKIELIST.html) [effective-url](https://curl.haxx.se/libcurl/c/CURLINFO_EFFECTIVE_URL.html) [ftp-entry-path](https://curl.haxx.se/libcurl/c/CURLINFO_FTP_ENTRY_PATH.html) [header-size](https://curl.haxx.se/libcurl/c/CURLINFO_HEADER_SIZE.html) [http-connectcode](https://curl.haxx.se/libcurl/c/CURLINFO_HTTP_CONNECTCODE.html) [httpauth-avail](https://curl.haxx.se/libcurl/c/CURLINFO_HTTPAUTH_AVAIL.html) [lastsocket](https://curl.haxx.se/libcurl/c/CURLINFO_LASTSOCKET.html) [local-ip](https://curl.haxx.se/libcurl/c/CURLINFO_LOCAL_IP.html) [local-port](https://curl.haxx.se/libcurl/c/CURLINFO_LOCAL_PORT.html) [namelookup-time](https://curl.haxx.se/libcurl/c/CURLINFO_NAMELOOKUP_TIME.html) [num-connects](https://curl.haxx.se/libcurl/c/CURLINFO_NUM_CONNECTS.html) [os-errno](https://curl.haxx.se/libcurl/c/CURLINFO_OS_ERRNO.html) [pretransfer-time](https://curl.haxx.se/libcurl/c/CURLINFO_PRETRANSFER_TIME.html) [primary-ip](https://curl.haxx.se/libcurl/c/CURLINFO_PRIMARY_IP.html) [primary-port](https://curl.haxx.se/libcurl/c/CURLINFO_PRIMARY_PORT.html) [proxyauth-avail](https://curl.haxx.se/libcurl/c/CURLINFO_PROXYAUTH_AVAIL.html) [redirect-url](https://curl.haxx.se/libcurl/c/CURLINFO_REDIRECT_URL.html) [request-size](https://curl.haxx.se/libcurl/c/CURLINFO_REQUEST_SIZE.html) [response-code](https://curl.haxx.se/libcurl/c/CURLINFO_RESPONSE_CODE.html) [rtsp-client-cseq](https://curl.haxx.se/libcurl/c/CURLINFO_RTSP_CLIENT_CSEQ.html) [rtsp-cseq-recv](https://curl.haxx.se/libcurl/c/CURLINFO_RTSP_CSEQ_RECV.html) [rtsp-server-cseq](https://curl.haxx.se/libcurl/c/CURLINFO_RTSP_SERVER_CSEQ.html) [rtsp-session-id](https://curl.haxx.se/libcurl/c/CURLINFO_RTSP_SESSION_ID.html) [size-download](https://curl.haxx.se/libcurl/c/CURLINFO_SIZE_DOWNLOAD.html) [size-upload](https://curl.haxx.se/libcurl/c/CURLINFO_SIZE_UPLOAD.html) [speed-download](https://curl.haxx.se/libcurl/c/CURLINFO_SPEED_DOWNLOAD.html) [speed-upload](https://curl.haxx.se/libcurl/c/CURLINFO_SPEED_UPLOAD.html) [ssl-engines](https://curl.haxx.se/libcurl/c/CURLINFO_SSL_ENGINES.html) [total-time](https://curl.haxx.se/libcurl/c/CURLINFO_TOTAL_TIME.html) --- # Received headers After a transfer, you can also check out the headers returned by the server: ``` say $curl.get-header('Content-Length'); say $curl.receiveheaders
; # Hash of all headers say $curl.Content-Length; ``` --- # Errors * Most real errors will throw an `X::LibCurl` exception * The `failonerror` option will force an exception on an HTTP code ≥ 400 (not usually an error from the `LibCurl` perspective). * You can check response code with $curl.response-code. --- # Debugging * `:verbose` option will just dump some good stuff to `STDOUT` * Create a debug subroutine: ``` sub debug(LibCurl::Easy $easy, CURL-INFO-TYPE $type, Buf $buf) {...} $curl.setopt(debugfunction => &debug); ``` Gets called periodically: - CURLINFO_TEXT - CURLINFO_HEADER_IN - CURLINFO_HEADER_OUT - CURLINFO_DATA_IN - CURLINFO_DATA_OUT - CURLINFO_SSL_DATA_IN - CURLINFO_SSL_DATA_OUT --- # Transfer progress You can enable the simple `curl` progress printing by `:!noprogress`. (Yes, this seems backwards..) ``` % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 364M 100 364M 0 0 104M 0 0:00:03 0:00:03 --:--:-- 104M ``` You can also install your own progress function: ``` sub xferinfo(LibCurl::Easy $easy, $dltotal, $dlnow, $ultotal, $ulnow) {...} $curl.setopt(xferinfofunction => &xferinfo); ``` --- # Multi-part Form POSTing ``` my $curl = LibCurl::Easy.new(URL => 'http://...'); # normal field $curl.formadd(name => 'fieldname', contents => 'something'); # upload a file from disk, give optional filename or contenttype $curl.formadd(name => 'fieldname', file => 'afile.txt', filename => 'alternate.name.txt', contenttype => 'image/jpeg'); # Send a Blob of contents, but as a file with a filename $curl.formadd(name => 'fieldname', buffer => 'some.file.name.txt', bufferptr => "something".encode); $curl.perform; ``` --- # Multi Interface Construct an `Easy` handle for each desired transfer, then `perform` them all simultaneously. ``` use LibCurl::Easy; use LibCurl::Multi; my $curl1 = LibCurl::Easy.new(:verbose, :followlocation, URL => 'http://example.com', download => './myfile1.html'); my $curl2 = LibCurl::Easy.new(:verbose, :followlocation, URL => 'http://example.com', download => './myfile2.html'); LibCurl:Multi.new.add-handle($curl1, $curl2).perform; say $curl1.statusline; say $curl2.statusline; ``` --- # Multi Interface Async ``` use LibCurl::Easy; use LibCurl::Multi; my $curl1 = LibCurl::Easy.new(:followlocation, URL => 'http://example.com', download => 'myfile1.html'); my $curl2 = LibCurl::Easy.new(:followlocation, URL => 'http://example.com', download => 'myfile2.html'); sub callback(LibCurl::Easy $easy, Exception $e) { die $e if $e; say $easy.response-code; say $easy.statusline; } my $multi = LibCurl::Multi.new(callback => &callback); $multi.add-handle($curl1, $curl2); $multi.perform; ``` --- # Conclusion * Perl 6 implementation still in development, please try it out and let me know what you like/don't like. * Multi interface is in particular could be extended to support a more perl6-ish interface. -- ##.center[Thank You!] .center[*Slides produced with [remark](https://remarkjs.com)*]