cURL on Linux: what you should know before you start
The free cURL free software is one of the oldest and most popular open source projects. The program, whose name comes from client URL , is used to send data on computer networks. The open license grants the right to use the program for any purpose. Today, cURL is used on a multitude of devices..
Index
- What is cURL?
- How does cURL work?
- How is libcurl used?
- How is cURL used?
- Practical examples of the cURL command on Linux
- First test with cURL
- Download files with cURL
- Access server data with cURL
- Access a file with cURL and save it to your local hard drive
- Resume download with cURL if it was interrupted
- Communicate with the server using cURL
- Check if a server is accessible with cURL
- Distribute header with cURL
- Analyze redirect chains with cURL
- Move data to server with cURL
- Read cookies with cURL
- General options
- Show additional information
- Enter username and password with cURL
- Use a proxy with cURL
What is cURL?
The cURL software consists of two components. The libcurl program library is the backbone of data transfer and supports the following protocols:
- DICT
- FILE
- FTP
- FTPS
- GOPHER
- HTTP
- HTTPS
- IMAP
- IMAPS
- LDAP
- LDAPS
- POP3
- POP3S
- RTMP
- RTSP
- SCP
- SFTP
- SMB
- SMBS
- SMTP
- SMTPS
- TELNET
- TFTP
For its part, the cURL command line program acts as a text-based interface and interacts with libcurl via the command line.
The program is an important tool for web development , because it allows the developer to communicate directly with the server instead of having to access it from a browser. The scripts based on cURL commands (in English: cURL commands ) are used to automate processes, debugging and testing..
How does cURL work?
The two components of cURL work in different ways.
How is libcurl used?
The libcurl program library includes functions for sending data over computer networks. There are language bindings for dozens of programming languages. Therefore, the libcurl functions are available to a large number of programs that communicate with servers..
How is cURL used?
The command line program cURL is used for web development. The easiest method is to enter the cURL commands on the command line. With the right knowledge you can test and debug servers and APIs.
Instead of entering the commands manually on the command line, they can be embedded in scripts . In this way, complex operations can be standardized and automated, such as uploading or downloading data, as well as scheduling form completion and copying entire websites.
The schema of a cURL command usually has the following structure:
# Esquema general de un cURL command curl [opciones] <url>
In the examples below we use the following form:
url="www.example.com" curl [opciones] "$url"
We have externalized the actual URL to a variable. In this way, with the options those that determine the operation of the cURL execution.
Practical examples of the cURL command on Linux
In order to use the following cURL examples, you will need:
- A computer with Linux or a UNIX-like operating system - also macOS.
- Access to the console or the command line.
- A code editor or plain text editor to compose the commands.
Note
Use only a code or plain text editor to compose your commands. Under no circumstances should word processors such as Word, OpenOffice, or LibreOffice be used to prepare text from the command line.
First test with cURL
First, make sure that cURL is installed on the system. To do this, open the command line and enter the following command (sometimes you need to press the return key to run the code).
# Comprobar que cURL está instalado curl --version
If you see text that begins with curl and is followed by a version number, cURL is installed. If cURL is not installed, follow the instructions in? Everything curl? to install the program.
advice
Use the curl --help and curl --manual commands to learn more about the command.
Also, you must open a new document in a code editor. You can copy the commands there to prepare them before entering them on the command line. When the command is ready, copy it to the command line and run it.
Note
Commands used on the command line have many repercussions. Basically a misspelled command could paralyze the entire system. This is why it is so important not to copy commands directly from the Internet and just run them on the local command line. Instead, it is a good idea to have an empty window open in the code editor. In this box, you can paste the commands: an intermediate step that will force you to carefully check each command and adapt it if necessary before executing it.
Access server data with cURL
In principle, with cURL you can access any URL you know. In this context, the verb curl ( curling ) describes the process of addressing a URL with curl. Try the following example:
# Dirígete al siguiente sitio con cURL site="www.google.com" curl "$site"
When you run the code this way, the result is garbled text. This is because the Google home page is returned in HTML in response to the cURL access. The HTML source is displayed on the command line without formatting. For its correct representation a browser is necessary. It is generally more useful to save the file locally.
advice
Enter the clear command on the command line to clear the screen. This will erase unnecessary data from the command line.
Access a file with cURL and save it to your local hard drive
We downloaded a logo from Wikipedia in English. The -O option ( or uppercase and not zero) directs cURL to use the name of the file at the end of the URL. With this option, cURL will save the downloaded file to the local hard drive with the same name.
However, how does it behave with a URL that does not include a filename? Try the following code:
# Página de inicio de Google homepage="www.google.com" # Abrir la página de inicio con opción -O curl "$homepage" -O
As you can see, an error appears because the URL of the Google home page does not contain a file name. In this case, you must use the -o ( or lowercase) option to give it a filename.
# Página de inicio de Google homepage="www.google.com" # Nombre del archivo que se escribirá nombre="homepage-google.html" # Abrir página de inicio y guardarla localmente con el nombre seleccionado curl "$homepage" -o "$name"
Resume download with cURL if it was interrupted
You may have suffered from it once - you've been waiting for hours for a large file to download when all of a sudden the download stops and you have to start over from the beginning. In this case, the -C- option can help you:
Note
Although it is true that the -C - option includes a space after the C, it is better read and it is easier to remember the -C- equivalence that we have used in the example.
With cURL you can also interrupt a download manually if necessary . With large files it can be an advantage; for example, when you have to leave the house and take your laptop with you.
advice
You can cancel the current execution of cURL. To cancel, press Ctrl + C repeatedly if necessary.
Communicate with the server using cURL
The cURL download function is similar to the popular wget command line tool. However, cURL is not specialized download software, it is designed for general communication over networks, offering even more benefits.
Check if a server is accessible with cURL
With cURL you can check if a server is accessible. In this sense, curl works similar to that of the ping command. However, due to the protocols and options available, cURL can be used more flexibly. Also, ping works on the Internet layer, while cURL works on the application layer. That means that ping checks if the machine is on the network. Instead, the following cURL command checks if and how a server reacts :
# Verificar si un servidor web está disponible server="google.com" curl -I "$server"
If status information is displayed after running the code , the server will be available.
Distribute header with cURL
With each HTTP request, in addition to transferring the document itself, different metadata is also exchanged . This data, mentioned in the HTTP header, describes both the document and the status of the HTTP request. Surely you have ever run into him ? Error 404 Not Found ? . It is precisely about metadata: the requested document has not been found.
Take a look at the layout of the URL header? Google.com ?. To do this, use curl with the --head option :
# Distribuir encabezado url="google.com" curl --head "$url"
It should display, among other messages:? 301 Moved Permanently ?. This is the HTTP status code for permanent redirection. What you see is the redirection from [google.com] (without? Www?) To [www.google.com].
advice
Instead of the --head option , you can also use the equivalent spelling -I ( uppercase i ). To remember, think you access the i nformation a URL and not its content.
Analyze redirect chains with cURL
The HTTP redirect (in English: redirects ) can be activated in series. In this context, the term redirect chain (or redirect chain) is used. Imagine a website whose home page is [https://www.example.com/]. Well, if we go to this site from [http://example.com], the following redirects will occur:
A concatenation of redirects leads to longer and unnecessary load times and should be avoided. Unfortunately, this is a difficult problem to analyze. When the page is loaded in the browser, redirects occur without the user hardly noticing it. In this case, the --location option may help , which tells cURL to follow through to the last redirect. Here we use the --head option because we are not interested in the content of the pages.
# Comprobar redireccionamientos url="google.com" curl --location --head "$url"
advice
Instead of the ? Location option , you can also use the equivalent spelling -L ( uppercase l ).
Move data to server with cURL
In addition to accessing data, with cURL you can also send data to a server . This is very practical, for example, if we want to automate the automatic completion of an online form. In addition to the POST request method, cURL also supports GET.
Since sending data is a more complex task than accessing it, we can only sketch a rough example. For more detailed information, check out the 'Everything curl' tips.
# Transferir datos al servidor url="example.com" # Introduce los datos como pares de valores clave separados por ?&? datos="nombre=Pedro&apellido=González&edad=42" # Esta ejecución cURL transfiere los datos mediante POST curl --data "$datos" "$url" # Alternativamente, indica a cURL que transfiera los datos mediante GET curl --data "$datos" "$url" --get
Read cookies with cURL
HTTP cookies are small text files that are stored locally on the user's device when visiting most websites. Within your browser you can see and delete the cookies of any page you have visited. However, if you want to do tests, this can be too cumbersome. With curl and the --cookie-jar option ( cookie box in Spanish), you can also directly access the cookies .
# Leer cookies url="www.google.com" cookies="cookies.txt" curl --head --cookie-jar "$cookies" "$url" # Mostrar cookies con el comando cat cat "$cookies"
advice
With the --cookie option you can also tell cURL to send cookies .
General options
Some of the available cURL options can be combined with the ones we have already mentioned.
Show additional information
In some cases, the information represented in the execution of a cURL command is not sufficient. You can then use the --verbose option . A cURL command that includes that option will give a greater amount of information .
# Mostrar información adicional url="www.google.com" curl --verbose -I "$url"
Enter username and password with cURL
Some URLs are encrypted with HTTP authentication to prevent unauthorized access. What happens if you try to cURL this type of URL? If you do not enter the username and password, the HTTP 401 error will appear on the screen. In that case, you should use the following scheme:
# Introducir usuario y contraseña con cURL # URL protegido por contraseña url="www.example.com/secure/" # Usuario usuario="" # Contraseña contraseña="" curl --user "${nutzer}:${passwort}" "$url"
advice
You can also use curl to access data from an FTP server. Use the --user option to specify a username and password.
Use a proxy with cURL
A proxy is an intermediate server. Using a proxy can be useful for a number of reasons. For example, the proxy makes it easier to meet certain security and performance requirements.
When you connect to the internet through a proxy , you must inform cURL. To do this, you must use the --proxy option :
# Utilizar un proxy con cURL url="www.google.com" proxy="proxy.example.com" port="8080" curl --proxy "${proxy}:${port}" "$url"
Note
If you need to specify the username and password on the proxy server , use the --proxy-user option . In this case, enter the username and password following the format? User: password ?.