How to use a Proxy with C# HttpClient
In this article, you will learn about how to use C#'s HttpClient
library behind a proxy. HttpClient
comes by default with C# and provides asynchronous requests out of the box which makes it a very attractive option as an HTTP client. Developers such as yourself prefer to use proxies while making web requests to retain anonymity and to prevent their primary IP from being blocked due to excessing web scraping. Whatever your reasons might be, you will be well equipped to use authenticated and unauthenticated proxies with HttpClient
after going through this tutorial.
Creating a new C# project
Let's start by creating a new C# project. I am using Visual Studio for Mac and creating a new console application using .net 6.
![New Project Wizard](png/new-project-wizard.png)
![.Net version Wizard](png/dot-net-wizard.png)
Once the new project is created, you will be greeted by the new console application template in the editor:
// See https://aka.ms/new-console-template for more information
Console.WriteLine("Hello, World!");
If you haven't used top-level statements like this before and are wondering where the Main
declaration is then it might be a good idea to read more about this new template in the official docs. We will be using the latest features in C# and .Net in this tutorial.
Making a basic HTTP request
You can create a basic HTTP request using HttpClient
like this:
using var client = new HttpClient();
var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);
The first line instantiates an HttpClient
object. The next line uses the GetStringAsync
method to send a GET request to https://api.ipify.org/
, and the last line prints the result.
Save the file and run the code. You should see your public IP in the terminal like below:
![Visual Studio Mac OS version](png/vs-mac-os.png)
Using a proxy with HttpClient
You can use a basic, unauthenticated, proxy with HttpClient
like this:
using System.Net;
var proxy = new WebProxy
{
Address = new Uri($"http://172.104.241.29:8081"),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
};
// Create a client handler that uses the proxy
var httpClientHandler = new HttpClientHandler
{
Proxy = proxy,
};
// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);
You need to create a custom WebProxy
object with the proxy details, pass that to a custom HttpClientHandler
, and finally pass that handler to HttpClient
. I also disable SSL verification as some free proxies cause an SSL verification error if you try to connect to them securely.
Multiple sources list free public proxies. I took the above proxy from a similar list hosted by Free Proxy List. Running the code above should result in the proxy host being printed in the terminal:
172.104.241.29
Using an authenticated proxy with HttpClient
You can use authenticated proxies with HttpClient
as well that require a user/pass combination. Just create a new NetworkCredential
Object and pass that to WebProxy
. HttpClient
will automatically use these credentials when connecting to the proxy server.
using System.Net;
var proxy = new WebProxy
{
Address = new Uri($"http://172.104.241.29:8081"),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
// Proxy credentials
Credentials = new NetworkCredential(
userName: proxyUserName,
password: proxyPassword)
};
// Create a client handler that uses the proxy
var httpClientHandler = new HttpClientHandler
{
Proxy = proxy,
};
// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);
If you run the above code (after replacing the placeholder proxy details with working proxy details), it should print a similar output as before:
172.104.241.29
Rotating proxies with requests
It is always a good idea to rotate your proxies with successive requests. This decreases the likelihood of any one proxy being blacklisted. You can do this by simply defining a list of proxies and using a random proxy with each new request.
using System.Net;
List<Dictionary<string, string>> proxy_list = new List<Dictionary<string, string>> {
new Dictionary<string, string> {
{"protocol", "http"},
{"host", "204.185.204.64"},
{"port", "8080"},
},
//...
new Dictionary<string, string> {
{"protocol", "http"},
{"host", "132.129.121.148"},
{"port", "8080"},
},
new Dictionary<string, string> {
{"protocol", "http"},
{"host", "154.129.98.156"},
{"port", "8080"},
},
new Dictionary<string, string> {
{"protocol", "http"},
{"host", "211.129.132.150"},
{"port", "8080"},
},
new Dictionary<string, string> {
{"protocol", "http"},
{"host", "164.129.114.111"},
{"port", "8080"},
}
};
// Generate a random number between 0 and 4 (inclusive)
Random rnd = new Random();
var randomIndex = rnd.Next(proxy_list.Count);
var proxy = new WebProxy
{
Address = new Uri($"{proxy_list[randomIndex]["protocol"]}://{proxy_list[randomIndex]["host"]}:{proxy_list[randomIndex]["port"]}"),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
};
// Now create a client handler that uses the proxy
var httpClientHandler = new HttpClientHandler
{
Proxy = proxy,
};
// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);
You can alter the code above to meet your specific needs. On each run, HttpClient
will use a random proxy from the proxy_list
.
Using ScrapingBee proxies
If you have done some decent amount of web scraping then you know how difficult it is to get a list of premium unblocked proxies. ScrapingBee helps you get around that problem by allowing you to use their premium IPs for web scraping. While the most powerful features of the platform are available only in the paid version, you do get access to 1000 free credits for making proxied requests. They only charge you credits when they return a successful response.
Let's look at a quick example of how you can use ScrapingBee. Go to the ScrapingBee website and sign up for an account:
![Signup scrapingbee](png/scrapingbee-signup-4.png)
After successful signup, you will be greeted with the default dashboard. Copy your API key from this page and start modifying the code in the app.js
file:
![Copy API key](png/scrapingbee-apikey-3.png)
You can make a request via ScrapingBee using HttpClient
like this:
using System.Net;
var proxy = new WebProxy
{
Address = new Uri($"http://proxy.scrapingbee.com:8886"),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
// Proxy credentials
Credentials = new NetworkCredential(
userName: "YOUR_API_KEY",
password: "render_js=False&premium_proxy=True")
};
// Create a client handler that uses the proxy
var httpClientHandler = new HttpClientHandler
{
Proxy = proxy,
};
// Disable SSL verification
httpClientHandler.ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;
// Finally, create the HTTP client object
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
var result = await client.GetStringAsync("https://api.ipify.org/");
Console.WriteLine(result);
Note: Don't forget to replace YOUR_API_KEY
with your API key from the ScrapingBee dashboard.
The password field is used to configure the various parameters supported by the ScrapingBee API. In this example, I am disabling JS rendering and asking ScrapingBee to use premium proxies.
Running the above code should print a random IP address in the terminal. Successive requests will print different IP addresses as ScrapingBee automatically rotates IP addresses so you don't have to do that on your own. ScrapingBee's proxy mode is so economical and easy to use that there is no reason not to give it a try!
Conclusion
In this article, you learned how to use a proxy with HttpClient
in C#. You saw how you can use an authenticated proxy and how you can write some basic C# code to use a random proxy from a list. You also learned that our web scraping API can take a lot of pain out of this process of using proxies and abstracts away most of the complexity of finding reliable, unblocked proxies, and rotating them.
Even though Python is more famous for web scraping, C# isn't too far behind either. There are multiple libraries like the HTML Agility Pack
library that provide support for parsing and extracting data from HTML documents. Give web scraping in C# a go and let us know about your experience!
Additional resources
![image description](png/yasoob.png)
Yasoob is a renowned author, blogger and a tech speaker. He has authored the Intermediate Python and Practical Python Projects books ad writes regularly. He is currently working on Azure at Microsoft.