How to use LWP::UserAgent with custom User-Agent string?

The LWP::UserAgent module in Perl is a popular way to make HTTP requests. By default, it sends a standard User-Agent string like libwww-perl/X.X, but sometimes you want to customize this string to mimic a browser or identify your application specifically. This is straightforward by setting the agent attribute on your LWP::UserAgent object.

What is a User-Agent string?

The User-Agent header tells the web server which client software is making the HTTP request. It can include information about the browser, OS, or bot identity. Setting a custom User-Agent can help:

• Bypass basic bot filters
• Provide your app's name/version for identification
• Test how servers respond to different clients

How to set a custom User-Agent string

When you create an LWP::UserAgent object, use the agent method to specify your custom string. It will be sent in the User-Agent HTTP header on all subsequent requests.

Example: Set a custom User-Agent and make a GET request

use strict;
use warnings;
use LWP::UserAgent;

# Create new UserAgent object
my $ua = LWP::UserAgent->new;

# Set a custom User-Agent string
$ua->agent('MyCustomAgent/1.0');

# Perform HTTP GET request
my $response = $ua->get('http://httpbin.org/user-agent');

if ($response->is_success) {
    print "Request succeeded!\n";
    print "Response content:\n";
    print $response->decoded_content;
} else {
    print "Error: " . $response->status_line . "\n";
}

This example:

• Imports LWP::UserAgent (core module, no extra install needed)
• Sets user agent to MyCustomAgent/1.0
• Makes a GET request to httpbin.org/user-agent, which echoes back the User-Agent header it received
• Prints the response showing your custom User-Agent

Important notes and tips

• Perl versions: LWP::UserAgent works across Perl 5.8+. No special version requirements for setting User-Agent.
• Sigils and context: The $ua scalar holds the object. Calling $ua->agent sets or gets the user agent string. Scalars are used for objects.
• TMTOWTDI (There's More Than One Way To Do It): You can also specify a user agent string at construction: my $ua = LWP::UserAgent->new(agent => 'MyAgent/1.0');
• Common gotchas:
  • Do not confuse the agent method with default_headers—they serve different purposes.
  • Remember to check your $response for success to avoid unintentional errors.
• Security: Don’t use misleading User-Agent strings while scraping or automating; some websites may block or ban if policies are violated.

Using LWP::UserAgent with a custom User-Agent header is simple and important for controlling how servers see your requests. This approach applies to any HTTP verb: GET, POST, PUT, etc.