Out of memory while using LWP::UserAgent
I am using LWP::UserAgent to retreive files from a FTP site. Everything appears to work fine until I attempt to get a file that is supposedly 65M in size. At this point the script dies with error:
"Out of memory during "large" request for 134221824 bytes, total sbrk() is 271669258 bytes at .."
The code I am using is below. I know the physical memory is available. Any thoughts on how to get around this problem? Thanks in advance.
my $ua = LWP::UserAgent->new();
my $response = $ua->get("$url");
if ($response->is_success) {
my $content = $response->content;
open (OUTPUT, ">out.txt") or die "Cannot open the output file: $!";
print OUTPUT "$content";
close (OUTPUT) or die "Cannot close the output file: $!";
}
else {
die $response->status_line;
}
Re: Out of memory while using LWP::UserAgent
created: 2006-01-04 16:28:40
See the section "Large Documents" in perldoc lwpcook.
Re: Out of memory while using LWP::UserAgent
created: 2006-01-04 17:24:34
Maybe Net::FTP would better suit your needs?
Re: Out of memory while using LWP::UserAgent
created: 2006-01-05 07:22:11
Search for content_cb and content_file in perldoc LWP::UserAgent. This shows how you can write the data to a file or process it with a callback instead of writing to a perl scalar.