I want LWP::UserAgent to log the following information:

requested uri, time it took to fetch, content length and header length.

The following code works unless somebody uses :content_cb, in this case $content_length is zero, because the content is not included in the HTTP::Response object. Has somebody an idea to get the content-length anyway?

Here is my code.

package LWP::UserAgent::Trace;

use Time::HiRes;
use Data::Dumper;
use base qw/LWP::UserAgent/;

sub send_request {
  my($self, $request, $arg, $size) = @_;

  my $t0 = [Time::HiRes::gettimeofday];
  my $response = $self->SUPER::send_request($request,$arg,$size);
  my $elapsed = Time::HiRes::tv_interval($t0); 

  my $content_length = length($response->content);
  my $header_length = length($response->headers_as_string);
  print STDERR sprintf "%s %f %d %d %d\n",
    $request->uri, $elapsed, $content_length,
    $header_length, $content_length+$header_length;

  return $response;
}

1;

Re: LWP::UserAgent hacking

Thelonius

created: 2006-03-04 12:10:59

perlmonks

• reply
• node
• thread
• author

• node
• author
• thread

prlmnks

• node
• thread
• author

As long as LWP doesn't support chunked responses (I think it still doesn't), this will work:

my $content_length = $response->content_length;

Re: LWP::UserAgent hacking

Thelonius

created: 2006-03-04 14:57:13

perlmonks

• reply
• node
• thread
• author

• node
• author
• thread

prlmnks

• node
• thread
• author

Here's a method that will work for chunked responses, too:

sub send_request {
  my($self, $request, $arg, $size) = @_;

  my $content_length = 0;
  if (ref $arg && ref $arg eq "CODE") {
    my $original_arg = $arg;
    $arg = sub { 
             $content_length += length($_[0]); 
	     return &$original_arg(@_);
	  };
  }

  my $t0 = [Time::HiRes::gettimeofday];
  my $response = $self->SUPER::send_request($request,$arg,$size);
  my $elapsed = Time::HiRes::tv_interval($t0); 

  $content_length = length($response->content) unless $content_length;
  my $header_length = length($response->headers_as_string);
  print STDERR sprintf "%s %f %d %d %d\n",
    $request->uri, $elapsed, $content_length,
    $header_length, $content_length+$header_length;

  return $response;
}