How to Remove Junk Characters (Rajeshk)

How to Remove Junk Characters

Rajeshk

created: 2006-01-05 04:46:09

perlmonks

node
author

prlmnks

node
author

Hi Monks,
I have problem while downloading HTML files using LWP::UserAgent.
There are some Junks Characters found in downloaded files.
Is any way to download the file without junks.
Note:I am using in Windows OS. Download the webpage to see junk Characters 'http://www.whitecase.com/attorneys/detail.aspx?attorney=1148';
Here is my try:

use LWP::UserAgent;
my $ua = new LWP::UserAgent;
$ua->proxy(['http']=> 'http://00.00.0.00:0000');
my  $url = 'http://www.whitecase.com/attorneys/detail.aspx?attorney=1148';
# Create a request

my $req = HTTP::Request->new('GET' => $url);
$req->proxy_authorization_basic("xxxxx", "xxxxx");

my $res = $ua->request($req);
if ($res->is_success) {
	my $file_cnt = $res->content;
	print "$file_cnt";
	open WOUT, ">out.html" or die "Can't open File: out.html";
	print WOUT $file_cnt;
	close WOUT;
}
else {
	print "Download Error\n";
}

Thanks & Regards,
Rajesh.K

Edit: [g0n] - replaced pre tags with code tags

Edit: [planetscape] - replaced [ ] with [ ]

Re: How to Remove Junk Characters

wfsp

created: 2006-01-05 05:18:05

perlmonks

prlmnks

Could you show us what it looks like?

Post a sample to give us some idea.

Update:
Try this

my $file_cnt = $res->content;
$file_cnt =~ s/\r//g;

Re^2: How to Remove Junk Characters

Rajeshk

created: 2006-01-05 07:20:50

perlmonks

prlmnks

Hi wfsp,

I tried your code. $file_cnt =~ s/\r//g;
It's not working.

Thanks,
Rajesh.K

Re: How to Remove Junk Characters

zentara

created: 2006-01-05 07:31:29

perlmonks

prlmnks

I took out your $ua->proxy line and your code runs fine. The out.html has no corruption. I'm on linux using Mozilla.

I'm not really a human, but I play one on earth. flash japh

Re^2: How to Remove Junk Characters

wfsp

created: 2006-01-05 08:51:01

perlmonks

prlmnks

He may mean this

....Dr. BÃ¶rries.....

It should, of course, be "Börries"
I tried

binmode(STDOUT, ':utf8');

with no success. Any idea what's happening?

wfsp

Re^3: How to Remove Junk Characters

holli

created: 2006-01-05 12:58:41

perlmonks

prlmnks

This works:

use Unicode::String qw(utf8);
#....
print utf8($file_cnt);

holli, /regexed monk/

Re: How to Remove Junk Characters

abcde

created: 2006-01-05 08:33:10

perlmonks

prlmnks

I am not sure what you mean by "junk" characters. May you post an example of what you mean?

I took out the proxy lines and ran the code; the file downloaded without any errors. However, I think you are referring to accented characters such as ö in the source - Use HTML::Entities if you want to encode them into the proper ö format.

But please post an example so we can be sure of what you want.

~abseed

Re^2: How to Remove Junk Characters

Rajeshk

created: 2006-01-06 00:47:58

perlmonks

prlmnks

Hi Monks,

Here are some sample junk characters 

Downloaded files Input  -- Original Output
===========================================         

1. jury trial.Â  For his   -- jury trial.  For his  
2. BÃ¶rries Ahrens  	-- Börries Ahrens  
3. Aldejohannâs main     -- Mr. Aldejohanns 
4. University of MÃ¼nster -- University of Münster
5. the â¬625 million senior and â¬130  -- 625 million senior and 130 
6. acquisition of a propertiesâ  -- acquisition of a properties
7. Westfield College â University  -- Westfield College  University 
8. TelÃ©fonos  -- Teléfonos
9.(CelumÃ³vil S -- (Celumóvil S
10. Dr. jur., 1990, with a dissertation on âDie UnabhÃ¤ngigkeit des genossenschaftlichen PrÃ¼fungsverbandesâ (âThe Independence of the Cooperative Inspection Associationâ) ---

Dr. jur., 1990, with a dissertation on "Die Unabhängigkeit des genossenschaftlichen Prüfungsverbandes" ("The Independence of the Cooperative Inspection Association")

Thanks,
Rajesh.K

Re^3: How to Remove Junk Characters

wfsp

created: 2006-01-06 14:56:18

perlmonks

prlmnks

Change

my $file_cnt = $res->content;

my $file_cnt = $res->decoded_content;

See [cpan://HTTP::Message] for an explanation of the difference.

Many thanks to the search artist [kwapping] for finding it and to [tye] for explaining it :-)