Hello guys,
I am using this great module for parsing, I like it, it is quite easy, but now I get into trouble - really dont know how to parse following:I read all nodes posted here, also tutorial. Main trouble is, I dont know create while() loop only for <div class="full"> html code - if I knew this, I would write such a parser.
thanks for any helpNot a TokeParser solution, but using HTML::TreeBuilder I'd use $t->look_down( _tag => "div", class => "full" ) to get a list of the divs you're interested and then call $div->look_down( _tag => 'a' ) on each of those. Sometimes the tree solution's just conceptually easier to get your brane around.
If you're dead set on tokeing it, build a state machine:
- start in looking_for_full until you see a div with class full, when you transition to looking_for_content
- when you see a div with class content in state looking_for_content, transition to looking_for_anchors
- when you see an anchor in looking_for_anchors, save the href attribute
- when you see a
Additional: Little note on implementation: you'd have a $state variable which keeps track of which state you're in (start with my $state = 'looking_for_full';). You'd then have a while( my $t = $stream->get_token ) { ... } loop, inside of which you'd implement the above behaviors. Any non-interesting token for the current state would be ignored (e.g. just next back to fetch the next token).
Thanks for nice explanation, Fletch. I think this is how it should be done. But...In real life is easier to make regexp for that content I want to parse, so when doesn't exist other simpler solution for HTML::TokeParser, I have to pick up regexs.
Also, which parser is better ? HTML::TreeBuilder or ? I want learn only one, which is able to parse these relative easy things and has no other glitches...If you really have a fixed format that you can guarantee isn't going to change (e.g. this is a one-off throw away program to convert old data into a new format), sure go ahead and use regexen. Otherwise you'll find out n months down the road that you're going to spend the time again re-implementing it when the HTML changes because the designer got a new version of Dreampage 06 X.
As for which parser is better: depends. Which is better, a Ferrari or a heavy duty pickup? Try moving a couple palates of bricks with the former, or winning a race with the later.
In my personal experience the answer is: depends. :) I used to use TokeParser more than TreeBuilder (writing my own RSS feeds before sites provided them themselves), but more often that's now the other way around. As you can see, for a task with more context sensitivity (foo elements 2 levels down inside bar elements) it's more scaffolding from the programmer to do things with TokeParser than with a tree. But there's other types of tasks (extract any foo elements with class zorch) that'll probably be simpler to think of in the TokeParser manner.
If you haven't looked at it you should also take a gander at HTML::TokeParser::Simple which provides an even nicer token interface.
I think, when we are talking about changing format, in many times there is also needed change also program, not only regexp. Now it is ok, assume it will not change.
Thanks for nice answer about comparing TokeParser and TreeBuilder, that is really enough for me and I see the difference. Maybe there should be some other module putting those two propertie together (take token and have also tree stored somewhere). Dont know if it is possible. The main thing is - problem is solved, and I hope this node will help also to other monks!Of about the last 10 "I need to parse this HTML/XML structure" questions asked here nine of the answers were trivial using ::TreeParser (there are XML and HTML versions) and the other was trivial using XML::Twig.
Personally I use TreeParser more often in an HTML context and XML::Twig for XHTML and XML. XML::Twig is very powerful for editing, TreeBuilder is very good at looking stuff up.
At the end of the day the more modules you know a little bit about the more quickly and reliably you get stuff done. Don't be afraid to read documentation! Sometimes a quick question in the CB can save a huge amount of time, if you have a general idea where you are headed in the first place.
Limiting yourself to a single module is ... limiting! There is no one tool that does every job, not even computers.
DWIM is Perl's answer to Gödel
#!/usr/bin/perl
use strict;
use warnings;
use HTML::TokeParser;
my $doc = do { local $/; };
my $p = HTML::TokeParser->new( \$doc );
while ( my $outer = $p->get_tag("div") ) {
next unless $outer->[1]{class} eq "full";
my $nested_div = 0;
while ( my $inner = $p->get_tag ) {
# keep count of nested divs
$nested_div++ if $inner->[0] eq "div";
$nested_div-- if $inner->[0] eq "/div";
# "full" div has closed
last if $nested_div == -1;
print $p->get_text, "\n" if $inner->[0] eq "a";
}
}
__DATA__
(assumes all links within class="full")
#!/usr/bin/perl
use strict;
use warnings;
use HTML::TokeParser::Simple;
my $html = do {local $/; };
my $p = HTML::TokeParser::Simple->new(\$html)
or die "can't parse: $!";
my ($in_full, @href);
while (my $t = $p->get_token){
next if
$t->is_start_tag('div')
and
$t->get_attr('class')
and
$t->get_attr('class') eq 'content';
$in_full++, next if
$t->is_start_tag('div')
and
$t->get_attr('class') eq 'full';
$in_full = 0, next if
$t->is_start_tag('div')
and
$t->get_attr('class') ne 'full';
next unless $in_full;
push @href, $t->get_attr('href') if
$t->is_start_tag('a');
}
print "$_\n" for @href;
__DATA__