perlmeditation

Polynomial JAPH revisited (tweetiepooh)

In my [id://577605|earlier submission] I showed a means of JAPHing using a polynomial generated by Excel. Here now is some code to calculate the equation from a string all in Perl.

It works fine with the JAPH string upto 19 characters above which I think the limitations on internal maths stuff breaks. It's not obfusctation at this point and suggestions to get it work with the full 25 characters in JAPH (include line feed) would be fun. Note the code has been cobbled together on a Friday afternoon and I've made no attempt to make it elegant

#!/usr/local/bin/perl -wuse strict;use Math::Polynomial;my %h = ();my $s = "Just another Perl h";# put string into a hash {0=>J, 1=>u ...)map { $h{$_}=ord(substr($s,$_,1))}(0..length($s)-1);# use the library to get the polynomial factorsmy @p = split / /, Math::Polynomial::interpolate(%h);# generate the calculation string from factorsmy $p;my $i = length($s)-1;map{ $p .= "+$_*\$i**$i";$i-- } @p;# tidy the string, remove $i**0 (=1)# add .5 to total for rounding purposes$p =~ s/\+\-/\-/g;$p =~ s/.*\(//;$p =~ s/\).*/.5/;print "$p\n";# test the equationfor $i (0..length($s)-1) {print chr eval $p;}

Update

Links and spelling corrected as per chatterbox

iThreads for OOP (Zukoff)

I've investigated what modules CPAN have about ithreads. Some of them are too old, some - doesn't work on Win32 as good as on *nix. I was impressed with:Perl Scalable ithreaded Component Helper Extensionshttp://www.presicient.com/psiche/

A faster?, safer, user transparent, shared variable "locking" mechanism. (BrowserUk)

In perl, all user data is stored in scalars (SV*s).

What if the user mutable shared data, shared scalars were allocated from a read-only memory section?

Every time the user attempted to write to them, an exception would be raised.

When an exception occurs, the operation, whatever it may be, is re-done within the auspices of a critical section having temporarially set the section read-write. It is set back to read-only before the critical section ends.

As I understand it (on x86), the read-only/read-write status of a section of memory is an attribute of its LDT entry. These are read and written using single opcodes (SLDT/LLDT) and so the process of switching between read-only to read-write and back ought to be pretty quick.

The steps are:

enter exception handler.
enter critical section.
set the attributes of the shared data memory section to read-write.
reattempt the excepting operation (opcode?)
reset the shared data memory section read-only.
exit critical section
exit exception handler
continue with original program at the next operation.

As a critical section cannot be preempted, this effectively locks the shared data without the need for semaphores.

The removal of semaphores from the picture, vastly reduces the complexity of the problem, and vastly reduces the overhead of checking for locks.

The absence of the requirement for semaphores means that the same (existing) code for manipulating non-shared SV*s can also be used for shared SV*s, with no impact on performance for operations on non-shared data.

All locking becomes redundant. And serialisation of write accesses to shared data is dealt with entirely by a single mechanism that is controlled by the CPU itself, that of raising exceptions.

These will only occur when writing. And only when writing to shared data.

All other accesses are entirely free of any impact attributable to the provision of shared data.

The only other impact is that when variables are declared shared, they get allocated from the shared (most times, read-only) memory section.

Can this work?
Would it be faster?
Could it be retro-fitted to perl 5?
Is there any merit in it for perl 6 (or Parrot or other implementation)?

Win32/win64 certainly have all the primitives to implement this. I suspect, but do not know, that many/most other OSs (on x86 at least) have similar primitives in order to implement it. What about other platforms/hardware?

I do not have the requisite XS/perlguts skills to try it, or reach a conclusion on whether it could be done there. So I am throwing the idea out there to see what pople think?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

OT: Improving IE7 SDK Docs (was Let's suppose...) (footpad)

The recent release of IE7 has gotten me to thinking. (Always a dangerous sign.)

Let's suppose you could set aside the political BS and actually talk to the documentation team for Internet Explorer's SDK, the folks that document IE's support for HTML, CSS, and other browser idiosyncracies. Let's say you could tell them what you didn't like about the existing documentation and could tell them what would make the documentation "pop."

Mind you, this isn't a request to bash IE; that's been done quite effectively elsewhere. Instead, it's a serious attempt to solicit feedback from a community that has typically not been consulted with respect to this documentation set.

Nor is it a request for feature support in the next release of IE. Yes, we all know it would be sweet if IE would support ACID2; however, today it doesn't and this isn't that sort of thread. This is strictly a focus on the developer documentation. What's confusing, what's missing, what would make that material a useful resource for you?

(And before you bash me for being off-topic, this does affect those of us using perl to write Web 1.0-style CGI scripts. There is an audience that I feel has been left out of the discussion and wonder what, if granted the opportunity, what feedback that community would provide.)

--f

Spot the bug! (tlm)

Just got my butt chomped by this one, so I thought I'd share.

What's wrong with this function?

sub foo {  return eval { bar( shift ) } || 0;}

The answer does not depend on the definition of bar. E.g. let bar be

sub bar { return shift; }

To see what's wrong, try this:

print foo( 3 ), "\n";$@ = 3;print foo( $@ ), "\n";__END__30

The moral of the story: never use @_ inside an eval.

the lowliest monk

Using DBM::Deep and Parallel::ForkManager for a generalized parallel hashmap function builder (followup to "reducing pain of parallelization with FP") (tphyahoo)

This is a followup to

Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison)

In this post, I was attempting to create numerous variations of a function builder I'm calling hashmap_parallel.

This function builder should theoretically allow me to implement, for example, many of the utilities in List::MoreUtils in a transparently parallel way. Eg, instead of using flexygrep to build up a parallel grepping function on top of a parallel mapping function and an element test, as shown below, I would use flexy_any, flexy_all, flexy_pairwise, flexy_mesh, or what have you. These could all build on top of diverse custom hashmap_parallel, which you could plug in at will depending on your resources and the paricular bottleneck for the computation you are faced with.

So, if you have multiple processors, you can take advantage of that without cluttering up your code. And if you happen to have a 1000 commodity-box cluster, perhaps you too will one day be able to sort a 10^10 element array in under two minutes, with a simple call to distributed_sort($cmp_function, $my_big_list)

Since posting the above meditation, I have been able to implement hashmap_parallel in two ways.

1) threads (with help from LanceDeeply)
2) forks, along with multiple DBM::Deep data stores (using Parallel::ForkManager on tilly's advice, using DBM::Deep because... well, I couldn't think of any other way to do it that would work for any type of data.)

In addition, I have a simple "serial hashmap" function that I can plug into my "flexygrep" grep builder, as a sanity check.

At present, the threading solution is faster. I'm attributing this to the multiple hard drive read/writes that are required with the multiple DBM::Deep solution.

I was hoping to also implement a solution using a single DBM::Deep db and locking, which should at least reduce the read time. However, I was unable to do so. My test is to match a collection of letters against /[abc]/, where this test is made after sleeping for one second (we delay so that there is an artificial "reason" to want to take advantage of parallelism). There should be 9 matches in each case.

perl test_hashmap.pltest strings: a b c a b c a b cok 1 - threads, matches: 9 -- a a b c c b b a c -- should be 9ok 2 - threads, time elapsed: 2, wanted max seconds: 2ok 3 - fork with many dbm::deep dbs, matches: 9 -- a a b c c b b a c -- should be 9ok 4 - fork with many dbm::deep dbs, time elapsed: 1, wanted max seconds: 2Use of uninitialized value in join or string at test_hashmap.pl line 48.not ok 5 - fork with 1 dbm::deep db, matches: 7 -- a b b b  c c -- should be 9#   Failed test 'fork with 1 dbm::deep db, matches: 7 -- a b b b  c c -- should be 9'#   in test_hashmap.pl at line 48.ok 6 - fork with 1 dbm::deep db, time elapsed: 1, wanted max seconds: 2test strings: a b c d e f g h i j k l m n o p q r s t u v w x y z a b c d e f g h i j k l m n o p q r s t u v w x y z a b c d e f g h i j k l m n o p q r s t u v w x y zok 7 - threads, matches: 9 -- a b a c c b c b a -- should be 9ok 8 - threads, time elapsed: 2, wanted max seconds: 2ok 9 - fork with many dbm::deep dbs, matches: 9 -- a b a c c b c b a -- should be 9not ok 10 - fork with many dbm::deep dbs, time elapsed: 9, wanted max seconds: 2#   Failed test 'fork with many dbm::deep dbs, time elapsed: 9, wanted max seconds: 2'#   in test_hashmap.pl at line 49.not ok 11 - fork with 1 dbm::deep db, matches: 8 -- b c a c b c a b -- should be 9#   Failed test 'fork with 1 dbm::deep db, matches: 8 -- b c a c b c a b -- should be 9'#   in test_hashmap.pl at line 48.not ok 12 - fork with 1 dbm::deep db, time elapsed: 9, wanted max seconds: 2#   Failed test 'fork with 1 dbm::deep db, time elapsed: 9, wanted max seconds: 2'#   in test_hashmap.pl at line 49.1..12# Looks like you failed 4 tests of 12.

In addition to the test failures for the single DBM::Deep file shown above, about 25% of the time the single DBM::Deep file case seems to lock up, and the test never completes.

I would be very interested in receiving advice for how to do hashmap_paralell with a single DBM::Deep. And if there are suggestions for other strategies I could try out, or changes to my function builders that would speed things up, I'll be much obliged.

Thanks.

test_hashmap.pl:#!/usr/bin/perluse strict;use warnings;use Test::More qw( no_plan );use Data::Dumper;use Grep; # several different versions of grep, built up functionally.my $slow_matches_b = sub { sleep 1;   return unless $_[0];   return 1 if $_[0] =~ /[abc]/; };my $justafew_letters = [ my @letters = (('a'..'c') x 3) ];my $lotsa_letters = [ (('a'..'z') x 3) ];for (     $justafew_letters,     $lotsa_letters    ){  testem($_)}sub testem {  my $test_strings = shift or die "no test strings";  print "test strings: @$test_strings\n";  my $paralell_tests = [ {testname => 'threads',function => sub { Grep::threadgrep( $_[0], $_[1]) } },       {testname=> 'fork with many dbm::deep dbs',function => sub { Grep::fork_manydbs_grep($_[0], $_[1]) }       },       {testname => 'fork with 1 dbm::deep db',function => sub { Grep::fork_onedb_grep($_[0], $_[1]) }       },       #{testname => 'simple serial map',#function => sub { Grep::slowgrep($_[0], $_[1]) }       #}     ];  my $max_seconds_parallel=2; # parallel execution should speed things up  for (@$paralell_tests ) {    my $timestarted=time;    my $test_name = $_->{testname};    my $matches = $_->{function}->($slow_matches_b, $test_strings);    my $timeelapsed=time - $timestarted;    my $num_matches = @$matches;    ok( $num_matches == 9, "$test_name, matches: $num_matches -- @$matches -- should be 9");    ok( $timeelapsed <= $max_seconds_parallel, "$test_name, time elapsed: $timeelapsed, wanted max seconds: $max_seconds_parallel");  }}#my ($timestarted, $timeelapsed);Grep.pm:package Grep;use strict;use warnings;use Data::Dumper;use Map;# grep can be parallelized by building it on top of map_parallel# which uses forks, threads, distributed computations with MapReduce# or some such black magic# in some cases this may be faster, but not always,# it depends where your bottleneck is.# Whatever black magic is going on in the background,# by abstracting it out, the code we get is clean and easy to read.sub threadgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_threads(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub fork_onedb_grep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_forks_onedb(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub fork_manydbs_grep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_forks_manydbs(@_)};  return flexygrep($test_function, $map_function, $in_array);}# or you could do it in a non-forked/threaded/distributed/whatever# way, by basing it on the conceptually simpler function map_serial.sub slowgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_serialized(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub flexygrep {  my $test_function = shift;  my $hashmap_function = shift;  my $in_array = shift;  my $in_hash = Map::hash_from_array($in_array);  my $result_hash = $hashmap_function->($test_function, $in_hash);  my $out_array = [];  for my $key (keys %$result_hash) {    if ( my $out_true = $result_hash->{$key}->{out} ) {      push @$out_array, $result_hash->{$key}->{in}    }  }  return $out_array;}1;Map.pm: package Map;use strict;use warnings;# Black magic for doing stuff in parallel is encapsulated here# use MapReduce; -- not yet, but it's on the list.use Data::Dumper;sub hash_from_array {  my $array = shift;  my $hash;  for my $index (0..$#$array) {    $hash->{$index}->{in} = $array->[$index];  }  return $hash;}# input is a funcion (eg, my $sub_multiply by ten = { return $_[0] * 10 } ), and# a hash like# my $input_values = { blee => { in => 1 },#                      blah => { in => 2}#                    }# output is a hash like#{ blee => { in => 1, out => 10 },#  blah => { in => 2, out => 20 }#}sub hashmap_serial {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  # hash keys are processed in whatever order  for my $key ( keys %$hash) {    my $in = $hash->{$key}->{in};    my $out = $function->($in);    #print "result for $in is $out\n";    $hash->{$key}->{out} = $out;  }  return $hash;}# does the same thing as hashmap_serial# but saves the value on the hard drive# (serialized in this context means a memory value gets put on the hard disk,# not to be confused with the sense of "serial as opposed to parallel"sub hashmap_serialized {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  use File::Path qw(mkpath);  my $dir="c:/tmp/map_serialized";  mkpath($dir) unless -d "$dir";  die "no directory: $dir" unless -d "$dir";  my $file="$dir/$$.db";  my $db = DBM::Deep->new( $file );  $db->{result}=$hash;  for my $key ( keys %$hash ) {    my $in = $hash->{$key}->{in};    my $out = $function->($in);    $hash->{$key}->{out} = $out;  }  #unlink $file;  #die "couldn't delete file" if -f $file;  return $hash;}# uses many DBM::Deep stores, along with forks# works# but slowsub hashmap_parallel_forks_manydbs {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  use File::Temp qw(tempdir);  my $dir = tempdir();  use Parallel::ForkManager;  use DBM::Deep;  my $pm=new Parallel::ForkManager(10);  for my $key ( keys %$hash ) {    $pm->start and next;    my $in = $hash->{$key}->{in};    my $out = $function->($in);    my $file="$dir/$key";    #print "file: $file\n";    my $db = DBM::Deep->new( $file );    $db->{result}=$out;    #print "in $in, out $out\n";    $pm->finish;  }  $pm->wait_all_children;  for my $key ( keys %$hash ) {    my $file="$dir/$key";    my $db = DBM::Deep->new( $file );    defined( my $out = $db->{result} ) or die "no result in $file for key $key";    $hash->{$key}->{out}=$out;  }  #print "hash: " . Dumper($hash);  return $hash;}# tries to use one locked DBM::Deep store, along with forks# doesn't worksub hashmap_parallel_forks_onedb {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  use File::Temp qw(tempfile);  my ($fh, $file) = tempfile();  use DBM::Deep;  my $db = DBM::Deep->new(                file => $file,locking => 1        );  use Parallel::ForkManager;  my $pm=new Parallel::ForkManager(10);  for my $key ( keys %$hash ) {    $pm->start and next;    my $in = $hash->{$key}->{in};    my $out = $function->($in);    #print "in $in, out $out\n";    #$db->lock();    $db->{result}->{$key}->{in}=$in;    $db->{result}->{$key}->{out}=$out;    #$db->unlock();    $pm->finish;  }  $pm->wait_all_children;  my $result = $db->{result};;  #print "result: " . Dumper($result);  return $result;  #return $hash;}#workssub hashmap_parallel_threads {    my $function = shift;    my $hash = shift;    use threads;    my @threads;    for ( keys %$hash ) {      my $in = $hash->{$_}->{in};      my $t = threads->create( sub { map_element($_, $function, $in ) } );      push @threads, $t;    }    #wait for threads to return ( this implementation is bound by slowest thread )    my %results = map { %{ $_->join() }; } @threads;    #print Dumper \%results;    return {%results};}sub map_element {      my $key = shift;      my $function = shift;      my $in = shift;      my $out = $function->($in);      return { $key => {in =>  $in,out => $out       }     };}1;

Things you should need to know before using Perl regexes. (Humour, with a serious point) (BrowserUk)

This is not a tutorial.

Perl's regex engine is not lightweight

Every time you use $`, $& or $', the entire scalar you are searching is copied.

What's more, not only are the scalars that you process using the regex contain a reference to one of those variables copied, but every scalar, processed by every regex in your entire program also gets copied.

Further, every time you use capturing brackets, all the captured chunks are also copied--again.

And, even correctly written regexes that use two or more variable length matches (* or + etc.) can consume prodigious amount of runtime stack and cpu.

Badly and/or naively written regexes that use nested qualifiers can have exponential runtimes, and if the scalar they operate on is anything more than modestly sized, can completely consume your process stack before finally trapping having consumed all your process memory allocation, or system swap space--whichever runs out first.

Dooom, gloom, despondency.

More doom gloom and despondency.

Blah, blah, blah.

Oh. and here is a solution that prevents some of the problems by wrapping each call to the regex engine.

It starts anothor process, sends your scalars and the regex to it via sockets. That other process runs the regex on your behalf, and sends the results back via another socket. This neatly eliminates the $& problem, and allows recovery from the stack runaway/memory exhaustion problems whilst keeping your main process' memory requirements to a minimum.

This is not a serious attack on the perl regex engine!

Whilst much of the above is and has been true for the past 5 (8?, 10?) years, most of it could not be otherwise.

And the point is that the regex engine isn't lightweight, and has some vagaries and caveats,

but that hasn't prevented thousands of programmers from writing 100s of thousands of perfectly functional, useful, beneficial scripts that use Perl's regex engine

Note:The stack problem has been very cleverly fixed in a recent build,

Perl is dead (cLive ;-))

Excuse me if this has been discussed before (I couldn't find it though). This is a rather amusing rant from an Amazon developer.

Even though it is a little OTT in places, it does definitely have some valid points.

I love Perl, but my faith in its future is not that great right now. I feel Python beckoning ;-)

How many words does it take? (Limbic~Region)

All,
This meditation is about not giving up on a problem just because it is [wp://Np_complete|NP complete]. Common responses from seasoned programmers to NP complete questions include:

Don't bother, it is NP complete
Find a heurstic approximation and hope it is good enough
Brute force is the only way to go

Response 2 is actually pretty good advice when it applies. Response 3 may be true but not all brute force approaches are created equal. The fact is that real world problems need to be solved regardless of CS theory. Just because something is NP complete, doesn't mean there isn't a smart(er|ish)? way of solving it.

I decided to work with [id://348444]. I was familiar with it because I had already provided a fairly [id://545167|fast approximation algorithm]. The problem stated is simple: Given a dictionary file, find the minimal number of words that contain a given set of letters.

I set out a list of constraints which turned the general problem into a specific one:

The dictionary file = 2of12 from the [http://wordlist.sourceforge.net/|Official 12Dicts Package]
Letters of both question and answer = a-z
Answer need only be fewest words, not fewest letters
Question may not contain duplicate letters - abc ok, abbc not ok

Originally, I wanted to precalculate enough information to allow each question to be solveable in polynomial time. While I still believe this approach would work with some problems, I ended up precalculating all solutions and functionally turned it into a lookup table. To do this, I needed to reduce the problem space enough to make a brute force solution feasible. Here are the guiding optimization principals I employed:

Not all solutions need to be found

There are a finite number of questions that can be asked because the letter set has been restricted to a-z. Using the combinatorial formula C = N! / K!(N - K)! when N = 26 and K = 1..26, there are 67_108_863 different possible questions. Since the solution that applies to 'abc' also applies to 'ab', 'ac', 'bc', 'a', 'b', and 'c' - even fewer questions actually have to be answered.

Not all words need be considered

Because of the constraints, we can reduce a word to the unique list of letters in contains. The word 'screwdriver' becomes 'cdeirsvw' and the word 'disc' becomes 'cdis'. Because there are no letters in 'disc' that are not contained in 'screwdriver', 'disc' need not be considered. Applying this principal further, we end up with only words of the longest string of unique characters.

Use a breadth-first search

Instead of solving questions at random, a [wp://Breadth-first_search|BFS] will provide the most efficient solution path. Start by finding all questions that can be solved using a single word from our reduced dictionary, then 2 at a time, then 3 at a time, etc until the longest possible question (a-z) can be answered.

Not all combinations of words need be considered

Since our goal is to get to a-z, when combining words we need only consider words that contain letters we don't already have. Further, we can use the reduction technique to only include the longest list of "new" characters. As an example if we start with 'hello' and are considering combining it with 'windowpane' and 'weapon', we are really considering 'adinpw' and 'anpw' and we can safely ignore 'weapon'.

Not all derived answers need be considered

Since we know that the answer for 'abc' also answers 'ab', 'ac', 'bc', 'a', 'b', and 'c' - we only need solve for 'abc' and we get the [wp://Powerset|powerset] for free. We can take advantage of this when solving new questions because we can skip the portions of the powerset that have been solved by some previous question. For instance, if we are looking at 'abcd', 'abce', and 'abcfg' - we know we can skip 'abc' and their descendants when solving the second and third question.

Even with these guiding principals in mind before I started, I still followed a number of dead-ends before finding the right approach. I will attempt to highlight the mistakes I made but will only be showing code for the actual solution.

Phase 1: Reduce the dictionary

Number of lines in input: 61_406
Number of lines in output: 3_477
Execution time: 1 min 15 seconds

#!/usr/bin/perluse strict;use warnings;use constant  WORD => 0;use constant   LEN => 1;use constant NFORM => 2;use Inline C =>;my $file = $ARGV[0] || 'dictionary.txt';open(my $fh, '<', $file) or die "Unable to open '$file' for reading: $!";my @word;while (<$fh>) {    $_ = lc;    tr/a-z//cd;    my %uniq = map {$_ => undef} split //;    my $len = keys %uniq;    push @word, [$_, $len, join '', sort keys %uniq];}@word = sort {$b->[LEN] <=> $a->[LEN] } @word;for my $i (0 .. $#word - 1) {    next if ! defined $word[$i];    for my $j ($i + 1 .. $#word) {        next if ! defined $word[$j];        $word[$j] = undef if ! distinct($word[$i][NFORM], $word[$j][NFORM]);    }}for (grep defined, @word) {    print join "\t", $_->[NFORM], $_->[WORD];    print "\n";}__END____C__int distinct(unsigned char *str1, unsigned char *str2) {    /* Actual code has 256 0s - truncated for post */    char exists[256] = {};    /* Turn array into a hash */    while (*str1) {        exists[*str1++] = 1;    }    /* Determine if str2 contains any chars str1 does not */    while (*str2) {        if (! exists[*str2++]) return 1;    }    return 0;}

Phase 2: Two at a time

Number of lines in input: 3_477
Number of lines in output: 636_186
Execution time: 9 min 35 seconds

#!/usr/bin/perluse strict;use warnings;use constant NEW_LET => 1;use constant     LEN => 2;use Inline C =>;my $file = $ARGV[0] || 'phase1.data';open(my $fh, '<', $file) or die "Unable to open '$file' for reading: $!";my @word;while (<$fh>) {    chomp;    push @word, [split /\t/];}my %seen_nform;for my $i (0 .. $#word - 1) {    my ($nform1, $str1) = @{$word[$i]};    my (@new_word, %seen);    for my $j ($i + 1 .. $#word) {        my ($nform2, $str2) = @{$word[$j]};        my $new_let = diff($nform2, $nform1);        next if ! $new_let || $seen{$new_let}++;        push @new_word, [$str2, $new_let, length($new_let)];     }    @new_word = sort { $b->[LEN] <=> $a->[LEN] } @new_word;    for my $i2 (0 .. $#new_word - 1) {        next if ! defined $new_word[$i2];        for my $j2 ($i2 + 1 .. $#new_word) {            next if ! defined $new_word[$j2];            $new_word[$j2] = undef if ! distinct($new_word[$i2][NEW_LET], $new_word[$j2][NEW_LET]);        }    }    for (grep defined, @new_word) {        my $str2 = $_->[0];        my %uniq = map {$_ => undef} split //, $nform1 . $str2;        my $new_nform = join '', sort keys %uniq;        next if $seen_nform{$new_nform}++;        print join "\t", $new_nform, $str1, $str2;        print "\n";    }}__END____C__SV* diff ( char *str1, char *str2 ) {    SV *sv= newSVpvn( "", 0 );    int result_index= 0;    char *result= SvGROW( sv, 257);    /* identify all chars present in str2 */    while ( *str1 && *str2 ) {        if ( *str1 < *str2)             result[ result_index++ ]= *str1++;        while ( *str1 && *str1 == *str2) {            str1++; str2++;        }        if ( *str1 > *str2 )            str2++;    }    while (*str1)         result[ result_index++ ]= *str1++;    result[ result_index ]= 0;    SvCUR_set( sv, result_index );        return sv;}int distinct(unsigned char *str1, unsigned char *str2) {    /* Actual code has 256 0s - truncated for post */    char exists[256] = {};    /* Turn array into a hash */    while (*str1) {        exists[*str1++] = 1;    }    /* Determine if str2 contains any chars str1 does not */    while (*str2) {        if (! exists[*str2++]) return 1;    }    return 0;}

Phase 3: Three at a time

Number of lines in input: 636_186
Number of lines in output: 8_809_183
Execution time: 401 min 10 seconds

Yes, that's java code you see. I originally wrote it in Perl which looked very much like phase 2 but it took between 6 1/3 hours to 11 1/2 hours depending on which machine and what variation of the algorithm I used. If anyone is interested in the perl counterpart - please let me know.

import java.io.File;import java.io.IOException;import java.util.ArrayList;import java.util.BitSet;import java.util.HashMap;import java.util.HashSet;import java.util.LinkedHashMap;import java.util.Map;import java.util.Scanner; public class Phase3 {    public static final String regex = "[a-z]+";    public static char alphabet[] = {        'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',        'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z',    };    public static BitSet[] seen_nform = {        new BitSet(),        new BitSet(26),       new BitSet(325),     new BitSet(2600),        new BitSet(14950),   new BitSet(65780),    new BitSet(230230),  new BitSet(657800),        new BitSet(1562275), new BitSet(3124550),  new BitSet(5311735), new BitSet(7726160),        new BitSet(9657700), new BitSet(10400600), new BitSet(9657700), new BitSet(7726160),        new BitSet(5311735), new BitSet(3124550),  new BitSet(1562275), new BitSet(657800),        new BitSet(230230),  new BitSet(65780),    new BitSet(14950),   new BitSet(2600),        new BitSet(325),     new BitSet(26),       new BitSet(1)    };        public static HashMap lookup = new HashMap(26);    public static void main(String[] args) {        initlookup();        Scanner scr1, scr2;        String  line;        LinkedHashMap word = new LinkedHashMap(3477);        // Load phase1.data into word        try {            scr1 = new Scanner(new File(args[0]));            while (scr1.hasNextLine()) {                line  = scr1.nextLine();                scr2  = new Scanner(line);                word.put(scr2.next(regex), scr2.next(regex));            }        }        catch (IOException ioe) {            ioe.printStackTrace();        }        // Process phase2.data        String nform, str1, str2;        try {            scr1 = new Scanner(new File(args[1]));            while (scr1.hasNextLine()) {                line  = scr1.nextLine();                scr2  = new Scanner(line);                nform = scr2.next(regex);                str1  = scr2.next(regex);                str2  = scr2.next(regex);                HashSet seen = new HashSet();                ArrayList new_word = new ArrayList();                for (Map.Entry entry : word.entrySet()) {                    String new_nform = uniq(nform, entry.getValue());                    int len = new_nform.length();                    int bit = getBit(len, new_nform);                    if (seen_nform[len].get(bit)) { continue; }                    String new_let = diff(entry.getKey(), nform);                    if (new_let.length() == 0 || seen.contains(new_let)) { continue; }                    seen.add(new_let);                    String[] e = {entry.getValue(), new_let, new_nform, Integer.toString(bit)};                    new_word.add(e);                }                String[] new1, new2;                int idx;                int end = new_word.size();                for (int i = 0; i < end - 1; ++i) {                    new1 = new_word.get(i);                    if (new1 == null) { continue; }                    for (int j = i + 1; j < end; ++j) {                        new2 = new_word.get(j);                        if (new2 == null) { continue; }                        idx = j;                        if (new2[1].length() > new1[1].length()) {                            String[] temp = new1;                            new1 = new2;                            new2 = temp;                            idx  = i;                        }                        int res = distinct(new1[1], new2[1]);                        if (res == 0) {                            new_word.set(idx, null);                        }                    }                }                for (String[] e : new_word) {                    if (e == null) { continue; }                    String str3 = e[0];                    String new_nform = e[2];                    int len = new_nform.length();                    int bit = Integer.decode(e[3]);                    /* Probably not needed */                    if (seen_nform[len].get(bit)) { continue; }                    seen_nform[len].set(bit);                    System.out.println(new_nform + "\t" + str1 + "\t" + str2 + "\t" + str3);                }            }        }        catch (IOException ioe) {            ioe.printStackTrace();        }    }    private static void initlookup () {        // Actual code goes a-z, reduced for post        lookup.put('a', new Integer("1"));        lookup.put('b', new Integer("2"));    }    // Determine Chars in str1 not present in str2    private static String diff(String str1, String str2) {        int len = str2.length();        HashSet have = new HashSet(len);        for (Character c : str2.toCharArray()) {            have.add(c);        }        StringBuilder new_let = new StringBuilder(len);        for (Character c : str1.toCharArray()) {            if (have.contains(c)) {                continue;            }            new_let.append(c);        }        return new_let.toString();    }    // Determine if str2 (shorter) contains any chars not in str1 (longer)    private static int distinct(String str1, String str2) {        HashSet have = new HashSet(str1.length());        for (Character c : str1.toCharArray()) {            have.add(c);        }        for (Character c : str2.toCharArray()) {            if (have.contains(c)) {                continue;            }            return 1;        }        return 0;    }     // List of unique chars in alphabetic order in str1 & str2    private static String uniq(String str1, String str2) {        HashSet have = new HashSet();        for (Character c : str1.toCharArray()) {            have.add(c);        }        for (Character c : str2.toCharArray()) {            have.add(c);        }        StringBuilder unique = new StringBuilder(26);        for (char let : alphabet) {            if (have.contains(let)) {                unique.append(let);            }        }        return unique.toString();    }    private static int binomial(int n, int k) {        int c = 1;        for (int i = 0; i < k; ++i) {            c *= n - i;            c /= i + 1;        }        return c;    }    private static int getBit(int len, String str) {        int    sum = 0;        for (int i = 0; i < len; ++i) {            sum += binomial(lookup.get(str.charAt(i)) - 1, i + 1);        }        return sum;    }}

Phase 4: Four at a time (new approach)

Number of lines in input: 8_809_183
Number of lines in output: 1
Execution time: 0 min 7 seconds

The last phase brought the solution close enough just to scan for a word that contained all the missing letters to reach a-z.

#!/usr/bin/perluse strict;use warnings;use Inline C =>;my $file = $ARGV[0] || 'phase1.data';open(my $fh, '<', $file) or die "Unable to open '$file' for reading: $!";my @word;while (<$fh>) {    chomp;    my ($nform, $word) = split /\t/;    push @word, $word;}$file = $ARGV[1] || 'phase3.data';open($fh, '<', $file) or die "Unable to open '$file' for reading: $!";while (<$fh>) {    my ($nform, $str1, $str2, $str3) = split /\t/;    for (@word) {        if (! diff('abcdefghijklmnopqrstuvwxyz', $_ . $nform)) {            print "abcdefghijklmnopqrstuvwxyz\t$_\t$str1\t$str2\t$str3";            exit;        }    }}print "No Cigar\n";__END____C__SV *diff(char *str1, char *str2) {    /* Actual code has 256 0s - truncated for post */    char exists[256] = {};    SV *ret_sv = newSVpvn("",0);        /* identify all chars present in str2 */    while (*str2) {        exists[(U8)*str2++] = 1;    }    /* Determine chars in str1 not in str2 */    for ( ; *str1 ; str1++ )         if (! exists[(U8)*str1])             sv_catpvn(ret_sv,str1,1);        return ret_sv;}

Phase 5: Generate powersets

Number of lines in input: 9_448_847
Number of lines in output: 67_108_863
Execution time: 66 min 20 seconds

Yep, more Java. I also have a Perl version if anyone is interested but I wanted the final solution to take less than 8 hours to run so that someone wanting to reproduce my results could.

import java.io.File;import java.io.IOException;import java.util.ArrayList;import java.util.BitSet;import java.util.HashMap;import java.util.Scanner; public class Phase5 {    public static final String regex = "[a-z]+";    public static BitSet[] seen = {        new BitSet(),        new BitSet(26),       new BitSet(325),     new BitSet(2600),        new BitSet(14950),   new BitSet(65780),    new BitSet(230230),  new BitSet(657800),        new BitSet(1562275), new BitSet(3124550),  new BitSet(5311735), new BitSet(7726160),        new BitSet(9657700), new BitSet(10400600), new BitSet(9657700), new BitSet(7726160),        new BitSet(5311735), new BitSet(3124550),  new BitSet(1562275), new BitSet(657800),        new BitSet(230230),  new BitSet(65780),    new BitSet(14950),   new BitSet(2600),        new BitSet(325),     new BitSet(26),       new BitSet(1)    };        public static HashMap lookup = new HashMap(26);    public static void main(String[] args) {        initlookup();        String  line = null;        Scanner s2   = null;        for (String filename : args) {            try {                Scanner scr = new Scanner(new File(filename));                while (scr.hasNextLine()) {                    line = scr.nextLine();                    s2 = new Scanner(line);                    if (! s2.hasNext(regex)) {                        continue;                    }                    getSets(s2.next(regex), line);                }            }            catch (IOException ioe) {                ioe.printStackTrace();            }        }    }    private static void initlookup () {        // Actual code goes a-z, reduced for post        lookup.put('a', new Integer("1"));        lookup.put('b', new Integer("2"));    }    private static void getSets(String str, String line) {        int len = str.length();        int bit = getBit(len, str);        if (! seen[len].get(bit)) {            seen[len].set(bit);            System.out.println(str + "\t" + line);            for (StringBuilder set : subsets(str)) {                getSets(set.toString(), line);            }        }    }    private static ArrayList subsets(String str) {        ArrayList subs = new ArrayList();        if (str.length() == 1) {            return subs;        }        for (int i = 0; i < str.length(); ++i) {            StringBuilder set = new StringBuilder(str);            set.deleteCharAt(i);            subs.add(set);        }        return subs;    }    private static int binomial(int n, int k) {        int c = 1;        for (int i = 0; i < k; ++i) {            c *= n - i;            c /= i + 1;        }        return c;    }    private static int getBit(int len, String str) {        int    sum = 0;        String key;        for (int i = 0; i < len; ++i) {            sum += binomial(lookup.get(str.charAt(i)) - 1, i + 1);        }        return sum;    }}

Creating a working final product

I decided to use a [cpan://DBD::SQlite] database with a very simple [http://www.gatcomb.org/dictdemo/demo.cgi|web front end to demo it]. The split into 26 tables was left over from an earlier disk space optimization idea that I didn't pursue so the following code could be simplified further:

Split the data by length and convert solution words to indices (took 12 minutes)

#!/usr/bin/perluse strict;use warnings;use Storable;my %fwd;my $file = $ARGV[0] || 'phase1.data';open(my $fh, '<', $file) or die "Unable to open '$file' for reading: $!";my $n;while ( <$fh> ) {    chomp;    my ($nform, $word) = split /\t/;    $fwd{$word} = ++$n;}my %sol_fh;for (1 .. 26) {    my $file_name = sprintf("%.2d", $_) . ".data";    open($sol_fh{$_}, '>', $file_name) or die $!;}$file = $ARGV[1] || 'phase5.data';open($fh, '<', $file) or die "Unable to open '$file' for reading: $!";while ( <$fh> ) {    chomp;    my ($nform, undef, @words) = split /\t/;    $_ = $fwd{$_} for @words;    print { $sol_fh{length($nform)} } $nform, "\t", (join "-", @words), "\n";}my %rev = reverse %fwd;store \%rev, 'sol.rev';

Build the database and use the sqlite3 shell interface to import the data

#!/usr/bin/perluse strict;use warnings;use DBD::SQLite;my $dbh = DBI->connect("dbi:SQLite:dbname=solution.db","","") or die $DBI::errstr;for my $size (1 .. 26) {    my $sql = "CREATE TABLE solution$size (nform TEXT, solution TEXT)";    $dbh->do($sql) or die $dbh->errstr;}$dbh->disconnect or die $dbh->errstr;

Create a simple web front end

#!/usr/bin/perl -Tuse strict;use warnings;use CGI::Simple;use DBD::SQLite;use HTML::Template;use Storable;$CGI::Simple::POST_MAX        = 1024;$CGI::Simple::DISABLE_UPLOADS = 1;#$CGI::Simple::DEBUG = 1;my $q = CGI::Simple->new();# TODO: if unable to get exclusive lock on $0 display_error()# TODO: Add error handling - duh!$q->param('answer') ? display_answer() : display_question();sub display_question {    my $template = HTML::Template->new(filename => 'question.tmpl');    print $q->header;    print $template->output();    exit(0);}sub display_answer {    my $template = HTML::Template->new(filename => 'answer.tmpl');    my $input = get_input();    my $len   = length($input);    my $rev   = retrieve('sol.rev');    my $dbh   = DBI->connect("dbi:SQLite:dbname=solution.db","","")        or die $DBI::errstr;    my $sth   = $dbh->prepare("SELECT solution FROM solution$len WHERE nform=?")        or die $dbh->errstr;    $sth->execute($input)                                                  or die $dbh->errstr;    my @word;    while (my @row = $sth->fetchrow_array() ) {        push @word, map { {WORD => $rev->{$_}} } split /-/, $row[0];    }    $template->param(USER_INPUT => $input);    $template->param(WORD_LIST  => \@word);    print $q->header;    print $template->output();    #$sth->finish();    #$dbh->disconnect;    exit(0);}sub get_input {    my $input = $q->param('question') || '';    $input = lc($input);    $input =~ tr/a-z//cd;    $input ||= 'abcdefghijklmnopqrstuvwxyz';    my %uniq = map {$_ => undef} split //, $input;    return join '', sort keys %uniq;}

Include a question template

How many words does it take.....How many words does it take....
Please enter a unique list of lower case letters (a-z) below.
Be warned that some of the "words" may not be safe for work (NSFW).

Include an answer template

It takes.....For  it takes....
    Word:

Conclusion

Answering all 67+ million questions using a dictionary of over 64K words can take less than 8 hours and the resulting lookup table (SQLite) is 2GB of disk space. If this were a real world problem, that is not a considerable investement.

Notes

I freely admit that I spent far more than 8 hours on this specific project. I also admit that each problem is unique and has no guarantee of being feasible. I came to the Monastery at least twice looking for help with optimizations:

[id://579130]
[id://576101]

I had to find the right balance of trading memory for speed since 1GB of RAM was not enough in many cases. I also had to find the right balance between reducing the output for the next phase with the amount of work required for that reduction.

The code as it appears can likely be optimized. I spent my time looking for algorithm related optimizations rather than micro-optimizations. While I used both [cpan://Inline::C] and Java in this project, I am not much more than a neophyte in either. In fact, this thread contains all the Java I have ever written in my life. I had help in those areas but the contributors were limited by my description of what I was trying to do so all deficiencies are mine rather than theirs.

I would like to thank everyone who helped me with this project. Rather than naming the few I can remember and offending those I can't, I will just say "you know who you are". You can try it out [http://www.gatcomb.org/dictdemo/demo.cgi|here]. I welcome any questions or comments on the code and algorithms.

Cheers - [Limbic~Region|L~R]

Parrot, threads & fears for the future. (BrowserUk)

The future is threaded.

At the commodity hardware level, this is already the case thanks to Intel Hyperthreading, Intel, AMD (and others) dual core processors. AMP, SMP & NUMA. Dual, quad & 8-way motherboards. In the future it will be more so. Eg. The Cell processor. Quad-core technology from Intel and AMD coming in 2007.

At the software level, very little existing software makes use of threading. There are several reasons for this:

Software needs to be written from the ground up with threading in mind in order to properly benefit from it.
Retro-fitting threading to existing applications is rarely effective because threading exacerbates the effect of every bad programming practice.
- Re-entrancy issues can not be glossed over. Whether in the application code, language runtime or OS.
- Global data-structures become even more vulnerable.
- Tight coupled code causes low granularity. Low granularity can make threading expensive.
- Memory management becomes paramount.
  - Stop the world GC will have a disastrous effect upon performance.
  - Monolithic heap management will suck the life out of efficiency.
Existing code that can benefit from threading, often uses event driven and/or state machine techniques to approximate those benefits, but code designed to utilise those techniques is usually structured such that it does not lend itself to conversion to threading.
Existing code that already achieves parallelism through forking, that could also benefit from shared state, is often difficult to adapt to threading due to the assumptions that can be safely made when using forking that no longer hold true with threading.

Redeveloping existing, successful applications that could benefit from threading is often slow to happen. Again, for a variety of reasons:

Mid-life, ground-up, redesign of an existing application is always a major undertaking, even where there are clear benefits to doing so.
In many environments threading is seen as hard.
In many environments, there is a lack of both understanding and the skills required to implement threading well.

Even when developing new applications, that could obviously benefit from threading, it is overlooked, ignored or explicitly ruled out. Again there are a variety reasons why this happens, most of which are already covered above.

Much of the resistance/reluctance to utilise threading can be attributed to a single factor--there are few if any good tools available.

Most languages--that the majority of people use--either do not support threading at all, or support it as an afterthought. And then only at the lowest level.
There are few good abstractions of shared state. If every program still had to deal with disk-bound data by directly manipulating physical blocks and freespace chains, very few programs that manipulate file-based data would exist--that's most programs in existence. Today's ubiquitous hierarchical filesystems make it seem as if there was never any other way, and that tends to imply that they are perfect. But we also have RDBMSs, which are most definitely not hierarchical, and not file-based, (Although they often live on hierarchical filesystems.).
Most of todays programming tools--compilers, interpreters, editors, debuggers, runtime libraries etc.--are the latest evolutions of the same tools that go back years., In many cases, decades.

And like any other type of existing application, adaption to threading is difficult, resisted ,expensive and risky. Redesign from the ground up with threading in mind, implies throwing away thousands or millions of development hours in thoroughly tried and tested tools and libraries.

Could that possibly be worth the trouble?

What if an existing, popular, powerful and flexible language was already being redesigned from the ground up.

If that language was already looking to support simple syntax and intuitive semantics for Distributed Operations?

    if( any( @list ) == constant ) ...    @list1 >> += << @list2

And those DistOps inherently lent themselves to being run concurrently on multiple hyperthreads/cores/CPUs?

What if the entire tool-chain to support that new language was also already being redesigned from the ground up?

Doesn't it make sense to write those tools with threading not just "in mind", but as a high priority?

Will those tools be all they could be, if their architects "Don't do threads"? If the implementors "do not see the need for threads"?

Indeed, does it bode well for the future of those tools if the implementors do not use the languages that those tools are to support, And don't have the slightest feel for what will drive the needs and uses of those languages in the future?

Does the

complete absence of a threads.pdd from the specification;
that the term "threads" appears only 35 times in the entire documentation set;
that the "failed" ithreads model, so widely denigrated and despised, is being nearly exactly replicated for the underpinnings of the new language;

inspire you with confidence?

What about userspace threading? Many languages provide this and many programmers find it's determinism and light weight lends itself to many things that they want to do. It's not a replacement for kernel threads, but if threads != interpreter, then providing primitives to allow cooperative, user threading to run within preemptive kernel threading becomes not just possible but almost trivial. Whether this is provided at the VM level or the language level. Trying to hack user space threading into a language when thread == interpeter becomes a completely different ballgame.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Creating a co-operative framework for testing (gmax)

Hi Monks,

I recently applied for a post of QA developer at a open source company and I started working there, with great delight, about one month ago. One of the reasons for my hiring was my active participation in the community, with blogging, writing articles, submitting bug reports, answering to forum and NG questions. My background as a tester was considered, of course, but the stress was on my community links.

Inviting an open source community to Quality Assurance tasks

Thus, one of my tasks is now to organize a framework for cooperation between the Quality Assurance department and the community.

It is not an easy task to tackle. I can see how much an active community can contribute (bug reports, test cases, code patches, usability feedback), but I can also see the clash between a QA department populated by professional testers and the erratic behavior of a large community. Somehow I have to find a way of reconciling these two worlds, and make them work harmoniously toward a common goal.

My plan (yes, in spite of the task being so though, I do have a plan) is to motivate potential contributors with durable benefits, to get the best possible outcome from this relationship.

During the past twelve months, the company has promoted such cooperation with a contest. Win an iPod if you submit many bugs, and/or write articles or blog posts about our products. The results were quite good in the beginning, but they faded away soon after a few weeks. Some of the prizes are yet to be awarded. Why did this policy fail to provide the expected results? IMO, because it was targeting the wrong quality of a potential contributor: it was appealing at the competitiveness, instead of tickling pride and desire of recognition, which work much better in an open source environment. But especially, it did not address the main aspect of open source success, i.e. mutual benefit.

Why does someone contribute to an open source project? There are many reasons, but the paramount one should be striving for improvement. The main point of open source is being able to modify something that barely works for you into something that fits your needs appropriately. Thus the mutual benefit: when you add a feature or fix a bug in an open source product that affects your work, you make that product better for you, but you are also improving its overall value.

The challenge of testing

Among the activities that make the quality of a software product, testing is perhaps the most visible one. There are other elements, such as policies for coding, internal code reviews, and several organizational issues that will affect the final quality. But testing is a key element in software development. Thus, my company has a huge regression test suite that does a good job of keeping each build free of bugs.

As everyone knows, testing is never enough. Despite all the efforts from the developers and the QA engineers, there is always a bug that escapes their attention and affects the user. Why? Surely because testing everything is impossible, but also because developers and QA engineers are highly trained people and they approach problems from a different perspective than the final users.

Common users take the product for what they need, and just throw at it the commands that will solve their problem, regardless of any other concern that the developers could take into account. This naive behavior is what finds the most appalling bugs.

The challenge, then, is to combine the scientific approach of a QA department with the cleverness of willing contributors, who are able to find bugs that elude most of the professionals.

The lessons of Perl

That's why (finally) I am coming here to ask for advice. My job is not as a pure Perl developer (although Perl is largely used in our testing infrastructure), so this problem is not related to Perl as a language, but with Perl as a community. I was accustomed to testing before using Perl, but it is in the Perl community that I found the most efficient way of testing.

I can see that the Perl community has a very good testing infrastructure. I see how it is organized, how it works, but the reasons why it is so good escape me.

Seeking advice

Here are the questions

Why is the Perl testing infrastructure so effective?
If I wanted to export some of the qualities of Perl testing to a non-Perl product, what should I focus on?
What motivates a (QA) contributor?

So, fire on. Any insight on this matter could be valuable.

Thanks in advance.

 _  _ _  _  (_|| | |(_|>< _|

Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison) (tphyahoo)

This is a followup from two recent posts:

Could there be ThreadedMapReduce (and/or ForkedMapReduce) instead of DistributedMapReduce?

using parallel processing to concatenate a string, where order of concatenation doesn't matter

In a sense it is a followup from an older post as well:

What is the fastest way to download a bunch of web pages?

The common theme to all these posts is doing something in parallel to accomplish a task, where each bit of the task is independent of the other bits.

It turns out that this is the essense of the MapReduce algorithm that google uses to build much of its most important code. Google uses cluster computing to accomplish this, but the same "parallelization" logic could also be used on a single computer running multiple processes, or even a single process with threads. (Ira Woodhouse has indicated that the next version of MapReduce released to the CPAN will probably have configuration options for single computer clusters in order to do exactly that.)

Currently, the canonical example of a situation where this would be useful for me, on my job, is to download a bunch of web pages and verify that each page regex-matches something that it should. In other words, grep, where part of the grep test is to download a web page.

Throughout my experience with perl, have had numerous other situations where this basic idea -- break down a task into components that can run in parallel, run them, and then reassemble the results -- would have been helpful. Sometimes I would do this, but because I'm not an experienced thread programmer, and sometimes I was on windows sometimes not, sometimes would have to recompile perl, etc etc, it was always an unpleasant experience. There were many occasions when I thought, maybe I could use parallelization here... but painful... not enough time to debug.... I'll just make do with having it be a little slower.

An example of a time when I could have, but didn't, use functional programming based on parallelization to speed things up, was when I had ~50,000 web pages to parse and reformat using a variety of functions based on HTML::TreeBuilder. None of the files cared what the other files looked like, so I could have processed multiples at the same time. But I got the basic system working serially, and it got the job done in a bit over 48 hours. This was too slow, so I did some simple things with forks and got it down to under a day, which was acceptable because the script only had to run once.

But I remember thinking the forking code the ugliest, and hardest to debug, in my program.

If I had had 5 million web pages instead of 50,000, and wanted to split the computation among multiple processors somehow, putting the job queue stuff together for it along the lines I had used till then, it would have been a nightmare. Even though all I was doing was grep, with a little network communication inside the grep test function.

Later when I came across the article

Why Threads are a Bad Idea

it all rang very true to me. Okay, I was working with forks, in a simple context, and this presentation is maybe more about threads with systems programming. But the basic difficulties apply to any situation where you are running stuff in an order that isn't guaranteed. It's different from running stuff through a simple, ordered, for loop in ways that keep quite subtle and hard to detect.

What frustrated me the most is the feeling that I couldn't encapsulate the logic that I want, that I have to keep writing the "ugly bits" again and again.

This was before I had heard of functional programming.Since then, I have been learning a lot about functional programming, and trying to incorporate it into my bag of tools. I originally became interested in this after reading Paul Graham's On Lisp. But I want to apply functional programming to make my life easier in perl. Joel Spolsky suggests in

Can Your Programming Language Do This

that functional programming is a good technique to hide the "ugly but important bits" of your code.

In the article, spolsky suggests that this is exactly what Google has done with their MapReduce algorithm. Google programmers can write code that says the equivalent of

my @results = distributed_mapreduce_grep ( $test_function, [ @in_array ]);

and this would do exactly what

my @results = grep { $test_function($) )} @array

would do in perl. Except that it works on a cluster. So you can process a lot more data, faster.

And the "ugly bits" are hidden.

I want to start hiding the ugly bits of my code, using functional programming.

The following is my attempt to do that.

Unfortunately it still isn't working. But I think it's an interesting read, and I'm also hoping someone can plug something in there that will make it work.

What's nice is that the "ugly bit" that isn't working is encapsulated. This is the function hashmap_parallel. Currently my "parallelization strategy" involves forking off processes and storing values in a DBM::Deep hard disk store. But actually I don't care about the implementation details, I just want it to work.

****************************

UPDATE: Thanks to LanceDeeply, I now have code that works, using threads for the parallelization.

I kept the function that doesn't work as hashmap_parallel_forks, which I am still hoping to get working. The code that does work is called hashmap_parallel_threads. The test script has also been updated accordingly.

If anyone else want to shoot me some candidates for their favorite way of implementing transparent parallelization with map, I will add them to the catalog.

****************************

Hashmap here means essentially the same as the "map" half in MapReduce. It processes a set, where order doesn't matter. I have several hashmap functions in this code, two of which work, and one (the one that executes in parallel) which doesn't.

These "mapping" functions are given as an argument to the function builder flexygrep, which returns a grepping function. So, as a consequence two of my grepping functions work and one (the parallel one) doesn't.

If I can get hashmap_parallelto work, I'm thinking could theoretically use this to build other functions, like sort_parallel, permute_parallel, you name it. Sometimes, for efficiency reasons, this will make sense. A lot of the time it won't. Depends what your bottleneck is -- cpu, memory, network, disk io, etc. But the good news is that once the parallel mapping function -- or mapping functions -- work, you can just plug them in and try. A lot easier than writing threading code for all scenarios.

Again, the bit that I need to get to work -- but which will pay major dividends in maintainability when I do -- is hashmap_parallel.

Now, here's a little test output.

$ ./test_hashmap.plok 1 - parallel-y threadgrep worksnot ok 2 - parallel-y forkgrep works#   Failed test 'parallel-y forkgrep works'#   in ./test_hashmap.pl at line 21.ok 3 - serially executing code works1..3# Looks like you failed 1 test of 3.

And here's the code

test_hashmap:#!/usr/bin/perluse strict;use warnings;use Test::More qw( no_plan );use Data::Dumper;use Grep;my $slow_matches_b = sub { sleep 1;                           return unless $_[0];                           return 1 if $_[0] =~ /b/;                         };my $test_strings = [ ('blee','blah','bloo', 'qoo', 'fwee' ) ];my $matches;$matches = Grep::threadgrep( $slow_matches_b, $test_strings );ok( @$matches == 3, "parallel-y threadgrep works"  );# should get blee, blah bloo, but not fwee or qoo$matches = Grep::forkgrep( $slow_matches_b, $test_strings );ok( @$matches == 3, "parallel-y forkgrep works"  );$matches = Grep::slowgrep( $slow_matches_b, $test_strings );ok( @$matches == 3, "serially executing code works"  );Grep.pm:package Grep;use strict;use warnings;use Data::Dumper;use Map;# grep can be parallelized by building it on top of map_parallel# which uses forks, threads, distributed computations with MapReduce# or some such black magic# in some cases this may be faster, but not always,# it depends where your bottleneck is.# Whatever black magic is going on in the background,# by abstracting it out, the code we get is clean and easy to read.sub threadgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_threads(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub forkgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_forks(@_)};  return flexygrep($test_function, $map_function, $in_array);}# or you could do it in a non-forked/threaded/distributed/whatever# way, by basing it on the conceptually simpler function map_serial.sub slowgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_serialized(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub flexygrep {  my $test_function = shift;  my $hashmap_function = shift;  my $in_array = shift;  my $in_hash = Map::hash_from_array($in_array);  my $result_hash = $hashmap_function->($test_function, $in_hash);  my $out_array = [];  for my $key (keys %$result_hash) {    if ( my $out_true = $result_hash->{$key}->{out} ) {      push @$out_array, $result_hash->{$key}->{in}    }  }  return $out_array;}1;Map.pm:package Map;use strict;use warnings;# Black magic for doing stuff in parallel is encapsulated here# use MapReduce;use Parallel::ForkManager;use threads;# use threads::shared qw(is_shared);use DBM::Deep;use Data::Dumper;sub hash_from_array {  my $array = shift;  my $hash;  for my $index (0..$#$array) {    $hash->{$index}->{in} = $array->[$index];  }  return $hash;}# input is a funcion (eg, my $sub_multiply by ten = { return $_[0] * 10 } ), and# a hash like# my $input_values = { blee => { in => 1 },#                      blah => { in => 2}#                    }# output is a hash like#{ blee => { in => 1, out => 10 },#  blah => { in => 2, out => 20 }#}sub hashmap_serial {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  # hash keys are processed in whatever order  for my $key ( keys %$hash) {    my $in = $hash->{$key}->{in};    my $out = $function->($in);    #print "result for $in is $out\n";    $hash->{$key}->{out} = $out;  }  return $hash;}# does the same thing as hashmap_serial# but saves the value on the hard drive# (serialized in this context means a memory value gets put on the hard disk,# not to be confused with the sense of "serial as opposed to parallel"sub hashmap_serialized {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  use File::Path qw(mkpath);  my $dir="c:/tmp/map_serialized";  mkpath($dir) unless -d "$dir";  die "no directory: $dir" unless -d "$dir";  my $file="$dir/$$.db";  my $db = DBM::Deep->new( $file );  $db->{result}=$hash;  for my $key ( keys %$hash ) {    my $in = $hash->{$key}->{in};    my $out = $function->($in);    $hash->{$key}->{out} = $out;  }  #unlink $file;  #die "couldn't delete file" if -f $file;  return $hash;}# but uses threads to compute "out" values in a parallel way# doesn't work.sub hashmap_parallel_forks {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  return {};  use File::Path qw(mkpath);  my $dir="c:/tmp/map_serialized";  mkpath($dir) unless -d "$dir";  die "no directory: $dir" unless -d "$dir";  my $file="$dir/$$.db";  my $db = DBM::Deep->new( $file );  $db->{result}=$hash;  my $pm=new Parallel::ForkManager(10);  for my $key ( keys %$hash ) {    $pm->start and next;    my $in = $hash->{$key}->{in};    my $out = $function->($in);    print "in $in, out $out\n";    $hash->{$key}->{out} = $out;    $pm->finish;  }  $pm->wait_all_children;  print "hash: " . Dumper($hash);  #unlink $file;  #die "couldn't delete file" if -f $file;  #die "forkgrep result: " . Dumper($hash);  return $hash;}#workssub hashmap_parallel_threads {    my $function = shift;    my $hash = shift;    my @threads;    for ( keys %$hash ) {      my $in = $hash->{$_}->{in};      my $t = threads->create( sub { map_element($_, $function, $in ) } );      push @threads, $t;    }    #   wait for threads to return ( this implementation is bound by slowest thread )    my %results = map { %{ $_->join() }; } @threads;    #print Dumper \%results;    return {%results};}sub map_element {      my $key = shift;      my $function = shift;      my $in = shift;      my $out = $function->($in);      return { $key => {                        in =>  $in,                        out => $out                       }             };}1;

UPDATE: Seemingly relevant comments from Jenda at [id://580019]:

"You can only transparently paralelize map{} if the executed block is side-effect-free....."

I'm actually not sure if my code here is side effects free or not. Hm...

*****************************************************

Posts I'm looking at to see if I can use something there to get hashmap_parallel to work...:

[id://237089]

C++ vs. Perl mention on shygypsy.com (dhoss)

Hey all,

If you've been in the CB within the past 24 hours, you've probably at least heard of [http://shygypsy.com/farm/p.cgi|Funny Farm], a pretty sweet little game by Igor Naverniouk.

Looking around [http://www.shygypsy.com|his site], i found on his [http://shygypsy.com/acm/|UVA tools page] that he says, and i quote:

ShyGypsy labsAugust 4, 2006 02:19I'm playing around with some better ways of ranking people and problems. Here is a preview of a prototype. I have also discovered that I can easily write CGI scripts in C++. Perl is great, but I can get stuff done a hundred times faster in C++.

"I can get stuff done a hundred times faster in C++"? Did I miss something? If I recall correctly, comparatively it takes at least 5-6 lines more of code in C++ that in perl to write a simple "Hello world" program.

Example:

C++:

#include using namespace std;int main (){  cout << "Hello World!";  return 0;}

Perl:

#!/usr/bin/perl -wuse strict;print "Hello world!\n";

or, even less:

C:\>perl -e "print 'hello world!'"

I'm not trying to dig up an old Jihad, and I normally would just search the site for responses on this, but what could possibly justify him saying that? It could very well be that he's more comfortable with C++ and can simply code quicker in C++, logically that's sound because he's got 2 C++ courses he teaches at UofToronto, but I just can't fathom this.

Another possiblity is he has many code libraries in C++, mainly algorithms and such, and maybe that puts C++ on a more level playing field with perl since it eliminates a lot of low end coding. But do libraries really making a "100 times faster" difference?

Please, correct me, downvote me, chastise me if I'm off base and haven't done my research, but I just can't possibly see how coding in C++ could possibly be any faster than coding in perl

Your wisdom is appreciated monks.

P.S> I just threw this in meditations, move as needed.

meh.

Book update: "Exploring Programming Language Architecture in Perl" (billh)

Is it ok to post updates here? I've got no-one else to talk to :-)
Anyway assuming it's ok I've just finished the first draft of chapter 13on continuationsover at at billhails.net.
It's still quite rough, and it needs rearranging (move the "what's so coolabout continuations" bits up to the front so the reader feels moremotivated to read the rest,) and subtitling.
I'm slightly worried that I'm leaning rather too heavily on one source, the venerable and wonderful "Essentials of Programming Languages", but hopefully by the time I've finished I'll be able to present a more balanced account.
Any and all constructive criticism is welcome: anything I've missed,any points that are too laboured etc.
And enjoy of course.
PS There are now PostScript and PDF versions linked from the cover page.
Update: changed title as per recommendation

Bill H

perl -e 'print sub { "Hello @{[shift]}!\n" }->("World")'

See also: [id://578390] and [id://543564]

Edited by [planetscape] - added "see also" links ( keep:0 edit:14 reap:0 )

Perl needs Zend (EvanCarroll)

I feel that it is important to say, I have down voted every response to this for lack of utility. Please post a response that has content, and not just mindless perl fanaticism.

Fellow monks, I think I've finally answered the question as to why PHP is, generally speaking, more successful. I have to point to their corporate backing which largely comes from Zend. Zend offers above and beyond the free PHP stuff other commercial products too, and they take the proceeds from these product, and dump them back into the language. Zen actually produces seven products all of which can be found on their PHP powered website at zend products. Zend even produces an Optimizer which makes all of your php code run up to "25times faster." That's really good news for php, especially considering php is pretty fast anyway.

I mean look at this, the Zend core actually runs Oracle and has been ported to IBM. Which makes me think maybe the core being more full featured than PERL's is the reason for PHP's success. Obviously, it has some advantages; or why would big blue, and Oracle have adopted it? Maybe its the additional free support Zend provides that the Perl foundation doesn't, IBM support and Oracle support.

Zend even offers certificates, and they are "confidant you will pass the first time." That's probably because the language has no learning curve whatsoever. You are almost born knowing it! And, if you get both certificates you get a $25 discount. Lets say though you find php cryptic, for 769.00 you can get Zend professional training. Another important thing Zend offers is consultation, which isn't readily available in the Perl community.

The solution to this problem is simple, Perl needs some really big corporate backing too, like either Google, Yahoo, Microsoft or Apple. Maybe there is some way we can get Larry to hand over the project to Zend. Or maybe some day Zend will build a Perl optimizer for us that will run our applications 25 times faster. We probably also need these big corporate guys to run on our core as well, at least Apple; surely the PERL core can handle them.

Evan Carroll
www.EvanCarroll.com

RFC: Perl meta programming (bennymack)

Dear Monks,

Herein lies my latest half-baked Perl extension. This one is for reducing the amount of boiler-plate code necessary for writing Perl subs.

As per usual, I will jump right to the code which I always assume will speak for itself ( but rarely does ).

Here is an example of a module that uses my "handy" new module.

SomeTestPackage2.pm

package SomeTestPackage2;use strict;use warnings;use Data::Dumper;use constant IVARS => qw[$none $href $aref $non_empty_aref];BEGIN {use base qw[SomeAttributes2];__PACKAGE__->import( IVARS );}use constant custom_aref => q{    confess( '!!!package!!! !!!ivar!!! must be an array ref' ) if 'ARRAY' ne ref !!!ivar!!!; };sub self_sub : Method( qw/ $none :none $href :href $aref custom_aref $non_empty_aref :non_empty_aref / ) { #    return ( ref $self, ref $href, ref $aref, scalar @{ $non_empty_aref } );}1;

The previous code snip shows how the meta programming interface works for the most part.

First, you specify a list of instance variables, or whatever, that you would like to pull into subs. This pre-declaration is necessary for syntax reasons.

Next, you can specify "custom" attributes as package subs if you desire. The difference between a custom attribute and a canned/common one is that canned attributes begin with a colon and custom do not.

Now, in your sub attribute you specify a list of variables you'd like pulled into your sub along with any attributes you'd like called on them.

This is the test script running the prior code

test2.pl

#!/usr/bin/perluse strict;use warnings;use Test::More qw[no_plan];use SomeTestPackage2;do { # Successfull call to self_sub    my( $test1 ) = bless( {}, 'SomeTestPackage2' );    my( @self_sub ) = $test1->self_sub( 'hi', { }, [ ], [ 1 ] );    ok( $self_sub[0] eq 'SomeTestPackage2', 'ref $self' );    ok( $self_sub[1] eq 'HASH',             'ref $href' );    ok( $self_sub[2] eq 'ARRAY',            'ref $aref' );    ok( $self_sub[3] > 0,                   'non empty array' );};

Here is the output of running the previous script.

$ perl test2.plok 1 - ref $selfok 2 - ref $hrefok 3 - ref $arefok 4 - non empty array1..4

And finally, here is the code that is compiled from the "Method" attribute on the "self_sub".

sub {    local *__ANON__='SomeTestPackage2::self_sub';    my( $self, $none, $href, $aref, $non_empty_aref ) = @_;    confess( 'SomeTestPackage2::self_sub $non_empty_aref needs a non-empty array reference' ) if 'ARRAY' ne ref $non_empty_aref or not @{ $non_empty_aref };    confess( 'SomeTestPackage2 $aref must be an array ref' ) if 'ARRAY' ne ref $aref;    confess( 'SomeTestPackage2::self_sub $href needs a hash reference' ) if 'HASH' ne ref $href;    confess( 'SomeTestPackage2::self_sub needs a SomeTestPackage2 reference' ) if ref $self ne 'SomeTestPackage2';    package SomeTestPackage2;    use warnings;    use strict 'refs';    return ref $self, ref $href, ref $aref, scalar @{$non_empty_aref;};}

That's it! Is it crap?

RFC: Perl Testing -- How to Introduce to a team (NovMonk)

Wise monks,

My company is facing a transition. We have been writing perl scripts-- little snippets of code to get certain text outputs-- or modifying longer programs (adding logic to affect printed text output). We have been using shell scripts to drive our processes-- the most ungodly mess you can imagine, and which varies from job to job based on who set it up, and how many years ago. The good news: we have recently purchased some new software and we are retooling our existing processes to use it, and moving from flat file to database (MySQL) data.

As we talked about designing this new framework, I timidly suggested adding testing modules to the process Up Front, and as a reward, I was tasked with researching testing, and training our developers to do it. The closest any of us have been to testing is adding print statements to code while debugging-- one of us (not me) actually uses the Perl debugger with some facility.

It's not a matter of convincing them we need to be testing-- everybody's sold on that. We are all just getting overwhelmed with where to start. I've read the posts on testing here found through Super Search, and I have printed some out to share. I've gone to CPAN and printed off several of the modules (Test::Simple, Test::More, the tutorial on testing, etc). I've been reading books on testing and software construction in general: Perl Medic, Perl Debugged, Perl testing: A Developer's Notebook , and Code Complete.

One problem is: most of us are self taught, or have only the haziest ideas about Object Oriented Programming, or writing/ using modules at all. Some things in our process are obvious- you want to test that needed files are where you expect them to be, are in the right format, and that the program generates the expected outfiles and sends them where they need to go. But-- What else? If youre supposed to design tests before you code, what else should we be looking for?

My plan for training is to pass out some of the CPAN material on Test::Simple and Test::More, and the posts from here, and parts of the books, some of which we are buying as a company. Im also going to touch on Perl Tidy and Perl Critic, as ways for us to enforce these coding standards were agreeing on. What Im looking for from you guys is other suggestions, things I might think about, ways youve presented testing to programmers youve mentored over the years. Things you wish youd known when you were starting out. Testing procedures you inherited when you started with a company, that you wish you could change.

Weve this fantastic opportunity to design procedures and policies that will make our lives easier, make us better programmers, and make our company, and us, lots of money. And that is also a little overwhelming.

Many thanks in advance,

NovMonk

OO in Perl 6 (Scott7477)

In his latest Apocalypse, Larry Wall indicates that Perl 6 "uses . instead of -> to dereference an object" due to the idea that "The use of arrow where most of the rest of the world uses dot was confusing." I think that this is a Very Good Thing, for a couple of reasons.

One reason is less keystrokes; going from the dash key to the ">" key has always seemed a little bit of a pain:) The second reason is that this change will likely facilitate introducing people to using Perl. This will be one less thing for folks new to Perl to go "Huh?" at when you show them some code.

(OT) X terminal output speed on linux (zentara)

This is just a curious discovery I made while testing the urxvt

I ran this simple script, to print out 500,000 lines and timed it.Both the xterm and the urxvt have 20k scrollback buffers.

#!/usr/bin/perl use warnings;use strict;my $start = time;for (1..500000){print "$_\n";}my $end = time;my $diff = $end - $start;print "$diff\n";

To my utter amazement ( and I'm still wondering if it's an xterm bug), the xterm took a whopping 1067 seconds, while the urxvt only took 19 seconds.

I would be interested in hearing the results from other people, on their xterm speed, and maybe k-terminal, etc.

Anyone have clues as to why this occurs?

I'm not really a human, but I play one on earth.Cogito ergo sum a bum

The state of audio processing with Perl (Joost)

Hi there, fellow monks!

My name is Joost and I'm going to talk about audio processing in perl.

I've been playing with "live" (not really realtime, but close enough) audio generation/processing in perl for some time now and I'm not really impressed by the code on CPAN to do that.

For one thing, there appears to be no portable module to play or record audio with any kind of precision. I mean just some code to throw an audio buffer at the soundcard (OIW, in-process handling of data vs playing an audio file). Sure there's Audio::Play but that only supports monophonic sound and doesn't record.

Also, most audio modules use incompatible formats for their data and lots of them are badly documented and/or haven't been updated for years. In other words, coupling these modules is a mess (and slow).

More issues: reading and writing different audio file formats. I've taken a stab at that just now - but it still lacks some important formats.

Here are some of the things I feel are needed:

One audio data interchange format. Audio::Data seems to me to be a good starting point, but it lacks low-level access (i.e. a C level interface and packed float/double strings for fast processing).
A portable audio playback/recording library. Preferably one that will handle everything that Audio::Play handles but with unlimited channels and a JACK interface.
Some kind of processing API that allows coupling of perl & XS code. Something like my Audio::LADSPA module, but more flexible and with an easier Perl interface.

So here's my question: is anyone interested in this kind of thing? By myself I might be able to get this stuff done in a year or so, considering the time I've got to spend, but I'd really like people to supply questions, burning needs, code and/or documentation to get this thing on the right track.

Also, if you are writing any audio modules on CPAN, I'd like to know about it.

Please let me know I'm not the only one interested in this kind of thing :-)

In other words: audio people of perl, let's hear your crazy ideas!

Cheers. J.

"What should it profit a man, if he should win a flame war, yet lose his cool?"

RFC: more of that almost-book (billh)

Hi,Tell me what you think:
Exploring Programming LanguageArchitecture in Perl.
Some of you may remember I posted a link to this many moons ago.
I haven't really given up on it but it's become something of a spare-timeproject, I've finished the object-oriented extension and started writingthe chapter on continuations. I've corrected a lot of bloopers in the earlierchapters and I've also learned far more about XML andCSS than anyone should have to :-), but at least it looks prettier now.

Bill H

perl -e 'print sub { "Hello @{[shift]}!\n" }->("World")'

GUI toolkit choice (jbert)

Hi,
A quick search only showed a Tk/Gtk discussion from 2001, so I hope I didn't miss something and this is a FAQ. Also, I'm hoping that we can discuss things without having a religious war about which is 'best'.

That said, I'm interested in finding out more about the various options for GUI coding, particularly - but not exclusively - from perl.

The main options I know of are:

Gtk+ - originally written for the GIMP, used by the GNOME people and some other apps
QT - written by Trolltech, used by the KDE people and some other apps
Tk - from Tcl and ported to perl early on
WxWindows - seems good but I know little of it (the DrScheme IDE is written in it and seems decent)

All of these are cross-platform (at least Linux and Windows and I would guess Mac and 'general unix/X windows as well').

I've used Gtk+ from perl and C++ on Linux with good results and the same scripts runs on Windows (once Gtk is installed), which is nice.

If anyone has experience of more than one of these toolkits and can provide some thoughts on what they see as the most significant strengths and weaknesses of them I'd be grateful.

And I'd hesitate from using this thread to ask the popularity of the different toolkits, but perhaps that would make a good poll?

The purpose of testing (g0n)

Testing has been a bit of a problem for me over the last few months, as some will have heard from my CB rants. Not the 'make test and watch the TAP output come pouring out testing' - oh dear no. For much of the last 8 months, I've been doing a great deal of manual testing, of the 'type these commands, compare 50 fields in the database output with the 50 fields printed in the test script, putting a p (pass) or f (fail) next to each one, sign and date, then go on to the next one of the 150 tests' type. Yuk! Human beings did not evolve to do this kind of thing.

So I enthused about automated testing, tried to get my colleagues on side (with a degree of success), but there is one major stumbling block. The Quality Assurance team. Correctly, they're independent. Unfortunately, they don't permit automated testing unless it's done with an approved, validated (company validated that is) automated testing tool (TestDirector, for example). They're also entirely non technical, and really only concerned with the quality (in terms of change tracking, consistency etc) of documents. The net result of this is that testing is an extraordinary time overhead, and we have to think carefully about what tests to run for a given release. This means that testing is not as thorough as it could, or should be, and bugs creep through. Not as many as you might expect in this situation, but nevertheless more bugs find their way into production than I consider acceptable.

This stalemate has been going on for some time. Years actually. Then on monday something happened. A big, fat bug in some of my code showed up in production. Embarassing. This bug meant that I now have to run a manual report daily for the next couple of weeks until we can patch, to take the place of the automated report that I broke. Embarassing and irritating, especially since another bug had been emergency fixed that morning.

At that point I realised that, just like the QA people, I'd lost sight of the real issue - testing is about finding bugs, not filling in forms. If the formal, QA approved testing is less thorough than it should be, we have to make sure that the code gets properly tested some other way.

So I got to work writing unit tests with [cpan://Test::More].

3 days work later I've got one of the components up to 50% test coverage and found 3 bugs in edge cases that have never showed up in production. Unfortunately we probably can't test everything this way, since the perl code is only one component, running in an embedded perl interpreter inside a proprietary application. Integration testing still needs to be done the old way, so our test overhead has gone up by the amount of effort needed to write unit tests, but at least the chances of bugs getting through is reduced.

Another advantage of testing with the Perl testing modules is the availability of [cpan://Devel::Cover]. Because the unit testing is informal and unvalidated, test cases can be added any time. If someone has a few minutes spare, a quick run of the test suite with [cpan://Devel::Cover] will show up opportunities for improving the testing.

Something else I'd lost sight of is the fact that we primarily want to test our code, not someone elses. A lot of our code depends heavily on [cpan://Net::LDAP], so the need to provide a correctly configured directory server looked like a barrier to automated testing. However, end to end integration testing covers the 'get data back from the directory server' test case. If there's no directory server easily available for unit testing, we can invade the dependency's name space to let us test our own code:

use strict;use warnings;use Test::More;require 'MyCode.pl';*Net::LDAP::bind = \&ldapbind;*Net::LDAP::new = \&ldapnew;MyCode::bindToLDAP("hostname","port","cn=binddn","password");sub ldapnew{my $host = shift;cmp_ok($host,"hostname:port","Check that Net::LDAP::new receives the right params");}sub ldapbind{my %params = @_;my %comparison ={dn=>"cn=binddn",password=>"password",};is_deeply(\%params,\%comparison,"Check that Net::LDAP::bind gets the right params");}

I'm hoping I can get the vendor of the core application to give us information on externally accessing the test functions in their application via XS, so that we can extend the unit tests to include the application config. I'm not hopeful on that front, but it's worth a try.

One final note: in the mindless drudgery of manual testing, I'd also forgotten how much fun one can have writing tests to try and break things :-)

--------------------------------------------------------------

"If there is such a phenomenon as absolute evil, it consists in treating another human being as a thing."
John Brunner, "The Shockwave Rider".

Make sure you're solving the right problem (talexb)

This is a followup on some recent posts of mine .. in case anyone new to Perl wonders if it is ever used for real work.

My web application handles document processing (conversion from PDF to XML), and it's currently running on a dozen servers, the busiest of which is used by two teams, one here in Toronto and the other in Mumbai, India. The work day starts about midnight local time (930am in Mumbai) and goes to about 8 or 9pm, and lately we've been running 7 days a week to keep up with demand. If things go South, it lands in my lap. I hate pages at 4am, so I do what I can to avoid that situation.

Anyway, recently, I wrote about a problem I'd been having with a sleepy CGI, and eventually rediscovered the excellent tool strace thanks to [jbert|some] [sgifford|respondents]. But that led me to my next problem, as to why the CGI would be freezing at

select(5, [4], [], [4], NULL

as strace was showing me. After some more research, I discovered this was the connection to the database. Could it be as simple as the CGI waiting for the database that was causing the problem?

Oh boy. The answer is yes, as it turns out. And the CGI was not 'going to sleep' -- it was continuing to run, but was patiently waiting for the database, not sleeping.

And the solution to the database slowdown (PostgreSQL, in my case) was the simple application of

ANALYZE VERBOSE DOCUMENTS;

and the response time for the main query went from 20 seconds to about 550ms. Yay!

Since I've seen this performance problem before, I'm now going to keep an eagle eye on the system today and find out how long it takes before the performance starts to drop, then set up a cron job to ANALYZE the suspect table again, most likely every four hours or so.

This is all a result of watching my system closely, something I think is extremely valuable in any Engineering job -- staying on top of the performance of your system is very important in my current situation of developer, supporter and maintainer.

The moral of the story is, follow the data (I know, it sounds like CSI). Processes very rarely 'go to sleep' unless they're told to. Where's the CPU time (or bandwidth, or your other resources) going? Follow that lead, and you'll find the answer to your problem.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Check out g2 .. a new graphics interface (zentara)

I just saw g2 today on http://freshmeat.net. It is a c lib for generating 2d graphics. I thought it would be worth mentioning here, because it contains a perl module interface, and it works quite well.

It's big advantage is that it can output to multiple types simultaneously. See screenshots at above link.

I'm not really a human, but I play one on earth.Cogito ergo sum a bum

Automated regression testing (rinceWind)

Where I am working, we have a requirement to provide an automatic regression test suite for an existing live application. The application is C++, not perl (though there is some perl glue already deployed for application monitoring). I'd quite like to do this using Perl, as I think the language is ideally suited. I also have permission to use any CPAN modules of my choice. I will still need to convince colleagues of the wisdom of this choice.

I use the Test::More and Test::Harness mechanisms in unit tests for modules used by the application glue. But, the full regression test is a much bigger prospect. On the automation side, I will need to be doing the following:

Starting and stopping application components (daemons)
Interfacing to the application through command line tools
Trawling and tailing log files to detect events
Database access, including updates

It occurs to me that there is a fundamental difference between an automation "step" and a test. If a test fails, I want the script to carry on and run more tests. If a step fails, I want it to pause requiring manual intervention. The pause could eventually time out, bailing the run, allowing the whole suite to be run in "lights out mode" unattended over a weekend.

My plan is to put the automation steps in first, and add the tests afterwards.

I'd be very interested in hearing from anybody who has attempted or achieved anything like this before. What tools have already been written that can help me? What problems and gotchas should I be careful to avoid?

--

Oh Lord, wont you burn me a Knoppix CD ?
My friends all rate Windows, I must disagree.
Your powers of persuasion will set them all free,
So oh Lord, wont you burn me a Knoppix CD ?
(Missquoting Janis Joplin)

Finding "Accidental Contributions" with Google Code Search (jasonk)

Someone mentioned today on digg that it was relatively easy to find thousands of developer email addresses by searching for a suitable regexp on Google Code Search. Since I was already bored (hey, I was reading digg, it couldn't have been too busy a day) I decided to search for myself and see what should pop up.

The first couple of pages were unsuprising, mostly links to my modules on CPAN, but then I stumbled across something interesting. In a little program I had never heard of was a copy of Geo::ShapeFile. The fact that they were distributing my module along with their program isn't a big deal, it's licensed under the Artistic license, and they credited me and everything else, what caught my eye about it was this comment in the LICENSE file:

The version of Geo::Shapfile in this distribution includes fixes for machines of different endianness, not yet available on CPAN.

I downloaded the source and perused it a bit and discovered that the author was even kind enough to include RCS files for the libraries that had been modified, which made it very easy to figure out what local changes he had made, and to find that indeed there were fixes for some endian-related issues in the module which have vexed me for quite a while (sadly my current job does not involve mapping, so I don't have as much time to put into this module as I would like). The most interesting part was that the solution he used seems to be better than my own solution had been, and appears to have been written before the problem was first reported to me.

So, have you checked Google to see what people might have neglected to contribute to your projects?

Along the way I also found some other interesting tidbits that I hadn't known...

And last (and certainly least) a reminder that I was even more of a geek in college

We're not surrounded, we're in a target-rich environment!

Collective Hacking Projects: Let's Hear About Them! (jkeenan1)

About two years ago this time I helped organize a Perl Seminar NY contingent of the Phalanx project. When I pitched the project to our members, I emphasized the joys of collective hacking. And it proved very enjoyable indeed (and profitable for the bar where we met to do this hacking!)

We didn't do as much collective hacking in the 2005-06 season, but with our 2006-07 Perl Seminar NY season about to begin, I'd like to rev up some interest among our members.

I know of some of the collective hacking projects going on in the Perl community right now, and I'll be participating in the Chicago Perl hackathon next month. But just in case I've missed any, I'd like to ask the Monks to respond to this Meditation by posting ongoing projects (e.g., Vanilla Perl, Module::Build) and their weblinks.

And, for extra credit, if you think that any of these projects could be broken up into components suitable for face-to-face hacking by local Perlmonger groups, please so indicate.

I like online projects, but I don't like to hack or drink alone.

Thanks in advance.

Jim Keenan

PPM - It's a GUI!! (mikasue)

I recently installed ActivePerl 5.8.8 Build 819 on a desktop. After reading the release notes, I was anxious to explore the new version of the Perl Package Manager. At first the GUI was a little confusing becuase it's not clear what is being listed. However, after playing with it, I think it's great.

Listing and installing packages are much easier tasks to complete. Adding repositories are just as easy as typing in the url and giving it a name. I can see all packages installed or all packages available for install. With the click of a button I can upgrade and verify packages.

All is not lost for those who like the command line shell. Doing a ppm repo will list all your repositories in a nice borded table. Other commands are available also just do a ppm help at the command line.

I like the new version of PPM. What do you monks thinks about it?

Oracle OpenWorld Pilgrimage (Thwack)

I will be traveling to Oracles OpenWorld conference in San Francisco (Oct. 22-26 2006). I am curious if any other Perl Monks and/or Perl related programmers will be there. I would like to meet up and hi and maybe do a lunch or something of that sort. It would be fun to meet other developers/administrators that use or focus on Perl with Oracle.

Startups Ideas from Perlmonks (monkfan)

Having visited many social networking sites like reddit, del.icio.us, Technorati, Digg, makes me think Perlmonks in some sense are far more superior than those guys.

There are too many features that is very fundamental which Perlmonks has that they don't have, e.g. Chatterbox, Bookmarklet, voting systems, and the closely knitted comunity (my knowledge is blatantly pale wrt PM wealth of features).

Do you guys have ever thought, what kind of business idea one can learn from our Perlmonks?

Regards,
Edward