perlmeditation

Polynomial JAPH revisited (tweetiepooh)

In my [id://577605|earlier submission] I showed a means of JAPHing using a polynomial generated by Excel. Here now is some code to calculate the equation from a string all in Perl.

It works fine with the JAPH string upto 19 characters above which I think the limitations on internal maths stuff breaks. It's not obfusctation at this point and suggestions to get it work with the full 25 characters in JAPH (include line feed) would be fun. Note the code has been cobbled together on a Friday afternoon and I've made no attempt to make it elegant

#!/usr/local/bin/perl -wuse strict;use Math::Polynomial;my %h = ();my $s = "Just another Perl h";# put string into a hash {0=>J, 1=>u ...)map { $h{$_}=ord(substr($s,$_,1))}(0..length($s)-1);# use the library to get the polynomial factorsmy @p = split / /, Math::Polynomial::interpolate(%h);# generate the calculation string from factorsmy $p;my $i = length($s)-1;map{ $p .= "+$_*\$i**$i";$i-- } @p;# tidy the string, remove $i**0 (=1)# add .5 to total for rounding purposes$p =~ s/\+\-/\-/g;$p =~ s/.*\(//;$p =~ s/\).*/.5/;print "$p\n";# test the equationfor $i (0..length($s)-1) {print chr eval $p;}

Update

Links and spelling corrected as per chatterbox

iThreads for OOP (Zukoff)

I've investigated what modules CPAN have about ithreads. Some of them are too old, some - doesn't work on Win32 as good as on *nix. I was impressed with:Perl Scalable ithreaded Component Helper Extensionshttp://www.presicient.com/psiche/

A faster?, safer, user transparent, shared variable "locking" mechanism. (BrowserUk)

In perl, all user data is stored in scalars (SV*s).

What if the user mutable shared data, shared scalars were allocated from a read-only memory section?

Every time the user attempted to write to them, an exception would be raised.

When an exception occurs, the operation, whatever it may be, is re-done within the auspices of a critical section having temporarially set the section read-write. It is set back to read-only before the critical section ends.

As I understand it (on x86), the read-only/read-write status of a section of memory is an attribute of its LDT entry. These are read and written using single opcodes (SLDT/LLDT) and so the process of switching between read-only to read-write and back ought to be pretty quick.

The steps are:

enter exception handler.
enter critical section.
set the attributes of the shared data memory section to read-write.
reattempt the excepting operation (opcode?)
reset the shared data memory section read-only.
exit critical section
exit exception handler
continue with original program at the next operation.

As a critical section cannot be preempted, this effectively locks the shared data without the need for semaphores.

The removal of semaphores from the picture, vastly reduces the complexity of the problem, and vastly reduces the overhead of checking for locks.

The absence of the requirement for semaphores means that the same (existing) code for manipulating non-shared SV*s can also be used for shared SV*s, with no impact on performance for operations on non-shared data.

All locking becomes redundant. And serialisation of write accesses to shared data is dealt with entirely by a single mechanism that is controlled by the CPU itself, that of raising exceptions.

These will only occur when writing. And only when writing to shared data.

All other accesses are entirely free of any impact attributable to the provision of shared data.

The only other impact is that when variables are declared shared, they get allocated from the shared (most times, read-only) memory section.

Can this work?
Would it be faster?
Could it be retro-fitted to perl 5?
Is there any merit in it for perl 6 (or Parrot or other implementation)?

Win32/win64 certainly have all the primitives to implement this. I suspect, but do not know, that many/most other OSs (on x86 at least) have similar primitives in order to implement it. What about other platforms/hardware?

I do not have the requisite XS/perlguts skills to try it, or reach a conclusion on whether it could be done there. So I am throwing the idea out there to see what pople think?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

OT: Improving IE7 SDK Docs (was Let's suppose...) (footpad)

The recent release of IE7 has gotten me to thinking. (Always a dangerous sign.)

Let's suppose you could set aside the political BS and actually talk to the documentation team for Internet Explorer's SDK, the folks that document IE's support for HTML, CSS, and other browser idiosyncracies. Let's say you could tell them what you didn't like about the existing documentation and could tell them what would make the documentation "pop."

Mind you, this isn't a request to bash IE; that's been done quite effectively elsewhere. Instead, it's a serious attempt to solicit feedback from a community that has typically not been consulted with respect to this documentation set.

Nor is it a request for feature support in the next release of IE. Yes, we all know it would be sweet if IE would support ACID2; however, today it doesn't and this isn't that sort of thread. This is strictly a focus on the developer documentation. What's confusing, what's missing, what would make that material a useful resource for you?

(And before you bash me for being off-topic, this does affect those of us using perl to write Web 1.0-style CGI scripts. There is an audience that I feel has been left out of the discussion and wonder what, if granted the opportunity, what feedback that community would provide.)

--f

Spot the bug! (tlm)

Just got my butt chomped by this one, so I thought I'd share.

What's wrong with this function?

sub foo {  return eval { bar( shift ) } || 0;}

The answer does not depend on the definition of bar. E.g. let bar be

sub bar { return shift; }

To see what's wrong, try this:

print foo( 3 ), "\n";$@ = 3;print foo( $@ ), "\n";__END__30

The moral of the story: never use @_ inside an eval.

the lowliest monk

Using DBM::Deep and Parallel::ForkManager for a generalized parallel hashmap function builder (followup to "reducing pain of parallelization with FP") (tphyahoo)

This is a followup to

Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison)

In this post, I was attempting to create numerous variations of a function builder I'm calling hashmap_parallel.

This function builder should theoretically allow me to implement, for example, many of the utilities in List::MoreUtils in a transparently parallel way. Eg, instead of using flexygrep to build up a parallel grepping function on top of a parallel mapping function and an element test, as shown below, I would use flexy_any, flexy_all, flexy_pairwise, flexy_mesh, or what have you. These could all build on top of diverse custom hashmap_parallel, which you could plug in at will depending on your resources and the paricular bottleneck for the computation you are faced with.

So, if you have multiple processors, you can take advantage of that without cluttering up your code. And if you happen to have a 1000 commodity-box cluster, perhaps you too will one day be able to sort a 10^10 element array in under two minutes, with a simple call to distributed_sort($cmp_function, $my_big_list)

Since posting the above meditation, I have been able to implement hashmap_parallel in two ways.

1) threads (with help from LanceDeeply)
2) forks, along with multiple DBM::Deep data stores (using Parallel::ForkManager on tilly's advice, using DBM::Deep because... well, I couldn't think of any other way to do it that would work for any type of data.)

In addition, I have a simple "serial hashmap" function that I can plug into my "flexygrep" grep builder, as a sanity check.

At present, the threading solution is faster. I'm attributing this to the multiple hard drive read/writes that are required with the multiple DBM::Deep solution.

I was hoping to also implement a solution using a single DBM::Deep db and locking, which should at least reduce the read time. However, I was unable to do so. My test is to match a collection of letters against /[abc]/, where this test is made after sleeping for one second (we delay so that there is an artificial "reason" to want to take advantage of parallelism). There should be 9 matches in each case.

perl test_hashmap.pltest strings: a b c a b c a b cok 1 - threads, matches: 9 -- a a b c c b b a c -- should be 9ok 2 - threads, time elapsed: 2, wanted max seconds: 2ok 3 - fork with many dbm::deep dbs, matches: 9 -- a a b c c b b a c -- should be 9ok 4 - fork with many dbm::deep dbs, time elapsed: 1, wanted max seconds: 2Use of uninitialized value in join or string at test_hashmap.pl line 48.not ok 5 - fork with 1 dbm::deep db, matches: 7 -- a b b b  c c -- should be 9#   Failed test 'fork with 1 dbm::deep db, matches: 7 -- a b b b  c c -- should be 9'#   in test_hashmap.pl at line 48.ok 6 - fork with 1 dbm::deep db, time elapsed: 1, wanted max seconds: 2test strings: a b c d e f g h i j k l m n o p q r s t u v w x y z a b c d e f g h i j k l m n o p q r s t u v w x y z a b c d e f g h i j k l m n o p q r s t u v w x y zok 7 - threads, matches: 9 -- a b a c c b c b a -- should be 9ok 8 - threads, time elapsed: 2, wanted max seconds: 2ok 9 - fork with many dbm::deep dbs, matches: 9 -- a b a c c b c b a -- should be 9not ok 10 - fork with many dbm::deep dbs, time elapsed: 9, wanted max seconds: 2#   Failed test 'fork with many dbm::deep dbs, time elapsed: 9, wanted max seconds: 2'#   in test_hashmap.pl at line 49.not ok 11 - fork with 1 dbm::deep db, matches: 8 -- b c a c b c a b -- should be 9#   Failed test 'fork with 1 dbm::deep db, matches: 8 -- b c a c b c a b -- should be 9'#   in test_hashmap.pl at line 48.not ok 12 - fork with 1 dbm::deep db, time elapsed: 9, wanted max seconds: 2#   Failed test 'fork with 1 dbm::deep db, time elapsed: 9, wanted max seconds: 2'#   in test_hashmap.pl at line 49.1..12# Looks like you failed 4 tests of 12.

In addition to the test failures for the single DBM::Deep file shown above, about 25% of the time the single DBM::Deep file case seems to lock up, and the test never completes.

I would be very interested in receiving advice for how to do hashmap_paralell with a single DBM::Deep. And if there are suggestions for other strategies I could try out, or changes to my function builders that would speed things up, I'll be much obliged.

Thanks.

test_hashmap.pl:#!/usr/bin/perluse strict;use warnings;use Test::More qw( no_plan );use Data::Dumper;use Grep; # several different versions of grep, built up functionally.my $slow_matches_b = sub { sleep 1;   return unless $_[0];   return 1 if $_[0] =~ /[abc]/; };my $justafew_letters = [ my @letters = (('a'..'c') x 3) ];my $lotsa_letters = [ (('a'..'z') x 3) ];for (     $justafew_letters,     $lotsa_letters    ){  testem($_)}sub testem {  my $test_strings = shift or die "no test strings";  print "test strings: @$test_strings\n";  my $paralell_tests = [ {testname => 'threads',function => sub { Grep::threadgrep( $_[0], $_[1]) } },       {testname=> 'fork with many dbm::deep dbs',function => sub { Grep::fork_manydbs_grep($_[0], $_[1]) }       },       {testname => 'fork with 1 dbm::deep db',function => sub { Grep::fork_onedb_grep($_[0], $_[1]) }       },       #{testname => 'simple serial map',#function => sub { Grep::slowgrep($_[0], $_[1]) }       #}     ];  my $max_seconds_parallel=2; # parallel execution should speed things up  for (@$paralell_tests ) {    my $timestarted=time;    my $test_name = $_->{testname};    my $matches = $_->{function}->($slow_matches_b, $test_strings);    my $timeelapsed=time - $timestarted;    my $num_matches = @$matches;    ok( $num_matches == 9, "$test_name, matches: $num_matches -- @$matches -- should be 9");    ok( $timeelapsed <= $max_seconds_parallel, "$test_name, time elapsed: $timeelapsed, wanted max seconds: $max_seconds_parallel");  }}#my ($timestarted, $timeelapsed);Grep.pm:package Grep;use strict;use warnings;use Data::Dumper;use Map;# grep can be parallelized by building it on top of map_parallel# which uses forks, threads, distributed computations with MapReduce# or some such black magic# in some cases this may be faster, but not always,# it depends where your bottleneck is.# Whatever black magic is going on in the background,# by abstracting it out, the code we get is clean and easy to read.sub threadgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_threads(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub fork_onedb_grep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_forks_onedb(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub fork_manydbs_grep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_forks_manydbs(@_)};  return flexygrep($test_function, $map_function, $in_array);}# or you could do it in a non-forked/threaded/distributed/whatever# way, by basing it on the conceptually simpler function map_serial.sub slowgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_serialized(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub flexygrep {  my $test_function = shift;  my $hashmap_function = shift;  my $in_array = shift;  my $in_hash = Map::hash_from_array($in_array);  my $result_hash = $hashmap_function->($test_function, $in_hash);  my $out_array = [];  for my $key (keys %$result_hash) {    if ( my $out_true = $result_hash->{$key}->{out} ) {      push @$out_array, $result_hash->{$key}->{in}    }  }  return $out_array;}1;Map.pm: package Map;use strict;use warnings;# Black magic for doing stuff in parallel is encapsulated here# use MapReduce; -- not yet, but it's on the list.use Data::Dumper;sub hash_from_array {  my $array = shift;  my $hash;  for my $index (0..$#$array) {    $hash->{$index}->{in} = $array->[$index];  }  return $hash;}# input is a funcion (eg, my $sub_multiply by ten = { return $_[0] * 10 } ), and# a hash like# my $input_values = { blee => { in => 1 },#                      blah => { in => 2}#                    }# output is a hash like#{ blee => { in => 1, out => 10 },#  blah => { in => 2, out => 20 }#}sub hashmap_serial {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  # hash keys are processed in whatever order  for my $key ( keys %$hash) {    my $in = $hash->{$key}->{in};    my $out = $function->($in);    #print "result for $in is $out\n";    $hash->{$key}->{out} = $out;  }  return $hash;}# does the same thing as hashmap_serial# but saves the value on the hard drive# (serialized in this context means a memory value gets put on the hard disk,# not to be confused with the sense of "serial as opposed to parallel"sub hashmap_serialized {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  use File::Path qw(mkpath);  my $dir="c:/tmp/map_serialized";  mkpath($dir) unless -d "$dir";  die "no directory: $dir" unless -d "$dir";  my $file="$dir/$$.db";  my $db = DBM::Deep->new( $file );  $db->{result}=$hash;  for my $key ( keys %$hash ) {    my $in = $hash->{$key}->{in};    my $out = $function->($in);    $hash->{$key}->{out} = $out;  }  #unlink $file;  #die "couldn't delete file" if -f $file;  return $hash;}# uses many DBM::Deep stores, along with forks# works# but slowsub hashmap_parallel_forks_manydbs {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  use File::Temp qw(tempdir);  my $dir = tempdir();  use Parallel::ForkManager;  use DBM::Deep;  my $pm=new Parallel::ForkManager(10);  for my $key ( keys %$hash ) {    $pm->start and next;    my $in = $hash->{$key}->{in};    my $out = $function->($in);    my $file="$dir/$key";    #print "file: $file\n";    my $db = DBM::Deep->new( $file );    $db->{result}=$out;    #print "in $in, out $out\n";    $pm->finish;  }  $pm->wait_all_children;  for my $key ( keys %$hash ) {    my $file="$dir/$key";    my $db = DBM::Deep->new( $file );    defined( my $out = $db->{result} ) or die "no result in $file for key $key";    $hash->{$key}->{out}=$out;  }  #print "hash: " . Dumper($hash);  return $hash;}# tries to use one locked DBM::Deep store, along with forks# doesn't worksub hashmap_parallel_forks_onedb {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  use File::Temp qw(tempfile);  my ($fh, $file) = tempfile();  use DBM::Deep;  my $db = DBM::Deep->new(                file => $file,locking => 1        );  use Parallel::ForkManager;  my $pm=new Parallel::ForkManager(10);  for my $key ( keys %$hash ) {    $pm->start and next;    my $in = $hash->{$key}->{in};    my $out = $function->($in);    #print "in $in, out $out\n";    #$db->lock();    $db->{result}->{$key}->{in}=$in;    $db->{result}->{$key}->{out}=$out;    #$db->unlock();    $pm->finish;  }  $pm->wait_all_children;  my $result = $db->{result};;  #print "result: " . Dumper($result);  return $result;  #return $hash;}#workssub hashmap_parallel_threads {    my $function = shift;    my $hash = shift;    use threads;    my @threads;    for ( keys %$hash ) {      my $in = $hash->{$_}->{in};      my $t = threads->create( sub { map_element($_, $function, $in ) } );      push @threads, $t;    }    #wait for threads to return ( this implementation is bound by slowest thread )    my %results = map { %{ $_->join() }; } @threads;    #print Dumper \%results;    return {%results};}sub map_element {      my $key = shift;      my $function = shift;      my $in = shift;      my $out = $function->($in);      return { $key => {in =>  $in,out => $out       }     };}1;

Things you should need to know before using Perl regexes. (Humour, with a serious point) (BrowserUk)

This is not a tutorial.

Perl's regex engine is not lightweight

Every time you use $`, $& or $', the entire scalar you are searching is copied.

What's more, not only are the scalars that you process using the regex contain a reference to one of those variables copied, but every scalar, processed by every regex in your entire program also gets copied.

Further, every time you use capturing brackets, all the captured chunks are also copied--again.

And, even correctly written regexes that use two or more variable length matches (* or + etc.) can consume prodigious amount of runtime stack and cpu.

Badly and/or naively written regexes that use nested qualifiers can have exponential runtimes, and if the scalar they operate on is anything more than modestly sized, can completely consume your process stack before finally trapping having consumed all your process memory allocation, or system swap space--whichever runs out first.

Dooom, gloom, despondency.

More doom gloom and despondency.

Blah, blah, blah.

Oh. and here is a solution that prevents some of the problems by wrapping each call to the regex engine.

It starts anothor process, sends your scalars and the regex to it via sockets. That other process runs the regex on your behalf, and sends the results back via another socket. This neatly eliminates the $& problem, and allows recovery from the stack runaway/memory exhaustion problems whilst keeping your main process' memory requirements to a minimum.

This is not a serious attack on the perl regex engine!

Whilst much of the above is and has been true for the past 5 (8?, 10?) years, most of it could not be otherwise.

And the point is that the regex engine isn't lightweight, and has some vagaries and caveats,

but that hasn't prevented thousands of programmers from writing 100s of thousands of perfectly functional, useful, beneficial scripts that use Perl's regex engine

Note:The stack problem has been very cleverly fixed in a recent build,

Perl is dead (cLive ;-))

Excuse me if this has been discussed before (I couldn't find it though). This is a rather amusing rant from an Amazon developer.

Even though it is a little OTT in places, it does definitely have some valid points.

I love Perl, but my faith in its future is not that great right now. I feel Python beckoning ;-)

How many words does it take? (Limbic~Region)

All,
This meditation is about not giving up on a problem just because it is [wp://Np_complete|NP complete]. Common responses from seasoned programmers to NP complete questions include:

Don't bother, it is NP complete
Find a heurstic approximation and hope it is good enough
Brute force is the only way to go

Response 2 is actually pretty good advice when it applies. Response 3 may be true but not all brute force approaches are created equal. The fact is that real world problems need to be solved regardless of CS theory. Just because something is NP complete, doesn't mean there isn't a smart(er|ish)? way of solving it.

I decided to work with [id://348444]. I was familiar with it because I had already provided a fairly [id://545167|fast approximation algorithm]. The problem stated is simple: Given a dictionary file, find the minimal number of words that contain a given set of letters.

I set out a list of constraints which turned the general problem into a specific one:

The dictionary file = 2of12 from the [http://wordlist.sourceforge.net/|Official 12Dicts Package]
Letters of both question and answer = a-z
Answer need only be fewest words, not fewest letters
Question may not contain duplicate letters - abc ok, abbc not ok

Originally, I wanted to precalculate enough information to allow each question to be solveable in polynomial time. While I still believe this approach would work with some problems, I ended up precalculating all solutions and functionally turned it into a lookup table. To do this, I needed to reduce the problem space enough to make a brute force solution feasible. Here are the guiding optimization principals I employed:

Not all solutions need to be found

There are a finite number of questions that can be asked because the letter set has been restricted to a-z. Using the combinatorial formula C = N! / K!(N - K)! when N = 26 and K = 1..26, there are 67_108_863 different possible questions. Since the solution that applies to 'abc' also applies to 'ab', 'ac', 'bc', 'a', 'b', and 'c' - even fewer questions actually have to be answered.

Not all words need be considered

Because of the constraints, we can reduce a word to the unique list of letters in contains. The word 'screwdriver' becomes 'cdeirsvw' and the word 'disc' becomes 'cdis'. Because there are no letters in 'disc' that are not contained in 'screwdriver', 'disc' need not be considered. Applying this principal further, we end up with only words of the longest string of unique characters.

Use a breadth-first search

Instead of solving questions at random, a [wp://Breadth-first_search|BFS] will provide the most efficient solution path. Start by finding all questions that can be solved using a single word from our reduced dictionary, then 2 at a time, then 3 at a time, etc until the longest possible question (a-z) can be answered.

Not all combinations of words need be considered

Since our goal is to get to a-z, when combining words we need only consider words that contain letters we don't already have. Further, we can use the reduction technique to only include the longest list of "new" characters. As an example if we start with 'hello' and are considering combining it with 'windowpane' and 'weapon', we are really considering 'adinpw' and 'anpw' and we can safely ignore 'weapon'.

Not all derived answers need be considered

Since we know that the answer for 'abc' also answers 'ab', 'ac', 'bc', 'a', 'b', and 'c' - we only need solve for 'abc' and we get the [wp://Powerset|powerset] for free. We can take advantage of this when solving new questions because we can skip the portions of the powerset that have been solved by some previous question. For instance, if we are looking at 'abcd', 'abce', and 'abcfg' - we know we can skip 'abc' and their descendants when solving the second and third question.

Even with these guiding principals in mind before I started, I still followed a number of dead-ends before finding the right approach. I will attempt to highlight the mistakes I made but will only be showing code for the actual solution.

Phase 1: Reduce the dictionary

Number of lines in input: 61_406
Number of lines in output: 3_477
Execution time: 1 min 15 seconds

#!/usr/bin/perluse strict;use warnings;use constant  WORD => 0;use constant   LEN => 1;use constant NFORM => 2;use Inline C =>;my $file = $ARGV[0] || 'dictionary.txt';open(my $fh, '<', $file) or die "Unable to open '$file' for reading: $!";my @word;while (<$fh>) {    $_ = lc;    tr/a-z//cd;    my %uniq = map {$_ => undef} split //;    my $len = keys %uniq;    push @word, [$_, $len, join '', sort keys %uniq];}@word = sort {$b->[LEN] <=> $a->[LEN] } @word;for my $i (0 .. $#word - 1) {    next if ! defined $word[$i];    for my $j ($i + 1 .. $#word) {        next if ! defined $word[$j];        $word[$j] = undef if ! distinct($word[$i][NFORM], $word[$j][NFORM]);    }}for (grep defined, @word) {    print join "\t", $_->[NFORM], $_->[WORD];    print "\n";}__END____C__int distinct(unsigned char *str1, unsigned char *str2) {    /* Actual code has 256 0s - truncated for post */    char exists[256] = {};    /* Turn array into a hash */    while (*str1) {        exists[*str1++] = 1;    }    /* Determine if str2 contains any chars str1 does not */    while (*str2) {        if (! exists[*str2++]) return 1;    }    return 0;}

Phase 2: Two at a time

Number of lines in input: 3_477
Number of lines in output: 636_186
Execution time: 9 min 35 seconds

#!/usr/bin/perluse strict;use warnings;use constant NEW_LET => 1;use constant     LEN => 2;use Inline C =>;my $file = $ARGV[0] || 'phase1.data';open(my $fh, '<', $file) or die "Unable to open '$file' for reading: $!";my @word;while (<$fh>) {    chomp;    push @word, [split /\t/];}my %seen_nform;for my $i (0 .. $#word - 1) {    my ($nform1, $str1) = @{$word[$i]};    my (@new_word, %seen);    for my $j ($i + 1 .. $#word) {        my ($nform2, $str2) = @{$word[$j]};        my $new_let = diff($nform2, $nform1);        next if ! $new_let || $seen{$new_let}++;        push @new_word, [$str2, $new_let, length($new_let)];     }    @new_word = sort { $b->[LEN] <=> $a->[LEN] } @new_word;    for my $i2 (0 .. $#new_word - 1) {        next if ! defined $new_word[$i2];        for my $j2 ($i2 + 1 .. $#new_word) {            next if ! defined $new_word[$j2];            $new_word[$j2] = undef if ! distinct($new_word[$i2][NEW_LET], $new_word[$j2][NEW_LET]);        }    }    for (grep defined, @new_word) {        my $str2 = $_->[0];        my %uniq = map {$_ => undef} split //, $nform1 . $str2;        my $new_nform = join '', sort keys %uniq;        next if $seen_nform{$new_nform}++;        print join "\t", $new_nform, $str1, $str2;        print "\n";    }}__END____C__SV* diff ( char *str1, char *str2 ) {    SV *sv= newSVpvn( "", 0 );    int result_index= 0;    char *result= SvGROW( sv, 257);    /* identify all chars present in str2 */    while ( *str1 && *str2 ) {        if ( *str1 < *str2)             result[ result_index++ ]= *str1++;        while ( *str1 && *str1 == *str2) {            str1++; str2++;        }        if ( *str1 > *str2 )            str2++;    }    while (*str1)         result[ result_index++ ]= *str1++;    result[ result_index ]= 0;    SvCUR_set( sv, result_index );        return sv;}int distinct(unsigned char *str1, unsigned char *str2) {    /* Actual code has 256 0s - truncated for post */    char exists[256] = {};    /* Turn array into a hash */    while (*str1) {        exists[*str1++] = 1;    }    /* Determine if str2 contains any chars str1 does not */    while (*str2) {        if (! exists[*str2++]) return 1;    }    return 0;}

Phase 3: Three at a time

Number of lines in input: 636_186
Number of lines in output: 8_809_183
Execution time: 401 min 10 seconds

Yes, that's java code you see. I originally wrote it in Perl which looked very much like phase 2 but it took between 6 1/3 hours to 11 1/2 hours depending on which machine and what variation of the algorithm I used. If anyone is interested in the perl counterpart - please let me know.

import java.io.File;import java.io.IOException;import java.util.ArrayList;import java.util.BitSet;import java.util.HashMap;import java.util.HashSet;import java.util.LinkedHashMap;import java.util.Map;import java.util.Scanner; public class Phase3 {    public static final String regex = "[a-z]+";    public static char alphabet[] = {        'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm',        'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z',    };    public static BitSet[] seen_nform = {        new BitSet(),        new BitSet(26),       new BitSet(325),     new BitSet(2600),        new BitSet(14950),   new BitSet(65780),    new BitSet(230230),  new BitSet(657800),        new BitSet(1562275), new BitSet(3124550),  new BitSet(5311735), new BitSet(7726160),        new BitSet(9657700), new BitSet(10400600), new BitSet(9657700), new BitSet(7726160),        new BitSet(5311735), new BitSet(3124550),  new BitSet(1562275), new BitSet(657800),        new BitSet(230230),  new BitSet(65780),    new BitSet(14950),   new BitSet(2600),        new BitSet(325),     new BitSet(26),       new BitSet(1)    };        public static HashMap lookup = new HashMap(26);    public static void main(String[] args) {        initlookup();        Scanner scr1, scr2;        String  line;        LinkedHashMap word = new LinkedHashMap(3477);        // Load phase1.data into word        try {            scr1 = new Scanner(new File(args[0]));            while (scr1.hasNextLine()) {                line  = scr1.nextLine();                scr2  = new Scanner(line);                word.put(scr2.next(regex), scr2.next(regex));            }        }        catch (IOException ioe) {            ioe.printStackTrace();        }        // Process phase2.data        String nform, str1, str2;        try {            scr1 = new Scanner(new File(args[1]));            while (scr1.hasNextLine()) {                line  = scr1.nextLine();                scr2  = new Scanner(line);                nform = scr2.next(regex);                str1  = scr2.next(regex);                str2  = scr2.next(regex);                HashSet seen = new HashSet();                ArrayList new_word = new ArrayList();                for (Map.Entry entry : word.entrySet()) {                    String new_nform = uniq(nform, entry.getValue());                    int len = new_nform.length();                    int bit = getBit(len, new_nform);                    if (seen_nform[len].get(bit)) { continue; }                    String new_let = diff(entry.getKey(), nform);                    if (new_let.length() == 0 || seen.contains(new_let)) { continue; }                    seen.add(new_let);                    String[] e = {entry.getValue(), new_let, new_nform, Integer.toString(bit)};                    new_word.add(e);                }                String[] new1, new2;                int idx;                int end = new_word.size();                for (int i = 0; i < end - 1; ++i) {                    new1 = new_word.get(i);                    if (new1 == null) { continue; }                    for (int j = i + 1; j < end; ++j) {                        new2 = new_word.get(j);                        if (new2 == null) { continue; }                        idx = j;                        if (new2[1].length() > new1[1].length()) {                            String[] temp = new1;                            new1 = new2;                            new2 = temp;                            idx  = i;                        }                        int res = distinct(new1[1], new2[1]);                        if (res == 0) {                            new_word.set(idx, null);                        }                    }                }                for (String[] e : new_word) {                    if (e == null) { continue; }                    String str3 = e[0];                    String new_nform = e[2];                    int len = new_nform.length();                    int bit = Integer.decode(e[3]);                    /* Probably not needed */                    if (seen_nform[len].get(bit)) { continue; }                    seen_nform[len].set(bit);                    System.out.println(new_nform + "\t" + str1 + "\t" + str2 + "\t" + str3);                }            }        }        catch (IOException ioe) {            ioe.printStackTrace();        }    }    private static void initlookup () {        // Actual code goes a-z, reduced for post        lookup.put('a', new Integer("1"));        lookup.put('b', new Integer("2"));    }    // Determine Chars in str1 not present in str2    private static String diff(String str1, String str2) {        int len = str2.length();        HashSet have = new HashSet(len);        for (Character c : str2.toCharArray()) {            have.add(c);        }        StringBuilder new_let = new StringBuilder(len);        for (Character c : str1.toCharArray()) {            if (have.contains(c)) {                continue;            }            new_let.append(c);        }        return new_let.toString();    }    // Determine if str2 (shorter) contains any chars not in str1 (longer)    private static int distinct(String str1, String str2) {        HashSet have = new HashSet(str1.length());        for (Character c : str1.toCharArray()) {            have.add(c);        }        for (Character c : str2.toCharArray()) {            if (have.contains(c)) {                continue;            }            return 1;        }        return 0;    }     // List of unique chars in alphabetic order in str1 & str2    private static String uniq(String str1, String str2) {        HashSet have = new HashSet();        for (Character c : str1.toCharArray()) {            have.add(c);        }        for (Character c : str2.toCharArray()) {            have.add(c);        }        StringBuilder unique = new StringBuilder(26);        for (char let : alphabet) {            if (have.contains(let)) {                unique.append(let);            }        }        return unique.toString();    }    private static int binomial(int n, int k) {        int c = 1;        for (int i = 0; i < k; ++i) {            c *= n - i;            c /= i + 1;        }        return c;    }    private static int getBit(int len, String str) {        int    sum = 0;        for (int i = 0; i < len; ++i) {            sum += binomial(lookup.get(str.charAt(i)) - 1, i + 1);        }        return sum;    }}

Phase 4: Four at a time (new approach)

Number of lines in input: 8_809_183
Number of lines in output: 1
Execution time: 0 min 7 seconds

The last phase brought the solution close enough just to scan for a word that contained all the missing letters to reach a-z.

#!/usr/bin/perluse strict;use warnings;use Inline C =>;my $file = $ARGV[0] || 'phase1.data';open(my $fh, '<', $file) or die "Unable to open '$file' for reading: $!";my @word;while (<$fh>) {    chomp;    my ($nform, $word) = split /\t/;    push @word, $word;}$file = $ARGV[1] || 'phase3.data';open($fh, '<', $file) or die "Unable to open '$file' for reading: $!";while (<$fh>) {    my ($nform, $str1, $str2, $str3) = split /\t/;    for (@word) {        if (! diff('abcdefghijklmnopqrstuvwxyz', $_ . $nform)) {            print "abcdefghijklmnopqrstuvwxyz\t$_\t$str1\t$str2\t$str3";            exit;        }    }}print "No Cigar\n";__END____C__SV *diff(char *str1, char *str2) {    /* Actual code has 256 0s - truncated for post */    char exists[256] = {};    SV *ret_sv = newSVpvn("",0);        /* identify all chars present in str2 */    while (*str2) {        exists[(U8)*str2++] = 1;    }    /* Determine chars in str1 not in str2 */    for ( ; *str1 ; str1++ )         if (! exists[(U8)*str1])             sv_catpvn(ret_sv,str1,1);        return ret_sv;}

Phase 5: Generate powersets

Number of lines in input: 9_448_847
Number of lines in output: 67_108_863
Execution time: 66 min 20 seconds

Yep, more Java. I also have a Perl version if anyone is interested but I wanted the final solution to take less than 8 hours to run so that someone wanting to reproduce my results could.

import java.io.File;import java.io.IOException;import java.util.ArrayList;import java.util.BitSet;import java.util.HashMap;import java.util.Scanner; public class Phase5 {    public static final String regex = "[a-z]+";    public static BitSet[] seen = {        new BitSet(),        new BitSet(26),       new BitSet(325),     new BitSet(2600),        new BitSet(14950),   new BitSet(65780),    new BitSet(230230),  new BitSet(657800),        new BitSet(1562275), new BitSet(3124550),  new BitSet(5311735), new BitSet(7726160),        new BitSet(9657700), new BitSet(10400600), new BitSet(9657700), new BitSet(7726160),        new BitSet(5311735), new BitSet(3124550),  new BitSet(1562275), new BitSet(657800),        new BitSet(230230),  new BitSet(65780),    new BitSet(14950),   new BitSet(2600),        new BitSet(325),     new BitSet(26),       new BitSet(1)    };        public static HashMap lookup = new HashMap(26);    public static void main(String[] args) {        initlookup();        String  line = null;        Scanner s2   = null;        for (String filename : args) {            try {                Scanner scr = new Scanner(new File(filename));                while (scr.hasNextLine()) {                    line = scr.nextLine();                    s2 = new Scanner(line);                    if (! s2.hasNext(regex)) {                        continue;                    }                    getSets(s2.next(regex), line);                }            }            catch (IOException ioe) {                ioe.printStackTrace();            }        }    }    private static void initlookup () {        // Actual code goes a-z, reduced for post        lookup.put('a', new Integer("1"));        lookup.put('b', new Integer("2"));    }    private static void getSets(String str, String line) {        int len = str.length();        int bit = getBit(len, str);        if (! seen[len].get(bit)) {            seen[len].set(bit);            System.out.println(str + "\t" + line);            for (StringBuilder set : subsets(str)) {                getSets(set.toString(), line);            }        }    }    private static ArrayList subsets(String str) {        ArrayList subs = new ArrayList();        if (str.length() == 1) {            return subs;        }        for (int i = 0; i < str.length(); ++i) {            StringBuilder set = new StringBuilder(str);            set.deleteCharAt(i);            subs.add(set);        }        return subs;    }    private static int binomial(int n, int k) {        int c = 1;        for (int i = 0; i < k; ++i) {            c *= n - i;            c /= i + 1;        }        return c;    }    private static int getBit(int len, String str) {        int    sum = 0;        String key;        for (int i = 0; i < len; ++i) {            sum += binomial(lookup.get(str.charAt(i)) - 1, i + 1);        }        return sum;    }}

Creating a working final product

I decided to use a [cpan://DBD::SQlite] database with a very simple [http://www.gatcomb.org/dictdemo/demo.cgi|web front end to demo it]. The split into 26 tables was left over from an earlier disk space optimization idea that I didn't pursue so the following code could be simplified further:

Split the data by length and convert solution words to indices (took 12 minutes)

#!/usr/bin/perluse strict;use warnings;use Storable;my %fwd;my $file = $ARGV[0] || 'phase1.data';open(my $fh, '<', $file) or die "Unable to open '$file' for reading: $!";my $n;while ( <$fh> ) {    chomp;    my ($nform, $word) = split /\t/;    $fwd{$word} = ++$n;}my %sol_fh;for (1 .. 26) {    my $file_name = sprintf("%.2d", $_) . ".data";    open($sol_fh{$_}, '>', $file_name) or die $!;}$file = $ARGV[1] || 'phase5.data';open($fh, '<', $file) or die "Unable to open '$file' for reading: $!";while ( <$fh> ) {    chomp;    my ($nform, undef, @words) = split /\t/;    $_ = $fwd{$_} for @words;    print { $sol_fh{length($nform)} } $nform, "\t", (join "-", @words), "\n";}my %rev = reverse %fwd;store \%rev, 'sol.rev';

Build the database and use the sqlite3 shell interface to import the data

#!/usr/bin/perluse strict;use warnings;use DBD::SQLite;my $dbh = DBI->connect("dbi:SQLite:dbname=solution.db","","") or die $DBI::errstr;for my $size (1 .. 26) {    my $sql = "CREATE TABLE solution$size (nform TEXT, solution TEXT)";    $dbh->do($sql) or die $dbh->errstr;}$dbh->disconnect or die $dbh->errstr;

Create a simple web front end

#!/usr/bin/perl -Tuse strict;use warnings;use CGI::Simple;use DBD::SQLite;use HTML::Template;use Storable;$CGI::Simple::POST_MAX        = 1024;$CGI::Simple::DISABLE_UPLOADS = 1;#$CGI::Simple::DEBUG = 1;my $q = CGI::Simple->new();# TODO: if unable to get exclusive lock on $0 display_error()# TODO: Add error handling - duh!$q->param('answer') ? display_answer() : display_question();sub display_question {    my $template = HTML::Template->new(filename => 'question.tmpl');    print $q->header;    print $template->output();    exit(0);}sub display_answer {    my $template = HTML::Template->new(filename => 'answer.tmpl');    my $input = get_input();    my $len   = length($input);    my $rev   = retrieve('sol.rev');    my $dbh   = DBI->connect("dbi:SQLite:dbname=solution.db","","")        or die $DBI::errstr;    my $sth   = $dbh->prepare("SELECT solution FROM solution$len WHERE nform=?")        or die $dbh->errstr;    $sth->execute($input)                                                  or die $dbh->errstr;    my @word;    while (my @row = $sth->fetchrow_array() ) {        push @word, map { {WORD => $rev->{$_}} } split /-/, $row[0];    }    $template->param(USER_INPUT => $input);    $template->param(WORD_LIST  => \@word);    print $q->header;    print $template->output();    #$sth->finish();    #$dbh->disconnect;    exit(0);}sub get_input {    my $input = $q->param('question') || '';    $input = lc($input);    $input =~ tr/a-z//cd;    $input ||= 'abcdefghijklmnopqrstuvwxyz';    my %uniq = map {$_ => undef} split //, $input;    return join '', sort keys %uniq;}

Include a question template

How many words does it take.....How many words does it take....
Please enter a unique list of lower case letters (a-z) below.
Be warned that some of the "words" may not be safe for work (NSFW).

Include an answer template

It takes.....For  it takes....
    Word:

Conclusion

Answering all 67+ million questions using a dictionary of over 64K words can take less than 8 hours and the resulting lookup table (SQLite) is 2GB of disk space. If this were a real world problem, that is not a considerable investement.

Notes

I freely admit that I spent far more than 8 hours on this specific project. I also admit that each problem is unique and has no guarantee of being feasible. I came to the Monastery at least twice looking for help with optimizations:

[id://579130]
[id://576101]

I had to find the right balance of trading memory for speed since 1GB of RAM was not enough in many cases. I also had to find the right balance between reducing the output for the next phase with the amount of work required for that reduction.

The code as it appears can likely be optimized. I spent my time looking for algorithm related optimizations rather than micro-optimizations. While I used both [cpan://Inline::C] and Java in this project, I am not much more than a neophyte in either. In fact, this thread contains all the Java I have ever written in my life. I had help in those areas but the contributors were limited by my description of what I was trying to do so all deficiencies are mine rather than theirs.

I would like to thank everyone who helped me with this project. Rather than naming the few I can remember and offending those I can't, I will just say "you know who you are". You can try it out [http://www.gatcomb.org/dictdemo/demo.cgi|here]. I welcome any questions or comments on the code and algorithms.

Cheers - [Limbic~Region|L~R]

Parrot, threads & fears for the future. (BrowserUk)

The future is threaded.

At the commodity hardware level, this is already the case thanks to Intel Hyperthreading, Intel, AMD (and others) dual core processors. AMP, SMP & NUMA. Dual, quad & 8-way motherboards. In the future it will be more so. Eg. The Cell processor. Quad-core technology from Intel and AMD coming in 2007.

At the software level, very little existing software makes use of threading. There are several reasons for this:

Software needs to be written from the ground up with threading in mind in order to properly benefit from it.
Retro-fitting threading to existing applications is rarely effective because threading exacerbates the effect of every bad programming practice.
- Re-entrancy issues can not be glossed over. Whether in the application code, language runtime or OS.
- Global data-structures become even more vulnerable.
- Tight coupled code causes low granularity. Low granularity can make threading expensive.
- Memory management becomes paramount.
  - Stop the world GC will have a disastrous effect upon performance.
  - Monolithic heap management will suck the life out of efficiency.
Existing code that can benefit from threading, often uses event driven and/or state machine techniques to approximate those benefits, but code designed to utilise those techniques is usually structured such that it does not lend itself to conversion to threading.
Existing code that already achieves parallelism through forking, that could also benefit from shared state, is often difficult to adapt to threading due to the assumptions that can be safely made when using forking that no longer hold true with threading.

Redeveloping existing, successful applications that could benefit from threading is often slow to happen. Again, for a variety of reasons:

Mid-life, ground-up, redesign of an existing application is always a major undertaking, even where there are clear benefits to doing so.
In many environments threading is seen as hard.
In many environments, there is a lack of both understanding and the skills required to implement threading well.

Even when developing new applications, that could obviously benefit from threading, it is overlooked, ignored or explicitly ruled out. Again there are a variety reasons why this happens, most of which are already covered above.

Much of the resistance/reluctance to utilise threading can be attributed to a single factor--there are few if any good tools available.

Most languages--that the majority of people use--either do not support threading at all, or support it as an afterthought. And then only at the lowest level.
There are few good abstractions of shared state. If every program still had to deal with disk-bound data by directly manipulating physical blocks and freespace chains, very few programs that manipulate file-based data would exist--that's most programs in existence. Today's ubiquitous hierarchical filesystems make it seem as if there was never any other way, and that tends to imply that they are perfect. But we also have RDBMSs, which are most definitely not hierarchical, and not file-based, (Although they often live on hierarchical filesystems.).
Most of todays programming tools--compilers, interpreters, editors, debuggers, runtime libraries etc.--are the latest evolutions of the same tools that go back years., In many cases, decades.

And like any other type of existing application, adaption to threading is difficult, resisted ,expensive and risky. Redesign from the ground up with threading in mind, implies throwing away thousands or millions of development hours in thoroughly tried and tested tools and libraries.

Could that possibly be worth the trouble?

What if an existing, popular, powerful and flexible language was already being redesigned from the ground up.

If that language was already looking to support simple syntax and intuitive semantics for Distributed Operations?

    if( any( @list ) == constant ) ...    @list1 >> += << @list2

And those DistOps inherently lent themselves to being run concurrently on multiple hyperthreads/cores/CPUs?

What if the entire tool-chain to support that new language was also already being redesigned from the ground up?

Doesn't it make sense to write those tools with threading not just "in mind", but as a high priority?

Will those tools be all they could be, if their architects "Don't do threads"? If the implementors "do not see the need for threads"?

Indeed, does it bode well for the future of those tools if the implementors do not use the languages that those tools are to support, And don't have the slightest feel for what will drive the needs and uses of those languages in the future?

Does the

complete absence of a threads.pdd from the specification;
that the term "threads" appears only 35 times in the entire documentation set;
that the "failed" ithreads model, so widely denigrated and despised, is being nearly exactly replicated for the underpinnings of the new language;

inspire you with confidence?

What about userspace threading? Many languages provide this and many programmers find it's determinism and light weight lends itself to many things that they want to do. It's not a replacement for kernel threads, but if threads != interpreter, then providing primitives to allow cooperative, user threading to run within preemptive kernel threading becomes not just possible but almost trivial. Whether this is provided at the VM level or the language level. Trying to hack user space threading into a language when thread == interpeter becomes a completely different ballgame.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Creating a co-operative framework for testing (gmax)

Hi Monks,

I recently applied for a post of QA developer at a open source company and I started working there, with great delight, about one month ago. One of the reasons for my hiring was my active participation in the community, with blogging, writing articles, submitting bug reports, answering to forum and NG questions. My background as a tester was considered, of course, but the stress was on my community links.

Inviting an open source community to Quality Assurance tasks

Thus, one of my tasks is now to organize a framework for cooperation between the Quality Assurance department and the community.

It is not an easy task to tackle. I can see how much an active community can contribute (bug reports, test cases, code patches, usability feedback), but I can also see the clash between a QA department populated by professional testers and the erratic behavior of a large community. Somehow I have to find a way of reconciling these two worlds, and make them work harmoniously toward a common goal.

My plan (yes, in spite of the task being so though, I do have a plan) is to motivate potential contributors with durable benefits, to get the best possible outcome from this relationship.

During the past twelve months, the company has promoted such cooperation with a contest. Win an iPod if you submit many bugs, and/or write articles or blog posts about our products. The results were quite good in the beginning, but they faded away soon after a few weeks. Some of the prizes are yet to be awarded. Why did this policy fail to provide the expected results? IMO, because it was targeting the wrong quality of a potential contributor: it was appealing at the competitiveness, instead of tickling pride and desire of recognition, which work much better in an open source environment. But especially, it did not address the main aspect of open source success, i.e. mutual benefit.

Why does someone contribute to an open source project? There are many reasons, but the paramount one should be striving for improvement. The main point of open source is being able to modify something that barely works for you into something that fits your needs appropriately. Thus the mutual benefit: when you add a feature or fix a bug in an open source product that affects your work, you make that product better for you, but you are also improving its overall value.

The challenge of testing

Among the activities that make the quality of a software product, testing is perhaps the most visible one. There are other elements, such as policies for coding, internal code reviews, and several organizational issues that will affect the final quality. But testing is a key element in software development. Thus, my company has a huge regression test suite that does a good job of keeping each build free of bugs.

As everyone knows, testing is never enough. Despite all the efforts from the developers and the QA engineers, there is always a bug that escapes their attention and affects the user. Why? Surely because testing everything is impossible, but also because developers and QA engineers are highly trained people and they approach problems from a different perspective than the final users.

Common users take the product for what they need, and just throw at it the commands that will solve their problem, regardless of any other concern that the developers could take into account. This naive behavior is what finds the most appalling bugs.

The challenge, then, is to combine the scientific approach of a QA department with the cleverness of willing contributors, who are able to find bugs that elude most of the professionals.

The lessons of Perl

That's why (finally) I am coming here to ask for advice. My job is not as a pure Perl developer (although Perl is largely used in our testing infrastructure), so this problem is not related to Perl as a language, but with Perl as a community. I was accustomed to testing before using Perl, but it is in the Perl community that I found the most efficient way of testing.

I can see that the Perl community has a very good testing infrastructure. I see how it is organized, how it works, but the reasons why it is so good escape me.

Seeking advice

Here are the questions

Why is the Perl testing infrastructure so effective?
If I wanted to export some of the qualities of Perl testing to a non-Perl product, what should I focus on?
What motivates a (QA) contributor?

So, fire on. Any insight on this matter could be valuable.

Thanks in advance.

 _  _ _  _  (_|| | |(_|>< _|

Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison) (tphyahoo)

This is a followup from two recent posts:

Could there be ThreadedMapReduce (and/or ForkedMapReduce) instead of DistributedMapReduce?

using parallel processing to concatenate a string, where order of concatenation doesn't matter

In a sense it is a followup from an older post as well:

What is the fastest way to download a bunch of web pages?

The common theme to all these posts is doing something in parallel to accomplish a task, where each bit of the task is independent of the other bits.

It turns out that this is the essense of the MapReduce algorithm that google uses to build much of its most important code. Google uses cluster computing to accomplish this, but the same "parallelization" logic could also be used on a single computer running multiple processes, or even a single process with threads. (Ira Woodhouse has indicated that the next version of MapReduce released to the CPAN will probably have configuration options for single computer clusters in order to do exactly that.)

Currently, the canonical example of a situation where this would be useful for me, on my job, is to download a bunch of web pages and verify that each page regex-matches something that it should. In other words, grep, where part of the grep test is to download a web page.

Throughout my experience with perl, have had numerous other situations where this basic idea -- break down a task into components that can run in parallel, run them, and then reassemble the results -- would have been helpful. Sometimes I would do this, but because I'm not an experienced thread programmer, and sometimes I was on windows sometimes not, sometimes would have to recompile perl, etc etc, it was always an unpleasant experience. There were many occasions when I thought, maybe I could use parallelization here... but painful... not enough time to debug.... I'll just make do with having it be a little slower.

An example of a time when I could have, but didn't, use functional programming based on parallelization to speed things up, was when I had ~50,000 web pages to parse and reformat using a variety of functions based on HTML::TreeBuilder. None of the files cared what the other files looked like, so I could have processed multiples at the same time. But I got the basic system working serially, and it got the job done in a bit over 48 hours. This was too slow, so I did some simple things with forks and got it down to under a day, which was acceptable because the script only had to run once.

But I remember thinking the forking code the ugliest, and hardest to debug, in my program.

If I had had 5 million web pages instead of 50,000, and wanted to split the computation among multiple processors somehow, putting the job queue stuff together for it along the lines I had used till then, it would have been a nightmare. Even though all I was doing was grep, with a little network communication inside the grep test function.

Later when I came across the article

Why Threads are a Bad Idea

it all rang very true to me. Okay, I was working with forks, in a simple context, and this presentation is maybe more about threads with systems programming. But the basic difficulties apply to any situation where you are running stuff in an order that isn't guaranteed. It's different from running stuff through a simple, ordered, for loop in ways that keep quite subtle and hard to detect.

What frustrated me the most is the feeling that I couldn't encapsulate the logic that I want, that I have to keep writing the "ugly bits" again and again.

This was before I had heard of functional programming.Since then, I have been learning a lot about functional programming, and trying to incorporate it into my bag of tools. I originally became interested in this after reading Paul Graham's On Lisp. But I want to apply functional programming to make my life easier in perl. Joel Spolsky suggests in

Can Your Programming Language Do This

that functional programming is a good technique to hide the "ugly but important bits" of your code.

In the article, spolsky suggests that this is exactly what Google has done with their MapReduce algorithm. Google programmers can write code that says the equivalent of

my @results = distributed_mapreduce_grep ( $test_function, [ @in_array ]);

and this would do exactly what

my @results = grep { $test_function($) )} @array

would do in perl. Except that it works on a cluster. So you can process a lot more data, faster.

And the "ugly bits" are hidden.

I want to start hiding the ugly bits of my code, using functional programming.

The following is my attempt to do that.

Unfortunately it still isn't working. But I think it's an interesting read, and I'm also hoping someone can plug something in there that will make it work.

What's nice is that the "ugly bit" that isn't working is encapsulated. This is the function hashmap_parallel. Currently my "parallelization strategy" involves forking off processes and storing values in a DBM::Deep hard disk store. But actually I don't care about the implementation details, I just want it to work.

****************************

UPDATE: Thanks to LanceDeeply, I now have code that works, using threads for the parallelization.

I kept the function that doesn't work as hashmap_parallel_forks, which I am still hoping to get working. The code that does work is called hashmap_parallel_threads. The test script has also been updated accordingly.

If anyone else want to shoot me some candidates for their favorite way of implementing transparent parallelization with map, I will add them to the catalog.

****************************

Hashmap here means essentially the same as the "map" half in MapReduce. It processes a set, where order doesn't matter. I have several hashmap functions in this code, two of which work, and one (the one that executes in parallel) which doesn't.

These "mapping" functions are given as an argument to the function builder flexygrep, which returns a grepping function. So, as a consequence two of my grepping functions work and one (the parallel one) doesn't.

If I can get hashmap_parallelto work, I'm thinking could theoretically use this to build other functions, like sort_parallel, permute_parallel, you name it. Sometimes, for efficiency reasons, this will make sense. A lot of the time it won't. Depends what your bottleneck is -- cpu, memory, network, disk io, etc. But the good news is that once the parallel mapping function -- or mapping functions -- work, you can just plug them in and try. A lot easier than writing threading code for all scenarios.

Again, the bit that I need to get to work -- but which will pay major dividends in maintainability when I do -- is hashmap_parallel.

Now, here's a little test output.

$ ./test_hashmap.plok 1 - parallel-y threadgrep worksnot ok 2 - parallel-y forkgrep works#   Failed test 'parallel-y forkgrep works'#   in ./test_hashmap.pl at line 21.ok 3 - serially executing code works1..3# Looks like you failed 1 test of 3.

And here's the code

test_hashmap:#!/usr/bin/perluse strict;use warnings;use Test::More qw( no_plan );use Data::Dumper;use Grep;my $slow_matches_b = sub { sleep 1;                           return unless $_[0];                           return 1 if $_[0] =~ /b/;                         };my $test_strings = [ ('blee','blah','bloo', 'qoo', 'fwee' ) ];my $matches;$matches = Grep::threadgrep( $slow_matches_b, $test_strings );ok( @$matches == 3, "parallel-y threadgrep works"  );# should get blee, blah bloo, but not fwee or qoo$matches = Grep::forkgrep( $slow_matches_b, $test_strings );ok( @$matches == 3, "parallel-y forkgrep works"  );$matches = Grep::slowgrep( $slow_matches_b, $test_strings );ok( @$matches == 3, "serially executing code works"  );Grep.pm:package Grep;use strict;use warnings;use Data::Dumper;use Map;# grep can be parallelized by building it on top of map_parallel# which uses forks, threads, distributed computations with MapReduce# or some such black magic# in some cases this may be faster, but not always,# it depends where your bottleneck is.# Whatever black magic is going on in the background,# by abstracting it out, the code we get is clean and easy to read.sub threadgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_threads(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub forkgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_parallel_forks(@_)};  return flexygrep($test_function, $map_function, $in_array);}# or you could do it in a non-forked/threaded/distributed/whatever# way, by basing it on the conceptually simpler function map_serial.sub slowgrep {  my $test_function = shift;  my $in_array = shift;  my $map_function = sub { Map::hashmap_serialized(@_)};  return flexygrep($test_function, $map_function, $in_array);}sub flexygrep {  my $test_function = shift;  my $hashmap_function = shift;  my $in_array = shift;  my $in_hash = Map::hash_from_array($in_array);  my $result_hash = $hashmap_function->($test_function, $in_hash);  my $out_array = [];  for my $key (keys %$result_hash) {    if ( my $out_true = $result_hash->{$key}->{out} ) {      push @$out_array, $result_hash->{$key}->{in}    }  }  return $out_array;}1;Map.pm:package Map;use strict;use warnings;# Black magic for doing stuff in parallel is encapsulated here# use MapReduce;use Parallel::ForkManager;use threads;# use threads::shared qw(is_shared);use DBM::Deep;use Data::Dumper;sub hash_from_array {  my $array = shift;  my $hash;  for my $index (0..$#$array) {    $hash->{$index}->{in} = $array->[$index];  }  return $hash;}# input is a funcion (eg, my $sub_multiply by ten = { return $_[0] * 10 } ), and# a hash like# my $input_values = { blee => { in => 1 },#                      blah => { in => 2}#                    }# output is a hash like#{ blee => { in => 1, out => 10 },#  blah => { in => 2, out => 20 }#}sub hashmap_serial {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  # hash keys are processed in whatever order  for my $key ( keys %$hash) {    my $in = $hash->{$key}->{in};    my $out = $function->($in);    #print "result for $in is $out\n";    $hash->{$key}->{out} = $out;  }  return $hash;}# does the same thing as hashmap_serial# but saves the value on the hard drive# (serialized in this context means a memory value gets put on the hard disk,# not to be confused with the sense of "serial as opposed to parallel"sub hashmap_serialized {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  use File::Path qw(mkpath);  my $dir="c:/tmp/map_serialized";  mkpath($dir) unless -d "$dir";  die "no directory: $dir" unless -d "$dir";  my $file="$dir/$$.db";  my $db = DBM::Deep->new( $file );  $db->{result}=$hash;  for my $key ( keys %$hash ) {    my $in = $hash->{$key}->{in};    my $out = $function->($in);    $hash->{$key}->{out} = $out;  }  #unlink $file;  #die "couldn't delete file" if -f $file;  return $hash;}# but uses threads to compute "out" values in a parallel way# doesn't work.sub hashmap_parallel_forks {  my $function = shift;  my $hash = shift;  die "bad hash" . Dumper($hash) if grep { ! defined($hash->{$_}->{in}) } (keys %$hash);  return {};  use File::Path qw(mkpath);  my $dir="c:/tmp/map_serialized";  mkpath($dir) unless -d "$dir";  die "no directory: $dir" unless -d "$dir";  my $file="$dir/$$.db";  my $db = DBM::Deep->new( $file );  $db->{result}=$hash;  my $pm=new Parallel::ForkManager(10);  for my $key ( keys %$hash ) {    $pm->start and next;    my $in = $hash->{$key}->{in};    my $out = $function->($in);    print "in $in, out $out\n";    $hash->{$key}->{out} = $out;    $pm->finish;  }  $pm->wait_all_children;  print "hash: " . Dumper($hash);  #unlink $file;  #die "couldn't delete file" if -f $file;  #die "forkgrep result: " . Dumper($hash);  return $hash;}#workssub hashmap_parallel_threads {    my $function = shift;    my $hash = shift;    my @threads;    for ( keys %$hash ) {      my $in = $hash->{$_}->{in};      my $t = threads->create( sub { map_element($_, $function, $in ) } );      push @threads, $t;    }    #   wait for threads to return ( this implementation is bound by slowest thread )    my %results = map { %{ $_->join() }; } @threads;    #print Dumper \%results;    return {%results};}sub map_element {      my $key = shift;      my $function = shift;      my $in = shift;      my $out = $function->($in);      return { $key => {                        in =>  $in,                        out => $out                       }             };}1;

UPDATE: Seemingly relevant comments from Jenda at [id://580019]:

"You can only transparently paralelize map{} if the executed block is side-effect-free....."

I'm actually not sure if my code here is side effects free or not. Hm...

*****************************************************

Posts I'm looking at to see if I can use something there to get hashmap_parallel to work...:

[id://237089]

C++ vs. Perl mention on shygypsy.com (dhoss)

Hey all,

If you've been in the CB within the past 24 hours, you've probably at least heard of [http://shygypsy.com/farm/p.cgi|Funny Farm], a pretty sweet little game by Igor Naverniouk.

Looking around [http://www.shygypsy.com|his site], i found on his [http://shygypsy.com/acm/|UVA tools page] that he says, and i quote:

ShyGypsy labsAugust 4, 2006 02:19I'm playing around with some better ways of ranking people and problems. Here is a preview of a prototype. I have also discovered that I can easily write CGI scripts in C++. Perl is great, but I can get stuff done a hundred times faster in C++.

"I can get stuff done a hundred times faster in C++"? Did I miss something? If I recall correctly, comparatively it takes at least 5-6 lines more of code in C++ that in perl to write a simple "Hello world" program.

Example:

C++:

#include using namespace std;int main (){  cout << "Hello World!";  return 0;}

Perl:

#!/usr/bin/perl -wuse strict;print "Hello world!\n";

or, even less:

C:\>perl -e "print 'hello world!'"

I'm not trying to dig up an old Jihad, and I normally would just search the site for responses on this, but what could possibly justify him saying that? It could very well be that he's more comfortable with C++ and can simply code quicker in C++, logically that's sound because he's got 2 C++ courses he teaches at UofToronto, but I just can't fathom this.

Another possiblity is he has many code libraries in C++, mainly algorithms and such, and maybe that puts C++ on a more level playing field with perl since it eliminates a lot of low end coding. But do libraries really making a "100 times faster" difference?

Please, correct me, downvote me, chastise me if I'm off base and haven't done my research, but I just can't possibly see how coding in C++ could possibly be any faster than coding in perl

Your wisdom is appreciated monks.

P.S> I just threw this in meditations, move as needed.

meh.

Book update: "Exploring Programming Language Architecture in Perl" (billh)

Is it ok to post updates here? I've got no-one else to talk to :-)
Anyway assuming it's ok I've just finished the first draft of chapter 13on continuationsover at at billhails.net.
It's still quite rough, and it needs rearranging (move the "what's so coolabout continuations" bits up to the front so the reader feels moremotivated to read the rest,) and subtitling.
I'm slightly worried that I'm leaning rather too heavily on one source, the venerable and wonderful "Essentials of Programming Languages", but hopefully by the time I've finished I'll be able to present a more balanced account.
Any and all constructive criticism is welcome: anything I've missed,any points that are too laboured etc.
And enjoy of course.
PS There are now PostScript and PDF versions linked from the cover page.
Update: changed title as per recommendation

Bill H

perl -e 'print sub { "Hello @{[shift]}!\n" }->("World")'

See also: [id://578390] and [id://543564]

Edited by [planetscape] - added "see also" links ( keep:0 edit:14 reap:0 )

Perl needs Zend (EvanCarroll)

I feel that it is important to say, I have down voted every response to this for lack of utility. Please post a response that has content, and not just mindless perl fanaticism.

Fellow monks, I think I've finally answered the question as to why PHP is, generally speaking, more successful. I have to point to their corporate backing which largely comes from Zend. Zend offers above and beyond the free PHP stuff other commercial products too, and they take the proceeds from these product, and dump them back into the language. Zen actually produces seven products all of which can be found on their PHP powered website at zend products. Zend even produces an Optimizer which makes all of your php code run up to "25times faster." That's really good news for php, especially considering php is pretty fast anyway.

I mean look at this, the Zend core actually runs Oracle and has been ported to IBM. Which makes me think maybe the core being more full featured than PERL's is the reason for PHP's success. Obviously, it has some advantages; or why would big blue, and Oracle have adopted it? Maybe its the additional free support Zend provides that the Perl foundation doesn't, IBM support and Oracle support.

Zend even offers certificates, and they are "confidant you will pass the first time." That's probably because the language has no learning curve whatsoever. You are almost born knowing it! And, if you get both certificates you get a $25 discount. Lets say though you find php cryptic, for 769.00 you can get Zend professional training. Another important thing Zend offers is consultation, which isn't readily available in the Perl community.

The solution to this problem is simple, Perl needs some really big corporate backing too, like either Google, Yahoo, Microsoft or Apple. Maybe there is some way we can get Larry to hand over the project to Zend. Or maybe some day Zend will build a Perl optimizer for us that will run our applications 25 times faster. We probably also need these big corporate guys to run on our core as well, at least Apple; surely the PERL core can handle them.

Evan Carroll
www.EvanCarroll.com

RFC: Perl meta programming (bennymack)

Dear Monks,

Herein lies my latest half-baked Perl extension. This one is for reducing the amount of boiler-plate code necessary for writing Perl subs.

As per usual, I will jump right to the code which I always assume will speak for itself ( but rarely does ).

Here is an example of a module that uses my "handy" new module.

SomeTestPackage2.pm

package SomeTestPackage2;use strict;use warnings;use Data::Dumper;use constant IVARS => qw[$none $href $aref $non_empty_aref];BEGIN {use base qw[SomeAttributes2];__PACKAGE__->import( IVARS );}use constant custom_aref => q{    confess( '!!!package!!! !!!ivar!!! must be an array ref' ) if 'ARRAY' ne ref !!!ivar!!!; };sub self_sub : Method( qw/ $none :none $href :href $aref custom_aref $non_empty_aref :non_empty_aref / ) { #    return ( ref $self, ref $href, ref $aref, scalar @{ $non_empty_aref } );}1;

The previous code snip shows how the meta programming interface works for the most part.

First, you specify a list of instance variables, or whatever, that you would like to pull into subs. This pre-declaration is necessary for syntax reasons.

Next, you can specify "custom" attributes as package subs if you desire. The difference between a custom attribute and a canned/common one is that canned attributes begin with a colon and custom do not.

Now, in your sub attribute you specify a list of variables you'd like pulled into your sub along with any attributes you'd like called on them.

This is the test script running the prior code

test2.pl

#!/usr/bin/perluse strict;use warnings;use Test::More qw[no_plan];use SomeTestPackage2;do { # Successfull call to self_sub    my( $test1 ) = bless( {}, 'SomeTestPackage2' );    my( @self_sub ) = $test1->self_sub( 'hi', { }, [ ], [ 1 ] );    ok( $self_sub[0] eq 'SomeTestPackage2', 'ref $self' );    ok( $self_sub[1] eq 'HASH',             'ref $href' );    ok( $self_sub[2] eq 'ARRAY',            'ref $aref' );    ok( $self_sub[3] > 0,                   'non empty array' );};

Here is the output of running the previous script.

$ perl test2.plok 1 - ref $selfok 2 - ref $hrefok 3 - ref $arefok 4 - non empty array1..4

And finally, here is the code that is compiled from the "Method" attribute on the "self_sub".

sub {    local *__ANON__='SomeTestPackage2::self_sub';    my( $self, $none, $href, $aref, $non_empty_aref ) = @_;    confess( 'SomeTestPackage2::self_sub $non_empty_aref needs a non-empty array reference' ) if 'ARRAY' ne ref $non_empty_aref or not @{ $non_empty_aref };    confess( 'SomeTestPackage2 $aref must be an array ref' ) if 'ARRAY' ne ref $aref;    confess( 'SomeTestPackage2::self_sub $href needs a hash reference' ) if 'HASH' ne ref $href;    confess( 'SomeTestPackage2::self_sub needs a SomeTestPackage2 reference' ) if ref $self ne 'SomeTestPackage2';    package SomeTestPackage2;    use warnings;    use strict 'refs';    return ref $self, ref $href, ref $aref, scalar @{$non_empty_aref;};}

That's it! Is it crap?

RFC: Perl Testing -- How to Introduce to a team (NovMonk)

Wise monks,

My company is facing a transition. We have been writing perl scripts-- little snippets of code to get certain text outputs-- or modifying longer programs (adding logic to affect printed text output). We have been using shell scripts to drive our processes-- the most ungodly mess you can imagine, and which varies from job to job based on who set it up, and how many years ago. The good news: we have recently purchased some new software and we are retooling our existing processes to use it, and moving from flat file to database (MySQL) data.

As we talked about designing this new framework, I timidly suggested adding testing modules to the process Up Front, and as a reward, I was tasked with researching testing, and training our developers to do it. The closest any of us have been to testing is adding print statements to code while debugging-- one of us (not me) actually uses the Perl debugger with some facility.

It's not a matter of convincing them we need to be testing-- everybody's sold on that. We are all just getting overwhelmed with where to start. I've read the posts on testing here found through Super Search, and I have printed some out to share. I've gone to CPAN and printed off several of the modules (Test::Simple, Test::More, the tutorial on testing, etc). I've been reading books on testing and software construction in general: Perl Medic, Perl Debugged, Perl testing: A Developer's Notebook , and Code Complete.

One problem is: most of us are self taught, or have only the haziest ideas about Object Oriented Programming, or writing/ using modules at all. Some things in our process are obvious- you want to test that needed files are where you expect them to be, are in the right format, and that the program generates the expected outfiles and sends them where they need to go. But-- What else? If youre supposed to design tests before you code, what else should we be looking for?

My plan for training is to pass out some of the CPAN material on Test::Simple and Test::More, and the posts from here, and parts of the books, some of which we are buying as a company. Im also going to touch on Perl Tidy and Perl Critic, as ways for us to enforce these coding standards were agreeing on. What Im looking for from you guys is other suggestions, things I might think about, ways youve presented testing to programmers youve mentored over the years. Things you wish youd known when you were starting out. Testing procedures you inherited when you started with a company, that you wish you could change.

Weve this fantastic opportunity to design procedures and policies that will make our lives easier, make us better programmers, and make our company, and us, lots of money. And that is also a little overwhelming.

Many thanks in advance,

NovMonk

OO in Perl 6 (Scott7477)

In his latest Apocalypse, Larry Wall indicates that Perl 6 "uses . instead of -> to dereference an object" due to the idea that "The use of arrow where most of the rest of the world uses dot was confusing." I think that this is a Very Good Thing, for a couple of reasons.

One reason is less keystrokes; going from the dash key to the ">" key has always seemed a little bit of a pain:) The second reason is that this change will likely facilitate introducing people to using Perl. This will be one less thing for folks new to Perl to go "Huh?" at when you show them some code.

(OT) X terminal output speed on linux (zentara)

This is just a curious discovery I made while testing the urxvt

I ran this simple script, to print out 500,000 lines and timed it.Both the xterm and the urxvt have 20k scrollback buffers.

#!/usr/bin/perl use warnings;use strict;my $start = time;for (1..500000){print "$_\n";}my $end = time;my $diff = $end - $start;print "$diff\n";

To my utter amazement ( and I'm still wondering if it's an xterm bug), the xterm took a whopping 1067 seconds, while the urxvt only took 19 seconds.

I would be interested in hearing the results from other people, on their xterm speed, and maybe k-terminal, etc.

Anyone have clues as to why this occurs?

I'm not really a human, but I play one on earth.Cogito ergo sum a bum

The state of audio processing with Perl (Joost)

Hi there, fellow monks!

My name is Joost and I'm going to talk about audio processing in perl.

I've been playing with "live" (not really realtime, but close enough) audio generation/processing in perl for some time now and I'm not really impressed by the code on CPAN to do that.

For one thing, there appears to be no portable module to play or record audio with any kind of precision. I mean just some code to throw an audio buffer at the soundcard (OIW, in-process handling of data vs playing an audio file). Sure there's Audio::Play but that only supports monophonic sound and doesn't record.

Also, most audio modules use incompatible formats for their data and lots of them are badly documented and/or haven't been updated for years. In other words, coupling these modules is a mess (and slow).

More issues: reading and writing different audio file formats. I've taken a stab at that just now - but it still lacks some important formats.

Here are some of the things I feel are needed:

One audio data interchange format. Audio::Data seems to me to be a good starting point, but it lacks low-level access (i.e. a C level interface and packed float/double strings for fast processing).
A portable audio playback/recording library. Preferably one that will handle everything that Audio::Play handles but with unlimited channels and a JACK interface.
Some kind of processing API that allows coupling of perl & XS code. Something like my Audio::LADSPA module, but more flexible and with an easier Perl interface.

So here's my question: is anyone interested in this kind of thing? By myself I might be able to get this stuff done in a year or so, considering the time I've got to spend, but I'd really like people to supply questions, burning needs, code and/or documentation to get this thing on the right track.

Also, if you are writing any audio modules on CPAN, I'd like to know about it.

Please let me know I'm not the only one interested in this kind of thing :-)

In other words: audio people of perl, let's hear your crazy ideas!

Cheers. J.

"What should it profit a man, if he should win a flame war, yet lose his cool?"

RFC: more of that almost-book (billh)

Hi,Tell me what you think:
Exploring Programming LanguageArchitecture in Perl.
Some of you may remember I posted a link to this many moons ago.
I haven't really given up on it but it's become something of a spare-timeproject, I've finished the object-oriented extension and started writingthe chapter on continuations. I've corrected a lot of bloopers in the earlierchapters and I've also learned far more about XML andCSS than anyone should have to :-), but at least it looks prettier now.

Bill H

perl -e 'print sub { "Hello @{[shift]}!\n" }->("World")'

GUI toolkit choice (jbert)

Hi,
A quick search only showed a Tk/Gtk discussion from 2001, so I hope I didn't miss something and this is a FAQ. Also, I'm hoping that we can discuss things without having a religious war about which is 'best'.

That said, I'm interested in finding out more about the various options for GUI coding, particularly - but not exclusively - from perl.

The main options I know of are:

Gtk+ - originally written for the GIMP, used by the GNOME people and some other apps
QT - written by Trolltech, used by the KDE people and some other apps
Tk - from Tcl and ported to perl early on
WxWindows - seems good but I know little of it (the DrScheme IDE is written in it and seems decent)

All of these are cross-platform (at least Linux and Windows and I would guess Mac and 'general unix/X windows as well').

I've used Gtk+ from perl and C++ on Linux with good results and the same scripts runs on Windows (once Gtk is installed), which is nice.

If anyone has experience of more than one of these toolkits and can provide some thoughts on what they see as the most significant strengths and weaknesses of them I'd be grateful.

And I'd hesitate from using this thread to ask the popularity of the different toolkits, but perhaps that would make a good poll?

The purpose of testing (g0n)

Testing has been a bit of a problem for me over the last few months, as some will have heard from my CB rants. Not the 'make test and watch the TAP output come pouring out testing' - oh dear no. For much of the last 8 months, I've been doing a great deal of manual testing, of the 'type these commands, compare 50 fields in the database output with the 50 fields printed in the test script, putting a p (pass) or f (fail) next to each one, sign and date, then go on to the next one of the 150 tests' type. Yuk! Human beings did not evolve to do this kind of thing.

So I enthused about automated testing, tried to get my colleagues on side (with a degree of success), but there is one major stumbling block. The Quality Assurance team. Correctly, they're independent. Unfortunately, they don't permit automated testing unless it's done with an approved, validated (company validated that is) automated testing tool (TestDirector, for example). They're also entirely non technical, and really only concerned with the quality (in terms of change tracking, consistency etc) of documents. The net result of this is that testing is an extraordinary time overhead, and we have to think carefully about what tests to run for a given release. This means that testing is not as thorough as it could, or should be, and bugs creep through. Not as many as you might expect in this situation, but nevertheless more bugs find their way into production than I consider acceptable.

This stalemate has been going on for some time. Years actually. Then on monday something happened. A big, fat bug in some of my code showed up in production. Embarassing. This bug meant that I now have to run a manual report daily for the next couple of weeks until we can patch, to take the place of the automated report that I broke. Embarassing and irritating, especially since another bug had been emergency fixed that morning.

At that point I realised that, just like the QA people, I'd lost sight of the real issue - testing is about finding bugs, not filling in forms. If the formal, QA approved testing is less thorough than it should be, we have to make sure that the code gets properly tested some other way.

So I got to work writing unit tests with [cpan://Test::More].

3 days work later I've got one of the components up to 50% test coverage and found 3 bugs in edge cases that have never showed up in production. Unfortunately we probably can't test everything this way, since the perl code is only one component, running in an embedded perl interpreter inside a proprietary application. Integration testing still needs to be done the old way, so our test overhead has gone up by the amount of effort needed to write unit tests, but at least the chances of bugs getting through is reduced.

Another advantage of testing with the Perl testing modules is the availability of [cpan://Devel::Cover]. Because the unit testing is informal and unvalidated, test cases can be added any time. If someone has a few minutes spare, a quick run of the test suite with [cpan://Devel::Cover] will show up opportunities for improving the testing.

Something else I'd lost sight of is the fact that we primarily want to test our code, not someone elses. A lot of our code depends heavily on [cpan://Net::LDAP], so the need to provide a correctly configured directory server looked like a barrier to automated testing. However, end to end integration testing covers the 'get data back from the directory server' test case. If there's no directory server easily available for unit testing, we can invade the dependency's name space to let us test our own code:

use strict;use warnings;use Test::More;require 'MyCode.pl';*Net::LDAP::bind = \&ldapbind;*Net::LDAP::new = \&ldapnew;MyCode::bindToLDAP("hostname","port","cn=binddn","password");sub ldapnew{my $host = shift;cmp_ok($host,"hostname:port","Check that Net::LDAP::new receives the right params");}sub ldapbind{my %params = @_;my %comparison ={dn=>"cn=binddn",password=>"password",};is_deeply(\%params,\%comparison,"Check that Net::LDAP::bind gets the right params");}

I'm hoping I can get the vendor of the core application to give us information on externally accessing the test functions in their application via XS, so that we can extend the unit tests to include the application config. I'm not hopeful on that front, but it's worth a try.

One final note: in the mindless drudgery of manual testing, I'd also forgotten how much fun one can have writing tests to try and break things :-)

--------------------------------------------------------------

"If there is such a phenomenon as absolute evil, it consists in treating another human being as a thing."
John Brunner, "The Shockwave Rider".

Make sure you're solving the right problem (talexb)

This is a followup on some recent posts of mine .. in case anyone new to Perl wonders if it is ever used for real work.

My web application handles document processing (conversion from PDF to XML), and it's currently running on a dozen servers, the busiest of which is used by two teams, one here in Toronto and the other in Mumbai, India. The work day starts about midnight local time (930am in Mumbai) and goes to about 8 or 9pm, and lately we've been running 7 days a week to keep up with demand. If things go South, it lands in my lap. I hate pages at 4am, so I do what I can to avoid that situation.

Anyway, recently, I wrote about a problem I'd been having with a sleepy CGI, and eventually rediscovered the excellent tool strace thanks to [jbert|some] [sgifford|respondents]. But that led me to my next problem, as to why the CGI would be freezing at

select(5, [4], [], [4], NULL

as strace was showing me. After some more research, I discovered this was the connection to the database. Could it be as simple as the CGI waiting for the database that was causing the problem?

Oh boy. The answer is yes, as it turns out. And the CGI was not 'going to sleep' -- it was continuing to run, but was patiently waiting for the database, not sleeping.

And the solution to the database slowdown (PostgreSQL, in my case) was the simple application of

ANALYZE VERBOSE DOCUMENTS;

and the response time for the main query went from 20 seconds to about 550ms. Yay!

Since I've seen this performance problem before, I'm now going to keep an eagle eye on the system today and find out how long it takes before the performance starts to drop, then set up a cron job to ANALYZE the suspect table again, most likely every four hours or so.

This is all a result of watching my system closely, something I think is extremely valuable in any Engineering job -- staying on top of the performance of your system is very important in my current situation of developer, supporter and maintainer.

The moral of the story is, follow the data (I know, it sounds like CSI). Processes very rarely 'go to sleep' unless they're told to. Where's the CPU time (or bandwidth, or your other resources) going? Follow that lead, and you'll find the answer to your problem.

Alex / talexb / Toronto

"Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

Check out g2 .. a new graphics interface (zentara)

I just saw g2 today on http://freshmeat.net. It is a c lib for generating 2d graphics. I thought it would be worth mentioning here, because it contains a perl module interface, and it works quite well.

It's big advantage is that it can output to multiple types simultaneously. See screenshots at above link.

I'm not really a human, but I play one on earth.Cogito ergo sum a bum

Automated regression testing (rinceWind)

Where I am working, we have a requirement to provide an automatic regression test suite for an existing live application. The application is C++, not perl (though there is some perl glue already deployed for application monitoring). I'd quite like to do this using Perl, as I think the language is ideally suited. I also have permission to use any CPAN modules of my choice. I will still need to convince colleagues of the wisdom of this choice.

I use the Test::More and Test::Harness mechanisms in unit tests for modules used by the application glue. But, the full regression test is a much bigger prospect. On the automation side, I will need to be doing the following:

Starting and stopping application components (daemons)
Interfacing to the application through command line tools
Trawling and tailing log files to detect events
Database access, including updates

It occurs to me that there is a fundamental difference between an automation "step" and a test. If a test fails, I want the script to carry on and run more tests. If a step fails, I want it to pause requiring manual intervention. The pause could eventually time out, bailing the run, allowing the whole suite to be run in "lights out mode" unattended over a weekend.

My plan is to put the automation steps in first, and add the tests afterwards.

I'd be very interested in hearing from anybody who has attempted or achieved anything like this before. What tools have already been written that can help me? What problems and gotchas should I be careful to avoid?

--

Oh Lord, wont you burn me a Knoppix CD ?
My friends all rate Windows, I must disagree.
Your powers of persuasion will set them all free,
So oh Lord, wont you burn me a Knoppix CD ?
(Missquoting Janis Joplin)

Finding "Accidental Contributions" with Google Code Search (jasonk)

Someone mentioned today on digg that it was relatively easy to find thousands of developer email addresses by searching for a suitable regexp on Google Code Search. Since I was already bored (hey, I was reading digg, it couldn't have been too busy a day) I decided to search for myself and see what should pop up.

The first couple of pages were unsuprising, mostly links to my modules on CPAN, but then I stumbled across something interesting. In a little program I had never heard of was a copy of Geo::ShapeFile. The fact that they were distributing my module along with their program isn't a big deal, it's licensed under the Artistic license, and they credited me and everything else, what caught my eye about it was this comment in the LICENSE file:

The version of Geo::Shapfile in this distribution includes fixes for machines of different endianness, not yet available on CPAN.

I downloaded the source and perused it a bit and discovered that the author was even kind enough to include RCS files for the libraries that had been modified, which made it very easy to figure out what local changes he had made, and to find that indeed there were fixes for some endian-related issues in the module which have vexed me for quite a while (sadly my current job does not involve mapping, so I don't have as much time to put into this module as I would like). The most interesting part was that the solution he used seems to be better than my own solution had been, and appears to have been written before the problem was first reported to me.

So, have you checked Google to see what people might have neglected to contribute to your projects?

Along the way I also found some other interesting tidbits that I hadn't known...

And last (and certainly least) a reminder that I was even more of a geek in college

We're not surrounded, we're in a target-rich environment!

Collective Hacking Projects: Let's Hear About Them! (jkeenan1)

About two years ago this time I helped organize a Perl Seminar NY contingent of the Phalanx project. When I pitched the project to our members, I emphasized the joys of collective hacking. And it proved very enjoyable indeed (and profitable for the bar where we met to do this hacking!)

We didn't do as much collective hacking in the 2005-06 season, but with our 2006-07 Perl Seminar NY season about to begin, I'd like to rev up some interest among our members.

I know of some of the collective hacking projects going on in the Perl community right now, and I'll be participating in the Chicago Perl hackathon next month. But just in case I've missed any, I'd like to ask the Monks to respond to this Meditation by posting ongoing projects (e.g., Vanilla Perl, Module::Build) and their weblinks.

And, for extra credit, if you think that any of these projects could be broken up into components suitable for face-to-face hacking by local Perlmonger groups, please so indicate.

I like online projects, but I don't like to hack or drink alone.

Thanks in advance.

Jim Keenan

PPM - It's a GUI!! (mikasue)

I recently installed ActivePerl 5.8.8 Build 819 on a desktop. After reading the release notes, I was anxious to explore the new version of the Perl Package Manager. At first the GUI was a little confusing becuase it's not clear what is being listed. However, after playing with it, I think it's great.

Listing and installing packages are much easier tasks to complete. Adding repositories are just as easy as typing in the url and giving it a name. I can see all packages installed or all packages available for install. With the click of a button I can upgrade and verify packages.

All is not lost for those who like the command line shell. Doing a ppm repo will list all your repositories in a nice borded table. Other commands are available also just do a ppm help at the command line.

I like the new version of PPM. What do you monks thinks about it?

Oracle OpenWorld Pilgrimage (Thwack)

I will be traveling to Oracles OpenWorld conference in San Francisco (Oct. 22-26 2006). I am curious if any other Perl Monks and/or Perl related programmers will be there. I would like to meet up and hi and maybe do a lunch or something of that sort. It would be fun to meet other developers/administrators that use or focus on Perl with Oracle.

Startups Ideas from Perlmonks (monkfan)

Having visited many social networking sites like reddit, del.icio.us, Technorati, Digg, makes me think Perlmonks in some sense are far more superior than those guys.

There are too many features that is very fundamental which Perlmonks has that they don't have, e.g. Chatterbox, Bookmarklet, voting systems, and the closely knitted comunity (my knowledge is blatantly pale wrt PM wealth of features).

Do you guys have ever thought, what kind of business idea one can learn from our Perlmonks?

Regards,
Edward

RFC: A Beginners Guide to Fuzzy Rules-Based Systems (lin0)

Greetings Fellow Monks,

This is a 50-minutes seminar I am giving next week as a part of an Image Analysis tutorial I am teaching. In this seminar, I introduce Fuzzy Sets and the AI::FuzzyInference module to the students.

The tutorial is divided into six parts:

Introduction to Image Analysis;
Introduction to the [http://www.itk.org|Insight Toolkit] (a toolkit for image processing, registration, and segmentation);
Introduction to Fuzzy Sets (the first part of this seminar)
Introduction to the AI::FuzzyInference module (the second part of this seminar);
Design strategies for Image Analysis Applications; and
Development of an Image Analysis Application.

The operating system is Linux

The students are supposed to have at least a basic knowledge of Perl.

The goal is to have the students walking out of the 50-minutes seminar knowing how to use the AI::FuzzyInference module and ready for a 1-hour seminar in which we use the module in an image analysis application.

Any comment or suggestion will be appreciated, especially with the goal of making this (either as-is or modified) into a piece suitable for the Tutorials section.

Thank you,

[http://perlmonks.org/?node_id=565709|lin0]

Updates:

Update 1:

When calling the new method, I am now using a Direct Object Syntax (my $fis = AI::FuzzyInference->new;) instead of the Indirect Object Syntax I was using before. Thank you to [http://perlmonks.org/index.pl?node_id=17000|ovid] for pointing that out.

Update 2:

I added some comments about what the output of the script is when you run it. Thanks to [http://perlmonks.org/?node=talexb | talexb] for pointing that out

Update 3:

Minor gramatical mistake fixed.

From:

Fuzzy Sets are a special type of sets that are able to express the notion of partial membership (as a value in the interval [0, 1] ) of an element to a particular group.

To:

A Fuzzy Set is a special type of set that is able to express the notion of partial membership (as a value in the interval [0, 1] ) of an element to a particular group.

Thanks to [http://perlmonks.org/?node_id=31503|runrig] for pointing it out

Introduction:

A [http://en.wikipedia.org/wiki/Fuzzy_set|Fuzzy Set] is a special type of [http://en.wikipedia.org/wiki/Set|set] that is able to express the notion of partial membership (as a value in the interval [0, 1] ) of an element to a particular group. [http://en.wikipedia.org/wiki/Fuzzy_set|Fuzzy Sets] were first proposed by [http://en.wikipedia.org/wiki/Lotfi_Zadeh|Prof. Lotfi Zadeh] of the [http://en.wikipedia.org/wiki/University_of_California%2C_Berkeley|University of California at Berkeley] in the 1965 paper [http://www-bisc.cs.berkeley.edu/Zadeh-1965.pdf|Fuzzy Sets]. Fuzzy Sets have been successfully used in applications that range from image processing and pattern recognition to decision support systems and fuzzy control systems. For a good overview on Fuzzy Sets and related terms ([http://en.wikipedia.org/wiki/Fuzzy_logic|Fuzzy Logic] and [http://en.wikipedia.org/wiki/Fuzzy_control_system|Fuzzy Control Systems]), I encourage you to have a look at the [http://en.wikipedia.org/wiki/Main_Page|Wikipedia].

Main Operations:

The AND, OR, and NOT operators of boolean logic exist in fuzzy logic. The AND operator is used to find the intersection between two sets (it is usually defined as the minimum between two Fuzzy Sets). The OR operator is used to find the union between two sets (it is usually defined as the maximum between two Fuzzy Sets). The NOT operator is used to find the complement of a set (it is usually defined as the subtraction of a membership function from 1)

Rules-Based Systems:

Rules are usually expressed in the form:

IF variable_1 IS fuzzy_set_input_1 AND variable_2 IS fuzzy_set_input_2 THEN action IS fuzzy_set_output

the IF part is called the "antecedent" and the THEN part is called the "consequent".

In this example, the two input variables are "variable_1" and "variable_2". They have values defined as fuzzy sets. The output variable, "action", is also defined by fuzzy sets.

To make a decision based on a set of rules, a rules-based system follows these steps:

All the rules that apply are invoked, using the membership functions and truth values obtained from the inputs (by a process called fuzzification), to determine the result of the antecedent.
This result in turn will be mapped into a membership function and truth value controlling the output variable. This process is known as implication. Two of the more common implication functions are: clipping (the fuzzy set is clipped to a value given by the level of activation of the input variables) and scaling ( the fuzzy set is multiplied by a value given by the level of activation of the input variables).
These results are combined by a process called aggregation. One common approach for the aggregation involves using the maximum of the implicated sets.
Finally, a process known as defuzzification is used to compute a single value that is representative of the aggregated fuzzy set. A typical defuzzification approach involves computing the centroid of the aggregated fuzzy set

The AI::FuzzyInference module:

Consider a system for assigning awards to Deserving Monks, directed by a Perl script. The script has to make decisions based on the "quality" of the nodes written by the Deserving Monks and the "speed" on which the Deserving Monk answer the cry for help from other monks. Therefore, one could have input variables: "quality" and "speed" and output variable "award".

The variable "quality" in this system can be subdivided into a range of values: "bad", "ok", "good", "excellent". The variable "speed" in this system can be subdivided into a range of values: "slow", "regular", "fast", "fastest". The variable "award" in this system can be subdivided into a range of values: "minimum", "small", "good", "excellent".

Among the rules for computing the 'award', we could have:

IF quality IS bad AND speed IS slow THEN award IS minimumIF quality IS good AND speed IS slow THEN award IS smallIF quality IS ok AND speed IS fast THEN  award IS goodIF quality IS excellent AND speed IS fast THEN award IS excellent

Here, we have a step by step guide to implement the corresponding Rules-Based System using the AI::FuzzyInference module:

First, we need to include the module

use AI::FuzzyInference;

Second, we need a FuzzyInference object to work with, so we create a new one

my $fis = AI::FuzzyInference->new;

Third, we define the input variables. For the input variable quality we have

$fis->inVar('quality',      0, 10,             bad      =>  [ 0, 1, 2, 1, 4, 0 ],             ok       =>  [ 2, 0, 4, 1, 6, 0 ],             good     =>  [ 4, 0, 6, 1, 8, 0 ],             excelent =>  [ 6, 0, 8, 1, 10, 1 ],           );

on the first line, from left to right:

$fis is the object identifier
inVar indicates that we are entering an input variable
'quality' is the name of the input variable
0 is the minimum acceptable value for the input variable
10 is the maximum acceptable value for the input variable

on the second line, from left to right:

bad is a fuzzy set identifier
0 is a value of the input variable
1 is the membership value for an input value equal to 0
2 is a value of the input variable
1 is the membership value for an input value equal to 2
4 is a value of the input variable
0 is the membership value for an input value equal to 4

The graphical representation of fuzzy set "bad" is:

    |1.00|BBBB       |    B   0.50|     B       |      B         |________B________________    0   2    4    6    8    10

Fourth, we define the output variable.

$fis->outVar('award',         0, 100,              minimum   => [  0,1, 20,1,   40,0 ],              small     => [ 20,0, 40,1,  60,0 ],              good      => [ 40,0, 60,1,   80,0 ],              excellent => [ 60,0, 80,1, 100,1 ],          );

on the first line, from left to right:

$fis is the object identifier
outVar indicates that we are entering an output variable
'award' is the name of the output variable
0 is the minimum value for the output variable
100 is the maximum value for the output variable

on the second line, from left to right:

minimum is a fuzzy set identifier
0 is a value of the input variable
1 is the membership value for an output value equal to 0
20 is a value of the input variable
1 is the membership value for an output value equal to 20
40 is a value of the input variable
0 is the membership value for an output value equal to 40

The graphical representation of fuzzy set "minimum" is:

    |1.00|MMMM       |    M   0.50|     M       |      M         |_______M_____________________    0  20  40  60  80 100

Fifth, we add the rules

$fis->addRule(     'quality=bad       & speed=slow' => 'award=minimum',    'quality=ok        & speed=slow' => 'award=minimum',    'quality=good      & speed=slow' => 'award=small',    'quality=excellent & speed=slow' => 'award=small',

on the first line, from left to right:

$fis is the object identifier
addRule indicates that we are entering a set of rules

on the second line, from left to right:

'quality=bad & speed=slow' is the antecedent
'award=minimum' is the consequent

Sixth, we compute the value of the award for a given set of values for the input variables.

$fis->compute( quality => $quality_of_service,               speed   => $speed_of_service);

Finally, to recover the value of the output variable, we use:

my $award = $fis->value('award');

If we wanted to know the values of the main operations and the main functions, we use:

$fis->operation( '&' );  #for the AND operation$fis->operation( '|' );  #for the OR operation$fis->implication();     #for the implication function$fis->aggregation();     #for the aggregation function$fis->defuzzification(); #for the defuzzification

In addition, we could select different values for the operations and the implication function by using:

$fis->operation( '&', 'difference');$fis->operation( '|', 'sum');$fis->implication( 'scale' );

As a final note, another tutorial for the AI::FuzzyInference module is available in [1]

[1] Ala Qumsieh, Fuzzy Logic in Perl, The Perl Journal, pp. 3-7, June 2003

The complete script to assign the awards to Deserving Monks is given below.

When you run the script, it will ask you first for the quality of service of the Deserving Monk (it should be a number between 0 and 10. This number represents the average quality of the nodes written by the Monk). Let's say that we think the quality of service is "ok" so we enter the value "4". Then, the script will ask you for the speed of service (it should be a number between 0 and 10. This number represents how fast the Monk replies to the requests for help). Let's say that we consider that the Monk answer "very fast" to the requests (the Monk is not the fastest but is closer to the fastest) so we enter the value "7". After you enter that value, the script will show you the value of the award (a number between 0 and 100 representing the percentage of the maximum possible award).

For this example, the award will be 60. In addition to the value of the "award", the script will also inform you of the values of "&", "|", "implication", "aggregation", and "defuzzification" it used in the computation. At the end of the script, I show you how to modify the values of "|", and "implication". This is why you also see a print out indicating the "new values of | and implication".

Enjoy it!

#!/usr/bin/perluse warnings;use strict;use AI::FuzzyInference;############### problem: to assign an award to a deserving Monk###############my $fis = new AI::FuzzyInference; # this indirect object                                   # notation was replaced                                   # for the direct                                   # notation below as                                    # suggested by ovidmy $fis = AI::FuzzyInference->new;############### input variables: inVar# 'service' relates to the quality of the nodes written by the Monk# 'speed'   relates to how fast the Monk offers a hand to a fellow#           monk in need##############$fis->inVar('quality',          0, 10,             bad         =>   [ 0, 1, 2, 1, 4, 0 ],             ok          =>   [ 2, 0, 4, 1, 6, 0 ],             good        =>   [ 4, 0, 6, 1, 8, 0 ],             excellent   =>   [ 6, 0, 8, 1,10, 1 ],           );$fis->inVar('speed',            0, 10,             slow        =>   [ 0, 1, 2, 1, 4, 0 ],             regular     =>   [ 2, 0, 4, 1, 6, 0 ],             fast        =>   [ 4, 0, 6, 1, 8, 0 ],             fastest     =>   [ 6, 0, 8, 1,10, 1 ],           );############### output variable: outVar# 'award'   relates to the quality of the award to be given##############$fis->outVar('award',           0, 100,              minimum    =>   [  0,1, 20,1,  40,0 ],              small      =>   [ 20,0, 40,1,  60,0 ],              good       =>   [ 40,0, 60,1,  80,0 ],              excellent  =>   [ 60,0, 80,1, 100,1 ],          );############### Adding the rules: addRule##############$fis->addRule(      'quality=bad        &   speed=slow'       =>  'award=minimum',     'quality=ok         &   speed=slow'       =>  'award=minimum',     'quality=good       &   speed=slow'       =>  'award=small',     'quality=excellent  &   speed=slow'       =>  'award=small',              'quality=bad        &   speed=regular'    =>  'award=minimum',     'quality=ok         &   speed=regular'    =>  'award=small',     'quality=good       &   speed=regular'    =>  'award=small',     'quality=excellent  &   speed=regular'    =>  'award=excellent',              'quality=bad        &   speed=fast'       =>  'award=small',     'quality=ok         &   speed=fast'       =>  'award=good',     'quality=good       &   speed=fast'       =>  'award=good',     'quality=excellent  &   speed=fast'       =>  'award=excellent',              'quality=bad        &   speed=fastest'    =>  'award=small',     'quality=ok         &   speed=fastest'    =>  'award=good',     'quality=good       &   speed=fastest'    =>  'award=excellent',     'quality=excellent  &   speed=fastest'    =>  'award=excellent',         );############### Requesting information from the user:# $quality_of_service, $speed_of_service##############my ($quality_of_service, $speed_of_service);print "Please, enter the quality of service [0, 10]: ";chomp ($quality_of_service = );print "Please, enter the speed of service   [0, 10]: ";chomp($speed_of_service =  );############### computing the "award" given the "quality_of_service"# and the "speed_of_service" as entered by the user# # Note: we are using the default values for:#       & (min),#       | (max),#       implication (clip), #       aggregation (max), and#       defuzzification (centroid)##############$fis->compute( quality => $quality_of_service,               speed   => $speed_of_service);############### reading and showing the value of the 'award'##############my $award = $fis->value('award');print "\nAward (quality = $quality_of_service, speed = $speed_of_service) [0, 100]: $award\n\n";############### showing the values of:# &, |, implication, aggregation, defuzzification##############print "values of &, |, implication, aggregation, and defuzzification\n";print "& = ", $fis->operation( '&' ),"\n";print "| = ", $fis->operation( '|' ),"\n";print "implication = ", $fis->implication(), "\n";print "aggregation = ", $fis->aggregation(), "\n";print "defuzzification = ", $fis->defuzzification(), "\n\n";############### modifying the values of | and implication##############$fis->operation( '|', 'sum');$fis->implication( 'scale' );############### showing the new values of: | and implication##############print "new values of | and implication\n";print "| = ", $fis->operation( '|' ),"\n";print "implication = ", $fis->implication(), "\n";

Maintaining State with Runtime Traits (Ovid)

Some of you might recall chromatics review of Class::Trait, a module which implements traits (PDF) in Perl. He gave a couple of good examples of how one can use traits and I figured a follow-up on one of my many real world uses of traits and how it has simplified my code.

I was recently working on a command line program and came upon the following line of code:

@servers = map { $_->{selected} ? $_->{obj} : () } @servers;

That is just ugly for a variety of reasons, but heres why it was created. We were presenting the users with lists and they needed to select one or more items from those lists. When the program was written, only one type of item (a server object) was created for this list. To track whether a server was selected from the list, the following data structure was used.

my $list_item = {     obj      => $server,     selected => 0,};

And that leads to to the above rather ugly map. In fact, the map statement could probably be written more clearly as follows:

@servers = map  { $_->{obj} }           grep { $_->{selected} } @servers;

Thats easier to read, but its still pretty ugly. Also, since were using hash keys, theyre really easy to mistype. And on top of that, I now need to reuse this construct for objects of different types so I was looking at potentially having to use that data structure in more than one spot. Now when writing code, having to essentially attach extra state data to objects is quite common. Ive worked with plenty of classes which have notes() methods or something similar which allow you to shove anything in there you like. For classes which dont provide this, we do all sorts of tricks like, in this case, wrapping them in data structures which can contain the extra data or separating the objects into different data structures depending on what extra data you want to capture (@selected and @unselected arrays come to mind). Or maybe you pass the objects one at a time to target methods which take extra arguments denoting the metadata. Who knows? Personally, I think traits are a better way to handle this.

All of these techniques have pros and cons, but what I really want is for the object to have that data available, not for me to jump through hoops to preserve that data. I cant simply build all possible metadata types into all objects which might need them because thats bloating my code for no reason. Instead, I want a trait which I can apply to a class at runtime and provides just what I need for my code. My first refactoring was a bit of a kludge, but what I have now looks a bit like this:

     1  package Trait::Selected;     2     3  use strict;     4  use warnings;     5     6  use Class::Trait 'base';     7     8  our @REQUIRES = qw(id);     9    10  my $get_id = sub {    11      my $self = shift;    12      return sprintf "%s %d" => ref $self, $self->id;    13  };    14    15  my %is_selected;    16    17  sub is_selected {    18      my $self = shift;    19      my $id   = $self->$get_id;    20      $is_selected{$id} ||= 0;    21      return $is_selected{$id} unless @_;    22      $is_selected{$id} = shift;    23      return $self;    24  }    25    26  1;

Line 6 says this is a trait. Line 8 says any object which uses this trait must supply an id() method. Line 10 is how you supply truly private methods or functions with traits and line 17 is the actual method that each class will now have.

In my actual code, its completely wrong for me to add an is_selected() method to a bunch of classes just because I need them for this one script, so Im going to apply the trait at runtime with this:

Class::Trait->apply( $some_class, 'Trait::Selected' );

Of course, if you dont want this to apply to all instances of that class, you can apply it just to a single instance by replacing $some_class with an instance of the object in question. (Prototype-based programming, anyone?)

Now that I have that, I can use is_selected() as a getter/setter (accessor/mutator, if you prefer) for this boolean property. So if a particular object from a list I present to the user is selected, I can just do this:

# This used to be $item->{selected} = 1; and is really easy to mistype$object->is_selected(1);  # Mistyping the method becomes a runtime failure

And this:

@servers = map  { $_->{obj} }  grep { $_->{selected} } @servers;

Becomes this:

@servers = grep { $_->is_selected } @servers

Thats much easier to read.

While this made much of my code easier to read and work with, it does show an ugly problem with trying to add runtime traits to languages not designed to support them. What happens when the object goes out of scope? The trait still retains that objects selected value and its tricky to remove. Fortunately for my code, this turns out to not be an issue, but using runtime traits to maintain state is problematic. While Perl 6 doesnt have this problem, for the time being, I need to solve it for Class::Trait. I have a reasonably workable approach, but it will take some time to implement. Once again, I find myself pining away for Perl 6 for the things that it makes easy.

Cheers,
Ovid

New address of my CGI Course.

Why did the 6-legged camel try to cross the sigil wall? (metaperl)

to avoid joining theHURD of Agile Languages

I've been a pretty heavy duty user of Python lately and at first it was rather tough on me because I had no stop symbols anywhere - there are very few special characters in Python. But after awhile I got used to that.

Then I realized something. Python has a number of sequence types built in and C++ has even more, which they refer to as containers.The neat thing about this army of sequence types is they share a common set of methods. Sometimes via inheritance and sometimes via typed function calls.

Perl5 ships with a single list type called an array. Other list-like types don't get the sigil treatment and have to be operated on via a completely different syntax.

Please RSVP as to whether Perl 6 will allow one to use various container implementations under the hood instead of being limited to a particular implementation for the @ sigil type.

We already have afull library of containers implemented in Perl, it's just a question of whether it will be easy to pop them into the array sigil on demand and use like any other array.

(!sigil_overload) and # hexacamel's fate is a no brainer. like pentacamel {   it will trip and fall    on   => the sigil wall   when => overloaded with various containers.}

Convert::Bencode_XS maintainership (giulienk)

I have a problem: I'm receiving several bug reports about my module Convert:Bencode_XS, but I'm not able to follow all of them. Also I programmed the XS part "just for fun", not having any big C skills. It seems the module catched up on popularity, people are using it and producing patches for the C part which I have problem evaluating. So I'm searching for some Perl programmer with experience in XS which would like to take over maintainership. Please post in this thread or send me a message, thanks.

$|=$_="1g2i1u1l2i4e2n0k",map{print"\7",chop;select$,,$,,$,,$_/7}m{..}g

RFC: UNIX shell to Perl converter (cdarke)

It is with trepidation that I propose this old chestnut.

I am aware that the collective wisdom of such as Tom Christiansen and the Llama book is that it is not worth the effort to just code a language conversion between (say) Korn shell and Perl.

The reasons seem to be that a) most of the functionality of a shell script is done by external programs, and b) the languages are so different that a different approach/design is required.

Valid points, although a well written shell script will not use more external programs than it needs, and differences have not prevented a2p, s2p or find2perl.

Well I'm giving it a go anyway someone should.

I want to take the tedium out of converting a shell script. The interesting and rewarding parts of shell to Perl conversion (for me) are mostly taking the multiple (and often unnecessary) calls to external programs and replacing them with fast Perl constructs. I'm happy doing that manually, or using various CPAN modules.

That's not to say that automatic conversion of many common external program calls is not possible.

What really bores the pants off me is converting [ and [[ to (; 'then', 'fi', 'do', 'done', to curlies, removing the 'in' from a 'for', changing gt to > and > to gt. Replacing rm with unlink, -n to 'defined', putting a $ on the left side of an assignment, and so on ad nauseum. Just don't ask about the trailing semi-colon.

Right now I can convert and run some scripts without manual intervention, but still have a long way to go. This is a huge job, I have been playing with a basic language parser and converter for about 6 months (off and on) and am still in an early phase.

Time to pause again to wonder how many others have trod this path. I can see footprints and traces of torn clothing and blood on the trail.

Cue a monk with a link to someone who did this ten years ago.

Portable Proc modules (xdg)

In reading Killing a hanging child process, most of the discussion seemed very Unix-oriented. (E.g. focusing on "fork" which is a terrible way to run a system command on Windows. See perlfork for details.)

One reply recommended Proc::Reliable, so I went to check that out, and then started looking at other similar modules on CPAN.

So far, the "best" -- loosely defined by me to mean portable, featureful and relatively efficient -- appears to be Proc::Background. (See this review, too.) Has anyone else used it? Any thoughts or reactions?

Here's a very short list of modules that I quickly examined with some observations.

Proc::Background: Separate sub-modules for Unix and Win32 modes; API could be extended to other OS as necessary.
Proc::Command: Piped open instead of backticks
Proc::Reliable: Uses fork, even on Win32
Proc::Simple: Tests fail/hang on Win32. (Not so simple, I guess).
qr/Proc::Fork/: All these would appear to use fork, of course.

What modules have I missed? What are your experiences or suggestions for portable process management?

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Tabs vs Spaces lets give this a go (EvanCarroll)

OK this is really getting on my nerves, not that people don't follow my convention, as much as people thinking their wrong convention is more righteous than mine. So I'm going to sum up the case, of spaces vs. indentation. I'll start out this diatribe by stating this is pro-tab, and DC is undeniably wrong, this is the most blatant flaw in the PBP.

Tabs belong in one place in a document; and that place only. That place is in between the left-margin and the first word character. This is the tab-zone.
Nothing other than a tabs, belongs in this tab-zone
This tab-zone encodes not display, but tabs as code-metadata, the level of indentation for your code or depth, which for my purposes can be used interchangeably.
This metadata can be utilized by external code preprocesses, editors, renderers, searching mechanisms or anything of that sort.
Because you are encoding a level of indentation rather than an element of presentation everyone with a decent modern editor can have their cake and eat it too. ie. you can set your presentational tab-space, or tabspace to 2, your friend can set it to 4, or you can have a script based on the time of day alternate it. It doesn't mater. This will eliminate the conversion toandfro spaces. Changing this indentation will still permit all operators to line up, they will just be shifted more, and that shift will be linear to their level of indentation.

Let me explain further why this metadata is important. Lets say you want to target where in your document the sub foo {} is. You preform a search on google.com's nifty codesearch, it yeilds what? foobar.pl, line 50, char X.

Should X include those spaces, or not? If so how many should it assume for each level of what appears visually as indentation? Should it display depth? You might make the argument that a line should only be 80characters? (or 78 if you are an O'reilly author who likes to justify modern coding practices with terminals from the 60s'. But thats alltogether a different argument, to which I personally subscribe, but because of preference rather than false-reasoning. ) The answer is oblivious to the length of the line, whither it be 5000 characters or 50 characters, the x-axis is important, and by using spaces to encode depth as presentation you are dually losing data, you no longer know the level of indentation, and you now have to assume what the programmer meant through heuristics.

This is all a fault of spaces, and the lack of a universal space->depth variable, but then again, spaces are being used in this context to alternate the presentation, which is fundamentally wrong. Another thing I would like to point out is something the space-proponents like to take advantage of, and it is often moot because it is an undeniable evil, and that is spaces do mark something up, and that is something we all use @words = split ' ', "foo bar baz" it is unfortunate we use them at all for presentation. But lets not make the same mistake for tabs.

Now finally the catch, I admit there is one thing I have neglected. The only fault in this argument is minor and disappearing more every day, and that is legacy editors, or non-programming editors. Lets be blunt now:
Qualitatively how useful is a programmer who uses notepad?, What does it say about a programmer who uses notepad?, How productive can the programmer be if he opts to use tools that are so inept in the productivity dept.?

You are only as good as your tools, "\t" is one that shouldn't be neglected.

Evan Carroll
www.EvanCarroll.com

(RFC) XML::TransformRules (Jenda)

Good `localtime` monks,
I was trying to find a better way to extract data from XML and I think I might have an idea. Maybe it's stupid, maybe there already is a module with that interface, but maybe not and maybe others will like the idea as well and there will be a point in implementing it.

The basic theme goes like this ... XML is basicaly a serialized data structure and what we want to get as we are parsing it is basicaly a data structure as well. What we get from the parsers is too generic and too complex (DOM, XML::Parser's Tree style) or too restricted (XML::Simple). Either we end up with a structure that's hard to use or we have to restrict the set of XMLs we can handle ... and end up with a structure that likewise may be more complex than necessary. What might help would be a way to specify the rules by which to transform the individual tags (with their attributes and content) to whatever data structure we need to end up with. And apply the rules from the leaves all the way to the root, either producing a simplified datastructure containing just the stuff we are interested in in a format that's convenient enough or process the partial structures as produced by applying the rules.

An example is worth a thousand words, so here's one

$xml = <<'*END*'   ...  ...  ...     ...   ...   ...   ...  
    ...  ...  ...     ...   ...   ...   ...  
 *END*%rules = (_default => sub {$_[0] => $_[1]->{_content}}, # by default we are only interested in the content and we want # the parent to access it as an attribute of the same name as was the tagbogus => undef, # means "ignore"address => sub {address => "$_[1]->{street}, $_[1]->{city} ($_[1]->{country})"},# let's convert the address to a single stringperson => sub {'@person' => "$_[1]->{lname}, $_[1]->{fname}\n<$_[1]->{email}>\n$_[1]->{address}"}# push the stringified data into the @{$parent->{person}}doc => sub { join( "\n\n", @{$_[1]->{person}})});print XML::TransformRules::Parse( $xml, \%rules);

or, a bit more complex

$xml = <<'*END*'   ...  ...  ...     ...   ...   ...   ...  
     123-456-7890   663-486-7890   663-486-7000      ...  ...  ...     ...   ...   ...   ...  
     663-486-7891   *END*%rules = (_default => sub {$_[0] => $_[1]->{_content}},bogus => undef,address => sub {address => "$_[1]->{street}, $_[1]->{city} ($_[1]->{country})"},phone => sub {$_[1]->{type} => $_[1]->{content}},# let's use the "type" attribute as the key and the content as the valuephones => sub {delete $_[1]->{_content}; %{$_[1]}}, # remove the text content and pass along the type => content from the child nodesperson => sub { # lets print the values, all the data is readily available in the attributesprint "$_[1]->{lname}, $_[1]->{fname} <$_[1]->{email}>\n";print "Home phone: $_[1]->{home}\n" if $_[1]->{home};print "Office phone: $_[1]->{office}\n" if $_[1]->{office};print "Fax: $_[1]->{fax}\n" if $_[1]->{fax};print "$_[1]->{address}\n\n";return; # the  tag is processed, no need to remember what it contained},);

Even though I talked about transforming the data structure there is nothing preventing us from applying the rules as we parse the document as soon as we encounter the closing tag. So we do not have to load the whole document to memory if we don't need it all at once. And even if we do we have a chance to trim it down as we read it and end up with a much smaller data structure.

The rules receive two parameters, the name of the tag and a hash containing the attributes and the content. For leaf nodes the attributes are the tag attributes and the _content is the textual content of the tag, for other tags it's a bit more complex, the data structure contains stuff returned by the rules of the subtags. The rules may return

nothing (empty list, undef of empty string) - nothing gets added to the parent's data structure
a single string - the string gets appended or pushed to the _content of the parent
a single reference - the parent's _content is converted to an array (if necessary) and the reference is pushed there
an even numbered list - add the keys (odd items) and values (even items) to the parent's data structure, if the key starts with '@' push the value at the end of the array referenced by the key (without the '@'). The value may be a reference.
everything else is an error

Hope the explanation makes sense. So the question is, is there something like this already? Does it make sense? Would you be interested in such a module? What parser should I build this on top of? Should it be a separate module or should I rather try to add this to XML::Parser as yet another style?

Jenda

Support Denmark!
Defend the free world!

Better Inside-Out Objects :) (Ovid)

When I was at YAPC::2006, a talk of [chromatic]'s got me to thinking about a better way of producing inside-out objects. My idea proved troublesome, but here's the interface I wanted:

use Encapsulation;sub new {  bless {}, shift;}sub foo {  my $self = shift;  return $self->{foo} unless @_;  $self->{foo} = shift;}1;

In other words, I wanted inside-out objects to act like a normal blessed hash. You know, if inside-out objects were that simple, more people would use them. I've stopped using them because they're so painful to me. I was talking about this idea to [adrianh] and he suggested a different approach than my foolish attempt to use a tied and blessed hash. Then he sent me the code.

#!/usr/bin/perluse strict;use warnings;{   package Encapsulate;    use Scalar::Util;    BEGIN {        my $Secrets = {};        use overload            '%{}' => sub {                my $id = Scalar::Util::refaddr( shift );                my $package = caller;                return $Secrets->{ $id }->{ $package } ||= {};            },            fallback => 1;        sub DESTROY {            my $id = Scalar::Util::refaddr( shift );            delete $Secrets->{ $id };        };    }}{   package Foo;    use base qw( Encapsulate );    sub new {        my $class = shift;        return bless {}, $class;    }    sub foo {        my $self = shift;        $self->{ secret } = shift if @_;        return $self->{ secret };    }}{   package Bar;    use base qw( Foo );    sub bar {        my $self = shift;        $self->{ secret } = shift if @_;        return $self->{ secret };    }}use Test::More tests => 6;isa_ok my $o = Bar->new, 'Bar';$o->foo( 42 );is $o->foo, 42, "can set and get value of Foo's secret";Bar->new->foo( 24 );is $o->foo, 42, "different objects get different secrets";$o->bar( 99 );is $o->bar, 99, "can set and get value of Bar's secret";is $o->foo, 42, "secrets of different classes do not interfere";ok !defined $o->{secret}, 'cannot reach into objects';

It's not complete, but wow, that's pretty easy! So go ahead, shoot holes in this approach.

Cheers,
Ovid

New address of my CGI Course.

I love XML::Twig (Skeeve)

Good Morning fellow monks!

Yesterday I got a task from my boss to come up with some XML code to be used as input for one of our programs. This program extracts information from XML documents and needs a kind of recipe I had to create.

I could have done this relatively fast "by hand". But this would have been boring, so I decided to do it the Perl way.

I already discovered XML::Twig about 3 years ago and I used it quite often. Since all the data I had to deal with for that task is XML, I used XML::Twig again.

Extracting the information from the document's templates was very easy and I was done real fast. I simply dumped it to STDOUT and wanted to create the input file by hand.

But then I realised: XML::Twig can't just parse! It can also create my XML output! The output part was done even faster! XML::Twig's perldoc pages are always a great help.

I really, really like this module! I only wish there were an "XML::Twig for Java". You know: Sometimes you simply have to use this Language.

s$$([},&%#}/&/]+}%&{})*;#$&&s&&$^X.($'^"%]=\&(|?*{%
+.+=%;.#_}\&"^"-+%*).}%:##%}={~=~:.")&e&&s""`$''`"e

Revisioning systems and the lackof (EvanCarroll)

I recently spent the time to learn SVN (Subversion); moreover it was my first Revision / Versioning management system. I'll start out by saying, while I was learning it I thought the whole way through, "This is a waste of time", with very few "thats neat" mixed in. I missed the boat when CVS was becoming the hip thing to do, and wrote it off the first ten thousands times as buzz or something that was just as easily accomplished with `tar cf`. Having learned it, I was wrong. I can't defend it yet, because I still had 50% of the functionality as before, which is 90% of the importance of the system, but having it; and, not being able to adequately justify the polish of it, doesn't mean I don't appreciate the system for what it is worth. If someone wanted a close analogy to working with a revision system, as compared to working without it, the closest thing I can think of is using CGI, vs utilizing a modern web framework.

While I realize that revisioning systems are often as polarized as text-editors I have to ask a more pertinent question How many monks don't use a version management system at all? And lets say we get a poll going for this!

Evan Carroll
www.EvanCarroll.com

[RFC] Stringifying a thread (jdhedden)

I'm considering a one line addition to the [docs://threads] module to overload the string value of a thread to return its ID:

  '""' => \&tid

Since thread IDs are unique, thread objects could be used as hash keys in a more rational way:

  $timeouts{$thr} = $secs;

Also, it would also simply such things as:

  print("Thread $thr running...\n");

While this is not completely backward compatible, I'm wondering if that really matters. For example, according to CPANTS there are no CPAN modules that have 'threads' as a requirement other than 'thread::shared', 'Thread::Suspend' and 'Thread::Cancel', and I know they won't break.

As to other user code out in the Perl universe, the only code that would break would be cases where the current stringified version of a thread (i.e., 'threads=SCALAR(0x...)') is explicitly tested for. IMHO, I think that such a case would be extremely rare, and as such, would not justify blocking the proposed change.

Comments?

Remember: There's always one more bug.

Why Oh Why oh Whyoh (Catalyst Woahs) (OverlordQ)

Earlier this year, you might have seen my writeup about how I felt some of the best (best is a relative term) Perl frameworks and modules have very . . . lacking documentation.

Having another potential project coming up, I decided to revisit my old nemesis, and see if anything has changed.

The Documentation, has come a long way in helping people like me who have hardly any experience with the MVC layout. It's appears to have been restructured and ordered in a way that flows better.

In addition the Walkthrough, has almost completely changed since I checked on it last and does an amazing job at explaining each step. While a few steps could have added a couple of addendums (like getting a strange error message if you use sqlite version 2 instead of 3), overall I feel that it's thorough and complete.

Last time I said Catalyst wasn't Fun, Rewarding, or Quick. As I havent scratched the surface yet, I can't say it's been Rewarding or Quick, atleast it has been Fun this time.

Want a million dollars? (kwaping)

Have you heard of the Netflix Prize? Well, I think the talent here has a good shot at winning it, so I registered a team named "PerlMonks.org".

If you'd like to join the team, please message me with your full name, email, and country and I'll add you to the list. Also reply to this node, so we can all see who's part of the team. Any prize money won will be evenly distributed (post-tax) among contributing team members.

I'll have more information posted to my home node as it comes available, such as private forums and CVS/SVN information.

---
It's all fine and dandy until someone has to look at the code.

Global variables (frodo72)

A recent [id://575508|thread] discussed about the possibility to use something like global variables in Perl, and how to do it. I don't want to discuss the the pros and cons of global variables in this post (but everyone is encouraged to contribute with their point of view, of course), but only a little toy module that derived from that thread.

Using plain package variables is a way to gain global variables, because they're accessible program-wide:

package A;print "the global k in package B is $B::k\n";

That is, if we fully qualify a variable, including its package, we're always allowed to access it. And yes, this works under strict too. OTOH, this approach is pretty weak with respect to typos.

The fact is that this really sucks. We're all lazy, and we don't want all that typing. One interesting suggestion is to code a sub, that can be easily made really global, so that we can use it wherever we need (see [id://575527|here]):

package main;my $MW = ...; # Create your main window here.sub UNIVERSAL::MW {$MW}

This really scares me a bit. This is really globally pervasive, and I fear (but it's my ignorance, for sure) that it could break when we're using some modules. For example: what happens when we use AUTOLOAD too? I really don't know, but I suspect that half of the audience won't be happy. Moreover, this approach is subject to typos as well, because the compiler will accept a call to a function pretty happily.

That's why I thought about creating something that could be used like this (from the SYNOPSIS of [cpan://vars::global]):

    # In the place/package where we want to create globals    use vars::global create => qw( $foo @bar %baz );    # Add some more global symbols    vars::global->create(qw( $hello @world %now ));    # Somewhere else, where we need to access those globals    use vars::global qw( $foo @bar %baz );    # Don't try to use globals that do not exist    use vars::global qw( $Foo ); # typo, croaks    use vars::global qw( @inexistent ); # we don't create by default                                        # use 'create' as above    # You can also import and create new globals    use vars::global qw( $foo %baz ), create => qw( $hey @joe );    # If you're lazy, you can import all the globals defined so far    use vars::global ':all';

The rationales are the following:

all global variables live in the vars::global
we declare that we want to use them much like we would do with [doc://use] [cpan://vars]. That is, when imported, you access them "lazily", i.e. without the need to use the fully qualified name
you cannot access a global unless it already exists or you create it explicitly

Here comes the implementation:

package vars::global;{   use version; $VERSION = qv('0.0.1');   use warnings;   use strict;   use Carp;   # Module implementation here   # Where we keep existing globals   my %ref_for;   sub import {      my $package = shift;      my $caller  = caller;      my @import;      GRAB_IMPORTS:      while (@_) {         my $param = shift;         last GRAB_IMPORTS if lc($param) eq 'create';         if (lc($param) eq ':all') {            @import = keys %ref_for;            while (@_) {                last GRAB_IMPORTS if shift eq 'create';            }            last GRAB_IMPORTS;         }                     push @import, $param;      }      $package->_create($caller, @_);      $package->_import($caller, @import, @_);      return;   } ## end sub import   sub create {      my $package = shift;      my $caller  = caller;      $package->_create($caller, @_);      $package->_import($caller, @_);      return;   } ## end sub create   sub has {      my $package = shift;      my ($symbol) = @_;      return unless exists $ref_for{$symbol};      return $ref_for{$symbol};   } ## end sub has   sub _create {      my $package = shift;      my $caller  = shift;      my @symbols = @_;      no strict 'refs';      no warnings 'once';      foreach my $symbol (@symbols) {         # Some checks         croak "undefined symbol" unless defined $symbol;         croak "empty symbol"     unless length $symbol;         my $identifier = substr $symbol, 1;         croak "invalid identifier '$identifier'"           unless $identifier =~ /\A \w+ \z/mxs;         my $fqn = $package . '::' . $identifier;         my $sigil = substr $symbol, 0, 1;         $ref_for{$symbol} =             $sigil eq '$' ? \${$fqn}           : $sigil eq '@' ? \@{$fqn}           : $sigil eq '%' ? \%{$fqn}           : croak "invalid sigil: '$sigil'";      } ## end foreach my $symbol (@symbols)      return;   } ## end sub _create   sub _import {      my $package = shift;      my $caller  = shift;      no strict 'refs';      foreach my $symbol (@_) {        my $ref = $package->has($symbol)           or croak "non existent global: '$symbol'";         *{$caller . '::' . substr $symbol, 1} = $ref;      }      return;   } ## end sub _import}1;    # Magic true value required at end of module__END__

The "core" of the import is the following line:

*{$caller . '::' . substr $symbol, 1} = $ref;

which strips the sigil out of $symbol to get the variable identifier, prepends the package name (in $caller) with the colons and defines the symbol in the symbol table. Something basically equal to what happens in vars itself, but I have to credit MJD for having learnt it (e.g. here).

We keep a hash %ref_for to remember which global variables have been created so far. New items are added only when the create() method is called, either explicitly or using the create parameter in the import list. For this reason, the import() method first selects the symbols that have to be simply imported from those that require creation too, so that you can write something like:

use vars::global qw( $these @exists ),        create => qw( $those @are %new );

One can check if a symbol already exists with the has() method, which returns a reference to the global variable (if this exists) or undef.

The two methods _import() and _create() factor out the code, so that we can call them from both import() and create(). They cannot call each other, because otherwise caller would be spoiled by the added call. Instead, we factor the code out, call caller from the "public" method (that without the underscore) and then pass this as a parameter to the "private" one (that with the underscore).

Conclusion: I had fun writing it... but shall I ever use it? I doubt :)

Flavio
perl -ple'$_=reverse' <<Don't fool yourself.

Are you using Vanilla/Strawberry Perl? Tell me about it! (xdg)

I'm putting together a presentation on Vanilla/Strawberry Perl for Perl Seminar NY.

I'd love to include some real-world examples of how people are using it.

As of this morning, there have been 1500 downloads of various Vanilla/Strawberry releases since my YAPC 2006 hackathon release in July. If you're one of those, please post a comment or send me an email with a couple lines about your experience.

Thanks!

-xdg

Getting GnuPG.pm to work with gpg > 1.25 (zentara)

I was testing some old [cpan://GnuPG] scripts today, with my latest version of GnuPrivacyGuard ( version 1.42 ) and found a little easy to fix bug. Since the module appears to be unmaintained anymore, I'll mention it here.

First, it has been previously known, that a "strict" bugfix was to change line 267 of GnuPG.pm , to avoid the 'bareword not allowed while using strict' error

#from my $max_fd = POSIX::sysconf( POSIX::_SC_OPEN_MAX  ) || 256;#to    my $max_fd = POSIX::sysconf( POSIX::_SC_OPEN_MAX()  ) || 256;

Second, when you use versions of gpg > 1.25 (according to a debian bug report) with GnuPG.pm, you will get a decrypt error:

protocol error: expected DECRYPTION_OKAY got PLAINTEXT:

So another fix is to change GnuPG.pm, at line 742

#from#    $self->abort_gnupg( "protocol error: expected DECRYPTION_OKAY got $cmd: \n" ) #      unless $cmd =~ /DECRYPTION_OKAY/; #to    $self->abort_gnupg( "protocol error: expected PLAINTEXT got $cmd: \n" )      unless $cmd =~ /PLAINTEXT/;

I'm not really a human, but I play one on earth.Cogito ergo sum a bum

The perils of DWIM (Aristotle)

The [doc://sort] function has extra magic to recognise bare sub names so you can write sort by_name @list_of_hashes. I cant think of any time I needed that feature however:

use strict;use File::Spec::Functions;use Test::More tests => 6;my @file = do {opendir my $dh, curdir or die "$!";readdir $dh;};sub check(&) { not grep $_ eq updir, shift->( @file ) };ok check { sort no_upwards @_ };ok check { sort no_upwards( @_ ) };ok check { sort no_upwards( @_ ) };ok check { sort ( no_upwards @_ ) };ok check { sort ( no_upwards( @_ ) ) };ok check { sort +( no_upwards @_ ) }=begin output1..6not ok 1#   Failed test in t.pl at line 10.not ok 2#   Failed test in t.pl at line 11.not ok 3#   Failed test in t.pl at line 12.not ok 4#   Failed test in t.pl at line 13.ok 5ok 6# Looks like you failed 4 tests of 6.

I just lost half an hour tracking this down.

Makeshifts last the longest.

Andy Lester Is My hero (jk2addict)

Last week, on their way to the Pittsburgh Perl Workshop, Andy can Pete stopped by the RubberyCity PM meeting to talk about Testing in Perl.

Now, I'm no stranger to tests. Handel has a test count of over 11k, and I like to think that I'm doing pretty well for not knowing what I'm doing. If nothing else, tests save me from myself. :-)

One of the slides Andy had was something like "10 things to test other than code". While I knew you could test other things like servers, data, file systems, websites, etc, I've always concentrated on tests related to the dist they're in. It finally clicked for me today at $work.

Once again, I was asked to check a data vs. server thing. Part X has an image assigned, but is the image actually on the servers?

I've done it before, and I have no doubt I'll do it again. So, why not write a test for it as an investment in my future free time? Even if it's not an official test in an official test suite, in an official language, it still exists, and I can still run them on a whim.

What's your methodology? (Mutant)

I've been thinking a lot about methodologies lately. This is partly just because it's a subject that's always interested me. Figuring out how you design & build your software is just as interesting as actually building and designing it (to me at least). It seems like a vast field, the theoretical aspect to development, and something that is often controversial.

The other thing that's got me thinking about it is that at work I'm being asked to build a system using a methodology that's pretty foreign to me. Although, in my professional life, I've rarely used formal methodologies, I've found that I've naturally used an Agile-like approach of my own (and the companies I've worked for). After reading a lot about Scrum lately, I've found it matches pretty closely how I've always worked, as well as adding a few extra controls (such as consistent length of iteration) which make life even easier.

Basically, I find it very difficult to create systems without the following things:

A single technical owner, and a single business owner (or at least the minimal group on each side).
Extremely important is direct and constant interaction between these two, especially in regards to requirements. To me, assembling requirements is a bargaining process. If business goes off and tries to figure out their requirements, they'll miss some really important issues relating to the reality of what they're asking.
The ability to manage constantly changing requirements
An iterative process, i.e. a feedback loop. The client needs to see the system to understand what their requirements are

The methodology I'm being asked to use is based on Waterfall. With the possible exception of the first point, it has none of the above. The preferred method of communication is via documentation. Requirements are locked down as much as possible, and there is no iterative process built in to the methodology.

While I'm sure I'll find a way to work within these contraints, I can't help wondering if I've missed the point of Waterfall (or similar) approaches. Agile is suffering from a backlash at the moment, with people playing the "No Silver Bullet" card (similar to some of the reaction to OO in the 90's). I don't want to jump on the Agile bandwagon and become one of the people who in their rabid evangelism actually harm the reputation of the very thing they're trying to advocate.

So what is your methodology of choice? What methodologies have you used and what works for you? What do you like / dislike about particular methodologies? (I'm sure this is a good way of starting a Holy War :)

Perldocs Enlightenment (Scott7477)

I was perusing perlfaq4 during my lunch break today and came across the topic "How do I handle linked lists?" The first sentence of the answer reads in part as follows: "In general, you usually don't need a linked list in Perl..." I did a double take and a silent "Hoo-aahh!"

I have rather nasty memories of trying to follow material on linked lists in certain algorithm texts years ago, so realizing that a programming language (i.e. Perl) can be widely accepted without linked lists as a feature was gratifying and enlightening.

Another quote from the faq: "If you really, really wanted, you could use structures as described in perldsc or perltoot and do just what the algorithm book tells you to do...but again, Perl's built-in are virtually always good enough." To me, this is another feature that pushes Perl to the top of my list when considering languages to use for coding projects.

Have any monks implemented linked-list structures in their Perl code, and if so did your performance improvements outweigh the additional complexity of the code?

Another "Find the bug!" Node (Ovid)

Much fun was spent today tracking down a mysterious bug. I even had fun shoving a coderef into @INC to find out what was loading the errant module. I've now reproduced a minimal test case. Once I put it together, the bug was clear, but before that, there were many red herrings. Here's a minimal test case.

package Package1;use base 'Exporter';our @EXPORT_OK = qw/this that/;our %EXPORT_TAGS = ( all => [qw/this that/], );sub this { 'this' }sub that { 'that' }1;

And another package:

package Package2;use base 'Exporter';use Package1 ':all';our @EXPORT_OK = ( @Package1::EXPORT_OK, 'foobar' );our %EXPORT_TAGS = %Package1::EXPORT_TAGS;push @{ $EXPORT_TAGS{all} } => 'foobar';sub foobar { 'foobar' }1;

Do you see the bug? That was very, very annoying, but fortunately the fix is easy. If you see the bug, post it as a spoiler.

Cheers,
Ovid

New address of my CGI Course.

Guess the output. (BrowserUk)

I was looking for something in perlop when I noticed this in the description for the tr/// operator:

 If no string is specified via the =~ or !~ operator, the $_ string is transliterated.

Never having thought of using !~ in conjunction with tr///, I wondered what it did, so I tried a few things:

C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[a][]"C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[b][]"1C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[bb][]"1C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[bc][]"1C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[b][]c"C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[b][]d"1C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[b][]"1C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[a][]c"1C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[ab][]"C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[a][]"C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[b][]"1C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[c][]"1C:\test>perl -wle"$_ = 'aaa'; print $_ !~ tr[bc][]"1C:\test>perl -wle"$_ = 'aab'; print $_ !~ tr[bc][]"

From which I concluded that in this 'inverted count-the-stars' mode, it returns a true of false value indicating whether the string doesn't contain any of the characters in the searchlist. Not exactly intuative, but could be useful sometime. So then I thought I'd try the delete flag, and the result surprised me no end.

Can you guess what this would do before you try it?

perl -wle"$_ = 'aaa'; $_ !~ tr[a][]d; print"

Were you right?

Can you explain it ;)

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

CheckWeb - a new module (pileofrogs)

In this node: 569177 I talked about my idea for a sort of test-harness for checking the functionality of your web server.

I've named it CheckWeb, and I've got my first version readyat http://seattlecentral.edu/~dmartin/pub/CheckWeb-v0.0.1.tar.gz

Before I upload to CPAN, does anyone have any last advice? Can anyone think of a better name than CheckWeb?

Thanks!

-Pileofrogs

RFC: Module::Prefer (radiantmatrix)

I often find myself needing to do things like:

BEGIN {  eval "require CGI::Simple; CGI::Simple->import;"  if ($@) {     eval "require CGI; CGI->import";     die ($@) if $@;     $main::q = CGI->new;  }  else {     $main::q = CGI::Simple->new;  }}

In other words, I want to load a preferred module, but if I can't, fall back on another. having no succinct term for this, I was unable to find a CPAN module to make this easier. So, I wrote something, and I'm curious for feedback from my fellow (usually much smarter...) monks.

NAME
SYNOPSIS
SUBROUTINES
DETAILS

modspec

NAME

This document describes Module::prefer

SYNOPSIS

Allows a package to effectively use preferred modules, while falling backon less-preferred modules when needed.

  use Module::prefer 'prefer';  # use CGI::Simple if possible, CGI otherwise  my $CGI = prefer(    [      'CGI::Simple' => [1],      'CGI'         => [1],    ],  );

See the section on prefer for the full explanation of the parameter spec.

Note that nothing is exported by default, so one must explicitly ask for prefer to be exported if this is desired.

SUBROUTINES

prefer

  $chosen = prefer( \@modspec, @default_args );

The prefer subroutine accepts a reference to a modspec array, anda list of default arguments to pass to the imported module's importmethod (if any). That is, the same list that would be passed after theuse statement.

If none of the modules can be required, prefer will die, dumpingthe errors given by each module, in order.

Once a module is sucessfully required, it will be imported, if possible,and the name of the chosen module will be returned.

DETAILS

modspec

The modspec is a reference to a ``hash-like'' array, in the following form:

  [ 'Module::Name' => [ $use_default_args, @special_args ], ... ]

Modules should occur in order of preference, from most to least preferred.The element immediately following the module name may either be undef oran ARRAYref containing at least one value.

If it is undef, this modules import function will be called withoutany parameters (if this module is selected for import).

The first element in that ARRAYref will be a true/false value that determineswhether or not the default args list will be applied if this module is selectedfor import. The remaining arguments, if any, will always be passed to thismodules import (if this module is chosen).

For example, assume Module::A must manually import foo, but Module::Bimports this automatically; also, both modules need to be told to import bar. Two approaches are possible:

  # 'bar' in default, special args for Module::A  $chosen = prefer (    [      'Module::A' => [1,'foo'], # use Module::A ('bar','foo');      'Module::B' => [1],       # use Module::B ('bar');    ],    'bar',  );

  # 'foo' and 'bar' in default, Module::B does it's own thing  $chosen = prefer (    [      'Module::A' => [1],       # use Module::A ('bar','foo');      'Module::B' => [0,'bar'], # use Module::B ('bar') - ignore defaults!    ],    'foo', 'bar',  );

And the code itself:

package Module::Prefer;use strict;use warnings;use base 'Exporter';our @EXPORT_OK = qw'prefer';sub prefer {  # prefer ([ module_name => [ $use_defaults, @param ], ... ], @default_args);  my $modspec = shift;  my $caller = caller;  my $module;  my @err;    if (@$modspec % 2 != 0) { die "need a hash-like ARRAYref in first argument" }    for (my $i = 0; $i <= @$modspec/2; $i+=2) {    eval 'require '.$modspec->[$i].';';    if ($@) { push @err, [ $modspec->[$i], $@ ]; next; }        $module = $i;    last;  }    if (defined $module && $$modspec[$module]->can('import')) {    # Required one of the modules!    my $call = "package $caller; ".$$modspec[$module]."->import";    my $param = $modspec->[$module+1];        # mod_name => undef same as mod_name => [0]    unless (defined $param && @$param) { $param = [0] }         my $use_defaults = shift @$param;    if ($use_defaults) {      # we should use defaults, if there are any      if (@_ || @$param) {        # only add things if we either have some defaults, or some specials        $call .= '('.(@_ ? "\@_," : '').(@$param ? "\@\$param" : '');        $call =~ s/,$//;        $call .= ')';      }    }    elsif (@$param) {      # only use the specials, if any exist      $call .= "(\@\$param)";    }    $call .= ';';          eval "$call";    die ($@) if $@;  }  else {    # couldn't require any module;    die "Failed to require any of the modules:\n\t"      .join("\n\t",map { sprintf('%20s => %s', @$_) } @err)      ."\n";  }    return $modspec->[$module]; #return name of module used}1;__END__=head1 NAMEThis document describes Module::Prefer=head1 SYNOPSISAllows a package to effectively C preferred modules, while falling backon less-preferred modules when needed.  use Module::Prefer 'prefer';  # use CGI::Simple if possible, CGI otherwise  my $CGI = prefer(    [      'CGI::Simple' => [1],      'CGI'         => [1],    ],  );See the section on C for the full explanation of the parameter spec.Note that nothing is exported by default, so one must explicitly ask for C to be exported if this is desired.=head1 SUBROUTINES=over 8=item C  $chosen = prefer( \@modspec, @default_args );The C subroutine accepts a reference to a L array, anda list of default arguments to pass to the imported module's Cmethod (if any).  That is, the same list that would be passed after theC statement.If none of the modules can be Cd, C will die, dumpingthe errors given by each module, in order.Once a module is sucessfully required, it will be Ced, if possible,and the name of the chosen module will be returned.=back=head1 DETAILS=head2 modspecThe modspec is a reference to a "hash-like" array, in the following form:  [ 'Module::Name' => [ $use_default_args, @special_args ], ... ]Modules should occur in order of preference, from most to least preferred.The element immediately following the module name may either be C oran ARRAYref containing at least one value.If it is C, this modules C function will be called withoutany parameters (if this module is selected for import).The first element in that ARRAYref will be a true/false value that determineswhether or not the default args list will be applied if this module is selectedfor C.  The remaining arguments, if any, will always be passed to thismodules C (if this module is chosen).For example, assume B must manually import C, but Bimports this automatically; also, both modules need to be told to import C. Two approaches are possible:  # 'bar' in default, special args for Module::A  $chosen = prefer (    [      'Module::A' => [1,'foo'], # use Module::A ('bar','foo');      'Module::B' => [1],       # use Module::B ('bar');    ],    'bar',  );  # 'foo' and 'bar' in default, Module::B does it's own thing  $chosen = prefer (    [      'Module::A' => [1],       # use Module::A ('bar','foo');      'Module::B' => [0,'bar'], # use Module::B ('bar') - ignore defaults!    ],    'foo', 'bar',  );=cut

The interface is a little clunky, but I haven't figured out any better (several worse ones, but we won't talk about those ;D). I'd love to hear suggestions on interface and comments on usefulness, including whether I'm doing anything stupid like reinventing a perfectly good wheel.

<–radiant.matrix–>
A collection of thoughts and links from the minds of geeks
The Code that can be seen is not the true Code
I haven't found a problem yet that can't be solved by a well-placed [http://en.wikipedia.org/wiki/Trebuchet|trebuchet]

What should be captured in automated test reports? (xdg)

As I've been developing CPAN::Reporter, I've been thinkingabout what information I wished I had when examining failure reports on CPAN Testers. Test::Reporter alreadyincludes the results of "perl -V". As part of CPAN::Reporter, I include a list of prerequisites specified and the module versionsloaded (and I believe that CPANPLUS does this as well).

Now I'm considering what else might be useful. From an initial small list Ihad, I've brainstormed a longer list of additional things that might be usefulto include. For example:

Toolchain module versions

It seemed like it might be useful to know what versions of the moduletoolchain are in use. These aren't listed in prerequisites and could bea potential source of test problems (particularly false negatives).

CPAN
CPANPLUS -- though not really relevant for CPAN::Reporter
ExtUtils::MakeMaker
Module::Build, along with ExtUtils::CBuilder and ExtUtils::ParseXS

Environment variables

Many test programs change behavior based on environment variables. In addition, it seemed to me that some environment variables might answer otherquestions about the operating environment at the time of the failure.

qr/PERL/ -- anything specific to Perl and kind enough to flag it in the name of the variable
PATH
SHELL
TERM
AUTOMATED_TESTING -- used by many authors to toggle tests for non-interactive situations
AUTHOR_TESTING -- I use this for pod/coverage tests, though I probably should have called it PERL_AUTHOR_TESTING
INCLUDE -- win32 gcc path to header files
LIB -- win32 gcc path to library files
LD_LIBRARY_PATH -- GNU/linux path to library files

Other stuff

This is more of a catch-all, but it seemed like many of these might beuseful to know when investigating a test failure report.

perl executable name -- $^X -- helpful for detecting spaces/non-standard locations
UID/EUID and GID/EGID -- $<, $>, $(, $) -- running as root or as a normal user?
umask -- affects files created during tests
Win32::GetOSVersion() -- only for win32, of course, but would give more detail than just "MSWin32" from $^O (e.g. Win98? Vista?)

Program versions

This is a much more speculative idea, as there is no standard way to ask fora version for an external program. It might or might not really matterexcept in very unusual XS cases.

compiler
$Config{make}

Feedback needed

I'm interested in what people think of these lists. Which of these seemuseful? Which are overkill? Are there things missing that you'd really liketo have in a test report. Which would you prioritize?

-xdg

RFC: Portable Alien Library System (xdg)

This post was orginally made as Portable Alien Library System on win32.perl.org. It is reposted here for widercommentary. Some additional design details, existing commentary and references can be found on that page.

Synopsis

Modules using "libfoo" list Alien::Foo as dependency
Alien::Foo namespace used to manage external libraries
Alien library files contained in the corresponding auto directory (auto/Alien/Foo)
Alien modules provide an API for CC flags in Makefile.PL/Build.PL
Alien modules verify library availability (e.g. in platform or Alien standard directories)
Alien modules responsible for installing binary library files when necessary
Alien modules support binary package download from a repository
Alien modules support source compilation or binary re-packaging to the binary package format used for installation

Introduction

The Vanilla Perl Project provides a Win32 Perl and bundled compiler to build Perl modules from CPAN rather than relying on binary module package systems like PPM. Early feedback on the project demonstrate the many popular Perl modules that have dependencies on external libraries -- ones often assumed to exist on a Unix-based operating system.

This document describes the Portable Alien Library System (PALS) -- a framework for external library dependency specification and resolution. While its inspiration is drawn from Win32, its design is intended to provide benefits to module authors and users on all platforms.

Requirements

This list of functional objectives is adapted from the original Alien module and commentary in External Library Handling at win32.perl.org.

Dependency specification

The solution must provide a way for Perl module authors to specify a dependency on an external library. An important consideration is how a solution integrates with or diverges from existing CPAN methods and tools.

Library detection

Libraries may be pre-installed in well known locations (e.g. "/lib") or may be installed ad hoc. The solution will need to detect if a library (and associated headers) are available and identify the associated paths.

Library configuration

Given the wide variety of locations for libraries, the solution needs to provide an abstraction layer to other tools that ensures proper configuration. Examples include:

Makefile.PL: compiler flags, including library and header paths
Shared/dynamic library loading: libraries in non-standard locations may need changes to environment settings such as PATH or LD_LIBRARY_PATH to ensure libraries are found at runtime
Other metadata: library metadata, such as version, may be required to manage dependencies

Binary library package generation

External library compilation from source can be substantially more time-consuming than typical XS compilation for Perl modules. Providing binaries of external libraries (whenever possible) from an online repository will provide a more user-friendly experience.

Binary library package installation

Whether compiled from source or downloaded as a binary, external modules will need to be installed such that they may be located later for library detection and configuration. The solution will need to address whether this should be platform independent or platform specific.

Design criteria

Portable

External library support is a potential problem all platforms -- not just Win32 -- and developers will be more inclined to use the solution to specify external dependencies if it addresses dependency problems portably. The solution should provide a standard interface for module developers and module users that abstracts platform and compiler details.

Modular

Library packagers may not wish to take on responsibility for creating a solution for all platforms or all compilers or may lack the necessary know-how. The solution should modularize functionality wherever possible to allow individual library packagers to work on separate parts of the solution for platforms of interest.

User-friendly

The solution should degrade gently. If library support is not available for a particular platform or compiler, the solution should fail cleanly with an explanation of the dependency problem prior to generating XS compilation failures during linking.

Maintainable

Ongoing availability or upgrades of an external library should not depend on a single individual or on tacit, undocumented packaging know-how. As with Perl::Dist::*, the published solution should contain an automated process to convert an external library from source (or binary package) to the binary package format used by the solution.

Design overview and commentary

The following design description will make more sense if you examinethe Alien Ecosystemdiagram.

Dependency specification

CPAN modules with external library dependencies will specify a module dependency on a corresponding Alien:: module, e.g. Alien::Foo for the "libfoo" module. Dependency management can thus be handled within the existing system using ExtUtils::MakeMaker, Module::Build or Module::Install prerequisite specifications.

Presence of an installed Alien::Foo should be considered evidence that libfoo is available. Alien::Foo must not install if libfoo cannot be found.

Library detection

Library searches will need to be done in platform specific ways. Perl Config values like "libpth" could be used as a starting point.

Win32: should also include the LIB environment variable and possibly the PATH environment variable for library searches
Unix: should include the LD_LIBRARY_PATH environment variable (or equivalents); could also use the output of "ldconfig -p"

Library configuration

Makefile.PL: Alien::Base will provide an API for generating CC flags for both libraries and headers.
Shared/dynamic library loading: Will need to set the shared library search path before bootstrapping XS. This could be done during import of "use Alien::Foo" or could be done via an explicit class method call.
Other metadata: Alien::Foo could be extended with methods for library metadata.

Binary library package generation

Alien::Foo::* should include the full procedure needed to generate a binary library package (a ".pal" file) for a particular platform. This should include downloading sources from a URL, any preparation work or patching, and generation of the binary.

This process could be done completely from the upstream library source, or could be a "repackaging" of another binary download, e.g. creating wrappers or definition files around a .dll on Win32.

At worst, this might just be a series of system() calls. The point is to ensure that the full process is available for future packagers to replicate, adapt or enhance.

Alien::Base should provide helper methods for URL downloading, PAL file generation, etc.

After the PAL file is generated, it can be uploaded to a repository for end-users to download for installation.

Binary library package installation

If not available already in platform-standard library directories, the library headers and binaries should be installed into the corresponding "auto" directory for the Alien module. E.g., "site/auto/Alien/Foo" for Alien::Foo. This location should be well-defined for all platforms, since it's the same as is used for XS modules.

Exact layout within this module is still open -- should there be "lib" and "include" subdirectories or should libraries and headers be combined in the top directory. Installation location of binary utilities included with libraries is still an open question.

The binary package may be either generated from source or downloaded from a URL (the default option).

-xdg

Interpolating method calls (TGI)

I was just reading [id://564791], which mentions that the key in a hash is evaluated as code when the hash is interpoloated. I had the sudden inspiration that this might be a way to clean up (slightly) the ugliness of method interpolation.

For example, if you have an object $foo, and you want to print one of its attributes you can choose one of the following approaches:

my $attr = $foo->attr; print "Attribute is $attr\n";print "Attribute is ". $foo->attr ."\n";print "Attribute is @{[$foo->attr]}\n";

In my opinion all of these options leave somthing to be desired.

My inspiration was that, since a hash key is evaluated, why not just use a tied hash that does nothing but returns the key. So $foo{'a value' will return 'a value'. I'm not sure about the wisdom of this move, but it seems to make the syntax a little cleaner.

Any thoughts on whether this sort of thing is a good idea? Or, perhaps, retched and foolish?

use strict;use warnings;# Create a dummy classuse Class::Struct Foo => [ attr => '$' ]; # with a dummy methodsub Foo::Repeater {    my $self = shift;    my $times = shift || 1;    return ($self->attr) x $times;  # Make sure x operates in list context}# Create a dummy objectmy $foo = Foo->new(attr => 'foo');# Create my interpolater.my %im;tie %im, 'InterpolateMethods';# Example: Concatenation.print "The Foo object's attr attribute is: " . $foo->attr . "\n";# Ugly inlineprint "The Foo object's attr attribute is: @{[$foo->attr]}\n";# Use the interpolaterprint "The Foo object's attr attribute is: @im{ $foo->attr }\n";# We can use the interpolater in list context as well.print "Calling Foo's Repeater(3) method yeilds: @im{ $foo->Repeater(3) }\n";package InterpolateMethods;use Tie::Hash;sub TIEHASH {    my $self = shift;    my $mi = \$self;    return bless $mi, $self;}sub FETCH {    my $self = shift;    my $key = shift;    return $key;}1;__END__The Foo object's attr attribute is: fooThe Foo object's attr attribute is: fooThe Foo object's attr attribute is: fooCalling Foo's Repeater(3) method yeilds: foo foo foo

TGI says moo

Categorizing a Perl Development Platform (jdrago_999)

Question: How do you categorize a software product that's a portal server/file server/MVC implementation/asynchronous eventing & messaging broker/etc. Together, they are codename "DevStack" (a rename may be pending).

After spending the last several years working on projects that ended up needing these kinds of tools, I finally decided to sit down and write them. The idea is that it would provide a good starting point to build applications, since about 98% of it would already be done.

"Portal Server" doesn't really do it. There's a lot more going on than just a portal and an MVC implementation. There's more than just file-server stuff. Lots of things together. What do you call something like that??

Thanks!