controlling child processes (ministry)

controlling child processes

ministry

created: 2006-04-04 18:07:55

perlmonks

node
author

prlmnks

node
author

Fellow monks,

I'm developing a script that will retrieve log files via a raptor (firewall) specific application called 'remotelogfile'.

Goal:
Fork a predetermined amount of child processes, each using the remotelogfile utility for simultaneous collection. The next child process must wait to be spawned until the number of child processes goes below the predetermined amount of children.

Caveat:
I can't do a collection of all firewalls listed in my array at the same time, i.e. - the wait.

Questions:
One - how can I make the script wait until all child processes have finished, before exiting?
Two - should I be decrementing the $counter every time a loop completes
Three - is there a better way to go about doing this?

Here's a perlmonks friendly version of what I have so far...

my($num_procs)='3';
my($counter)='0';
foreach my $fw qw(fw1 fw2 fw3 fw4 fw5){
  $counter++;
  if($counter <= $num_procs){
    &do_stuff($fw);
  }
  else{
    wait;
    &do_stuff($fw);
  }
}

sub do_stuff{
  my($fw)=shift;
  local $SIG{CHLD}='IGNORE';
  defined(my $pid=fork) || die "Can not fork!";
    exec "remotelogfile $fw logfile > /var/$fw.log" if(!$pid);
}

Any help would be most appreciated.

cheers!
-Ev

Good judgement comes with experience. Unfortunately, the experience usually comes from bad judgement.

Re: controlling child processes

ikegami

created: 2006-04-04 18:10:35

perlmonks

prlmnks

It sounds like you are duplicating the functonality of Parallel::ForkManager.

Re: controlling child processes

johngg

created: 2006-04-04 19:02:21

perlmonks

prlmnks

I have a script that does tape backups. It forks a child process for each tape device on the host so that multiple tape devices are written concurrently rather than consecutively. This is the skeleton of the process, which uses pipes so that the children can log messages that the parent can read later.

use IO::Pipe;
...
foreach $tapeDevice (@tapeDevices)
{
    $pipeFH = IO::Pipe->new();
    ...
    if($pid = fork())
    {
        # This is the parent.
        $pipeFH->reader();
        push @pipes, $pipeFH;
        ...
    }
    elsif(defined $pid)
    {
        # This is the child.
        $pipeFH->writer();
        # Do lots of stuff writing to tape
        ...
        $pipeFH->print("some message or other");
        ...
    }
    else
    {
        # Fork failed for some reason.
        die "fork: $!\n";
    }
}
# We have now forked a child for each tape device so
# wait for our children to finish. Print exit status.
while((my $returnPID = wait()) != -1)
{
    print "Child $returnPID returned status ",
       $? >> 8, "\n";
}
# All children have returned, get messages.
foreach $pipe (@pipes)
{
    while(defined($_ = $pipe->getline()))
    {
        # Do something
        ...
    }
}

That is the bare bones of the mechanism. I have not attempted to reconcile use strict; and my or our here as it is just an illustration.

I hope this is useful.

Cheers,

JohnGG

Re: controlling child processes

bluto

created: 2006-04-04 19:03:25

perlmonks

prlmnks

It is easy to get these problems wrong, so I second using Parallel::ForkManager. In your solution you need to keep looping not only if there is more work to spawn, but also if there are children to reap. I like to use a hash to maintain the children. Here's an untested solution, probably with bugs ...

use strict;
use warnings;

$SIG{CHLD} = 'IGNORE';
my $max_procs = 3;  # don't use strings when you want numbers
my @work = qw(fw1 fw2 fw3 fw4 fw5);
my %kids;
while (@work or keys %kids) {
   if (@work and $max_procs > keys %kids) {
      my $fw = shift @work;
      my $pid = fork;
      die "Can not fork!" if ! defined $pid;
      if ($pid) {
         $kids{$pid} = $fw;
      } else {
         exec "remotelogfile $fw logfile > /var/$fw.log";
         die;
      }
   } else {
     my $pid = waitpid(-1, 0);
     delete $kids{$pid};
   }
}