Arbitrarily Nested Loops
Limbic~Region
created: 2006-02-03 10:51:43
All,
While this meditation is about arbitrarily nested loops, it has come as a result of a specific problem I was trying to solve. Please try to focus on the meditation and not the specific problem.

Introduction

Many of you know my affinity for challenging problems and puzzles. For me, they are fun ways to expand my knowledge. There is a currently unsolved math problem of an order 10 [http://mathworld.wolfram.com/PerfectMagicCube.html|Perfect Magic Cube]. To prove that an order 10 magic cube is not possible by brute force, 1000! permutations of arranging the numbers 1 - 1000 must be tried. I was hoping to find and prove the existence of a cube so I knew that I was going to have to find ways to significantly reduce my search space and hope I got lucky. I succeeded in finding ways to reduce the search space but failed in getting lucky. This meditation is about the methodology I used and not about the problem I was trying to solve. If anyone is interested in a list of techniques I used, /msg me and I will be glad to go over them. What was important to me was the knowledge not the answer.

Explanation of arbitrarily nested loops

If you were trying to figure out how many different outfits you could make with your current wardrobe, you may create the following code:
for my $feet ( qw/black brown sneaker/ ) {
    for my $legs ( qw/shorts dress_pants jeans/ ) {
        for my $torso ( qw/t-shirt dress_shirt sweater/ ) {
            print "$torso $legs $feet\n";
        }
    }
}
Now imagine that you wanted to do the same thing but you were doing it for your upcoming birthday gifts. Since you won't know until the gifts arrive (runtime) what you are going to receive, you can't pre-construct the loops. You know you may need to extend the number of loops to include things like gloves to cover your hands and hats to cover your head and sunglasses for your eyes.

Abritrary nested loops is not knowing in advance the number of loops or the number of items in each loop. It may seem like a difficult problem to solve, but in fact it is quite easy.

How to handle arbitrarily nested loops

[Dominus] uses the concept of an odometer with dials of different ranges in [http://perl.plover.com/hop/|HOP] to give a visual representation of the process. When I first [462987|figured out] how to it, I thought of a combination lock on a briefcase with different size dials. The idea is the same.

Assume that you have received a list of lists that need to be looped through. Each list represents a dial that must be turned. When you the dial you are turning reaches its maximum value, the dial to its left increments by one and the current one rolls over to the start. When the leftmost dial reaches its maximum number - you know you are finished.

(black, brown, sneakers), (shorts, dress_pants, jeans), (t-shirt, dress_shirt, sweater)

# Start position
0, 0, 0 = black, shorts, t-shirt
0, 0, 1 = black, shorts, dress_shirt
0, 0, 2 = black, shorts, sweater
0, 1, 0 = black, shorts, t-shirt
0, 1, 1 = black, shorts, dress_shirt
0, 1, 2 = black, shorts, sweater
0, 2, 0 = black, dress_pants, t-shirt
...
2, 2, 2 = sneakers, jeans, sweater
To make this an iterator you need to use closures which I won't cover. Instead of rolling your own you should probably use [tye]'s [cpan://Algorithm::Loops] 'NestedLoops'. There is still value in removing the veil of mystery to understand how it works even if you are not going to implement it yourself.

How to dynamically construct the loops

What if the problem you are trying to solve requires the construction of a loops values to be based off 1 or more of the preceeding values selected? For instance, when you are selecting wardrobes - perhaps you don't want to mix business wear with beach wear. This can be handled with a code ref. In [cpan://Algorithm::Loops], $_ is the currently selected value immediately to the left of the current dial and @_ is all of the selected values with 0 being the outer most dial.

Now it is a simple matter of checking when a dial rolls over to the beginning if the dial was a fixed list or generated from a code ref. If it was from a code ref, replace the current list with the return value of the the executed code ref.

What else might I want?

This meditation is not intended to be an explanation of [cpan://Algorithm::Loops]. In fact, I rarely grok [tye]'s code and this is no exception. If my explanation fits the [cpan://Algorithm::Loops] code it is not a result of me studying the code. He however does have some great ideas which I shamelessly steal.

You might want the ability to filter out some of the results. We may allow the user to pass in a code ref that we will execute at the end of each dial turn. If the code ref returns a true value we return the list and if not, we spin the dial again - wash, rinse, repeat.

You may also want to execute some arbitrary piece of code after each loop. Again, this is as simple as designing your API such that the user can pass in a code ref that you execute at the end of each dial turn.

Conclusion

Handling arbitrarily nested loops that are dynamically constructed in Perl is fairly simple. This does not mean you should go rolling your own without due cause as [cpan://Algorithm::Loops] is a wonderful module. I needed to for my task because Perl was just too slow. I ended up using C. While I was able to produce some extremely fast code (25-30 million iterations per second), I was unable to make a generic reusable library. Perl rawks!

I didn't want this to be a tutorial showing step by step code for 2 reasons. The first is because there is an existing ready made solution. The second is because one of the best ways to learn is by doing. Try it out, make sure you can do it on your own. You will likely learn something. If you have problems or questions - feel free to reply.

If anyone has the inclination to make this work in C as a reusable library I will be very much interested.

Cheers - [Limbic~Region|L~R]

Re: Arbitrarily Nested Loops
created: 2006-02-03 11:44:56

(Moving here what I had on my pad for you.)


I wrote this to get the problem clearly in my head. It doesn't really solve anything, since your problem is "...work with element_ptr...". There's also the issue of whether individual elements should be freed. Perl solved both of these problems by using SVs. Your requirement that your lists can contains more than one kind of data complicates things greatly. It would be much simpler if you could deal with a single type at a time (even if the type was different in different calls to NestedLoops). See "Take 2", below, where I came up with something along the lines of a simple SV.

struct NestedLoopList {
   int   elem_size;
   int   length;
   void* ptr;
   int   free_ptr;  /* 0 = no, 1 = yes */
};

struct NestedLoopArrayArg {
   int   elem_size;
   int   length;
   void* ptr;
};

struct NestedLoopFuncArg 
   void  (*ptr)(struct NestedLoopList* list_ptr, void* pad);
   void* pad;
};

struct NestedLoopArg {
   int ptr_type;  /* 0 = array, 1 = func */
   union {
      struct NestedLoopArrayArg array;
      struct NestedLoopFuncArg  func;
   } ptr;
};

Calling the function

struct NestedLoopArg* args = (NestedLoopArg*)calloc(2, sizeof(NestedLoopArg));

args[0].ptr_type            = 0;  /* array */
args[0].ptr.array.elem_size = sizeof(int);
args[0].ptr.array.length    = 5;
args[0].ptr.array.ptr       = calloc(arg[0].array.length, arg[0].array.elem_size);

args[1].ptr_type            = 1;  /* func */
args[1].ptr.func.ptr        = my_func;
struct MyFuncPad arg1_pad;
arg1_pad.count = 5;
args[1].ptr.func.pad        = &arg1_pad;

NestedLoops(args);

... free memory allocated for args ...

Getting the list

struct NestedLoopList list;
if ((*arg_ptr).ptr_type == 0)
{
   /* array */
   struct NestedLoopArrayArg* array_ptr = (*arg_ptr).ptr;
   list.elem_size = (*array_ptr).elem_size;
   list.length    = (*array_ptr).length;
   list.ptr       = (*array_ptr).ptr;
   list.free_ptr  = 0;
}
else 
{
   /* function */
   struct NestedLoopFuncArg* func_ptr = (*arg_ptr).ptr;
   (*(*func_ptr).ptr)(&list, (*func_ptr).pad);
}

Example function pointer argument

struct MyFuncPad {
   int count;
};

/* Creates a list of ints, numbered 1 to the int located at *pad. */
void my_func(struct NestedLoopList* list_ptr, void* pad) {
   int  count = (*(MyFuncPad*)pad).count;
   int* array = (int*)calloc(count, sizeof(int));
   int i;
   for (i=0; i

Going over the list

/* Convert the pointer to an int since ++ and other  */
/* arithmentic operators are overloaded for pointers */
int p = (int)list.ptr; 

for (i=0; i



Take 2.

Much better. ElementType probably should not contain handler, and handler needs more arguments, but you get the idea.

struct ElementType {
   int  size;
   void (*handler   )(void* ptr);
   void (*destructor)(void* ptr);
};

struct Element {
   struct ElementType* type_ptr
   void*               ptr;
};

struct NestedLoopList {
   int      length;
   Element* ptr;
   int      free_ptr;    /* 1 when ptr needs to be freed. */
   int      free_elems;  /* 1 when each element of ptr[] needs to be freed and destroyed. */
};

struct NestedLoopArrayArg {
   struct ElementType* type_ptr;
   int                 length;
   void*               ptr;
};

struct NestedLoopMixedArg {
   int             length;
   struct Element* ptr;
};

struct NestedLoopFuncArg {
  void  (*ptr)(struct NestedLoopList* list_ptr, void* pad);
  void* pad;
};

struct NestedLoopArg {
   int ptr_type;  /* 0 = array, 1 = mixed array, 2 = func */
   union {
      struct NestedLoopArrayArg array;
      struct NestedLoopMixedArg mixed;
      struct NestedLoopFuncArg  func;
   } ptr;
};

Calling the function

struct NestedLoopArg* arg = (NestedLoopArg*)calloc(2, sizeof(NestedLoopArg));

args[0].ptr_type           = 0;  /* array */
args[0].ptr.array.type_ptr = &IntType;
args[0].ptr.array.length   = 5;
args[0].ptr.array.ptr      = calloc(arg[0].array.length, arg[0].array.elem_size);

args[1].ptr_type           = 1;  /* mixed array */
args[1].ptr.mixed.length   = 4;
args[1].ptr.mixed.ptr      = (Element*)calloc(arg[0].array.length, sizeof(struct Element));
{
   ... populate args[1].mixed.ptr[] ...
}

args[2].ptr_type           = 2;  /* func */
args[2].ptr.func.ptr       = my_func;
struct MyFuncPad arg2_pad;
arg2_pad.count = 5;
args[2].ptr.func.pad       = &arg2_pad;

NestedLoops(args);

... free memory allocated for args ...

Getting a list

struct NestedLoopList list;

switch ((*arg_ptr).ptr_type) {
   case 0: /* array */
   {
      struct NestedLoopArrayArg* array_ptr = (*arg_ptr).ptr;

      int                 length;
      struct ElementType* type_ptr
      int                 elem_size;
      void*               data_arr;
      struct Element*     elem_arr;
      int                 src;
      Element*            dst;
      int                 i;

      length    = (*array_ptr).length;
      type_ptr  = (*array_ptr).type_ptr;
      elem_size = type_ptr.size;
      data_arr  = (*array_ptr).ptr;

      elem_arr = (struct Element*)calloc(length, sizeof(struct Element));

      /* Convert the pointer to an int since ++ and other  */
      /* arithmentic operators are overloaded for pointers */
      src = (int)data_arr;
      dst = elem_arr;

      for (i=length; i--; ) {
         (*dst).type_ptr = type_ptr;
         (*dst).ptr      = (void*)src;

         src += elem_size;
         dst++;
      }

      list.length     = length;
      list.ptr        = elem_arr;
      list.free_ptr   = 1;
      list.free_elems = 0;
      break;
   }

   case 1: /* mixed array */
   {
      struct NestedLoopMixedArg* mixed_ptr = (*arg_ptr).ptr;
      list.length     = (*mixed_ptr).length
      list.ptr        = (*mixed_ptr).ptr;
      list.free_ptr   = 0;
      list.free_elems = 0;
      break;
   }

   case 2: /* func */
   {
      struct NestedLoopFuncArg* func = (*arg_ptr).ptr;
      (*(*func_ptr).ptr)(&list, (*func_ptr).pad);
      break;
   }
}

Example function pointer argument

struct MyFuncPad {
   int count;
};

/* Creates a list of ints, numbered 1 to the int located at *pad. */
void my_func(struct NestedLoopList* list_ptr, void* pad) {
   int             count;
   struct Element* elem_arr;
   Element*        dst;
   int             i;

   count = (*(MyFuncPad*)pad).count;

   elem_arr = (struct Element*)calloc(count, sizeof(struct Element));

   dst = elem_arr;

   for (i=0; i

Going over the list (and freeing it as we go along)

Element* elem_ptr = list.ptr;

for (i=0; i

Freeing memory

if (list.free_elems) {
   int i;
   Element* elem_ptr = list.ptr;

   for (i=list.length; i--; ) {
      (*(*(*elem_ptr).type_ptr).destructor)(elem_ptr.ptr);
      free(elem_ptr.ptr);
      elem_ptr++;
   }
}

if (list.free_ptr) {
   free(list.ptr);
}
Re: Arbitrarily Nested Loops
created: 2006-02-03 12:17:30
When the leftmost dial reaches its maximum number
Maybe you meant to say when all the dials reach their respective maxima? Consider two dials, each ranging from 0 to 2. By your original statement, you'd get the following:
0,0
0,1
0,2
2,0
In this, you've missed two combinations ( (2,1) and (2,2) ). I'm sure it was just a type or a mind-o, though. :)

thor

The only easy day was yesterday

Re^2: Arbitrarily Nested Loops
created: 2006-02-03 12:35:24
thor,
No, but there is a typo. The word reaches should have read exceeds. I will update the original.

Cheers - L~R

Re: Arbitrarily Nested Loops
created: 2006-02-03 15:53:39

Here's a complete (but untested) solution:

#ifndef _NESTED_LOOPS_H
#define _NESTED_LOOPS_H


/* -------------------- */
/* nested_loops.h       */


struct NestedLoop_List {
   int    length;
   void** ptr;                                    /* arr of ptrs  */
   void   (*free)(struct NestedLoop_List* list);  /* frees ptr    */
   void*  free_pad;                               /* closure-like */
};


#define NestedLoop_PT_ARRAY 0
#define NestedLoop_PT_FUNC  1

struct NestedLoop_ArrayArg {
   int    length;
   void** ptr;
};

struct NestedLoop_FuncArg 
   void  (*ptr)(struct NestedLoop_List* list, void* pad);
   void* pad;  /* closure-like */
};

struct NestedLoop_Arg {
   int ptr_type;  /* NestedLoop_PT_* */
   union {
      struct NestedLoop_ArrayArg array;
      struct NestedLoop_FuncArg  func;
   } ptr;
};


struct NestedLoop_Iter {
   int                     num_loops;
   struct NestedLoopArg*   loops;
   int*                    indexes;
   struct NestedLoop_List* vals;
   struct NestedLoop_List  list;
};


void NestedLoop_no_free(void*);

void NestedLoops_array_to_list(
   struct NestedLoop_List* list,
   void*                   array,
   int                     array_size,
   int                     elem_size
);

void NestedLoops_array_to_ptr_array(
   void** ptr_array_ptr,
   void*  array,
   int    array_size,
   int    elem_size
);

void NestedLoops_free_list(struct NestedLoop_List* list);


void NestedLoops_get_iter(
   struct NestedLoop_Iter* iter,
   int                     num_loops,
   struct NestedLoop_Arg*  loops
);

BOOL NestedLoops_fetch(
   struct NestedLoop_Iter* iter
);

void NestedLoops_done(
   struct NestedLoop_Iter* iter
);


#endif
/* -------------------- */
/* nested_loops.c       */


#include "nested_loops.h"


void NestedLoop_list_free_none(void*) {
   /* Do nothing */
}


void NestedLoops_array_to_list(
   struct NestedLoop_List* list,
   void*                   array,
   int                     array_size,
   int                     elem_size
) {
   (*list).length = array_size;
   (*list).free   = &free;

   NestedLoops_to_ptr_array(
      &((*list).ptr),
      array,
      array_size,
      elem_size
   );
}


void NestedLoops_to_ptr_array(
   void** ptr_array_ptr,
   void*  array,
   int    array_size,
   int    elem_size
) {
   int   src;
   void* dst;
   int   i;

   ptr_array_ptr = (void**)calloc(ptr_array_size, sizeof(void*));

   src = (int)array;
   dst = *ptr_array_ptr;

   for (i=array_size; i--; ) {
      *dst = (void*)src;

      dst++;
      src += elem_size;
   }
}


void NestedLoops_free_list(struct NestedLoop_List* list) {
   if ((*list).ptr != NULL) {
      (*list).free(list);

      (*list).length   = 0;
      (*list).ptr      = NULL;
      (*list).free     = &NestedLoop_no_free;
      (*list).free_pad = NULL;
   }
}


void NestedLoops_get_iter(
   struct NestedLoop_Iter* iter,
   int                     num_loops,
   struct NestedLoop_Arg*  loops,
) {
   (*iter).num_loops   = num_loops;
   (*iter).loops       = loops;

   (*iter).indexes     = (int*)calloc(num_loops, sizeof(int));
   (*iter).vals        = (struct NestedLoop_List*)calloc(num_loops, sizeof(struct NestedLoop_List))

   (*iter).list.length = 0;
   (*iter).list.ptr    = (void**)calloc(num_loops, sizeof(void*));
   (*iter).list.free   = &free;

   {
      int i                             = (*iter).num_loops;
      struct NestedLoop_Arg*  loop      = (*iter).loops;
      struct NestedLoop_List* loop_vals = (*iter).vals;

      while (i--) {
         if ((*loop).ptr_type == NestedLoop_PT_ARRAY)
         {
            (*loop_vals).length   = (*loop).ptr.array.length;
            (*loop_vals).ptr      = (*loop).ptr.array.ptr;
            (*loop_vals).free     = &NestedLoop_no_free;
            (*loop_vals).free_pad = NULL;
         }
         else /* if ((*loop).ptr_type == NestedLoop_PT_FUNC) */
         {
            (*loop_vals).length   = 0;
            (*loop_vals).ptr      = NULL;
            (*loop_vals).free     = &NestedLoop_no_free;
            (*loop_vals).free_pad = NULL;
         }

         loop++;
         loop_vals++;
      }
   }
}


BOOL NestedLoops_fetch(
   struct NestedLoop_Iter* iter
) {
   int  last_loop_idx = (*iter).num_loops   - 1;
   int  last_list_idx = (*iter).list.length - 1; /* Save on return! */

   int* indexes = (*iter).indexes;

   if (last_list_idx < last_loop_idx) {
      struct NestedLoopArg* loops = (*iter).loops;

      indexes[++last_list_idx] = -1;

      if (loops[last_list_idx].ptr_type == NestedLoop_PT_FUNC) {
         struct NestedLoop_Arg*  loop      = &{loops[last_list_idx]);
         struct NestedLoop_List* loop_vals = &(vals [last_list_idx]);

         NestedLoops_free_list(loop_vals);

         (*(*arg).ptr.func.ptr)(loop_vals, (*arg).ptr.func.pad);
      }
   }

   /* If the last loop is done,                 */
   /* increment the index of the previous loop. */
   /* Exit if every loop is at its last index.  */
   for (;;) {
      struct NestedLoop_List* loop_vals = &(vals[last_list_idx]);

      ++indexes[last_list_idx];

      if (indexes[last_list_idx] >= (*loop_vals).length) {
         break;
      }

      --last_list_idx;

      if (last_list_idx < 0) {
         (*iter).list.length = last_list_idx + 1;
         return FALSE;
      }
   }

   /* Populate the list. */
   (*iter).list.ptr[last_list_idx] =
      vals[last_list_idx].ptr[indexes[last_list_idx]];

   (*iter).list.length = last_list_idx + 1;
   return TRUE;
}


void NestedLoops_done(
   struct NestedLoop_Iter* iter
) {
   NestedLoops_free_list((*iter).list);

   {
      int i = (*iter).num_loops;
      struct NestedLoop_List* loop_vals = (*iter).vals;

      while (i--) {
         NestedLoops_free_list(loop_vals++);
      }
   }

   free((*iter).indexes);
   (*iter).indexes = NULL;
}
/* -------------------- */
/* example.c            */


/* Simple example:                   */
/* - No function pointers.           */
/* - Data type is already a pointer. */


#include 
#include "nested_loops.h"

void main() {
   /* This could be built dynamically.            */
   /* The args must remain accessible until done. */
   /* The caller handles cleanup of args.         */

   struct NestedLoop_Arg args[3];

   char* fruits[3];
   char* veggies[3];
   char* meats[2];

   fruits[0] = "apple";
   fruits[1] = "banana";
   fruits[2] = "coconut";

   veggies[0] = "asparagus";
   veggies[1] = "brocoli";
   veggies[2] = "cauliflower";

   meats[0] = "beef";
   meats[1] = "chicken";

   args[0].ptr_type = NestedLoop_PT_ARRAY;
   args[0].ptr.array.length = sizeof(fruits)/sizeof(fruits[0]);
   args[0].ptr.array.ptr    = fruits;

   args[1].ptr_type = NestedLoop_PT_ARRAY;
   args[1].ptr.array.length = sizeof(veggies)/sizeof(veggies[0]);
   args[1].ptr.array.ptr    = veggies;

   args[2].ptr_type = NestedLoop_PT_ARRAY;
   args[2].ptr.array.length = sizeof(meats)/sizeof(meats[0]);
   args[2].ptr.array.ptr    = meats;

   {
      struct NestedLoop_Iter iter;

      NestedLoops_get_iter(&iter, sizeof(args)/sizeof(args[0]), args);

      while (NestedLoops_fetch(&iter)) {
         int i;

         /* Skip results without an element from each loop. */
         if (iter.list.length != iter.num_loops) {
            continue;
         }

         for (i=0; i



/* -------------------- */
/* example2.c           */


/* Example using a non-pointer type. */


#include 
#include "nested_loops.h"

void main() {
   /* This could be built dynamically.            */
   /* The args must remain accessible until done. */
   /* The caller handles cleanup of args.         */

   struct NestedLoop_Arg args[3];

   char* chars = "abc!@#123";

   args[0].ptr_type = NestedLoop_PT_ARRAY;
   args[0].ptr.array.length = 3;
   NestedLoops_to_ptr_array(&(args[0].ptr.array.ptr), chars+0*3, 3, sizeof(char));

   args[1].ptr_type = NestedLoop_PT_ARRAY;
   args[1].ptr.array.length = 3;
   NestedLoops_to_ptr_array(&(args[1].ptr.array.ptr), chars+1*3, 3, sizeof(char));

   args[2].ptr_type = NestedLoop_PT_ARRAY;
   args[2].ptr.array.length = 3;
   NestedLoops_to_ptr_array(&(args[2].ptr.array.ptr), chars+2*3, 3, sizeof(char));

   {
      struct NestedLoop_Iter iter;

      NestedLoops_get_iter(&iter, sizeof(args)/sizeof(args[0]), args);

      while (NestedLoops_fetch(&iter)) {
         int i;

         /* Skip results without an element from each loop. */
         if (iter.list.length != iter.num_loops) {
            continue;
         }

         for (i=0; i



/* -------------------- */
/* example3.c           */


/* Example using function pointers. */


#include 
#include "nested_loops.h"


struct RangePad {
   int  min;
   int  max;
   int* data;
};


struct RangeIncPad {
   int  max;
   int* data;
};


void range_free(struct NestedLoop_List* list) {
   RangePad* pad;

   pad = (RangePad*)((*list).free_pad);

   free((*pad).data);
   (*pad).data = NULL;

   free((*list).ptr);
   (*list).ptr = NULL;
}


void range(struct NestedLoop_List* list, RangePad* pad) {
   int min;
   int max;
   int count;

   min = 1;
   max = (*pad).max;

   count = max - min + 1;   
   if (count < 0) { count = 0; }

   free((*pad).data);
   (*pad).data = (int*)calloc(count, sizeof(int*));

   {
      int* dst = (*pad).data;
      int  i;
      for (i=min; i<=max; i++) {
        *(dst++) = i;
      }
   }

   (*list).length   = count;
   (*list).free     = &range_free;
   (*list).free_pad = pad;
   NestedLoops_to_ptr_array(&((*list).ptr), (*pad).data, count, sizeof(int));
}


void range_inc_free(struct NestedLoop_List* list) {
   RangeIncPad* pad;

   pad = (RangeIncPad*)((*list).free_pad);

   free((*pad).data);
   (*pad).data = NULL;

   free((*list).ptr);
   (*list).ptr = NULL;
}


void range_inc(struct NestedLoop_List* list, RangeIncPad* pad) {
   int min;
   int max;
   int count;

   min = (*pad).min;
   max = (*pad).max;

   count = max - min + 1;   
   if (count < 0) { count = 0; }

   {
      int* dst = (*pad).data;
      int  i;
      for (i=min; i<=max; i++) {
        *(dst++) = i;
      }
   }

   free((*pad).data);
   (*pad).data = (int*)calloc(count, sizeof(int*));

   (*list).length   = count;
   (*list).free     = &range_inc_free;
   (*list).free_pad = pad;
   NestedLoops_to_ptr_array(&((*list).ptr), (*pad).data, count, sizeof(int));
}


void main() {
   /* This could be built dynamically.            */
   /* The args must remain accessible until done. */
   /* The caller handles cleanup of args.         */

   struct NestedLoop_Arg args[3];

   struct RangePad    pad0;
   struct RangeIncPad pad[2];

   pad0.min  = 1;
   pad0.max  = 3;
   pad0.data = NULL;
   args[0].ptr_type = NestedLoop_PT_FUNC;
   args[0].ptr.func.ptr = ⦥
   args[0].ptr.func.pad = &(pad[0]);

   pad[1-1].max  = 4;
   pad[1-1].data = NULL;
   args[1].ptr_type = NestedLoop_PT_FUNC;
   args[1].ptr.func.ptr = &range_inc;
   args[1].ptr.func.pad = &(pad[1-1]);

   pad[2-1].max = 4;
   pad[2-1].data = NULL;
   args[2].ptr_type = NestedLoop_PT_FUNC;
   args[2].ptr.func.ptr = &range_inc;
   args[2].ptr.func.pad = &(pad[2-1]);

   {
      struct NestedLoop_Iter iter;

      NestedLoops_get_iter(&iter, sizeof(args)/sizeof(args[0]), args);

      while (NestedLoops_fetch(&iter)) {
         int i;

         /* Skip results without an element from each loop. */
         if (iter.list.length != iter.num_loops) {
            continue;
         }

         for (i=0; i


The core logic is a direct translation of [tye]'s [mod://Algorithm::Loops]'s NestedLoops.

Re: Arbitrarily Nested Loops
created: 2006-02-07 07:08:06
Re: Arbitrarily Nested Loops
created: 2006-02-07 10:38:39

I ran into a similar challenge a bit ago - the generation of [http://en.wikipedia.org/wiki/Rainbow_table|rainbow tables]. Yes, there are a number of programs out there that do this, but I was attempting to apply the concept to create a generic module that could use any of the [cpan://Digest] modules. In any case, I didn't know about [cpan://Algorithm::Loops] then, so I reinvented the subset I needed. Whoops. ;-)

The nifty thing about the implementation, though, is that it's restartable -- or, rather, one can start at any state in the series (pre-set the odometer, if you will) and then get results for that and all following states. This is useful if, for example, you need all combinations of a given set of chars that are between 3 and 10 chars long.

Anyhow, I'm posting here because I think my code, being so special-purpose, might be easier to follow than the wonders of [cpan://Algorithm::Loops] (how [tye] manages to repeatedly pull off such deep magic, I will never know. Go [tye]!). It's also not complete, production code, so feel free to rip it to shreds (though I'd prefer patches).

package RainbowTable;

=pod ==========================================================================

=head1 NAME

RainbowTable - A module to assist in the creation of hash-based 
rainbow tables

=head1 VERSION

This document describes RainbowTable version 0.01

=head1 SYNOPSIS

Generates a range of printable character combinations (such as might 
be used for passwords) and converts them to hashes via a specifiable
hash function, caching the results for easy and fast searching later.
A hash type is required, other parameters are optional.

	use RainbowTable;
	
	#generate a table of MD5 hashes for all sets 1-10 chars in length
	my $rainbow = RainbowTable->new( 
		hash => 'MD5', maxlength => 10, minlength=>1,
	);
	while ($rainbow->next) {
		open my $fh, '>', $rainbow->hash or die($!);
		print $fh $rainbow->string;
		close $fh;
	}

The iterative model is used, since it can take a I long time to
generate the tables, and the interruption should be restartable.

	use RainbowTable;
	
	#resume processing using the 'start_at' parameter
	my $rainbow = RainbowTable->new(
		hash => 'MD5', start_at => 'aaaba',
	);

You can specifiy your own character set:

	# process only alphanumerics (no spaces, even)
	my $rainbow = RainbowTable->new(
		hash => 'MD5', chars => [ ('A'..'Z'), ('a'..'z'), (0..9) ]
	);

Any hash type with a related Digest::* module can be specified by
name. For example, 'MD5' will load Digest::MD5 for hashing. The
module interface must have a 'hexdigest' method in OO mode.

=cut

use strict;
use warnings;
use Params::Validate qw/validate_with :types/;

#________________________________________________________________________ new()_
sub new {

	# new ( )
	my $self = bless {}, shift;    # creates blessed-hash object
	my %args = validate_with(
		params => \@_,
		spec   => {
			hash      => 1,
			minlength => { type => SCALAR  , default => 1     },
			maxlength => { type => SCALAR  , default => 10    },
			start_at  => { type => SCALAR  , default => ''    },
			chars     => { type => ARRAYREF, default => undef },
		},
	);

	# some extra param checking and initialization
	unless ( defined $args{chars} ) {
		for ( 32  .. 126 ) { push @{ $args{chars} }, chr($_) }
		for ( 128 .. 168 ) { push @{ $args{chars} }, chr($_) }
	}

	foreach ( keys %args ) {
		$self->{$_} = $args{$_};
	}
	$self->{hash} = _loadHash( $args{hash} );

	$self->{SET} = join( '', @{ $args{chars} } );
	die("Must have sets of at least 1 char") if $self->{minlength} < 1;

	return $self;
}

#__________________________________________________________________ _loadHash()_
sub _loadHash {

	# _loadHash ( $name ) - loads hashing module by name
	#     Returns codeRef to hash object or dies on failure.
	my ($name) = @_;

	my $hashobj;
	eval {
		eval "require Digest::$name;" or die($@);
		$hashobj = "Digest::$name"->new()
		  or die("Unable to construct object for '$name'. $@");
		  
		die("'$name' can't 'hexdigest'") unless $hashobj->can('hexdigest');
	};
	if ($@) {
		die "Unable to initialize hashing function '$name': $@";
	}

	return $hashobj;
}

#__________________________________________________________________ spaceSize()_
sub spaceSize {

	# spaceSize ( ) - size of search space, ignoring starting location
	my $self = shift;
	my $base = @{ $self->{chars} };

	my $size =
	  $base**$self->{maxlength} - $base**( $self->{minlength} - 1 ) + 1;
	return $size;
}

#_______________________________________________________________________ next()_
sub next {

	# next ( ) - generates the next string to deal with, updates 'start_at'
	my $self = shift;
	my $size = length $self->{start_at};
	my $max  = length( $self->{SET} ) - 1;

	while ( length( $self->{start_at} ) < $self->{minlength} ) {
		$self->{start_at} .= substr( $self->{SET}, 0, 1 );
	}

	#return if we started with a too-short string.  This is because we will
	#have just initialized it.
	return 1 if $size < $self->{minlength};

	# increment
	my $col = $size - 1;

	while ( $col >= 0 ) {

		#find current char in set
		my $loc = index( $self->{SET}, substr( $self->{start_at}, $col, 1 ) );
		if ( $loc == $max ) {

			# roll over
			substr( $self->{start_at}, $col, 1 ) = substr( $self->{SET}, 0, 1 );
			if ( $col == 0 ) {

				# leftmost column rollover adds a new place
				$self->{start_at} =
				  substr( $self->{SET}, 0, 1 ) . $self->{start_at};
				last;
			}
			$col--;
		}
		else {
			substr( $self->{start_at}, $col, 1 ) =
			  substr( $self->{SET}, $loc + 1, 1 );
			last;
		}
	}    #/while

	#	$self->next if length($self->{start_at}) < $self->{minlength};
	return 0 if length( $self->{start_at} ) > $self->{maxlength};

	return 1;
}

#_______________________________________________________________________ hash()_
sub hash {

	# hash ( ) - calculates and returns hashed value
	my $self = shift;
	$self->{hash}->reset;
	$self->{hash}->add( $self->string );
	my $hash = $self->{hash}->hexdigest;

	return $hash;
}

#_____________________________________________________________________ string()_
sub string {

	# string ( ) - returns current working string
	my $self   = shift;
	my $string = $self->{start_at};

	return $string;
}

1;
<-radiant.matrix->
A collection of thoughts and links from the minds of geeks
The Code that can be seen is not the true Code
I haven't found a problem yet that can't be solved by a well-placed [http://en.wikipedia.org/wiki/Trebuchet|trebuchet]

perlmonks.org content © perlmonks.org and dokkeldepper, ikegami, Limbic~Region, radiantmatrix, thor

prlmnks.org © 2006 edmund von der burg (eccles & toad)

v 0.03