The problem
Installing a Perl web application should be as easy as "1. Unpack 2. Upload 3. Visit URL for configuration". Typically it isn't this easy, especially when the application follows the best practice of re-using CPAN modules.
By contrast, my average experience installing a PHP web application has been impressively easy, approximating the three step process I outlined above.
There's certainly nothing about the language itself that restricts this ease use. DadaMail has basically got it.
The DadaMail difference is that bucks the norms of CPAN culture. It doesn't exist on CPAN, and it bundles the modules it depends on from CPAN rather than requiring the users to fetch them themselves.
CPAN was designed with geeks in mind, not end users.
Having end users get modules themselves from would a better experience if it were designed for them. As an installation cpan appears to primary desired for users who run as root and have a shell and and compiler handy. Sure, there are documented ways to install modules as a non-root user, but it Shouldn't Be That Hard.
As a result, Perl apps aren't seeing the user base that they could. For example, selenium-regen is a nice helper script for use with Selenium. It comes bundled in the WWW-Selenium-Utils distribution. It would be useful to a lot of people outside of the Perl community, but fussing with installing related CPAN modules is required to use it.
The merits of bundling prequisites
Some people react negatively to the idea of bundling prerequisites, as DadaMail has done. When you do this, the argument goes, it will be difficult to deploy upgrades of the bundled modules because you now have several copies in use with one copy in each bundle.
Doing sysadin for a website development and hosting company, I've had the chance to experience the recommended "everyone uses one copy" philosophy. I'll be clear this has significant drawbacks as well. This design assumes that upgrades tend to make things better and that backwards compatibility is maintained. In practice, the only thing for certain about an upgrade is that Something Changed. The more projects that depend on that module, the greater chance there is that one them depended on the behavior before the change happened.
So while this approach may save module installation time, it potentially destabilizes the overall system, and requires quality assurance testing of all the projects that use the required module. Attention may still be needed to be given to many projects hosted on the server.
A little infrastructure may go a long way.
I have a vision for an alternate way to access CPAN that addresses these concerns.
This site would be designed for end-users and end user applications, not necessarily programmers and modules. However, unlike all the "script archives" out there, this system would be designed for projects built primary with CPAN modules.
When a user clicks a download link here, they would receive a tar ball which bundles all the distributions the project depends on. The distribution authors would only need to generate hints about these in a normal CPAN upload. The infrastracture would take care of generating the tar balls.
The result would be something that could be unpacked and uploaded, without additional installations or command line fussing.
Handling Perl XS
Handling Perl XS code would be the trickiest part of this system. "XS" code is C code compiled for a particular architecture.
Here we can follow in the footsteps of how Linux distributions already handle this for Perl XS code: by making binary packages for each target architecture.
From my initial poking around, it seems we have the key tools to address this systemically, by automatically (or easily) generating and uploading platform-specific binaries for each distribution which directly uses XS.
PAR generates platform-specific binaries for Perl distributions. Adding PAR integration to a typical Perl distribution is so easy that it could be automated in many cases. For a typical Makefile.PL, you can simply use inc::Module::Install; instead of ExtUtils::MakeMaker, and then add a line at the bottom like: par_base('MARKSTOS'); . Now make par is available to generate a PAR file made just for your platform.In Conclusion
Making Perl applications easier to deploy helps us all. Our work becomes more accessible to others, lowering the bar for gaining new users and contributors.
Historically, The infrastructure of CPAN has been a tremendous assest our community. Let's take the lesson of the value of good infrastructure and use it to expand the reach of Perl.
- Perl6 will address the XS issue by assuming you have Parrot installed. Since Parrot does all the heavy-lifting, nearly every CP6AN distro should be "pure-perl".
- The primary issue isn't "How do I install this as a non-root user?" but "How do I install this in a place that my code will find?" You are generally expected to be root because the perl executable has root-only paths hard-coded. It seems to me that we need to do two things:
- Provide more documentation/tutorials/whatever to get "use lib" into more use.
- Provide an easier way in CPAN(PLUS) to install into a directory on a per-install basis. Maybe something like "install Foo into lib" or "installdir lib^Vinstall Foo" or something like that. But, configuring CPAN as a non-root user when there's already a root user requires source-diving and crazy configuration. If you're not a root user, it shouldn't use the root config.
- Building on the prior comment and using Dadamail as an example, Dadamail should be able to specify to CPAN(PLUS) that it wishes to install all the prerequisites into a given directory, assuming that the prereqs aren't already there. You should also be able to specify to Dadamail where your personal installs are, if you wish. Then, Dadamail should be able to build a string of "use lib" items that it places in a special .pm file that it sources in immediately, probably through 'do' because 'use' scopes too much. That kind of work needs to be automated.
- The XS issue in Perl5 is a tough one, especially as most interesting apps will want to use MySQL and DBD::mysql is problematic in a shared-hosting environment. You either compile against 4.1+ or pre-4.1 and the two cannot live together, just like MP1 and MP2 cannot live together. Maybe C-compilers are just a requirement with free ones listed (gcc, MingWcc, etc).
- The interface to CPAN needs improving. The three .gz files that get downloaded shouldn't be that way by default. Whenever I go to cpan, I'm usually issuing 1-3 "install Foo" commands and that's it. cpan should be able to ask some CGI front-end "What's the correct FTP command for grabbing the Foo module?" and let a centralized database do the rest. Here's why - I often want to build my own personal CPAN so that I can use CPAN.pm to update my company's modules on production instead of using SVN. That's just one more thing to install on production machines and I don't like that. (The alternative is bundling and installing by hand which is fraught with danger as befits any by-hand solution.) If the API for CPAN was a clear and simple solution (with maybe a personal CPAN app?), then doing this would be a piece of cake. Imagine that as your upgrade solution for your company's modules!
My criteria for good software:
- Does it work?
- Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
If tools like they are involved at all, they should come into play as part of the web-based configuration step. There the software can ask permission, and then install extra modules under the covers.
My criteria for good software:
- Does it work?
- Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
Ugh, I'm sorry, but having each application install its own version of common libraries in its own location sounds to me like a maintenance nightmare. What happens if there's a security bug in an underlying module which makes it necessary for you to upgrade it? You then have two options:
- Find every instance of that module and update it in place, making sure that you break none of the applications in the process.
- Wait for all applications to release a new version with the upgraded module, which can take days to months, leaving you vulnerable in the meantime.
Both of these are extremely nasty. Additionally, encouraging the practice of using "private" versions of modules will lead to brittle applications which assume they know exactly which kind of environment they're working in, as well as a plethora of "tweaked" versions of modules for each app.
The CPAN way works extremely well for the two more common usage scenarios, that of a multi-user machine maintained by a (team of) admins whose responsibility it is to check that upgrades do not break apps (this can be helped a lot by using an OS/distribution which does sane dependency checking and provides clean upgrades for the included packages), and that of a single-user machine where the admin is also the developer/user. It does not work quite so well (as you have illustrated) for the shared web hosting environment where untrusted users need to install their own modules. Frankly, I don't consider that usage scenario important enough to make things harder for the rest of us. Maybe some specialized applications which are most commonly used in webhosting environments could go this way, but I'd recommend against making this common practice.
As an addendum, there is a single PHP web app I use, and I run multiple instances of it. Every time a security-related bug is discovered somewhere in the package I need to upgrade each and every instance of the app instead of just one underlying library, which pisses me off no end.
A computer is a state machine. Threads are for people who can't program state machines. -- Alan Cox
What happens if there's a security bug in an underlying module which makes it necessary for you to upgrade it?
You have disqualified yourself from the target user if have multiple versions of the same things install, and you have a clue about security.
1. Find every instance of that module and update it in place, making sure that you break none of the applications in the process.The part about "making sure you break none of the applications" is required step for all applications that use the module, whether you upgrade it one place or multiple places. And that's where most of the work. The "find every instance" would be a simple "find", if you didn't already know by memory or from documentation.
2. Wait for all applications to release a new version with the upgraded module, which can take days to months, leaving you vulnerable in the meantime.
This could be solved be the infrastructure. Remember the "auto generated packages"? When a new module is published to CPAN the package depends on, a new bundle could be available immediately. (Perhaps because the bundles would be generated on the fly or with smart caching). Like Linux distro packaging systems, the bundle name might include a revision in addition to the version, indicating that the bundled prereqs have been updated.
Additionally, encouraging the practice of using "private" versions of modules will lead to brittle applications which assume they know exactly which kind of environment they're working in, as well as a plethora of "tweaked" versions of modules for each app.Remember the design? The distributions would be regular CPAN distributions with regular dependencies. I'm just talking about generating an additional alternate view for end users. Since the authors are CPAN uploaders, they won't suddenly lose a clue about good software practices.
Frankly, I don't consider [the shared web hosting] usage scenario important enough to make things harder for the rest of us.
Ouch. This attitude of not caring about end users may do something to explain why several PHP applications now at the top of my list for features and end-user ease-of-use, even though I would prefer a Perl solution: osCommerce, PhpPgAdmin, PhpBB, PhpAds, WordPress, Drupal.
The net effect is that I'm now learning some PHP to light customizations to osCommerce and Drupal, because as a pragmatic user, they already have extensions written for them that do about everything I want.
Unless I've missed something in my research, Perl's got some catching up to do in the end-user killer web-app category, and I think some infrastructure focused on that could really help.
The result would actually be better, because in the background we would be using many of the well-documented, well-coded and well-tested we're already familiar with from CPAN.
You have disqualified yourself...
I'm sorry, I don't understand that sentence, care to explain what you mean here?
..."making sure you break none of the applications" is required step...whether you upgrade in one place or multiple places
Yes, but it's much harder, because you've got dog-knows how many different versions of the module installed for the different apps (that's the whole point of this scheme after all, right, to be able to have the version of a module which the app is written against) and you now have to find out what changes between each of those versions and the newly fixed one. Much, much harder. And yeah, finding the module file is easy enough, but then you have to care enough to upgrade the gazillion instances you have strewn all over your disk, and time being a scarce resource this will often not happen. This is a prime recipe for increasing the number of out-dated Perl modules installed on production machines and giving Perl a bad security rap.
..auto-generated packages..
I'm confused, if you can auto-generate the packages you're presuming that the bundle will work with the latest version of the module. So what's the difference to installing the latest module version into a central place? The only difference I can see is a false sense of security on the part of the user who thinks "this must work because I downloaded the Bundle" and will be all the more disappointed if it breaks.
..won't suddenly lose a clue about good software practices
Anything that gets made easier will be done more often. If module authors no longer need to care about their app working with the latest version of an underlying module version they will be tempted to use deprecated features, module internals etc., because "it works if you use the distribution". Other things (like adding the latest feature) will take precedence over robustness. That is part of my point, keeping different versions of modules on the same machine should be something that requires manual intervention, it's not good practice (most of the time) and you should have to think about what you're doing.
This attitude of not caring about end users
You misunderstand, I do care about the end user, and installation of a software package is indeed an important factor in its uptake (I've more than once refrained from trying out a Java app because of the horribly horrible horror that is Java installation and package handling). But Perl isn't a one-trick pony like PHP, it has applications that go far beyond the web-hosting sector. The requirements of all of these users needs to be taken into account when trying to improve some aspect of the language, not just of a relatively small subset of the user base.
If someone wants to use PHP instead of Perl (or Ruby or Python or any other language) and has good reasons for doing so, more power to them. To me the disadvantages of PHP are numerous enough that I'll prefer Perl any day, and one of these is the area of security. I care much more about Perl getting a bad reputation for its security than I care about the convenience of amateur webmasters (just to be clear, I'm not saying all users who use shared webhosting are amateurs, but those for whom this particular issue is a problem probably are).
I do agree that dependency handling an installation from CPAN is not as end-user friendly as it could be, but this is a very hard problem to solve correctly. I don't see any merit in auto-bundling software together for the purpose of spewing multiple versions of module all over your harddisk.
A computer is a state machine. Threads are for people who can't program state machines. -- Alan Cox
This is a prime recipe for increasing the number of out-dated Perl modules installed on production machines and giving Perl a bad security rap.Having a bad security rap involves having users (so someone actually cares about your reputation), and having bad security, so there's actually a problem). To have high profile security problems with Perl web apps, we have to have the high profile web applications in the first place. Moveable Type is a high profile web app using this strategy, and I'm not aware that it is contributing to Perl having a reputation of poor security. Having watching many ChangeLogs of Perl modules go by over the years, I rarely see updates because of security vulnerabilities. Changes tend to be bugfixes, new features, and sometimes incompatible changes (which are a real problem). Security updates are rare for Perl modules. When I think of visible Perl code with poor security, I think of the original Matt Wright script archive code, before the NMS project. If users want web software that's easy to install, they will find it, whether it's through projects that bundle like Moveable Type or Dada Mail, in a simple script format, or in another language like PHP.
I'm confused, if you can auto-generate the packages you're presuming that the bundle will work with the latest version of the module. So what's the difference to installing the latest module version into a central place? The only difference I can see is a false sense of security on the part of the user who thinks "this must work because I downloaded the Bundle" and will be all the more disappointed if it breaks.Module::Build allows you to be very precise about the versions of the prerequistes you require. You can set a minimum version, a maximum version, and make exceptions as well. So an author that wants to optimize for stability that can do that, by giving precise version numbers, so a newer prereq wouldn't automatically be included in the bundle. An author could also take the approach of simply setting a minimum version of prereqs, expecting that future versions will be compatible improvement or security improvements. This kind of choice is already exists in Module::Build for module authors, so this is aspect requires no infrastructure changes.
Anything that gets made easier will be done more often.Agreed. This proposal allows people to have an additional definition of "easy". It seems your definition would involve continuing to install and upgrade Perl web apps as you do now, which you can keep doing.
I care much more about Perl getting a bad reputation for its security than I care about the convenience of amateur webmasters
I'm a professional website developer, and this is a problem for me. I maintain CGI::Application and Data::FormValidator and have used them on many projects. Both are installed in a central location on our production web server. Although I'm in charge of reviewing and releasing the code for each, at least once in each case I managed to release a version with an incompatibile change that would break other applications in production if it was deployed, although the older versions worked fine.
This wasn't about security, this was about the realistic impossibility of maintaining complete compatibility when moving a software project forward.
So even as a professional website developer, it's safer choice to deploy specific module versions in production than to use a shared version.
It sounds like Perrin, another respected professional web developer, takes the same approach with Krang, another higher profile Perl web app. I know Jesse Erlbaum, the creator of CGI::Application, is also on the record as recommending putting public modules under source control and deploying private versions of them. (It looks like someone attended a talk he gave, and summarized that point as point 10 under "Implementation").
So, professional web developers are already using the approach deploying specific versions of prereqs with applications, and we know it works well for us.I do agree that dependency handling an installation from CPAN is not as end-user friendly as it could be, but this is a very hard problem to solve correctly. I don't see any merit in auto-bundling software together for the purpose of spewing multiple versions of module all over your harddisk.
I like to think of it as several applications which have the same features and stability as when they were deployed...
Thanks for sharing your concerns with this approach. I'm sure they are shared by others, and it gave me a chance to flesh out the details and justifications some more.
Mark
having each application install its own version of common libraries in its own location sounds to me like a maintenance nightmare.Having to worry that some of the applications you have installed will break whenever you upgrade any module is also a maintenance nightmare. In the end, you should be able choose which kind of nightmare you prefer, ;-) which may depend on your situation. A user who just wants to install one blog application is not in the same situation as a webmaster maintaining several applications.
A good compromise could be to provide both options: a standard CPAN package that lets you install the dependencies separately wherever you want, and a "user-friendly" bundle that you can just unwrap and works out of the box.
A good compromise could be to provide both options:
Exactly. I wonder why noone else brought this up. Why do we always have to ask either-or questions? Often, theres no problem pursuing both alternatives.
Makeshifts last the longest.
markjugg, thanks for this. Automatic bundlement on the hoof through infrastructure - I like it! Are you aware that you can have multi-architecture PAR balls, which might be a better fit than platform specific ones?
You might be interested in the slides for a talk I have given at YAPC::Europe 2005 and the London Perl Workshop.
--
Oh Lord, wont you burn me a Knoppix CD ?
My friends all rate Windows, I must disagree.
Your powers of persuasion will set them all free,
So oh Lord, wont you burn me a Knoppix CD ?
(Missquoting Janis Joplin)
I spent Christmas working on the documentation for installing catalyst on a shared server. It is *not* easy. It needs to be, or this technology just won't get adopted and will die. The catalyst approach for this is, use CPANPLUS... but CPANPLUS, though better, is still not that user friendly.
What my gut wants is, default installations for CPAN and CPANPLUS without having to answer all those questions, working 90% of the time, on a shared hosting environment. Because that's the reality, lots of people use shared hosting. I was able to make the CPAN install pretty painless by accepting the defaults and editing Config.pm after the fact, but I wasn't able to do this for CPANPLUS.
I guess the bottom line is, all those CPAN/CPANPLUS install questions are intimidating to people who just want to start developing, and they will switch to other solutions if this doesn't get streamlined.
Been a while since I've looked but I seem to recall that Movable Type does it the same way. No need for the end user to ever install modules via a shell as all common non-core libraries are bundled in $INSTALL_DIR/extlib.
And what I do is exactly that - bundle all those modules that are used with the code that I wrote. But unfortunately that is only part of the problems that Perl for Web is facing :(
Even that technique still requires that the end user is messing up with some *.cgi or *.pl files to set up the shebang line ... and to chmod it. So I usually install the applications myself. As I use CGI::App ;) it's only one file that dispatches to modules.
The other great problem is performance. CGI is out because of it performance problems, yet mod_perl still isn't quite suitable for any "Joe" on shared hosting. Mod_php still does it much better. I admit, it doesn't give you the power that mod_perl does - but Joe doesn't care ...
Sure there are other solutions like FastCGI or _____ (just put whatever you like/use here). But do web servers come with that as default? I don't think so - so who (which administrator) would even bother to install that? Especially when php does the job - and is mostly used anyway ...
I believe that Ruby is facing similar performance (CGI vs mod_ruby) problems when it comes to hosting.
When we solve those problems - we can start thinking about better advocating Perl ...
Let's first see what others are doing good and how they are doing it. Then let's adopt it ...
Have you tried freelancing? Check out Scriptlance - I work there.
For some reasons, i am trying to develop own web application framework, running under Apache/mod_perl. My idea is not to wrap Apache module nor force some templating schema, but to support easy usage of mod_perl features without detail knowledge about everything of mod_perl and to support possibility of code reusability.
I like clarity and simplicity, and I tried to use h2xs to build standardized perl project. Perhaps I am ignorant, but I did not successive, specially, when I did need to include some non-perl directories.
So, I have my devel directory, potentially synchronized with CVS or SVN repository, including perl modules and configurations and example applications. No autoconf, automake, reconf, Makefile.PL or another building/installing stuff. One row have to be included into httpd.conf (to include my piece of apache configuration) and apache must be restarted. Thats all. Other configurations are described in a framework conf and application confs, which are in the xml format.
I am writing this note to imagine, how it is easy to develop, deploy and/or use some software - do cvs checkout, include one row into the service configuration and restart the service. I know that I have the very specific constraints - I assume installed Apache/mod_perl and I am not dependant on other modules yet...
Heya,
I'm the author of Dada Mail, the app used as an example.
I thought I'd give a quick, "how do I do" for bundling the CPAN modules into Dada Mail.
As markjugg pointed out, the audience for Dada Mail is more likely to be your casual website owner, small business person, etc - not a fellow developer. As such, these people may know how to sling some HTML to a very good degree and will know how to configure fairly complex web apps that have simple configurations, like Dada Mail, Moveable Type, etc but WILL NOT be able to easily jump into a shell and fire up CPAN. It's also one more step in setting up a program and with this audience, too many steps leads to your application being passed by for a different alternative. Some examples of similar applications that I have been told are too hard to either use or to install are Gnu Mailman and actually a php application, called PHPList. I haven't installed either myself.
Anyways,
I keep a record of the CPAN modules I need for Dada Mail in a CPAN Bundle - Here it is: http://cvs.sourceforge.net/viewcvs.py/*checkout*/mojomail/dada_mail_stable/dada/extras/developers/DadaMail.pm
It's in a CVS repository, so any changes, of course, will be annotated to a ridiculous degree. Any other security advisories about the program would come from the regular sources. These are usually few and far between.
Since the Bundle file is POD, it's also used to make documentation: http://mojo.skazat.com/support/documentation/DadaMail.pm.html
When I make a distribution of Dada Mail, I'll run cpan myself and get a fresh copy of all the needed modules and test with this setup. If something breaks, I'll know about it before it gets released. Just to clarify: the only thing in CVS is the Bundle File - no actual CPAN modules are kept in CPAN - always plucked fresh for each release (each release seems to be now about every 2 - 4 weeks)
As for, "What happens if there's a security problem". If there's a change in a CPAN module, the new version of Dada Mail will automatically have the fix and any testing done will see if anything breaks. (I don't have a test suite, but it's in the pipeline)
Fixing this type of problem basically means reinstalling the entire distribution of Dada Mail again. If you're a little slicker and know how to use cpan and have root access, you can basically just make a custom distribution of Dada Mail for your specific environment by removing the, "perllib" directory that comes with Dada Mail and running the DadaMail.pm bundle locally. This following scenario is mentioned in the POD of the Bundle file itself. I don't see this being any more dangerous than allowing a user to have their of person perllib in the home directory.
I think there's an added benifit of bundling your own CPAN modules with your application and that is: different version of CPAN modules do act differently, as markjugg pointed out. For example, Dada Mail relies on the HTML::FromText CPAN module, but can only use version 1.005 of the module and nothing later. This is because the 2.x version of the module needs a much larger CPAN module requirement itself, some of these modules being XS modules, which, as a rule, isn't something I attempt to require, since the target audience won't know/doesn't have the correct resources to compile code.
What markjugg is dreaming of may be actually easier to handle if it was the responsibilty of the app distributor (for the time being). I strung my current build tools with a simple shell script and a slightly custom CPAN config. I talk about how I did this here: CPAN Shell, Bundles and downloading specific versions and here: Install a CPAN module at x location
There has been an instance of Dada Mail using a CPAN module and customizing the CPAN module itself to make it actually work very different than before. This CPAN module is now not a prereq and isn't a part of the Bundle. I don't see this as a problem, as it's sort of the whole idea of Free Software: you put something out because you think it's useful, allowing people the freedom to use and modify it as they see fit.
Just quickly glancing at the Bundle file for Dada Mail, you'll notice that it relies on a fair amount of CPAN code - about 3.2 megs. This includes some big ones, like MIME-Tools. I would say the rest of the app is about .5 megs. Dada Mail would not be even close to what it is now without CPAN. Dada Mail is very far in its life as a program and any program of this complexity will have problems dealing with all the different pieces. Knowing exactly which CPAN modules are being used in each version greatly simplifies making sure the app you ship will work. I can bet you the popularity of the program would not be as great as it is now if the CPAN modules were not installed automatically, or, if I only used the core modules.
Hope that enlightens a bit.
-justin simoni
skazat me
An alternative approach would be to have a standardised dispatch script for using perl modules as web applications. It would be a single script that specified a module, a method in that module, and query arguments. All web applications would have to be a module, and could all be installed from CPAN.
To make such an approach viable for people with no Perl skill, an alternative interface to CPAN could be devised. One that didn't require shell access. Such a system would need to be able to operate entirely through the browser, downloading modules in segments, as a page refreshed. Alternatively if no web access modules where available (such as LWP), a wizard could be created to help users upload modules themselves.
With a pair of standardised scipts (dispatch and install), CPAN could become an excellent way of distributing web apps.
I'm not sure what you mean by the "alternative interface" to CPAN, because that's the core of what the idea is. It provides an interface to CPAN that generates tarballs which are easy to unpack and upload without shell access. For fancy configurations, a web-based installation wizard might well talk to CPAN directly to install additional stuff.