Read on at your own peril ;-)
Introduction
Some time ago I was toying with writing my own "modular audio composer" / sequencer. Sort of like [http://web.hibo.no/~mva/whatisbuzz.php|Buzz] for Linux, only better ;-)I started with an XS library that wraps the [http://www.ladspa.org|LADSPA API] in a nice perl OO API, just so I could play with the many existing LADSPA plugins (these plugins are mostly written in C / C++ and can be loaded into the program at runtime).
Then I added a bunch of perl code to make it all easier to use and manage. I'm still finetuning it, but that part is reasonably stable and [cpan://Audio::LADSPA|available on CPAN].
So far so good, but this project needed a GUI. After two initial attempts in Perl/Tk that got quite messy and didn't work as well as I'd hoped, I abandoned the whole thing for a two years.
Right now, I've got some spare time and I'm rewriting the interface in [cpan://Gtk2]. Once you get used to it, coding in Gtk is pretty easy, you can use nice OO techniques if you want, it's easy to refactor and it looks a whole lot prettier than Tk.
Though I'm still not finished, I'm mostly happy with how the code looks and the progress there.
Problems
This program should be able to generate audio streams that can be sequenced and changed on the fly via the GUI (and eventually also via MIDI controls). It works, with some latency, but:Some GUI updates are a bit slow and cause breakups in the audio stream.
It's possible that I could buffer more output to offset the problem but if the delay between GUI actions and output bcomes too long, the whole thing will feel sluggish and it just won't work very well.
For larger networks the whole thing slows down.
A large network here is one with lots of connected plugins. I still need to do some work to figure out where the bottle neck is, exactly.
If needed I could rewrite some of the perl code in [cpan://Audio::LADSPA::Network] in C/XS, but to get real good speed I might have to break the possibility of writing plugins in Perl. That would be a shame, since I use that option right now for "special" plugins that glue the network to the GUI and add extra functionality. I could also rewrite those plugins in C/XS, which wouldn't be too bad, but I like having the option to quickly make & change prototypes in Perl first and optimize later.
I'm looking for a good audio IO library for perl
At the moment I'm using [cpan://Audio::Play]. Works well and it's very easy to use, but it only supports monophonic output and doesn't allow me to check the output buffer state.Since I've got a 6 in / 6 out audio card that works very well with ALSA, I'd like to be able to use all of those inputs & outputs, either directly using the ALSA api or via [http://jackaudio.org|Jack] or both.
I might end up writing my own XS code to handle this anyway, but I would appreciate suggestions for existing solutions.
Constraints & design decisions
Currently, the highlevel design is simple: there's one perl process that initializes the Audio::LADSPA::Network code and sets up the GUI. GUI actions modify the network, and network changes are reflected in the GUI via Observer /Observable messages. (Note: due to [http://rt.cpan.org/Public/Bug/Display.html?id=19507|a bug] in [cpan://Class::Observable], this functionality is not yet available in the CPAN release of Audio::LADSPA).
Every few milliseconds, a Gtk2 timeout callback will ask the network to generate a bit of audio, which may be send to the audio output if the network contains a Play plugin (which uses Audio::Play)
Simplified diagram of the current design
+--------------------------------------------------------+ | Main process | | +--------------+ +----------------------+ | | | GUI (Gtk2) | ---update--> | Network with plugins | | | | | <--update--- | | | | +--------------+ | +------------------+ | | | +--------------+ | | audio output | | | | | Timer (Gtk2) | ---- run --> | +------------------+ | | + +--------------+ +----------------------+ | +--------------------------------------------------------+
Updates are things like: adding & removing plugins, adjusting parameters etc. The run action is what actually generates the audio stream (every run action generates about 2.5 milliseconds worth of audio).
Now, because all this runs in a single thread, the timer isn't completely reliable (gui updates can delay timer calls). Also, as noted above, it seems that the network code could use a few improvements to make it faster.
What I really want to do is to seperate out the run actions from the rest of the process. I can think of three ways to do that: threading, multi-process or a combination of the two.
Threading
Now that perl has some measure of thread support (based on pthreads it seems), I might be able to use that. Basically, put the run() calls in a seperate thread.
This seems ideal, especially if I want to use the Jack audio API, which more or less requires a seperate thread for its callback routine (Jack callbacks look like they could be fairly easily mapped to network runs).
Since I haven't really played with perl's threading for ages, and I haven't done any pthread programming in C, I do have some questions:
- Is it possible to share perl objects between threads?
- If so, how efficient is that?
- Does it also work with XS backed objects?
- If I can't share perl objects, would it be possible to create a 100% C thread that directly calls into the C code/data in another thread?
- Are there any pitfalls I should look out for?
Forking / multi-process
Since the Network/Plugin API is getting pretty stable, I might be able to run the whole network in a seperate process. This would mean I need some kind of IPC mechanism. I'd prefer not to write a whole IPC library from scratch if I can help it :-)More questions:
- What are the choices for good, lightweight, preferably OO IPC libraries like Java RMI, or Distributed Ruby?
- Since I might end up rewriting the Network/Plugin code in C, what are my choices if I want to do IPC from Perl to C and vice-versa?
- I still might need to run a seperate C thread to handle callbacks if I want to use Jack.
Conclusion
This is more or less everything I can think up now. I hope it wasn't too incoherent.I'm looking for suggestions, answers, links, anecdotes, questions, anything. In any case, if you've made it this far, thank you for reading it all. :-)
Cheers, Joost.
One alternative to generic perl that is great for processing vector data (like audio streams) is the Perl Data Language, or PDL. The vector and matrix operations are implemented in highly optimized C code and are programmed in a Matlab-like dialect that makes it easy to express DSP algorithms.
-Mark
The perl based plugins only do "control channels", that is - they generate only 1 float per channel each "tick" (with ticks once every 40 .. 200 samples). PDL can't really help with the control data, I think, since you're not acting on arrays of numbers, only single values.
I do see there's now a PDL::Audio module on CPAN. That wasn't there when I started this (2 years ago). I'll have to check it out.
If you can get SDL::Mixer to work for you, you should be able to incorporate it with Gtk2. You may be able to use the "do 1 loop" trick of Gtk2
Gtk2->main_iteration while Gtk2->events_pending;to avoid having to use the Gtk2 Mainloop. You could let the SDL event loop run things, just repeadetly call the Gtk2->main_iteration to keep the gui responsive. As an example with Tk
#!/usr/bin/perl -w
use strict;
use Gtk2;
use Tk;
my $mw = MainWindow->new(-title=>'Tk Window');
Gtk2->init;
my $window = Gtk2::Window->new('toplevel');
$window->set_title('Gtk2 Window');
my $glabel = Gtk2::Label->new("This is a Gtk2 Label");
$window->add($glabel);
$window->show_all;
my $tktimer = $mw->repeat(10, sub{
Gtk2->main_iteration while Gtk2->events_pending;
});
$mw->Button(-text=>' Quit ',
-command => sub{exit}
)->pack();
MainLoop;
__END__
As far as threads go, the recent versions of Gtk2 have a way to share Gtk2 objects across threads, but it is tricky and not generally gauranteed to always work. It has an enter and leave method, to tell Gtk2 to enter and leave the thread.
use Gtk2 qw/-init -threads-init/;
die "Glib::Object thread safetly failed"
unless Glib::Object->set_threadsafe (TRUE);
.....
.....
sub thread_work{
Gtk2::Gdk::Threads->enter;
.........
.........
Gtk2::Gdk::Threads->leave;
}
I'm not really a human, but I play one on earth. flash japh
SDL has a lot of nice routines for playing & mixing long samples & streams, but nothing that gives me that kind of response. Or so it seems to me reading the docs. If you or anyone else have tried something like this with SDL, I'd be happy to hear your stories :-)
Now that perl has some measure of thread support (based on pthreads it seems), I might be able to use that.Perl's threading support has improved greatly since its introduction in 5.8.0, and continues to improve. As such, you should use the most recent version of Perl (currently 5.8.8) for your system, and be sure to install the lastest versions of the threads and threads::shared modules. This will give you the latest feature sets as well as the best reliability.
* Is it possible to share perl objects between threads?Objects can be shared between threads. However, the class must be written to specifically support threads and sharing. You can do all this from scratch, of course, but I strongly recommend an object support module that handles all the necessary technicalities for you. Object::InsideOut is one such module that is very full-featured and fully supports threads and sharing.
* If so, how efficient is that?Thread data sharing is implemented using a tie methodology. Sharing objects between threads is as efficient as sharing any other data.
* Does it also work with XS backed objects?Yes, but you have to make sure it is done correctly. As an example, you might look at Math::Random::MT::Auto which has XS-backed objects, and uses Object::InsideOut for its object paradigm.
* If I can't share perl objects, would it be possible to create a 100% C thread that directly calls into the C code/data in another thread?Since you can share objects between threads, this question is moot. However, regarding the technicalities of using C-based threads from Perl, it should be possible if you don't access any Perl data in those threads, but I would imagine that such an approach would be fraught with difficulities.
* Are there any pitfalls I should look out for?If you want to write your classes to handle all the details of threading and sharing yourself, then you need to be cognizant of the CLONE method (see perlmod#Making your module threadsafe), and how to create sharable objects (see annotations for threads::shared).
However, if you use Object::InsideOut, all these details are taken care of for you.
Regardless of your approach, you need to be conversant with the threads and threads::shared PODs.
Remember: There's always one more bug.
For my C++ generation code, the perl functions specify a signal-processing netlist. I used the graph modules to solve the dependencies in the netlist, so that the C++ code came out in the correct order. It works great, and the generated code is clean. It looks like it was typed in by a very patient programmer! A typical example is a 51 line perl program that turned into a 9102 line Jack application.
At the moment, the graph I'm dealing with is a net of connected plugins that are mostly written in C (LADSPA plugins). I just calculate the right order once it's needed (in perl, using the Graph modules) and cache that until the the network is changed. After that the speed is acceptable - not great, but good enough for small experiments at least - since all I do is call one method for each plugin to generate a partial stream of X samples. The rest is normally handled in C.
I used HTML::Template on the C++ code, and I was in the process of switching to Text::Fillin when I stopped working on it. My code is from the early days of Jack when it was a moving target. I hope to get back to it someday and publish it in an updated form.
Our approaches are somewhat similar, and they are also similar to the PD (Pure Data) approach. The difference is that we like perl code, and PD uses a GUI instead.
I don't know if you are using the low-latency linux patches, etc, but if you are not, you should visit Planet CCRMA at home. The PlanetCCRMA web site and mailing list is a great place to find help and advice on linux audio development!
Currently I'm working on the sequencing part, which still needs a lot of GUI work. I don't know when that'll be done, since I'm also pretty busy doing paid work.
By the way, I've just uploaded the latest version (0.018) of Audio::LADSPA to CPAN. This doesn't include the GUI but at least it allows you to play around with any ladspa plugins you might have installed.