a developer's pipeline framework
What is pypiper?
Pypiper is a development-oriented pipeline framework. It is a python package that helps you write robust pipelines directly in python, handling mundane tasks like restartability, monitoring for time and memory use, monitoring job status, copious log output, robust error handling, easy debugging tools, and guaranteed file output integrity.
What makes pypiper better?
With Pypiper, simplicity is paramount. Prerequisites are few: base python and 2 common packages (
psutil). It should take fewer than 15 minutes to build your first pipeline and only an hour or two to learn the advanced features. Pypiper pipelines are:
- written in pure python, so they do not require learning a new language;
- easy to modify, so they are simple to update and maintain;
- simple to understand for an outsider, so they can be approached by others.
These traits make pypiper ideally suited for pipelines under active development. Read more about the pypiper philosophy.
Releases are posted as GitHub releases, or you can install from PyPI using
Global scope for single user:
pip install --user --upgrade piper
Within an active virtual environment:
pip install --upgrade piper
To employ pypiper, you build something like a shell script, but pass the commands through the
run method on a
PipelineManager object. Build your pipeline in pure python:
#!/usr/bin/env python import pypiper outfolder = "hello_pypiper_results" # Choose a folder for your results # Create a PipelineManager, the workhorse of pypiper pm = pypiper.PipelineManager(name="hello_pypiper", outfolder=outfolder) # Timestamps to delineate pipeline sections are easy: pm.timestamp("Hello!") # Now build a command and pass it to pm.run() target_file = "hello_pypiper_results/output.txt" command = "echo 'Hello, Pypiper!' > " + target_file pm.run(command, target_file) pm.stop_pipeline()
Then invoke your pipeline via the command-line:
python my_pipeline.py --help
Pypiper differs from existing frameworks in its focus on simplicity. Pypiper requires learning no new language, as pipelines are written in pure python. Pypiper is geared toward developing pipelines that are contained in a single file, easy to update, and easy to understand.