New downloads and fixed webpages

New releases of SeqSuite and SLiMSuite are now available. The webpages have now hopefully been fixed too, including the broken Manual links. (A bit of trouble parsing some the docstrings had messed up the HTML, in case you care!) Please report any more anomalies.

There are not many major updates since the last release. The biggest are that SLiMFinder (and QSLiMFinder) now produce a single *.occ.csv containing motif instances for all datasets, in addition to the old dataset-specific files. This is to make the output more consistent with SLiMProb although do note that some of the column headers are different. The new file contains the same data as the old dataset-specific *.occ.csv files plus two additional columns: Dataset and RunID. (These match the main *.csv output.)

Dataset-specific results files have also been cleaned up a little for (Q)SLiMFinder and SLiMProb (i.e. the SLiMCore Class in libraries/rje_slimcore) to make the targz=T/F and savespace=X options a little more useful and consistent. This will be the subject of another post shortly.

Other miscellaneous updates are listed below.

Updates since last release:

• comparimotif_V3: Updated from Version 3.10.
→ Version 3.10: Added forking.
→ Version 3.11: Added additional overlap/matchfix checks during basic comparison to try and speed up.

• qslimfinder: Updated from Version 1.6.
→ Version 1.7: Fixed "MustHave=LIST" correction of motif space.

• slimfinder: Updated from Version 4.5.
→ Version 4.6: Minor modification to seqocc=T function. !Experimental! Added main occurrence output and modified savespace.

• rje_pydocs: Updated from Version 2.8.
→ Version 2.8: Added docsource=PATH : Input path for Python Module documentation (manuals etc.) ['../docs/']
→ Version 2.9: Attempts to fix some broken links and sort out manuals confusion

• rje_slimcore: Updated from Version 1.12.
→ Version 1.13: Modified the savespace settings to reduce numbers of files. targz file now uses RunID not Build Info.

• rje_uniprot: Updated from Version 3.14.
→ Version 3.14: Added dblist=LIST and dbsplit=T/F for additional DB link output control. Set unipath default to url.
→ Version 3.15: Added extraction of taxonomic groups. Add UniFormat to improve pure downloads.

Wonky webpages

It has come to my attention that the formatting has got a bit messed up at the SLiMSuite download pages. A new release of the downloads will be made soon and hopefully these kinks can get ironed out at the same time. (I'm not sure what's happened!)

SLiMSuite Down Under

Rich has recently moved to Sydney, Australia to take up a position at the University of New South Wales (UNSW). As a result, things are a bit disrupted at present but a better-than-normal service should resume shortly, as should continuing to update the documentation. There are also plans to mirror the Bioware servers in UNSW, so watch this space.

If you are in Sydney and fancy a SLiM-related job, Rich also has a postdoc opportunity at present.

A note on using BLAST+ with SLiMSuite

One of the major changes in the last release was the incorporation of BLAST+ as a replacement for BLAST. It should be noted that BLAST+ has not been benchmarked with SLiMSuite and it is not clear how and when it will behave differently, particularly with regards to UPC generation (i.e. generating clusters of unrelated proteins).

Early indications are that BLAST+ has a greater tendency to return no hits for short sequences. This can cause issues with SLiMSuite programs if oldblast=F. This will be fixed in the next release but running with dev=T gets round this issue in the meantime.

Please note that UPC may be different with BLAST versus BLAST+. This will need to be the focus of further study.

Log Files

Every program generates a log file when it is run. By default, this file will be named after the calling program (e.g. gasp.py will produce a log called gasp.log) but this can be changed with the log=FILE option. The basefile=X option will also set the base name of the log file, as well as the main results files (for most programs). Logs will be appended unless the newlog (or newlog=T) option is used.

The log file records information that may help subsequent interpretation of results or identify problems. Each line is tab delimited in the form:

#XXX    HH:MM:SS    Log Message.

Where #XXX is an identifier that can be used to parse out specific types of information, HH:MM:SS is the runtime in hours, minutes and seconds, and Log Message will be something (hopefully) informative.

All log files start with the same few lines:

#~~#    #~~#    #~~#
#LOG    00:00:00    Activity Log for PROGRAM X.X: DATE TIME YEAR
#DIR    00:00:00    Run from directory: RUNPATH
#ARG    00:00:00    Commandline arguments: ARGLIST
#CMD    00:00:00    Full Command List: [FULL ARGLIST]

This should contain all the information required to repeat the analysis:

• PROGRAM X.X: DATE TIME YEAR will have the program name, version number and the date/time of the run.
• RUNPATH is the directory from which the program was run.
• ARGLIST is the list of command-line arguments given to the program.
• FULL ARGLIST is the full list of command-line arguments including any arguments read in from ini files.

The last line can help identify the source of any unexpected behaviour due to default settings etc.

(The #~~# #~~# #~~# line is simply to act as a separator if appending an existing log file.)

If the program runs to completion successfully, it will end with another #LOG line:

#LOG    HH:MM:SS    PROGRAM V:X.X End: DATE TIME YEAR

If this line is not present then something went wrong during the run (see Error Messages, below - or it is still in progress. Other information is also recorded along with the runtime (HH:MM:SS since the program started). For help interpreting log files, please check the relevant software manual or contact me if the information is missing. (Hopefully, the log content is mostly self-explanatory but I shall add any explanations I have to send people to the relevant manual’s appendix.)

Error Messages

One of the most important aspects of the log file is to register any error messages. These are marked by an #ERR line header. Hopefully, there will not be any but if there was a problem with the run then these lines should contain the details. To catch these lines separately, errorlog=FILE will output error messages to an additional file.

New Software Release

New releases of SLiMSuite and SeqSuite are now available. Please note that RJESuite has now been discontinued - for simplicity, all of the extra gubbins is now part of the SeqSuite release. SLiMSuite still represents a cut-down version that focuses on Short Linear Motif analysis tools.

There have been a number of updates since the last release, which will be the focus of future posts. The biggest change since the last release is the implementation of BLAST+ as the default in place of BLAST for most tools. The old BLAST can still be invoked using the oldblast=T switch. In addition to blastpath=PATH, a new blast+path=PATH parameter will need to be set.

Apart from some file organisation tweaks, the other major change is that CompariMotif now has a memsaver=T mode, which will process very large motif lists much quicker and avoid memory issues. The XGMML output is not (yet) available in this mode. For multi-processor CPUs and large searchdb motif lists, CompariMotif now also supported forking (forks=X).

Documentation is in the process of having an overhaul and is still lagging behind as a result. Please ask if anything is unclear and that section of documentation will be prioritised.

Updates since last release:

• aphid: Updated from Version 2.0.
→ Version 2.1: Reduced import commands.

• budapest: Updated from Version 2.1.
→ Version 2.2: Removed unrequired rje_dismatrix import.
→ Version 2.3: Updated to use rje_blast_V2. Needs further updates for BLAST+. Deleted obsolete OLDreadMascot() method.

• comparimotif_V3: Updated from Version 3.9.
→ Version 3.10: Added MemSaver option, which will read and process input motifs (not searchdb) one motif at a time.
→ Version 3.10: Added forking.

• fiesta: Updated from Version 1.5.
→ Version 1.6: Removed HAQESAC import (uses MultiHAQ).
→ Version 1.7: Updated to use rje_blast_V2. Needs work to make function with BLAST+.

• gablam: Updated from Version 2.10.
→ Version 2.11: Altered to use BLAST+ and rje_blast_V2.

• gasp: Updated from Version 1.3.
→ Version 1.4: Minor tweaks to imports.

• gfessa: Updated from Version 1.2.
→ Version 1.3: Tidied module imports.
→ Version 1.4: Switched to rje_blast_V2. More work needed for BLAST+.

• haqesac: Updated from Version 1.8.
→ Version 1.9: Added rje_blast_V2 implementation and BLAST+. Use oldblast=T for old BLAST.

• peptcluster: Updated from Version 1.3.
→ Version 1.4: Bug fixes for end of sequence characters and different length peptides.

• picsi: Updated from Version 1.0.
→ Version 1.1: Updated to blast_V2 and BLAST+.

• pingu: Updated from Version 3.8.
→ Version 3.9: Tidied imports.

• qslimfinder: Updated from Version 1.5.
→ Version 1.6: Removed excess module imports.

• slimbench: Updated from Version 1.8.
→ Version 1.9: Added memsaver option. Replaced SLiMSearch with SLiMProb. Altered default IO paths.
→ Version 1.9: Removed 3DID again: new ELM interaction_domains file has position-specific PPI details.
→ Version 2.0: Major overhaul of input options to standardise/clarify. Implemented auto-downloads and PPI datasets.

• slimprob: Updated from Version 1.0.
→ Version 1.1: Tidied import commands.

• slimsuite: Created.
→ Version 0.0: Initial Compilation with downloadelm function.

• rje_pydocs: Updated from Version 2.6.
→ Version 2.7: Added rje_ppi output for module links.
→ Version 2.8: Added parsing of commandline options from docstring and cmdRead calls.
→ Version 2.8: Added docsource=PATH : Input path for Python Module documentation (manuals etc.) ['../docs/']

• rje: Updated from Version 4.6.
→ Version 4.7: Added self.warn list and self.warnLog() functions to Log object. Modified i=-1 quitchoice to raise not quit.
→ Version 4.8: Added perc cmdtype = float that is multiplied by 100.0 if < 1.0. Removed server option from iniCmds().

• rje_ancseq: Updated from Version 1.2.
→ Version 1.3: Changed "biproblem" error handling in gaspProbs()

• rje_blast_V1: Updated from Version 1.14.
→ Version 1.15: Added OldBLAST/Legacy option to Object for compatibility with rje_blast_V2. (Always True!)

• rje_blast_V2: Updated from Version 2.1.
→ Version 2.2: Added gablamData() to return old-style GABLAM dictionary from table.
→ Version 2.3: Added blastCluster() method to return UPC clustering and GABLAM distance matrix from a file.
→ Version 2.4: Scrapped BLAST "Run" field to simplify code - keep a single run per BLASTRun object.

• rje_db: Updated from Version 1.0.
→ Version 1.1: Added sortedEntries() function.
→ Version 1.2: Added Table.hasField(field). Add openTable(), readEntry() and readSet() methods.

• rje_forker: Created.
→ Version 0.0: Initial Compilation.

• rje_iridis: Updated from Version 1.8.
→ Version 1.9: Added scanning of legacy folder - moving GOPHER_V2!

• rje_obj: Updated from Version 1.0.
→ Version 1.1: Added rje_zen import and self.zen() to call rje_zen.Zen().wisdom().
→ Version 1.2: Added warnLog functions.
→ Version 1.3: Added perc cmdtype = float that is multiplied by 100.0 if < 1.0. Also added cmdtype = date for YYYY-MM-DD.

• rje_ppi: Updated from Version 2.7.
→ Version 2.8: Tweaked Spring Layout. Stores original Hub and Spoke Field.

• rje_seq: Updated from Version 3.16.
→ Version 3.17: Updated to use BLAST+ and rje_blast_V2

• rje_sequence: Updated from Version 2.2.
→ Version 2.3: Added alternative self.info keys for sequence (for UniProt splice variants). Added SpliceVar dict.

• rje_slimcore: Updated from Version 1.10.
→ Version 1.11: Tidied some of the module imports.
→ Version 1.12: Upgraded BLAST to BLAST+. Can use old BLAST with oldblast=T.

• rje_slimlist: Updated from Version 1.1.
→ Version 1.2: Added some extra functions for CompariMotif Memsaver mode

• rje_tree: Updated from Version 2.9.
→ Version 2.10: Added cleanup of *.r.csv file following R-based PNG generation.

• rje_uniprot: Updated from Version 3.13.
→ Version 3.14: Added direct retrieval of UniProt entries from URL, including full proteomes. Updated output file naming.
→ Version 3.14: Added dblist=LIST and dbsplit=T/F for additional DB link output control. Set unipath default to url.

• rje_xml: Updated from Version 0.1.
→ Version 0.2: Added parsing from URL.

• rje_xref: Updated from Version 0.0.
→ Version 1.0: Added xfrom and xto fields and xMap() function for mapping from one ID set to another.

External Components of SLiMSuite

In addition to the python modules included in the main downloads, some of the programs make use of the additional published programs. Wherever possible, these are freely available for downloading and installing. It is recommended that the user downloads and installs these programs according to the instructions given on the appropriate website.

Common programs

Some of the more common programs are listed below. The websites and instructions listed are subject to change, so it is advisable to Google for updated information if in doubt.

ALIGN: This is part of the Fasta package (Pearson, 1994; Pearson, 2000) and can be downloaded from the University of Virginia. Make sure that align is part of the download. For some reason it seems to have been dropped from later packages. You may need to install an earlier package first (e.g. 2.1) and then a later package. ALIGN is a legacy tool that is not a core component of any SLiMSuite program and need not be installed.

BLAST(+): BLAST (Altschul, et al., 1990) and BLAST+ are freely available for download from NCBI. BLAST has now largely been superseded by BLAST+ but some programs are still restricted to BLAST at the moment. Other tools can be made to use BLAST using oldblast=T.

CLUSTALW: ClustalW (Higgins and Sharp, 1988; Thompson, et al., 1994) is an old stalwart for bioinformatics and is freely available from http://www.clustal.org/. CLUSTALW is used by default to draw trees. See Replacing Components with Other Programs (below) for details of how to incorporate other tree-drawing packages. It is generally advisable to install the more recent version, clustalw2 and give SLiMSuite the updated clustal command using clustalw=clustalw2.

CLUSTAL Omega: CLUSTALO is a newer multiple alignment program from the Clustal team, available from clustal.org. (See below for more multiple alignment options.)

"The last alignment program you'll ever need."

R: The statistical programming language, R, is used for PNG visualisation by some SLiMSuite programs. R is freely available from: http://cran.r-project.org/. Note that some installations of R can require a bit of tweaking of the R scripts provided (in libraries/r/). Please email seqsuite@gmail.com if you require some help with this and/or have problems with the R-coded PNG visualisations.

It is recommended that paths to these programs are placed into an INI file (see Command-line Options). These can usually be replaced with different programs if desired (Replacing Components with Other Programs).

Replacing Components with Other Programs

Two of the most important functions performed by the external programs are multiple sequence alignment and tree-drawing. This section lists some ways to incorporate alternative programs for these functions into RJE programs. I am always interested to add more functionality, so if there is a program you would like to use instead of those listed, then please contact me and I may be able to add them in a more controlled fashion than below.

Alignment programs

By default, Clustal Omega is used for alignments as I have found this to be both fast and accurate. If there are problems with memory allocation for larger datasets etc., you might find the older ClustalW (Higgins and Sharp, 1988; Thompson, et al., 1994) to be safer. Any of these programs can be replaced by another program that uses the same command-line format call the programs.

The following alignment program options can currently be used with SLiMSuite programs:

clustalw=COMMAND : Path to CLUSTALW program ['clustalw']
clustalo=COMMAND : Path to CLUSTAL Omega alignment program ['clustalo']
mafft=COMMAND    : Path to MAFFT alignment program ['mafft']
muscle=COMMAND   : Path to MUSCLE alignment program ['muscle']
fsa=COMMAND      : Path to FSA alignment program ['fsa']            
pagan=COMMAND    : Path to PAGAN alignment program ['pagan']            
alnprog=X        : Choice of alignment program to use (clustalw/clustalo/muscle/mafft/fsa/pagan) [clustalo]

Any of these could be replaced with another script or program with the same input/output. For example, muscle=PATH could be used to redirect to any program using the system: program -in INFILE -out OUTFILE, where INFILE and OUTFILE are both fasta format. (Remember to set alnprog=muscle.)

For ClustalW, the system call is clustalw INFILE, where INFILE is in fasta format (*.fas) and the output file (*.aln) is in ClustalW align format. The path to ClustalW can be changed to redirect to another program using the clustalw=COMMAND option. (This maybe written as clustalw=PATH in places but the full path including the clustalw program should be given.)

Tree-drawing programs

The default for SLiMSuite programs is to use the Neighbour-joining method implemented in ClustalW for drawing trees. Although this is not the most accurate phylogeny construction algorithm around, it is fast and efficient and reasonable for trees of closely-related sequences with high bootstrap support, such as those HAQESAC was designed to build and work with.

Again, this program can be replaced with another using the maketree=PATH option. The system call used is:

clustalw -infile=INFILE -bootstrap=X -seed=X [-kimura]

for UNIX, or

clustalw INFILE -bootstrap=X -seed=X [-kimura]

for Windows, where INFILE is in fasta format (*.fas) and the output file (*.phb) is in bootstrapped Phylip format (I think).

It should work to have a program output a Newick Standard Format tree as *.nsf but I have not tested that. Phylip tree-drawing is also implemented. See rje_tree module documentation for details. Other phylogenetics programs can be added on request - anything able to generate Phylip or Newick format trees should be easy to add.

Wrapper scripts

If the chosen program does not accept the same input/output commands/formats then a wrapper script should be written. It is suggested to use Perl or Python for this. Although I cannot promise help in every suggestion, you are welcome to e-mail me for help with this and I will see what I can do.

Incorporating Other Programs into the Python Code

If you are feeling brave, you can actually edit the Python modules themselves. The key methods for this are rje_seq.muscleAln(), rje_seq.clustalAln() and rje_tree.makeTree(). Obviously, I cannot promise to give technical support for any changes that are made but, if you know what you are doing, you should be OK and I will help where I can.

References

This reference list needs completing but references for the older software listed include:

• Altschul SF, Gish W, Miller W, Myers EW and Lipman DJ (1990). Basic local alignment search tool. J Mol Biol, 215: 403-410.
• Edgar RC (2004). MUSCLE: a multiple sequence alignment method with reduced time and space complexity. BMC Bioinformatics, 5: 113.
• Higgins DG and Sharp PM (1988). CLUSTAL: a package for performing multiple sequence alignment on a microcomputer. Gene, 73: 237-244.
• Pearson WR (1994). Using the FASTA program to search protein and DNA sequence databases. Methods Mol Biol, 24: 307-331.
• Pearson WR (2000). Flexible sequence similarity searching with the FASTA3 program package. Methods Mol Biol, 132: 185-219.
• Thompson JD, Higgins DG and Gibson TJ (1994). CLUSTAL W: improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice. Nucleic Acids Res, 22: 4673-4680.