From 0ba1c2880a60f05bbc01a578d410b09580c16fbf Mon Sep 17 00:00:00 2001 From: Colin Date: Thu, 13 Aug 2015 17:44:11 -0500 Subject: [PATCH] Reformat code blocks --- docs/Apollo2Build.md | 4 ++-- docs/Command_line.md | 6 +++--- docs/Configure.md | 20 ++++++++++---------- docs/Data_loading.md | 33 +++++++++++++++++---------------- docs/Migration.md | 41 +++++++++++++++++++++++++++-------------- docs/Permissions.md | 11 ++++++----- docs/Setup.md | 10 +++++----- docs/Troubleshooting.md | 22 +++++++++++----------- 8 files changed, 81 insertions(+), 66 deletions(-) diff --git a/docs/Apollo2Build.md b/docs/Apollo2Build.md index 9ccd39ebcb..9cd1d0c38d 100644 --- a/docs/Apollo2Build.md +++ b/docs/Apollo2Build.md @@ -30,7 +30,7 @@ There won't be any difference in the workflow either way that you choose to get We can now perform a quick-start of the application in "development mode" with this command: -``` +``` ./apollo run-local ``` @@ -45,7 +45,7 @@ Also note: if there are any errors at this step, check the setup.log file for er After we have a server setup, we will want to add a new organism to the panel. If you are a new user, you will want to setup this data with the jbrowse pre-processing scripts. You can see the [data loading guide](Data_loading.md) for more details, but essentially, you will want to load a reference genome and an annotations file at a minimum: -``` +``` bin/prepare-refseqs.pl --fasta yourgenome.fasta --out /some/directory bin/flatfile-to-json.pl --gff yourannotations.gff --type mRNA --trackLabel AnnotationsGff --out /some/directory ``` diff --git a/docs/Command_line.md b/docs/Command_line.md index a8cc42d2fe..205a912f96 100644 --- a/docs/Command_line.md +++ b/docs/Command_line.md @@ -12,7 +12,7 @@ The command line tools are located in docs/web_services/examples, and they are m Example: -``` +``` get_gff3.groovy -organism Amel_4.5 -username admin@webapollo.com -password admin_password -url http://localhost:8080/apollo > my output.gff3 ``` @@ -27,7 +27,7 @@ The -username and -password refer to the admin user, and they can also be specif Example: -``` +``` add_users.groovy -username admin@webapollo.com -password admin_password -newuser newuser@test.com -newpassword newuserpass -destinationurl http://localhost:8080/apollo ``` @@ -40,7 +40,7 @@ A list of users specified in a csv file can also be used as input. Example: -``` +``` add_organism.groovy -name yeast -url http://localhost:8080/apollo/ -directory /opt/apollo/yeast -username admin@webapollo.com -password admin_password ``` diff --git a/docs/Configure.md b/docs/Configure.md index 7a2491219a..ef5ac51bdd 100644 --- a/docs/Configure.md +++ b/docs/Configure.md @@ -15,7 +15,7 @@ The main configuration settings for Apollo are stored in `grails-app/conf/Config `apollo-config.groovy` file (i.e. the same file that contains your database parameters). Here are the defaults that are defined in the Config.groovy file: -``` +``` // default apollo settings apollo { default_minimum_intron_size = 1 @@ -66,7 +66,7 @@ These settings are essentially the same familiar parameters from a config.xml fi The defaults are generally sufficient, but as noted above, you can override any particular parameter in your apollo-config.groovy file, e.g. you can add override configuration any given parameter as follows: -``` +``` grails { apollo.get_translation_code = 1 apollo { @@ -88,7 +88,7 @@ Canned comments are configured via the admin panel on the web interface. Web Apollo can be configured to work with sequence search tools. The tool UCSC BLAT is commonly used and can be easily configured via the config file, with the general parameters given as follows: -``` +``` sequence_search_tools = [ blat_nuc: [ search_exe: "/usr/local/bin/blat", @@ -119,7 +119,7 @@ installation binaries reside elsewhere, edit the search_exe location to point to Data adapters are currently configured as follows -``` +``` data_adapters = [[ permission: 1, key: "GFF3", @@ -196,17 +196,17 @@ One thing to consider with this proxy setup is the websocket calls. We use the S The most simple setup on apache is a reverse proxy. Note that a reverse proxy _does not_ use `ProxyRequests On` (i.e. if you want you can set `ProxyRequests Off`, it is not relevant to reverse proxies). Here is the most basic configuration: -``` +``` ProxyPass /apollo http://localhost:8080/apollo ProxyPassReverse /apollo http://localhost:8080/apollo ProxyPassReverseCookiePath / http://localhost:8080/apollo -``` +``` This setup will use AJAX long-polling unless websockets are also configured to be proxied. To setup the proxy for websockets, you can use mod_proxy_wstunnel (available for httpd 2.4): http://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html First load the module or use a2enmod enable it (on ubuntu / debian): -``` +``` LoadModule proxy_wstunnel_module libexec/apache2/mod_proxy_wstunnel.so ``` @@ -221,7 +221,7 @@ Then in your server config, i.e. httpd.conf, add extra ProxyPass calls for the w Note: you can also proxy all apps on your tomcat server to a subdirectory like this, but the important part of this configuration is that ProxyPassReverseCookiePath is configured so that sessions are set up properly (otherwise it will look like the app is loaded, but you will not be able to login) -``` +``` ProxyPass /testing http://localhost:8080 ProxyPassReverse /testing http://localhost:8080 ProxyPassReverseCookiePath / /testing @@ -235,7 +235,7 @@ With this configuration, an app that was originally at http://localhost:8080/apo Your setup may vary, but setting the upgrade headers can be used for the websocket configuration http://nginx.org/en/docs/http/websocket.html -``` +``` map $http_upgrade $connection_upgrade { default upgrade; '' close; @@ -275,7 +275,7 @@ There are a variety of different ways to include new CSS into the browser, but t Add the following statement to your trackList.json: -``` +``` "css" : "data/yourfile.css" ``` diff --git a/docs/Data_loading.md b/docs/Data_loading.md index b666fe3052..71d03b96e2 100644 --- a/docs/Data_loading.md +++ b/docs/Data_loading.md @@ -12,20 +12,20 @@ guide](Troubleshooting.md). The first step to setup the genome browser is to load the reference genome data. We'll use the `prepare-refseqs.pl` script to output to the data directory that we will point to later in the organism tab. -``` +``` bin/prepare-refseqs.pl --fasta pyu_data/scf1117875582023.fa --out /opt/apollo/data -```` +``` ### flatfile-to-json.pl The flatfile-to-json.pl script can be used to setup a GFF3 tracks with flexible feature types. Here, we'll start off by loading data from the MAKER generated GFF for the Pythium ultimum data. The simplest loading command specifies a --trackLabel, the --type of feature to load, the --gff file and the --out directory. -``` +``` bin/flatfile-to-json.pl --gff pyu_data/scf1117875582023.gff --type mRNA --trackLabel MAKER --out /opt/apollo/data -```` +``` -Note: you can also use simple scripts like bin/maker2jbrowse to expedite loading of MAKER data. +Note: you can also use simple scripts like `bin/maker2jbrowse` to expedite loading of MAKER data. Also see the section [Customizing features](Data_loading.md#customizing-features) section for more information on customizing the CSS styles of the Web Apollo 2.0 features. @@ -34,7 +34,7 @@ Also see the section [Customizing features](Data_loading.md#customizing-features Once data tracks have been created, you can generate a searchable index of names using the generate-names.pl script: -``` +``` bin/generate-names.pl --verbose --out /opt/apollo/data ``` @@ -47,17 +47,18 @@ option to disable the completion on certain tracks. WebApollo natively supports BAM files and the file can be read (in chunks) directly from the server with no preprocessing. -To add a BAM track, copy the BAM+BAI files to your data directory, and then use the add-bam-track.pl to add the file to the tracklist. +To add a BAM track, copy the .bam and .bam.bai files to your data directory, and then use the add-bam-track.pl to add the file to the tracklist. -``` +``` mkdir /opt/apollo/data/bam - cp pyu_data/simulated-sorted.bam* /opt/apollo/data/bam + cp pyu_data/simulated-sorted.bam /opt/apollo/data/bam + cp pyu_data/simulated-sorted.bam.bai /opt/apollo/data/bam bin/add-bam-track.pl --bam_url bam/simulated-sorted.bam \    --label simulated_bam --key "simulated BAM" -i /opt/apollo/data/trackList.json ``` -Note: the `bam_url` parameter is a URL that is relative to the data directory. It is not a filepath! +Note: the `bam_url` parameter is a URL that is relative to the data directory. It is not a filepath! Also, the .bai will automatically be located if it is simply the .bam with .bai appended to it. ### add-bw-track.pl @@ -66,7 +67,7 @@ WebApollo also has native support for BigWig files (.bw), so no extra processing To use this, copy the BigWig data into the jbrowse data directory and then use the add-bw-track.pl to add the file to the tracklist. -``` +``` mkdir /opt/apollo/data/bigwig cp pyu_data/*.bw /opt/apollo/data/bigwig bin/add-bw-track.pl --bw_url bigwig/simulated-sorted.coverage.bw \ @@ -81,7 +82,7 @@ To change how the different annotation types look in the "User-created annotatio annotation type to the appropriate CSS class. This data resides in `client/apollo/json/annot.json`, which is a file containing WebApollo tracks that is loaded by default. You'll need to modify the JSON entry whose label is `Annotations`. Of particular interest is the `alternateClasses` element. Let's look at that default element: -``` +``` "alternateClasses": { "pseudogene" : { "className" : "light-purple-80pct", @@ -145,7 +146,7 @@ It is relatively easy to add other stylesheets that have custom style classes th `flatfile-to-json.pl`. For example, you can create `/opt/apollo/data/custom_track_styles.css` which contains two new styles: -``` +``` .gold-90pct, .plus-gold-90pct, .minus-gold-90pct { @@ -179,7 +180,7 @@ adding this to the trackList.json Then you may use these new styles using --subfeatureClasses, which uses the specified CSS classes for your features in the genome browser, for example: -``` +``` bin/flatfile-to-json.pl --gff MyFile.gff --type mRNA --trackLabel MyTrack --subfeatureClasses '{"CDS":"gold-90pct", "UTR": "dimgold-60pct"}' ``` @@ -191,7 +192,7 @@ Then you may use these new styles using --subfeatureClasses, which uses the spec You can use the `tools/data/add_transcripts_from_gff3_to_annotations.pl` script to bulk load GFF3 files with transcripts to the user annotation track. Let's say we want to load our `maker.gff` transcripts. -``` +``` tools/data/add_transcripts_from_gff3_to_annotations.pl \ -U localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin \ -i scf1117875582023.gff -type mRNA @@ -204,7 +205,7 @@ You can still use this script even if the GFF3 file that you are loading does no Let's say we want to load `match` and `match_part` features as transcripts and exons respectively. We'll use the `blastn.gff` file as an example. -``` +``` tools/data/add_transcripts_from_gff3_to_annotations.pl \ -U localhost:8080/WebApollo -u web_apollo_admin -p web_apollo_admin \ -i cf1117875582023gff -t match -e match_part diff --git a/docs/Migration.md b/docs/Migration.md index 93120e9253..e500460a6f 100644 --- a/docs/Migration.md +++ b/docs/Migration.md @@ -7,23 +7,33 @@ In all cases you will need to follow the [guide for setting up your 2.0 instance ## Migration from Evaluation to Production: -If you are running your evaluation/development version using ```./apollo run-local``` +If you are running your evaluation/development version using `./apollo run-local` when you setup your production instance, any prior annotations will use a separate database. -If you are using the same production instance you can use scripts to [delete all annotations and preferences](../scripts/delete_all_features.sh) -or [just the annotations](../scripts/delete_only_features.sh). +If you are using the same production instance you can use scripts to delete all annotations and preferences: + +``` + scripts/delete_all_features.sh +``` + +or just the annotations: + +``` + scripts/delete_only_features.sh +``` If you want to start from scratch (including reloading organisms and users), you can just drop the database (when the server is not running) and the proper tables will be recreated on startup. ## Migration from 1.0 to 2.0: -In each case you will need to use [Apollo 2 web services](web_services_link.png). We provide examples in the form of [migration scripts](web_services/examples). +We provide examples in the form of [migration scripts](web_services/examples) in the docs/web_services/examples directory. These tools are also described in the [command line tools section](Command_line.md). In each case these scripts will the use Apollo 2 web services API, which you can find out more about via the user interface: + +![](web_services_link.png) -To use the newer examples written in ```groovy```, you'll need to install it in a similar manner as you did grails: ```gvm install groovy```. If you have not installed [gvm](http://gvmtool.net/) (or grails) you should be able to run '''curl -s get.gvmtool.net | bash -'''. +If you are upgrading to a new version of grails or groovy, you'll need to install it in a similar manner as you did grails, e.g. `gvm install groovy`. If you have not installed [gvm](http://gvmtool.net/) (or grails) you should be able to download it easily following their instructions. -We have written new examples using the groovy language, but mostly any language will work (Perl, shell/curl, Python, etc.). +We have written many of the [command line tools](Command_line.md) examples using the groovy language, but mostly any language will work (Perl, shell/curl, Python, etc.). ### Migrate Annotations @@ -32,17 +42,22 @@ We provide a [migration script](web_services/examples/groovy/migrate_annotations connects to a single Web Apollo 1 instance and populates the annotations for an organism for a set of sequences / (confusingly called tracks as well). It would be best to develop your script on a development instance of Apollo2 for restricted sequences. To get the scripts working properly, you'll need to provide the list of sequences (or tracks) to migrate for each organism. -You can get the list of tracks by either using the database (```select * from tracks ;```) or looking -in the Web Apollo annotations directory (e.g., ```/opt/apollo/annotations/ | grep Annotations | grep -v history | paste -s -d"," - ```). The latter will only yield tracks that have annotations. +You can get the list of tracks by either using the database (`select * from tracks ;`) or looking +in the Web Apollo annotations directory + +``` + /opt/apollo/annotations/ | grep Annotations | grep -v history | paste -s -d"," - +``` + ### Migrate Users -You have to add users de novo using something like the [add_users script](web_services/examples/groovy/add_users.groovy). +You have to add users de novo using something like the [add_users.groovy script](web_services/examples/groovy/add_users.groovy). In this case you create a csv file (you can start in Excel) with the email, name, password, and role ('user' or 'admin'). This is passed into the add_users.groovy script and users are added. -From Web Apollo 1, you should be able to pull user names out of the database ```select * from users ;```, but there is not much overlap between users in Web Apollo 1 and Apollo2. +From Web Apollo 1, you should be able to pull user names out of the database `select * from users ;`, but there is not much overlap between users in Web Apollo 1 and Apollo2. If you have only a few users, however, just adding them manually on the users will likely be easier. @@ -50,8 +65,6 @@ If you have only a few users, however, just adding them manually on the users wi If possible adding organisms on the organisms tab is the easiest option if you only have a handful of organisms. -However, we provide a mechanism via the [add_organism.groovy script](web_services/examples/groovy/add_organism.groovy). -In this case organisms are added individually as organisms are on separate tomcat instances. - +The [add_organism.groovy script](web_services/examples/groovy/add_organism.groovy) can help automate this process if you have a large number of migrations to handle. diff --git a/docs/Permissions.md b/docs/Permissions.md index cf42e1e53f..cb4909b481 100644 --- a/docs/Permissions.md +++ b/docs/Permissions.md @@ -11,7 +11,8 @@ Can only view things related to that organism. * **read**: view / search only, no annotation -``` + +``` Annotations: lock detail / coding RefSeq: hide export Organism: hide @@ -22,20 +23,20 @@ Can only view things related to that organism. ``` * **export**: same as read, but can use the export screen -``` +``` RefSeq: show export ``` * **write**: same as above, but can add / edit annotations -``` +``` Annotations: allow editing JBrowse: enable UcA track ``` * **admin**: access to everything for that organism -``` +``` Organism: show User: show Group: show @@ -45,7 +46,7 @@ Can only view things related to that organism. Table of permissions: -``` +``` Permission | Annotator | Tracks | Ref Seq | Organism | Users/groups | ------------- | --------------- | ------- | -------- | -------- | ------- | READ | Vis / locked | None | no export| hide | hide | diff --git a/docs/Setup.md b/docs/Setup.md index 8fe2208adc..8247eb07e8 100644 --- a/docs/Setup.md +++ b/docs/Setup.md @@ -54,7 +54,7 @@ The `apollo run-local` command only launches a temporary server and should reall -``` +``` ./apollo deploy ``` @@ -64,7 +64,7 @@ This command will package the application and it will download any missing pre-r Alternatively, as we alluded to previously, you can also launch a temporary instance of the server which is useful for testing -``` +``` ./apollo run-local 8080 ``` @@ -118,7 +118,7 @@ The `apollo release` command also requires some extra Perl modules, namely Text: To build a Web Apollo release with Javascript minimization, you can use the command -``` +``` ./apollo release ``` @@ -131,7 +131,7 @@ In all other respects, `apollo release` is exactly the same as `apollo deploy` t To perform active development of the codebase, use -``` +``` ./apollo debug ``` @@ -144,7 +144,7 @@ If you modify the javascript files (i.e. the client directory), you can run `scr A temporary tomcat server can be automatically launched without having you configure your own Tomcat server with the `apollo run-local` command: -``` +``` ./apollo run-local [8085] ``` diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md index 74b6c81a0a..cc7d35cc7e 100644 --- a/docs/Troubleshooting.md +++ b/docs/Troubleshooting.md @@ -7,14 +7,14 @@ Typically, the default memory allowance for the Java Virtual Machine (JVM) is to #### Suggested Tomcat memory settings -``` +``` export CATALINA_OPTS="-Xms512m -Xmx1g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=256m" ``` In cases where the assembled genome is highly fragmented, additional tuning of memory requirements and garbage collection will be necessary to maintain the system stable. Below is an example from a research group that maintains over 40 Apollo instances with assemblies that range from 1,000 to 150,000 scaffolds (reference sequences): -``` +``` export CATALINA_OPTS="-Xmx12288m -Xms8192m -XX:PermSize=256m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=64m -XX:+UseG1GC -XX:+CMSClassUnloadingEnabled -Xloggc:$CATALINA_HOME/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" ``` @@ -24,13 +24,13 @@ To change your settings, you can edit the setenv.sh script in #### Confirm your settings -``` +``` ps -ef | grep -i tomcat ``` You should see your memory settings in the command that runs tomcat. -``` +``` /usr/local/bin/java -Dfile.encoding=UTF-8 -Xmx2048M -Xms64M -XX:MaxPermSize=1024m ``` @@ -43,8 +43,8 @@ If you have named your web application named "WebApollo.war" then you can remove - `apollo deploy` (or `apollo release` for javascript-minimization) - stop tomcat - `rm -f $TOMCAT_WEBAPPS_DIR/webapps/WebApollo.war && sudo rm -rf $TOMCAT_WEBAPPS_DIR/webapps/WebApollo` -- start tomcat - `cp target/apollo-XXX.war $TOMCAT_WEBAPPS_DIR/webapps/WebApollo.war` +- start tomcat ### Tomcat permissions @@ -84,7 +84,7 @@ Note that you can also configure tomcat to run on different ports, or you can la If you receive an error similar to this: -``` +``` SEVERE: Unable to create initial connections of pool. org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /var/lib/tomcat7/prodDb.lock.db" [8000-176] ``` @@ -94,7 +94,7 @@ If you use H2 (which is great for testing or single-user user, but not for full- You can modify the specified data directory for the H2 database in the apollo-config.groovy. For example, using the /tmp/ directory, or some other directory: -``` +``` url = "jdbc:h2:/tmp/prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE" ``` @@ -109,7 +109,7 @@ that tomcat is running in e.g., ``/var/lib/tomcat7/``. In some instances you can't write to the default cache location on disk. Part of an example config log: -``` +``` 2015-07-03 14:37:39,675 [main] ERROR context.GrailsContextLoaderListener - Error initializing the application: null java.lang.NullPointerException at grails.plugin.cache.ehcache.GrailsEhCacheManagerFactoryBean$ReloadableCacheManager.rebuild(GrailsEhCacheManagerFactoryBean.java:171) @@ -121,7 +121,7 @@ There are several solutions to this, but all involve updating the ```apollo-conf 1) Disabling the cache: -``` +``` grails.cache.config = { cache { enabled = false @@ -132,7 +132,7 @@ There are several solutions to this, but all involve updating the ```apollo-conf 2) Disallow writing overflow to disk (best solution for small installations): -``` +``` grails.cache.config = { // avoid ehcache naming conflict to run multiple WA instances provider { @@ -150,7 +150,7 @@ There are several solutions to this, but all involve updating the ```apollo-conf 3) Specify the overflow directory (best for high load servers, which will need the cache). Make sure your tomcat / web-server user can write to that directory: -``` +``` // copy from Config.groovy except where noted grails.cache.config = { ...