ADD: cross-compatibility with either Python2 or Python3 #52

kevinkle · 2018-02-04T01:13:02Z

No description provided.

…able dependencies for platform linux-64

…tting ectyper version above that on bioconda

…m w setup

ADD: travis badge

…ping into pasteurize

… predictions

…ctions

…mpty, where are we losing our data?

…, so tell me why youre continuing

kevinkle · 2018-02-12T00:37:52Z

The main functions work in py27 as of 0f8cdf2. There's no rush to merge this PR, but would recommend doing so before making any changes.

…between Py27 and Py36

kevinkle · 2018-02-12T02:28:03Z

Tests passing in py36, 1 failing in py27:

$ nosetests
2018-02-12 01:16:48,609 ectyper.ectyper INFO     Temporary files and directory created
2018-02-12 01:16:48,609 ectyper.ectyper INFO     Temporary files and directory created
.2018-02-12 01:16:48,610 ectyper.ectyper INFO     Starting ectyper
Serotype prediction with input:
             Namespace(detailed=False, input=u'test/Data/test_dir', output=u'test_folder_input', percentIdentity=90, percentLength=50, species=False, verify=False)
             Log file is: /home/travis/build/phac-nml/ecoli_serotyping/ectyper.log
2018-02-12 01:16:48,613 ectyper.ectyper INFO     Temporary files and directory created
2018-02-12 01:16:48,613 ectyper.ectyper INFO     Gathering genome files
2018-02-12 01:16:48,613 ectyper.genomeFunctions INFO     Gathering genomes from directory test/Data/test_dir
2018-02-12 01:16:48,613 ectyper.ectyper INFO     Removing invalid file types
2018-02-12 01:16:48,618 ectyper.genomeFunctions WARNING  test/Data/test_dir/badfasta.fasta is not a valid file
2018-02-12 01:16:48,619 ectyper.genomeFunctions WARNING  test/Data/test_dir/sample.fasta.zip is not a valid file
2018-02-12 01:16:48,621 ectyper.genomeFunctions WARNING  Compressed file is not supported: test/Data/test_dir/sampletar
2018-02-12 01:16:48,622 ectyper.genomeFunctions WARNING  test/Data/test_dir/test_junk.txt is not a fasta/fastq file
2018-02-12 01:16:48,622 ectyper.ectyper INFO     Preparing genome files for blast alignment
2018-02-12 01:16:48,625 ectyper.ectyper INFO     4 final fasta files
2018-02-12 01:16:48,625 ectyper.ectyper INFO     Standardizing the genome headers
2018-02-12 01:16:48,627 ectyper.ectyper INFO     Start creating blast database #1
2018-02-12 01:16:48,627 ectyper.ectyper INFO     Using SEROTYPE_FILE: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_data.fasta
2018-02-12 01:16:48,627 ectyper.ectyper INFO     Using SEROTYPE_ALLELE_JSON: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_dict.json
2018-02-12 01:16:48,754 ectyper.ectyper INFO     Start blast alignment on database #1
2018-02-12 01:16:50,872 ectyper.ectyper INFO     Start serotype prediction for database #1
2018-02-12 01:16:50,872 ectyper.predictionFunctions INFO     Predicting serotype from blast output
2018-02-12 01:16:50,913 ectyper.predictionFunctions INFO     Serotype prediction completed
2018-02-12 01:16:50,921 ectyper.ectyper INFO     Outputs stored in /home/travis/build/phac-nml/ecoli_serotyping/output/test_folder_input
2018-02-12 01:16:50,921 ectyper.ectyper INFO     
Reporting result...
2018-02-12 01:16:50,926 ectyper.predictionFunctions INFO     
genome O_prediction              O_info H_prediction           H_info
 sample            -  No alignment found          H10  Alignment found
sample2         O148     Alignment found          H44  Alignment found
sample3         O148     Alignment found          H44  Alignment found
sample4         O148     Alignment found          H44  Alignment found
.2018-02-12 01:16:50,928 ectyper.ectyper INFO     Starting ectyper
Serotype prediction with input:
             Namespace(detailed=False, input=u'', output=None, percentIdentity=90, percentLength=50, species=False, verify=False)
             Log file is: /home/travis/build/phac-nml/ecoli_serotyping/ectyper.log
2018-02-12 01:16:50,928 ectyper.ectyper INFO     Temporary files and directory created
2018-02-12 01:16:50,929 ectyper.ectyper INFO     Gathering genome files
2018-02-12 01:16:50,929 ectyper.ectyper INFO     Removing invalid file types
2018-02-12 01:16:50,929 ectyper.ectyper INFO     Preparing genome files for blast alignment
2018-02-12 01:16:50,929 ectyper.ectyper INFO     0 final fasta files
2018-02-12 01:16:50,929 ectyper.ectyper INFO     No valid genome files. Terminating the program.
.usage: nosetests [-h] -i INPUT [-d PERCENTIDENTITY] [-l PERCENTLENGTH]
                 [--verify] [-s] [--detailed] [-o OUTPUT]
nosetests: error: argument -d/--percentIdentity: 999 is an invalid positive int percentage value
.2018-02-12 01:16:50,933 ectyper.ectyper INFO     Starting ectyper
Serotype prediction with input:
             Namespace(detailed=False, input=u'test/Data/test_dir/sample.fasta,test/Data/test_dir/sample.fasta', output=u'test_list_input', percentIdentity=90, percentLength=50, species=False, verify=False)
             Log file is: /home/travis/build/phac-nml/ecoli_serotyping/ectyper.log
2018-02-12 01:16:50,934 ectyper.ectyper INFO     Temporary files and directory created
2018-02-12 01:16:50,934 ectyper.ectyper INFO     Gathering genome files
2018-02-12 01:16:50,934 ectyper.genomeFunctions INFO     Using genomes in the input list
2018-02-12 01:16:50,934 ectyper.ectyper INFO     Removing invalid file types
2018-02-12 01:16:50,936 ectyper.ectyper INFO     Preparing genome files for blast alignment
2018-02-12 01:16:50,937 ectyper.ectyper INFO     2 final fasta files
2018-02-12 01:16:50,937 ectyper.ectyper INFO     Standardizing the genome headers
2018-02-12 01:16:50,938 ectyper.ectyper INFO     Start creating blast database #1
2018-02-12 01:16:50,938 ectyper.ectyper INFO     Using SEROTYPE_FILE: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_data.fasta
2018-02-12 01:16:50,939 ectyper.ectyper INFO     Using SEROTYPE_ALLELE_JSON: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_dict.json
2018-02-12 01:16:51,064 ectyper.ectyper INFO     Start blast alignment on database #1
2018-02-12 01:16:52,525 ectyper.ectyper INFO     Start serotype prediction for database #1
2018-02-12 01:16:52,525 ectyper.predictionFunctions INFO     Predicting serotype from blast output
2018-02-12 01:16:52,553 ectyper.predictionFunctions INFO     Serotype prediction completed
2018-02-12 01:16:52,562 ectyper.ectyper INFO     Outputs stored in /home/travis/build/phac-nml/ecoli_serotyping/output/test_list_input
2018-02-12 01:16:52,562 ectyper.ectyper INFO     
Reporting result...
2018-02-12 01:16:52,567 ectyper.predictionFunctions INFO     
genome O_prediction              O_info H_prediction           H_info
sample            -  No alignment found          H10  Alignment found
sample            -  No alignment found          H10  Alignment found
.2018-02-12 01:16:52,569 ectyper.ectyper INFO     Starting ectyper
Serotype prediction with input:
             Namespace(detailed=False, input=u'test/Data/Escherichia.fastq', output=u'test_valid_fastq_file', percentIdentity=90, percentLength=50, species=False, verify=False)
             Log file is: /home/travis/build/phac-nml/ecoli_serotyping/ectyper.log
2018-02-12 01:16:52,569 ectyper.ectyper INFO     Temporary files and directory created
2018-02-12 01:16:52,569 ectyper.ectyper INFO     Gathering genome files
2018-02-12 01:16:52,570 ectyper.genomeFunctions INFO     Using genomes in file test/Data/Escherichia.fastq
2018-02-12 01:16:52,570 ectyper.ectyper INFO     Removing invalid file types
2018-02-12 01:16:52,571 ectyper.ectyper INFO     Preparing genome files for blast alignment
E2018-02-12 01:16:56,271 ectyper.genomeFunctions INFO     Using genomes in the input list
.2018-02-12 01:16:56,271 ectyper.genomeFunctions WARNING  123 is not found
2018-02-12 01:16:56,272 ectyper.genomeFunctions WARNING  test/Data/test_dir/test_junk.txt is not a fasta/fastq file
2018-02-12 01:16:56,273 ectyper.genomeFunctions WARNING  Compressed file is not supported: test/Data/test_dir/sampletar
2018-02-12 01:16:56,273 ectyper.genomeFunctions WARNING  test/Data/test_dir/sample.fasta.zip is not a valid file
.2018-02-12 01:16:56,275 ectyper.ectyper INFO     Start creating blast database #1
2018-02-12 01:16:56,275 ectyper.ectyper INFO     Using SEROTYPE_FILE: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_data.fasta
2018-02-12 01:16:56,275 ectyper.ectyper INFO     Using SEROTYPE_ALLELE_JSON: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_dict.json
2018-02-12 01:16:56,394 ectyper.ectyper INFO     Start blast alignment on database #1
2018-02-12 01:16:57,566 ectyper.ectyper INFO     Start serotype prediction for database #1
2018-02-12 01:16:57,566 ectyper.predictionFunctions INFO     Predicting serotype from blast output
2018-02-12 01:16:57,593 ectyper.predictionFunctions INFO     Serotype prediction completed
.2018-02-12 01:16:57,605 ectyper.ectyper INFO     Start creating blast database #1
2018-02-12 01:16:57,605 ectyper.ectyper INFO     Using SEROTYPE_FILE: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_data.fasta
2018-02-12 01:16:57,605 ectyper.ectyper INFO     Using SEROTYPE_ALLELE_JSON: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_dict.json
2018-02-12 01:16:57,733 ectyper.ectyper INFO     Start blast alignment on database #1
2018-02-12 01:16:58,612 ectyper.ectyper INFO     Start serotype prediction for database #1
2018-02-12 01:16:58,612 ectyper.predictionFunctions INFO     Predicting serotype from blast output
2018-02-12 01:16:58,613 ectyper.predictionFunctions INFO     No hit found for this blast query
2018-02-12 01:16:58,632 ectyper.predictionFunctions INFO     Serotype prediction completed
.2018-02-12 01:16:58,650 ectyper.ectyper INFO     Start creating blast database #1
2018-02-12 01:16:58,650 ectyper.ectyper INFO     Using SEROTYPE_FILE: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_data.fasta
2018-02-12 01:16:58,650 ectyper.ectyper INFO     Using SEROTYPE_ALLELE_JSON: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_dict.json
2018-02-12 01:16:58,773 ectyper.ectyper INFO     Start blast alignment on database #1
2018-02-12 01:16:59,867 ectyper.ectyper INFO     Start serotype prediction for database #1
2018-02-12 01:16:59,867 ectyper.predictionFunctions INFO     Predicting serotype from blast output
2018-02-12 01:16:59,893 ectyper.predictionFunctions INFO     Serotype prediction completed
.2018-02-12 01:16:59,906 ectyper.ectyper INFO     Start creating blast database #1
2018-02-12 01:16:59,907 ectyper.ectyper INFO     Using SEROTYPE_FILE: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_data.fasta
2018-02-12 01:16:59,907 ectyper.ectyper INFO     Using SEROTYPE_ALLELE_JSON: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_dict.json
2018-02-12 01:17:00,034 ectyper.ectyper INFO     Start blast alignment on database #1
2018-02-12 01:17:01,593 ectyper.ectyper INFO     Start serotype prediction for database #1
2018-02-12 01:17:01,593 ectyper.predictionFunctions INFO     Predicting serotype from blast output
2018-02-12 01:17:01,623 ectyper.predictionFunctions INFO     Serotype prediction completed
.2018-02-12 01:17:01,635 ectyper.ectyper INFO     Start creating blast database #1
2018-02-12 01:17:01,636 ectyper.ectyper INFO     Using SEROTYPE_FILE: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_data.fasta
2018-02-12 01:17:01,636 ectyper.ectyper INFO     Using SEROTYPE_ALLELE_JSON: /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/ectyper_dict.json
2018-02-12 01:17:01,766 ectyper.ectyper INFO     Start blast alignment on database #1
2018-02-12 01:17:03,172 ectyper.ectyper INFO     Start serotype prediction for database #1
2018-02-12 01:17:03,172 ectyper.predictionFunctions INFO     Predicting serotype from blast output
2018-02-12 01:17:03,198 ectyper.predictionFunctions INFO     Serotype prediction completed
..2018-02-12 01:17:04,113 ectyper.speciesIdentification INFO     Salmonella.fasta is identified as an invalid e.coli genome file by marker approach
.
======================================================================
ERROR: test_valid_fastq_file (test.ectyper_test.TestEctyper)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/phac-nml/ecoli_serotyping/test/ectyper_test.py", line 58, in test_valid_fastq_file
    ectyper.run_program()
  File "/home/travis/build/phac-nml/ecoli_serotyping/ectyper/ectyper.py", line 69, in run_program
    raw_files_dict, temp_files, verify=args.verify, species=args.species
  File "/home/travis/build/phac-nml/ecoli_serotyping/ectyper/ectyper.py", line 265, in filter_for_ecoli_files
    ffile, f, temp_dir, verify=verify, species=species)
  File "/home/travis/build/phac-nml/ecoli_serotyping/ectyper/ectyper.py", line 294, in filter_file_by_species
    genomeFunctions.assemble_reads(genome_file, combined_file, temp_dir)
  File "/home/travis/build/phac-nml/ecoli_serotyping/ectyper/genomeFunctions.py", line 260, in assemble_reads
    return split_mapped_output(output)
  File "/home/travis/build/phac-nml/ecoli_serotyping/ectyper/genomeFunctions.py", line 284, in split_mapped_output
    SeqIO.write(identif_seqs, output_handle, "fasta")
  File "/home/travis/miniconda/envs/test-environment/lib/python2.7/site-packages/Bio/SeqIO/__init__.py", line 491, in write
    count = writer_class(fp).write_file(sequences)
  File "/home/travis/miniconda/envs/test-environment/lib/python2.7/site-packages/Bio/SeqIO/Interfaces.py", line 214, in write_file
    count = self.write_records(records)
  File "/home/travis/miniconda/envs/test-environment/lib/python2.7/site-packages/Bio/SeqIO/Interfaces.py", line 199, in write_records
    self.write_record(record)
  File "/home/travis/miniconda/envs/test-environment/lib/python2.7/site-packages/Bio/SeqIO/FastaIO.py", line 200, in write_record
    self.handle.write(">%s\n" % title)
TypeError: write() argument 1 must be unicode, not str
-------------------- >> begin captured logging << --------------------
ectyper.ectyper: INFO: Starting ectyper
Serotype prediction with input:
             Namespace(detailed=False, input=u'test/Data/Escherichia.fastq', output=u'test_valid_fastq_file', percentIdentity=90, percentLength=50, species=False, verify=False)
             Log file is: /home/travis/build/phac-nml/ecoli_serotyping/ectyper.log
ectyper.ectyper: DEBUG: Namespace(detailed=False, input=u'test/Data/Escherichia.fastq', output=u'test_valid_fastq_file', percentIdentity=90, percentLength=50, species=False, verify=False)
ectyper.ectyper: INFO: Temporary files and directory created
ectyper.ectyper: DEBUG: {u'output_file': u'/home/travis/build/phac-nml/ecoli_serotyping/output/test_valid_fastq_file/output.csv', u'assemble_temp_dir': u'/tmp/tmpGQSplF/assemblies', u'output_dir': u'/home/travis/build/phac-nml/ecoli_serotyping/output/test_valid_fastq_file', u'fasta_temp_dir': u'/tmp/tmpGQSplF/fastas'}
ectyper.ectyper: INFO: Gathering genome files
ectyper.genomeFunctions: INFO: Using genomes in file test/Data/Escherichia.fastq
ectyper.ectyper: DEBUG: [u'/home/travis/build/phac-nml/ecoli_serotyping/test/Data/Escherichia.fastq']
ectyper.ectyper: INFO: Removing invalid file types
ectyper.ectyper: DEBUG: raw fasta files: []
ectyper.ectyper: DEBUG: raw fastq files: [u'/home/travis/build/phac-nml/ecoli_serotyping/test/Data/Escherichia.fastq']
ectyper.ectyper: DEBUG: {u'fasta': [], u'fastq': [u'/home/travis/build/phac-nml/ecoli_serotyping/test/Data/Escherichia.fastq']}
ectyper.ectyper: INFO: Preparing genome files for blast alignment
ectyper.subprocess_util: DEBUG: Running: [u'bowtie2', u'--score-min L,1,-0.5', u'--np 5', u'-x', u'/home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/bowtie_index/combined', u'-U', u'/home/travis/build/phac-nml/ecoli_serotyping/test/Data/Escherichia.fastq', u'-S', u'/tmp/tmpGQSplF/assemblies/reads.sam']
ectyper.subprocess_util: DEBUG: Subprocess finished successfully in 0.640 sec.
ectyper.subprocess_util: DEBUG: Running: [u'samtools', u'view', u'-F 4', u'-q 1', u'-bS', u'/tmp/tmpGQSplF/assemblies/reads.sam', u'-o', u'/tmp/tmpGQSplF/assemblies/reads.bam']
ectyper.subprocess_util: DEBUG: Subprocess finished successfully in 0.206 sec.
ectyper.subprocess_util: DEBUG: Running: [u'samtools', u'sort', u'/tmp/tmpGQSplF/assemblies/reads.bam', u'-o', u'/tmp/tmpGQSplF/assemblies/reads.sorted.bam']
ectyper.subprocess_util: DEBUG: Subprocess finished successfully in 0.179 sec.
ectyper.subprocess_util: DEBUG: Running: samtools mpileup -uf /home/travis/build/phac-nml/ecoli_serotyping/ectyper/Data/combined.fasta /tmp/tmpGQSplF/assemblies/reads.sorted.bam | bcftools call -c | vcfutils.pl vcf2fq | seqtk seq -A - > /tmp/tmpGQSplF/assemblies/Escherichia.fasta
ectyper.subprocess_util: DEBUG: Subprocess finished successfully in 2.669 sec.
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
Ran 15 tests in 15.825s
FAILED (errors=1)
The command "nosetests" exited with 1.
Done. Your build exited with 1.

…en()

kevinkle · 2018-02-12T02:56:39Z

Nosetests passing on both py27 and py26 now.

kevinkle added 27 commits February 2, 2018 16:12

ADD: travis badge

35371f7

START: python2/3 support

51017c4

CHANGE: dont email me

19b2041

ADD: future deps

6688d55

FIX: python3/2 definition for os.getcwdu() and os.getcwd()

ced3832

FIX: was throwing an AttributeError not import

128fd13

START: tests passing for pasturize, now making conda build for python2/3

3cb657b

UPDATE: bump the version number

447afe8

FIX: noarch:python

e1ba92d

DEBUG: conda-build issues

4eb042c

DEBUG: conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfi…

b8a20f1

…able dependencies for platform linux-64

ADD: git branch

537bcc8

ADD: git branch

35de085

FIX: req future in conda too

9da129f

FIX: future

50d4aca

DEBUG: conda install doesnt take --channel-priority seriously, try pu…

65bf59c

…tting ectyper version above that on bioconda

DEBUG: looks like its still forcing python3

59ab8b9

CHANGE: unpin the req

35c033a

DEBUG: more py versioning

9dbaa7a

CHANGE: use local src

6cdbadf

DEBUG: build me a py27 ver conda!!!

bb7e369

DEBUG: build me a py27 ver conda!!!

cec8e49

DEBUG: more py27

2fd7131

START: can now start building py27, but getting compatibility probles…

c3cdc9a

…m w setup

CHANGE: looks like package_data isnt probably used in our case

e855c3c

UPDATE: pre-PR changes

6a3a600

Merge pull request #51 from phac-nml/kevinkle-patch-1

01ffe08

ADD: travis badge

kevinkle requested a review from chadlaing February 4, 2018 01:13

kevinkle added 2 commits February 3, 2018 20:14

ADD: readd version pinning for deps

7965483

Merge branch 'pasteurize' of https://github.com/phac-nml/ecoli_seroty…

d1fd1d1

…ping into pasteurize

kevinkle added 21 commits February 11, 2018 16:23

DEBUG: didnt convert?

6e33393

DEBUG: print types

f43424f

DEBUG: looks like still objects

05413c6

DEBUG: more unicode checking

fe14096

DEBUG: keys being read as ?

5a35f24

DEBUG: open

0bfd344

DEBUG: dont have pandas encode utf-8 on write

2cd8ef5

DEUBG: bypass storing the output_df for now

9b3da2e

DEBUG: dont use open from builtins

967b82f

DEBUG: rm encoding from open

b96e334

DEL: prints

94ac049

CHECK: writing csv ok now, but unsure if line conversion is effecting…

f41aa86

… predictions

FIX: convert the sframe value for py27 and ADD: checks for pandas fun…

fd0895b

…ctions

DEBUG: see if the get_prediction() is being effected by backport

d40a1ae

DEBUG: looks like there is a predictors_df being created which isnt e…

e5918d1

…mpty, where are we losing our data?

FIX: typo

860f0a1

DEBUG: something isnt working in the get_prediction for py27

49d81c5

DEBUG: pandas is performing as it should (there are valid predictors)…

8a2a623

…, so tell me why youre continuing

FIX: should be usuing equality comparison not object comparison

29b4cc5

DEL: stuff used for debugging

79cd169

CHANGE: dont try to upload

0f8cdf2

kevinkle reopened this Feb 12, 2018

FIX: set a global for read flags in predictionFunctions that differs …

dbf190c

…between Py27 and Py36

CHANGE: read_flags for genomeFunctions too and no longer force py3 op…

7594134

…en()

kevinkle self-assigned this Feb 12, 2018

kevinkle added 2 commits February 12, 2018 10:37

EDIT: add .bak files to .gitignore

05481ae

EDIT: cleanup

73cb919

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADD: cross-compatibility with either Python2 or Python3 #52

ADD: cross-compatibility with either Python2 or Python3 #52

kevinkle commented Feb 4, 2018

kevinkle commented Feb 12, 2018

kevinkle commented Feb 12, 2018

kevinkle commented Feb 12, 2018

ADD: cross-compatibility with either Python2 or Python3 #52

Are you sure you want to change the base?

ADD: cross-compatibility with either Python2 or Python3 #52

Conversation

kevinkle commented Feb 4, 2018

kevinkle commented Feb 12, 2018

kevinkle commented Feb 12, 2018

kevinkle commented Feb 12, 2018