Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

applications crashes when reading shapefile #28

Open
james2432 opened this issue Sep 7, 2016 · 8 comments
Open

applications crashes when reading shapefile #28

james2432 opened this issue Sep 7, 2016 · 8 comments

Comments

@james2432
Copy link

james2432 commented Sep 7, 2016

james@debian:~/Desktop/test$ etl2osm "/home/james/Desktop/test/Tree-Inventory.shp" --config "/home/james/Desktop/test/lol.json" --outfile "/home/james/Desktop/test/out.osm"
ERROR:root:No Models Found - Verify or <FILE TYPE: .yml .json>.
Traceback (most recent call last):
File "/usr/local/bin/etl2osm", line 9, in
load_entry_point('etl2osm==0.0.3', 'console_scripts', 'etl2osm')()
File "/usr/local/lib/python2.7/dist-packages/click-5.1-py2.7.egg/click/core.py", line 700, in call
return self.main(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/click-5.1-py2.7.egg/click/core.py", line 680, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/dist-packages/click-5.1-py2.7.egg/click/core.py", line 873, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "/usr/local/lib/python2.7/dist-packages/click-5.1-py2.7.egg/click/core.py", line 508, in invoke
return callback(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/etl2osm-0.0.3-py2.7.egg/etl2osm/cli.py", line 18, in cli
etl2osm.process(infile, outfile, config=config, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/etl2osm-0.0.3-py2.7.egg/etl2osm/api.py", line 16, in process
data.transform(config, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/etl2osm-0.0.3-py2.7.egg/etl2osm/extract.py", line 193, in transform
self.properties = config_to_properties(config)
File "/usr/local/lib/python2.7/dist-packages/etl2osm-0.0.3-py2.7.egg/etl2osm/transform.py", line 43, in config_to_properties
for key, value in config.items():
AttributeError: 'NoneType' object has no attribute 'items'

@DenisCarriere
Copy link
Member

@james2432 Fixed some issues, it might not be related to this error, but might work now... I'm only supporting YAML at the moment.

Try parsing the address file using etl2osm to see if it works for you, I was able to parse it using this config:

http://opendata.peelregion.ca/data-categories/regional-geography/address-points.aspx

'addr:street':
    - model: suffix
      field: STREETTYPE
    - title: True
      field: STREETNAME
    - model: direction
      field: STREETDIRE
'addr:housenumber':
    int: True
    field: STREETNUM
'addr:unit':
    UNIT_IDENT
ref:
    ROPADRID
'source:ref':
    text: 'Region of Peel'
source:
    text: 'Region of Peel'

@james2432
Copy link
Author

would it just be:
etl2osm "/home/james/Desktop/test/Tree-Inventory.shp" --config "/home/james/Desktop/test/lol**.yml**" --outfile "/home/james/Desktop/test/out.osm"

that the input is a yml file?

@DenisCarriere
Copy link
Member

This is how you would use it.

$ etl2osm --config lol.yml --outfile out.osm Tree-Inventory.shp

You might still be able to provide a JSON, but it's probably broken.

@james2432
Copy link
Author

C:\TEMP\addresses>etl2osm --config config.yml --outfile out.osm AddressPoints.shp
ERROR:root:No Models Found - Verify or <FILE TYPE: .yml .json>.
ERROR:root:No Models Found - Verify or <FILE TYPE: .yml .json>.
Traceback (most recent call last):
File "C:\Python27\Scripts\etl2osm-script.py", line 9, in
load_entry_point('etl2osm==0.1.0', 'console_scripts', 'etl2osm')()
File "C:\Python27\lib\site-packages\click\core.py", line 700, in call
return self.main(_args, *_kwargs)
File "C:\Python27\lib\site-packages\click\core.py", line 680, in main
rv = self.invoke(ctx)
File "C:\Python27\lib\site-packages\click\core.py", line 873, in invoke
return ctx.invoke(self.callback, *_ctx.params)
File "C:\Python27\lib\site-packages\click\core.py", line 508, in invoke
return callback(_args, *_kwargs)
File "C:\Python27\lib\site-packages\etl2osm\cli.py", line 18, in cli
etl2osm.process(infile, outfile, config=config, *_kwargs)
File "C:\Python27\lib\site-packages\etl2osm\api.py", line 16, in process
data.transform(config, *_kwargs)
File "C:\Python27\lib\site-packages\etl2osm\extract.py", line 205, in transform
feature = transform_columns(feature, config, *_kwargs)
File "C:\Python27\lib\site-packages\etl2osm\transform.py", line 261, in transform_columns
feature['properties'] = transform_fields(feature['properties'], config, *_kwargs)
File "C:\Python27\lib\site-packages\etl2osm\transform.py", line 252, in transform_fields
value = clean_field(properties, item, *_kwargs)
File "C:\Python27\lib\site-packages\etl2osm\transform.py", line 223, in clean_field
value = titlecase_except(value)
File "C:\Python27\lib\site-packages\etl2osm\transform.py", line 182, in titlecase_except
if word in models['title_except']:
TypeError: argument of type 'NoneType' is not iterable

@DenisCarriere
Copy link
Member

Hum.... seems to not be able to find the models...

What version of Python?

Try setting up an virtualenv using Python 2.7+

I've had this code working on Windows before, so I know it's not the file path.

Is anything working?

Try using a simplier YML config

'addr:unit':
    UNIT_IDENT
ref:
    ROPADRID

@DenisCarriere
Copy link
Member

I will end up re-writing this entire app for NodeJS (Javascript). It's way too fragile and lots of possibilities for errors.

@james2432
Copy link
Author

Seems to be working with the new config it gave me an error that No Medels were found but is taking up 1GB of memory and 25% cpu(after 10 minutes).
Are the "-" infront of model supposed to designate something?

@DenisCarriere
Copy link
Member

The application has memory leak issues and can't process large files, another reason for the rewrite.

I modified the title case syntax, use:

name:
  - title: True
  - field: (source field name)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants