EmailMessageObj:Header/Sender causes error (in 1.1.1.1 and 1.1.1.2)

[marlontaylor]

input.xml

<?xml version="1.0" encoding="UTF-8" ?>
<stix:STIX_Package 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:stix="http://stix.mitre.org/stix-1" 
xmlns:stixCommon="http://stix.mitre.org/common-1" 
xmlns:cybox="http://cybox.mitre.org/cybox-2" 
xmlns:EmailMessageObj="http://cybox.mitre.org/objects#EmailMessageObject-2" 
xmlns:AddressObj="http://cybox.mitre.org/objects#AddressObject-2" 
xsi:schemaLocation="
http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.1.1/stix_core.xsd 
http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd 
http://cybox.mitre.org/cybox-2 http://cybox.mitre.org/XMLSchema/core/2.1/cybox_core.xsd 
http://cybox.mitre.org/objects#EmailMessageObject-2 http://cybox.mitre.org/XMLSchema/objects/Email_Message/2.1/Email_Message_Object.xsd 
http://cybox.mitre.org/objects#AddressObject-2 http://cybox.mitre.org/XMLSchema/objects/Address/2.1/Address_Object.xsd 
"
version="1.1.1" 
xmlns:example_namespace="example_namespace/">


    <stix:Observables cybox_major_version="2" cybox_minor_version="1" cybox_update_version="0">
    <cybox:Observable id="example_namespace:Observable-1d2e49f1-1fef-4dd6-b8d2-cdad9e538039">
      <cybox:Title>Email</cybox:Title>
      <cybox:Description>Email Description</cybox:Description>
      <cybox:Object id="example_namespace:Object-98cdb2c0-d8e8-4a5b-adee-8255af99a893">
        <cybox:Properties xsi:type="EmailMessageObj:EmailMessageObjectType">
          <EmailMessageObj:Header>
            <EmailMessageObj:Sender>
                <AddressObj:Address_Value condition="Equals" is_case_sensitive="false" apply_condition="ANY" delimiter="##example_namespace##">sender</AddressObj:Address_Value>
            </EmailMessageObj:Sender>
          </EmailMessageObj:Header>
        </cybox:Properties>
      </cybox:Object>
    </cybox:Observable>
  </stix:Observables>

</stix:STIX_Package>

script.py

#!/usr/bin/env python
# Copyright (c) 2014, The MITRE Corporation. All rights reserved.
# See LICENSE.txt for complete terms.
'''
Description: Round-trip example. This script takes a STIX instance document from XML to
a binding object, then to a api object and then to a dictionary. That dictionary is then
converted back into an api object, which is then used to generate an XML document.
'''
import io
from pprint import pprint
from stix.core import STIXPackage
from stix.indicator import Indicator
import stix.bindings.stix_core as stix_core_binding

def main():
    fn = 'input.xml'
    stix_package = STIXPackage.from_xml(fn)
    stix_dict = stix_package.to_dict() # parse to dictionary
    pprint(stix_dict)
    stix_package_two = STIXPackage.from_dict(stix_dict) # create python-stix object from dictionary
    xml = stix_package_two.to_xml() # generate xml from python-stix object
    print(xml)

if __name__ == '__main__':
    main()

error

Traceback (most recent call last):
  File "script.py", line 25, in <module>
    main()
  File "script.py", line 17, in main
    stix_package = STIXPackage.from_xml(fn)
  File "/usr/lib/python2.6/site-packages/stix/core/stix_package.py", line 431, in from_xml
    return parser.parse_xml(xml_file)
  File "/usr/lib/python2.6/site-packages/stix/utils/parser.py", line 185, in parse_xml
    stix_package = STIXPackage().from_obj(stix_package_obj)
  File "/usr/lib/python2.6/site-packages/stix/core/stix_package.py", line 395, in from_obj
    return_obj.observables = Observables.from_obj(obj.Observables)
  File "/usr/lib/python2.6/site-packages/cybox/core/observable.py", line 326, in from_obj
    obs.add(Observable.from_obj(o))
  File "/usr/lib/python2.6/site-packages/cybox/core/observable.py", line 212, in from_obj
    obs.object_ = Object.from_obj(observable_obj.Object)
  File "/usr/lib/python2.6/site-packages/cybox/core/object.py", line 130, in from_obj
    obj.properties = ObjectProperties.from_obj(object_obj.Properties)
  File "/usr/lib/python2.6/site-packages/cybox/common/object_properties.py", line 158, in from_obj
    defobj = klass.from_obj(defobj_obj)
  File "/usr/lib/python2.6/site-packages/cybox/common/object_properties.py", line 145, in from_obj
    return super(ObjectProperties, cls()).from_obj(defobj_obj)
  File "/usr/lib/python2.6/site-packages/cybox/__init__.py", line 194, in from_obj
    val = field.type_.from_obj(val)
  File "/usr/lib/python2.6/site-packages/cybox/__init__.py", line 195, in from_obj
    setattr(entity, field.attr_name, val)
  File "/usr/lib/python2.6/site-packages/cybox/__init__.py", line 586, in __set__
    value = self.type_(value)
  File "/usr/lib/python2.6/site-packages/cybox/objects/address_object.py", line 62, in __init__
    super(EmailAddress, self).__init__(addr_string, Address.CAT_EMAIL)
  File "/usr/lib/python2.6/site-packages/cybox/objects/address_object.py", line 40, in __init__
    self.address_value = address_value
  File "/usr/lib/python2.6/site-packages/cybox/__init__.py", line 586, in __set__
    value = self.type_(value)
  File "/usr/lib/python2.6/site-packages/cybox/common/properties.py", line 27, in __init__
    self.value = value
  File "/usr/lib/python2.6/site-packages/cybox/common/properties.py", line 68, in value
    self._value = self._parse_value(value_)
  File "/usr/lib/python2.6/site-packages/cybox/common/properties.py", line 349, in _parse_value
    raise ValueError("Cannot set String type to non-string value")
ValueError: Cannot set String type to non-string value

[ikiril01]

As best I can tell, this is due to some weirdness with the EmailAddress class being used in this field. It appears to be happening because the sender field is being set twice in from_obj of cybox.Entity, with the second instance being of type cybox.objects.Address.EmailAddress:

sender <type 'str'>
sender <class 'cybox.objects.address_object.EmailAddress'>

The following change (for https://github.com/CybOXProject/python-cybox/blob/master/cybox/__init__.py#L195) seemed to fix it, but @gtback should probably verify :)

if not getattr(entity, field.attr_name):
    setattr(entity, field.attr_name, val)

[marlontaylor]

I think reply_to had a similar error.

[ikiril01]

Yeah, any field that uses the EmailAddress class will have the same issue.

[gtback]

I can reproduce this. I'm not sure why it's happening, but I agree it's probably related to the EmailAddress class.

The point of the EmailAddress class is to ensure that you don't need to always set category='e-mail' in Python manually all the time.

I noticed that by adding category="e-mail" to the XML, the problem resolves itself. Even so, we still need to fix python-cybox to keep this from raising an error.

Address_Value should be of type String. We should be automatically "upgrading" from a str to a String, but not to an EmailAddress. I'm not sure if @ikiril01 's solution is the best one, or if it would have other unintended consequences. I'll take a deeper look.

[ikiril01]

@gtback speaking of unintended consequences, I just found a fairly significant one with my proposed approach in conjunction with the parsing of MAEC content with python-maec. Because most classes in python-maec that have ID properties set them by default in the constructor (example below), this would result in these auto-set properties not being properly populated with the parsed in value.

    def __init__(self, id = None, malware_instance_object_attributes = None):
        super(MalwareSubject, self).__init__()
        if id:
            self.id_ = id
        else:
            self.id_ = maec.utils.idgen.create_id(prefix="malware_subject")