Missing @is_case_sensitive in JSON (when set to true)

[marlontaylor]

@is_case_sensitive is missing from JSON when set to true in xml (see AddressObject example below)

input.xml

<stix:STIX_Package  xmlns:example_namespace="http://example_namespace"
xmlns:cyboxCommon="http://cybox.mitre.org/common-2" 
xmlns:cybox="http://cybox.mitre.org/cybox-2"
xmlns:cyboxVocabs="http://cybox.mitre.org/default_vocabularies-2"   
xmlns:AddressObj="http://cybox.mitre.org/objects#AddressObject-2"   
xmlns:stixCommon="http://stix.mitre.org/common-1"   
xmlns:stixVocabs="http://stix.mitre.org/default_vocabularies-1" 
xmlns:stix="http://stix.mitre.org/stix-1"   
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"   
xsi:schemaLocation="    http://cybox.mitre.org/common-2 http://cybox.mitre.org/XMLSchema/common/2.1/cybox_common.xsd    http://cybox.mitre.org/cybox-2 http://cybox.mitre.org/XMLSchema/core/2.1/cybox_core.xsd http://cybox.mitre.org/default_vocabularies-2 http://cybox.mitre.org/XMLSchema/default_vocabularies/2.1/cybox_default_vocabularies.xsd  http://cybox.mitre.org/objects#AddressObject-2 http://cybox.mitre.org/XMLSchema/objects/Address/2.1/Address_Object.xsd  http://stix.mitre.org/common-1 http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd http://stix.mitre.org/default_vocabularies-1 http://stix.mitre.org/XMLSchema/default_vocabularies/1.1.1/stix_default_vocabularies.xsd   http://stix.mitre.org/stix-1 http://stix.mitre.org/XMLSchema/core/1.1.1/stix_core.xsd" version="1.1.1">


  <stix:Observables cybox_major_version="2" cybox_minor_version="1" cybox_update_version="0">
    <cybox:Observable id="example_namespace:Observable-db99dc4d-47f6-4d45-ada3-a4fbab1e8b97">
      <cybox:Title>AddressObject</cybox:Title>
      <cybox:Description>blank</cybox:Description>
      <cybox:Object id="example_namespace:Object-092e0904-f8c7-407c-a5aa-cf38f2b950d3" />
    </cybox:Observable>
    <cybox:Observable id="example_namespace:Observable-2e70c7f0-9a90-4473-b8be-3225cfe0d829">
      <cybox:Object id="example_namespace:Object-80b676f9-0f0e-45c1-a088-2c3861acafbc">
        <cybox:Properties xsi:type="AddressObj:AddressObjectType" category="ipv4-addr">
          <AddressObj:Address_Value condition="DoesNotEqual" is_case_sensitive="true" apply_condition="ALL" delimiter="##example_namespace##">A##example_namespace##B##example_namespace##C</AddressObj:Address_Value>
        </cybox:Properties>
      </cybox:Object>
    </cybox:Observable>
    <cybox:Observable id="example_namespace:Observable-e1a3aba2-f46e-4c66-891b-6bb19a17956b">
      <cybox:Title>3 ip</cybox:Title>
      <cybox:Object id="example_namespace:Object-58adb7ed-fdcf-4bed-923c-d4044760b3d4">
        <cybox:Properties xsi:type="AddressObj:AddressObjectType" category="ipv4-addr">
          <AddressObj:Address_Value condition="Equals" is_case_sensitive="false" apply_condition="ANY" delimiter="##example_namespace##">1.1.1.1##example_namespace##2.2.2.2##example_namespace##3.3.3.3</AddressObj:Address_Value>
        </cybox:Properties>
      </cybox:Object>
    </cybox:Observable>
  </stix:Observables>

</stix:STIX_Package>

#!/usr/bin/env python
# Copyright (c) 2014, The MITRE Corporation. All rights reserved.
# See LICENSE.txt for complete terms.

'''
File: ex_01.py

Description: Round-trip example. This script takes a STIX instance document from XML to
a binding object, then to a api object and then to a dictionary. That dictionary is then
converted back into an api object, which is then used to generate an XML document.
'''

import io
from pprint import pprint

from stix.core import STIXPackage

def main():
   fn = 'input.xml'
   stix_package = STIXPackage.from_xml(fn)
   stix_dict = stix_package.to_dict() # parse to dictionary
   pprint(stix_dict)

   stix_package_two = STIXPackage.from_dict(stix_dict) # create python-stix object from dictionary
   xml = stix_package_two.to_xml() # generate xml from python-stix object
   print(xml)

if __name__ == '__main__':
   main()

output.json

{
   "observables":{
      "observables":[
         {
            "description":"blank",
            "object":{
               "id":"example_namespace:Object-092e0904-f8c7-407c-a5aa-cf38f2b950d3"
            },
            "id":"example_namespace:Observable-db99dc4d-47f6-4d45-ada3-a4fbab1e8b97",
            "title":"AddressObject"
         },
         {
            "object":{
               "id":"example_namespace:Object-80b676f9-0f0e-45c1-a088-2c3861acafbc",
               "properties":{
                  "category":"ipv4-addr",
                  "xsi:type":"AddressObjectType",
                  "address_value":{
                     "apply_condition":"ALL",
                     "delimiter":"##example_namespace##",
                     "condition":"DoesNotEqual",
                     "value":[
                        "1.1.1.1",
                        "2.2.2.2",
                        "3.3.3.3"
                     ]
                  }
               }
            },
            "id":"example_namespace:Observable-2e70c7f0-9a90-4473-b8be-3225cfe0d829"
         },
         {
            "object":{
               "id":"example_namespace:Object-58adb7ed-fdcf-4bed-923c-d4044760b3d4",
               "properties":{
                  "category":"ipv4-addr",
                  "xsi:type":"AddressObjectType",
                  "address_value":{
                     "apply_condition":"ANY",
                     "delimiter":"##example_namespace##",
                     "is_case_sensitive":false,
                     "condition":"Equals",
                     "value":[
                        "1.1.1.1",
                        "2.2.2.2",
                        "3.3.3.3"
                     ]
                  }
               }
            },
            "id":"example_namespace:Observable-e1a3aba2-f46e-4c66-891b-6bb19a17956b",
            "title":"3 ip"
         }
      ],
      "major_version":2,
      "update_version":0,
      "minor_version":1
   },
   "version":"1.1.1"
}

[gtback]

Hey, @marlontaylor:

Our goal with the JSON representation is to keep it as small as possible, so we leave out default values. The default value for is_case_sensitive is true, so we intentionally don't include it in JSON output.

Does this make sense? Do you think we should re-evaluate that decision?

[marlontaylor]

hi @gtback ,

Small JSON representation is usually(if not always) the best decision. Going with the approach of not including the defaults, I have 2 questions:

1. How does someone determine if is_case_sensitive(or any other value) is either included and set to the default or not included at all?
2. Is this the intent: If is_case_sensitive(or any other value with a default setting) is not included the API will include it with the default setting (a free upgrade/enhancement if you will)?

[gtback]

@marlontaylor: You generally can't determine whether the attribute was explicitly set to the default value or if it was not set (and has the default value by default). This is the intent of the default value. It is the value whether or not it was explicitly set.

I believe in most cases, if you try to "get" the value of an attribute, it will return the default value even if it wasn't explicitly set. If not, that's something we should fix.

Does that answer your question(s)?

[marlontaylor]

@gtback:

I would consider the uncertainty of determining whether an attribute was included (or not) an issue.

1. Since the Python object provides the defaults, by default, the JSON representation should do the same.
2. Since the inclusion of the default values might be different from the original document/object, developers should have the ability to get the JSON representation of the original document without modification.

I suggest adding a flag to to_dict() to determine whether to return either a 'defaulted' version or the 'original' version of the STIX object (maybe to_dict() with no parameters would return the 'original' version).

Choose the best flag structure/format/value. See an example below.

# defaulted implies the inclusion of all default values (possible upgrade)
# unmarked implies the unmodified original representation on the STIX object
flag =  'defaulted ' | 'unmarked' | {'output':'defaulted '} | {'output':'unmarked'} | etc. # choose the best format
pprint(stix_package.to_dict(flag))

I think this should handle the cases discussed so far.

[gtback]

@marlontaylor:

I don't know if I agree. Particularly because the JSON is supposed to be concise and not include unnecessary information, including the default information should not be required. Semantically, there is no difference between the default value being explicitly included and the value being omitted entirely (this is true even in the XML). Also, if you ask for the value of a field in Python, you should get the default value, regardless of whether if it was provided explicitly or if it was omitted. I don't think distinguishing between those two is particularly valuable; if it is, it might be better to just parse with LXML directly.

If we want to pass an argument to the to_dict function, it is probably safe to just use a boolean field. I've been reluctant to add things like this, since you end up passing a ton of arguments through to recursive function calls (meaning you have to modify the same code in a lot of different places). If we move to a class-based Parser/Serializer model, we could think about doing this, I suppose.