How to desrialise some complex STIX json into objects

[KangoV]

I'm on my last part of writing a STIX parser, but am having difficulty with this last part. It involves a Map of cyber security extensions on a Stix File object, for example:

{
  "id": "bundle--6c9a1180-994e-4082-9a11-80994e308254",
  "type": "bundle",
  "objects": [
    {
      "type": "file",
      "spec_version": "2.1",
      "id": "file--9a1f834d-2506-5367-baec-7aa63996ac43",
      "name": "foo.zip",
      "hashes": {
        "SHA-256": "35a01331e9ad96f751278b891b6ea09699806faedfa237d40513d92ad1b7100f"
      },
      "mime_type": "application/zip",
      "extensions": {
        "archive-ext": {
          "contains_refs": [
            "file--019fde1c-94ca-5967-8b3c-a906a51d87ac",
            "file--94fc2163-dec3-5715-b824-6e689c4de865",
            "file--d07ff290-d7e0-545b-a2ff-04602a9e0b73"
          ]
        }
      }
    }
  ]
}

The problem here is the extensions node. This is a map where the type of the object is the key. In this case, the type is archive-ext. I am using immutables.org.io to generate the implementations:

The File object:

public interface File extends ScoObject {
    ...
    @JsonProperty("extensions")
    Map<String, ScoExtension> getExtensions();
    ...
}

The archive-ext extension:

@JsonTypeName("archive-ext")
public interface ArchiveFileExtension extends ScoExtension {
    @JsonProperty("type") String getType();
    @JsonProperty("contains_refs") Set<String> getContainsRefs();
    @JsonProperty("version") Optional<String> getVersion();
    @JsonProperty("comment") Optional<String> getComment();

The ScoExtension interface:

@JsonTypeInfo(
    use = JsonTypeInfo.Id.NAME,
    include = JsonTypeInfo.As.EXISTING_PROPERTY,
    property = "type",
    visible = true
//    defaultImpl = GenericCustomObject.class
)

public interface ScoExtension extends ScoExtensionCommonProperties, CustomProperties {
}

I'm really not sure if Jackson can handle this or if i should just go ahead and produce custom de/serializers. If you use a Map instead of ScoExtension as the value: Map<String, Map<String, Object>> it works fine, but this is missing the point, as it's not using the type name.

Any help on this would be greatly appreciated.

[KangoV]

Doh, forgot the error message:

Could not resolve subtype of [simple type, class io.kangov.stix.v21.core.sco.extension.ScoExtension]: missing type id property 'type' (for POJO property 'extensions')
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: io.kangov.stix.v21.core.sco.objects.File$Builder["extensions"]->java.util.LinkedHashMap["archive-ext"])
com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve subtype of [simple type, class io.kangov.stix.v21.core.sco.extension.ScoExtension]: missing type id property 'type' (for POJO property 'extensions')
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: io.kangov.stix.v21.core.sco.objects.File$Builder["extensions"]->java.util.LinkedHashMap["archive-ext"])
	at app//com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:43)
	at app//com.fasterxml.jackson.databind.DeserializationContext.missingTypeIdException(DeserializationContext.java:2094)
	at app//com.fasterxml.jackson.databind.DeserializationContext.handleMissingTypeId(DeserializationContext.java:1607)
	at app//com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleMissingTypeId(TypeDeserializerBase.java:307)
	at app//com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedUsingDefaultImpl(AsPropertyTypeDeserializer.java:211)
	at app//com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:145)
	at app//com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
	at app//com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:625)
	at app//com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:449)
	at app//com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32)
	at app//com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:158)
	at app//com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserializeFromObject(BuilderBasedDeserializer.java:338)
	at app//com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:240)
	at app//com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170)
	at app//com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136)
	at app//com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
	at app//com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:74)
	at app//com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at app//com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4801)
	at app//com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2974)
	at app//com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3438)
	at app//io.kangov.stix.parser.Reader.processNode(Reader.java:84)
	at app//io.kangov.stix.parser.Reader.processBundle(Reader.java:65)
	at app//io.kangov.stix.parser.Reader.read(Reader.java:46)
	at app//io.kangov.stix.parser.Parser.read(Parser.java:42)
	at app//io.kangov.stix.parser.$Parser$Definition$Intercepted.$$access$$read(Unknown Source)
	at app//io.kangov.stix.parser.$Parser$Definition$Exec.dispatch(Unknown Source)
	at app//io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:442)
	at app//io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:128)
	at app//io.micronaut.validation.ValidatingInterceptor.validateReturnMicronautValidator(ValidatingInterceptor.java:153)
	at app//io.micronaut.validation.ValidatingInterceptor.intercept(ValidatingInterceptor.java:139)
	at app//io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:137)

[pjfanning]

This is just my personal experience. If I can't get the Jackson (or some other serialization framework) to deserialize my data into my target object model - I flip the problem. I create some model instances in code and serialize them. I look at what the format of the output data looks like. It may not match the format that I expected. I can tweak the object model to try to get the serialized data to more closely match what I want. If you can get the serialization to work, then deserialization is more likely to work.

[KangoV]

I've had a go at the serialisation. The object model is used like this:

var bundle = Bundle.create(b -> b
        .addObject(File.createFile(f -> f
            .name("foo.zip")
            .putHash("SHA-256", "35a01331e9ad96f751278b891b6ea09699806faedfa237d40513d92ad1b7100f")
            .mimeType("application/zip")
            .putExtension("archive-ext", ArchiveFileExtension.create(a -> a
                .addContainsRefs(
                    "file--019fde1c-94ca-5967-8b3c-a906a51d87ac",
                    "file--94fc2163-dec3-5715-b824-6e689c4de865",
                    "file--d07ff290-d7e0-545b-a2ff-04602a9e0b73"
                )
            ))
        ))
    );

Which, when run through the ObjectMapper, produces:

{
  "type" : "bundle",
  "id" : "bundle--8a12c3b2-b5aa-4060-94b4-17f106e1dc8d",
  "spec_version" : "2.1",
  "objects" : [ {
    "type" : "file",
    "id" : "file--d595d147-8ec1-4c9e-a8ec-3f10f8fe69b7",
    "extensions" : {
      "archive-ext" : {
        "contains_refs" : [ 
          "file--019fde1c-94ca-5967-8b3c-a906a51d87ac", 
          "file--94fc2163-dec3-5715-b824-6e689c4de865", 
          "file--d07ff290-d7e0-545b-a2ff-04602a9e0b73" 
        ]
      }
    },
    "hashes" : {
      "SHA-256" : "35a01331e9ad96f751278b891b6ea09699806faedfa237d40513d92ad1b7100f"
    },
    "name" : "foo.zip",
    "mime_type" : "application/zip",
    "defanged" : false
  } ]
}

Which is a correct STIX bundle with a correct STIX File object. But the reverse fails.
This may need a custom deserialiser as Jackson cannot seem to handle it as is.

I am going to have a play around with the model to see if I can "massage" it a bit ;)

[cowtowncoder]

So, this:

include = JsonTypeInfo.As.EXISTING_PROPERTY,

won't work but I wonder if using instead

include = JsonTypeInfo.As.WRAPPER_OBJECT,

could help: it will add Type Id as a key of 1-entry Map with value serialization.
But it won't work for cases where polymorphic values themselves are in Map