Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue trying to deploy a new SR path on Juniper MX #16

Open
avisnevschi opened this issue Apr 19, 2019 · 28 comments
Open

Issue trying to deploy a new SR path on Juniper MX #16

avisnevschi opened this issue Apr 19, 2019 · 28 comments

Comments

@avisnevschi
Copy link

avisnevschi commented Apr 19, 2019

Hi, we are an ISP and we are thinking of migration from static RSVP tunnels to Segment Routing with an external controller. Our network is build mostly on Juniper MX devices, we cretead a lab with few routers and with an ODL Neon controller + Pathman-SR. Unfortunatelly we can't deploy any path.
So, what we have:
ODL Neon Controller
Pathman-SR
Juniper vMX devices running Junos 17.2R1.13

All the routers are seen in Pathman-SR, we have following topologies in our ODL controller:
example-linkstate-topology(all the routers/links are seen there)
example-pcep-topology(all the routers are connected to PCEP controller and session is UP, configured with Stateful, Active).
example-netconf-topology(the status is connected with all the routers)
I've attached a file, with the result of "/restconf/operational/network-topology:network-topology/"

When I try to deploy a path, I see the following error:
"Response indicated the error in frontend-backend communication."

This the error in karaf.log:
"2019-04-19T18:56:12,806 | INFO | remote-connector-processing-executor-19 | NetconfDevice | 407 - org.opendaylight.netconf.sal-netconf-connector - 1.9.0 | RemoteDevice{EQUINIX}: Netconf connector initialized successfully
2019-04-19T18:56:13,589 | WARN | epollEventLoopGroup-5-2 | Stateful07TopologySessionListener | 381 - org.opendaylight.bgpcep.pcep-topology-provider - 0.11.0 | Unhandled PCErr message PcerrMessage{_errors=[Errors{_errorObject=ErrorObject{_type=6, _value=10, _ignore=false, _processingRule=false, augmentation=[]}, augmentation=[]}], augmentation=[]}.
2019-04-19T18:56:13,590 | WARN | epollEventLoopGroup-5-2 | AbstractTopologySessionListener | 381 - org.opendaylight.bgpcep.pcep-topology-provider - 0.11.0 | Unhandled message Pcerr{_pcerrMessage=PcerrMessage{_errors=[Errors{_errorObject=ErrorObject{_type=6, _value=10, _ignore=false, _processingRule=false, augmentation=[]}, augmentation=[]}], augmentation=[]}, augmentation=[]} on session PCEPSessionImpl{channel=[id: 0x35c28cba, L:/10.10.1.134:4189 - R:/10.100.100.11:62749], localOpen=Open{_deadTimer=120, _keepalive=30, _sessionId=0, _tlvs=Tlvs{augmentation=[Tlvs1{_srPceCapability=SrPceCapability{_msd=0, augmentation=[]}}, Tlvs1{_stateful=Stateful{_lspUpdateCapability=true, augmentation=[Stateful1{_initiation=true}, Stateful1{_deltaLspSyncCapability=true, _includeDbVersion=true, _triggeredInitialSync=true, _triggeredResync=true}]}}]}, augmentation=[]}, remoteOpen=Open{_deadTimer=120, _keepalive=30, _sessionId=49, _tlvs=Tlvs{augmentation=[Tlvs1{_srPceCapability=SrPceCapability{_msd=5, augmentation=[]}}, Tlvs1{_stateful=Stateful{_lspUpdateCapability=true, augmentation=[Stateful1{_initiation=true}, Stateful1{}]}}, Tlvs3{}]}, _version=ProtocolVersion{_value=1}, _ignore=false, _processingRule=false, augmentation=[]}}"

This is the log in /tmp/pathman.log:
"6591 2019-04-19 15:56:13.511 UTC root:initialize INFO: Init 2 done - debug saved
6591 2019-04-19 15:56:13.512 UTC root:rest_interface_parser INFO: Commands Relieved: {u'path': [u'CORE0', u'EQUINIX'], u'option': u'create', u'name': u'CORE0 -> EQUINIX'}
6591 2019-04-19 15:56:13.512 UTC root:get_loop_list INFO: Path: [u'CORE0', u'EQUINIX']
6591 2019-04-19 15:56:13.512 UTC root:get_loop_list INFO: Loop list: [u'10.100.100.20']
6591 2019-04-19 15:56:13.512 UTC root:get_sid_list INFO: Path: [u'CORE0', u'EQUINIX']
6591 2019-04-19 15:56:13.513 UTC root:get_sid_list INFO: SID list: [2020]
6591 2019-04-19 15:56:13.592 UTC root:createSRtunnel INFO: Create SR Tunnel response: {u'output': {u'failure': u'failed', u'error': [{u'error-object': {u'ignore': True, u'processing-rule': True, u'type': 10, u'value': 5}}]}}
6591 2019-04-19 15:56:15.595 UTC root:post INFO: {"response": [{"cause": {"failure": "failed", "error": [{"error-object": {"ignore": true, "processing-rule": true, "type": 10, "value": 5}}]}, "option": "create", "success": false}]}
6591 2019-04-19 15:56:15.665 UTC root:initialize INFO: Init 2 done - debug saved
6591 2019-04-19 15:56:15.665 UTC root:rest_interface_parser INFO: Commands Relieved: {u'option': u'list_all'}
6591 2019-04-19 15:56:15.679 UTC root:get_url INFO: Url get Status: 200
6591 2019-04-19 15:56:15.679 UTC root:listAllLsp INFO: list: [], formatted: []
6591 2019-04-19 15:56:15.680 UTC root:post INFO: {"response": [{"list": [], "option": "list_all", "success": true}]}"

This is the error which I see on Juniper MX in the log file of PCEP controller:
"Apr 19 18:19:10 [4829] pcep_msg_parse: Received PCEP message type: PCCreate, version: 1 length: 80, flags: 0x0
Apr 19 18:19:10 [4829] pcep_obj_trace: SRP object: remove: 0 srp_id: 2 lsp_setup_type: 0
Apr 19 18:19:10 [4829] pcep_tlv_trace: TLV path setup setup: 1
Apr 19 18:19:10 [4829] pcep_obj_trace: LSP object: plsp ID: 0, remove: 0, operational: 0, last_update: 0, delegated: 1 administrative: 1 is_p2mp 0, PCE Created: 0
Apr 19 18:19:10 [4829] pcep_tlv_trace: TLV LSP Symbolic Name: 'EQUINIX -> CORE0'
Apr 19 18:19:10 [4829] pcep_obj_trace: END-POINTS IPv4 object: source: 10.100.100.20, destination: 10.100.100.11
Apr 19 18:19:10 [4829] pcep_obj_trace: ERO object: subobjects: 1
Apr 19 18:19:10 [4829] pcep_msg_parse: PCCreate message received:
Apr 19 18:19:10 [4829] pcep_msg_parse: ...P!........... 20 0C 00 50 21 12 00 14 00 00 00 00 00 00 00 02
Apr 19 18:19:10 [4829] pcep_msg_parse: ................ 00 1C 00 04 00 00 00 01 20 10 00 1C 00 00 00 09
Apr 19 18:19:10 [4829] pcep_msg_parse: ....EQUINIX.->.C 00 11 00 10 45 51 55 49 4E 49 58 20 2D 3E 20 43
Apr 19 18:19:10 [4829] pcep_msg_parse: ORE0.....dd..dd. 4F 52 45 30 04 12 00 0C 0A 64 64 14 0A 64 64 0B
Apr 19 18:19:10 [4829] pcep_msg_parse: .........}...dd. 07 10 00 10 05 0C 10 01 00 7D B0 00 0A 64 64 0B
Apr 19 18:19:10 [4829] pcep_msg_parse:
Apr 19 18:19:10 [4829] process_message: Deadtimer has been touched
Apr 19 18:19:10 [4829] pccd_pcep_onmessage_cb() -->
Apr 19 18:19:10 [4829] pccd_pcep_onmessage_cb(): onmessage objp: 0x886e000 objc: 4 msg_type:12 pce:'odl'
Apr 19 18:19:10 [4829] pccd_core_received_pccreate() -->
Apr 19 18:19:10 [4829] pccd_core_received_pccreate(): Copying LSP name: 'EQUINIX -> CORE0' from TLV
Apr 19 18:19:10 [4829] pccd_core_parse_lsp_instantion() -->
Apr 19 18:19:10 [4829] pccd_core_parse_spring_ero() -->
Apr 19 18:19:10 [4829] pcep_send_message: Sending PCEP message 'Error'
Apr 19 18:19:10 [4829] pcep_obj_trace: SRP object: remove: 48 srp_id: 2 lsp_setup_type: 48
Apr 19 18:19:10 [4829] pcep_obj_trace: ERROR object: type: 10, value: 5
Apr 19 18:19:10 [4829] pcep_obj_trace: TLV Missing Request ID: 808464432
Apr 19 18:19:10 [4829] pccd_core_parse_spring_ero() <--
Apr 19 18:19:10 [4829] pccd_core_parse_lsp_instantion() <--
Apr 19 18:19:10 [4829] pccd_pcep_onerror_cb() -->
Apr 19 18:19:10 [4829] pccd_pcep_onerror_cb(): onerror_cb pcep 0x880b060 ctx 0x882c000 error 0 (No error: 0)
Apr 19 18:19:10 [4829] pccd_pcep_onerror_cb(): Outgoing PCErr msg with TYPE:6 and VALUE:10
Apr 19 18:19:10 [4829] pccd_pcep_onerror_cb() <--
Apr 19 18:19:10 [4829] pcep_send_message: Sending PCEP message 'Error'
Apr 19 18:19:10 [4829] pcep_obj_trace: ERROR object: type: 6, value: 10
"

Can anybody advice what to do to solve the problem ?
Any help would be very appreciated!!!

operational-network-topology.txt

@avisnevschi
Copy link
Author

avisnevschi commented Apr 23, 2019

Is there anybody from guru's who can help me please ? @alwye @nikmon2 @gilesheron @chrismetz09

As I understood from debug of my ODL controller, the problem seems to be in loading yang schemes/modules... I've put all the schemes/modules into "karaf-0.8.0/cache/schema/" directory, and set the name accordingly:
-rw-r--r-- 1 root root 7988475 Apr 23 11:05 [email protected] - for Junos 17.1R8
-rw-r--r-- 1 root root 10686894 Apr 23 10:49 [email protected] - for Junos 17.2R13

But when they try to load, I got this error into debug log of my controller:
unsatisfied imports: {RevisionSourceIdentifier [name=configuration]=[ModuleImportImpl [name=junos-extension, revision=null, semanticVersion=null]]}
2019-04-23T12:50:18,623 | WARN | remote-connector-processing-executor-14 | NetconfDevice | 418 - org.opendaylight.netconf.sal-netconf-connector - 1.7.0 | RemoteDevice{EQUINIX}: Unable to build schema context, failed to resolve source RevisionSourceIdentifier [name=configuration@2017-01-01], will reattempt without it
org.opendaylight.yangtools.yang.model.repo.api.SchemaResolutionException: Failed to resolve required models, failed source: RevisionSourceIdentifier [name=configuration@2017-01-01], resolved sources: [], unsatisfied imports: {}
2019-04-23T12:28:31,378 | WARN | remote-connector-processing-executor-12 | SharedSchemaContextFactory | 337 - org.opendaylight.yangtools.yang-parser-impl - 2.0.1 | Source identifier mismatch for module "configuration", requested as RevisionSourceIdentifier [name=configuration@2015-09-11] but actually is RevisionSourceIdentifier [name=configuration]. Using actual id
2019-04-23T12:28:31,382 | WARN | remote-connector-processing-executor-12 | NetconfDevice | 418 - org.opendaylight.netconf.sal-netconf-connector - 1.7.0 | Unable to map identifier to a devices reported capability: RevisionSourceIdentifier [name=configuration] Available: [(http://xml.juniper.net/xnm/1.1/xnm?revision=2015-09-11)configuration]
2019-04-23T12:28:31,382 | WARN | remote-connector-processing-executor-12 | NetconfDevice | 418 - org.opendaylight.netconf.sal-netconf-connector - 1.7.0 | RemoteDevice{NXDATA}: Unable to build schema context, unsatisfied imports {RevisionSourceIdentifier [name=configuration]=[ModuleImportImpl [name=junos-extension, revision=null, semanticVersion=null]]}, will reattempt with resolved only
org.opendaylight.yangtools.yang.model.repo.api.SchemaResolutionException: Failed to resolve required models, failed source: null, resolved sources: [], unsatisfied imports: {RevisionSourceIdentifier [name=configuration]=[ModuleImportImpl [name=junos-extension, revision=null, semanticVersion=null]]}
configuration
http://yang.juniper.net/yang/1.1/jc/configuration/junos/17.2R1.13
show-auto-configuration
http://yang.juniper.net/yang/1.1/jrpc/show-auto-configuration/junos/17.2R1.13

@avisnevschi
Copy link
Author

When I switched to Junos 17.1R8, I get another error:

cat /tmp/pathman.log
16126 2019-04-23 12:27:48.427 UTC root:rest_interface_parser INFO: Commands Relieved: {u'path': [u'NXDATA', u'CORE1'], u'option': u'create', u'name': u'NXDATA -> CORE1'}
16126 2019-04-23 12:27:48.428 UTC root:get_loop_list INFO: Path: [u'NXDATA', u'CORE1']
16126 2019-04-23 12:27:48.428 UTC root:get_loop_list INFO: Loop list: [u'10.100.100.10']
16126 2019-04-23 12:27:48.428 UTC root:get_sid_list INFO: Path: [u'NXDATA', u'CORE1']
16126 2019-04-23 12:27:48.428 UTC root:get_sid_list INFO: SID list: [801010]
16126 2019-04-23 12:27:48.504 UTC root:createSRtunnel INFO: Create SR Tunnel response: {u'output': {u'failure': u'no-ack'}}
16126 2019-04-23 12:27:50.507 UTC root:post INFO: {"response": [{"cause": {"failure": "no-ack"}, "option": "create", "success": false}]}
16126 2019-04-23 12:27:50.531 UTC root:initialize INFO: Init 2 done - debug saved
16126 2019-04-23 12:27:50.531 UTC root:rest_interface_parser INFO: Commands Relieved: {u'option': u'list_all'}
16126 2019-04-23 12:27:50.546 UTC root:get_url INFO: Url get Status: 200
16126 2019-04-23 12:27:50.547 UTC root:listAllLsp INFO: list: [], formatted: []
16126 2019-04-23 12:27:50.547 UTC root:post INFO: {"response": [{"list": [], "option": "list_all", "success": true}]}

This are the logs from Juniper vMX router:
Apr 23 15:27:46 [4658] pcep_msg_parse: Received PCEP message type: PCCreate, version: 1 length: 80, flags: 0x0
Apr 23 15:27:46 [4658] pccd_pcep_onerror_cb() -->
Apr 23 15:27:46 [4658] pccd_pcep_onerror_cb(): onerror_cb pcep 0x880b060 ctx 0x882d000 error 0 (No error: 0)
Apr 23 15:27:46 [4658] pccd_pcep_onerror_cb(): Outgoing PCErr msg with TYPE:2 and VALUE:0
Apr 23 15:27:46 [4658] pccd_pcep_onerror_cb() <--
Apr 23 15:27:46 [4658] pcep_send_message: Sending PCEP message 'Error'
Apr 23 15:27:46 [4658] pcep_obj_trace: ERROR object: type: 2, value: 0
Apr 23 15:27:46 [4658] pccd_pcep_log_cb(): pcep_obj_parse(): Invalid SRP object (length=20)
Apr 23 15:27:46 [4658] pcep_msg_parse: Incorrect message received:
Apr 23 15:27:46 [4658] pcep_msg_parse: ...P!........... 20 0C 00 50 21 12 00 14 00 00 00 00 00 00 00 01
Apr 23 15:27:46 [4658] pcep_msg_parse: ................ 00 1C 00 04 00 00 00 01 20 10 00 1C 00 00 00 09
Apr 23 15:27:46 [4658] pcep_msg_parse: ....NXDATA.->.CO 00 11 00 0F 4E 58 44 41 54 41 20 2D 3E 20 43 4F
Apr 23 15:27:46 [4658] pcep_msg_parse: RE1......dd..dd. 52 45 31 00 04 12 00 0C 0A 64 64 16 0A 64 64 0A
Apr 23 15:27:46 [4658] pcep_msg_parse: .............dd. 07 10 00 10 05 0C 10 01 C3 8F 20 00 0A 64 64 0A
Apr 23 15:27:46 [4658] pcep_msg_parse:
Apr 23 15:27:46 [4658] pccd_pcep_log_cb(): pcep_msg_parse(): Could not parse all objects from message
Apr 23 15:27:46 [4658] pccd_pcep_log_cb(): connection_readable(): Unable to parse message
Apr 23 15:27:46 [4658] pccd_pcep_log_cb(): close_conn(): Closing connection with state 4, err: 22, close_msg: 3
Apr 23 15:27:46 [4658] pccd_pcep_onerror_cb() -->
Apr 23 15:27:46 [4658] pccd_pcep_onerror_cb(): onerror_cb pcep 0x880b060 ctx 0x882d000 error 22 (Invalid argument)

@nikmon2
Copy link
Contributor

nikmon2 commented Apr 23, 2019

I have not looked at ODL Neon at this time so it would be a little while before I have a setup to compare with.
But error messages logged in pathman's log are as received from ODL, and you can decode them by looking at PCEP error codes at https://tools.ietf.org/html/draft-ietf-pce-pce-initiated-lsp-04

Cheers,
Nik

@avisnevschi
Copy link
Author

avisnevschi commented Apr 23, 2019 via email

@avisnevschi
Copy link
Author

avisnevschi commented Apr 24, 2019

I think that the problem is that ODL is not impoting/resolving correctly yang modules which I put into /cache/schema/ directory, and because of this Pathman can't send/retrieve correct commands to/from vMX.

What is wrong ? I've put all the yang modules from https://github.com/Juniper/yang for specific version and put it into /cache/schema/ folder, then I added nodes to my ODL Controller's netconf_topology with:
{
"node": [
{
"node-id": "CORE0",
"netconf-node-topology:host": "10.100.100.11",
"netconf-node-topology:password": "netconf123!",
"netconf-node-topology:username": "netconf",
"netconf-node-topology:port": 830,
"netconf-node-topology:tcp-only": false,
"netconf-node-topology:keepalive-delay": 0
}
]
}

Now I see my nodes into netconf_topology but most of capabilities are unresolved:
{
"node": [
{
"node-id": "CORE0",
"netconf-node-topology:unavailable-capabilities": {
"unavailable-capability": [
{
"capability": "(http://yang.juniper.net/junos/conf/routing-instances?revision=2017-01-01)junos-conf-routing-instances",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/set?revision=2017-01-01)junos-rpc-set",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/conf/routing-options?revision=2017-01-01)junos-conf-routing-options",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/conf/system?revision=2017-01-01)junos-conf-system",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/ted?revision=2017-01-01)junos-rpc-ted",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/conf/interfaces?revision=2017-01-01)junos-conf-interfaces",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/rsvp?revision=2017-01-01)junos-rpc-rsvp",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/conf/protocols?revision=2017-01-01)junos-conf-protocols",
"failure-reason": "unable-to-resolve"
},
{
{
"capability": "(http://yang.juniper.net/junos/rpc/spring-traffic-engineering?revision=2017-01-01)junos-rpc-spring-traffic-engineering",
"failure-reason": "unable-to-resolve"

},
{
"capability": "(http://yang.juniper.net/junos/rpc/clear?revision=2017-01-01)junos-rpc-clear",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/conf/forwarding-options?revision=2017-01-01)junos-conf-forwarding-options",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/bfd?revision=2017-01-01)junos-rpc-bfd",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/conf/policy-options?revision=2017-01-01)junos-conf-policy-options",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/dynamic-profile?revision=2017-01-01)junos-rpc-dynamic-profile",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/ping?revision=2017-01-01)junos-rpc-ping",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/help?revision=2017-01-01)junos-rpc-help",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/ripng?revision=2017-01-01)junos-rpc-ripng",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/path-computation-client?revision=2017-01-01)junos-rpc-path-computation-client",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/conf/vmhost?revision=2017-01-01)junos-conf-vmhost",
"failure-reason": "unable-to-resolve"
},
{
"capability": "(http://yang.juniper.net/junos/rpc/telemetry-agentd?revision=2017-01-01)junos-rpc-telemetry-agentd",
"failure-reason": "unable-to-resolve"
}
]
},
"netconf-node-topology:available-capabilities": {
"available-capability": [
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:xml:ns:netconf:capability:candidate:1.0"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:netconf:capability:confirmed-commit:1.0"
},
{
"capability-origin": "device-advertised",
"capability": "http://xml.juniper.net/dmi/system/1.0"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:netconf:base:1.0"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:xml:ns:netconf:base:1.0"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:netconf:capability:url:1.0?protocol=http,ftp,file"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:netconf:capability:validate:1.0"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:netconf:capability:candidate:1.0"
},
{
"capability-origin": "device-advertised",
"capability": "http://xml.juniper.net/netconf/junos/1.0"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:xml:ns:netconf:capability:url:1.0?protocol=http,ftp,file"
},
{
"capability-origin": "device-advertised",
"capability": "urn:ietf:params:xml:ns:netconf:capability:validate:1.0"
},
{
"capability": "(http://yang.juniper.net/junos/common/types?revision=2017-01-01)junos-common-types"
},
{
"capability": "(http://yang.juniper.net/junos/conf/diameter?revision=2017-01-01)junos-conf-diameter"
}
]
},
}
]
}

Still, all the files are present:
[root@localhost ~]#ll karaf-0.8.0/cache/schema/ | grep junos-rpc-spring-traffic-engineering
-rw-r--r-- 1 root root 6803 Apr 24 11:25 [email protected]

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 24, 2019 via email

@avisnevschi
Copy link
Author

@gilesheron

I tried to DEBUG netconf and yangtools, but still, in the logs I get only:
ImmutableUnkeyedListEntryNode{nodeIdentifier=(urn:opendaylight:netconf-node-topology?revision=2015-01-14)unavailable-capability, value=[ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:netconf-node-topology?revision=2015-01-14)capability, value=(http://yang.juniper.net/junos/conf/protocols?revision=2017-01-01)junos-conf-protocols, attributes={}}, ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:netconf-node-topology?revision=2015-01-14)failure-reason, value=unable-to-resolve, attributes={}}], attributes={}},

But all the modules are present in /cache/schema :
[root@localhost ~]# ll karaf-0.8.0/cache/schema/ | grep junos-conf-protocols
-rw-r--r-- 1 root root 1338410 Apr 24 11:25 [email protected]

And to be honest I have no idea what else I can do... Already read a lot on google/forums, still don't know how to figure It out to resolve Junos yang modules/schema...

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 24, 2019 via email

@avisnevschi
Copy link
Author

avisnevschi commented Apr 24, 2019

@gilesheron
I made a debug on Yangtools and Netconf... Tried to grep for ERROR, still didn't find anything related to yang modules..

I attached all log files from the moment when I added devices to netconf-topology and until the status from ODL to my nodes changed to Connected.

Maybe you can find something that can help me to figure it out.

karaf_logs.zip

@avisnevschi
Copy link
Author

I finally managed to add nodes to topology_netconf without any erros.. I simply changed the JunOS version of my vMX routers to 17.4R1.16, and uploaded yang modules for this version. Now, I see no errors with my nodes in ODL Controller, all modules are added, and all capabilities are resolved.

I attached a file with a reply for my GET request "/restconf/operational/network-topology:network-topology/"

Still, now when I try to deploy a LSP, I still see this logs:

Juniper vMX Router pcep trace:
Apr 25 13:42:11 [6397] pcep_msg_parse: Received PCEP message type: PCCreate, version: 1 length: 92, flags: 0x0
Apr 25 13:42:11 [6397] pcep_obj_trace: SRP object: remove: 0 srp_id: 1 lsp_setup_type: 0
Apr 25 13:42:11 [6397] pcep_tlv_trace: TLV path setup setup: 1
Apr 25 13:42:11 [6397] pcep_obj_trace: LSP object: plsp ID: 0, remove: 0, operational: 0, last_update: 0, delegated: 1 administrative: 1 is_p2mp 0, PCE Created: 0
Apr 25 13:42:11 [6397] pcep_tlv_trace: TLV LSP Symbolic Name: 'CORE1 -> EQUINIX'
Apr 25 13:42:11 [6397] pcep_obj_trace: END-POINTS IPv4 object: source: 10.100.100.10, destination: 10.100.100.20
Apr 25 13:42:11 [6397] pcep_obj_trace: ERO object: subobjects: 2
Apr 25 13:42:11 [6397] pcep_msg_parse: PCCreate message received:
Apr 25 13:42:11 [6397] pcep_msg_parse: ...!........... 20 0C 00 5C 21 12 00 14 00 00 00 00 00 00 00 01
Apr 25 13:42:11 [6397] pcep_msg_parse: ................ 00 1C 00 04 00 00 00 01 20 10 00 1C 00 00 00 09
Apr 25 13:42:11 [6397] pcep_msg_parse: ....CORE1.->.EQU 00 11 00 10 43 4F 52 45 31 20 2D 3E 20 45 51 55
Apr 25 13:42:11 [6397] pcep_msg_parse: INIX.....dd..dd. 49 4E 49 58 04 12 00 0C 0A 64 64 0A 0A 64 64 14
Apr 25 13:42:11 [6397] pcep_msg_parse: .........}...dd. 07 10 00 1C 05 0C 10 01 00 7D B0 00 0A 64 64 0B
Apr 25 13:42:11 [6397] pcep_msg_parse: [email protected]. 05 0C 10 01 00 7E 40 00 0A 64 64 14
Apr 25 13:42:11 [6397] process_message: Deadtimer has been touched
Apr 25 13:42:11 [6397] pccd_pcep_onmessage_cb() -->
Apr 25 13:42:11 [6397] pccd_pcep_onmessage_cb(): onmessage objp: 0x84cf3c0 objc: 4 msg_type:12 pce:'odl'
Apr 25 13:42:11 [6397] pccd_core_received_pccreate() -->
Apr 25 13:42:11 [6397] pcep_send_message: Sending PCEP message 'Error'
Apr 25 13:42:11 [6397] pcep_obj_trace: SRP object: remove: 48 srp_id: 1 lsp_setup_type: 48
Apr 25 13:42:11 [6397] pcep_obj_trace: ERROR object: type: 10, value: 3
Apr 25 13:42:11 [6397] pcep_obj_trace: TLV Missing Request ID: 808464432
Apr 25 13:42:11 [6397] pccd_core_received_pccreate() <--
Apr 25 13:42:11 [6397] pccd_pcep_onmessage_cb(): Handling of PCCreate failed
Apr 25 13:42:11 [6397] pccd_pcep_onmessage_cb() <--

ODL Nitrogen Controller:
2019-04-25T14:02:54,505 | INFO | epollEventLoopGroup-11-21 | PCEPSessionImpl | 383 - org.opendaylight.bgpcep.pcep-impl - 0.9.0 | Closing PCEP session: PCEPSessionImpl{channel=[id: 0x3ba13f73, L:/10.10.1.134:4189 - R:/10.100.100.10:55337], localOpen=Open [_deadTimer=120, _keepalive=30, _sessionId=2, _tlvs=Tlvs [augmentation=[Tlvs1 [_srPceCapability=SrPceCapability [_msd=0, augmentation=[]]], Tlvs1 [_stateful=Stateful [_lspUpdateCapability=true, augmentation=[Stateful1 [_initiation=true], Stateful1 [_deltaLspSyncCapability=true, _includeDbVersion=true, _triggeredInitialSync=true, _triggeredResync=true]]]]]], augmentation=[]], remoteOpen=Open [_deadTimer=120, _keepalive=30, _sessionId=1, _tlvs=Tlvs [augmentation=[Tlvs1 [_stateful=Stateful [_lspUpdateCapability=true, augmentation=[Stateful1 [_initiation=true], Stateful1 []]]], Tlvs3 [], Tlvs1 []]], _version=ProtocolVersion [_value=1], _ignore=false, _processingRule=false, augmentation=[]]}
2019-04-25T14:02:54,505 | INFO | epollEventLoopGroup-11-21 | PCEPSessionImpl | 383 - org.opendaylight.bgpcep.pcep-impl - 0.9.0 | Closing PCEP session channel: [id: 0x3ba13f73, L:/10.10.1.134:4189 - R:/10.100.100.10:55337]
2019-04-25T14:02:54,506 | INFO | epollEventLoopGroup-11-21 | AbstractTopologySessionListener | 387 - org.opendaylight.bgpcep.pcep-topology-provider - 0.9.0 | Session PCEPSessionImpl{channel=[id: 0x3ba13f73, L:/10.10.1.134:4189 ! R:/10.100.100.10:55337], localOpen=Open [_deadTimer=120, _keepalive=30, _sessionId=2, _tlvs=Tlvs [augmentation=[Tlvs1 [_srPceCapability=SrPceCapability [_msd=0, augmentation=[]]], Tlvs1 [_stateful=Stateful [_lspUpdateCapability=true, augmentation=[Stateful1 [_initiation=true], Stateful1 [_deltaLspSyncCapability=true, _includeDbVersion=true, _triggeredInitialSync=true, _triggeredResync=true]]]]]], augmentation=[]], remoteOpen=Open [_deadTimer=120, _keepalive=30, _sessionId=1, _tlvs=Tlvs [augmentation=[Tlvs1 [_stateful=Stateful [_lspUpdateCapability=true, augmentation=[Stateful1 [_initiation=true], Stateful1 []]]], Tlvs3 [], Tlvs1 []]], _version=ProtocolVersion [_value=1], _ignore=false, _processingRule=false, augmentation=[]]} terminated by peer with reason PCEPCloseTermination{reason=UNKNOWN}
2019-04-25T14:02:54,517 | INFO | epollEventLoopGroup-11-22 | PCEPSessionNegotiator | 383 - org.opendaylight.bgpcep.pcep-impl - 0.9.0 | Replacing bootstrap negotiator for channel [id: 0x95caddb9, L:/10.10.1.134:4189 - R:/10.100.100.10:52328]
2019-04-25T14:02:54,518 | INFO | epollEventLoopGroup-11-22 | AbstractPCEPSessionNegotiator | 383 - org.opendaylight.bgpcep.pcep-impl - 0.9.0 | PCEP session with [id: 0x95caddb9, L:/10.10.1.134:4189 - R:/10.100.100.10:52328] started, sent proposal Open [_deadTimer=120, _keepalive=30, _sessionId=3, _tlvs=Tlvs [augmentation=[Tlvs1 [_srPceCapability=SrPceCapability [_msd=0, augmentation=[]]], Tlvs1 [_stateful=Stateful [_lspUpdateCapability=true, augmentation=[Stateful1 [_initiation=true], Stateful1 [_deltaLspSyncCapability=true, _includeDbVersion=true, _triggeredInitialSync=true, _triggeredResync=true]]]]]], augmentation=[]]
2019-04-25T14:02:54,536 | INFO | epollEventLoopGroup-11-22 | AbstractPCEPSessionNegotiator | 383 - org.opendaylight.bgpcep.pcep-impl - 0.9.0 | PCEP peer [id: 0x95caddb9, L:/10.10.1.134:4189 - R:/10.100.100.10:52328] completed negotiation
2019-04-25T14:02:54,536 | INFO | epollEventLoopGroup-11-22 | PCEPSessionImpl | 383 - org.opendaylight.bgpcep.pcep-impl - 0.9.0 | Session /10.10.1.134:4189[3] <-> /10.100.100.10:52328[2] started
2019-04-25T14:02:54,538 | INFO | epollEventLoopGroup-11-22 | AbstractTopologySessionListener | 387 - org.opendaylight.bgpcep.pcep-topology-provider - 0.9.0 | Session with /10.100.100.10 attached to topology node KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=pcep-topology]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=pcc://10.100.100.10]]]]}

Pathman logs:
27200 2019-04-25 11:03:22.052 UTC root:rest_interface_parser INFO: Commands Relieved: {u'path': [u'CORE1', u'CORE0', u'EQUINIX'], u'option': u'create', u'name': u'CORE1 -> EQUINIX'}
27200 2019-04-25 11:03:22.053 UTC root:get_loop_list INFO: Path: [u'CORE1', u'CORE0', u'EQUINIX']
27200 2019-04-25 11:03:22.053 UTC root:get_loop_list INFO: Loop list: [u'10.100.100.11', u'10.100.100.20']
27200 2019-04-25 11:03:22.053 UTC root:get_sid_list INFO: Path: [u'CORE1', u'CORE0', u'EQUINIX']
27200 2019-04-25 11:03:22.053 UTC root:get_sid_list INFO: SID list: [2011, 2020]
27200 2019-04-25 11:03:22.106 UTC root:createSRtunnel INFO: Create SR Tunnel response: {u'output': {u'failure': u'failed', u'error': [{u'error-object': {u'ignore': True, u'processing-rule': True, u'type': 10, u'value': 5}}]}}
27200 2019-04-25 11:03:24.108 UTC root:post INFO: {"response": [{"cause": {"failure": "failed", "error": [{"error-object": {"ignore": true, "processing-rule": true, "type": 10, "value": 5}}]}, "option": "create", "success": false}]}
27200 2019-04-25 11:03:24.128 UTC root:initialize INFO: Init 2 done - debug saved
27200 2019-04-25 11:03:24.129 UTC root:rest_interface_parser INFO: Commands Relieved: {u'option': u'list_all'}
27200 2019-04-25 11:03:24.140 UTC root:get_url INFO: Url get Status: 200
27200 2019-04-25 11:03:24.140 UTC root:listAllLsp INFO: list: [], formatted: []
27200 2019-04-25 11:03:24.141 UTC root:post INFO: {"response": [{"list": [], "option": "list_all", "success": true}]}

So, all the time, I get the PCEP Error-type:10 with value:5, which means:
Error-Type:10 -> Reception of an invalid object
Value:5 -> ERO mixes SR-ERO subobjects with other subobject types

I am not sure... but maybe I have to add some other yang modules/schema to ODL Controller so that it can convert Segment Routing LSP configuration provided from Pathman-SR to JunOS language ?
As I understand, the SR-ERO provided from Pathman-SR is not recongized or is not read correctly by JunOS...

Any ideas ?

operational-network-topologynetwork-topology.txt

@avisnevschi
Copy link
Author

I think I found the problem.. It is because I have to change iana-sr-subobject as it is written in https://tools.ietf.org/html/draft-ietf-pce-segment-routing-06

In Open Day Light documentation, I see:

PCEP Segment Routing
Conforms draft-ietf-pce-segment-routing-01 - PCEP extension for Segment Routing,

The default configuration file is located in etc/opendaylight/karaf.

33-pcep-segment-routing.xml
MIGHT NOT BE NEEDED: you might need a switch from sub-object EXPLICIT_ROUTE and ROUTE_RECORD code point 5/6 to code point 36 defined by draft https://tools.ietf.org/html/draft-ietf-pce-segment-routing-06. Switch the field iana-sr-subobjects-type to true if your router supports code point 36.

prefix:pcep-parser-segment-routing pcep-parser-segment-routing true--> 4: iana-sr-subobjects-type - IANA is requested to allocate code points (36) in the RSVP Parameters registry for SR-ERO and SR-RRO. To use IANA assigned type set the value to true, to preserve code points 5/6, set value bellow to false or remove (false by default);

But unfortunatelly I don't have 33-pcep-segment-routing.xml in my karaf directory and I cant change it via REST API as well.. When I PUT
"URL: /restconf/config/pcep-segment-routing-app-config:pcep-segment-routing-app-config"
"Body":

true

It returns me:
{
"errors": {
"error": [
{
"error-type": "protocol",
"error-tag": "invalid-value",
"error-message": "URI has bad format. Possible reasons:\n 1. "pcep-segment-routing-app-config:pcep-segment-routing-app-config" was not found in parent data node.\n 2. "pcep-segment-routing-app-config:pcep-segment-routing-app-config" is behind mount point. Then it should be in format "/yang-ext:mount/pcep-segment-routing-app-config:pcep-segment-routing-app-config"."
}
]
}
}

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 29, 2019 via email

@avisnevschi
Copy link
Author

avisnevschi commented Apr 29, 2019

I've managed to change iana-sr-subobject as it is written in https://tools.ietf.org/html/draft-ietf-pce-segment-routing-06 with the help of DLUX interface and yang modules there...
Still, now I get another error.. When I try to create a simple RSVP tunnel, It works, but BGP Pathman doesn't work :). If I try to create Segment Routing LSP, Pathman-SR works well, but I got this error:
Error-Type: 24
Error-Value: 1 or Error-Value:2.

This is how I try to create the LSP dirrectly from ODL Controller:

<input xmlns="urn:opendaylight:params:xml:ns:yang:topology:pcep">
   <node>pcc://10.100.100.11</node>
   <name>TEST-LSP</name>
   <arguments>
      <lsp xmlns="urn:opendaylight:params:xml:ns:yang:pcep:ietf:stateful">
         <delegate>true</delegate>
         <administrative>true</administrative>
      </lsp>
      <endpoints-obj>
         <ipv4>
            <source-ipv4-address>10.100.100.11</source-ipv4-address>
            <destination-ipv4-address>10.100.100.22</destination-ipv4-address>
         </ipv4>
          <processing-rule>true</processing-rule>
      </endpoints-obj>
      <path-setup-type xmlns="urn:opendaylight:params:xml:ns:yang:pcep:ietf:stateful">
         <pst>1</pst>
      </path-setup-type>
      <ero>
         <subobject>
            <loose>false</loose>
            <sid-type xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">ipv4-node-id</sid-type>
            <m-flag xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">true</m-flag>
            <sid xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">2022</sid>
            <ip-address xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">37.233.0.14</ip-address>
        </subobject>
      </ero>
   </arguments>
   <network-topology-ref xmlns:topo="urn:TBD:params:xml:ns:yang:network-topology">/topo:network-topology/topo:topology[topo:topology-id="pcep-topology"]</network-topology-ref>
</input>

And I get this error:

{
    "output": {
        "failure": "failed",
        "error": [
            {
                "error-object": {
                    "type": 23,
                    "value": 1,
                    "ignore": true,
                    "processing-rule": true
                }
            }
        ]
    }
}

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 29, 2019 via email

@avisnevschi
Copy link
Author

avisnevschi commented Apr 29, 2019 via email

@avisnevschi
Copy link
Author

avisnevschi commented Apr 29, 2019

I tried now once again, and with such a request:

<input xmlns="urn:opendaylight:params:xml:ns:yang:topology:pcep">
   <node>pcc://10.100.100.11</node>
   <name>TEST-LSP1</name>
   <arguments>
      <lsp xmlns="urn:opendaylight:params:xml:ns:yang:pcep:ietf:stateful">
         <delegate>true</delegate>
         <administrative>true</administrative>
      </lsp>
      <endpoints-obj>
         <ipv4>
            <source-ipv4-address>10.100.100.11</source-ipv4-address>
            <destination-ipv4-address>10.100.100.22</destination-ipv4-address>
         </ipv4>
          <processing-rule>true</processing-rule>
      </endpoints-obj>
      <path-setup-type xmlns="urn:opendaylight:params:xml:ns:yang:pcep:ietf:stateful">
         <pst>1</pst>
      </path-setup-type>
      <ero>
         <subobject>
            <loose>false</loose>
            <sid-type xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">ipv4-node-id</sid-type>
            <m-flag xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">true</m-flag>
            <sid xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">50011</sid>
            <ip-address xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">37.233.0.14</ip-address>
        </subobject>
      </ero>
   </arguments>
   <network-topology-ref xmlns:topo="urn:TBD:params:xml:ns:yang:network-topology">/topo:network-topology/topo:topology[topo:topology-id="pcep-topology"]</network-topology-ref>
</input>

It returns me this error:

{
    "output": {
        "failure": "failed",
        "error": [
            {
                "error-object": {
                    "type": 24,
                    "value": 2,
                    "ignore": true,
                    "processing-rule": true
                }
            }
        ]
    }
}

Then, if I send the same request once again, I get the Error-Type 23 and Error-Value:1.

The logs from the router:

Apr 29 12:53:00 [6833] pcep_msg_parse: Received PCEP message type: PCCreate, version: 1 length: 76, flags: 0x0
Apr 29 12:53:00 [6833] pcep_obj_trace: SRP object: remove: 0 srp_id: 1 lsp_setup_type: 0
Apr 29 12:53:00 [6833] pcep_tlv_trace: TLV path setup setup: 1
Apr 29 12:53:00 [6833] pcep_obj_trace: LSP object: plsp ID: 0, remove: 0, operational: 0, last_update: 0, delegated: 1 administrative: 1 is_p2mp 0, PCE Created: 0
Apr 29 12:53:00 [6833] pcep_tlv_trace: TLV LSP Symbolic Name: 'TEST-LSP1'
Apr 29 12:53:00 [6833] pcep_obj_trace: END-POINTS IPv4 object: source: 10.100.100.11, destination: 10.100.100.22
Apr 29 12:53:00 [6833] pcep_obj_trace: ERO object: subobjects: 1
Apr 29 12:53:00 [6833] pcep_obj_trace_ero: SPRING SUB-OBJECT: loose: 0, sid_type: 1, sid: 50011
Apr 29 12:53:00 [6833] pcep_msg_parse: PCCreate message received:
Apr 29 12:53:00 [6833] pcep_msg_parse: ...L!...........  20 0C 00 4C 21 12 00 14 00 00 00 00 00 00 00 01
Apr 29 12:53:00 [6833] pcep_msg_parse: ................  00 1C 00 04 00 00 00 01 20 10 00 18 00 00 00 09
Apr 29 12:53:00 [6833] pcep_msg_parse: ....TEST-LSP1...  00 11 00 09 54 45 53 54 2D 4C 53 50 31 00 00 00
Apr 29 12:53:00 [6833] pcep_msg_parse: .....dd..dd.....  04 12 00 0C 0A 64 64 0B 0A 64 64 16 07 10 00 10
Apr 29 12:53:00 [6833] pcep_msg_parse: $....5..%...      24 0C 10 01 0C 35 B0 00 25 E9 00 0E
Apr 29 12:53:00 [6833] process_message: Deadtimer has been touched
Apr 29 12:53:00 [6833] pccd_pcep_onmessage_cb() -->
Apr 29 12:53:00 [6833] pccd_pcep_onmessage_cb(): onmessage objp: 0x8425140 objc: 4 msg_type:12 pce:'odl'
Apr 29 12:53:00 [6833] pccd_core_received_pccreate() -->
Apr 29 12:53:00 [6833] pccd_core_received_pccreate(): Copying LSP name: 'TEST-LSP1' from TLV
Apr 29 12:53:00 [6833] pccd_core_parse_lsp_instantion() -->
Apr 29 12:53:00 [6833] pccd_core_parse_spring_ero() -->
Apr 29 12:53:00 [6833] pccd_core_parse_spring_ero(): Added SID lebel-20 is: 50011
Apr 29 12:53:00 [6833] pccd_core_print_sr_ero_info(): Number of SR-ERO subobjects: 1
Apr 29 12:53:00 [6833] pccd_core_print_sr_ero_info(): SR-ERO loose_type: 0
Apr 29 12:53:00 [6833] pccd_core_print_sr_ero_info(): SR-ERO SID lebel type: 2
Apr 29 12:53:00 [6833] pccd_core_print_sr_ero_info(): SR-ERO sid value: 1539506176
Apr 29 12:53:00 [6833] pccd_core_print_sr_ero_info(): SR-ERO NAI type: 1
Apr 29 12:53:00 [6833] pccd_core_print_sr_ero_info(): SR-ERO IPv4-Node addr for subobject 0: 37.233.0.14
Apr 29 12:53:00 [6833] pccd_core_parse_spring_ero() <--
Apr 29 12:53:00 [6833] pccd_core_parse_lsp_instantion() <--
Apr 29 12:53:00 [6833] pccd_core_provisioned_lsps_increase_counter(): Provisioned LSPs counter increased: 1
Apr 29 12:53:00 [6833] pccd_core_received_pccreate(): Creating LSP name: 'TEST-LSP1'
Apr 29 12:53:00 [6833] pccd_core_received_pccreate(): Got LSP name: 'TEST-LSP1', Path name: '(null)' and priority: '(null)'
Apr 29 12:53:00 [6833] pccd_rpd_pccreate() -->
Apr 29 12:53:00 [6833] pccd_rpd_pccreate(): sending create request to RPD for LSP: 'TEST-LSP1'
Apr 29 12:53:00 [6833] pccd_rpd_debug_lsp_info(): tx pccreate name(TEST-LSP1) path() template() src(10.100.100.11) dst(10.100.100.11) msg_trigger() type(3) state(4) control(0) flags(0x0) admin grp exclude(0) admin grp include any(0) admin grp include all(0) setup prio(0) hold prio(0) bw(0bps) metric(0)
Apr 29 12:53:00 [6833] pccd_rpd_debug_lsp_info():     lsp_id(0) tunnel_id(0) setup_type(1) ero(0x0) ero_len(0) rro(0x0) rro_len(0) sr-ero(0x84a10f0), sr-ero_len(0) sr-rro(0x0) sr-rro_len(0) err_spec(0x0) err_spec_len(0) #assoc(0)
Apr 29 12:53:00 [6833] pccd_rpd_print_sr_ero_info(): Number of SR-ERO subobjects: 1
Apr 29 12:53:00 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO loose_type: 0
Apr 29 12:53:00 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO SID level type: 2
Apr 29 12:53:00 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO sid value: 1539506176
Apr 29 12:53:00 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO NAI type: 1
Apr 29 12:53:00 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO IPv4-Node addr for subobject 0: 37.233.0.14
Apr 29 12:53:00 [6833] pccd_rpd_debug_lsp_info():     path_flags(0) new AuBw(0bps) #paths(0) assoc()local_cspf(0)
Apr 29 12:53:00 [6833] pccd_rpd_pcupdate_send() -->
Apr 29 12:53:00 [6833] pccd_rpd_encode_pcupdate() -->
Apr 29 12:53:00 [6833] pccd_rpd_encode_pcupdate(): lsp_params->setup_type: 1
Apr 29 12:53:00 [6833] pccd_rpd_encode_pcupdate() <--
Apr 29 12:53:00 [6833] pccd_rpd_pcupdate_send() <--
Apr 29 12:53:00 [6833] pccd_rpd_store_new_lsp() -->
Apr 29 12:53:00 [6833] pccd_rpd_reconfig_lsp(): LSP 0x847d000, template name
Apr 29 12:53:00 [6833] pccd_rpd_store_new_lsp(): Stored new LSP as :TEST-LSP1 for LSP TEST-LSP1  with path '' id 4
Apr 29 12:53:00 [6833] pccd_rpd_store_new_lsp() <--
Apr 29 12:53:00 [6833] pccd_rpd_pccreate() <--
Apr 29 12:53:00 [6833] pccd_core_received_pccreate(): LSP 'TEST-LSP1'
Apr 29 12:53:00 [6833] pccd_core_received_pccreate(): LSP 'TEST-LSP1'
Apr 29 12:53:00 [6833] pccd_core_store_lsp() -->
Apr 29 12:53:00 [6833] pccd_core_store_lsp(): Storing LSP ID 4
Apr 29 12:53:00 [6833] pccd_core_store_lsp() <--
Apr 29 12:53:00 [6833] pccd_core_received_pccreate(): PCE: 'odl' created provisioned LSP: 'TEST-LSP1' id: 4
Apr 29 12:53:00 [6833] pccd_core_received_pccreate() <--
Apr 29 12:53:00 [6833] pccd_pcep_onmessage_cb() <--
Apr 29 12:53:00 [6833] pccd_rpd_sock_readable_cb(): There's something to read on rpd's fd 11
Apr 29 12:53:00 [6833] pccd_rpd_lsp_err_recv() -->
Apr 29 12:53:00 [6833] pccd_rpd_lsp_err() -->
Apr 29 12:53:00 [6833] pccd_rpd_lsp_err(): RPD LSP Error: Attempt to update local LSP (errno:4), lsp: 'TEST-LSP1'
Apr 29 12:53:00 [6833] pccd_rpd_lsp_err(): Unable to Update/Delete LSP hence send error msg to PCE
Apr 29 12:53:00 [6833] pccd_core_del_lspvec_senderr(): Found cached LSP id: 4 in pce 'odl'
Apr 29 12:53:00 [6833] pcep_send_message: Sending PCEP message 'Error'
Apr 29 12:53:00 [6833] pcep_obj_trace: SRP object: remove: 48 srp_id: 1 lsp_setup_type: 48
Apr 29 12:53:00 [6833] pcep_obj_trace: ERROR object: type: 24, value: 2
Apr 29 12:53:00 [6833] pcep_obj_trace: TLV Missing Request ID: 808464432
Apr 29 12:53:00 [6833] pccd_core_del_lspvec_senderr() <--

And the second request:

Apr 29 12:53:46 [6833] pcep_msg_parse: Received PCEP message type: PCCreate, version: 1 length: 76, flags: 0x0
Apr 29 12:53:46 [6833] pcep_obj_trace: SRP object: remove: 0 srp_id: 2 lsp_setup_type: 0
Apr 29 12:53:46 [6833] pcep_tlv_trace: TLV path setup setup: 1
Apr 29 12:53:46 [6833] pcep_obj_trace: LSP object: plsp ID: 0, remove: 0, operational: 0, last_update: 0, delegated: 1 administrative: 1 is_p2mp 0, PCE Created: 0
Apr 29 12:53:46 [6833] pcep_tlv_trace: TLV LSP Symbolic Name: 'TEST-LSP1'
Apr 29 12:53:46 [6833] pcep_obj_trace: END-POINTS IPv4 object: source: 10.100.100.11, destination: 10.100.100.22
Apr 29 12:53:46 [6833] pcep_obj_trace: ERO object: subobjects: 1
Apr 29 12:53:46 [6833] pcep_obj_trace_ero: SPRING SUB-OBJECT: loose: 0, sid_type: 1, sid: 50011
Apr 29 12:53:46 [6833] pcep_msg_parse: PCCreate message received:
Apr 29 12:53:46 [6833] pcep_msg_parse: ...L!...........  20 0C 00 4C 21 12 00 14 00 00 00 00 00 00 00 02
Apr 29 12:53:46 [6833] pcep_msg_parse: ................  00 1C 00 04 00 00 00 01 20 10 00 18 00 00 00 09
Apr 29 12:53:46 [6833] pcep_msg_parse: ....TEST-LSP1...  00 11 00 09 54 45 53 54 2D 4C 53 50 31 00 00 00
Apr 29 12:53:46 [6833] pcep_msg_parse: .....dd..dd.....  04 12 00 0C 0A 64 64 0B 0A 64 64 16 07 10 00 10
Apr 29 12:53:46 [6833] pcep_msg_parse: $....5..%...      24 0C 10 01 0C 35 B0 00 25 E9 00 0E
Apr 29 12:53:46 [6833] process_message: Deadtimer has been touched
Apr 29 12:53:46 [6833] pccd_pcep_onmessage_cb() -->
Apr 29 12:53:46 [6833] pccd_pcep_onmessage_cb(): onmessage objp: 0x8425140 objc: 4 msg_type:12 pce:'odl'
Apr 29 12:53:46 [6833] pccd_core_received_pccreate() -->
Apr 29 12:53:46 [6833] pccd_core_received_pccreate(): Copying LSP name: 'TEST-LSP1' from TLV
Apr 29 12:53:46 [6833] pccd_core_parse_lsp_instantion() -->
Apr 29 12:53:46 [6833] pccd_core_parse_spring_ero() -->
Apr 29 12:53:46 [6833] pccd_core_parse_spring_ero(): Added SID lebel-20 is: 50011
Apr 29 12:53:46 [6833] pccd_core_print_sr_ero_info(): Number of SR-ERO subobjects: 1
Apr 29 12:53:46 [6833] pccd_core_print_sr_ero_info(): SR-ERO loose_type: 0
Apr 29 12:53:46 [6833] pccd_core_print_sr_ero_info(): SR-ERO SID lebel type: 2
Apr 29 12:53:46 [6833] pccd_core_print_sr_ero_info(): SR-ERO sid value: 1539506176
Apr 29 12:53:46 [6833] pccd_core_print_sr_ero_info(): SR-ERO NAI type: 1
Apr 29 12:53:46 [6833] pccd_core_print_sr_ero_info(): SR-ERO IPv4-Node addr for subobject 0: 37.233.0.14
Apr 29 12:53:46 [6833] pccd_core_parse_spring_ero() <--
Apr 29 12:53:46 [6833] pccd_core_parse_lsp_instantion() <--
Apr 29 12:53:46 [6833] pccd_core_received_pccreate(): PCE: 'odl' cannot create LSP(TEST-LSP1) - name exists
Apr 29 12:53:46 [6833] pcep_send_message: Sending PCEP message 'Error'
Apr 29 12:53:46 [6833] pcep_obj_trace: SRP object: remove: 48 srp_id: 2 lsp_setup_type: 48
Apr 29 12:53:46 [6833] pcep_obj_trace: ERROR object: type: 23, value: 1
Apr 29 12:53:46 [6833] pcep_obj_trace: TLV Missing Request ID: 808464432
Apr 29 12:53:46 [6833] pccd_core_received_pccreate() <--
Apr 29 12:53:46 [6833] pccd_pcep_onmessage_cb(): Handling of PCCreate failed

Any ideas on why I get Error 24 with Value 2 ? I've read that it is
LSP instantiation error and 2: Internal error but I don't know what to do to troubleshoot it or solve it... Could it be because of virtual enviroment or.. I don't know..

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 29, 2019 via email

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 29, 2019 via email

@avisnevschi
Copy link
Author

avisnevschi commented Apr 29, 2019

Probably the ODL controller doesn't clean it up because on the router I don't see any LSPs.

Now I tried once again to deploy a path, with Loose set as true.

<input xmlns="urn:opendaylight:params:xml:ns:yang:topology:pcep">
   <node>pcc://10.100.100.11</node>
   <name>LSP-to-NXDATA</name>
   <arguments>
      <lsp xmlns="urn:opendaylight:params:xml:ns:yang:pcep:ietf:stateful">
         <delegate>true</delegate>
         <administrative>true</administrative>
      </lsp>
      <endpoints-obj>
         <ipv4>
            <source-ipv4-address>10.100.100.11</source-ipv4-address>
            <destination-ipv4-address>10.100.100.22</destination-ipv4-address>
         </ipv4>
          <processing-rule>true</processing-rule>
      </endpoints-obj>
      <path-setup-type xmlns="urn:opendaylight:params:xml:ns:yang:pcep:ietf:stateful">
         <pst>1</pst>
      </path-setup-type>
      <ero>
         <subobject>
            <loose>true</loose>
            <sid-type xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">ipv4-node-id</sid-type>
            <m-flag xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">true</m-flag>
            <sid xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">2010</sid>
            <ip-address xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">37.233.0.17</ip-address>
        </subobject>
      </ero>
   </arguments>
   <network-topology-ref xmlns:topo="urn:TBD:params:xml:ns:yang:network-topology">/topo:network-topology/topo:topology[topo:topology-id="pcep-topology"]</network-topology-ref>
</input>

The topology is the following:
Router(IP:10.100.100.11, SID:2011) -> Router(IP:10.100.100.10, SID:2010) -> Router(IP:10.100.100.22, SID:2022)

This is the log from the router:

Apr 29 13:11:32 [6833] pcep_msg_parse: Received PCEP message type: PCCreate, version: 1 length: 80, flags: 0x0
Apr 29 13:11:32 [6833] pcep_obj_trace: SRP object: remove: 0 srp_id: 3 lsp_setup_type: 0
Apr 29 13:11:32 [6833] pcep_tlv_trace: TLV path setup setup: 1
Apr 29 13:11:32 [6833] pcep_obj_trace: LSP object: plsp ID: 0, remove: 0, operational: 0, last_update: 0, delegated: 1 administrative: 1 is_p2mp 0, PCE Created: 0
Apr 29 13:11:32 [6833] pcep_tlv_trace: TLV LSP Symbolic Name: 'LSP-to-NXDATA'
Apr 29 13:11:32 [6833] pcep_obj_trace: END-POINTS IPv4 object: source: 10.100.100.11, destination: 10.100.100.22
Apr 29 13:11:32 [6833] pcep_obj_trace: ERO object: subobjects: 1
Apr 29 13:11:32 [6833] pcep_obj_trace_ero: SPRING SUB-OBJECT: loose: 1, sid_type: 1, sid: 2010
Apr 29 13:11:32 [6833] pcep_msg_parse: PCCreate message received:
Apr 29 13:11:32 [6833] pcep_msg_parse: ...P!...........  20 0C 00 50 21 12 00 14 00 00 00 00 00 00 00 03
Apr 29 13:11:32 [6833] pcep_msg_parse: ................  00 1C 00 04 00 00 00 01 20 10 00 1C 00 00 00 09
Apr 29 13:11:32 [6833] pcep_msg_parse: ....LSP-to-NXDAT  00 11 00 0D 4C 53 50 2D 74 6F 2D 4E 58 44 41 54
Apr 29 13:11:32 [6833] pcep_msg_parse: A........dd..dd.  41 00 00 00 04 12 00 0C 0A 64 64 0B 0A 64 64 16
Apr 29 13:11:32 [6833] pcep_msg_parse: .........}..%...  07 10 00 10 A4 0C 10 01 00 7D A0 00 25 E9 00 11
Apr 29 13:11:32 [6833] pcep_msg_parse:
Apr 29 13:11:32 [6833] process_message: Deadtimer has been touched
Apr 29 13:11:32 [6833] pccd_pcep_onmessage_cb() -->
Apr 29 13:11:32 [6833] pccd_pcep_onmessage_cb(): onmessage objp: 0x8425140 objc: 4 msg_type:12 pce:'odl'
Apr 29 13:11:32 [6833] pccd_core_received_pccreate() -->
Apr 29 13:11:32 [6833] pccd_core_received_pccreate(): Copying LSP name: 'LSP-to-NXDATA' from TLV
Apr 29 13:11:32 [6833] pccd_core_parse_lsp_instantion() -->
Apr 29 13:11:32 [6833] pccd_core_parse_spring_ero() -->
Apr 29 13:11:32 [6833] pccd_core_parse_spring_ero(): Added SID lebel-20 is: 2010
Apr 29 13:11:32 [6833] pccd_core_print_sr_ero_info(): Number of SR-ERO subobjects: 1
Apr 29 13:11:32 [6833] pccd_core_print_sr_ero_info(): SR-ERO loose_type: 1
Apr 29 13:11:32 [6833] pccd_core_print_sr_ero_info(): SR-ERO SID lebel type: 2
Apr 29 13:11:32 [6833] pccd_core_print_sr_ero_info(): SR-ERO sid value: -637075456
Apr 29 13:11:32 [6833] pccd_core_print_sr_ero_info(): SR-ERO NAI type: 1
Apr 29 13:11:32 [6833] pccd_core_print_sr_ero_info(): SR-ERO IPv4-Node addr for subobject 0: 37.233.0.17
Apr 29 13:11:32 [6833] pccd_core_parse_spring_ero() <--
Apr 29 13:11:32 [6833] pccd_core_parse_lsp_instantion() <--
Apr 29 13:11:32 [6833] pccd_core_provisioned_lsps_increase_counter(): Provisioned LSPs counter increased: 2
Apr 29 13:11:32 [6833] pccd_core_received_pccreate(): Creating LSP name: 'LSP-to-NXDATA'
Apr 29 13:11:32 [6833] pccd_core_received_pccreate(): Got LSP name: 'LSP-to-NXDATA', Path name: '(null)' and priority: '(null)'
Apr 29 13:11:32 [6833] pccd_rpd_pccreate() -->
Apr 29 13:11:32 [6833] pccd_rpd_pccreate(): sending create request to RPD for LSP: 'LSP-to-NXDATA'
Apr 29 13:11:32 [6833] pccd_rpd_debug_lsp_info(): tx pccreate name(LSP-to-NXDATA) path() template() src(10.100.100.11) dst(10.100.100.11) msg_trigger() type(3) state(4) control(0) flags(0x0) admin grp exclude(0) admin grp include any(0) admin grp include all(0) setup prio(0) hold prio(0) bw(0bps) metric(0)
Apr 29 13:11:32 [6833] pccd_rpd_debug_lsp_info():     lsp_id(0) tunnel_id(0) setup_type(1) ero(0x0) ero_len(0) rro(0x0) rro_len(0) sr-ero(0x84a1210), sr-ero_len(0) sr-rro(0x0) sr-rro_len(0) err_spec(0x0) err_spec_len(0) #assoc(0)
Apr 29 13:11:32 [6833] pccd_rpd_print_sr_ero_info(): Number of SR-ERO subobjects: 1
Apr 29 13:11:32 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO loose_type: 1
Apr 29 13:11:32 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO SID level type: 2
Apr 29 13:11:32 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO sid value: -637075456
Apr 29 13:11:32 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO NAI type: 1
Apr 29 13:11:32 [6833] pccd_rpd_print_sr_ero_info(): SR-ERO IPv4-Node addr for subobject 0: 37.233.0.17
Apr 29 13:11:32 [6833] pccd_rpd_debug_lsp_info():     path_flags(0) new AuBw(0bps) #paths(0) assoc()local_cspf(0)
Apr 29 13:11:32 [6833] pccd_rpd_pcupdate_send() -->
Apr 29 13:11:32 [6833] pccd_rpd_encode_pcupdate() -->
Apr 29 13:11:32 [6833] pccd_rpd_encode_pcupdate(): lsp_params->setup_type: 1
Apr 29 13:11:32 [6833] pccd_rpd_encode_pcupdate() <--
Apr 29 13:11:32 [6833] pccd_rpd_pcupdate_send() <--
Apr 29 13:11:32 [6833] pccd_rpd_store_new_lsp() -->
Apr 29 13:11:32 [6833] pccd_rpd_reconfig_lsp(): LSP 0x847d280, template name
Apr 29 13:11:32 [6833] pccd_rpd_store_new_lsp(): Stored new LSP as :LSP-to-NXDATA for LSP LSP-to-NXDATA  with path '' id 5
Apr 29 13:11:32 [6833] pccd_rpd_store_new_lsp() <--
Apr 29 13:11:32 [6833] pccd_rpd_pccreate() <--
Apr 29 13:11:32 [6833] pccd_core_received_pccreate(): LSP 'LSP-to-NXDATA'
Apr 29 13:11:32 [6833] pccd_core_received_pccreate(): LSP 'LSP-to-NXDATA'
Apr 29 13:11:32 [6833] pccd_core_store_lsp() -->
Apr 29 13:11:32 [6833] pccd_core_store_lsp(): Storing LSP ID 5
Apr 29 13:11:32 [6833] pccd_core_store_lsp() <--
Apr 29 13:11:32 [6833] pccd_core_received_pccreate(): PCE: 'odl' created provisioned LSP: 'LSP-to-NXDATA' id: 5
Apr 29 13:11:32 [6833] pccd_core_received_pccreate() <--
Apr 29 13:11:32 [6833] pccd_pcep_onmessage_cb() <--
Apr 29 13:11:32 [6833] pccd_rpd_sock_readable_cb(): There's something to read on rpd's fd 11
Apr 29 13:11:32 [6833] pccd_rpd_lsp_err_recv() -->
Apr 29 13:11:32 [6833] pccd_rpd_lsp_err() -->
Apr 29 13:11:32 [6833] pccd_rpd_lsp_err(): RPD LSP Error: Attempt to update local LSP (errno:4), lsp: 'LSP-to-NXDATA'
Apr 29 13:11:32 [6833] pccd_rpd_lsp_err(): Unable to Update/Delete LSP hence send error msg to PCE
Apr 29 13:11:32 [6833] pccd_core_del_lspvec_senderr(): Found cached LSP id: 5 in pce 'odl'
Apr 29 13:11:32 [6833] pcep_send_message: Sending PCEP message 'Error'
Apr 29 13:11:32 [6833] pcep_obj_trace: SRP object: remove: 48 srp_id: 3 lsp_setup_type: 48
Apr 29 13:11:32 [6833] pcep_obj_trace: ERROR object: type: 24, value: 2
Apr 29 13:11:32 [6833] pcep_obj_trace: TLV Missing Request ID: 808464432
Apr 29 13:11:32 [6833] pccd_core_del_lspvec_senderr() <--

When I try to deploy the path from Pathman-SR, I get the same error, but I don't think it compute wrong ERO, because in logs I see all the SIDs, and they are correct.

If you want, I can give you access to ODL Controller and my LAB enviroment in EVE-NG.

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 29, 2019 via email

@avisnevschi
Copy link
Author

Yes, thats the node-id, but still, get the same error..

Request:

<input xmlns="urn:opendaylight:params:xml:ns:yang:topology:pcep">
   <node>pcc://10.100.100.11</node>
   <name>NEW-TEST-NXDATA</name>
   <arguments>
      <lsp xmlns="urn:opendaylight:params:xml:ns:yang:pcep:ietf:stateful">
         <delegate>true</delegate>
         <administrative>true</administrative>
      </lsp>
      <endpoints-obj>
         <ipv4>
            <source-ipv4-address>10.100.100.11</source-ipv4-address>
            <destination-ipv4-address>10.100.100.22</destination-ipv4-address>
         </ipv4>
          <processing-rule>true</processing-rule>
      </endpoints-obj>
      <path-setup-type xmlns="urn:opendaylight:params:xml:ns:yang:pcep:ietf:stateful">
         <pst>1</pst>
      </path-setup-type>
      <ero>
         <subobject>
            <loose>true</loose>
            <sid-type xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">ipv4-node-id</sid-type>
            <m-flag xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">true</m-flag>
            <sid xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">2010</sid>
            <ip-address xmlns="urn:opendaylight:params:xml:ns:yang:pcep:segment:routing">10.100.100.10</ip-address>
        </subobject>
      </ero>
   </arguments>
   <network-topology-ref xmlns:topo="urn:TBD:params:xml:ns:yang:network-topology">/topo:network-topology/topo:topology[topo:topology-id="pcep-topology"]</network-topology-ref>
</input>

Response:

{
    "output": {
        "failure": "failed",
        "error": [
            {
                "error-object": {
                    "type": 24,
                    "value": 2,
                    "ignore": true,
                    "processing-rule": true
                }
            }
        ]
    }
}

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 29, 2019 via email

@avisnevschi
Copy link
Author

avisnevschi commented Apr 29, 2019

Ok, will be waiting for any help from other members...

Are you also able to help with BGP PCEP Pathman which deploys RSVP tunnels ?

The idea is that we want to migrate to SDN controller for path computation, and... with Pathman-SR we get this error when trying to deploy paths while with BGP PCEP Pathman, we can deploy paths(as I tested directly from ODL controller and RSVP tunnels are created) but we don't see linkstate topology on pathman page:

/pathman/client/pathman/index.html -> don't see anything, it stucks on load...
/pathman/client/BGP/index.html -> we see the entire topology...

And in the logs when I open pathman/index.html I have no errors, but when I open BGP/index.html, I get this errors:

16544 2019-04-29 10:43:48.372 UTC root:get_url      INFO: Url get Status: 200
16544 2019-04-29 10:43:48.373 UTC root:parseNodes   INFO: BGP build node topology
16544 2019-04-29 10:43:48.374 UTC root:parseNodes   ERROR: BGP get node error2: list index out of range
16544 2019-04-29 10:43:48.374 UTC root:parseNodes   INFO: BGP Nodelist Len: 5
16544 2019-04-29 10:43:48.374 UTC root:parseLinks   INFO: BGP compose links
16544 2019-04-29 10:43:48.375 UTC root:get          INFO: BGP Nodes: 5
16544 2019-04-29 10:43:48.375 UTC root:get          INFO: BGP Links: 8

Should I open an Issue in Pathman project ?

@gilesheron
Copy link
Collaborator

gilesheron commented Apr 29, 2019 via email

@nikmon2
Copy link
Contributor

nikmon2 commented May 1, 2019

I can look at the Pathman error - Can I get access to the setup?

16544 2019-04-29 10:43:48.374 UTC root:parseNodes   ERROR: BGP get node error2: list index out of range

Cheers,
Nik

@avisnevschi
Copy link
Author

Of course I can give you access. Please give me any contact details where I can send you all credentials.

@nikmon2
Copy link
Contributor

nikmon2 commented May 1, 2019

please mailto nmontin @ cisco . com

@avisnevschi
Copy link
Author

avisnevschi commented May 1, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants