AssertionError: Cannot start restore, another operation running: {'type': 'backup', 'opID': '66ebc863db6d44488677de9f', 'name': '2024-09-19T06:44:51Z', 'startTS': 1726728291, 'status': 'copyDone'} 2024-09-19T06:44:39Z I [rs1/rs103:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:22_UTC GoVersion: go1.23.1 2024-09-19T06:44:39Z I [rs1/rs101:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:22_UTC GoVersion: go1.23.1 2024-09-19T06:44:39Z I [rs1/rs103:27017] starting PITR routine 2024-09-19T06:44:39Z I [rs1/rs102:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:22_UTC GoVersion: go1.23.1 2024-09-19T06:44:39Z I [rs1/rs102:27017] starting PITR routine 2024-09-19T06:44:39Z I [rs1/rs101:27017] starting PITR routine 2024-09-19T06:44:39Z I [rs1/rs102:27017] node: rs1/rs102:27017 2024-09-19T06:44:39Z I [rs1/rs103:27017] node: rs1/rs103:27017 2024-09-19T06:44:39Z I [rs1/rs101:27017] node: rs1/rs101:27017 2024-09-19T06:44:39Z I [rs1/rs102:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T06:44:39Z I [rs1/rs103:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T06:44:39Z I [rs1/rs101:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T06:44:39Z I [rs1/rs102:27017] listening for the commands 2024-09-19T06:44:39Z I [rs1/rs103:27017] listening for the commands 2024-09-19T06:44:39Z I [rs1/rs101:27017] listening for the commands 2024-09-19T06:44:44Z W [rs1/rs101:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T06:44:44Z W [rs1/rs103:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T06:44:44Z W [rs1/rs102:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T06:44:44Z E [rs1/rs101:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:44:44Z E [rs1/rs103:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:44:44Z E [rs1/rs102:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:44:46Z I [rs1/rs103:27017] got command resync <ts: 1726728286> 2024-09-19T06:44:46Z I [rs1/rs102:27017] got command resync <ts: 1726728286> 2024-09-19T06:44:46Z I [rs1/rs101:27017] got command resync <ts: 1726728286> 2024-09-19T06:44:46Z I [rs1/rs102:27017] got epoch {1726728284 12} 2024-09-19T06:44:46Z I [rs1/rs103:27017] got epoch {1726728284 12} 2024-09-19T06:44:46Z I [rs1/rs101:27017] got epoch {1726728284 12} 2024-09-19T06:44:46Z D [rs1/rs103:27017] [resync] lock not acquired 2024-09-19T06:44:46Z D [rs1/rs101:27017] [resync] lock not acquired 2024-09-19T06:44:46Z I [rs1/rs102:27017] [resync] started 2024-09-19T06:44:46Z D [rs1/rs102:27017] [resync] uploading ".pbm.init" [size hint: 5 (5.00B); part size: 10485760 (10.00MB)] 2024-09-19T06:44:46Z D [rs1/rs102:27017] [resync] got backups list: 0 2024-09-19T06:44:47Z D [rs1/rs102:27017] [resync] got physical restores list: 2 2024-09-19T06:44:47Z D [rs1/rs102:27017] [resync] epoch set to {1726728287 4} 2024-09-19T06:44:47Z I [rs1/rs102:27017] [resync] succeed 2024-09-19T06:44:51Z I [rs1/rs102:27017] got command backup [name: 2024-09-19T06:44:51Z, compression: s2 (level: default)] <ts: 1726728291> 2024-09-19T06:44:51Z I [rs1/rs103:27017] got command backup [name: 2024-09-19T06:44:51Z, compression: s2 (level: default)] <ts: 1726728291> 2024-09-19T06:44:51Z I [rs1/rs101:27017] got command backup [name: 2024-09-19T06:44:51Z, compression: s2 (level: default)] <ts: 1726728291> 2024-09-19T06:44:51Z I [rs1/rs103:27017] got epoch {1726728287 4} 2024-09-19T06:44:51Z I [rs1/rs102:27017] got epoch {1726728287 4} 2024-09-19T06:44:51Z I [rs1/rs101:27017] got epoch {1726728287 4} 2024-09-19T06:44:51Z D [rs1/rs101:27017] [backup/2024-09-19T06:44:51Z] init backup meta 2024-09-19T06:44:51Z D [rs1/rs101:27017] [backup/2024-09-19T06:44:51Z] nomination list for rs1: [[rs103:27017 rs102:27017] [rs101:27017]] 2024-09-19T06:44:51Z D [rs1/rs101:27017] [backup/2024-09-19T06:44:51Z] nomination rs1, set candidates [rs103:27017 rs102:27017] 2024-09-19T06:44:52Z D [rs1/rs103:27017] [backup/2024-09-19T06:44:51Z] skip: lock not acquired 2024-09-19T06:44:52Z I [rs1/rs102:27017] [backup/2024-09-19T06:44:51Z] backup started 2024-09-19T06:44:52Z D [rs1/rs101:27017] [backup/2024-09-19T06:44:51Z] skip after nomination, probably started by another node 2024-09-19T06:44:53Z D [rs1/rs102:27017] [backup/2024-09-19T06:44:51Z] backup cursor id: a042be2d-f8f2-4398-9122-84ed56b4026f 2024-09-19T06:44:56Z D [rs1/rs102:27017] [backup/2024-09-19T06:44:51Z] set journal up to {1726728293 1} 2024-09-19T06:44:56Z D [rs1/rs101:27017] [backup/2024-09-19T06:44:51Z] bcp nomination: rs1 won by rs102:27017

Process completed with exit code 1.

AssertionError: Cannot start restore, another operation running: {'type': 'backup', 'opID': '66ebc36e4f8618667bb44a65', 'name': '2024-09-19T06:23:42Z', 'startTS': 1726727022, 'status': 'copyDone'} 2024-09-19T06:23:20Z I [rs1/rs102:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:22_UTC GoVersion: go1.23.1 2024-09-19T06:23:20Z I [rs1/rs102:27017] starting PITR routine 2024-09-19T06:23:20Z I [rs1/rs101:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:22_UTC GoVersion: go1.23.1 2024-09-19T06:23:20Z I [rs1/rs103:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:22_UTC GoVersion: go1.23.1 2024-09-19T06:23:20Z I [rs1/rs101:27017] starting PITR routine 2024-09-19T06:23:20Z I [rs1/rs103:27017] starting PITR routine 2024-09-19T06:23:20Z I [rs1/rs101:27017] node: rs1/rs101:27017 2024-09-19T06:23:20Z I [rs1/rs102:27017] node: rs1/rs102:27017 2024-09-19T06:23:20Z I [rs1/rs103:27017] node: rs1/rs103:27017 2024-09-19T06:23:20Z I [rs1/rs103:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T06:23:20Z I [rs1/rs101:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T06:23:20Z I [rs1/rs102:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T06:23:20Z I [rs1/rs103:27017] listening for the commands 2024-09-19T06:23:20Z I [rs1/rs101:27017] listening for the commands 2024-09-19T06:23:20Z I [rs1/rs102:27017] listening for the commands 2024-09-19T06:23:22Z I [rs1/rs104:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:22_UTC GoVersion: go1.23.1 2024-09-19T06:23:22Z I [rs1/rs104:27017] node: rs1/rs104:27017 2024-09-19T06:23:22Z I [rs1/rs104:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T06:23:22Z I [rs1/rs104:27017] listening for the commands 2024-09-19T06:23:25Z W [rs1/rs101:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T06:23:25Z W [rs1/rs102:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T06:23:25Z E [rs1/rs101:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:23:25Z E [rs1/rs102:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:23:25Z W [rs1/rs103:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T06:23:25Z E [rs1/rs103:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:23:27Z W [rs1/rs104:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T06:23:27Z E [rs1/rs104:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:23:30Z E [rs1/rs101:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:23:30Z E [rs1/rs102:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:23:30Z E [rs1/rs103:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T06:23:32Z W [rs1/rs104:27017] [agentCheckup] storage is not initialized 2024-09-19T06:23:32Z I [rs1/rs104:27017] got command resync <ts: 1726727012> 2024-09-19T06:23:32Z I [rs1/rs104:27017] got epoch {1726727010 6} 2024-09-19T06:23:32Z I [rs1/rs104:27017] [resync] started 2024-09-19T06:23:32Z D [rs1/rs104:27017] [resync] uploading ".pbm.init" [size hint: 5 (5.00B); part size: 10485760 (10.00MB)] 2024-09-19T06:23:32Z D [rs1/rs104:27017] [resync] got backups list: 0 2024-09-19T06:23:32Z D [rs1/rs104:27017] [resync] got physical restores list: 0 2024-09-19T06:23:32Z D [rs1/rs104:27017] [resync] epoch set to {1726727012 13} 2024-09-19T06:23:32Z I [rs1/rs104:27017] [resync] succeed 2024-09-19T06:23:32Z I [rs1/rs103:27017] got command resync <ts: 1726727012> 2024-09-19T06:23:32Z I [rs1/rs101:27017] got command resync <ts: 1726727012> 2024-09-19T06:23:32Z I [rs1/rs102:27017] got command resync <ts: 1726727012> 2024-09-19T06:23:32Z I [rs1/rs103:27017] got epoch {1726727012 13} 2024-09-19T06:23:32Z I [rs1/rs101:27017] got epoch {1726727012 13} 2024-09-19T06:23:32Z I [rs1/rs102:27017] got epoch {1726727012 13} 2024-09-19T06:23:32Z D [rs1/rs101:27017] [resync] lock not acquired 2024-09-19T06:23:32Z D [rs1/rs102:27017] [resync] lock not acquired 2024-09-19T06:23:32Z D [rs1/rs103:27017] [resync] get lock: duplicate operation: 66ebc364b969bcdb2b5775da [Resync storage] 2024-09-19T06:23:32Z D [rs1/rs103:27017] [resync] lock not acquired 2024-09-19T06:23:42Z I [rs1/rs104:27017] got command backup [name: 2024-09-19T06:23:42Z, compression: s2 (level: default)] <ts: 1726727022> 2024-09-19T06:23:42Z I [rs1/rs104:27017] got epoch {1726727012 13} 2024-09-19T06:23:42Z I [rs1/rs101:27017] got command backup [name: 2024-09-19T06:23:42Z, compression: s2 (level: default)] <ts: 1726727022> 2024-09-19T06:23:42Z I [rs1/rs102:27017] got command backup [name: 2024-09-19T06:23:42Z, compression: s2 (level: default)] <ts: 1726727022> 2024-09-19T06:23:42Z I [rs1/rs103:27017] got command backup [name: 2024-09-19T06:23:42Z, compression: s2 (level: default)] <ts: 1726727022> 2024-09-19T06:23:42Z I [rs1/rs101:27017] got epoch {1726727012 13} 2024-09-19T06:23:42Z I [rs1/rs103:27017] got epoch {1726727012 13} 2024-09-19T06:23:42Z I [rs1/rs102:27017] got epoch {1726727012 13} 2024-09-19T06:23:42Z D [rs1/rs101:27017] [backup/2024-09-19T06:23:42Z] init backup meta 2024-09-19T06:23:42Z D [rs1/rs101:27017] [backup/2024-09-19T06:23:42Z] nomination list for rs1: [[rs103:27017] [rs102:27017] [rs101:27017]] 2024-09-19T06:23:42Z D [rs1/rs101:27017] [backup/2024-09-19T06:23:42Z] nomination rs1, set candidates [rs103:27017] 2024-09-19T06:23:43Z I [rs1/rs103:27017] [backup/2024-09-19T06:23:42Z] backup started 2024-09-19T06:23:43Z D [rs1/rs101:27017] [backup/2024-09-19T06:23:42Z] skip after nomination, probably started by another node 2024-09-19T06:23:43Z D [rs1/rs104:27017] [backup/2024-09-19T06:23:42Z] skip after nomination, probably started by another node 2024-09-19T06:23:43Z D [rs1/rs102:27017] [backup/2024-09-19T06:23:42Z] skip after nomination, probably started by another node 2024-09-19T06:23:44Z D [rs1/rs103:27017] [backup/2024-09-19T06:23:42Z] backup cursor id: ef0c2a35-41db-407b-9a79-002b02aaa0cc 2024-09-19T06:23:47Z D [rs1/rs103:27017] [backup/2024-09-19T06:23:42Z] set journal up to {1726727024 1} 2024-09-19T06:23:47Z D [rs1/rs101:27017] [backup/2024-09-19T06:23:42Z] bcp nomination: rs1 won by rs103:27017

Process completed with exit code 1.

AssertionError: Cannot start restore, another operation running: {'type': 'backup', 'opID': '66ebcd9c8045b066f344518b', 'name': '2024-09-19T07:07:08Z', 'startTS': 1726729629, 'status': 'copyDone'} 2024-09-19T07:06:56Z I [rs1/rs101:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:44_UTC GoVersion: go1.23.1 2024-09-19T07:06:56Z I [rs1/rs102:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:44_UTC GoVersion: go1.23.1 2024-09-19T07:06:56Z I [rs1/rs102:27017] starting PITR routine 2024-09-19T07:06:56Z I [rs1/rs103:27017] pbm-agent: Version: 2.6.0 Platform: linux/amd64 GitCommit: 4e467842b7e52c2892ff69bd24dac19a231db215 GitBranch: dev BuildTime: 2024-09-19_06:44_UTC GoVersion: go1.23.1 2024-09-19T07:06:56Z I [rs1/rs103:27017] starting PITR routine 2024-09-19T07:06:56Z I [rs1/rs101:27017] starting PITR routine 2024-09-19T07:06:56Z I [rs1/rs103:27017] node: rs1/rs103:27017 2024-09-19T07:06:56Z I [rs1/rs102:27017] node: rs1/rs102:27017 2024-09-19T07:06:56Z I [rs1/rs101:27017] node: rs1/rs101:27017 2024-09-19T07:06:56Z I [rs1/rs103:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T07:06:56Z I [rs1/rs102:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T07:06:56Z I [rs1/rs101:27017] conn level ReadConcern: majority; WriteConcern: majority 2024-09-19T07:06:56Z I [rs1/rs103:27017] listening for the commands 2024-09-19T07:06:56Z I [rs1/rs102:27017] listening for the commands 2024-09-19T07:06:56Z I [rs1/rs101:27017] listening for the commands 2024-09-19T07:07:01Z W [rs1/rs101:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T07:07:01Z W [rs1/rs103:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T07:07:01Z W [rs1/rs102:27017] [agentCheckup] get current storage status: query mongo: mongo: no documents in result 2024-09-19T07:07:01Z E [rs1/rs101:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T07:07:01Z E [rs1/rs103:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T07:07:01Z E [rs1/rs102:27017] [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result 2024-09-19T07:07:04Z I [rs1/rs103:27017] got command resync <ts: 1726729623> 2024-09-19T07:07:04Z I [rs1/rs102:27017] got command resync <ts: 1726729623> 2024-09-19T07:07:04Z I [rs1/rs101:27017] got command resync <ts: 1726729623> 2024-09-19T07:07:04Z I [rs1/rs103:27017] got epoch {1726729622 3} 2024-09-19T07:07:04Z I [rs1/rs102:27017] got epoch {1726729622 3} 2024-09-19T07:07:04Z I [rs1/rs101:27017] got epoch {1726729622 3} 2024-09-19T07:07:04Z D [rs1/rs101:27017] [resync] lock not acquired 2024-09-19T07:07:04Z D [rs1/rs102:27017] [resync] lock not acquired 2024-09-19T07:07:04Z I [rs1/rs103:27017] [resync] started 2024-09-19T07:07:04Z D [rs1/rs103:27017] [resync] uploading ".pbm.init" [size hint: 5 (5.00B); part size: 10485760 (10.00MB)] 2024-09-19T07:07:04Z D [rs1/rs103:27017] [resync] got backups list: 0 2024-09-19T07:07:04Z D [rs1/rs103:27017] [resync] got physical restores list: 2 2024-09-19T07:07:04Z D [rs1/rs103:27017] [resync] epoch set to {1726729624 20} 2024-09-19T07:07:04Z I [rs1/rs103:27017] [resync] succeed 2024-09-19T07:07:09Z I [rs1/rs103:27017] got command backup [name: 2024-09-19T07:07:08Z, compression: s2 (level: default)] <ts: 1726729628> 2024-09-19T07:07:09Z I [rs1/rs102:27017] got command backup [name: 2024-09-19T07:07:08Z, compression: s2 (level: default)] <ts: 1726729628> 2024-09-19T07:07:09Z I [rs1/rs101:27017] got command backup [name: 2024-09-19T07:07:08Z, compression: s2 (level: default)] <ts: 1726729628> 2024-09-19T07:07:09Z I [rs1/rs102:27017] got epoch {1726729624 20} 2024-09-19T07:07:09Z I [rs1/rs103:27017] got epoch {1726729624 20} 2024-09-19T07:07:09Z I [rs1/rs101:27017] got epoch {1726729624 20} 2024-09-19T07:07:09Z D [rs1/rs101:27017] [backup/2024-09-19T07:07:08Z] init backup meta 2024-09-19T07:07:09Z D [rs1/rs101:27017] [backup/2024-09-19T07:07:08Z] nomination list for rs1: [[rs102:27017 rs103:27017] [rs101:27017]] 2024-09-19T07:07:09Z D [rs1/rs101:27017] [backup/2024-09-19T07:07:08Z] nomination rs1, set candidates [rs102:27017 rs103:27017] 2024-09-19T07:07:09Z D [rs1/rs102:27017] [backup/2024-09-19T07:07:08Z] skip: lock not acquired 2024-09-19T07:07:09Z I [rs1/rs103:27017] [backup/2024-09-19T07:07:08Z] backup started 2024-09-19T07:07:09Z D [rs1/rs101:27017] [backup/2024-09-19T07:07:08Z] skip after nomination, probably started by another node 2024-09-19T07:07:10Z D [rs1/rs103:27017] [backup/2024-09-19T07:07:08Z] backup cursor id: 64226ddf-19d8-48c1-bdf7-d5b5bdf03344 2024-09-19T07:07:13Z D [rs1/rs103:27017] [backup/2024-09-19T07:07:08Z] set journal up to {1726729630 1} 2024-09-19T07:07:14Z D [rs1/rs101:27017] [backup/2024-09-19T07:07:08Z] bcp nomination: rs1 won by rs103:27017

Process completed with exit code 1.

AssertionError: Starting backup '2024-09-19T06:31:39Z'.... Waiting for '2024-09-19T06:31:39Z' backup................ done Backup '2024-09-19T06:31:39Z' to remote store 'http://nginx-minio:21114/bcp/pbme2etest' assert 0 != 0 + where 0 = CommandResult(backend=<testinfra.backend.docker.DockerBackend object at 0x7f9f79618a90>, exit_status=0, command=b'pbm ........... done\nBackup '2024-09-19T06:31:39Z' to remote store 'http://nginx-minio:21114/bcp/pbme2etest'\n", _stderr=b'').rc

AssertionError: Starting restore 2024-09-19T06:51:30.146643805Z to point-in-time 2024-09-19T06:49:59 from '2024-09-19T06:48:36Z'...Started logical restore. Waiting to finish....Error: operation failed with: reply oplog: replay chunk 1726728519.1726728597: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1726728567,"I":17},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"config.databases","Object":[{"Key":"_id","Value":"test"},{"Key":"primary","Value":"rs2"},{"Key":"partitioned","Value":true},{"Key":"version","Value":[{"Key":"uuid","Value":{"Subtype":4,"Data":"qbIUu0/jQ3CRBdO0q5WLvw=="}},{"Key":"timestamp","Value":{"T":1726728567,"I":15}},{"Key":"lastMod","Value":1}]}],"Query":null,"UI":{"Subtype":4,"Data":"qZgbxuM1R+yoMiO94gzEGg=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null,"MultiOpType":null} | merr <nil>: applyOps: (NamespaceNotFound) cannot apply insert or update operation on a non-existent namespace config.databases: { ts: Timestamp(1726728567, 17), t: 1, v: 2, op: "i", ns: "config.databases", o: { _id: "test", primary: "rs2", partitioned: true, version: { uuid: UUID("a9b214bb-4fe3-4370-9105-d3b4ab958bbf"), timestamp: Timestamp(1726728567, 15), lastMod: 1 } }, ui: UUID("a9981bc6-e335-47ec-a832-23bde20cc41a") }

Process completed with exit code 1.

AssertionError: Starting backup '2024-09-19T06:45:25Z'.... Waiting for '2024-09-19T06:45:25Z' backup.......................... done Backup '2024-09-19T06:45:25Z' to remote store 'http://nginx-minio:21114/bcp/pbme2etest' assert 0 != 0 + where 0 = CommandResult(backend=<testinfra.backend.docker.DockerBackend object at 0x7f7a46c15150>, exit_status=0, command=b'pbm ........... done\nBackup '2024-09-19T06:45:25Z' to remote store 'http://nginx-minio:21114/bcp/pbme2etest'\n", _stderr=b'').rc

AssertionError: Starting restore 2024-09-19T07:05:40.242397046Z to point-in-time 2024-09-19T07:04:04 from '2024-09-19T07:02:41Z'...Started logical restore. Waiting to finish....Error: operation failed with: reply oplog: replay chunk 1726729365.1726729442: apply oplog for chunk: applying an entry: op: {"Timestamp":{"T":1726729413,"I":2},"Term":1,"Hash":null,"Version":2,"Operation":"i","Namespace":"config.databases","Object":[{"Key":"_id","Value":"test"},{"Key":"primary","Value":"rs2"},{"Key":"partitioned","Value":false},{"Key":"version","Value":[{"Key":"uuid","Value":{"Subtype":4,"Data":"RkJgQzaMQhCaGZoKNJqwEQ=="}},{"Key":"timestamp","Value":{"T":1726729412,"I":15}},{"Key":"lastMod","Value":1}]}],"Query":[{"Key":"_id","Value":"test"}],"UI":{"Subtype":4,"Data":"QGp36c8PQdSDE85QA4+n2A=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null,"MultiOpType":null} | merr <nil>: applyOps: (NamespaceNotFound) cannot apply insert or update operation on a non-existent namespace config.databases: { ts: Timestamp(1726729413, 2), t: 1, v: 2, op: "i", ns: "config.databases", o: { _id: "test", primary: "rs2", partitioned: false, version: { uuid: UUID("46426043-368c-4210-9a19-9a0a349ab011"), timestamp: Timestamp(1726729412, 15), lastMod: 1 } }, o2: { _id: "test" }, ui: UUID("406a77e9-cf0f-41d4-8313-ce50038fa7d8") }

Process completed with exit code 1.

AssertionError: Starting backup '2024-09-19T06:57:06Z'.... Waiting for '2024-09-19T06:57:06Z' backup.......................... done Backup '2024-09-19T06:57:06Z' to remote store 'http://nginx-minio:21114/bcp/pbme2etest' assert 0 != 0 + where 0 = CommandResult(backend=<testinfra.backend.docker.DockerBackend object at 0x7fae6e7c9cd0>, exit_status=0, command=b'pbm ........... done\nBackup '2024-09-19T06:57:06Z' to remote store 'http://nginx-minio:21114/bcp/pbme2etest'\n", _stderr=b'').rc

AssertionError: Starting restore 2024-09-19T07:17:04.22898465Z to point-in-time 2024-09-19T07:15:29 from '2024-09-19T07:14:06Z'...Started logical restore. Waiting to finish....Error: operation failed with: reply oplog: replay chunk 1726730050.1726730127: apply oplog for chunk: applying a transaction entry: apply txn: { "Timestamp": { "T": 1726730098, "I": 1 }, "Term": 1, "Hash": null, "Version": 2, "Operation": "c", "Namespace": "admin.$cmd", "Object": [ { "Key": "applyOps", "Value": [ [ { "Key": "op", "Value": "c" }, { "Key": "ns", "Value": "config.$cmd" }, { "Key": "ui", "Value": { "Subtype": 4, "Data": "8+c6Ptq3Qi2vIlaOhCpTxw==" } }, { "Key": "o", "Value": [ { "Key": "create", "Value": "databases" }, { "Key": "idIndex", "Value": [ { "Key": "v", "Value": 2 }, { "Key": "key", "Value": [ { "Key": "_id", "Value": 1 } ] }, { "Key": "name", "Value": "_id_" } ] } ] } ], [ { "Key": "op", "Value": "i" }, { "Key": "ns", "Value": "config.databases" }, { "Key": "ui", "Value": { "Subtype": 4, "Data": "8+c6Ptq3Qi2vIlaOhCpTxw==" } }, { "Key": "o", "Value": [ { "Key": "_id", "Value": "test" }, { "Key": "primary", "Value": "rs1" }, { "Key": "partitioned", "Value": false }, { "Key": "version", "Value": [ { "Key": "uuid", "Value": { "Subtype": 4, "Data": "LxPZ1siyR4O38qK3935D/Q==" } }, { "Key": "timestamp", "Value": { "T": 1726730097, "I": 15 } }, { "Key": "lastMod", "Value": 1 } ] } ] }, { "Key": "o2", "Value": [ { "Key": "_id", "Value": "test" } ] } ], [ { "Key": "op", "Value": "i" }, { "Key": "ns", "Value": "config.placementHistory" }, { "Key": "ui", "Value": { "Subtype": 4, "Data": "tLfUxD8NRY+Af+grGB3Erg==" } }, { "Key": "o", "Value": [ { "Key": "_id", "Value": "66ebcf72cd5d23614db0abe6" }, { "Key": "nss", "Value": "test" }, { "Key": "timestamp", "Value": { "T": 1726730097, "I": 15 } }, { "Key": "shards", "Value": [ "rs1" ] } ] }, { "Key": "o2", "Value": [ { "Key": "_id", "Value": "66ebcf72cd5d23614db0abe6" } ] } ] ] } ], "Query": null, "UI": null, "LSID": "ZgAAAAVpZAAQAAAABEmrjt+obEB/hQNOzRv5olMFdWlkACAAAAAAY5mrDaxi8gv8RmdTsQ+1j7fmkr7JUsabhNmXAheU0fgFdHhuVVVJRAAQAAAABB8KS2vzlUkZnWV23DlBVbsA", "TxnNumber": 0, "PrevOpTime": "HAAAABF0cwAAAAAAAAAAABJ0AP//////////AA==", "MultiOpType": null }: applying transaction op: op: {"Timestamp":{"T":1726730098,"I":1},"Term":1,"Hash":null,"Version":0,"Operation":"i","Namespace":"config.databases","Object":[{"Key":"_id","Value":"test"},{"Key":"primary","Value":"rs1"},{"Key":"partitioned","Value":false},{"Key":"version","Value":[{"Key":"uuid","Value":{"Subtype":4,"Data":"LxPZ1siyR4O38qK3935D/Q=="}},{"Key":"timestamp","Value":{"T":1726730097,"I":15}},{"Key":"lastMod","Value":1}]}],"Query":[{"Key":"_id","Value":"test"}],"UI":{"Subtype":4,"Data":"8+c6Ptq3Qi2vIlaOhCpTxw=="},"LSID":null,"TxnNumber":null,"PrevOpTime":null,"MultiOpType":null} | merr <nil>: applyOps: (UnknownError) applyOps had one or more errors applying ops

Process completed with exit code 1.

AssertionError: Unexpected exit code 1 for CommandResult(backend=<testinfra.backend.docker.DockerBackend object at 0x7fc7b70ef010>, exit_status=1, command=b'supervisorctl start mongod-ce', _stdout=b'mongod-ce: ERROR (no such file)\n', _stderr=b'') assert 1 == 0 + where 1 = CommandResult(backend=<testinfra.backend.docker.DockerBackend object at 0x7fc7b70ef010>, exit_status=1, command=b'supervisorctl start mongod-ce', _stdout=b'mongod-ce: ERROR (no such file)\n', _stderr=b'').rc