-
-
Notifications
You must be signed in to change notification settings - Fork 99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Auto-Reconnect or How to listen for disconnections ? #198
Comments
You are right. |
Thanks for your reply. Regards |
Hi, I am hosting a server (gRPC, not REST, but I think this is irrelevant) and I created the following scenario. I do a first request with the database already open and everything works fine. I am checking if the database is open and reopening if not. If I shutdown MongoDb and start it up again and then perform another request, this works flawlessly. Then I will shutdown the server, do a request and it will fail, as expected. Now, however, because I made a request while MongoDb was down, I ended doing a call to db.open while the db server is unavailable. Next, I start MongoDb once more and do another request. The call to db.open will fail because it says it is in an invalid state. I did try to do a call to db.close() while the state was opening, but it also threw. So, what I ended doing was the following code: // I am not so sure if I really need the _db.state == State.OPENING condition
if (!_db.isConnected || _db.state == State.OPENING) {
if (_db.state == State.OPENING) {
_db.state = State.CLOSED; // I am manually updating the state to CLOSE to prevent the invalid state exception.
}
await _db.open();
}
It works, though it seems like a hack. I am running MongoDb as a replica-set. Kind Regards, Rui |
You are right. There was an inconsistency in how the "
Could you please give a try and tell me if everything is OK? I would greatly appreciate. |
It is much, much, better now. I am doing some stress testing and the client performs several requests in burst sequence. After starting MongoDb again, the very first request to db.open() is working fine now. The following few requests will throw, and then further requests will be OK. Though it's not completely desirable behavior, it is kind of expected and understandable. The first request asks to create the connection, and then the others will find a connection that is still not open, but is underway trying to be opened, so it throws with a complaint that the state is still opening. Maybe I'll just leave it as it is or later on add a flag and some code in lieu of a semaphore, which Dart does not have. Anyhow, your fix solves the problem. Thanks you! 😊 Best Regards! Rui |
Published. |
Actually I have more than an idea, but a companion package to share in a future date. MongoDB drivers for C# actually map the bson they receive from MongoDb into plain-old C# objects, objects of classes that represent the business domain. So, in C#, I never need to work with the equivalent of Map<String, dynamic>, but only with stuff like Person, Contract, Car, Tree,..., and so on. I wanted to have the same thing with Dart, so I have been working on a prototype of a code generator that creates code that receives Map and does the conversion to business domain objects and back. It is inspired by json_serializable, but doesn't add it as a dependency (the problem I am solving is very similar). The problem with my package is that I am trying to solve too many problems on that prototype and now I need to split the implementation and do a whole lot of refactoring before I share it. That same prototype does a whole lot of other things. Because I am planning on using gRPC, my prototype is also generating the corresponding .proto files, which are then fed into protoc to generate the Dart representation of protocol buffer message. My code generator also does the mapping between my business domain classes and those generated protobuffer dart classes. Finally, for each Class, the code generator also generates a Validator class that adds validation methods for custom validation annotations I defined. An example: @map
class Person
{
ObjectId id; // by default will be mapped to '_id' when the object is mapped to Map<String, dynamic>
@required
String name;
@MapField(name: 'dob') // will be mapped to 'dob' key instead of birthDate
DateTime birthDate;
List<Contract> contracts;
@mapIgnore
int dummy; // not mapped
}
Class person will have a class PersonMapper. Class Contract will also have a ContractMapper. PersonMapper will map all the fields into Map<String, dynamic> and back, using ContractMapper to map each contract in Contracts, and so on. There will also be a new generated person.proto file. gRPC related protoc utility also generates classes and I am prefixing them with G (configurable), so the equivalent protoc class will be GPerson. My code generator also creates a PersonProtoMapper that will map between Person and GPerson. It will also create a PersonValidator that will tell if the instance is valid according to business rules. In this example there is only one rule, that name must be a none-empty string. Finally, in case Person is an immutable class, the code generator also creates a PersonBuilder, a corresponding mutable class that has a build() method to generate immutable Person instances. As if all this weren't enough, I am also generating classes to represent CRUD permission points for each of the business classes. The purpose of all this mess and complexity is actually to simplify the development workflow. In the end, each business concept will be represented by a single root source of truth, from which everything else is generated code, thus avoiding tons of manually created boilerplate. As soon as I find the time to split this into more fine-grained packages, I intend sharing this code on github. If you like the idea and would like to integrate this into mongo-dart project, say, for instance as mongo_dart_generator, it would be fine by me. |
Interesting. I do have a similar environment, but it is too complex and tailored on my needs to be shared. I'd like to make a simpler version, but at present I have no time... |
What if we have a function db.onDisconnect({required Function function})? |
The idea is to have a client (like NodeJs driver), and let the client manage the connections. This is in the work list, but it is a complex task, that implies also some breaking changes. |
Are there any updates on this reconnection effort? Currently running into it myself. |
Unfortunately no news |
A couple years ago you posted the following:
How about a backoff strategy where the retries get further and further apart? I've seen other clients (message queues, databases, etc) also implement a maxRetries configuration. An event stream could be helpful as a way to allow the user to handle and implmement their own reconnection logic. Personally, I had to fully wrap both class MongoDbImpl implements mongo.Db {
mongo.Db? __delegate;
mongo.Db get _delegate => (__delegate != null ? __delegate! : throw new mongo.MongoDartError('Instance not connected. Must call `open()` first'));
final String? _debugInfo;
final String _uriString;
Completer<void> _whenReadyCompleter = new Completer<void>();
Future<void> get whenReady => _whenReadyCompleter.future;
bool get isReady => _whenReadyCompleter.isCompleted && this.state == dbConnection.State.OPEN;
MongoDbImpl(String uriString, [String? debugInfo]): this._uriString = uriString, this._debugInfo = debugInfo;
MongoDbImpl.pool(List<String> uriList, [String? debugInfo]): this(uriList.join(','), debugInfo);
static Future<MongoDbImpl> create(String uriString, [String? _debugInfo]) async {
if (uriString.startsWith('mongodb://')) {
return MongoDbImpl(uriString, _debugInfo);
} else if (uriString.startsWith('mongodb+srv://')) {
var temp = await mongo.Db.create(uriString);
return MongoDbImpl.pool(temp.uriList, _debugInfo);
} else {
throw mongo.MongoDartError('The only valid schemas for Db are: "mongodb" and "mongodb+srv".');
}
}
@override
Future<void> open({
mongo.WriteConcern writeConcern = mongo.WriteConcern.ACKNOWLEDGED,
bool secure = false,
bool tlsAllowInvalidCertificates = false,
String? tlsCAFile,
String? tlsCertificateKeyFile,
String? tlsCertificateKeyFilePassword
}) async {
this.__delegate ??= await mongo.Db.create(_uriString, _debugInfo);
unawaited(_delegate.open(
writeConcern : writeConcern,
secure : secure,
tlsAllowInvalidCertificates : tlsAllowInvalidCertificates,
tlsCAFile : tlsCAFile,
tlsCertificateKeyFile : tlsCertificateKeyFile,
tlsCertificateKeyFilePassword: tlsCertificateKeyFilePassword,
)
.catchError((_) {
var retryDelay = Duration(seconds: 3);
log.config('MongoDb not ready, retrying in $retryDelay ...');
return Future.delayed(retryDelay, () => this.open(
writeConcern : writeConcern,
secure : secure,
tlsAllowInvalidCertificates : tlsAllowInvalidCertificates,
tlsCAFile : tlsCAFile,
tlsCertificateKeyFile : tlsCertificateKeyFile,
tlsCertificateKeyFilePassword: tlsCertificateKeyFilePassword,
));
})
.then((x) => (!_whenReadyCompleter.isCompleted ? _whenReadyCompleter.complete(x) : x)) // TODO: why is this sometimes already completed / tries to complete twice?
);
return whenReady;
}
Future<void> _reconnect() async {
log.config('Lost connection to MongoDB - reconnecting...');
await close();
await open().then((_) => log.config('Reconnected to MongoDB'));
return whenReady;
}
// -- passthrus
@override mongo.Db? get authSourceDb => _delegate.authSourceDb;
@override set authSourceDb(mongo.Db? _authSourceDb) => _delegate.authSourceDb = _authSourceDb;
... etc, etc, etc for 36 other getters/setters/methods ...
@override mongo.DbCollection collection(String collectionName) => MongoCollection(this, collectionName);
}
class MongoCollection implements mongo.DbCollection {
final MongoDbImpl _db;
final mongo.DbCollection _delegate;
MongoCollection(this._db, String collectionName): _delegate = _db._delegate.collection(collectionName);
Future<void> get whenReady => _db.whenReady;
Future<void> _reconnect() {
return _db._reconnect().then((_) => this.whenReady);
}
Stream<T> _tryWithReconnectStream<T>(Stream<T> Function() computation) async* {
try {
yield* computation();
} on mongo.MongoDartError catch (e) { // ignore: avoid_catching_errors
if (e.message == 'No master connection') {
await _reconnect();
yield* _tryWithReconnectStream(computation);
} else {
rethrow;
}
}
}
Future<T> _tryWithReconnectFuture<T>(Future<T> Function() computation) async {
try {
return computation();
} on mongo.MongoDartError catch (e) { // ignore: avoid_catching_errors
if (e.message == 'No master connection') {
await _reconnect();
return _tryWithReconnectFuture(computation);
} else {
rethrow;
}
}
}
// -- passthrus
@override Future<Map<String, dynamic>?> findOne([dynamic selector]) => _tryWithReconnectFuture(() => _delegate.findOne(selector));
@override Stream<Map<String, dynamic>> legacyFind([dynamic selector]) => _tryWithReconnectStream(() => _delegate.legacyFind(selector));
... etc, etc, etc for 45 other getters/setters/methods ...
} Obviously, this won't work for everyone, (besides being a ton of work to override every single method), since it'll just indefinitely continue trying to reconnect, but maybe it can provide some inspiration. (it also implements a |
Thanks for your suggestion. My idea is to take example from tne Nodejs driver and manage messaging on the status and auto-reconnect based on a polling check every 2-3 seconds. |
Any news on this?, I've come to realise my implementation isn't that great😓, well one of them at least but I still need a better one. |
Give a look to discussion #228 . Maybe it could help. |
Would it be a good idea to use those reconnect methods in a Middleware for a web server to verify connection before each api call and return an internal server error or something on failure to reconnect? Or is it better before each DB call like they use it? |
Hello I'm thinking of a server usage.
In my dart server I want to hold a few connections to a mongodb.
In case of disconnection of the mongodb server (such as network failure),
I would like to auto-reconnect. Currently it seems to simply throw the following error:
How is it possible to auto-reconnect?
Or if it is not implemented It would be nice If i could listen for disconnections so I could write some code to re-connect.
If I missed it, I'm sorry.
Regards
The text was updated successfully, but these errors were encountered: