Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-Reconnect or How to listen for disconnections ? #198

Open
hiroshihorie opened this issue Sep 28, 2020 · 17 comments
Open

Auto-Reconnect or How to listen for disconnections ? #198

hiroshihorie opened this issue Sep 28, 2020 · 17 comments

Comments

@hiroshihorie
Copy link

Hello I'm thinking of a server usage.

In my dart server I want to hold a few connections to a mongodb.
In case of disconnection of the mongodb server (such as network failure),
I would like to auto-reconnect. Currently it seems to simply throw the following error:

throw MongoDartError('No master connection');

How is it possible to auto-reconnect?
Or if it is not implemented It would be nice If i could listen for disconnections so I could write some code to re-connect.

If I missed it, I'm sorry.
Regards

@giorgiofran
Copy link
Contributor

You are right.
I thought about this problem, but I couldn't find any good solution.
My doubt was which algorithm to implement. I mean, If it is a network problem, maybe retrying a couple of times could be enough, but, if it is a server problem we could end retrying indefinitely. So, even a fixed number of retries could not be a correct solution.
I ended up doing nothing.
Maybe your suggestion of a listener is a good solution.
Anyone else has a suggestion?
Briefly, at present in my server I test if it is connected before of each operation (or small logically linked group of operations).
I do check the db.masterconnection.connected flag (it is a quite new getter). If it is false I close and reopen the database.
I know that it is not an elegant solution, but it works.

@hiroshihorie
Copy link
Author

Thanks for your reply.
For now, I will just use a simple timer to monitor db.masterconnection.connected and re-connect if disconnected.

Regards

@ruicraveiro
Copy link

ruicraveiro commented Oct 29, 2020

Hi, I am hosting a server (gRPC, not REST, but I think this is irrelevant) and I created the following scenario. I do a first request with the database already open and everything works fine. I am checking if the database is open and reopening if not. If I shutdown MongoDb and start it up again and then perform another request, this works flawlessly. Then I will shutdown the server, do a request and it will fail, as expected. Now, however, because I made a request while MongoDb was down, I ended doing a call to db.open while the db server is unavailable. Next, I start MongoDb once more and do another request. The call to db.open will fail because it says it is in an invalid state. I did try to do a call to db.close() while the state was opening, but it also threw. So, what I ended doing was the following code:

     // I am not so sure if I really need the _db.state == State.OPENING condition
    if (!_db.isConnected || _db.state == State.OPENING) {
      if (_db.state == State.OPENING) {
        _db.state = State.CLOSED; // I am manually updating the state to CLOSE to prevent the invalid state exception.
      }
      await _db.open();
    }

It works, though it seems like a hack. I am running MongoDb as a replica-set.

Kind Regards,

Rui

@giorgiofran
Copy link
Contributor

giorgiofran commented Oct 30, 2020

You are right. There was an inconsistency in how the "OPENING" state was managed. The idea of "OPENING" is to prevent different calls to db.open() at the same time, so your hack cannot be a definitive solution
I made a fix and posted on GitHub.
You can try it temporarily changing your pubspec.yaml settings to the following for mongo_dart dependency:

mongo_dart:
    git:
      url: "https://github.com/mongo-dart/mongo_dart.git"   
      ref: master 

Could you please give a try and tell me if everything is OK? I would greatly appreciate.
Once tested I will publish it.

@ruicraveiro
Copy link

It is much, much, better now. I am doing some stress testing and the client performs several requests in burst sequence. After starting MongoDb again, the very first request to db.open() is working fine now. The following few requests will throw, and then further requests will be OK. Though it's not completely desirable behavior, it is kind of expected and understandable. The first request asks to create the connection, and then the others will find a connection that is still not open, but is underway trying to be opened, so it throws with a complaint that the state is still opening. Maybe I'll just leave it as it is or later on add a flag and some code in lieu of a semaphore, which Dart does not have.

Anyhow, your fix solves the problem. Thanks you! 😊

Best Regards!

Rui

@giorgiofran
Copy link
Contributor

Published.
If you have any idea on how to improve the package, please let me know.
I'm quite busy right now, but I will surely consider it.

@ruicraveiro
Copy link

Actually I have more than an idea, but a companion package to share in a future date. MongoDB drivers for C# actually map the bson they receive from MongoDb into plain-old C# objects, objects of classes that represent the business domain. So, in C#, I never need to work with the equivalent of Map<String, dynamic>, but only with stuff like Person, Contract, Car, Tree,..., and so on. I wanted to have the same thing with Dart, so I have been working on a prototype of a code generator that creates code that receives Map and does the conversion to business domain objects and back. It is inspired by json_serializable, but doesn't add it as a dependency (the problem I am solving is very similar). The problem with my package is that I am trying to solve too many problems on that prototype and now I need to split the implementation and do a whole lot of refactoring before I share it. That same prototype does a whole lot of other things. Because I am planning on using gRPC, my prototype is also generating the corresponding .proto files, which are then fed into protoc to generate the Dart representation of protocol buffer message. My code generator also does the mapping between my business domain classes and those generated protobuffer dart classes. Finally, for each Class, the code generator also generates a Validator class that adds validation methods for custom validation annotations I defined. An example:

@map
class Person
{
    ObjectId id; // by default will be mapped to '_id' when the object is mapped to Map<String, dynamic>

   @required
   String name;

   @MapField(name: 'dob') // will be mapped to 'dob' key instead of birthDate
   DateTime birthDate;

   List<Contract> contracts;
   
   @mapIgnore
   int dummy; // not mapped
}

Class person will have a class PersonMapper. Class Contract will also have a ContractMapper. PersonMapper will map all the fields into Map<String, dynamic> and back, using ContractMapper to map each contract in Contracts, and so on. There will also be a new generated person.proto file. gRPC related protoc utility also generates classes and I am prefixing them with G (configurable), so the equivalent protoc class will be GPerson. My code generator also creates a PersonProtoMapper that will map between Person and GPerson.

It will also create a PersonValidator that will tell if the instance is valid according to business rules. In this example there is only one rule, that name must be a none-empty string. Finally, in case Person is an immutable class, the code generator also creates a PersonBuilder, a corresponding mutable class that has a build() method to generate immutable Person instances. As if all this weren't enough, I am also generating classes to represent CRUD permission points for each of the business classes.

The purpose of all this mess and complexity is actually to simplify the development workflow. In the end, each business concept will be represented by a single root source of truth, from which everything else is generated code, thus avoiding tons of manually created boilerplate. As soon as I find the time to split this into more fine-grained packages, I intend sharing this code on github. If you like the idea and would like to integrate this into mongo-dart project, say, for instance as mongo_dart_generator, it would be fine by me.

@giorgiofran
Copy link
Contributor

Interesting. I do have a similar environment, but it is too complex and tailored on my needs to be shared. I'd like to make a simpler version, but at present I have no time...

@brunoggt
Copy link

brunoggt commented Apr 8, 2021

What if we have a function db.onDisconnect({required Function function})?
It could trigger a function every time the db change it state to disconnect, then would be possible to reconnect and keep the connection on

@giorgiofran
Copy link
Contributor

The idea is to have a client (like NodeJs driver), and let the client manage the connections. This is in the work list, but it is a complex task, that implies also some breaking changes.
As I had to spend a lot of time with the null-safety version, I cannot start immediately.
Aside the automatic management from the client, my intention is to let the user be informed about connection status (if required), with a mechanism similar to the one you pointed out.
I cannot tell you when it will be ready... :-(

@bobjackman
Copy link

Are there any updates on this reconnection effort? Currently running into it myself.

@giorgiofran
Copy link
Contributor

Unfortunately no news

@bobjackman
Copy link

bobjackman commented Mar 9, 2022

A couple years ago you posted the following:

You are right. I thought about this problem, but I couldn't find any good solution. My doubt was which algorithm to implement. I mean, If it is a network problem, maybe retrying a couple of times could be enough, but, if it is a server problem we could end retrying indefinitely. So, even a fixed number of retries could not be a correct solution. I ended up doing nothing. Maybe your suggestion of a listener is a good solution. _Anyone else has a suggestion? ...

How about a backoff strategy where the retries get further and further apart? I've seen other clients (message queues, databases, etc) also implement a maxRetries configuration. An event stream could be helpful as a way to allow the user to handle and implmement their own reconnection logic. Personally, I had to fully wrap both Db and Collection classes like so:

class MongoDbImpl implements mongo.Db {
  mongo.Db? __delegate;
  mongo.Db get _delegate => (__delegate != null ? __delegate! : throw new mongo.MongoDartError('Instance not connected. Must call `open()` first'));

  final String? _debugInfo;
  final String  _uriString;

  Completer<void>   _whenReadyCompleter = new Completer<void>();
  Future<void> get  whenReady => _whenReadyCompleter.future;
  bool         get  isReady   => _whenReadyCompleter.isCompleted && this.state == dbConnection.State.OPEN;

  MongoDbImpl(String uriString, [String? debugInfo]): this._uriString = uriString, this._debugInfo = debugInfo;

  MongoDbImpl.pool(List<String> uriList, [String? debugInfo]): this(uriList.join(','), debugInfo);

  static Future<MongoDbImpl> create(String uriString, [String? _debugInfo]) async {
    if (uriString.startsWith('mongodb://')) {
      return MongoDbImpl(uriString, _debugInfo);
    } else if (uriString.startsWith('mongodb+srv://')) {
      var temp = await mongo.Db.create(uriString);
      return MongoDbImpl.pool(temp.uriList, _debugInfo);
    } else {
      throw mongo.MongoDartError('The only valid schemas for Db are: "mongodb" and "mongodb+srv".');
    }
  }

  @override
  Future<void> open({
    mongo.WriteConcern writeConcern                = mongo.WriteConcern.ACKNOWLEDGED,
    bool               secure                      = false,
    bool               tlsAllowInvalidCertificates = false,
    String?            tlsCAFile,
    String?            tlsCertificateKeyFile,
    String?            tlsCertificateKeyFilePassword
  }) async {
    this.__delegate ??= await mongo.Db.create(_uriString, _debugInfo);
    unawaited(_delegate.open(
        writeConcern                 : writeConcern,
        secure                       : secure,
        tlsAllowInvalidCertificates  : tlsAllowInvalidCertificates,
        tlsCAFile                    : tlsCAFile,
        tlsCertificateKeyFile        : tlsCertificateKeyFile,
        tlsCertificateKeyFilePassword: tlsCertificateKeyFilePassword,
      )
      .catchError((_) {
        var retryDelay = Duration(seconds: 3);
        log.config('MongoDb not ready, retrying in $retryDelay ...');
        return Future.delayed(retryDelay, () => this.open(
          writeConcern                 : writeConcern,
          secure                       : secure,
          tlsAllowInvalidCertificates  : tlsAllowInvalidCertificates,
          tlsCAFile                    : tlsCAFile,
          tlsCertificateKeyFile        : tlsCertificateKeyFile,
          tlsCertificateKeyFilePassword: tlsCertificateKeyFilePassword,
        ));
      })
      .then((x) => (!_whenReadyCompleter.isCompleted ? _whenReadyCompleter.complete(x) : x)) // TODO: why is this sometimes already completed / tries to complete twice?
    );

    return whenReady;
  }

  Future<void> _reconnect() async {
    log.config('Lost connection to MongoDB - reconnecting...');
    await close();
    await open().then((_) => log.config('Reconnected to MongoDB'));
    return whenReady;
  }

  // -- passthrus

  @override mongo.Db?            get authSourceDb                          => _delegate.authSourceDb;
  @override                      set authSourceDb(mongo.Db? _authSourceDb) => _delegate.authSourceDb = _authSourceDb;

  ... etc, etc, etc for 36 other getters/setters/methods ...

  @override mongo.DbCollection collection(String collectionName) => MongoCollection(this, collectionName);
}

class MongoCollection implements mongo.DbCollection {
  final MongoDbImpl        _db;
  final mongo.DbCollection _delegate;

  MongoCollection(this._db, String collectionName): _delegate = _db._delegate.collection(collectionName);

  Future<void> get whenReady => _db.whenReady;

  Future<void> _reconnect() {
    return _db._reconnect().then((_) => this.whenReady);
  }

  Stream<T> _tryWithReconnectStream<T>(Stream<T> Function() computation) async* {
    try {
      yield* computation();
    } on mongo.MongoDartError catch (e) { // ignore: avoid_catching_errors
      if (e.message == 'No master connection') {
        await _reconnect();
        yield* _tryWithReconnectStream(computation);
      } else {
        rethrow;
      }
    }
  }

  Future<T> _tryWithReconnectFuture<T>(Future<T> Function() computation) async {
    try {
      return computation();
    } on mongo.MongoDartError catch (e) { // ignore: avoid_catching_errors
      if (e.message == 'No master connection') {
        await _reconnect();
        return _tryWithReconnectFuture(computation);
      } else {
        rethrow;
      }
    }
  }

  // -- passthrus

  @override Future<Map<String, dynamic>?> findOne([dynamic selector]) => _tryWithReconnectFuture(() => _delegate.findOne(selector));

  @override Stream<Map<String, dynamic>> legacyFind([dynamic selector]) => _tryWithReconnectStream(() => _delegate.legacyFind(selector));

  ... etc, etc, etc for 45 other getters/setters/methods ...
}

Obviously, this won't work for everyone, (besides being a ton of work to override every single method), since it'll just indefinitely continue trying to reconnect, but maybe it can provide some inspiration. (it also implements a whenReady completer so your main script can do something like await db.whenReady)

@giorgiofran
Copy link
Contributor

Thanks for your suggestion. My idea is to take example from tne Nodejs driver and manage messaging on the status and auto-reconnect based on a polling check every 2-3 seconds.
I have started recently to give a look to this solution, but I have a few time in this period, so it will take a long time :-(

@JohnF17
Copy link

JohnF17 commented Mar 29, 2023

Any news on this?, I've come to realise my implementation isn't that great😓, well one of them at least but I still need a better one.
If it not much of a bother, a simple explanation on how to do it better or a short code snippet would be highly appreciated, I'd be glad to see how y'all approached it.

@giorgiofran
Copy link
Contributor

Give a look to discussion #228 . Maybe it could help.

@FillipMatthew
Copy link

Give a look to discussion #228 . Maybe it could help.

Would it be a good idea to use those reconnect methods in a Middleware for a web server to verify connection before each api call and return an internal server error or something on failure to reconnect? Or is it better before each DB call like they use it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants