[Revision Ideas] DWX_ZeroMQ_Server_v2.0.1_RC8.mq4

[reppolice]

Hi,

I created an 8GB RAM VM for this, which has the latest Windows 10 Enterprise , live MT4 and Python 3.7 32 bit (I suspect with 0mq there is no difference between python 32 bit and 64). The VM itself has given me no weirdness so far. With this new DWX I am getting 99% of the time problems, though they do not seem 100% reproducible. I got no messages at all in "Experts" once or twice, or I got just the Binding message, or I got pricings for a few ticks and then it was gone(not stopped or crashed, it just wasn't printing any more prices). Restarting the computer seems to take a long time to shut MT4 down, and on my current run I only got the Binding messages and decided to turn on the "Live trading" property of the EA, which seems to have crashed my MT4 (Not responding for 10 minutes or more).

Somewhere in the middle of all this the EA triggered Windows Firewall (do you want to allow...) to which I replied yes, of course. Sometimes turning the firewall off helps with such issues, but I also suspect the server could exit/crash more gracefully. I will probably try on Windows 7 later. From other comments it is clear previous versions run OK for some people, no comments yet on the new one.

Finally it is Friday and the ticks are few but not that few!

Thanks!

[reppolice]

Exact same behavior in Windows 7. Hard to believe it is not some kind of bug.

[TimKabue]

It has been a while before I used ZeroMQ but I can distinctly remember a situation where I would fail to receive quotes in Python. However, it was always resolved by restarting my MQL4 program and Python program.
I will take a look at my past projects and give more helpful insight.

[talaikis]

I have worked previously with MT4 and abandoned it. The platform is unable to handle even hourly quotes without freezing for 100+ instruments, and ticks, obviously, require more resources.

[integracore2]

Hi @reppolice,

Thank you for bringing this to our attention, we'll try and reproduce the scenario at our end.

Could you please confirm more precisely the exact sequence of events that lead to the behaviour you experienced, how many symbols were being published, etc

We've tested this with 28 simultaneous symbols publishing from MetaTrader 4 simultaneously with some cases where ticks would drop.

This issue has in fact raised the need to for a test log for this here on GitHub - so thank you for being thorough and raising the flag!

What we need to ascertain is whether this is an issue with the Python script or MetaTrader's abilities as regards handling inbound requests / processing - should the latter be the case, we'll probably need to stress test this more and make the process known more clearly here for everyone's benefit.

[reppolice]

I did not change anything from your README and tutorials, so the symbols were this long list you have in the example file, perhaps 28. I am trying to figure out how to retry in some more predictable and documented way, but meanwhile here's an idea. What if you published a "known working" configuration, in increasing difficulty: 1) a portable MT4 zip with the bridge and example installed. 2) A Windows VM with a working demo 3) A Linux VM with the demo in Wine

[Steviey]

I guess it's all about the right sockets choosen. I played a lot with the script (incl. strategy tester).
Meanwhile I defined MT4 as client and R as server (sockets changed). This makes much more sense for me and it works! Even now, I'm able to shut down the server via client deInit-handler and ZeroMQ. Anyways, MT4 and the original concept of the server script could be discussed. Like xenu256, I'm not convinced yet, whether MT4 is the right platform, for such approaches- in general.

[reppolice]

I guess it's all about the right sockets choosen. I played a lot with the script (incl. strategy tester).
Meanwhile I defined MT4 as client and R as server (sockets changed). This makes much more sense for me and it works! Even now, I'm able to shut down the server via client deInit-handler and ZeroMQ. Anyways, MT4 and the original concept of the server script could be discussed. Like xenu256, I'm not convinced yet, whether MT4 is the right platform, for such approaches- in general.

Are you hinting at MT5, or altogether at greener pastures? I don't know if you went down the "portable MT4" path, the only sensible way to deploy code if you need to move it around between testing and release environments, for example, but certainly the world could use a working template.

[Steviey]

@reppolice
I'm hinting/recoding at MT4. As I said, I changed the complete concept. Not sure if this is worth it.
I identified two major problems (in personal context).

1. kind of sockets
2. changing from onTimer() to onTick() - handler provides strategy tester compatibility.

Since I turned MT4 into a client and refactored nearly everything, a pull request doesn't makes sense anymore.

[reppolice]

so MT4 as a client " MT4 is the right platform, for such approaches- in general."? I thought you denounced MT4 altogether for this bridge-automation. Anyway, if it made sense for you, it may make sense for Darwinex devs. Of course when Darwinex says it is running OK-ish for them and it fails me on both Win 10 and Win 7, all I can do is scratch my head :)

[Steviey]

@reppolice Yes, perhaps it makes sense for them too. Another question would be, what about the 'one-tick-data' of 28 symbols? Elaborated algo-strategies will use extensive history data per symbol to trade. Maybe one should use this 'one-tick-data' as scanning service. I don't know. But if so, this functionality should resist in a separate modul. This would keep the template much cleaner and reject a possible missleading focus on HFT with MT4. There are also to less exceptions. Socket-programmings need an extensive amount of exceptions (incl. informative warnings). These are essential for this kind of programmings. And connections have to be terminated correctly. Anything else is a nightmare :-). Last but not least, in my opinion, program logic should rely on external languages and the MT4 script should focus on IO and a maximum clean documentation of that. If everything is clean and separated, a broader community could participate and contribute.

[integracore2]

@reppolice and @Steviey

Guys, this issue has turned into an amazing thread of extremely constructive feedback - thank you so much for your participation, everything you've discussed/shared here has been duly noted!

There is certainly a strong case for separating the market data element from the MQL implementation.

Do bear in mind that this bridge was not intended to be used for backtesting - compatibility with the MT Strategy Tester was therefore not considered in this iteration.

As regards onTick() vs onTimer():

1. The choice of onTimer() over onTick() is simply to not have a pending order sent to MT to experience intra-tick latency, especially at lower liquidity hours of the day.

2. The market data publishing aspect of the EA likely requires a shift in thinking -> from iterating through the entire list of desired symbols as it is now every tick, to something else.

3. We've considered (and are working on a Python implementation of) an entirely different alternative: external scripts (resident on the same host machine) reading flat HST data from MetaTrader history directories instead of MetaTrader actively publishing the data each time a tick is recorded on the symbol the EA is deployed on (and creating the uncertainties some users are experiencing with hangups etc)

This would also open up the realm of backtesting to be kept inside external scripts entirely, with no dependency on the MetaTrader terminal itself, just its directory base and access rights to files therein.

Once again, please keep this feedback coming - extremely helpful indeed! 😃 👍

[Steviey]

@integracore2 Good answer, I'm happy you guys keep care of it and glad to hear that!
Pleas keep in mind, not everyone prefers Python as the 'logic basis' for algo-tradings.
There is a big R-community in quant environments too. I can code both, but see certain advantages over Python by R. There are also C#-devs out there. It would be nice, to keep the server script open for these platforms. This would provide a much larger user base and more interested, possible members of DarwineX. If the next iteration of the server script keeps compatible- I may would contribute a corresponding R-client/server. In the longer run a MQL5-branch would make sense too. Thank you in advanced so far.

[integracore2]

Hi @Steviey ,

Yes, totally agree regarding additional programming languages being facilitated 👍

The plan is to make most implementations that we publish on GitHub, available in Python, R, C#/C++ and Java.

The first release of anything will almost always be in Python or R to enable extensive user-testing and incorporating any feedback as necessary - as demonstrated by how this thread has turned out so far, simply fantastic! 😃

Regarding the Server EA again -> I think the solution we discussed earlier about separating the market data publication entirely is the best way forward.

We've written the necessary code since yesterday to achieve this (in Python for now) - ready data directly from MetaTrader's history directory - and are currently testing out the use-case thoroughly prior to sharing it here.

* It would however require the user to ensure that all symbols for which up-to-date is required, have their corresponding charts open in the MetaTrader terminal to ensure the history database files remain updated (a seemingly unavoidable one-off at this time)

Will keep you posted!

And thank you so much for your interest in providing an R version down the line, that's awesome 👏

We'll make sure its referenced as yours and acknowledged as your contribution by our entire user-base 🙏

[kuhtooie]

Has there been any consideration to laying a GraphQL server over MT? No matter which language implements it that should effectively open it up to any language that supports the API.
Live data, Trades, and historical data could each have their own tag in the schema.

[kuhtooie]

I am extremly interested in adding this library to the following framework, or one of it's peers:
https://github.com/backtrader/backtrader

Any thoughts on the subject, or individuals also interested in the idea, are welcome.