zeromq + julia examples

[joearms]

Hello, I'm a total beginner as regards Julia.

I want to do the following: send a 3x3 matrix of integers from Erlang to Julia
invert the matrix and sent it back. I want to use zeroMQ as the transport.
Assume each integer is an unsigned 32 bit integer, and that I send the integers
in network byte order.

Assume the size of the matix (NxM) is encoded as two unsigned bytes

To invert the matrix I send 3,3,X11,X12,X13, ... etc. where the X's are encoded as
unsigned 32 bit integers in network byte order.

How do I get the data in and out of Julia.

A simple example of this would be a great help.

The encoding I used above (NxM) encoded as two bytes is pretty ad.hock.
is there a generic way of doing this?

Should I send Julia code "over the wire" and "eval" it?

Cheers

/Joe

[jiahao]

This is an interesting question, but this is not really a codebase issue and I suspect you will get an answer if you post this question to the julia-users mailing list.

[Keno]

I'm not entirely sure what you're trying to accomplish. Do you have a Julia and an Erlang process that you want to have communicate (i.e. two programs that are communicating) or a single Julia server process that Erlang processes connect to and send code and data to? What kind of ZMQ sockets are you using (I'm not an expert on ZMQ, but I know there are several different modes of operation which work slightly different)?

[timholy]

A long time ago I wrote something for communication between Julia and MATLAB. I'm sure it doesn't work anymore, but you could use it as a model. It targets the Julia serializer; surely, a much better solution would be to use JSON (for which there was no Julia support at the time I wrote that). Even better would be to target a more efficient representation than JSON; you can probably find old mailing list discussions about alternatives like Thrift, MessagePack, or Google Protocol Buffers. For just plain data (not code), MessagePack would probably be pretty easy, and I see there's already a start here.

[timholy]

I think the IPC parts of ZeroMQ.jl got ripped out, but I can probably find them on my hard drive somewhere.

[StefanKarpinski]

Hi, Joe. You'd want to do something along the lines of

using ZMQ

ctx = Context(1)
s = Socket(ctx, REP)
ZMQ.bind(s, "tcp://*:5555")

while true
  msg = ZMQ.recv(s)
  io = seek(convert(IOStream, msg), 0)
  m = read(io, Uint8)
  n = read(io, Uint8)
  A = map(ntoh, read(io, Float64, m, n))
  @show A
  B = pinv(A)
  ZMQ.send(s, Message(map(hton,B)))
end

A Julia client for this server would look like this:

using ZMQ

ctx = Context(1)
s = Socket(ctx, REQ)
ZMQ.connect(s, "tcp://localhost:5555")

function remote_pinv(A::Matrix{Float64})
  m, n = size(A)
  io = IOBuffer()
  write(io, uint8(m), uint8(n), map(hton,A))
  ZMQ.send(s, Message(io))
  msg = ZMQ.recv(s)
  io = seek(convert(IOStream, msg), 0)
  map(ntoh, read(io, Float64, n, m))
end

This highlights a bunch of interface problems with the ZMQ module that I should open issues for, but it gets the job done. A couple of notes:

• The inverse of an integral matrix typically won't be integral, so I changed the element type to Float64.
• Since the matrix inverse isn't defined for general rectangular matrices, I used the pseudo-inverse (i.e. pinv), which is.
• The response message doesn't send dimensions since the pseudo-inverse of an m-by-n matrix will always be n-by-m.

It's easy to implement a general serialization protocol along these lines – we do it ourselves in base/serialize.jl, but that serialization assumes that both ends are Julia processes and moreover that they're the same version of Julia – or at least that they agree about "magic numbers" that represent core Julia types.

It would be really cool to see some good Erlang/Julia interop. Drop me a line if you want to discuss.

[StefanKarpinski]

I just opened an issue about some of the problems with the Message type interface: #46.

[joearms]

Thanks Stefan I'll drop you a line.

I'm not sure how interop with Erlang would work out.

With my erlang hat on I'd say (In erlang)

 Julia ! "inv([1 2 3; 4 5 6; 7 8 9])"
 receive
       M -> ...
 end

I'd send you some Julia "down the wire" and you'd send me back the result

• "down the wire" means "via zeroMQ" - why zeroMQ - because a lot of people find it
  easy to use.

Cheers

/Joe

[stevengj]

Ideally, you would call Julia directly via the C API, using Erlang's NIF or port driver interface, and something like PyCall for argument conversions and marshalling. Sending programs and data back and forth as strings is a lot of overhead.

[joearms]

I'm imagining the case were julia is on a remote machine. I want to control
large numbers
of remote Julias :-)

/Joe

On Tue, Feb 4, 2014 at 8:23 PM, Steven G. Johnson
[email protected]:

Ideally, you would call Julia directly via the C API, using Erlang's NIF
or port driver interface, and something like PyCall for argument
conversions and marshalling. Sending programs and data back and forth as
strings is a lot of overhead.

Reply to this email directly or view it on GitHubhttps://github.com//issues/44#issuecomment-34096465
.

[ViralBShah]

Closed by #225. Could always use better examples, but at least there's a pointer. We could also point to other packages that use ZMQ.jl.