Skip to content

Commit

Permalink
moves parallel to enhances
Browse files Browse the repository at this point in the history
  • Loading branch information
shikokuchuo committed Nov 1, 2023
1 parent 6621efc commit 6b22b25
Show file tree
Hide file tree
Showing 3 changed files with 53 additions and 53 deletions.
6 changes: 3 additions & 3 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Package: mirai
Type: Package
Title: Minimalist Async Evaluation Framework for R
Version: 0.11.0.9021
Version: 0.11.0.9022
Description: Lightweight parallel code execution and distributed computing.
Designed for simplicity, a 'mirai' evaluates an R expression asynchronously,
on local or network resources, resolving automatically upon completion.
Expand All @@ -23,9 +23,9 @@ Encoding: UTF-8
Depends:
R (>= 3.5)
Imports:
nanonext (>= 0.10.2.9021),
parallel
nanonext (>= 0.10.2.9024)
Enhances:
parallel,
promises
Suggests:
later,
Expand Down
4 changes: 2 additions & 2 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# mirai 0.11.0.9021 (development)
# mirai 0.11.0.9022 (development)

* Adds 'mirai' method for 'as.promise()' from the {promises} package (if available). This functionality is merged from the package {mirai.promises}, allowing use of the promise pipe `%...>%` with a 'mirai'.
* Parallel clusters (the alternative communications backend for R) now work with existing R versions, no longer requiring R >= 4.4.
Expand All @@ -12,7 +12,7 @@
* Fixes supplying an `ssh_config()` specifying tunnelling to the 'remote' argument of `daemons()`.
* Fixes the print method for a subset 'miraiCluster' (thanks @HenrikBengtsson #83).
* Removes the deprecated deferred evaluation pipe `%>>%`.
* Requires nanonext >= [0.10.2.9019].
* Requires nanonext >= [0.10.2.9024].

# mirai 0.11.0

Expand Down
96 changes: 48 additions & 48 deletions vignettes/mirai.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -69,15 +69,15 @@ Upon completion, the 'mirai' resolves automatically to the evaluated result.

```r
m$data |> str()
#> num [1:100000000] 57.804 -1.941 -7.661 0.387 0.777 ...
#> num [1:100000000] 0.196 2.357 -6.271 0.685 2.239 ...
```

Alternatively, explicitly call and wait for the result using `call_mirai()`.


```r
call_mirai(m)$data |> str()
#> num [1:100000000] 57.804 -1.941 -7.661 0.387 0.777 ...
#> num [1:100000000] 0.196 2.357 -6.271 0.685 2.239 ...
```

For easy programmatic use of `mirai()`, '.expr' accepts a pre-constructed language object, and also a list of named arguments passed via '.args'. So, the following would be equivalent to the above:
Expand All @@ -94,7 +94,7 @@ args <- list(m = runif(1), n = 1e8)
m <- mirai(.expr = expr, .args = args)

call_mirai(m)$data |> str()
#> num [1:100000000] 0.123 0.733 0.649 1.472 5.71 ...
#> num [1:100000000] 0.953 0.357 -1.377 -0.939 1.481 ...
```

[&laquo; Back to ToC](#table-of-contents)
Expand Down Expand Up @@ -221,12 +221,12 @@ status()
#>
#> $daemons
#> i online instance assigned complete
#> abstract://2c91f63f732bdd18c473ec80 1 1 1 0 0
#> abstract://3b690e927a6d8d9be7b8eb64 2 1 1 0 0
#> abstract://2c4aa81c2d6e20d10998049f 3 1 1 0 0
#> abstract://4521e936800ac3da2a5cce9e 4 1 1 0 0
#> abstract://96c4b425a2eda1175e24a33d 5 1 1 0 0
#> abstract://12f6dc8cdb6aa4fabed7300e 6 1 1 0 0
#> abstract://d0acbf797fca7d8ad57ce6db 1 1 1 0 0
#> abstract://9193e16129b32a48f5af1953 2 1 1 0 0
#> abstract://4c9c493f1b310490627d4809 3 1 1 0 0
#> abstract://52f8135cb023b13b443da89a 4 1 1 0 0
#> abstract://31cbcb2b9ff93c123c110b51 5 1 1 0 0
#> abstract://4423774e68363970ddb3578c 6 1 1 0 0
```

The default `dispatcher = TRUE` creates a `dispatcher()` background process that connects to individual daemon processes on the local machine. This ensures that tasks are dispatched efficiently on a first-in first-out (FIFO) basis to daemons for processing. Tasks are queued at the dispatcher and sent to a daemon as soon as it can accept the task for immediate execution.
Expand Down Expand Up @@ -261,7 +261,7 @@ status()
#> [1] 6
#>
#> $daemons
#> [1] "abstract://d5a7ab71b50c27f6e81196d0"
#> [1] "abstract://f452a388caafaa22825cb163"
```

This implementation sends tasks immediately, and ensures that tasks are evenly-distributed amongst daemons. This means that optimal scheduling is not guaranteed as the duration of tasks cannot be known *a priori*. As an example, tasks could be queued at a daemon behind a long-running task, whilst other daemons are idle having already completed their tasks.
Expand Down Expand Up @@ -352,7 +352,7 @@ By specifying `dispatcher = FALSE`, remote daemons connect directly to the host

```r
daemons(url = host_url(), dispatcher = FALSE)
#> [1] "tcp://hostname:35033"
#> [1] "tcp://hostname:42189"
```

Note that above, calling `host_url()` without a port value uses the default of '0'. This is a wildcard value that will automatically cause a free ephemeral port to be assigned. The actual assigned port is provided in the return value of the call, or it may be queried at any time via `status()`.
Expand All @@ -368,7 +368,7 @@ status()
#> [1] 0
#>
#> $daemons
#> [1] "tcp://hostname:35033"
#> [1] "tcp://hostname:42189"
```

To reset all connections and revert to default behaviour:
Expand Down Expand Up @@ -445,10 +445,10 @@ daemons(n = 2, url = host_url())

launch_remote(1:2)
#> [1]
#> Rscript -e "mirai::daemon('tcp://hostname:40815',rs=c(10407,-1298605964,-1613285147,639709282,1751824507,-1051788928,-932623487))"
#> Rscript -e "mirai::daemon('tcp://hostname:36925',rs=c(10407,12447993,776969318,-1122538577,132652580,1015805653,-1905100398))"
#>
#> [2]
#> Rscript -e "mirai::daemon('tcp://hostname:39985',rs=c(10407,1124331931,1808224896,-1742832536,-639982197,-1545947632,1369966981))"
#> Rscript -e "mirai::daemon('tcp://hostname:34367',rs=c(10407,-1677977516,618003857,-1886215845,-1504767133,2041967051,556928152))"

daemons(0)
#> [1] 0
Expand Down Expand Up @@ -480,37 +480,37 @@ The generated self-signed certificate is available via `launch_remote()`. This f
```r
launch_remote(1)
#> [1]
#> Rscript -e "mirai::daemon('wss://hostname:44085/1',tls=c('-----BEGIN CERTIFICATE-----
#> Rscript -e "mirai::daemon('wss://hostname:32953/1',tls=c('-----BEGIN CERTIFICATE-----
#> MIIFNzCCAx+gAwIBAgIBATANBgkqhkiG9w0BAQsFADAzMREwDwYDVQQDDAhrdW1h
#> bW90bzERMA8GA1UECgwITmFub25leHQxCzAJBgNVBAYTAkpQMB4XDTAxMDEwMTAw
#> MDAwMFoXDTMwMTIzMTIzNTk1OVowMzERMA8GA1UEAwwIa3VtYW1vdG8xETAPBgNV
#> BAoMCE5hbm9uZXh0MQswCQYDVQQGEwJKUDCCAiIwDQYJKoZIhvcNAQEBBQADggIP
#> ADCCAgoCggIBAJfcvcHmr4Io9XXaE/2T7QAEBoxtdPZxF30hgu5wjNJA5d2SrKzY
#> SQRTmwTT4yTjFIn6D7Z12g1blK7cPZ7IQbD53z76jJe0ySMQA/7q4koYtLxhIUfm
#> ISoBkxZs1uN91cM0epVEcr+NBiqYBbwXS1KfBPPyxJA4/pBnyOA+X/bFXJEj6ldX
#> 6ahWNHuMPrXdrI8T1YZX7ox+kW/PqwHdVdbrniAC7Z/IFOrjA8h7fxVuHnFcZ0Nv
#> yYV2fWTiBIZg9YnRsmCDt7uUPQg+P2JcpgNRJd4Y5dlQRjL1hF1gNld5lkRwiIE0
#> 1n9Sm7ep+BUKRqNJPriT8hRcXSD0p+vsUmr0i81uUJNWEr1iOZDKkr03pZgsuaA7
#> 9d+tEaq8mgrwsrYBw3+VPFu1gzPumZztkmPTw6DQfHGA5ivcmj7q7fEWgAsLOZxi
#> 0IZ3PKBdbvrvV1XZtP7wdH4d8E/+ZDHAy0VnugAjEPRIgsV0liCK6KLpUWyOn/WG
#> 6oBMpWg2mqhtomN6vO/k0l8muHuFgHOyVIoerrc6OeTPbfUWVXASj4BnMJ+9xN9+
#> wVNUwEYp58X1NQxvVmuaGx9gsxoCmeR48AoLE1ehee6vlIsebX5Ec+Z8ntA7a+ks
#> T6YxZUvigddaTinNZ/ZXuPTy84cb9ixK6q7KkWz1htauADW7auQHPIDJAgMBAAGj
#> VjBUMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFJT2qxTuAnDrnwTdzs40
#> 4fGJBdHQMB8GA1UdIwQYMBaAFJT2qxTuAnDrnwTdzs404fGJBdHQMA0GCSqGSIb3
#> DQEBCwUAA4ICAQA3IFY7QadfMJS8NxUYTQD7FJa2eOICE9pog+gxRXFXkcV+xXVJ
#> oJ7EnG87vwvH96GOvrV5Q9882zA6tecXgIYCFYKGlOXTVw0OYE80yxIZvEuMgFdi
#> oDIc6xgfJ8xvKfx2c9ePEhRQQqm2c9jxeFsvzqSBtKq/LEBFvD3ZgFv3I7mQjaO9
#> MmfCzOkNshq7F9G3U39k+k4+Q4f+nk6smtZQbDWrBnAUxUd2jvgO0fk0WR6DjwSI
#> Ji+gjOqGYFc85mPcFGmtAFbaAe/FfhPyLHvVSMMfqjIDDR5oyYkXTk8eCd2rYbsE
#> I5AmAsr86R9HOA5fjmA0/iV0AZPiLZlAzVqsPaTbVtd7hwvLsro2IyYvrdPwNTHO
#> FOaCnIk6xaQaud0ICY3/RmBRVCYi7tbuZfSf5khPvRrC/6FMrQFT442Pq9ZMRaF2
#> 7b7lbdjfX+JCnR5xKbVS4aw/pJrtG/D2z+I5qMeCLJ4MqJNECE6kxC/w1Iq3N4iN
#> Cd+gbstgHEAihS+ySife7uv4gbWu++xOESNJ4RRNZmksjhddRkQfy+nCMsFBbisC
#> T5JL43Jx2ASZXEaOPX7Q3URXO5xgTRTkFP/or10s4g2BaUya71AQheD6Z5uc/6kp
#> EaPys8D8HzD27GQ/ASLDeNlf5bTJaDhx/uzb8Q9KX2GA0x6gwKfGb+iX1w==
#> ADCCAgoCggIBALYs88dY5lNbd7GA06s4pfsGIZYxSkLcC+pqQUol5vTksiQWgSJd
#> t+Amof6wdgEeHEtrN+w4P7Z+mOSpbobd8vOro5Vr+iZdMuh99gy5aleQvb0Yjmh1
#> crOD+CeLYW0QXEU8q44ZicIZ+JyMmpyEqgtBD1lvqSPzwXDx3URvNjkujByvVaUk
#> gj+u/XlrSGitbScR9CO50cQwK+eQ625FH/ur1NgC10vSajKGF91NjKwfpBl7t5Po
#> hQzGVjKq+87o8WSY37EZ3MRvGh5pKRrm7kP2eSd3rtSS0EkN8D5yr4iqWcr8SZcv
#> 2mYQ5KGTyv6+Gvsy79PIA0jc2rVj4sS7TgOsrYYIGqkJlPVRGbiiyNtk00veptPc
#> bOevkBnG3hPvgYMGK6nRKNAP9IqHwAym+CW7JaWKKQrp5uuIv9Z3K/W5LCqS7qwY
#> U+sNBlAesRJfUzQYlOK7jiUfBMl5YRKXc54Tp0KcDQ+7NM9j9rd69Bvb6YK1I4lA
#> EAI/P2AN+su6pyPnOYeUCO5AS1Q6dnUWKbDeYjJc7xzPClrmtQ4/H+NzQbYTps2S
#> p0C1fxcTpo79SP7L19tHTjsPRafQ8CBavEI8EQwfngBkkeZ6QLefZUgEU+/4wkec
#> 5m972an+IjfCunkhzEKlWqRhAROXNx0IUWiqoZFWrTZ24Ra2ppH022jrAgMBAAGj
#> VjBUMBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFGEzbYVYUZemrBN4XC2d
#> HxUiwJ2DMB8GA1UdIwQYMBaAFGEzbYVYUZemrBN4XC2dHxUiwJ2DMA0GCSqGSIb3
#> DQEBCwUAA4ICAQCT1qUa+qZ0xEdwl/5xYL8tHqZ2Qn6eKk6XeIVjC/aJ/0ihLrG9
#> b0gDYSZwcSEuNP6Tt45bm/9mrHCTSwg1qjbE4t7japhkAc4Qq0+m+ncSeTUjeBhI
#> PvdqPJUzXrgQ3n2gKyMpNxSpFmN/84PkVMfSGeorAmG9WMMH6sakR6fmphKEnV20
#> nF1j396ZY4yEJtfJCR6u/ZUKtyz0lkAp/mRNqUbDygvDGgkrOMaULTVtnutT/CsB
#> 0g8pgDRJrYp5BoFatC44yRQq9aK6G1TAh0lROPIm2LvN1otkXOabDmfO/TKd9+Ol
#> xJn66a+EzW1oLiVMF7SCntiAsg6mx5gotsEuIhyto1ihe7IxdHgu2eqCt4pE1rEw
#> aC5/aDKYkfHNVAeeIzPUklS7SNrJHw2Ou66juVVWmQBAeTkPf6ZggN9W4tEV5oqv
#> bEdYGvJ6HiFnferXjkk+xwkeVDJ+c50R9z+qfBmVXU2raS9zUo8B4XAxgSkkt2/P
#> Xrokv8iO9c7qrPsRqh0tGnetYgR4ProHSORwbbaX++2b89R8LAfBExu6oCh1ENNh
#> hFbJR2VjNcZyeDXa6BZ+o/mITobStzAzSWzqEshEWZF4Xc/wD+45lKNSj2uFCXtL
#> gb+xtSJx4wiSbiihSwgS4v/OfIIGx8gwhdR87kaVnYc66yCj+xcj6mwyiw==
#> -----END CERTIFICATE-----
#> ',''),rs=c(10407,1009925696,-5957311,851346958,-714207433,-1893795316,680318877))"
#> ',''),rs=c(10407,1573322550,847649855,1628623220,-2111267355,-395308702,-469637765))"
```

The printed value may be deployed directly on a remote machine.
Expand Down Expand Up @@ -634,36 +634,36 @@ cl

parallel::parLapply(cl, 1:5, rnorm)
#> [[1]]
#> [1] -0.9836203
#> [1] -2.100108
#>
#> [[2]]
#> [1] 1.0656311 0.5503744
#> [1] -1.204032 -1.042151
#>
#> [[3]]
#> [1] 0.2525613 -0.8804021 0.5376508
#> [1] 0.1248739 1.1350673 -0.1112634
#>
#> [[4]]
#> [1] -1.968415e-05 1.351948e+00 1.176134e+00 1.551765e+00
#> [1] 0.6114177 -0.8331200 -0.1468930 -0.4842804
#>
#> [[5]]
#> [1] -0.5270231 0.4128853 -0.1984081 0.8376930 -1.1472594
#> [1] -0.7784605 -0.7929214 -1.2540343 -1.0441703 0.3732057

status(cl)
#> $connections
#> [1] 4
#>
#> $daemons
#> [1] "abstract://757f162f5903b2ccf58e3782"
#> [1] "abstract://193e22ffb1ef8c82ced3ff97"

stop_cluster(cl)

cl <- make_cluster(n = 2, url = host_url())
#> Shell commands for deployment on nodes:
#> [1]
#> Rscript -e "mirai::daemon('tcp://hostname:39797',rs=c(10407,-1845852469,625033104,1695168081,-1539269666,1803884231,-226338724))"
#> Rscript -e "mirai::daemon('tcp://hostname:33829',rs=c(10407,1306966007,-787489844,329801309,-1459926662,1567557811,2114850392))"
#>
#> [2]
#> Rscript -e "mirai::daemon('tcp://hostname:39797',rs=c(10407,-951338746,-573234744,-1153439598,1348855247,1553839924,-21456304))"
#> Rscript -e "mirai::daemon('tcp://hostname:33829',rs=c(10407,1230136452,360255358,885527131,991567976,-445341772,345044727))"

stop_cluster(cl)
```
Expand Down

0 comments on commit 6b22b25

Please sign in to comment.