Rewrite process internals

.github/workflows/elixir.yml

@@ -7,7 +7,7 @@ env:
   ELIXIR_VERSION: "1.14.0"
 jobs:
   lint:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     name: Lint OTP ${{matrix.otp}} / Elixir ${{matrix.elixir}}
     strategy:
       matrix:
@@ -23,7 +23,7 @@ jobs:
 
       - name: Cache Dependencies
         id: mix-cache
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: |
             deps
@@ -42,26 +42,25 @@ jobs:
       - run: mix deps.get
       - run: mix deps.unlock --check-unused
       - run: mix format --check-formatted
+      - run: (cd go_src/ && test -z $(gofmt -l .))
       - run: mix compile --warnings-as-errors
       - run: mix credo --strict
       - run: mix dialyzer --plt
 
   linux:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     name: Linux OTP ${{matrix.otp}} / Elixir ${{matrix.elixir}}
     strategy:
       matrix:
         include:
-          - elixir: 1.12.x
-            otp: 23.x
-          - elixir: 1.13.x
-            otp: 24.x
           - elixir: 1.14.x
-            otp: 25.x
+            otp: 24.x
           - elixir: 1.15.x
-            otp: 26.x
+            otp: 25.x
           - elixir: 1.16.x
             otp: 26.x
+          - elixir: 1.17.x
+            otp: 27.x
     steps:
       - uses: erlef/setup-beam@v1
         with:
@@ -71,7 +70,7 @@ jobs:
 
       - name: Cache Dependencies
         id: mix-cache
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: |
             deps
@@ -89,7 +88,7 @@ jobs:
       - run: mix test --trace
 
   macos:
-    runs-on: macos-11
+    runs-on: macos-14
     steps:
       - uses: actions/checkout@v4
       - uses: DeterminateSystems/nix-installer-action@main
@@ -100,23 +99,23 @@ jobs:
       - run: nix develop --command mix test --trace
 
   windows:
-    runs-on: windows-2019
+    runs-on: windows-latest
     name: Windows OTP ${{matrix.otp}} / Elixir ${{matrix.elixir}}
     strategy:
       matrix:
         include:
-          - elixir: "1.14"
-            otp: "25"
+          - elixir: "1.17"
+            otp: "27"
     steps:
       - uses: erlef/setup-beam@v1
         with:
           otp-version: ${{matrix.otp}}
           elixir-version: ${{matrix.elixir}}
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
 
       - name: Cache Dependencies
         id: mix-cache
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: |
             deps

README.md

@@ -4,98 +4,149 @@
 [![Hex.pm](https://img.shields.io/hexpm/v/ex_cmd.svg)](https://hex.pm/packages/ex_cmd)
 [![docs](https://img.shields.io/badge/docs-hexpm-blue.svg)](https://hexdocs.pm/ex_cmd/)
 
+ExCmd is an Elixir library for running and communicating with external programs using a back-pressure mechanism. It provides a robust alternative to Elixir's built-in [Port](https://hexdocs.pm/elixir/Port.html) with improved memory management through demand-driven I/O.
 
-ExCmd is an Elixir library to run and communicate with external
-programs with back-pressure mechanism. It makes use os backed stdio
-buffer for this.
+## The Port I/O Problem
 
-Communication with external program using
-[Port](https://hexdocs.pm/elixir/Port.html) is not demand driven. So
-it is easy to run into memory issues when the size of the data we are
-writing or reading from the external program is large. ExCmd tries to
-solve this problem by making better use of os backed stdio buffers
-and providing demand-driven interface to write and read from external
-program. It can be used to stream data through an external
-program. For example, streaming a video through `ffmpeg` to serve a
-web request.
+When using Elixir's built-in [Port](https://hexdocs.pm/elixir/Port.html), running external programs that generate large amounts of output (like streaming video using `ffmpeg`) can quickly lead to memory issues. This happens because Port I/O is not demand-driven - it consumes output from stdout as soon as it's available and sends it to the process mailbox. Since BEAM process mailboxes are unbounded, the output accumulates there waiting to be received.
 
-Getting audio out of a video stream is as simple as
+### Memory Usage Comparison
 
-``` elixir
-ExCmd.stream!(~w(ffmpeg -i pipe:0 -f mp3 pipe:1), input: File.stream!("music_video.mkv", [], 65336))
-|> Stream.into(File.stream!("music.mp3"))
-|> Stream.run()
+Let's look at how ExCmd handles memory compared to Port when processing large streams:
+
+Using Port (memory grows unbounded):
+```elixir
+Port.open({:spawn_executable, "/bin/cat"}, [{:args, ["/dev/random"]}, {:line, 10}, :binary, :use_stdio])
 ```
 
-### Major Features
+![Port memory consumption](./images/port.png)
+
+Using ExCmd (memory remains stable):
+```elixir
+ExCmd.stream!(~w(cat /dev/random))
+|> Enum.each(fn data ->
+  IO.puts(IO.iodata_length(data))
+end)
+```
 
-* Unlike beam ports, ExCmd puts back pressure on the external program
-* Stream abstraction
-* No separate shim installation required
-* Ships pre-built binaries for MacOS, Windows, Linux
-* Proper program termination. No more zombie process
-* Ability to close stdin and wait for output (with ports one can not selectively close stdin)
+![ExCmd memory consumption](./images/ex_cmd.png)
 
+ExCmd solves this by implementing:
+- Demand-driven I/O with proper back-pressure
+- Efficient use of OS-backed stdio buffers
+- Stream-based API that integrates with Elixir's ecosystem
 
-## Examples
+## Key Features
+
+- **Back-pressure Support**: Controls data flow between your application and external programs
+- **Stream Abstraction**: Seamless integration with Elixir's Stream API
+- **Memory Efficient**: Demand-driven I/O prevents memory issues with large data transfers
+- **Cross-platform**: Pre-built binaries for MacOS, Windows, and Linux
+- **Process Management**: Proper program termination with no zombie processes
+- **Selective I/O Control**: Ability to close stdin while keeping stdout open
+- **No Dependencies**: No separate middleware or shim installation required
+
+## Installation
+
+Add `ex_cmd` to your list of dependencies in `mix.exs`:
 
 ```elixir
+def deps do
+  [
+    {:ex_cmd, "~> x.x.x"}
+  ]
+end
+```
+
+## Quick Start Examples
+
+### Basic Command Execution
+
+```elixir
+# Simple command execution
+ExCmd.stream!(~w(echo Hello))
+|> Enum.into("")
+# => "Hello\n"
+
+# Get your IP address
 ExCmd.stream!(~w(curl ifconfig.co))
 |> Enum.into("")
 ```
 
-Binary as input
+### Working with Input Streams
 
-  ```elixir
-  ExCmd.stream!(~w(cat), input: "Hello World")
-  |> Enum.into("")
-  # => "Hello World"
-  ```
+```elixir
+# String input
+ExCmd.stream!(~w(cat), input: "Hello World")
+|> Enum.into("")
+# => "Hello World"
 
-  ```elixir
-  ExCmd.stream!(~w(base64), input: <<1, 2, 3, 4, 5>>)
-  |> Enum.into("")
-  # => "AQIDBAU=\n"
-  ```
+# List of strings
+ExCmd.stream!(~w(cat), input: ["Hello", " ", "World"])
+|> Enum.into("")
+# => "Hello World"
+
+# Binary data
+ExCmd.stream!(~w(base64), input: <<1, 2, 3, 4, 5>>)
+|> Enum.into("")
+# => "AQIDBAU=\n"
+
+# IOData
+ExCmd.stream!(~w(base64), input: [<<1, 2>>, [3], [<<4, 5>>]])
+|> Enum.into("")
+# => "AQIDBAU=\n"
+```
 
-List of binary as input
+### Media Processing Examples
 
-  ```elixir
-  ExCmd.stream!(~w(cat), input: ["Hello ", "World"])
-  |> Enum.into("")
-  # => "Hello World"
-  ```
+```elixir
+# Extract audio from video with controlled memory usage
+ExCmd.stream!(~w(ffmpeg -i pipe:0 -f mp3 pipe:1),
+  input: File.stream!("music_video.mkv", [], 65536))
+|> Stream.into(File.stream!("music.mp3"))
+|> Stream.run()
+
+# Process video streams efficiently
+ExCmd.stream!(~w(ffmpeg -i pipe:0 -c:v libx264 -f mp4 pipe:1),
+  input: File.stream!("input.mp4", [], 65536),
+  max_chunk_size: 65536)
+|> Stream.into(File.stream!("output.mp4"))
+|> Stream.run()
+```
 
-iodata as input
+### Error Handling
 
-  ```elixir
-  ExCmd.stream!(~w(base64), input: [<<1, 2,>>, [3], [<<4, 5>>]])
-  |> Enum.into("")
-  # => "AQIDBAU=\n"
-  ```
+```elixir
+# stream!/2 raises on non-zero exit status
+ExCmd.stream!(["sh", "-c", "exit 10"])
+|> Enum.to_list()
+# => ** (ExCmd.Stream.AbnormalExit) program exited with exit status: 10
+
+# stream/2 returns exit status as last element
+ExCmd.stream(["sh", "-c", "echo 'foo' && exit 10"])
+|> Enum.to_list()
+# => ["foo\n", {:exit, {:status, 10}}]
+```
 
-If you want pipes and globs, you can spawn shell process and pass your
-pipeline as argument
+### Advanced Features
 
 ```elixir
-cmd = "echo 'foo bar' | base64"
-ExCmd.stream!(["sh", "-c", cmd])
+# Redirect stderr to stdout
+ExCmd.stream!(["sh", "-c", "echo foo; echo bar >&2"],
+  stderr: :redirect_to_stdout)
 |> Enum.into("")
-# => "Zm9vIGJhcgo=\n"
+# => "foo\nbar\n"
 ```
 
-Read [stream documentation](file:///Users/akash/repo/elixir/ex_cmd/doc/ExCmd.html#stream!/2) for information
-about parameters.
+## Alternatives
 
-**Check out [Exile](https://github.com/akash-akya/exile) which is an
-alternative solution based on NIF without middleware overhead**
+- For NIF-based solutions without middleware overhead, consider [Exile](https://github.com/akash-akya/exile)
+- For simple command execution without streaming, Elixir's built-in Port might be sufficient
 
-## Installation
+## Documentation
 
-```elixir
-def deps do
-  [
-    {:ex_cmd, "~> x.x.x"}
-  ]
-end
-```
+Detailed documentation is available at [HexDocs](https://hexdocs.pm/ex_cmd/).
+
+## License
+
+See [LICENSE](LICENSE) file for details.