Skip to content

Commit

Permalink
added documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
Ben J. Ward committed Mar 1, 2018
1 parent 31ad60a commit 26abc4f
Show file tree
Hide file tree
Showing 7 changed files with 220 additions and 72 deletions.
2 changes: 2 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,5 @@ matrix:

after_success:
- julia -e 'cd(Pkg.dir("BufferedStreams")); Pkg.add("Coverage"); using Coverage; Codecov.submit(process_folder())'
- julia -e 'cd(joinpath(Pkg.dir("BufferedStreams"), "docs/src")); mkdir("assets"); cd("assets"); run(pipeline(`curl https://codeload.github.com/BioJulia/assets/tar.gz/master`, `tar -xz --strip=2 assets-master/doc_assets`))'
- julia --color=yes -e 'cd(Pkg.dir("BufferedStreams")); Pkg.add("Documenter"); include(joinpath("docs", "make.jl"))'
121 changes: 49 additions & 72 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,98 +1,75 @@
[![Diagram of locks](https://biojulia.github.io/BufferedStreams.jl/locks.gif)](http://www.pc.gc.ca/eng/lhn-nhs/qc/annedebellevue/natcul/natcul2/b.aspx)
[![Build Status](https://travis-ci.org/BioJulia/BufferedStreams.jl.svg?branch=master)](https://travis-ci.org/BioJulia/BufferedStreams.jl)
[![codecov.io](http://codecov.io/github/BioJulia/BufferedStreams.jl/coverage.svg?branch=master)](http://codecov.io/github/BioJulia/BufferedStreams.jl?branch=master)
# BufferedStreams

[![Diagram of locks](https://biojulia.github.io/BufferedStreams.jl/locks.gif)](https://www.pc.gc.ca/eng/lhn-nhs/qc/annedebellevue/natcul/natcul2/b.aspx)

| **Release** | **Documentation** | **Maintainers** |
|:---------------------------------------------------------------:|:-------------------------------------------------------------------------------:|:-------------------------------------------:|
| [![][release-img]][release-url] [![][license-img]][license-url] | [![][docs-stable-img]][docs-stable-url] [![][docs-latest-img]][docs-latest-url] | ![][maintainer-a-img] |


## Description

BufferedStreams provides buffering for IO operations. It can wrap any IO type
automatically making incremental reading and writing faster.


## BufferedInputStream
## Installation

```julia
BufferedInputStream(open(filename)) # wrap an IOStream
BufferedInputStream(rand(UInt8, 100)) # wrap a byte array
using Pkg
add("BufferedStreams")
# Pkg.add("BufferedStreams") on julia v0.6-
```

`BufferedInputStream` wraps a source. A source can be any `IO` object, but more
specifically it can be any type `T` that implements a function:
```julia
BufferedStreams.readbytes!(source::T, buffer::Vector{UInt8}, from::Int, to::Int)
```
If you are interested in the cutting edge of the development, please check out
the master branch to try new features before release.

This function should write new data to `buffer` starting at position `from` and
not exceeding position `to` and return the number of bytes written.

`BufferedInputStream` is itself an `IO` type and implements the source type so
you can use it like any other `IO` type.
## Testing

Calling `close` propagates to the underlying source object if applicable; once
you wrap an `IO` object with a buffered stream, you can automatically close it
by calling `close` on the buffered stream. Hence, if you are using a source that
requires closing of an underlying system resource like file IO or socket, ensure
you implement a `close` method in your source. This may be important for
on-access locking file systems as in the MS Windows OS.
BufferedStreams.jl is tested against Julia `0.6` and current `0.7-dev` on Linux, OS X, and Windows.

### Anchors
| **PackageEvaluator** | **Latest Build Status** |
|:---------------------------------------------------------------:|:------------------------------------------------------------------------------------------------------:|
| [![][pkg-0.6-img]][pkg-0.6-url] [![][pkg-0.7-img]][pkg-0.7-url] | [![][travis-img]][travis-url] [![][appveyor-img]][appveyor-url] [![][codecov-img]][codecov-url] |

Input streams also have some tricks to make parsing applications easier. When
parsing data incrementally, one must take care that partial matches are
preserved across buffer refills. One easy way to do this is to copy it to a
temporary buffer, but this unecessary copying can slow things down.

Input streams instead support the notion of "anchoring", which instructs the
stream to save the current position in the buffer. If the buffer gets refilled,
then any data in the buffer including or following that position gets shifted
over to make room. When the match is finished, one can then call `takeanchored!`
return an array of the bytes from the anchored position to the currened
position, or `upanchor!` to return the index of the anchored position in the
buffer.
## Contributing and Questions

```julia
# print all numbers literals from a stream
stream = BufferedInputStream(source)
while !eof(stream)
b = peek(stream)
if '1' <= b <= '9'
if !isanchored(stream)
anchor!(stream)
end
elseif isanchored(stream)
println(ASCIIString(takeanchored!(stream)))
end

read(stream, UInt8)
end
```
We appreciate contributions from users including reporting bugs, fixing issues,
improving performance and adding new features.
Please go to the [contributing section of the documentation](https://biojulia.net/Contributing/latest)
for more information.

If you have a question about
contributing or using this package, you are encouraged to use the
[Bio category of the Julia discourse
site](https://discourse.julialang.org/c/domain/bio).

## BufferedOutputStream

```julia
stream = BufferedOutputStream(open(filename, "w")) # wrap an IOStream
```
[release-img]: https://img.shields.io/github/release/BioJulia/BufferedStreams.jl.svg
[release-url]: https://github.com/BioJulia/BufferedStreams.jl/releases/latest

`BufferedOutputStream` is the converse to `BufferedInputStream`, wrapping a sink
type. It also works on any writable `IO` type, as well the more specific sink
interface:
[license-img]: https://img.shields.io/badge/license-MIT-green.svg
[license-url]: https://github.com/BioJulia/BufferedStreams.jl/blob/master/LICENSE

```julia
writebytes(sink::T, buffer::Vector{UInt8}, n::Int, eof::Bool)
```
[docs-latest-img]: https://img.shields.io/badge/docs-latest-blue.svg
[docs-latest-url]: https://biojulia.github.io/BufferedStreams.jl/latest
[docs-stable-img]: https://img.shields.io/badge/docs-stable-blue.svg
[docs-stable-url]: https://biojulia.github.io/BufferedStreams.jl/stable

This function should consume the first `n` bytes of `buffer`. The `eof` argument
is used to indicate that there will be no more input to consume. It should
return the number of bytes written, which must be `n` or 0. A return value of 0
indicates data was processed but should not be evicted from the buffer.
[maintainer-a-img]: https://img.shields.io/badge/BioJulia%20Maintainer-Ward9250-orange.svg

[pkg-0.6-img]: https://pkg.julialang.org/badges/BufferedStreams_0.6.svg
[pkg-0.6-url]: https://pkg.julialang.org/detail/BufferedStreams
[pkg-0.7-img]: https://pkg.julialang.org/badges/BufferedStreams_0.7.svg
[pkg-0.7-url]: https://pkg.julialang.org/detail/BufferedStreams

### `BufferedOutputStream` as an alternative to `IOBuffer`
[travis-img]: https://img.shields.io/travis/BioJulia/BufferedStreams.jl/master.svg?label=Linux+/+macOS
[travis-url]: https://travis-ci.org/BioJulia/BufferedStreams.jl

`BufferedOutputStream` can be used as a simpler and often faster alternative to
`IOBuffer` for incrementally building strings.
[appveyor-img]: https://img.shields.io/appveyor/ci/BioJulia/BufferedStreams.jl/master.svg?label=Windows
[appveyor-url]: https://ci.appveyor.com/project/Ward9250/bufferedstreams-jl/branch/master

```julia
out = BufferedOutputStream()
print(out, "Hello")
print(out, " World")
str = String(take!(out))
```
[codecov-img]: https://codecov.io/gh/BioJulia/BufferedStreams.jl/branch/master/graph/badge.svg
[codecov-url]: https://codecov.io/gh/BioJulia/BufferedStreams.jl
42 changes: 42 additions & 0 deletions appveyor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
environment:
matrix:
- Julia_URL: "https://julialang-s3.julialang.org/bin/winnt/x64/0.6/julia-0.6-latest-win64.exe"
- Julia_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x64/julia-latest-win64.exe"

matrix:
allow_failures:
- JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x64/julia-latest-win64.exe"

branches:
only:
- master
- /release-.*/

notifications:
- provider: Email
on_build_success: false
on_build_failure: false
on_build_status_changed: false

install:
- ps: "[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12"
# if there's a newer build queued for the same PR, cancel this one
- ps: if ($env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:APPVEYOR_BUILD_NUMBER -ne ((Invoke-RestMethod `
https://ci.appveyor.com/api/projects/$env:APPVEYOR_ACCOUNT_NAME/$env:APPVEYOR_PROJECT_SLUG/history?recordsNumber=50).builds | `
Where-Object pullRequestId -eq $env:APPVEYOR_PULL_REQUEST_NUMBER)[0].buildNumber) { `
throw "There are newer queued builds for this pull request, failing early." }
# Download most recent Julia Windows binary
- ps: (new-object net.webclient).DownloadFile(
$env:JULIA_URL,
"C:\projects\julia-binary.exe")
# Run installer silently, output to C:\projects\julia
- C:\projects\julia-binary.exe /S /D=C:\projects\julia

build_script:
# Need to convert from shallow to complete for Pkg.clone to work
- IF EXIST .git\shallow (git fetch --unshallow)
- C:\projects\julia\bin\julia -e "versioninfo();
Pkg.clone(pwd(), \"BufferedStreams\"); Pkg.build(\"BufferedStreams\")"

test_script:
- C:\projects\julia\bin\julia -e "Pkg.test(\"BufferedStreams\")"
21 changes: 21 additions & 0 deletions docs/make.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
using Documenter, BufferedStreams

makedocs(
format = :html,
sitename = "BufferedStreams",
pages = [
"Home" => "index.md",
"Input Streams" => "inputstreams.md",
"Output Streams" => "outputstreams.md"
],
authors = "D. C. Jones, Ben J. Ward"
)

deploydocs(
repo = "github.com/BioJulia/BufferedStreams.jl.git",
julia = "0.6",
osname = "linux",
target = "build",
deps = nothing,
make = nothing
)
1 change: 1 addition & 0 deletions docs/src/index.md
66 changes: 66 additions & 0 deletions docs/src/inputstreams.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
```@meta
CurrentModule = BufferedStreams
DocTestSetup = quote
using BufferedStreams
end
```
# BufferedInputStream

```@example
BufferedInputStream(open(filename)) # wrap an IOStream
BufferedInputStream(rand(UInt8, 100)) # wrap a byte array
nothing # hide
```

`BufferedInputStream` wraps a source. A source can be any `IO` object, but more
specifically it can be any type `T` that implements a function:
```julia
BufferedStreams.readbytes!(source::T, buffer::Vector{UInt8}, from::Int, to::Int)
```

This function should write new data to `buffer` starting at position `from` and
not exceeding position `to` and return the number of bytes written.

`BufferedInputStream` is itself an `IO` type and implements the source type so
you can use it like any other `IO` type.

Calling `close` propagates to the underlying source object if applicable; once
you wrap an `IO` object with a buffered stream, you can automatically close it
by calling `close` on the buffered stream. Hence, if you are using a source that
requires closing of an underlying system resource like file IO or socket, ensure
you implement a `close` method in your source. This may be important for
on-access locking file systems as in the MS Windows OS.

## Anchors

Input streams also have some tricks to make parsing applications easier. When
parsing data incrementally, one must take care that partial matches are
preserved across buffer refills. One easy way to do this is to copy it to a
temporary buffer, but this unecessary copying can slow things down.

Input streams instead support the notion of "anchoring", which instructs the
stream to save the current position in the buffer. If the buffer gets refilled,
then any data in the buffer including or following that position gets shifted
over to make room. When the match is finished, one can then call `takeanchored!`
return an array of the bytes from the anchored position to the currened
position, or `upanchor!` to return the index of the anchored position in the
buffer.

```@example
# print all numbers literals from a stream
stream = BufferedInputStream(source)
while !eof(stream)
b = peek(stream)
if '1' <= b <= '9'
if !isanchored(stream)
anchor!(stream)
end
elseif isanchored(stream)
println(ASCIIString(takeanchored!(stream)))
end
read(stream, UInt8)
nothing # hide
end
```
39 changes: 39 additions & 0 deletions docs/src/outputstreams.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
```@meta
CurrentModule = BufferedStreams
DocTestSetup = quote
using BufferedStreams
end
```
# BufferedOutputStream

```@example
stream = BufferedOutputStream(open(filename, "w")) # wrap an IOStream
nothing # hide
```

`BufferedOutputStream` is the converse to `BufferedInputStream`, wrapping a sink
type. It also works on any writable `IO` type, as well the more specific sink
interface:

```@example
writebytes(sink::T, buffer::Vector{UInt8}, n::Int, eof::Bool)
nothing # hide
```

This function should consume the first `n` bytes of `buffer`. The `eof` argument
is used to indicate that there will be no more input to consume. It should
return the number of bytes written, which must be `n` or 0. A return value of 0
indicates data was processed but should not be evicted from the buffer.


## `BufferedOutputStream` as an alternative to `IOBuffer`

`BufferedOutputStream` can be used as a simpler and often faster alternative to
`IOBuffer` for incrementally building strings.

```@example
out = BufferedOutputStream()
print(out, "Hello")
print(out, " World")
str = String(take!(out))
```

0 comments on commit 26abc4f

Please sign in to comment.