Skip to content

Commit

Permalink
Fix unit tests (#170)
Browse files Browse the repository at this point in the history
* fix unit tests

* Fix s3 example
  • Loading branch information
meggart authored Nov 21, 2024
1 parent 2a2ad03 commit 1dd0da5
Show file tree
Hide file tree
Showing 3 changed files with 22 additions and 16 deletions.
20 changes: 13 additions & 7 deletions docs/src/s3examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,31 @@ account, you can access the dataset without credentials as follows:

````@example aws
using Zarr, AWS
AWS.global_aws_config(AWSConfig(creds=nothing, region = "eu-west-2"))
Zarr.AWSS3.AWS.global_aws_config(Zarr.AWSS3.AWS.AWSConfig(creds=nothing, region="us-west-2"))
````

Then we can directly open a zarr group stored on s3

````@example aws
z = zopen("s3://zarr-demo/store/foo/bar")
z = zopen("s3://mur-sst/zarr-v1")
````

So we see that the store points to a zarr group with a single variable `baz`.
So we see that the store points to a zarr group with a few arrays

````@example aws
v = z["baz"]
v = z["analysed_sst"]
````

The variable seems to contain an ASCIIString.
And we can read the attributes from the array

````@example aws
String(v[:])
v.attrs
````

Or some data

````@example aws
v[1:1000,1:1000,1]
````

## Accessing CMIP6 data on GCS
Expand Down Expand Up @@ -156,7 +162,7 @@ a[:,:,:] = reshape(1.0:24.0, (2,3,4))
Now we test if the data can be accessed

````@example minio
a2 = zopen("s3://zarrdata/group_1/array_1")
a2 = zopen("s3://zarrdata/group_1/bar")
a2[2,2,1:4]
````
`````
Expand Down
2 changes: 1 addition & 1 deletion src/Storage/consolidated.jl
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ _pdict(d::ConsolidatedStore,p) = filter(((k,v),)->startswith(k,p),d.cons)
function subdirs(d::ConsolidatedStore,p)
p2 = _unconcpath(d,p)
d2 = _pdict(d,p2)
_searchsubdict(d2,p,(sp,lp)->length(sp) > lp+1)
_searchsubdict(d2, p2, (sp, lp) -> length(sp) > lp + 1)
end

function subkeys(d::ConsolidatedStore,p)
Expand Down
16 changes: 8 additions & 8 deletions test/storage.jl
Original file line number Diff line number Diff line change
Expand Up @@ -171,15 +171,15 @@ end
end

@testset "AWS S3 Storage" begin
Zarr.AWSS3.AWS.global_aws_config(Zarr.AWSS3.AWS.AWSConfig(creds=nothing, region="eu-west-2"))
S3,p = Zarr.storefromstring("s3://zarr-demo/store/foo")
@test storagesize(S3,p) == 0
@test Zarr.is_zgroup(S3,p) == true
Zarr.AWSS3.AWS.global_aws_config(Zarr.AWSS3.AWS.AWSConfig(creds=nothing, region="us-west-2"))
S3, p = Zarr.storefromstring("s3://mur-sst/zarr-v1")
@test Zarr.is_zgroup(S3, p)
@test storagesize(S3, p) == 10551
S3group = zopen(S3,path=p)
S3Array = S3group.groups["bar"].arrays["baz"]
@test eltype(S3Array) == Zarr.ASCIIChar
@test storagesize(S3Array) == 69
@test String(S3Array[:]) == "Hello from the cloud!"
S3Array = S3group["time"]
@test eltype(S3Array) == Int64
@test storagesize(S3Array) == 72184
@test S3Array[1:5] == [0, 1, 2, 3, 4]
end

@testset "GCS Storage" begin
Expand Down

0 comments on commit 1dd0da5

Please sign in to comment.