Skip to content

Commit

Permalink
first update
Browse files Browse the repository at this point in the history
  • Loading branch information
deusebio committed Aug 2, 2024
1 parent a9f919a commit 5cf5af8
Show file tree
Hide file tree
Showing 7 changed files with 16 additions and 14 deletions.
1 change: 1 addition & 0 deletions .github/workflows/on_push.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ on:
push:
branches:
- '3.4-22.04/*'
- '3.5-22.04/*'

jobs:
publish:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/trivy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ on:
push:
branches:
- 3.4-22.04/edge
- 3.5-22.04/edge
pull_request:
jobs:
build:
Expand Down
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The Charmed Spark rock images come with some built-in tooling embedded:

Rocks will be named as `<version>-<series>_<risk>`.

`<version>` is the software version; `<series>` is the Ubuntu LTS series that rocks supports; and the <risk> is the type of release, if it is edge, candidate or stable. Example versioning will be 3.4-22.04_stable which means Charmed Spark is a version 3.4.x of the software, supporting the 22.04 Ubuntu release and currently a 'stable' version of the software. See versioning details [here](https://snapcraft.io/docs/channels).
`<version>` is the software version; `<series>` is the Ubuntu LTS series that rocks supports; and the <risk> is the type of release, if it is edge, candidate or stable. Example versioning will be 3.5-22.04_stable which means Charmed Spark is a version 3.5.x of the software, supporting the 22.04 Ubuntu release and currently a 'stable' version of the software. See versioning details [here](https://snapcraft.io/docs/channels).

Channel can also be represented by combining `<version>_<risk>`

Expand Down Expand Up @@ -65,15 +65,15 @@ Charmed Spark rock with Spark Rapids support is available at https://github.com/
The image can be used straight away when running Spark on Kubernetes by setting the appropriate configuration property:

```shell
spark.kubernetes.container.image=ghcr.io/canonical/charmed-spark:3.4-22.04_edge
spark.kubernetes.container.image=ghcr.io/canonical/charmed-spark:3.5-22.04_edge
```

### Using `spark8t` CLI

The `spark8t` CLI tooling interacts with the K8s API to create, manage and delete K8s resources representing the Spark service account.
Make sure that the kube config file is correctly loaded into the container, e.g.
```shell
docker run --name chamed-spark -v /path/to/kube/config:/var/lib/spark/.kube/config ghcr.io/canonical/charmed-spark:3.4-22.04_edge
docker run --name chamed-spark -v /path/to/kube/config:/var/lib/spark/.kube/config ghcr.io/canonical/charmed-spark:3.5-22.04_edge
```

Note that this will start the image and a long-living service, allowing you to exec commands:
Expand All @@ -83,7 +83,7 @@ docker exec charmed-spark spark-client.service-account-registry list

If you prefer to run one-shot commands, without having the Charmed Spark image running, use `\; exec` prefix, e.g.
```shell
docker run -v ... ghcr.io/canonical/charmed-spark:3.4-22.04_edge \; exec spark-client.service-account-registry list
docker run -v ... ghcr.io/canonical/charmed-spark:3.5-22.04_edge \; exec spark-client.service-account-registry list
```

For more information about spark-client API and `spark8t` tooling, please refer to [here](https://discourse.charmhub.io/t/spark-client-snap-how-to-manage-spark-accounts/8959).
Expand All @@ -95,7 +95,7 @@ Charmed Spark rock Image is delivered with Pebble already included in order to m
#### Starting History Server

```shell
docker run ghcr.io/canonical/charmed-spark:3.4-22.04_edge \; start history-server
docker run ghcr.io/canonical/charmed-spark:3.5-22.04_edge \; start history-server
```

### Running Jupyter Lab
Expand All @@ -110,7 +110,7 @@ To start a JupyterLab server using the `charmed-spark-jupyter` image, use
docker run \
-v /path/to/kube/config:/var/lib/spark/.kube/config \
-p <port>:8888
ghcr.io/canonical/charmed-spark-jupyterlab:3.4.2-4.0.11-22.04_edge \
ghcr.io/canonical/charmed-spark-jupyterlab:3.5.1-4.0.11-22.04_edge \
--username <spark-service-account> --namespace <spark-namespace>
```

Expand All @@ -123,7 +123,7 @@ JupyterLab server to act as a Spark driver and request resources on the K8s clus

## Developers and Contributing

Please see the [CONTRIBUTING.md](https://github.com/canonical/charmed-spark-rock/blob/3.4-22.04/edge/CONTRIBUTING.md) for guidelines and for developer guidance.
Please see the [CONTRIBUTING.md](https://github.com/canonical/charmed-spark-rock/blob/3.5-22.04/edge/CONTRIBUTING.md) for guidelines and for developer guidance.

## Bugs and feature request

Expand All @@ -135,7 +135,7 @@ If you find a bug in this rock or want to request a specific feature, here are t

## Licence statement

Charmed Spark is free software, distributed under the [Apache Software License, version 2.0](https://github.com/canonical/charmed-spark-rock/blob/3.4-22.04/edge/LICENSE).
Charmed Spark is free software, distributed under the [Apache Software License, version 2.0](https://github.com/canonical/charmed-spark-rock/blob/3.5-22.04/edge/LICENSE).

## Trademark Notice

Expand Down
2 changes: 1 addition & 1 deletion images/charmed-spark/conf/spark-defaults.conf
Original file line number Diff line number Diff line change
@@ -1 +1 @@
spark.kubernetes.container.image=ghcr.io/canonical/charmed-spark:3.4-22.04
spark.kubernetes.container.image=ghcr.io/canonical/charmed-spark:3.5-22.04
6 changes: 3 additions & 3 deletions images/charmed-spark/rockcraft.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ description: |
license: Apache-2.0

version: "3.4.2"
version: "3.5.1"

base: [email protected]

Expand Down Expand Up @@ -51,8 +51,8 @@ services:
parts:
spark:
plugin: dump
source: https://github.com/canonical/central-uploader/releases/download/spark-3.4.2-ubuntu5/spark-3.4.2-ubuntu5-20240702211805-bin-k8s.tgz
source-checksum: sha512/2d620730b31ddb6532dc0f08c3da92aa4fbc0e3d7eea7628bc9705ae26b3a38db5b776be27e65271d7c375cfc676def233c8de37917c6cb6becd9cbbe6fabb22
source: https://github.com/canonical/central-uploader/releases/download/spark-3.5.1-ubuntu1/spark-3.5.1-ubuntu1-20240801193221-bin-k8s.tgz
source-checksum: sha512/4c9ec4dc3f67602ec9f83fa08486985647b8ab7eaf3748c3042a382ed367fc776b7623a067ceef408337cf5c2171a010da0ff063dcbf76c4b84674b661d2330b
overlay-script: |
sed -i 's/http:\/\/deb.\(.*\)/https:\/\/deb.\1/g' /etc/apt/sources.list
apt-get update
Expand Down
2 changes: 1 addition & 1 deletion tests/integration/integration-tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ cleanup_user_failure() {
teardown_test_pod() {
kubectl logs testpod-admin -n $NAMESPACE
kubectl logs testpod -n $NAMESPACE
kubectl logs -l spark-version=3.4.2 -n $NAMESPACE
kubectl logs -l spark-version=3.5.1 -n $NAMESPACE
kubectl -n $NAMESPACE delete pod $ADMIN_POD_NAME
}

Expand Down
2 changes: 1 addition & 1 deletion tests/integration/resources/testpod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ metadata:
name: testpod
spec:
containers:
- image: ghcr.io/canonical/test-charmed-spark:3.4.2
- image: ghcr.io/canonical/test-charmed-spark:3.5.1
name: spark
ports:
- containerPort: 18080

0 comments on commit 5cf5af8

Please sign in to comment.