Image copy support #1122
Image copy support #1122
Conversation
- Adds a simple extensible framework for supporting multiple build operations. Right now only RUN and COPY are implemented. - Refactored Dockerfile generation into the Image class. Rewired existing builders to use this. - Re-implemented `build-platform-image` to use new builder service API.
| @@ -161,6 +158,32 @@ def build_image( | |||
| _create_image(obj, python_sdk_path) | |||
|
|
|||
|
|
|||
| @app.command(help="Build platform images for function names") | |||
There was a problem hiding this comment.
@j3m7 this needs to be in the Tensorlake cli, and wrapped in Tensorlake deploy
for OSS, we could translate .copy to COPY in the local build context
There was a problem hiding this comment.
Agreed, it's final home will be in the tensorlake cli. added it here so we could start leveraging this functionality on our internal build pipelines before writing the tensorlake cli.
The .copy() calls are already rendered as COPY for both local and remote build contexts.
There was a problem hiding this comment.
We discussed the approach of postponing creating the tensorlake CLI + SDK to later since it is not needed for document AI.
Since everything will be open source, having it live here for now is seen as a good way for us to make progress and not have to setup a new repo, new pipy, new CI.
server/data_model/src/lib.rs
Outdated
| image_hasher.update(base_image.clone()); | ||
| image_hasher.update(run_strs.clone().join("")); | ||
| image_hasher.update(sdk_version.clone().unwrap_or("".to_string())); // Igh..... | ||
| let mut compat_image_hash: String = "".to_string(); |
There was a problem hiding this comment.
@seriousben as discussed we preserve backwards compat while accepting blank image_hashes.
Co-authored-by: Benjamin Boudreau <[email protected]>
Co-authored-by: Benjamin Boudreau <[email protected]>
| res = client.get( | ||
| f"{service_endpoint}/builds/{build.id}", headers=headers | ||
| ) | ||
| build = Build.model_validate(res.json()) |
There was a problem hiding this comment.
Is this where we'll be able to get the logs and display them?
There was a problem hiding this comment.
After the build is completed another call will need to be made to retrieve the logs, that is not implemented at this time.
|
|
||
| import docker.api.build | ||
| case "failed": | ||
| print(f"Building failed, please see logs for details") |
There was a problem hiding this comment.
What is the current expectation when this fails? Is there a way to get the logs?
There was a problem hiding this comment.
Not for the client side, they are logged by the builder task. A subsequent feature would be to store them in the blob store and add some API tissue to retrieve them.
Context
This adds COPY support for images as well as setting up a foundation for supporting more build operations.
What
Refactors image building to execute on a list of BuildOps and ports all existing image building functionality. Also adds support for cloud bases image service.
Testing
Run
indexify-cli build-default-imageandindexify-cli build-imagewith existing image definitions. Viable images should be built.Contribution Checklist
make fmtinpython-sdk/.make fmtinserver/.