Integration with TrustyAI service

[yhwang]

TrustyAI service is The TrustyAI service is a REST service that provides the integration between XAI and metrics algorithms provided by the TrustyAI core and external models.

TrustyAI operator is used to deploy the TrustAI service.

Integrating the LM-Eval-aaS from this repo with the TrustyAI service would be good. I'd like to use this to discuss

• How to integrate LM-Eval-aaS with the TrustyAI
• How to consolidate the code in multiple repos
• How the high-level architecture could be when LM-Eval-aaS integrates with the TrustyAI service

and maybe other topics

[RobGeada]

Hey @yhwang.

To clarify, are we talking about having some integration of actual service bits of the LM-eval-service with the TrustyAI service, in addition to merging the lm-eval-service-controller into the TrustyAI operator? And the actual LM-eval-service is currently a Python+Flask server, right?

If so, that could potentially pose an interesting project, but perhaps from the opposite direction: TrustyAI has been contemplated creating a Python-based TrustyAI Service, that could run alongside/replace the current Java service, to allow for closer alignment with other ML/data science projects. If we want to align these, perhaps the lm-eval-service could be the first module of this broader Python based TrustyAI service.

[yhwang]

And the actual LM-eval-service is currently a Python+Flask server, right?

Yes, but since we have the CRD and controller now. the Flask server will be stripped down to only serve the API purpose and create LMEvalJob CRs based on the requests. It would be good to convert it to go and use the go pkg of the CRD. In the current design, the LM-Eval job is happening in the Pod that the controller creates according to the content in the CRD + settings. From the LM-Eval-service point of view, as long as the image can run the lm-eval tasks and produce the results, then we can expose the tasks through the APIs.

[yhwang]

I guess the scope could be:

• integrate the CRD + controller of LM-Eval-service into TrustyAI operator

The TrustyAI operator will not only monitor the existing TrustyAIService but also LMEvalJob. In the long run, the jobs will be managed by the job management system.

[ruivieira]

@yhwang Since the work for the CRD and reconciler logic is already made/in progress I agree this is a good first step.
The TrustyAI operator only manages one CRD at the moment (TrustyAIServices). It would be good to have a first discussion about the integrated architecture 👍

[yhwang]

I started with a general idea and then we can dive into the specific details and scope, i.e. integrating the LMEvalJob CRD and controller into the TustyAI operator.

Let me change the first item to: How to integrate LM-Eval-aaS with the TrustyAI

[ruivieira]

@yhwang Just some initial thoughts:

• I think we could create a development branch on the TrustyAI operator and do PRs against that for now
• The TrustyAI operator does not have cert-manager as a dependency at the moment. I don't see that as problem regarding TrustyAI itself.
• When unifying this, IMO, we should contemplate several available scenarios (i.e. user wants TrustyAI service (TAS) and not LM-Eval-aaS (LME), wants LME and not TAS or both). I think this will mean looking at the operator's default configuration options.

[yhwang]

I think we could create a development branch on the TrustyAI operator and do PRs against that for now

This is a great suggestion and I vote for it

The TrustyAI operator does not have cert-manager as a dependency at the moment. I don't see that as problem regarding TrustyAI itself.

I'd love to have the cert-manager as the dependency because we need it for the webhook. And if you check another discussion thread, we plan to have mTLS between the controller and the driver although the mTLS would be optional. It would be good to have the cert-manager as the dependency to facilitate the deployment.

we should contemplate several available scenarios (i.e. user wants TrustyAI service (TAS) and not LM-Eval-aaS (LME), wants LME and not TAS or both)

This makes sense to me too. It's great to provide such options for users to choose which service they want the operator to manage. And I think it's good to use webhooks to validate the options, at least one of the options should be chosen.

Currently, I am working on this issue. My thinking is to finish the RBAC-free mechanism first. Then I can start to port the controller + CRD here to the TrustyAI operator repo and use a dev branch for the PRs. While I am working on the issue, we can keep discussing details of the integration, including the dependencies, options, new architecture of the TrustyAI operator, etc.

[yhwang]

One thing we definitely need to think through is this: Why should one avoid a system design where a single controller is responsible for managing multiple CRDs (Custom Resource Definitions)(for example, an ‘install_all_controller.go’)? so definitely two controllers for 2 CRDs and the architecture could be:
A)

• TrustyAI operator has one controller that controls the deployment of TAS (TrustyAI Service), LMES(LM-Eval Service)
  Add an LMEvalService field under TrustyAIService to enable it. But this makes LMES optional and TAS is always on unless we explicitly add TrustyAIService as well.
• If LMES is selected, the LMEValJob CRD + controller is deployed, the controller runs as a separate deployment and this controller manages the LMEvalJob CRs.

B)

• TrustyAI operator has two controllers

  • one controller manages the TrustyAIService CR
  • one controller manages the LMEvalJob CRs

  Then we need a mechanism at the operator level to turn on/off the TAS and LMES.

Open for discussion (I guess this is why this discussion thread was created)

[ruivieira]

Hi @yhwang, I like option B)

If we take the ODH case, if the TrustyAI operator is installed (i.e. Managed) it will be in an idle state, simply monitoring for the creation of TrustyAIServices.
I think that even without a mechanism at the operator level to turn on/off the TAS/LMES, this gives the flexibility and separation between TASs/LMESs. I will look at the possibility of extending the TrustyAI's DSC component to enable on/off customisation too.

cc @danielezonca

[yhwang]

Thanks for all the input, let me summarize what we have discussed so far:

• The integration work will lay on a dev branch on the TrustyAI operator repo:
  • Create a dev branch for the LMES integration
  • Create PR(s) to move the LMES controller into the TrustyAI operator
  • Have a flag or config setting to load either the TAS/LMES controller or both when starting up the TrustyAI operator
  • Need to add LMEvalJob CRD and related manifests to the TrustyAI repo as well (should go to the config directory)
• Adding a toggle/switch mechanism for TAS/LMES on TrustyAI DSC component
• Nice to have: Make TrustyAI operator depend on the cert-manager.
  • Two LMES components need certificates/keys: the webhook and the driver. If there is no cert-manager, then we can make webhook optional and the driver uses unsecured gRPC to communicate with the controller.

@RobGeada @ruivieira feel free to chime in and correct me if something is wrong.

[terrytangyuan]

Have a flag or config setting to load either the TAS/LMES controller or both when starting up the TrustyAI operator

Is this a boolean flag or something like a list of additional capabilities, e.g. ["Trusty", "TAS", "LMES"]?

One thing we need to handle is the discovery of available CRDs. If LMEvalJob CRD does not exist in the cluster but this capability is turned on, we need to make sure to handle this gracefully in the operator.

[danielezonca]

Have a flag or config setting to load either the TAS/LMES controller or both when starting up the TrustyAI operator

If this is a global (and static) configuration we can probably just create different overlays to install or not a controller. In this way we can decide at deploy time which feature you want to enable

Nice to have: Make TrustyAI operator depend on the cert-manager.

There is already a mechanism in ODH operator to inject certificates that each component can use. We are considering cert-manager but it has a way larger impact across ODH components so the idea is to try to centralize this in ODH operator.

If LMEvalJob CRD does not exist in the cluster

Isn't this part of the installation manifest too?

[yhwang]

Is this a boolean flag or something like a list of additional capabilities, e.g. ["Trusty", "TAS", "LMES"]?

I believe it would be better to have this as a list, expecting more capabilities to be added in the future.

One thing we need to handle is the discovery of available CRDs. If LMEvalJob CRD does not exist in the cluster but this capability is turned on, we need to make sure to handle this gracefully in the operator.

Good advice. noted.

If this is a global (and static) configuration

This is a good question. I would leave it to the TrustyAI or ODH operator team to decide. But if the decision is global but editable, then the TrustyAI operator needs to pick up the new configuration/settings by reloading the deployment, i.e. rollout restart deploy trustyai-operator

Isn't this part of the installation manifest too?

I believe it's kind of due diligence at the TrustyAI operator level.

[yhwang]

There is already a mechanism in ODH operator to inject certificates that each component can use

Thanks for the information. As long as the needed keys/certs can be generated for the webhook and driver, at the TrustyAI operator level, it doesn't matter what mechanism is used for the keys/certs