[CSPL-2699]: adding Azure and GCP bucket access using sdk

[vivekr-splunk]

Signed-off-by: Vivek Reddy [email protected]

The Splunk Operator for Kubernetes currently supports AWS S3 buckets, allowing users to integrate Splunk with AWS storage seamlessly. However, customers utilizing Google Cloud Platform (GCP) and Microsoft Azure are required to use interoperability solutions to access their respective Blob storage services. This pull request introduces native support for GCP storage buckets and enhances Azure integration by implementing Azure AD authentication alongside traditional credential methods.

Key Enhancements:

• GCP Storage Support:

  • GCP SDK Integration: Introduced the Google Cloud Platform (GCP) SDK to enable direct interaction with GCP storage buckets.
  • GCP SDK Wrapper: Developed a wrapper for the GCP SDK, mirroring the existing AWS S3 client wrapper, to facilitate operations such as listing and downloading objects from GCP storage buckets.

• Azure Storage Enhancements:

  • Azure AD Authentication: Implemented Azure Active Directory (Azure AD) authentication to provide secure and streamlined access to Azure Blob Storage.
  • Azure SDK Integration: Leveraged the Azure SDK to manage credentials and handle Blob storage operations effectively.

Motivation

Expanding the Splunk Operator's storage capabilities to include GCP and Azure addresses the needs of a broader user base operating in multi-cloud environments. Native support eliminates the complexities and limitations associated with interoperability solutions, offering users a more robust, secure, and efficient way to manage their storage integrations within Splunk.

Changes

GCP Integration

1. GCP SDK Wrapper:

   • Created a new module gcpbucketclient that encapsulates GCP SDK functionalities.
   • Implemented methods for:
     • Listing objects in GCP storage buckets.
     • Downloading objects from GCP storage buckets.

2. Configuration Updates:

   • Added configuration options in the operator's settings to specify GCP credentials and bucket details.
   • Updated Kubernetes ServiceAccount and Role definitions to accommodate GCP access permissions.

3. Documentation:

   • Updated README and user guides to include instructions for configuring GCP storage integration.

Azure Enhancements

1. Azure AD Authentication:

   • Integrated Azure AD authentication mechanisms to secure access to Azure Blob Storage.
   • Configured the operator to use Azure AD tokens for authenticating storage operations.

2. Azure SDK Integration:

   • Utilized the Azure SDK to handle credential management and Blob storage interactions.
   • Ensured compatibility with existing Azure storage configurations and workflows.

3. Configuration Updates:

   • Enhanced the operator's configuration to support Azure AD authentication parameters.
   • Updated Kubernetes manifests to include necessary Azure RBAC roles and permissions.

4. Documentation:

   • Expanded documentation to cover Azure AD authentication setup and troubleshooting steps.

Testing

• Unit Tests:

  • Added unit tests for the new GCP SDK wrapper to ensure reliable interactions with GCP storage.
  • Enhanced existing tests to cover Azure AD authentication flows.

• Integration Tests:

  • Performed end-to-end testing with GCP storage buckets to validate listing and downloading functionalities.
  • Verified Azure Blob Storage access using Azure AD authentication, ensuring secure and authorized operations.

• Regression Testing:

  • Ensured that existing AWS S3 functionalities remain unaffected by the new integrations.
  • Conducted performance testing to confirm that the operator maintains optimal performance with multi-cloud support.

Impact

This enhancement broadens the Splunk Operator's compatibility, making it a more versatile tool for organizations leveraging multi-cloud strategies. By providing native support for GCP and Azure storage services, users can manage their Splunk integrations more effectively, reducing dependency on third-party interoperability solutions and simplifying their infrastructure management.

Related Issues

• Issue CSPL-2824: Support for GCP Storage Buckets
• Issue CSPL-2825: Implement Azure AD Authentication for Blob Storage Access

Future Work

• GCP Storage Operations: Extend GCP storage support to include uploading objects and managing bucket configurations.
• Azure Storage Features: Incorporate additional Azure Blob Storage functionalities such as container management and advanced access controls.
• Multi-Cloud Optimization: Explore optimizations for simultaneous multi-cloud storage operations to enhance performance and reliability.
• User Interface Enhancements: Update the operator's UI/dashboard to provide better visibility and control over multi-cloud storage integrations.

Checklist

• Code follows the project's coding standards
• Relevant documentation has been updated
• All new and existing tests pass
• Changes have been tested in a development environment