Skip to content

Latest commit

 

History

History
111 lines (87 loc) · 11.1 KB

File metadata and controls

111 lines (87 loc) · 11.1 KB

Release CI License Registry Source

terraform-azurerm-lake-loader-vmss

A Terraform module for saving Delta / Iceberg format files into Azure Blob Storage for direct querying by warehouses.

Usage

See the examples/default directory for a working usage example of this module.

Requirements

Name Version
terraform >= 1.0.0
azuread >= 2.39.0
azurerm >= 3.58.0

Providers

Name Version
azuread >= 2.39.0
azurerm >= 3.58.0

Modules

Name Source Version
service snowplow-devops/service-vmss/azurerm 0.1.1
telemetry snowplow-devops/telemetry/snowplow 0.5.0

Resources

Name Type
azuread_application.app_registration resource
azuread_application_password.app_pasword resource
azuread_service_principal.sp resource
azurerm_eventhub_consumer_group.enriched_topic resource
azurerm_network_security_group.nsg resource
azurerm_network_security_rule.egress_tcp_443 resource
azurerm_network_security_rule.egress_tcp_80 resource
azurerm_network_security_rule.egress_udp_123 resource
azurerm_network_security_rule.ingress_tcp_22 resource
azurerm_role_assignment.sc_app_ra resource
azuread_client_config.current data source
azurerm_resource_group.rg data source
azurerm_storage_container.sc data source

Inputs

Name Description Type Default Required
bad_topic_kafka_password Password for connection to Kafka cluster under PlainLoginModule (note: as default the EventHubs topic connection string for writing is expected) string n/a yes
bad_topic_name The name of the bad Event Hubs topic that the loader will push failures into string n/a yes
enriched_topic_kafka_password Password for connection to Kafka cluster under PlainLoginModule (note: as default the EventHubs topic connection string for reading is expected) string n/a yes
enriched_topic_name The name of the enriched Event Hubs topic that the loader will pull data from string n/a yes
kafka_brokers The brokers to configure for access to the Kafka Cluster (note: as default the EventHubs namespace broker) string n/a yes
name A name which will be pre-pended to the resources created string n/a yes
resource_group_name The name of the resource group to deploy the service into string n/a yes
ssh_public_key The SSH public key attached for access to the servers string n/a yes
storage_account_name Name of the output storage account string n/a yes
storage_container_name Name of the output storage container string n/a yes
subnet_id The subnet id to deploy the service into string n/a yes
accept_limited_use_license Acceptance of the SLULA terms (https://docs.snowplow.io/limited-use-license-1.0/) bool false no
app_version Lake Loader app version to use. This variable facilitates dev flow, the modules may not work with anything other than the default value. string "0.1.4" no
associate_public_ip_address Whether to assign a public ip address to this instance bool true no
bad_topic_kafka_username Username for connection to Kafka cluster under PlainLoginModule (default: '$ConnectionString' which is used for EventHubs) string "$ConnectionString" no
custom_iglu_resolvers The custom Iglu Resolvers that will be used by Enrichment to resolve and validate events
list(object({
name = string
priority = number
uri = string
api_key = string
vendor_prefixes = list(string)
}))
[] no
default_iglu_resolvers The default Iglu Resolvers that will be used by Enrichment to resolve and validate events
list(object({
name = string
priority = number
uri = string
api_key = string
vendor_prefixes = list(string)
}))
[
{
"api_key": "",
"name": "Iglu Central",
"priority": 10,
"uri": "http://iglucentral.com",
"vendor_prefixes": []
},
{
"api_key": "",
"name": "Iglu Central - Mirror 01",
"priority": 20,
"uri": "http://mirror01.iglucentral.com",
"vendor_prefixes": []
}
]
no
eh_namespace_name The name of the Event Hubs namespace (note: if you are not using EventHubs leave this blank) string "" no
enriched_topic_kafka_username Username for connection to Kafka cluster under PlainLoginModule (default: '$ConnectionString' which is used for EventHubs) string "$ConnectionString" no
java_opts Custom JAVA Options string "-XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75" no
kafka_source The source providing the Kafka connectivity (def: azure_event_hubs) string "azure_event_hubs" no
ssh_ip_allowlist The comma-seperated list of CIDR ranges to allow SSH traffic from list(string)
[
"0.0.0.0/0"
]
no
storage_container_path Path within the storage container to write output into string "events/" no
tags The tags to append to this resource map(string) {} no
telemetry_enabled Whether or not to send telemetry information back to Snowplow Analytics Ltd bool true no
user_provided_id An optional unique identifier to identify the telemetry events emitted by this stack string "" no
vm_instance_count The instance count to use number 1 no
vm_sku The instance type to use string "Standard_B2s" no

Outputs

Name Description
nsg_id ID of the network security group attached to the Lake Loader Server nodes
vmss_id ID of the VM scale-set

Copyright and license

Copyright 2023-present Snowplow Analytics Ltd.

Licensed under the Snowplow Limited Use License Agreement. (If you are uncertain how it applies to your use case, check our answers to frequently asked questions.)