Non-Azure option?

[ghost]

Is there any way to make this work with regular old on-prem vanilla SQL Server?

[Aniruddh25]

Yes, you can simply do a dotnet tool install Microsoft.DataApiBuilder and get started using the docs here:
https://learn.microsoft.com/en-us/azure/data-api-builder/get-started/get-started-with-data-api-builder

Just specify the database type to be mssql and provide the connection string with the credentials necessary to read metadata and query SQL Server.