Proxy setting using HttpClientFactory is not working

[patilravikiran]

I am trying to download buckets from Google cloud storage using dotnet APIs over proxy, but its failing with below error.

Error Message:
Authentication failed because the remote party has closed the transport stream at Google.Cloud.Storage.V1.StorageClientImpl.DownloadObjectImpl(RequestBuilder request builder, Stream destination, DownloadObjectOptions options, IProgress'1 progress)

I have followed below issue for proxy code and used bucket download script.
Issue- #1537

Could you please help me with script for downloading buckets over proxy?

[jskeet]

(Note: this issue would have been better filed at https://github.com/googleapis/google-cloud-dotnet as that's where the Google.Cloud.Storage.V1 lives. I can't transfer it right now, but will try to do so later in the week.)

I'll look tomorrow (it's a public holiday today). It would help if you could provide more information though:

• Do other API operations work, e.g. listing buckets and objects, uploading objects?
• Do you specifically need a different proxy for Storage operations than for other operations? (There are simpler ways of setting a process-wide proxy.)
• What version of .NET are you using, and where is the code running?
• Please could you provide a short but complete example that demonstrates the problem, ideally as a console app for simplicity?

[patilravikiran]

Thanks for quick response. Please find my answers below.

Do other API operations work, e.g. listing buckets and objects, uploading objects?
Its not working with current proxy script. The list and upload operations works fine when I follow below link, but not download.
googleapis/google-cloud-dotnet#1832

Do you specifically need a different proxy for Storage operations than for other operations? (There are simpler ways of setting a process-wide proxy.)
I have tried with WebProxy as well, but no luck. I will need proxy script which allows me to set bypass list.

What version of .NET are you using, and where is the code running?
.Net core 3.1
Google.apis-1.49.0
Google.Cloud.Storage.V1 - 3.3.0
I am running code on Azure cloud.

Please could you provide a short but complete example that demonstrates the problem, ideally as a console app for simplicity?
I will share code shortly.

[jskeet]

Please could you copy/paste the code as text (or a zip file with a complete project) into this issue? Screenshots are much harder to work with than plain text.

[patilravikiran]

var scopes = new string[] { "https://www.googleapis.com/auth/devstorage.full_control" };

var serviceCredentialFromFile = (ServiceAccountCredential) GoogleCredential.FromFile(sa_file).UnderlyingCredential;

var initializer = new ServiceAccountCredential.Initializer(serviceCredentialFromFile.Id)
{
ProjectId = serviceCredentialFromFile.ProjectId,
Key = serviceCredentialFromFile.Key,
User = serviceCredentialFromFile.User
HttpClientFactory = new ProxyHttpClientFactory(proxy),
Scopes = scopes
};
var credential = GoogleCredential.FromServiceAccountCredential(new ServiceAccountCredential(initializer));

var client = StorageClient.Create(crediential);
using ( var filestream = File.Create("");
{
client.DownloadObject("", "", filestream);
}

[patilravikiran]

using ( var filestream = File.Create("localpath");
{
client.DownloadObject("bucketname", "filename", filestream);
}

[amanda-tarafa]

I have been able to reproduce and I know what is happening. I'm now going into a 1 hour meeting, but I'll be posting a workaround after and we'll probably be able to provide a better solution later during the week.

[amanda-tarafa]

@patilravikiran I've sent googleapis/gax-dotnet#464 for review.

After this has been merged and release, you can either reference the new Gax version directly or wait for an updated Storage release. Then your code should look something like this:

// Your implementation of Google.Apis.Http.IHttpClientFactory.
var proxiedHttpClientFactory = new ProxyHttpClientFactory(proxy);

var scopes = new string[] { "https://www.googleapis.com/auth/devstorage.full_control" };
var serviceCredentialFromFile = (ServiceAccountCredential) GoogleCredential.FromFile(sa_file).UnderlyingCredential;
var initializer = new ServiceAccountCredential.Initializer(serviceCredentialFromFile.Id)
{
    ProjectId = serviceCredentialFromFile.ProjectId,
    Key = serviceCredentialFromFile.Key,
    User = serviceCredentialFromFile.User
    HttpClientFactory = proxiedHttpClientFactory,
    Scopes = scopes
};
var credential = GoogleCredential.FromServiceAccountCredential(new ServiceAccountCredential(initializer));

var clientBuilder = new StorageClientBuilder
{
    Credential = credential,
    // All HTTP client instances for API calls will be obtained from this factory
    HttpClientFactory = proxiedHttpClientFactory
}

var client = clientBuilder.Build();

Notice also that once #1851 is merged and released, setting a proxy for a credential will be much more simple as well.

[amanda-tarafa]

I'll signal here when this has been released. It shouldn't be long.
In the meantime, if you need a workaround you can create the StorageService yourself setting the HttpClientFactory and then use that service to create a StorageClientImpl instance. Let me know if you'd like a code snippet for that.

[amanda-tarafa]

I wrote the code for the workaround anyways and here it is, let me know if you have questions. I'll signal here when we have released the proper fix anyway.

// Users implementation of Google.Apis.Http.IHttpClientFactory.
// They can extent Google.Apis.Http.HttpClientFactory and override CreateClientHandler to
// return an HttpClientHandler with a proxy set.
var proxiedHttpClientFactory = new ProxyHttpClientFactory(proxy);

var scopes = new string[] { ... };
var serviceCredentialFromFile = (ServiceAccountCredential) GoogleCredential.FromFile(sa_file).UnderlyingCredential;
var initializer = new ServiceAccountCredential.Initializer(serviceCredentialFromFile.Id)
{
    ProjectId = serviceCredentialFromFile.ProjectId,
    Key = serviceCredentialFromFile.Key,
    User = serviceCredentialFromFile.User
    HttpClientFactory = proxiedHttpClientFactory,
    Scopes = scopes
};
var credential = GoogleCredential.FromServiceAccountCredential(new ServiceAccountCredential(initializer));

var storageService = new StorageService(new BaseClientService.Initializer
{
    HttpClientInitializer = credential,
    HttpClientFactory = HttpClientFactory
});

var client = new StorageClientImpl(storageService);

That should work for now, will update here when we have release.

[patilravikiran]

I wrote the code for the workaround anyways and here it is, let me know if you have questions. I'll signal here when we have released the proper fix anyway.

// Users implementation of Google.Apis.Http.IHttpClientFactory.
// They can extent Google.Apis.Http.HttpClientFactory and override CreateClientHandler to
// return an HttpClientHandler with a proxy set.
var proxiedHttpClientFactory = new ProxyHttpClientFactory(proxy);

var scopes = new string[] { ... };
var serviceCredentialFromFile = (ServiceAccountCredential) GoogleCredential.FromFile(sa_file).UnderlyingCredential;
var initializer = new ServiceAccountCredential.Initializer(serviceCredentialFromFile.Id)
{
    ProjectId = serviceCredentialFromFile.ProjectId,
    Key = serviceCredentialFromFile.Key,
    User = serviceCredentialFromFile.User
    HttpClientFactory = proxiedHttpClientFactory,
    Scopes = scopes
};
var credential = GoogleCredential.FromServiceAccountCredential(new ServiceAccountCredential(initializer));

var storageService = new StorageService(new BaseClientService.Initializer
{
    HttpClientInitializer = credential,
    HttpClientFactory = HttpClientFactory
});

var client = new StorageClientImpl(storageService);

That should work for now, will update here when we have release.

Thanks a lot... Are you saying new code will be available for public use in a weeks time?

[amanda-tarafa]

Yes, new code will be available in weeks time so you can use the StorageClientBuilde instead of the above workaround. I'll comment here when that's been released.

[patilravikiran]

Yes, new code will be available in weeks time so you can use the StorageClientBuilde instead of the above workaround. I'll comment here when that's been released.
Sure. Better I will wait for new code.

[patilravikiran]

Hi,
Do we have similar script for BigQueryClient? I would like to replace my existing proxy script(googleapis/google-cloud-dotnet#1832).

[amanda-tarafa]

Both the fix and the workaround will work the same for BigQueryClient. Basically replace where it says Storage... with BigQuery..., check your scopes as well which should be different.

[patilravikiran]

Hi,
I know its only 7 days since your last comment, but I would like to know if you have managed to merge your code?

[jskeet]

No, we haven't done the release yet - we have various other things that we want to get in first. (There are several layers of libraries here, and new features being added in all of them, so we want to get the whole set ready before we start releasing.) But as Amanda said, the same basic workaround should work with BigQueryClient too.

[patilravikiran]

Thanks. Can I expect final code in couple of weeks? Or it will take few months?

[jskeet]

I'd hope it will be in the next couple of weeks.

[jskeet]

@patilravikiran: the Google.Api.Gax libraries version 3.4.0 have now been pbulished. If you add an explicit dependency on Google.Api.Gax.Rest version 3.4.0, you should be able to use the builder to create a client using a proxy easily.

[amanda-tarafa]

To add to Jon's comment, your code now could look something like this:

IWebProxy proxy = ...;
Google.Apis.Http.HttpClientFactory factory = HttpClientFactory.ForProxy(proxy);

var scopes = new string[] { "https://www.googleapis.com/auth/devstorage.full_control" };
var credential= GoogleCredential.FromFile(sa_file);
credential = credential.CreateScoped(scopes).CreateWithHttpClientFactory(factory);

var clientBuilder = new StorageClientBuilder
{
    Credential = credential,
    // All HTTP client instances for API calls will be obtained from this factory
    HttpClientFactory = factory
}

var client = clientBuilder.Build();

[patilravikiran]

Hi,
Thanks for quick resolution. I was enjoying my holidays, so didn't get time to test above solution. Today when I tested above fix, I got below error. Do we have any internal dependencies?

Error Message: Could not load file or assembly 'Google.APIs, Version=1.50.0.0, Culture=neutral, PublicKeyToken=4b01fa6e34db77ab' or one of its dependencies. The system cannot find the file specified.

Packages Used:
Google.Cloud.BigQuery.V2-v2.2.0
Google.Apis-v1.52.0
Google.Api.Gax-v3.4.0

Type of Project- .net framework 4.6.2

[jskeet]

@patilravikiran: That looks like a problem with assembly binding redirects. It's looking for Google.Apis version 1.50.0, but you've got 1.52.0 - so I'd expect there to be an autogenerated redirect from "anything before 1.52.0" to 1.52.0.

(If you were to use an SDK style project, this would be completely invisible to you.) If you've got a config file with assembly binding redirects, you could try editing it by hand - or see if you can persuade VS to regenerate the redirects for you. (I've seen it get "stuck" a few times before now, but I'm afraid I don't have any recommendations for getting it to regenerate the redirects, short of basically starting with a new project.)

[patilravikiran]

@patilravikiran: That looks like a problem with assembly binding redirects. It's looking for Google.Apis version 1.50.0, but you've got 1.52.0 - so I'd expect there to be an autogenerated redirect from "anything before 1.52.0" to 1.52.0.

(If you were to use an SDK style project, this would be completely invisible to you.) If you've got a config file with assembly binding redirects, you could try editing it by hand - or see if you can persuade VS to regenerate the redirects for you. (I've seen it get "stuck" a few times before now, but I'm afraid I don't have any recommendations for getting it to regenerate the redirects, short of basically starting with a new project.)

Thanks Jon. I will look into this. Do you have any reference links for similar issue. I have tried to modify app.config of secondary (referred) project, but no luck.

[jskeet]

I don't have any references I'm afraid, no. I also don't know the details of your project set-up, or what the app.config looked like before, or how you changed it. If you could provide a minimal but complete example, I'd be happy to try to trouble-shoot it - but otherwise I'm really making suggestions in the dark.

[patilravikiran]

See below example.
I have changed old version from 0.0.0.0-1.52.0.0 to 0.0.0.0-1.50.0.0

<dependentAssembly>
<assemblyIdentity name="Google.Apis" publicKeyToken="4b01fa6e34db77ab" culture="neutral" />
<bindingRedirect old version="0.0.0.0-1.52.0.0" new version="1.52.0.0" />
</dependentAssembly>

[jskeet]

No, it needs to be the other way round - you need to redirect 1.50.0 to 1.52.0.

[jskeet]

(Although you didn't attach anything, so it's not entirely clear what you meant )

[patilravikiran]

Sorry not able to add XML content

[jskeet]

That's broadly what it should look like - but again, it would be much easier to help you if you could just provide a zip file with a minimal example in.

[patilravikiran]

I have referred below issue
googleads/googleads-dotnet-lib#210

[jskeet]

I'm not sure what you intended to upload - potentially an image - but again, please provide a zip file with the complete solution. It's very unlikely that I'll be able to help you until you do so.

[patilravikiran]

Abc.GcpConnection.zip

[patilravikiran]

Sorry, was not able to upload this solution from corporate laptop. Hence uploaded solution from personal machine.

To run solution, please follow below steps:

1. Run Abc.GcpConnection project. It will open Microsoft Visual Studio Debug Console.
2. After some time, it will show url like http://localhost:7071/api/Function1
3. Use this url and paste in browser.

[jskeet]

I'll look on Monday

[patilravikiran]

Thanks @jskeet

[jskeet]

@patilravikiran: Ah, you hadn't mentioned that this was an Azure Function. We've seen issues with Azure Functions dependency resolution before - in particular with Functions v1 which you appear to be using.

It looks like the problem you're facing is really somewhat separate to proxy settings - I see the same issue after commenting out the proxy configuration lines. I suspect you first saw this problem due to needing a new version of Google.Api.Gax.Rest, but fundamentally you've got a problem with Azure Functions not managing dependencies properly.

I suspect this would be fixed by using a newer version of Azure Functions - I believe it was fixed in Azure Functions 2.0. If you could migrate to .NET Core that would also help massively in avoiding all these problems.

This issue isn't specific to Google libraries at all - see https://codopia.wordpress.com/2017/07/21/how-to-fix-the-assembly-binding-redirect-problem-in-azure-functions/ for some details and a workaround.

I'll close this issue now as I believe this is more of a problem with Azure Functions v1 than with our libraries.

[patilravikiran]

Yes thanks @jskeet. I am using Azure function version 1 for obvious reasons. As there is no option, I will stick to earlier workaround shared by you.