There are various ways of authenticating against Azure Functions but the default method is by providing a code value as a URL parameter in the call to the endpoint. If you’re using RestSharp to call your function there’s no specific authenticator to manage handling this parameter but it is pretty Read more
In the v3 version of Azure Functions DI is better supported and there’s now some offical documentation demonstrating how it’s done. It’s pretty similar to how it was done previously, below is updated code from my previous v2 example. The IFunctionsHostBuilder instance is created by the Azure functions host and Read more
I’ve been using IAsyncEnumerable recently for the streaming of SQL query rows and of deserialized objects from API calls and while it seems to work like a charm it does seem to have a few configuration issues. Firstly, on creating a new .NET Core project in Visual Studio 2019 (v16.3.5) Read more
Outputting a SQL table to a CSV file on an SFTP site is pretty much the same as outputting it to a local CSV file but with the added step of using SSH.NET to upload the stream. Full code for this can be found here. GetSqlRowsAsync Usage Usage – GZip Read more
This post demonstrates how write a SQL table out to a local CSV file, the full code for this can be found here. GetSqlRowsAsync This returns an IAsyncEnumerable of DataRows, IAsyncEnumerable has been newly introduced with C# 8.0 and it allows results to be streamed back to the caller as Read more
Similar to how I’m uploading data to an SFTP the below example streams a SQL table directly to a Google Cloud Storage object. Full code for this is here. Usage Here the StreamWriter needs to be flushed before the upload to GCM is initiated as otherwise the StorageClient will finish Read more
Moving files from Google Cloud Storage to Azure Blob Storage is pretty simple as they can just be streamed directly. My source file in Google Cloud Storage was a compressed gzip file which I needed to be unzipped at the Azure Blob Storage destination. Usage
As I seem to have been shuffling a lot of data about recently I thought I’d try to document as much of it as possible. Google BigQuery to Google Cloud Storage Google Cloud Storage to SQL SQL to Google Cloud Storage Google Cloud Storage to Azure Blob Storage SQL to Read more
In order to transfer data from a Google BigQuery table into SQL I first export it to Google Cloud storage and then stream it directly into into my SQL table using SqlBulkCopy and a modified version of the CsvDataReader from CsvHelper. This post demonstrates how to load data from Google Read more
There is a variety of ways of getting data out of BigQuery using the API including streaming the results into memory and creating a file for download, however if you need to download a large amount of data then this will initially need to be exported to Google Cloud Storage Read more