Similar to how I’m uploading data to an SFTP the below example streams a SQL table directly to a Google Cloud Storage object. Full code for this is here. Usage Here the StreamWriter needs to be flushed before the upload to GCM is initiated as otherwise the StorageClient will finish Read more…
Moving files from Google Cloud Storage to Azure Blob Storage is pretty simple as they can just be streamed directly. My source file in Google Cloud Storage was a compressed gzip file which I needed to be unzipped at the Azure Blob Storage destination. Usage
As I seem to have been shuffling a lot of data about recently I thought I’d try to document as much of it as possible. Google BigQuery to Google Cloud Storage Google Cloud Storage to SQL SQL to Google Cloud Storage Google Cloud Storage to Azure Blob Storage SQL to Read more…
In order to transfer data from a Google BigQuery table into SQL I first export it to Google Cloud storage and then stream it directly into into my SQL table using SqlBulkCopy and a modified version of the CsvDataReader from CsvHelper. This post demonstrates how to load data from Google Read more…
There is a variety of ways of getting data out of BigQuery using the API including streaming the results into memory and creating a file for download, however if you need to download a large amount of data then this will initially need to be exported to Google Cloud Storage Read more…
As part of a recent project I’ve had to download a gzipped file of product details from a client endpoint and to load these products into a local SQL database. Although this is easy enough to do in discrete synchronous steps as below what I really wanted to do was Read more…
I usually access Azure Blob Storage by using a SAS key but I am trying to move everything over to KeyVault and Managed Service Identities (MSI) to improve security and secret management. During this process I kept on receiving the following error. I could connect to the container using my Read more…
This is a list of useful articles which I’ll try and update when I find them rather than immediately forgetting them after reading! 2020/01/22 – Designing a microservices architecture A basic introduction to designing a microservice architecture. 2020/01/22 – Make secure .NET Microservices and Web Applications An overview of how Read more…
While there seems to be plenty of resources and guides out there to help users create an API there seems to be much less guidance on how to create a client for accessing an API. I’m fine with calling an API using HttpClient or through a library such as RestSharp Read more…
I’ve recently started getting to grips with the API Management service in Azure which seems to be a good way of managing access to resources and to document their usage. Part of this is the Developer Portal which acts as a centralized store of knowledge for using your APIs which Read more…