Uncategorised
Data Transfer
As I seem to have been shuffling a lot of data about recently I thought I’d try to document as much of it as possible. Google BigQuery to Google Cloud Storage Google Cloud Storage to SQL SQL to Google Cloud Read more…
Programming
Data Transfer – Google Cloud Storage to SQL
In order to transfer data from a Google BigQuery table into SQL I first export it to Google Cloud storage and then stream it directly into into my SQL table using SqlBulkCopy and a modified version of the CsvDataReader from Read more…
Programming
Data Transfer – Google BigQuery to Google Cloud Storage
There is a variety of ways of getting data out of BigQuery using the API including streaming the results into memory and creating a file for download, however if you need to download a large amount of data then this Read more…
Programming
Streaming API Data into SQL
As part of a recent project I’ve had to download a gzipped file of product details from a client endpoint and to load these products into a local SQL database. Although this is easy enough to do in discrete synchronous Read more…
Azure
Azure Storage Authentication Using Managed Service Identities
I usually access Azure Blob Storage by using a SAS key but I am trying to move everything over to KeyVault and Managed Service Identities (MSI) to improve security and secret management. During this process I kept on receiving the Read more…
Uncategorised
Useful Reading
This is a list of useful articles which I’ll try and update when I find them rather than immediately forgetting them after reading! 2020/01/22 – Designing a microservices architecture A basic introduction to designing a microservice architecture. 2020/01/22 – Make Read more…
Programming
Creating a Client for Accessing an API in .NET Core
While there seems to be plenty of resources and guides out there to help users create an API there seems to be much less guidance on how to create a client for accessing an API. I’m fine with calling an Read more…
Azure
Disabling Signup in the Azure API Management Developer Portal
I’ve recently started getting to grips with the API Management service in Azure which seems to be a good way of managing access to resources and to document their usage. Part of this is the Developer Portal which acts as Read more…
Programming
Publishing Referenced Projects to Azure Web Apps
I’ve been updating our company’s Christmas bake-off site and as part of the update I added the Microsoft.Identity.Web project to help with authentication and accessing the Graph API. This project doesn’t currently have a NuGet package though one is apparently Read more…
Programming
Passing NULL Values to a SQL Stored Procedure from .NET Core
I’ve previously posted about how to cope with nullable values retrieved from a SqlDataReader but it’s also necessary to account for nulls when passing values through to a SQL operation such as a stored procedure. sp__set_completed.sql If you want you’re Read more…