Outputting a SQL table to a CSV file on an SFTP site is pretty much the same as outputting it to a local CSV file but with the added step of using SSH.NET to upload the stream. Full code for this can be found here. GetSqlRowsAsync Usage Usage – GZip Read more…
This post demonstrates how write a SQL table out to a local CSV file, the full code for this can be found here. GetSqlRowsAsync This returns an IAsyncEnumerable of DataRows, IAsyncEnumerable has been newly introduced with C# 8.0 and it allows results to be streamed back to the caller as Read more…
Similar to how I’m uploading data to an SFTP the below example streams a SQL table directly to a Google Cloud Storage object. Full code for this is here. Usage Here the StreamWriter needs to be flushed before the upload to GCM is initiated as otherwise the StorageClient will finish Read more…
Moving files from Google Cloud Storage to Azure Blob Storage is pretty simple as they can just be streamed directly. My source file in Google Cloud Storage was a compressed gzip file which I needed to be unzipped at the Azure Blob Storage destination. Usage
In order to transfer data from a Google BigQuery table into SQL I first export it to Google Cloud storage and then stream it directly into into my SQL table using SqlBulkCopy and a modified version of the CsvDataReader from CsvHelper. This post demonstrates how to load data from Google Read more…
As part of a recent project I’ve had to download a gzipped file of product details from a client endpoint and to load these products into a local SQL database. Although this is easy enough to do in discrete synchronous steps as below what I really wanted to do was Read more…
If you have a blob in Azure Storage that you want to have a look at first before deciding whether to download it or not you can open up a stream from it and use a StreamReader to read as many rows as you want from it. In my case Read more…
You can host images (or other files) in Azure Storage by making the container with them in public but if you want them to be private and only accessible to authorized users then you can either generate a short-lived SAS key granting read permissions on the blob or you can Read more…
The Azure computer vision API can be used to retrieve information about an image including tags, descriptions and categories along with their associated confidence scores. The Microsoft quickstart guide to using this is here and my releated code is below. This retrieves an image file from Azure Storage and streams Read more…
Azure Data Warehouse has a nice feature where it can access GZipped file blobs directly from Azure Storage as linked tables. The below code is for downloading zipped files from an SFTP site, extracting them, gzipping them and then uploading them to Azure Storage. This all takes place in memory Read more…