In order to transfer data from a Google BigQuery table into SQL I first export it to Google Cloud storage and then stream it directly into into my SQL table using SqlBulkCopy and a modified version of the CsvDataReader from CsvHelper. This post demonstrates how to load data from Google Read more…
CsvHelper is handy for reading and writing CSV files but there’s no obvious way to write out a DataTable to a CSV file. Therefore in order to do this we need to loop through all the rows and columns of the DataTable and write the values as individual fields as Read more…
Using SQL Task Results in a Script Task in SSDT In SSDT if you want to access the results of a SQL query in a script task you can do this by outputting the results as a full result set into a variable of type Object and then load this Read more…
Thread Safe DataTable Inserts and SqlBulkCopy As part of a project where I was running multiple threads making API calls like this I was making a stored procedure call to log the results to an SQL database for each result returned. This worked fine when a low number of threads Read more…
Checking for CSV Column Headers in C# Importing CSV files into a .NET object isn’t too tricky and for this I usually use CsvHelper. However, sometimes I need to import a CSV file and only extract a couple of columns from it and these columns aren’t always guaranteed to exist. Read more…