In order to transfer data from a Google BigQuery table into SQL I first export it to Google Cloud storage and then stream it directly into into my SQL table using SqlBulkCopy and a modified version of the CsvDataReader from CsvHelper. This post demonstrates how to load data from Google Read more…
CsvHelper is handy for reading and writing CSV files but there’s no obvious way to write out a DataTable to a CSV file. Therefore in order to do this we need to loop through all the rows and columns of the DataTable and write the values as individual fields as Read more…
In SSDT if you want to access the results of a SQL query in a script task you can do this by outputting the results as a full result set into a variable of type Object and then load this into a DataTable within your script task. It’s also possible Read more…
As part of a project where I was running multiple threads making API calls like this I was making a stored procedure call to log the results to an SQL database for each result returned. This worked fine when a low number of threads was used but then started to Read more…
Importing CSV files into a .NET object isn’t too tricky and for this I usually use CsvHelper. However, sometimes I need to import a CSV file and only extract a couple of columns from it and these columns aren’t always guaranteed to exist. In order to do this I use Read more…