Azure Table storage is a way of storing structured NoSQL data in the cloud, as such it’s more geared towards rapid read access rather than manipulation of data in the table.

If you want to perform any operations such as updates or deletes on data in your table then the most efficient way to do this is as a batch operation which groups up to 100 operations into a single batch provided they all have the same partition key.

In my case I wanted to download all rows from a history table before the current point in time and to then delete them from the history table.

Initially I get all of the history messages (the row key in this table is a timestamp and the partition key is a customer ID).

public async Task<List<HistoryEntity>> GetHistoryMessagesAsync(string tableName, CloudStorageAccount storageAccount, string timestamp)
	List<HistoryEntity> historyEntities = new List<HistoryEntity>();
	CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
	CloudTable historyTable = tableClient.GetTableReference(tableName);

	TableQuery<historyTable> dowloadQuery = new TableQuery<historyTable>().Where(TableQuery.GenerateFilterCondition("RowKey", QueryComparisons.LessThan, timestamp));
	TableContinuationToken continuationToken = null;

		var tableQueryResult = historyTable.ExecuteQuerySegmentedAsync(dowloadQuery, continuationToken);

		continuationToken = tableQueryResult.Result.ContinuationToken;

		foreach (HistoryEntity result in tableQueryResult.Result)
	while (continuationToken != null);

	return historyEntities;

I then get all of the distinct partition keys from my entities so that I can batch up all of my deletes into as few operations as possible.

List<HistoryEntity> historyEntities = GetHistoryMessagesAsync("HistoryTable", storageAccount, "2019-03-07 22:10:53.8785130");
List<string> partitionKeys = historyEntities.Select(x => x.PartitionKey).Distinct();

In order to most efficiently delete these keys I want to create batches of 100 or fewer entities, if there’s more than 100 records belonging to a partition key then I want to split these into separate batches.

public async Task DeleteHistoryMessagesAsync(string partitionKey, CloudTable historyTable)
	TableQuery<TableEntity> deleteQuery = new TableQuery<TableEntity>()
			TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, partitionKey),
			TableQuery.GenerateFilterCondition("RowKey", QueryComparisons.LessThan, timestamp)
		.Select(new string[] { "PartitionKey", "RowKey" });

	TableContinuationToken continuationToken = null;

		var tableQueryResult = historyTable.ExecuteQuerySegmentedAsync(deleteQuery, continuationToken);

		continuationToken = tableQueryResult.Result.ContinuationToken;

		// Split into chunks of 100 for batching
		List<List<TableEntity>> rowsChunked = tableQueryResult.Result.Select((x, index) => new { Index = index, Value = x })
			.Where(x => x.Value != null)
			.GroupBy(x => x.Index / 100)
			.Select(x => x.Select(v => v.Value).ToList())

		// Delete each chunk of 100 in a batch
		foreach (List<TableEntity> rows in rowsChunked)
			TableBatchOperation tableBatchOperation = new TableBatchOperation();
			rows.ForEach(x => tableBatchOperation.Add(TableOperation.Delete(x)));

			await historyTable.ExecuteBatchAsync(tableBatchOperation);
	while (continuationToken != null);

We can also run these deletes in parallel using TPL to further speed up the processing.

// Create a block with an asynchronous action
var block = new ActionBlock<(string partitionKey, CloudTable historyTable)>(
	async x => await DeleteHistoryMessagesAsync(x.partitionKey, x.historyTable),
	new ExecutionDataflowBlockOptions
		BoundedCapacity = 100, // Cap the item count
		MaxDegreeOfParallelism = 16

foreach (string partitionKey in partitionKeys)
	await block.SendAsync((partitionKey, historyTable));

await block.Completion;


Diego Guerrero · 3rd September 2019 at 4:39 pm

Great Post, I had a similar problem (Only delete data with multiple filters), but this solution was solved.

Thanks a lot

Robert Maurer · 4th November 2019 at 6:31 pm

Great post. I also solved this, just not as elegantly as you did. How do you get around the the the request unit limit when you have 16 threads query and deleting data simultaneously? It seems that unless I set my RU to 10,000, I still get 429 errors.

    Shinigami · 5th November 2019 at 9:30 am

    I haven’t actually hit the request limit yet, I haven’t made any specific changes to avoid this so it’s likely that the number of rows I need to delete hasn’t been high enough to trigger it.

    It sounds like the request limit is there to prevent DDoS attacks so there’s likely not much that can be done about it, though maybe requests coming from within Azure are treated more leniently than external requests.

    An option would be to partition your data by table and then drop tables when they’re no longer needed though this probably works best for scenarios where you only want to keep data for a limited time period as you could create a separate table for each day and then drop all tables older than your cut off.

Ramesh Janjyam · 9th July 2020 at 3:29 pm

thank you. this article helped me today

Leave a Reply

Your email address will not be published. Required fields are marked *