Ggthjy

How can I block email signup overlays or javascript popups in Safari?

How to implement a feedback to keep the DC gain at zero for this conceptual passive filter?

Strong empirical falsification of quantum mechanics based on vacuum energy density

Is Witten's Proof of the Positive Mass Theorem Rigorous?

What should you do when eye contact makes your subordinate uncomfortable?

How do you respond to a colleague from another team when they're wrongly expecting that you'll help them?

Infinite dials to reset ever?

A social experiment. What is the worst that can happen?

Basic combinatorial probability problem

How could a planet have erratic days?

What is this called? Old film camera viewer?

Is it better practice to read straight from sheet music rather than memorize it?

The probability of Bus A arriving before Bus B

Store Credit Card Information in Password Manager?

Creepy dinosaur pc game identification

How should I respond when I lied about my education and the company finds out through background check?

What is the evidence for the "tyranny of the majority problem" in a direct democracy context?

What does "Scientists rise up against statistical significance" mean? (Comment in Nature)

What does routing an IP address mean?

Is (0,1] a closed or open set?

Why electric field inside a cavity of a non conducting not zero

Rising and falling intonation

If a character has darkvision, can they see through an area of nonmagical darkness filled with lightly obscuring gas?

Open a doc from terminal, but not by its name

Sending device data of 10k items with Parallel.ForEach

Implementation of Parallel.ForEach for .NET 3.5Queue with task parallel libraryMonitoring progress in Parallel.ForEach every minuteSpeeding up Parallel.ForEach iterating through datatable and rendering reportNon-blocking, non-threaded HTTP client implementationUsing BlockingCollection with Parallel.ForEachParsing NuGet packages.config with Parallel.ForEachCreating membership list with Parallel.ForEachReceiving and sending notifications from databaseSocket sending/receiving data

2

1

$begingroup$

I have a deviceList of more than 10k items and want to send data by calling another method.

I tried to use Parallel.Foreach but I'm not sure is this the correct way to do it.

I have published this webapp on azure,
I have tested this for 100 it works fine but for 10k it got timeout issue.

This is working code , I need a improvement here :)

private List<Task> taskEventList = new List<Task>();

public async Task ProcessStart()

{

    string messageData = "{"name":"DemoData","no":"111"}";

    RegistryManager registryManager;



    Parallel.ForEach(deviceList, async (device) =>

    {

        // get details for each device and use key to send message

        device = await registryManager.GetDeviceAsync(device.DeviceId);

        SendMessages(device.DeviceId, device.Key, messageData);

    });



    if (taskEventList.Count > 0)

    {

        await Task.WhenAll(taskEventList);

    }

}



private void SendMessages(string deviceId, string Key, string messageData)

{

    DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);

    //created separate Task

    var taskEvents = Task.Run(() => ProcessMessages(deviceId, string messageData));

    taskEventList.Add(taskEvents);

}





private async Task ProcessMessages(string deviceId, string messageData)

{

    var startTime = DateTime.UtcNow;

    while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))

    {

        await deviceClient.SendEventAsync(messageData);

    }

}

c# performance multithreading task-parallel-library azure

share|improve this question

edited Mar 15 at 18:46

t3chb0t

35k752124

asked Mar 15 at 16:58

NeoNeo

1114

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  You are probably getting into a thread starvation as you just scheduling tasks and also borrowing threads from the thread pool. You need some kind of batching or throttling, as the thread pool is limited ( a thread has 2 MB, imagine 10k requests for threads)
  $endgroup$
  – Adrian Iftode
  Mar 15 at 17:34
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Also if DeviceClient.Create creates and disposes a new HttpClient everytime, then you'll get out of available sockets
  $endgroup$
  – Adrian Iftode
  Mar 15 at 17:40
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Hey @AdrianIftode that seems to be the beginnings of an answer. Are you sure you don't want to write one, now that the post has been reopened?
  $endgroup$
  – Vogel612♦
  Mar 15 at 18:47
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks, I would like also to know about DeviceClient.Create, possible @Neo ?
  $endgroup$
  – Adrian Iftode
  Mar 15 at 20:40
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @Neo, are you sure everything is ok with this check: while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15)) ?
  $endgroup$
  – Adrian Iftode
  Mar 15 at 21:07
  

  
  
  
  

 | 
show 6 more comments

2

1

$begingroup$

I have a deviceList of more than 10k items and want to send data by calling another method.

I tried to use Parallel.Foreach but I'm not sure is this the correct way to do it.

I have published this webapp on azure,
I have tested this for 100 it works fine but for 10k it got timeout issue.

This is working code , I need a improvement here :)

private List<Task> taskEventList = new List<Task>();

public async Task ProcessStart()

{

    string messageData = "{"name":"DemoData","no":"111"}";

    RegistryManager registryManager;



    Parallel.ForEach(deviceList, async (device) =>

    {

        // get details for each device and use key to send message

        device = await registryManager.GetDeviceAsync(device.DeviceId);

        SendMessages(device.DeviceId, device.Key, messageData);

    });



    if (taskEventList.Count > 0)

    {

        await Task.WhenAll(taskEventList);

    }

}



private void SendMessages(string deviceId, string Key, string messageData)

{

    DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);

    //created separate Task

    var taskEvents = Task.Run(() => ProcessMessages(deviceId, string messageData));

    taskEventList.Add(taskEvents);

}





private async Task ProcessMessages(string deviceId, string messageData)

{

    var startTime = DateTime.UtcNow;

    while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))

    {

        await deviceClient.SendEventAsync(messageData);

    }

}

c# performance multithreading task-parallel-library azure

share|improve this question

edited Mar 15 at 18:46

t3chb0t

35k752124

asked Mar 15 at 16:58

NeoNeo

1114

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  You are probably getting into a thread starvation as you just scheduling tasks and also borrowing threads from the thread pool. You need some kind of batching or throttling, as the thread pool is limited ( a thread has 2 MB, imagine 10k requests for threads)
  $endgroup$
  – Adrian Iftode
  Mar 15 at 17:34
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Also if DeviceClient.Create creates and disposes a new HttpClient everytime, then you'll get out of available sockets
  $endgroup$
  – Adrian Iftode
  Mar 15 at 17:40
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Hey @AdrianIftode that seems to be the beginnings of an answer. Are you sure you don't want to write one, now that the post has been reopened?
  $endgroup$
  – Vogel612♦
  Mar 15 at 18:47
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks, I would like also to know about DeviceClient.Create, possible @Neo ?
  $endgroup$
  – Adrian Iftode
  Mar 15 at 20:40
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @Neo, are you sure everything is ok with this check: while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15)) ?
  $endgroup$
  – Adrian Iftode
  Mar 15 at 21:07
  

  
  
  
  

 | 
show 6 more comments

$begingroup$

I have a deviceList of more than 10k items and want to send data by calling another method.

I tried to use Parallel.Foreach but I'm not sure is this the correct way to do it.

I have published this webapp on azure,
I have tested this for 100 it works fine but for 10k it got timeout issue.

This is working code , I need a improvement here :)

private List<Task> taskEventList = new List<Task>();

public async Task ProcessStart()

{

    string messageData = "{"name":"DemoData","no":"111"}";

    RegistryManager registryManager;



    Parallel.ForEach(deviceList, async (device) =>

    {

        // get details for each device and use key to send message

        device = await registryManager.GetDeviceAsync(device.DeviceId);

        SendMessages(device.DeviceId, device.Key, messageData);

    });



    if (taskEventList.Count > 0)

    {

        await Task.WhenAll(taskEventList);

    }

}



private void SendMessages(string deviceId, string Key, string messageData)

{

    DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);

    //created separate Task

    var taskEvents = Task.Run(() => ProcessMessages(deviceId, string messageData));

    taskEventList.Add(taskEvents);

}





private async Task ProcessMessages(string deviceId, string messageData)

{

    var startTime = DateTime.UtcNow;

    while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))

    {

        await deviceClient.SendEventAsync(messageData);

    }

}

c# performance multithreading task-parallel-library azure

share|improve this question

edited Mar 15 at 18:46

t3chb0t

35k752124

asked Mar 15 at 16:58

NeoNeo

1114

$endgroup$

private List<Task> taskEventList = new List<Task>();

public async Task ProcessStart()

{

    string messageData = "{"name":"DemoData","no":"111"}";

    RegistryManager registryManager;



    Parallel.ForEach(deviceList, async (device) =>

    {

        // get details for each device and use key to send message

        device = await registryManager.GetDeviceAsync(device.DeviceId);

        SendMessages(device.DeviceId, device.Key, messageData);

    });



    if (taskEventList.Count > 0)

    {

        await Task.WhenAll(taskEventList);

    }

}



private void SendMessages(string deviceId, string Key, string messageData)

{

    DeviceClient deviceClient = DeviceClient.Create(hostName, new DeviceAuthenticationWithRegistrySymmetricKey(deviceId, deviceKey), Microsoft.Azure.Devices.Client.TransportType.Mqtt);

    //created separate Task

    var taskEvents = Task.Run(() => ProcessMessages(deviceId, string messageData));

    taskEventList.Add(taskEvents);

}





private async Task ProcessMessages(string deviceId, string messageData)

{

    var startTime = DateTime.UtcNow;

    while (DateTime.UtcNow - startTime < TimeSpan.FromMinutes(15))

    {

        await deviceClient.SendEventAsync(messageData);

    }

}

share|improve this question

edited Mar 15 at 18:46

t3chb0t

35k752124

asked Mar 15 at 16:58

NeoNeo

1114

share|improve this question

edited Mar 15 at 18:46

t3chb0t

35k752124

asked Mar 15 at 16:58

NeoNeo

1114

edited Mar 15 at 18:46

t3chb0t

35k752124

edited Mar 15 at 18:46

t3chb0t

35k752124

asked Mar 15 at 16:58

NeoNeo

1114

asked Mar 15 at 16:58

NeoNeo

1114