score:1
Why not try something like:
//Load all the database entries into Memory
List<string> prodASINs = dc.aboProducts.Select(a => a.asin).ToList();
var count = prodASINs.Count();
//Loop through passing 10 at a time to AWS
for (var i = 0; i < count; i++)
{
var prodASINToSend = prodASINs.Skip(i * 10).Take(10).ToList();
//Send to AWS
}
Or if you don't want to load them all into memory.
var count = dc.aboProducts.Count();
for (var i = 0; i < count; i++)
{
List<string> prodASIN = dc.aboProducts.OrderBy(a => a.Id).Select(a => a.asin).Skip(i * 10).Take(10).ToList();
//Send to AWS
}
score:-1
List<T>
has a built-in function called GetRange()
which was made specifically for what you're trying to do. It's extremely fast and doesn't need Linq, casting, etc...
List<string> prodASINs = dc.aboProducts.Select(a => a.asin).ToList();
for(int i = 0; i < prodASINs.Count; i += 10)
{
List<string> buffer = prodASINs.GetRange(i, 10);
// do something with buffer
}
That's it. Very simple.
Test results: GetRange
vs. Slice
vs. Linq
with 5000 strings in List<string>
As you can clearly see, the Skip/Take approach using Linq is over 383 times slower than Slice<T>()
and 4,736 times slower than GetRange()
==================================================================================
GetRange took on average 168 ticks
Slice took on average 2073 ticks
Linq took on average 795643 ticks
Test method used (try it yourself):
private static void GetRangeVsSliceVsLinq()
{
List<string> stringList = new List<string>();
for (int i = 0; i < 5000; i++)
{
stringList.Add("This is a test string " + i.ToString());
}
Stopwatch sw = new Stopwatch();
long m1 = 0, m2 = 0, m3 = 0;
for (int x = 0; x < 10; x++)
{
Console.WriteLine("Iteration {0}", x + 1);
Console.WriteLine();
sw.Reset();
sw.Start();
for (int i = 0; i < stringList.Count; i += 10)
{
List<string> buffer = stringList.GetRange(i, 10);
}
sw.Stop();
Console.WriteLine("GetRange took {0} msecs", sw.ElapsedMilliseconds);
Console.WriteLine("GetRange took {0} ticks", sw.ElapsedTicks);
m1 += sw.ElapsedTicks;
sw.Reset();
sw.Start();
string[] sliceArray = stringList.ToArray();
for (int i = 0; i < sliceArray.Length; i += 10)
{
List<string> buffer = sliceArray.Slice(i, 10).ToList();
}
sw.Stop();
Console.WriteLine("Slice took {0} msecs", sw.ElapsedMilliseconds);
Console.WriteLine("Slice took {0} ticks", sw.ElapsedTicks);
m2 += sw.ElapsedTicks;
sw.Reset();
sw.Start();
var count = stringList.Count();
for (var i = 0; i < count; i++)
{
var buffer = stringList.Skip(i * 10).Take(10).ToList();
}
sw.Stop();
Console.WriteLine("Skip/Take took {0} msecs", sw.ElapsedMilliseconds);
Console.WriteLine("Skip/Take took {0} ticks", sw.ElapsedTicks);
m3 += sw.ElapsedTicks;
Console.WriteLine();
}
Console.WriteLine();
Console.WriteLine("GetRange took on average {0} ticks", m1 / 10);
Console.WriteLine("Slice took on average {0} ticks", m2 / 10);
Console.WriteLine("Linq took on average {0} ticks", m3 / 10);
}
score:0
Slice Extension (for Arrays):
public static T[] Slice<T>(this T[] source, int index, int length)
{
T[] slice = new T[length];
Array.Copy(source, index, slice, 0, length);
return slice;
}
Array.Copy is extremely fast, a lot faster than the Select/Skip/Take pattern. Although this method is not the fasted I've found, recents tests show that it's nearly 400 times faster than the Skip/Take pattern used to split Lists and Arrays.
To use it as is:
const int arraySize = 10;
List<string> listSource = whatever;
string[] source = listSource.ToArray();
for (int i = 0; i < source.Length; i += arraySize)
{
List<string> buffer = source.Slice(i, arraySize).ToList();
DoSomething(buffer);
}
score:1
Sorry this isnt LINQ specific, but perhaps it will help...
One of the things I have done when working with data with MWS and ERP software is adding a control column to the database, something like "addedASIN'. In the database I define the control column as a boolean value ( or TINYINT(1) in MySQL ) and default the flag to 0 for all new entries and set it to 1 when the entry has been added.
If you are able to do that then you can do something like
SELECT asin FROM datasource WHERE addedASIN = 0 LIMIT 10;
Then once MWS returns successful for the additions update the flag using
UPDATE datasource SET addedASIN = 1 WHERE asin = 'asinnumber';
The benefit I have found with this is that your database will be able to stop and start with a minimal repetition of data - for instance in my case ( and what started this control column ) our network connection can be flaky, so I was finding during order imports I would lose connectivity resulting in lost orders, or orders being uploaded to our system twice.
This solution has mitigated that by having at most 1 order being added twice as a result of a connectivity loss, and in order for that order to be uploaded twice, connectivity needs to be lost between sending the data to our ERP system, our ERP system acknowledging it was added and the database being updated, which for a round trip takes approximately 30 seconds.
score:2
I know the question is answered but I can't withhold from you this little extension method I once made and that has served me well since.
You can do:
foreach(var list in prodASINs.ToChunks(10))
{
// send list
}
Source: stackoverflow.com
Related Query
- Split results from DB into "chunks" of 10
- Split an IEnumerable<T> into fixed-sized chunks (return an IEnumerable<IEnumerable<T>> where the inner sequences are of fixed length)
- Accessing Results View from variable in code
- creating Linq to sqlite dbml from DbLinq source code
- Does LINQ load all items from SQL Server into memory or chunks only?
- Linq/C#: How to split List into variable length chunks based on list item information?
- Can't add a new record with an integer value into database by using linq from code C#
- Data from List<T> split into another List<T> with 2 nested List<T>
- C# Group results from List<T> into another List<T2> with selected fields
- Split into xml files retaining few tags from base xml file using linq
- c# Linq or code to extract groups from a single list of source data
- Split CSV and concat suffix from multiple properties into a list
- Moving code from the view into the LINQ query
- Split List into Sublists with LINQ
- Split a collection into `n` parts with LINQ?
- Limit Number of Results being returned in a List from Linq
- Can I split an IEnumerable into two by a boolean criteria without two queries?
- LINQ: How to get items from an inner list into one list?
- How To Project a Line Number Into Linq Query Results
- How can I split an IEnumerable<String> into groups of IEnumerable<string>
- How to split an array into a group of n elements each?
- Split string into list of N-length strings using LINQ
- Split a list into multiple lists at increasing sequence broken
- Combining The Results Of Two Linq Queries Into A Single Var?
- Split datatable into multiple fixed sized tables
- How to return a page of results from SQL?
- How do I load these LINQ results into my ViewModel class?
- Enumerable.Empty<T>().AsQueryable(); This method supports the LINQ to Entities infrastructure and is not intended to be used directly from your code
- Generating the Shortest Regex Dynamically from a source List of Strings
- Split a List<int> into groups of consecutive numbers
More Query from same tag
- How to convert a group_concat, join, and union query into linq lambda?
- How do I convert this looped code to a single LINQ implementation?
- PLinq parallel summing performance inconsistency
- Combining two streams of numbers using linq or rx
- Unable to cast object of type 'System.Collections.Generic.List TO System.Collections.Generic.IList
- LINQ - Left Outer Join with multiple parameters in Where clause
- Iterating Linq result set using indexers
- Anonymous type and intersection of 2 lists
- How to merge two lists of different objects?
- Reading an Xml to the DataTable
- How to use linqkit like
- Query a list LINQ-style in PowerShell
- Deleting elements from dictionary
- Updating a SQL table through DataGrid in WPF - I cannot get the updated data from the datagrid
- How can I sort a list of images by the number of matching tags?
- what' the simplest way to bind my grid view on one record(distinct different data)
- Linq -- Join on joined table's primary key
- How to combine two lists without duplication? (possible dup)
- Is there a cleaner and faster way to order this query in Entity Framework?
- select multiple column from include table in LINQ
- orderby Count(other_table) in linq query
- Calculating Weighted Average with LINQ
- Expression.IsFalse is Unknown?
- Entity framework LINQ where attribute equals of ICollection
- How to find if a string exists only with numbers? (in LINQ)
- OrderBy an INT Column in LINQ to Entities using dynamic Expression in C#
- Split a line from a CSV to a List<Key, Value>
- Expression Trees with subquery
- use substring and ternary operator linq
- How to return a list of strings using LINQ