Hello guys, its me again 
Thanks to god for this forum, or else i would be screwed. I am making an API call, where it has a ton of data ( converted to data table, 50k rows). And the thing is, that it almost has 250 pages. I wanted to go for a quicker approach and use parallel for each for the HTTP request, looping it with the page numbers, but i cant even imagine, how to store the XML format answers that i will be getting from each of the calls. Is there a way to store them all into one big XML format file and then extract the needed data using invoke code activity? Your help is as always, highly appreciated. Thanks!
Hi! Just thinking out loud⌠what if you treat the extraction from each page as an individual transactional activity which writes the answer to an XML file?
For example, each parallel HTTP request could create one XML file in a target folder.
By the end of your for each loop, the folder will contain an XML file for each page.
Then you could just loop through the files in this folder to get the info you need.
However Iâm not sure whether Iâve 100% grasped your need, so donât hesitate to share more info if you believe I went off track. 
1 Like
As you are converting to datatableâŚbetter would be inside parallel for each use your conversion steps also and use a merge datatable activity âŚso that you get all the data merged into one table
Cheers
2 Likes
Thanks for your suggestion @Irene,
But I still dont understand one step of your suggestion. How can I store the answers into a file using HTTP request in parallel for each? Is there a parameter, that the HTTP request activity would just place a txt file with the answer? Thanks
Thanks for your suggestion @Anil_G ,
But i still have some forward questions for you. How should the workflow look like then? For example, I use invoke code to extract the needed data from the XML answer and store it in a data table. If i put HTTP request, invoke code, merge datatable activities in a parallel for each, there wont be no sequence. Could you please be so kind and share a screenshot of how the workflow could look like? Thanks!
If it would help, I will share how I understand the workflow should go and what code I use. I have it like this now:
Invoke code (extracts needed data from XML and places in datatable):
string xmlResponse = @in_XMLResponse;
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.LoadXml(xmlResponse);
XmlNamespaceManager nsManager = new XmlNamespaceManager(xmlDoc.NameTable);
nsManager.AddNamespace("ebay", "urn:ebay:apis:eBLBaseComponents");
XmlNode totalPagesNode = xmlDoc.SelectSingleNode("//ebay:PaginationResult/ebay:TotalNumberOfPages", nsManager);
int totalNumberOfPages = 0;
if (totalPagesNode != null && int.TryParse(totalPagesNode.InnerText, out totalNumberOfPages))
{
// Assign the parsed total number of pages to an output variable
Out_TotalNumberOfPages = totalNumberOfPages;
}
else
{
// Handle the error or assign a default value if needed
Out_TotalNumberOfPages = -1; // example error value
}
dataTable = new DataTable();
dataTable.Columns.Add("ItemID", typeof(string));
dataTable.Columns.Add("Title", typeof(string));
dataTable.Columns.Add("SKU", typeof(string));
dataTable.Columns.Add("CurrentPrice", typeof(string));
dataTable.Columns.Add("BuyItNowPrice", typeof(string));
dataTable.Columns.Add("StartTime", typeof(string));
dataTable.Columns.Add("ViewItemURL", typeof(string));
dataTable.Columns.Add("ListingDuration", typeof(string));
dataTable.Columns.Add("Quantity", typeof(string));
dataTable.Columns.Add("ShippingServiceCost", typeof(string));
dataTable.Columns.Add("QuantityAvailable", typeof(string));
dataTable.Columns.Add("ShippingProfileName", typeof(string));
dataTable.Columns.Add("ReturnProfileName", typeof(string));
dataTable.Columns.Add("PaymentProfileName", typeof(string));
// Summary details
dataTable.Columns.Add("ActiveAuctionCount", typeof(int));
dataTable.Columns.Add("AuctionSellingCount", typeof(int));
dataTable.Columns.Add("TotalAuctionSellingValue", typeof(decimal));
dataTable.Columns.Add("TotalSoldCount", typeof(int));
dataTable.Columns.Add("TotalSoldValue", typeof(decimal));
dataTable.Columns.Add("SoldDurationInDays", typeof(int));
XmlNodeList itemNodes = xmlDoc.SelectNodes("//ebay:ActiveList/ebay:ItemArray/ebay:Item", nsManager);
foreach (XmlNode itemNode in itemNodes)
{
string itemID = itemNode.SelectSingleNode("ebay:ItemID", nsManager)?.InnerText;
string title = itemNode.SelectSingleNode("ebay:Title", nsManager)?.InnerText;
string sku = itemNode.SelectSingleNode("ebay:SKU", nsManager)?.InnerText;
string currentPrice = itemNode.SelectSingleNode("ebay:SellingStatus/ebay:CurrentPrice", nsManager)?.InnerText;
string buyItNowPrice = itemNode.SelectSingleNode("ebay:BuyItNowPrice", nsManager)?.InnerText;
string startTime = itemNode.SelectSingleNode("ebay:ListingDetails/ebay:StartTime", nsManager)?.InnerText;
string viewItemURL = itemNode.SelectSingleNode("ebay:ListingDetails/ebay:ViewItemURL", nsManager)?.InnerText;
string listingDuration = itemNode.SelectSingleNode("ebay:ListingDuration", nsManager)?.InnerText;
string quantity = itemNode.SelectSingleNode("ebay:Quantity", nsManager)?.InnerText;
string shippingServiceCost = itemNode.SelectSingleNode("ebay:ShippingDetails/ebay:ShippingServiceOptions/ebay:ShippingServiceCost", nsManager)?.InnerText;
string quantityAvailable = itemNode.SelectSingleNode("ebay:QuantityAvailable", nsManager)?.InnerText;
string shippingProfileName = itemNode.SelectSingleNode("ebay:SellerProfiles/ebay:SellerShippingProfile/ebay:ShippingProfileName", nsManager)?.InnerText;
string returnProfileName = itemNode.SelectSingleNode("ebay:SellerProfiles/ebay:SellerReturnProfile/ebay:ReturnProfileName", nsManager)?.InnerText;
string paymentProfileName = itemNode.SelectSingleNode("ebay:SellerProfiles/ebay:SellerPaymentProfile/ebay:PaymentProfileName", nsManager)?.InnerText;
// Extracting summary details
int activeAuctionCount = int.Parse(xmlDoc.SelectSingleNode("//ebay:Summary/ebay:ActiveAuctionCount", nsManager)?.InnerText ?? "0");
int auctionSellingCount = int.Parse(xmlDoc.SelectSingleNode("//ebay:Summary/ebay:AuctionSellingCount", nsManager)?.InnerText ?? "0");
decimal totalAuctionSellingValue = decimal.Parse(xmlDoc.SelectSingleNode("//ebay:Summary/ebay:TotalAuctionSellingValue", nsManager)?.InnerText ?? "0");
int totalSoldCount = int.Parse(xmlDoc.SelectSingleNode("//ebay:Summary/ebay:TotalSoldCount", nsManager)?.InnerText ?? "0");
decimal totalSoldValue = decimal.Parse(xmlDoc.SelectSingleNode("//ebay:Summary/ebay:TotalSoldValue", nsManager)?.InnerText ?? "0");
int soldDurationInDays = int.Parse(xmlDoc.SelectSingleNode("//ebay:Summary/ebay:SoldDurationInDays", nsManager)?.InnerText ?? "0");
dataTable.Rows.Add(itemID, title, sku, currentPrice, buyItNowPrice, startTime, viewItemURL, listingDuration, quantity, shippingServiceCost, quantityAvailable, shippingProfileName, returnProfileName, paymentProfileName, activeAuctionCount, auctionSellingCount, totalAuctionSellingValue, totalSoldCount, totalSoldValue, soldDurationInDays);
}
pageNumber = Enumerable.Range(2, numberofpages) ((A list on page numbers that the API has))
But when I do the for each, log message activity with the needed answer doesnât have anything.
EDIT: I can assure you guys, that the XML response string is not empty. I print it out after every http call i make, but it seems like the Invoke code code doesnt take out any information. When i use debug mode, i can see, that at first, HTTP calls are being made, then after all of them are completed, the invoke code and log message gets its turn. What can be the problem?
1 Like
Hi! How are you setting the value to DT_Response? Is it correctly configured as output argument in the Invoke Code activity? Can you show a screenshot of the âEdit Argumentsâ popup?
1 Like
Of course @Irene, here you are:
Thanks! Are you getting any value set to HasMoreItemsValue and numberofpages, or those output arguments are empty too?
1 Like
Additional question: can you show a screenshot of the Variables panel, so that we can see the scope of each variable? Thanks
1 Like
Of course my friend.
When i print out numberofpages variable, it gets me the answer - 406 (this is the number of pages that the API has with my parameters.
The variables panel:
I know its messed up
Because you see, before going into parallel sequence, i make the exact same api call, but with page set to 1, to get the number of pages that the API has, so i could loop it. And the API call with page 1 uses the same Invoke code activity to extract the data from the first page. Thanks a lot for your help!
How the whole process looks like now:
1 st:
2nd:
EDIT: For each is used to loop through different Ebay seller accounts. I get datatable filled with tokens and site names and etc from Mysql querry
Ok, thanks a lot for the info and screenshots!
Can you make a test by reducing the scope of the DT_Response variable in the Variables panel, so that itâs only visible within the For Each loop? Iâm not sure whether this is the cause of your problem, but itâs worth a try imho! If youâre using the same DataTable outside of your loop (for the 1st call to the API), then can you try creating a different DataTable in the For Each loop, with a reduced scope? (Only visible within the loop)
1 Like
One more question (sorry, they never come at the same time 
): can you confirm that you donât have the âContinue on Errorâ property of the âInvoke Codeâ activity set to True?
Thanks again
1 Like
Ok, will do, but give me like 10-15 minutes, the invoke code activity is still really confusing for me
And for this one no problems, im really greatful that your trying to help me, so dont worry bout anything.
And yes, the invoke code does not have âContinue on errorâ set to True.
1 Like
Ok, so after i ran some tests and changed the variables scopes and created different Datatable variables for the first HTTP request and the parallel one, i found something strange. When i tried to run the first HTTP request, that is not in a parallel, It printed out the number of pages, but the variable, that should hold the XML data stored as a datatable is empty as well. I presume, that the problem is in the code part, where it tries to take out the data and store it in a datatable variable. But what can be wrong? I used this code in a different file, it worked great.
Just for the sake of testing, can you try replacing the parallel for each with a regular for each? If it still doesnât work, then indeed the issue is somehow in the code (either with the scope of the variables, or the code itself). If it works with the regular for each, then we can investigate further the parallel for each
1 Like
I am almost certain, that the problem is either with the variables inside of Invoke code, or the code itself. Because you see, when I do it outside of any For each loop, it still doesnt work. It extracts the numberofpages from the XML, but the item nodes cant be found with the code that i used before. This is an example of the XML file, that im getting:
<?xml version='1.0' encoding='UTF-8'?><GetMyeBaySellingResponse xmlns="urn:ebay:apis:eBLBaseComponents"><Timestamp>2024-03-27T09:30:51.081Z</Timestamp><Ack>Success</Ack><Version>1271</Version><ActiveList><ItemArray><Item><BuyItNowPrice currencyID="EUR">115.87</BuyItNowPrice><ItemID>354301097369</ItemID><ListingDetails><StartTime>2022-09-27T10:31:23.000Z</StartTime><ViewItemURL>https://www.ebay.de/itm/F-r-MITSUBISHI-Outlander-07-09-Vorne-Sto-f-nger-Mit-Nebelscheinwerfer-L-chern-/354301097369</ViewItemURL><ViewItemURLForNaturalSearch>http://cgi.ebay.de/F-r-MITSUBISHI-Outlander-07-09-Vorne-Sto-f-nger-Mit-Nebelscheinwerfer-L-chern?item=354301097369&category=33640&cmd=ViewItem</ViewItemURLForNaturalSearch></ListingDetails><ListingDuration>GTC</ListingDuration><ListingType>FixedPriceItem</ListingType><Quantity>2</Quantity><SellingStatus><CurrentPrice currencyID="EUR">115.87</CurrentPrice></SellingStatus><ShippingDetails><ShippingServiceOptions><ShippingServiceCost currencyID="EUR">39.0</ShippingServiceCost></ShippingServiceOptions><ShippingType>Flat</ShippingType></ShippingDetails><TimeLeft>PT2H32S</TimeLeft><Title>FĂźr MITSUBISHI Outlander 07-09 Vorne StoĂfänger Mit Nebelscheinwerfer LĂśchern</Title><QuantityAvailable>2</QuantityAvailable><SKU>PMB04137BA</SKU><ClassifiedAdPayPerLeadFee currencyID="EUR">0.0</ClassifiedAdPayPerLeadFee><SellerProfiles><SellerShippingProfile><ShippingProfileID>222669608017</ShippingProfileID><ShippingProfileName>DIDELES SIUNTOS 39Eur</ShippingProfileName></SellerShippingProfile><SellerReturnProfile><ReturnProfileID>220817326017</ReturnProfileID><ReturnProfileName>Standard return</ReturnProfileName></SellerReturnProfile><SellerPaymentProfile><PaymentProfileID>220171935017</PaymentProfileID><PaymentProfileName>SIGNEDA PAYMENT</PaymentProfileName></SellerPaymentProfile></SellerProfiles></Item></ItemArray><PaginationResult><TotalNumberOfPages>20242</TotalNumberOfPages><TotalNumberOfEntries>20242</TotalNumberOfEntries></PaginationResult></ActiveList><Summary><ActiveAuctionCount>0</ActiveAuctionCount><AuctionSellingCount>0</AuctionSellingCount><TotalAuctionSellingValue currencyID="EUR">0.0</TotalAuctionSellingValue><TotalSoldCount>254</TotalSoldCount><TotalSoldValue currencyID="EUR">27174.78</TotalSoldValue><SoldDurationInDays>31</SoldDurationInDays><ClassifiedAdCount>0</ClassifiedAdCount><TotalListingsWithLeads>0</TotalListingsWithLeads><QuantityLimitRemaining>27897</QuantityLimitRemaining><AmountLimitRemaining currencyID="EUR">1655614.06</AmountLimitRemaining></Summary></GetMyeBaySellingResponse>
Oh my god, im so sorry. Its my dumb mistake. The Invoke code works perfectly in the first bit The thing is, that when im using log message and trying to print out a Datatable variable, it shows that its empty, because its a datatable variable. When i used it in Watch panel in debug, it showed that it had data and all is fine
1 Like
But one more, and the last question for you @Irene,
Could you please be so kind, and explain to me, how can I merge the datatables that im getting from Invoke Code inside the parallel for each activity into one and then bulk insert the combined datatable in one go?
Anil_G
jast1631
yuichi