Ecommerce Shopify WordPress Discussion

Problem with large PHP cURL request (JSON)

We are programming a WooCommerce webshop for a customer. The goal is to sync the products from the client's new ERP system. The ERP system also is in development right now and provides a JSON interface for testing. The product import JSON contains more than 5.000 products with multiple variations right now. I tried to do the request with postman for testing. The request took a relatively long time (approx. 8 minutes), but was successful. The response JSON file has a size of approx. 2 MB. Now I tried to do the request with PHP but it doesn't work. After 1 hour (max_execution_time = 3600) In the server protocol I get the following error message: "18454#0: *1484295 upstream timed out (110: Connection timed out) while reading response header from upstream" I have set the following PHP settings: memory_limit = 256M post_max_size = 256M upload_max_filesize = 256M max_execution_time = 3600 max_input_time = 3600 The PHP code for the cURL request looks like this: $curl = curl_init(); curl_setopt_array($curl, array( CURLOPT_URL => 'https://api.anqodo.com:8088/anqDataSvc/api/TofererExternal/webshopproducts?ws_id=' . $webshop_id . '&date_from=' . $date_from, // the date_from parameter can be set to limit the products to all edited products since this date. If I limit the date (e.g. 2024-01-01) the request works, because the response is smaller. But for the initial import I need all products CURLOPT_RETURNTRANSFER => true, CURLOPT_ENCODING => '', CURLOPT_MAXREDIRS => 10, CURLOPT_TIMEOUT => 3600, CURLOPT_FOLLOWLOCATION => true, CURLOPT_CUSTOMREQUEST => 'GET', CURLOPT_HTTPHEADER => array( 'RequestTimeClient: ' . date("Y-m-d\TH:i:s\Z"), 'Content-Type: application/json', 'Authorization: Bearer ' . $token // I retrieve the token in another function, which works fine ), )); $response = json_decode(curl_exec($curl)); curl_close($curl); return $response; I do not have much experience with such interfaces, which is why I need help. I don't really know what causes the problem. Is there a limit to the size of such a JSON response? Why does it work with Postman but not with PHP? What would be the best way to deal with this problem? I tried to do the request with Postman, which worked. After that I tried to to this request within my WordPress plugin, but also in a "normal" PHP file in the root path, to exclude possible issues with WordPress, which didn't work. As mentioned before, I also tried to set the "date_from" parameter, to limit the response to all products, that were edited within the last 2 weeks or so. This worked perfectly.
function fetchProducts($webshop_id, $date_from, $token) { $batchSize = 100; // Number of products to fetch in each batch $currentPage = 1; $allProducts = []; do { $start = ($currentPage - 1) * $batchSize; $url = 'https://api.anqodo.com:8088/anqDataSvc/api/TofererExternal/webshopproducts?ws_id=' . $webshop_id . '&date_from=' . $date_from . '&start=' . $start . '&limit=' . $batchSize; $response = fetchBatch($url, $token); if (!$response || empty($response->products)) { // Handle error or break the loop if no more products break; } // Merge products from the current batch into the result array $allProducts = array_merge($allProducts, $response->products); $currentPage++; } while (count($response->products) == $batchSize); // Continue fetching until fewer products are returned than the batch size return $allProducts; } function fetchBatch($url, $token) { $curl = curl_init(); curl_setopt_array($curl, array( CURLOPT_URL => $url, CURLOPT_RETURNTRANSFER => true, CURLOPT_ENCODING => '', CURLOPT_MAXREDIRS => 10, CURLOPT_TIMEOUT => 3600, CURLOPT_FOLLOWLOCATION => true, CURLOPT_CUSTOMREQUEST => 'GET', CURLOPT_HTTPHEADER => array( 'RequestTimeClient: ' . date("Y-m-d\TH:i:s\Z"), 'Content-Type: application/json', 'Authorization: Bearer ' . $token ), )); $response = json_decode(curl_exec($curl)); curl_close($curl); return $response; } // Example usage: $webshop_id = 123; $date_from = '2024-01-01'; $token = 'your_access_token'; $products = fetchProducts($webshop_id, $date_from, $token); Batch Processing: The fetchProducts function gets products in smaller groups (batches) by setting a starting point and a limit in the web address. The batch size is initially 100 but can be changed according to your requirements. Loop through Batches: A do-while loop continues to get batches until the number of products received is smaller than the set batch size. Merge Results: Products from each batch are combined into the $allProducts collection. Separate Function: The fetchBatch function handles single requests to the ERP system. Error Handling: Properly dealing with errors is crucial. If there's an issue or no more products are returned, the loop stops. By using batch processing, you can handle a large dataset more effectively and decrease the risk of running into execution time limits. Adjust the batch size based on your server's capabilities and the ERP system's response times.

January 12, 2024

TurboCommerce make the better internet purchasing globaly

Turbo Multi-language Translator

Make the better internet purchasing globaly

Turbosify SEO Speed Booster

5.0 (7) • Free plan available
Get better conversions by improving store loading speed Installed

Turbo Multi-language Chat - AI Customer service in one hand

TurboCommerce make the better internet purchasing globaly
Our products

The help you need, when you need it

App by Turbo Engine

3 apps • 5.0 average rating

Turbosify Speed Booster

5.0 (7)
Get better conversions by optimizing shopify store Google page speed Installed

Turbosify Translator for Wordpress Woocommerce

5.0 (74) • Free Wordpress Woocommerce Plugin
Translate your wordpress website to multiple language within 1 click, no configuration needed, no No technical required

Grow your business here

Whether you want to sell products down the street or around the world, we have all the tools you need.