Location Import / Export

Copying To A New Site

If you are moving something like a staging/test site to a new site and want to bring everything over intact I strongly recommend looking into the WordPress backup tool VaultPress.   One of the features included in VaultPress, besides near-real-time incremental backups of your site with nearly infinite restore points, is the ability to restore the site to an alternate site.

The beauty of this tool is that it will correct all domain name references.  If you are on site test.xyz.com and restore to live.abc.com the tool will change all the site references during the restore.   I use this tool exclusively when moving between live and staging sites.    You do need to make sure that your test/staging site copy has the latest data from the live site, such as new ecommerce system orders, blog posts, forum posts, etc. as it is an “all or none” option.   If you are an uber-geek you can always download the modified-domain zip file, unzip it, edit the data stream to take out things you do not want to obliterate on the restore site (like an ecommerce orders table), re-zip, and restore.

If you don’t need/want to move an entire site, the export locations can help.   Login to your source site, make sure you have Pro Pack installed, and go to Locations / Export.    It will export and download a CSV file.   If you have issues with the export see the troubleshooting guide below.

There is a quirk in some data sets that will require you to re-arrange the header row (first row) of the export file.  Before importing your export file make sure you’ve checked that the headers have not shifted.  I am actively seeking this bug but have not been able to reproduce it on the test and development servers.

Exports are Incomplete

Multiple large-scale exports have been tested.  On a relatively small server, a virtual machine with 2GB RAM running CentOS 6, I have been able to export 10,000 locations within 15 minutes with no issues.   Special characters and extended data have exported with no issues.

If you are running into issues there are several things to check that can cause problems during an export:

Web Server Process Limits

Apache, and nginx as far as I am aware, have per-process limits.  This is built into the web server to prevent “runaway” processes that eat up all of the CPU and memory on the server over time.   On shared servers this limit is very low and typically allows 30 seconds to 2 minutes of processing time.  Large data sets can use up to 5 minutes or more of processing time.   Make sure your server has a high enough limit to allow the entire data export to complete execution.

The types of limits that typically impact and export:

  • Process time (CPU cycles used) limitations.
  • Connection time (how long a single user is connected) limitations.
  • Memory limitations.

This setting is typically in the httpd.conf file, however there are also system configuration limits for the operating system (rare) and php process limits (more common) that can impact this setting.

[box]This setting is a common issue.[/box]

PHP Memory Limit

Check the PHP memory limit in your php.ini file.

This determines how much RAM php sessions can consume.  While the export is writing to disk there is ongoing overhead to track the process. This should increase marginally as the export data set grows, however there are number of known memory leaks in PHP that can have a larger impact on some sites depending on which versions of PHP you are using and which extensions you have enabled for PHP.  Some PHP installations have no memory leaks and will not exceed the memory limit once the base process has started.  Others can exceed the limit after just a few thousand locations.

[box]This problem is common.[/box]

WordPress Memory Limit

WordPress has a wp-config.php file and various defines set within the main code that set limits on how much memory can be consumed by a single WordPress session.  This will override the PHP memory limits and should be checked.  Many larger applications, especially ecommerce applications, require you to edit this setting to change the default value from 32MB to a larger value.    Store Locator Plus consumes little additional memory per exported location, however the aforementioned memory leaks in some environments can be an problem for this setting.

[box]This problem is common.[/box]

Operating System Limits

There are various operating system limits that can cause issues including general process limitations,  directory security issues, and memory limitation issues.  All system-level applications that run on a server are beholden to the over-arching system limitations set by the server administrator.   For WordPress sites Apache (or whatever flavor web server app you use) is the primary communicator with the operating system and is most likely to be flagged for “over resource limitations”.   PHP can be flagged depending on how you have it configured to work with Apache.    Finding server-level limits means search system-level log files.

[box]This problem is rare.[/box]

Network Limits

Some firewalls, especially on shared hosting environments, can throttle and time out high bandwidth data streams.  This is likely to become more common in the United States with the demise of Net Neutrality.   Most hosting companies have stopped this practice.

[box]This problem is rare.[/box]

Browser Limits

Some browsers can try to cache the download file as it streams from the server instead of writing directly to disk.  This can cause the browser to run out of memory.

[box]This problem is rare.[/box]

Related Notes

Keep in mind that data is exported in “first in” order.   It is not always alphabetical or based on the location ID.   It is based on which records were saved to the database first.    If you are missing locations on  your export it is likely because the export did not complete before being stopped by the server.

There are several factors that work in tandem when dealing with long-running web processes.    All of these elements can cause limitations on exports.  The primary factors include:

  • Server Operating System configuration and settings.
  • PHP environment settings.
  • Web Server Application (Apache for example) configuration and settings.
  • Web Application, in this case WordPress, configuration settings.
  • Network environment settings (hosting company firewalls, bandwidth throttles, etc.)
  • User’s browser environment.

An environment that is properly tuned to deal with large data sets will have not issues with import, exporting, or managing large lists of locations in Store Locator Plus.  The core of the product is coded to work well with large data sets with consideration for things like per-event execution time, per-event memory limitations, and reduced disk I/O operations.

The “Debugging WordPress” section on the main Troubleshooting page can help.   Turning on WordPress debug logs can quickly identify and resolve export issues.   The web server log files (/var/log/httpd/*log on Apache RHEL servers) is also a good place to find limitation hints.