If you are experiencing lag with 50+ markers dropped on the map at a time it can be your server response times when processing AJAX (REST JSONP actually) requests, your local browser and local Internet speed, and Google server response times. Most likely is your hosting server response times; Google response times is the least likely culprit. The reason I recommend only showing 50 markers or less is more due to the user experience issues than performance issues. When performance comes into play it is the Google JS and/or user browser performance that is more concerning that WP database or SLP plugin performance. A user running old IE versions, for example, will show notable performance degradation due to the extra Google API JS code that executes for each marker to support implementation older than IE9.
There are sites I have worked with that show 100+ markers at a time and they load in < 3 seconds. Some sites have 400+ markers (I do not advise this, just from an experience perspective) and still load in 5-7 seconds. However both those customers are running higher-end dedicated servers. If you are hosting on a shared or virtualized server plan you are likely to have more performance problems.
To determine if it is the initial request being processed on your server or a local PC/Internet connectivity issue you can look into the network analysis tools in the browser. It will show you the total time in ms for a request to be posted to a server, processed, and returned. I prefer to view the Net tab and review the data results and watch for timing issues there. A well-configured server processing MySQL for WordPress should be able to return hundreds of locations in a few seconds.
Here is 114 locations , all markers displayed, using a laptop as a WordPress 4.3 server running SLP. You can see the WordPress + SLP + Google Map throughput is sufficient and loads all 114 markers in < 600ms. 10 tests, all under 600ms.
[caption id="" align="alignnone" width="920"] 114 Markers Rendered < 600ms[/caption]