It appears the only thing it needs is admin-ajax.php after all, the rest of the folder can still be locked out for the security focused. The directives to isolate just the one file took me a minute so here it is if anyone else wants it.
#allows anyone to use admin-ajax.php but only certain IPs can access the rest of the wp-admin folder
<Directory [fullpath]/wp-admin>
Order allow,deny
allow from 192.168.0.1
allow from 192.168.0.2
allow from 192.168.0.3
<Files admin-ajax.php>
Order deny,allow
</Files>
</Directory>
