Fast software performance is often the focus when developing real-time vision-based control applications for robot simulators. in this paper we have developed a thin, high performance middleware for USARSim and other simulators designed for real-time vision-based control aplications. it includedes a fast image server providing images in openCV, matlab or web formats and a simple command/sensor processor. the interface has been tested in USARSim with an unmanned aerial vehicle using two control applications; landing using a reinforcement learning algorithm and altitude control using elementary motion detection. the middleware has been found to be fast enough to control the flying robot as well as very easy to set up and use.