Could you try to deactivate all modules, one at a time, to find out if one of them creates some performance issues on your site?
You can follow the instructions here to disable a Jetpack module:
http://jetpack.me/support/#turn-off-jetpack-modules
↧
Jeremy Herve on "[Plugin: Jetpack by WordPress.com] Slow Loading"
↧