Jeremy Herve on "[Plugin: Jetpack by WordPress.com] Tiled mosaic; border between pictures adjustable?"

You could change the margin around the images with some custom CSS, but since the image sizes are defined via javascript, you will experience issues when trying to make these changes.

If you want to customize the look of the Tiled Galleries, it would probably be better to completely remove Jetpack's Tiled Gallery libraries, like so:

function tweakjp_rm_tiledlibs(){
	wp_dequeue_script( 'tiled-gallery' );
	wp_dequeue_style( 'tiled-gallery' );
}
add_action( 'wp_footer', 'tweakjp_rm_tiledlibs' );

You can then load your own custom css and js files instead, based on Jetpack's ones (css / js) but with your changes.