Naval battles work in exactly the same way as land battles: the battlefield is a seascape, individual ships replace units, and combat is controlled using essentially the same user interface as when fighting on land. Although during the period covered by Total War: ROME II fighting at sea consisted of firing missiles and then getting in close to board and melee, ramming was also a viable battle tactic at this time.
During a boarding action the crew of one ship will attempt to take over another with a combination of melee and support from missile troops. With your own ship selected, click on the boarding button and then pick a target vessel to order your ship to close with it and send an armed group aboard to capture it. Boarding actions call for a high degree of skill and courage in the troops taking part, and can be cancelled at any time by issuing a contradictory order.