You need to make your boat name large enough to be read. In fact, if done well, the name can be almost as big as you want. There are a lot of boats whose names cover the entire transom or half the length of the boat. Keep in mind, these could be very expensive and very difficult to apply. If your vessel is NOT a US Coast Guard registered vessel, there's no limit to how small you can make the name. If it IS a US Coast Guard registered vessel, the name must be 4" or higher. You must also include the hailing port if your vessel is USCG registered and the 4" rule also applies to that. Also for USCG registered vessels, the name can't exceed 33 characters. So, a boat name like 'Monty Python's Incredible Flying Circus' at 34 letters won't cut it for a registered vessel.
For up-to-date info on USCGÂ Documentation Requirments, check out their FAQ.
For small registered vessels like ours, 4" hailing ports can be a bit distracting. One way around it is to use a narrow font.

The font styles (letter styles) available are practically endless. But, that doesn't mean you can or should use any letter style that floats your boat. Below are some good and bad examples:

What makes a good font good is its readability. The more complex it gets, the harder it is to read. In addition, that complexity gets lost once you get 10' away from the vessel. And, unless you're racing, no other vessel should be within 10' of yours!
Not only is the readability a factor, but the more complex the font, the more of a PITA it is to cut, weed, mask, and apply. While we can pretty much cut any name you want, I will impose a strong warning and very high cost if you pick something like the font you see in the 'terrible' column.
Special Effects
The most common type of special effect are the drop shadow and the outline. I'll say right up front, these are a complete waste of money and time!
Drop Shadow
Drop shadows are added to give a bit of depth to the graphic. This works fine for a storefront or graphics on a car, but on a boat where the only close up will be at the dock, it's not worth it (unless, of course, your boat never leaves the dock). The big thing about drop shadows is that it requires TWICE the amount of vinyl.

The other common effect is the outline. This is done, again, to emphasize the text and offset it from the background. If the desired color for your name is a light color, a dark outline can help set it off. Like the drop shadow, outlines require twice the vinyl. My first recommendation regarding outlines is to select a color that highly contrasts your hull color and forget using outlines.They are invisible (unless very thick) from about 10' away.

You can also have names run along a curve or some other shaped path. In addition, you can have text slanted left or right. These are less of an issue than drop shadows and outlines. However, you need to consider the overall size. If you start out with 10" letters (using a 15" wide roll of vinyl) and curve them to the point where the overall size is now over 15", you've just stepped up to the next size in vinyl widths (24"). This will increase the cost. Also, slanted text will use more length than non-slanted.