Folk dance etiquette is simple and based on courtesy and consideration for your fellow dancers. First and very important, NEVER make disparaging remarks about other dancers while at the club or class. So far all the folkdancers I've seen are human, and we all make mistakes. We bring an enormous range of ability and fitness levels to folk dancing, so be tolerant of others.
One tried & true way to learn a new dance, if it isn't actively being taught, is to dance behind the line until you are comfortable you can join in. You can also ask other dancers if a dance is easy, or if it holds any big surprises - some dances start off pretty easy but have tricky parts. Pick a good dancer to follow and you can learn a lot! Traditionally, recreational dancers learn a majority of the dances by following behind the line, then joining in as they become familiar with the steps and styling of the dance.
Join a dance near the middle or end of the line, if it is a line dance. To know which is the "end", you'll need to know which direction the dance travels. Some dances travel both to the right and the left...join these dances in the middle. Join a closed circle dance anywhere in the circle, but be sensitive to the fact that certain people may prefer to dance together. Also, if the dance is done in a shoulder, belt, or basket hold, you'll be more comfortable dancing next to someone near your own height.
Basket hold note: Don't cross your arms! Reach out to each side, skip the person next to you, and hold the hand of the person once removed from you on either side.
If you find yourself in a dance that's just too hard, smile and excuse yourself and DROP OUT rather than fumbling through the rest of the dance. Continue the dance behind the line, where you can concentrate on learning without worrying about your neighbors.