If they refuse to give visa, will they refund my visa fee?

No, if you are refused of the required visa your visa fee and the diversity processing fee will not be returned back. The American Government’s green card lottery requires some eligibility for the applicants to meet.
Country eligibility.
Education/work experience.
Medical Clearance.
Police clearance.
Affidavit of Support.

Before you start to process the lottery after the winning the notification, you will be informed by the authorities not process the application further, if you don't have the required eligibility as given above. There are many other reasons that you may get refusal to issue with a visa.

Don't have the required education or work experience.

Didn't enter the spouse and children details in the online entry and entering them in the later application.
Submitting fake documents.
Failing medical Test.
Failing the police clearance
And many other things, which the immigration officer can find out at the time of Interview.

So if the consular officer refuses you to issue the visa, your visa fees will not be returned back.