In the military if a clearance is obtained and you wish to leave or retire, you only have 2 years to find a job that requires a clearance (if you already have one, you have preference over those without) or you lose it.
I presume it costs so much because it's a 10+ page application wanting to know everything about you (friends, school, work, marriages, divorces, family members, addresses lived, etc.). It's one of the most in-depth background check ever as they want to figure out cultures, societies, people, etc. that you have been/are associated with. They follow up on every aspect of information on the application.
I once heard a story where the investigators called a friend of an applicant and the friend stated that they smoked weed years ago. Guess what? Ultimate buddy f*** as not obtaining the clearance also means not getting the desired $200,000/yr job.
I was unaware of the 2 year carry over rule and applied for a $140k job... to find out I no longer had clearance (and the contractor wouldn't sponsor it... because it is a $100,000 gamble that in those years you didn't even get a parking ticket in Bumfaq nowhere... yes, it's that in-depth).