I did my first track day about 1 month ago. I had some of the same concerns as you, but after the first session, I realized that I could control the risk by just riding at my own pace, being aware of those around me and not getting overly excited about the fun I was having!
I started in the beginners group and found the others in that group to be very cautious. Also, marshalls were riding with us and constantly on the watch for people riding way off lines, being to eratic, or doing anything that might endanger themselves or others.
I think my skill set is more inline with the intermediate group but it was good to start slow.
My first four sessions went great (I felt) and then sessions 5 and 6, I felt to "stiff" and not relaxed. It was time to call it a day at that point.
There is always a chance that someone could take you out from behind or in a corner but I felt so much more safe on the track with others than on the street. I was supprised how much safer I felt.
The lack of insurance sucks, but the speeds aren't that great, (I did hit 169mph on the straights), and when in a corner, you're leaned over and not that far off the ground. You should be limited to lowsides and not much damage. There is always that chance of a tumble and a total though.
Take it easy, ride your own pace and be aware of how a faster rider would navigate the track and if you feel slow in certain corners or straights, leave room for the faster riders to choose their ideal lines.
It is definetly a great experience!