Data structure to keep things in memory

Without going into all the details, I have the following situation:

There are a set of objects RTS, all of the same type, that gather run-time
statistics about the running of a program. Various related objects with
different lifecycles pass them around and use them. I'm interested in when
each member of RTS is no longer being updated, because I want to collect its
final statistics , but there's no good way to tell when it's no longer in
use except via its collection, so when it's created I attach a weak
reference to it and associate the weak reference with a queue. When the
reference is delivered, I record the statistics. [1]

So far so good. The question is, what's the best way to keep the references
in memory until they can be delivered to the queue? Obviously, I could put
them into a synchronized HashSet, but I'd prefer to optimize for concurrency
by minimizing locking. At the moment, I'm using a ConcurrentHashMap, since
it seems to give the best concurrency for updates. (The usage pattern is a
bit unusual, since there are effectively no lookups. There's a put() when
the reference is created, and a remove() when it's pulled out of the
reference queue.) This is an area where I have very little experience, so
I'd welcome input from anyone who's worked with these classes.

1. Obviously, a member of RTS points to a separate statistics object which
is also pointed to by the weak reference.