Yes, this dirt is "attached to outside dirt".
This is the same ground that's under all of the foundation. There'd be no reason to bring in dirt to put on top of concrete that's been poured. Which is why this is so confusing - there's no reason I can think of for leaving this open and installing a wood floor rather than being finished with concrete like the rest. This just leaves an opportunity for water intrusion.
If you're down to clay, I'd just fill with gravel (and insulation, as mentioned by someone else) and pour concrete - this is the best approach in my opinion. Otherwise it will always be a moisture problem.
You could extend the clean out to be just above your finished floor level.
Alternatively, level the concrete to just barely below the clean out and install a false floor using wood. Then you've sealed the floor, left the clean out, and the false floor provides an even surface without the clean out being in the way (it would be under the false floor).