What really are units? Why can we ignore them, like in class?

[haruspex]

these things I learned early on by applying them on numbers, also work with dimensional quantities

Yes, but they don't work always. You can multiply a speed by a time and get a distance, but is it the distance you wanted?
When we write a physical law, the claim is that if you match up the entities in the equation to the right entities in the real world then the equation, interpreted as arithmetic, will produce (near enough) the right answer. So your question becomes why do certain real world entities have relationships that correspond to mathematical operations.
This is referred to as 'the unreasonable effectiveness of mathematics'. But I feel it's not such a mystery.
First, we never can know how accurately the real world corresponds to the maths. Newtonian mechanics looked pretty good for a while.
Secondly, we defined numbers and standard operations on them as we have because they have such use. Group Theory covers all sorts of real world things that don't behave much like numbers.
Thirdly, the Anthropic Principle: maths works at all in the real world because the real world has a degree of predictability. In a universe without that, intelligent beings would not evolve, since they'd have no advantage.

 

[Drakkith]

Do you think it could be a good thing to stop worrying about it and just proceed with math in a mechanical way (even though I'd hate to do such a thing)?

Yes. While trying to fully grasp the underlying concepts of something can be good, if it's stopping you from learning the math you need to succeed in the real world then I'd put your question on the back burner for now. You can always come back to it later.

 

[Svein]

Here "dimensional quantity" implies that "oranges" is some kind of mathematical object that is also manipulated around in the equation.

No, Physical objects. Mathematics does not care. It just solves the problem you give it,whether it makes any sense or not.