Why do marginal cost and marginal product move opposite to each other?

Marginal cost and marginal product are essentially looking at the same thing from two different directions.

Marginal cost is asking how much money you need to spend to produce a given output. The money you need is the price of inputs times the quantity of inputs you'll need to buy. So it's basically how much input you need per output.

Marginal product is asking how much output you'll get from a given amount of input. It's how much output you get per input.

So for example let's suppose we're buying apples to make apple pie. Let's suppose each pie requires 8 apples and each apple costs $0.50.

Marginal cost is the price of an apple times the number of apples needed to make a pie: So that's $4.00.

Marginal product is the number of pies we can make per apple, which is only a fraction: 1/8 or 0.125.

Holding the price of apples constant, our marginal cost and marginal product would be in inverse proportion. Suppose we find a way to make pies with only 4 apples. Our marginal product would double to 0.25, and our marginal cost would fall by half to $2.00.

Put another way, marginal cost times marginal product simply is the price of the inputs; notice that (0.125)($4.00) = (0.25)($2.00) = $0.50