Explanation

We use by when we talk about “means of transport”.

We travel by bus.

“On” is used with a specific bicycle, bus train, ship plane.

Come on my bicycle.