Having ridden a Honda Dominator through plenty of mud, sand and loose gravel, with 2 heavily loaded ammo boxes as home-made panniers, I suggest that comparing the ride characteristics with these boxes with carrying a pillion is unrealistic.

A pillion's mass is vertical through the frame, generally forward of the rear axle, and self compensates (hopefully) to riding conditions.

The biggest problem with rear mounted heavy panniers is sideways rear end momentum, especially at high speeds on loose surfaces. You can get some crazy swinging action going on, as you hope you can sufficiently control your foot onto the rear brake, before you end up eating dust. Conversely they make for speccy power slides.

For durability alone I recommend ammo box panniers, especially if you are doing really rough stuff, and expect to drop your bike.

For strength and rideability I suggest welding the boxes onto a frame that bolts semi-permanently to the bike. The most important load carrying attachment point to the bike should be through the frame holes provided by the pillion's footpegs, thus achieving weight distribution as low as possible. Everyone understands the extra stability gained by standing up when riding, despite the increase in height of centre of gravity. Do the same for your panniers. Use mounting points around the seat for stability and strength only, not weight support. Make a brace that joins both boxes over the back of the bike for extra strength. Mount your boxes as far forward over the rear axle as possible. Making a semipermanent set-up will allow you to mount the boxes much closer to the bike than having removable boxes. It is easy to secure ammo boxes.

The criteria above increases in importance if your bike has a shorter wheelbase and is lighter.

If you or a friend are handy with a welder an ammo box setup much more durable than aftermarket panniers can be had for a fraction of the cost. I also recommend putting in a decent load adjustable rear shock if you use ammo boxes.