What works better (ceteres paribus), a lighter and hence faster bullet or a slower and heavier bullet with a higher sectional density?

Depends on the armour array as much as anything else. A paint fleck at a couple of km/s will smash relatively thick plate of steel but would be stopped by a spaced array, while a more massive but slower projectile would be stopped easily by monolithic plate but go through a spaced array like it wasn’t there.

at lower energies (pistol energies),  smaller calibers have better penetration.  The MP7 (4. 6mm) can defeat a vest at 250m,  despite tiny energies of 330ft lbs. 

By comparison the FN 5. 7 (similar energies)  has about 20ft lbs remaining,  after defeating a vest @ 100yards.  Much worse

As you move up to higher rifle energies and tougher armour,  I suspect the gap closes and a 6. 5mm might match or even outpenetrate a smaller caliber

The opposite effect is found with barrier penetration.  An AK in 762 will defeat walls and cover far better then smaller calibers 

I am currently playing with energies around 1300J and ranges around 300m. You saying that I'd be better of with 5mm than 6mm overall?

"What works better (ceteres paribus), a lighter and hence faster bullet or a slower and heavier bullet with a higher sectional density"

The problem is different armors have different mechanisms of penetration.

Kevlar soft armor is defeated in multiple ways: High velocity (impact velocity over 2000fps) narrow cross section / pointiness (.224 55gr spitzer FMJ will pierce IIIA at 1000fps) or high hardness (steel core 9mm will penetrate 2x IIIA vests stacked on top of one another.)  M855A1 is reportedly capable of piercing IIIA beyond 1000m, due to being both pointy and hardened steel. 

AR500 steel is destroyed by velocity; a 5.56 55gr M193 at 3180fps+ will burn through a 'Level III' AR500 steel plate via adibiatic sheer. This same plate will stop steel tip M855 at a slightly lower velocity. 

On the flipside, Level III UHMWPE will catch 3300fps+ lead core M193 and .308 at 2800fps, yet can be penetrated by a steel tip 5.7x28 at 2100fps. Here, projectile hardness and pointiness is what defeats the armor. 

And ceramic defeats both high velocity and steel tips; here its really a matter of Tungsten cores of sufficient length and speed to penetrate. 

Making this even worse, most 'Level III+/Level IV' hard armor is a mixture of Ceramic and UHMWPE. So one must design to punch through two different materials back to back. 

so I am fundamentally thinking a hardened steel core like the 855A1 in a 1300J PDW round (modernised version of 5.56x30 MARS).

But what targets are you intending to defeat?

PDW role - so near peer combatants.

So soft armour with rifle-proof plates, probably ceramic.

I doubt you’d make it through the plates with a PDW, so you’re left going for soft or no armour. I suspect that narrow, sharp and fast would be the order of the day.

Fast also compensates for aiming errors by inexperienced users.

The real trick is to transform the high velocity into terminal ballistics, either post armour or hitting unarmoured parts

With energies of roughly 1000ft lbs,  a 19.  Caliber would have by far the best energy retention at 300m.  A 6mm wouldn't come close.  

If penetration must be extreme,  you'd want a fairly 'full' shaped bullet,  closer to 556 shape than modern,  pointy,  VLD shapes.  This would allow an extremely large steel core,  which is far more economical than tungsten. 

Finally though,  slimmer calibers do struggle in short barrels,  energy wise.  Because the swept volume is less. 

A mars-style round would need at least 19 inches of barrel to reach 1300jouls,  if the caliber was. 19 (5mm)

Your PDW should therefore be a bullpup,  which uses reverse feeding (like the boberg pistol) to shortern the overall length.   Lastly,  you'd want the magazine right at the very rear of the bullpup.... far away from the grip (again this shortens length).... also,  don't use a flash hider.  That would add an inch of length. 

Ultimately you can get extreme 300m penetration with far less energy.  An. 18 caliber,  with about 750 ft lbs and a steel core,  would do the same job.  From a 15inch barrel.