50ish ft/lbs is 'lethal' ie will penetrate 12" of ballistics gel, so long as the bullet is of sufficient mass and doesn't tumble. 

The 50gr 00 buckshot (modern 'OO' is actually more like a real 0 due to the use of wads) is penetrating 12" of ballistic gel at ~700fps impact velocity. 50gr @ ~700fps being similar to a .25 ACP from a pocket pistol.

50gr x 700fps = 54 ft/lbs

That said, 'lethal' is not necessarily incapacitating. For example, a guy about to fire a gun gets tagged in the chest with a .25 ACP. He'll no doubt die or need serious medical attention - but will this occur before he's able to aim and fire his weapon?

For 'maximum effective range' I like to use Supersonic Velocity - 1125fps - as the cutoff.

Pretty much any FMJ bullet of 40gr or more at 1125fps will penetrate 12", and few shoulder fired weapons are accurate outside of their supersonic range. So its an easy, mostly universal metric, thats better then using a minimum energy figure. 

So in my head I am using a shortened 6.5 Grendel with 1.5cc case capacity at 440 MPa with a case length of 25mm and a 65gr bullet with a hardened steel core and a copper "sabot" and an ogive length of 15mm, for a COAL of 40mm. E0 is ca 1500J. 

Drop at 300m is 9,5", transsonic speed is reached at around 600 yds. 
 

Given that the cartridge is a lot fatter than 5.7x28, I assume that the mag capacity would be somewhere around 30, but it would be thicker. 

I would suggest an integrated moderator, swallowing a lot of bang and some of the recoil. Add a folding stock and call it a PDW ;)

This could be - with a bit of difficulty - carried in a thigh holster. Now I get that this is probably not for the US, since the M4 probably fills this role as long as there are billions of rounds of 5.56 available.

It would struggle to reach 1500j in a PDW barrel.  The 300blackout has that energy in short barrels,  but this is because theres more swept volume due to wider and far heavier bullets. 

Slimmer caliber equals low short barrel energies. 

If you did reach those energies you'd want a fairly hefty barrel for any controllability.  The Sig rattler is a tiny 300black gun and it has a  real heavy barrel. 

You earlier mentioned ogive length.  A 6. 5 round in a 40mm OAL will be very short-for-caliber in terms of bullet length.  Whatever a ballistic calculator might be telling you,  I can assure you that 300m would see a massive drop in retained energy. 

Why not use an MP5 shape with reverse feeding or a mini bullpup. For a small reduction in clip size you could have a longer round.  In a far smaller gun

He's using a sabot.

You could also use the 6.5 CBJ, for that matter.

Point is, PDW is somewhat loosely defined - the way I see it, PDW is something a non-rifleman (REMF) uses to fight his way to a rifle, so it has to be somewhat effective up to some 20 meters at most.

One must take into consideration the fact that REMF aren't trained well enough to consistently hit anything far away, so a handgun is as good as anything else.

M1 Carbine was a brilliant exception - for a multitude of reasons, including historical moment, it was actually much closer to an assault rifle than to a PDW - but don't expect other weapons to repeat that feat.

You mentioned the Rattler, and you're correct in one thing - a weapon that won't offer significantly better external ballistics from a significantly smaller package is doomed to fail just like the P90 and MP7 have (in the PDW role, that is).

renatohm said:

He's using a sabot.

I used that term somewhat ironically. I mean a bullet that stays together when it hits a soft target, but discards the hardened steel core when hitting armour. That is why I put "sabot" in quotation marks. And my cartridge has significantly better external ballistics. IMO it is a decent successor to the M1 carbine. Which was used by REMFs and at reasonable ranges. Precisely because handguns were insufficient as proven in WWI. So my gun can reach as far as an M1 Carbine or more, has a bit more energy than .30 Carbine, can perforate at least up to IIIA, if not III armour and can be carried as a somewhat unwieldy handgun, while having a folding stock of probably something like 14" length, which, unfolded, should turn it into a reasonable carbine.

Like I said the practical hit probability at 300m,  for typical PDW-type personell,  is zero. You could change that by squeezing in a longer 12 inch barrel with reverse feeding. 

If you Downsized the caliber,  the saved weight could be spent on an integrated optic with a long 4x zoom.  Then it would be a 300m gun.  

How about considering 556 in a mini bullpup.  The logistical savings would make it more appealing to militaries 

PDW is meant for use well under 100m . No point in trying to make it effective to 300m although you can hit man-sized targets with relative ease with both FNs 5.7mm and Hks 4.6mm with any red dot equipped firearm. The optics extend the range far more than barrel length/ gains in velocity.

renatohm said:

...the way I see it, PDW is something a non-rifleman (REMF) uses to fight his way to a rifle, so it has to be somewhat effective up to some 20 meters at most.

I think you have an erroneous understanding of the PDW's real purpose, as well as its necessary range of employment.

PDWs are intended primarily for combat personnel who are not riflemen, such as helicopter pilots and tank crewmen.

Also, the PDW is not meant to enable the user "to fight his way to a rifle."  The PDW is meant to be used in lieu of a rifle.

Armies seem to agree more with 'my' definition - it explains better why dedicated PDW aren't nearly as widespread as pistols and submachine guns for the role.