Encuentre el conjunto S que maximiza la distancia mínima entre puntos en S unión A

For a given k with X and A as input brute forcing is obviously in P (as C(|X|,k) is polynomial in |X|).

If k is also an input then it might depends on 'distance' :

If 'distance' is arbitrary then your problem is equivalent to find a fixed size clique in a graph (which is NP-complete):

NP-Dureza :

Take an instance of the clique problem, that is a graph (G,E) and a integer k. Add to this graph a vertex 'a' connected to every other vertex, let (G',E') be your modified graph.

(G',E') k+1 is then an equivalent instance of your first instance of the clique problem.

Create a map Phi from G' to R^d (you can map G' on N anyway ...) and defined 'distance' such that distance(Phi(c),Phi(d')) = 1 if (c,d) ⊆ E', 0 otherwise.

X = Phi(G'), A = Phi({a}), k+1 give you an instance of your problem.

You can notice that by construction s ≠ Phi(a) <=> distance(s,Phi(A)) = 1 = max distance(.,.), i.e. min_{s⊆S, s'≠s ⊆ S∪A} distance(s,s') = min_{s⊆S, s'≠s ⊆ S} distance(s,s') if |S| = k >= 2

Solve this instance (X,A,k+1) of your problem : this give you a set S of cardinality k+1 such that min( distance(s,s') |s,s'⊆ S, s≠s') is maximal.

Check whether there are s,s'⊆ S, s≠s', distance(s,s') = 0 (this can be done in k^2 as |S| = k) :

• if it is the case then there is no set of cardinality k+1 such that forall s,s' distance(s,s') = 1 i.e. there is no subgraph of G' which is a k+1-clique
• if it's not the case then map back S into G', by definition of the 'distance' there is an edge between any vertices of Phi^-1 (S) : it's a k+1 clique

This solve with a polynomial reduction a problem equivalent to the clique problem (known as NP-hard).

NP-Easiness :

Let X, A, k be an instance of your problem.

For any subset S of X min_{s⊆S, s'≠s ⊆ S∪A} distance(s,s') can only take value in {distance(x,y), x,y ⊆ X} -which has a polynomial cardinal-.

To find a set that maximize this distance we will just test every possible distance for a correct set in a decreasing order.

To test a distance d we first reduce X to X' containing only points at distance >= d of A{the point itself}.

Then we create a graph (X',E) ¤ where (s,s') ⊆ E iff distance(s,s') >= d.

Test this graph for a k-clique (it's NP-easy), by construction there is one iff there it's vertices S are a set with min_{s⊆S, s'≠s ⊆ S∪A} distance(s,s') >= d

If 'distance' is euclidean or have any funny property it might be in P, I don't know but I won't hope too much if I were you.

¤ I assumed that X (and thus X') was finite here

This is probably NP-hard. Greedily choosing the point furthest away from previous choices is a 2-approximation. There might be a complicated approximation scheme for low d based on the scheme for Euclidean TSP by Arora and Mitchell. For d = 10, forget it.

The top 2 results for the search sphere packing np complete turned up references to a 1981 paper proving that optimal packing and covering in 2d is np complete. I do not have access to a research library, so I can't read the paper. But I expect that your problem can be rephrased as that one, in which case you have a proof that it you have an NP-complete problem.