Infimal Convolution
Contents
9.19. Infimal Convolution#
Note
Some results in this section appear without complete proof at the moment. They have been collected from various sources. The proofs will be added later.
9.19.1. Definition#
(Infimal convolution)
Let \(\VV\) be a real vector space. Let \(f, g : \VV \to \ERL\) be extended realvalued functions. Their infimal convolution, denoted by \(f \infimal g : \VV \to \ERL\), is defined as
The infimal convolution of \(f\) with \(g\) is called exact at \(\bx\) if the infimum in (9.11) is attained at some \(\by, \bz \in \VV\) such that \(\by + \bz = \bx\).
The infimal convolution is called exact if it is attained at every \(\bx \in X\).
(Alternative definitions)
It is easy to see that
Another formulation is
This definition is reminiscent of the convolution operation in signal processing. The word infimal reminds of the infimum appearing in the definition. This is the motivation for the name infimal convolution.
Infimal convolution is also known as epigraphical addition because performing an infimal convolution of \(f\) and \(g\) amounts to the adding of the strict epigraphs of \(f\) and \(g\). Recall from Definition 2.52 that the strict epigraph of a function \(f : \VV \to \ERL\) is given by
We show in Property 9.42 that \(\epi_s f \infimal g = \epi_s f + \epi_s g\).
From the definition, it is clear that \(f \infimal g\) is the largest of all the functions \(h: \VV \to \ERL\) satisfying
(Cost minimization)
Consider a situation where a person sources a certain product from two different manufactures.
The person needs to acquire \(x\) total quantity.
The total supply is given by the equation \(x = y + z\) where \(y\) is the quantity sourced from manufacturer A and \(z\) is the quantity sourced from manufacture B.
Suppose the total cost of acquisition from manufacture A is given by a function \(f(y)\). It may not be linear as the manufacturer may have different levels of discounts as \(y\) increases.
Similarly, the cost of acquisition from manufacture B is given by the function \(g(z)\).
Then the total cost is given by \(f(y) + g(z)\) subject to the condition that \(y + z = x\).
Then the cost minimization problem is given by
\[ h(x) = \inf \{ f(y) + g(z) \ST x = y + z \} = (f \infimal g) (x). \]We will also be interested in the set of solutions to the infimal problem; i.e.,
\[ \{ (y,z ) \ST f(y) + g(z) = (f \infimal g) (x) \}. \]
9.19.2. Basic Properties#
9.19.2.1. Domain#
(Domain of infimal convolution)
Let \(f, g : \VV \to \ERL\) be given extended valued functions. Then
Proof. Let \(\bz \in \dom f + \dom g\).
Then there exist \(\bx \in \dom f\) and \(\by \in \dom g\) such that \(\bz = \bx + \by\).
Then \(f(\bx) < \infty\) and \(g(\by) < \infty\).
By definition of infimal convolution
\[ (f \infimal g)(\bz) \leq f(\bx) + g(\by) < \infty. \]Hence \(\bz \in \dom f \infimal g\).
Hence \(\dom f + \dom g \subseteq \dom f \infimal g\).
For the converse, let \(\bz \in \dom f \infimal g\).
Then \((f \infimal g)(\bz) < \infty\).
First consider the case where \((f \infimal g)(\bz) = t \in \RR\).
Then for any \(\epsilon > 0\), there exist \(\bx, \by\) with \(\bx + \by = \bz\) such that
\[ t \leq f(\bx) + g(\by) \leq t + \epsilon. \]Since \(f(\bx) + g(\by)\) is finite, hence \(f(\bx) < \infty\) and \(g(\by) < \infty\).
Hence \(\bx < \dom f\) and \(\by \in \dom g\).
Thus \(\bz = \bx + \by \in \dom f + \dom g\).
Now consider the case where \((f \infimal g)(\bz) = \infty\).
Then for every \(M \in \RR\), there exist \(\bx, \by\) with \(\bx + \by = \bz\) such that
\[ f(\bx) + g(\by) \leq M. \]Thus \(f(\bx) < \infty\) and \(g(\by) < \infty\) for such pairs of \(\bx, \by\).
Consequently \(\bz = \bx + \by \in \dom f + \dom g\).
Combining the two cases \(\dom f \infimal g \subseteq \dom f + \dom g\).
Finally
We have already shown that \(\dom f + \dom g \subseteq \dom f \infimal g\).
Thus \(\dom f + \dom g = \dom f \infimal g\).
9.19.2.2. Epigraph#
(Connection with epigraph)
Proof. Let \(t = (f \infimal g) (\bx)\). Define \(S = \{ r \in \RR \ST (\bx, r) \in \epi f + \epi g \}\). We need to show that \(t\) is the greatest lower bound of the set \(S\).
Let \((\bx,r) \in \epi f + \epi g\).
Then there exists \((\by, r_1) \in \epi f\) and \((\bz, r_2) \in \epi g\) so that \(\bx = \by + \bz\) and \(r = r_1 + r_2\).
Thus \(f(\by) + g(\bz) \leq r_1 + r_2 = r\) (by epigraphs).
By infimal convolution definition \(t \leq f(\by) + g(\bz) \leq r\).
Thus we have shown that \(t \leq r\) if \((\bx, r) \in \epi f + \epi g\).
Thus \(t\) is a lower bound of the set \(S\).
Now choose any \(\epsilon > 0\).
Then there exists \(\by \in \dom f\) and \(\bz \in \dom g\) with \(\bx = \by + \bz\) such that
\[ f(\by) + g(\bz) \leq t + \epsilon. \]Note that \((\by, f(\by)) \in \epi f\) and \((\bz, g(\bz)) \in \epi g\).
Hence \((\by + \bz, f(\by) + f(\bz)) \in \epi f + \epi g\).
Consequently \((\by + \bz, t + \epsilon) \in \epi f + \epi g\).
Replacing \(\bx = \by + \bz\), we have \((\bx, t + \epsilon) \in \epi f + \epi g\).
Thus, for every \(\epsilon > 0\), \(t + \epsilon \in S\).
Hence \(t\) must be the greatest lower bound of the set \(S\).
In other words, \(t = \inf S\) as desired.
(Strict epigraph of infimal convolution)
Let \(f, g : \VV \to \ERL\) be given extended valued functions. Then
Proof. We show the set inclusion from both sides.
Let \((\bx, r) \in \epi_s f \infimal g\).
Then \((f \infimal g)(\bx) < r\).
Then there exist \(\by \in \dom f\) and \(\bz \in \dom g\) with \(\bx = \by + \bz\) such that
\[ f(\by) + g(\bz) < r. \]This is directly from the definition of infimal convolution.
Consequently, we can split \(r\) as \(r = r_1 + r_2\) such that \(f(\by) < r_1\) and \(g(\bz) < r_2\).
Let \(s = f(\by)\), \(t = g(\bz)\).
Let \(d = r  (s + t) > 0\).
Let \(e = \frac{d}{2} > 0\).
Let \(r_1 = s + e\) and \(r_2 = t + e\).
Then \(r_1 + r_2 = s + t + 2e = s + t + d = r\).
Also \(f(\by) = s < r_1\) and \(g(\bz) = t < r_2\).
Then \((\by, r_1) \in \epi_s f\) and \((\bz, r_2) \in \epi_s g\) by definition of strict epigraphs.
Hence \((\by + \bz, r_1 + r_2) = (\bx, r) \in \epi_s f + \epi_s g\).
Thus \(\epi_s f \infimal g \subseteq \epi_s f + \epi_s g\)
For the converse
Let \((\bx, r) \in \epi_s f + \epi_s g\).
Then there is \((\by, r_1) \in \epi_s f\) and \((\bz, r_2) \in \epi_s g\) so that
\[ \bx = \by + \bz \text{ and } r = r_1 + r_2. \]Thus \(f(\by) < r_1\) and \(g(\bz) < r_2\).
Thus \(f(\by) + g(\bz) < r_1 + r_2 = r\).
Hence \((f \infimal g)(\bx) \leq f(\by) + g(\bz) < r\).
Hence \((\bx, r) \in \epi_s f \infimal g\).
Hence \(\epi_s f + \epi_s g \subseteq \epi_s f \infimal g\).
(Epigraph of infimal convolution)
Let \(f, g : \VV \to \ERL\) be given functions. Then
The equality holds if and only if the infimal convolution is exact at each \(\bx \in \VV\) where \((f \infimal g) (\bx) \in \RR\).
Proof. We first show the inclusion.
Let \((\bx, r) \in \epi f + \epi g\).
Then there is \((\by, r_1) \in \epi f\) and \((\bz, r_2) \in \epi g\) so that
\[ \bx = \by + \bz \text{ and } r = r_1 + r_2. \]Thus \(f(\by) \leq r_1\) and \(g(\bz) \leq r_2\).
Thus \(f(\by) + g(\bz) \leq r_1 + r_2 = r\).
Hence \((f \infimal g)(\bx) \leq f(\by) + g(\bz) \leq r\).
Hence \((\bx, r) \in \epi f \infimal g\).
Hence \(\epi f + \epi g \subseteq \epi f \infimal g\).
Next we show that the reverse inclusion holds if infimal convolution is exact at each \(\bx \in \VV\) where \((f \infimal g) (\bx) \in \RR\).
Let \((\bx, r) \in \epi f \infimal g\).
Then \((f \infimal g)(\bx) \leq r\).
Consider first the case where \((f \infimal g)(\bx) = \infty\).
Then for any \(M \in \RR\), there exist \(\by \in \dom f\) and \(\bz \in \dom g\) with \(\bx = \by + \bz\) such that \(f(\by) + f(\bz) < M\).
In particular, there exist \(\by, \bz\) so that \(f(\by) + f(\bz) < r\).
Hence \((\bx, r) \in \epi f + \epi g\).
Now if \((f \infimal g)(\bx) \in \RR\) then the infimal convolution is exact.
Since the infimal convolution is exact, hence there exist \(\by \in \dom f\) and \(\bz \in \dom g\) with \(\bx = \by + \bz\) such that
\[ f(\by) + g(\bz) = (f \infimal g)(\bx) \leq r. \]This is directly from the definition of infimal convolution.
Consequently, we can split \(r\) as \(r = r_1 + r_2\) such that \(f(\by) \leq r_1\) and \(g(\bz) \leq r_2\).
Then \((\by, r_1) \in \epi f\) and \((\bz, r_2) \in \epi g\) by definition of epigraphs.
Hence \((\by + \bz, r_1 + r_2) = (\bx, r) \in \epi f + \epi g\).
Thus \(\epi_s f \infimal g \subseteq \epi f + \epi g\).
Finally, to show that this condition is necessary, assume that there exists a point \(\bx \in \VV\) where \((f \infimal g) (\bx) \in \RR\) and the infimal convolution is not exact.
Let \(t = (f \infimal g) (\bx)\).
Then \((\bx, t) \in \epi f \infimal g\).
Since the infimal convolution is not exact, hence for any \(\by \in \dom f\) and \(\bz \in \dom g\) with \(\bx = \by + \bz\)
\[ t < f(\by) + g(\bz). \]For contradiction, assume that \((\bx, t) \in \epi f + \epi g\).
Then there exist \((\by, t_1) \in \epi f\) and \((\bz, t_2) \in \epi g\) so that \(\bx = \by + \bz\) and \(t = t_1 + t_2\).
Thus \(f(\by) \leq t_1\) and \(g(\bz) \leq t_2\).
Thus \(f(\by) + g(\bz) \leq t_1 + t_2 = t\).
We have arrived at a contradiction.
Hence \((\bx, t) \notin \epi f + \epi g\).
9.19.2.3. Convexity#
(Infimal convolution of convex functions)
Let \(f : \VV \to \RERL\) be a proper convex function and \(g : \VV \to \RR\) be a real valued convex function. Then \(f \infimal g\) is convex.
Proof. Define \(h(\bx, \by) = f(\by) + g(\bx  \by)\).
Since \(f\) and \(g\) are convex, hence \(h\) is convex.
We can easily show that for any \(\bx \in \VV\), there exists \(\by \in \VV\) such that \(h(\bx, \by) < \infty\).
Pick any \(\bx \in \VV\).
Choose any \(\by \in \dom f\).
Then \(f(\by) < \infty\).
Since \(g\) is real valued, hence \(g(\bx  \by) < \infty\).
Thus \(h(\bx, \by) = f(\by) + g(\bx  \by) < \infty\).
Then due to Theorem 9.118
\[ (f \infimal g) (\bx) = \inf_{\by \in \VV} h(\bx, \by) = \inf_{\by \in \VV} [f(\by) + g(\bx  \by)] \]is convex as a partial minimization of the function \(h(\bx, by)\) w.r.t. \(\by\).
It is still possible that \(f \infimal g\) is not a proper function and may be equal to \(\infty\) at some \(\bx\).
(Set distance function as infimal convolution)
Let \(C \subseteq \VV\) be a nonempty convex set.
The indicator function \(I_C\) is a proper convex function.
The norm function \(\ \cdot \\) is a real valued convex function.
By Theorem 9.274, \(I_C \infimal \ \cdot \\) is convex.
But then
\[ (I_C \infimal \ \cdot \) (\bx) = \inf_{\by \in \VV} (I_C(\by) + \ \bx  \by \) = \inf_{\by \in C} \ \bx  \by \ = d_C(\bx). \]Thus \(d_C\) (the distance function to a convex set) is convex.
9.19.3. Conjugates#
(Conjugate of infimal convolution of proper functions)
For two proper functions \(h_1, h_2: \VV \to \RERL\), it holds that:
Proof. Pick \(\by \in \VV^*\). Then
(Conjugate of sum of convex functions)
Let \(h_1 : \VV \to \RERL\) be a proper convex function and \(h_2 : \VV \to \RR\) be a real valued convex function. Then
Proof. The proof is more complicated and requires the application of Fenchelâ€™s duality theorem (Theorem 10.85).
Pick any \(\by \in \VV^*\).
Elaborating the conjugate function
\[\begin{split} (h_1 + h_2)^* (\by) &= \sup_{\bx \in \VV} \{ \langle \bx, \by \rangle  (h_1 + h_2)(\bx) \}\\ &= \sup_{\bx \in \VV} \{ \langle \bx, \by \rangle  h_1(\bx)  h_2(\bx) \}\\ &= \inf_{\bx \in \VV} \{ h_1(\bx) + h_2(\bx)  \langle \bx, \by \rangle \}\\ &= \inf_{\bx \in \VV} \{ h_1(\bx) + g(\bx) \}\\ \end{split}\]where \(g(\bx) = h_2(\bx)  \langle \bx, \by \rangle\).
We note that since \(h_2\) is a real valued convex function, hence \(g\) is also a real valued convex function.
We note that
\[ (\relint \dom h_1) \cap (\relint \dom g) = (\relint \dom h_1) \cap \VV = \relint \dom h_1 \neq \EmptySet. \]Applying the Fenchelâ€™s duality theorem (Theorem 10.85),
\[ \inf_{\bx \in \VV} \{ h_1(\bx) + g(\bx) \} = \sup_{\bz \in \VV} \{ h_1^*(\bz)  g^*(\bz) \}. \]Further
\[\begin{split} g^*(\bz) &= \sup_{\bx \in \VV} \{ \langle \bx,  \bz \rangle  g(\bx) \}\\ &= \sup_{\bx \in \VV} \{ \langle \bx,  \bz \rangle  h_2(\bx) + \langle \bx, \by \rangle \}\\ &= \sup_{\bx \in \VV} \{ \langle \bx, \by  \bz \rangle  h_2(\bx) \}\\ &= h_2^*(\by  \bz ). \end{split}\]Consequently
\[\begin{split} (h_1 + h_2)^* (\by) &= \inf_{\bx \in \VV} \{ h_1(\bx) + g(\bx) \} \\ &=  \sup_{\bz \in \VV} \{ h_1^*(\bz)  g^*(\bz) \}\\ &= \inf_{\bz \in \VV} \{h_1^*(\bz) + g^*(\bz) \}\\ &= \inf_{\bz \in \VV} \{h_1^*(\bz) + h_2^*(\by  \bz )\} \\ &= (h_1^* \infimal h_2^*) (\by). \end{split}\]
This completes the proof.
(Function sum as conjugate of infimal convolution of conjugates)
Let \(h_1 : \VV \to \RERL\) be a proper closed convex function and \(h_2 : \VV \to \RR\) be a real valued convex function. Then
Proof. We proceed as follows.
Since \(h_1\) is proper and \(h_2\) is real valued, hence \(h_1 + h_2\) is proper.
Since both \(h_1\) and \(h_2\) are closed functions, hence \(h_1 + h_2\) is closed.
Since both \(h_1\) and \(h_2\) are convex functions, hence \(h_1 + h_2\) is convex.
Thus \(h_1 + h_2\) is a proper, closed convex function.
By Theorem 9.242,
\[ (h_1 + h_2)^{**} = h_1 + h_2. \]By Theorem 9.276,
\[ (h_1 + h_2)^* = h_1^* \infimal h_2^*. \]Hence
\[ h_1 + h_2 = (h_1 + h_2)^{**} = [(h_1 + h_2)^*]^* = [h_1^* \infimal h_2^*]^*. \]
(Representation of the infimal convolution by conjugates)
Let \(h_1 : \VV \to \RERL\) be a proper convex function and \(h_2 : \VV \to \RR\) be a real valued convex function. Suppose \(h_1 \infimal h_2\) is a real valued function. Then
Proof. We proceed as follows.

\[ (h_1 \infimal h_2)^* = h_1^* + h_2^*. \]
Since \(h_1\) is proper and convex and \(h_2\) is real valued and convex, hence by Theorem 9.274, \(h_1 \infimal h_2\) is convex.
By hypothesis \(h_1 \infimal h_2\) is also real valued.
Thus \(h_1 \infimal h_2\) is proper and closed.
Since \(h_1 \infimal h_2\) is proper, closed and convex; hence by Theorem 9.242
\[ (h_1 \infimal h_2)^{**} = h_1 \infimal h_2. \]Hence
\[ h_1 \infimal h_2 = [(h_1 \infimal h_2)^*]^* = [h_1^* + h_2^*]^*. \]
9.19.4. Convexity#
An interesting application of constructing a convex function from a convex set in \(\VV \oplus \RR\) is the fact that addition of epigraphs leads to another convex function. This is also known as infimal convolution.
(Infimal convolution)
Let \(f_1, \dots, f_m : \VV \to \RERL\) be proper convex functions. Let \(f : \VV \to \RERL\) be defined as
Then, \(f\) is a proper convex function.
Proof. Let \(F_i = \epi f_i\) and \(F = F_1 + \dots + F_m\).
Since \(f_i\) is convex, hence \(F_i = \epi f_i\) are convex sets for \(i=1,\dots,m\).
Sum of convex sets is convex. Hence, \(F\) is a convex set.
By definition of \(F\), \((\bx, t) \in F\) if and only if there exists \(\bx_i \in \VV\) and \(t_i \in \RR\) with \((\bx_i, t_i) \in F_i\) such that \(\bx = \bx_1 + \dots + \bx_m\) and \(t = t_1 + \dots + t_m\).
It follows that \(f_i(\bx_i) \leq t_i\) for \(i=1,\dots,m\).
Consequently, \(f_1(\bx_1) + \dots + f_m(\bx_m) \leq t_1 + \dots + t_m\).
By Theorem 9.111, a function \(g: \VV \to \RERL\) defined as
\[ g(\bx) \triangleq \inf \{t \in \RR \ST (\bx, t) \in F \} \]is a proper convex function.
Let \(T = \{t \in \RR \ST (\bx, t) \in F \}\). Then \(g(\bx) = \inf T\).
We note that
\[ T = \{t \ST (\bx_i, t_i) \in F_i, t = t_1 + \dots + t_m, \bx = \bx_1 + \dots + \bx_m \}. \]We claim that \(g = f\).
To show that \(g=f\), we need to do the following.
Show that \(\dom f = \dom g\).
Then, show that for every \(\bx \in \dom f\), \(f(\bx) = g(\bx)\).
Assume that \(\bx \notin \dom g\).
Then, there is no \(t \in \RR\) such that \((\bx, t) \in F\).
Thus, there is no \((\bx_i, t_i) \in F_i\) for \(i=1,\dots,m\) such that \(\bx = \bx_1 + \dots + \bx_m\).
Thus, for every \(\bx_i \in \VV\) such that \(\bx = \bx_1 + \dots + \bx_m\) there is at least one \(i\) such that there is no \(t_i \in \RR\) such that \((\bx_i, t_i) \in F_i\).
Thus, for every \(\bx_i \in \VV\) such that \(\bx = \bx_1 + \dots + \bx_m\) there is at least one \(i\) such that \(f_i(\bx_i) = \infty\).
Thus, for every \(\bx_i \in \VV\) such that \(\bx = \bx_1 + \dots + \bx_m\), \(f_1(\bx_1) + \dots + f_m(\bx_m) = \infty\).
Thus, \(f(\bx) = \infty\).
Thus, \(\bx \notin \dom f\).
Assume that \(\bx \notin \dom f\).
Then, \(f(\bx) = \infty\).
Thus, for every \(\bx_i \in \VV\) such that \(\bx = \bx_1 + \dots + \bx_m\), \(f_1(\bx_1) + \dots + f_m(\bx_m) = \infty\).
Thus, for every \(\bx_i \in \VV\) such that \(\bx = \bx_1 + \dots + \bx_m\), there exists at least one \(i\) such that \(f_i(\bx_i) = \infty\).
Thus, for every \(\bx_i \in \VV\) such that \(\bx = \bx_1 + \dots + \bx_m\), there exists at least one \(i\) such that \(\bx_i \notin \dom f_i\).
Thus, for every \(\bx_i \in \VV\) such that \(\bx = \bx_1 + \dots + \bx_m\), there exists at least one \(i\) such that there is no \(t_i \in \RR\) with \((\bx_i, t_i) \in F_i\).
Thus, there is no \((\bx_i, t_i) \in F_i\) for \(i=1,\dots,m\) such that \(\bx = \bx_1 + \dots + \bx_m\).
Thus, there is no \(t \in \RR\) such that \((\bx, t) \in F\).
Thus, \(\bx \notin \dom g\).
Thus, \(\bx \notin \dom f \iff \bx \notin \dom g\). Thus, \(\dom f = \dom g\).
Let us now consider some \(\bx \in \dom f = \dom g\). Then, both \(f(\bx)\) and \(g(\bx)\) are finite. Our goal is to show that \(f(\bx) = g(\bx)\).
We first show that \(f(\bx) \leq g(\bx)\).
For every \(\bx_i \in \VV\) with \(\bx = \bx_1 + \dots + \bx_m\), and \(f_i(\bx_i)\) finite, we have:
\[ f(\bx) \leq f_1(\bx_1) + \dots + f_m(\bx_m). \]Thus, for every \((\bx_i, t_i) \in F_i\) with \(\bx = \bx_1 + \dots + \bx_m\), we have
\[ f(\bx) \leq f_1(\bx_1) + \dots + f_m(\bx_m) \leq t_1 + \dots + t_m. \]Thus, for every \((\bx, t) \in F\) with \((\bx_i, t_i) \in F_i\), \(\bx = \bx_1 + \dots + \bx_m\), \(t = t_1 + \dots + t_m\), we have
\[ f(\bx) \leq t. \]Taking the infimum on the R.H.S. over the set \(T\), we get
\[ f(\bx) = f(\bx) \leq g(\bx) = z. \]
Now, for contradiction, assume that \(f(\bx) < g(\bx)\).
For every \(t \in \RR\) such that \((\bx, t) \in F\), we have \(g(\bx) \leq t\).
Thus, for every \((\bx_i, t_i) \in F_i\) with \(\bx = \bx_1 + \dots + \bx_m\), we have
\[ g(\bx) \leq t_1 + \dots + t_m. \]Then, for every \((\bx_i, t_i) \in F_i\) with \(\bx = \bx_1 + \dots + \bx_m\), we have
\[ f(\bx) < t_1 + \dots + t_m. \]Thus, there exists \(r > 0\) such that for every \((\bx_i, t_i) \in F_i\) with \(\bx = \bx_1 + \dots + \bx_m\), we have
\[ f(\bx) + r \leq t_1 + \dots + t_m. \]
TO BE COMPLETED.