Fun with Functionals by dokuDoku Math 🔍 ## What's a Functional? **Note** for this we will be considering only the reals $\mathbb{R}$ for fields, but note that $\mathbb{C}$ is also a valid field, but for simplicity, only consider $\mathbb{R}$ here. Informally, a *Functional* is a function whose input is a function and whose output is a scalar (assume the reals). Symbolically $$ F[f]: f \rightarrow \mathbb{R} $$ For comparison a *Function* takes in a scalar $x$ and outputs another scalar $f(x)$, which we'll assume are both in the reals. $$ f(x): x \rightarrow f(x) \quad x,f(x) \in \mathbb{R}$$ It is now natural to ask *What if I want to input a function and output another function?*, this is called an *Operator*. This is defined as a map between function/vector spaces $$ T: V \rightarrow W$$ To summarize in a table: | Object | Input | Output | | ------------ | -------- | -------- | | Function | number | number | | Functional | function | number | | Operator | function | function | ### Linearity When these mappings are linear in general (with scalars a,b $\in \mathbb{R}$) they satisfy: <center> Map(a input$_1$ + b input$_2$) = a Map(input$_1$) + b Map(input$_2$) </center> The only difference between the definition of linearity across these mappings are - input type - output type **Function Space** To study functions and functionals, it's useful to be able to define what a function space is where functions live, as a vector space for functions. For instance, for n dimensional vectors, we would consider the space $\mathbb{R}^n$, where all real-valued n dimensional vectors live. However, for functions there typically is not a universal space and the nature of the problem determines the space. Here will list commonly considered function spaces. - $C([a,b])$: Continuous Functions from $a$ to $b$ $$C([a,b]) = \{f:[a,b] \to \mathbb{R} | f\ continuous\}$$ - Norm defined as maximum of absolute value $||f|| = max_{x \in [a,b]}|f(x)|$ - $C^1([a,b])$: 1st Differentiable Functions from $a$ to $b$ $$C^1([a,b]) = \{f:[a,b] \to \mathbb{R} | f\ continuous\ and\ f'(x)\ continuous\}$$ - Norm defined as maximum of absolute value plus maximum of absolute value of first derivative $||f|| = max_{x \in [a,b]}|f(x)| + max_{x \in [a,b]}|f'(x)|$ - $C^1([a,b]) \subseteq C([a,b])$ - $C^n([a,b])$: nth Differentiable Functions from $a$ to $b$ $$C^n([a,b]) = \{f:[a,b] \to \mathbb{R} | f\ continuous\ and\ derivatives\ up\ to\ f^{(n)}(x)\ continuous\}$$ - Norm defined as maximum of absolute value plus maximum of absolute value of n derivatives $||f|| = max_{x \in [a,b]}|f(x)| + \sum_i^n max_{x \in [a,b]}|f^{(i)}(x)|$ - $L^2([a,b]) = \left\{ f : [a,b] \to \mathbb{R} \;\middle|\; \int_a^b |f(x)|^2 \, dx < \infty \right\}$: Space of functions that are square integrable - Inner product defined as $\langle f,g \rangle = \int_a^b f(x)g(x)dx$ (as we'll see later) - Norm from inner product $||f|| = \sqrt{\int_a^b f^2(x) dx}$ - Hilbert space since it's complete & has inner product A functional is simply defined as a mapping from a function space to the reals. Using the function space $C([a,b])$ we get: $$ \{F: C([a,b]) \to \mathbb{R}\}$$ where this denotes the set of all functionals on $C([a,b])$, which is a vector space. Now to define linearity conditions for our three mappings, we have the following table. Note that function space $C$ arbitrarily chosen, but it could have just as validly been $C^1$, $C^n$, or another function space. | Object | Map | Linearity condition | | ---------- | ------------------------------ | ---------------------- | | Function | $f: \mathbb{R} \to \mathbb{R}$ | $f(ax+by)=af(x)+bf(y)$ | | Functional | $F: C([a,b]) \to \mathbb{R}$ | $F(af+bg)=aF(f)+bF(g)$ | | Operator | $T: V \to W $ | $T(af+bg)=aT(f)+bT(g)$ | ### Local Functionals A *local functional* is a type of functional that depends only on local information about that function. Specifically a functional $F[f]$ is local if can be written in the form $$ F[f] = \int L(x, f(x), f'(x), f''(x),...)dx$$ A local functional only depends on the value of $f$ at each point $x$, and possibly its derivatives at that same point $x$. Of note it does not depend on values of $f$ far away in a non-local way. We can notice that this formula $ F[f] = \int L(x, f(x), f'(x), f''(x),...)dx$ has great significance in physics where $L()$ is denoted as the *Lagrangian* and we can use the *Calculus of Variations* to find the optimal function $f(x)$ that extremizes $F[f]$ given the Lagragian $L$. This can be used for solving problems such as optimizing the KL-divergence in continuous space and principle of least action. ## Functionals as Infinite Dimensional Vectors First consider an n dimensional vector in $\mathbb{R}^n$ $$v = [v_1, v_2, ..., v_n]^T$$ this can be thought of as a list of coefficients where each multiplies a basis vector. Now consider replacing the finite list with a function $f(x)$, where instead of indexing by $i = 1,2,...n$ you index by $x$, giving $$ f = \{f(x) | x \in \mathbb{R}\}$$ Which is an infinite list of values, one for every $x$. To make this analogy precise, consider that for a vector, we have coefficients relative to the basis vectors $\{e_i\}$ where each $e_i$ is a one hot vector with a 1 in the ith position. In function spaces, we have the basis functions $\{\phi_i(x)\}$, that's defined as a set of functions that's the basis of the function space. One such sequence of basis functions for the subset of polynomials are the polynomial functions where $$ \{\phi_i(x)\} = \{1,x,x^2,x^3,...\}$$ Hence we that the from linear algebra in finite dimensions, each vector is a linear combination of basis vectors $$ v = \sum_i^n v_i e_i$$ we get that in many function spaces, functions can be represented by a linear combination of basis functions. For polynomial functions for example $$ f(x) = \sum_i^\infty a_i \phi_i(x) $$ Just like in linear algebra, a basis must both span the range (any function $f$ can be written as linear combination of basis functions) and be linear independent (no function can be written as combination of others). Another commonly used basis is the *Fourier basis* $$\phi_n(x) = \{sin(nx), cos(nx)\}_{n=1}^\infty$$ Forms an orthogonal basis of $L^2([-\pi,\pi])$. Note that n used instead of i to not confuse with imaginary numbers. ### What's the inner product between two functions in $L^2$? In the discrete case, the inner product between two vectors, $v,w \in \mathbb{R}^n$ is defined as $$ \langle v, w \rangle = \sum_i^n v_i \times w_i$$ Similarly, if for continuous functions $f,g$ we have that we want to sum over all the product of the points in $f$ and $g$. This naturally would be an integral, which is a continuous summation $$ \langle f,g \rangle = \int f(x) \times g(x) dx$$ Note also that the integral satisfies all three properties needed for an inner product: 1. Linearity $\langle au + bv,w \rangle = a\langle u,w \rangle + b \langle v,w \rangle$ - Integrals similarly $$ \langle au + bv,w \rangle = \int (a\cdot u(x) + b \cdot v(x)) \times w(x) dx = \int a\cdot u(x)w(x) dx + \int b \cdot v(x)w(x) dx $$ $$ a \int u(x)w(x) dx + b \int v(x)w(x) dx = a \langle u,w \rangle + b \langle v, w \rangle$$ 2. Symmetry (reals) $\langle u,v \rangle = \langle v,u \rangle$ - Integrals similarly $$ \langle u,v \rangle = \int u(x)v(x) dx = \int v(x)u(x) dx = \langle v,u \rangle$$ 3. Positive Definiteness $\langle v,v \rangle \geq 0$ - Integrals similarly $$ \langle v,v \rangle = \int v(x)v(x) dx = \int v^2(x) dx$$ - Note that $\forall x$, $v^2(x) \geq 0$ thus giving us $\int v^2(x) dx \geq 0$ - Hence $\langle v,v \rangle = \int v^2(x) dx \geq 0$ Great, we showed that integrals are valid inner products for functions in $L^2$! ## What's a Functional? **Note** for this we will be considering only the reals $\mathbb{R}$ for fields, but note that $\mathbb{C}$ is also a valid field, but for simplicity, only consider $\mathbb{R}$ here. Informally, a *Functional* is a function whose input is a function and whose output is a scalar (assume the reals). Symbolically $$ F[f]: f \rightarrow \mathbb{R} $$ For comparison a *Function* takes in a scalar $x$ and outputs another scalar $f(x)$, which we'll assume are both in the reals. $$ f(x): x \rightarrow f(x) \quad x,f(x) \in \mathbb{R}$$ It is now natural to ask *What if I want to input a function and output another function?*, this is called an *Operator*. This is defined as a map between function/vector spaces $$ T: V \rightarrow W$$ To summarize in a table: | Object | Input | Output | | ------------ | -------- | -------- | | Function | number | number | | Functional | function | number | | Operator | function | function | ### Linearity When these mappings are linear in general (with scalars a,b $\in \mathbb{R}$) they satisfy: <center> Map(a input$_1$ + b input$_2$) = a Map(input$_1$) + b Map(input$_2$) </center> The only difference between the definition of linearity across these mappings are - input type - output type **Function Space** To study functions and functionals, it's useful to be able to define what a function space is where functions live, as a vector space for functions. For instance, for n dimensional vectors, we would consider the space $\mathbb{R}^n$, where all real-valued n dimensional vectors live. However, for functions there typically is not a universal space and the nature of the problem determines the space. Here will list commonly considered function spaces. - $C([a,b])$: Continuous Functions from $a$ to $b$ $$C([a,b]) = \{f:[a,b] \to \mathbb{R} | f\ continuous\}$$ - Norm defined as maximum of absolute value $||f|| = max_{x \in [a,b]}|f(x)|$ - $C^1([a,b])$: 1st Differentiable Functions from $a$ to $b$ $$C^1([a,b]) = \{f:[a,b] \to \mathbb{R} | f\ continuous\ and\ f'(x)\ continuous\}$$ - Norm defined as maximum of absolute value plus maximum of absolute value of first derivative $||f|| = max_{x \in [a,b]}|f(x)| + max_{x \in [a,b]}|f'(x)|$ - $C^1([a,b]) \subseteq C([a,b])$ - $C^n([a,b])$: nth Differentiable Functions from $a$ to $b$ $$C^n([a,b]) = \{f:[a,b] \to \mathbb{R} | f\ continuous\ and\ derivatives\ up\ to\ f^{(n)}(x)\ continuous\}$$ - Norm defined as maximum of absolute value plus maximum of absolute value of n derivatives $||f|| = max_{x \in [a,b]}|f(x)| + \sum_i^n max_{x \in [a,b]}|f^{(i)}(x)|$ - $L^2([a,b]) = \left\{ f : [a,b] \to \mathbb{R} \;\middle|\; \int_a^b |f(x)|^2 \, dx < \infty \right\}$: Space of functions that are square integrable - Inner product defined as $\langle f,g \rangle = \int_a^b f(x)g(x)dx$ (as we'll see later) - Norm from inner product $||f|| = \sqrt{\int_a^b f^2(x) dx}$ - Hilbert space since it's complete & has inner product A functional is simply defined as a mapping from a function space to the reals. Using the function space $C([a,b])$ we get: $$ \{F: C([a,b]) \to \mathbb{R}\}$$ where this denotes the set of all functionals on $C([a,b])$, which is a vector space. Now to define linearity conditions for our three mappings, we have the following table. Note that function space $C$ arbitrarily chosen, but it could have just as validly been $C^1$, $C^n$, or another function space. | Object | Map | Linearity condition | | ---------- | ------------------------------ | ---------------------- | | Function | $f: \mathbb{R} \to \mathbb{R}$ | $f(ax+by)=af(x)+bf(y)$ | | Functional | $F: C([a,b]) \to \mathbb{R}$ | $F(af+bg)=aF(f)+bF(g)$ | | Operator | $T: V \to W $ | $T(af+bg)=aT(f)+bT(g)$ | ### Local Functionals A *local functional* is a type of functional that depends only on local information about that function. Specifically a functional $F[f]$ is local if can be written in the form $$ F[f] = \int L(x, f(x), f'(x), f''(x),...)dx$$ A local functional only depends on the value of $f$ at each point $x$, and possibly its derivatives at that same point $x$. Of note it does not depend on values of $f$ far away in a non-local way. We can notice that this formula $ F[f] = \int L(x, f(x), f'(x), f''(x),...)dx$ has great significance in physics where $L()$ is denoted as the *Lagrangian* and we can use the *Calculus of Variations* to find the optimal function $f(x)$ that extremizes $F[f]$ given the Lagragian $L$. This can be used for solving problems such as optimizing the KL-divergence in continuous space and principle of least action. ## Functionals as Infinite Dimensional Vectors First consider an n dimensional vector in $\mathbb{R}^n$ $$v = [v_1, v_2, ..., v_n]^T$$ this can be thought of as a list of coefficients where each multiplies a basis vector. Now consider replacing the finite list with a function $f(x)$, where instead of indexing by $i = 1,2,...n$ you index by $x$, giving $$ f = \{f(x) | x \in \mathbb{R}\}$$ Which is an infinite list of values, one for every $x$. To make this analogy precise, consider that for a vector, we have coefficients relative to the basis vectors $\{e_i\}$ where each $e_i$ is a one hot vector with a 1 in the ith position. In function spaces, we have the basis functions $\{\phi_i(x)\}$, that's defined as a set of functions that's the basis of the function space. One such sequence of basis functions for the subset of polynomials are the polynomial functions where $$ \{\phi_i(x)\} = \{1,x,x^2,x^3,...\}$$ Hence we that the from linear algebra in finite dimensions, each vector is a linear combination of basis vectors $$ v = \sum_i^n v_i e_i$$ we get that in many function spaces, functions can be represented by a linear combination of basis functions. For polynomial functions for example $$ f(x) = \sum_i^\infty a_i \phi_i(x) $$ Just like in linear algebra, a basis must both span the range (any function $f$ can be written as linear combination of basis functions) and be linear independent (no function can be written as combination of others). Another commonly used basis is the *Fourier basis* $$\phi_n(x) = \{sin(nx), cos(nx)\}_{n=1}^\infty$$ Forms an orthogonal basis of $L^2([-\pi,\pi])$. Note that n used instead of i to not confuse with imaginary numbers. ### What's the inner product between two functions in $L^2$? In the discrete case, the inner product between two vectors, $v,w \in \mathbb{R}^n$ is defined as $$ \langle v, w \rangle = \sum_i^n v_i \times w_i$$ Similarly, if for continuous functions $f,g$ we have that we want to sum over all the product of the points in $f$ and $g$. This naturally would be an integral, which is a continuous summation $$ \langle f,g \rangle = \int f(x) \times g(x) dx$$ Note also that the integral satisfies all three properties needed for an inner product: 1. Linearity $\langle au + bv,w \rangle = a\langle u,w \rangle + b \langle v,w \rangle$ - Integrals similarly $$ \langle au + bv,w \rangle = \int (a\cdot u(x) + b \cdot v(x)) \times w(x) dx = \int a\cdot u(x)w(x) dx + \int b \cdot v(x)w(x) dx $$ $$ a \int u(x)w(x) dx + b \int v(x)w(x) dx = a \langle u,w \rangle + b \langle v, w \rangle$$ 2. Symmetry (reals) $\langle u,v \rangle = \langle v,u \rangle$ - Integrals similarly $$ \langle u,v \rangle = \int u(x)v(x) dx = \int v(x)u(x) dx = \langle v,u \rangle$$ 3. Positive Definiteness $\langle v,v \rangle \geq 0$ - Integrals similarly $$ \langle v,v \rangle = \int v(x)v(x) dx = \int v^2(x) dx$$ - Note that $\forall x$, $v^2(x) \geq 0$ thus giving us $\int v^2(x) dx \geq 0$ - Hence $\langle v,v \rangle = \int v^2(x) dx \geq 0$ Great, we showed that integrals are valid inner products for functions in $L^2$! Comments (0) Please log in to comment. No comments yet. Be the first to comment! ← Back to Blog
Comments (0)
Please log in to comment.
No comments yet. Be the first to comment!