Add an upper bound of the operator norm for limited-memory qn

[MaxenceGollier]

@dpo,

Following JuliaSmoothOptimizers/RegularizedOptimization.jl#275 (comment).

[MaxenceGollier]

@dpo,

You can get the bound by doing the following:
We have

$$a_k = \frac{B_{k-1}s_k}{\sqrt{s_k^T B_{k-1}s_k}},$$

define

$$u_k = \frac{B_{k-1}^{1/2}s_k}{\|B_{k-1}^{1/2}s_k\|}.$$

Then,

$$a_k a_k^T = \frac{B_{k-1}s_k s_k^T B_{k-1}}{s_k^T B_{k-1} s_k} = B_{k-1}^{1/2} (u_k u_k^T) B_{k-1}^{1/2}.$$

Therefore,

$$B_{k-1} - a_k a_k^T = B_{k-1}^{1/2}(I - u_k u_k^T) B_{k-1}^{1/2},$$

and

$$\| B_{k-1} - a_k a_k^T \|_2 \leq \|B_{k-1}\|_2\|I - u_k u_k^T\|_2 = \|B_{k-1}\|_2,$$

therefore we can ignore the $$∥a_k∥^2$$ in the formula.

[dpo]

Great! Thank you.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.35%. Comparing base (32dbc5e) to head (4719536).
⚠️ Report is 59 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #395      +/-   ##
==========================================
+ Coverage   95.00%   95.35%   +0.35%     
==========================================
  Files          17       20       +3     
  Lines        1100     1140      +40     
==========================================
+ Hits         1045     1087      +42     
+ Misses         55       53       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[MaxenceGollier]

If you approve, please merge @dpo.

[dpo]

@MaxenceGollier I thought that we should store those norms instead of recomputing them. That should be an easy change.

[MaxenceGollier]

In RegularizedOptimization, I computed it like this:

@inbounds for i = 1:data.mem
   bound += norm(data.b[i], 2)^2
end

which is $$\mathcal{O}(mn)$$, here I only remove the norm of b being inserted and add the new norm which is $$\mathcal{O}(2n)$$.

[dpo]

It would be $O(n)$ if we stored the norm. The norm of the vector being removed was computed earlier.

[MaxenceGollier]

Done.

[MaxenceGollier]

Can we add a release ?

Then I can update JuliaSmoothOptimizers/RegularizedOptimization.jl#275.