One is that a new and simple rule to update the penalty parameter ρ in (P ρ ) is derived, the other is that, same as in [18], the uniformly positive definite restriction on the Lagrangian Hessian estimate is relaxed.
(d) Under suitable CQ and assumptions including a relaxed positive definite restriction on the Lagrangian Hessian estimate (H_{k}), but without the isolatedness of the stationary points, the proposed algorithm is globally and superlinearly convergent.
Under suitable CQ and assumptions including a relaxed positive definite restriction on the Lagrangian Hessian estimate (H_{k}), but without the isolatedness of the stationary points, the proposed algorithm is globally and superlinearly convergent.
At each iteration, the algorithm needs to solve two or three reduced systems of linear equations with a common coefficient matrix, where a slightly new working set technique for judging the active set is used to construct the coefficient matrix, and the positive definiteness restriction on the Lagrangian Hessian estimate is relaxed.
During the numerical experiments, by modifying the technique in Section 5 of (SIAM J. Optim. 14(1): 173-199, 2003), we introduce a slightly new computation measure for the Lagrangian Hessian estimate based on second order derivative information, which can satisfy the associated assumptions.
All available priors were subsequently included in the Hessian approximation.
In our hybrid framework, the Hessian matrix is estimated solely using the public data set, and we use public and private data sets to compute the gradient.
Based on the simultaneous perturbation idea, Spall further provided a second-order SPSA method, which estimates the Hessian matrix at each iteration.
It would be, nonetheless, interesting to study different implementations for those oracles, e.g. using a backprop-based estimate for the Hessian diagonal, or smart line-searches to estimate the distance from a local minimum.
