Data Mining
I Cohen
arXiv preprint arXiv:2504.02083 submitted
Measuring the Data analytically finds the intrinsic manifold in big data. First, Optimal Transport generates the tangent space at each data point from which the intrinsic dimension is revealed. Then, the Koopman Dimensionality Reduction procedure derives a nonlinear transformation from the data to the intrinsic manifold. Measuring the data procedure is presented here, backed up with encouraging results.
System and Method of Training a Neural Network Model
G Gilboa, R Turjeman, T Berkov, I Cohen
US Patent App. 18/231,968
A method and system for implementing a machine-learning (ML) based function may include providing a NN model comprising a plurality of NN parameters; training the NN model over a plurality of training epochs, to implement a predefined ML function, based on a training dataset; for one or more NN parameters of the plurality of NN parameters:(i) calculating a profile vector, representing evolution of the NN parameter through the plurality of training epochs; and (ii) calculating an approximated value of the at least one NN parameter, based on the profile vector; and replacing at least one NN parameter value in the trained NN model with a respective calculated approximated value, to obtain an approximated version of the trained NN model.
Enhancing Neural Training via a Correlated Dynamics Model
J Brokman, R Betser, R Turjeman, T Berkov, I Cohen, G Gilboa
The Twelfth International Conference on Learning Representations (ICLR)
As neural networks grow in scale, their training becomes both computationally demanding and rich in dynamics. Amidst the flourishing interest in these training dynamics, we present a novel observation: Parameters during training exhibit intrinsic correlations over time. Capitalizing on this, we introduce Correlation Mode Decomposition (CMD). This algorithm clusters the parameter space into groups, termed modes, that display synchronized behavior across epochs. This enables CMD to efficiently represent the training dynamics of complex networks, like ResNets and Transformers, using only a few modes. Moreover, test set generalization is enhanced. We introduce an efficient CMD variant, designed to run concurrently with training. Our experiments indicate that CMD surpasses the state-of-the-art method for compactly modeled dynamics on image classification. Our modeling can improve training efficiency and lower communication overhead, as shown by our preliminary experiments in the context of federated learning.
presentation | video | arXiv | code
BASIS: Batch Aligned Spectral Embedding Space
O Streicher, I Cohen, G Gilboa
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Graph is a highly generic and diverse representation, suitable for almost any data processing problem. Spectral graph theory has been shown to provide powerful algorithms, backed by solid linear algebra theory. It thus can be extremely instrumental to design deep network building blocks with spectral graph characteristics. For instance, such a network allows the design of optimal graphs for certain tasks or obtaining a canonical orthogonal low-dimensional embedding of the data. Recent attempts to solve this problem were based on minimizing Rayleigh-quotient type losses. We propose a different approach of directly learning the graph’s eigensapce. A severe problem of the direct approach, applied in batch-learning, is the inconsistent mapping of features to eigenspace coordinates in different batches. We analyze the degrees of freedom of learning this task using batches and propose a stable alignment mechanism that can work both with batch changes and with graph-metric changes. We show that our learnt spectral embedding is better in terms of NMI, ACC, Grassman distnace, orthogonality and classification accuracy, compared to SOTA. In addition, the learning is more stable.
presentation | video | arXiv | code
The underlying correlated dynamics in neural training
R Turjeman, T Berkov, I Cohen, G Gilboa
Arxiv
Training of neural networks is a computationally intensive task. The significance of understanding and modeling the training dynamics is growing as increasingly larger networks are being trained. We propose in this work a model based on the correlation of the parameters’ dynamics, which dramatically reduces the dimensionality. We refer to our algorithm as \emph{correlation mode decomposition} (CMD). It splits the parameter space into groups of parameters (modes) which behave in a highly correlated manner through the epochs. We achieve a remarkable dimensionality reduction with this approach, where networks like ResNet-18, transformers and GANs, containing millions of parameters, can be modeled well using just a few modes. We observe each typical time profile of a mode is spread throughout the network in all layers. Moreover, our model induces regularization which yields better generalization capacity on the test set. This representation enhances the understanding of the underlying training dynamics and can pave the way for designing better acceleration techniques.
presentation | video | arXiv | code
