AI Needs Relationships?: A Relation-Centric Transformer Architecture for Discrete Mathematical Reasoning

RLCT: A Relation-Centric Transformer Architecture for Discrete Mathematical Reasoning Despite the great success of the standard Transformer architecture in Natural Language Processing, precise and algorithmic non-linear discrete mathematical reasoning remains a major limitation. The standard Self-Attention mechanism treats tokens as independent points in a high-dimensional space, discarding the computed attention matrix immediately after context aggregation. This stateless relational computation approach significantly hinders the preservation of the sequence's structural topology as the network deepens. RLCT (Relation-Centric Transformer, Kappatsu) is a fundamental sequence modeling architecture that elevates 'Token Relationships' themselves to first-class citizens. Alongside traditional token representations, RLCT explicitly preserves and non-linearly updates a dedicated Relation Tensor across network layers, creating a continuous "relation-stream" that encodes the structural coupling states between variables. To solve the O(T2D)O(T2D) memory complexity issue inherent in high-dimensional relational tensors, RLCT introduces a 3-Stage Low-Rank Einsum Decomposition technique that linearly reduces computational overhead while maintaining the informational fidelity of the relationships. Empirical evaluations on highly challenging non-linear discrete mathematical sequence modeling tasks demonstrate that RLCT achieves vastly superior sample efficiency, stable convergence, and structural integrity compared to Vanilla Transformers of the same parameter scale.We are currently conducting large-scale experiments by scaling the RLCT architecture up to a 1.5B parameter language model to evaluate its performance on broader natural language and complex reasoning tasks.and There is currently a technical issue with uploading the paper, and a fix is in progress.