For an MLP probability of default model, what all validation techniques can be used .

Other than AUC,KS,GINI,etc standard techniques.

Also advice on how do one check for fairness/bias for such a model.

This is a credit risk model

I got tired of switching to a browser just to see if my model was alive.

Training lives in the terminal, but monitoring lives in TensorBoard / W&B. That context switch breaks flow, especially over SSH.

So I built nn-htop — runs right in your terminal where training happens.

What it does now:

- live layer stats (mean, std, grad norm) per layer

- catches dead ReLUs / exploding grads early

- loss + lr sparkline in terminal

- zero browser, zero extra ports, works over SSH

Why TUI and not web?

Honestly I wanted to see the shape the network is drawing. Loss curves hide layer misbehavior. I've been obsessed with visualizing up to 10 dimensions — we can only really see in 3D, but if we could see more, we'd know what's happening under the hood. nn-htop is step 1.

Early and hacky, but I use it daily instead of opening a tab.

Stack: Python + Rich / Textual

Repo: https://github.com/marviecephas/nn-htop

Would love feedback from people who train on remote boxes — what stats would you want in an htop for nets? What makes you keep opening TensorBoard?

hey everyone

I've been studying the mathematics behind neural networks and recently finished building my first trainable network from scratch.

It's a 2 → 3 → 1 architecture for XOR, implemented using NumPy without PyTorch or TensorFlow.

I implemented forward propagation, backpropagation, gradient descent and trained it for 5,000 epochs.

The network correctly classifies all four XOR combinations, with a final MSE of 0.004715.

Also built a small Gradio playground to explore the predictions, loss curve and hidden layer activations.

I've documented the implementation and mathematics here:

https://github.com/Vaibh37/Neural-Network

https://github.com/Vaibh37/ML-MATH

Would appreciate feedback on the code and mathematical explanations.

Hey guys! Quick update on my renderer, which I've been working on and off for the past 7–8 months or so.

The library is quite opinionated about DOD (data-oriented design) and the explicit nature of WebGPU, and I've recently added support for neural inference and training.

1. Neural networks for agent behavior

You can now train small neural networks (MLPs), around 1–4 layers deep, using supervised learning directly as part of the library.

In the herd and wolves example below, the agents have simple behavioral rules, like chasing prey when it's nearby or fleeing when a predator gets close. The idea is to train a small model to learn these behaviors and then use it to drive the agents.

For this example, we're talking about a model with roughly 2,000 weights, so obviously not a general-purpose AI model, but enough for these kinds of behaviors.

The library also supports ONNX Runtime for working with larger models or models trained externally with something like PyTorch. For these smaller, more specialized models, though, the built-in MLP implementation will generally be faster in my use case.

Live demo (WebGPU-supported browsers only):

https://nullgraph.dev/dashboard/agents/herd-wolves-gpu

Github

https://github.com/flarelink-in/NullGraph

1. Offline rendering and simulation baking

Another feature I've been working on is offline rendering, with a workflow somewhat similar to what's available in tools like Blender or Houdini.

It's split into two phases:

\- Compute records: Run the expensive simulation and bake the data that needs to be simulated.

\- Render records: Replay those records through the rendering execution chain using a screen texture, generating frames/tracks that can then be stitched together.

The goal is to handle simulations or visuals that would otherwise be too expensive to run in real time, or would take a long time to render, and then replay the result at 60 FPS.

I have a demo of this on my YouTube channel. The bake took up close to 30 GB of space, while the final result was around 500 MB.

Offline rendering demo:

https://youtu.be/pTD-7PCPGrg?si=vfENWIpR9YU-JEbZ

I've also added another demo to the video, this time focusing more on visual aesthetics rather than the science/simulation side.

1. What's next?

NullGraph exists as part of a larger web-native open-world simulation runtime that I work on from time to time.

Right now, I'm looking into improving my neural network implementation to better match or potentially outperform ONNX Runtime for my use cases. One of the things I've noticed is that the ONNX integration can introduce a 1–3 frame delay in my current setup, along with an additional runtime download of around 6.4 MB.

I'm also starting research into co-evolution, with the goal of bringing evolutionary learning and reinforcement learning into the runtime. The co-evolution approach draws on ideas from OpenAI research and the hall-of-fame technique. I've written some design docs for this, but I'll share more once there's a working implementation to show.

One thing I'd love to get your thoughts on:

Would you guys be interested in offline rendering driven by AI prompts directly in the browser? The idea is to let you generate demos or visually complex scenes through code, spend as much time as needed baking them locally, and then replay the results smoothly. I'm also exploring ways for the system to verify its own output rather than just blindly generating code.

I'm curious whether this kind of workflow would be useful to people working on creative coding, simulations, procedural graphics, or web-based experiences.

Let me know what you think!

Fine-tuning a pretrained neural network on a small dataset creates a trade-off for BatchNorm layers. Updating their running statistics can adapt normalization to the new data distribution, but small batches can make those estimates noisy. Freezing the statistics preserves the pretrained estimates, which may be a poor match if the target data differs substantially.

Freezing running statistics is also distinct from freezing BatchNorm's learned scale and shift: those parameters can remain trainable while the stored statistics stay fixed.

For small-batch fine-tuning, what evidence would favor updating the statistics rather than freezing them? How would the choice change when the target distribution differs strongly from the pretraining data?

Disclosure: This discussion prompt was generated through automation.

I am searching for people who are exploring graft neural networks, and i'm also searching for people who are willing to look outside of the language model paradigm.

I'm working on a system right now. I have the scripts running functionally as a toy. Because deployment and scalability are difficult without a team.

Would anybody know where to go to find people that would be interested in working on a system that is safer and more legitimate?

i have generated lookup tables using WASP. Now, we still have error when compared to the actual dataset. Wasp does predict wind power and at some points it's working perfectly fine but since there's a lot of room for improvement. We have to incorporate Deep Learning into it. Since it's a time series data, im thinking rnns/lstm. But how do i start deep learning? Any guide will help!

Is it possible for a simple neural net with a neuron that can only emit +1, -1 or 0 without biases or weights to learn (evolve) to balance a cart - pole? Or is this just starting off too hard?

I built MaRN (Manifold Regularized Networks), a PyTorch library that lets you optimize a compact latent representation instead of directly training every model parameter.

Some results from my current benchmarks:

• MNIST CNN: Reduced a 537,748-parameter CNN to 4,080 trainable parameters (131.8× reduction), with accuracy decreasing from 99.07% → 98.10% (−0.97 pp). A smaller 107,998-parameter CNN was reduced to 1,872 trainable parameters (57.7× reduction), with accuracy decreasing from 98.83% → 97.18% (−1.65 pp).

These results come with trade-offs: mapped models can train substantially slower, and performance varies by task. The benchmarks are exploratory, using synthetic data for some tasks, and aren't evidence of general superiority over direct training.

The library includes global and layer-wise mappings, regularization options, and pruning/LRD integrations.

Code: https://github.com/arjunmnath/MaRN
Docs: https://marn.readthedocs.io/

I'd value feedback on the approach, benchmark design, and where this kind of parameter-efficient optimization could be useful.

Can anyone tell me how I can learn about neural networks with zero prior knowledge.

Maybe suggest a roadmap...

I’m testing a simple idea: combining young-CSF factor FGF17, 7,8-DHF, and an enriched environment to see whether it helps rats learn better. It came from my work with AI neural networks (convolutional and adversarial), where small-world network structures form and the connections between them reason across all the small worlds together, creating generalized reasoning, the sense of self, and unified sequential communication, which is what we call “consciousness.” These connections act as a translator module, using patterns and logic from the many small worlds to generate an infinity of ideas and generalized knowledge, while language is just a bridge: the input resonates as a pulse between the small worlds and returns an output (sound, movement, text, etc) depending on how the language module is trained and on its input/output architecture. We can teach AI models to learn faster with exploration strategies and refined training data that avoids shortcuts and memorization, and true AGI is born when a model stops memorizing and starts generalizing the patterns and logic of its small worlds through a universal language model that bridges them. Consciousness is just the language of the collective small worlds, so I think we can do the same in our own organic neural network by biohacking its architecture with FGF17 and 7,8-DHF and creating a refined training environment with the right inputs and expected outputs, enhancing learning in both velocity and scale (I wonder how smart a rat can get, lol). This month’s Nobel Prize in Medicine for optogenetics will help here too, since it lets us modulate the input used to train the rats in phase, for optimized learning. It’s also a cheap first step toward brain aging and Alzheimer’s research. Learn more about the protocol here.

Remembering Dave Rumelhart, the Grandfather of AI

Oct 9, 1986: The Paper that Started the AI Revolution

Over the past few years, AI has profoundly transformed our world. With the release of ChatGPT and similar Large Language Models (LLMs), companies are racing to build ever more powerful and useful AI systems. The AI tech leaders have become household names: Sam Altman, Demis Hassabis, Rahul Patil, Mustafa Suleyman, and others. But the first giant step in the development of these amazing AI systems happened 40 years ago. Today’s LLMs use many of the same basic algorithms and architecture from the seminal paper written by Rumelhart, Hinton, and Williams in1986[[i]](#_edn1).

Most people probably recognize the second name in that paper. Geoffrey Hinton won the Nobel Prize in Physics in 2024 along with John Hopfield for their contributions to AI and neural networks. Hinton has often been called “The Godfather of AI,” and he is also famous for quitting Google to warn of the dangers of unbridled AI development.

But the first name in this paper, Rumelhart, is almost never mentioned. That saddens me because he contributed so much to this field. In this short writing, I will attempt to shine some light on Rumelhart and his tremendous contributions. Indeed, if Geoff Hinton is the “Godfather of AI,” I believe that Dave Rumelhart should properly be called the “Grandfather of AI”.

The LLMs of today are basically multi-layer neural networks of the type described in that seminal 1986 paper, and they are trained via the backpropagation algorithm that Rumelhart pioneered. Some of you may look at the AI systems of today and say that they are nothing like those described in the 1986 paper, but that is like saying that an F-22 Raptor is nothing like the Wright Brothers’ first airplane. Yes, today's systems are billions of times larger than anything Rumelhart and his colleagues worked on in the 80s, and there have been substantial breakthroughs like the seminal Attention is All You Need paper[[ii]](#_edn2), but those would not be possible without the initial multi-layer neural network architecture and backpropagation algorithms developed by Rumelhart et al.

I will not attempt to cover the entire corpus of work that Dave Rumelhart produced. For that, I point you to Rumelhart’s Wikipedia page. Rather, I would like to share my personal perspectives on Dave’s work and how that work has been foundational to the current AI breakthroughs. In addition, I will share some of my personal experiences of working with Dave Rumelhart over a fifteen-year period. Spoiler alert: I found him to be one of the smartest people I’ve ever encountered.

Rumelhart’s PDP group

In the 80s, Dave Rumelhart and James McClelland organized the PDP group. PDP stands for parallel distributed processing, and many of the topics covered in those early years were compiled into a two-volume book Parallel Distributed Processing by David E. Rumelhart and James L. McClelland[[iii]](#_edn3).  The group met weekly, and included people from many different disciplines: Neuroscience, computer science, physics, cognitive science, and even philosophy. Francis Crick was a member, as was Goeff Hinton while he was visiting there. I became a member after I had visited Caltech to talk with John Hopfield about part of my Ph. D. research on fractal basin boundaries of Hopfield neural networks. John told me I should talk to Rumelhart. I asked what department he was in. I was shocked when John told me that Rumelhart was in the psychology department. As a young, arrogant physicist, I couldn’t imagine any relevant research being done outside of physics, mathematics, or computer science, but, I took John’s advice and went to see Rumelhart.

Rumelhart described his backpropagation algorithm. I remember thinking, “Gradient descent. I think I learned that in kindergarten.” It didn’t seem like something that would change the world, but it did, and it is still changing the world even after 40 years. I told Rumelhart about my research, and he suggested that I use the Hamming distance as a measure for exploring the space of 2¹⁰⁰ states in the Hopfield network. That turned out to be a great tip. I used that measure to define orthogonal vectors to slice through the enormous space. This wouldn’t be the last time Rumelhart gave me sound advice. Indeed, we worked together for the next fifteen years, and, of the 50 patents I have, he probably pointed me in the right direction on more than half of them.

In these PDP meetings, a speaker would present their research, then we would have a discussion about the research. For example, Goeff Hinton gave a talk about the Boltzmann machine. McClelland talked about more conventional rule-based approaches. But the talk that sticks in my mind was as mind-blowing back then as ChatGPT was a few years ago. Terry Sejnowski presented NetTalk. It was a text-to-speech converter that learned the relationships between text and phonemes; the phonemes were then fed into an audio system to produce sound. He came in with a boom box, popped in a cassette, and we listened as the system learned. At the start, it sounded like random noise. Soon, vowel sounds could be identified. Then it sounded like baby babbling and learning to talk. Then, fast-forward to the end of training, the text-to-speech was almost perfect. All learned through backpropagation.

Now this doesn’t sound too impressive today, but back then, it opened everyone’s eyes to the possibility of machine learning. I was hooked. So were many others. It created a frenzy of research and startup companies based on machine learning.

The other thing I remember from these PDP meetings was that, after discussing and arguing several points, Rumelhart would speak. He would summarize the issues, then give his view. Heads would nod around the table in agreement with Rumelhart’s assessment. He had a deep understanding of all of the topics, and he was gifted at explaining the key points and flaws. Even Francis Crick seemed to defer to Rumelhart’s judgment.

Stanford, MCC, and Pavilion

I finished my Ph. D. in Physics at UC San Diego in 1987. I started a post-doc position at Stanford later that year, and, coincidentally, Rumelhart moved to the Psychology Department at Stanford. This was great for me, because I could continue to bash around ideas with Rumelhart. He wasn’t just brilliant in the area of neural networks. He had an encyclopedic knowledge of everything relating to the brain and how it processes information. By way of example, I told him about some research I had completed investigating second-order interactions in neural networks. As the temperature, or signal-to-noise ratio, decreased, the system exhibited a phase transition and critical slowing down. I didn’t expect him to understand an esoteric physics phenomenon, but he instantly understood the implications and pointed me to some neuroscience research showing that norepinephrine functions as a signal-to-noise modulator in the brain. I read the articles and the behavior matched perfectly with our theoretical model, as described in a paper published in the Proceedings of the National Academy of Sciences[[iv]](#_edn4).

Later that year, I got an offer to lead the neural networks research group at MCC – the Microelectronics and Computer Technology Corporation in Austin, TX. MCC was a research consortium kind of like Bell Labs, but supported by many different high-tech companies. Like any good Californian, I thought Texas was hot, flat, racist, and humid. Why would anyone want to live there? But they flew me out to Austin for an interview, and I fell in love with the city.

While at MCC, Rumelhart and I collaborated on several research topics. We worked on character recognition, speech recognition, fraud detection, and process prediction. One of the most important breakthroughs we discovered was the Integrated Segmentation and Recognition system[[v]](#_edn5). It solved the problem of identifying characters that were touching and, more generally, created a mechanism for training a system on a dataset where more than one object can be simultaneously presented in the inputs and target outputs. This was a multi-year investigation, and I was impressed not only with Rumelhart’s mastery of neural network architectures, but his skill as a mathematician and as a programmer. He wrote the neural network software that we used for the research, and he was a damn fine C programmer. This system used a convolutional neural network architecture, as did the work of Martin and Pittman. Note that in the Martin and Pittman paper[[vi]](#_edn6), they also showed that the lower-level neurons developed activation patterns reminiscent of the visual system. This was decades before the convolutional neural network “breakthrough” that was touted in the 2010s. There is an old video where Rumelhart and I discussed this research[[vii]](#_edn7).

The VISA CRIS fraud detection system research was also developed at MCC by Steve Piche (who would become my chief scientist at Pavilion). This system saved $20 million in the first year of operation[[viii]](#_edn8). Rumelhart consulted with Piche on the problem.

Another thing that Rumelhart helped me with was a process control problem. We had lots of data from a chemical process at Eastman Chemical – pressures, temperatures, flow rates, as inputs, and the product quality and flow rate as outputs. The goal was to control the process to get the highest quality and quantity. Of all the machine learning algorithms I could have used, I knew that backpropagation was the best. I trained a model to predict the outputs, and it worked beautifully, but turning the problem around and asking which inputs to change to get better outputs was an unsolved problem. After consulting with Rumelhart, I came up with the Residual Activation Neural Network (RANN) architecture. Applying this to the problem led to a savings of about a million/year on the distillation column that we were working on.

But there were tens of thousands of distillation columns in the US alone. Do the math. Based on this result, I licensed the technology from MCC and co-founded a company, Pavilion Technologies, Inc. It became one of the fastest-growing companies in Austin in the 90s. We applied neural network machine learning techniques to process monitoring, control, and optimization. That’s right. Neural networks have been controlling thousands of manufacturing processes all over the world. Not just in chemical manufacturing, but in refining, pulp-and-paper, cement, food processing, and power production. Many of those processes can go boom in the night, but there has never been an instance of the neural network hallucinating and causing problems.

Pavilion was a powerhouse of innovation, as one can tell from the scores of patents. But back then, machine learning was much harder than it is now. Often we would have data sets with only a few thousand points. We were overjoyed if a dataset had tens of thousands of records. We had to be very clever about cleaning the data and using it to train and validate the models. Moreover, we didn’t have Nvidia DSP chips to accelerate learning, so we used a stiff-differential equation solver to speed up learning (backprop is a set of stiff differential equations). We also invented methods to meld first-principles models with the neural networks. This helped extend the range of validity of the models into regions outside the realm of the training data. We did a lot of work in environmental monitoring and optimization. One of our products, the Software Continuous Emissions Monitor, was approved by the EPA for monitoring emissions. I am proud to say that the Pavilion software systems have abated millions of pounds of pollutants over the years, and the systems are still running. Pavilion’s Process Perfecter software is a closed-loop multivariable nonlinear controller. The math behind that was shockingly hard, but it worked amazingly well. Pavilion was sold to Rockwell Automation, and the software is still running in thousands of manufacturing plants.

During the time I was at Pavilion, Rumelhart served as a technical advisor and consultant. Of all the people I have worked with, I admired and respected Dave Rumelhart the most. He seemed so wise, and, as I said, he always seemed to have the instinct to point people in the right direction on a hard problem. He was also very fun to work with, and we became friends over the years.

The reason I believe Rumelhart should be remembered as the “Grandfather of AI” is that he was the driving force behind the backpropagation algorithm. His insight into the problem stemmed from the famous XOR problem. Single-layer systems could not solve this problem. It required “hidden units”. That realization led to the multi-layer neural network architectures used in today’s LLMs, and backpropagation is how they are trained. If Rumelhart didn’t figure this out, I am sure someone else would have (indeed, a grad student figured it out in his thesis before Rumelhart, but nobody knew about it). But without Rumelhart, it might have been many years, perhaps decades, before people started using backpropagation. Just like someone would have figured out relativity sometime if Einstein didn’t present it, but who knows how long it would have taken? Rumelhart should be remembered for the invention that has changed the world.

Sadly, Rumelhart developed Pick’s disease in the late 90s. He died in 2011. I left Pavilion in 2000, and didn’t talk much with him during those later years. What would he think of the current LLMs? Knowing him as well as I did, I think he would be amazed at the capabilities of these modern AI systems, but I also think he would view the 92-layer neural networks as brute-force, ghastly, and inelegant. He would probably say something like “The cortex only has six layers. We should try to make that work.” And, knowing Dave Rumelhart, he would probably have made great progress in that direction. He was that gifted. His name should be remembered among the great contributors in this field as “The Grandfather of AI”.

-James D. Keeler

[[i]](#_ednref1)  Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. (1986-10-09). "Learning representations by back-propagating errors". Nature. 323 (6088): 533–536.

[[ii]](#_ednref2) Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (December 2017). "Attention is All you Need" (PDF). In I. Guyon and U. Von Luxburg and S. Bengio and H. Wallach and R. Fergus and S. Vishwanathan and R. Garnett (ed.). 31st Conference on Neural Information Processing Systems (NIPS). Advances in Neural Information Processing Systems. Vol. 30. Curran Associates, Inc. arXiv):1706.03762.

[[iii]](#_ednref3) David E. Rumelhart; James L. McClelland; PDP Research Group (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. ISBN) 9780262680530.

[[iv]](#_ednref4) J.D. Keeler, E.E. Pichler, J. Ross “Noise in Neural Networks: Thresholds, Hysteresis, and Neuromodulation of Signal-to-noise,” Proceedings of the National Academy of Sciences, USA, 86 (1989) 1712-1716.

[[v]](#_ednref5) Keeler, James D., Rumelhart, David E., Leow, Wee-Kheng. “Integrated Segmentation and Recognition of Hand-Printed Numerals”. Printed in Neural Information Processing Systems, 3. R. Lippmann, J. Moody, D. Touretzky, Eds. Morgan Kaufmann Publishing, San Mateo, CA.

[[vi]](#_ednref6) G. Martin, J. Pittman (1990) “Recognizing Hand-Printed Letters and Digits”. In D. S. Touretzky (ed). Advances in Neural Information Processing Systems 2, Morgan Kaufmann Publishing, San Mateo, CA.

[[vii]](#_ednref7) https://youtu.be/fG-9ILWI2u4

[[viii]](#_ednref8) Visa to expand fraud detection system; the system saved 15 issuers over $20 million last year. American Banker, Sept. 30, 1994.

I’ve been testing the idea of running models locally rather than depending entirely on hosted inference, and the interesting part isn’t just getting a model to run. The bigger challenge seems to be integrating it into an application while dealing with memory usage, inference speed, documents and retrieval.

LM-Kit is one project I’ve been evaluating for this approach. It supports local AI workloads along with things like RAG and document processing, so I’m interested in how it compares with other ways of building local ML applications.

For those who have worked with local inference in production or serious projects, what has mattered most in practice: model performance, hardware requirements, latency, or integration?

Does anyone know if there is a library or code that uses a stiff differential equation solver to speed up neural network training? I recall reading a paper in the early 90s that claimed 1,000x speed improvement, but I haven't seen anyone using this. Anyone know about this?

Hey r/neuralnetworks ,

I just wrapped up the first release of AnyInit, a Python library designed to handle neural network initialization for custom architectures—without forcing you to manually calculate complex variance-scaling rules to ensure proper signal propagation.

Instead of spending time figuring out math tweaks for non-standard layers or dealing with vanishing and exploding gradients, AnyInit automates stable initializations so custom models work reliably out of the box.

Since it's brand new, I'd love to get some feedback from the community:

• Do you find this useful for your current workflows?
• Are there specific initialization methods, frameworks, or features you'd like to see added?
• Any feedback on the API or implementation?

GitHub repo: https://github.com/jmiravet/AnyInit . Any thoughts, critiques, or feature requests are greatly appreciated!

I've been working on a project where I'm trying to build a Denoising Diffusion Probabilistic Model completely from scratch.

The main rule I'm following is: no using Diffusers as a crutch.

I want to actually understand what's happening inside the model instead of just calling a pipeline and getting an image.

So far I've implemented/learned:

• The forward diffusion process
• Beta schedules
• The reparameterization trick
• A noise scheduler
• Group Normalization
• SiLU activation
• Sinusoidal timestep embeddings
• Residual blocks with timestep conditioning
• Self-attention
• Downsampling and upsampling

The interesting part for me has been realizing that a diffusion U-Net isn't just a normal CNN.

The network needs to know how noisy the current image is, which is why timestep information has to be injected into the network.

I'm building toward training on CelebA at 64×64 and eventually generating faces completely from noise.

My longer-term goal is to understand these architectures deeply enough that I can read, modify and eventually contribute to projects like Hugging Face Diffusers.

Phase 3 is where things start getting interesting:

U-Net assembly.

I'll be documenting the progress as I go. 🔥

What was the hardest part of diffusion models for you when you first learned them?

CAN ANYONE EXPLAIN THIS?

The announcement describes RL for Population Scaling: training models together to develop complementary capabilities.

https://x.com/MLCatttt/status/2107147690450817259

A while ago, I published a preprint proposing a novel approximation of the Mish activation function, which appears to be significantly more computationally efficient while preserving its learning capability. Interested to hear your thoughts.

Paper: https://doi.org/10.36227/techrxiv.176591866.68698045/v2