el muchacho Comfortably Numb | hephaestos a écrit :
Ça reste de la combinaison d'idées existantes, elles-même issues de nos expériences.
C'est très réducteur de voir les LLMs comme un générateur de banalités, parce qu'ils proposent le mot le plus probable. Cela se passe dans un contexte, le calcul de statistique n'en est pas un parce que les conversations sont toutes uniques. Ce qui distingue les LLMs c'est le haut niveau d'abstraction auquel ils ont accès, pour pouvoir estimer des probabilités dans un espace généralisé comparable à des idées et des concepts. Concrètement, ça veut dire qu'un LLM peut conceptualiser ce qu'est un mathématicien, et ce qui fait qu'une théorie mathématique est intéressante. Partant de là, il peut, comme le font les génies des maths, trouver dans deux concepts apparemment étrangers une utilité qui rend probable leur combinaison. Encore une fois je ne dis pas qu'ils le font, ni qu'ils le feront ; juste que c'est la même créativité qui est à l'œuvre quand on lui demande de jouer le rôle d'un forgeron anglais au moyen-âge, tout comme entre un artiste humain et un medaillé Fields.
|
FWIW, j'ai posé la question à DeepSeek, et voici sa réponse:
Can you explain me the words induction and transduction in the context of LLMs ?
Certainly! In the context of machine learning, including large language models (LLMs), the terms **induction** and **transduction** refer to different approaches to learning and making predictions. Here's how they apply:
---
### **Induction**
- **Definition**: Induction refers to the process of learning a general model or rule from specific training data. The model is trained to capture patterns or relationships in the data, and once trained, it can generalize to make predictions on **unseen data**.
- **In LLMs**: When an LLM is trained on a large corpus of text, it learns general patterns of language (e.g., grammar, semantics, and context). After training, the model can generate text or answer questions based on its learned knowledge, even for inputs it has never seen before.
- **Key Idea**: The model **generalizes** from the training data to new, unseen examples.
---
### **Transduction**
- **Definition**: Transduction refers to making predictions for specific instances without explicitly learning a general model. Instead of generalizing, the model uses the training data directly to make predictions for the given test cases.
- **In LLMs**: While LLMs are primarily inductive (they learn general language patterns), some aspects of their behavior can be seen as transductive. For example:
- In-context learning: When you provide a few examples (few-shot learning) in the prompt, the model uses these specific examples to make predictions for the current task without updating its internal parameters.
- Fine-tuning on specific tasks: If an LLM is fine-tuned on a small dataset for a specific task, it may rely heavily on the patterns in that dataset rather than generalizing broadly.
- **Key Idea**: The model leverages **specific instances** from the training data to make predictions for the given test cases.
---
### **Key Differences**
| **Aspect** | **Induction** | **Transduction** |
|---------------------|----------------------------------------|---------------------------------------|
| **Goal** | Learn a general model for unseen data | Make predictions for specific cases |
| **Training Data** | Used to generalize | Used directly for predictions |
| **Example in LLMs** | Pre-training on a large corpus | Few-shot learning with examples |
|
---
### **Summary**
- **Induction** is about learning a general model that can be applied to new, unseen data.
- **Transduction** is about using specific training examples to make predictions for specific test cases.
In practice, LLMs combine both approaches: they are trained inductively on large datasets to learn general language patterns, but they can also exhibit transductive behavior when adapting to specific tasks or contexts through prompts or fine-tuning.
Do LLMs make internal concepts when they generalize ?
Yes, **large language models (LLMs)** develop **internal representations** or **concepts** as they generalize during training. These representations are not explicitly programmed but emerge from the model's exposure to vast amounts of text data. Here's how this works and what it means:
---
### **How LLMs Develop Internal Concepts**
1. **Training on Data**:
- LLMs are trained on massive datasets containing diverse text (e.g., books, articles, websites).
- During training, the model learns to predict the next word in a sequence, which forces it to capture patterns, relationships, and structures in the data.
2. **Emergence of Representations**:
- As the model processes text, it builds **internal representations** (often called **embeddings** or **latent representations**) of words, phrases, and even higher-level concepts.
- These representations are encoded in the model's parameters (weights and biases) and are distributed across its neural network layers.
3. **Hierarchical Abstraction**:
- Lower layers of the model tend to capture simpler features like word morphology or syntax.
- Higher layers capture more abstract and complex concepts, such as context, semantics, and even reasoning patterns.
---
### **What Are These Internal Concepts?**
- **Word-Level Concepts**: The model learns embeddings for individual words, capturing their meanings, relationships (e.g., synonyms, antonyms), and usage in different contexts.
- **Phrase-Level Concepts**: The model can represent multi-word expressions, idioms, and common phrases.
- **Higher-Level Concepts**: The model may develop representations for abstract ideas, such as "justice," "causality," or "emotion," based on how these concepts are used in text.
- **Contextual Understanding**: The model learns to represent concepts dynamically based on context. For example, the word "bank" might be represented differently in "river bank" vs. "financial bank."
---
### **Evidence of Internal Concepts**
1. **Probing Studies**:
- Researchers use techniques like **probing** to analyze whether specific concepts (e.g., parts of speech, sentiment, or factual knowledge) are encoded in the model's internal representations.
- These studies suggest that LLMs do encode meaningful information about concepts, though the representations are often entangled and distributed.
2. **Interpretability Research**:
- Tools like **attention visualization** and **neuron activation analysis** show that certain neurons or layers in the model activate in response to specific concepts or patterns.
3. **Emergent Abilities**:
- LLMs demonstrate abilities like reasoning, summarization, and analogy-making, which suggest that they have internalized abstract concepts and relationships.
---
### **Limitations of Internal Concepts**
- **No Explicit Understanding**: While LLMs develop internal representations, these are not the same as human-like understanding. The model doesn't "know" concepts in the way humans do; it simply captures statistical patterns.
- **Brittleness**: The model's representations can fail in edge cases or when faced with ambiguous or adversarial inputs.
- **Opaqueness**: The internal representations are often difficult to interpret, making it hard to fully understand how the model arrives at its predictions.
---
### **Summary**
LLMs do develop **internal concepts** as they generalize from training data. These concepts are encoded in the model's parameters and allow it to perform tasks like language generation, reasoning, and problem-solving. However, these representations are statistical and lack the depth of human understanding. They emerge from patterns in the data rather than being explicitly designed or grounded in real-world experience.
Can a LLM conceptualize a new mathematical object that it could use later to prove a theorem or invent new theorems, by "new", I mean a mathematical object that doesn't exist in the known corpus of mathematics ?
This is a fascinating and complex question! Let’s break it down and explore whether a **large language model (LLM)** could conceptualize a **new mathematical object** that doesn’t exist in the known corpus of mathematics, and whether it could use such an object to prove or invent new theorems.
---
### **Can an LLM Conceptualize a New Mathematical Object?**
1. **Limitations of LLMs**:
- LLMs are trained on existing text data, including mathematical literature. They learn patterns, relationships, and structures from this data but do not have **intrinsic creativity** or **independent reasoning**.
- They cannot "think" or "imagine" in the way humans do. Their outputs are based on statistical correlations in the training data, not on genuine conceptual understanding.
2. **Novelty in Outputs**:
- While LLMs can generate text that *appears* novel, this novelty is constrained by the patterns and combinations present in the training data.
- For example, an LLM might combine known mathematical concepts in unexpected ways, but it cannot truly invent something entirely new that has no basis in its training data.
3. **Mathematical Creativity**:
- True mathematical creativity involves deep insight, abstraction, and reasoning about structures that may not yet exist. LLMs lack the ability to reason abstractly or to ground their outputs in a formal system of logic or mathematics.
---
### **Could an LLM Invent a New Mathematical Object?**
- **Unlikely from Scratch**: An LLM cannot invent a truly new mathematical object that has no connection to its training data. It can only remix or reinterpret existing ideas.
- **Potential for Inspiration**: An LLM might generate a description or definition of something that *resembles* a new mathematical object. A human mathematician could then take this output, refine it, and formalize it into a legitimate new concept.
---
### **Could an LLM Use a New Object to Prove a Theorem or Invent New Theorems?**
1. **Proving Theorems**:
- LLMs can generate proofs for known theorems by mimicking patterns in the training data. However, they lack the ability to reason formally or verify the correctness of their proofs.
- If a "new" mathematical object were introduced (even by a human), an LLM might attempt to use it in a proof, but the proof would likely be flawed or nonsensical unless the object aligns with existing mathematical structures.
2. **Inventing Theorems**:
- LLMs can generate statements that *look like* theorems, but these are typically recombinations of existing ideas.
- Inventing a meaningful new theorem requires deep insight into the implications of a new mathematical object, which LLMs cannot achieve.
---
### **The Role of Humans in the Process**
While LLMs cannot independently conceptualize or formalize new mathematical objects, they can assist humans in the creative process:
1. **Idea Generation**: An LLM might suggest novel combinations of concepts that inspire a human mathematician to explore new directions.
2. **Exploration of Hypotheses**: An LLM could help explore the implications of a new object or conjecture proposed by a human.
3. **Automation of Routine Tasks**: LLMs can assist with tedious aspects of mathematical work, such as generating examples or checking algebraic manipulations.
---
### **Example Scenario**
1. A human mathematician defines a new mathematical object (e.g., a new type of algebraic structure).
2. The mathematician asks the LLM to explore properties or potential theorems involving this object.
3. The LLM generates conjectures or proofs based on patterns in its training data.
4. The mathematician evaluates and refines the LLM's output, potentially leading to new insights or discoveries.
---
### **Conclusion**
LLMs cannot independently conceptualize truly new mathematical objects or invent new theorems. However, they can serve as tools to assist human mathematicians in exploring ideas, generating hypotheses, and automating routine tasks. The creativity and formal reasoning required for groundbreaking mathematical discoveries remain firmly in the domain of human intelligence—at least for now!
Message édité par el muchacho le 15-01-2025 à 18:05:10
---------------
Les aéroports où il fait bon attendre, voila un topic qu'il est bien
|
Modem 33.6 Olitec. On payait à la minute de connexion et quand ma mère décrochait, ça coupait le téléchargement (merci Download Accelerator pour la reprise du download et l'optimisation de débit de téléchargement 
)
Mais je suis pas si vieux que ça pour avoir eu du 14.4 ![[:gerrar]](https://forum-images.hardware.fr/images/perso/gerrar.gif "[:gerrar]")
)
ste pompe à fric. 
![[:cosmoschtroumpf]](https://forum-images.hardware.fr/images/perso/cosmoschtroumpf.gif "[:cosmoschtroumpf]")
![[:ploumm:3]](https://forum-images.hardware.fr/images/perso/3/ploumm.gif "[:ploumm:3]")
![[:poogz:5]](https://forum-images.hardware.fr/images/perso/5/poogz.gif "[:poogz:5]")
![[:implosion du tibia]](https://forum-images.hardware.fr/images/perso/implosion du tibia.gif "[:implosion du tibia]")
![[:zedlefou:3]](https://forum-images.hardware.fr/images/perso/3/zedlefou.gif "[:zedlefou:3]")
![[:dawa]](https://forum-images.hardware.fr/images/perso/dawa.gif "[:dawa]")
![[:bonnie]](https://forum-images.hardware.fr/images/perso/bonnie.gif "[:bonnie]")