<li><ahref="#/explainability-in-artificial-intelligence"id="/toc-explainability-in-artificial-intelligence">Explainability in Artificial Intelligence</a></li>
<li><ahref="#/presentation-and-demonstration-of-mechir"id="/toc-presentation-and-demonstration-of-mechir">Presentation and Demonstration of MechIR</a></li>
<li><ahref="#/demonstration-tasb-vaswani"id="/toc-demonstration-tasb-vaswani">Demonstration (TASB & Vaswani)</a></li>
<figcaption>Advanced RAG Achitecture <spanclass="citation"data-cites="fan_survey_2024">(<ahref="#/references"role="doc-biblioref"onclick="">Fan et al. 2024</a>)</span></figcaption>
</figure>
</div>
</div>
</section>
<sectionid="example"class="slide level3 center">
<h3>Example</h3>
<imgdata-src="images/RAGHN-perso.drawio.png"class="r-stretch quarto-figure-center"><pclass="caption">RAG Architecture used here (inspired from <spanclass="citation"data-cites="tran_retrieval_2024">(<ahref="#/references"role="doc-biblioref"onclick="">Tran et al. 2024</a>)</span>)</p></section></section>
<h3>Explainability in Artificial Intelligence</h3>
<p><em>Open black box models</em></p>
<h4id="aims">Aims</h4>
<ul>
<li>Trustability</li>
<li>Understandability</li>
<li>Model Rectification</li>
</ul>
</section>
<sectionid="definitions"class="slide level3">
<h3>Definitions</h3>
<p><strong>XAI</strong> : Make model’s behavior understandable for human <spanclass="citation"data-cites="bell_its_2022">(<ahref="#/references"role="doc-biblioref"onclick="">Bell et al. 2022</a>)</span><br>
<strong>Understand</strong> : Predict model’s behavior <spanclass="citation"data-cites="bell_its_2022">(<ahref="#/references"role="doc-biblioref"onclick="">Bell et al. 2022</a>)</span><br>
<strong>Explanation</strong> : Any way to make the decision process of the model understandable for human</p>
<imgdata-src="images/VieModele-TempsXAI.drawio.png"class="r-stretch quarto-figure-center"><pclass="caption">Explanation through creation of a model</p></section></section>
<p>Understand the internal mechanisms of neural networks by <strong>performing causal interventions</strong> on specific model components</p>
<h4id="mechir-parry_mechir_2025">MechIR <spanclass="citation"data-cites="parry_mechir_2025">(<ahref="#/references"role="doc-biblioref"onclick="">Parry et al. 2025</a>)</span></h4>
<ul>
<li><p>Encoder-only models</p></li>
<li><p>For Information Retrieval models</p></li>
<li><p>Identify components responsible for some behavior</p></li>
<h3>Activation Patching <spanclass="citation"data-cites="chen_axiomatic_2024">(<ahref="#/references"role="doc-biblioref"onclick="">Chen et al. 2024</a>)</span></h3>
<p>Let <spanclass="math inline">\(Q \times D \subset \mathcal{Q}\times\mathcal{D}\)</span> be a set of pairs of questions and documents<br>
Let <spanclass="math inline">\(Q \times \tilde{D}\)</span> the same set of pairs but with perturbed documents</p>
<oltype="1">
<liclass="fragment">Forward pass all <spanclass="math inline">\(Q\times D\)</span>
<ul>
<liclass="fragment">record <spanclass="math inline">\(o_{i,j}^e\)</span> the output of each component <spanclass="math inline">\(n_{i,j}, \forall e \in Q\times D\)</span></li>
<liclass="fragment">record <spanclass="math inline">\(p_D\)</span> the performance of the model</li>
</ul></li>
<liclass="fragment">Forward pass all <spanclass="math inline">\(Q\times \tilde{D}\)</span>
<ul>
<liclass="fragment">record <spanclass="math inline">\(o_{i,j}^\tilde{e}\)</span> the output of each component <spanclass="math inline">\(n_{i,j}, \forall \tilde{e} \in Q\times \tilde{D}\)</span></li>
<liclass="fragment">record <spanclass="math inline">\(p_\tilde{D}\)</span> the performance of the model<br>
</li>
</ul></li>
<liclass="fragment">Rewrite <spanclass="math inline">\(D, e, \tilde{D} \text{ and } \tilde{e}\)</span> as
<ul>
<liclass="fragment"><spanclass="math inline">\(\hat{D}, \hat{e}, \check{D} \text{ and } \check{e}\)</span> if <spanclass="math inline">\(p_D > p_\tilde{D}\)</span></li>
<liclass="fragment"><spanclass="math inline">\(\check{D}, \check{e}, \hat{D} \text{ and } \hat{e}\)</span> otherwise</li>
<h3>Activation Patching <spanclass="citation"data-cites="chen_axiomatic_2024">(<ahref="#/references"role="doc-biblioref"onclick="">Chen et al. 2024</a>)</span></h3>
<olstart="3"type="1">
<li>Rewrite <spanclass="math inline">\(D, e, \tilde{D} \text{ and } \tilde{e}\)</span> as
<ul>
<li><spanclass="math inline">\(\hat{D}, \hat{e}, \check{D} \text{ and } \check{e}\)</span> if <spanclass="math inline">\(p_D > p_\tilde{D}\)</span></li>
<li><spanclass="math inline">\(\check{D}, \check{e}, \hat{D} \text{ and } \hat{e}\)</span> otherwise</li>
</ul></li>
</ol>
<olstart="4"type="1">
<liclass="fragment">For each component <spanclass="math inline">\(n_{i,j}\)</span> forward pass <spanclass="math inline">\(Q\times\check{D}\)</span> but replace <spanclass="math inline">\(o_{i,j}^{\check{e}}\)</span> by <spanclass="math inline">\(o_{i,j}^{\hat{e}}\)</span> for each <spanclass="math inline">\(\check{e}\)</span>. Record the performance <spanclass="math inline">\(\bar{p}\)</span></li>
<liclass="fragment"><spanclass="math inline">\(P = \frac{\bar{p} - p_\hat{D} }{p_\check{D} - p_\hat{D}}\)</span> gives the impact of the perturbation on the model performance</li>
<spanid="cb1-7"><ahref=""></a><spanclass="cf">return</span> doc.replace(<spanclass="st">"microwave"</span>, <spanclass="st">"toaster"</span>)</span></code></pre></div><buttontitle="Copy to Clipboard"class="code-copy-button"><iclass="bi"></i></button></div>
<spanid="cb2-2"><ahref=""></a>dot_model <spanclass="op">=</span> Dot(dot_model_name)</span></code></pre></div><buttontitle="Copy to Clipboard"class="code-copy-button"><iclass="bi"></i></button></div>
<divclass="cell-output cell-output-stdout">
<pre><code>Moving model to device: cpu
Loaded pretrained model sebastian-hofstaetter/distilbert-dot-tas_b-b256-msmarco into HookedEncoder</code></pre>
Query: [CLS] measurement of dielectric constant of liquids by the use of microwave techniques [SEP]
Baseline Document: [CLS] broadband millimetre wave paramagnetic resonance spectrometer the specimen and waveguide which can be cooled by means of a cryostat are placed between close pole pieces giving high uniform magnetic fields design details and some measurements on zero field splittings are given [SEP]
Perturbed Document: [CLS] broadband millimetre wave paramagnetic resonance spectrometer the specimen and waveguide which can be cooled by means of a cryostat are placed between close pole pieces giving high uniform magnetic fields design details and some measurements on zero field splittings are given [SEP]
Query: [CLS] measurement of dielectric constant of liquids by the use of microwave techniques [SEP]
Baseline Document: [CLS] broadband millimetre wave paramagnetic resonance spectrometer the specimen and waveguide which can be cooled by means of a cryostat are placed between close pole pieces giving high uniform magnetic fields design details and some measurements on zero field splittings are given [SEP]
Perturbed Document: [CLS] broadband millimetre wave paramagnetic resonance spectrometer the specimen and waveguide which can be cooled by means of a cryostat are placed between close pole pieces giving high uniform magnetic fields design details and some measurements on zero field splittings are given [SEP]
<divclass="code-copy-outer-scaffold"><divclass="sourceCode cell-code"id="cb8"><preclass="sourceCode numberSource python number-lines code-with-copy"><codeclass="sourceCode python"><spanid="cb8-1"><ahref=""></a><spanclass="co"># Initialize lists to store baseline and perturbed performances for each dataloader</span></span>
<spanid="cb8-9"><ahref=""></a>plot_score_dists_mult(all_baseline_performance, all_perturbed_performance, plot_type<spanclass="op">=</span><spanclass="st">"kde"</span>)</span></code></pre></div><buttontitle="Copy to Clipboard"class="code-copy-button"><iclass="bi"></i></button></div>
Bell, Andrew, Ian Solano-Kamaiko, Oded Nov, and Julia Stoyanovich. 2022. <span>“It’s <span>Just</span><span>Not</span><span>That</span><span>Simple</span>: <span>An</span><span>Empirical</span><span>Study</span> of the <span>Accuracy</span>-<span>Explainability</span><span>Trade</span>-Off in <span>Machine</span><span>Learning</span> for <span>Public</span><span>Policy</span>.”</span><em>2022 <span>ACM</span><span>Conference</span> on <span>Fairness</span><span>Accountability</span> and <span>Transparency</span></em> (Seoul Republic of Korea), June, 248–66. <ahref="https://doi.org/10.1145/3531146.3533090">https://doi.org/10.1145/3531146.3533090</a>.
Chen, Catherine, Jack Merullo, and Carsten Eickhoff. 2024. <span>“Axiomatic <span>Causal</span><span>Interventions</span> for <span>Reverse</span><span>Engineering</span><span>Relevance</span><span>Computation</span> in <span>Neural</span><span>Retrieval</span><span>Models</span>.”</span><em>Proceedings of the 47th <span>International</span><span>ACM</span><span>SIGIR</span><span>Conference</span> on <span>Research</span> and <span>Development</span> in <span>Information</span><span>Retrieval</span></em> (Washington DC USA), July, 1401–10. <ahref="https://doi.org/10.1145/3626772.3657841">https://doi.org/10.1145/3626772.3657841</a>.
Lewis, Patrick, Ethan Perez, Aleksandra Piktus, et al. 2020. <span>“Retrieval-<span>Augmented</span><span>Generation</span> for <span>Knowledge</span>-<span>Intensive</span><span>NLP</span><span>Tasks</span>.”</span><em>arXiv: Computation and Language</em>.
Tran, The Trung, Carlos-Emiliano González-Gallardo, and Antoine Doucet. 2024. <span>“Retrieval <span>Augmented</span><span>Generation</span> for <span>Historical</span><span>Newspapers</span>.”</span><em>Proceedings of the 24th <span>ACM</span>/<span>IEEE</span><span>Joint</span><span>Conference</span> on <span>Digital</span><span>Libraries</span></em> (Hong Kong China), December, 1–5. <ahref="https://doi.org/10.1145/3677389.3702542">https://doi.org/10.1145/3677389.3702542</a>.
3. Rewrite $D, e, \tilde{D} \text{ and } \tilde{e}$ as
- $\hat{D}, \hat{e}, \check{D} \text{ and } \check{e}$ if $p_D > p_\tilde{D}$
- $\check{D}, \check{e}, \hat{D} \text{ and } \hat{e}$ otherwise
:::{.incremental}
4. For each component $n_{i,j}$ forward pass $Q\times\check{D}$ but replace $o_{i,j}^{\check{e}}$ by $o_{i,j}^{\hat{e}}$ for each $\check{e}$. Record the performance $\bar{p}$
5. $P = \frac{\bar{p} - p_\hat{D} }{p_\check{D} - p_\hat{D}}$ gives the impact of the perturbation on the model performance
...
...
@@ -121,14 +233,22 @@ Let $Q \times \tilde{D}$ the same set of pairs but with perturbed documents
### Animation de l'execution de Activation patching
### Perturbation
### Step 1: Choose a perturbation
Function that applies the same modification on each document.
"Let $Q \\times D \\subset \\mathcal{Q}\\times\\mathcal{D}$ be a set of pairs of questions and documents \n",
"Let $Q \\times \\tilde{D}$ the same set of pairs but with perturbed documents \n",
"\n",
"1. Forward pass all $Q\\times D$\n",
" - record $o_{i,j}^e$ the output of each component $n_{i,j}, \\forall e \\in Q\\times D$\n",
" - record $p_D$ the performance of the model\n",
"2. Forward pass all $Q\\times \\tilde{D}$\n",
" - record $o_{i,j}^\\tilde{e}$ the output of each component $n_{i,j}, \\forall \\tilde{e} \\in Q\\times \\tilde{D}$\n",
" - record $p_\\tilde{D}$ the performance of the model \n",
"3. Rewrite $D, e, \\tilde{D} \\text{ and } \\tilde{e}$ as\n",
"- $\\hat{D}, \\hat{e}, \\check{D} \\text{ and } \\check{e}$ if $p_D > p_\\tilde{D}$\n",
"- $\\check{D}, \\check{e}, \\hat{D} \\text{ and } \\hat{e}$ otherwise\n",
"4. For each component $n_{i,j}$ forward pass $Q\\times\\check{D}$ but replace $o_{i,j}^{\\check{e}}$ by $o_{i,j}^{\\hat{e}}$ for each $\\check{e}$. Record the performance $\\bar{p}$\n",
"5. $P = \\frac{\\bar{p} - p_\\hat{D} }{\\p_\\check{D} - p_\\hat{D}}$ gives the impact of the perturbation on the model performance\n",
"\n",
"\n",
"### Animation de l'execution de Activation patching\n",
"\n",
"### Perturbation\n",
"\n",
"Function that applies the same modification on each document.\n",
"Example : "
],
"id": "fa2ddf7d"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"def perturbation(doc):\n",
" return doc.replace(\"microwave\", \"toaster\")"
],
"id": "2898ee6e",
"execution_count": null,
"outputs": []
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"#### Perturbation creation technique\n",
"\n",
"- Identify vocabulary specific to the dataset \n",
"- find in the vocabulary words with several meaning $m_D$ and $m_D$\n",
"- Replace that word by a synonym of the $m_D$ meaning\n",
"\n",
"\n",
"### What is a good perturbation \n",
"\n",
"- Have an impact of the documents representation\n",
"_Des images de courbes à ajouter ici_\n",
"- Be useful for interpretation\n",
"\n",
"### Enhance a model with MechIR\n",
"TODO\n",
"\n",
"\n",
"---"
],
"id": "069b4e0f"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"# Brouillon\n",
"- Mechir\n",
" - but et concept : cartographier les sensibilités des modèles encoder-based \n",
" - Activation Patching [Chen et al,. 2024]\n",
" - Étapes\n",
" - recul sur le résultat obtenu\n",
" - Perturbation\n",
" - Definition\n",
" - approche de création par étude du vocabulaire important et utilisation des mots poly-sémantiques\n",
"Let $Q \\times D \\subset \\mathcal{Q}\\times\\mathcal{D}$ be a set of pairs of questions and documents \n",
"Let $Q \\times \\tilde{D}$ the same set of pairs but with perturbed documents \n",
"\n",
"1. Forward pass all $Q\\times D$\n",
" - record $o_{i,j}^e$ the output of each component $n_{i,j}, \\forall e \\in Q\\times D$\n",
" - record $p_D$ the performance of the model\n",
"2. Forward pass all $Q\\times \\tilde{D}$\n",
" - record $o_{i,j}^\\tilde{e}$ the output of each component $n_{i,j}, \\forall \\tilde{e} \\in Q\\times \\tilde{D}$\n",
" - record $p_\\tilde{D}$ the performance of the model \n",
"3. Rewrite $D, e, \\tilde{D} \\text{ and } \\tilde{e}$ as\n",
"- $\\hat{D}, \\hat{e}, \\check{D} \\text{ and } \\check{e}$ if $p_D > p_\\tilde{D}$\n",
"- $\\check{D}, \\check{e}, \\hat{D} \\text{ and } \\hat{e}$ otherwise\n",
"4. For each component $n_{i,j}$ forward pass $Q\\times\\check{D}$ but replace $o_{i,j}^{\\check{e}}$ by $o_{i,j}^{\\hat{e}}$ for each $\\check{e}$. Record the performance $\\bar{p}$\n",
"5. $P = \\frac{\\bar{p} - p_\\hat{D} }{\\p_\\check{D} - p_\\hat{D}}$ gives the impact of the perturbation on the model performance\n",
"\n",
"\n",
"### Animation de l'execution de Activation patching\n",
"\n",
"### Perturbation\n",
"\n",
"Function that applies the same modification on each document.\n",
"Example : "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d02801b1",
"metadata": {},
"outputs": [],
"source": [
"def perturbation(doc):\n",
" return doc.replace(\"microwave\", \"toaster\")"
]
},
{
"cell_type": "raw",
"id": "a4e2efff",
"metadata": {},
"source": [
"#### Perturbation creation technique\n",
"\n",
"- Identify vocabulary specific to the dataset \n",
"- find in the vocabulary words with several meaning $m_D$ and $m_D$\n",
"- Replace that word by a synonym of the $m_D$ meaning\n",
"\n",
"\n",
"### What is a good perturbation \n",
"\n",
"- Have an impact of the documents representation\n",
"_Des images de courbes à ajouter ici_\n",
"- Be useful for interpretation\n",
"\n",
"### Enhance a model with MechIR\n",
"TODO\n",
"\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "a80fea01",
"metadata": {},
"source": [
"---\n",
"# Brouillon\n",
"- Mechir\n",
" - but et concept : cartographier les sensibilités des modèles encoder-based \n",
" - Activation Patching [Chen et al,. 2024]\n",
" - Étapes\n",
" - recul sur le résultat obtenu\n",
" - Perturbation\n",
" - Definition\n",
" - approche de création par étude du vocabulaire important et utilisation des mots poly-sémantiques\n",
"Let $Q \\times D \\subset \\mathcal{Q}\\times\\mathcal{D}$ be a set of pairs of questions and documents \n",
"Let $Q \\times \\tilde{D}$ the same set of pairs but with perturbed documents \n",
"\n",
"1. Forward pass all $Q\\times D$\n",
" - record $o_{i,j}^e$ the output of each component $n_{i,j}, \\forall e \\in Q\\times D$\n",
" - record $p_D$ the performance of the model\n",
"2. Forward pass all $Q\\times \\tilde{D}$\n",
" - record $o_{i,j}^\\tilde{e}$ the output of each component $n_{i,j}, \\forall \\tilde{e} \\in Q\\times \\tilde{D}$\n",
" - record $p_\\tilde{D}$ the performance of the model \n",
"3. Rewrite $D, e, \\tilde{D} \\text{ and } \\tilde{e}$ as\n",
"- $\\hat{D}, \\hat{e}, \\check{D} \\text{ and } \\check{e}$ if $p_D > p_\\tilde{D}$\n",
"- $\\check{D}, \\check{e}, \\hat{D} \\text{ and } \\hat{e}$ otherwise\n",
"4. For each component $n_{i,j}$ forward pass $Q\\times\\check{D}$ but replace $o_{i,j}^{\\check{e}}$ by $o_{i,j}^{\\hat{e}}$ for each $\\check{e}$. Record the performance $\\bar{p}$\n",
"5. $P = \\frac{\\bar{p} - p_\\hat{D} }{\\p_\\check{D} - p_\\hat{D}}$ gives the impact of the perturbation on the model performance\n",
"\n",
"\n",
"### Animation de l'execution de Activation patching\n",
"\n",
"### Perturbation\n",
"\n",
"Function that applies the same modification on each document. \n",
"Example : "
],
"id": "99dbc701"
},
{
"cell_type": "code",
"metadata": {},
"source": [
"#| echo: true\n",
"def perturbation(doc):\n",
" return doc.replace(\"microwave\", \"toaster\")"
],
"id": "a87c9cff",
"execution_count": null,
"outputs": []
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"#### Perturbation creation technique\n",
"\n",
"- Identify vocabulary specific to the dataset \n",
"- find in the vocabulary words with several meaning $m_D$ and $m_D$\n",
"- Replace that word by a synonym of the $m_D$ meaning\n",
"\n",
"\n",
"### What is a good perturbation \n",
"\n",
"- Have an impact of the documents representation\n",
"_Des images de courbes à ajouter ici_\n",
"- Be useful for interpretation\n",
"\n",
"### Enhance a model with MechIR\n",
"TODO\n",
"\n",
"\n",
"---"
],
"id": "dab3024c"
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"# Brouillon\n",
"- Mechir\n",
" - but et concept : cartographier les sensibilités des modèles encoder-based \n",
" - Activation Patching [Chen et al,. 2024]\n",
" - Étapes\n",
" - recul sur le résultat obtenu\n",
" - Perturbation\n",
" - Definition\n",
" - approche de création par étude du vocabulaire important et utilisation des mots poly-sémantiques\n",
"Let $Q \\times D \\subset \\mathcal{Q}\\times\\mathcal{D}$ be a set of pairs of questions and documents \n",
"Let $Q \\times \\tilde{D}$ the same set of pairs but with perturbed documents \n",
"\n",
"1. Forward pass all $Q\\times D$\n",
" - record $o_{i,j}^e$ the output of each component $n_{i,j}, \\forall e \\in Q\\times D$\n",
" - record $p_D$ the performance of the model\n",
"2. Forward pass all $Q\\times \\tilde{D}$\n",
" - record $o_{i,j}^\\tilde{e}$ the output of each component $n_{i,j}, \\forall \\tilde{e} \\in Q\\times \\tilde{D}$\n",
" - record $p_\\tilde{D}$ the performance of the model \n",
"3. Rewrite $D, e, \\tilde{D} \\text{ and } \\tilde{e}$ as\n",
"- $\\hat{D}, \\hat{e}, \\check{D} \\text{ and } \\check{e}$ if $p_D > p_\\tilde{D}$\n",
"- $\\check{D}, \\check{e}, \\hat{D} \\text{ and } \\hat{e}$ otherwise\n",
"4. For each component $n_{i,j}$ forward pass $Q\\times\\check{D}$ but replace $o_{i,j}^{\\check{e}}$ by $o_{i,j}^{\\hat{e}}$ for each $\\check{e}$. Record the performance $\\bar{p}$\n",
"5. $P = \\frac{\\bar{p} - p_\\hat{D} }{\\p_\\check{D} - p_\\hat{D}}$ gives the impact of the perturbation on the model performance\n",
"\n",
"\n",
"### Animation de l'execution de Activation patching\n",
"\n",
"### Perturbation\n",
"\n",
"Function that applies the same modification on each document. \n",
"Example : "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "70279707",
"metadata": {},
"outputs": [],
"source": [
"#| echo: true\n",
"def perturbation(doc):\n",
" return doc.replace(\"microwave\", \"toaster\")"
]
},
{
"cell_type": "raw",
"id": "d1b08c4e",
"metadata": {},
"source": [
"#### Perturbation creation technique\n",
"\n",
"- Identify vocabulary specific to the dataset \n",
"- find in the vocabulary words with several meaning $m_D$ and $m_D$\n",
"- Replace that word by a synonym of the $m_D$ meaning\n",
"\n",
"\n",
"### What is a good perturbation \n",
"\n",
"- Have an impact of the documents representation\n",
"_Des images de courbes à ajouter ici_\n",
"- Be useful for interpretation\n",
"\n",
"### Enhance a model with MechIR\n",
"TODO\n",
"\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "3f975028",
"metadata": {},
"source": [
"---\n",
"# Brouillon\n",
"- Mechir\n",
" - but et concept : cartographier les sensibilités des modèles encoder-based \n",
" - Activation Patching [Chen et al,. 2024]\n",
" - Étapes\n",
" - recul sur le résultat obtenu\n",
" - Perturbation\n",
" - Definition\n",
" - approche de création par étude du vocabulaire important et utilisation des mots poly-sémantiques\n",
