[{"data":1,"prerenderedAt":311},["ShallowReactive",2],{"post:bulk-upsert-does-not-apply-ingest-pipeline":3},{"id":4,"title":5,"body":6,"date":299,"description":12,"draft":300,"extension":301,"meta":302,"navigation":303,"path":304,"seo":305,"stem":306,"tags":307,"__hash__":310},"posts\u002Fposts\u002Fbulk-upsert-does-not-apply-ingest-pipeline.md","TIL: Elasticsearch Bulk Upsert Does Not Apply the Default Ingest Pipeline To Existing Documents",{"type":7,"value":8,"toc":296},"minimark",[9,13,16,105,108,111,127,130,144,149,156,283,292],[10,11,12],"p",{},"I was today-years-old when I learned that ElasticSearch will not apply an ingest pipeline to existing documents during a bulk upsert operation.",[10,14,15],{},"As a quick example, consider the following bulk upsert request:",[17,18,23],"pre",{"className":19,"code":20,"language":21,"meta":22,"style":22},"language-sh shiki shiki-themes github-light github-dark","POST my-index\u002F_bulk\n{\"update\":{\"_id\":\"123\"}}\n{\"doc\":{\"status\":\"active\",\"count\":1},\"doc_as_upsert\":true}\n","sh","",[24,25,26,39,70],"code",{"__ignoreMap":22},[27,28,31,35],"span",{"class":29,"line":30},"line",1,[27,32,34],{"class":33},"sScJk","POST",[27,36,38],{"class":37},"sZZnC"," my-index\u002F_bulk\n",[27,40,42,46,49,53,55,58,61,64,67],{"class":29,"line":41},2,[27,43,45],{"class":44},"sVt8B","{",[27,47,48],{"class":33},"\"update\"",[27,50,52],{"class":51},"sj4cs",":",[27,54,45],{"class":37},[27,56,57],{"class":44},"\"",[27,59,60],{"class":33},"_id",[27,62,63],{"class":33},"\":\"",[27,65,66],{"class":33},"123",[27,68,69],{"class":33},"\"}}\n",[27,71,73,76,79,82,85,87,90,93,96,99,102],{"class":29,"line":72},3,[27,74,75],{"class":33},"{\"",[27,77,78],{"class":33},"doc",[27,80,81],{"class":33},"\":{\"",[27,83,84],{"class":33},"status",[27,86,63],{"class":33},[27,88,89],{"class":33},"active",[27,91,92],{"class":33},"\",\"",[27,94,95],{"class":33},"count",[27,97,98],{"class":33},"\":1},\"",[27,100,101],{"class":33},"doc_as_upsert",[27,103,104],{"class":33},"\":true}\n",[10,106,107],{},"I ran into an issue where a pre-existing document was retaining field values after bulk index which should removed by the default pipeline.",[10,109,110],{},"I asked Perplexity about this, and I got this response:",[112,113,114,117,120],"blockquote",{},[10,115,116],{},"Yes—Bulk API upserts can use an index’s index.default_pipeline when the upsert results in a new document being created, provided you do not override the pipeline at request or per-item level. Existing-document updates are not generally re-ingested through the default pipeline.",[10,118,119],{},"...",[10,121,122,123,126],{},"If ",[27,124,125],{},"the document"," already exists, Elasticsearch applies the partial update and does not run the ingest pipeline over the merged stored document.",[10,128,129],{},"This feels a bit \"wrong\" to me, instinctually, just because I would expect an ingest pipeline to allow transformations to be applied consistently, whether creating NEW documents or merely updating existing ones.",[10,131,132,133,137,138,143],{},"However...they are called ",[134,135,136],"em",{},"ingest"," pipelines for a reason! And you can still ",[139,140,142],"a",{"href":141},"#manually-applying-a-pipeline-during-bulk-insert","apply pipelines manually",". Lessons learned!",[145,146,148],"h2",{"id":147},"manually-applying-a-pipeline-during-bulk-insert","Manually Applying a Pipeline During Bulk Insert",[10,150,151,152,155],{},"One easy fix is to manually specify the pipeline as a ",[24,153,154],{},"?pipeline=my-pipeline"," query parameter in your indexing requests:",[17,157,159],{"className":19,"code":158,"language":21,"meta":22,"style":22},"POST my-data-stream\u002F_doc?pipeline=my-pipeline\n{\n  \"@timestamp\": \"2099-03-07T11:04:05.000Z\",\n  \"my-keyword-field\": \"foo\"\n}\nPUT my-data-stream\u002F_bulk?pipeline=my-pipeline\n{ \"create\":{ } }\n{ \"@timestamp\": \"2099-03-07T11:04:06.000Z\", \"my-keyword-field\": \"foo\" }\n{ \"create\":{ } }\n{ \"@timestamp\": \"2099-03-07T11:04:07.000Z\", \"my-keyword-field\": \"bar\" }\n",[24,160,161,168,173,183,194,200,209,228,249,264],{"__ignoreMap":22},[27,162,163,165],{"class":29,"line":30},[27,164,34],{"class":33},[27,166,167],{"class":37}," my-data-stream\u002F_doc?pipeline=my-pipeline\n",[27,169,170],{"class":29,"line":41},[27,171,172],{"class":44},"{\n",[27,174,175,178,180],{"class":29,"line":72},[27,176,177],{"class":33},"  \"@timestamp\"",[27,179,52],{"class":51},[27,181,182],{"class":37}," \"2099-03-07T11:04:05.000Z\",\n",[27,184,186,189,191],{"class":29,"line":185},4,[27,187,188],{"class":33},"  \"my-keyword-field\"",[27,190,52],{"class":51},[27,192,193],{"class":37}," \"foo\"\n",[27,195,197],{"class":29,"line":196},5,[27,198,199],{"class":44},"}\n",[27,201,203,206],{"class":29,"line":202},6,[27,204,205],{"class":33},"PUT",[27,207,208],{"class":37}," my-data-stream\u002F_bulk?pipeline=my-pipeline\n",[27,210,212,215,218,220,222,225],{"class":29,"line":211},7,[27,213,214],{"class":44},"{ ",[27,216,217],{"class":33},"\"create\"",[27,219,52],{"class":51},[27,221,45],{"class":37},[27,223,224],{"class":37}," }",[27,226,227],{"class":37}," }\n",[27,229,231,233,236,238,241,244,247],{"class":29,"line":230},8,[27,232,214],{"class":44},[27,234,235],{"class":33},"\"@timestamp\"",[27,237,52],{"class":51},[27,239,240],{"class":37}," \"2099-03-07T11:04:06.000Z\",",[27,242,243],{"class":37}," \"my-keyword-field\":",[27,245,246],{"class":37}," \"foo\"",[27,248,227],{"class":37},[27,250,252,254,256,258,260,262],{"class":29,"line":251},9,[27,253,214],{"class":44},[27,255,217],{"class":33},[27,257,52],{"class":51},[27,259,45],{"class":37},[27,261,224],{"class":37},[27,263,227],{"class":37},[27,265,267,269,271,273,276,278,281],{"class":29,"line":266},10,[27,268,214],{"class":44},[27,270,235],{"class":33},[27,272,52],{"class":51},[27,274,275],{"class":37}," \"2099-03-07T11:04:07.000Z\",",[27,277,243],{"class":37},[27,279,280],{"class":37}," \"bar\"",[27,282,227],{"class":37},[10,284,285,291],{},[139,286,290],{"href":287,"rel":288},"https:\u002F\u002Fwww.elastic.co\u002Fdocs\u002Fmanage-data\u002Fingest\u002Ftransform-enrich\u002Fingest-pipelines#add-pipeline-to-indexing-request",[289],"nofollow","See ES documentation for more details",".",[293,294,295],"style",{},"html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":22,"searchDepth":41,"depth":41,"links":297},[298],{"id":147,"depth":41,"text":148},"2026-09-23T16:43:22-04:00",false,"md",{},true,"\u002Fposts\u002Fbulk-upsert-does-not-apply-ingest-pipeline",{"title":5,"description":12},"posts\u002Fbulk-upsert-does-not-apply-ingest-pipeline",[308,309],"elasticsearch","til","FW6iBxv4sTjhzm2_qCd93bbuyQTsFZHFF29r3JvJ3M0",1793891606389]