One of the stage in my pipeline has a web activity and I have set the Secure output and Secure input to true. Now if my pipeline fails due to some other activity and if I rerun it from the failed activity, this web activity is skipped and the status of that stage comes out as failed with the error message Operation on target failed: Operation on target : Activity '' references the SecureOutput from previous activity ''. Since ADF does not cache secure output, please rerun from activity '' or rerun the whole pipeline run instead. Any thoughts on how to handle this?
Thanks for the question. This one seems to be a bit at odds with the functionality of rerunning a pipeline. I tested this in my enviornment with a Get Metadata and a Filter activity that referenced the output of the Get Metadata. I set the secureOutput to true on the Get Metadata and attempted to rerun and it failed because of the same error message that you mentioned. From what I have read, you would lose the ability to do accomplish this if you are referencing activities upstream in your pipeline. The end result would be to either run the whole pipeline again as mentioned in the error due to the security of not storing the outputs.
There should be two pipeline runs on the original failed run that has the failed pipeline and the one that succeeded. There may be a carrot or arrow icon that you need to click to view it.
That's a great question! There is some awesome conditional logic you can bring into a pipeline to only run activities upon the success or failure of other activities using precedent constraints. You can also use the expression builder in pipelines and utilize the outputs of activities that are further upstream in a pipeline. Those are two ways that you can automate your ETL and run large loads in one batch sequentially. For this specific instance though there was an error that needed to be corrected in order to rerun from the failed activity. Once a pipeline run fails you need to manually correct that issue before it can be rerun. Therefore, there would not be a great way to automate this specific process using that feature from the Monitor Hub.
![Azure Synapse Analytics: Introduction to Copy Activity [Introduction to Synapse Analytics - Ep. 10]](http://i.ytimg.com/vi/o7-GNQxY7P4/mqdefault.jpg)
![Azure Synapse Analytics: Introduction to Copy Activity [Introduction to Synapse Analytics - Ep. 10]](/img/tr.png)
One of the stage in my pipeline has a web activity and I have set the Secure output and Secure input to true.
Now if my pipeline fails due to some other activity and if I rerun it from the failed activity, this web activity is skipped and the status of that stage comes out as failed with the error message
Operation on target failed: Operation on target : Activity '' references the SecureOutput from previous activity ''. Since ADF does not cache secure output, please rerun from activity '' or rerun the whole pipeline run instead.
Any thoughts on how to handle this?
Thanks for the question.
This one seems to be a bit at odds with the functionality of rerunning a pipeline.
I tested this in my enviornment with a Get Metadata and a Filter activity that referenced the output of the Get Metadata. I set the secureOutput to true on the Get Metadata and attempted to rerun and it failed because of the same error message that you mentioned.
From what I have read, you would lose the ability to do accomplish this if you are referencing activities upstream in your pipeline.
The end result would be to either run the whole pipeline again as mentioned in the error due to the security of not storing the outputs.
How to check the rerun pipeline I can't locate after rerunning from failed activity
There should be two pipeline runs on the original failed run that has the failed pipeline and the one that succeeded. There may be a carrot or arrow icon that you need to click to view it.
How do we automate this?
That's a great question! There is some awesome conditional logic you can bring into a pipeline to only run activities upon the success or failure of other activities using precedent constraints. You can also use the expression builder in pipelines and utilize the outputs of activities that are further upstream in a pipeline. Those are two ways that you can automate your ETL and run large loads in one batch sequentially.
For this specific instance though there was an error that needed to be corrected in order to rerun from the failed activity. Once a pipeline run fails you need to manually correct that issue before it can be rerun. Therefore, there would not be a great way to automate this specific process using that feature from the Monitor Hub.