Add support for explicit batch for all models #1522
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@dusty-nv Can you do a review of these changes? Is there a better way to handle this scenario? |
|
Hi @ligaz, thank you for the PR - yes, I believe this looks good but have not had a chance to test it yet. I have added it to my TODO queue however to be merged/tested into the |
|
@dusty-nv Do you want me to base this change on top of the |
|
@ligaz it's not that big of a code change so I would probably just apply it to |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently only the ONNX models support the explicit batch strategy. I have my own TRT model that has the
NCHWinput size and I need to be able to use it.The fix is to remove the hardcoding of the ONNX models and check the input dimensions size. If the number of dimensions are 4 (
NCHW) then we are in explicit batch mode.Additionally the initialization of
mModelTypeandmModelPathfields for Engine file types is done before loading the engine. Those fields are potentially used in the loading itself (they were used when we were checking the ONNX model type before the fix).